Generation Capability

Suppose you want to build a coding assistant, or an application that translates Latin into English. It is tempting to start with the application logic: the user interface, the prompts, the way results are displayed. But before you write any of that, you need to know whether the model underneath can do the job at all. A beautifully built application cannot make a model understand a language it has never learned. This is the central idea of generation capability: a model's domain-specific capabilities are constrained by its configuration and training data, so evaluation has to check whether the needed capability is actually present.

A model's domain-specific capabilities are limited by two things. The first is its configuration, such as its architecture and size. The second is the data it was trained on. Configuration determines what kinds of abilities the model could in principle develop, while training data determines which abilities it actually picked up. Both factors set the ceiling on what the model can do in a given domain.

The consequence is blunt. If a model never saw Latin during training, it will not understand Latin, no matter how well the surrounding application is built. A model that lacks the capability your application needs will not work for you, and a good application wrapper does not change that.

The tool for gathering evidence is the domain-specific benchmark. Benchmarks can be public or private. Public benchmarks cover a very wide range of abilities, including code generation, code debugging, grade school math, science knowledge, common sense, reasoning, legal knowledge, tool use, and game playing. Choosing a benchmark that matches your target task is the first decision in any evaluation.

Multiple-choice questions are one of the most common ways to evaluate non-coding capabilities. A typical item presents a question followed by lettered options, and the model must select one. The MMLU benchmark has this form: it asks why governments regulate monopolies and offers four lettered options, and the model must choose one. Every item in this format comes with an answer key, which is the single correct letter. That is what makes scoring mechanical.

Each question produces exactly one result: the model's letter either matches the answer key or it does not. Accuracy is then the number of questions answered correctly divided by the total number of questions. It is the most common metric for multiple-choice benchmarks, and it is simple to compute.

The right evaluation method depends on the kind of output the task produces. For coding-related capabilities, the usual method is functional correctness, which checks whether the generated code does what it is supposed to do. Functional correctness is one kind of exact check applied to code. For domain-specific capabilities in general, the common approach is exact evaluation, which checks whether an output is exactly correct. Non-coding capabilities are often tested with close-ended tasks, and multiple-choice questions are the most common form, because their answers can be verified directly.

What decides the method is whether the output can be checked exactly. Code that runs can be tested for function, and its runtime and memory can be measured. Readability is different. There is no obvious exact procedure for judging whether code is easy to understand, so evaluators often fall back on subjective evaluation, such as AI judges.

Correctness is rarely the only concern. A car that runs but burns an excessive amount of fuel is a useful comparison: code can be correct and still be a poor fit for a real system. An SQL query from a text-to-SQL model may return the right rows yet take too long or need too much memory to be usable. The BIRD-SQL benchmark illustrates this. It combines the execution accuracy of a generated query with its efficiency, measured by comparing the generated query's runtime with the runtime of the ground-truth SQL query.

Readability matters for a related reason. Code that runs but that nobody can understand is hard to maintain and hard to incorporate into a larger system. So a complete picture of coding capability usually combines an exact check of correctness and efficiency with a judgment-based check of readability.

Close-ended testing has a deeper limit. Answering a multiple-choice question tests the ability to tell good responses from bad ones. That is a classification-style skill. It is not the same as the ability to produce a good response from scratch. Multiple-choice questions suit two kinds of evaluation well: knowledge, such as whether a model knows that Paris is the capital of France, and reasoning, such as whether it can infer from a table of business expenses which department spends the most.

Generation capabilities such as summarization, translation, and essay writing are a poor fit for multiple-choice tests. These tasks ask a model to produce text, and the quality of that text cannot be captured by picking a letter. Generation capability is the ability of a model to generate text or code, and it has to be evaluated with different metrics than multiple-choice questions use.