Multiple-Choice Question Evaluation

Imagine you want to build a coding assistant, or an application that translates Latin into English. Before you write any application logic, you need to know whether the model underneath can do the job at all. Abilities like coding or understanding Latin are domain-specific capabilities: abilities tied to a particular domain or task rather than to language use in general. Evaluation is the process of finding out whether a model has the capability your application needs, before you build on top of it.

A model's domain-specific capabilities are limited by two things. The first is its configuration, such as its architecture and size. The second is the data it was trained on. These two factors set the ceiling on what the model can do in a given domain. Configuration determines the structure and capacity of the model, while training data determines what the model has actually been exposed to. A capable structure with no relevant training exposure still will not produce the capability.

The practical consequence is blunt. If a model never saw Latin during training, it will not understand Latin, regardless of how carefully the surrounding application is built. A model that lacks the capability your application needs will not work for you. Evaluation therefore has to verify two things: that the configuration can support the task, and that the training data gave the model the relevant ability.

The tool for gathering this evidence is the domain-specific benchmark. Benchmarks can be public or private. Public benchmarks cover a very wide range of abilities, including code generation, code debugging, grade school math, science knowledge, common sense, reasoning, legal knowledge, tool use, and game playing. Choosing a benchmark that matches your target task is the first decision in any evaluation.

Once you know which capability you need, the next question is how to check it. The right evaluation approach depends on whether the task produces code or some other kind of output. The two families look quite different, and the comparison below summarizes what each one checks and how it judges a response.

For coding-related capabilities, the usual method is functional correctness, which checks whether the generated code does what it is supposed to do. Correctness alone is not the whole picture, and the rest of the coding evaluation is covered in the subsections below. For non-coding capabilities, evaluators often use close-ended tasks, and multiple-choice questions are the most common form. The reason is verification: close-ended outputs are easier to check and easier to reproduce.

A multiple-choice item from the MMLU benchmark shows the format. It presents a question about why governments regulate monopolies, followed by four lettered options, and the model must select one of them. Every item in this format has an answer key, which records the correct option. Because the key already exists, scoring is mechanical: the evaluator compares the model's chosen letter with the key and records whether they match.

Multiple-choice questions can have one correct answer or several. Some tasks use a point system instead, where harder questions are worth more points, or where a model earns one point for each correct option it selects when several options are correct. Classification is a special case in which every question has the same set of choices. A tweet sentiment task, for example, offers NEGATIVE, POSITIVE, and NEUTRAL for every item. For classification tasks, metrics beyond accuracy include F1 scores, precision, and recall.

Follow one question through the scoring process. The model is shown a question and its options. It selects one option. The evaluator then compares that selection with the answer key. If the two match, the question is counted as correct. If they do not match, it is counted as incorrect. Each question contributes one result, and the results from all questions are combined into a single score. The flowchart below traces this path for a single item and then shows how the individual results combine.

The most common metric for multiple-choice evaluation is accuracy, which counts how many questions the model gets right. Accuracy is simple to compute. It is the number of questions answered correctly divided by the total number of questions.

Multiple-choice benchmarks are popular because they are easy to create, verify, and evaluate. They have a drawback, though. A model's performance on them can change with small changes in how the question and options are presented. Alzahrani et al. (2024) found that adding an extra space between the question and the answer, or adding an instructional phrase such as 'Choices:', could cause the model to change its answers.

There is a deeper limit to close-ended testing. Answering multiple-choice questions tests the ability to tell good responses from bad ones, which is a classification-style skill. It is not the same as the ability to produce a good response from scratch. Multiple-choice questions suit two kinds of evaluation well: knowledge, such as whether a model knows that Paris is the capital of France, and reasoning, such as whether it can infer from a table of business expenses which department spends the most.

Multiple-choice questions are a poor fit for generation capabilities such as summarization, translation, and essay writing. Those tasks ask a model to produce text, and the quality of that text cannot be captured by picking a letter. Generation capability is the ability of a model to generate text or code, and it has to be evaluated with different metrics than multiple-choice questions use. The comparison below contrasts the two situations.