Evaluating Models on Reasoning Tasks

Suppose you want a language model to reason through a problem before it commits to an answer. The simplest approach is to ask it to show its work in the prompt. That only works if the model already knows how to show its work, so a more basic question sits behind the prompt: where does that skill come from? This section approaches chain-of-thought reasoning from the training data side. The question to keep in mind is what the example responses in a finetuning dataset must look like if a model is going to learn to think in steps.

The word chain is doing real work. In a chain-of-thought response, each step is built on the step before it. The final answer is therefore supported by a sequence of smaller claims rather than appearing from nowhere. A response of this kind states a fact, performs an operation on that fact, and then uses the result of that operation as the input to the next operation. Because every link is visible, a reader can check each one in order.

The diagram above uses a made-up problem, not one from the source, to show the pattern. The fact at the start is used once, the first operation produces a number, and that number is the input to the second operation. If you removed the intermediate line 12, the final answer 10 would no longer be visibly connected to the facts it depends on.

Consider two training examples a dataset builder might write for a question about nitrogen. In the first, the instruction asks for the boiling point of nitrogen, and the response is a single value, -320.4F. The response contains no reasoning at all. In the second, the instruction explicitly asks the model to reason step by step, and the response walks through a problem one stage at a time before arriving at the total. Both examples come from the same research source and illustrate the two response styles side by side.

The contrast is not only about length. The bare answer gives the reader a result to accept, while the chain-of-thought response gives the reader a sequence of claims to check. The second style is what the course means by a visible chain supporting the final answer.

The source makes this point directly: to teach a model to generate step-by-step responses, its training data should include chain-of-thought responses. The skill is not reliably supplied by the prompt alone. A prompt can ask for steps, but if the examples the model learned from never showed reasoning, the request has no matching pattern to draw on.

The evidence cited in the source supports this. In work by Chun et al. (2024), adding step-by-step responses to finetuning data improved performance on chain-of-thought tasks for models of several different sizes. For some tasks, accuracy nearly doubled. The improvement is attributed to the data, which is why the construction of chain-of-thought examples matters for anyone building a training set.

Notice what would go wrong if the middle step were missing. A response that jumped from the facts directly to 10 would still be correct, but it would give the reader nothing to check. A model trained on responses like the one above learns to produce intermediate results in the same order, which is exactly the behavior the source describes.

If chain-of-thought data helps so much, why not simply collect plenty of it? The source identifies the bottleneck: generating multi-step responses is slow and laborious. Explaining how to solve a problem step by step is far more demanding than writing down only the final answer. A bare answer such as 9 takes seconds to record, while a step-by-step response requires the writer to state each intermediate value correctly and in order. Because of this extra effort, chain-of-thought datasets are less common than other instruction datasets.