Task Decomposition and Chain-of-Thought Prompting

When you ask a language model a question, the most direct prompt often produces the most direct answer. Sometimes that is exactly what you want. For harder questions, though, you may want the model to slow down and work through the problem in a more systematic way. The source material describes two techniques for this purpose: chain-of-thought (CoT) prompting and self-critique prompting. This article focuses on CoT and introduces self-critique because the two are discussed together.

The phrase explicitly asking matters. CoT is not a hidden setting and it is not a special model. It is a choice you make in the wording of the prompt. The source describes CoT as among the first prompting techniques that work well across models, which is part of why it is widely taught.

The source gives two broad ways to write a CoT prompt. The simplest is to add a phrase such as think step by step or explain your decision to your prompt. In that case the model works out for itself which steps to take. The second way is more directed: you specify the steps the model should take, or you include examples of what those steps should look like inside the prompt.

The source shows four CoT variations written against the same original query: Which animal is faster: cats or dogs? Keep that single question fixed and watch what changes in what you send to the model. The question itself never changes. Only the surrounding prompt text changes, and each change is one of three kinds: an instruction to reason, a list of steps, or an example of the output shape.

Look at the rows in order. The first two variations are both zero-shot, even though their wording differs. Each asks for reasoning, but neither includes an example. The third variation is also zero-shot, but it goes further by supplying the steps the model should follow. The last variation is the only one that includes an example of the output shape. Because the source says the best variation depends on the application, read this set as a list of options to choose from rather than a ranking from worst to best.

CoT prompts are commonly grouped by how many examples they contain. The grouping comes directly from the source, and it is best understood as one question: does the prompt include an example of the reasoning you want? Zero-shot CoT includes no examples. One-shot CoT includes exactly one example in the prompt.

The source reports that CoT improves model performance on various benchmarks. The Wei et al. (2022) paper, which introduced CoT, showed improvements for models of different sizes, namely LaMDA, GPT-3, and PaLM. The source's figure for this result plots performance against model scale, measured in billions of parameters, across the MAWPS, SVAMP, and GSM-8K benchmarks. In words: for each model family, the CoT version of the prompt scored higher than the standard approach on these math word problem and related benchmarks.

The source also notes that LinkedIn found CoT reduces models' hallucinations. This is a reported finding, and the source does not describe the conditions in detail. Treat it as evidence that CoT can help, not as a guarantee for every prompt.

The source describes self-critique as similar to CoT in spirit: both nudge the model to think critically about a problem. CoT pushes the model to work through steps. Self-critique pushes the model to review what it produced. Used together, the model first reasons and answers, then examines that answer. This is a common way the two techniques are combined.

Read the flowchart as a sequence. The first answer is not the end of the process. The model examines that answer as a separate step, and the revised output is what the user sees at the end. Each stage must finish before the next can begin, so the work accumulates in order rather than happening all at once.

The source is direct about a cost of these techniques. Similar to prompt decomposition, CoT and self-critique can increase the latency users perceive. A model may perform several intermediate steps before the user sees the first output token. This is especially challenging when you encourage the model to come up with its own steps, because the resulting sequence can take a long time to finish. That increases latency and can make the approach costly.