Structured Output From Language Models
Autoregressive models generate output one token at a time, so a stopping condition is needed to decide when to end.
Why a Maximum Size Seems Like Enough
When a language model writes a response, it does not produce the whole answer in one step. A very common first idea for controlling how long that response gets is simple: if the output is too long, set a maximum size and stop once it is reached. The size limit feels like a complete solution because it guarantees the output will never run past a certain length. Keep that belief in mind, because the rest of this article tests it.
What do you think happens?
A model is told to stop once it reaches a fixed maximum size. What does the reader actually see at the end of the output?
Reveal answer
A response that may be cut off partway through a sentence.
A fixed token limit halts generation whenever the count is reached, regardless of where the text happens to be. The limit does not know whether a sentence, a phrase, or a required structure has been finished, so the output can stop in the middle.
Output Built One Token at a Time
Autoregressive generation is a loop. The model takes the tokens already present, predicts a next token, appends it to the sequence, and then repeats the process with the longer sequence. Each new token depends on the ones before it. This is what makes the output a sequence built one piece at a time. It is also why the length of the sequence is directly tied to how much work the model does: every additional token means another pass through the loop.
The loop has no built-in sense of when a response is finished. Without a stopping rule, the length of the output has to be controlled from outside the generation loop. The diagram below traces one pass through the loop and shows where the stop decision branches off.
The Stopping Condition as a Test
A stopping condition is a test placed at the check step of the generation loop. Each time a token is appended, the test asks whether the output should end. If the answer is no, the loop continues. If the answer is yes, generation halts and the output so far is returned.
Everything about a stopping condition comes down to one question: what does the test look at? The source material describes two main kinds of test. One counts how much has been written. The other watches for a particular signal in what has been written. The first is about how much has been written; the second is about what has been written. Most of the trade-offs between them follow from that difference.
When Early Stopping Breaks Structure
Now return to the belief from the start. A fixed token limit does not always produce a well-formatted output. Its failure is not only a truncated sentence. When the output must follow a structure, an early stop can leave that structure broken, and the damage may not be visible until someone tries to use the output. The source specifically warns that early stopping can produce malformatted outputs, such as JSON missing closing brackets.
{"name": "Ada", "city": "London"}Generation finished at the correct boundary, so the object is complete.
{"name": "Ada", "city": "LonA token limit stopped generation mid-value, so the closing quote and closing brace are missing.
The Cost, Latency, and Completeness Trade-off
Length matters for practical reasons. A long output takes more time to produce, uses more compute, and therefore costs more money. A long, rambling response can also frustrate the reader. A stopping condition is therefore a rule for when to halt, and choosing it well is part of getting useful output at a reasonable cost.
Stopping early pulls in one direction: it reduces latency and cost. Stopping late, or only when the output is truly finished, pulls in the other: it protects completeness and structure. The source explicitly names this tension, since early stopping saves time and money but can produce malformatted outputs. A good stopping rule keeps the output as short as the task allows without cutting off the parts that must be present.
Choosing a stopping rule for a JSON answer
A program asks a model to return a JSON object with three fields: name, city, and score. The program parses the output directly. Which halting mechanism protects the output best, and what trade-off remains?
- 1Identify what must be preserved
The output must be a complete JSON object, so the closing brace must be present. A count alone cannot guarantee that.
- 2Test a fixed token limit
If the limit is set too low, generation can stop before the third field, leaving the object missing its closing brackets. The parser then fails.
- 3Test a stop signal
A stop token tied to the end of the object halts generation at the natural boundary, so the structure is more likely to be complete.
- 4Weigh the cost
Allowing more tokens before the stop signal can increase time and compute, so the rule must balance completeness against latency and cost.
A stop signal tied to the end of the structure protects completeness better than a bare count, but it still takes more generation time than an aggressive early cutoff. The trade-off between cost and completeness remains and must be chosen deliberately.
Mistakes About Stopping Rules
Common mistake Assuming a size limit guarantees a finished response
What it looks likeA model is capped at a fixed count, and the reader sees a sentence that stops mid-word.
Why it failsA fixed token limit only counts. It does not check whether the text is complete or whether a required structure has been closed.
Do this insteadTreat a fixed limit as a cost control, and pair it with a signal or a structural check when the output must be complete.
Common mistake Believing the model knows when it is finished
What it looks likeA developer expects the model to end a response on its own after it has answered the question.
Why it failsThe generation loop has no built-in sense of when a response is finished, so the ending has to be controlled from outside the loop.
Do this insteadDefine an explicit stopping condition at the check step of the loop.
Common mistake Ignoring malformatted output because it looks almost correct
What it looks likeA JSON object is missing its closing brace, but the text reads fine by eye.
Why it failsEarly stopping can leave a structure broken, and the damage may not show until the output is used.
Do this insteadValidate the output's structure before relying on it, especially when it must be machine-readable.
Common mistake Treating a shorter output as always better, without checking whether required parts were removed
Choose a Stopping Rule
A team asks a model to produce a JSON list of product records for a database import. Some outputs arrive with missing closing brackets, and the import fails. The team wants to keep costs low. Which stopping mechanism should they examine first, and what trade-off must they accept? Explain your reasoning in terms of what the stopping test looks at.
Hints
- Ask whether the test counts how much has been written or watches for a signal in what has been written.
- Consider which mechanism is tied to the end of a complete structure.
- Remember that stopping earlier reduces cost and latency but can break structure.
Key Takeaways
- An autoregressive model produces output one token at a time, and it has no built-in sense of when to finish, so a stopping condition must be supplied from outside the generation loop.
- A stopping condition is a test at the check step of the loop. A fixed token limit counts how much has been written, while a stop token or stop word watches for a signal in what has been written.
- A fixed token limit is simple but can cut outputs off mid-sentence, and it can break a required structure such as JSON.
- Early stopping reduces latency and cost but can produce malformatted outputs, so the stopping rule must balance cost, latency, and completeness.
- Stopping rules are a design choice, and the right one depends on what the output has to look like.