Preparing High-Quality Finetuning Data
Finetuning is the natural remedy because it gives the model more examples of that specific form.
When Every Sentence Is True and the Output Still Fails
Suppose you ask a model for a document and it returns text in which every sentence is true, yet the document is still unusable. Or suppose you ask for code and the code will not compile. In both cases the model is not failing at knowing facts in the usual sense. Something about how it behaves is wrong. This kind of failure is called a behavioral issue, and this article is about recognizing it and deciding when finetuning is the right tool.
Behavioral issues show up in two main ways: as outputs that are accurate but irrelevant, or as outputs that do not follow the expected format. These are different symptoms, and they point toward different corrections. Learning to tell them apart is the first skill this article builds, because a bad output can only be fixed well once you know what kind of bad it is.
Two Ways a Model Output Can Go Wrong
The first behavioral problem is an output that is factually correct but irrelevant to the task. Every claim in the output may check out, but the output does not address what was asked. A true statement about the wrong product, or a correct explanation of a neighboring topic when the request was for something specific, is accurate and still useless. Nothing is false, so a fact check alone will not reveal the problem. The question to ask is whether the output is on task.
The second behavioral problem is an output that does not follow the expected format. Here the content may be sound, but its shape is wrong. A structured object with a missing bracket, an extra key, or a misnamed field has failed even when the underlying meaning was understood correctly. The question to ask is whether the output matches the required structure.
These two problems are not interchangeable, and that matters for the fix. Both are behavioral, and the guidance here is that finetuning may help with either one. Finetuning can address a behavioral problem by training on well-defined specifications of the desired behavior, or by exposing the model to more examples of the required format. In each case the correction comes from the examples the model is trained on, not from looking up new facts.
Why Semantic Parsing Depends on Output Form
Semantic parsing is a task in which a model turns a request written in everyday language into a structured representation that a program can act on. The structured representation has a required form, and whether it works depends on that form being exactly right. This is why semantic parsing is so sensitive to the second behavioral problem: a parse that understood the sentence perfectly can still fail if its syntax is off.
Strong off-the-shelf models are generally good at common, less complex syntaxes such as JSON, YAML, and regular expressions. They can struggle with syntaxes that have fewer examples available on the internet. Two cases are named: a domain-specific language for a less popular tool, and a complex syntax. In both, a general model has had less exposure to the exact form it must produce. Finetuning is the natural remedy because it gives the model more examples of that specific form.
Consider a generated example. A customer writes: cancel my order 1234. A semantic parser is asked to produce a JSON object with an intent key and an order_id key. A correct parse has an opening brace, the two keys with their values, and a closing brace. Each part has a job, as the diagram below shows token by token.
What do you think happens?
A parse has every key and every value correct, but its closing brace is missing. Is this a format problem or a facts problem?
Reveal answer
Format problem
The content is right and only the shape is broken. Shape problems belong to the form side of the distinction, which is the side finetuning addresses.
{"intent": "cancel_order", "order_id": 1234Near-miss parse: the closing brace is missing, so the object is never closed and a program cannot read it
{"intent": "cancel_order", "order_id": 1234}Well-formed parse: the object closes and the keys match the expected form
Notice what this example is testing. Nothing about the customer's request needs to be looked up. The question is whether the model reliably emits a well-formed object with the right keys. A parser that produces a near-miss, such as a missing bracket or an extra key, has failed even if it understood the sentence perfectly.
How Finetuning on Targeted Examples Changes Behavior
Finetuning changes a model's behavior through the examples it is trained on. For a format problem, the useful examples are ones that show the required form clearly and correctly. For a problem with an off-task output, the useful training data are examples written against a well-defined specification of what the model should do. In both cases, the curated examples are the lever. The model does not receive a new fact to repeat; it is shown the behavior it should produce more often.
The diagram is a simplified picture. What it shows is the direction of the change: the curated examples come first, the model is trained on them, and the output that changed is the one with the wrong shape. The same logic applies to the output that is accurate but off task, where the training examples should reflect the specification of the task itself.
Choosing Between RAG and Finetuning
The source summarizes the distinction in one line: RAG is for facts, and finetuning is for form. A RAG system supplies external knowledge so the model can build more accurate and informative answers, and it can help mitigate hallucinations. Finetuning instead helps the model understand and follow syntaxes and styles. The two tools aim at different failures, so the first question to ask about a bad output is what kind of bad it is.
| Question | RAG | Finetuning |
|---|---|---|
| Which failure does it target? | Facts that are missing or unreliable | Output shape or style that is wrong |
| What does it supply? | External knowledge the model can use when building an answer | Training examples that teach a syntax or style |
| Effect on hallucinations | Can help mitigate them | Can reduce them with enough high-quality data, and can worsen them with low-quality data |
| Typical symptom | Confident but outdated or unsupported claims | Near-miss structured output, missing brackets, extra keys |
Matching a model problem to the tool aimed at it
The decision path starts with one question: is the output wrong in what it says, or in how it is shaped? If the facts are missing or unreliable, the guidance points toward RAG. If the shape or style is wrong, it points toward finetuning. A worked example applies this path to a case where both kinds of problem appear.
Sorting a Model Problem That Has Two Parts
A support assistant answers a question about a product's current warranty length with a confident but outdated number. The same assistant also writes its answer as plain prose when the integration expects a JSON object with a warranty_months key. Which tool addresses which problem, and where should work begin?
- 1Separate the two symptoms
The wrong warranty length is a problem with what the output says. The prose instead of JSON is a problem with how the output is shaped. Treat them as two problems rather than one.
- 2Match each symptom to a tool
The unreliable warranty fact is a facts problem, which points toward RAG. The prose-instead-of-JSON output is a form problem, which points toward finetuning on examples of the required structure.
- 3Apply the ordering guidance
When both kinds of problem appear, the guidance is to start with RAG. Supplying the current warranty information comes first.
- 4Check the training data before finetuning
Because low-quality finetuning data can worsen hallucinations, the examples used for the format fix must be carefully prepared rather than assembled quickly.
Start with RAG to supply reliable warranty information, then finetune on high-quality examples of the required JSON form. Each tool handles the problem it was designed for.
When Both Problems Appear Together
A model can be both irrelevant and format-breaking at once. In that case the two problems do not have to be fixed in one step, and they should be addressed in order. The guidance is to start with RAG for the facts, because the form fix depends on the content being right in the first place. Only after the facts are reliable does finetuning on format examples make sense as the next step.
Mistakes When Diagnosing a Bad Output
Common mistake Treating a format problem as a facts problem
What it looks likeA parse is missing its closing brace, and the response is to add retrieved documents.
Why it failsRetrieved knowledge supplies facts. A missing bracket is a shape problem, and no new facts will close the object.
Do this insteadAsk first whether the output is wrong in what it says or in how it is shaped, then choose the tool that matches.
Common mistake Assuming that an accurate output is a good output
What it looks likeA true statement about the wrong product is judged acceptable because nothing in it is false.
Why it failsAn output can be factually correct and still irrelevant to the task, which is a separate behavioral problem.
Do this insteadCheck whether the output addresses the request, not only whether its claims are true.
Common mistake Assuming finetuning is always safe for factual accuracy
Why it failsFinetuning can reduce hallucinations with enough high-quality data, but it can make them worse when the data quality is low.
Do this insteadTreat the quality of the training examples as a central part of the decision.
Common mistake Assuming a general model handles every syntax equally well
Common mistake Ignoring how few examples a syntax has online
Why it failsStrong general models are generally good at common syntaxes such as JSON, YAML, and regex but can struggle with syntaxes that have fewer examples available.
Do this insteadFor a less popular domain-specific language or a complex syntax, expect that finetuning on targeted examples may be needed.
Classifying Failing Outputs
Three outputs were produced for the same request. Output A is a true statement about a different product than the one asked about. Output B is a JSON object that is missing its closing brace. Output C has the correct JSON structure, but it cites a retired policy as current. For each output, decide whether it shows an irrelevance problem, a format problem, or a facts problem, and say which tool you would reach for first.
Hints
- Start with the question from the decision path: is the output wrong in what it says, or in how it is shaped?
- Output A is accurate but off task, which is the first behavioral problem.
- Output C has the right shape, so its problem is in the facts it carries, which points toward RAG.
Key Takeaways
- Behavioral issues appear as accurate but irrelevant outputs, or as outputs that break the expected format.
- Semantic parsing depends on correct output form, so it often calls for finetuning, especially for syntaxes with few examples available online.
- RAG is for facts and finetuning is for form. With both kinds of problem, the guidance is to start with RAG, and low-quality finetuning data can worsen hallucinations.
- Sort a bad output first by asking whether it is wrong in what it says or in how it is shaped, and only then choose the tool.
At a glance
- Two behavioral problems are described: outputs that are accurate but irrelevant, and outputs that break the expected format.
- Finetuning on targeted examples, or on well-defined specifications, can address behavioral problems, and semantic parsing often depends on getting the output form exactly right.
- RAG is for facts and finetuning is for form; when both appear, start with RAG, and remember that low-quality finetuning data can worsen hallucinations.
- Diagnose a bad output by asking first whether it is wrong in what it says or in how it is shaped.