Data Cleaning and Deduplication for Training
Training data must be in the format the target model expects, and that format includes the tokenizer and the chat template. Two datasets with identical content can still be unusable for a given model if they are laid out differently.
The Shortcut That Feels Reasonable
Suppose you have already spent time getting a base model to do a task by writing a careful prompt. The prompt contains a description of the job and a few worked examples, and the model now performs reasonably well. Once you decide to finetune, the natural next step seems to be reusing everything you already wrote. This article looks at why that shortcut often leads to trouble, and what a sound way of preparing finetuning data looks like instead.
What do you think happens?
Should the instructions that worked for prompt engineering also be the instructions used for finetuning?
Reveal answer
No, the instructions usually need to change for finetuning
The belief that prompt-engineering instructions should be reused for finetuning feels reasonable because the instructions describe the task in both cases. The source says this belief causes most of the problems, because finetuning instructions typically do not need task descriptions or examples.
Tracing the Preparation Sequence
Preparing data for finetuning is a sequence of steps, and formatting is the last one. The source describes the broader process as covering deduplication, cleaning, and formatting. Deduplication removes duplicate records, and cleaning improves the content. Only after those two stages do the remaining examples get arranged into the exact shape the model expects. Once that is done, the data is ready to be used for training.
Notice where the model-specific requirements enter the process. Formatting is the stage where they apply, and the source says formatting is model-specific. Cleaning and deduplication still matter, but the final shape of the data depends on the model you plan to finetune.
Why the Model Sets the Format
Each model uses a specific tokenizer and expects data in a specific chat template. The tokenizer determines how text is split into the units the model reads. The chat template determines how the data is laid out around those units. The source ties these two facts together, and both belong to the model you are finetuning, not to the task you are trying to teach.
Training data must be in the format the target model expects, and that format includes the tokenizer and the chat template. Two datasets with identical content can still be unusable for a given model if they are laid out differently.
Anatomy of One Training Example
If you are doing supervised finetuning, the source says your data most likely takes the form of an instruction paired with a response. So a single training example has two main parts. The instruction part may itself have two components: a system prompt and a user prompt. The chat template organizes these parts, and the tokenizer then turns the formatted text into the units the model reads.
Keep this structure in mind. The chat template is what lays these parts out in the order and form the model expects, so a training example is not complete until it has been formatted for the specific model.
Prompt Instructions Versus Finetuning Instructions
The key contrast is between the instruction you wrote for prompting and the instruction you need for finetuning. A prompt has to carry everything the model needs at inference time, so it often includes a task description and demonstrations. A finetuned model has been trained on many examples, so the training data itself can teach the behavior. That is why finetuning instructions typically do not need task descriptions or examples, and why the instructions may be different from the ones used during prompt engineering.
| Aspect | Prompt engineering | Finetuning |
|---|---|---|
| Where the teaching happens | Inside the prompt, through the description and examples | In the training data, through many examples |
| Task description | Typically needed | Typically not needed |
| Demonstrations | Typically included in the prompt | Become training examples |
| Role of the instruction | Must carry everything needed at inference time | Only needs to frame each training example |
Typical differences between what a prompt needs and what finetuning needs
Read the comparison carefully. The left column is what a prompt typically needs. The right column is what finetuning typically needs. The contrast is about where the teaching happens, not about whether the task matters.
Turning a Few-Shot Prompt into Records
The source gives a concrete scenario. Imagine you have been using a three-shot instruction for a food classification task with a base model. Each of the three demonstrations shows an input and its correct label, and the instruction also describes the task. The source states that for finetuning, all the examples included in the three-shot prompt can be converted into training examples. The walkthrough below uses an illustrative set of demonstrations of this kind.
Converting a three-shot food prompt into training records
A three-shot prompt for food classification contains an instruction describing the task and three demonstrations: apple with the label fruit, carrot with the label vegetable, and rice with the label grain. Convert the demonstrations into finetuning training examples.
- 1Identify the demonstrations
Each demonstration pairs an input with its correct label. Here the inputs are apple, carrot, and rice, and the labels are fruit, vegetable, and grain.
- 2Convert each demonstration into one record
Each demonstration becomes its own training example. The input goes into the instruction part and the correct label goes into the response part.
- 3Decide what to keep from the task description
The prompt's task description was written to guide a base model at inference time. Finetuning instructions typically do not need a task description, so the training records can use a simpler instruction.
- 4Format the records for the target model
Apply the chat template and tokenizer of the model being finetuned so that each record has the layout that model expects.
Three separate training records, one per demonstration, each with an input in the instruction part and the correct label as the response, formatted for the target model.
Notice what changed. Before conversion, the examples were embedded in one block of prompt text. After conversion, each example is its own record. The teaching no longer depends on the examples sitting inside the instruction, because each record now carries its own input and correct output.
Mistakes When Preparing Finetuning Data
Common mistake Copying the prompt-engineering instructions unchanged into the training data
Why it failsThe source notes that prompt-engineering instructions should not be copied unchanged, because finetuning instructions typically do not need task descriptions or examples.
Do this insteadRewrite the instruction to fit the training record, and let the training data carry the examples.
Common mistake Formatting the data before deduplicating and cleaning it
Why it failsFormatting is the final stage after deduplication and cleaning, so formatting first would lay out records that may still be duplicated or messy.
Do this insteadRemove duplicates and clean the content first, then apply the model-specific formatting.
Common mistake Using a chat template or tokenizer from a different model
Why it failsEach model uses a specific tokenizer and expects a specific chat template, so the data layout would not match the model being finetuned.
Do this insteadUse the tokenizer and chat template that belong to the model you are finetuning.
Common mistake Treating the examples in a prompt as a single unit instead of separate records
Practice Tracing a Prompt Conversion
A four-shot prompt for sentiment labeling contains an instruction describing the task and four demonstrations, each pairing a sentence with a label of positive or negative. Describe how many training records you would create, what goes into the instruction part and the response part of each record, and what you would change about the original task description before formatting.
Hints
- Each demonstration in the prompt becomes its own training example.
- Finetuning instructions typically do not need task descriptions or examples.
- Formatting happens last, after deduplication and cleaning.
Key Takeaways
- Training data for finetuning must be in the format the target model expects, including the tokenizer and chat template, which are specific to that model.
- Data preparation runs in stages: deduplication, then cleaning, then formatting as the final step.
- Prompt-engineering instructions should not be copied unchanged, because finetuning instructions typically do not need task descriptions or examples.
- Each example in a few-shot prompt can be converted into its own training example for finetuning.