Refining Pre-Trained Models for Conversational AI
Demonstration data is a collection of (prompt, response) pairs that show a model what appropriate responses look like.
A Decision Before Reading On
Imagine a user asks an assistant a question about making pizza. You must decide which continuation would actually help that user. One option adds more detail about who the pizza is for. Another adds follow-up questions about ingredients and cooking time. A third gives the instructions directly. Before reading on, decide which option you think a helpful assistant should produce, and notice what made you choose it.
Pick one of the three options for the pizza question (more detail about who the pizza is for, follow-up questions about ingredients and cooking time, or direct instructions). Write down the reason for your choice in one or two sentences before continuing.
Hints
- Think about what the user actually asked, and whether the reply helps them with that request.
- Notice whether your choice depended on what you assumed about the user.
That small decision captures the problem demonstration data is designed to solve. A model learns patterns from what it is trained on, so if we want it to answer in a helpful way, we need to show it what helpful answers look like. The rest of this section and the ones that follow explain how that is done and what makes it harder than it first appears.
What One Demonstration Pair Contains
Demonstration data is a collection of (prompt, response) pairs that show a model what appropriate responses look like. Each pair has two parts: the prompt, which states what was asked, and the response, which shows the kind of answer that should follow.
A useful way to picture one pair is as a worked answer key entry. The prompt is the question on the sheet, and the response is the model answer written beside it. The entry does not explain the rules in general; it simply shows one concrete case of a good answer to a specific request.
The diagram shows that each prompt is matched to exactly one response in its own pair. The dataset is a list of such matched entries. During supervised finetuning, the model is exposed to many of these entries so that it can learn the relationship between the kinds of requests it receives and the kinds of replies that suit them.
Behavior Cloning Step by Step
The source describes a model as something that mimics its training data. Given that tendency, the practical strategy is to demonstrate the behavior you want and let the model copy it. This strategy is called behavior cloning: you demonstrate how the model should behave, and the model clones that behavior by learning from your examples.
Read the diagram as a sequence. Demonstration happens first, and the model's later behavior depends on what those examples contained. The user in the last step is not one of the training examples. The goal is generalization: the model should carry the demonstrated pattern over to requests that were never written down. That is why the dataset is a set of examples of behavior rather than a lookup table of answers.
Why One Task Is Not Enough
Different kinds of requests call for different kinds of responses, so a demonstration dataset cannot consist of a single task repeated many times. A question deserves a direct answer, a long article may need a summary, and a sentence in one language may need to be rendered in another. The source lists question answering, summarization, and translation as examples of the range a dataset should contain.
The source also points to a distribution of task types that OpenAI used when finetuning InstructGPT. That distribution is a useful reminder that task coverage is a deliberate design choice rather than something that happens automatically. It also shows a scope boundary: the distribution does not include multimodal tasks, because InstructGPT is a text-only model. A dataset should match the modalities the model is meant to handle. If a task type is missing, the model has no demonstrated pattern for it, so any behavior it shows on that task has to come from elsewhere.
Writing Demonstrations Versus Labeling
Ordinary data labeling often needs little or no domain expertise: a labeler can assign a category by following clear instructions. Demonstration data is different. Its prompts may be complex, and writing a good response can require critical thinking, gathering information, and judging whether a user's request is appropriate in the first place. The source gives InstructGPT's labeler-written pairs as the example of this work.
| Aspect | Ordinary data labeling | Demonstration data |
|---|---|---|
| Typical output | A category or label chosen from clear instructions | A complete response written for a specific prompt |
| Expertise needed | Little or none in many cases | Can require domain knowledge and careful thought |
| Work per item | Assign a label | Think, gather information, and write the answer |
| Extra judgment | Usually follows the instructions as given | Decide whether the request is appropriate to answer at all |
| Planning implication | Budget by the number of items | Budget for skill and time per pair |
Writing demonstration data compared with ordinary data labeling
The contrast matters for planning. If you budget demonstration data the way you would budget ordinary labeling, you will underestimate how much skill each pair requires. A response written with care may need research, and some requests need a decision about whether to answer at all. Those judgments are what the model is meant to absorb from the examples.
Mistakes Teams Make With Demonstration Data
The following mistakes follow directly from the ideas above. Each one treats the dataset as a pile of text rather than a set of examples of behavior.
Common mistake Treating each pair as just more text to collect
What it looks likeGathering many prompts and pasting in any reply that looks plausible
Why it failsEach pair is meant to show what an appropriate response looks like, so an unhelpful or inappropriate reply teaches the wrong pattern.
Do this insteadWrite or select each response so that it demonstrates the behavior you want the model to copy.
Common mistake Covering only one request type
What it looks likeA dataset made entirely of short factual questions
Why it failsDifferent request types call for different responses, so the model has no demonstrated pattern for the types that are absent.
Do this insteadInclude question answering, summarization, translation, and other task types the model is expected to handle, matching its modalities.
Common mistake Budgeting demonstration data like ordinary labeling
What it looks likeEstimating the same time per item for choosing a category and for writing a careful answer
Why it failsComplex prompts can require critical thinking, information gathering, and judgment about appropriateness.
Do this insteadPlan for research, drafting, and decisions about whether to respond, not just for the number of items.
Common mistake Expecting the model to learn only from the exact examples
What it looks likeWriting pairs and assuming the model will handle only those same prompts
Why it failsThe goal of behavior cloning is generalization to requests that were never written down.
Do this insteadUse a range of examples that show the underlying pattern, then test on new requests.
Judging a New Request
Return to the pizza question. Suppose your team has a dataset that contains only pairs where the prompt asks for a fact and the response gives a one-line answer. Trace what happens when a user asks for step-by-step pizza instructions.
Using the diagram of behavior cloning, explain in two or three sentences why a model trained only on one-line factual answers may struggle with a request for step-by-step instructions, and what kind of demonstration pair would help.
Hints
- Ask whether the step-by-step request appears anywhere in the demonstrations.
- Think about the kind of prompt and response that would show the model how to handle a multi-step request.
Key Takeaways
- A (prompt, response) demonstration pair shows a model what an appropriate answer looks like for a specific request, and supervised finetuning exposes the model to many such pairs.
- Behavior cloning means demonstrating the desired behavior so the model learns to imitate it, with the aim of generalizing to requests it has not seen.
- A useful dataset covers a range of request types such as question answering, summarization, and translation, and matches the modalities the model handles.
- Writing demonstration data is harder than ordinary labeling because complex prompts can require critical thinking, information gathering, and judgment about whether to answer.