Pre-Trained Language Models and Completion
A pre-trained model is likely optimized for completion, so it may extend a request instead of answering it.
When Good Completion Is Not Good Conversation
Imagine you type a request into a language model and it simply keeps writing. It adds more words that seem to belong after your text, but it never actually answers you. This is not a bug in the usual sense. It follows from what the model was built to do. A pre-trained model has learned to continue text, and when you give it a request, it treats your request as the beginning of a passage to extend.
What do you think happens?
You type 'What is a pre-trained model?' into a pre-trained model that has only been trained to continue text. What do you think it will most likely do?
Reveal answer
It may extend the text with more words that seem to belong after your request, rather than answering it.
The model was optimized for completion. It has no built-in notion that your text is a message from a person who wants something, so the request and the opening of a document look alike to it.
The rest of this article explains why this happens and what can change it. The key idea is supervised finetuning, the step that adjusts a pre-trained model so that it behaves like a helpful respondent in a conversation.
Why a Request Looks Like a Document Opening
A pre-trained model is likely optimized for completion rather than conversation. It has no built-in notion that the text in front of it is a message from a person who wants something. From the model's point of view, a request and a document opening look alike: both are text that can be extended.
Because both look alike, many different continuations are reasonable. A model could continue a question with another question, a list of related topics, or a paragraph of background. Only one of these continuations is useful to the person who asked. Completion quality alone does not tell you which one the model will choose.
Read the diagram as a single input with two possible outcomes. The input does not change between the two rows. What changes is the behavior the model has learned to favor, and supervised finetuning is the step that changes it.
Demonstration Data as Prompt-Response Pairs
The main ingredient of supervised finetuning is demonstration data. Each item in demonstration data is a pair. The first part is a prompt, which is the request a user might send. The second part is a response, which is the answer you want the model to give. Written as a pair, the format is (prompt, response).
Showing the model many such pairs is how you encourage it to generate appropriate responses. The word supervised describes this: the training uses examples that show the correct output for each input.
The demonstration data does not add a new rule written in English. It shows the behavior by example. The model tends to reproduce patterns found in its training data, so if the examples consistently pair a request with a direct, helpful answer, the model becomes more likely to answer requests directly.
Behavior Cloning and the Finetuning Pipeline
The sections below separate two ideas that work together. Behavior cloning describes the strategy. Supervised finetuning describes the training step that carries it out.
Following One Support Team Through the Process
It helps to follow one request through the whole process. Picture a support team that wants a model to answer questions about a software product. The steps below trace what the team does and what the model learns.
Building a product-support model
A support team wants a pre-trained model to answer customer questions about its software product, rather than extending the questions with more text.
- 1Gather demonstration pairs
The team collects pairs in which a customer's question is paired with a clear, helpful answer.
- 2Start from a pre-trained model
The team does not build a model from scratch. It starts from a model that already knows a great deal about language.
- 3Run supervised finetuning
The team trains the pre-trained model on the pairs, so the training shows the correct output for each question.
- 4Test a question of the same kind
For a new question similar to the examples, the model's output tends to read as an answer rather than as text that merely continues the question.
The finetuned model favors answering over extending for questions like the ones in the demonstrations. Its general language knowledge is kept, and its tendencies are reshaped.
Judging Whether the Examples Cover Enough Ground
Different types of requests call for different types of responses. Question answering, summarization, and translation each ask for a different kind of output. This means demonstration data should cover the range of requests the model is meant to handle.
A set of demonstrations made only of questions and answers would teach the model how to answer questions. It would give little guidance for a request to condense a long passage or to convert a sentence into another language. The model would have seen no examples of those behaviors, so there is no pattern for it to copy.
Coverage is a design decision. Source material on finetuning InstructGPT, a model from OpenAI, includes a distribution graph of the prompts used, based on numbers reported in OpenAI's paper. The useful lesson is not the exact proportions. The useful lesson is that the mix of prompts was deliberately chosen. The same source notes that the distribution contains no multimodal tasks, because InstructGPT is a text-only model. The mix you choose should match the modalities the finished model needs to handle.
| Request type | Example demonstration | Covered in a question-answer-only set? |
|---|---|---|
| Question answering | A question paired with a direct answer | Yes |
| Summarization | A long passage paired with a condensed version | No |
| Translation | A sentence paired with its rendering in another language | No |
A coverage check of a hypothetical demonstration set built only from question-answer pairs
Mistakes When Choosing Demonstrations
Common mistake Assuming a model that completes text well will also answer well
What it looks likeA pre-trained model extends a question with more words instead of answering it.
Why it failsThe model was optimized for completion. It has no built-in notion that the text is a request from a person.
Do this insteadUse supervised finetuning with demonstration pairs that show direct answers.
Common mistake Treating the demonstration pairs as a list of rules to explain
What it looks likeWriting a sentence that says the model should always answer directly, and expecting that alone to change behavior.
Why it failsThe demonstration data shows the behavior by example. The model reproduces patterns in its training data.
Do this insteadSupply many pairs that show the desired behavior across the request types the model must handle.
Common mistake Filling the set with one request type
What it looks likeA demonstration set made only of question-answer pairs, used for a model that must also summarize.
Why it failsThe model has no examples of summarization to copy, so it gives little guidance for that request type.
Do this insteadList the request types the model must handle and check that each has examples.
Common mistake Including request types the finished model will never receive
What it looks likeAdding image-based tasks to the demonstrations for a text-only model.
Why it failsThe mix of prompts should match the modalities the finished model needs to handle.
Do this insteadMatch the demonstration mix to the inputs the deployed model must accept.
Common mistake Discarding the pre-trained model and training from scratch when a narrower goal is needed
Checking Coverage in a Demonstration Set
A team building a text-only assistant has written six demonstration pairs. Four are questions paired with direct answers. Two are requests to condense a passage paired with shorter versions. No pairs show translation. Judge whether this set covers a useful range of request types, and name any request type that is missing or underrepresented.
Hints
- List the request types the assistant must handle, then compare them with the types present in the pairs.
- Count how many examples each type has. A type with no examples is missing; a type with very few is underrepresented.
- Consider whether all of the request types are text-based, since the assistant is text-only.
Key Takeaways
- A pre-trained model is optimized for completion, so it may extend a request instead of answering it, because it has no built-in notion that the text is a message from a person.
- Demonstration data is made of (prompt, response) pairs, and showing many such pairs teaches the model to favor appropriate responses by example rather than by written rules.
- Behavior cloning is the strategy of demonstrating desired behavior so the model copies it, and supervised finetuning is the training step that applies this to a pre-trained model.
- Demonstration data should cover the range of request types the model must handle, such as question answering, summarization, and translation, and its mix should match the modalities the finished model needs.