Foundation Model Training Stages
In-context learning teaches a model a behavior through examples or instructions placed in the prompt, without updating the model's weights.
Teaching a Model Without Retraining
Suppose you want a language model to do a task it was never specifically built for, such as sorting customer messages into categories it has never seen labeled. There are two broad routes. One option is to gather labeled data, adjust the model, and train it again. The other option is to place a few worked examples directly in the text you send to the model and ask it to continue the pattern. The second option is the one this article is about, and it is the core of in-context learning.
What do you think happens?
You place three labeled customer messages in your prompt, then ask the model to label a fourth. Do the model's weights change as a result?
Reveal answer
No, the weights stay the same and the examples only shape the response
In-context learning teaches a behavior through examples or instructions placed in the prompt, without updating the model's weights. The examples influence how the model responds to this one input, but the model itself is not modified.
Training Stages That Update Weights
To understand why the prompt-based approach stands out, start with the traditional route. A model learns the behavior it is expected to show during training. That training happens in several stages: pre-training, post-training, and finetuning. Each stage involves updating the model's weights, which are the numerical parameters that determine how the model transforms an input into an output.
- Pre-training: the first stage in which the model's weights are learned
- Post-training: a later stage that also updates the weights
- Finetuning: a stage that adjusts the weights further toward a particular behavior
- Once training is complete, the weights are fixed for ordinary use, so a change in behavior normally means another round of training
The key point is that every stage in this traditional route touches the weights. If you want the model to behave differently, the usual remedy is to change the weights through more training, which takes time and requires gathered data.
Tracing the Model State During a Prompt
Now follow what happens when you use in-context learning. Both the examples and the new query arrive as text the model reads in one pass. The model is not modified along the way. Whatever it learns from the examples exists only in how it responds to this particular input. The diagram below compares the model's state before and after it picks up a behavior from examples in the prompt.
Read the diagram carefully. The weights in the two states are the same. What changes is the response, because the examples in the prompt steer it for that input. This is why in-context learning is described as learning without a weight update.
The GPT-3 paper made a striking point. The model was trained for next token prediction, meaning its basic training objective was to guess the next piece of text. Even so, the authors showed that it could pick up from the context what to do in tasks like translation, reading comprehension, simple math, and answering SAT questions. Importantly, this could happen even when the behavior requested differed from what the model had originally been trained to do.
Zero-Shot and Few-Shot Prompts
In-context learning comes in two common forms. The difference between them is simply how many examples appear in the prompt. Zero-shot learning provides no examples, while few-shot learning provides a small number of examples. Both are forms of in-context learning.
Answering Beyond the Training Cut-Off
A model's knowledge is fixed at a certain point in time. Consider a model trained on older JavaScript documentation. If you ask it about a newer JavaScript version, you would normally have to retrain it so it absorbs the changes. With in-context learning, you can instead include the new JavaScript changes in the model's context. The model can then answer queries that fall beyond its training cut-off date.
Answering About a Newer Language Version
A model was trained on documentation that predates a newer JavaScript version. You want it to answer a question about a change that appears only in the newer version.
- 1Identify the gap
The model's knowledge is fixed at its training cut-off, so the newer change is missing from what it absorbed during training.
- 2Add the new information to the context
Place the description of the new JavaScript change in the text sent to the model, together with the question.
- 3Ask the question
The model reads the context and the query in one pass and responds using the information it was given at the moment of use.
- 4Confirm no retraining occurred
No weights were changed. The answer depends on the information supplied in this input.
The model answers a query that falls beyond its training cut-off date, without any retraining.
Why This Counts as Continual Learning
The source describes in-context learning as a form of continual learning. Continual learning is the ability of a model to keep learning new information over time. In-context learning supplies new information at the moment of use, which keeps the model from becoming outdated without a fresh round of training.
Common Mistakes With In-Context Learning
Common mistake Believing that giving examples retrains the model
What it looks likeAdding three labeled messages to a prompt and assuming the model's weights were updated afterward.
Why it failsIn-context learning teaches a behavior through examples placed in the prompt, without updating the model's weights.
Do this insteadTreat the examples as part of the input. They shape the response to that input, and the weights remain the same.
Common mistake Assuming few-shot learning is a different kind of learning from zero-shot learning
What it looks likeThinking zero-shot and few-shot are unrelated techniques.
Why it failsBoth are forms of in-context learning. The only difference is the number of examples supplied.
Do this insteadCompare them by counting examples: zero examples for zero-shot, a small number for few-shot.
Common mistake Expecting the model to remember a prompt's examples in later, separate inputs
What it looks likeSupplying a new JavaScript change once and assuming the model now knows it for every future question.
Why it failsWhatever the model learns from examples exists only in how it responds to that particular input.
Do this insteadInclude the information in the context of each input that needs it.
Common mistake Forgetting that the new query must come after the examples in the prompt
Practice Sorting Customer Messages
Write a few-shot prompt that teaches a model to label customer messages as Refund or Technical Issue. Include two worked examples and one new query at the end with the label left blank. Then state whether the weights of the model change after the prompt is sent.
Hints
- Start with a single sentence that states the task.
- Place each example as a message followed by its label.
- Put the new query last, with the label left blank for the model to complete.
- Remember that the answer to the weights question is about the model, not the prompt text.
Key Takeaways
- In-context learning teaches a model a behavior through examples or instructions placed in the prompt, without updating the model's weights.
- Traditional learning updates weights through pre-training, post-training, and finetuning, after which the weights are fixed for ordinary use.
- Zero-shot prompts supply no examples and few-shot prompts supply a small number; both are forms of in-context learning.
- Because new information can be placed in the context at the moment of use, a model can respond to queries beyond its cut-off date, which is why in-context learning is considered a form of continual learning.