Continual Learning in Language Models
In-context learning teaches a model a behavior through examples or instructions placed in the prompt, without updating the model's weights.
A Model That Sorts Messages It Was Never Taught
Imagine you want a language model to do a task it was never specifically built for. For example, you want it to sort customer messages into categories that it has never seen labeled before. There are two broad ways to make that happen. The first is to gather labeled data, adjust the model, and train it again. The second is to place a few worked examples directly in the text you send to the model and ask it to continue the pattern. The second option is the subject of this article, and it is the core of in-context learning.
What do you think happens?
When you give a model a few worked examples in the prompt and it follows the pattern, do you expect its weights to be changed by that request?
Reveal answer
No, the weights stay the same.
The source states that the model is not modified along the way. Whatever it learns from the examples exists only in how it responds to this particular input.
Two Routes to a New Behavior
To see why the prompt-based approach stands out, start with the traditional route. A model learns the behavior it is expected to show during training. That training happens in several stages: pre-training, post-training, and finetuning. Each stage involves updating the model's weights, which are the numerical parameters that determine how the model transforms an input into an output. Once training is complete, the weights are fixed for ordinary use, and a change in behavior normally means another round of training.
In-context learning takes a different route. It teaches a model a behavior through examples or instructions placed in the prompt, and it does this without updating the model's weights. The word context refers to the text the model reads at the moment it answers. Everything the model is shown in that text can shape its response, but the model itself is not changed by being shown it.
The source also points out a striking result from the GPT-3 work. That model was trained for next token prediction, so its basic training objective was to guess the next piece of text. Even so, the authors showed that it could pick up from the context what to do in tasks such as translation, reading comprehension, simple math, and answering SAT questions. This could happen even when the requested behavior differed from what the model had originally been trained to do.
Tracing the Model Across Two Requests
The clearest way to see in-context learning is to compare two requests sent to the same model. In the first request, the model receives only the new query. In the second request, the model receives a few examples followed by the same query. The weights are identical in both cases. What differs is the text the model reads, and therefore the response it produces.
Read the diagram from left to right within each group. The weights column is the same on both sides, so nothing about the model has been modified between the requests. The changed behavior lives in the second request's prompt, which is why the source says that whatever the model learns from the examples exists only in how it responds to this particular input. When the next request arrives without those examples, the behavior is gone.
Zero-Shot and Few-Shot Prompts
In-context learning comes in two common forms. The difference between them is simply how many examples appear in the prompt. Both forms are considered in-context learning, because in both forms the behavior is taught through the text rather than through weight updates.
Comparing the Two Prompt Styles
Sorting Customer Messages Two Ways
You want the model to label each customer message with one of several categories that it has never seen labeled. Write the prompt in the zero-shot style and then in the few-shot style, and identify how the number of examples changes what the model receives.
- 1Zero-shot prompt
Write a description of the task and the category names, then add the new message. No labeled examples appear anywhere in the text. For instance: Sort the following customer message into one of the categories Billing, Shipping, or Technical. Message: My package has not arrived after two weeks.
- 2Few-shot prompt
Add two or three labeled examples before the new message, such as Message: I was charged twice this month. Category: Billing. Then Message: The app crashes when I log in. Category: Technical. Then place the new message at the end with the Category field left empty for the model to complete.
- 3What changes
Only the text differs. The number of examples supplied goes from zero to a small number, and the model is shown the pattern it should continue. Neither prompt updates the model's weights.
The zero-shot prompt provides no examples, and the few-shot prompt provides a small number. Both rely on in-context learning because the behavior is conveyed through the prompt rather than through training.
| Feature | Zero-shot prompting | Few-shot prompting |
|---|---|---|
| Examples in the prompt | None | A small number |
| What the model sees before the query | Task description only | Worked examples that show the pattern |
| Form of in-context learning | Yes | Yes |
| Weights updated by the request | No | No |
| Difference between them | Number of examples supplied | Number of examples supplied |
Zero-shot and few-shot prompting differ in the number of examples, not in whether weights change.
Answering Beyond the Training Cut-Off
A model's knowledge is fixed at a certain point in time. Consider a model trained on older JavaScript documentation. If you ask it about a newer JavaScript version, you would normally have to retrain it so that it absorbs the changes. With in-context learning, you can instead include the new JavaScript changes in the model's context. The model can then answer queries that fall beyond its training cut-off date.
The diagram follows the information from its source to the answer. The new material never enters the weights. It enters through the prompt, and the model reads it alongside the query in the same pass. That is how a model whose training stopped at an earlier date can still respond to material that appeared later.
Why Prompt-Based Learning Counts as Continual Learning
This is why the source describes in-context learning as a form of continual learning. Continual learning is the ability of a model to keep learning new information over time. In-context learning supplies new information at the moment of use, which keeps the model from becoming outdated without a fresh round of training.
The map shows the contrast the source draws. Traditional training stages involve weight updates. In-context learning leaves the weights fixed and relies on the prompt, and because it allows new information to enter at the moment of use, it fits the description of continual learning. What is kept is the model itself. What changes from request to request is the context supplied with each query.
Mistakes to Avoid When Reading In-Context Learning
Common mistake Believing that in-context learning permanently teaches the model
What it looks likeAssuming that after one request with examples, the model will follow the same pattern in every later request without any examples.
Why it failsThe source states that whatever the model learns from the examples exists only in how it responds to this particular input.
Do this insteadInclude the examples in each request where you need the behavior, because the behavior lives in the prompt.
Common mistake Thinking that few-shot prompting is a different category from in-context learning
What it looks likeDescribing few-shot prompting as training the model on a handful of examples.
Why it failsThe source treats zero-shot and few-shot learning as two forms of in-context learning, and the only difference is the number of examples in the prompt.
Do this insteadDescribe few-shot prompting as in-context learning that supplies a small number of examples.
Common mistake Assuming that the weights are updated whenever a model adapts to a task
Why it failsTraditional learning updates weights during training stages, but in-context learning works without any weight update.
Do this insteadAsk whether the change came from training stages or from the prompt text to identify which mechanism is at work.
Common mistake Confusing a training cut-off date with the limit of what a prompt can supply
Practice: Predict Before You Check
A team wants a model to classify support tickets into categories it has never seen labeled. They try a prompt with no examples, and the answers are inconsistent. They then add three labeled tickets before the new ticket and the answers become consistent. Before reading the hints, decide which prompt style each attempt used, and explain whether the weights changed between the two attempts and where the improvement came from.
Hints
- Count the examples in each prompt. Zero examples means zero-shot; a small number means few-shot.
- Recall the source's point that the model is not modified along the way.
- The improvement came from the text the model read in the second request, not from a change to the model.
Key Takeaways
- In-context learning teaches a model a behavior through examples or instructions placed in the prompt, without updating the model's weights.
- Zero-shot prompting supplies no examples, while few-shot prompting supplies a small number; both are forms of in-context learning.
- Because new information placed in the context is read at the moment of use, a model can respond to queries beyond its training cut-off date.
- In-context learning is considered a form of continual learning because it keeps the model from becoming outdated without a fresh round of training.