Contextualized Output Embeddings
Because LLM input embeddings are learned as part of training, they are not fixed ahead of time. They are changed along with the rest of the model, so they can be shaped to the specific task and data.
Where Embeddings Come From
Earlier in this module you met pretrained embedding models such as Word2Vec. These models turn words into numeric vectors, and those vectors can then be fed into other machine learning models. That approach works, but it raises a practical question: where should the embeddings come from? Should they be produced by a separate model that was trained ahead of time, or can the language model itself create the embeddings it needs? This article answers that question for LLMs. The short version is that LLMs commonly produce their own embeddings, and those embeddings are part of the input layer and are updated during training.
Following a Token Into the Input Layer
Before a model can work with text, each token has to be connected to a numeric representation. In the input layer, each token is mapped to its embedding. Think of the embeddings as a learned table: each token has a slot in that table, and the token's ID tells the model which slot to read. The diagram below traces that lookup for two tokens. The IDs shown are illustrative only, since the source does not give specific ID numbers.
Tracing one token through the input layer
A model receives the sentence 'the cat sat'. Trace what happens to the token 'cat' at the input layer, using the illustrative IDs from the diagram.
- 1Identify the token
The token 'cat' has the illustrative ID 812.
- 2Find its slot
The ID tells the model which row of the embedding table to read, so row 812 is the slot for 'cat'.
- 3Pass the embedding forward
The embedding in that row is the input representation of 'cat'. It flows into the rest of the model, as the source describes.
The token 'cat' enters the model as the vector stored in its row of the learned embedding table, and that vector then takes part in the model's computations.
Embeddings That Keep Learning
The key idea is that the embedding layer is part of the model being trained. When training begins, the embeddings participate in the model's computations. As training proceeds, the embeddings are changed along with the rest of the model, so they gradually take on values that help the model perform its task. Nothing in this loop depends on a separate embedding model being trained first. The model learns its own embeddings as it learns everything else.
The diagram above shows the change in words. Each token's embedding starts with initial values. During training, an update step adjusts the model, and this includes the embedding layer itself. The embeddings therefore move from their starting values toward values that suit the task and data at hand. The source's main point is that this makes them optimized to the specific task and data, which is the reason training-time embeddings can be a good fit for a particular problem.
Because LLM input embeddings are learned as part of training, they are not fixed ahead of time. They are changed along with the rest of the model, so they can be shaped to the specific task and data.
Two Sources of Embeddings
The source contrasts two ways of getting embeddings for a machine learning model. Pretrained models like Word2Vec can generate embeddings that are then used by the model. LLMs, by contrast, commonly learn their own embeddings inside the input layer during training. The comparison below lays out the differences the source identifies.
| Aspect | Pretrained embedding model (Word2Vec) | LLM input layer embeddings |
|---|---|---|
| Where the embeddings come from | A separate model that generates them | The language model itself, during training |
| How they are used | Fed into other machine learning models | Participate in the model's computations from the start of training |
| Changed by training? | Not described as part of the LLM's training loop in the source | Yes, updated along with the rest of the model |
| Main advantage the source names | Works as a pretrained source of vectors | Task fit: optimized to the specific task and data at hand |
Two ways to obtain embeddings, as the source describes them
Mistakes Beginners Make With Embedding Origins
Common mistake Thinking an LLM's input embeddings must come from a separate model trained first
Why it failsThe source says the embedding layer is part of the model being trained, and that nothing in the training loop depends on a separate embedding model being trained first.
Do this insteadTreat the input embeddings as learned by the LLM itself during training.
Common mistake Assuming input embeddings stay frozen at their starting values
Why it failsAs training proceeds, the embeddings are changed along with the rest of the model.
Do this insteadThink of the embedding values as moving over the course of training.
Common mistake Confusing input embeddings with contextualized output embeddings
Why it failsThe source says the embeddings described in this article sit in the input layer and are updated during training, while contextualized output embeddings are a separate capability mentioned alongside them.
Do this insteadKeep the two apart: input-layer embeddings are one thing, and contextualized output embeddings are a related but distinct capability.
Common mistake Treating 'pretrained' and 'learned during LLM training' as the same origin, when the source presents them as two distinct approaches
Predict Before You Reveal
What do you think happens?
Does a separate embedding model have to be trained before an LLM can learn its input embeddings?
Reveal answer
No, the LLM learns its input embeddings as part of its own training.
The source says the embedding layer is part of the model being trained, and that the embeddings change along with the rest of the model during training. Nothing in that loop depends on a separate embedding model being trained first.
Trace the update loop in your own words. Starting from a token at the input layer, describe in order: how the token reaches its embedding, how that embedding flows through the model, what the update step changes, and why the embedding values end up better suited to the task than their starting values.
Hints
- Start with the token ID and the learned embedding table.
- Then follow the embedding into the rest of the model.
- Finish with the update step, and remember that it adjusts the embedding layer too.
A Related Capability: Contextualized Outputs
The source adds a related point: LLMs can also create contextualized output embeddings. The source notes that this topic is discussed in a later chapter, so this article only introduces the phrase. For now, the important distinction is this. The embeddings described in this article sit in the input layer and are updated during training. Contextualized output embeddings are a separate capability that the source mentions alongside them. The two ideas sit in the same model, but they are not the same thing.
Summary of Embedding Origins
At a glance
- LLMs commonly produce their own embeddings, which are part of the input layer and are updated during training.
- Each token is looked up in a learned embedding table, and that embedding then flows into the rest of the model.
- Because embeddings are learned during training, they can be optimized to the specific task and data at hand, unlike embeddings produced ahead of time by a model such as Word2Vec.
- Contextualized output embeddings are a related but distinct LLM capability, which the source defers to a later chapter.