Word2Vec and Pretrained Word Embeddings
An LLM's input embeddings are not a separate artifact that gets plugged in. They are a learned part of the model, and training adjusts them for the task in front of the model.
Where Embeddings Come From
Earlier, you met pretrained embedding models such as Word2Vec. These models turn words into numeric vectors, and those vectors can then be fed into other machine learning models. That approach works, but it leaves an open question: where should the embeddings come from? Should they be produced by a separate model that was trained ahead of time, or can the language model itself create the embeddings it needs? This article follows that question through one central idea: in an LLM, the embedding layer is part of the model being trained.
What do you think happens?
Two ways to get word vectors for a model: (A) use a table produced by a separate model trained ahead of time, or (B) let the model learn its own vectors during its own training. Which one can be tuned to the specific task and data at hand?
Reveal answer
B: the vectors learned inside the model during training
The main advantage identified for learning embeddings during LLM training is task fit. Because the embeddings are learned as part of training, they can be optimized to the specific task and data at hand. A pretrained table such as Word2Vec is produced ahead of time and is not shaped by the training of the main model in this way.
Tracing a Token Into the Input Layer
Before a language model can compute anything, each input token has to become a vector. The input layer handles this step. Think of the embedding layer as a lookup table: each token is identified by a number, and that number picks out one row of a table of learned vectors. The table is the embedding matrix, and the row chosen for a token is the embedding for that token. The diagram traces this lookup with a small, illustrative set of token IDs.
In the diagram, the token has ID 7, so the model reads row 7 and uses that vector as the token's representation going forward. The other rows are not used for this token. Keep this picture in mind: the rows are not fixed labels. They are numbers that the model holds and can change, which is the point of the next section.
Embeddings That Change During Training
The key idea is that the embedding layer is part of the model being trained. When training begins, the embeddings take part in the model's computations, just like the rest of the network. As training proceeds, the embeddings are changed along with the rest of the model, so they gradually take on values that help the model perform its task. The embeddings are not a finished product handed to the model; they are one of the things training adjusts.
Read the diagram as a loop. Input tokens are mapped to their embeddings in the input layer. Those embeddings flow into the rest of the model. During training, an update step adjusts the model, and this includes the embedding layer itself, so the embeddings change over time. Nothing in this loop depends on a separate embedding model being trained first. The embeddings and the model are learned together.
Learned Inputs Versus Pretrained Tables
The source contrasts two ways of getting embeddings for a machine learning model. Pretrained models such as Word2Vec can generate embeddings, and those embeddings are then used by the model. LLMs, by contrast, commonly learn their own embeddings inside the input layer during training. The distinction is about origin and adjustment: where the vectors come from, and whether they are shaped by the task the model is being trained to do.
| Question | Learned input embeddings in an LLM | Pretrained embedding model such as Word2Vec |
|---|---|---|
| Where are the embeddings produced? | Inside the input layer of the model being trained | By a separate model trained ahead of time |
| Are they updated during training? | Yes, they change along with the rest of the model | They are not updated as part of the main model's training |
| What is the main advantage named for this approach? | They can be optimized to the specific task and data at hand | They provide vectors that can be fed into other models |
Two sources of embeddings for a machine learning model
An LLM's input embeddings are not a separate artifact that gets plugged in. They are a learned part of the model, and training adjusts them for the task in front of the model.
Worked Example: Adapting Vectors to a Domain
Why task-tuned embeddings can fit a specific dataset
A team wants a model that classifies short customer messages for a software product. Some words in these messages carry product-specific meanings. Should the team rely on a general pretrained table, or let the model learn its input embeddings during its own training?
- 1Identify what the embeddings must support
The model needs vectors that help it perform this classification task on this particular kind of text. The goal is task fit.
- 2Consider the pretrained option
A pretrained table is produced by a separate model ahead of time. Its vectors can be used, but they were not shaped by this team's task and data.
- 3Consider the learned option
If the embedding layer is part of the model being trained, the embeddings take part in the computations and are changed along with the rest of the model. Over training, they take on values that help this task.
- 4Compare the outcomes
The learned option is the one the source identifies as optimized to the specific task and data at hand, so it is the better fit for this team's aim.
The team should let the model learn its own input embeddings during training, because those embeddings are adjusted to the specific task and data, which is the advantage described for training-time embeddings.
Contextualized Output Embeddings
The source adds a related point: LLMs can also create contextualized output embeddings. This is a related but distinct capability. This article only introduces the phrase, because the source notes that the topic is discussed in a later chapter. The distinction to hold onto now is this: the embeddings described in the rest of this article sit in the input layer and are updated during training, while contextualized output embeddings are a separate capability that the source mentions alongside them.
Common Mistakes With Embeddings
Common mistake Thinking the input embedding table is fixed once it is created
What it looks likeAssuming the vector for a token stays the same from the first training step to the last.
Why it failsThe embedding layer is part of the model being trained. As training proceeds, the embeddings are changed along with the rest of the model.
Do this insteadTreat input embeddings as learned parameters that gradually take on values that help the model perform its task.
Common mistake Assuming every LLM needs a separate embedding model trained first
What it looks likeBelieving a Word2Vec table must be built and then handed to the LLM before training can begin.
Why it failsThe source describes LLMs as commonly learning their own embeddings inside the input layer during training, and the loop in that case does not depend on a separate embedding model being trained first.
Do this insteadRecognize two options: pretrained embeddings produced ahead of time, or embeddings learned inside the model during training.
Common mistake Equating the advantage of learned embeddings with a guarantee of better results everywhere
What it looks likeClaiming that learned embeddings always beat a pretrained table in every setting.
Why it failsThe source identifies task fit as the main advantage of training-time embeddings. It does not claim that this advantage holds in every situation.
Do this insteadState the advantage precisely: the embeddings can be optimized to the specific task and data at hand.
Common mistake Confusing input embeddings with contextualized output embeddings, which are a separate capability
Check Your Understanding
Explain, in your own words, what happens to a token's embedding vector between the start and end of training when the embedding layer is part of the model being trained. Then say which property of training-time embeddings makes them a better fit for a specific task and dataset than a pretrained table such as Word2Vec.
Hints
- Look at the loop from input tokens through the embedding layer, the rest of the model, and the update step.
- The key word for task fit is that the embeddings are optimized to the specific task and data at hand.
Key Takeaways
- An LLM commonly produces its own embeddings as part of its input layer, and these embeddings are updated along with the rest of the model during training.
- Because training-time embeddings are learned as part of training, they can be optimized to the specific task and data at hand, which is their main advantage over a pretrained table such as Word2Vec.
- Pretrained embedding models such as Word2Vec are produced by a separate model ahead of time and then used by other models, which is a different arrangement from learned input embeddings.
- Contextualized output embeddings are a related but distinct capability of LLMs, introduced here only as a phrase and covered in a later chapter.