Tokenization and Vocabulary Construction
Special tokens such as <|unk|> and <|endoftext|> enhance a model's understanding and handle contexts like unknown words and boundaries between unrelated texts.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
Same Word, Different Places
A language model does not read text the way a person does. It never works directly with letters or whole words on the page. Instead, each token in the input is mapped to a token embedding, which is a vector representation for that token. The model then does its work on these vectors. This setup has a useful property: two appearances of the same token always receive the same vector. That consistency is helpful, but it also opens a gap. The vector for a token does not, by itself, say where that token appears in the sequence.
What do you think happens?
The word 'the' appears near the start of a sentence and again near the middle. Does the model receive the same token embedding vector both times, or does the vector change with position?
Reveal answer
The vector is the same both times.
Two appearances of the same token always receive the same token embedding vector. The vector records the identity of the token, but nothing in it records where the token sits. Position information has to be supplied some other way, which is the job of positional embeddings.
Why Token Vectors Alone Lose Order
Token embeddings give each token a consistent vector, but they lack any sense of the token's position in a sequence. Consider a short sentence in which the word 'the' appears at positions 1, 5, and 9. Its token embedding is identical in all three places. Nothing in that vector records whether the token came first, fifth, or ninth. If the model only had the token embeddings, it would see the same three vectors regardless of the order in which they appeared.
The fix is to add a second piece of information. A positional embedding carries the position index, so the model can tell the three occurrences apart. The token embedding answers the question 'what is this token?', and the positional embedding answers 'where is it?'. The two answers are then combined, as the next sections explain.
Special Tokens for Unknown Words and Boundaries
A regular token stands for a piece of the text itself. A special token stands for a condition or a structural event rather than a piece of text. The source identifies two main jobs for special tokens, and each one is shown by a named example. Special tokens are written with angle brackets and a vertical bar, such as <|unk|> and <|endoftext|>.
| Property | Regular token | Special token |
|---|---|---|
| What it stands for | A piece of the text itself | A condition or structural event |
| Example | An ordinary word or word piece | <|unk|> or <|endoftext|> |
| Typical situation | Ordinary text the model has a token for | An unknown word, or a boundary between unrelated texts |
Regular tokens versus special tokens
Absolute and Relative Positional Embeddings
The source describes two main types of positional embeddings: absolute and relative. An absolute positional embedding assigns position information according to each token's place in the sequence. Each position gets its own slot, determined by its index. Relative positional embeddings are the other main type. The source names them but does not elaborate on how they are computed, so this article does not describe that mechanism in detail. What matters for this topic is the distinction between the two families and which one GPT models use.
Tracing Position 2 Through the Input
In GPT models, the absolute positional embedding is added to the token embedding vector. The source also states that these positional embeddings are optimized during model training, which means their values are learned rather than fixed by hand. Once combined, the single vector carries both the identity of the token and its position. The diagram below traces this flow for one token.
Building the input vector for one token
A sequence contains the token 'cat' at position 2. Trace how the vector the model receives is formed from the token embedding and the absolute positional embedding.
- 1Identify the token
The token at position 2 is 'cat'. Its identity is captured by its token embedding vector, which is the same every time 'cat' appears.
- 2Identify the position
The token sits at position 2, so the absolute positional embedding for position 2 is selected. Its values are learned during training.
- 3Combine the two vectors
The absolute positional embedding is added to the token embedding vector. The result is one vector.
- 4Check what the result carries
The combined vector encodes both that the token is 'cat' and that it is at position 2. If 'cat' appeared at position 7 instead, its token embedding would be the same, but the added positional embedding would differ, so the combined vector would differ.
The model receives a single input vector that carries both the identity of the token and its position in the sequence.
Mistakes Learners Make with Embeddings
Common mistake Assuming the token embedding already encodes position
What it looks likeBelieving the word 'the' at position 1 and at position 9 has two different token embeddings.
Why it failsTwo appearances of the same token always receive the same token embedding vector. The token embedding gives a consistent vector but lacks any sense of position.
Do this insteadTreat the token embedding as identity only. Position must come from a positional embedding that is combined with it.
Common mistake Thinking <|unk|> is a real word in the text
What it looks likeReading <|unk|> as a word the author wrote.
Why it failsA special token stands for a condition, such as an unknown word, rather than a piece of the text itself.
Do this insteadRead <|unk|> as a placeholder that signals a word the model had no ordinary token for.
Common mistake Assuming <|endoftext|> marks a word inside a document
What it looks likeExpecting <|endoftext|> to appear between two words of the same sentence.
Why it failsThe token <|endoftext|> marks the boundary between unrelated texts.
Do this insteadLook for <|endoftext|> where one unrelated text ends and another begins.
Common mistake Confusing absolute and relative positional embeddings
What it looks likeSaying GPT models use relative positional embeddings because relative schemes are also described.
Why it failsThe source states that GPT models use absolute positional embeddings. Relative embeddings are the other main type, not the one GPT uses.
Do this insteadRemember that GPT models use absolute positional embeddings, which are added to the token embedding vector.
Common mistake Believing positional embeddings are fixed by hand rather than learned, when the source states they are optimized during training
Check Your Understanding
A sentence contains the word 'river' at positions 3 and 11. Explain why the token embedding for 'river' is the same at both positions, and describe what must be added to it so the model can tell the two occurrences apart. Then say which positional scheme GPT models use.
Hints
- Recall what a token embedding records and what it does not record.
- Think about which vector is added to the token embedding, and what it carries.
- The source names two families of positional embeddings; check which one GPT models use.
Key Takeaways
- 1Token embeddings give each token a consistent vector, but that vector does not record where the token sits in the sequence.
- 2Special tokens such as <|unk|> handle unknown words, and <|endoftext|> marks the boundary between unrelated texts.
- 3The two main families of positional embeddings are absolute and relative. GPT models use absolute positional embeddings.
- 4In GPT models, the absolute positional embedding is added to the token embedding vector, and the positional embeddings are optimized during training, so the combined vector carries both identity and position.
At a glance
- Token embeddings give each token a consistent vector, but that vector does not record where the token sits in the sequence.
- Special tokens such as <|unk|> handle unknown words, and <|endoftext|> marks the boundary between unrelated texts.
- The two main families of positional embeddings are absolute and relative. GPT models use absolute positional embeddings.
- In GPT models, the absolute positional embedding is added to the token embedding vector, and the positional embeddings are optimized during training, so the combined vector carries both identity and position.