Token IDs and Vocabularies
A token embedding is retrieved by looking up the row of the embedding layer's weight matrix that matches the token ID, with rows counted from zero.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
A Word That Appears Twice
Imagine a short input that contains the same word twice, once near the beginning and once near the end. A human reader sees two copies of the same word and knows they are the same word. Does a large language model treat them the same way? The answer depends on what the first stage of the model, the embedding layer, actually does with each token. That first stage is where this article begins.
What do you think happens?
Before reading on: when the embedding layer receives two copies of the same word, what vector does it return for each copy?
Reveal answer
Two identical vectors, because the lookup uses the same token ID.
The embedding layer retrieves the row that matches the token ID. Both copies carry the same ID, so both retrieve the same row. In the worked example later in this article, both copies of the word fox return the vector 1.2753 -0.2010 -0.1606.
Tracing a Token ID Through the Weight Matrix
An embedding layer holds a weight matrix. Each row of that matrix is a vector, and each token ID selects exactly one row. When the layer receives a token ID, it does not compute anything complicated. It retrieves the vector stored in the row that corresponds to that ID, and that retrieved row is the layer's output. A useful way to picture this is a table in which the token ID acts as the address of a row: you give the layer the address, and it hands back the contents of that row.
Row counting starts at zero, and this is the detail that trips people up. Token ID 5 corresponds to the sixth row of the matrix, not the fifth. This follows from Python's zero-based indexing, and it is a common source of off-by-one errors when people read embedding matrices by hand. In the illustrated matrix, the rows are numbered from the top, starting at 0. Token ID 5 retrieves the row at that position, whose entries are -2.8400, -0.7849 and -1.4096.
Looking up token ID 5 by hand
A matrix has its rows numbered from 0 at the top. Which row does token ID 5 retrieve, and what vector comes out?
- 1Read the ID
The token ID is 5. The lookup key is the number itself.
- 2Count from zero
Row 0 is the first row, row 1 the second, and so on. Row 5 is therefore the sixth row, not the fifth.
- 3Retrieve the row
The entries of row 5 are -2.8400, -0.7849 and -1.4096. These three numbers form the output vector for token ID 5.
Token ID 5 produces the vector -2.8400 -0.7849 -1.4096. The lookup used only the ID, so the same vector would appear wherever ID 5 occurs in the input.
Why Repeated Tokens Get Identical Vectors
Because the lookup depends only on the token ID, the layer produces the same vector for a given ID wherever that ID appears. The embedding layer has no input for position, so position cannot influence its output. This gives a clear prediction rule: identical IDs always produce identical vectors, and different IDs pull different rows.
Predicting the vectors for two copies of fox
In a sentence, fox appears at positions one and four, and both copies carry token ID 2. What vectors does the embedding layer return for these two positions?
- 1Identify the IDs
Both occurrences of fox carry token ID 2. Position number does not enter the lookup, so the position difference is irrelevant here.
- 2Retrieve the row
ID 2 selects the row numbered 2, counting from zero, in the weight matrix.
- 3Compare the outputs
Both positions return the same vector, 1.2753 -0.2010 -0.1606.
Position 1 and position 4 receive identical embedding vectors. The layer's output depends on the token ID alone.
Where Word Order Drops Out
Self-attention is the mechanism that relates tokens to each other inside an LLM. It takes the vectors that reach it and mixes information between them. The important limitation is that self-attention, on its own, has no notion of where a token falls in the sequence or in what order tokens appear. It receives the token vectors and nothing that encodes their positions.
Consider the two occurrences of fox from the earlier example, at positions one and four. The layer returns the same vector both times, and nothing else in the pipeline records which copy came first. The same reasoning holds for any token at any position. If the words were rearranged so that the sentence read fox fox jumps over, the vectors would form the same set of values, only in a different arrangement. The embedding layer would produce no distinct signal for that change in order.
Mistakes Beginners Make With Token IDs
Common mistake Assuming the same word gets a different vector when it appears in a different place
What it looks likeBelieving the second fox in a sentence returns a new vector because it comes later.
Why it failsThe lookup uses only the token ID. Position is not an input to the embedding layer, so the output cannot depend on it.
Do this insteadIdentical IDs give identical embedding vectors wherever they occur.
Common mistake Reading token ID 5 as the fifth row
What it looks likeCounting rows from one and taking the fifth row for token ID 5.
Why it failsRow counting starts at zero, so token ID 5 corresponds to the sixth row.
Do this insteadCount rows from zero. Token ID 5 retrieves the row at index 5, with entries -2.8400, -0.7849 and -1.4096.
Common mistake Believing self-attention knows word order by itself
What it looks likeExpecting the attention step alone to tell apart fox jumps over fox from fox fox jumps over.
Why it failsSelf-attention has no notion of position or order, and the token vectors it receives carry no position signal.
Do this insteadWord order must come from a separate source of position information, which the next stage of the model supplies.
Common mistake Treating the vocabulary as a list of words rather than a set of integer IDs, which hides the fact that the model only ever looks up rows by number
Checking the Lookup Rule
A weight matrix has rows numbered from zero. The sentence contains token ID 5 at position 2 and again at position 6. Without looking anything else up, what vector does each occurrence receive, and does the position of each occurrence change the answer? Explain your reasoning using only the lookup rule.
Hints
- Start by asking which row token ID 5 selects, counting from zero.
- Ask whether position appears anywhere in the lookup rule.
Summary of the Lookup Stage
- 1An embedding layer holds a weight matrix, and each token ID selects exactly one row of it, with rows counted from zero.
- 2The output of the embedding layer is the retrieved row, so the same token ID always yields the same vector regardless of position.
- 3Token ID 5 retrieves the sixth row, whose entries are -2.8400, -0.7849 and -1.4096; the two occurrences of fox, with token ID 2, both return 1.2753 -0.2010 -0.1606.
- 4Self-attention on its own has no notion of position or order, so position-blind token vectors cannot tell a sentence apart from a scrambled version of the same words; a separate source of position information is needed.
At a glance
- An embedding layer retrieves one row of its weight matrix for each token ID, counting rows from zero.
- Identical token IDs produce identical embedding vectors, because the lookup depends only on the ID and not on position.
- Self-attention on its own has no notion of where a token sits or in what order tokens appear.
- Position-blind token vectors mean the model needs an additional source of position information to read word order.