Tokenization and Vocabulary Basics
Each vector position corresponds to a vocabulary entry, so a particular output number can be associated with that entry.
Try it: Text Vectorization: Tokens to Vectors
The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.
How it works
- Standardize: lowercase and delete punctuation.
- Split on whitespace into tokens.
- Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
- Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
- Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.
Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.
Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.
Loading the simulation…
Predict the Output Shape
Imagine a model receives two text samples, with four tokens in each sample. Before reading the output, ask what the model must return for each token position. Would one number be enough, or would the model return a collection of numbers? The answer determines how to read the shape that follows.
What do you think happens?
For a batch of two text samples with four tokens per sample, what shape does this run report for the model output?
Reveal answer
[2, 4, 50,257]
The first two dimensions represent the two text samples and the four token positions per sample. The final dimension has one entry for each of the 50,257 vocabulary tokens.
Read the Three Axes
The reported output shape is torch.Size([2, 4, 50257]). Read it from left to right: the first axis selects a text sample, the second selects a token position within that sample, and the third selects a vocabulary entry. The shape describes nested results: for each sample and each token position, there is a whole vector of vocabulary-sized output values.
Finding One Output Value
Use the shape [2, 4, 50,257] to describe how you would locate a value for one particular text sample, one token position, and one vocabulary entry.
- 1Choose a sample
Use the first axis to select which of the two text samples the result belongs to.
- 2Choose a token position
Within that sample, use the second axis to select one of its four token positions.
- 3Choose a vocabulary entry
Within the selected position's vector, use the third axis to select one of the 50,257 vocabulary entries.
One value in the output is identified by a sample, a token position within that sample, and a vocabulary entry.
Why Each Position Has a Vocabulary Vector
At a particular sample and token position, the output is not a single guessed word. It is a vector with 50,257 entries, one for each unique token in the vocabulary. Each vector position corresponds to a vocabulary entry, so a particular output number can be associated with that entry. The model output is commonly called logits. These numbers are an intermediate result, not yet decoded words.
In a later postprocessing step, the vectors are converted into token IDs, and those IDs can then be decoded into words. Keep the stages distinct: logits are the model's output numbers; token IDs and decoded words come afterward. The output shape tells you that every sample position has a vocabulary-sized vector, but it does not mean the model has already returned readable text.
Track Changes in Shape
The three dimensions correspond to separate quantities: the number of text samples, the number of token positions in each sample, and the vocabulary size. If the batch contains more samples, the first dimension changes. If each sample contains more token positions, the second dimension changes. The final dimension remains tied to the tokenizer's vocabulary size, which is 50,257 in this run.
The changed shapes in the diagram are generated examples applying the same axis meanings. In both comparisons, the vocabulary dimension stays at 50,257. The examples show why it is useful to read every dimension separately instead of treating the shape as an unexplained sequence of numbers.
Trace the Data Path
When a run produces an unexpected result, check the stages in order rather than guessing from the final tensor alone. Start with the batch of tokenized input: this run has two samples and four tokens per sample. Then inspect the model output, whose expected shape is [2, 4, 50,257]. After that, keep the later postprocessing separate: logits are converted to token IDs and then decoded into words. The source places layer normalization inside the model, between its input and output, but its normalization details are outside this article's scope.
- 1Check the input batch: confirm how many text samples it contains and how many tokens are in each sample.
- 2Check the model output: compare its first two dimensions with the sample count and token count, and check whether its final dimension matches the vocabulary size.
- 3Check which stage you are inspecting: logits are model output numbers; token IDs and decoded words belong to later postprocessing.
Mistakes That Confuse the Output
Common mistake Reading the dimensions in the wrong order.
What it looks likeTreating [2, 4, 50,257] as four samples with two token positions.
Why it failsThe axes are ordered as text samples, tokens per sample, and vocabulary entries.
Do this insteadRead from left to right: 2 samples, 4 token positions per sample, and 50,257 vocabulary entries.
Common mistake Treating each token position's output as one word or one number.
What it looks likeAssuming one position in the output contains only a single guessed token.
Why it failsEach position has a vector with one entry for every token in the vocabulary.
Do this insteadThink of a position as holding a 50,257-entry vector of logits, not decoded text.
Common mistake Calling the logits decoded words.
Why it failsThe logits are an intermediate model output; conversion to token IDs and decoding into words happen later.
Do this insteadIdentify whether you are looking at logits, postprocessed token IDs, or decoded words.
Practice Reading a New Shape
A generated example has output shape [3, 6, 50,257]. Describe what each dimension represents. Then say which dimension would change if the batch contained one more text sample, and which would change if every sample had one more token position.
Hints
- Use the same axis order: text samples, token positions per sample, vocabulary entries.
- The vocabulary dimension is determined by the tokenizer's vocabulary size.
Key Takeaways
- The output shape [2, 4, 50,257] describes two text samples, four token positions per sample, and 50,257 vocabulary entries.
- At each sample and token position, the model returns a vocabulary-sized vector rather than a single decoded word.
- The output numbers are commonly called logits; later postprocessing converts them into token IDs and then decoded words.
- Changing the sample count changes the first dimension, changing the token count changes the second, and the vocabulary size determines the third.
- To investigate an unexpected result, check the input batch, model output shape, and postprocessing stage in order.