Tokenization and Token Embeddings in LLMs

Suppose someone hands you a word embedding and asks how big it is. You might expect the answer to describe the word itself, but an embedding is simply a list of numbers attached to that word. The size of that list is what we call the dimensionality of the embedding. Word embeddings can have anywhere from one dimension to thousands in practice.

Keep one question in mind as you read: if each word is described by a list of numbers, what happens to the meaning captured by those numbers when the list gets longer or shorter? The rest of this article traces that question step by step, from a single number per word up to the very large vectors used in GPT models.

The clearest way to see what dimensionality counts is to watch a single word while only the length of its vector changes. Nothing else about the word is altered. Only the number of values in its list is allowed to vary.

Read the diagram as a row of boxes. A 768-dimensional embedding has 768 such boxes, numbered from zero, and each box holds one number. The index of a box tells you which dimension it belongs to, and the number inside is that dimension's value for the word. The word has the same identity in every case; only the count of boxes changes.

A two-dimensional embedding can be drawn directly. Each word becomes a point on a flat plane, with one number setting its horizontal position and the other setting its vertical position. Once words are points, we can look at which points sit close together. When a technique such as Word2Vec is used, words for similar concepts tend to land near one another in the embedding space.

The plotted example places several kinds of birds in one region of the plane, and a country paired with its capital city in another region. Comparative forms of the word long form a third group. Each cluster is a neighborhood of similar points, and the grouping is what the two-dimensional picture communicates.

Two numbers per word are enough for a picture, but they leave little room to represent the many ways words relate to one another. Each extra dimension is another value the embedding can use to separate words that differ in subtle ways. Two words might be close on one feature, such as being animals, yet still differ on another, such as size or habitat. A richer vector has more room to record those fine distinctions.

Dimensionality is a tradeoff rather than a setting where bigger is always better. The gain is nuance; the price is computation. The two effects grow together, so a designer has to decide how much of each to accept.

The two-dimensional plane worked because two dimensions fit on a page. Human senses and ordinary graphs are limited to three dimensions or fewer, so a vector with hundreds of values cannot be drawn as points in one picture. The earlier bird-and-capital example used a two-dimensional embedding precisely because that is the only size that can be plotted without loss.

When we move to large language models, the dimensionality is far higher than anything we can plot. Visualizing such an embedding directly is therefore not practical, and a learner should not expect to see the full vector of a large model as a picture. Any picture of such an embedding has to be a reduced view, and that reduction is a separate step from the embedding itself. A reduced view shows some structure while leaving the rest of the vector out of sight.

The GPT family makes the idea concrete. In GPT models the embedding size varies by variant and size, and the embedding size is often called the dimensionality of the model's hidden states. Larger models generally use larger embeddings, and the choice is framed as a tradeoff between performance and efficiency.

Between the smallest GPT-2 model at 768 dimensions and the largest GPT-3 model at 12,288 dimensions, the embedding length grows by a factor of 16. Every word in the largest GPT-3 model is therefore described by sixteen times as many values as a word in the smallest GPT-2 model. This is the same tradeoff at scale: more room for nuance, bought with more storage and arithmetic per word.

Dimensionality is chosen as part of how an LLM turns text into numbers. Text is split into words, those words are converted into tokens, and the tokens are turned into embedding vectors. The embedding size is the length of the final vector produced in that last step, so it sets the shape that every later stage of the model must handle.

The ideas in this article are easy to misread. The three mistakes below come up often.