Word2Vec and Similarity in Embedding Space

Suppose someone hands you a word embedding and asks how big it is. You might expect the answer to describe the word itself, but an embedding is simply a list of numbers attached to that word. The size of that list is what we call the dimensionality of the embedding. In this article we will trace, step by step, what that size controls, why it matters for the relationships an embedding can express, and why bigger is not automatically better.

Keep one question in mind as you read: if each word is described by a list of numbers, what happens to the meaning captured by those numbers when the list gets longer or shorter?

Dimensionality is the number of value slots in the vector that represents one word. Each slot holds one number, and the vector is the full ordered list of those numbers. Counting the slots gives the dimensionality, and the source notes that in practice this count ranges from one to thousands.

Read the diagram as a single word. Its 768 slots together form one embedding. Changing the dimensionality changes how many slots exist, but not which word the vector belongs to. A word with a 2-dimensional embedding has two slots, and a word with a 1-dimensional embedding has one.

A two-dimensional embedding can be drawn directly. Each word becomes a point on a flat plane, with one number setting its horizontal position and the other setting its vertical position. Once words are points, we can look at which points sit close together. The source explains that when a technique such as Word2Vec is used, words for similar concepts tend to land near one another in the embedding space.

The plotted example in the source groups several kinds of birds in one region of the plane, and it places a country and its capital city as a pair in another region. Comparative forms of the word long form a third group. The useful information is in the arrangement of the points: words that share a meaning or role sit in the same neighborhood.

Two numbers per word are enough for a picture, but they leave little room to represent the many ways words relate to one another. The source states that a higher dimensionality might capture more nuanced relationships. Each extra dimension is another value the embedding can use to separate words that differ in subtle ways, so a richer vector has more room to record fine distinctions.

That extra room has a price. The same source notes that higher dimensionality comes at the cost of computational efficiency. Every word now carries more numbers, so storing the embeddings takes more memory, and every calculation that touches them does more arithmetic. Designers therefore weigh how much nuance they need against how much computation they can afford.

In a low-dimensional space, different meanings are forced to overlap because there are few slots to separate them. In a high-dimensional space, words have more room to differ along separate aspects of meaning. The source frames this as a tradeoff rather than a rule that more is always better.

The plane works because two dimensions fit on a page. Our senses and ordinary graphs are limited to three dimensions or fewer, so a vector with hundreds of values cannot be drawn as points in one picture. The source explains that this limit is exactly why the earlier example used a two-dimensional embedding in a two-dimensional scatterplot.

When we move to large language models, the embeddings have a dimensionality far higher than what we can plot. Visualizing them directly is therefore not practical, and you should not expect to see the full vector of a large model as a picture. Any picture of such an embedding has to be a reduced view of it, and that reduction is a separate step from the embedding itself.

The source uses two members of the GPT family to make the idea concrete. For both GPT-2 and GPT-3, the embedding size, often called the dimensionality of the model's hidden states, changes with the specific variant and size of the model. Larger models generally use larger embeddings, and the source frames the choice as a tradeoff between performance and efficiency.

Dimensionality is chosen as part of how a large language model turns text into numbers. The source points to the steps that come next: splitting text into words, converting words into tokens, and turning tokens into embedding vectors. The embedding size is the length of the final vector produced in that last step, so it sets the shape that every later stage of the model must handle.