Introduction to Word Embeddings

Once you learn that a neural network can turn text into a list of numbers, a tempting conclusion follows. If the numbers capture meaning, then the same network should be able to handle any kind of input. Sound, images, and video all carry information too, so why not feed them through the same machinery and collect the same kind of output? This article examines that idea, explains why it breaks down, and shows how the different families of text embeddings fit together.

The conversion can be carried out in two ways. It can use a specific neural network layer, or it can use some other pretrained neural network model. Both routes produce the same kind of result: a numeric representation of the input. The important point is the output. Whatever the input was, the embedding is a vector that the rest of a computation can work with.

Embedding is not restricted to one kind of input. The same general idea applies to video, audio, and text. What changes from one kind to another is the model doing the conversion, not the idea of turning the data into a vector.

Each format has its own structure. Video has frames, sound has a waveform, and text has a sequence of symbols. An embedding model has to be built around the structure of the data it reads. This is why the source emphasizes that different data formats require distinct embedding models, and why a text model should not be used for audio or video.

The mistaken belief is that a single embedding model can be applied to any data type. Treating a text model as a universal converter does not fail loudly. It produces vectors that look perfectly valid in form but carry no useful meaning for the input they came from. This silent failure is what makes the mistake dangerous.

Within the text domain, the unit being embedded can vary. The most common form is the word embedding, which gives a vector for an individual word. The source also describes embeddings for larger units of text: sentences, paragraphs, and whole documents. Choosing the unit is a design decision, because the size of the unit determines what the resulting vector summarizes.

The source gives one concrete use case. Sentence and paragraph embeddings are popular in retrieval-augmented generation, which combines two activities. Generation is producing text. Retrieval is searching an external knowledge base. The technique pulls in relevant information while text is being generated. The source treats the technique itself as beyond its scope, so this article only notes where it connects to embeddings. The source does not describe a specific use case for whole-document embeddings, so you should treat the document level as the largest unit of text rather than as a use case with its own method.

Suppose you have already split a piece of text into tokens and assigned each token an integer ID. You now have a list of numbers that stands in for the sentence. What does the model receive as its input once those integers exist? The short answer is that the integers are converted into embedding vectors, and those vectors are what the LLM works with during training.

The full preparation path runs from input text to tokenized text, then to token IDs, then to embedding vectors. Each stage changes the form of the data while keeping track of the same content. The last stage gives the model numeric input that it can adjust during learning. The diagram below follows one short sentence through all three stages.

The last transition is the least obvious, so it deserves a closer look. Each token ID is paired with its own embedding vector. The conversion keeps the identity of each token while changing its form. One ID leads to one embedding vector, and the pairing is the important idea to hold onto. The source does not describe the internal arithmetic of this lookup, so the diagram below shows only the correspondence.

The IDs in this diagram are illustrative values chosen for the example sentence The cat sat. The pairing is what matters. Each ID picks out one vector, and the vector is a different thing from the ID. The ID is an integer that names a token. The vector is a list of numbers the model can adjust during learning.

The values inside the embedding vectors do not come from a dictionary of meanings. Embedding weights start with random values. They are then optimized during LLM training, which means the training process updates them so they become more useful for the task. At the start, a vector for a token carries no meaning. Over training, the same vector is adjusted.