Embedding Models in Practice

Imagine a system that must answer questions about a sentence such as 'the cat sits on a mat'. To a person, that sentence is words with meaning. To the machine doing the work, it is a string of letters and spaces. The machine is built to handle numbers, so something has to bridge that gap before any reasoning can happen.

The idea that performs this bridging is called an embedding. Embeddings are one of the central ideas behind modern language and image models. In this article you will follow a single input step by step, see what the numbers look like, and learn which kinds of models produce them.

A model cannot act on raw text in the way a person does, because the hardware and the arithmetic inside it operate on numbers. That is the starting constraint. Whatever a model receives, whether text or an image, must first be turned into a form the arithmetic can use.

This raises the central question: what kind of numbers are useful? Arbitrary numbers would not help. The numbers need to carry something about the original data forward into later computation. An embedding is the answer the field uses for this problem.

Follow the sentence from the moment it is submitted to the moment the model has something numerical to work with. The diagram below shows the sequence. The sentence enters as text, is converted into a numerical representation, takes the form of an embedding vector, and only then is passed to the model's computation.

The conversion is the step that makes the rest of the pipeline possible. Without it, the model would have nothing it could compute with. Notice that the original sentence and its embedding are not two different pieces of content. They carry the same underlying content, built for different consumers.

The word 'aims' matters. An embedding is an attempt to encode meaning in numbers, not a guaranteed or complete copy of meaning. The numbers are useful precisely because they carry that encoded meaning forward into later computation. A model that receives a good embedding can use the numbers to reason about what the input represents, while the raw letters alone give it nothing numerical to combine.

Because an embedding is a vector, it can be written as an ordered list. The source illustrates this with the sentence 'the cat sits on a mat', which might be represented by the vector [0.11, 0.02, 0.54]. Each slot in that list is one element. The diagram below labels each position so you can see that the list is an indexed structure.

The three-element example used earlier is far below that range. It should be read as an illustration of structure, meaning what a vector is and how its slots are ordered, rather than as an illustration of scale. Real embeddings are much longer lists.

Embedding size is not one fixed value across the field. Different models produce embeddings of different sizes, and the source lists sizes for several common models, including BERT, CLIP, and the OpenAI Embeddings API. Some models list separate sizes for different variants, and some list separate sizes for image and text inputs. When you pick a model, its embedding size is one of the properties you must check in its documentation, because the choice of model determines the size and other properties of the vectors it outputs.

An embedding is produced by a model, and some models are trained specifically to produce embeddings. Embedding models are a family rather than a single model. They fall into two broad groups: open-source models and proprietary models offered as APIs. The diagram below groups the named examples from the source.

The open-source group includes BERT, CLIP, and Sentence Transformers. CLIP stands for Contrastive Language-Image Pre-training, which signals that it works across language and images. The proprietary group is represented in the source by models provided as APIs, with the OpenAI Embeddings API as the named example. The distinction is about how the model is built and accessed, while both groups produce embeddings that the model's later computation can use.