Embedding Size and Model Choice

Imagine a system that must answer questions about a simple sentence such as 'the cat sits on a mat'. The sentence is made of letters and spaces, but the machine doing the work is built to handle numbers. Before any reasoning can happen, something has to bridge that gap. The idea that does this bridging is called an embedding, and it is one of the central ideas behind modern language and image models.

A model cannot act on raw text in the way a person does, because the hardware and the arithmetic inside it operate on numbers. That is the starting constraint. Every input, whether it is a sentence or an image, must first be turned into a numerical form that the model can compute with. The practical question is what kind of numbers are useful for this job.

The word 'aims' matters. An embedding is an attempt to encode meaning in numbers, not a guaranteed copy of that meaning. The numbers are useful precisely because they carry the encoded meaning forward into later computation. The original sentence is meant for people reading text, while the embedding is meant for a model that needs numbers to compute with. The two forms carry the same underlying content but are built for different consumers.

Because an embedding is a vector, it can be written as an ordered list. The source illustrates this with the sentence 'the cat sits on a mat', which might be represented by the vector [0.11, 0.02, 0.54]. Each slot in that list is one element, and the position of an element in the list matters. Changing the order of the numbers produces a different vector, even when the same numbers appear.

Read the diagram from left to right. Each box is one slot, and the number inside it is that slot's element. The three slots together form one vector. The source does not say what any single slot means on its own, so do not read a single position as a named feature. What it does say is that the vector as a whole, with its numbers in their particular order, is the representation.

The conversion happens before any reasoning. The raw input, such as a sentence or an image, is handed to a model that produces a numerical representation. That representation is the embedding, and it becomes the material the rest of the computation works on. The order of these steps is fixed: conversion first, then processing.

Trace the sentence through the pipeline. It starts as text, which a person can read but a model cannot compute with directly. An embedding model converts it into an embedding vector. Only then does the model's computation begin, and that computation works on the numbers in the vector.

The three-element vector used earlier is far below the typical range. It should be read as an illustration of structure, showing how a vector is an ordered list, rather than as an illustration of scale. A real embedding usually has many more elements than three.

An embedding is produced by a model, and some models are trained especially to produce embeddings. Embedding models form a family rather than a single model. The choice of model determines the embedding size and other properties of the vectors it outputs, so model choice and embedding size are linked.

Notice what this check can and cannot tell you. It can tell you whether a size is typical. It cannot tell you whether the vector captures meaning well, because the source describes an embedding as an attempt to encode meaning rather than a guaranteed result.