From Token IDs to Embedding Vectors

A tokenizer that works on one sentence has not yet been proven. The real test is whether it still behaves correctly on a much longer text. In this article we apply a basic tokenizer to an entire short story by Edith Wharton and check three things: how many tokens come out, whether the first tokens look right, and how those tokens are turned into integers.

Before you trace anything, take a guess at how many tokens the story produces. Most people count words and punctuation marks, and that is close to what the tokenizer does. The gap between a rough guess and the exact count is exactly the kind of detail a step-by-step trace can settle for you. Keep your guess in mind; the count appears later in the article.

The whole path has two stages that you should keep separate. Tokenization decides where the boundaries fall in the text. Token ID conversion decides how each resulting token is represented as a number. A problem in the first stage carries into the second, so verifying the token list before you trust the IDs built from it is a useful habit.

The core of the tokenizer is a single call to re.split from Python's re module. You give it a pattern and a string, and it cuts the string wherever the pattern matches. The pattern in this tokenizer matches three kinds of boundary: a character class listing commas, periods, colons, semicolons, question marks, underscores, exclamation marks, parentheses, and quotation marks; the double dash; and any whitespace character.

The pattern uses a capturing group, which is what keeps matched punctuation in the output. Without the group, the punctuation marks that trigger a split would disappear from the result. With the group, each punctuation mark survives as its own item, so a word and the comma after it end up as separate tokens rather than one fused string.

The stages in the diagram follow the order in which the tokenizer works. First the text is cut at every boundary the pattern describes. Matched punctuation is retained through the capturing group. The raw result then goes through a second pass, which is where blank items are handled. Reading the pipeline in this order lets you check each stage separately instead of trusting the final list as a whole.

Because the split happens at every whitespace character, the raw result also contains empty strings and strings that hold only spaces. Those are not tokens. A second pass removes them, keeping only items that still contain something after stripping spaces. This is why the count you get is not simply the number of pieces the first split produced.

A quick visual check helps before you trust a long count. Print a slice of the first thirty tokens and look at the order of the items. Ask whether any word has been cut in half and whether any punctuation mark is stuck to a word. In the source's first-thirty check the output looked right: every word was separate, and the capital letter in HAD was kept exactly as it appeared in the source.

The tokenizer does not lowercase anything. Case differences therefore remain visible in the tokens, so HAD and had would be different strings. This matters later, because the token list is the input to the conversion into integers.

A list of tokens is still a list of Python strings. The next step converts those tokens from strings into an integer representation, producing token IDs. The source describes this conversion as an intermediate step, sitting between the tokens and the embedding vectors that come after it. A token ID is an integer, and it is the value the embedding layer will use as a key.

A language model does not read text directly. Text is first split into tokens, and each token is represented by an integer token ID. The question this article answers is how those integer IDs become embedding vectors, which are the numeric representations a model can compute with.

The source works with a small, concrete case: four input tokens whose IDs are 2, 3, 5, and 1. The IDs are stored in a tensor, which is the data container PyTorch uses for numbers. torch.tensor() creates one from a Python list. The variable input_ids holds the integers that the embedding layer will look up.

The embedding layer performs a lookup operation, not a calculation on the token ID. Integer token IDs enter the layer, and the layer uses each ID to select a row of its weight matrix. The selected rows leave the layer as embedding vectors. Nothing in this path performs arithmetic on the IDs; the layer's job is retrieval.

The diagram shows the lookup for the four token IDs from the opening example. Each ID points to exactly one row, and the row number equals the ID. The values inside those rows are not part of this section, because the source's matrix is not reproduced here. What matters for now is the mapping from ID to row.

The rows are counted from index zero, which is the convention Python uses. This means the row for ID 3 is the fourth row in the matrix, not the third. Keep this in mind whenever you read a row number off a matrix by eye, because counting from one is the most common source of an off-by-one mismatch.

The next step applies the embedding layer to a single token ID. The layer is named embedding_layer in this example, and calling it like a function is how you use it. The call passes a tensor containing only the ID 3. Before you run it, notice what you are asking for: one token ID goes in, and the layer must return a vector that belongs to that token.

The returned result is displayed inside double square brackets. The outer pair of brackets means the value is a two-dimensional tensor with a single row. The three numbers inside that row are the components of the embedding vector for token ID 3. In this example each embedding vector has three components.

The display also includes a label named grad_fn, with the value EmbeddingBackward0. The source does not explain that label in this section, so treat it as metadata that PyTorch attaches to the result. The important part is the three numbers, because they are the values stored in the matching row of the weight matrix.

The source makes the point by comparison. The embedding vector returned for token ID 3 is identical to the fourth row of the embedding matrix from the previous discussion. Because Python starts with a zero index, token ID 3 corresponds to the row at index 3. The layer does not compute a new vector from the ID. It reads a stored row.

The following mistakes come up often when people first trace a tokenizer and an embedding layer. Each one is a point where the output can quietly diverge from what you expected.