Input Tokenization and Embedding

Every time a chatbot answers you, it is producing text in small pieces, one word or token after another. A GPT model is built for exactly that job. The letters stand for Generative Pretrained Transformer, and the model is a large deep neural network whose purpose is to generate new text one token at a time.

A large language model cannot read a sentence the way you do. Before any learning can happen, text has to be turned into a form that a deep learning model can process. This article follows that preparation in two stages. The first stage converts raw text into discrete integer identifiers. The second stage converts those identifiers into continuous vectors that a model can work with. As you read, keep one question in mind: what does the text look like after each step?

A natural first question is how a system that large can be understood at all. The answer is that the design is less tangled than its size suggests. Many of its parts are the same piece repeated over and over, so once you understand one repeated unit and how the units are wired together, most of the architecture comes into focus.

Think of a building made of identical floors. An architect only needs to draw one floor in detail and then specify how many times to stack it. A GPT-style model works in a similar spirit. Its core contains one or more transformer blocks, and each transformer block contains a masked multi-head attention module. Adding more blocks makes the model deeper without requiring a new kind of part.

Beyond the transformer blocks, a GPT model has embedding layers before them and output layers after them. The embedding layers turn tokenized text into numerical vectors. The transformer blocks do the main processing, and the output layers produce the model's result. The masked multi-head attention module is treated in this course as a finished component that gets placed inside the transformer block; its internal arithmetic is studied separately.

Read the diagram as a set of nested containers. The whole model is the outer box. Inside it are the embedding layers, one or more transformer blocks, and the output layers. Inside each transformer block is the masked multi-head attention module. The tokenized text enters at the embedding stage, which is why the input connects to the embedding layers first.

Take the tokenized input 'Every effort moves you'. It moves forward through the model in a fixed order. It enters the embedding layers first, then passes through the transformer blocks, where the masked multi-head attention module does its work, and it arrives at the output layers last.

This order tells you which stage hands its result to which other stage. It does not show the arithmetic inside a transformer block. Keeping the stages separate in your head makes the rest of the chapter easier to follow.

The first step is tokenization. Raw text is broken into tokens, and a token can be a word or a character. After that, each token is converted into an integer representation called a token ID. So the pipeline runs from a string of characters, to a list of tokens, to a list of integers.

Notice that the output of tokenization is discrete. Each token ID is a whole number that names one token. The text is no longer a continuous stream of characters, but a sequence of labels drawn from a fixed set.

Tokenization is the general process of breaking text into tokens. Byte pair encoding (BPE) is one specific tokenizer that performs that process, and it is the approach used for LLMs such as GPT-2 and GPT-3.

A tokenizer built only on whole words runs into trouble when it meets a word it has never seen. The BPE tokenizer handles this efficiently. It breaks an unfamiliar word down into subword units or, if necessary, into individual characters. The unknown word does not have to be discarded, because its pieces can still be represented.

Here is how the fallback works in principle. Start with an unfamiliar word. If the whole word is not in the vocabulary, look for smaller pieces that are. If even those pieces are not known, keep splitting until each remaining unit is a single character. Every stage of this process produces a unit the tokenizer can still represent, so nothing is lost.

Token IDs are integers, and integers alone are not a rich representation for a deep learning model. An embedding layer in PyTorch solves this by functioning as a lookup operation. It takes a token ID and retrieves the vector that corresponds to it. The retrieved embedding vectors provide continuous representations of tokens, and this is crucial for training deep learning models such as LLMs.

The diagram uses an illustrative toy embedding matrix with made-up numbers; a real model's values are learned during training. The point is the mechanism: the ID 2 is a position, not a vector. The lookup picks out row 2, and that row is the vector the model works with. The integer itself is only a label, while the retrieved row carries several continuous values.

Parameters are the trainable internal variables adjusted during training. A useful way to picture them is as a large set of adjustable dials. Each dial starts at some value, training turns the dials a little at a time based on how wrong the model currently is, and the final settings of all the dials together determine how the model responds to input. When people talk about the size of a GPT model, they are counting these dials.

A weight matrix holds one parameter per element, so its parameter count is its number of rows multiplied by its number of columns.

This chapter builds the core structure and does not yet train it or load pretrained weights. Loading pretrained weights and adapting the code to larger GPT-2 variants is deferred to a later chapter.