Understanding the Transformer Architecture

Read a recent machine learning paper, a product announcement, or a forum thread, and you will often see the words transformer and LLM used as if they meant the same thing. The habit is understandable. The language models that have attracted the most attention in recent years are built on the transformer architecture, so the two labels tend to travel together.

The two labels do not name the same kind of thing. One is a neural network architecture, which is a design for how a network's computations are organized. The other is a category of AI model defined by what it does, namely processing language. Keeping these two ideas apart is the purpose of this article.

A common error here is a pair of mirror-image assumptions. The first says that every transformer is an LLM. The second says that every LLM is a transformer. Each assumption feels reasonable because the two sets overlap heavily in practice, but each one fails for a different reason. It helps to test them rather than accept them.

A useful way to hold the relationship in your head is to separate the design from the purpose. The transformer is a design. An LLM is a purpose-driven category: a model built to handle language. Today's LLMs happen to be constructed from transformers, which is why the two overlap so much. Overlap is not identity, though.

The diagram below shows the three places where the terms part ways. Today's LLMs sit in the region where the two categories overlap, because they are built from the transformer design. Outside that region, the transformer design is also used beyond language, and LLMs are also built on other designs.

Now test the second assumption, that every LLM is a transformer. The source states that some LLMs are based on recurrent and convolutional architectures. An LLM built on one of those designs is still an LLM, because its purpose is processing language, but it is not a transformer. The second assumption fails for the opposite reason: the category is defined by purpose, and purpose does not force a particular architecture.

The key idea is the direction of the relationship. The transformer architecture is the starting point, and LLMs are what get derived from it. The architecture is therefore the more general design, and LLMs are built on top of it. Understanding the architecture is one of the routes to understanding LLMs.

The architecture is a design, not a single trained model. LLMs are built on top of it. The direction runs from the base architecture to the derived LLMs, not the reverse. The diagram below shows that direction, naming only the two things the source establishes.

A useful way to reason about a base design is to imagine it as a template that several products are built from. The template is not any one product. Each product starts from the same template and then takes its own shape. The template idea is an illustration of the relationship, not a description of how transformers are built internally.

If transformer-based LLMs are the dominant design, why would anyone build LLMs on recurrent or convolutional foundations? The source gives a direct answer: the main motivation behind these alternatives is to improve the computational efficiency of LLMs. The goal is to make language models cheaper to run in terms of computation, not to make a different kind of language model.

This explains why recurrent and convolutional LLMs still count as LLMs. Their purpose, processing language, is unchanged. What differs is the architecture chosen to carry out that purpose, and the stated reason for choosing it is computational efficiency.

The comparison below sorts the cases the source supports. A transformer that is not an LLM is one used for computer vision. An LLM that is not a transformer is one based on a recurrent or convolutional architecture. The middle region holds transformer-based LLMs, the systems where the two labels coincide.

Notice what the source does and does not name. It names computer vision as a setting where transformers are used, and recurrent and convolutional designs as the basis for some LLMs. It does not provide a full list of transformer-based LLMs or of computer vision systems, so the diagram shows only the categories the source supports.

The available material establishes the derivation relationship only: LLMs are derived from the transformer architecture, and the architecture is a design rather than a trained model. Several links are not stated, so they should not be added as facts.