Transformer Architecture and Variants

Modern large language models are often assumed to share one design. Many of the best-known ones do build on a specific neural network design called the transformer architecture. The source for this section makes a direct point, however: LLMs need not all share that design. Some researchers are actively building language models with different internal machinery.

This article gives you the family map before the mechanics. You will start with the original transformer, see how its variants divide into two styles, look at the experimental alternatives that sit beside it, and then pin down a few supporting terms. Keep two questions in mind as you read: which architecture does each model use, and why is each reference included?

The transformer architecture was proposed in the paper titled Attention Is All You Need, published in 2017 by Vaswani et al. Every variant discussed in this section is a descendant of that original design. When you see the phrase original transformer, it refers to the architecture introduced in that paper.

Think of the original transformer as the root of a family tree. Researchers took that design and adapted it into several variants. The variants that matter most for this course are BERT and GPT-3, and knowing which variant is which is the first step toward reading the rest of the module with confidence.

The source groups the variants of the transformer into two styles. The name of each style tells you which kind of transformer it is, and the two styles are the main distinction to carry into the rest of the module. The diagram below shows how information moves through each style at a general level. It is a conceptual orientation, not a description of every internal layer.

Read the diagram from top to bottom in spirit: the original architecture sits at the top, and the two styles sit beneath it as the two branches. The left branch reads an input and produces representations of it. The right branch works from a prompt and produces the next token. The source uses one named model as the example for each branch.

The source gives each model a paper, a year, and a style. Placing these facts side by side makes the matching exercise concrete. Use the table to check which fact belongs to which model.

The source presents the three landmarks in order: the original architecture first, then the encoder-style BERT, then the decoder-style GPT-3. Following that order helps you see how the ideas arrived.

The source names three experimental designs that are not based on the transformer: RWKV, Hyena Hierarchy, and Mamba. Each reference is a 2023 paper, and each title hints at its core idea. The source uses the titles as clues. The hints point toward recurrent, convolutional, and state-space ideas respectively. Treat these as clues to be confirmed against the papers before relying on any detail.

The diagram places the transformer and the experimental group as siblings under the general heading of LLM architectures. It is a classification aid, not a ranking. The transformer is not the root of the tree, and the experimental group is not a lesser branch. They are two answers to the same question of how to build an LLM.

The rest of the section is a reading path. Each reference explains a term used elsewhere in the material. Three of them connect directly to GPT-style models and their training: Llama 2, The Pile, and InstructGPT.

Llama 2 is Meta AI's model. The source describes it as a popular implementation of a GPT-like model that is openly available, in contrast to GPT-3 and ChatGPT. The reference is Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) by Touvron et al.

The Pile is a publicly available dataset curated by Eleuther AI. The source calls it an 800GB dataset of diverse text for language modeling. It is the subject of the paper The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020) by Gao et al. Its role in the reading path is to connect to the dataset references mentioned earlier in the material.

InstructGPT is a fine-tuning method for GPT-3 that uses human feedback. Human feedback is not applied to GPT-3 as a single direct action. It guides a fine-tuning step, and that fine-tuning step starts from GPT-3 and produces a model that follows instructions.

Follow the chain from left to right: GPT-3 is the starting model, human feedback guides a fine-tuning step applied to it, and the result is a model that follows instructions. Llama 2 sits in the same GPT-like family as an openly available example, and The Pile is the public dataset reference that belongs to the language modeling material.

A vector representation is an ordered list of numbers. Each position in the list is a slot, and the whole list stands for one item. A short vector with four slots looks like a list of four numbers. The slot values in any example are placeholders, not real data.

The source also uses the terms embedding space and latent space. It notes that these are related concepts, and that the first chapter of its book discusses how they compare and how vector representations work in general. The source does not define either term, so treat the following only as orientation. A vector representation is one list of numbers. Embedding and latent space are the broader terms the source points you to for further study.