Datasets for Language Modeling
When people talk about large language models, they often assume one internal design.
Must Every Language Model Be a Transformer?
When people talk about large language models, they often assume one internal design. The transformer is the design most learners meet first, and the reference material for this section makes a deliberate point: the transformer is one successful design among several. Some researchers are actively building language models with different internal machinery. This article follows that idea step by step, tracking which architecture each model uses and why each reference is included.
What do you think happens?
Does a large language model have to be built on the transformer architecture?
Reveal answer
No, LLMs need not all share the transformer design
The source uses experimental LLM architectures, which are designs not based on the transformer, to show that LLMs do not all have to share that design.
What Experimental Means in This Section
An experimental LLM architecture is a design that is not based on the transformer architecture. The source calls these architectures experimental but less popular. In this section, the word experimental describes the design's status relative to the mainstream transformer.
Transformer and Experimental Families Side by Side
The source organizes the ideas as a classification aid. The transformer group and the experimental group sit beside each other under the general heading of LLM architectures. Neither is the root of the tree, and the experimental group is not a lesser branch. Both groups are two answers to the same question: how to build a large language model.
The diagram places the transformer in one column and the three named experimental designs in another. Read it as a map of groups, not a ranking. The source does not give the internal mechanics of each experimental model in this section, so treat the diagram as a statement about classification only.
Reading the Three Names as Design Clues
The source names three experimental designs: RWKV, Hyena Hierarchy, and Mamba. Each reference is a 2023 paper, and each title hints at its core idea. The source's own wording is that the titles hint at recurrent, convolutional, and state-space ideas respectively. Treat these hints as clues to the design, then confirm them against the paper before relying on any detail.
| Model | What the name signals | Design idea hinted at |
|---|---|---|
| RWKV | A name pointing toward a recurrent design | Recurrent |
| Hyena Hierarchy | A name pointing toward a convolutional design | Convolutional |
| Mamba | A name pointing toward a state-space design | State-space |
Design clues from the names, to be verified against each paper
Tracing the Reference Path from Llama 2 to InstructGPT
The rest of the section is a reading path. Each entry points to a source that explains a term used elsewhere in the material. Llama 2 is Meta AI's model, described as a popular implementation of a GPT-like model that is openly available, in contrast to GPT-3 and ChatGPT. The reference is Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) by Touvron et al.
The Pile is a publicly available dataset curated by Eleuther AI. The source calls it an 800GB dataset of diverse text for language modeling. It is the subject of the paper The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020) by Gao et al. The source links this paper to dataset references mentioned earlier in the material.
The InstructGPT entry describes a sequence of influences. Human feedback is not applied to GPT-3 as a single action. It guides a fine-tuning step, and that fine-tuning step starts from GPT-3 and produces a model that follows instructions. The diagram lays out this flow from left to right so you can track where each input enters and what comes out. The Pile and Llama 2 appear as separate references: The Pile is a dataset, and Llama 2 is an openly available GPT-like model.
Tracing where human feedback enters the InstructGPT flow
Using only the source's description, identify which step human feedback enters, what that step starts from, and what it produces.
- 1Locate the input
Human feedback is described as guiding a fine-tuning step. It is not described as a separate model or a direct edit to GPT-3.
- 2Locate the starting point
The fine-tuning step starts from GPT-3. GPT-3 is the base that the fine-tuning modifies.
- 3Locate the output
The fine-tuning step produces a model that follows instructions, which the source names InstructGPT.
Human feedback guides a fine-tuning step that starts from GPT-3 and produces InstructGPT, a model that follows instructions.
Vector Representations as Ordered Lists
A vector representation is an ordered list of numbers. Each position in the list is a slot, and the whole list stands for one item. The slots are ordered, so the first slot and the second slot are different positions even when their values match. The diagram below shows a short vector with four slots. The slot values are placeholders, not real data, and the source does not give specific values.
The four boxes are one vector, not four separate items. Reading the vector means reading all slots in order. If you swapped two slot values, you would have a different vector even though the same numbers appear.
Vector, Embedding Space, and Latent Space
The source notes that embedding spaces and latent spaces are related concepts, and it points to the first chapter of its book for how they compare and how vector representations work in general. This section does not define either term. The following is a general orientation rather than a summary of the source: a single vector is one item, while an embedding space or latent space is a wider set in which such vectors live. Check the source chapter before using these terms precisely.
Mistakes Learners Make with These Terms
Common mistake Treating experimental as meaning the model is unreliable or unused
Why it failsThe source says experimental describes architectures that are less popular and not based on the transformer. It does not make a quality claim.
Do this insteadRead experimental as a status relative to the mainstream transformer design.
Common mistake Assuming the titles of RWKV, Hyena Hierarchy, and Mamba fully explain their mechanisms
Why it failsThe titles only hint at recurrent, convolutional, and state-space ideas, and the source asks readers to confirm details against each paper.
Do this insteadUse the names as clues and verify the mechanism in the paper.
Common mistake Thinking that The Pile is a model
What it looks likeListing The Pile next to Llama 2 as if both were GPT-style models.
Why it failsThe source describes The Pile as a publicly available dataset curated by Eleuther AI, and Llama 2 as a GPT-like model.
Do this insteadKeep datasets and models in separate categories when you trace the references.
Common mistake Treating a vector, an embedding space, and a latent space as the same thing
Why it failsThe source calls these related concepts. A vector is an ordered list of numbers, while the other two are wider terms that the source does not define in this section.
Do this insteadKeep the single item separate from the broader spaces, and check the source chapter for precise definitions.
Practice Tracing the Reference Path
Without looking back, write one sentence for each of these: (1) why an LLM does not have to be a transformer, (2) what the word experimental means in this section, (3) how human feedback connects to GPT-3 in InstructGPT, and (4) what a vector representation is. Then compare your sentences against the sections above.
Hints
- Recall the statement that experimental architectures are designs not based on the transformer.
- The InstructGPT flow has three parts: GPT-3, human feedback, and a fine-tuning step.
- A vector representation is an ordered list in which each position is a slot.
Key Takeaways
- Not every large language model has to be built on the transformer architecture; experimental LLM architectures are designs not based on the transformer, and the source uses them to make this point.
- RWKV, Hyena Hierarchy, and Mamba are the three named experimental alternatives, and their titles hint at recurrent, convolutional, and state-space ideas, which must be confirmed against each paper.
- Llama 2 is an openly available GPT-like model, The Pile is a publicly available 800GB dataset curated by Eleuther AI, and InstructGPT is produced by fine-tuning GPT-3 with human feedback.
- A vector representation is an ordered list of numbers in which each slot is a position; embedding space and latent space are related but distinct terms that the source points to a book chapter to explore.