Data Preparation and Sampling for Language Models
The pretrained model is the shared starting point. Only the fine-tuning dataset decides whether the finished result sorts text into categories or responds as a personal assistant.
Deciding What Comes First
Once you understand the basic ideas behind large language models, the natural next question is how one actually gets built. The approach taken here is to start from the core idea behind GPT and use it as a blueprint for a model you write yourself. That blueprint is carried out in three stages, and each stage hands its result to the next one.
What do you think happens?
If you were building a model that can hold a conversation, which would you expect to come first: training it to chat, or deciding how its internal structure should look?
Reveal answer
Deciding how its internal structure should look comes first.
At the very start there is no trained model, only the intention to build one. Two things must exist before any learning can happen: a description of the model's architecture, and a body of data the model can learn from. Training only makes sense once those foundations are in place, so the order is not arbitrary.
The Three Stages as a Single Path
The three stages form a single path. Stage one covers data preparation and sampling, the attention mechanism, and the overall LLM architecture. Stage two pretrains the model on unlabeled data, producing what is called a foundation model. Stage three fine-tunes that foundation model so that it serves a specific purpose, such as acting as a personal assistant or classifying text.
The diagram shows the path as a sequence of handoffs. The first three steps all fall under stage one. The pretraining step opens stage two, and the training loop and model evaluation complete it. Stage three begins by loading the pretrained weights and then fine-tuning, and at that point the path divides into two possible finished results.
- Stage one: get ready by preparing data, implementing the attention mechanism, and defining the LLM architecture.
- Stage two: pretrain on unlabeled data to produce a foundation model.
- Stage three: fine-tune the foundation model so it serves a specific purpose.
The State Before Any Training
A useful way to think about the three stages is this: stage one builds the empty machine and gathers its raw material, stage two runs that machine over a large amount of general material, and stage three steers the already-capable machine toward one particular job.
Trace the state at the start. There is no trained model to work with, only the intention to build one. Two things have to exist before any learning can happen: a description of the model's architecture, and a body of data that the model can learn from. Both are prepared in the first stage. Training, which is the process of adjusting the model based on data, only makes sense once those foundations are in place.
Stage One: Getting Ready
Stage one is about getting ready. Its goal is to make sure the model has both a structure to learn inside and material to learn from. The stage has three parts, and each one is covered below.
Stage Two: Pretraining on Unlabeled Data
Stage two is pretraining. The model is trained on unlabeled data, meaning the training text does not come with annotations telling the model what each example is supposed to mean. The purpose of this stage is to produce a foundation model, a general-purpose result that can be specialized later. Pretraining is not yet about any single job; it is about building a broad base.
Worked Scenario: From Raw Text to a Foundation Model
Tracking the handoffs in one project
A small team wants to build a model that can hold a conversation. They have a large collection of general text with no annotations, and they also have a smaller dataset of instruction examples. Trace what each stage hands to the next.
- 1Stage one: prepare
The team defines the LLM architecture, including the attention mechanism inside it, and decides how the general text will be sampled so the model can be fed in a structured way.
- 2Stage two: pretrain
The model trains on the unlabeled general text. The team does not add annotations. The output is a foundation model, a broad base that is not yet a chat assistant.
- 3Stage three: fine-tune
The team loads the pretrained weights and fine-tunes on the instruction dataset. Because the data is instructions, the path leads toward a personal assistant rather than a classifier.
Each stage hands its output forward. Stage one hands over an architecture and sampled data, stage two hands over a foundation model, and stage three turns that foundation model into an assistant. Had the team used a dataset with class labels instead, the same foundation model would have become a text classifier.
Stage Three: Fine-Tuning Branches the Path
Stage three takes the foundation model from stage two and adjusts it for a particular application. Fine-tuning can produce two kinds of result named in the source material: a personal assistant or a text classifier. The kind of data used determines which one you get. A dataset with class labels leads toward a classifier that sorts text into categories, while an instruction dataset leads toward a personal assistant or chat model.
The pretrained model is the shared starting point. Only the fine-tuning dataset decides whether the finished result sorts text into categories or responds as a personal assistant.
Mistakes Beginners Make with the Three Stages
Common mistake Treating the foundation model as a finished chat application
Why it failsPretraining on unlabeled data produces a general-purpose base. It is not yet specialized for a particular job.
Do this insteadRecognize that a foundation model still needs stage three before it serves a specific purpose.
Common mistake Thinking the attention mechanism is a topic that appears only later
Why it failsAttention is one of the required pieces of the architecture, so it belongs to stage one.
Do this insteadUnderstand and implement attention as part of building the architecture, before training begins.
Common mistake Assuming any dataset will produce the same kind of fine-tuned result
Why it failsThe kind of data used determines the outcome. Class-labeled data leads toward a classifier, while an instruction dataset leads toward a personal assistant.
Do this insteadChoose the fine-tuning dataset according to the application you want.
Common mistake Starting training before the architecture and the data exist
Practice: Sorting Stages and Outcomes
A team has finished pretraining a model on unlabeled text and wants a text classifier that sorts support emails into categories. Which stage comes next, what kind of data should it use, and what does the team have at the end of that stage? Explain your answer using the stage that each step belongs to.
Hints
- Pretraining is stage two, so the next step is stage three.
- The kind of data used determines the result. Class-labeled data leads toward a classifier.
- The finished result of stage three is the application, built on the foundation model from stage two.
Key Takeaways
- Building an LLM happens in three stages: stage one implements the architecture and prepares data, stage two pretrains the model, and stage three fine-tunes it.
- Stage one covers data preparation and sampling, the attention mechanism, and the LLM architecture, and it must finish before training can begin.
- Pretraining on unlabeled data produces a foundation model, which is a general-purpose base rather than a finished application.
- Fine-tuning the foundation model yields a personal assistant when the dataset is instructions, or a text classifier when the dataset has class labels.