Fine-Tuning Large Language Models with Human Feedback

Imagine you have built a language model that has already gone through the earlier work of this course. It can produce fluent text, but it has not been shaped for any particular job. Ask it to sort customer emails into categories, or to answer a specific question about a product, and it has no particular reason to do either task well. The practical question this article asks is simple: once a pretrained model exists, how do you turn it into one that performs a specific job?

The answer this course gives is fine-tuning. Fine-tuning is the step that comes after pretraining, and it is where the model starts to be aimed at real work. The source describes it as adjusting the pretrained model so that it follows instructions for named tasks. Keep the starting point in mind as you read the rest of the article, because fine-tuning does not begin from an empty model. It begins from a pretrained one.

Most of the language models that a learner meets first are built on the transformer architecture, the design used by GPT-style models. The source makes a point that is easy to miss: this design is one successful choice, not a requirement. It describes a group of experimental LLM architectures that are not based on the transformer. The word experimental here describes architectures that are less popular and are still being explored by researchers. It does not mean the models are unreliable or unused.

The diagram below places the transformer group and the experimental group side by side under the general heading of LLM architectures. The transformer appears as one design, and the three named experimental models appear as members of a second group. Read the diagram as a classification aid, not a ranking. The transformer is not the root of the tree, and the experimental group is not a lesser branch. Both groups are two answers to the same question of how to build an LLM.

The source names three experimental designs: RWKV, Hyena Hierarchy, and Mamba. Each reference is a 2023 paper, and each title carries a clue about the core idea. Reading the names as clues is a useful habit, because the name tells you where to look inside the design. The source does not explain the internals in this section, so the clues below are orientation only and should be confirmed against the paper before you rely on any detail.

The rest of this section is a reading path. Each reference points to a source that explains a term used elsewhere in the material, and each one contributes a different piece to the picture of GPT-style models and their training. Follow the path in order and track what each reference adds. The three entries below show how the references connect to GPT-style models.

The course builds an LLM in stages, and the third stage is the one this article covers. In that stage, the course takes a pretrained LLM and fine-tunes it for specific work. Read the sentence as a sequence: the pretrained model comes first, and fine-tuning is applied to it afterward. The diagram below shows that order as a flow of data from the pretrained model, through the fine-tuning step, to a model ready for specific tasks.

The two halves of that definition are worth separating. The first half names the starting object and the action performed on it. The second half names the purpose. A learner who keeps both halves in mind is less likely to confuse fine-tuning with building a model from nothing.

The easiest way to separate a pretrained LLM from a fine-tuned LLM is to ask two questions: where did this model start, and what is it for? The diagram below shows the same model before and after fine-tuning. In the before group, the pretrained model is the end point of earlier work, aimed at fluent text in general. In the after group, the fine-tuned model is produced from the pretrained model and is aimed at specific tasks.

A vector representation is an ordered list of numbers. Each position in the list is a slot, and the whole list stands for one item. The diagram below shows a short vector with four slots. The values in the slots are placeholders, not real data, and the slots are read from left to right by their position.

Embedding spaces and latent spaces are related concepts that the source points to a chapter of its book to explore. This section does not define either term, so treat the following as general orientation rather than a summary of the source. An embedding is a vector representation of an item, and a latent space is the collection of such vectors that a model works within. In this sense, a vector is the basic unit, an embedding is one kind of vector representation, and a latent space is the wider space those vectors live in.

The source names two example tasks for a fine-tuned model: answering queries and classifying texts. It also explains why they matter: they are among the most common tasks in real-world applications and in research. The two tasks ask different things of the model. Answering a query asks the model to produce a response to a question. Classifying a text asks the model to assign an input to a category. Both are ways of following an instruction, which is the behavior the fine-tuning stage aims to build.