Inference Optimization for Large Models

When a language model does not behave the way a project needs, a tempting response is to train it further on your own examples. Finetuning sounds like the direct fix: the model learns your task, and the problem goes away. Many people reach for it early, often before they have tried anything else.

This article examines the belief that finetuning is always the first step to take when a model underperforms. That belief is usually wrong. The more careful sequence starts with prompting, moves to finetuning only when evidence says simpler methods have run out of room, and includes decisions about separate models and newer base models along the way.

Instead of asking 'should I finetune?' at the start, follow the state of the project one decision at a time. Begin with a model that underperforms. The first question is not about training. It is whether prompting has been carefully tested. If it has not, or if it has been tried only once or twice, the project stays on the prompting branch and the work is to refine and test prompt variants. Only when carefully tested prompting proves inadequate does the path lead toward finetuning.

Read the diagram as a series of gates. A failed prompt sends you to the first gate, which asks whether prompting has truly been tested. Only a 'Yes' there opens the way to finetuning, and even then the finetuning gate asks whether the evidence justifies the cost. The later gates about newer base models and separate models are reached only after the earlier decisions have been made honestly.

AI engineering experiments should begin with prompting, and that prompting work should follow sound practices. The reason is cost and evidence. Finetuning is expensive to set up and maintain, so it needs to be justified by proof that simpler methods have run out of room. Prompting is the cheapest way to find out how far a model can already go with the right instructions.

The phrase 'thoroughly tested' carries most of the weight in this advice. A model's behavior can shift considerably depending on how a request is worded, so one or two attempts tell you very little. A fair test means trying several variants of the prompt and checking each result against a clear measure of success. Without a clear measure, you cannot tell whether a variant helped.

The lesson is not that prompting always works. It is that a claim such as 'prompting does not work' should be checked against how thoroughly prompting was actually tried. Most failed prompting efforts trace back to a small set of weaknesses.

The steps of an AI engineering experiment happen in a fixed order. Each step is meant to be completed before the next one is considered. Prompting comes first, and finetuning becomes an option only at a later point, once the earlier steps have shown their limits.

Notice that the step of gathering annotated data appears only after prompting has been shown to fall short. Data collection is one of the largest up-front costs of finetuning, so placing it after the prompting test is what keeps the project from paying for training it did not need.

Finetuning requires high up-front investment and continual maintenance. These costs are easy to underestimate, because the training step itself often looks like the simplest part of the project. Costs fall into two groups: those paid before the model is useful, and those that continue for as long as the model is in use.

Finetuning is also a bet against a moving target. New base models are being developed quickly, and they may improve faster than you can improve a finetuned model. A model you spent months finetuning may be overtaken before the work pays off. That creates two decisions the team must answer in advance: whether a switch to a newer base model is worth it, and whether the work should be prioritized at all.

A switch to a better base model often yields only a small incremental improvement. In that case the task may deserve lower priority than projects with larger returns, such as enabling new use cases. Deciding how to spend effort is part of engineering judgment, not only a question of which model scores highest on a benchmark.

Sometimes a finetuned model improves one kind of query while harming others. A model might handle one type of query better after finetuning but perform worse on two other tasks. To avoid this, finetune on all the queries you care about, rather than making narrow adjustments that favor one task.

If you still cannot get one model to perform well across all your tasks, consider separate models for different tasks. If serving several models becomes inconvenient, they can be combined into one later through merging, a technique the course discusses in a later part of the same chapter. The order matters: separate models are considered after the team has seen that one model cannot serve every task, and merging is a later simplification, not a first step.

The course gives finetuning for a specialized domain its own treatment, which signals that domain work raises considerations beyond the general case. The costs of data, training knowledge, and serving apply with particular force here, because domain tasks often depend on expertise that is hard to label.