Batch Size and Gradient Accumulation

It is easy to assume that somewhere there is a correct learning rate, a number that works for any finetuning job. Tutorials often show one specific value, and it feels natural that a best setting should exist, the way a recipe has a best oven temperature. The source explains why this assumption trips learners up: the learning rate that works depends on the model, the data, and the stage of training. Instead of searching for a magic number, you find a workable one by experimenting within a typical range and reading the loss curve.

Finetuning starts from a base model that has already been trained, then adjusts it for a specific use case. Along the way you make many choices that are not learned from the data. These settings are called hyperparameters. The exact list depends on the base model and the finetuning method, and the documentation for your model or framework is always the authoritative reference. This article focuses on four hyperparameters that come up again and again and strongly influence how efficiently finetuning works: the learning rate, the batch size (together with gradient accumulation), the number of epochs, and the prompt model weight.

The learning rate sets how fast the model's parameters change with each learning step. A helpful picture is learning as a walk toward a goal, where the learning rate is the length of each step. Short steps are safe but slow: if the steps are too small, reaching the goal takes too long. Long steps cover ground quickly, but they can carry you past the goal. If the steps are too big, the model can overshoot and, as a result, may never converge.

Experiments are much easier to interpret when you watch the loss curve, because its shape tells you how the step length is behaving. The source gives two diagnostic patterns. If the loss curve fluctuates a lot, the learning rate is probably too big, because each update overshoots and the loss bounces around. If the loss curve is stable but takes a long time to decrease, the learning rate is probably too small, because each update moves the model only a little. The guidance that follows is to increase the learning rate as high as you can while the loss curve remains stable.

Because the right step length depends on the situation, the source recommends experimenting across the typical range, which is roughly 1e-7 to 1e-3, and seeing which value works best for your run. The range is wide, so the search needs a starting point. The source offers one: take the learning rate used at the end of the pre-training phase and multiply it by a constant between 0.1 and 1. That gives you a sensible neighborhood to search instead of a blind guess.

The intermediate values in the diagram are illustrative points spaced across the range. The source does not name a specific winner, so each run's loss curve decides which candidate to keep. Try a value, read the curve, then move toward a larger rate if the curve is stable and too slow, or back off if it fluctuates.

The learning rate does not have to stay fixed during training. You can use larger learning rates at the beginning and smaller learning rates near the end. An algorithm that determines how the learning rate should change throughout training is called a learning rate schedule. A schedule is therefore a rule that maps training progress to a learning rate, and it lets you act on the diagnostic patterns from the loss curve as training moves forward.

The batch size determines how many examples the model learns from in each step before it updates its weights. Batch size affects both stability and speed, and it is limited by the hardware you have. A batch that is too small, such as fewer than eight examples, can lead to unstable training. A larger batch helps aggregate the signals from different examples, which produces more stable and reliable updates.

Bigger is not free. In general, a larger batch lets the model get through the training examples faster, but it also needs more memory to run. Because memory is finite, batch size is capped by the hardware you use. This is the cost versus efficiency trade-off: more expensive compute allows faster finetuning. The source notes that compute remains a bottleneck for finetuning, since models are often so large and memory so constrained that only small batch sizes fit. Small batches can produce unstable weight updates, and that tension is what gradient accumulation is designed to address.

Gradient accumulation changes when the weights get updated, not how many examples each batch holds. Instead of updating the model weights after every batch, you accumulate gradients across several batches and update the weights once enough reliable gradients have been collected. The weight update is therefore based on information gathered from several batches, which helps when each individual batch is too small to give a stable update on its own.

Read the diagram as a flow with a pause in the middle. Each batch contributes its gradients to a running total, and the weights change only at the end of that stretch. The batches still flow through the model one at a time, so the memory limit that forced small batches still applies, but the update you finally make reflects several batches rather than one.

An epoch is one pass over the training data, so the number of epochs determines how many times each training example is trained on. Small datasets often need more epochs than large ones. For a dataset with millions of examples, one or two epochs might be sufficient. A dataset with thousands of examples might still improve after four to ten epochs.

The gap between training loss and validation loss gives you hints about the right number of epochs. The comparison below summarizes the two patterns the source describes.

In instruction finetuning, each training example consists of a prompt and a response, and both can contribute to the model's loss during training. At inference time, however, users usually supply the prompt, and the model only needs to generate the response. Because of this, response tokens should contribute more to the loss than prompt tokens. The prompt model weight is the setting that controls how much prompts count toward the loss compared with responses.

The two extremes are instructive. At 100 percent, prompts contribute to the loss as much as responses, so the model learns equally from both. At 0 percent, the model learns only from responses. The default is typically 10 percent, which means the model learns a little from prompts but mostly from responses.