Evolutionary Strategy for Training Neural Networks
Memory calculation is crucial for both model training and model inference, and the two are treated as separate calculations.
Why Memory Planning Comes First
Training and running a neural network both consume memory. A model that does not fit in the memory available to it cannot be trained or served. Engineers sometimes learn this the hard way. A lighthearted saying in the field holds that you are not really doing AI until you have seen the error message 'RuntimeError: CUDA out of memory'. The joke points at a real experience: running out of memory is a common way people discover that their plans did not account for the resources a model needs.
What do you think happens?
If a model runs comfortably for inference on a machine, can you assume the same machine will be able to train it?
Reveal answer
No. Training and inference are separate memory calculations, so you cannot assume that a setup which works for one will work for the other.
The source treats memory calculation as crucial for both training and inference, and it treats the two as separate problems, each with its own reference material. Before estimating anything, you need to know which activity you are planning for.
Training and Inference Are Separate Calculations
The source organizes memory calculation into two subconcepts. Inference memory calculation concerns running a model. Training memory calculation concerns training it. The two are different problems, so a planner should identify which activity is being planned before doing any estimate.
The diagram shows the structure the source establishes. Each activity has its own calculation and its own reference. The source does not walk through the arithmetic itself, so this article does not present a memory formula. What it does establish is that the planner must decide, first, whether the question is about running the model or training it.
Deciding Which Memory Calculation to Use
A team wants to use a neural network that already exists in two ways: first to answer user requests in a product, and second to adapt it to a new dataset. Which memory calculation applies to each plan?
- 1Identify the activity for the first plan
Answering user requests means running the model, so this is an inference plan.
- 2Apply the inference memory calculation
The source directs inference planning to the inference reference, Transformer Inference Arithmetic.
- 3Identify the activity for the second plan
Adapting the model to a new dataset is further training, so this is a training plan. Finetuning is training work, as the next section explains.
- 4Apply the training memory calculation
The source directs training planning to the training reference, Transformer Math 101. The second plan carries training-side memory costs even though the model already exists.
The first plan uses the inference memory calculation. The second plan uses the training memory calculation, because adapting the model is training work.
Finetuning as Further Training
Finetuning is further training of a pre-trained model on a specific task or dataset. You begin with a model that has already been trained, then continue training it on data that targets a particular purpose.
Skipping Gradients for Frozen Parameters
Trainability connects directly to the cost of an update. The source states that if a parameter is not trainable, it does not need to be updated. Because it is not updated, there is no need to compute its gradient. Gradients are only needed for the parameters that will change, so marking parameters as not trainable removes work for them.
Read the diagram as a tracing exercise. Both kinds of parameter pass through the same training process, but only the trainable branch does the gradient work. The frozen branch reaches the end of the process without a gradient ever being computed. This is the reason marking parameters as not trainable reduces the work involved in an update.
Numerical Formats in Memory Use
Memory use depends on how numbers are stored. The source names two numerical formats. It does not describe how many bits either one uses, so this article does not compare their storage sizes. The source points to outside resources for the detailed arithmetic.
Mistakes Beginners Make When Planning Memory
Common mistake Treating training and inference as one memory question
Why it failsThe source treats them as separate calculations, each with its own reference material.
Do this insteadDecide first whether the plan is for running the model or training it, then choose the matching calculation.
Common mistake Assuming finetuning is always required
Why it failsThe source notes that finetuning is not always strictly necessary.
Do this insteadConfirm that finetuning is actually needed before budgeting for its training-side memory costs.
Common mistake Believing a model that already exists needs no training memory planning
What it looks likeA team plans to adapt an existing model to a new dataset and budgets only for answering requests.
Why it failsFurther training on a dataset is training work, so it carries training-side memory costs.
Do this insteadPlan a training memory calculation for any adaptation step.
Common mistake Computing gradients for parameters that will never change
Why it failsA parameter that is not trainable does not need to be updated, so its gradient does not need to be computed.
Do this insteadMark parameters that should stay fixed as not trainable so their gradients are skipped.
Common mistake Discovering memory limits only when the run crashes
What This Source Does Not Cover
Practice Planning a Memory Budget
A colleague says, 'We already have the model running for inference, so we are done with memory planning. We only need to add a finetuning step on our own dataset.' Explain what is wrong with this reasoning, and name the two things your plan must cover. Use the terms from this article.
Hints
- Ask whether finetuning is inference work or training work.
- Recall which memory calculation the source associates with training.
- Think about whether every parameter in the model needs to change during finetuning.
Key Takeaways
- Memory calculation is crucial for both model training and model inference, and the two are treated as separate calculations, each with its own reference material.
- Finetuning is further training of a pre-trained model on a specific task or dataset. It is not always strictly necessary, and it carries training-side memory costs.
- A parameter that is not trainable does not need to be updated, so its gradient does not need to be computed.
- The source names BFloat16 and integer (fixed point) formats as relevant to memory use, but it does not give their bit layouts or describe the evolutionary strategy method named in the title.