Calculating Model Memory Usage (Chapter 7)

When people talk about large models, the first number they usually mention is the size of the model. Size matters, but the source passage behind this article asks a different set of questions about the design and cost of a neural network. Which mathematical ingredients make a network work at all? How can training be understood? How do hardware and memory constrain a design? What do compute costs look like? Together, these questions explain why architecture and scale are chosen with care rather than by simply making a network larger.

The source is a short passage, so this article stays close to what it actually establishes. Where it makes a claim, we explain that claim. Where it does not go further, we do not fill the gap with outside detail. The detailed method for calculating a model's memory usage is handled in Chapter 7 of the source, and this article only sets up the key idea behind it.

The source explains that a feedforward layer by itself keeps the computation linear, and that the model needs a nonlinear function to break that linearity. The nonlinear step is what lets the network go beyond what a purely linear computation can express. The source does not require this function to be exotic. What matters is that the nonlinearity is present and that it is cheap enough to compute during training.

The reason linearity is a problem can be seen with a small generated example. Suppose one layer multiplies its input by 2, and a second layer multiplies its output by 3. Passing a number x through both layers gives 3 times (2 times x), which is 6 times x. The two layers together are no more powerful than one layer that multiplies by 6. Stacking more purely linear layers never escapes this pattern, which is why a nonlinear step is inserted between them.

Once you know that a nonlinear step is required, the next question is which one to use. The source states that simpler functions that are faster to compute are preferred, because sophisticated ones use too much training compute and memory. The source does not name specific functions, so the comparison below sticks to the trade-off it describes.

Gradient descent is introduced in the source as a technique to train neural networks. The source then offers a less familiar way of thinking about it. Neural networks are good at simulating many computer programs, and gradient descent can be understood as a search algorithm. It searches through all the programs a neural network can simulate in order to find the best one for its target task.

Read this diagram from left to right. The set of programs a network can simulate is the space being searched, the search is carried out by gradient descent, and the output is the program that best fits the target task. The diagram shows the flow of this idea. It does not describe the individual update steps of training, which the source does not spell out.

The source presents an argument, attributed to Ilya Sutskever, about why it is so hard to develop new neural network architectures that outperform existing ones. The argument rests on the search idea from the previous section. Because existing networks can simulate many programs, a new architecture can potentially be simulated by existing networks as well.

This changes how architecture work should be judged. A designer is not simply looking for a new shape of network. The real question is whether the new shape reaches a set of programs that the current shapes cannot reach. The source states that a new architecture must be able to simulate programs that existing architectures cannot in order to outperform them. For the full argument, the source points to a talk by Sutskever at the Simons Institute at Berkeley in 2023.

Architecture design is also influenced by the hardware that runs the model. The source notes that the transformer was originally designed by Google to run fast on Tensor Processing Units, usually called TPUs, and was only later optimized for Graphics Processing Units, or GPUs. The hardware a design targets therefore leaves a visible mark on the design itself.

The source links these pieces as a chain. The transformer connects to the TPU as its original target, and then to the GPU as a later optimization target. When you read an architecture, it helps to ask which hardware it was built to run on first, because that choice constrains what is fast, what is affordable, and what is practical to train.

The source states that the actual memory needed for a model is higher than the model's size. The memory budget therefore has to be planned against more than the bare size of the model. The source does not list every component of that extra memory. It says only that Chapter 7 discusses how to calculate a model's memory usage, so the detailed method belongs there.

An epoch is one full pass through the training data. The source notes that, as of its writing, large models are typically pre-trained on only one epoch of data. In other words, the common practice for large models is a single pass rather than many repeated passes over the same data. The source is careful about timing. The phrase as of this writing marks this as a snapshot of common practice at one point in time, not a permanent rule.

Compute is often described in FLOP/s, a count of floating-point operations per second. The source states that the FLOP/s count is measured in FP32, which is one floating point format. A FLOP/s figure should therefore always be read together with the format it was measured in. The source says floating point formats are discussed in Chapter 7, and it does not describe them here, so this article does not go into the individual formats. The practical point is that a FLOP/s number is tied to a stated format.

The source also gives a concrete price point for compute. As of its writing, cloud providers offer H100 GPUs for around 2 to 5 dollars per hour, and the source adds that compute is getting rapidly cheaper, so this number is expected to fall considerably. The price is a dated figure and should be read as a snapshot, not as a current quote.