Memory Hierarchy for Deep Learning Workloads

Imagine a product page that advertises an accelerator capable of several hundred trillion operations every second. It is tempting to read that figure as the speed at which your model will run. That reading is usually too optimistic. To judge an AI accelerator sensibly, you need to answer three questions: what the advertised number actually measures, how it changes when you change the number format your model uses, and what else the chip needs in order to keep its processing units busy. This lesson follows those three questions in order, and along the way it traces what happens to a piece of data as it travels from memory to the arithmetic units.

Because FLOP/s is a rate, a larger value means the chip can, in principle, push through more arithmetic in the same amount of time. A rating of this kind describes a ceiling. It says what the hardware could do under ideal conditions, not what a particular program will do on a particular day.

The peak figure is a theoretical maximum, and the gap between that ceiling and real performance matters a great deal. An application rarely reaches the peak because real workloads spend time waiting for data, moving results around, and handling steps that are not pure arithmetic. Every time a processing unit sits idle while it waits for its next operand, the achieved rate falls below the peak.

The ratio of the throughput an application actually achieves to the theoretical peak is a useful utilization measure. It tells you what fraction of the chip's capacity your workload is really using. A utilization of one would mean the workload delivers the full peak rate; a lower value means that part of the chip's capacity is going unused.

The number of operations a chip can complete per second depends on the precision of the numbers it works with. Precision here refers to how finely a number is represented. A format that stores more detail for each value requires more work for each operation, so fewer operations fit into each second. Consider adding two 32-bit values compared with adding two 16-bit values: the wider format involves more work per addition.

The relationship is not a clean proportion. The count does not simply double each time the width is halved, because chips are optimized differently for each format. The specific ratings in the H100 SXM specification table are quoted in teraFLOP/s with sparsity, and they should be read directly from that table rather than estimated from the bit width. The lesson for practice is that switching to a narrower format can raise the advertised peak substantially, but the gain is a property of the chip's design for that format, not a simple arithmetic consequence of fewer bits.

A GPU contains many cores that operate in parallel. Those cores can only do useful work when the data they need arrives in time, so the speed at which data moves from memory to the cores becomes a limiting factor. AI workloads make this worse, because they involve very large weight matrices and training datasets that must be moved quickly to keep the cores fed. An accelerator that can perform many operations per second will sit partly idle if its memory cannot supply operands fast enough.

Memory is not a single pool. The accelerator interacts with several tiers of memory. The tiers closest to the compute units are the smallest and fastest, such as registers and on-chip caches. Larger and slower tiers sit further away, and high bandwidth memory, usually called HBM, is the large off-chip store that holds the weights and data a model needs. Data moves down the hierarchy when it is stored and back up toward the cores when it is needed for arithmetic.

Two numbers describe an accelerator's memory: its size and its bandwidth. For the feeding problem, there is a second property that matters just as much, which is latency. Bandwidth is how many bytes can be moved per second. Latency is how long a single access has to wait before its data starts to arrive. The two are different properties, and an accelerator needs both.

Accelerator memory must deliver high bandwidth and low latency. High bandwidth lets large weight matrices and datasets stream to many cores at once. Low latency keeps each individual request from stalling the cores while they wait. Meeting both demands requires advanced memory technology such as HBM.

Meeting the demands of high bandwidth and low latency requires more advanced memory technology, and that technology costs more. This is one reason GPU memory is more expensive than the memory attached to a CPU. The contrast between the two designs is easiest to see side by side, with the caveat that the source material describes the cost difference qualitatively rather than giving per-byte prices.