Memory Hierarchies in Computer Architecture

Imagine two accelerator chips are advertised with very large numbers, and you must choose one for a model that has to answer user requests quickly. The printed figures tell you something, but not everything. Two questions matter most. The first is how many arithmetic operations the chip can complete each second. The second is how quickly it can feed data to the processing units that perform those operations. This article takes those two questions in turn, so that you can read an accelerator specification with the right questions in mind.

The two questions are linked. A chip with enormous arithmetic capacity still delivers little if its data arrives late. Reading a spec well means asking how the compute figure and the memory figures interact inside the full system the accelerator belongs to.

Scale matters when reading these figures. Modern accelerators are quoted in teraFLOP/s, where one tera is a trillion. A figure in the hundreds or thousands of teraFLOP/s therefore describes hundreds of trillions up to quadrillions of operations every second. A number like that is only meaningful when you also know which precision format it was measured with, and the next section explains why.

Numerical precision is the amount of detail used to represent each number during computation. A format that stores more detail needs more bits per value, and the circuitry that handles those bits does more work for each operation. Compare adding two 32-bit numbers with adding two 16-bit numbers. The 32-bit addition has more bits to process, so a chip running at a fixed rate completes fewer 32-bit operations than 16-bit operations in the same interval.

Read a table of throughput figures as a pattern rather than a list to memorize. Formats with fewer bits per value tend to appear with larger throughput figures. Some formats can also share the same listed figure even though they lay out their bits differently. The practical lesson is that a throughput number is comparable to another only when both were measured at the same precision.

A GPU contains many cores that operate in parallel. Each core needs operands to compute on, and those operands usually reside in memory outside the cores. Data must therefore travel from memory to the cores at a rate that keeps them busy. AI workloads make this demanding, because models contain large weight matrices and training passes through large volumes of data.

Follow one workload step by step. Operands start in memory. They move toward the processing units. The cores perform arithmetic on them. Results are produced and the next operands are requested. At each handoff, the question is whether the next item is already in place. If data arrives more slowly than the cores can consume it, the cores sit idle, and the chip's compute capacity goes unused no matter what its peak figure says.

To decide which resource limits a workload, ask a sequence of questions. The diagram below turns that reasoning into a decision path. Start with the workload. Then ask whether the cores are waiting for operands. If they are, memory bandwidth is the constraint. If the data keeps up, the processing units themselves set the limit.

An accelerator's memory is described by two numbers. Size is how much data the memory can hold. Bandwidth is how fast that data can be moved. Neither number is meaningful alone. A large memory with slow transfer leaves cores waiting. A fast memory that is too small to hold the model's weights forces data to be shuffled in and out repeatedly. Both figures must be read within the full system.

GPU memory must provide high bandwidth, meaning a large amount of data moved per unit time, and low latency, meaning a short delay before a requested item begins to arrive. Meeting both requirements calls for more advanced memory technology than the memory typically attached to a CPU. That engineering is one reason GPU memory costs more than CPU memory.

The accelerator also does not talk to a single memory. It works with a hierarchy of memory levels, each with its own size and speed, and data passes between those levels as the computation proceeds. When assessing a chip, ask where the working data sits at each step and how quickly it can move between those locations.

DDR SDRAM is the kind of memory typically attached to a CPU. HBM is the memory technology that GPUs favor. The reason is the demand described above. Because a GPU runs many cores in parallel, its memory must supply data with high bandwidth and low latency, and that requirement drives the use of HBM over DDR SDRAM. The comparison below summarizes the points this material establishes.