CPU and GPU Memory in AI Accelerators

Picture a kitchen where a cook can chop vegetables at great speed, but every ingredient has to be fetched from a storage room across the building. The cook spends most of the shift walking, not chopping. An AI accelerator faces a similar problem. Its arithmetic units can process numbers extremely quickly, yet they can only do useful work when the numbers they need are already within reach.

This article answers a practical question: when a model runs, where do its numbers live, and how long does it take to get them to the place where they are used? The short answer is that an accelerator does not have one memory. It has a layered arrangement in which each layer trades capacity for speed. Learning to read that arrangement is the first step toward understanding why some workloads run quickly and others stall.

An AI accelerator can draw on three memory tiers: CPU system memory, the GPU's own high-bandwidth memory, and on-chip memory built into the silicon. Each tier offers a different balance of speed and size. The three are described one at a time below, and then compared side by side.

Position matters as much as speed. The on-chip tier is inside the same silicon as the compute units, so it is the closest. GPU HBM is a separate pool dedicated to the GPU and placed near it, so it sits one step farther out. CPU system memory is the most distant: the accelerator reaches it through the connection to the host CPU. Read the diagram below from the compute outward. Each step adds capacity and adds distance.

Two properties describe each tier, and they are separate. Bandwidth is how fast data can be read or moved. Capacity is how much data the tier can hold. A chip can be strong on one and weak on the other, and both matter when judging whether a chip suits a memory-heavy workload. The table below collects the source's typical figures so the three tiers can be compared directly.

Read across the table and a pattern appears: every step toward the compute increases speed and reduces room. The on-chip tier can be read at the highest rate but is measured in tens of megabytes, while CPU memory can hold tens or hundreds of gigabytes at a fraction of that rate. Designers cannot simply make the fastest memory bigger without changing the chip. That constraint is why the hierarchy exists at all.

Speed numbers become meaningful when you watch a specific piece of data move. The scenario below is an invented illustration, not a measured benchmark. It uses the tier characteristics from the earlier sections to show how the same chunk of data can be far from the compute at one moment and close to it the next.

The slowdown is most likely to come from the outer edge of the path. The first transfer out of CPU system memory is the slowest link, and any data the compute units need that has not yet been staged into the fast tiers forces a wait. A workload that fits comfortably in the fast tiers behaves very differently from one that spills into the slow ones.

Because the hierarchy determines speed, a large share of GPU optimization is about keeping useful data in the fast tiers for as long as possible. The source notes that, at the time it was written, mainstream frameworks such as PyTorch and TensorFlow did not yet offer fine-grained control over memory access. That gap pushed many engineers toward lower-level GPU programming languages, including CUDA, OpenAI's Triton, and ROCm. CUDA was originally called Compute Unified Device Architecture. ROCm is AMD's open source alternative to NVIDIA's proprietary CUDA.

The same reasoning applies when choosing an accelerator. Raw compute tells you how fast the arithmetic can run, but it does not tell you whether the numbers will be in reach. Bandwidth and capacity are separate properties of memory, and both matter when judging whether a chip suits a memory-heavy workload. A chip with strong compute but small or slow memory can stall on the same chunks of data that a better-balanced chip handles smoothly.