Training Versus Inference Workloads

Two programs can run the same model and still feel very different in speed and running cost. Part of the reason is the software, but a large part is the hardware underneath it. Some optimization techniques work on any machine, while others only pay off once you understand the machine you are running on. This article starts from the hardware side: it looks at the chips that run AI workloads and asks what those chips are built to do.

The link between models and hardware is older than the current boom. A shortage of sufficiently powerful computers was one factor in the first AI winter in the 1970s. The return of interest in deep learning in 2012 was also closely tied to compute. Keep this in mind as you read: the question of which chip runs a workload has shaped which AI research was practical at all.

The word specific is the important part of the definition. A general-purpose processor is built to handle a wide range of tasks. An accelerator gives up some of that breadth in exchange for doing one kind of work well. An AI accelerator is tuned for the arithmetic that AI models repeat, and that focus is the reason it exists alongside the general-purpose processor rather than replacing it.

A historical episode shows why this matters. Before the AlexNet paper, training a model at AlexNet's scale generally meant running on thousands of CPUs. One commonly cited reason for that paper's influence is that it was the first to successfully use GPUs, graphics processing units, to train neural networks. A couple of GPUs were far more affordable for PhD students and researchers than thousands of CPUs, and that accessibility helped set off the deep learning research boom.

The central difference between a CPU and a GPU is their purpose. CPUs are designed for general-purpose use, meaning they are meant to handle a wide mix of tasks. GPUs are designed for parallel processing, meaning they are meant to do many pieces of work at the same time. Most of the other differences follow from that single choice.

A CPU has a few powerful cores. High-end consumer machines can have up to 64 of them. Many CPU cores handle multi-threaded workloads well, but they are at their best when a task needs high single-thread performance. Running an operating system, managing input and output operations, and handling complex sequential processes all fall into this category.

A GPU takes the opposite approach. It has thousands of smaller, less powerful cores. Those cores are optimized for work that can be broken into many small, independent calculations, such as graphics rendering and machine learning. Each core is weaker than a CPU core, but the sheer number of them is what matters when the work splits cleanly.

Now imagine a batch of independent matrix entries arriving at each kind of chip. On the CPU, the batch is spread across a small number of strong workers. Each worker takes a share of the entries and handles it quickly, so the batch is finished by a handful of workers rather than a crowd.

On the GPU, the same entries are spread across thousands of smaller cores. Each core is less powerful, but because the entries are independent, none of those cores has to wait for another. At any moment, many entries are in progress together. The state of the batch changes from a few entries being worked on slowly to a large set being worked on at once.

GPUs are optimized for tasks that can be broken into many smaller, independent calculations. The operation that makes up most machine learning workloads is matrix multiplication, and it is highly parallelizable. The key property is independence: if one piece of the answer does not need another piece's result, the two pieces can be worked out at the same time on different cores.

Read the formula as a recipe for one output cell. The entry in row i and column j of the result C is built from row i of A and column j of B. Two different output cells can be computed without knowing each other's results, which is exactly the independence the source describes.

The same logic scales up. A real model multiplies matrices far larger than two by two, which means far more independent entries. This is why a chip with thousands of cores can keep many of them busy on one multiplication.

Specialized compute units exist in hardware because different computations work on different shapes of data. The source identifies three shapes. A scalar is a single value. A vector is an ordered list of values. A tensor generalizes these to higher-dimensional arrays.

Compute units are specialized for scalars, vectors, or tensors, and chips often mix them. GPU tensor cores and TPUs are built around tensor operations, which is one reason the matrix arithmetic of AI models maps well onto them. Different hardware architectures have different memory layouts and specialized compute units that evolve over time, and these units are optimized for specific data types.

Many chips can handle both training and inference, but one clear trend is the rise of chips specialized for inference. Training builds a model, while inference means running a trained model to produce outputs. A survey by Desislavov and colleagues (2023) reports that inference can exceed the cost of training in commonly used systems, and that inference accounts for up to 90 percent of machine learning costs for deployed AI systems. For anyone deploying models, inference hardware becomes a central decision.

A chip's specification sheet holds many details, and only some of them matter for a given use case. Across use cases, three characteristics stand out: computational capability, memory size and bandwidth, and power consumption. Computational capability describes how much work the chip can do. Memory size and bandwidth describe how much data it can hold and how quickly that data can move. Power consumption describes the energy it draws while doing that work.

The three characteristics should be weighed together rather than one at a time. A chip with the most computational capability may draw too much power for a deployment that cares about running cost, and a chip with plenty of memory bandwidth may still be a poor fit if the workload does not move much data. The workload determines how much weight each characteristic deserves.