Memory Requirements for Serving Models

Serving an AI model is a resource problem. Accelerators such as GPUs are built to do many calculations at once, and that property is what makes them useful for serving models. It also gives engineers a set of choices when a service has to grow. Parallelism strategies are the family of methods that spread work across several chips or machines so the service can keep up with demand, or so it can run a model too large for one device.

Serving problems usually arrive in one of two forms. Sometimes there are too many requests, and each request is quick enough on its own, so the goal is to answer more of them at the same time. Other times the model itself is too large to fit on one device, so the goal is to split the model. Some strategies also respond to a third pressure, which is very long input sequences. The rest of this article follows these pressures one at a time, and it tracks where the memory and the work end up in each case.

Replica parallelism is the simplest strategy to put into practice. You make several identical copies of the model you want to serve. Each copy handles its own requests, so the service as a whole can process more requests at the same time than a single copy could.

Trace one request through a replicated service. A request arrives, it is routed to one replica, and that replica runs the full model from start to finish. A second request can be routed to a different replica and run at the same moment. Neither request runs through a shorter path than it would on a single copy. What changes is how many requests are in progress at once.

The trade-off is hardware cost. Every extra replica needs its own chips to run on. More replicas let more simultaneous requests be handled, but the number of chips you pay for grows with them. Replica parallelism therefore scales throughput by buying capacity rather than by making any one request cheaper or faster.

Once you create replicas, or serve several models at once, you face a placement question. You have a fixed number of chips, and you must decide how many replicas of each model to create and which chip hosts each replica. Chips also differ in how much memory they have, so a model that fits on one chip may not fit on another.

This placement question is called the bin-packing problem. Each chip is a container with a fixed capacity. Each model replica is an item that must be placed. The goal is to arrange the items so they fit without overflowing. The problem gets harder as you add more models, more replicas, and more chips, because the number of possible arrangements grows quickly.

Start with the arithmetic. Assume every model is stored at 8-bit precision, which means each parameter takes one byte. A 13B model therefore needs roughly 13 GB just for its weights. This estimate ignores extra memory used at runtime, so a real placement needs more headroom than the raw numbers suggest.

The diagram shows one arrangement that works, and it leaves free space on every chip. The free space is not wasted by accident. It is headroom for runtime memory, which the simple arithmetic ignores. Other arrangements are possible, and choosing among them is the bin-packing problem. A 70B model cannot go on a 24 GB, 40 GB, or 48 GB chip at all, so some chips are simply ruled out for some models before any packing choice is made.

Sometimes a model is so large that no single machine can hold it. Model parallelism addresses this by splitting the same model across multiple machines. Each machine holds part of the model, so the model as a whole can run even though no single machine could host it alone.

Model parallelism makes placement harder. You are no longer deciding only which chip gets which whole model. You also decide how to cut the model into pieces and how those pieces fit across chips. There are several ways to make the cuts. The two most important for inference are tensor parallelism and pipeline parallelism.

Context parallelism and sequence parallelism are less common than the strategies above, but both were developed to make the processing of long input sequences more efficient. The two names sound similar, so it helps to remember what each one splits. Context parallelism splits the input. Sequence parallelism splits the operations.

In context parallelism, each device receives a different portion of the input sequence and processes that portion separately. For example, the first half of a long input might be handled on one machine while the second half is handled on another.

In sequence parallelism, the input is not cut into portions. Instead, the operators needed for the whole input are divided. Every part of the input may still need several kinds of computation. If the input requires both attention and feedforward computation, one machine might carry out the attention step while another carries out the feedforward step, with the whole input passing between them.

Each strategy answers a different pressure on a serving system. Match the strategy to the problem you are trying to solve, then check the trade-off you accept in return.