Cost Per Request for Inference Services

Imagine two inference services running on identical hardware. The first finishes a large number of requests every minute, and the second finishes fewer. If you look only at that first number, the first service seems better. Yet the users of the first service may wait far too long for their answers to begin, while the second service keeps its responses prompt. Counting finished work alone cannot separate these two situations. This article introduces two measurements that do: throughput, which counts how much work a service completes, and goodput, which counts only the work that meets a set of performance targets. Throughput also connects directly to cost, so the same measurements tell you what each served request actually costs.

Throughput can also be reported in requests rather than tokens. A service that completes requests can be described by how many requests it finishes per second or per minute. The two units describe the same service from different angles. A token count tells you how much text the service produces, while a request count tells you how many separate jobs it finishes. The table below lays out both units.

The service-level number is a sum. Each active request contributes the output tokens it is generating right now, and the service reports the total. The following worked example follows that addition step by step, using a small generated scenario with three users.

Throughput is directly tied to compute cost. A higher throughput generally means a lower cost per unit of output, because the same hourly hardware price is spread across more generated tokens. The source works through the arithmetic with specific numbers. Follow the same chain of reasoning step by step, because the same method applies to any hardware price and any throughput figure.

In this formula, P is the hardware price per hour and T is the throughput in output tokens per second. The factor 3600 converts seconds into hours, so T times 3600 is the number of tokens the service can produce in one hour. Dividing the hourly price by that number gives the cost of one token, and multiplying by one million gives the cost per million tokens.

Applying the formula to the source's example: the hardware costs 2 USD per hour and generates 100 output tokens per second. That is 360,000 tokens per hour. The cost of one token is 2 divided by 360,000, which is about 0.00000556 USD. Scaled to one million tokens, the cost is about 5.556 USD per 1M output tokens.

Like many software systems, AI applications face a trade-off between latency and throughput. Techniques such as batching can raise throughput, but they tend to lengthen latency. Each user waits a little longer so that the service as a whole finishes more work. The source reports that a team deploying generative AI products found it was not unusual to double or triple throughput, provided they accepted worse TTFT and TPOT figures. TTFT and TPOT are the two latency measures the source uses: the wait before a response begins, and the pace of the response once it starts.

Follow a single request from arrival to its final count. The request is served, its output tokens are generated, and it finishes. At the moment it finishes it is counted in throughput, whatever its speed. Goodput asks one more question: did the request also meet the stated objectives? Only requests that pass that check are counted in goodput.

The key difference from throughput is which requests are counted. Throughput counts every request that finishes. Goodput counts only the finished requests that also meet the stated objectives, which makes it a stricter measure. A service level objective, often called an SLO, is the set of targets the service promises to meet.

Consider an application whose objectives are a TTFT of at most 200 ms and a TPOT of at most 100 ms. If the service completes 100 requests per minute but only 30 of them meet both targets, its goodput is 30 requests per minute, even though its throughput in requests is 100 per minute. The diagram below places the two readings side by side.

The source gives a larger-scale version of the same point. If a service completes 10 requests per second but only 3 of them satisfy the SLO, its goodput is 3 requests per second. The gap between the two numbers, here 7 requests per second, is the portion of the service's output that users would experience as too slow. A service can look productive on its throughput line while most of its finished work misses the targets that users care about.