Token-Based Pricing and Model APIs

This article follows how those three forces interact and how to make a sensible choice among them. Quality, latency, and cost pull against one another in practice, and the best model for a given application is rarely the one that wins on every axis. Choosing a model is therefore less about finding the single strongest model and more about deciding which constraints are fixed and which are preferences, then selecting the model that fits the application's real needs.

Latency sounds like a single quantity, but for foundation models it is measured in several ways. Each metric captures a different part of the experience a user has while waiting, so the right one depends on what your application is trying to protect. A streaming chat interface cares a great deal about when the first words appear, while a batch report generator may care only about when the entire response is finished. The diagram below traces where each common metric starts and stops within a single response.

Read the timeline from left to right. The first interval is the wait before any output appears. The middle intervals describe the spacing of the tokens as they arrive. The final interval runs until the whole response is done. Each metric measures one of these stretches, which is why a single number can hide a problem that another metric would reveal.

Latency is not a fixed property of a model. It also depends on the prompt that is sent and on the sampling settings used to generate output. Since output length is a major driver, the most direct levers are the ones that control how much the model writes.

Optimizing several objectives at once is a field of study known as Pareto optimization. The core technique is to rank your objectives by how firm they are. A firm objective acts as a filter: any candidate that fails it is removed outright. Only after filtering do you compare the survivors on the softer objectives. This ordering matters, because it stops a strong result on one dimension from quietly excusing a failure on a dimension you cannot give up.

A useful habit is to ask, for every objective, whether it is a fixed requirement or a preference. Many teams, once they see the trade-off clearly, deliberately choose a model that is somewhat weaker in quality because it responds faster or charges less. Whether that trade is worth making depends entirely on what the application needs.

If you call a model through an API, the provider usually charges by the number of tokens, counting both what you send in and what comes back. Every additional input token and every additional output token raises the bill. The diagram below labels which parts of a single exchange count toward the total.

The source of cost depends on how the model is delivered. With a model API, the provider handles the infrastructure and bills you per token. If you host your own model, the picture changes. Apart from outside engineering cost, your main expense is compute, meaning the machines that run the model. Because machines come with fixed amounts of memory, there is a strong incentive to choose the largest model that fits on the hardware you already have. Memory sizes such as 16 GB, 24 GB, 48 GB, and 80 GB shape which models are practical, and this is one reason many popular models have roughly seven billion or sixty-five billion parameters.

Scale changes which delivery approach is cheaper. Suppose you have already built a cluster able to serve one billion tokens a day. Whether you serve one million tokens or one billion, the compute cost is the same, so the per-token cost at the lower volume is far higher. At high volume the self-hosted option can become cheaper per token, but only if the usage actually arrives. Because the economics shift with scale, the choice between an API and self-hosting is not permanent, and companies need to reconsider it as their usage changes.

Turning the three forces into something you can check requires writing criteria down. A criteria table lists each dimension, the metric used to measure it, a benchmark that supplies the test conditions, a hard requirement that a model must meet, and an ideal target that a model should aim for. The illustrative table below uses a fictional application so that the hard requirements and ideal targets sit side by side.

Notice that each row names a benchmark. Two models can look different only because they were measured on different test sets, so the benchmark column is what makes the comparison fair. A factual consistency criterion, measured with an internal metric against an internal hallucination dataset, can be added the same way when accuracy is a concern for the application.

Once the criteria are written, selection proceeds in two stages. The first stage applies the fixed hard requirements as filters and removes any candidate that fails one. The second stage compares the survivors on the softer goals, placing them on a map whose axes are cost and performance, and picks the one that delivers the most performance for the money. The diagram below shows the path from criteria to a chosen model, including the return to the start when usage changes.

The process is more nuanced than two stages suggest, and it does not happen only once. When you use prompt engineering, you might begin with the strongest available model to test whether the task is feasible at all, then work backward toward smaller models to see whether they are good enough. When you finetune, you might begin with a small model so you can test your code cheaply, then move toward the largest model that still fits your hardware constraints, such as a single GPU.