Model Selection Criteria

A model can produce excellent answers and still be a poor fit. If users wait so long that they give up, or each request costs more than the product can support, raw answer quality is not enough. Model selection means weighing three forces together: the quality of the output, the time users wait, and the cost of serving the model. The right balance depends on what the application needs.

Pareto optimization is the study of optimizing several objectives at once. For model selection, its practical lesson is to make trade-offs explicit: decide which objectives are requirements and which are preferences. A model may be worth choosing even if it is weaker on one dimension, provided it meets the application's firm needs and gains enough on another dimension, such as response speed or cost. There is not necessarily one model that wins on every objective.

Start by separating firm requirements from softer goals. A firm requirement is a pass-or-fail filter: remove any candidate that cannot meet it. Then compare the remaining candidates against preferences such as better quality, lower latency, or lower cost. This order matters because a strong score on one dimension should not excuse failure on something the application cannot give up.

The exact order of exploration can vary during development. With prompt engineering, a team might first try the strongest available model to check whether the task is feasible, then test smaller models to see whether they are good enough. When testing fine-tuning, a team might start with a small model to test its code cheaply, then move toward the largest model that fits its hardware constraints. These experiments inform selection, but the final choice still needs to meet the application's firm requirements.

Latency is not just one number. Different metrics describe different parts of a request, so choose the metric that matches what the application needs to protect. A streaming chat interface may care about when the first words appear; a batch report generator may care more about when the complete response is ready.

Latency depends on more than the model itself. The prompt and sampling settings affect it, and autoregressive models generate output one token at a time. As a result, longer outputs generally take longer to finish. Concise prompting, a stopping condition, and other optimization techniques can reduce the latency users observe.

For a model API, the provider usually charges according to the tokens sent to the model and the tokens returned. Both the input prompt and the generated output contribute to the bill, so reducing unnecessary tokens in either part is a common way to control API cost.

Self-hosting changes the cost driver: the main expense is compute, or the machines running the model, apart from engineering costs. Once a cluster is in place, compute spending is largely fixed. If usage grows while that spending stays the same, the cost per token can fall. At lower usage, the same fixed compute can make the per-token cost much higher. By contrast, API cost per token stays fairly steady as volume grows, so the total bill rises with usage. The API must also be able to handle the required volume.

A criteria table makes model comparisons more disciplined. For each dimension, record the metric, the benchmark or test conditions, the hard requirement, and the ideal target. The benchmark matters because two models can appear different simply because they were tested on different sets. If factual consistency matters, it can be added as another criterion with an appropriate measurement method and benchmark.