Cost Planning for Model Inference

Imagine a small product team that wants its application to produce text from a language model. The team has two broad routes. It can send requests to a model that someone else operates and pay for each use, or it can take a model and run it on infrastructure it controls or rents. Both routes can produce useful answers. They differ in who sees the data, who decides what the model may be used for, and what the bill looks like over time.

This article focuses on that choice. The practical question is which considerations should drive a team's decision when it picks hosting or an API. The answer depends less on technical convenience than on three questions that are easy to overlook at the start of a project: where the model's knowledge came from, what the license allows, and which parties the data is allowed to reach.

The difference is easiest to see by following a request. With an API, the team's application sends a request to the provider, and the provider's model handles it. The data crosses a boundary of trust the moment it leaves the team's application. With hosting, the request stays on infrastructure the team operates or has rented, and the model runs there. The source puts the central point this way: every request sent to an outside service crosses a boundary of trust.

Read the diagram as two paths from the same starting point. The upper path sends data to a party outside the team, the provider. The lower path keeps the model's execution on infrastructure the team operates or rents. Neither path removes the need for the license and lineage checks covered below, because those checks apply to the model itself, whichever path delivers it.

A useful way to think about the decision is as a sequence of gates rather than a single preference. Each gate asks a question, and the answer narrows the options before the team reaches the final check.

Read the diagram from top to bottom. The team first asks whether its data needs strict privacy protection. If it does, it asks whether a large cloud provider is an acceptable place for that data. A yes leads to hosting on that cloud, and a no pushes the team toward self-hosting or another provider. If the data is not especially sensitive, the team can consider a third-party API directly. Every route converges on the same final step. Whether a model is hosted or called through an API, its license and the origin of its training data still determine whether it may be used at all.

If a model was trained on outputs produced by another model, the terms governing that other model can still bind how the new model is used. A restriction does not stop at the model it was attached to. It travels down the chain to every model and deployment built from that material. The tree below follows one such chain, using the case the source describes.

A model's license is the set of terms and conditions under which the model can be used. It is tempting to read a license as a complete statement of permissions, but a license may restrict use in ways a quick reading misses. The Model X case shows one such way. A license can permit one activity, such as using outputs to improve other models, while the broader chain of obligations still forbids the use.

The source also describes a related pattern in the open source world. It compares models to the Elastic License, which forbids companies from offering the open source version of Elastic as a hosted service that competes with the Elasticsearch platform. The comparison shows that a license can allow access to source material while still limiting who may commercially offer it. Model licenses can work in a similar spirit. Open or accessible therefore does not automatically mean unrestricted.

Data privacy concerns the protection of sensitive information. In the hosting-versus-API decision, the central question is which parties will receive that information. A team's privacy policy therefore often reduces to a list of organizations it is willing to trust with its data.

The source makes a point that may seem surprising at first. Some companies with strict data privacy requirements have said that they usually cannot send data to third-party services, yet they are comfortable sending data to models hosted on GCP, AWS, and Azure. For these organizations, the policy is less about whether an outside party is involved and more about which services they can trust. They trust large cloud providers but do not trust other startups.

Regulation adds another consideration. The source notes that rules are evolving around the world and that requirements for auditable information about models and training data may increase. Commercial models may be able to provide certifications, which can spare a company the effort of producing that evidence itself. A hosting decision and an API decision can therefore differ in how much audit work falls on the team.

Many users want models to be open source because openness means more information and more options. The harder question is what the developer gains. Building a model can require very large investment, so a developer who releases it openly needs some reason to do so. The source offers three possibilities. One is that a wave of companies has formed to offer inference and finetuning services on top of open source models, and many people need those services to use open models. A second, offered as an argument rather than a settled fact, is that a large developer may support open source models to keep its competitors in check. A third is that open source is good for society, and that goodwill may be enough to sustain it.

The same model family can appear on both sides of the choice. The source cites Mistral and Cohere as companies that both release open source models and offer APIs. Once a company's inference service runs on its own models, it may become a competitor to the developer that released them. Open availability and commercial services can coexist, but their interests do not always align.

API cost is the other side of the picture, and it is not uniform. The source observes that the companies hit hardest by per-use API costs are probably not the biggest companies. Large organizations may be important enough to service providers that they can negotiate favorable terms. Smaller companies usually have less bargaining power and feel the full weight of the usage bill. A route that looks reasonable for a large company with negotiating leverage can be a heavy burden for a small team.