Model Build Versus Buy Decision: the broader framework this article belongs to

Once you have decided that your application needs a model, you still have to decide how that model reaches your users. One path is to rent access to a model that someone else runs, paying for each use. The other path is to run a model on infrastructure that your own team manages. These two paths are not interchangeable. They change what the model can do for you, what it costs over time, how much control you keep, and how exposed you are when something outside your team changes.

This article is the framework that sits above the choice. It does not tell you which model to pick. It tells you which considerations should steer the decision between buying access through a commercial API and building on a model you host yourself. The article follows that framework in four stages: the performance gap between open and proprietary models, the supporting features a model needs in production, the cost comparison, and finally control and provider risk. Keep one question in mind throughout: what would your application need to keep working in a year, and which option makes that easier to guarantee?

A common assumption is that open source models will eventually match the best proprietary models. The evidence points in a more nuanced direction. Across published benchmarks, the distance between open source and proprietary models has been shrinking. One benchmark, MMLU, is often used to illustrate this. If you plotted the scores of each group of models against time, you would see two curves moving toward each other. The proprietary curve stays higher, but the space between the two curves gets narrower with each new generation.

Think of the gap as a distance that changes over time, not as a yes or no question. A narrowing distance can still leave a meaningful difference. The right question is not whether the gap has closed but how large it is for the capability your task needs.

A narrowing gap does not mean the gap is gone, and the source makes a pointed argument about incentives. A developer who holds the strongest model available has a strong reason to keep it behind an API, where it can be monetized, rather than releasing it for others to use. For that reason, companies often keep their strongest models proprietary while open sourcing weaker ones. The likely result is that the strongest open source model keeps trailing the strongest proprietary models for the foreseeable future.

There is a second, quieter reason for the lag, and it is separate from raw model quality. Commercial providers learn how their models are used in practice through the feedback and behavior of their users. Once an open source model is released, its developers usually have no direct view of how it is being used or how well it holds up in the real world. That missing feedback loop is a structural disadvantage for the open side.

A model on its own rarely serves a use case. Several supporting functions have to exist around it before users can rely on the result. The source names scalability, function calling, structured outputs, and output guardrails as examples. Each one answers a different production question, and each one is a place where the deployment choice matters. Scalability asks whether the service can keep up with demand. Function calling lets the model trigger actions in your software. Structured outputs make responses predictable enough for code to consume. Output guardrails check what the model produces before it reaches a user.

Building these functions yourself is slow and demanding, which is one reason many teams choose a commercial API. Providers often ship some of these functions ready to use. The trade-off is that you can only use what the provider chooses to offer. The source gives a useful example in logprobs, which expose the model's probability estimates for its outputs. Logprobs are valuable for classification, evaluation, and interpretability. Commercial providers may hesitate to expose them, partly because others could use them to imitate the model. In practice, many APIs do not expose logprobs at all, or expose only a limited amount.

The cost comparison is less intuitive than it first appears. Using a model API means your spending scales with your usage, and at heavy volume that spending can become very large. Self-hosting trades that recurring charge for a different kind of expense. You take on the time, talent, and engineering effort needed to optimize the model, keep the inference service running at the right scale, and build the guardrails that protect your product. The source sums up the tension this way: APIs are expensive, but engineering can be even more so.

When a business depends on a model, it is reasonable to want real influence over how that model behaves. A 2024 study by a16z, cited in the source, found that enterprises care about open source models mainly for two reasons: control and customizability. These two reasons are the ones that deployment choices most directly affect. The choice of how the model reaches you determines how much you can change it, whether you can finetune it or freeze a version, and how much you depend on decisions made outside your team.

A good model choice is one that is easy to use and easy to manipulate. The source observes a practical split between the two families. Proprietary models are generally easier to get started with and to scale, while open models can be easier to manipulate because their components are more accessible. Whichever family you pick, the source recommends that the model follow a standard API, because a standard interface makes it easier to swap one model for another.

The source treats the choice as a sequence of filters rather than a single verdict. Start with your hard requirements, such as whether the model must run locally, whether you need access to signals like logprobs, and how much control and customization you need. These requirements narrow the field. Then refine the shortlist with publicly available performance data. The flowchart below is a simplified way to walk through those filters. It is an illustration of the criteria the source names, not a fixed rule with thresholds.

After the hard requirements narrow the field, you still need evidence about how each candidate performs. There are thousands of benchmarks designed to measure different model capabilities. Google's BIG-bench, published in 2022, contains 214 benchmarks on its own, and the number keeps growing as AI use cases multiply. Benchmarks also wear out. As models improve, older benchmarks stop separating strong models from weak ones, so new benchmarks have to be introduced.

To run many benchmarks at once, people use an evaluation harness, a tool that evaluates a model across multiple benchmarks. The source names EleutherAI's lm-evaluation-harness, which supported over 400 benchmarks at the time of writing, and OpenAI's evals, which lets you run roughly 500 existing benchmarks and register new ones for OpenAI models. These benchmarks cover a wide range of abilities, from math and puzzle solving to recognizing words written as ASCII art.

The build-versus-buy decision is a framework of connected questions rather than a single verdict. Performance differences are real and narrowing, but the strongest open source models are expected to keep trailing. Supporting functions such as scalability, function calling, structured outputs, and guardrails must exist around any model, and a provider may withhold some of them, such as logprobs. Cost shifts between a usage bill and engineering effort. Control, customizability, and provider risk decide how much you can change the model and how exposed you are when the outside world changes. Keep the one-year question in view: which option makes it easier to guarantee that your application keeps working?