Foundation Models and Model APIs

Many people think of a product as finished once it reaches its goal. In practice, planning has to look further ahead. You need to ask how the product might change over time and how it should be kept running and improved. For an AI product this question is harder than usual, because the field itself keeps shifting under your feet. The tools you choose this quarter may look very different a year from now, and the decisions you make today will shape how much effort maintenance requires later.

This article follows one line of reasoning. It starts with the speed of change in AI, then looks at which kinds of change are easy or hard to live with, then places AI engineering within the broader field of machine learning engineering, and finally describes the layered structure that AI applications are usually built on. Keep one question in mind as you read: when the ground keeps moving, what should a team re-check, and what can it safely ignore?

The AI space has been moving incredibly fast over the last decade, and it is widely expected to keep moving fast for the next one. Building on top of foundation models therefore amounts to committing to a fast-moving environment. This is the core reason AI products are harder to maintain than conventional ones: the pace of the underlying technology is part of the deal you accept when you build on it.

Speed is not uniformly bad. Many of the changes are improvements. The limitations of many models are being addressed, context lengths are getting longer, and model outputs are getting better. A team that never updates is not automatically safe, because it misses these gains. A team that updates constantly is not automatically productive, because every update carries a workflow cost. Maintenance is the work of managing that tension.

The source illustrates the trend with a chart that tracks the cost of inference and the performance of models on Massive Multitask Language Understanding, a popular foundation model benchmark, between 2022 and 2024. The chart shows the cost of AI reasoning dropping rapidly over that period. The point to take from it is directional rather than a set of exact figures: the economics of building with models are shifting quickly.

Not all changes cost the same to live with. The useful distinction is between changes you can plan for as a technical task and changes that alter what you are allowed to do at all. The diagram below sorts the examples the source gives into those two groups. Read the left column as work you can schedule, and the right column as changes that can force a redesign or stop a product entirely.

The easier group is not free. As model providers converge on the same API, it becomes easier to swap one model API for another. That convergence is a real benefit, but it does not remove the work entirely. Each model has its own quirks, strengths, and weaknesses, so developers who switch models will usually need to adjust their workflows, their prompts, and their data. Without proper infrastructure for versioning and evaluation, that adjustment can become a source of considerable frustration.

The source singles out regulation as the main example of a change that is harder to adapt to. The key idea is that a swap of API is a technical change you can plan for, while a regulatory change can alter what you are allowed to do. Regulation affects the resources AI depends on, including compute, talent, and data, because technologies surrounding AI are considered national security issues in many countries.

Regulation can have large direct costs. The source gives the example of Europe's General Data Protection Regulation, which was estimated to cost businesses 9 billion dollars to become compliant with. Regulation can also change supply overnight. Compute availability can shift suddenly when new laws place more restrictions on who can buy and sell compute resources. The source cites a US executive order from October 2023 as an example of this kind of rule.

The practical consequence for a product team is stark. If the company that supplies your graphics processing units is suddenly barred from selling to your country, the product that depends on that hardware is in trouble. Intellectual property adds a further risk. Rules around intellectual property and AI usage are still evolving, and if you build your product on a model that was trained using other people's data, it is not obvious that you can always be certain your product's intellectual property will belong to you. The source reports that many intellectual-property-heavy companies, such as game studios, hesitate to use AI because they fear losing their intellectual property later on.

The source describes a team deciding whether to host its own model or pay a model provider, and it asks you to reason through the decision before reading the answer. The source frames the lesson rather than a verdict: each technology investment needs an ongoing cost-benefit analysis, not a one-time decision. The worked example below applies that framing. It is an illustration built for this article, not a result the source reports.

Once a team has committed to building an AI product, it needs to understand the engineering stack behind it. Any AI application stack has three layers: application development, model development, and infrastructure. Think of them as a vertical arrangement, with application development at the top and infrastructure at the bottom. The diagram shows that arrangement and the direction in which work usually moves.

The source explains the working order. When developing an AI application, you will likely start from the top layer and move down as needed. You do not have to master every lower layer before you begin. You should, however, be ready to descend into a lower layer when a problem at the top requires it. In other words, the stack is worked top-down, but it is not learned top-down in a strict sequence.

A Model API is the way an application reaches a foundation model that someone else hosts. The diagram traces one exchange: your application sends a request, the hosted model produces a result, and a response returns to your application. The point the diagram makes is where the external dependency sits. It lies between your product code and a model you do not control.

This placement explains the maintenance problem. When the provider changes its model, its pricing, or its terms, your product feels it through the interface, even though your team did not write that change. That is why the earlier question of whether you can swap an API matters so much: the API is the seam where your product meets a dependency you do not own.

AI engineering evolved out of ML engineering, and the two overlap heavily even where companies draw separate job lines. When a company starts experimenting with foundation models, it is natural for its existing machine learning team to lead that effort. The relationship between the two roles is therefore one of heritage and overlap rather than a clean separation.

Companies handle the two roles differently. Many organizations place AI engineering and ML engineering under the same umbrella, and some treat them as identical. Others publish separate job descriptions for AI engineering. Existing ML engineers can add AI engineering to their skill lists to widen their job prospects, and there are also AI engineers who arrive with no previous ML experience. The rapid growth of the field has produced a great deal of hype and fear of missing out, so the source advises focusing on the fundamental building blocks rather than trying to keep up with every new tool.