Model Adaptation in AI Engineering
The infrastructure layer is the bottom of the AI engineering stack and contains tooling for model serving, data and compute management, and monitoring.
A Reply-Drafting Feature Meets Reality
Imagine a team that has just finished a clever AI feature, such as a tool that drafts customer replies. The feature works in the team's own tests, but before a single user can benefit from it, several questions need answers. Where does the model run? How does a request reach it? Who keeps track of how much data and computing power the system is consuming, and who notices when responses suddenly become slow or wrong?
These questions belong to the bottom of what can be called the AI engineering stack. This article is about that bottom layer, which is called the infrastructure layer. The useful picture to keep in mind is simple: the layers above infrastructure are where people build products and shape models, while infrastructure is the ground those activities stand on. Everything the team builds depends on that ground being in place.
What do you think happens?
The reply-drafting feature suddenly starts producing slow or wrong responses in production. Which of the three infrastructure duties is the one that notices this?
Reveal answer
Monitoring
Monitoring is the job of watching the running system so that problems are visible, including a failing service, unusual resource use, or output quality that has drifted. Serving is about delivering inputs and outputs, and data and compute management covers the resources underneath the exchange.
Following One Request Through the Stack
To understand the infrastructure layer, follow a single customer message through the reply-drafting feature. The message starts in the application, which is one of the layers above infrastructure. The application needs to hand that message to a model and get a draft back. That handoff is not free: it needs a running model, a path for the request to reach it, and capacity to process it.
Read the diagram from the top down. The stack is made of layers, and the infrastructure layer sits at the bottom, beneath the layers where products are built and models are shaped. The three duties on the lowest branch are the work that makes the layers above possible. The tracing exercise in the next sections will show which duty handles each step of the request.
The Three Duties of the Infrastructure Layer
The infrastructure layer is the bottom of the AI engineering stack, and it contains tooling for three responsibilities: model serving, data and compute management, and monitoring. Each one addresses a need that exists in every serving system. The subsections below name each duty and explain what it covers.
Watching the Stack Grow After 2023
To see how the landscape changed, one analysis searched GitHub in March 2024 for AI-related repositories with at least 500 stars and found 920 of them. The repositories were grouped by the layer they belonged to: applications, application development, model development, and infrastructure. The analysis then tracked the cumulative count of repositories in each category, month over month.
The cumulative curves show a sharp rise in the number of AI tooling repositories during 2023, following the arrival of Stable Diffusion and ChatGPT. The steepest climbs belonged to the application and application development categories. Infrastructure also grew, but its growth was far smaller than what happened in the layers above it. The chart shows the top layers pulling away from the bottom one, rather than every layer rising together.
Notice what the diagram does and does not claim. The application layers changed sharply, while infrastructure changed far less. The infrastructure layer's growth was smaller, but the layer did not disappear or stop mattering. Its needs were simply not reshaped to the same degree.
Why Infrastructure Needs Stayed Stable
The explanation offered for the smaller infrastructure growth is that the core infrastructure needs stayed the same even though models and applications changed. Resource management, serving, and monitoring still have to happen whatever model sits on top of them. A new model may be larger or wired to a different product, but it still needs compute, a way to receive requests, and a way to be watched.
This stability also points to a broader lesson. Much of what machine learning engineers learned over the previous decade still applies to AI applications. Systematic experimentation, mapping business goals to measurable model metrics, making models faster and cheaper, and keeping a feedback loop running with production data all carry over. What is new is built on top of these principles, and the infrastructure layer is where the carried-over principles mostly live.
How AI Engineering Differs From Traditional ML
Building applications on foundation models differs from traditional ML engineering in three major ways: model adaptation, larger models with higher latency that demand efficient inference, and open-ended outputs that are harder to evaluate. The table below sets out each difference and connects it to the infrastructure layer where the source allows.
| Dimension | What differs in AI engineering | Connection to the infrastructure layer |
|---|---|---|
| Model adaptation | Adapting a pretrained model for a specific use is a defining activity of AI engineering. | The source names this difference but does not detail how it changes infrastructure. |
| Model size and latency | Models are larger and have higher latency, which demands efficient inference. | Efficient inference depends on serving, the duty that delivers inputs to the model and returns outputs. |
| Open-ended outputs | Outputs are open-ended and harder to evaluate than fixed results. | Monitoring watches output quality, including drift, so this difference falls on the monitoring duty. |
Three differences between AI engineering and traditional ML engineering
The model size and latency row deserves a closer look. Larger models take longer to respond, so a team building on them must pay attention to how efficiently requests are handled. That attention lands in the infrastructure layer, because serving and compute capacity determine how quickly a draft can come back to the customer.
Common Misreadings of the Infrastructure Layer
Common mistake Treating the three duties as separate products to buy one at a time
What it looks likeA team assumes serving, data and compute management, and monitoring each require an independent product that can be added or removed alone.
Why it failsThe duties are needs that exist in every serving system. The source states they are not separate products that a team can pick up and drop.
Do this insteadThink of the duties as requirements the layer must meet. One tool may cover several of them, and several tools may cover one.
Common mistake Assuming slow infrastructure growth means the infrastructure layer matters less
What it looks likeBecause infrastructure repositories grew far less than application repositories after 2023, a reader concludes that infrastructure is a minor concern.
Why it failsThe slower growth reflects stable needs, not reduced importance. Serving, resource management, and monitoring still have to happen whatever model is on top.
Do this insteadRead slow growth as stability. The layer's needs stayed the same, so the layer is the constant ground the rest of the stack depends on.
Common mistake Believing that new AI means all old engineering principles are obsolete
What it looks likeA team drops systematic experimentation and production feedback loops because the work is now about foundation models.
Why it failsMany classical ML engineering principles still apply, and the infrastructure layer is where much of that continuity lives.
Do this insteadKeep the carried-over principles and build the new practices, such as model adaptation, on top of them.
Classifying Duties in a New Scenario
A team's reply-drafting feature now runs on a larger model. Users report that drafts take noticeably longer to arrive. Another engineer notices that the cluster hosting the model is close to its processing limit. Name the infrastructure duty that most directly covers the cluster capacity problem, and explain in one or two sentences which duty would catch a drop in draft quality over several weeks.
Hints
- Recall that data and compute management covers the processing capacity, such as GPUs and clusters, that the models need in order to run.
- Recall that monitoring watches for output quality that has drifted, not only failing services.
What the Infrastructure Layer Carries
- 1The infrastructure layer sits at the bottom of the AI engineering stack and contains tooling for model serving, data and compute management, and monitoring.
- 2Model serving lets other software send inputs to a model and receive outputs. Data and compute management covers stored data and processing capacity. Monitoring watches for failing services, unusual resource use, and drifting output quality.
- 3Infrastructure growth after 2023 was far smaller than growth in the layers above it, because its core needs stayed the same whatever model or product sat on top.
- 4AI engineering differs from traditional ML engineering through model adaptation, larger models with higher latency that demand efficient inference, and open-ended outputs that are harder to evaluate.
- 5Many classical ML engineering principles still apply, and the infrastructure layer is where much of that continuity lives.
Key Takeaways
- The infrastructure layer is the bottom of the AI engineering stack and contains tooling for model serving, data and compute management, and monitoring.
- Model serving connects software to a model, data and compute management supplies storage and processing capacity, and monitoring makes failures and quality drift visible.
- Infrastructure needs stayed stable after 2023, which is why infrastructure grew far less than the application layers above it.
- AI engineering differs from traditional ML engineering through model adaptation, larger models with higher latency, and open-ended outputs that are harder to evaluate.
- Many classical ML principles still apply, and the infrastructure layer is where most of that continuity lives.