Modularity in AI Systems

Imagine you are handed a finished AI product and asked where its moving parts are. You could start by guessing which piece handles the user's request, which piece calls the model, and which piece decides whether an answer is safe to show. Without a map, each guess stands alone, and you have no way to check whether one guess relates to another. This article gives you that map: a high-level architecture for applications built on top of foundation models.

The goal is not to memorize one fixed layout. The goal is to learn a way of thinking about how the pieces relate to each other, and what each new piece costs. This article belongs to a larger chapter on building applications on foundation models as a whole. Earlier material looked at individual aspects of AI engineering one at a time. This chapter shifts the lens: it asks how those individual aspects combine into a working application, and where things tend to go wrong along the way.

The word framework matters here. A framework gives you a vocabulary and a set of questions to ask about any system, even when the system in front of you is organized differently from the common pattern. A blueprint, by contrast, would say exactly where every part goes. The architecture of a particular application may vary from the common pattern because teams make different choices about where work happens and which features they need.

What stays useful across every application is the shared way of reasoning. For any system you encounter, identify the components, identify how they connect, and identify what each one is responsible for. Those three questions work whether the system follows the common pattern closely or departs from it substantially.

The source builds this framework step by step. It walks through the construction of the architecture one stage at a time and, at each stage, discusses the challenges that appear and the techniques engineers can use to address them. This means you meet each component together with its typical problems, rather than as an abstract box on a diagram.

A guardrail is a check that constrains what the system accepts or produces. The easiest way to understand where guardrails sit is to follow a single request from start to finish. The diagram below shows one generic path: the request is checked before the model is called, the model produces a response, and that response is checked before it is returned. Trace each arrow and notice where the request can stop early.

Follow the two outcomes. If the input check fails, the request never reaches the model, so the blocked result is produced without a model call. If the input check passes, the model runs. If the output check then fails, the model has already produced a response, but that response is not shown to the user. In both cases the guardrail decides what the user finally receives. The diagram is a generic illustration of this control flow; the source itself does not prescribe a fixed number of checks or their order.

Engineers separate components on purpose. Keeping responsibilities distinct makes a system easier to modify and maintain, because a change in one area is less likely to break another. That separation is the core idea behind modularity in AI systems.

The separation, however, is not rigid. Real systems do not divide into neat, non-overlapping slices. Several components can take on the same kind of job, and the boundary between two of them can blur. Modularity is therefore best understood as a useful guideline for drawing boundaries, not as a guarantee that every responsibility has exactly one home.

Guardrails are the clearest illustration. The source notes that guardrails can be implemented in the inference service, in the model gateway, or as a standalone component. The same kind of protection, in other words, has several legitimate homes. The diagram below shows that one piece of guardrail logic can be placed in any of these three components.

Read this diagram as a set of options rather than a single correct answer. The guardrail logic is the same in each case. What changes is which component is responsible for running it. Placing it inside the inference service keeps the check close to the model. Placing it in the model gateway puts it where requests and responses pass through a shared entry. A standalone component keeps it separate from both. The source does not rank these choices, so the decision depends on what the application needs.

Adding a component is never free. The source states that each additional component can make a system more capable, safer, or faster. These are three different kinds of payoff, and a team might be chasing any one of them. But the same addition also increases the system's complexity and exposes it to new failure modes. Each gain comes bundled with a cost.

The practical lesson is that every component should justify itself. When you consider a new piece, ask which of the three gains it targets, and then ask what new failures it could introduce and how you would notice them. A component that adds capability but makes the system much harder to reason about may be a poor trade.

Monitoring and observability are described in the source as an integral part of any complex system. Observability is not a single tool. It has three connected parts: understanding how your system fails, designing metrics and alerts around those failures, and building the system so that failures can be detected and traced back to their source.

The diagram below shows where signals can be captured as a request moves through an AI application. Each stage the request passes through is a place where data about its behavior can be recorded and sent onward to a monitoring system.

Observability is partly a design decision made early. A system that was not built to expose the right signals will be hard to diagnose later, no matter what monitoring tool is attached at the end. This is why observability belongs in the architecture discussion rather than being an afterthought.

Many observability practices carry over from software engineering and from traditional machine learning. The source is clear, however, that foundation models introduce new failure modes. Those failures call for additional metrics and additional design considerations beyond what older practices provide.