Model Gateways and Routing

Imagine a customer-support assistant built on a generative model. A user types a question, the application sends it to a model, and a reply comes back. That sounds simple, but each step in the path is a place where something can go wrong. The user might try to manipulate the model. The model might produce an unsafe or off-topic reply. The reply might leak information it should not contain. Guardrails are the checks that catch these problems.

This article follows one request through such an application. The goal is to see where guardrails sit, what they cost, and how a team chooses among the options. A useful way to hold the whole topic in mind is that guardrails are not free. Adding them changes how fast the application responds, how much the team depends on outside providers, and how much engineering work the team takes on. A good implementation is a set of deliberate choices, not a checklist of every check that could be added.

Start with the request path. The user's input does not go directly to the model. It first passes through input guardrails, which examine what the user sent. Only then does the request reach the model API. The model's raw output does not go directly back to the user either. It passes through output guardrails before the response is shown. The path therefore has a gate on each side of the model call.

Read the diagram as a pipeline with two gates. The input gate protects the model from what comes in. The output gate protects the user from what goes out. The scorers are the tools that help make those judgments. The source places scorers under the model APIs because they are often AI-powered themselves, even though they are usually smaller and faster than the generative models. Scorers can also sit inside the output guardrails stage instead, so the scoring work can attach to either side of the model call depending on the design.

Because many different parts of a system can fail, guardrails can be placed at several levels. Two groups do most of this work: the companies that build and host models, and the teams that build applications on top of those models. Model providers and application developers can both implement guardrails, and each level has its own responsibilities. The two groups have different tools, so a careful design usually uses both.

The gateway layer deserves its own check. Some model gateways also provide guardrail functionality, so a team that already runs a gateway should look at what that gateway covers before duplicating the same checks in its own code. The team's own checks then fill the gaps the gateway and provider do not cover.

The number of guardrails a team needs depends on whether it hosts its own models or uses third-party APIs. Guardrails can be built on top of either option. However, third-party API providers typically offer many guardrails out of the box, which reduces how much a team must build itself.

Self-hosting pulls in the opposite direction on one front. When models run inside your own system, requests do not have to be sent to an external service. That reduces the need for many input guardrails that exist to handle data crossing that outside boundary. The engineering burden shifts rather than disappearing. A self-hosted team gives up provider-supplied protections and gains control over where its checks run.

The central trade-off in guardrail implementation is reliability versus latency. Stronger checking usually takes more time, and users notice delay. The source reports that some teams felt latency mattered more than the guardrails and chose not to implement them, because those checks can significantly increase an application's latency. That is a real engineering decision, and the right answer depends on the product and its risks.

The trade-off becomes sharper with output guardrails and the way responses are delivered. By default, the whole response is generated before anyone sees it, and that can take a long time. In stream completion mode, tokens are sent to the user as they are generated, so the user waits less. The cost of streaming is that partial responses are hard to evaluate. An unsafe response might reach the user before the guardrails decide to block it.

The two delivery modes ask different things of the output guardrail. A check on a complete response sees everything the model produced before any decision is made. A check during streaming sees only the part that has been generated so far. The table below summarizes the trade-off described in the source.

Instead of building every check from scratch, a team can adopt an existing guardrail solution. The source notes that several can be used out of the box, and knowing these exist is the first step. Choosing among them depends on what each one covers. The source pack for this article does not name specific products, so treat any product list as something to verify against current provider documentation.

Map each risk to the side of the model call it threatens. Input risks include a user trying to manipulate the model, which is checked on the way in. Output risks include an unsafe or off-topic reply and a reply that leaks information it should not contain, which are checked on the way out. Because risks overlap between inputs and outputs, a single solution will often protect both sides, so a team should not assume it needs one tool for input and another for output.