Inference Servers for Self-Hosted Models

Imagine you are asked to design the software behind an AI product that is going into real use. The finished system may end up with many moving parts, and the diagram can look intimidating on first contact. A useful habit is to set the finished system aside and sketch the smallest design that could possibly work. Then you add pieces one at a time, each for a reason you can name. This article starts from that smallest design and follows the same path a team tends to follow in production.

The simplest AI application architecture is a relay. A query goes from the application to the model, and the model's response goes back to the user. There is nothing else in the path. Follow one request step by step: the user submits a question to the application; the application forwards that question to a model; the model produces an answer; and the application passes that answer back to the user. The diagram below traces exactly those parts, in that order.

Notice what the application does not do on this path. It does not look anything up, does not screen the text, and does not remember which answers it has already produced. Its job is only to move text between the user and the model.

The simplest architecture has only three roles in play: the person, the application, and the model. The phrase model API covers two different situations. One is a third-party service offered by a model provider, which the application reaches over the network. The other is a model that the team hosts itself. From the application's point of view, both look the same: a place it sends a request and receives a response. How a self-hosted model is actually served is a separate topic handled in the course's inference server material, so this article treats the model API as a destination for requests.

It helps to name what is missing, because each absence corresponds to a later enhancement. The simplest design has three gaps. It has no context augmentation, so the model sees only what the user typed. It has no guardrails, so nothing checks what goes in or comes out for protection. It has no optimization, so every request is sent to the model fresh, and the system does not try to reduce response time or spend.

From the simple starting point, the material lists enhancements a team might consider: context input enhancement, guardrails, a model router and gateway, caching, and complex logic. Each item answers a different question: where the model gets its information, how the system is protected, how requests are directed, how cost and speed are improved, and how far the system can go beyond answering questions. Monitoring and observability are described as integral to any application, but they are not counted among these enhancements.

The listed order is a typical progression, not a required one. The source is explicit that everyone's needs differ, and that the right order is whichever order makes the most sense for the application being built. Treat the list as a menu with a suggested typical order, and decide from your own requirements which entries you need at all.

Each enhancement is chosen to answer one question. When a team has a specific problem, the useful move is to name the problem first and then pick the enhancement that answers it. The comparison below pairs three stated problems with the enhancement that addresses each one. The pairings are a teaching illustration of the question each enhancement answers, not a complete design rule.