Preprocessing Requests for Model Serving

Training a model produces a set of learned parameters, but those parameters do nothing for anyone on their own. Someone typing a question into a chat window never touches the parameters directly. Their words travel across a network, get handled by software, run through the model, and come back as an answer. Everything between the user's keyboard and the model's arithmetic is the machinery that makes a model usable, and that machinery is the subject of this article.

Keep one practical question in mind as you read. When a request arrives at a deployed model, which piece of software should be responsible for deciding which model to run and which hardware to use? The rest of the article answers that question by following one request from start to finish.

Imagine a single user prompt leaving a client application. The first stop is not the model. The request enters an inference service, which is a larger body of software whose responsibilities include receiving the request, deciding where it should go, and sometimes cleaning up or transforming its contents before anything else sees it. Only after that work is the request handed to the component that actually performs the computation.

Read the diagram as a relay. The service is the outer shell that the application talks to. Inside it, routing and any preprocessing happen first. The inference server is the stage that receives a prepared request, assigns the resources needed to run the appropriate model, and produces an output. That output then travels back through the same chain to the user. The server is not the whole story: a request can be received and reshaped before the server ever sees it, and the service is the part responsible for that front portion.

Each part of that definition describes a responsibility. Hosting models means the server holds the models it is able to run. Having access to hardware means it is connected to the computing resources those models need in order to run. Returning responses means it takes an application request, uses the right model and hardware to execute it, and sends back what the model produced.

The server's job is to turn a prepared request into a model response. The request arrives in a form the server can work with, the server assigns the resources required to run the appropriate model, and the model executes on that hardware. The result is the response. Notice that the server's work begins only after the request has been received and prepared, which is why the surrounding service matters.

The relationship between the two terms is one of containment. An inference service is the broader unit, and an inference server is one part of it. The service handles receiving, routing, and possibly preprocessing requests before they reach the server. The server's own responsibilities are narrower: it holds the models it can run, it is connected to the hardware those models need, and it answers requests by using that hardware to execute the right model.

A useful way to hold this in memory is to ask, for any given piece of work, whether it happens before or after the request reaches the server. Receiving, routing, and preprocessing belong to the service. Choosing and executing a model on available hardware belongs to the server. Both are needed for a working deployment, but they are separate jobs.

There are two broad ways to get a model response. The first is to call a hosted model API, such as those offered by OpenAI and Google. These are inference services that you consume rather than build. When your application calls one of these APIs, the provider has already put the receiving, routing, server, and hardware layers in place. If you rely on such a service, you will not be implementing most of the techniques covered in this chapter.

The second way is hosting a model yourself. In that case the whole inference service becomes your responsibility. You must build the layers that receive and route requests, optimize how the model runs, and keep the system maintained over time.