Optimizing Model Inference
The inference server is the stage that decides which model runs and on which resources, then produces the output. Its responsibilities are narrower than the whole inference service that surrounds it.
The Question Behind Every Deployed Request
Training a model produces a set of learned parameters, but those parameters do nothing for anyone on their own. A person typing a question into a chat window never touches the parameters directly. Their words travel across a network, get handled by software, run through the model's arithmetic, and come back as an answer. Everything between the user's keyboard and the model's arithmetic is the machinery that makes a model usable, and that machinery is what this article examines.
What do you think happens?
When a request arrives at a deployed model, which piece of software should be responsible for deciding which model to run and which hardware to use? Make your best guess before reading on.
Reveal answer
The inference server. It holds the models it can run, is connected to the hardware those models need, and assigns the resources needed to run the appropriate model.
The rest of the article follows one request from start to finish to show why this job belongs to the server and why the software around it handles the work that comes before.
Following a Request Through the Relay
Imagine a single user prompt leaving a client application. The first stop is not the model. The request enters an inference service, a larger body of software whose responsibilities include receiving the request, deciding where it should go, and sometimes cleaning up or transforming its contents before anything else sees it. Only after that work is the request handed to the component that actually performs the computation.
Read the diagram as a relay. The service is the outer shell that the application talks to. Inside it, routing and any preprocessing happen first. The inference server is the stage that receives a prepared request, assigns the resources needed to run the appropriate model, and produces an output. That output then travels back through the same chain to the user. The server is not the whole story: a request can be received and reshaped before the server ever sees it, and the service is the part responsible for that front portion.
What the Inference Server Does
An inference server is the component that runs model inference. It hosts models, has access to the hardware they need, and returns responses based on application requests.
The definition has three parts, and each one describes a responsibility. Hosting models means the server holds the models it is able to run. Access to hardware means the server is connected to the computing resources those models require. Returning responses means the server answers requests from applications by using that hardware to execute the right model. Taken together, these are what turn an incoming request into a model response.
The inference server is the stage that decides which model runs and on which resources, then produces the output. Its responsibilities are narrower than the whole inference service that surrounds it.
The Service Wrapped Around the Server
The relationship between the two terms is one of containment. An inference service is the broader unit, and an inference server is one part of it. The service handles receiving, routing, and possibly preprocessing requests before they reach the server. The server, by contrast, takes over once a request is ready to be run.
A useful test for any piece of work is to ask whether it happens before or after the request reaches the server. Receiving, routing, and preprocessing belong to the service. Choosing and executing a model on available hardware belongs to the server. Both are needed for a working deployment, but they are separate jobs, and mixing them up is one of the most common errors when reasoning about a serving system.
Tracing One Prompt Through the Layers
A single prompt from client to response
A user types a question into a chat application. Trace which layer handles each step, and mark the point where the request crosses from the inference service into the inference server.
- 1Step 1: The application sends the prompt
The request leaves the client application. It has not reached the model yet, and the model's parameters have not been touched.
- 2Step 2: The service receives it
The request enters the inference service, which is the outer shell the application talks to. Receiving is a service responsibility.
- 3Step 3: The service decides where it goes
Routing happens inside the service, before the server sees anything. The routing decision is part of the service's front portion.
- 4Step 4: The service may reshape the contents
Preprocessing, if it is applied, cleans up or transforms the request. It still belongs to the service, not the server.
- 5Step 5: The server takes over
The prepared request crosses into the inference server. The server assigns the resources needed for the appropriate model and runs it.
- 6Step 6: The response returns
The output travels back through the same chain, service first, and reaches the user as an answer.
The service handled everything that happened before the server, and the server handled model selection, resource assignment, and execution. The crossing point between them is the boundary to remember.
Mistakes When Reading the Serving Stack
Common mistake Treating the inference server and the inference service as the same thing
What it looks likeSaying the server receives and routes the user's request, so the server must also be where preprocessing happens.
Why it failsThe service is the broader unit that contains the server. Receiving, routing, and possibly preprocessing sit in the service, before the server.
Do this insteadAssign front-end work such as receiving and routing to the service, and assign model selection, resource assignment, and execution to the server.
Common mistake Assuming the model parameters alone answer a user's request
What it looks likeBelieving that once training finishes, the learned parameters can respond to a question directly.
Why it failsThe parameters do nothing for anyone on their own. Software must receive the request, route it, run the model on hardware, and return the answer.
Do this insteadThink of the parameters as one input to a larger system that includes the service, the server, and the hardware.
Common mistake Assuming a model API removes the need to understand the stack
What it looks likeBelieving that calling a hosted API means no layer of the serving system matters to you at all.
Why it failsThe provider has already built the layers. You still need to understand what the layers do so you can reason about requests and their results.
Do this insteadUse the layer model to describe what happens to your request, even when you are not the one building those layers.
Third-Party APIs Versus Hosting Yourself
Model APIs offered by large providers, such as those from OpenAI and Google, are examples of inference services that you use rather than build. When your application calls one of these APIs, the provider has already put the receiving, routing, server, and hardware layers in place. The other path is hosting a model yourself. In that case the whole inference service becomes your responsibility, including the layers that receive and route requests, the work of optimizing how the model runs, and keeping the system maintained over time.
| Concern | Third-party model API | Hosting a model yourself |
|---|---|---|
| Inference service | Built and run by the provider | Built and run by your team |
| Receiving and routing | Already in place | You must build these layers |
| Inference server and hardware | Already in place | You must set up access to the hardware |
| Optimizing how the model runs | Largely handled by the provider | Your responsibility |
| Maintenance over time | Handled by the provider | Your responsibility |
What each choice leaves for you to build
If you rely on a hosted model API, you will not be implementing most of the techniques covered in this chapter. If you host the model yourself, you are responsible for the entire inference service, including the parts that run before the server.
Checking Your Reasoning About Requests
A team hosts its own model. A request arrives, and an engineer asks which component is responsible for deciding which model to run and which hardware to use. Then the engineer asks which component is responsible for rewriting the request before it arrives at that decision point. Name the layer responsible for each question, and explain which work belongs to the service and which belongs to the server.
Hints
- Think about which software decides the model and hardware, and whether it is the outer shell or the stage inside it.
- Preprocessing happens before the server sees the request, so ask which layer sits in front of the server.
- The answer to the first question is the server, and the answer to the second is the service.
List three pieces of work a team would take on if it hosted a model itself, and sort each one into service work or server work.
Hints
- Receiving, routing, and preprocessing are service work.
- Hosting models, accessing hardware, and executing the right model are server work.
Key Takeaways
- An inference server is the component that runs model inference. It hosts models, has access to the hardware they need, and returns responses based on application requests.
- An inference service is the broader unit that contains the inference server. It receives requests, routes them, and possibly preprocesses them before they reach the server.
- The server's job starts when a prepared request arrives. It assigns the resources needed for the appropriate model, produces an output, and that output travels back through the service to the user.
- Using a third-party model API means the provider has already built the receiving, routing, server, and hardware layers. Hosting a model yourself makes your team responsible for the entire inference service, including optimization and maintenance.