Inference Servers and Hardware Allocation
A trained model is a set of parameters. A deployed model is a set of parameters plus the machinery that receives requests and runs them on hardware.
A Prompt Before the Model Sees It
Imagine typing a question into a chat window. The words you type do not reach a trained model in any direct way. They leave your device, cross a network, get handled by software on a server, pass through the model's computation, and come back to you as an answer. Each of those steps is part of the machinery that makes a model usable.
This article follows a single request through that machinery. Along the way it asks one practical question: when a request arrives at a deployed model, which piece of software should decide which model to run and which hardware to use? Keep that question in mind, because the answer emerges as the request moves from start to finish.
Why Trained Parameters Are Not Enough
Training a model produces a set of learned parameters. On their own, those parameters do nothing for anyone. A model file sitting on disk answers no questions. Something has to load the model, receive a request, run the model's arithmetic on that request, and send the result back. The software that performs those duties is what turns a set of parameters into a usable service.
A trained model is a set of parameters. A deployed model is a set of parameters plus the machinery that receives requests and runs them on hardware.
Tracing a Request Through the Stages
Start from the moment a user prompt leaves a client application. The first stop is not the model. The request enters an inference service, which is a larger body of software. Its responsibilities include receiving the request, deciding where it should go, and sometimes cleaning up or transforming its contents before anything else sees it. Only after that work is the request handed to the component that actually performs the computation, the inference server.
The diagram below traces that path in order. Read it as a relay. The request moves forward through receiving, routing, and any preprocessing, then reaches the server, which assigns the resources needed to run the appropriate model and produces an output. That output travels back through the same chain to the user.
Service Versus Server
An inference server is the component that runs model inference. It hosts models, has access to the hardware they need, and returns responses based on application requests.
An inference service is the broader unit that contains the inference server and handles receiving, routing, and possibly preprocessing requests before they reach the server.
The relationship between the two terms is containment. The service is the outer shell that the application talks to, and the server is one part inside it. The diagram below shows that nesting. Notice that the server's own responsibilities are narrower than the service's: it holds the models it can run, it is connected to the hardware those models need, and it answers requests by using that hardware to execute the right model.
Following One Request Step by Step
Tracing a Prompt Through a Deployed Model
A user submits the prompt 'Summarize this paragraph' from a client application to a deployed model. Trace which component handles each stage and where the request is in the chain at each point.
- 1Leave the client
The application sends the prompt. The model is not the first stop; the request first enters the inference service.
- 2Receive and route
The service receives the request and decides where it should go, which is the routing responsibility that belongs to the service.
- 3Preprocess if needed
If the contents need cleaning or transforming, the service does that before anything else sees the request.
- 4Hand to the server
The prepared request now reaches the inference server, the component that runs the computation.
- 5Assign resources and run
The server assigns the resources needed to run the appropriate model and executes it on the hardware it can access.
- 6Return the response
The output travels back through the same chain of the service to the user.
The model performs the computation only after the service has received, routed, and possibly preprocessed the request. The server's job is the middle of the journey, not the whole journey.
Using a Model API Versus Hosting Your Own
There are two broad ways to get a model's answers into an application. The first is to call a model API offered by a large provider, such as those from OpenAI and Google. Those APIs are inference services that you consume rather than build. When your application calls one, the provider has already put the receiving, routing, server, and hardware layers in place.
The second way is to host a model yourself. In that case the whole inference service becomes your responsibility. You must build the layers that receive and route requests, optimize how the model runs, and keep the system maintained over time. The comparison below sets the two positions side by side.
| Concern | Third-party model API | Self-hosted model |
|---|---|---|
| Inference service (receiving, routing, preprocessing) | Provided by the provider | You build it |
| Inference server and hardware layers | Already in place | You set them up and connect them to hardware |
| Optimizing how the model runs | Largely handled by the provider; you avoid most of this chapter's techniques | You build and optimize it |
| Maintaining the system over time | Handled by the provider | You maintain the entire inference service |
What each choice leaves for you to build
Common Mistakes About Where Work Happens
Common mistake Treating the inference server and the inference service as the same thing
Why it failsThe service is the broader unit that contains the server and handles receiving, routing, and possibly preprocessing before the server is reached.
Do this insteadUse the service for the outer work and the server for running the model on hardware.
Common mistake Believing a trained model file can answer requests by itself
Why it failsLearned parameters do nothing for anyone on their own. Something must receive the request and run the model.
Do this insteadThink of a deployed model as parameters plus the software that receives, routes, and executes requests.
Common mistake Assuming that calling a hosted API means you still have to build the whole serving stack
Why it failsWhen you use a provider's API, the receiving, routing, server, and hardware layers are already in place, so you avoid implementing most of the chapter's techniques.
Do this insteadDecide first whether you are consuming an inference service or hosting one, because that choice determines what you must build.
Common mistake Assuming routing and preprocessing happen inside the model's arithmetic rather than before the request reaches the server
Checking Your Understanding
A team decides to host its own model instead of calling a provider's API. Name the layers they must now build that the provider would otherwise have supplied, and state which component in the chain is responsible for actually running the model.
Hints
- Think about the outer layers that receive and route requests before the server sees them.
- The server is the component that hosts models, has access to hardware, and executes the appropriate model.
Key Takeaways in Context
Work through the request chain once more in your head: client, inference service, routing and preprocessing, inference server, hardware, and back. Each stage has a distinct job, and the server's job sits in the middle of the chain.
At a glance
- An inference server runs model inference: it hosts models, has access to the hardware they need, and returns responses based on application requests.
- An inference service is the broader unit that contains the server and handles receiving, routing, and possibly preprocessing requests before they reach it.
- A request travels from the client into the service, through routing and any preprocessing, to the server, which assigns resources and runs the model, and the output returns through the same chain.
- A third-party model API is an inference service you consume, so the provider handles most layers; self-hosting makes you responsible for building, optimizing, and maintaining the entire inference service.