Monitoring Models and Collecting User Feedback

Imagine you are building a customer support assistant and you need a language model to power it. The fastest-looking route is to open a public leaderboard, sort the entries by their scores, and pick the one at the top. Many teams do exactly this. It feels rational: the ranking is public, the numbers look precise, and someone else has already done the testing.

This article examines why that shortcut is incomplete and what a more reliable process for choosing a model looks like. The process is called the model evaluation workflow. It is a four-stage method for deciding which model an application should use, and the rest of the article traces how a pool of candidate models is narrowed as it moves through those stages.

The four stages of the workflow share one defining property: each stage takes a larger pool of candidates and produces a smaller, better-informed one. The work also becomes more expensive and more specific as you move along. Early stages are cheap and coarse. Later stages are costly and tailored to your situation.

The workflow is normally applied in a fixed order, starting with the cheapest filter and ending with the stage that runs once a model is in use. Each stage is described below in the order it is normally applied.

A public leaderboard answers one question well: how did each model perform on the tasks and criteria that someone else chose? It does not answer the question your application actually poses, which is how a model will behave with your users, your wording, and your definition of a good answer. The two sources of information differ in who designed the test and whose needs it reflects.

Within the team's own evaluation pipeline, quality is only one of three objectives. The pipeline balances quality, latency, and cost for the team's specific needs. This matters because these three objectives pull against one another. A model that produces slightly better answers may respond more slowly or cost more at the expected volume of requests, and a faster or cheaper model may give slightly weaker answers.

The source material does not give a formula for weighing these objectives against each other. What it establishes is that the weighing depends on your own needs, which is why the pipeline is built from prompts and metrics you control rather than from a universal ranking.

The workflow is drawn from left to right, but it is not a strict one-way line. The stages are iterative, which means a decision made in an earlier stage can be reopened once newer information arrives from a later one. The return path in the diagram represents that possibility. Production monitoring is one of the main sources of such new information, because it reveals how the chosen model behaves with real users.

The build-or-buy question is worth singling out. Most companies will not build foundation models from scratch, so the real choice is between commercial model APIs and hosting an open source model. This decision can significantly reduce the pool of candidate models, which is why it belongs near the front of the process and why it is often reconsidered later.

The four stages run in order: filter by hard attributes, narrow with public information, experiment with your own pipeline, and monitor in production. Each stage hands a smaller, better-informed pool of candidates forward, and each is more specific and more costly than the one before. Because the stages are iterative, production feedback can reopen an earlier decision, such as the build-or-buy choice.