Measuring Success in Deployed AI Systems

Imagine a team that has built a system for sorting incoming support tickets by topic. In testing, the system works well, and the builders are proud of its classification accuracy. Then the support managers report a problem: tickets still take about two days to get resolved. The sorter is doing its technical task correctly, yet the outcome the managers care about has not changed.

This gap between a working system and a useful one is the subject of this article. The Jobs-to-be-Done lens is a way of asking what the customer actually needs done, and of checking whether a deployed AI system is built around that need. Before reading on, pause and consider what the team would need to ask first, and why accuracy alone might not change the outcome the managers care about.

Many AI projects begin with a sentence like this: we have a capable model, so let us find a place to use it. The team then looks for a feature to attach the technology to, and the project is judged by how well the technology performs. This starting point feels productive because it begins with something concrete, but it quietly assumes that the product's purpose is to run the AI. The Jobs-to-be-Done lens exists to test that assumption before any system is built.

The lens replaces the question what should our AI do? with a different question: what job did this person hire your product to do? The word hire is deliberate. It treats the customer as someone who has a task in their life or work and who brings in a product to get that task done. The product is judged by whether it did the job, not by whether it contains impressive technology.

Applied to AI deployments, the answer to the hiring question is rarely use the AI. The answer is usually an outcome in the customer's work. A useful way to see the difference is that a feature describes what the product contains, while a job describes what the customer is trying to accomplish. A job is therefore not a restatement of a feature with the word customer added in front of it. It is a fact about the customer's situation that the product is supposed to change.

Consider an invented example to make this concrete. A team building a tool for a clinic's scheduling staff could describe its feature as a model that predicts no-shows. The job the staff hired the tool for might instead be closer to keeping the appointment book full without spending the morning on phone calls. The first statement names a mechanism. The second names a change in the staff member's day. Only the second tells the team whether the product has worked.

A job is expressed in the customer's language and measured in the customer's metrics. This is a deliberate discipline. It keeps the team focused on the customer's world rather than on the system's internals. When the job is stated as the customer would state it, the team cannot quietly swap in a question about how the system works and call the swap a success.

The contrast between the two vocabularies matters. The customer speaks of tickets resolved, appointments kept, or hours saved. A model reports accuracy, precision, or error rate. Both can be real measurements of the same work, but only one of them is the measure the customer uses to decide whether the product helped. Tracking the model metric alone means the team is measuring something true about the system while leaving the customer's question unanswered.

Stating the job is not yet enough to build anything. The source identifies a distinct step between the customer's job and the technical specification, and it describes this translation as the central work of the deployment. The person doing the translation takes a statement in the customer's language and turns it into a description of what the system must do, what inputs it receives, and how its behavior will be checked against the customer's measure.

The translation is the point at which the job becomes buildable. A job written as a change in the customer's day cannot be implemented directly. A specification can be implemented, because it names the system's behavior, its inputs, and the check that will show whether the job is being done. Follow the chain from the customer's job through the translation to the model metric, and notice where each stage gets its meaning.

If the translation is skipped, the specification is written from whatever the team already knows how to build. The system's behavior then drifts away from the job the customer hired it to do. The source warns that skipping the translation can produce systems that do the technically interesting thing without doing the job. The ticket sorter in the opening example is a case of this drift: its accuracy is real, but nothing in its specification ties that accuracy to the managers' resolution time.