Chapter 12: the loop from telemetry to evals and prompt changes
Designing for handoff means shaping the project from its earliest days so the customer's team can run the system after the builders leave.
Why Handoff Starts on Day One
Picture a team that spends months building an AI system for a customer. When the system is running, the builders leave. The people who built it are no longer around, but the system still has to keep working. The question this chapter answers is how a project should be shaped so that the customer's own team can operate the system after the builders step away.
The source treats this as something decided early, not a chore at the end. Designing for handoff means shaping the project from its earliest days. In the same way that reliability or security is a property a system must have from the start rather than a feature bolted on later, handoff is a property the system needs from the beginning.
This chapter follows one thread through the project: the signals you collect early are what let you evaluate the system and adjust it later. The sections below trace where those signals come from, how they reach a live dashboard, and how they feed back into evaluations and prompt changes.
Three Checkpoints Along the Project
The source gives three checkpoints that line up along the life of a project. Instrumentation begins at the first commit. Design for handoff begins in the first week. The customer's dashboard goes live on the day the system enters production. Laid out in order, these checkpoints show that handoff preparation is spread across the whole project rather than concentrated at the end.
What do you think happens?
When does preparing for handoff actually begin in a project?
Reveal answer
Preparation begins early and is spread across the project: instrumentation starts at the first commit, design for handoff begins in the first week, and the customer's dashboard goes live when the system enters production.
The source's three checkpoints line up along the project timeline, so handoff work is distributed over the whole life of the project rather than concentrated at the end.
Instrumentation at the First Commit
Instrumentation means collecting telemetry from the system: the signals that show what it is doing. The source states that instrumentation begins at the first commit because telemetry is what makes a launch possible. It is not something added after the system is already in use.
The reasoning follows from the loop described later in this chapter. Telemetry is the input to evaluation, and evaluation is the input to prompt changes. If the signals were never collected while the system was being built, there is nothing to evaluate when it first runs and nothing to compare against when it changes.
The Live Customer Dashboard
A customer dashboard is usually imagined as a place to read numbers. The source asks for a different view: its main job is to build confidence. When the customer can watch what the system is doing in real time, their worry about the unknown decreases.
This confidence matters for the engineer as well. A customer who can see the system working is more willing to let the engineer make scope decisions while the work is underway. The dashboard therefore supports the delivery itself, not only the reporting that follows it.
The dashboard matters both during and after the delivery. While the builders are present, it is part of how scope is negotiated. After they leave, the same visibility is what the customer's own team relies on to see how the system is behaving. The source's checkpoint places the dashboard going live on the day the system enters production.
The Four Handoff Artifacts
Designing for handoff produces four specific artifacts for the customer's internal team: a runbook, a set of alerts, an eval set, and a playbook. The source describes them as part of the delivery. They are not extra documents written in a rush at the end, and they should be built alongside the system.
- Runbook: part of the delivery, built alongside the system for the customer's team
- Alerts: part of the delivery, built alongside the system for the customer's team
- Eval set: part of the delivery, built alongside the system for the customer's team
- Playbook: part of the delivery, built alongside the system for the customer's team
Each artifact serves a different purpose. Together they let the customer's team notice problems, check quality, and respond. The diagram below shows how the four artifacts connect to the delivered system and to the team that operates it. The source does not spell out each link in detail, so read the edge labels as the purposes the source assigns: noticing, checking quality, and responding.
Following the Telemetry Loop
The source points forward to a loop that runs from telemetry back into evaluations and prompt changes, which this chapter covers in depth. The loop has a clear direction of dependence: telemetry is gathered first, and the evaluation and prompt changes depend on it. The diagram below shows the order of the steps and how the result carries into the next release.
Tracing one failure through the loop
A hypothetical AI assistant for a customer starts giving answers the customer's team considers wrong on a certain kind of request. Trace how the loop would handle it. This scenario is invented to illustrate the loop; the source does not describe this case.
- 1Telemetry shows the failure
Because instrumentation began at the first commit, the production signals show the pattern of problematic requests, not just a vague complaint.
- 2The eval set captures it
The failing kind of request is recorded in the eval set, so the team has a fixed way to check quality on this case.
- 3A prompt change is made and re-checked
The prompt is adjusted, and the eval set is used to see whether the adjusted system now handles the case.
- 4The change ships and produces new telemetry
The next release carries the change, and its behavior generates new telemetry, which closes the loop.
The loop turns a production failure into a recorded check and a targeted change, and the next release's telemetry shows whether the change worked.
Mistakes in Designing for Handoff
Common mistake Treating handoff as a final chore
What it looks likeThe team plans to write the runbook and playbook in the last week before the builders leave.
Why it failsThe source describes handoff as a set of decisions shaped from the project's earliest days, not a task concentrated at the end.
Do this insteadBuild the runbook, alerts, eval set, and playbook alongside the system, starting in the first week.
Common mistake Adding instrumentation after launch
What it looks likeTelemetry is added once the system is already in customers' hands.
Why it failsTelemetry is what makes a launch possible, so the earliest period of the system's life goes unrecorded.
Do this insteadBegin instrumentation at the first commit.
Common mistake Seeing the dashboard as only a place to read numbers
Why it failsThe source's main point is that the dashboard builds confidence, which affects how much the customer trusts scope decisions during the work.
Do this insteadGive the customer a live, visible view of the system from the start, so it supports both trust and decision-making.
Practice: Tracing What Goes Missing
A team begins instrumenting its system only after launch. Using the three checkpoints from this chapter, explain which parts of the project's early life would have no telemetry, and describe what the customer's team would be unable to evaluate or adjust in the first weeks of production as a result.
Hints
- Start from the first commit checkpoint and ask what the system was doing before launch.
- Connect telemetry to the evaluation and prompt change steps of the loop.
- Think about what the dashboard can show when no signals were collected earlier.
Key Takeaways
- Handoff is a property the system must have from the start, shaped by decisions made in the project's earliest days rather than a task at the end.
- Instrumentation begins at the first commit because telemetry is what makes a launch possible and is the input to later evaluation and adjustment.
- A live customer dashboard builds trust, both during the work and after the builders leave, and supports the engineer's scope decisions.
- The runbook, alerts, eval set, and playbook are part of the delivery and are built alongside the system so the customer's team can notice problems, check quality, and respond.
- The telemetry loop runs from production signals into evals and prompt changes, and each release produces the telemetry that feeds the next cycle.