Telemetry and Observability in Production Systems
For AI systems, testing does not end at launch, because production behavior is the only signal that fully matches what production behavior is.
Why Launch Is Not the Finish Line
Once a system is running in production and its integration has met the customer's real environment, it is tempting to think the hard part is over. For AI systems, the opposite is closer to the truth. Launch marks the point where the real work of learning from the system begins. This article explains why testing continues after deployment, what makes AI systems different from classical software in this respect, and how the ongoing work is organized into a loop that runs at three speeds.
Keep one question in mind as you read: when a system is live, what tells you whether it is behaving correctly? The answer depends on whether the system is deterministic, and that distinction is the thread running through the whole article.
Testing Ends at Deployment in Classical Software
In a classical software system, the test phase happens before deployment. A developer writes a test, the test passes, and the code ships. The behavior of the code is deterministic, meaning the same input produces the same behavior each time. Because of that, the passing tests carry real weight.
Observation in production still has a role in classical software, but it is a fallback. It catches the cases the tests missed. Under this model, the boundary is clear: testing belongs to the period before the code ships, and observing belongs to the period after, where it acts as a safety net.
Nondeterministic Outputs and the Merged Boundary
An AI system does not behave in the deterministic way classical software does. Its outputs are nondeterministic, which means the system's behavior cannot be fully predicted from a fixed set of pre-deployment tests. Because of this, the test phase in an AI system never really ends. The behavior in production is the only signal that fully matches what production behavior is.
The practical consequence is that no set of checks run before launch can stand in for watching what the system actually does with real traffic. Testing and observing stop being two separate phases and become one continuous activity. The diagram below contrasts the two boundaries.
Read the top row as the classical pattern, where a passing test is strong evidence and observation only catches leftovers. Read the bottom row as the AI pattern, where pre-launch checks are necessary but never sufficient, and observation of production runs continuously alongside testing.
The Telemetry, Feedback, and Iteration Loop
The response to nondeterministic production behavior has three parts. The system ships, it is instrumented so that production behavior can be seen, and production traces are fed back into the prompts and evaluations the system relies on. These three actions form the core of post-deployment work. The ongoing work is organized as a loop made of three stages, described below.
Three Speeds of the Loop
The loop runs at three speeds: per-trace inside the engagement, per-week on the customer dashboard, and per-month back into the vendor's product. All three run at once, and all three are the FDE's responsibility. Each speed has its own scope, so it helps to look at them one at a time.
| Loop speed | Where it operates | Who sees it | Typical scope |
|---|---|---|---|
| Per-trace | Inside the engagement | The FDE working the specific case | A specific production case |
| Per-week | The customer dashboard | The customer | How the system is trending over time |
| Per-month | The vendor's product | The vendor's product organization | Lessons carried into the product itself |
The three loops differ in where they operate and who sees them. The scope column is drawn from the descriptions of each loop; the visibility column follows from where each loop operates.
A single production event can be relevant to more than one loop at once. The per-trace loop handles the specific case, the per-week loop shows the customer how the system is trending, and the per-month loop carries the lesson into the vendor's product. Keeping the three distinct is what allows the work to be divided and tracked correctly.
Sorting Post-Launch Work into a Loop
When a piece of post-launch work appears, the practical question is which loop it belongs to. Use the scope of each loop as the test. Ask first whether the work responds to one specific production case inside the engagement. If it does, it is per-trace work. If not, ask whether it concerns how the system is trending for the customer on the dashboard. If so, it is per-week work. Otherwise, it concerns a lesson to carry into the vendor's product, which is per-month work.
Routing a Single Piece of Post-Launch Work
After launch, the team notices that one customer request produced a response that ignored part of the instruction. The team plans to adjust the prompt so this kind of case is handled. Which loop does the work belong to?
- 1Check the scope against one specific production case
The work starts from one trace in the engagement, a single request and its response. That makes it concern a specific production case, so the first decision answers yes.
- 2Stop at the first matching loop
Because the work concerns one specific case, it is per-trace work. The per-week and per-month questions do not need to be asked for this item.
- 3Note the change it leads to
The fix is a change to the prompt, which is an iteration. Under the loop, that iteration is shipped back into production and produces new telemetry, so it is still tracked within the per-trace loop until its effect is known.
The work belongs to the per-trace loop, because its scope is one specific production case inside the engagement.
Common Mistakes About Post-Launch Work
Common mistake Treating launch as the end of testing for an AI system
What it looks likeA team ships an AI feature after its pre-launch checks pass and then stops reviewing production behavior.
Why it failsNondeterministic outputs mean the behavior cannot be fully predicted from a fixed set of pre-deployment tests. Production behavior is the only signal that fully matches what production behavior is.
Do this insteadKeep observing production as a continuous part of testing, using the loop to feed traces back into prompts and evaluations.
Common mistake Assuming observation is only a fallback, as in classical software
Why it failsIn classical software, observation catches what the deterministic tests missed. For AI systems, observing production is part of the test activity itself, not a backup to it.
Do this insteadTreat the testing and observing boundary as merged for AI systems.
Common mistake Running only one of the three loops
What it looks likeA team reviews only the monthly lessons for the product and ignores the per-trace cases an FDE is handling.
Why it failsAll three loops run at once, and each has its own scope. A single production event can be relevant to more than one of them.
Do this insteadTrack per-trace, per-week, and per-month work separately so each is divided and followed correctly.
Common mistake Confusing the three stages with the three speeds
Why it failsTelemetry, feedback, and iteration describe the stages of one cycle. Per-trace, per-week, and per-month describe how fast and in what scope the loop runs. Mixing them up makes the loop hard to reason about.
Do this insteadUse the stages to describe what happens and the speeds to describe where and how often it happens.
Sorting Practice
For each item below, decide whether it belongs to the per-trace, per-week, or per-month loop. Item A: a fix for one production request where the response ignored part of the instruction. Item B: a summary showing how often a certain kind of response has appeared on the customer dashboard over the last few weeks. Item C: a lesson from many engagements that the vendor's product should handle more consistently. Use the scope of each loop to justify your choice.
Hints
- Ask first whether the item concerns one specific production case inside the engagement.
- A trend shown to the customer on the dashboard points to the per-week loop.
- A lesson carried into the vendor's product points to the per-month loop.
Key Takeaways
- Testing does not end at launch for AI systems, because nondeterministic outputs mean pre-deployment tests cannot fully predict behavior, and production behavior is the only signal that fully matches what production behavior is.
- In classical software, passing deterministic tests carry real weight and observation in production is a fallback; for AI systems, testing and observing merge into one continuous activity.
- The ongoing work is a loop of telemetry (the record of production behavior), feedback (what that record reveals about performance), and iteration (changes to prompts and evaluations shipped back into production).
- The loop runs at three speeds at once: per-trace inside the engagement, per-week on the customer dashboard, and per-month back into the vendor's product, all the FDE's responsibility. Sort work by asking which scope it belongs to.