Instrumenting AI Systems from the First Commit

Imagine an AI system has just been switched on for a customer's real users. The customer has spent months waiting for this moment, and they are watching closely. Their first question is rarely about model quality. It is usually some version of: what is this thing doing right now? The answer to that question is what the customer dashboard is for.

This article looks at three connected ideas from the same source passage. The first is timing: the dashboard should be live from the day the system enters production. The second is purpose: the dashboard is better understood as a trust-building mechanism than as a metrics screen. The third is handoff: the delivery has to be designed so that the customer's internal team can keep the system running after the delivery team leaves.

The point of this question is that it shows what a customer needs in the first hours of a live system. They are not primarily asking whether the system is clever. They are asking whether it is behaving as expected, in the moment, for their users. A dashboard that answers this question is doing the job the source assigns to it.

The timing rule is simple to state and easy to underestimate. The dashboard is not a report that gets assembled once the project has settled down. It is live and visible from the moment the system enters production. Visibility in this sense means the customer can look at it and see the system's current behaviour, not a summary prepared for them afterward.

This timing ties into a broader point in the same passage: instrumentation should start early. The source says to instrument the system from the first commit, and it describes telemetry as the thing that makes launch possible rather than something added after launch. A live dashboard is only as informative as the signals feeding it, so the dashboard and the instrumentation behind it belong to the same early plan.

Read the timeline from left to right. Instrumentation begins at the first commit, the system enters production, and the dashboard is live at that same moment. The alternative branch, a dashboard added after launch, is the one the source warns against. The reason is that the customer's questions arrive on production day, so a dashboard that arrives later leaves them without an answer during the period when they are most watchful.

It is tempting to think of a dashboard as a metrics screen, a place where numbers are collected and charted. The source deliberately pushes against that picture. It describes the dashboard as a trust-building mechanism more than a metrics mechanism. The dashboard's main job is not to be a complete analytical tool. Its job is to make the customer feel able to see what is going on.

The source gives two results of that visibility. The first is that the customer's anxiety drops when they can see, in real time, what the system is doing. The second is that the FDE's authority to make in-flight scope decisions rises. A customer who can see the system is more likely to trust the people running it, and that trust is what gives the delivery team room to make decisions while work is underway.

The second practical pattern in the source is about who runs the system after delivery. The source states plainly that the FDE is going to leave, and that the system has to keep running. Those two facts together mean that a handoff is not an event that happens at the end of a project. It is a property the system must be built with from the start. The source's guidance is to design for handoff from the first week.

The source names four things that must be ready for the customer's internal team to take over: a runbook, alerts, an eval set, and a playbook. It makes a specific claim about them. They are not afterthoughts that get written when the project is nearly finished. They are part of the delivery itself, which means they are a deliverable in the same sense that the working system is. The diagram shows the four artifacts as parts of the delivery, each one feeding the customer's internal team's ability to take over.

The source does not go into the internal contents of each artifact in this passage, so this article does not describe them in detail. What the source does establish is their role: each is necessary for the team that stays behind to keep the system running after the FDE is gone.