Failure Mode Analysis for AI Systems

Picture a team that has spent months building an AI system for a retail customer. The product is nearly ready. The sponsor inside the customer's company is eager to see it working. Someone proposes a simple plan: skip the formal checking stage and run a small pilot in a single store. The pilot will feel like progress. Customers will talk to the system, staff will react to it, and everyone will have something real to point at.

This article follows a situation where a team was tempted by exactly that plan and chose a different path. The path it chose is called a validation gate, and by the end you should be able to explain why that gate has to come before the pilot, how the test material for it is built, and how its results drive a decision to launch or hold back.

The belief behind the shortcut is that a pilot counts as testing. A pilot does put the system in front of real people, and that is useful for learning how users behave. But a pilot produces excitement, while a validation report produces evidence about whether the system meets a standard. A pilot tells you the system is being used. A validation gate tells you whether it is good enough to be used.

The gate's job is not to confirm what the team hopes is true. Its job is to expose gaps in performance that the team cannot see from the inside. Because the gate produces a measured result, the team gets something concrete to fix and something concrete to retest against. A pilot, by contrast, can look successful while the underlying performance gaps remain hidden.

The order of events matters. The validation gate is placed before the pilot, not after it. The timeline below shows the sequence the retail case followed, with the gate standing between the build and the pilot. A system that has not cleared the gate does not move on to the pilot stage at all.

Read the sequence as a gate with a single passage through it. The gate sits between the build and the pilot. In the source case, the system's failures were discovered at the gate, and the pilot was not allowed to begin until the threshold was met.

A validation gate needs test material, and the material in this engagement was called an evaluation set. It was assembled in week three from two sources. The first source was real interactions. Eighty cases came from actual conversations between customers and associates, transcribed from the company's call recordings. The second source was expectation. Twenty cases came from questions the company's own training material listed as common.

The lesson of this step is that a single aggregate score hides structure. The team needed the gate to show where performance was lost, not merely that it was lost. Only named failure modes can direct targeted repairs.

Once the failure modes were named, the team spent two weeks repairing them. The loyalty-tier problem required tighter integration with the company's CRM, along with an explicit instruction in the prompt for handling the tiers. The clearance-markdown problem required a daily refresh of the rule information into the agent's tool-use context, so the system worked from current rules rather than stale ones. The retest run at the end of week five reached 73 percent, and the threshold was cleared.

Read the diagram as a loop with one exit. The decision point asks whether the measured result meets the threshold. A failing result sends the work back through naming, repair, and retesting. Only a passing result allows the pilot to begin. The team stayed at the decision point until the threshold was cleared.