Agent Failure Modes and Evaluation

Imagine an agent that has been asked to book a multi-leg trip. It builds a plan, carries out each step, and reports success at the end. Nothing crashed, so from the outside the run looks healthy. Yet the agent may have booked a connection that arrives after the hotel check-in window closes, because nothing in its process ever asked whether the plan made sense.

This is the gap that reflection is meant to close. An agent can keep operating without examining its own work, but it is far less likely to actually accomplish what it was asked to do. The rest of this article explains what reflection is, where it fits in a task, how the parts of a reflective system cooperate to find and fix errors, and what it costs to build in.

Running steps and succeeding at a task are not the same thing. An agent that only executes steps moves forward no matter what each result looks like. If a step produces a wrong result, the next step simply builds on it, and the run ends with a report of completion. Reflection adds a pause after the result arrives, asking whether the outcome is actually right before the agent commits to its next direction.

Reflection is not a single event that happens at the end of a run. It can be placed at several points along the way, and each placement answers a different question about the work so far. There are four such points in a task: after the request arrives, after the initial plan is made, after each execution step, and after the whole plan has run. The question changes as the agent moves further into the task.

Read left to right, the diagram shows the agent's path through a task. The first checkpoint sits right after the request arrives, before any work begins. The second follows the initial plan, asking whether that plan makes sense before it is carried out. The third follows each execution step, so a bad result can be caught while it is still one step old rather than after the whole run. The fourth comes after the whole plan has run, judging whether the task as a whole was accomplished. The trip agent in the opening example would have been caught by the second checkpoint, because a connection arriving after check-in closes is a plan that does not make sense.

Reflection needs a judge, and the judge can be arranged in more than one way. The simplest arrangement uses the same agent. It is prompted with self-critique questions that ask it to look back over its own output. A second arrangement hands the judging to a separate component, such as a specialized scorer, which is a model that assigns a concrete score to each outcome.

Separating the judge from the actor is useful because the agent that produced an answer is often poorly placed to notice that answer's flaws. The same separation can be extended across multiple agents. One agent plans and acts, while another agent evaluates the result after each step or after a set number of steps.

Reflexion, proposed by Shinn and colleagues in 2023, makes the path from a finished step to a corrected direction explicit by splitting reflection into two modules. The first is an evaluator that judges the outcome. The second is a self-reflection module that analyzes what went wrong. After both have run, the agent proposes a new trajectory, which is the authors' term for a plan. Each step therefore passes through evaluation and self-reflection before the agent commits to its next direction.

In the diagram, the evaluator comes first. It receives the outcome of the executed step and judges it. If the judgment is that the task has not been accomplished, that failure is passed to self-reflection, which looks for why the attempt went wrong. The evaluator tells the system that something is wrong; self-reflection tells it where the problem started. Only after both have run does the agent propose a new trajectory, and only then does it move on to the next step.

When the evaluator judges that the task has not been accomplished, the actor is prompted to think about why the attempt failed and how the approach could be improved. That improvement becomes the basis for a new plan. The diagram below follows one flawed plan through that process, from its original form to its replacement. Each arrow is labeled with the event that triggers the move to the next state.

Reflection is relatively easy to add to an agent compared with improving how it generates plans in the first place, and it can produce a surprisingly large gain in performance. The price is paid in latency and cost. Thoughts, observations, and sometimes actions all consume tokens, and that consumption grows with the number of intermediate steps. Users feel the extra delay, and the bill rises accordingly. Reflection also adds context pressure, because the extra thoughts and observations take up room in what the agent is working with.