Evaluating Complex AI Systems

The goal of evaluating a complex AI system is not only to observe that a problem exists, but to locate where the problem lives. A good evaluation tells you which stage failed, so that you do not have to guess.

Programming

When a Correct-Looking Answer Hides the Problem

Real-world AI applications are complex. A single application may be built from many components, and a task may not finish in one exchange: it can be completed after many turns. Each part of the system produces its own output along the way, and the overall result depends on all of those outputs together. This means that when you test such a system, the finished product alone rarely tells you enough.

Consider an application that reads a person's resume PDF and reports that person's current employer. Suppose the reported employer is wrong. The final answer is the only thing you can see directly, but it is the product of several steps. The text may never have been pulled out of the PDF correctly. Or the text may have been extracted well, and the employer was picked out of it incorrectly. A single pass or fail judgment on the final answer cannot tell these two cases apart.

Key point

Tracing Data Through the Resume Pipeline

The source uses a concrete application to make this idea precise. The application extracts a person's current employer from their resume PDF, and it works in two steps. The first step extracts all the text from the PDF. The second step extracts the current employer from that extracted text. Read carefully, the design has three distinct values in play: the system's input (the PDF), the first component's output (a block of text), and the second component's output (the employer name).

The important structural fact is that each stage consumes what the previous stage produced. The second step never sees the PDF directly. It only sees the text that the first step handed to it. That dependency is what makes a problem in the first step quietly affect everything downstream.

What does step 1 produce, where does that output go, and what does step 2 receive? Each arrow shows one stage's output becoming the next stage's input.

Notice that the block of text is an intermediate output. It is not what the learner asked for, and it is not the final answer, but it is a real product of the system that can be inspected. Evaluation becomes much more useful once you accept that intermediate outputs deserve their own judgments.

Three Levels of Evaluation

Evaluation can happen at three levels: per task, per turn, and per intermediate output. The same system can be examined at each of these levels, which is why it helps to name them clearly before you start judging anything. The three levels are not competing methods. They describe different slices of the same run, and they nest inside one another.

How do per-task, per-turn, and per-intermediate-output evaluation relate? The whole task contains its turns, and each turn contains the intermediate outputs produced while handling it.

Read the diagram from the top down. The per-task level covers the whole job from start to finish. The per-turn level covers one exchange within a job that may need several. The per-intermediate-output level covers the product of an individual component. The diagram is an illustration of how the levels relate. The turn labels and the two-turn structure are generated for illustration, since the source does not specify a particular number of turns.

Judging Each Component on Its Own

The source gives a clear rule: evaluate the end-to-end output and each component's intermediate output independently. Component-level evaluation assesses the intermediate output of an individual component. End-to-end evaluation assesses the overall output of the whole system. The two answer different questions. A component-level check asks whether a stage did its own job well. An end-to-end check asks whether the system as a whole produced an acceptable result.

Question	Component-level evaluation	End-to-end evaluation
What is judged?	The intermediate output of one component	The overall output of the whole system
What does it ask?	Did this stage do its own job well?	Did the system as a whole produce an acceptable result?
Resume example	Was the extracted text a faithful copy of the PDF content?	Is the reported employer correct?
What it can locate	Which stage produced the problem	Whether a problem exists at all

Two kinds of judgment applied to the same two-step pipeline

A pipeline can pass one kind of check and fail the other. A system can produce a correct final answer while one of its intermediate outputs was sloppy, and a system can produce a wrong final answer even when every stage worked as intended. This is exactly why the source recommends doing both kinds of evaluation rather than choosing one.

Key point

When judging a later stage, use the output the earlier stage actually produced. Do not substitute what you expected that stage to produce. This keeps each judgment honest and lets a problem be located rather than guessed.

How a Bad Intermediate Output Spreads

Suppose the first step produces a bad intermediate output. Because the second step consumes whatever the first step handed over, the error travels forward. The second step may then produce a wrong employer name, even if it performed its own job correctly on the text it received. From the outside, all you see is a wrong final answer.

If step 1 produces an incomplete text block, how does that error reach step 2 and the final answer? The point to catch it is at the text block, before step 2 runs on it.

The diagram shows where the error can be caught. Checking the output of step 1 reveals the missing employer line directly. If you only checked the final answer, you would see a wrong employer and have no direct way to tell whether step 1 or step 2 was responsible. The earlier checkpoint is the one that locates the problem.

Mistakes When Scoring Only the End Result

Common mistake
Judging the system only by its final output
What it looks like
A wrong employer name is marked as a failure, and no one checks whether the PDF text was extracted correctly.
Why it fails
A single pass or fail on the final answer cannot tell apart a failed extraction from a failed selection.
Do this instead
Add a judgment for each component's intermediate output so the failing stage can be identified.
Common mistake
Checking a later stage against what you expected rather than what was produced
What it looks like
You test the employer step using a clean text you wrote yourself, instead of the text the first step actually handed over.
Why it fails
The later stage's real behavior depends on the real input it received, including any flaws in that input.
Do this instead
Feed each stage the output the earlier stage actually produced, then judge that stage's output.
Common mistake
Assuming a correct final answer means every stage worked
Why it fails
A pipeline can produce an acceptable result while one intermediate output was poor, and that fragility will show up later.
Do this instead
Evaluate intermediate outputs even when the end-to-end result looks correct.
Common mistake
Treating per-task evaluation as the only level that matters, even for work that spans many turns

Tracing the Resume Pipeline Step by Step

Worked example

Locating a wrong employer in the two-step pipeline

The resume application reports the wrong current employer. Use component-level evaluation to decide which stage to examine first.

1
Record the system input
The input is the resume PDF. This is what the whole system received.
2
Judge the first output
Inspect the block of text that step 1 produced. Ask whether it contains the employer information that appears in the PDF. Suppose the employer line is missing from it.
3
Judge the second output
Inspect the employer name that step 2 produced, using the text step 1 actually handed over. Because the employer line was absent, step 2 had nothing correct to select from.
4
Locate the failure
The first intermediate output is where the problem begins. Fixing step 2 alone would not address it, because step 2 was working from incomplete input.

Result

The failure lives in the extraction stage, not the selection stage. The end-to-end result was wrong, but the intermediate judgment located the cause.

The example above is illustrative. The specific case where the employer line is missing is generated for teaching. The reasoning it demonstrates follows the source: judge each component's output on its own, using what the earlier stage actually produced.

Practice: Finding the Faulty Stage

EASY

A two-step application reads a resume PDF and reports the current employer. The final employer name is wrong. Step 1 output is a complete block of text that includes the correct employer line. Step 2 output is a wrong employer name. Which component should you investigate first, and why?

Hints

Check which stage's output you have already confirmed as correct.
Step 2 received the text that step 1 produced, so ask what that text contained.
If the text was complete and step 2 still chose wrong, the failure lives in the selection stage.

Your attempt (not saved)

MEDIUM

A conversation with the application takes four exchanges to complete a task. The final result is acceptable, but the second exchange gave an answer that would have misled a user if they had acted on it. Which evaluation levels would catch this, and which would miss it?