Evaluating Multi-Agent Systems

Ask questions about the customer's needs before you propose or reject a solution, because the answers decide which design fits.

Programming

A Design Already Drawn in the Customer's Head

A customer walks into a design conversation and names the system they want. The architecture is already drawn in their head. There are two easy responses. The first is to nod, say that it sounds good, and start building. The second is to say that their design is wrong and offer something else. Both replies skip the same step: finding out what the customer actually needs.

This article is about that step. It covers how to ask the right questions, how to name the job the system has to do, how to judge whether a multi-agent design fits, and how to offer a different design without fighting the customer for it. The central idea is that the number of agents is not what decides whether a multi-agent system works. What decides it is less visible, and it is the part teams most often neglect.

Asking Before Accepting or Rejecting

The first move in a strong response is a question, not a counterproposal. Before a candidate can judge whether a multi-agent design fits, they need to know the conditions under which the design has worked for this customer or for others. A question such as asking what part of a previous multi-agent deployment worked, and what part was hard, gathers that context without challenging the customer's choice head-on.

The reason for asking first is that the answers decide which design fits. A candidate who has not asked has only the customer's picture of the system to work from. A candidate who has asked has information about what the system must achieve and where the previous attempt struggled. The customer's reasoning becomes evidence rather than an obstacle.

What do you think happens?

A team has a multi-agent design that performs poorly on a task. They can either double the number of agents or redesign how the agents share the context they work from. Which change is more likely to matter, and why?

Reveal answer

Answer:

Redesigning how the agents share context is the change more likely to matter.

The source states that the number of agents is not the lever for effectiveness and that the engineering of the shared context is. Adding agents while leaving the shared context unchanged does not address the mechanism that determines whether the agents behave consistently.

Naming the Job Behind a Surface Request

The customer's proposal is a surface request. Beneath it is a job the system has to do. Consider the scenario the source describes. The customer's team favors the multi-agent shape because they want clear ownership of the compliance review step. The candidate restates this as the real requirement: the review needs a clear boundary. Once the job is named, the candidate can evaluate several designs against it, rather than arguing over the one design that was drawn first.

The source's annotation points to three things the strong candidate did. They took the stated solution seriously enough to ask what worked. They named the underlying job instead of the surface request. And they proposed a path that respects the customer's authority while testing a simpler architecture.

Two Versions of the Same Review Process

Once the job is clear, the candidate can describe two versions of a design that satisfies it. In the first version, the review is a separate process with its own agent, its own logs, and its own sign-off. This matches the picture the customer arrived with. In the second version, the intake and draft work happen in a single workflow driven by one prompt that returns structured output, and the review boundary is kept explicit within that workflow.

Aspect	Separate review agent	Single workflow, one prompt
Shape	Multi-agent, with a dedicated review agent	One workflow returning structured output
Logs and sign-off	Its own logs and its own sign-off	One workflow producing one record
Ease of debugging	Harder to trace across agents	Easier to debug, per the source
Ease of evaluation	Harder to evaluate as a whole	Easier to evaluate, per the source
Explaining to auditors	Harder to explain	Easier to explain to auditors, per the source
Fit with customer's picture	Matches what the customer came in with	Does not match the customer's picture

The two versions of the compliance review design, using the tradeoffs the source names

The source names the tradeoffs of the simpler version directly. It is easier to debug, easier to evaluate, and easier to explain to auditors. Its drawback is that it does not match the picture the customer came in with. A candidate should present both versions honestly. The customer keeps the authority to choose, and validation, not the candidate's opinion, is what settles the question.

Why More Agents Is the Wrong Lever

Multi-agent systems attract attention because they seem to promise a simple kind of scaling. If one agent can do useful work, it seems natural that several agents working together should do more. Many teams reach for this idea first when a task feels too large or too complicated for a single agent.

The misconception to watch for is that effectiveness rises with the number of agents. Under this belief, a weak result is treated as a sign that more agents are needed, and a strong demonstration is taken as proof that the same approach will hold up at scale. The source identifies this as a misconception. Adding agents is not the lever that improves a multi-agent system.

The source's wording is direct: the number of agents is not the lever for effectiveness, the engineering of the shared context is. Adding more agents adds more participants who must stay consistent with one another, but it does not by itself change how well they share what they know. That is why a larger team of agents can still behave poorly.

Shared Context as the Real Lever

Shared context is the information that agents rely on in order to act consistently with one another. When several agents work on a task, the quality of the final result depends heavily on whether each agent can access and build on what the others have done. The source's central claim is that the effectiveness of a multi-agent system depends on how this shared context is engineered.

How does information move between agents, and where does the shared context store sit that every agent reads from or writes to?

The diagram shows the idea in simplified form. Each agent's work becomes part of the information the next agent reads. If that information is engineered well, the agents act consistently with one another. If it is treated as an afterthought, the agents may act on different understandings of the same task.

Read-Heavy Versus Write-Heavy Work

The source sorts tasks into two broad families. The first family is read-heavy and parallelizable, with research, retrieval, and exploration as its examples. The second family is write-heavy and stateful. For this second family the source is blunt: multi-agent setups fail badly, because sub-agents cannot cheaply share context. The distinction turns on what the agents must do with shared information, not on how large the task is.

Which parts of a task can run side by side without conflicts, and which parts change shared state in an order that must be preserved?

Read the two columns as a contrast in what the agents need from each other. In read-heavy work, agents gather or examine information, and their results can be combined afterward without each agent repeatedly changing a common record. In write-heavy, stateful work, agents must keep a shared state consistent as they act, and the source says that sub-agents cannot cheaply share the context this requires.

Judging Whether a Multi-Agent Fit Is Real

A candidate who rejects multi-agent systems outright makes the opposite error from one who accepts them blindly. The source gives a usable test. The four-agent shape works when each agent has a genuinely different context window and tool set. When agents mostly pass the same context back and forth, the shape gets harder to manage. A multi-agent system fits when each agent has a genuinely different context window and tool set, not automatically.

1
Name the job the system has to do, beyond the surface request.
2
Check whether the task is read-heavy and parallelizable, or write-heavy and stateful.
3
For each proposed agent, ask whether it has a genuinely different context window and tool set.
4
If agents would mostly pass the same context back and forth, treat the separation as adding management cost rather than value.
5
Propose the simpler alternative with its honest tradeoffs and a validation plan, and let the results decide.

Mistakes Teams Make When Evaluating Designs

Common mistake
Opening with a verdict instead of questions
What it looks like
A candidate says the design is wrong and cites an outside source before learning anything about the customer's situation.
Why it fails
The customer's reasoning is treated as an obstacle, and the candidate lacks the information needed to judge fit.
Do this instead
Open with questions about what worked and what was hard in past deployments, and treat the customer's reasoning as information.
Common mistake
Accepting the proposed design without naming the job
What it looks like
The team starts building the drawn architecture because the customer described it.
Why it fails
The surface request is not the same as the job the system must do, so the design may not serve that job.
Do this instead
Restate the underlying requirement, such as a clear boundary around the review step, and evaluate designs against it.
Common mistake
Adding agents when the weak result is about shared context
Why it fails
The source identifies the engineering of shared context as the lever, not the number of agents.
Do this instead
Examine how information moves between agents before deciding to add more of them.
Common mistake
Trusting a strong demonstration as proof of scale
Why it fails
Designs that ignore shared context may look impressive in a demonstration but behave erratically in production.
Do this instead
Test with real inputs and real timing, and let validation results decide between designs.
Common mistake
Using multiple agents for write-heavy, stateful work
Why it fails
Sub-agents cannot cheaply share the context that consistent state updates require.
Do this instead
Consider a simpler design for write-heavy, stateful work unless the agents have a genuinely different context window and tool set.

Practice: Judging a Proposed Task

MEDIUM

A customer proposes a multi-agent system in which several agents each search different document collections and report what they find, and the results are combined afterward. Is this a good or poor fit for a multi-agent design? Explain your answer using the read-heavy versus write-heavy distinction, and say what question you would ask the customer first.

Hints

Ask whether the agents must change a shared record as they work, or whether they gather information that is combined later.
Consider which family the search-and-report pattern resembles in the source.
Think about whether the customer's previous experience, if any, would change your judgment.

Your attempt (not saved)

HARD

A second customer proposes a multi-agent system in which several agents each update the same order record in sequence, and each update depends on the previous one. Name the task family, judge the fit, and describe the simpler alternative you would propose along with a validation plan that lets the customer's choice be tested honestly.