Designing the Pre-Call Briefing Agent

Imagine an account executive about to join a sales call. A pre-call briefing agent has gathered information about the prospect and summarized it. Every fact in the summary checks out, yet the executive finishes reading and still does not know which of those facts matter for the next twenty minutes. Asking whether the agent got the facts right does not capture this failure at all.

This article looks at how a team judged a pre-call briefing agent in a way that could catch exactly this kind of problem, and how they used the results to improve it. The method is a useful model for evaluating any AI-generated research product. Instead of asking one general question about quality, the evaluation breaks quality into named criteria, runs the agent on real inputs, reads where it misses, changes specific parts of the system, and measures again.

The scenario behind the design is a familiar one. An account executive, the salesperson who owns a client relationship, may have three calls on the calendar in one afternoon. Each call involves a different company with its own history: earlier conversations, recent press coverage, and people on the other side of the table whose backgrounds the salesperson may not remember. Walking into every call with that context in mind takes time the salesperson often does not have.

A useful way to read the design is as a timed process with four stages: a trigger, a gathering step, a generation step, and a delivery step. The trigger is time-based and tied to the calendar. Nothing in the design asks the salesperson to request a briefing. The schedule itself starts the work.

The diagram traces the order of the stages. A scheduled call starts the trigger twenty minutes before the meeting. The agent then gathers information from the four sources, generates one page, and delivers the same document to two places: the account executive's email and a Slack channel the AE monitors.

The delivery step deserves a careful reading. The design sends the same briefing to each destination rather than producing a different document for each. The Slack destination is a channel the AE already watches, not a private message or a new place the AE must learn to check. That means the delivery depends on the AE's habits as well as on the system's output. A portfolio brief should name both channels and note this.

The briefing is assembled from four kinds of information. A portfolio brief should be able to account for every input, so the list is worth memorizing.

These four sources answer different questions. Taken together, a briefing that draws on all four covers the account, its recent context, its history of conversations, and the audience for the next call. Each source contributes something the others cannot, which is why the design lists all four rather than relying on one system of record.

The briefing was scored on a four-criterion rubric. A rubric turns a vague sense that the briefing is good into separate judgments that can be compared across runs. A briefing can do well on one criterion and poorly on another, which is why keeping them separate matters. The table names each criterion and describes what it asks about the briefing.

The evaluation ran during a validation phase in weeks three and four. The team assembled a set of sixty real upcoming calls, and three account executives each scored the briefing on the four-criterion rubric. Using real upcoming calls rather than invented scenarios means the briefings were judged against the questions a rep actually faces when the meeting is coming up.

The first scored run is called the baseline. Before reading its result, consider what a single average tells you and what it hides.

The baseline diagnosis is the step that turns a score into a plan. The source reports only the baseline average and the final average, so the diagram shows those two numbers and does not show intermediate scores.

After the baseline diagnosis, the team changed two parts of the system. The prompt shapes what the agent is asked to produce. The source-prioritization logic governs which information sources the agent favors when it assembles the briefing. Because the salience problem concerned relevance, these two levers were the natural places to act.

The source describes the work as two iterations on these components. Two iterations raised the average from 6.4 to 7.6, and the threshold was cleared. The source does not say which individual change produced which share of the gain, so a brief should not attribute the improvement to one specific edit.

When you present this design, the most common errors come from treating a proposed design as a finished product or from adding details the source does not contain.