Evaluating AI Application Outputs
Application development is the process of using a model to build an application by supplying good prompts and the context the model needs.
Why a Capable Model Still Needs a Developer
Models have become easy to get hold of, so almost anyone can now start using one to build something. The layer where this happens is called application development. Within the broader picture of building AI systems, it is the layer that has drawn the most activity in roughly the last two years, and it is still changing quickly.
What do you think happens?
If a model's answer reads smoothly and confidently, is it safe to assume the answer is accurate for your task?
Reveal answer
No, fluent wording says little about whether the answer is correct.
The source explains that a model can produce output that reads well and still be wrong for the task, and that a fluent answer gives little signal about its accuracy. This is the reason the layer requires deliberate evaluation.
Keep one question in mind as you read: if a model is already capable, what work is left for the developer? The rest of this article answers that question.
From a General Engine to a Specific Job
A useful way to think about this is that the model supplies a capability, and the developer decides how that capability is aimed. A model on its own is a general-purpose engine that responds to whatever text it receives. An application is something narrower. It has a particular job, particular users, and a particular shape of output.
Application development is the work of closing that gap between a general engine and a specific job. The developer does not change what the model can do. The developer shapes which of its abilities gets used, in what form, and for whom.
Imagine a small team wants a tool that turns long meeting notes into a five-bullet summary for a project manager. The model can already summarize text. The application work is deciding that the output must be exactly five bullets, that the reader is a project manager, and that the summary should focus on decisions and owners rather than small talk.
Tracing the Two Inputs Into the Model
Two inputs do most of the work in this layer. The first is the prompt, which is the instruction the model reads. The second is the context, which is the material the model needs in order to act on that instruction sensibly. The two are related but not the same.
Follow the path in the diagram below. The prompt and the context flow into the model. The model's output becomes the application's result. That result must then pass through evaluation before it reaches users through an interface. Evaluation and interface are not optional stops at the end of the path. They sit between the model and the people who use the application.
Notice the direction of the arrows. The prompt and the context enter the model. The model's output leaves it and becomes the application's result. Nothing reaches the user directly from the model, because the source places evaluation and interface on the path in between.
Prompt Versus Context
The prompt states the task and the expected form of the output. The context supplies the background that the instruction does not contain. Each can be examined on its own, and each shapes what the model produces in a different way.
| Aspect | Prompt | Context |
|---|---|---|
| Role | The instruction the model reads | The material the model needs to act on that instruction sensibly |
| Answers the question | What task to do and what form the output should take | What background the instruction does not already contain |
| Shapes the output by | Setting the task and the expected form | Supplying the facts and situation the model would otherwise lack |
| Typical question it resolves | What should the model do? | What does the model need to know to do it well? |
The prompt and the context are related but distinct inputs
Returning to the meeting-notes tool: the prompt says to produce exactly five bullets focused on decisions and owners. The context is the set of meeting notes plus a short description of the project, which the model needs in order to know which decisions matter. Changing the prompt changes the form of the output. Changing the context changes which facts the output can draw on.
Where Evaluation Sits in the Layer
The source describes this layer as one that requires rigorous evaluation. Evaluation is the deliberate step of testing an application's outputs against what the job actually requires, rather than trusting the first pleasing result. The outputs must be checked against the job before they are trusted.
The map below shows what contains what. The model supplies the capability from outside the application. The application layer is where the developer works, and evaluation and interface both belong inside that layer.
Why a Fluent Answer Is Not Enough
A model can produce output that reads well and still be wrong for the task. Because of this, a fluent answer gives little signal about its accuracy. The first pleasing result is not evidence that the application works.
Suppose the summary tool returns a neatly written five-bullet summary. It looks professional. Evaluation asks a different question: do the bullets name the decisions that were actually made in the notes, and do they name the owners correctly? A summary can pass a reading test and still invent a decision that never happened. Only a check against the job reveals that.
Revising Prompts and Context After a Failed Check
Evaluation changes what happens next. When an output fails the check, the developer does not simply accept it. The flow below shows the loop: the output is evaluated against the job, and a failure leads to a revision of the prompt or context, followed by another attempt.
Tracing one failed evaluation through the loop
The meeting-notes tool returns a summary that lists a decision the notes never mention.
- 1Evaluate the output
The developer checks each bullet against the notes and the job. The invented decision fails the check, so the output is not trusted.
- 2Locate the cause
The developer asks whether the prompt is unclear about the task, or whether the context is missing or misleading. Suppose the notes were pasted in with the wrong meeting attached, so the context lacks the real decisions.
- 3Revise the input
The developer corrects the context so it holds the right notes, and keeps the prompt's requirement of exactly five bullets focused on decisions and owners.
- 4Run and evaluate again
The model runs on the revised inputs. The new output is checked against the job, and it is trusted only if it passes.
The failure is traced to an input, the input is revised, and the output is re-evaluated before it reaches users.
Designing the Interface Alongside the Rest
The interface is the surface through which a person gives the application input and receives its results. The source states that good applications demand good interfaces. A model may produce an excellent answer, yet the application still fails if that answer is hard to find, hard to act on, or presented without the information the user needs to judge it.
Design the interface alongside the prompts and evaluation, not after the model work is finished. An interface that shows the output without the evidence a user needs to judge it can undo the value of careful evaluation.
Mistakes Developers Make With Outputs
Common mistake Trusting the first output because it reads well
What it looks likeA neatly written summary is shipped without checking its bullets against the meeting notes.
Why it failsA fluent answer gives little signal about accuracy, and a model can be wrong for the task while sounding right.
Do this insteadEvaluate every output against what the job actually requires before trusting it.
Common mistake Treating evaluation as a final step after the application is done
Why it failsThe source treats evaluation as a required part of this layer that sits on the path between the model and users.
Do this insteadBuild evaluation into the layer from the start, as part of the path outputs must travel.
Common mistake Mixing up the prompt and the context
What it looks likeA developer puts the background notes into the instruction and then wonders why the output format is inconsistent.
Why it failsThe prompt states the task and expected output form, while the context supplies background the instruction does not contain.
Do this insteadWrite the task and output form in the prompt, and supply the background material separately as context.
Common mistake Designing the interface last
Why it failsA good output can still fail users if it is hard to find, hard to act on, or missing the information needed to judge it.
Do this insteadDesign the interface alongside the prompts and evaluation.
Tracing a Support Reply Through the Layer
A team builds an application that drafts replies to customer support emails. Its prompt asks for a polite reply of under 100 words that answers the customer's question. Its context is the customer's email and the relevant product policy. A draft reply reads smoothly but quotes a refund period that is not in the policy. Trace what should happen next, naming the inputs, the evaluation step, the revision, and the interface.
Hints
- Start by identifying which of the two inputs is the instruction and which is the background material.
- Ask whether a smooth-reading draft has been checked against the job, and what the check should compare the draft against.
- If the check fails, decide whether the prompt or the context needs revising, and then describe the new attempt before the reply reaches the customer interface.
Key Takeaways
- Application development is the work of using a model to build an application by supplying good prompts and the context the model needs, closing the gap between a general engine and a specific job.
- The prompt states the task and the expected form of the output, while the context supplies background the instruction does not contain.
- Outputs must pass through rigorous evaluation against the job, because a fluent answer gives little signal about accuracy.
- Evaluation and interface sit on the path between the model and users, and the interface should be designed alongside the prompts and evaluation rather than after them.