Systematic Evaluation Design for AI Applications

Building an AI application is only half of the work. Once something works well enough to demo, the harder question becomes how you know whether it is working, whether it got better after your last change, and whether it still behaves well on the inputs real users send. This article looks at evaluation from two angles: how much attention the topic receives in public open-source work, and how well the supporting tools have kept pace with the rest of the AI stack.

One way to gauge interest in a topic is to count the public code projects devoted to it. The source examined the top 1,000 AI-related repositories on GitHub, ranked by number of stars, and found more than 50 repositories dedicated to evaluation as of May 2024. On its own, that count is a meaningful figure for one slice of the open-source landscape.

The more telling observation comes from timing. When the evaluation repositories are plotted by creation date, the curve rises in a way that looks exponential rather than steady and linear. The number of new evaluation projects is not just increasing; each period has been adding more than the one before. The source's key takeaway is that evaluation has become an active area of building rather than a niche afterthought.

Growth in repositories tells one part of the story. The source then compares the amount of tooling available for different tasks. The number of evaluation tools is small when set against the number of tools for modeling and training, and against the number of tools for AI orchestration. The comparison is relative: evaluation is not missing entirely, but it is clearly underserved next to the areas around it.

The source connects this gap to investment. Inadequate investment leads to inadequate infrastructure, and that shortfall makes it hard for people to carry out systematic evaluations. Read the chain from left to right: less money goes into evaluation, so fewer supporting tools get built, so teams find it harder to evaluate in a disciplined way. The result is that the tools you would reach for to measure quality are thinner than the tools you would reach for to build the model or wire the application together.

The source also reports what people say they do when asked how they evaluate their AI applications. Many said they simply eyeballed the results, reading outputs and deciding whether they looked reasonable. Many others kept a small set of go-to prompts that they ran against models. Follow the state of an evaluation step by step: first the team has an output, then a judgment is made, then the same judgment is reused for the next change.

Notice what the state contains at each point. The judgment lives in one person's head, and the prompt set reflects that person's history. Nothing in that state records what the application actually needs to do, so the state cannot tell you whether the application has regressed or improved after a change.

Read the two columns as a pair. The ad hoc column describes how an evaluation gets done when the judgment behind it is personal and informal. The systematic column describes an evaluation whose prompts are chosen to match the application and which is supported by real infrastructure. The source's concern is that the first column is the common default, and that the second column is what application iteration requires.

The source is fair to the ad hoc habit. An ad hoc approach can be enough when a project is getting off the ground, because early on the main goal is to see whether an idea can work at all. Eyeballing a handful of outputs answers that question quickly. The trouble starts when the same informal method is kept after the application needs to be changed and improved again and again.

In the first stage the informal method matches the goal. In the second stage the goal has changed to improvement across repeated changes, yet the method is the same. This mismatch is the core reason ad hoc evaluation falls short during iteration: the method was never designed to answer whether the latest change helped and whether earlier behavior still holds.

The following scenario is an original illustration, not a case from the source. It shows how the ideas above play out over the life of one project. Think about what the team is relying on at each stage and whether that reliance still fits the job.