Setting Measurable Goals for AI Products

Once a team has defined measurable goals for an AI product, the next question is how to reach them. A goal such as handling a share of customer support tickets automatically tells the team where it wants to end up, but it does not say how to get there. Planning the route is the work that comes next.

This article follows one idea from start to finish: the route a team should take depends heavily on where it begins. Before reading on, hold a simple question in mind. If a product already does part of the job, how much of the job is actually left?

Milestone planning comes after goal-setting because a goal alone is too distant to act on directly. Intermediate targets break the distance into stages. Each stage is measured, so the team can see whether the product is moving toward the final target.

The plan is not written in the abstract. The starting point is usually a set of existing models whose abilities can be measured before any custom work begins. That measurement is what makes the plan concrete.

Evaluating existing models reveals the baseline, meaning the level of performance a team gets before doing any custom work. The baseline is the first point on the route. A stronger baseline means less custom work is needed to reach the goal, and a weak baseline means more building.

The evaluation also makes the effort visible. It shows what the models can already do, so the team can see how far each intermediate target sits from the starting point. The diagram below shows how milestones sit between the baseline and the final measurable goal.

A very common belief goes like this: if a prototype already looks impressive after a short build, the product must be most of the way finished. The reasoning feels sensible. When effort is visible and results are visible, people tend to assume the remaining distance is proportional to what has already been done.

The source gives two reasons this pattern appears. First, foundation models already have impressive base capabilities, so a convincing demo can be built quickly. Second, that speed hides the work of refining behavior. The source names product kinks and hallucinations as areas that consumed a large share of the time. The early success is real, but it says little about how much refinement is still ahead.

Early success can mislead, because the last mile of quality often takes far longer than the first stretch of progress. The first improvements tend to come quickly, since the base model already does a lot. As the product approaches its final quality level, each further gain tends to require more work than the one before it.

Because of this, a team cannot judge remaining effort by looking at how fast it has moved so far. The pace of early progress is not a reliable guide to total effort. The most demanding stretch lies close to the target, which is exactly where a team is most tempted to believe it is nearly done.