Building and Maintaining Competitive AI Products

Imagine a team that has just built a prototype AI feature in a short amount of time. It works, it impresses the people who see it, and everyone can see the effort that went into it. It is tempting to conclude that the product is most of the way finished. This article follows a team from a measurable goal to a plan for reaching it, and shows why that conclusion is often wrong.

Once a team has defined measurable goals for an AI product, the next question is how to reach them. Milestone planning is the work of building a plan that moves a product from its current state toward those measurable targets. A milestone is an intermediate target along the way, so milestone planning turns one goal into a sequence of smaller checkpoints.

Milestone planning follows goal-setting because it cannot begin until the goal is measurable. A goal such as handling a share of support tickets automatically can be checked against a number. Without that measurement, there is no way to say which intermediate target has been reached. The plan is also not written in the abstract. The route depends on the starting point, and the starting point is usually a set of existing models whose abilities can be measured before any custom work begins.

Evaluating existing models is how a team learns its starting point. The evaluation shows what the models can already do, which makes the effort needed to reach each goal visible. The principle is direct: the stronger the off-the-shelf models are, the less work the team has to perform, and weak baselines mean more building.

Evaluation does more than measure progress. It can also change the goal itself. Suppose the work needed to bring an application up to the level where it is useful turns out to cost more than the value the application could return. In that case, the team may decide not to pursue the product at all. For this reason, milestone planning includes a checkpoint where the goal is revisited, not only a schedule for reaching it.

A very common belief holds that if a prototype already looks impressive after a short build, the product must be most of the way finished. The reasoning feels sensible. When effort is visible and results are visible, people tend to assume the remaining distance is proportional to what has already been done. The source gives two reasons this pattern appears. Foundation models already have impressive base capabilities, so a convincing demo can be built quickly. But that speed hides the work of refining behavior, and the source names product kinks and hallucinations as areas that consumed a large share of the time.

The central pattern is that the last mile of quality often takes far longer than the first stretch of progress. Early on, a team can move a product a long way with a strong existing model and a small amount of work. Near the final quality level, each further improvement tends to require much more effort. This is why early success can mislead: the first stretch looks like the whole story when it is only the beginning of the route.