Evaluating Vendors for Forward Deployed AI

When an organization looks for an outside provider of forward deployed AI work, the vendor's own presentation is usually one of the first things it sees. Case studies, client logos, and confident claims about capability fill the pitch. These materials are useful, but they share one feature: they show the successes. A case study rarely describes the project that went badly, and a logo wall says nothing about the engagements that ended in disappointment.

This article asks a different question. How can you learn about a vendor's experience from the way they talk about something that went wrong? The source treats this as one of the more practical tests available during vendor evaluation. To see why, we first need to look at a common assumption that gets in the way.

A common assumption goes like this: a vendor who presents a spotless history, with no failed projects mentioned and every story ending well, must be the safest choice. The logic seems reasonable on the surface. If the vendor is experienced and honest, why would they hide anything? The problem is that this reasoning treats a missing failure story as evidence of good performance. The source points in the opposite direction.

The source makes a specific claim about forward deployed (FDE) work: every working FDE function has failed engagements. Failure is a normal part of this kind of work. A vendor who claims to have none is therefore not describing a flawless operation. By the source's own standard, they are describing something that does not match experience. The summary of the source puts it directly: a vendor who cannot describe a failed engagement is either inexperienced or dishonest.

The source describes a different kind of answer. A vendor who can describe a failed engagement in specific detail has been doing the work long enough to develop real judgment. The strength of the answer lies in its specificity. A vague statement such as we learned a lot from a difficult project does not show judgment. A specific account does, because it names concrete things that a person could only know by having been there.

The source gives a structure for what a specific account contains. It has four elements: the customer's industry, the problem that was being addressed, what went wrong, and what the vendor would do differently. Each element tests something different, so an evaluator can check each one separately.

The four elements are not interchangeable. An account that names a real industry but cannot say what went wrong has shown some context without showing judgment. An account that names a mistake but never says what the vendor would change has shown that something went wrong without showing that anything was learned. A useful answer covers all four, because together they show that the vendor did real work, understood it, and changed as a result.

Consider two answers to the same interview question: tell us about a project that did not go as planned. The example below is an illustration written for this article, not a case from the source. Read each answer and check it against the four elements before reading the evaluator's notes.

The source calls honesty about failure the differentiator between vendors. Many vendors can look similar on paper: similar logos, similar capability claims, similar case study formats. What separates them is how they talk about what went wrong. A vendor with real judgment and a vendor presenting only a polished record may sound equally confident about their successes, but they diverge when asked about failure.