Data Access in AI Engineering
A cargo cult engagement has the title and vocabulary of a successful AI engagement but lacks the operating model that makes it work.
Same Title, Different Project
A common assumption in AI work is that a project becomes a real AI project once it is described in the right language. The team has engineers with AI-oriented job titles, the kickoff uses the vocabulary of modern machine learning, and the work is labeled as an AI engagement. Under that surface, however, the project may run like any ordinary consulting job with a few new words attached.
What do you think happens?
A project has AI job titles, uses machine learning vocabulary at kickoff, and is labeled an AI engagement. Does that prove it has the operating model that makes AI work succeed?
Reveal answer
No. The title and vocabulary can be present while the operating model is missing.
The source describes this gap directly: an engagement can have the title and vocabulary of a successful AI engagement but lack the operating model that makes it work. Vocabulary describes a project; it does not establish how the project is run.
This article follows one case in which exactly that happened and teaches a way to tell the two apart. The method is to trace the state of the engagement, compare it against the elements a working engagement needs, and ask a small set of diagnostic questions in a fixed order.
Defining a Cargo Cult Engagement
A cargo cult engagement has the title and vocabulary of a successful AI engagement but lacks the operating model that makes it work. The name points to the outward form of success without the underlying mechanism that produces it.
The distinction matters because the two kinds of engagement look similar from a distance. Both have a kickoff, a team, a set of tools, and a steady flow of messages. The difference only becomes visible when you check whether the parts that determine outcomes are actually in place.
The Engagement Under Examination
The engagement in this case was a portfolio-narrative generation system, an agent that drafted commentary for a regional financial-services firm's clients about their portfolio performance. The deal was 620,000 dollars for a fourteen-week engagement, and at week four it was visibly stuck.
Rather than rely on a general feeling that something was wrong, an examiner put five concrete questions to the project. Each question checks one part of the operating model. The questions cover evaluation approval, production data access, internal alignment, stakeholder involvement, and customer-facing visibility.
Tracing the Engagement at Week Four
Before reading the diagnostic answers, trace what the engagement looked like from the inside. Follow the state of each element of the operating model as the project moved forward. This is the step where a stalled project reveals its real condition.
Following the Four Elements Through the Project
Track the status of the data request, the compliance team, the eval set, and the sponsor's view of progress across the first weeks of a fourteen-week engagement.
- 1Week one: data request filed
The data-access request for the customer's production portfolio data was filed. The data sat inside a system with strict access controls, so the request was the only path to real inputs.
- 2Weeks two to four: request does not move
The request stayed stuck. Instead of escalating it, the vendor team kept building against synthetic data it had generated itself. The team accumulated four weeks of work against inputs the deployed system would never receive.
- 3Eval set: built by vendor alone
The eval set, which determines how the system is judged, was built entirely by the vendor. The customer's sponsor never approved it, so the team was optimizing for a target the customer had not chosen.
- 4Compliance: informed but not involved
The customer's compliance team, which would have to sign off on any portfolio commentary sent to clients, had been told about the engagement but was not drawn into the design conversation. The vendor planned to bring compliance in at the end of week eight.
- 5Sponsor view: Slack without a dashboard
The engagement had a Slack channel but no dashboard. The sponsor had not seen a metric since the kickoff meeting, so she had nothing concrete to point to when her CEO asked how the project was going.
At week four, all four elements were missing or weakened, even though the project carried the full vocabulary and title of a working AI engagement.
Five Diagnostic Questions
Five diagnostic questions cover evaluation approval, production data access, internal alignment, stakeholder involvement, and customer-facing visibility. Several no answers together signal a cargo cult engagement. The order matters, because the data question is the one that most directly exposes an engagement that looks productive from outside.
Read the diagram as a checklist with a single rule. A yes at each step means that element of the operating model is present. A no at any step points toward the cargo cult pattern, and the source notes that several no answers together are the stronger signal. The diagram shows the path for each question, but in practice you ask all five and count the gaps.
How Four Missing Elements Undermine a Project
Each missing element sets off its own chain of effects. Individually, each one weakens the project. Together, they explain why the engagement stalled even though it looked active. The concept map below connects the four failure points to the stalled outcome.
Mistakes Teams Make With the Title Trap
Common mistake Treating the AI title as evidence of an operating model
What it looks likeA kickoff uses machine learning vocabulary and the engineers hold AI job titles, so the team assumes the project is on track.
Why it failsTitle and vocabulary describe the project; they do not show whether approval, data access, compliance, and visibility are in place.
Do this insteadAsk the five diagnostic questions and check each element directly.
Common mistake Building on synthetic data while a production request is blocked
What it looks likeThe data request is filed in week one and never moves, so the team generates its own inputs and keeps going.
Why it failsThe system is then tested only against inputs it will never receive in production, and the work accumulates against the wrong target.
Do this insteadEscalate the blocked request, and confirm that any synthetic data resembles what production will supply.
Common mistake Writing the eval set without the customer
What it looks likeThe vendor builds the eval set entirely alone and the sponsor never approves it.
Why it failsThe team optimizes for a success definition the customer never chose.
Do this insteadHave the sponsor shape and approve the eval set from the beginning.
Common mistake Assuming an active Slack channel is enough visibility
Why it failsMessages can be plentiful while the decision-maker still lacks any measurable picture of progress.
Do this insteadProvide a regular, customer-facing dashboard with concrete metrics.
Practice: Diagnosing a Stalled Engagement
A project has an AI title, uses machine learning vocabulary, and has a weekly Slack update. The eval set was written by the vendor. The production data request was filed three weeks ago and has not moved, and the vendor is building on generated data. The compliance team will be consulted after the system is finished. Which of the five diagnostic questions get a no answer, and which element of the operating model is the project missing most severely?
Hints
- Go through the five questions in order: evaluation approval, production data access, internal alignment, stakeholder involvement, and customer-facing visibility.
- Ask whether a weekly Slack update is the same as a dashboard the sponsor can use to see progress.
- The source treats the data question as the one most directly exposing an engagement that looks productive from outside.
Summary
A cargo cult engagement carries the title and vocabulary of a successful AI engagement but lacks the operating model that makes it work. The distinction is found by checking the elements that determine outcomes, not the labels the project uses.
At a glance
- An AI engagement can have the right title and vocabulary while lacking the operating model that makes it succeed.
- Five diagnostic questions cover evaluation approval, production data access, internal alignment, stakeholder involvement, and customer-facing visibility; several no answers together signal a cargo cult engagement.
- Missing customer approval leads a team to optimize for an unchosen target; missing production data leaves the system untested against real inputs.
- Late compliance involvement risks a six-week delay or forced redesign, and missing visibility leaves the sponsor unable to judge or explain progress.
- Escalate blocked production data requests and treat a regular customer-facing dashboard as a required part of the engagement, not an optional extra.