Data Challenges in AI Deployments
Production data should be examined early in a project, not late.
Why Teams Postpone Production Data
When an AI system is deployed into a customer's environment, the production data is the material it must eventually handle. Teams often treat getting access to that data as a task to do later, once the system has been built and tested on something easier. The guidance in this article argues against that habit. Look at the production data on day one of the project rather than near the end, because when you look at it determines what kind of problem you will have to deal with.
What do you think happens?
If the team only discovers production data access problems late in a project, what is the likely consequence?
Reveal answer
Delivery can fail outright.
The source compares two timings. Starting the access work early results in a validation that takes longer than planned. Starting it late can result in a delivery that fails outright. The effort is roughly the same in both cases, but the cost of the setback is very different.
Tracing the Same Fight Through Two Timelines
The source makes a specific point: the fight to get production data into validation has to happen regardless of when it is started. The only real choice is when it happens. The diagram below traces the same sequence of events through two project timelines. In the early timeline, the team requests access on day one, so the approvals and agreements are resolved while there is still time to plan around them. In the late timeline, the team builds and tests on easier data first, and the same fight arrives near delivery.
Notice that the work in each timeline is the same kind of work: negotiation, approvals, and data-handling agreements. Only the position of that work changes. Early in the project, its consequence is a validation that takes longer than planned. Late in the project, its consequence can be a delivery that fails outright.
The Access Fight Cannot Be Skipped
The central idea is that the access work is required for validation no matter what. Waiting does not remove it. The negotiation, approvals, and data-handling agreements needed to see production data are needed for validation anyway. Delaying them only moves them to a point in the project where they are more expensive to resolve. This is why the source treats the timing of the access work as the only real decision a team makes about it.
Validation of an AI system against production data requires access to that data. The question a team controls is not whether the access work happens, but whether it happens early enough to be planned for.
Following a Deployment's First Steps
Applying the early-access principle means putting the access work near the start of the plan. The flowchart below is one way to order the first steps of a deployment under that principle. It is a plan derived from the source's reasoning, not a sequence the source prescribes step by step. Trace it from the top: each box depends on the one before it, and the first two boxes are where the production data access work begins.
Planning the first week of a deployment
A team is starting a deployment into a customer's environment. Use the early-access principle to decide what the team does first and what depends on the production data access work.
- 1Identify the production data work
The team recognizes that production data will be needed for validation no matter when the work starts, so the access work is scheduled, not postponed.
- 2Start the approvals on day one
Negotiation, approvals, and data-handling agreements begin immediately, because they must be resolved before production data can be examined.
- 3Schedule background study
The engineer studies storage engines, replication, and batch versus stream processing so the discussion with the customer's data team can be credible.
- 4Hold the first examination
Once access is granted, the team looks at production data early, which lets any mismatch surface while there is still room to plan for it.
The first week is spent on access, approvals, and background study, and validation planning begins with production data already in view rather than as a late surprise.
Background the Data Team Will Expect
Getting production data early is not only about validation. The customer's data team will ask how the AI system behaves under real operating conditions. Without a working understanding of the underlying data systems, an engineer cannot answer those questions credibly. The source names three areas to understand. The concept map below shows how those background areas connect the engineer's view of system behavior to the customer's production data and the questions the data team asks.
Training Data Versus Production Data
The source contrasts the data a system is built and tested on with the production data it must eventually handle. The table below summarizes that contrast using the source's own wording. It shows why waiting to look at production data leaves the team's understanding anchored to the easier material.
| Aspect | Data used to build and test | Production data |
|---|---|---|
| Described in the source as | Something easier | The material the system must eventually handle |
| Typical team habit | Used first, while access is deferred | Looked at on day one, per the source's guidance |
| Effect on problem discovery | Mismatches may stay hidden until late | Mismatches surface early, while there is time to plan |
| Consequence of late discovery | Not stated by source for this row | Delivery can fail outright if discovered late |
Comparison of the data used to build and test an AI system and the production data it must handle, based on the source's framing.
Mistakes Teams Make About Data Access
Common mistake Assuming the access difficulty goes away if the team waits
What it looks likeA team defers the production data request until the system has passed its tests on easier data.
Why it failsThe source says the negotiation, approvals, and data-handling agreements are required for validation anyway. Waiting does not remove them.
Do this insteadStart the access work early, because it is needed regardless of timing.
Common mistake Treating the early-access principle as only a validation concern
What it looks likeAn engineer requests production data but never studies the systems behind it.
Why it failsThe source also says the customer's data team will ask how the AI system behaves under real conditions, and an engineer needs background to answer credibly.
Do this insteadPair the access work with study of storage engines, replication, and batch versus stream processing.
Common mistake Believing that late discovery only makes validation slower
Planning Your First Steps
A team is about to start a deployment into a customer's environment. Using the early-access principle, list the first three things the team should do, and for each one, say whether it depends on getting production data access.
Hints
- Ask what the source says about the timing of the access work.
- Remember that approvals and agreements come before the team can examine production data.
- Consider which background topics the engineer needs before discussing system behavior with the customer's data team.
Key Takeaways on Early Production Data
- 1Production data should be examined early in a project, not late.
- 2The access work is required for validation regardless of timing, so the only choice is when it happens.
- 3Early access leads to a validation that takes longer than planned, while late access risks a delivery that fails outright.
- 4Engineers need background in storage engines, replication, and batch versus stream processing to discuss system behavior credibly with the customer's data team.
At a glance
- Production data should be examined early in a project, not late.
- The access work is required for validation regardless of timing, so only its timing is a choice.
- Early access results in a validation that takes longer than planned; late access can result in a delivery that fails outright.
- Engineers need background in storage engines, replication, and batch versus stream processing to discuss system behavior credibly.