Scoping Data Work in AI Deployments
A clean sample is evidence about the sample; production data can differ, and the source reports exactly that gap.
A Team Reads a Tidy Sample
Imagine a team preparing to deliver a data system for a customer. The customer has handed over a sample export, several documents describing the fields, and a verbal summary of how the business runs. The team reads through the sample, finds it tidy, and begins designing around what they see. Weeks later, the system meets the real data and behaves in ways nobody planned for.
This pattern is one of the most common failures in FDE engagements, and it usually starts with a quiet assumption made early in the project. The assumption is that a data sample is representative of production data. Nothing in the team's process forces them to question it, because the sample looks correct and comes from the right place. This article examines that assumption, explains why it is wrong, and describes the three kinds of data quality problems that follow from it.
Why a Sample Feels Trustworthy
The reasoning behind the sample assumption feels sensible. A sample comes from the same source as production data, it has the same columns, and it looks well-behaved when inspected. Because of these surface similarities, it seems reasonable to treat whatever is true of the sample as true of the whole system. The problem is that these similarities describe the shape of the data, not its contents across every record.
What do you think happens?
Does a sample that has the same columns and comes from the same source tell you what the full production dataset contains?
Reveal answer
No. A clean sample is evidence about the sample. It is not automatically evidence about the rest of the production data.
The source states that the data in the sample can be clean while the data in production is not. Shared columns and a shared origin do not guarantee that the records outside the sample look the same.
How a Clean Sample Hides Production Problems
The source states the gap directly: the data in the sample is clean, but the data in production is not. A sample can be clean for reasons that have nothing to do with the production data it was meant to represent. It may have been assembled by hand, filtered by someone who wanted it to look right, or pulled from a narrow slice of the business. Whatever the reason, a clean sample tells you about the sample and very little beyond it.
The diagram shows the key asymmetry. The sample column can look orderly while the problematic records sit in the part of the business the sample never reached. The sample's tidiness is a fact about the sample, so it cannot settle what the production data contains. The three problem types discussed later are exactly the kinds of issues that hide in that unseen portion.
Tracing the Chain from Documentation to Rework
The source explains that the three problem types share a single root: the expectation built from documentation and samples diverges from what the production data contains. Each problem is therefore a point where the data the customer described and the data the customer actually holds have drifted apart. Tracing the chain of events shows why this matters for the delivered system.
Read the flowchart from top to bottom. The first step is a documentation set and sample that look correct. The team builds on that view, which is a reasonable choice given what they have. When production records arrive with gaps or inconsistencies the design did not anticipate, the system fails on the actual data. The source notes that a system built around the documented structure can fail in exactly this way, and that the bridging work needed to reconcile the two is often absent from the original estimate. That missing work is the unplanned rework at the bottom of the chain.
The Three Recurring Problem Types
The source groups the common issues into three kinds. Each one is a place where the documented or sampled picture and the production data disagree. The three kinds are documented required fields that are often empty, schemas that disagree across business units or systems, and date formats that vary within a single column. The sections below take each kind in turn.
Checking Production Data Directly
The source identifies the safeguard against the sample misconception as checking production data directly. Reading the sample and the documents is a starting point, not a finish line. The checks below target the three problem types directly, so that gaps are found before the design is locked in.
- Measure how often each documented required field is actually populated in production data
- List the field names and types that each business unit or system uses, and compare them against the assumed schema
- Find every format that appears in each date column, not only the format seen in the sample
- Carry the reconciliation work into the project estimate, since the source notes it is often absent from the original estimate
Mistakes That Hide Production Gaps
Common mistake Treating a clean sample as representative of production data
What it looks likeA team designs a system around a tidy sample export and never checks the rest of the business.
Why it failsThe source states that a sample can be clean for reasons unrelated to production, such as hand assembly, filtering, or a narrow slice of the business.
Do this insteadTreat the sample as evidence about the sample only, and check production data directly before relying on it.
Common mistake Trusting a documented required field to be populated
What it looks likeA design assumes every record has a value for a field the documentation lists as required.
Why it failsThe source reports that documented required fields are often empty in practice.
Do this insteadMeasure how often the field is populated in production and plan for blank values.
Common mistake Assuming one schema applies across every source
Why it failsThe source reports that schemas often disagree across business units or systems.
Do this insteadCompare the field names and types of each source against the assumed schema before building.
Common mistake Leaving the reconciliation work out of the estimate
Why it failsThe source notes that the bridging work needed when a system meets actual data is often absent from the original estimate.
Do this insteadBudget time for reconciling documented and actual data as a planned part of the work.
Practice: Auditing a Sample Before Design
A customer gives your team a 200-row sample export from one regional office. The documents list customer_email as a required field, and the sample shows it filled in for every row. Your team plans to build a system that depends on customer_email for every record. Before designing anything, name the three checks you would run against production data, one for each of the three problem types discussed in this article. For each check, state what you are looking for.
Hints
- Consider how often a documented required field is actually populated across the full production data, not just in the regional sample.
- Think about whether other business units or systems might name or type the same fields differently.
- Look for every format used in each date column across the full data, not only the format that appears in the sample.
Key Takeaways
- A clean sample is evidence about the sample; production data can differ from it, and the source reports exactly that gap.
- The three recurring problem types in FDE engagements are documented required fields that are often empty, schemas that disagree across business units or systems, and date formats that vary within a single column.
- A system built around the documented structure can fail on actual data, and the bridging work to reconcile the two is often absent from the original estimate.
- The safeguard is to check production data directly, including how often required fields are populated and which formats appear.