Data Quality and Cleaning in Large Archives
Email addresses and domain names change over time as people move between organizations, requiring a mapping system to consolidate multiple variants into single canonical addresses
Why Identity Fragments
A large email archive may contain several addresses for the same person. Email addresses and domain names change as people move between organizations. If each address is treated as a separate identity, the archive can represent one contributor as several unrelated contributors. Mapping rules provide a way to consolidate those variants into canonical forms during processing.
The central idea is transformation rather than deletion: an original address remains in the archive, while processing can treat it as equivalent to a chosen canonical address.
Consolidating Individual Addresses
The Mapping table in mapping.sqlite stores individual email mappings in source-to-destination form. The source is the address that appears in the archive, and the destination is the canonical address that the processing system should use instead. When gmodel.py encounters an address, it checks the Mapping table. If a mapping exists, it uses the destination address. If no mapping exists, it uses the address as-is.
Building a Career History
Steve Githens and Three Addresses
Consolidate Steve Githens' three email addresses into the most recent address, swgithen@mtu.edu.
Choose the destination: Use swgithen@mtu.edu as the canonical destination because it is Steve Githens' most recent address.
Add the first source mapping: Create a Mapping entry from the Northwestern address to swgithen@mtu.edu.
Add the second source mapping: Create a Mapping entry from the Cambridge address to swgithen@mtu.edu.
Leave the canonical address unmapped: Do not add a third entry from swgithen@mtu.edu to itself. It is already the final form.
Two Mapping entries consolidate the three addresses. Whenever gmodel.py encounters either the Northwestern address or the Cambridge address, it uses swgithen@mtu.edu instead. The canonical address is used as-is.
For three known addresses, two entries are sufficient when one address is selected as canonical. The mapping count is based on the noncanonical addresses, not on the total number of addresses.
Normalizing Domains First
The DNSMapping table works at the domain level rather than the individual address level. For an address such as someone@iupui.edu, gmodel.py first checks whether iupui.edu has a DNS mapping. If the table maps iupui.edu to indiana.edu, the address becomes someone@indiana.edu. Individual email address mappings are checked after this domain transformation.
Following One Archive Record
When gmodel.py processes the archive, it reads each message and extracts sender and recipient addresses. It checks DNSMapping first, then checks the individual-address Mapping table. If a relevant rule exists at either stage, the destination is used. If no rule exists, the current address continues unchanged. This process is repeated for every email in the archive, so one mapping entry can affect thousands of messages when the associated person was a prolific contributor.
Protecting the Original Archive
Email and DNS mappings are applied during data processing, not during data collection or storage. They act as a transformation layer, so the original email archive remains unchanged. If a mapping later proves incorrect, it can be modified or removed without changing the archived source data. This makes it possible to test different mapping strategies while preserving the original records.
Treat mappings as adjustable rules for analysis, not as replacements for the underlying archive. Preserve the original data and revise the transformation rules when your understanding of an identity or domain changes.
Mistakes in Archive Cleaning
Adding a self-mapping for the canonical address
The canonical address is already in its final form and does not need a mapping entry.
Fix:
Map only the two noncanonical addresses to swgithen@mtu.edu.Using DNSMapping for an individual address change
DNSMapping operates on domain names, while the Mapping table stores individual email address mappings.
Fix:
Use the Mapping table for individual address variants and DNSMapping for domain variants.Checking individual address mappings before DNS mappings
The documented processing order places the DNSMapping check before the individual email address check.
Fix:
Process the domain-level mapping first, then apply the individual address mapping.Editing the original archive to normalize addresses
Mappings are designed as non-destructive transformation rules and are applied during processing.
Fix:
Keep the original archive unchanged and adjust the Mapping or DNSMapping rules.Assuming that an unmapped address is discarded
If no mapping exists, gmodel.py uses the address as-is.
Fix:
Distinguish between an address that has no mapping and an address that has been mapped to a canonical destination.
Practice the Mapping Decisions
A contributor appears in an archive under three addresses, and you choose the most recent address as the canonical destination. Decide how many Mapping entries are needed, which addresses should be sources, and whether the canonical address needs an entry. Then explain whether a domain change should be handled in Mapping or DNSMapping.
Hints
- Count the noncanonical addresses rather than counting all addresses.
- A Mapping entry points from a source address to a destination address.
- Use DNSMapping when the transformation concerns a domain name rather than one complete email address.
What do you think happens?
If gmodel.py encounters an address with no entry in the Mapping table, what happens to that address?
Reveal answer
Answer: It is used as-is.
The processing rule checks for a mapping entry. When no mapping exists, the address remains as encountered.
Unified Archive View
- The Mapping table consolidates individual email address variants by sending source addresses to a canonical destination.
- When three addresses belong to one person and one is canonical, two Mapping entries are sufficient; the canonical address does not map to itself.
- DNSMapping consolidates domain names and is applied before individual email address mappings.
- Mappings are non-destructive processing rules: the original archive remains unchanged and the rules can be revised or removed.
- Email and DNS mappings work together to create a unified view of contributors across organizational and career changes.
Key Takeaways
- Use the Mapping table for individual email address changes and DNSMapping for domain changes.
- Choose one canonical destination and map each noncanonical source address to it.
- Apply DNS mappings before individual email mappings when processing archive records.
- Keep the original archive unchanged because mappings form a reversible transformation layer.
- A small number of carefully chosen mappings can normalize many archive messages.