Why Enterprise Data Quality Quietly Deteriorates After Migrations

Data migrations are sold as acts of hygiene. The old system has accumulated years of manual fixes, duplicate customer records, and spreadsheet workarounds. A new platform promises a single source of truth, enforced validation rules, and a clean break from the past. Six months after go-live, the data quality in the new system is often worse than what was left behind.

The deterioration is rarely caused by a single mistake. It accumulates from a sequence of decisions made under deadline pressure, each reasonable in isolation, each eroding the integrity of the transferred records. Understanding why this happens requires looking at how migrations are actually run, not how they are described in project plans.

The extraction stage loses meaning silently

Every migration begins with extraction, pulling records out of the source database into a staging area. At this point, the source system contains fields that no longer have documented meanings. A column labelled ACCOUNT_TYPE may hold values that were last updated in 2014 by a batch job that no longer exists. A STATUS field may carry free-text entries typed by operators who left the company years ago.

Extraction scripts tend to treat these fields charitably. They copy the values across on the assumption that someone, somewhere, still uses them. In reality, many fields are read by no downstream process. Once they arrive in the new system, they become orphaned columns that nobody owns and nobody can confidently delete. The new system now carries more fields than the old one, and fewer of them have a clear purpose.

The worst case is silent truncation. Source fields defined as free text often contain longer values than the target schema allows. Migration scripts truncate without warning, turning a shipping note into its first fifty characters. The records look clean because they fit the new schema, but the information that was removed is gone permanently. Queries against the new system return shorter values and no error.

Transformations encode assumptions that do not survive contact with real data

Between extraction and load sits the transformation layer, where source values are mapped to target schema fields. This is where business rules are encoded, and where most quality loss happens. A transformation that maps two source customer identifiers into a single target field must decide which one wins when they conflict. If the rule is last-write-wins, a stale record can overwrite a current one. If the rule is first-write-wins, an early error propagates through the entire history.

Transformations also apply lookups against reference tables. A source record containing a country code that is not in the reference table fails the lookup. Migration teams handle these failures in one of three ways: they reject the record, they map it to a default value, or they leave the field null. Each choice has different consequences, and the choice is often made by a consultant under time pressure who will not be present when the consequences surface.

The result is a dataset where a measurable fraction of records have been silently altered. The records that passed through cleanly are indistinguishable from the records that were patched, because the migration log is rarely consulted after the project is signed off. Downstream users querying the new system have no way to know which records were transformed and which were copied verbatim.

Validation rules are weakened to hit the deadline

Every new system ships with validation rules that are stricter than the old one. Required fields, foreign key constraints, and format checks are part of the justification for the migration. During load testing, a meaningful percentage of source records fail these rules. The project now faces a choice: clean the data, or relax the rules.

Under deadline pressure, the rules are relaxed. Required fields become optional. Foreign key constraints are dropped and replaced with soft references. Format checks are converted to warnings rather than errors. The migration completes on time, and the new system goes live with rules that are weaker than the ones it replaced. The single source of truth now accepts records that the old system would have rejected.

This is the structural reason data quality deteriorates. The new system is not more rigorous. It is more permissive, because the only way to move the data on time was to stop enforcing the standards that justified the project in the first place.

Duplicate detection fails when identity rules are inconsistent

One of the promised benefits of migration is deduplication. The old system contains multiple records for the same customer, created over years by different teams using different identifiers. The migration is supposed to merge these into a single canonical record.

Deduplication requires an identity rule, a way to decide that two records refer to the same entity. The rule is usually based on a combination of name, address, and a external identifier such as a tax number. When any of those fields is missing or inconsistent, the rule fails. Records that should be merged remain separate. Records that should stay separate are merged.

The false merge is more damaging than the false split. Two distinct customers combined into a single record create a history that belongs to neither. Invoices, support tickets, and contractual terms are interleaved. Correcting a false merge after go-live requires manual investigation, which is rarely funded because the migration project has already closed.

Ownership disappears at the boundary

Before the migration, each source system had owners who understood its quirks. After the migration, the data belongs to everyone and no one. The team that ran the old system has moved on or been reassigned. The new system is administered by a central team that maintains the infrastructure but does not understand the business context of individual records.

When a data quality problem surfaces months later, there is no clear owner to escalate to. The help desk logs a ticket, which routes between teams until it expires. The problem persists and accumulates, because the organisational structure that maintained data quality in the old system was dismantled during the migration and never rebuilt around the new one.

What reliable migrations actually do

The migrations that preserve data quality share a few characteristics. They run the old and new systems in parallel for longer than the project plan allows, so that discrepancies are caught before the old system is decommissioned. They log every transformation, including the records that were altered and the rules that were relaxed, and they make that log queryable. They assign a data owner to each major entity before go-live, not after.

Most importantly, they accept that data quality is not a migration outcome. It is an ongoing operational discipline. The new system will degrade over time unless someone is responsible for monitoring it. A migration that does not plan for that discipline is a project that moves dirty data into a more expensive database and calls it progress.