Retiring a Hadoop estate: 470 tables, one migration framework

Every large data platform eventually becomes two problems at once. The first is the bill — clusters sized for peak, running whether anyone queries them or not. The second is quieter: the people who knew how the thing was wired have moved on, and nobody wants to touch it.

A retail marketing organisation had reached that point with an on-premise Hadoop estate, alongside Netezza and Oracle instances that had grown up around it. The goal was not "improve Hadoop". It was to sunset it entirely, along with the marketing areas sitting in SQL Server and Oracle, and land everything in BigQuery.

We delivered this migration for a US retail client, engaged through the contracting structure we use for US work.

The size of it

The scope was specified in tables, which is the honest unit for this kind of work:

Source	Scope
Hadoop — marketing	302 tables
Hadoop — enterprise data hub	68 tables
Oracle — marketing ETL instance	100 tables
Netezza	Marketing subject areas

Roughly 470 tables, with the explicit understanding that the final list would arrive later — a normal and honest position at the start of a migration, and one that shapes the whole approach.

Build the framework, not the migration

The single most important decision was to treat the work as building a migration framework, not as migrating 470 tables.

The contract made this concrete: adding tables would not significantly increase effort, because the framework absorbed them. Only a table demanding structural change to the framework or to downstream jobs would trigger a change request.

That commitment is only safe if the framework genuinely generalises. It shaped everything:

• Initial load and incremental load were both framework concerns from the start, not a second project after the first cutover.
• Deduplication and column pruning happened during migration rather than after. A lift-and-shift that faithfully reproduces every historical mistake is a wasted opportunity; the migration is the one moment when touching every table is already the plan.
• Exception handling and validation were treated as deliverables in their own right, not as whatever the scripts happened to do on a bad day.

Proving it actually worked

The part of a migration that earns trust is not moving data. It is demonstrating that what landed matches what left.

We built validation scripts that confirmed source matched target, run as part of the pipeline rather than as a manual reconciliation. With 470 tables, sampling and hoping is not a strategy — and the failure mode of a silent mismatch is a marketing team making decisions on numbers that quietly drifted.

Delivery ran in sprints, each with defined activities and acceptance criteria, so tables moved in reviewable batches instead of one large cutover.

The target that mattered most

Alongside retiring the old platform, the engagement carried a specific commitment: improve data availability SLAs by at least 15%.

The baseline was concrete — one day for Hadoop, two days for Netezza. For a marketing organisation, that latency is the difference between acting on last week and acting on yesterday. It is also the kind of number that makes a migration defensible to people who do not care which database anything runs on.

What was deliberately left out

Two exclusions are worth stating, because they are the interesting ones.

PII was not migrated. Personally identifiable data was explicitly out of scope. In a marketing estate that constraint is not incidental — it shapes which tables move, which columns survive, and what the target schema is allowed to look like.

Campaign tooling stayed put. SAS Enterprise Guide and the campaign management system were out of scope. Migrating the data platform without dragging every attached tool along keeps the project finishable, and lets those tools be dealt with on their own timetable.

Why we would do it this way again

The instinct on a migration this size is to start moving tables, because progress is visible and satisfying. The framework-first approach looks slower for the first few weeks and is faster by the end — and, more importantly, it makes the scope question ("what if there are more tables than we thought?") a routine one rather than a renegotiation.

The other lesson is that validation is the deliverable. Anyone can move data. Being able to show, table by table, that the target matches the source is what lets a client switch off the old system — which was, after all, the entire point.