Three platforms, three teams, one data lake

Large retailers accumulate data platforms the way they accumulate systems: one at a time, each justified, none retired. The result here was three — Teradata, Netezza and Hadoop — each with its own engineering team building broadly the same pipelines against broadly the same sources.

Challenges

Parallel teams doing parallel work. Separate data engineering teams for each platform meant the same ingestion, the same cleansing and the same business logic implemented three times, in three technologies, by three groups who rarely compared notes.

Duplication and silos. The direct cost was wasted engineering effort. The indirect cost was worse: three implementations of "revenue" that agree until they do not, and no authority to appeal to when they disagree.

No unified platform. Nowhere to point when someone asked where the data is.

The architecture

Shared ingestion frameworks. Frameworks we had designed were deployed for both batch and streaming data — one implementation, used by everyone, instead of three.

Pipelines rebuilt across internal and external sources. Not lifted and shifted. The consolidation was the moment to fix the data structures rather than carry three sets of historical decisions forward.

A customer data hub. Transaction data and customer master data integrated in Hadoop, enriched with an analytics layer. Behavioural data was joined to first-party data so a customer journey could be followed rather than inferred.

Results

• A unified, core data platform — one place, one answer.
• A consistent view for digital marketing, and standardised KPIs for sales and inventory. This is the change people felt: meetings stopped beginning with an argument about whose number was right.
• A 360-degree view aligning data across merchandising, logistics, sales, marketing and customer behaviour.
• Self-service enabled for product managers, analysts and data scientists — the real productivity unlock, because the platform team stopped being a queue for every question.

What we would take from this

Consolidation is a people problem with a technical component. Three platforms existed because three groups each had good reasons. The engineering was tractable; agreeing standardised KPIs was the work.

Standardise the definitions, then the platform. A unified platform serving three definitions of revenue has solved storage and nothing else.

Self-service is the outcome worth aiming at. A consolidated platform that still requires the central team to answer every question has moved the bottleneck without removing it.

On numbers

The source documentation records the capabilities delivered rather than measured savings, so no percentages are quoted. The engineering-effort reduction was real but was not, as far as our records show, ever quantified.