One version of the truth, at Kafka scale

Large retailers rarely suffer from a shortage of reports. They suffer because the reports disagree — each built by a different team, on a different extract, with a subtly different definition of the same metric. Everyone knows the numbers are inconsistent; nobody can say which one is right.

We designed and built the omnichannel analytics platform that replaced that situation for a US department store chain. The stated goals were unusually clear, which made the architecture easier to argue about:

• Consistency in how KPIs are calculated across the organisation's reports.
• A single source of truth everyone could reference.
• Lower cost than the collection of proprietary systems it replaced.
• Keep everything — as much data, from as many sources, for as long as the business wanted.
• Real-time analytics, not overnight batch.

That last pair is in tension with the third. Keeping everything and serving it in real time is exactly what makes proprietary analytics systems expensive.

The shape of it

Order and shipment events originated in the order management system and were published to its own Kafka cluster. The analytics platform mirrored those topics into its own Kafka cluster rather than reading from the source directly — a boundary that matters more than it looks, because it means the analytics platform can be down, slow, or replaying history without ever affecting the system taking customer orders.

Alongside the event stream, flat files arrived from several other systems on their own schedules. KPIs were computed both in real time and in batch, and the results landed in Redshift, where the existing reporting tools could query them as an ordinary warehouse.

That last decision was deliberate. A new analytics platform that also demands everyone learn a new reporting tool is two migrations, and the second one is the one that fails.

The genuinely hard part: data that arrives late

Anyone can compute yesterday's sales from yesterday's events. The difficulty is that not all of yesterday's events arrive yesterday.

A message broker goes down for maintenance. A mirroring process stalls. A network partition heals hours later and a backlog floods through. The event was created at 23:50 on Tuesday and arrived at 02:00 on Wednesday. Which day's sales does it belong to?

Get this wrong and the platform quietly produces wrong numbers — the worst failure mode, because it looks like it is working.

We handled it with bucketing and reconciliation, built on a precise distinction:

• Creation timestamp — when the event was generated at source.
• Ingestion timestamp — when the platform actually processed it.
• Day — a calendar date change, regardless of elapsed hours. 23:00 on the 9th and 02:00 on the 10th are a day apart, even though they are three hours apart.

Every record was bucketed as CURRENT (created and ingested on the same day) or DELAYED (not), and the design enumerated the failure scenarios explicitly, each with its expected behaviour and required corrective action:

Situation	Bucketing	Action
Both clusters healthy	CURRENT	None
Source broker down, recovers same day	CURRENT	None
Source broker down, recovers after a day	DELAYED	Run reconciliation
Analytics-side mirroring down	DELAYED	Run reconciliation

Anything landing in DELAYED went through a reconciliation flow that restated the affected day rather than silently folding old events into today's totals.

Why write the scenarios down

The valuable part of that table is not the logic. It is that the failure modes were enumerated during design, before anything ran in production.

Most late-data bugs are not failures of intelligence. They are cases nobody thought about, discovered months later when a finance team notices a Tuesday that changed. Writing out "what if the mirroring process is down for two days" costs an afternoon during design, and the answer becomes both an implementation requirement and an operational runbook.

It also forces a conversation that is genuinely a business decision, not a technical one: when late data arrives, do we restate history or not? Restating means yesterday's number can change, which finance may hate. Not restating means the number is knowingly wrong. Someone has to choose, and it should not be whoever is writing the ingestion job.

What generalises

Three things we would carry to any similar platform:

Mirror, do not tap. Put a boundary between the operational system and the analytics platform. It buys the freedom to replay, backfill and fail without consequence to the business.

Separate creation time from ingestion time everywhere. Not just for late data — every question about "when" becomes ambiguous the moment those two collapse into one column, and separating them later is a migration.

Land in something the existing tools already speak. The platform earns adoption by making current reports better, not by requiring everyone to change how they work on the same day.

On numbers

We have not quoted cost savings or latency figures for this platform. The architecture document sets out goals rather than measured outcomes, and we would rather describe the design accurately than attach numbers we cannot source.