The event was created yesterday and arrived today. Which day is it?
Late-arriving data is where most analytics pipelines are quietly wrong. A practical treatment of event time versus ingestion time, bucketing, reconciliation, and the business decision nobody wants to make.
Every analytics pipeline has a bug of this shape, and most teams do not know it yet.
A broker goes down for maintenance at 22:00. It comes back at 01:30. The backlog drains through in a few minutes. Nobody is paged, nothing errors, and the dashboards are green.
But three hours of events created on Tuesday were processed on Wednesday. If your pipeline buckets by processing time, Tuesday is now understated and Wednesday overstated. Nobody notices, because both numbers are plausible.
Two timestamps, and the discipline of keeping them apart
The root cause is almost always the same: a single timestamp column doing two jobs.
- Event time (creation time) — when the thing actually happened. An order was placed. A shipment status changed.
- Ingestion time (processing time) — when your system got around to it.
These are usually seconds apart, which is exactly why they get conflated. Every incident — an outage, a network partition, a stalled consumer, a replay — is a moment when they diverge, and those are the moments when correctness matters most.
Carry both, always. Retrofitting this distinction later is a migration across every table and every downstream query. Adding a column at the start costs nothing.
Define "a day" before you need to
This sounds pedantic and is the source of real bugs. Is 23:00 on the 9th and 02:00 on the 10th one day apart, or three hours?
For a business reporting daily sales, it is one day apart — a calendar boundary was crossed, regardless of elapsed time. That definition needs writing down, because the intuitive engineering answer (three hours) produces different bucketing than the business answer, and the difference only shows up at the edges.
Timezone makes it sharper. A day boundary in whose timezone? The customer's, the warehouse's, or UTC? A retailer operating across several timezones has to pick one and apply it consistently, and every reasonable choice is wrong for someone.
Bucketing: name the state explicitly
Rather than hoping late data is rare, classify every record as it arrives:
- CURRENT — event time and ingestion time fall on the same day.
- DELAYED — they do not.
The value is not the label. It is that the pipeline now knows when it has seen something unusual, which means it can act rather than silently averaging the problem into a total.
Then enumerate the failure scenarios during design, not during an incident:
| Situation | Bucket | Action |
|---|---|---|
| Everything healthy | CURRENT | None |
| Source down, recovers same day | CURRENT | None |
| Source down, recovers next day | DELAYED | Reconcile |
| Consumer stalled, backlog drains later | DELAYED | Reconcile |
| Deliberate replay or backfill | DELAYED | Reconcile |
Writing this table costs an afternoon. It becomes the implementation spec, the alerting spec and the operational runbook simultaneously, and it converts late data from an incident into a routine, expected condition.
Reconciliation: the part that is not technical
When delayed data arrives, you restate the affected period — recompute Tuesday now that Tuesday's missing events are present.
Here is the part engineering cannot decide alone: restating means yesterday's number can change.
A finance team that has already reported Tuesday's figure upward may strongly object to Tuesday quietly becoming a different number. Equally, refusing to restate means knowingly publishing a figure you know is wrong.
There is no universally correct answer. What is universally wrong is letting this be decided implicitly by whoever writes the ingestion job. Common resolutions:
- Restate freely, with an "as of" timestamp on every report so consumers know a figure can move.
- Restate within a window — three days, say — and treat anything later as a permanent adjustment posted to the current period.
- Never restate, and carry late data as an explicit correction line.
All three are defensible. Pick one deliberately and document it.
How you find out you already have this bug
If you are unsure whether your pipeline handles this, the cheapest test:
- Take a day at least a week old and record its headline metric.
- Recompute that same day from raw events now.
- Compare.
If the numbers differ, you have late data and no reconciliation — the recomputation is picking up events the original run missed. If they match, the day was complete when it was first computed, which tells you the pipeline held that day but not that it always will.
The second test is to check whether anything alerts when a record arrives with an event time older than the current period. Most pipelines do not, and that silence is the whole problem: a pipeline that is quietly wrong looks exactly like a pipeline that is right.