40 engineers, 10+ projects, 5 frameworks that held it together

Long engagements fail in a specific way. Each project is delivered, each is defensible on its own, and after three years there are eleven codebases that solve the same problems eleven different ways. Nobody decided that; it is simply what happens when delivery pressure is constant and nothing rewards generalising.

We spent several years embedded with a large US retail chain's data organisation, and the thing worth writing about is not any individual project. It is what we did to avoid that outcome.

The shape of the work

At its peak the engagement ran to roughly 40 people — Scala and Spark developers, a larger group working in Pig and Hive, architects, delivery leads, and data scientists.

Ten or more projects were delivered or in flight, across the areas a retailer actually runs on:

• Customer data hub — consolidating customer records scattered across systems that had never been designed to agree with each other.
• Merchandise tier transformation — the merchandising hierarchy that determines what is sold where, and at what price tier.
• Sales hub — sales data aggregated for the people who need to act on it.
• Digital marketing measurement — attribution and campaign performance.
• Marketing models — the analytical layer above all of it.
• Cloud migration — moving the estate off on-premise infrastructure.

Each of those is a substantial project in its own right. Run consecutively by different sub-teams, they are also eleven opportunities to reinvent the same plumbing.

What we built once and used everywhere

Five frameworks came out of the engagement, each extracted because the same problem had appeared for the third time:

A data quality framework. Every pipeline needs to answer "is this data sane?" — nulls where they should not be, row counts that moved more than they should, referential gaps. Solved per-project, this becomes a dozen slightly different opinions on what "sane" means. Solved once, it becomes a shared definition every team can extend.

A data sync framework. Moving data between systems on a schedule, with incremental loads and failure recovery, is the least glamorous and most frequently rewritten code in any data platform.

Cloud migration patterns. Once the first workload moved, the second and third should not have been research projects. Documented patterns turned a migration into a repeatable procedure.

A unit testing framework. Data pipelines are notoriously under-tested, largely because setting up representative test data is tedious enough that people skip it. Making that setup cheap is the whole battle.

A build and deployment framework. Shared CI and release tooling, so a new project inherited a working pipeline on day one instead of copying someone else's and slowly diverging.

None of these are exciting individually. Collectively they are the difference between a team that gets faster over three years and one that gets slower.

Frameworks as a hiring and onboarding strategy

There is a second-order effect that is easy to miss. On a 40-person engagement with normal turnover, shared frameworks are how new people become productive quickly. A developer joining their first project meets the same data quality checks, the same deployment pipeline and the same testing approach they will meet on their fourth. The learning curve is paid once, by the organisation, rather than repeatedly by every individual.

The team invested heavily on this front — structured technical interviewing, enterprise learning subscriptions and certifications, regular knowledge-sharing sessions, and open source contribution as an explicit expectation. Fifteen or more internal frameworks were released under an Apache licence to the central Maven repository.

That last part matters more than it looks. Code you intend to publish gets written differently: interfaces get thought about, assumptions about one client's environment get removed, and documentation stops being optional. Open-sourcing a framework is a forcing function for making it genuinely reusable.

What we would tell a team starting something similar

Extract the third time, not the first. Generalising too early produces frameworks that fit one use case and are contorted into fitting the second. Waiting until a pattern has appeared three times means the abstraction is supported by evidence.

Budget for it explicitly. Framework work never wins against a delivery deadline if it has to compete for the same hours. It has to be planned as part of the work rather than hoped for.

Judge a framework by adoption, not existence. A shared library nobody uses is worse than none, because it looks like the problem is solved. The frameworks above earned their place by being the path of least resistance.

We have no published performance figures to attach to this engagement, and we would rather say that than quote numbers we cannot source. What we can say is that it ran for years, grew to forty people, and delivered ten-plus projects on shared foundations rather than eleven separate ones.