Open source is an architecture decision, not a cost decision
The large data lakes I have built stand on open-source products because they compose better. The savings are a consequence of that choice, not the reason for it, and treating them as the reason leads you to build the wrong thing.
Open source gets sold internally as the cheap option. It is a persuasive argument and a corrosive one, because a decision justified purely on price is re-litigated the moment the price argument weakens, and it always weakens once you have staffed the thing properly.
The durable argument is composition.
Why composition matters at scale
A large data estate is never one problem. It is ingestion, storage, catalogue, transformation, governance, query, and serving, each with its own rate of change and its own idea of what good looks like. An integrated commercial platform makes a bet that these should evolve together, on one release cycle, under one set of design assumptions.
At small scale that bet pays. At large scale it stops paying, because your components stop changing at the same speed. Storage economics move on one timeline, query engines on another, and governance on whatever timeline the regulator sets. A stack you can recompose absorbs that. A stack you cannot forces you to hold the slowest component's pace across the whole estate.
Two ways to hold a data estate together
One integrated platform
- Every component on a single release cycle
- One set of design assumptions across the estate
- Replacing a part means migrating off the whole
A composed stack
- Storage, query and governance move at their own pace
- The regulator sets the timeline for one part, not all of them
- A component that stops being right is replaced, not escaped
The costs that are real
Open source is not free, and pretending otherwise is how these programmes lose credibility in year two. The real costs are:
- Integration work that a commercial vendor would have absorbed, which is now permanently yours.
- Version and compatibility management across components that do not coordinate their releases.
- Deep expertise in a handful of projects, which means real people with real market value.
- The discipline to say no to components: a stack of thirty tools composes worse than a stack of eight.
Put those in the business case. A programme that has already named its costs is far harder to attack than one that has to discover them in public.
What it buys you
The point is not that you can read the source. It is that no single vendor's roadmap can hold your data estate still.
Optionality is the actual asset. When a component stops being the right answer (and at a long enough horizon, each one will), you replace that component rather than negotiating a migration off a platform. That is worth considerably more over a decade than the licence line you removed in year one, and it is the thing to lead with when the decision is being made.
Enterprise Technology Executive
Anurag Kumar Mishra
I lead large-scale transformation across telecom and enterprise technology, connecting technology strategy, operating models and capital allocation to measurable business outcomes.
LinkedIn