The future of big data – consolidations , partnerships and mostly open source

In theory, I am on vacation this week – except that I have not managed to get out of email yet. And to top that, I caught up on a lot of reading and as usual, have some thoughts to share. As always – these are just my personal opinions, not that of my employer.

The good(?) and bad thing about big data is that it has no rigid definition – it is more or less what we want it to be at that point in time. I like to think of “big” in terms status quo of a customer. If the customer only reports out of their ERP data, and now wants to combine CRM and SCM data with it , or maybe combine social data with it – it is already BIG for them. For a stock exchange – they have already figured out how to manage lots of fast and furious data. It is hard to tell them data is BIG now – data was always big for them. It might be a bit “bigger” now, but you get the point. “big” is relative to status quo.

At the moment – there are very few big data projects (as a % of all projects ) that add outrageous business value. Its like a white tiger – a majestic animal with great abilities and all, but rarely seen in the wild. But they are not a myth and they really exist. I saw one just this week and took a photo. I know many friends who tend to look at big data as a white elephant – and I now like to think of the current situation of  big data as a white tiger 🙂


So where is it all trending to? I have some opinions/guesses – not backed by any scientific research. So take it with a pinch (or pound) of salt.

1. Vendor Consolidation

I definitely think there is some consolidation about to happen in 2 to 5 years time. This could take many forms – established firms like Oracle, IBM, MS, Intel, SAP etc could buy some Hadoop/NoSQL type companies. Some of the newer players could merge. The big ones have enough cash in their war chest and those that don’t have cash might not mind a bit of stock dilution to be successful in the brave new world . Big data is what is going to sell more analytics, more hardware and more databases in future – so the incumbent big players in those areas will have all the incentive to jump in as soon as they feel a big data start up is up for grabs.

Oracle might be the one exception to the rule here. A lot of big data startups were founded by people who can’t stand Oracle for some reason or other . So they might not sell to Oracle (but then we know what happened to peoplesoft few years ago, so who knows). End of the day, almost everyone has a price at which they will sell – and these big vendors might pay such a premium given the long term gains. Oracle, IBM etc have an enviable customer base, and have some very driven leaders – I won’t for a second under estimate them.

2. Partnerships

I am a guy that heads a team focused on partnerships, and if I didn’t think partnerships are an integral part of this big data story, I would not have taken my current job. I am firmly convinced that no one vendor can solve an end to end business problem for a customer. Most customers don’t want to spend a lot of time and money in integration work. While no integration is exactly seamless – customers expect a significant part of the integration to be available off the shelf.

Not only that, they probably don’t want the hassle of chasing multiple vendors for making a solution. So I expect to see a lot of OEM/reseller licensing in big data world where even competitors sell each other’s products. It is not new – IBM and Oracle, SAP and Oracle etc are all relationships based on co-ompetition. I think we will see this become mainstream across the board in less than 5 years. I also think that some companies will shy away from this model – and probably perish in the process.

Even the vendors who buy out big data startups cannot avoid this partnership thingy – customers will force the issue if they shy away.

3. Open source will become the norm for software business models

I will go out on a limb to say in 5 to 7 years, open source will be the normal way a company thinks of software. Even the closed source vendors will embrace it more openly (pun intended). The reason for this is simple – as the newer big data vendors take a stance that they don’t need to maximize any given transaction, and are willing to give customers more value than they pay for , it becomes harder and harder for closed source, maintenance revenue based businesses to keep up their current model. Even if they get bought out by closed source vendors – customer expectation will not change much I think.

So why should it take 5 to 7 years ? It takes time to make a dent in the universe 🙂

a. Open source big data vendors need time to mature their code base. They need to build tooling around their products that enterprises need. Remember the time before say Oracle 8i? I think open source vendors will go through that kind of evolution, except that they have to do it in 5 years instead of 20 .

b. Also, for the incumbent vendors to take notice of the new companies seriously – they need to be at least a $500M revenue (or a  trajectory to that kind of revenue) business . I saw this first hand with SAP Hana – Oracle and IBM did not say anything against Hana for a while till SAP started showing big numbers in hundreds of millions of dollars. I fully expect the same to play out against open source too.

c. Unlike ERP etc – I don’t think there will be one big mothership of a big data project at most customers. Instead, I think there will be a lot of parallel projects of smaller size. Some will go live and some will get killed as better use cases evolve. I think big data will drive a trend towards “disposable apps” in enterprise.  But it aint happening overnight.

And all this will take time to shake out – which is why I am pegging it at 5 to 7 years. But I have no doubts that open source will dent the universe !


Published by Vijay Vijayasankar

Son/Husband/Dad/Dog Lover/Engineer. Follow me on twitter @vijayasankarv. These blogs are all my personal views - and not in way related to my employer or past employers

3 thoughts on “The future of big data – consolidations , partnerships and mostly open source

  1. Hi Vijay,

    Nice vacation! I think that white tiger is just over and down the hill from our AZ property–apparently doesn’t mind the heat.

    Couple of thoughts while waiting on a couple of calls–

    I’m with you pretty solid on most of this post, just slightly different flavor. Related comments on your old friend’s blog over at Diginomica–Jon’s hit and misses.

    1) Consolidation of value doesn’t necessarily mean consolidation via M&A — indeed the biggest successes in this industry have done just that. I do agree a lot of M&A will continue to take place due to dollars alone.

    2) On partnering–obviously we both agree on that, and it is related. In conversations again this week I pointed this out several times–if vendors don’t partner wisely, including large with small, OEM/tech, the risk of being completely run over goes through the roof, and it may not be from anything even on the radar if even thought of yet. Tech consolidation is almost certain in my view unless vendors make a real proactive effort in partnering.

    3) On open source–I think in some respects we are already there–I distinctly recall discussing the model that became Red Hat, but…. the open source models today are looking more and more like the old enterprise models of the past before they become unwieldily. It’s great for infrastructure, and can be good for costs–provided all those SW fees aren’t just swapped for labor overhead, not necessarily ROI or even survival due to inherent lack of differentiation. There is a real risk both in OSS and data scientists of simply exchanging ESW lock-in with internal or outsourced service lock-in. And there are a great many proprietary systems out there that are not well matched to OSS–like mine, but are very well aligned with OSS vendors and make natural partners. This is where I think the religion of passionate techies needs to be tempered with wisdom of senior management at times.

    Enjoy the rest of your ‘vacation’. Cheers, MM


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: