Big data – because it “can” deliver you huge value doesn’t mean it “will”


Big data is the talk of the town in social media, and has picked up some interest amongst customers too.I had a series of big data conversations this week with customers, colleagues and friends and thought I will share some here. As always – these are just my personal ramblings, not my employer’s views.

In social media – “big” usually means close to petabytes or at least several tens of  TB rushing at you from all over the place. At customer sites, the expectation seems to be much more modest – 50 to 100 TB is considered excessively big data even for some very large customers I know.

Cost of big data is bigger on all fronts compared to status quo volumes (and velocity and all other factors) of data in most shops. Storage is cheaper than few years ago, but it is not free – and when you talk about petabytes and all, it needs a LOT of storage. And then there is the multiples needed for HA/DR/Archiving etc. And this needs more data center space, cooling , power and so on.

What about the quality of data? As we know – poor quality is a big problem in all kinds of data related stuff. Quality becomes a bigger problem when volume and speed increases. Existing tools may be stretched to deal with that kind of data. But assuming tools can somehow do this – there is a question of the human effort to fix data. A lot of data projects fail to deliver value because no one owns data from business side. Big data will most probably make this problem worse, unless software improves by leaps and bounds in short order to make data quality a non issue. How many of us will hold our breath on that?

What about security? even with just 2 TB of structured data – there are companies who struggle to make sure everything is secure, and everyone is kept honest, and all the legal compliance is ensured. I have seen the amount of trouble they go through when status quo is changed (like an M&A , or even a small CRM system is introduced).   Most of them are not equipped to deal with more data unless they beef up on more sophisticated governance, and probably more staff.

Some companies love BYOD and others do not. The ones who do not, frequently worry about support cost and security. Imagine the effort they have to go through if BYOD will happen in their companies, and they have to protect much larger data than they are used to?

We are right now in the middle of a small POC for a customer – and the data in the datawarehouse is miniscule compared to what “Big Data” can be. We are talking about something only like 150 to 200 million lines per cube. The data comes back at lightning speed from database to appserver. But the user did not see this speed from his iPad connected from a starbucks wifi via VPN. He did see some improvement, but not enough for a big WOW.  And every drill down needs a roundtrip that also chokes up the network yet again.  Essentially, the bottleneck moved from the DB/App server side to the network/client side. These networks will need serious upgrades in capacity to cope with big data. And the mobile software should be smart enough to use the processing power and memory of the device to minimize the use of bandwidth when it is not required. Carriers will probably need big upgrades too, and if big data catches on – we should start seeing different types of data plans from them, dissimilar to the rates that we see now when we buy tablets and smart phones.

Then there is the cost of licensing – and the models of licensing evolving. But if licenses are tied to the quantity of data that is processed/stored – then that adds up quickly.  And even with sophisticated software – you need smart data analysts who can make use of it to generate value. These analysts – or architects, scientists, artists, artisans or whatever it is they are called this week – don’t come in big numbers, and they won’t be cheap either. And long term – I am not sure if this is given enough importance in universities.

The other side of the equation – the more important side, is the value that big data delivers. There is definitely value in big data – significant value – for sure. But it is not value that gets delivered overnight, and it is value that takes significant investment before reaping benefits. And this value will not be spread evenly across industries, or even companies across industries.  So it is a decision that needs to be taken carefully.  Given the cost, the insights from big data has to be not just “big” but  “BIIIIGGGG” – for the investment to be worthwhile.  And because it “can” deliver value does not mean it “will” – it is not a secret that several companies could not even make good use of much smaller quantities of structured data available to them readily all these years.

Several CXOs I have spoken to are willing to dip their toes despite the cost.  And they are all trying to find out where it is that they can gain competitive advantage by jumping in. Several are interested in a cloud offering for big data – mostly from a cost point of view. This is an area where SIs and SW vendors and analysts et al need to do a better job in my opinion. There seems – in my limited visibility – a serious shortage of  specific use cases to help companies make a business decision. There are a few – like in healthcare for example – where compelling arguments were made, and customers and vendors are partnering effectively.  Given the investment needed for big data – evolutionary change might not make it look appealing to the buyers.  It needs to be revolutionary . And as my ex-manager used to tell me –  almost every project that pays for itself will get funded irrespective of the economy.

PS: If big data catches on big time, then we can seriously expect a boom for the tech stocks across the board since several companies will benefit from the vendor side. The economy – at least in history books – will probably thank big data for the good that it did 🙂

IBM Watson – what better use of analytics than fighting cancer ?


From 1992 – when I joined the mechanical engineering degree class in TKM College, till today – I have been a fan of Analytics.In fact, I am pretty sure it is the engineering education that put this fascination in me. And it was my statistics professor Mr. Kalyanaraman who took it to the next level.

Nothing fascinated me more than numbers and making inferences based on them. It was not as if I didn’t realize that texts and pictures and all the so called “unstructured” data was very important – it is just that I always felt that there was plenty more to be done in the “small data” world of numbers, before any one worried about “big data” . I have kept on questioning the value of the insight that will come out of big data for most companies, since they cannot even make decisions based on relatively small and highly structured data from pre-defined sources.

And then, along came IBM Watson and it changed my perspective on analytics and big data completely. Although I work in IBM, I don’t work in the team that works on Watson directly. If I am envious of any one professionally – it is that group of colleagues who get to work on Watson. Watson captured my imagination from the first day I heard about its plans to play Jeopardy on public TV.

Now, god knows how I don’t suffer marketing . I attribute it to the compulsory marketing classes I had to take in B School. And the irony is that IBM has world class marketing. So when IBM trumpeted Watson from the roof tops, my natural instinct was to cringe. But as I thought through the implications – it became clear that Jeopardy was the perfect way for IBM to avoid evolutionary steps, and make a grand leap into the future of analytics. Jeopardy needed everything – ability to consume big data with no structure, ability to understand natural language, truly massively parallel processing, ability to work on commodity hardware, lightning speed, ability to make a decision, ability to learn and many more. And it was a safe test bed to see if technology can stand up to that stress in an environment that is not “life and death” types, but useful enough to make a determination if this has a future.

Right after Watson won Jeopardy against the “human” champions, the IBM team started focusing it on real world problems. And this is where I was most fascinated by the choices of that team and its leader, Manoj Saxena.

IBM has a huge army of smart sales people, who could have very easily sold Watson in some form to a large number of clients across the globe.It would not have been hard at all – my own clients have asked me multiple times how they can use Watson to help their business, without me having to bring it up. As we know, IBM is a publicly traded company with a published roadmap for earnings till 2015. But instead of taking a short term view of cashing in right away, they took a long term view of proving it out thoroughly in the real world with real customers.

Instead of trying to do too many things across all the geographic regions that IBM does business in, they chose to focus on a small number of very specific high value use cases in healthcare, insurance, banking etc. And they partnered with some of the best in class clients in those industries to do so – and in public,not behind closed doors. Now, that is good marketing – the kind I can relate to. Let the customers declare the vision and the success, not the vendor.

The use case that makes me most excited is the cancer treatment one where IBM is teaming with Memorial Sloan-Kettering Cancer Center. Like everything else, there is of course a commercial angle to this – and I can imagine this to earn IBM good revenue. But that revenue could also have come from many other use cases. It is the humanitarian angle that impresses me the most. Cancer research and knowledge can now be spread across the world in very short time once this project succeeds. Doctors outside major research hospitals can reduce a lot of dependence on opinions and guess work and experience, and do a lot more “evidence based” decisions. Of course I don’t expect Watson to ever replace a doctor, but Watson has the potential to be the strongest weapon an oncologist has in the fight against the deadly disease. That is not evolutionary – it is revolutionary. It makes me wonder how many other big problems can be solved by the judicious use of analytics theory and technology.

Please watch this and listen to Dr Norton explain this

And finally, I like the sense of reality the IBM team and the clients who are partnering with them display. They clearly explain what Watson can and cannot do , and how long it will take to get there. Now, I know a lot of my friends like innovations to be out in the market quickly – and I understand where they come from. So on this cancer treatment use case, I pinged a few friends who are doctors in India, who I know from childhood to understand more. It sounded like on an average it takes anywhere from 8 to 12 years according to them for information on diagnosis and treatment to become common knowledge from the time it is published. According to them, they will be thrilled if they can cut that time by a third. So even if Watson can start being of help to cancer patients in couple of years, it will be a big deal, and quite fast in “time to market” .

I am sure the Watson team will have its ups and downs in this journey – but I think it will be well worth the proverbial blood, sweat and tears. I wish them the best. And tonight I will dream of doctors in my Grandom’s village having a pocket Watson with them when they help their patients fight their diseases.

 

Now I believe SAP is serious about collaboration


I had an unusually busy day today, and consequently missed out “officially” congratulating my buddy  Sameer Patel and SAP when he broke the news on twitter and on his blog http://www.pretzellogic.org/blog/2012/03/13/im-joining-sap-ag/.  Of course I knew about it for a while since he and I had discussed this a few times as he was considering the job. And I am very happy for SAP- they have a prize catch.

 

Collaboration is not new at SAP – just that I never got the impression that they were serious about it. Of course they have Streamwork as a collaboration tool. I even crashed a dinner with that team and Sameer, Jon Reed and Dennis Howlett in Palo Alto few weeks ago. It is just that I never took it very seriously, despite Mark Finnern and the SAP Mentors using it extensively. To me – it just did not make sense that SAP developed a collaboration tool that primarily stood alone, without being in the context of the world of business. It especially looked silly to me what SAP has the best business suite application and a pretty good BI platform – and yet did not capitalize on the opportunity to put collaboration in context of those business processes.

 

Over the last few months – a few of my customers have started asking me about collaboration capabilities . Some of them had seen streamwork demos – especially the ones with SAP CRM. But as I dug in to it a little more – it was clear that this was not “out of the box” , but more of a services offering from SAP PSO. Every time SAP comes out with something that only their own services arm can implement, I get a little disheartened – primarily since it raises questions on the maturity of the product, and the ability of a customer to support it after go-live. But since customers had started asking – I was ready to start taking collaboration a little more seriously than before.

 

Sameer is amongst the handful of people I know in the whole Enterprise 2.0 and collaboration and social business world who have a sense of reality in terms of what is possible today and what is not.  And he is very passionate about treating collaboration as a business problem, and less as a technical problem. This is EXACTLY what SAP needs in my opinion.  As much as SAP likes to be a technology or a platform company – its strength and leadership is more on the application side.  And with a guy like Sameer at the helm of the collaboration initiative – I am sure that team will find more focus in capitalizing on SAP’s traditional strengths in Business Suite and BI.  And of course, I am not discounting mobile – it is after all 2012, and no one will collaborate without a mobile device today, will they?

 

I think Vishal’s goals on HANA will come to faster fruition if SAP’s collaboration solutions can act as the glue – as in, an enabler. HANA has proven that it is genuinely fast and real time. The only thing standing between Hana and its future glory is in tethering this speed to several business use cases. I think Collaboration has the potential to be that secret sauce. I am looking forward to see what Vishal and Sameer will tell us at SAPPHIRE. I will honestly be disappointed if I don’t hear anything about this.

 

I don’t think for a minute that Sameer is going to have an easy time at SAP. It is a large and complex organization – and he will have to quickly learn how to navigate there. Everything at SAP is all about HANA – which is good and bad. Good since as I mentioned above, HANA could use a hand from collaboration. Bad because HANA will probably suck all the O2 in the room 🙂 . It is a great time for Sameer to be there though – Snabe already made it clear that Collaboration is high on SAP’s agenda. Something tells me that SAP might make an acquisition in this space  – and of course I cannot resist restating my long held view that they should buy TIBCO , and solve multiple problems in one shot.

 

And then there is the SFSF angle. SFSF has Jam, and SAP has Streamwork. There is some overlap – and I am curious to see which one will survive.  Other than for political reasons if any – I cannot imagine any good reason why customers will ever want to choose between two competing offerings from SAP. They already face that difficulty in SAP’s Planning solutions, and I am sure they don’t want an encore.

 

I am also not sure I understand Collaboration team falling under the Analytics organization, instead of applications organization under Sanjay. It probably does not matter – but just does not make sense to me at first sight.  But an even bigger question for me is if there is an engineering organization under Vishal’s group that works on Collaboration technology.  Given the nature of collaboration, I think it makes sense to treat it as a platform wide pervasive thing, rather than as a point solution. To say the least – I am super curious to get updates on how Vishal and Sethu are going to treat collaboration from the Technology & Platform POV.

 

Sameer, I wish you all the very best as you begin this new chapter of your career – and I am sure your leadership and passion will lead SAP to newer heights.