Big data opportunities and challenges are getting a bit clearer

From time to time, I take a few days off work to reflect on things I don’t get to think about in “regulation time” . Its a bit of spring cleaning of my mind.


I am in the middle of one such break today. Other than sleeping a lot, and recovering from India’s loss to Australia last week in cricket world cup – I have been busy reading, listening to Ilayaraja songs non-stop, installing a new patio door, following the progress of my dog who is on a dogshow circuit in midwest with his handler (probably the closest to a good training for me and my wife for when our kiddo leaves for college) , catching up with my friends/mentors/old customers/school mates etc.

Yesterday night, I finally put an end to my month long misery of not being able to crack the 2048 game ( it is a super addictive game – my advice is to not start on it unless you don’t mind spending every spare minute on your phone playing it, and it is a big culprit on the battery drain front). The first thing I did after getting the 2048 tile was to take a screenshot to show my daughter who challenged me to do it, and the next thing I did was to delete the game on my phone. All of today, I have been fighting the withdrawal . As of 5 PM PST, I can report that I could resist the temptation of not downloading the game again and playing it all over again 🙂

Spending the time talking shop with all the peeps I managed to get a hold of these last few days – one thing hit me immediately. Customers and vendors who have started on their big data journey in the last year or two have a new appreciation for the opportunities and challenges in front of them. The opportunity part is pretty straightforward – customers are recognizing that some of the hype around big data is justified, and that real verifiable customer stories are now available. Of course they also know the koolaid firehose is still running full 🙂

Here are some recurring themes on the challenges.

1. Talent shortage

Vendors need technical pre-sales people and developers the most. Customers need developers and ops people either in house or from consulting companies. And such people are apparently in unicorn category. And when these people are available – the employers just don’t know how to evaluate their skills.

Another issue that customers seem to be running into is breadth vs depth. They can usually find an expert in one technology for the right money. But a project typically needs more than one new technology – like maybe hive, mongodb and say elastic search. People who can integrate all of them in real life are rarer than unicorns in rainbow color.

2. How exactly does open source work ?

The people who understand the nuances of open source are overwhelmingly on the vendor side of the house. This includes legal experts. Some customers are also finding their trusted buyer’s agents are not yet smart on open source models. There is some silver lining though – Subscription models are better understood compared to a year or two ago.

3. Procurement cannot figure out what motivates sales people any more.

This one made me smile quite a bit. A good part of my grey hair can be attributed directly to wrestling with procurement folks over the years. Here is how one guy explained it to me ” It was pretty simple in the past – the larger the check I could write, the more benefits I could extract from the salesman. It no longer seems to be the case across the board. Sales reps selling BI and big data things to me all seem to have incentives that are rather unique. Some don’t even want big checks anymore. Some like cloud and some others talk me out of it . I feel like I need to take classes on dealing with them”.

And an IT director buddy – someone who has planned and executed 100s of millions of dollars worth of projects in his career told me “I have a hard time with financial models for projects now given the mix of perpetual and subscription models for all the different software I need. I can barely understand all the pricing and terms nuances , let alone explain the full picture to the controllers and other stakeholders”.  The impact is a weird situation – he takes more time planning a project than actual execution, and he hates it.

4. Development is not the big worry anymore – maintenance is 

They all unanimously agree that these new technologies all reduce development time significantly and give great flexibility to make changes relatively quickly. However, they all have the same worry on maintenance – especially my friends who work in consulting/outsourcing companies. These new technologies all have different security models, different ways to backup and restore and different ways to provision new instances. Each one is built individually to be maintainable and scalable – their worry is how to do all of them together with tight SLAs.

5. Minimal vertical messaging 

I never thought I would hear customers ask for more marketing – but that did happen!  What is the world coming to ? 🙂

These folks have all heard it loud and clear that data is big and bad these days and these new technologies can all help them to tame the bad ass big data beast. But they are looking for specific examples of how it helps customers in their industry. On the bright side most of them are not hesitant to try proof of concepts for new use cases.

I did not offer any solutions to these challenges – my intention was just to listen and get a feel for where we are headed at a big picture level. But now that I have thought about it a little bit, I have some rough initial thoughts on things that can help make life easier on this front. When these thoughts are a little better formed, I will make an attempt to scribble them and share.

I am very curious to hear from all of you on whether these themes are showing up in your big data journey. Let me know !


Wishing Godspeed To Marilyn and Mark, The Godmother And Godfather of SAP Community

I have had a writing block for a couple of months now. Then I saw the news of Marilyn and Mark moving on from SAP and that reminded me of the two year hesitation I over came in 2008 to write my very first blog.

That blog was written almost exactly 7 years ago on SDN–part-1 . The first time I felt like I should blog was probably two years prior to that – but I was scared, and rather clueless on how it is done. Finally I mustered the courage (mostly with endless encouragement from friends at IBM and SAP who said something like “its going to be great – but you go first” 🙂 ) and wrote it – and it got read by a couple of thousand people or so. Some kind folks even took time to post some encouraging comments. I liked that first experience enough to write two more to finish my thought process on that topic . Next thing I know, it got featured on SDN home page and I was moved to “expert blogger” status – which meant my content needed no further reviews before publishing. That was not really me being a great blogger right off the block – it was all Marilyn working her magic behind the scenes to encourage a newbie 🙂

While blogging was new to me – SAP community itself was not. I had been in the field since the mid 90s and was commenting on forums and blogs for a while. I was also a regular at Techeds, mostly as a participant but also occasionally as a presenter. And that is how I first ran into Marilyn Pratt. Looking back, it was a turning point in my life and career. Marilyn, as she had done for several others, took me under her wings and nurtured my interest in blogging. She has the most facilitative style of anyone I know – never once did she tell me that there was a better way of doing things (I was pretty bad – I know it, but she never made me worry about it), but would just keep encouraging me to blog more, talk more at events, develop my network within the community and so on. My confidence grew sufficiently to be a regular blogger on SCN and then a couple of years later – I started this blog to rant about non-SAP stuff.

What I learned the most – and continue to learn – from Marilyn is the concept of paying it forward. There is a second lesson too – she is a wizard when it comes to connecting people with common interests. There was not a lot that I could do for her for everything she has done for me. She in fact constantly tells me she did not do much for me ( that is dead wrong – but she is way too modest). I learned from her , and later from others too, that all I should make sure is that I pay it forward by helping the people who come after me. It is not just about blogging – I have certainly helped others get started on blogging , but it is just as important in every aspect of life. I can’t say I have mastered it at all – but I have indeed been trying. It has made a big impact on how I look at life and I mostly have Marilyn to thank for that. I am sure I am not the only one who thinks that way.

Soon after I started blogging and presenting at SAP events, I started meeting some interesting people across the world . No two were alike – what exactly is common between Thorsten Franz, Dennis Howlett, Jon Reed, Sue Koehan, Tammy Powlas, John Appleby and Vitaliy Rudnitsky ? And I met more than a 100 such unique souls and many became close personal friends. That is the beauty of community – random individuals come together due to some common interest, and they stick together for a long time and develop new set of interests together. Its very rare that I talk about SAP topics with any of these folks I met through SCN – but each have enriched my life in meaningful ways. And that would not have happened without Marilyn.

I had already seen Mark Finnern from a distance at that time. And a little while after I became a regular blogger on SCN, I got a phone call from him to say I have been selected as an SAP mentor. I literally got a feeling similar to winning the Nobel prize or something of that magnitude. He would not tell me who nominated me. I tried asking Mark Yolton and Chip Rodgers – and they would not tell me either. It was only after I joined SAP that I figured it was Marilyn Pratt who nominated and nudged Finnern into taking me into the mentor wolf pack . For a second time, my life (and career) took a meaningful turn. I now had access to some of the most interesting leaders like Vishal Sikka, Jim Snabe, Bjoern Goerke, Sanjay Poonen et al at SAP and knew a lot more about how decisions were made on many products and programs. I knew many of these SAP execs from my time at IBM – but being in the mentor program made it a much stronger relationship and many of them became friends and mentors for me.

Mark and I became good friends and I was very active in mentor program. I always knew that he was working hard to make it easy for mentors to add value to SAP. But I did not realize the extent of his challenges till I actually joined SAP myself couple of years ago. It is a large and complex organization – and at any point there would be someone in the executive team who thought mentors added a lot of value. Unfortunately, many of those executives moved on from SAP too from time to time. So Mark had to rebuild his relationships constantly to make sure the program continued to have sponsorship at the highest level all the time . Mentor program is under SAP marketing from an HR reporting line – although its intent is not to be a classic marketing initiative.  I have always admired how Mark managed to keep the program thriving because a good part of the ROI of mentor program is intangible , or at a minimum cannot be measured in classic marketing KPIs. A lesser man or woman would have thrown in the towel and walked away from leading this program. But then Mark is not ordinary – he is as extraordinary as someone can get and he made it work. And a lot of us who are mentors (or are alums like me) are grateful to him and SAP for the opportunities it gave us.

I have no doubts in my mind that Mentor program has served SAP really well over the years – more than anything else, it is a much needed and invaluable reality check on how the ecosystem perceives SAP.  Each and every mentor does something unique that adds value to the SAP community and to SAP the company. Mark’s shoes are really big to fill – but SAP is not short on talent . Whoever has taken over Mentor initiative – I wish you well and I hope you will keep the flag flying high and take it to its next level. The brand of mentors is intertwined with the brand of Mark – and rightfully so for everything he has done. But as he leaves for his next adventure, I am excited to see how the program evolves.

Marilyn and Mark – I wish you the very best in the next phase of your personal and professional lives. You have made a dent in the universe and I will be cheering for you all the way. Godspeed my friends ! I look forward to grabbing a drink with you real soon.