From time to time, I take a few days off work to reflect on things I don’t get to think about in “regulation time” . Its a bit of spring cleaning of my mind.
I am in the middle of one such break today. Other than sleeping a lot, and recovering from India’s loss to Australia last week in cricket world cup – I have been busy reading, listening to Ilayaraja songs non-stop, installing a new patio door, following the progress of my dog who is on a dogshow circuit in midwest with his handler (probably the closest to a good training for me and my wife for when our kiddo leaves for college) , catching up with my friends/mentors/old customers/school mates etc.
Yesterday night, I finally put an end to my month long misery of not being able to crack the 2048 game ( it is a super addictive game – my advice is to not start on it unless you don’t mind spending every spare minute on your phone playing it, and it is a big culprit on the battery drain front). The first thing I did after getting the 2048 tile was to take a screenshot to show my daughter who challenged me to do it, and the next thing I did was to delete the game on my phone. All of today, I have been fighting the withdrawal . As of 5 PM PST, I can report that I could resist the temptation of not downloading the game again and playing it all over again 🙂
Spending the time talking shop with all the peeps I managed to get a hold of these last few days – one thing hit me immediately. Customers and vendors who have started on their big data journey in the last year or two have a new appreciation for the opportunities and challenges in front of them. The opportunity part is pretty straightforward – customers are recognizing that some of the hype around big data is justified, and that real verifiable customer stories are now available. Of course they also know the koolaid firehose is still running full 🙂
Here are some recurring themes on the challenges.
1. Talent shortage
Vendors need technical pre-sales people and developers the most. Customers need developers and ops people either in house or from consulting companies. And such people are apparently in unicorn category. And when these people are available – the employers just don’t know how to evaluate their skills.
Another issue that customers seem to be running into is breadth vs depth. They can usually find an expert in one technology for the right money. But a project typically needs more than one new technology – like maybe hive, mongodb and say elastic search. People who can integrate all of them in real life are rarer than unicorns in rainbow color.
2. How exactly does open source work ?
The people who understand the nuances of open source are overwhelmingly on the vendor side of the house. This includes legal experts. Some customers are also finding their trusted buyer’s agents are not yet smart on open source models. There is some silver lining though – Subscription models are better understood compared to a year or two ago.
3. Procurement cannot figure out what motivates sales people any more.
This one made me smile quite a bit. A good part of my grey hair can be attributed directly to wrestling with procurement folks over the years. Here is how one guy explained it to me ” It was pretty simple in the past – the larger the check I could write, the more benefits I could extract from the salesman. It no longer seems to be the case across the board. Sales reps selling BI and big data things to me all seem to have incentives that are rather unique. Some don’t even want big checks anymore. Some like cloud and some others talk me out of it . I feel like I need to take classes on dealing with them”.
And an IT director buddy – someone who has planned and executed 100s of millions of dollars worth of projects in his career told me “I have a hard time with financial models for projects now given the mix of perpetual and subscription models for all the different software I need. I can barely understand all the pricing and terms nuances , let alone explain the full picture to the controllers and other stakeholders”. Â The impact is a weird situation – he takes more time planning a project than actual execution, and he hates it.
4. Development is not the big worry anymore – maintenance isÂ
They all unanimously agree that these new technologies all reduce development time significantly and give great flexibility to make changes relatively quickly. However, they all have the same worry on maintenance – especially my friends who work in consulting/outsourcing companies. These new technologies all have different security models, different ways to backup and restore and different ways to provision new instances. Each one is built individually to be maintainable and scalable – their worry is how to do all of them together with tight SLAs.
5. Minimal vertical messagingÂ
I never thought I would hear customers ask for more marketing – but that did happen!  What is the world coming to ? 🙂
These folks have all heard it loud and clear that data is big and bad these days and these new technologies can all help them to tame the bad ass big data beast. But they are looking for specific examples of how it helps customers in their industry. On the bright side most of them are not hesitant to try proof of concepts for new use cases.
I did not offer any solutions to these challenges – my intention was just to listen and get a feel for where we are headed at a big picture level. But now that I have thought about it a little bit, I have some rough initial thoughts on things that can help make life easier on this front. When these thoughts are a little better formed, I will make an attempt to scribble them and share.
I am very curious to hear from all of you on whether these themes are showing up in your big data journey. Let me know !
And Vijay Says... 
Great blog – it seems as though individual technology threads are progressing faster than our ability to put all the pieces together, SI’s to train their staff, and procurement/business to make sense of it all.
LikeLike
Vijay, when you say customers ask for more marketing, what do you mean?
LikeLike
Interesting read as always. It’s not exactly in the big data space specifically but FWIW two of your points in particular really ring true in the enterprise application space as well, where the mixed financial model you mention of course means cloud/as-a-service model integrating with traditional on-premise ERP:
-breadth vs depth – and finding people with the right cross-section of skills (and ensuring we’re working towards building them ourselves). The traditional idea of the “T” cross-section where you gain expertise in one area and spread out from there is more applicable than ever IMO. What’s interesting here from the technical angle is the combination of skills that is often most valuable:
-skills which had gone out of fashion are suddenly very relevant again e.g. low-level networking skills
-while at the same time, skills-du-jour (such as deep web dev skills to be able to understand the art of the possible when integrating systems with no mature middleware/integration technologies built up e.g. what can be published and consumed as a web service and can support the volume and type of data etc.)
I don’t know that any one person can do much about this shortage but do expect the market demand to eventually drive improvement in this area (I actually expect it to get worse in the short term as when new areas of demand blossom like this, very often there is an initial surge of people looking to get into it for the wrong reasons).
-financial models – I haven’t had the pleasure of working too extensively in this area but having been on the fringes of discussions to get agreement on models and costs between multiple SIs, vendors and a customer with internal divisions that, let’s say, don’t always work together as well as they should… Let’s just say having some parts of each stakeholder talking about completely different models hasn’t helped. What’s interesting here is that more and more the finance people need technical input to be able to resolve these incompatibilities. It’s always been true but perhaps more so now than ever, it’s a great attribute for any technical person to build their skills in this area (or vice-versa of course, but I see this less often). Being able to speak intelligently about the traditional cost structures and resourcing models, fixed-price versus T&M or just bodyshopping (not always a bad thing) and also the OPEX model/run-rate etc. around as-a-service, while having the technical understanding of each area means you can really help push these discussions forward (perhaps an even rarer combination of skills again – what’s rarer than a rainbow-coloured unicorn? 🙂
The other big issue I see in this space is the culture clash between the traditional ERP world’s waterfall development methodology, well suited to more monolithic off-the-shelf software from the SAP world, and the cloud/as-a-service world’s more agile methodology. Not a new insight by any means but having been hands on in technical leadership of hybrid projects, I’ve seen how things can go badly wrong before it even starts if the respective PMs don’t work very closely together and _understand_ the other model. It’s a really tricky issue to manage (I’m not sure there is a “resolution” as such) that is not helped by the general unwillingness and distrust across the aisle I’ve seen from both sides.
Just some quick rambling thoughts anyway, apologies for the slight diversion from the big data focus (I do think these areas are all interrelated though). Looking forward to your follow-on post!
LikeLike
Great article! Thank you. Please write a blog on Big data opportunities that you think will be a great addition for SAP functional consultants, Business Analyst etc.
LikeLike