From time to time, I take a few days off work to reflect on things I don’t get to think about in “regulation time” . Its a bit of spring cleaning of my mind.
I am in the middle of one such break today. Other than sleeping a lot, and recovering from India’s loss to Australia last week in cricket world cup – I have been busy reading, listening to Ilayaraja songs non-stop, installing a new patio door, following the progress of my dog who is on a dogshow circuit in midwest with his handler (probably the closest to a good training for me and my wife for when our kiddo leaves for college) , catching up with my friends/mentors/old customers/school mates etc.
Yesterday night, I finally put an end to my month long misery of not being able to crack the 2048 game ( it is a super addictive game – my advice is to not start on it unless you don’t mind spending every spare minute on your phone playing it, and it is a big culprit on the battery drain front). The first thing I did after getting the 2048 tile was to take a screenshot to show my daughter who challenged me to do it, and the next thing I did was to delete the game on my phone. All of today, I have been fighting the withdrawal . As of 5 PM PST, I can report that I could resist the temptation of not downloading the game again and playing it all over again 🙂
Spending the time talking shop with all the peeps I managed to get a hold of these last few days – one thing hit me immediately. Customers and vendors who have started on their big data journey in the last year or two have a new appreciation for the opportunities and challenges in front of them. The opportunity part is pretty straightforward – customers are recognizing that some of the hype around big data is justified, and that real verifiable customer stories are now available. Of course they also know the koolaid firehose is still running full 🙂
Here are some recurring themes on the challenges.
1. Talent shortage
Vendors need technical pre-sales people and developers the most. Customers need developers and ops people either in house or from consulting companies. And such people are apparently in unicorn category. And when these people are available – the employers just don’t know how to evaluate their skills.
Another issue that customers seem to be running into is breadth vs depth. They can usually find an expert in one technology for the right money. But a project typically needs more than one new technology – like maybe hive, mongodb and say elastic search. People who can integrate all of them in real life are rarer than unicorns in rainbow color.
2. How exactly does open source work ?
The people who understand the nuances of open source are overwhelmingly on the vendor side of the house. This includes legal experts. Some customers are also finding their trusted buyer’s agents are not yet smart on open source models. There is some silver lining though – Subscription models are better understood compared to a year or two ago.
3. Procurement cannot figure out what motivates sales people any more.
This one made me smile quite a bit. A good part of my grey hair can be attributed directly to wrestling with procurement folks over the years. Here is how one guy explained it to me ” It was pretty simple in the past – the larger the check I could write, the more benefits I could extract from the salesman. It no longer seems to be the case across the board. Sales reps selling BI and big data things to me all seem to have incentives that are rather unique. Some don’t even want big checks anymore. Some like cloud and some others talk me out of it . I feel like I need to take classes on dealing with them”.
And an IT director buddy – someone who has planned and executed 100s of millions of dollars worth of projects in his career told me “I have a hard time with financial models for projects now given the mix of perpetual and subscription models for all the different software I need. I can barely understand all the pricing and terms nuances , let alone explain the full picture to the controllers and other stakeholders”. Â The impact is a weird situation – he takes more time planning a project than actual execution, and he hates it.
4. Development is not the big worry anymore – maintenance isÂ
They all unanimously agree that these new technologies all reduce development time significantly and give great flexibility to make changes relatively quickly. However, they all have the same worry on maintenance – especially my friends who work in consulting/outsourcing companies. These new technologies all have different security models, different ways to backup and restore and different ways to provision new instances. Each one is built individually to be maintainable and scalable – their worry is how to do all of them together with tight SLAs.
5. Minimal vertical messagingÂ
I never thought I would hear customers ask for more marketing – but that did happen!  What is the world coming to ? 🙂
These folks have all heard it loud and clear that data is big and bad these days and these new technologies can all help them to tame the bad ass big data beast. But they are looking for specific examples of how it helps customers in their industry. On the bright side most of them are not hesitant to try proof of concepts for new use cases.
I did not offer any solutions to these challenges – my intention was just to listen and get a feel for where we are headed at a big picture level. But now that I have thought about it a little bit, I have some rough initial thoughts on things that can help make life easier on this front. When these thoughts are a little better formed, I will make an attempt to scribble them and share.
I am very curious to hear from all of you on whether these themes are showing up in your big data journey. Let me know !
