And Vijay Says…

Some of you may know that I am a big dog lover . I also used to compete actively in dog shows with German shepherds , labs and goldens . As career started taking more and more of my time – I gave up on dog shows . I actually don’t care very much about winning a show any more – I don’t miss that thrill anymore . But I terribly miss the fun of training a puppy from scratch and competing and the constant problem solving .

When you compete seriously – you need a puppy or young adult who is suitable for your specific needs and that needs a lot of research to find one. I haven’t been at the big shows or following pedigrees closely for a long time – so I had to make a fresh start . I have several friends who are active in the field and I will get plenty of good advice from them . But for me to follow along their advice intelligently, I need to do some homework first .

So during the vacation period, I decided to start my research. Seeing several people walking their beautiful and well trained dogs in the streets of London and Paris added to my motivation 🙂

First step of any good research is of course to stock up on coffee . And we had just returned from Europe and the Parisian cafes had set a very high bar . Thanks to Amazon prime – it only took a couple of days to get a few different coffee bean packs . For good measure I brought out the old Espresso machines from their boxes so that I have enough options to find the perfect match .

Then I opened up YouTube and started watching clips from the big german Shepherd shows in Germany . About ten videos in – and probably 2 double shot espressos – I realized there is an unaddressed problem . I am in no shape physically to train a dog at the level I need to . I need strength , endurance , flexibility and things of that nature .

Two years ago – I had lost a lot of weight by signing up with a personal trainer and working out at the gym . Well that ended up with muscle tears in both arms and that kept me away from the gym for a year . I still lost some weight but it was negligible . At that pace – I wouldn’t get to the physical state I need to be anytime soon . Darn !

I love outdoors – and I hate doing cardio at the gym with a passion . Thanks to our mild winter in Chandler – I put on tennis shoes , and started walking . About 45 mins later – I couldn’t figure out how much distance I covered or how many calories I burned . This is the curse of being an analytics guy at work – I get this intense urge to measure ! Thankfully there are plenty of apps that can do it all – and I dutifully downloaded the requisite apps and started measuring weight , exercise , food , water etc .

I started at around 2 miles and can now do 5 miles of brisk walk . And in the first week I lost about 3 pounds . I hope I can continue the good habits and get into the right shape by the time I get the pup .

Thanks to the insistence of my teenage daughter who has been using AirPods for a while – I had bought a set for myself too but never used them . It’s one of those things – I always felt like a total idiot talking with Bluetooth headsets and generally have avoided using them . But the daily walks proved to be a turning point – I am now a big fan , using them for work calls as well as music from the phone .

Anyways – back to my research on dogs. So I read through the breed standard again – and surprisingly could more or less recite it verbatim from memory . I started looking at my favorite dogs from 80s and 90s to get my eye tuned to the ideal version . And then I started watching the videos and pictures of the more recent dogs .

I did not like most dogs . I started saying things in my mind like “great head but too long in the body” and “great movement but don’t like the tail set” . Highly frustrating to say the least !

And then came some sad news – my mentor in dog shows passed away and a dear friend from India sent me a message about it . I was so terribly sad and spent a lot of time thinking about the times I have spent with him when I was in college . And then the light bulb went on – I was doing a cardinal sin of evaluation of dogs called “fault judging”. My mentor – Nawab Nazeer – had warned me about it several times .

Purebred dogs have a written standard . Everyone knows the standard is for an ideal dog and no dog is ideal . So the principle of judging is to figure out how close a given dog comes to the ideal . However it is a lot easier for less experienced and lesser skilled people to judge all the faults and compare that way . That’s what fault judging means – and it’s a bad idea . I knew that from before – and yet I was doing it with full enthusiasm . Go figure !

Any way – I have some unlearning and resetting to do before I continue my research to find my puppy . Probably that gives me a bit more time to lose the weight 🙂

Today, I am typing this from the plane to NY – the first business trip of the year . And as I am slowly getting back to the normal work routine, I started thinking about how much fault judging happens in the work place . It’s much more easy for many of us to spot weaknesses and criticize people about it than recognize strengths and give them credit for it .

It’s funny how one thing leads to another when you are powered by great espresso ! For good measure , I just ordered some more espresso beans 🙂

I had an extensive debate recently on why monoliths are not universally such a terrible idea, where a young developer told me “Most of the code you would have written in your time as a developer is now what we call legacy code” .

I don’t blame the young man for thinking “legacy = bad”. I believed it too when I had his role many years ago . And then my thinking evolved over time and I am sure his will too.

I wrote this blog “Nuke that code, lets start over” nearly a decade ago, and I think many of the points discussed in there still hold true . There is a lot of wisdom in the comments section under the blog that I just finished reading again .

The young man’s comment made me think a bit about all the code I have written – from games in BASIC that I have sold to others for 50 rupees on cassettes while in high school to C/C++ , ABAP and java Programs that did more “Enterpisey” stuff for big companies across the world .

“Most” of the code I wrote probably have not lasted very long at all !

Some of the best code I wrote never made it to production – because priorities changed, projects were shelved , acquistion happened and a host of other reasons .

Most of the code I wrote has been refactored – some times by me , but mostly by developers who owned those repos after my tenure . I know of a couple of places where a bit of my code still runs in production – but it’s a tiny fraction of what I have written in total .

Legacy code survives because of many reasons – some good and some bad

1. It does something exceedingly well and it’s hard to make a case to replace II

2. It is way too complex for anyone in the team to rewrite now – usually due to poor knowledge management , not because the code itself is bad . Even with sophisticated tooling – it’s hard to understand logic completely from code, DB schemas and logs .

3. Refactoring is a routine activity in good tech shops . But the truth is that vast majority of large enterprise shops have ignored the importance of refactoring for a long time . They paint themselves to a corner and then need massive modernization projects to get out of the mess and cater to the business needs of today’s and tomorrow’s market . It also shocks me that some shops after finishing the modernization, they still refuse to follow any structured process to refactor along the way !

4. Another common pattern I have seen is that modernization is attempted purely for “tech fashion” – and usually with tragic results . I was quite amused once to watch the code of MRP process be rewritten to a new technology only to realize that even though the results now showed up in a fraction of the time of the old system, it was all wrong and hence useless !

Being honest about the need to rewrite is a good starting point in all modernization projects . There is only limited time, effort and money available and there is never a good reason to waste it by doing a project for the wrong reasons .

“we need to modernize because mainframe is dead” or “we need to move to NoSQL because RDBMS is dead” or “Our code base is a big monolith and we need to be serverless for our future” . It’s very common to hear these kinds of reasons being touted as the reason to start a modernization project . These are not bad reasons by themselves !

These statements are all good starting points to explore what is the real issue – do you take more time to deliver a new feature that business wants , and is that leading to less than optimal business results ? If the answer is yes – is it a tech problem or a process problem primarily that causes it ? Can you quantify the value of changing to a new system and prove that it is greater than the estimated cost ? What happens if the modernization project fails – how much risk can you take ?

Bottom line – you need an honest case for why you want to modernize . This case will then serve as your North Star in your modernization journey . It also will serve as a dose of reality check – which we will all need from time to time . The case itself has a shelf life – so it’s important to revisit periodically to check what changes are needed .

Big Bang vs Incremental changes – how do you decide which is the best way to go ? I will share some thoughts on that in another post . For now , I need to get back to my day job 🙂

As 2019 is heading to its last few weeks, I had a chance to catch up with a lot of friends, mentees and my own mentors and also take some time to reflect how things worked out for me this year . The consultant in me immediately started grouping and classifying the common themes, apply MECE , and map it all to a 2 X 2 . Don’t worry – I somehow resisted the temptation and stopped at just grouping 🙂

I thought three repeated themes were worth sharing here on my blog .

1. You need to tell some managers what they want to hear

So there is this guy who comes from a non tech background and does a good job managing a team of developers to hit milestones ahead of schedule (mostly by overworking them) . He is a flight risk, and the only way to keep him is to promote him to an executive rank . If he flees, his manager probably won’t earn his own promotion any time soon – so there is plenty of motivation to make this happen.

The only trouble is that this promotion needs the blessing of a very senior tech exec . There is no way this non tech dude is going to pass that level of tech scrutiny . That’s when the manager has a flash of brilliance in his thinking . The senior techie is somewhat of a philosopher and his current pet topic is “culture”. He coaches the up and comer to focus heavily of culture when the interview happens – which he masterfully does and gets promoted .

About half the people I caught up with had a similar story where promotions or raises were secured by telling their bosses what they wanted to hear .

2. Diversity seems to be a bit easier than inclusion

While it’s still a long ways to go, a lot of managers have been acting on improving diversity in their teams . There is a lot more awareness and training at all levels . In many cases there are top down directives like “by end of the year there needs to be X% women in your team , Y% URM in your team” etc . In a some of these cases, these are KPIs tied to the manager’s bonus too . All of these are great of course – and I hope it’s not just a one time exercise .

Then comes the question of inclusion . Are these “new” members of the team supported and set up for success ? Are their voices considered with the same importance as when your team was homogenous ? I was a bit surprised that even the people who actively champion the need for diverse teams haven’t done as much thinking on how to make inclusion happen in their day to day work.

That said – I was quite happy to hear that almost everyone I spoke to had done something good about making sure people doing similar roles are paid the same.

3. Being a newly promoted executive continues to be really hard

Before I made Partner , I had attended an exceptional internal training course in IBM called “Cornerstone”. And that’s where it was drilled into me that “what got you here won’t get you there”. Excellent advice which helped me and many others who attended that course . It is also very hard to put it into practice !

Everyone wants to make an immediate impact as an exec . Most have someone else they treat as their role model and want to be like him or her. Nothing wrong with any of it – just that what works for one exec in one context might not translate 1:1 to you in your context . Almost every single story I heard from the newly promoted folks made me say in my inner voice “oh no – I did that too” .

Here is one story I heard from someone who got promoted and took over a new team as its leader in January 2019. All six direct reports were asked to make 1 hr presentation to the new boss . The first one made a less than stellar presentation in the new manager’s opinion and the next day it was announced that this person will have a new role to be announced soon . Long story short – we are in December now and that position is still not filled , and two of the key next-in-line leaders quit because they couldn’t stand the ongoing chaos.

I asked them what were the words of wisdom they got from their leaders after their promotion and it was the usual list of “Have a bias to action”, “Lead with courage”, “Make your voice heard” etc . Those are all invaluable in their own rights – but perhaps we should add “slow down a bit now to speed up later” to the list .

A young engineer I met last week asked me “Sir, you have had an impressive career since the time you left college . Did you always ace every job interview? ”

That question took me back a couple of decades and I realized I only aced one from the first four interviews – which was when TCS hired me from Business School as one of their first SAP consultants.

The three I failed were all painful at the time – but funny enough in hindsight . So I will share those three here – just for some fun 🙂

1. INFOSYS

It must have been 1996 or 97 . Infy was THE place to be – hot young company where all the cool kids got hired . They did not visit our college – but they had an open hiring day where we could apply and go through their evaluation process .

If I remember right – it was a three step process . Step 1 was a multiple choice test on math etc . Step 2 was a problem solving round where they gave a puzzle and you have to solve it and walk the interviewer through your solution . And the third – I am told – would have been an interview with a senior exec and an HR partner .

I had no trouble with the written test and was asked to appear for the problem solving round . The interviewer was a young lady not much older than me. She gave me a printed sheet of paper which explained the question . Funny enough – I had once solved this exact question and in full honesty I told her that I already knew the answer . No problem – she found me another sheet with a different question . And she sat across me , crossed her legs and started reading a copy of CHIP magazine . Well – I couldn’t solve the problem at all and I gave up . She promptly kicked me out of the process and said I can apply in 6 months again .

I never got around to applying again 🙂

2. IBS

When I was in Business School in 1998 , IBS was just getting started at Technopark in Trivandrum . Our Dean , Dr M.N.V Nair , asked me and few others to check it out . A bunch of us took him up on it . I remember a very fancy office and some well dressed people conducting the test .

The written test was on logic and quantitative ability – I aced it . The very pleasant HR lady told me that I only got one question wrong and that I was the highest scorer she had seen till then . So at this point my confidence was sky high and I had no doubts that I am going to kill it in the interview as well .

There were four interviewers in the panel – including the company CEO and the CEO of Technopark . Right off the bat they congratulated me on the high test score and asked me what was my strategy for the test . I said I solved the easy ones first , then the medium complexity ones next and finally attacked the hard problems for which I had conserved time .

That was the end of the interview . One of the panel members cut me short and said “You won’t be a good engineer for us . We are looking for people who tend to attack the hardest problems first” .

Uh oh !

Never tried to get a job there after that . When I reported back to the Dean – he said “everyone makes mistakes”. I didn’t quite understand whether he was referring to me or my interviewers 🙂

3. SAP

While I was working as an ABAP programmer at TCS in Colorado Springs, I interviewed for a programmer role in Washington D.C with SAP in their public sector development team . I did fine in the phone interview and the in-person HR round . Then a Dev manager did a technical interview with me – and asked me to write some code on the white board . We had a great discussion on optimizing performance of the code I wrote on the board .

Pretty soon the interviewer and I were furiously writing and editing code on the board and then at a random point he shook his head and declared “I can’t hire you man . You got the syntax of FOR ALL ENTRIES in ITAB wrong . I cannot look past that . Sorry – you can leave now” .

This was painful ! I knew the syntax since I used that construct quite often in daily work . Somehow I messed it up that day and if it was done on a computer – obviously I would have fixed it in a second . But it was not to be !

Several years later – I did get hired by SAP . And I recently read somewhere that “for all entries” is no longer cool in ABAP once ERP moved to HANA . Oh well 🙂

This weekend I ran into an unusual problem – I did not have a single book to read . To the best of my recollection, I have never had this situation in my adult life . Now – the rational thing to do would be to get a book on kindle . But that’s not my thing . I need a physical book – I don’t like to read on kindle .

I don’t even own a kindle . My preferred way of getting a book is to order on amazon prime on my mobile and get it delivered to where ever I am . This way I don’t need to carry more than two books in my bag when I travel . If I particularly like a book – I give my copy as a gift to someone that I think will find it useful , and buy a couple more .

I do miss the experience of walking into the local library, chatting with the librarian , browsing the shelves and driving back with a half dozen books . I might do it today after a few years gap . Amazon with its large collection of books and easy access to reviews have helped me not miss the physical library experience to a large extent – but that strangely has not translated to reading books electronically .

There is a contradiction here that I find weird . I rarely print anything at work to read . I read big documents on my MacBook and redline them as needed . And yet when it comes to books I need to hold it in my hand or else I feel I can’t get past the first page . I have tried and I have failed .

As much as I love reading work related stuff on my laptop – I rarely use my laptop to take notes . I need to write it in my notebook with a pen myself . And consequently I love great pens to write with . Those pens are as expensive today as it was when I entered the workforce – and never once have I thought they are a luxury item . I can’t rationalize an expensive pen that gives me pleasure to write with – because I barely write 5% of all the content I create . Other 95% is electronic – including this blog that I am typing on an iPhone using the WordPress app . I run most of my business from my phone . There is literally no business case for me to justify an expensive pen other than “that’s how I like it” 🙂

Talking about writing – and expensive pens – I have my name engraved in cursive on the pen I use frequently . A young kid recently looked at it and could not make out that it is written in English . I realized – with some shock – that not all schools teach cursive writing now – which of course makes all practical sense . I have horrible hand writing myself – and often write in block letters if someone else needs to read it . But I have always been a big fan of beautiful cursive writing – and I think my dad is one of best in that craft . It is painful to note that cursive writing will become a lost art in my life time .

Times are of course changing fast and ideally we have to change with it . Talking about time – I like the old school watches to put on my wrist every day and not glance at my phone for time , or use a modern electronic one line an Apple Watch that can also tell me that I haven’t walked enough steps today .

The old school watch – as much as I admire the excellent craftsmanship – is not super practical for a guy like me who travels across time zones frequently . I have to check time on my phone when the plane lands to correct my analog watch . Rationally – there is no reason to keep doing this for the millions of air miles I have travelled , but again never once have I felt this was a pain .

In my day job, I am reasonably good at convincing my clients to let go of their past ( that we lovingly refer to as LEGACY ) and move to cutting edge new solutions . A big part of that is building a rational case on why change is good – and try to get their head to over rule their heart . But everyone – literally everyone – has an emotional connection with their past that they won’t let go .

I fondly remember my very first client who let me automate two complex reports – and then looked at them both to manually prepare a third report . Why ? Because – he said with a genuine smile – “That is just how I like it” .

As a consultant, I have learned over my career that my clients all have a unique mix of a genuine fascination for the new and a tendency to hang on to the old – and I can’t push for all the change in one step .The trick is to know how much to push and when to stop and agree this is as far we can go for now .

This is a big reason why transformations are always journeys – and not destinations !

Pressure is a fact of life – especially so in businesses. You cannot eliminate it, but you can probably minimize it with good preparation and training and so on. But whatever you do – you are bound to end up in high pressure situations from time to time. It could be your boss promising more profit to the board or the market changing too fast and all your carefully laid plans going for a toss – there are many reasons why you may end up there.

The part you have a greater chance to control is how you handle it as a leader. And I think this is where a lot of traditional tactics that we seem to take for granted become counter productive. I want to highlight just three such “common strategies” and why they may not yield the expected results .

Centralized decision making 

The most common tactic when pressure builds is for people higher in the hierarchy to take direct control of execution – especially of what appears as low hanging fruit. You will start seeing memos that go “All travel needs to be approved by the CFO” , “All purchases over $100 need to be approved by the GM” and so on.

On first blush it seems like a real good idea. People will be careful about what they spend money on because no one wants to go to the scary CFO to ask for a $500 air ticket. And multiplied by hundreds of people – that is real cost that is saved.

But what is the reality ? The good sellers will switch to a competitor with less draconian rules. The next best will stay paralyzed thinking they can push their deals when the rules ease off. The CFO would rather spend $500 to get the $100K deal – if only he knew. But this set of people won’t always ask. They will wait. It proceeds along similar lines across all segments. CFO might win the cost battle for a short time and still  lose the larger business war over time.

The accounts payable clerk could have handled this $100 restriction – or automation could have, if the company had the IT skills needed. So on top of killing everyone else’s productivity – the CFO now has a lot more low value work to do. Is that the best use of a highly paid executive’s time ? You cannot manufacture more time for yourself even if you have the most powerful title !

What about the mid level managers who actually lead their teams ? They feel powerless, and for the most part their teams realize it in no time and lose respect for them. So now you have a morale issue and the good managers start floating their CVs in the market too.

A CFO I respect a lot once told me – about a decade or so ago –  “When the business knows what they are doing, my job is to report what they do. When the business does not seem to know what they do – my job is to get it to a shape where I can get back to reporting again”. I think my appreciation for his words have increased every year since then.

Cutting the workforce 

When the going gets hard and saving travel and stationary cost is not enough, leaders have to let go of people. It is a harsh reality of business. Many companies manage cost very effectively on an ongoing basis and even they will occasionally be pushed to cutting head count significantly from time to time.

The traditional wisdom is to cut the bottom rungs first. What gets ignored or forgotten typically is the difference in org structure to handle peace time vs war time. In peace time – you need the matrix and hierarchy to make sure you are investing sufficiently for future. So you will see roles like “Chief of transformation” , “Chief of culture” and so on – and with great conviction, the big bosses will put their best leaders in those roles. That is absolutely the right thing to do as well in that context.

War time is very different. You need your best leaders leading the charge – dealing with the market and your employees directly. If they truly are your best leaders, they cannot be hidden in internal roles managing spreadsheets and on vague ideas.

It is common to hear “all hands on deck” messages from the top at war time. But how many of these peace time roles are actually redeployed to the front line immediately at war time? How many of those senior and expensive people are shown the door if they don’t have the war time skills needed to keep the company alive ? And what saves more money – getting rid of peace time leadership roles at war time , or getting rid of a lot of lower cost less experienced people ? Same question about over lay and ops support roles – matrix requires significant operational overhead. When you are fighting to stay alive, does it really matter much how many ways you can slice and dice your results ?

In my admittedly limited exposure – I have always felt that most leaders are optimists. They think of all troubles in business as temporary – and hence will go away very soon. So why go through the trouble of redeploying etc when there is a less complex way that looks good on a spreadsheet ?

Over communicating 

One of the things that leaders often encourage their teams to do the moment pressure starts mounting is “Please over communicate”. This often happens after most of the critical decision making has already been centralized – thereby reducing the usefulness of the lower level managers . Pleas to over-communicate  is done with great intentions as well –  for example, if people can alert their bosses of important issues early – they can help solve it before it becomes a disaster.

But what really happens when leaders try to over communicate ?

Even when there is no real pressure, communication is not usually a real strength for many business leaders. When they start to over communicate under pressure – the team starts to wonder a few things – “Wow this sounds desperate – should I brush up my CV” , “Clearly you have no respect for my time – and I seriously doubt you understand what my job really is” , “I lost you after the first five minutes – can we get to the point?”, and so on. The reason is simple – the more senior you are, your only way to over communicate is via some “one size fits all” strategy. When people are under pressure – they need clear instructions and specifics, the exact opposite of “one size fits all”.

When the bosses talk in generalities – what are the chances of the lower level employees to go back to them with specific topics ?

So what would be a sensible approach for leaders to handle pressure in their business ?

1. If you are going to war – declare it explicitly so that all your troops hear it loud and clear .
2. Define what it takes to win the war. Delegate battles effectively and stay focused on the war.
3. Act decisively to get rid of the peace-time org structures and redeploy troops for war.
4. Treat those who are not fit to be warriors with extreme kindness – and try to make it up to them when peace-time returns
5. Take the lower level leaders into confidence and empower them – centralize strategy as needed without choking information flow, and decentralize execution and communication.
6. Keep everyone posted as needed – focus on specifics. Resist the urge to over-communicate especially in “one size fits all” fashion.

“Football is two things. It’s blocking and tackling. I don’t care about formations or new offenses or tricks on defense. You block and tackle better than the team you’re playing, you win.” – Vince Lombardi

Just like in football, there is a perception that blocking and tackling has no glamor and it is beneath us to focus on those activities if we were to be successful. Not a month passes without me getting an SOS ping from someone with “Save me from this madness – I don’t want to be stuck in creating decks, business cases, managing utilization and creating pricing sheets. I want to be in the strategic stuff”. I have some empathy – I have made such calls too when I had their roles in the past. And one such call today is the reason I started typing this post 🙂

The main reason to hate the blocking and tackling work is because we generally don’t know why we do these “boring” tasks at all when we start out.

My favorite way of getting people (usually my younger colleagues) to see the WHY aspect is to bring them with me to a meeting where their work is used. A few years ago, a young senior manager came to me for some advice and told me how much she hated the mindless work around pricing deals. So I asked her boss to have this senior manager present the deal on his behalf for the deal review next time.

Our CFO asked her “Why should we make this investment instead of putting the money in an index fund”. She did not have a good answer, but her boss did. He walked us through the fundamentals of the analysis she had come up with and showed why it is a better deal than alternate ideas we had suggested.  She knew the mechanics of pricing very well – she just did not know the context of how her work was used by others for their decisions. Now she is masterful in how she crafts business propositions and is well set for an amazing career. To her credit – she now explains the WHY aspects of the work to the people in her team when she assigns them such work.

People who grow up learning to block and tackle well will have some advantages in business that are hard to learn later in life. I learned this from the head of manufacturing of a car company some twenty years ago when I was a young consultant collecting business requirements for an SAP implementation project.

He used to let me sit in his Tuesday morning staff meetings and for every problem that came up – he would ask “Should we make this problem go away, or should we solve it?”. That was a test to see who knew WHY the problem exists and hence can explain how to make it go away by making upstream changes. Over the course of several such meetings, I realized that people who passed the test were the ones that got better roles and more money and so on. It was counter intuitive for me on why he thought less of the people who just thought about ways of putting in a solution for the problem without questioning why the problem came up in the first place. Those were the people who got a lot of extra coaching and formal training. The irony was that often the solutions proposed by this second set of people were the “requirements” I captured for the ERP implementation 🙂

I am a big fan of delegation and have written about my thoughts on how to do so effectively a few times already.  As I look back at my own career, a good part of my success over the years happened because my bosses felt comfortable delegating more and more to me. And the reason they had that confidence in delegating to me was because they knew I had a good command over the blocking and tackling aspects of the job, and they were positive I can make “some problems go away for good” because I have a first principles understanding of the issue. Now – the honest truth is that I did not always have that understanding . There is a little halo effect that you earn for yourself when you are good at blocking and tackling – and that buys you a bit of extra time and room for mistakes to make the problems go away.

Blocking and tackling are less boring when you understand the WHY aspects . But that does not mean that boredom and grief will not return if all you do is block and tackle. When you are really good at blocking and tackling – you become good at solving problems the RIGHT way . The next logical step up from there is solving the RIGHT problems.

The people who have the most fulfilling time at work are the ones who can do both – identify the right problems to solve, and then solve it the right way. The reality is that majority of leaders become good at only one of these two things, and hence companies need multiple leaders working in tandem to bring both these much needed skills together . Personally, I prefer leaders who are awesome at execution to the ones who are hailed as strategic thinkers. To be more precise – I believe it is a step in the wrong direction to separate execution from strategy. There is no such thing, in my opinion, as a great strategy, poorly executed. Strategy that does not take into account the ability to execute is at best called marketing/PR/advertising. At worst it is just called a MISTAKE.

Yesterday evening, while flying from PHX to JFK, I had a chance to read this excellent blog by Arun Murthy . If you have not read it yet – pls read it first before you read my rant below. As always – these are strictly my personal views .

First – I think Arun is probably the best person to write a blog like this and needs to congratulated for doing so. He has been part of the Hadoop story from the very beginning and continues to play a significant part in shaping its evolution. It also felt that it came straight from the heart – and extra points for starting lists from 0 and not 1 🙂

My own point of view was developed over the last couple of decades as a data geek who worked on a range of technologies on data management as well as analytics for a wide variety of clients across the world.

Data is largely an unsolved problem in the large enterprise world. Just when you think you have it under control, you realize that the problem got bigger and more complex. And you also realize the tech has improved and now you have more options on how to solve the bigger problems. This is one reason why most techies eventually use philosophy as a framework for explaining the evolutionary nature of their work.

It is hard to predict which way an exploratory project will go. This is great for developers as it gets their creative juices flowing. But that is not how enterprise CIOs think. They value high doses of stability and predictability , and very low doses of complexity. They very well know that the pundits will use terms like “legacy thinking” to shame them all the time. But their world comes with flat and declining budgets and there are always a lot of lights to be kept on. Within all those constraints – the good CIOs try to foster as much innovation as possible. And of all the innovation they have sponsored in the last decade or so – Hadoop definitely was top of the list.

World was ready for Hadoop . Classic datawarehousing had been pushed to its limits. Data warehouses became data dumps. Cost of maintaining those datawarehouses started driving everyone nuts. And Hadoop promised a solution for all these and more. Plus the open source nature gave all the geeks even more incentive to introduce it in their shops.

In my opinion, 4 things led to Hadoop’s alleged demise

1. Too many options for clients to choose from
2. Unskilled people implementing it
3. Multiple changes in market positioning 
4. High operational complexity

Pretty soon – everyone ran into challenges. MapReduce was no longer sufficient to do most of what enterprises wanted to do. No worries there – Spark etc came up just at the right time and took over. The world realized that you just cannot run away from SQL even if you criticize it heavily. So many different SQL on Hadoop projects came into being and that did not always work in the way traditional IT shops expected. IT shops are not used to having tremendous choice in solving problems. When Cloudera and Hortonworks proposed different solutions to a problem – be it SQL, be in security or whatever – it became very confusing for the people who were trying to implement a long term solution in their shops. In short – “Hadoop is a philosophy” started getting interpreted as “there are no real best practices here – just keep experimenting” by a lot of clients. Just to keep it brief, I am skipping the divergent direction MapR took – and that story did not end well either.

Then came the question of skills. For enterprises to adopt technology faster – you need a lot of people with that skill. Much like how SAP market got flooded with poor skills when ERP was hot – Hadoop market did too . That had a direct effect on the quality of implementations. Many clients are still struggling with tech debt caused by using developers and architects who did not have good fundamentals in data management. Net net – hardly anyone replaced any data warehouses , and data lakes became the new data swamps. To be fair – the growth of classic data warehouses have been significantly curtailed since hadoop became mainstream.

Arun has already explained the “What is Hadoop?” question in great detail. So I will skip that entirely.

What also did not help a lot was the positioning of hadoop companies changed over time – perhaps to sustain the insane valuations in private markets. It swung from data management to analytics and ML to managing everything in cloud. When you try to do everything – even if it is a great problem to solve – it is hard to execute to perfection, and it confuses clients a great deal.

The last point that made it difficult for Hadoop was operational complexity. Data management is a lot of fun for developers. But the moment it is in production – rock solid operations is what keeps it going. Even for RDBMS based systems – DBAs and other Ops experts with all the mature tooling still spend significant time managing their landscapes. Hadoop ( to be fair – most NoSQL DB too ) did not prioritize ops sufficiently. In my view at least – this was perhaps the biggest miss and one I think Cloudera and others should urgently address. Every client I know will be grateful if managing Hadoop was significantly simplified – especially between on-prem and cloud.

All this said – I don’t think Hadoop is dead , or that it will die. It will continue to evolve and world of data management needs that innovation and open source communities to thrive. But if the four points I raised are not addressed – I seriously doubt Hadoop will reach its potential any time soon.