Root Cause Analysis – it needs a grain of salt

Vast amounts of time and resources are spent in the corporate world in what is called “root cause analysis” (or RCA for short). I have done my fair share – countless spreadsheets and PPT decks have been created by me too like most of you who are reading this . And from the time I was in engineering college to today , I have read a lot of books, white papers and blogs on this topic . I have asked my team to do root cause analysis countless times – at least occasionally in a mindless fashion.

And I have to say this – skills and luck are both needed to do a decent job . It’s more art and less science – and it will take a lot to convince me otherwise.

Here is something I once ran into at the beginning of my career . An incoming high value sales order EDI document failed to post in the ERP system . It was the second time in one week that this happened and the customer ordered a root cause analysis – and finally it was me who had to do it at the bottom of the hierarchy . It took me all of 10 minutes to find that the translation system had wrong mapping for a field or two . It took a day to work with that vendor to change the map . We didn’t see the problem for a good while after that . My ability to analyze was praised – and although I didn’t get a raise , I did get some lovely emails and all that .

And then at a party few days later, I was introduced to a VP at the customer. He thanked me profusely and said the business impact was pretty high for missing the order twice . He also asked me if I thought EDI was the wrong medium for an order of that high a value . His other question was whether the contract with my employer had any SLA’s based on business impact . I had never thought of that angle – and in all honesty I told him I can’t answer it . My boss picked it up from there – except, 15 years or so later I haven’t forgotten the incident .

1. There is seldom “A” root cause

I – and many others – assume there is this one clean reason why something (bad) happened . There is never one reason . It’s always a combination of reasons . ( I can’t resist the temptation of saying it is like a nested JSON structure in a document database than the near rows and columns of a relational table )

2. It’s not repeatable for the most part

The way I do root cause analysis is not how someone else does it . More often than not – you will get different causes if you ask two persons to do RCA .

And it will cost an arm and a leg to come up with MECE answers ( MECE is what consultants learn in their bootcamps – mutually exclusive, collectively exhaustive) . And even at that cost – it might not be MECE after all.

3. Asking “Why” 5 times doesn’t solve it either

Asking “Why” is necessary – but to the best I can jog my memory , I can’t remember one incident where I exactly asked it 5 times to get to the root cause(s). It is an arbitrary number that on occasion might be 5.

To begin with, you can’t ask the right “why” questions about things you don’t know at a deep enough level even if all data is available . When programs fail and we debug and look at logs – it makes sense only if we understand correctly what we are looking at . A C programmer might not be the best to analyze a Unix dump .

Asking too many “why” questions is also counter productive from a time and budget perspective – you can only bark up so many trees at a time before the cows come home 🙂

4. You can’t always determine cause and effect

What we find as root cause might be the effect of another cause . It almost always is the case . It’s like a recursive function which can’t get out of itself .

Over a few beers, a friend and I were once discussing the root cause for the poor performance of an application we had coded . I will spare you the many colorful details (all very logical if I might say so) – but by the time we left the bar , it was crystal clear that the reason for sub par performance of the code could be traced back to some traumatic childhood experiences of a common friend .

Parting comment – set the right expectations before you dive into root cause analysis . Make peace with the fact that most probably you can only treat some symptoms , and over time and with luck on your side – you will solve enough symptoms and some causes to make things work at a decent level .

RCA is a dish that always needs some salt – some times a grain , and at other times a pound . I will leave you with that .


Looking back and forward on the same day

So it’s my birthday – and my mind is oscillating furiously between being too old and not growing up fast enough . It’s a strange feeling !


the boat that my 9 year old daughter made for my birthday

On one hand – I do feel rather old . I work in a company where vast majority of people I run into are smart, good looking twenty somethings. At a company party, in my first week there , I heard a woman colleague lamenting that she can’t keep up with the young crowd . So I asked her how old she was – and her answer was 29 ! I nearly broke down in tears ( and under my breath said something like – “shoot me dead now – actually poke me in both eyes before you shoot me, and drop kick me as I fall” 🙂

And yet – talking to several friends my age and older who continue to work at large companies , and not all of them are thrilled with their status quo but for their own reasons they can’t move out – I feel like the young guy with free
spirit 🙂

Ok that was a stretch – practically no one equates me with free spirit , but hey it’s my blog and I am going to take some blogger’s license here .

When I finished my MBA – I had a clear plan for retiring at 35. 35 came and went and I am nowhere close to retirement – which made it rather depressing , but I wrote it off as my MBA naïveté . Now I am firmly convinced that I don’t ever want to retire – I think I will go nuts and will drive my family nuts if I do . What I need is periodic vacations – and I am slowly getting better at taking time off work . It’s one of my goals for when I grow up . The guy I need to thank the most for finally taking time off from work is my mate Dennis Howlett . He told me many times – with and without beer in our hands – how time off work helped him, and it kind of worked out that way for me too .

I used to take pride in my programming abilities – and looking back today, I had no right to be . I can still code in C , ABAP and a little bit in java . And over last few years, I figured out JavaScript and Hana – but clearly I am nowhere as good as I thought . Bad news – I doubt I can get a programmer job anymore to pay bills . I better forget that security net and focus on improving skills I need for my current job . There was a time when I could write 500 lines of code in a stretch with no syntax or compilation errors – and I used to take that for granted . Can’t do it any more – and it hurts !

On the bright side – what didn’t kill me made me stronger . All those hours spent tackling programming stuff is coming in very handy when I learn MongoDB – given my natural way of understanding something new is by comparing and contrasting with stuff I know from past . There is some light at the end of the tunnel after all .

Looking back – I am most thankful to some fascinating leaders I got to work for who paid it forward to let me be successful in my career . And I am incredibly proud of the people who came after me who I could help along their way . I know they will go places – and I would be proud to work for them one day . If I am ever successful in my life to deserve a good legacy – I want to be remembered as a guy who helped create a few more leaders in this world . I could do a lot better on that front than I have done in the past – and I intend to do just that going forward .

Last but not least – if there is one true regret above all others , it is that I didn’t spend enough time with my family and our dogs . I can’t go back in time and fix that retrospectively – but I hope to make up for it, and then some, from now on .

Definitely need to lose some weight too – but then , some promises seem to be meant for following year – on a rolling one year basis 🙂

IBM seems to have everything , but TIME

IBM is a good case study for balancing out everyone singing the “transformation is cool” tune . Transformation is utterly painful – almost directly proportional to the size of the company .

So IBM announced its Q1 results and no surprises there – HW business is not in good health , and software and services are not exactly stellar either .
But that is the top line story . Bottom line – or specifically EPS, is a different story . EPS continues to grow as IBM promised in its plan .

Several people commented today that The best days of IBM are behind them . I actually think the opposite is true . I think IBM will come back with a roar in couple of years .

Why do I think so ? For two reasons primarily

1. Look at their investments in last couple of years while their top line was shrinking . They formed the Watson group – and put their best leaders in charge of growing it . They bought Softlayer and are moving full speed to make it the base of their cloud strategy up and down the stack . They partnered well with growing technologies ( capitalized on Hana momentum from SAP on the hardware side – and yes they sold it off to Lenovo , supported MongoDB API in DB2, etc) . They haven’t let the slow down on top line from affecting their investments – that is how IBM works . They bet on the long term and don’t panic (at least in public). On top of that they continue to invest heavily in their research capabilities . iBM spends more money on research than many hot shot companies make in revenue .

2. They have a very deep leadership bench – and has the brand and money to attract more leaders if needed . This is by far the most important reason for me to believe IBM will come back with a roar . There aren’t a lot of leaders of the caliber of Ginni Rommety , Steve Mills etc. The people working for them are equally talented and it goes a few layers deep . And if that is not enough – they have plenty of cash and brand value to attract away most top leaders from other companies .

So what is it that IBM doesn’t have ?
They don’t have time on their side !

Investors are not always a patient bunch . On the bright side , people buy IBM for guaranteed dividends and long term appreciation . Come what may – I am sure IBM has enough capability to squeeze costs to make earnings promises be made good . But if the investment community doesn’t show patience – they might have to cut deeper than necessary to deliver and that helps no one .

As much as I care for IBM – I feel terrible every time they let go of people, since I have a lot of friends who work there . And in every round, some of them get affected . That makes it difficult for me to justify what looks like a good decision on economic front . That does not do IBM any long term good – a business is only as good as the people who work there, and not just the leaders . But that is the price to pay when IBM does not have time on its side .

For the sake of all my friends who still work there , and for my own sake as an ex-IBMer who cares for the company – I hope that IBM can buy some time to make its come back and not cut much more deep into its payroll to deliver on its commitments to the street .

We need “A teams” not “All A Players”

This is the greatest mistake that happens when we try building teams . We start out with the intention of hiring an “A team” – and pretty soon it magically transforms into a hunt for “A players” . It took me a long while to realize it’s not the same – and often hiring all A players into one team hardly makes any sense .

There are only two ways of building an A team that I know of .
1. If you have money and luck on your side – hire all A players you can to fill each role in the team.
2. OR, hire a few A players with strong leadership skills and fill the rest with B and if needed some C players . Then make it the Responsibility of A players to bring along the B and C players

Option 1 is a no brainer . A players love working with other A players – but there are only so many of that caliber. And you probably can’t hire them all.

Individuals can win a few matches with their sheer talent . But to win consistently and become championship material – it takes a team . Not just any team – but an A team .

My favorite example is the Indian cricket team that won the 1983 World Cup . There were two top talent players – Sunil Gavaskar and Kapil Dev. Rest were average or slightly above average players if you look at their life time stats . Yet – Kapil led that team to win the World Cup , defeating the best West Indies team ever in the finals . That is the nature of A teams .

The Australian side of Steve Waugh in 90s was probably close an A team built with all A players . And they kept winning for a very long time , except when that crop of players retired – the team could not make another A team of A players . Now they are taking an approach which is closer to building an A team with couple of A players and the rest with B players .

I have had the fortune of working in A teams where every one was a star player , and also in A teams where only some players could be described as stars . But in both cases – output was stellar . And my current thinking is that the viable model is to discourage people from trying to build an A team of solely A players .

However, an A team with A, B and C players needs some ducks to line up in a row before their execution ability gets into stellar territory. Here are my top 6 thoughts .

1. An A team needs strong leadership – ideally an A player who can take responsibility of bringing along and improving the non-A players to play at major league. It’s the acid test of motivational ability of a leader.

2. Hire one or more A players to keep the leader honest throughout the time the team works together . Most often, of the leader alone is an A player – their ideas will go unchallenged . That can only hurt in the long term . You always need people in the team who can challenge the leader when needed

3. Never allow the B and C players to recruit . If you do – then the chance of them hiring an A player is close to zero. The biggest long term risk is the standard of incoming players decreasing over time and the team moving from stellar to mediocre status.

4. Don’t drink the “culture fit” koolaid in big gulps . It is important that people get along – but end of the day, fun needs to be balanced with output. And output needs specific skills and experience . This is a grey area at best – and a true test of a good leader is the ability to find the right compromise.

5. Stretch, but don’t break . A players typically can stretch a lot in the assignments they take on. Don’t let that make you try to stretch the B and C players too in similar fashion . Everyone has a breaking point – and the sooner the leader and team realizes everyone’s breaking point , the better the chance of great execution. In most cases , B and C players and can stretch the same as A players over time – as long as it is done in small increments .

6. Rotate players when they are successful . Let new players come in with new ideas . Let your A players go to other teams and expand their horizons . Share the wealth of talent – that is how it multiplies and scales .

Education is the key for unity in diversity in India

It is parliament election season in India now . The country is so vast and infrastructure hasn’t caught up yet – so elections have to be held over several weeks in a phased manner . Whichever way I look at it – a coalition government is bound to rule India for the next term . And that usually is only slightly better than no government . And a country so diverse as India – with 1000s of languages and tens of states and countless religious beliefs – good solid governance is a must have to ensure continuous progress. It’s not a trivial challenge – to say the least .

There are three strong political groupings – one led by Congress , one by BJP and a third by the new AAP .

Congress led United democratic front – which in reality is neither united nor democratic – ruled for the last two terms . They barely got to govern with their structural issues . Regional parties held the government to random throughout the term. And given Manmohan Singh was just a figure head – whom the party could not find a safe seat to get elected, and hence had to get him in through the back door via Rajyasabha – his ministers seem to have had no intentions to take his lead . One of the greatest presidents we had – Dr Kalam – wasn’t even given a second term . And now they are projecting Rahul Gandhi as the PM candidate . If Congress can’t find a leader outside the Gandhi family in all these years , how exactly are they going to watch out for the common man ?

It’s not as if Gandhi family is the only one looking for politics as a family business venture . Veteran leaders like Chidambaram , Jaswant Singh and many others found no moral issues in asking for seats for their kin . The country has a billion plus people – why does this keep happening across all political parties ?

There is BJP – and they have some very strong leaders . The big issue with them is that despite almost a guaranteed secular rule if they come to power – like their last term – they find it necessary to stir up a Religious agenda to get votes . That regressive nature is not fit for a secular republic that the constitution envisions . India needs economic development – which happens with unity of purpose, not with communal tensions . BJP tends to keep church and state separate when they govern – but not when they are running for offices. It’s pretty sad that this is the best way for them to come to power .

And finally there is the AAP . I was elated and disillusioned back to back with AAP . Corruption is a very serious issue in India and Arvind Kejriwal took a great step in rooting that issue . Unfortunately – they lack political maturity and ended up as a one trick pony . Running away from office in 49 days meant that they couldn’t take the heat of actual execution . Talk is cheap – and now they have to deal with the stigma of being a party that ran away . Real life governance is not binary – you cannot govern with the attitude that you will do only one thing . Fighting corruption is fantastic – but the state needed more and AAP didn’t stay to deliver . However , I am positive that in 5 to 10 years , AAP will be a very viable party to form a good government . And in the process, they will force other parties to reform and get closer to their citizens

In my mind – the reason for this mess is mostly a lack of education. We always get leaders that we deserve . The politicians who are after our votes would rather have a large uneducated and poor vote bank – which can be swayed by simple and mostly false promises . If I can’t find three square meals a day – I will vote for the guy who offers me free or subsidized food . If I am an oppressed factory worker who has no skills to find another job – the communist ideology might be too attractive for me to see other options . If I can’t read – I probably won’t realize that Rahul Gandhi is not the great grandson of Mahatma Gandhi . If I didn’t have access to a variety of TV channels and newspapers – I might not realize that not all Muslims are terrorists and that many Muslims have served India at highest offices . If I don’t go to school – I might never hear about gender equality .

And for that – education is key. Education improves life across multiple dimensions – and gives us options . And it teaches us how to evaluate options . It allows us to have different beliefs but unites us as a country . It makes us Indians first and followers of a religion second . Education is what India needs – in abundance . It takes a lot of political maturity for a political party to push education at grassroots level across the country – as opposed to pontificating in an election manifesto.
The onus is on those of us who had the good fortune to get education to help our fellow citizens who were less fortunate . Part of that responsibility is in choosing wisely the people we want to represent us in government . Let’s choose wisely every step of the way for this election and every single one coming after it .

What on earth does VP of Global Channel at MongoDB mean ?

Ever since I announced I got a job at MongoDB , the question that I got the most has been “what does a VP of global channels do ?”

I grew up as a technologist – not as a business development / channel person. And apart from engineering , I have been a successful sales guy too in later stages of my life . That is the primary reason why I get the question – why did I take a job running channels .

This is how I explained my vision to my friends and family .

Let’s ignore software business and think of a manufacturer of home plumbing supplies . The company’s goal is to revolutionize how modern houses do their plumbing . For this to happen – the important thing is that adoption of the new way of plumbing goes viral – and enough money gets made to keep the business running and growing .

If the goal was to become a high margin company in the shortest time – the idea would have been to sell the minimum possible with the highest prices for product and services . That is not what this company wants to do in our example – it is exactly the opposite . Here the course will be to get as many customers to use the product as possible – irrespective of how much each pays . It is an important distinction – the goal is to radically modernize how people do plumbing .

There are two types of people who generally buy the products – home owners and contractors . One way or other – it’s the home owner who eventually picks up the tab . So it goes without saying that the company must convince home owners to use their product . You cannot take focus away from home owner ever . No one is more important a stake holder to make your goal a reality .

But this is rarely sufficient . If the sales person sold the product – who is going to do the installation of the plumbing ? Who would connect old plumbing to the modern plumbing ? When the bathroom plumbing changes – it usually happens along with redesigning the flooring , faucets, showers etc . Who does that ?

So it is imperative that all the plumbers and bathroom remodelers and general contractors need to be trained in using the new way to plumb . That usually needs them to be convinced this is a better way of doing plumbing , and that home owners will give them business if they have the skills and experience .

To begin with – how many sales people can be hired to go after each home owner ? At some point you want the supplies store to stock your product . Sales person can make a sale and send the home owner to a store to pick up the stuff .

There are other things like bathroom vanities and showers that will need to work with the new way of plumbing . Initially, it might be enough to hire a contractor to connect disparate things by hand with a connector . But some home owners might expect pre-fitted shower heads and vanities that are DIY friendly . So someone needs to convince those manufacturers to work with you to put your way of plumbing into their products for the benefit of your mutual customers.

As the business grows – you want new products to come up that are specifically built for your way of plumbing . You want a large number of people to work with your plumbing to create even more sophisticated solutions for the home owner – to the point that people take it for granted that if they are plumbing , they are using your product as the default . That is when the world of plumbing has been revolutionized .

As people make optimized widgets for your way of plumbing, you might need a one stop shop to help them and your customers – like your own marketplace.

In this entire process – the idea is to help the customer have a better quality of life enabled by your plumbing. You want to reach a state where your plumbing is behind everything in the home – to the extend that no one thinks twice about its presence . You can’t get there by yourself – the plumbers , the general contractors , the retailers , the manufacturers of vanities and shower heads and many others need to work together with a unity of purpose (improving the quality of life of customer) to get there .

And as more and more people work with the product – there will be more things that come up that needs the company to tweak its products and services . The old plumbing companies and you might learn a trick or two from each other – it’s a big world with a place for everyone , especially if everyone plays together and talks to each other .

That is the principle and spirit of channels – and it applies just about the same to software as it applies to plumbing supplies or soaps or cars. No one company can revolutionize the world by itself – but by working in a trusted relationship with several good partners, we can get there together and make some happy and excited and loyal customers .

That is the journey I am embarking on at MongoDB – to help and build and nurture an ecosystem around an amazing product to revolutionize how the world builds applications. I know it will be quite a ride – and I am very sure I will get to work with a lot of people who read this to reach that goal .

Now, Let’s go make our customers successful – and let’s do it TOGETHER !

Higher standards , double standards and no standards in leaders

So the CEO of Mozilla resigned – and I saw the news on twitter last night when I came back to my hotel room . And the question that came to mind was – should CEOs and other leaders be held to a higher standard ? Do double standards apply to some ? And are there any lines that can be drawn on what is appropriate and what is not ?

I am a HUGE fan of equality . And I have many friends in the LGBT community . I know what pain prop 8 caused those of them in California – and hence when I first saw this news on twitter , my instinct was that it is a good thing that he is no longer the CEO.

And then I thought of politicians with extra marital affairs that became public – including a past president . Some of them left public life , some – like Clinton – didn’t . Well, Clinton apologized – but he didn’t resign from office . And years later , it seems his other good deeds compensated for his bad deeds and he is touted as a great leader .

If rewind a lot more in history – there is Adolf Hitler who ruled over Germany . He was a vegetarian and was faithful to his wife . He loved his dog dearly . But he also did extremely bad things and the fact that he didn’t have extra marital affairs nor his vegetarian diet was used by anyone to cut him some slack in history . And that is the way it should be – there is no justification .

Nehru, India’s first PM, had a public romantic relationship with Edwina Mountbatten even before Independence . That didn’t stop him from being elected back to back till he died .

The point here is – we have historically judged leaders on the impact of their “day job” – not on other things they did or said . In the grand scheme of things – Clinton, Hitler and Nehru all had more powerful jobs than the CEO of Mozilla . This is why I am wondering if there are double standards in how we evaluate leaders .

America is a republic – and citizens have rights guaranteed by constitution . Whether I agree with him or not – and I don’t – He is free to have his own opinions and beliefs . He doesn’t need to conform to anyone else’s belief system . Even if he is in an absolute minority – he can stay true to his beliefs as a citizen of USA . So this is where I wonder maybe there are no standards to be applied – if someone like a CEO doesn’t conform to popular sentiment, then maybe all bets are off these days , and his constitutional rights don’t give him much of a benefit . I feel kind of uncomfortable with that line of thinking – everyone is a minority in some way , and that is the reason why rights are guaranteed in the first place . But legal rights don’t mean society holds the same opinion .

Could he have protected his company’s equality culture while personally being opposed to gay marriage ? I don’t know – there wasn’t enough time to find out given he had the job only for 11 days . But going by the examples of Nehru and Clinton – those guys didn’t try to make rules legitimizing adultery, despite their own weaknesses on that front . So maybe Eich could have pulled it off – or maybe he could not have .

Increasingly , we tend to not have lines drawn in sand . World just seems to have a lot more grey areas with every passing day . I usually have a strong opinion on these things – but this time I am just pretty thoroughly confused .