Root Cause Analysis – it needs a grain of salt


Vast amounts of time and resources are spent in the corporate world in what is called “root cause analysis” (or RCA for short). I have done my fair share – countless spreadsheets and PPT decks have been created by me too like most of you who are reading this . And from the time I was in engineering college to today , I have read a lot of books, white papers and blogs on this topic . I have asked my team to do root cause analysis countless times – at least occasionally in a mindless fashion.

And I have to say this – skills and luck are both needed to do a decent job . It’s more art and less science – and it will take a lot to convince me otherwise.

Here is something I once ran into at the beginning of my career . An incoming high value sales order EDI document failed to post in the ERP system . It was the second time in one week that this happened and the customer ordered a root cause analysis – and finally it was me who had to do it at the bottom of the hierarchy . It took me all of 10 minutes to find that the translation system had wrong mapping for a field or two . It took a day to work with that vendor to change the map . We didn’t see the problem for a good while after that . My ability to analyze was praised – and although I didn’t get a raise , I did get some lovely emails and all that .

And then at a party few days later, I was introduced to a VP at the customer. He thanked me profusely and said the business impact was pretty high for missing the order twice . He also asked me if I thought EDI was the wrong medium for an order of that high a value . His other question was whether the contract with my employer had any SLA’s based on business impact . I had never thought of that angle – and in all honesty I told him I can’t answer it . My boss picked it up from there – except, 15 years or so later I haven’t forgotten the incident .

1. There is seldom “A” root cause

I – and many others – assume there is this one clean reason why something (bad) happened . There is never one reason . It’s always a combination of reasons . ( I can’t resist the temptation of saying it is like a nested JSON structure in a document database than the near rows and columns of a relational table )

2. It’s not repeatable for the most part

The way I do root cause analysis is not how someone else does it . More often than not – you will get different causes if you ask two persons to do RCA .

And it will cost an arm and a leg to come up with MECE answers ( MECE is what consultants learn in their bootcamps – mutually exclusive, collectively exhaustive) . And even at that cost – it might not be MECE after all.

3. Asking “Why” 5 times doesn’t solve it either

Asking “Why” is necessary – but to the best I can jog my memory , I can’t remember one incident where I exactly asked it 5 times to get to the root cause(s). It is an arbitrary number that on occasion might be 5.

To begin with, you can’t ask the right “why” questions about things you don’t know at a deep enough level even if all data is available . When programs fail and we debug and look at logs – it makes sense only if we understand correctly what we are looking at . A C programmer might not be the best to analyze a Unix dump .

Asking too many “why” questions is also counter productive from a time and budget perspective – you can only bark up so many trees at a time before the cows come home 🙂

4. You can’t always determine cause and effect

What we find as root cause might be the effect of another cause . It almost always is the case . It’s like a recursive function which can’t get out of itself .

Over a few beers, a friend and I were once discussing the root cause for the poor performance of an application we had coded . I will spare you the many colorful details (all very logical if I might say so) – but by the time we left the bar , it was crystal clear that the reason for sub par performance of the code could be traced back to some traumatic childhood experiences of a common friend .

Parting comment – set the right expectations before you dive into root cause analysis . Make peace with the fact that most probably you can only treat some symptoms , and over time and with luck on your side – you will solve enough symptoms and some causes to make things work at a decent level .

RCA is a dish that always needs some salt – some times a grain , and at other times a pound . I will leave you with that .

Looking back and forward on the same day


So it’s my birthday – and my mind is oscillating furiously between being too old and not growing up fast enough . It’s a strange feeling !

20140423-085658.jpg

the boat that my 9 year old daughter made for my birthday

On one hand – I do feel rather old . I work in a company where vast majority of people I run into are smart, good looking twenty somethings. At a company party, in my first week there , I heard a woman colleague lamenting that she can’t keep up with the young crowd . So I asked her how old she was – and her answer was 29 ! I nearly broke down in tears ( and under my breath said something like – “shoot me dead now – actually poke me in both eyes before you shoot me, and drop kick me as I fall” 🙂

And yet – talking to several friends my age and older who continue to work at large companies , and not all of them are thrilled with their status quo but for their own reasons they can’t move out – I feel like the young guy with free
spirit 🙂

Ok that was a stretch – practically no one equates me with free spirit , but hey it’s my blog and I am going to take some blogger’s license here .

When I finished my MBA – I had a clear plan for retiring at 35. 35 came and went and I am nowhere close to retirement – which made it rather depressing , but I wrote it off as my MBA naïveté . Now I am firmly convinced that I don’t ever want to retire – I think I will go nuts and will drive my family nuts if I do . What I need is periodic vacations – and I am slowly getting better at taking time off work . It’s one of my goals for when I grow up . The guy I need to thank the most for finally taking time off from work is my mate Dennis Howlett . He told me many times – with and without beer in our hands – how time off work helped him, and it kind of worked out that way for me too .

I used to take pride in my programming abilities – and looking back today, I had no right to be . I can still code in C , ABAP and a little bit in java . And over last few years, I figured out JavaScript and Hana – but clearly I am nowhere as good as I thought . Bad news – I doubt I can get a programmer job anymore to pay bills . I better forget that security net and focus on improving skills I need for my current job . There was a time when I could write 500 lines of code in a stretch with no syntax or compilation errors – and I used to take that for granted . Can’t do it any more – and it hurts !

On the bright side – what didn’t kill me made me stronger . All those hours spent tackling programming stuff is coming in very handy when I learn MongoDB – given my natural way of understanding something new is by comparing and contrasting with stuff I know from past . There is some light at the end of the tunnel after all .

Looking back – I am most thankful to some fascinating leaders I got to work for who paid it forward to let me be successful in my career . And I am incredibly proud of the people who came after me who I could help along their way . I know they will go places – and I would be proud to work for them one day . If I am ever successful in my life to deserve a good legacy – I want to be remembered as a guy who helped create a few more leaders in this world . I could do a lot better on that front than I have done in the past – and I intend to do just that going forward .

Last but not least – if there is one true regret above all others , it is that I didn’t spend enough time with my family and our dogs . I can’t go back in time and fix that retrospectively – but I hope to make up for it, and then some, from now on .

Definitely need to lose some weight too – but then , some promises seem to be meant for following year – on a rolling one year basis 🙂

IBM seems to have everything , but TIME


IBM is a good case study for balancing out everyone singing the “transformation is cool” tune . Transformation is utterly painful – almost directly proportional to the size of the company .

So IBM announced its Q1 results and no surprises there – HW business is not in good health , and software and services are not exactly stellar either .
But that is the top line story . Bottom line – or specifically EPS, is a different story . EPS continues to grow as IBM promised in its plan .

Several people commented today that The best days of IBM are behind them . I actually think the opposite is true . I think IBM will come back with a roar in couple of years .

Why do I think so ? For two reasons primarily

1. Look at their investments in last couple of years while their top line was shrinking . They formed the Watson group – and put their best leaders in charge of growing it . They bought Softlayer and are moving full speed to make it the base of their cloud strategy up and down the stack . They partnered well with growing technologies ( capitalized on Hana momentum from SAP on the hardware side – and yes they sold it off to Lenovo , supported MongoDB API in DB2, etc) . They haven’t let the slow down on top line from affecting their investments – that is how IBM works . They bet on the long term and don’t panic (at least in public). On top of that they continue to invest heavily in their research capabilities . iBM spends more money on research than many hot shot companies make in revenue .

2. They have a very deep leadership bench – and has the brand and money to attract more leaders if needed . This is by far the most important reason for me to believe IBM will come back with a roar . There aren’t a lot of leaders of the caliber of Ginni Rommety , Steve Mills etc. The people working for them are equally talented and it goes a few layers deep . And if that is not enough – they have plenty of cash and brand value to attract away most top leaders from other companies .

So what is it that IBM doesn’t have ?
They don’t have time on their side !

Investors are not always a patient bunch . On the bright side , people buy IBM for guaranteed dividends and long term appreciation . Come what may – I am sure IBM has enough capability to squeeze costs to make earnings promises be made good . But if the investment community doesn’t show patience – they might have to cut deeper than necessary to deliver and that helps no one .

As much as I care for IBM – I feel terrible every time they let go of people, since I have a lot of friends who work there . And in every round, some of them get affected . That makes it difficult for me to justify what looks like a good decision on economic front . That does not do IBM any long term good – a business is only as good as the people who work there, and not just the leaders . But that is the price to pay when IBM does not have time on its side .

For the sake of all my friends who still work there , and for my own sake as an ex-IBMer who cares for the company – I hope that IBM can buy some time to make its come back and not cut much more deep into its payroll to deliver on its commitments to the street .