And Vijay Says…

Yesterday evening, while flying from PHX to JFK, I had a chance to read this excellent blog by Arun Murthy . If you have not read it yet – pls read it first before you read my rant below. As always – these are strictly my personal views .

First – I think Arun is probably the best person to write a blog like this and needs to congratulated for doing so. He has been part of the Hadoop story from the very beginning and continues to play a significant part in shaping its evolution. It also felt that it came straight from the heart – and extra points for starting lists from 0 and not 1 🙂

My own point of view was developed over the last couple of decades as a data geek who worked on a range of technologies on data management as well as analytics for a wide variety of clients across the world.

Data is largely an unsolved problem in the large enterprise world. Just when you think you have it under control, you realize that the problem got bigger and more complex. And you also realize the tech has improved and now you have more options on how to solve the bigger problems. This is one reason why most techies eventually use philosophy as a framework for explaining the evolutionary nature of their work.

It is hard to predict which way an exploratory project will go. This is great for developers as it gets their creative juices flowing. But that is not how enterprise CIOs think. They value high doses of stability and predictability , and very low doses of complexity. They very well know that the pundits will use terms like “legacy thinking” to shame them all the time. But their world comes with flat and declining budgets and there are always a lot of lights to be kept on. Within all those constraints – the good CIOs try to foster as much innovation as possible. And of all the innovation they have sponsored in the last decade or so – Hadoop definitely was top of the list.

World was ready for Hadoop . Classic datawarehousing had been pushed to its limits. Data warehouses became data dumps. Cost of maintaining those datawarehouses started driving everyone nuts. And Hadoop promised a solution for all these and more. Plus the open source nature gave all the geeks even more incentive to introduce it in their shops.

In my opinion, 4 things led to Hadoop’s alleged demise

1. Too many options for clients to choose from
2. Unskilled people implementing it
3. Multiple changes in market positioning 
4. High operational complexity

Pretty soon – everyone ran into challenges. MapReduce was no longer sufficient to do most of what enterprises wanted to do. No worries there – Spark etc came up just at the right time and took over. The world realized that you just cannot run away from SQL even if you criticize it heavily. So many different SQL on Hadoop projects came into being and that did not always work in the way traditional IT shops expected. IT shops are not used to having tremendous choice in solving problems. When Cloudera and Hortonworks proposed different solutions to a problem – be it SQL, be in security or whatever – it became very confusing for the people who were trying to implement a long term solution in their shops. In short – “Hadoop is a philosophy” started getting interpreted as “there are no real best practices here – just keep experimenting” by a lot of clients. Just to keep it brief, I am skipping the divergent direction MapR took – and that story did not end well either.

Then came the question of skills. For enterprises to adopt technology faster – you need a lot of people with that skill. Much like how SAP market got flooded with poor skills when ERP was hot – Hadoop market did too . That had a direct effect on the quality of implementations. Many clients are still struggling with tech debt caused by using developers and architects who did not have good fundamentals in data management. Net net – hardly anyone replaced any data warehouses , and data lakes became the new data swamps. To be fair – the growth of classic data warehouses have been significantly curtailed since hadoop became mainstream.

Arun has already explained the “What is Hadoop?” question in great detail. So I will skip that entirely.

What also did not help a lot was the positioning of hadoop companies changed over time – perhaps to sustain the insane valuations in private markets. It swung from data management to analytics and ML to managing everything in cloud. When you try to do everything – even if it is a great problem to solve – it is hard to execute to perfection, and it confuses clients a great deal.

The last point that made it difficult for Hadoop was operational complexity. Data management is a lot of fun for developers. But the moment it is in production – rock solid operations is what keeps it going. Even for RDBMS based systems – DBAs and other Ops experts with all the mature tooling still spend significant time managing their landscapes. Hadoop ( to be fair – most NoSQL DB too ) did not prioritize ops sufficiently. In my view at least – this was perhaps the biggest miss and one I think Cloudera and others should urgently address. Every client I know will be grateful if managing Hadoop was significantly simplified – especially between on-prem and cloud.

All this said – I don’t think Hadoop is dead , or that it will die. It will continue to evolve and world of data management needs that innovation and open source communities to thrive. But if the four points I raised are not addressed – I seriously doubt Hadoop will reach its potential any time soon.

 

Having “Chief” in your title is pretty awesome . I am friends with several people who have “Chief” in their title and they are rightfully proud of it – and I am very proud of them for getting to those important career milestones. I have my own ambitions on this matter – some day I hope to the the “Chief Trainer” at the dog training business I intend to establish post retirement. While I don’t exactly know at the moment what people with the title “Growth hacker” actually do – that is also a cool title I intend to take on in that post retirement business.

The baseline case is that the person running the company is called CEO and the direct reports to the CEO carry “Chief ” in their titles like COO, CMO, CFO, CRO, CHRO etc. CEOs cannot have an indefinite number of people who directly report to him – so some other C level roles get one notch down in the pecking order. For example – CIO working for CFO, a Chief Risk Officer working for the COO and so on. From that point on – it gets out of hand pretty quickly.

While I chose to rant on “C” titles here – this does happen in many flavors. EVP/SVP/GM roles for example all have a habit of playing out the same way.

There are about a half dozen common reasons I can think of on the top of my head why additional “Chief” roles get created. I am sure there are many more.

1. Some “must have” movement happens in the industry and pundits will advise the board and CEO that this needs a C level role. Hence roles Chief AI officer, Chief Analytics officer, Chief Digital Officer and so on get created with great intentions to drive transformation – but not always with any real budget or authority to make decisions.
2. Some “top talent” types won’t join the company without a C level title. Off late I have seen “Chief Growth Officer” as a title cropping up – and many of the people occupying those roles are people who otherwise would have only joined for a CEO or CRO role.
3. A way to retain senior talent for a period of time till another plan can be created . A famous big data company had a chief or technology, chief of engineering, chief of engineering and a few other similar sounding titles co-existing . This scenario is very common when the company is limited in making compensation increases. Similarly it could also be a way to show that a growth milestone has been met, usually for startups. The VP of Legal will become General Counsel, VP of sales becomes CRO, VP of marketing becomes CMO and so on – even with very little or no change in actual responsibilities.
4. A way for the boss to punish someone without actually firing them immediately . I know of a CTO who created a title of “Chief of intellectual renewal” for a very senior engineering leader who was out of favor. It could as well have been “Chief Librarian” . It could also be the other way around – A way for the boss to save face after hiring someone with a great resume only to find out quickly that the candidate is not going to work out, but cannot let the person go immediately due to contractual or PR issues . Not that long ago, a high profile CTO hire was converted to a Chief Business Officer with an unclear charter – who then left the company several months later.
5. A way to temporarily make sure a big transformation effort does not get off the rails – like a business reorg, a new product line , a big M&A etc . Lots of overlay roles are put in place as guard rails – especially for making sure organizational antibodies don’t kill the change before it takes root.
6. As largely aggregation functions – where the role in reality just adds up the work of everyone below then in the hierarchy, and reports it to the top. And similarly serves as a traffic cop on decisions coming from above.

All the above reasons might have been absolutely valid for the context in which those decisions were made . But the trouble is that once these roles are created – they don’t get eliminated even if there is limited proof of value add. And there is a tipping point where everyone will agree that a C level title is meaningless because it is given away cheaply.

What will absolutely happen every time is that the overall cost will go up – and revenue might not proportionately go up. These additional CXOs will need support functions. They create additional operational overhead. They exponentially increase communication overhead. They will all measure the same things across various dimensions and ask similar but repeated questions to the same people on the frontlines, wasting the time they should spend on value adding activities . And at a certain scale – invariably they will create conflicting messages and confuse the heck out of everyone.

Over time, a top heavy organization is a diminishing returns proposition. In the simplest case, it makes a company less agile. If market needs it to shed overhead quickly – it is a lot harder to let go of people at the top than at the bottom of the pyramid even if that is the right solution. Too many people sitting in meaningless roles creates a problem in promoting organically, and worse still – there will always be people aspiring for those low value jobs because the title is impressive.

So what can we do about this ?

Clearly the corporate world will not eliminate every low value CXO role – so what we need is a framework to minimize the trouble it comes with .

First – The CEO and the Board should come to terms with how much dilution can they live with C level titles being dispensed liberally. Obviously if they are ok with significant dilution, they deserve the pain it will invariably generate.

Second – Don’t create any leadership role, especially a C level role, unless there is a clear charter that makes it a unique necessity. That charter should then come with adequate budget and decision making power to deliver on the charter. Agree on a clear way to measure the success of the role before announcing it.  This cannot be a one time activity just when roles are created – all roles need to be reviewed critically every few years.

Third – if you need to make an exception for any of the half dozen reasons I listed above, then put a firm time frame around the life time of the role and define periodic check points to see if you still need the role. Make it a CEO level decision if the role is still needed after the role gets to its expiry date.

Fourth – Ideally, hire people on a time and outcome bound contract into those additional C level roles – so that it is an easier conversation with the people when the role expires, or performance does not match up to agreed levels.

Fifth – Automation ( as in analytics, AI, BI etc ) and business process changes should be the first way of solving aggregation issues. It may not be enough to eliminate all redundant roles at once – but in general it will help point out the glaring holes in organizational design very quickly.

 

 

Over the last few days – I had multiple conversations about different aspects of anxiety in the work place . I also chimed in on a couple of threads in social media about it . So when I woke up today – I felt like I should share some thoughts about how I deal with it myself .

When I was a young consultant – I was anxious all the time . Every Sunday I would get anxious about getting on a plane on Monday morning . Every time a Partner visited my project, I would get anxious . Every time I had to make a presentation or report status – I would get anxious . It didn’t take much to for me to feel anxious .

It would manifest in many ways – ranging from sweaty palms on the mild side to acute acid reflex on the harsh side . I was misery personified . What made matters worse was that I also tried really hard to hide my anxiety from my colleagues . And strangely I never asked anyone for help – and just chose to suffer through it myself . Not a lot of my friends even know today that I had to deal with those problems back when we worked together .

It had an impact on my career progression too – At 29, I was still a senior consultant when most others who started with me were managers and one or two were already senior managers .

While struggling through anxiety for several years – I also got better at analyzing problems and experimenting with solutions . I started analyzing what was causing my anxiety and what could I do about it . And at some point – I think I cracked the code !

Life turned for the better – and rather dramatically . I started feeling better physically , and started enjoying Sundays . I no longer threw up before big meetings . And my career took off – in another 5 or 6 years I got into the executive ranks .

So what did I find out ?

The primary cause of my anxiety was fear – or more precisely the fear of getting fired !

I had very good skills for my line of work . And my line of work – SAP development – was in hot demand . And if I messed up at work – the first thing a Partner or my client would do was certainly not to fire me . In hindsight all of this is plain obvious . Just that it took me half a decade to realize that the odds of getting fired were really low !

I also realized I had two big weaknesses to overcome .

1. I could have come to this conclusion quickly if only I had asked for help sooner .

2 And there was a possibility that even though I had great skills – I just didn’t know enough good people to find a job if and when I needed one.

So I started actively seeking help .

Thanks to my wife insisting on it – I went and saw a doctor and he prescribed something that helped with my acid reflux . He almost immediately diagnosed that it was stress related – and he was right . As soon as my approach to work changed – acid reflux went away and I didn’t need the medicine any more .

Looking back – a part of the problem was that I was (and still am) an introvert whom many people who know me mistakenly take for an extrovert . By now I know I am not alone with this situation, and I can joke about it 🙂

I started asking for help early and often and that made a big difference . I no longer felt the need to be the smartest person in the room who knew all the answers . This also made me realize that everyone has some difficulty asking for help . The moment I started asking for help – others in my team did so as well . Collectively – we figured out solutions much faster and with less stress . Another learning was that it is a very limited exercise if you only helped people who could help you back . You have to help what you can and then over a long period of time , you tend to always get more help than you ever expected .

I also made it a point to keep my skills sharp all the time . Every quarter I set a goal to master a new skill and I would use the weekly plane rides to get it done . It’s a habit that has become second nature . It has helped me change my line of work several times over the years – from ABAP to SAP functional consulting to BI to CRM to AI and so on . Off late – my interests have also widened to ethics , psychology and history . Eventually it became no longer about job risk mitigation – but it doesn’t hurt that it serves as an insurance in case I ever need it .

Then came the need to network . It didn’t take long to realize that just by connecting to a lot of people on LinkedIn and Twitter didn’t do me any favors in having a useful network . It’s a painstaking process of building meaningful relationships one at a time – starting with strengthening existing friendships and business relationships and then working from there to extend to others . It takes a long period of time and there is no end to it – it’s something you do all the time , and again without making it a “I will only help people who help me ” transaction .

There are other factors involved like living under your means and saving for a rainy day , taking good care of your health , prioritizing your family over work and so on . You can’t take those for granted – just by optimizing on work alone will not get you to a good space .

The confidence – and especially the peace of mind – that comes from removing fear from your mind is something that you need to experience for yourself . Words are not adequate to explain it . It’s a huge feeling of liberation from a jail that you created for yourself .

It’s not that I no longer feel anxious – I absolutely still do . It’s just that I have learned how to use it to my advantage instead of letting it stress me out . The “trick” for me essentially is to have a routine about things I feel anxious about .

For me – that includes listening to music – usually Carnatic , getting plenty of sleep (I need 7 to 8 hours) and focusing hard on just the first couple of things I need to do to get into a rhythm . If I have a presentation to make that I am starting to worry about – I focus on making sure I know what I have to talk to for the first 2 or 3 minutes . Once I get through that – my experience kicks in and I can get through the rest quite easily . If I have to review my business with my bosses – I think about what they would want to know and figure out that aspect of my answer very well . I spend less time worrying about peripheral things . If I still feel the stress – I know it’s because I need more help . I call one of my mentors and spend a few minutes talking with them and pretty quickly I am back in a good mental space .

Dealing with your own anxiety is one thing . That in itself takes a lot of effort – but it still might not be enough . There is a high chance that people in your team are anxious – and you may actually be the reason for that . As I grew into leadership roles – this started becoming more and more a topic of interest for me .

My approach to this is as follows –

1. I cannot be insecure at all if I have to help some one in my team with their anxiety . This means I need to think carefully about how I hire , how I communicate and so on . Insecure managers compound the insecurity of their team .

2. Everyone is different . What worked for you to minimize your anxiety might not work for them at all . I remember a young colleague who got anxious about flying – fearing that the plane will crash if there is turbulence . That led to a couple of glasses of wine every trip and some times even before getting into the plane . I tried to help but this was beyond me – and I was happy that this person went to a professional and got the help he needed .

3. You need to proactively and consistently take fear away from the work place – and then make sure that other people in your team are reinforcing that behavior .

4. Your primary expectation as a leader should not be to be liked – it should be to be respected and trusted . If they like you – that’s a nice side effect . The truth is that you will have to take hard decisions that affect people in your team . As long as they know you have been consistent and fair with your decision – they will understand and respect your decision even if they don’t like you for what you did . I try to be as transparent as I can be with my team – and give them headlights into what will happen next for each course of action we take .

5. All that said – there is one area where I haven’t been able to minimize my anxiety . That is about firing people . Almost invariably the moment I take that decision – I feel sick and the acid reflux comes back full swing . It’s predictable and that makes me realize it’s my body preparing me and I get through it with some pain . It is one area I definitely need to improve .