“You have to draw the lines” and some other thoughts


Jon Reed wrote an excellent article on work life balance http://diginomica.com/2015/08/21/pulling-truth-from-the-viral-absurdity-of-the-amazon-workplace-debate/ and that prompted some thoughts in my mind based on my own work and life over the last two decades.

There are two simple principles in life that keep me sane when I follow them both religiously. And when I don’t follow them , it invariably screws me over 

1. We have to draw lines when it comes to how we want to lead our lives . If we don’t do that, someone else will draw it for us and we won’t always like it – in fact most times we will hate it .

2. There will always be some people smarter and luckier than us . We have to make peace with that quickly and not let it drive us nuts every day

Work life balance is just one example for this . I have a job that needs me to travel most weeks . If I don’t draw a line – I could easily be traveling every day of the week , all year , for my employer’s business . So I do draw reasonable lines and try to be home as much as I can . In the process of drawing these lines , I might lose out on some opportunities and the price to pay for that might be high. What often makes it hard is that we have to draw these lines without perfect data . But if I don’t draw that line – I won’t have a life outside work at all.

In my job, I spend a lot of time negotiating with customers, partners and my own colleagues . One of the  golden rules of negotiations is that “they will keep asking till you say no”. Saying no doesn’t come naturally to me . But if I don’t say no when required , then I can’t fairly complain when I am asked to do things that I don’t like . So I learned to say No and it helps a lot in managing my life on my terms . This is true outside work too.

Then there are the people who think being difficult is a badge of honor . In reality, it’s exactly the opposite . Being easy to work with is much harder than being difficult . If you want to be easy to work with, you need to make principled compromises on the fly . Being difficult is just an easy way to buy more time to make decisions . It slows things down and then no one will want to work with you any way. Being easy to work with comes at a high risk and it could be taken as a sign of weakness by some . But when you can combine being easy to work with AND the ability to draw lines when required , you will get somewhere you like .

I have always worked with a lot of over achievers . It is hard to not compare myself from time to time with them. There are many who are significantly more successful than me in any dimension I choose to compare – money , how fast they progressed in their career, how big is their house – or pretty much anything at all. It’s partly good fodder for motivating me to try harder , but for the most part it just made me miserable . Then I realized that there are two aspects that I am not considering 

1. They might be better at one dimension – like say they made it to executive ranks sooner than I did . But they have also paid a price for that – like sacrificing on quality time with family . Some chose to not marry or not have kids , unlike me . I wanted a family and when I add that to the framework, I no longer felt like an under achiever . 

2. There are always elements of luck, risk taking ability and intelligence that they may have that I have no control over whatsoever  . So why stress over that at all ?

This perspective is not something I have had all my life . I got it while I took a time out between my last job and my current job . It just made me realize that the only way to do a meaningful comparison is to see if I am better today than I was yesterday . Words cannot explain how peaceful I felt when I came to that conclusion . I just needed to get off the hamster wheel before I could get that clarity in my mind .

I will stop this with one last thought . If we don’t keep skills sharp , and keep learning all the time – our ability to say no will decrease at an alarming rate . And at a certain level , it might get to a stage where you cannot make peace with it . So it’s in our own best interests to stay relevant .

Social media and diminishing marginal utility


It was a cold evening in February 2008, and I was out having a beer with my boss in Beaverton, OR that the idea of starting a blog first came up. She and I had both read an article about social media in our intranet and we discussed the idea at length . We left the pub convinced that we should both start a blog and see how it goes . 

The only place I knew of people writing blogs at that time was SAP community network – SCN. The problem was that I had no interest in writing another “how to” blog , even though hundreds of them had helped me fix stuff in my own projects . I was already active reading and commenting on blogs at that time . So I thought maybe if I try to share my experiences on actual projects , maybe it will be interesting to a few people . So I started this in March 2008 http://scn.sap.com/people/vijay.vijayasankar/blog

That was a pleasant experience and pretty soon I joined Twitter , Facebook etc and started a personal blog here at http://andvijaysays.com for random topics . My social media habits have evolved over time – the latest being moving some conversations over to whatsapp – and has reached a stage where I need to evaluate if it’s worth my time to spend time on it . 

There are many plus points 

1. It is now my primary source of news . I scan all of them a few times a day to see if anything catches my eye . 

2. It helps me re-establish old friendships from school days . Even managed to find and meet buddies from primary school that I haven’t seen in 25+ years and that is invaluable 

3. It helps my work in many ways – customers know my POV when they google me , I find great candidates for open positions in my team and so on . Also gave me several friends whom I otherwise would not have met otherwise 

But it comes at a steep price in terms of time and noise and lost friendships

1. It’s hard to filter social media to just useful things unless you are on it all the time . Just when I thought I got rid of all the “let me post every famous quote” and “top ten list” people on Twitter feed , I found that I had a huge “I am smart and rest of you are idiots” set of people to deal with. Now I follow less than 200 and even that doesn’t fully help me – so I spend an hour every few weeks tweaking it . I know election season will make it an impossible task for almost a year 🙂

2. The onus is always on the receiver to decide what is of value and what is noise . I just need to look at the mirror to find a perfect example of noise creation on Facebook . I have friends from my professional world , friends from dog shows , family and school mates in my friend list . If I post my thoughts on software world , it is noise to my dog lover friends . If I write something about dogs , my class mates will find it weird . I don’t have the time or inclination to pick and choose who gets to see what – so I add to the problem that I am trying to solve for myself 

3. The best part of social media is people – and they are also the worst . People don’t always behave as adults in social media , especially in groups . I am right in the middle of one such group where 40 year olds act as 4 year olds throwing a tantrum . There is a cycle that repeats – handful of people with common interests come together and form a group , have a great time , pull in others , have a greater time , everyone gets comfortable and want to do greater things and solve world hunger ,  people take sides , smaller groups start, original group disintegrates – and so it goes over and over . Inevitably , the “founders” of the group are the ones who feel the pain the most . Moral : people are people , social media just provides a time lag for their true colors to show up . And when it shows up , social can help amplify the good or bad of it quite significantly 

4. Social media tries to be an equalizing force and it eventually fails . I have now seen this in three groups back to back in a short period of time . There is an assumption when people come together that everyone is an equal in a social media group . This falls apart as soon as the immediate goals are met and the group finds other goals . It sickens me to see people who were once great friends threaten and belittle each other in public for no apparent good reason . This is the biggest reason I sign off from many groups as soon as I smell trouble . I have enough stress as it is – why would I ever want to get more trouble voluntarily 🙂

5. In any population I have seen in social media , 80% of content comes from 20% of people . That doesn’t worry me by itself – it’s normal . What does worry me is that a sense of hero worship for some part of that 20% kicks in too over time and it always seem to end up in a mess . People without the guts or inclination to say something sometimes choose to live vicariously through the “holier than thou” gang  . This mindless support ( also funny that some of this is just sarcastic but gets interpreted as real support )  in turns strokes egos and a sense of invincibility apparently gets into the mind of some contributors . It’s all downhill from that point – apparently for me . There is no solution to this in a free world other than filtering out by people who find it unbearable . 

6. The interface between virtual and real worlds is not anywhere close to seamless . If your boss or your mom is in the same group as you are, you are limited in many cases in expressing what you really think . And the same is true about actually acting on decisions taken in the virtual world . For example – a group of us once decided on FB that we can pool some money and help people affected by a natural disaster in Kerala . A kind soul in the group who lived in Kerala offered to be the guy on the ground disbursing cash to the needy. At the end , he did it on good faith and then got questioned endlessly by others in the group . He quit in disgust and is still upset and I feel terrible that he had to go through it . The ease with which social gets us through decisions gives a false sense of how things work in real world . Thankfully there are many examples of thoughtful people using social media in useful fashion  – but the trouble they take should not be underestimated .

All things considered , I am still a big fan of social media . But I am also firmly convinced that it is less valuable than what I thought it once was . It adds enough value for me to stop using it , but I just need to filter way more aggressively and prioritize other things over social in many cases . 

Defining Hadoop with a straight face


Unlike datalake, I don’t think of Hadoop as a buzzword at all. It is a real thing – a real project you can touch and feel. Also, I love elephants (The one in the picture is an eighty year old Guruvayoor Padmanabhan, owned by a famous temple in Kerala)  – which might explain why I have a particular soft spot for Hadoop 🙂 . Hadoop is perhaps not yet ready to be an elephant this size, but it is not such a little baby elephant as shown in the marketing pictures either.

20130727-195627.jpg

By now, it is stable enough even for an obsolete programmer like me to play with it minimum fuss. In my day job, I don’t program any more (sadly – but probably a good thing for my team, and perhaps for the rest of humanity). But there is hardly a week that passes me without a need for explaining hadoop to someone. Even when I was between jobs and enjoying a nice vacation, I was pulled into “help me understand hadoop” discussions.

The primary problem for me personally is that the folks I usually have to explain hadoop to are not always conversant with open source. I get to talk mostly to people who are smart enterprise technologists, but who have very little idea of open source per se. Vast majority of them equate open source to free software. And it is not as if open source is a good all encompassing way to explain hadoop (hello MapR). And if I somehow manage to get these folks to understand open source and a few licensing models, I lose them again on the idea of what a distribution is and what is different between the distros. I think its mostly a mental block that there are commercial vendors who make money of software that is supposedly free – and they all do it in different ways. Can’t blame them – I had those questions too for a long time.

Once you get past open source, then there is a question of what constitutes Hadoop.

No one in my circles apparently cares about Hadoop common , so I no longer utter a word about that. And the few times I have mentioned it, mostly to friends who come from coding backgrounds – I have to deal with “why would anyone use java for building this?” 🙂 . Hence no “common” – nothing but grief from that conversation.

Then I talk about HDFS and MapReduce . Enter more rat holes – why does the world need yet another file system – why not use GPFS? I have heard of Spark – so when do we use MapReduce and why not just use Spark all the time? If I get a breather, I also get to mention YARN and Arun’s explanation of datacentre operating system, and that there is a MapReduce2. People get why YARN is a good idea almost instantaneously. Everyone appreciates the vision of pluggability – but invariably someone will ask about its compatibility with stuff that came before YARN .

This is usually where I start with the idea of different kind of nodes, mechanics of replication, why 3 is a default value, and why the whole thing is built for commodity servers daisy chained together. You would think this is the easy part – but it is not. We are dealing with people who have spent their entire careers working on high end servers that have all kinds of resiliency. It is a really hard thing for them to visualize the world hadoop is built for. Usual starting questions include “if I just repurpose all my high end servers, can I just avoid replicating thrice?” or “I already have tooling that takes care of HA and DR in all my data centers. Surely there are APIs that I can use to connect my existing tools to this hadoop thing ?”. This is a game of how many times you get to say “it depends” in any one conversation without taking a breath.

Just when I am ready to wind down the conversation – impatient listener will ask “can you fill me in on hive, HBase etc”. Sure why not – so I explain how there are a bunch of other projects that play alongside hadoop. “Are they all java?” – well, they are not ALL java, but ideally you should not have to worry given they all have interfaces you can use relatively easy. I can see relief !

HBase seems to be a trigger for starting on the NoSQL part of this Odyssey. This is particularly so because my friends know I spent some time working at MongoDB. Sadly – as in REALLY sadly – 90% of my conversations include a part of convincing people that MongoDB is not hadoop. And at a minimum at that point I have to touch on Cassandra to explain there are more NoSQL options out there. Invariably this opens up the question of “does MongoDB work with Hadoop” – thankfully it does and I explain the connector. It also usually leads to my friends from ops background sigh aloud “can’t we just use a general purpose database that does all these things” ? . I no longer fight them on this topic – mostly for lack of energy.

“So you have explained how data sits in hadoop, but you have not explained how I can put it in there or how I get these awesome insights from that data”. Ah I forgot all about that. So I go on to explain even more projects ( at some point it is overwhelming to remember all the names and the right spellings ) – and also manage to get in a plug that a lot of closed source tools can access hadoop. This is usually the point at which my ops friends give up on me and say “good Lord,  it is complicated to manage all this” and my dev friends get all excited that there are so many cool new toys to play with.

Usually the final question is on security. On the bright side, Ranger and Sentry are both easy names to remember. On the not too bright side – I don’t want ( or even know fully well enough) to explain why two major distros have two different approaches. And this usually leads to other examples like impala to show that not all distros share the same implementation philosophy. I should use something other than impala in future as my example – since that goes into “how many possible ways can you do SQL on hadoop?”. My usual temptation is to say “more ways than enough for you to get a costco size jar of aspirin” 🙂

I have not counted, but there are certainly more than 30 projects that can somehow be called a part of hadoop. I have personally played with less than half of them. And every day more projects are starting and it is getting hard to keep up. No wonder why my friends/customers/random people all get stressed out trying to understand hadoop and how to make use of it. And yet – everyone is excited about their hadoop journey, despite all its twists and turns. Marketers like my friend Ingrid who recently joined HortonWorks as CMO should have a fun time articulating a message that makes it much simpler to understand.

Alright, so who wants to ask me about Hadoop next ?