Big Data Solutions – Do Questions Matter ?


I have Ray Wang to thank for this post. Off late, I have a serious problem of writers block. I just cant find a topic interesting enough to write about, and consequently have become a ratherirregular blogger – at least compared to last year. Any way – back to the topic of this post.

Ray tweeted this few minutes ago

A lot of BI blueprinting sessions from my consulting career flashed through my mind when I saw that. A key principle for a good BI system design is in finding out upfront most of the questions a user would ask the system, and then designing a solution around that. Unfortunately this is a blessing and a curse – while we can really optimize getting fast and accurate responses to predefined questions , this also curtails our ability to change our mind and ask different questions. More experienced BI experts will second guess other questions that users “may” ask and leave some room in design to cater for that, but it is clearly not a scalable way to do things.

Somehow, users were also trained along the way to agree to some lack of flexibility in BI systems. While the complaints never went away fully, most users think by now that it is normal for BI team to ask for some time to change the datamodels and create new reports and so on. It is a sort of “marriage of convenience” if you will – with tradeoffs understood by both sides.

So when we let go of “ordinary” data and embrace “big” data – what should change? I think we should use the big data momentum to make BI systems more intelligent than the rudimentary things it is capable of doing today. And this intelligence should be done with some business savvy. In other words both “B” and “I” of BI need some serious tweaking.

In my opinion, what should change right away is the expectation of business users needing to state most of their potential questions upfront at design time of the system . Or more clearly – the expectation should be significantly lowered, and business users should be allowed to ask more ad-hoc questions than they have done so far. Of course we can never guarantee full flexibility – so some subjectivity is necessary on where we draw the line. Just that the line should be drawn musch farther from where it is drawn today.

Accuracy of result for ad-hoc questions is not enough – the results should come back in a predictable and short time frame too. Ideally, all questions should come back with answers ( or a heads up to user that this is going to take longer ) within a predefined timeframe (say like 3 to 5 seconds or less).

Then there is the question of how the users ask these questions. SQL or NoSQL – querying languages do not provide democratic access to data. People should be allowed to ask questions in English ( or whatever language they use for business ). Some training might be needed for the system and for the users to understand the restrictions – but no user should be constrained with the need to know how things work behind the scenes. A minority of people should have the skills to educate the computer – the rest of us should not be burdened with that. Instead, the computers should be smart enough to tell them answers to what questions users ask.

There are very seldom exact answers to questions in business ( or life) – even apparently simple questions like “what is my margin in North America ? ” is ambiguous to answer. Most clients I have had have many different meanings to “margin” and “North America” and “My” within their organization. In real life, if these questions are asked of a human analyst, she will ask follow up questions to you to clarify and then provide an answer with necessary caveats. Why can’t systems do that? Wouldn’t life of users be vastly improved if systems answered problems like humans did, in a way humans understand? of course with more speed than humans 🙂

Big data or otherwise, there is always an issue of trust in the data from user’s perspective. Most analysts spend nearly as much time explaining how they arrived at their results, as they take for compiling and analyzing the data. The system goes through all the computation any way – even today in the non big data world. Why can’t our BI systems explain to the user how it arrived at the result all the way from source to target or backwards? Wouldn’t that increase productivity a lot?

When users ask questions – they usually will also combine it with external data (google, spreadsheets etc) before they take a final decision. Would it be possible for a BI system to present some useful contextual data to the questions from internet and intranet and allow the user to choose/combine what he needs?

And one last thing – if the system is intelligent enough to find answers, why can’t it have the smarts to also figure out the best possible presentation for the results? Today – we mostly have to predefine how output looks like visually. Why put that load on users? Can’t systems be smart enough to look at the question and the answers and figure out the best way to represent it to the user? This is not a “big data” problem – this should have been the case all along, but somehow never quite happened in a mainstream kind of way.

This is by no means an exhaustive list – I left out plenty of things like collaboration, predictive responses, closed loop BI and so on. I didn’t do so because they are unimportant, but only because of the boredom factor. These types of things are already happening to some extent, and hopefully will catch on more as time progresses.

So there you have it – its my birthday wishlist. And thanks again Ray for that much needed spark to blog again 🙂

Is “Out Of The Box” a Myth ?


I had a brief exchange of tweets with Ram Manohar Tiwari ( @rmtiwari on twitter ) recently on out of the box thinking , and since then a lot of thoughts have been brewing in my mind about this .

Everyone I know in corporate circles is a fan of “out of the box” . I don’t think a day has passed in my working life these past few years without someone mentioning it explicitly . It almost gives me an impression that there is this huge big box , full of regular joes like me – and a handful of smart people who stand outside and try their best to get us on their side .

My view on this matter is that there is no such thing as “out of the box”. People maybe able to get out of “A” box , but they will be in “some box” all the time . And because people are different from each other – most people should be able to tell others to get out of the box . All they can do is get out of their current box, get into a box with more space , and when that box fills up – then jump to yet another box .

Some times you might even have to visit the box you were in earlier – the one that you took pains to get out . If you need an example : think of people moving out of mainframes and now going to a similar model in cloud computing 🙂

I also doubt we are just in one box at a time – my feeling is that we are at the intersection of several ones at any point in time . This makes me claustrophobic just thinking about it . This is also the reason getting out if a box is hard – it is like a relational DB. All the dependencies need to be taken care off before you whack a table 🙂

All things considered , I now think teams will have more success by making use of diversity in its members than making a homogeneous team try hard to get out of the boxes they are in .

I think I have sufficiently bored you by now . If you need to be “un-bored”, try some out of the box thinking ….NOT

IT In India Could Use Some Help – Are You In ?


I woke up this weekend to this depressing news http://toi.in/ot3e_a

I have worked at various SIs all my life before deciding to join SAP labs in January of this year . So this problem hit me hard – and in some way, I felt that I am responsible too somehow for this dismal situation that the younger generation is facing .

This is not just an HCL problem – every SI I know of has had this issue of having a big mismatch between supply and demand . The irony is that these SIs all have very capable S&OP type experts who have done fantastic work for their clients solving this exact problem . Yet they can’t seem to solve it for their own business .

The academic world in India does not work as closely with Industry as it should. I am a mechanical engineer by training – but there were hardly any good mechanical engineering jobs when I came out of college . The only decent jobs were in IT – that too in SIs . There was practically nothing that I learned in Engineering that I could directly apply to my IT job . And yet, vast number of mechanical engineering students come out of colleges every year and look for IT jobs . Why isn’t there a supply adjustment to suit demand ?

It is not as if the education is much better for core mechanical engineering needs itself. The labs in most colleges still use engines that were obsolete 40 years ago . When I went to college – auto transmission was popular outside India . I remember just passing references to it in my text book – and that was it . I am glad I did not have to do mechanical engineering for a living . I just wasn’t well prepared for it . As I talk with young students now – I think the syllabus has barely changed from when I did engineering 20 years ago.

And yet – thousands of new engineering grads are churned out every year . This cannot be good – the average quality is not good for their core discipline nor is it good for IT .

IT education is not much better . People still learn C and java and come out of school looking for software jobs . I met a recent computer science grad last week who did not know why servers use fusion I/O cards or even SSDs. They very rarely have seen good code in college , because there is very little interaction with corporates . So corporates take them, train them and few years later they are productive . If academicians took a look at what the industry wants – these unproductive years could be absolutely minimized . But most of them don’t – they just love status quo.

The VC culture in India is nascent at best . All the major VCs have presenxe here – but several people who could use their investment have no idea where they are or how to get their attention . And vast majority of people don’t understand even basics of how startups work . It pains me to see some of them fall prey to local loan sharks . This lack of awareness results in several brilliant students live a “next best life” as a programmer at an SI, and rise through its ranks to varying degrees of success . It depresses me to no end . I try to talk to as many people as I can and try to give them pointers and help them build a network – and I know several others do it too . But it is not done at a scale that matters , given the magnitude of the opportunity cost .

This should change – and every Indian who knows better should spend some time and effort in helping those who don’t . If enough people take an interest at grass roots level – I am sure this could change for the better . Actually, I am not sure any more – but I sure hope and pray that it will change !