It’s about time we set a higher bar for analytics

First off – I have no interest in being nuanced here about data vs information , big data vs regular data , BI vs Analytics vs Reporting and so on . Use the terms you like in the following rant . If all we missed today was the nuance between these terms , we would have been in a better place already .

There might not be a more tired part of technology out there today than Analytics . It’s been a top CIO concern for as long as I can remember . It is a top concern for me personally as an executive running a business at my employer . There are very few people who are truly happy with their ability to make sense of data . And no wonder BI companies young and old are all thriving and marketing their hearts out that they are already in “next gen”.

Big data – and the Gartner 3V model – brought good focus on information management side . But it did not exactly democratize “data based decision making “.

We don’t need to get into high volume or velocity or variety of data to see the failings of today’s BI . The thought leadership in analytics is along the lines of “ask good questions to get good answers”. This is a much needed part of analytics for sure – we should have the sophistication in our systems to answer the best and most complex questions . However – that should be more of a table stakes thing in the world of data .

For analytics to take a leap into “next gen” – pretty visualization is not enough either . It is a little more along the way into future than the ability to answer complex questions . An answer is not good if the user does not understand what the system is trying to say . So yes – let’s find more and more ways to visualize data . It’s a good thing that some companies got started with it and are making progress .

Before I leave the topic of visualization – I have to say this . For a given question and its answer , computers must be able to provide a default representation that is the “theoretical best” , without a user being asked to create a representation from scratch every time . By all means – allow the user to change things around , but there is very little value in making a user guess what is the best representation visually . The type of chart , positioning in screen , scale , default filters are all things that software should be able to figure out by itself . Personalization is important – but again it should be treated as table stakes by now .

But there is a gap that still remains at the core . If the best we can do is “ask good questions to get good answers”, we are still stuck in the world of art , and refusing to move a bit closer to science .

My hope is that we start building systems that can
1. look at available data and start by
2. offering clues to what kind of questions can be asked of it ,
3. what kind of meaningful patterns the software can be seen already ,
4. and what kind of data would be great to add to currently available data set to make even better inferences .

Sure , the person who can ask better questions will still retain their edge in making better use of the data – but if the system can prompt decent questions to users , I think this whole promised land of “data driven decisions” would be a lot closer to where we are now than if we inched along the current trajectory .

This might be really hard to do as a generic horizontal platform capability in near future . But if vendors focused on taking this approach for targeted vertical apps , a lot of these challenges can be mitigated . Such learning can then be used to build horizontal general purpose solutions over time if it makes sense .

It all starts with setting a higher bar of expectations of analytics – incremental innovation won’t cut it .