What percentage of decisions need precise data right upfront ? My guess is less than 10% or even lower .
A big decision for an average person is purchasing a home . Having gone through that exercise in two countries – and knowing many people who have done this before and after me , I am convinced that the data needed to make that decision did not need much precision . Based on your financial position – you could judge affordability within a plus or minus range of some amount . Another factor was school district rating – how many of us will care if the score was “9 out of 10” vs “10 out of 10” ?
Decision making is progressive – you find a “cluster” based on some characteristics like location , price etc . As you narrow down – you cluster again – flooring , yard size etc comes into play ( but again you don’t need super precision – if I am looking for a 2500 sqft house , I won’t overlook a 2400 sqft house because it didn’t precisely meet my criteria ) . And then comes the ultimate short list that needs some precision , and a final decision that needs excellent precision since you need to pay it to the seller .
Some version of this process is followed in all decision making , including sales , marketing, purchasing etc . Apart from legal and financial – i think almost no business function needs the type of upfront precision that it has chased since the beginning of time .
Look at an average BI project in enterprise world – 90% of time is spent on plumbing data – designing schemas , defining exception workflows , writing transformations and so on . Remaining 10% is used to make reports useful to its audience . This is unavoidable because BI is very static in nature – even what is called as-hoc analysis is limited by schemas in back end . In short – even the best BI solutions cannot mimic how human beings make decisions .
The quest for extreme upfront precision is what works against BI being useful – ironic as it might sound . And BI has no chance of being seriously disrupted till it stops expecting tightly defined schemas on back end , and high precision right upfront in all cases .
Context is way more valuable than precision . That is how we make decisions eventually in real life . And context changes with time – which means BI has no chance to keep up given its hard dependency on static things . BI world needs to think in terms of real world entities – not in some arbitrarily defined data models .
Good news is that technology and data science have progressed enough to do that in (more or less) repeatable and cost effective ways . Bad news is that the world of BI won’t go to the promised land without blood curdling shrieks , kicking and screaming .
Keep calm – our world of BI is changing , hopefully for the better .