The cost of precision in BI


What percentage of decisions need precise data right upfront ? My guess is less than 10% or even lower .

A big decision for an average person is purchasing a home . Having gone through that exercise in two countries – and knowing many people who have done this before and after me , I am convinced that the data needed to make that decision did not need much precision . Based on your financial position – you could judge affordability within a plus or minus range of some amount . Another factor was school district rating – how many of us will care if the score was “9 out of 10” vs “10 out of 10” ?

Decision making is progressive – you find a “cluster” based on some characteristics like location , price etc . As you narrow down – you cluster again – flooring , yard size etc comes into play ( but again you don’t need super precision – if I am looking for a 2500 sqft house , I won’t overlook a 2400 sqft house because it didn’t precisely meet my criteria ) . And then comes the ultimate short list that needs some precision , and a final decision that needs excellent precision since you need to pay it to the seller .

Some version of this process is followed in all decision making , including sales , marketing, purchasing etc . Apart from legal and financial – i think almost no business function needs the type of upfront precision that it has chased since the beginning of time .

Look at an average BI project in enterprise world – 90% of time is spent on plumbing data – designing schemas , defining exception workflows , writing transformations and so on . Remaining 10% is used to make reports useful to its audience . This is unavoidable because BI is very static in nature – even what is called as-hoc analysis is limited by schemas in back end . In short – even the best BI solutions cannot mimic how human beings make decisions .

The quest for extreme upfront precision is what works against BI being useful – ironic as it might sound . And BI has no chance of being seriously disrupted till it stops expecting tightly defined schemas on back end , and high precision right upfront in all cases .

Context is way more valuable than precision . That is how we make decisions eventually in real life . And context changes with time – which means BI has no chance to keep up given its hard dependency on static things . BI world needs to think in terms of real world entities – not in some arbitrarily defined data models .

Good news is that technology and data science have progressed enough to do that in (more or less) repeatable and cost effective ways . Bad news is that the world of BI won’t go to the promised land without blood curdling shrieks , kicking and screaming .

Keep calm – our world of BI is changing , hopefully for the better .

Talent cannot be managed


Most people can and should be managed in an organization – but not talent . And talent is not a word that should be used in a light hearted and generalist way . That is how it loses relevance .

Interestingly, I did not learn this originally from corporate world or business school – I learned this from training and working with dogs . My intention is not to compare dogs to humans – just that I learned something from my hobby that empirically seem to also apply in my real career .

Most novice trainers go around looking for smart and intelligent dogs . What they don’t realize is that you can’t train smart dogs with novice training skills . It is one of those things that people won’t learn without making their own mistakes . Been there done that and learned , I think I learned at least . The dog for novice trainers is the one that is highly motivated by food , and an over eagerness to please the owner . That is – a pretty dumb dog , the opposite of an intelligent dog . Once they are trained – the dumb dogs will perform spectacularly to please their handler – but they usually can’t think for themselves to save their lives .

That is pretty much the case with management too . If you really want to work with smart people – you cannot manage them . They don’t thrive under management . They need collaboration and leadership . Set them in a direction , and get out of their way . Check in periodically and let them know you can be approached for help whenever they need . And then don’t let them down when they come to you .

“Talent” is not scalable – there is no such thing as a talented team of 100 people . I wish there was but I have never seen or heard of it . So be prepared to run smaller teams if you want to work with real talent .

“Talent” doesn’t mix with non talent – the moment they mix with people less smart than them , they lose interest . That is when turf wars and politics and all start – and they will out wit everyone else , even if the result is that the team will not deliver on goals . So if you decide that talent is what you are after – you need to be super careful as a leader to not lower your hiring standards .

“Talent” is fiercely loyal to their own ideas – and this is why they cannot be managed . It is a test of character for the leader to see if you can gain their agreement on a team goal . Or at least get them to disagree , but commit . And you will be the biggest idiot if you don’t consider their ideas carefully – because “I said so” is not what they consider as rational criteria . If you dismiss their idea – you need to beat them to the punch at an intellectual level they are at . Very very hard to do .

“Talent thrives on loyalty” – they value integrity in their leaders . If you fight for them when they needed , they will usually lay down on the tracks for you. Conversely – screw them over and they will screw you over harder than your worst nightmare . Don’t make promises you can’t keep and don’t give them standard company lines as excuses . They know exceptions can be made almost in every case . If they suspect you are not doing everything for them – that loyalty goes away in a hurry .

“Talent” will challenge you every step of the way. Don’t hire them if you can’t deal with it constantly . They will make you think when you would rather sleep or have a beer . It is not easy – you are either in the game with them , or you are in the cheap stands . No bench in this game. Always on !

“Talent” doesn’t tolerate breach of trust – they know that they won’t always get managers who are as smart as them . But as long as they trust the leader – this is not an issue . The issue is that they won’t usually give you a second chance if you break their trust . Be open and fair as a leader . And be consistent in being open and fair.

“Talent” needs direction – especially because they are quite capable of thinking about many ways to do things . And they will get bored if there is not enough challenge in their jobs . They will also be pissed off if you set unrealistic goals . It is a fine balance to strike .

“Talent” needs money , but money won’t keep them there for long – these folks don’t come cheap and most of them know their value quite well . But that is the easier part . The harder part is retaining them and keeping them positive without turning them into bitter employees .

“Talent” will walk away – but they probably will give you some time to get your act together . They might even politely remind you that they are losing patience . But they won’t sit around for ever to bitch and moan . They know that they are in demand irrespective of the economy or general market conditions . So they will walk away . And they won’t usually come back if you do counter offers – because they would have computed that in their decision making process before they chose to walk . So if they are leaving , allow them to leave on pleasant terms . You never know when you need each other . It is a small world .

Nothing but problems in general – so why bother with talent at all then ? Because these are the few people who will take the big swings and hit home runs . And that is what separates you at the end from competitors . If you are happy with status quo – don’t worry about talent

But here is the thing – you need the rest of the organization to work at peak efficiency so that the “talent” can be given the freedom to make big swings . If the rest of the organization is not disciplined – it is reckless to just depend on big swings to change your fortunes . That is hope – hope is not a strategy . There is a management concept called “policy by lapse” – that is not an admirable strategy , to say the least .

This is where rubber meets the road for companies with big innovation agendas . They tend to over do it by trying to light as many fires as they can – hoping that something will catch on . In this process – they forget that “talent” can easily deal with it , but others probably cannot do it to the same degree . And when “talent” sees the lack of differentiation in what got assigned to them – their passion will evaporate. And pure recklessness results – with no goals being met . The smarter companies know what part of the team needs to be industrialized and what is the portion of the team that can be allowed to make those big swings .

And in the process of (mis)managing “talent” – the real hard job is to take adequate care of everyone else . Today’s bills are paid by everyone else . “Talent” can only pay tomorrows bills . And every leader needs to keep that in mind .

Now , which part of the team would you like to lead ? All “talent” , All “everyone else”, or a mix ?

Big data – You can start small, but start today !


I am back at Houston Airport after an excellent customer event where I did a key note on big data . One of the side conversations that happened today was on where to start big data programs . Jon Reed had nudged me couple of days ago on the same topic . So here are a few things that come to mind .

I am a big fan of starting small but starting soon . Technology lets you scale as you go . So don’t wait for the most amazing earth shattering use case . You will need some time to get a first hand feel of how things work in big data land .

There are many ways to skin the animal that big data is – but if you want the easiest way to do it , my suggestion is to start with corporate data and work your way into bringing real world entities into the mix to see how they work together usefully .

My favorite way of finding a big data problem – which allegedly is influenced by an interest in corporate finance in b school back in the day – is to start with your financial data reported to the street . This is a summary representation of what is happening in your company and if you compare it across periods – you can spot what needs most attention .

Lets say Accounts Receivable is the one you spotted , and of course verified with other sources of data to make sure it is worth exploring . Now – what makes AR ? It is all those customers who bought your wares who haven’t paid you yet . What do you know about those customers ? Have they paid you in time before ? How is their credit rating ? Did collections department make a hundred calls before they paid last invoice ? This needs more data – granular data that you probably need to load from somewhere else .

What is happening with those customers in social media ? Not just twitter and FB – what do financial analysts think of them ?

Can you combine the AR and collections info with social media input to assess the chance of you getting paid ? How about seeing if the customer is holding payments because of poor service ? Have you tried to analyze the trouble tickets and service documents and see any trends ?

You could keep narrowing down till you find something you didn’t know before . Once you figured out what influences other data have on the AR info you start with – you can map it to the transaction processing . Should you discount more ? Should you put a billing block on the customer ? Can you have an API that can make there decisions while a transaction is in process ?

That is a full closed loop – moving from a small set of data to a bigger set data , only to find what tiny insights you can find in the context of a given transaction . And once you have solved that – cast the net wider , and keep going after all the opportunities you can find .

The trick here is to not waste time chasing every fork in the road . You will get a lot mod false signals that you need to smartly move past . Fail fast and move on . And if everything looks ok on AR , try another part of your financial statement trend that stand out.

Another way I attempt this is to talk to the person in charge and the persons who are at the farthest end of the business process . Ignore the layers in the middle . Like the VP of Sales and the shipping clerk / Store manager / sales person etc . More often than not – their perspectives will be different . Start from summary data and see of you can relate the source data from each of the others to see impact . As you add more and more sources – you probably will run into interesting trends , either with the impact on summary or between sources themselves . Stop and analyze and ask the business users on possible explanations . If you smell a rat , don’t assume something is wrong . But if the actual business users smell a rat – you have enough validation usually to dig deeper.

There are many caveats here . Organizational and Political motivations can easily derail you if you are dealing with data from multiple sources . Constantly verify data with multiple sources – systems and people . It is as much an art as it is a science .

Also keep in mind that data can be interpreted in many ways to suit someone’s needs . And just because you see correlation doesn’t mean you can assume something is good or bad . This is why I repeat myself all the time that data interpretation needs to be verified by people in the business who understand the context .

And always remember to close the loop – end of the day , data is good only if you can act on it . So make sure that what the data tells you is factored into your business process – maybe you need to enhance a transaction , maybe you need to empower a call
Center agent way more than today , may be you need less approvals in your work flow . You haven’t done justice till the loop is closed . Even then – you will need to keep tweaking it as the world around us changes faster than we can keep up .

As technology improves and more machine learning and AI gets mainstream , the effort to do all of these will come down for sure . But then, the complexity of problems we will attempt to solve in mainstream business will also be orders of magnitude bigger . So – start small , but start today .

Got to run – my flight is boarding 🙂