I am not the biggest football fan around – I am a big fan of cricket though. And despite my day job is about making sense of data – I don’t use much of quantitative methods when it comes to sports . I think it takes away my excitement .
After the Super Bowl game finished – I saw on twitter that SAP had predicted that Denver will win over Seattle in a close match . As it turned out – Seattle won a rather one sided match with a very young side . A few friends on twitter pointed out that SAP made a bad prediction before the game , and they are not wrong .
In parallel, I decided to skip watching the India vs NewZealand cricket series thinking India will win this 5-0 and it will be boring . I was close on my gut prediction – the score was 4-0, just that India was on the losing side of that equation . On the bright side , I am happy that I didn’t have to watch the massacre and live with the nightmares .
I didn’t work on the predictive Analytics solution that made the prediction for Super Bowl and I am not authorized by SAP to provide a response . But since I am sitting at PHX waiting for my flight in the midst of many dejected Denver fans who are analyzing the result in painful detail – I wanted to share my personal views on this matter .
Predictive Analytics in general cannot be used to make absolute predictions when there are so many variables involved . In fact – I think there is no place for absolute predictions at all . And when the results are explained to the non-statistical expert user – it should not be dumbed down to the extent that it appears to be an absolute prediction .
Predictive models make assumptions – and these should be explained to the user to provide the context . And when the model spits out a result – it also comes with some boundaries (the probability of the prediction coming true , margin of error , confidence etc). When those things are not explained – predictive Analytics start to look like reading palms or tarot cards . That is a disservice to predictive Analytics .
If the chance of Denver winning is 49% and Seattle winning is 51% – it doesn’t exactly mean Seattle will win . And not all users will look at it that way unless someone tells them more details .
In business , there is hardly any absolute prediction ever . Analytics provide a framework for decision making for the business leaders . Analytics can say that if sales increases at the same historic trend , Latin America will outperform planned numbers next year compared to Asia. However , the global sales leader might know more about the nuances that the predictive model had no idea of, and hence can decide to prioritize Asia . The additional context provided by predictive Analytics enhances the manager’s insight and over time will trend to better decisions . The idea definitely is not to over rule the intuition and experience of the manager . Of course the manager should understand clearly what the model is saying and use that information as a factor in decision making .
When this balance in approach is lost – predictive Analytics gets an unnecessary bad rap.
That being said , I heard next year Super Bowl is played in Arizona . Maybe I should start following the game a bit more closely 🙂
And Vijay Says... 
Vijay,
Predictive is only as good as the population size. A small group of men and a single event is just to small. Also as you noted, confidence intervals are everything. If you want to see the future, read Asimov’s foundation series. Great stuff. The universe is the population.
LikeLike
Great blog Vijay. I believe the SB result was more of an anomaly than it was that the technology got it wrong. Sometimes these things happen – especially in sport. I would, however, like to see a series of games predicted by the technology to see how close it gets on a number of games. A big occasion like the SB has even more factors than a “normal” game.
LikeLike
Hey Vijay,
Can’t agree more with what you’ve said here, but I would expand.
I’ll just point out that SAP makes software that is marketed as “predictive”. I’m sure this comes as a surprise to you 😉
The point is, when you say, “not all users will look at it that way unless someone tells them more details”, it’s not an abstract “someone” we are talking about here. This is the responsibility of SAP’s software, education, and marketing. In my opinion, SAP is really pushing “ease” of predictive software without addressing the issue of understanding, and in the process is dropping the ball in terms of both software design and marketing.
Regards,
Ethan
LikeLike
I personally think one of things holding back wider adoption of Predictive Analytics is not the software but getting people/customers to understand the results and then putting programs in place for actionable change in their organizations . I think Josh Bersin does a solid job explaining that in this article
http://www.forbes.com/sites/joshbersin/2013/10/07/big-data-in-human-resources-a-world-of-haves-and-have-nots/
On a side although I feel predicting the score and talking about how “white uniforms” might influence the outcome we not the best choice I do feel that SAP Marketing has executed very well on the using sports to help popularize big data (which obviously helps Vishal’s “little girl” longer term) 🙂
LikeLike
Agreed – and how to use, interpret, and understand analytics is more important than the technology itself (although that’s still important).
LikeLike
Here ya go: https://twitter.com/SAP/status/302150227756994560 – Don’t be too hard on them. They’re in marketing. They don’t know much. 😉
The other challenge with predictive analytics that we’ve found is that there is no “one size fits all” algorithm or model. It still requires data scientists or people knowledgeable about the domain to frame the questions and the data in a way that can potentially result in the desired (or new) insights.
One of the aspects of HANA that always impressed me was the breadth of analytics capabilities “baked in”, the inclusion of “R”, and the extensibility of the platform. That enables your partners and ecosystem to add in the domain-specific magic.
LikeLike
Hi Rick – OK so, to be fair to our SAP Marketing colleagues, in the tweet they DO say that businesses use analytics to make INFORMED decisions :). They didn’t say that human decisions are irrelevant. Analytics can provide additional dimensions on which (hopefully) better decisions can be made (by people).
Your point about “one size fits all” algorithms is right on. Models and algorithms can help to find correlations. Understanding of Causation and Interpretations are still best done with human instinct and judgement.
LikeLike
Interestingly, earlier in the year, one of your SAP Marketing colleagues tweeted that “executives shouldn’t trust their gut instincts” and should instead trust analytics. I disagreed then and still do. 😉
LikeLike
If someone said that – I will disagree too , Rick
LikeLike
Not sure I totally agree. Gut instincts are useful, but why would you ignore the analytics? Are you basing it on the fact that SAP didn’t get the superbowl right? That’s a matter of probabilities, like Vijay pointed out. Wouldn’t you want to get the data, the analysis, and then see what your gut says? I would.
LikeLike
Charles, I could show you some glaring examples of why analytics need interpretation. One easy one: A former CEO of a company I worked with a couple years ago, fresh off a Wharton MBA, fed a lot of data about our business into a predictive analytics model. It told him that our revenues were most directly influenced by the square footage of our office space. Uh huh.
LikeLike
So I think we’re saying the same thing. In extending your analogy, what was the alternative? Bring in a new CEO and have him use ‘gut instinct’ without any data/analytics? Realize that we’re basing this on the fact that SAP (along with every TV analyst) blew the call on the SuperBowl. That event was an anomaly, it exists in the long tail. If you apply that logic in a casino, you may win the occasional game, but in the long run you lose. No gut instinct required here, just probabilities…..
LikeLike