Election 2016 might shake our faith in data science


First off – I am about as independent and undecided as one can get in this country. In general I am fiscally conservative and socially liberal. I am not a big fan of Clinton, Sanders or Trump – and can’t even decide who I dislike the least. But – as the primaries wind down, I am starting to follow the election with great interest. This is not merely political interest or for entertainment – it just gets me all geeked out on the potential of data science to help these candidates.

Trump is the “presumptive” candidate for GOP. That does not mean anything about his chance of winning. He is not anywhere close on any dimension to the kind of GOP candidates that went before him in past elections. So the idea of “red states” and “blue states” as it exists today do not really matter to figure out how he can win.

I am not sure if his campaign used extensive data analysis so far. On TV, it seemed like he just used his big personality (and the free media coverage in drew) as the primary weapon and it threw off the conventional campaigns of his competitors. Even the grand daddy of all data scientists who cover elections – Nate Silver – was thrown off his game by the Trump campaign. But now that he is the presumptive candidate – his campaign will probably take a more data driven approach to his approach in general elections.

The modeling approaches could get very complex for figuring out what Trump should do to attract votes. I read somewhere that his big voter base is white males without a college degree. Since education has generally increased over the years, It will be interesting how the percentages work in each state. The general theory is that black community do not like Trump because of him challenging Obama’s birth certificate issue. But if all things remain equal, even if he takes a small chunk of black community votes – he might carry the state. But then women don’t like him either apparently – which adds to the complexity of a predictive model . Past polling data and all kinds of analysis that RNC must have done – my bet is that it won’t be of much use and new models will need to be created and tweaked.

Its not any easier to predict what works best for Clinton to get the 270 magic number in general election. Math looks to be in her favor to win the primaries of her party. That said – Bernie Sanders has an extremely loyal base , especially amongst young voters. Even if Bernie himself endorsed Clinton at the convention as I think he will – I am not sure whether his supporters will care for Clinton. In that case – will they vote for her, stay home or god forbid, vote for Trump ? Also – just as GOP data scientists  will have to find what exactly works for Trump – Clinton camp will need to find messages that work against him. Given no history exists for a candidate like Trump – this exercise should wear out a lot of keyboards.

While Trump is famous for his gut instinct driving his primary wins, there is one aspect that makes me think there is a bit of data analysis that has helped his cause. His idea of tagging Bush as “Low energy”, Rubio as “Little”, Ted Cruz as “Lying” and finally Clinton as “Crooked” seems to me like a data driven strategy. May be it did not start that way and his gut instinct gave him the idea to begin with Jeb Bush. But my best guess is that his team picked up on it and tested the other adjectives before he used them effectively in his debates and stumps. Of course I can only guess – and I would love to see what comes out after the election cycle is over and someone writes a book.

I am sure a book or two will be written – this election will put to test a lot of data science and its practitioners. I can’t wait.

 

 

 

What on earth is “cognitive computing” really ?


When I first heard the term few years ago , I thought “this is somewhere between marketing buzzword and science fiction” . It is now the single most often asked question for me in my work life , and pretty soon this is going to be no different in my social circles too . Rather than provide a definition – I am sure IBM website and Wikipedia will have some articulate definitions – let me try to provide some background on how we make decisions and then try to tie it back to WTH cognitive is .

How do we make important decisions in real life – like say ordering a beverage at Starbucks ?

  
We use some data (sugar is bad for me) , we use some past experience (the Starbucks in chandler doesn’t make a good capuccino all the time)  , we evaluate some options (should I have a frapuccino instead for this one time) and then we pick a path (no I will take a capuccino but with an extra shot) that we think has best odds ( need heavy caffeine dosage to keep up with my calendar) . It’s almost never on the basis of perfect information . Decisions are mostly made on directionally correct information (frapuccino is higher in calories) , not on precise information (it’s exactly 375 calories more ) . Speed , flexibility and agility holds higher value in decision making compared to extreme accuracy and precision . If you are a barrista making a cappuccino  – there is probably a perfect ratio of ingredients , temperature etc that you use to make the perfect beverage . But getting it roughly in those ratios in quick time is more useful to you as a barrista (and probably to the customer) , than spending an extra ten minutes in making it perfect . Beyond a point , the strive for precision gets into diminishing marginal returns .

Then there is another important aspect – we have to tune out distractions to get the job done . If you are a barrista at Starbucks at Phoenix airport – at any given moment , there are three people shouting complex orders at you . You need to focus on one , and tune out the rest temporarily , and when done – move on to the next order, rinse and repeat . If you are really good at what you do – you might even be able to take the second order roughly in parallel without making the customer repeat the instructions .
Given how quickly demand can change – getting to an answer usually needs a dialog between two or more parties . If my daughter and I are ordering together our favorite green tea frapuccino  , one of us typically will change our mind at the last minute that will need the barrista to engage in a conversation with us (or at least listen to both of us and ask clarifying questions ). This is especially true when my thick Indian accent and my long name makes it harder for them . 

Perhaps the most amazing part of how we solve problems is our ability to learn and avoid reinventing the wheel . Over time , Phoenix airport barrista has figured out my preferences and only asks me “capuccino or frapuccino today?”. She knows by now that I will always take an extra espresso shot in mornings and that I don’t like whip cream in my frapuccino in evenings . 

Finally , after we take a decision – we are usually able to explain how we arrived at the conclusion we made  . And such explanation will need varying  amount of detail depending on who you are dealing with ( this is good for you is fine with my daughter , but a longer winded explanation on why I chose the frapuccino is often needed with my wife) .

While it all sounds rather complex – we do this everyday as human beings with no problem at all . Human brain is amazing – way more amazing than any computer . Trying to mimic human thought processes  with computers is incredibly complex on many fronts – and that is essentially what cognitive computing tries to do . It tries to solve problems like how we do as human beings – except at a larger scale , using more data and faster . 

Most computer programs are built on deterministic (fancy for if-then-else) logic . That is a piece of cake for computers . Cognitive computing can of course reuse deterministic logic , but it primarily needs probabilistic logic ( option A has higher odds than option B type logic ) . Often times the computer might not know the exact answer at all -and then it perhaps needs to figure out what option has lowest risk . If a fellow human being can barely understand my accent, imagine how difficult it will be for a computer to understand the words that are coming out of my mouth ! 

Now the good news is that the smart women and men working in this field have already done quite a bit of work on making computing. More cognitive . While there are many more miles to go , what is already there can bring immediate value to customers . Siri for example is a starting point on humans talking to computers for simple questions and answers . Siri has limitations – like keeping the context between successive questions asked by a user . IBM Watson can keep context across questions and can have a more meaningful dialog with a human user. But in its current form , it will be hard for Watson to tune out noise (say my wife and I both ask a question each in parallel). But over time ( and I am positive it won’t take that long)  – those challenges will go away and technology will get more and more savvy at making computers closer to human way of thinking and solving problems .

While I am all excited about the idea of cognitive computing as the future of this industry  ( well, and present too actually given we do multiple projects already using cognitive computing ) – I don’t think it has the potential to replace human information workers in huge numbers . I will explain my rationale in another post . 

Big Data : it’s about time we stopped putting the cart before the horse 


  
It rained heavily in Chandler yesterday and our front yard looked a whole lot nicer than the usual dusty appearance . That somehow made me think of the new look and feel of the world of data I live in as a professional .

For those of us who grew up implementing big data warehousing projects , it should not be a shock really to look back and figure out that most DW projects started because business had an analytics problem , but in the end 90% of effort was spent on the plumbing – the management of data (ETL, data modeling and so on) and only 10% on actual analytics (or even just basic reporting in many cases)  . 

This is true not just in design and build – it’s the case with supporting and maintaining the Data warehouse too . Companies have spent countless dollars on DW implementations and no one is truly happy about it . Yet, no one I know has any plans of fully replacing their DW implementations either (which of course is the right thing to do ).

Along came “big data” promising to make life better for everyone and setting very high expectations . Vast majority of customer executives that I speak to think of big data as an answer to their analytics solutions . Even amongst the CIO community , very few realize that most of the conversation they have heard is about the data management aspects ( 3V model is familiar to everyone and it’s about data management , not analytics). So in the past few years , I have seen several of my clients jump into big data initiatives to accelerate the realization of their analytics needs . 

The fall from grace is rather rapid – mostly because of unrealistic expectations . To begin with the minimum requirement for big data projects in many cases is to meet the SLAs of their existing data warehouses and data marts . It doesn’t take too long to realize that ain’t gonna happen . 

Then comes the dejavous realization that big data projects also need most of the time spent in ETL just like data warehouses did in past . Usually this leads to a quick reduction in scope of the projects – usually by eliminating some sources of data that are more complex or less clean , and of course this means analytics is compromised too .

Finally the reality of “data lakes need a lot of curation” kicks in . No company has enough man power to curate all the data that it needs for analysis . And at some point , the data lake just becomes a data dump with the idea that “curation can wait while we figure out what we need to analyze”. That is rarely practical – data scientists won’t always know the context of the data unless an expert curated it beforehand . And the world doesn’t have enough data scientists today to make them do data cleansing for most of their time .

Till such time as AI/Cognitive capabilities take the stress of curation away , I think analytics will continue to get short changed and the promise of big data ( and specifically data lakes) giving powerful analytics for busines users will not exactly work as advertised . 

It’s not all gloomy though . Customers who start small with well defined analytics requirements have already started realizing benefits from their big data investments . They don’t take a “build, and they will come” approach . They just build intelligently as requirements come up and plan to have more comprehensive solutions down the road . They value business flexibility and agility over technical elegance . Many of them have taken the time to formulate a strategy and a roadmap on what they want to do – leading with analytics that satisfy specific business requirements and working back to data management , and not the other way around . 

Of course we need both – but It’s time we put the horse (analytics) in front of the cart (data management).