Is data science doomed with Trump being elected ?

Ever since Trump won the election , the question I have faced the most from family and friends is “is predictive analytics dead?”. I also got asked if Watson would have picked the correct winner . The more savvy doubts were about how Clinton missed the trends in places like Wisconsin and Michigan .

Here are my thoughts – and pls treat them as my personal opinions only as always !
To begin with – the analytics was not all wrong , and did many things right . It also did many things wrong . Rather than saying data science  is dead , I think all it really is that it’s cloudy and some work needs to be done to make it less cloudy . 
The thing we forget the most about data science is that it is all about odds . When Nate Silver said Trump had 35% chance of winning – he meant exactly that ! Having about 2/3 chance of winning for Clinton should not have been interpreted as Clinton will win ! This problem is one I face every day with my clients too on all kinds of predictive scenarios . It’s not a binary thing as we like it to be in most cases .

That said , the predictive models all had given significant odds for Clinton and now we know something was wrong with them . So yes – data science on politics should absolutely take some significant blame for what they missed . 

To begin with – All analytics about people are hard . I wrote about it few weeks ago here . 

Models are based on history and assumptions to give them context . It’s not uncommon in this business for calibration to go out of whack  – usually because context changes , but the model continues to depend on old assumptions . Since all public analysis of this election trended the same way – I guess we can safely say that “establishment thinking” about polls needs an overhaul . 

Then there is the actual data itself that comes from polls and the bias ( like selection bias , confirmation bias etc ) that gets associated with it . I often post twitter polls to get a pulse on topics I care about – and I should know about the selection bias when I look at the results . People who collected and analyzed the data should have been way more careful about bias . 

Pollsters need to know the markets they are polling . Respondents don’t always literally say what they mean . This is nothing new – any kind of market research would have run into this scenario and there are ways to get around it . When I have done collection and analysis about foreign markets using folks who are technical experts , but largely ignorant of those markets – I have always had poor results . I have a feeling that a lot of polling was “lazy” this time around in election season . For example – if your call list only has landline numbers , you won’t know what I have to say ( I haven’t had a land line for quite some time and I am hardly alone in that ). 

Weather forecasting is something we are all familiar with since it’s been around for a long time . However , our ability to accurately predict beyond the next week or ten days is actually not that high . Little events can change weather big time.  If we extend that thought to how the sex tapes and FBI actions all came back to back – we probably can have some sympathy for the statisticians who had to deal with the data . 

Even if all the models worked well , late happening events – like FBI director’s two notes to Congress – don’t leave a lot of room to actually act on what the model tells you . We were recently working on predictive maintenance solution at a client . The maintenance VP was very clear that if all I can give him is a 2 day window with failure prediction  , there isn’t a whole lot he can do to avoid down time . While I don’t know for sure – I wouldn’t mind making a small bet that analytics used by Clinton campaign probably highlighted the issues of Michigan and Wisconsin , just that it was too late to do anything about it . 

I am sure I am missing several other aspects – and some technical aspects are probably too boring for most of my usual readers – but I think I have given a fair idea of the thoughts I have on this topic . I am sure you will add more , or correct me in the comments . 

Some changes in the polling and predictions industry is needed , but we just need to try to NOT throw the baby with the bath water . And while I am the biggest fan of Watson , I don’t really know for sure if it would have done better . Knowing what went wrong this time – I am sure this industry will use it to its advantage and reclaim its position quite quickly . 

Parting thought – for all my pals who think AI will take over the world soon , this might be worth noting that for foreseeable future these models will need significant human help to be useful . It’s man AND machine , and we should stop obsessing about man VS machine . 

Few more post election thoughts 

In no particular order …
1. Peaceful protests are legal and a part of our culture . It can’t be denied and it’s a much needed venting mechanism . It might also give the young people a life lesson on how to be active about things they should care about , including voting 
2. Trump won fair and square . Clinton won popular vote by just 0.2% and there is no trophy for that . Saying “Trump didn’t win, she lost” doesn’t change the truth that he won by a bigger margin than most of us ever expected .
3. Trump is the president elect now . There is only one President for the whole country . There is no such thing as “he is not my president”. Calling him names is just an insult to the office he got elected to . If we care about democracy , we should give the guy a chance unlike how GOP obstructed Obama throughout . Nobody wins if we keep obstructing for the sake of obstructing ! 
4. If you don’t like Trump or GOP , work at grass root level to get democrats to power – both in legislature as well as Whitehouse next time . If he does something crazy – let’s protest it by all means. Same for GOP – listen to people and inspire them , pls don’t resort to fear mongering as primary strategy 
5. If we consider the global economy , every thing we try to get to a stalemate is an opportunity we are giving China and others to gain an upper hand . 
6. If you really want to make a long term difference and don’t want a repeat of Trump vs Clinton type for other elections – invest in education today so that a more intelligent generation follows ours . 
7. Lets give leadership opportunities to the next generation of leaders on both parties . Let’s change what “establishment” means by driving change at all levels . Let’s bring the focus to “people” ! 
8. Social media is fun , but not representative of the nation . Let’a please not forget that many of us live in a bubble we create by selection bias 

The sun rose today, and it will rise tomorrow as well 

Like most of you , I too was up till 2 AM following the election results and experiencing it’s associated mood swings . I certainly was not a Trump supporter and didn’t expect him to win . I don’t like Clinton either – just for different reasons . Between the two , I was hoping for Clinton to win . I went through publicly available polling data over last few days and concluded , semi scientifically , that Trump at best only had a 30% or so chance to win . I was wrong – dead wrong – and so were a lot of others that I know .

I am middle of the road on my political leanings – fiscally conservative and socially liberal . I also firmly believe that in a democracy , I can’t pick and choose and apply random criteria . People of this country voted and Trump won fair and square . He now has my respect as the President , irrespective of whether I like him or not as a person . It’s difficult , but I am optimistic that it will be fine . The sun rose today , and it will rise tomorrow as well .

But there are important learnings for all of us here , well at least for me for sure 🙂

1. Trump listened to the pain of the people – and Clinton and many of us did not get it . He appealed to the insecurities ( perceived or real ) and got rewarded . Rest of us didn’t – and he won . If you miss the pulse of the majority of  people you deserve to lose . Few months ago I. Atlanta – I had an African American uber driver who told me he will vote for Trump because he is against gay marriage . On a national scale , I didn’t realize how many such people exist and how ignorant we are of the “silent trump supporters” .  Social media created a bubble for many of us – real world is still far removed from social media and that is something I will never forget ever again !

2. Campaigning and governing are not the same . When you campaign , essentially you are pleading  to the majority as defined by electoral college votes . When governing you are everyone’s president and you can’t just think of the voters who took your side . It remains to be seen if Trump has any real solutions to back up his rhetoric of making America great again. I am not a fan of Obamacare – but I am not sure if he has. Ought through sufficiently on what to replace it with . Last thing we need is it to be repealed and replaced with something that doesn’t work .

3. Big solutions might be out of the box in nature . Building the wall , deporting illegals etc are not pragmatic . But then I didn’t see Narendra Modi making 1000 and 500 rupees notes illegal over night to curb black money either . Does Trump have such tricks up his sleeve ? I haven’t seen clues – but I hope he does have great big plans that will pleasantly surprise us .

4. House , Senate and President are all going to be from the same party now . Checks and balances were built into the constitution for a good reason . When democrats had all three , they passed Obamacare that was poorly written and poorly implemented . I am waiting to see if GOP will do any better when they own all parts of government

5. Trump is a business man and I am sure he understands the difference between marketing and execution . I have a lot of Muslims , LGBTs, minorities  and immigrants as my friends and I am super worried for their rights . I hope and pray their rights are not infringed like he told us in his campaign speeches . I don’t honestly think – despite my own multiple jabs – that Old Testament will replace the constitution of this country . Over time I fully expect Trump and the legislature to win our confidence and trust .

6. We should worry about each other more than about the president honestly . A nasty Election season has driven big wedges amongst us . Clinton and Trump were rich and famous and will continue to be so , and will both shake hands and move on . I worry about the rest of us being able to forgive and forget . A good start will be to stop shaming each other from today on how we voted . It’s over , and there are miles to go before we sleep !

Role Of Services In Artifical Intelligence

Please don’t interpret the following as IBM’s position – this is just my view . Also, after a long time, I am typing a blog post on my mac instead of on my iPhone 🙂

I am not an official spokesperson for IBM . That said, I am not a spectator from the peanut gallery either. Almost every day for last couple of years, I have been involved in the sales and delivery of projects that include some flavor of Artificial intelligence. I am sticking with AI here instead of “Cognitive” to avoid the distraction of what is AI, ML , Cognitive etc . While I do not have an academic degree in AI, I have hands on design and coding experience in this domain, and I am learning more every opportunity I get.

Couple of weeks ago, I was in Vegas for a week – attending IBM’s World of Watson event. I spoke with tens of clients and prospects, several analysts and a bunch of my colleagues from other parts of the world that I don’t get to interact frequently. One of the common themes in this conversation was the role of services in AI. Variants of this topic also caught my attention on social media as well – mostly from analysts. The big difference between the people I spoke with was that many analysts were dismissive or pessimistic about the role of services, and clients did not seem to care what was product and what was services – they did not seem to care much about the distinction. Either way, this topic picked my interest bigly 🙂

There are many reasons to be pessimistic about the role of services in AI

  1. Things that caught the imagination in past like ERP, data warehousing etc all had big services components and many projects were plagued with cost over runs. Why repeat the mistake with AI ?
  2. Most AI initiatives – from IBM, Google etc – are cloud based. Cloud should not need as much services, right ?
  3. AI in general comes with a cloud of uncertainty about the future of mankind. Almost every person I know has a strong position on Universal Basic Income thanks to this fear. Why accelerate the apocalypse by promoting services on AI?
  4. AI is in its nascent stage. Services is what hides the immaturity of the product and hence deserves a public take down. In any case, it is more cool to be on the side of product and be opposed to services.

If you are running out of patience already on what my take is – I believe services will be integral to the present and future of AI, for a long time . Here is why ( and its long – get a beer or coffee ).

The nirvana state of AI for me is when computers can mimic ( and accelerate) human thoughts instead of just human tasks. We are a long way away from that stage. But what is possible today is already good enough to help individuals and businesses in tangible ways – often with “order of magnitude” benefits. People who criticize AI as immature often do not recognize that many companies use it heavily today, and its a part of many apps on your phone. The future is already here, and its getting more evenly distributed as well 🙂

A lot of people think – and I get asked almost every day – that systems like IBM Watson are big monoliths like traditional ERP systems. While the start was certainly that way , AI today is largely a set of APIs that application developers can mix and match for their unique needs. These are quite well defined APIs and even someone like me who is not a full time developer can figure out how to use it pretty quickly.

One such API is speech to text . You input the audio, and you get text as output – that is pretty much it. There is no end to the useful scenarios this “simple” API can be used in ( closed captioning in TV for example). However, there are very few people relatively who know such an API exists, and even fewer who know how to integrate into existing workflows in their business to improve efficiency and effectiveness of the process. Consultants can help their clients explore how AI can be incorporated into their existing processes. This kind of integration work is perhaps the most tactical use of AI services today.

While tactical uses are awesome, most companies have little interest in things that make incremental changes. I have a client who is in the entertainment business. The leader in their industry is about 4 times bigger than them. They are betting on AI as the way to leapfrog the leader and become the top dog in their industry. A good part of that project is the strategic planning – what bets to make, what trade offs to consider, how to mitigate risks etc. AI Advisory is probably the most exciting part of the future of consulting business. It needs a unique mix of AI knowledge , the intimate knowledge of specific industries, and business process expertise. In other words – product is not the lone differentiator.

I explain this to my clients as “It is not what we make with AI, it is what we make possible with AI. We will help you build the bridge from where you are to where you need to be” . 


There are some “tangential” scenarios too for AI in the Enterprise IT world. Between ERP and Data warehousing, countless days of my life was spent in getting data cleansed and transformed. For the most part, even with SaaS and big data – we worry a lot about GIGO ( garbage in garbage out). AI will help us take a fresh look at this problem. What we call as “garbage” data might contain many useful nuggets. We might not need to transform it row by row to get meaningful insights – AI can figure out useful patterns. Coupled with the modern big data management systems, this could make a lot of ETL disappear for scenarios where directionally correct insights are more important than precise information. It needs services – the kind that generates actual value and gets rid of a lot of terribly inefficient code. Actually not just code, also the precious human effort wasted in fixing the “garbage” data !

This post is already becoming quite long, so I will stop with making just one more topic – a favorite for both my clients and my analyst friends. Companies around the world have invested heavily in ERP systems ( and satellites for SRM, CRM etc). Those assets have not all been sweated sufficiently quite yet. Some of these vendors have modernized ( mostly about moving to cloud, and making the systems faster ). However, in the promise of “better, faster, cheaper” – the “better” part is largely unfulfilled today. I firmly believe the missing ingredient is purposeful use of AI.

Lets take the mundane case of managing collections – where optimization today means you outsource it and make it someone else’s problem. Cost of collection is not trivial for most clients I have met – but they have mostly just accepted it as a fair cost of doing business. Efficiency is improved by mostly fine tuning the process  part of it – like how many dunning cycles need to be run, but with very little personalization. We recently implemented some elementary AI ( trade off analytics, speech to text , text to speech etc) for a client who is a giant in the world of education ( took less than a month to implement it) and the client saw 20% better collections immediately after that.

Optimizing collections is really a trivial use case (and even that has tremendous ROI)  if we look at the power of AI available today. Exponentially better implementations will be possible (and already in the works) when we combine AI with other emerging stuff like say blockchain .

While I have no doubts that services will play a major role in the field of AI – I will be the first to admit that practitioners have a lot of learning and retooling to do to add value to their clients. The good consultants have always been life long learners – what is perhaps different for them is just the speed at which they need to learn now. Its a fast evolving field and there is just no time to sit back and say “I am an expert” any more.

PS: If working in AI, integrating cognitive capabilities to existing systems, Advising clients on making use of AI etc are things that you are good at, and you promise to keep learning – I am hiring. Ping me at Vijay dot Vijayasankar at US dot IBM dot COM .