I have a whole new appreciation of Business Intelligence now

As many of you know, I grew up as a BI and Data warehousing guy – I have implemented BI for a lot of users across the world . I have collected requirements from shipping clerks as well as CXOs , and in at least a dozen different languages. I have spent countless hours thinking and rethinking data models, how best to transform data and how best to present data to my users.

Along the way, I became a manager and then an executive, and thus became an active consumer of BI myself. But in the big companies like IBM and SAP that I worked at – I learned to live with someone else’s BI design. As far as I can remember, I never had to create significant new requirements . These were rather stable businesses that could be run with minor tweaks to existing BI capabilities. 

And then few months ago, I joined MongoDB and my whole perspective on BI changed. We are for the most part still a startup. We don’t have a huge IT arm that can cater to endless requirements from me and other leaders of the business. Our IT landscape is almost completely SaaS based. If we can hire one more person – we would rather hire to fill a front line technical role to make the product and customer experience better,  or a sales or channel type role to make the business better. For foreseeable future, I don’t expect that to change either.  

We are a global business – and we are growing incredibly fast. And to keep that pace – we need good data, especially when it comes to customer facing business whether it is direct sales or channels (which I run) . Having grown up as a programmer and then a BI guy after that, I have a great affinity for making decisions by numbers. With the speed at which we grow and our lean policies, I don’t really have a lot of time to wait for information to see how things are going – which essentially means I need good quality operational BI at all times. 

We use Salesforce.com for our CRM. I am a first time user of this solution – which might surprise a lot of people. My past experience with CRM has all been in Seibel and SAP CRM. The best part of salesforce.com was that I did not need any training to use it – none at all. My past experience was almost immediately transferable to use the system as a non-expert. My primary use is not as a transactional user who creates or updates opportunities etc. My main goal is that of gaining quick visibility into the aggregate opportunity to order process for channel business, with the ability to drill down into details as needed.

Once I got settled in my new role, and got to know my team better – my immediate priority was to get a full view of the global business. I mocked up initial requirements into spreadsheets and discussed it on phone with my partner manager Guillaume in Dublin, who is an experienced salesforce user. From my past consultant life – I estimated the effort required as a few months in the technologies I grew up with (assuming I got the most skilled people I could find). Next day morning, I saw Guillaume already had 3 dashboards ready for me which showed most of the information I needed. And then in 2 more daily scrums – I had the 6 dashboards I needed to view the business from every dimension I care about. That is much less than the time it would have taken me to write a proposal for a customer for this work in my past life. 

What did I learn from this experience ?

A lot of good things for sure

1. Business users like Guillaume (and Luca, his boss who runs channels in EMEA for MongoDB) are better BI consultants than anyone I could have ever hired from outside. He not only knew the technology well, he knew my business well and could challenge my assumptions and give me new ideas. It has convinced me that rest of my team including me should step up our skills in salesforce.

2. The technology to build operational reports should be extremely simple so that business teams can iterate quickly. Till I saw it with my own eyes, I did not believe that it could be this easy.

3. From prior life as a programmer and a BI guy, I am well aware of the limitations in reporting – so I can minimize the churn in requirements gathering and make good compromises on what needs to be measured.

4. The simplicity of reports and the report writing technology – and my big time aversion to any transformations (having seen how data loses meaning way too many times) – helps us stay nimble and make changes on the fly. 

There are also some areas of improvement of technology , which I am sure Alex Dayon and team will fix at some point, hopefully soon .

1. Charting and visualization is very limited – so when multiple graphs are put next to each other it is quite a strain to discern information quickly. Granted, the ease of changing things on the fly is more important to me than flashy reports.

2. Only 20 controls possible in a dashboard. I can compromise on it for now, but as business grows – this is a pretty serious limitation for me to get a global view across everything I need to monitor and act on. 

3. Reporting across objects looks limited – but this could just be my lack of experience.

4. Operational reporting does not replace the need for a data warehouse . I still need some other place to combine the lead to order process with information from from Finance, HR etc. For my current purposes, I have work arounds – but if all the SaaS vendors for CRM, Finance, HR co-operated and built a BI solution to seamlessly provide me with integrated data – I will swipe my credit card happily to buy it. 


What Does BI Mean To You?

Totally an unstructured rant – be very warned ! 🙂

Over the last few weeks , a lot of discussions happened on BI between me and some friends . Essentially – why is it that BI projects seem to stay the same way it was always done even while tools became a lot better ?

To begin with – tools have become better , but not to the level where a user can use it in self service fashion in most cases . So, education is probably the number one issue in the context of BI. Education comes in many flavors – and the easiest one in my mind for a user is the actual tool training . So lets move past that.

There are a few things that have not changed from a BI project POV in several years

1. Users do not understand the data available to them

This needs a lot of bottom up and top down effort to fix . I think top down governance will hit a wall sooner than most of us estimate . Data needs to be explained for each user with real life examples, but that approach is hardly scalable.

BI project plans need to budget time and resources to fix this – but having seen no change in many years , I guess the inertia is just too much

2. Users do not understand what reports are telling them , and most of them have limitations in explaining requirements to IT or consultants

I have done elementary statistics type classes to explain data to users in some past projects . I have a feeling this needs a deeper look to see if such an approach needs to be standard in BI project life cycle.

Users have unrealistic expectations of BI in many cases. This movie is now playing again with predictive Analytics . I see a lot of users and consultants having limited idea of what a predictive model is telling them and what it’s limitations are . No wonder several of them crash and burn .

3. IT and consultants takes a one size fits all approach for BI , and spends little effort in acknowledging BI is personal , not generic

This won’t change till BI is no longer treated as a side kick for ERP . People personalize ERP quite a bit ( at least compared to BI) , and that rigor needs to get into more BI projects .

There are a dozen more things , but I feel better already after typing 3 to get it off my chest

Sorry – had to rant . Will try to not do this too often .

Big Data Solutions – Do Questions Matter ?

I have Ray Wang to thank for this post. Off late, I have a serious problem of writers block. I just cant find a topic interesting enough to write about, and consequently have become a ratherirregular blogger – at least compared to last year. Any way – back to the topic of this post.

Ray tweeted this few minutes ago

A lot of BI blueprinting sessions from my consulting career flashed through my mind when I saw that. A key principle for a good BI system design is in finding out upfront most of the questions a user would ask the system, and then designing a solution around that. Unfortunately this is a blessing and a curse – while we can really optimize getting fast and accurate responses to predefined questions , this also curtails our ability to change our mind and ask different questions. More experienced BI experts will second guess other questions that users “may” ask and leave some room in design to cater for that, but it is clearly not a scalable way to do things.

Somehow, users were also trained along the way to agree to some lack of flexibility in BI systems. While the complaints never went away fully, most users think by now that it is normal for BI team to ask for some time to change the datamodels and create new reports and so on. It is a sort of “marriage of convenience” if you will – with tradeoffs understood by both sides.

So when we let go of “ordinary” data and embrace “big” data – what should change? I think we should use the big data momentum to make BI systems more intelligent than the rudimentary things it is capable of doing today. And this intelligence should be done with some business savvy. In other words both “B” and “I” of BI need some serious tweaking.

In my opinion, what should change right away is the expectation of business users needing to state most of their potential questions upfront at design time of the system . Or more clearly – the expectation should be significantly lowered, and business users should be allowed to ask more ad-hoc questions than they have done so far. Of course we can never guarantee full flexibility – so some subjectivity is necessary on where we draw the line. Just that the line should be drawn musch farther from where it is drawn today.

Accuracy of result for ad-hoc questions is not enough – the results should come back in a predictable and short time frame too. Ideally, all questions should come back with answers ( or a heads up to user that this is going to take longer ) within a predefined timeframe (say like 3 to 5 seconds or less).

Then there is the question of how the users ask these questions. SQL or NoSQL – querying languages do not provide democratic access to data. People should be allowed to ask questions in English ( or whatever language they use for business ). Some training might be needed for the system and for the users to understand the restrictions – but no user should be constrained with the need to know how things work behind the scenes. A minority of people should have the skills to educate the computer – the rest of us should not be burdened with that. Instead, the computers should be smart enough to tell them answers to what questions users ask.

There are very seldom exact answers to questions in business ( or life) – even apparently simple questions like “what is my margin in North America ? ” is ambiguous to answer. Most clients I have had have many different meanings to “margin” and “North America” and “My” within their organization. In real life, if these questions are asked of a human analyst, she will ask follow up questions to you to clarify and then provide an answer with necessary caveats. Why can’t systems do that? Wouldn’t life of users be vastly improved if systems answered problems like humans did, in a way humans understand? of course with more speed than humans 🙂

Big data or otherwise, there is always an issue of trust in the data from user’s perspective. Most analysts spend nearly as much time explaining how they arrived at their results, as they take for compiling and analyzing the data. The system goes through all the computation any way – even today in the non big data world. Why can’t our BI systems explain to the user how it arrived at the result all the way from source to target or backwards? Wouldn’t that increase productivity a lot?

When users ask questions – they usually will also combine it with external data (google, spreadsheets etc) before they take a final decision. Would it be possible for a BI system to present some useful contextual data to the questions from internet and intranet and allow the user to choose/combine what he needs?

And one last thing – if the system is intelligent enough to find answers, why can’t it have the smarts to also figure out the best possible presentation for the results? Today – we mostly have to predefine how output looks like visually. Why put that load on users? Can’t systems be smart enough to look at the question and the answers and figure out the best way to represent it to the user? This is not a “big data” problem – this should have been the case all along, but somehow never quite happened in a mainstream kind of way.

This is by no means an exhaustive list – I left out plenty of things like collaboration, predictive responses, closed loop BI and so on. I didn’t do so because they are unimportant, but only because of the boredom factor. These types of things are already happening to some extent, and hopefully will catch on more as time progresses.

So there you have it – its my birthday wishlist. And thanks again Ray for that much needed spark to blog again 🙂

The Second Question

As many of you have pointed out in private and in public – I have not blogged much since I have joined SAP . Trust me it was not intentional – just that other things like kitchen remodeling , getting used to my new job etc took all my bandwidth. But now that remodeling project is complete , I can get back to more blogging etc .

Any way – my first assignment at SAP is to work with our customers and partners to make sure they realize the value of BW on Hana . Since the day I joined SAP – I have been talking to customers about this, and something interesting became clear to me. So I thought it might be useful to summarize my customer and partner conversations in my blog so that we can have a more extensive conversation virtually .

When Suite runs on Hana , BW runs on Hana and assorted data marts run on Hana – what would be different for a business user ? In my opinion – after talking to several customers – it is the “ease of answering the second question” that is the most value adding scenario for a business user – especially the “business analyst” types.

Let me explain with an example that should resonate with many of the readers here

Analysts live in an ad-hoc world – their “real” work starts when an executive asks them a non routine question like “how many bottles of soda did we sell to top 5 distributors in Arizona last summer and how did we do against plan” . Question is simple enough – but there might not be an easy way to answer this .

Analysts will probably go to BW or CRM to find who their top 5 distributors are , how much was sold to them etc. and then they will probably log on elsewhere to find plan information . Finally, all of that gets dumped to an excel sheet , massaged with various vlookup functions etc and a prettied up table and graph will be presented to the executive . Most probably , one of the sources of information is yet another spreadsheet stored in a share point site.

Now, I have never met an executive who had asked all she wants to know in one question – and that includes me !

So as soon as the first answer comes in – the exec would ask her next question . “Hmm – that is interesting , how is that split across the various brands we sold them ?” .

The best analysts know this and will come with as much info that they can second guess. But that is a limited approach with most executives – invariably , more data dumps and analysis will be needed . And this takes time – hours at a minimum, days to weeks usually .

Now, what would change with Hana ?

For starters – a lot of data sitting in BW can be crunched and compared to spreadsheet data on the fly by the analyst without IT help using workspaces . This can be done without Hana too – if speed is not an issue and patience is over abundant .

BW sitting on Hana can be combined easily with other datamarts modeled directly on Hana via composite providers. So – adhoc queries spanning multiple sources become all the more easy . And of course the front end like BI 4.x Analysis makes this an excel friendly exercise .

Now when suite also works on Hana – and suite has the Hana Analytics Foundation under it – this becomes all the more easy . SHAF is nothing but Hana views that can also understand BW data . And of course it understands BI front end tools . So in effect – data sitting in suite, BW and other Hana datamarts are all available to users without a lot of manual work to make sense of it all .

Back to our analyst friend who got the second question – now he can quickly change the query to include more parameters and respond to the executive much faster than in the pre-Hana world . What is not to like ? 🙂

Of course , Hana does not replace the need to have a good solid BI discipline in place . The right way to look at this in my opinion is to think of how powerful is the scenario of having Hana and a good BI system together than just one or the other (or god forbid, neither).

It is also important to note that suite on hana does not eliminate the need for BW. Neither is it always a good idea to move everything in a BW system to a custom data warehouse even if it sits on Hana . If you are not convinced , try to implement a reporting scenario that is based on an ERP cost center hierarchy with time dependencies directly in Hana and also in BW on Hana . You can see why having these solutions to compliment each other is better than trying to force fit every requirement into one of them .

Now tell me – what is your “second question” I can help with 🙂

Strategy Based On Data – Will What Worked For The Campaign Work For Governance Too?

So President Obama won a second term. Congratulations to POTUS and all my friends who supported him.  And I do feel bad for several friends who genuinely feel bad that Romney did not win . I didn’t think either candidate had it in them to help this country prosper.  The ideal candidate in my mind needed more Obama like stances on social issues, and a mix of Obama and Romney stances on economic issues.  But that is not going to happen any time soon I guess.

What fascinated me most in this election was the use of quantitative methods in Obama campaign. At a high level, there are 3 things campaigns need to know

1. Who to target to secure enough votes  ?

2. Who to target to get funding  ?

3. What to tell these 2 groups, and using which media ?

And one way or other, the Obama campaign seem to have succeeded on all three, and hence Obama gets to be President for 4 more years. So far so good – but that is not the part that fascinates me.

Now, with the elections behind them – will the administration still invest in big data analytics to govern better? That is the possibility that fascinates me. I sure hope the answer is YES (Not YES WE CAN, but YES WE WILL 🙂 )

Now, the administration does not create laws – Congress does. And given the control of house and senate are not changing according to latest poll results- the chance of a different type of law making is pretty low if nothing changes. But things could change – and if all arms of government can use a little more of decisions based on data, and a little less of “political gut” based decisions – maybe this country can start prospering again. The type of analytics needed for policy making is not exactly the same as the narrow objectives of campaigning – but I would argue that similar principles apply. And just like in enterprise software projects, I suspect that the hard work is less of tech work, its probably change management.

Second presidential terms are all about leaving a legacy, I am told – I would love to remember Obama as the president who used data effectively to win 2 elections, and then used data to effectively govern this great country. Probably wishful thinking on my part – but hey, I make a living doing BI work for my clients, AND I am jet lagged as I am typing this at 2 AM 🙂

I am also rather curious now on whether Indian politicians will use big data analytics in their campaigns. If they do, India might see some interesting times.

How Many Communities Do We Really Need ?

If you do a search in internet for SAP communities, I am sure you will get a lot of hits. If you take the time to dig deeper, you will probably find significant overlap in their content and membership too. There will always be a loyalty/intertia factor that inhibits widespread adoption of a new community. Every time SAP acquires a company, this problem of integrating communities come up again. In short, I doubt the world is short on online communities.

Just to avoid annoying close friends – let me use a more neutral example. We saw Google+ arrive some time ago. Almost everyone I know who was active in twitter, jumped to G+. There were plenty of debates on whether twitter will die, and all action will be in G+. End of the day, they both survived, and peeps on twitter I followed are still more active on twitter than G+ by orders of magnitude. Sure there are nuances and all – but in general, world did not become a better place in a hurry because of G+ in my opinion. I wrote about it sometime ago too https://andvijaysays.wordpress.com/2011/07/24/google-plus-tribes-wont-follow-till-chiefs-jump-in-with-both-feet

Yesterday afternoon, I caught up a little with twitterverse. There was an interesting conversation going on there with my friends at SAP and my fellow SAP mentor Mico Yuk. The general topic was whether SCN was a comprehensive community.

Mico raised two points on SCN – apparently based on what she heard from her customers .

1. SCN is way too technical, and hence not a good place for business people to visit. So for people who sign the BI checks, as she put it, SCN does not fit the bill

2. The top ten contributors in SCN are “no-names” in BI.

Now, Mico holds strong views about SAP and its ecosystem, as I do too. I respect her for that. In this case, I have a slightly different view point.

It is true that SCN primarily has technical content. With great fanfare, SAP brought in a community for Business Process Experts (BPX) few years ago. I don’t know if it is alive or dead now, but in my mind it did not catch the imagination of majority of the community. May be SAP thinks other wise, and that is fine with me. I am just expressing my personal views here.

So do we need a separate community for Business ? In this context – I have some opinions on the Business/IT divide. http://https://andvijaysays.wordpress.com/2012/09/10/on-business-it-and-artificial-distinctions/

I do agree fully with Mico that business typically will tend to tune out of topics they don’t find an interest in. But that does not necessarily mean SCN does not have a place in the scheme of things, and that we need another community.

The way I have seen purchases happen in BI are in 2 ways – a “shadow IT” purchase by a department within a company (typically CFO office) , or an IT purchase by CIO organization. In shadow IT purchases, they typically depend on vendor references, a couple of analyst studies and probably google searches. The CIO purchases typically include more formal RFP type processes, and might include a buyer’s agent if it is a big decision. Of course this is a generalization.

In either case, the actual buyer in most cases may not get a lot of direct online community information. But he/she will be heavily influenced – on purpose – by people actively listening and/or participating in the online BI community. And that is where a community like SCN comes in. I have seen multiple business buyers resist the temptation of buying new departmental solutions, after someone in their organization points them to content in online communities that sheds less than favorable light on the solution. SCN does not get the credit it deserves due to many reasons – but it is still vastly influential amongst people who give their input to the ultimate buyers of BI software and consulting services.

There is no hard and fast distinction of business vs IT content , as it applies to a buying decisions, in my opinion. Things like security and scalability are important to both sides, but probably need to be expressed in different language for each audience. I would much rather have one community with content for everyone, and then strong filtering and search capabilities built in. Just as the world of canned BI reports and dashboards is shrinking, so is the world of content management changing. One of my biggest criticism about HANA content was that it was split between SCN and experience HANA website. I hear there is some integration happening now , but in general it did not hurt me much as long as I used Google as my starting point, instead of an SAP site.

Next , about the “no names” comment about top 10 BI contributors. If some customer of mine said that, I will go out and try to correct them. Everyone has an opinion, but this by no means makes it fair to criticize the top 10 contributors. They don’t always serve a technical purpose. I will take 2 names from mentor community in that list as an example. Ingo Hilgefort is an SAP employee who delivers excellent content on BOBJ. His audience at teched etc is technical. But I also know first hand that he deals with business users all the time at my clients, and gives them sound advice. Similarly, there is Tammy Powlas. She is an SAP customer. She is a total god-sent to BI community, and SAP should be sending her flowers and chocolates all the time in my opinion. It is not always that a customer takes time to post content on a vendor’s product. If I am a buyer, I would give a lot of weightage to her opinion, than content coming from an SAP Partner ( like yours truly). So unless I am mistaken on what was the rationale for the customer to make the no-names comment, my view is that that was an unfair assessment.

The trouble with twitter is that the 140 character limit does not allow meaningful conversations back and forth all the time. So my plan is to see if I can get Mico and Mark Finnern together for a few minutes at Teched next week and get a bitter grip on this discussion. Maybe some other mentors and community members including customers will join us? I know the community clubhouse usually has great coffee to go with such discussions .



Is There Anything Left To Be Said About SAP HANA – Yeah, There Is

I have written a lot about HANA, and have participated in several podcasts, webinars and so on about HANA. And I am probably the least prolific person around on the topic. SAP and its influencers have all done a lot to spread the word to the extent that I was joking on twitter recently that I wonder if there is anything left to be said about HANA.

I guess there is – about what is going to happen next about HANA.

Apart from the stand alone HANA option and BW on HANA that are both in GA, a lot of things are in the pipeline from what we have heard from SAP – ECC on HANA, CRM on HANA, and every other business application from SAP on HANA, the HANA App cloud and so on. There is also a lot of activity going on about HANA start ups. We know this – but what remains as a question is what will SAP prioritize when it comes to HANA ?

We have heard Vishal Sikka state that ECC running on HANA in some shape will come out in Q4 this year. I know it is a topic close to Vishal’s heart – and for good reason. Having heard Hasso on the topic too multiple times, I know this is a big priority for him as well. I think the primary reason to do this is to prove to the world that HANA can take transactional workloads as well or better than Oracle and other traditional DB vendors. It is a coming of age of sorts for HANA. So this definitely has merit as a worthy goal for SAP to aim at. But is this a good commercial goal for SAP?

HANA apparently has about 600 customers for what is available today as functionality. Not all of them have projects on HANA, but probably will have one at some point. Now, it has only been out for a limited time – so we have to give it time. For a 16000 strong install base, HANA has barely skimmed the surface. Probably it will sell a lot more – and it is a safe bet, since BW is not always a mission critical system, and more customers can be persuaded to buy HANA for BI purposes even though it has only gone into GA recently. There is enough value to offset cost for most BW customers.

However, that is not the story for ECC. Most companies will come to a screeching halt if anything at all goes wrong with ECC implementations. Although to a lesser extent, so is the case with CRM. So I am curious as to how many customers will make the leap into ECC or CRM on HANA in near future. I am sure there are a few that always will buy into SAP’s vision and jump into it. But the vast majority of SAP customers might take a conservative stance on changing their database for a production ECC or CRM instance.

As I and many others have pointed out before – just by porting ECC from some database to HANA, things won’t get incredibly fast by default. The reason is that most of the bottleneck is in the ABAP layer. Unless all that code is rewritten, the schemas simplified and so on – it is difficult to imagine ECC consistently getting a huge boost from Hana. I am sure SAP will probably rewrite some code to make use of HANA, at least for long running batch jobs. But even in that case – the best case I can see is most customers using it as a secondary database for accelerating some parts of ECC at best. Add to this the efforts of all the other DB vendors to attack HANA to protect their turf. So all things considered, running ECC etc on HANA is at best a technical triumph, but not a sure shot commercial success in itself.

Then there is the HANA App cloud. When I first heard about it from Vishal, I was super excited at the potential. But not a lot of action has happened for customers to jump into it. But I am a bit confused on the whole cloud strategy at SAP now. There is the big cloud division under Lars Dalgaard that has BByD, LOB On Demand solutions, SFSF etc. Then they bought Ariba – and I am curious to see where that will fit in. Will Ariba folder under the existing cloud organization? I doubt it – but curious to see is it does. And then there is the Hana App cloud, and netweaver cloud. Where does all that fit? will it stand alone ? Will it fold under Lars’ organization? Why does this matter ? Only because now SAP has so many disparate technologies and data centers, that I am not sure they have the time and effort at their disposal to get synergy out of all of this – in data centers, in applications, and in platforms. Sure we can assume that user experience will converge at some point, but if the back end is heterogeneous, it is hard to drive down costs. I hope I can get some clarity from the SAP people next time I meet them.

Many years ago – SAP made an all out effort , though not quite as impactful as current efforts on HANA , on Netweaver. Netweaver was supposed to be the super platform that would make everything work together seamlessly, and with great performance and low development effort and all that good stuff. As it turns out, it was only a moderate success. Not many people – except some die hard SAP shops – trust netweaver as the platform to run non-SAP stuff. With HANA – SAP has a chance to correct that, and make a big platform play.

Hana as a Platform is where I think SAP has biggest bang for their buck. I also think that is where SAP customers and the developer ecosystem also has the biggest bang for their buck. I am not sure if SAP sees it that way – but surely there are people there who are exploring that. Look at all the start ups, the interest in HANA during techcrunch and so on. These are not the traditional SAP ecosystem people. Their needs are different, and they have no loyalty from past that will keep them tied to SAP . And SAP has a relatively short window to keep their interest and make them loyal.

But there in lies the problem – there is only so much money and time SAP can throw at HANA, despite their deep pockets and excellent commitment at all levels of the company. The needs of HANA as a Platform are not always consistent with the features it needs to be an excellent database for ECC and other business suite products. If SAP tries to do this sequentially by getting HANA ready for business suite first, and then think of Platform plays, they might not have enough of interest from the ecosystem by the time they are ready for pltforms. Of course when I say platform, I don’t mean just the technical features – I mean everything from Go to market, pricing, support – the whole enchilada.

There is a blogger meeting with SAP coming up in few days – I will try to find answers to these while I attend those sessions, and post a part 2 if needed.