Geeks , Suits and Their Database Choices


Who chooses the database for an application? how do those decisions get made in a company ? Is it a suit or a geek or both ?

Conventional wisdom says geeks are the best qualified to decide on a database, and suits are better for selecting applications – especially packaged applications. I think the conventional wisdom is still largely true – but would add that suits have a more important role to play in selecting a database today than say a decade ago.

Say, there is a conversation happening between a suit and a geek about budgeting for building a custom order management application . And to get to my point quickly (?) – lets say the last remaining question on the table is which database to buy. Since it is a technical decision, the suit is mostly looking at the geek to start the thought process.

BEGINNING OF THE FICTIONAL STORY

The geek says – “well, it is a transactional application – so unfortunately we cannot consider these modern technologies like MongoDB which are not ACID. We should consider one of the traditional relational databases.”

Suit asks “what does that mean in terms of time and money?”

Geek : “Oh these things are expensive. Think big $$. We probably need to nail down schema design and put some strong change control in place for the project.  Once we start development, schema changes will set us back big time in schedule. Probably best to stick to a waterfall methodology. On the bright side – there are plenty of people who know this technology, and there are many tools and utilities to manage it”

Suit asks ” err…as you know, we don’t exactly know what to expect from customers since it is a brand new market. Can we deploy quickly and then change as we learn more?”

Geek : “I hear you – and if it was not a transactional system, I could have saved you a lot of time and money by choosing something way more agile and less expensive like MongoDB. See for order management transactions, you need this thing we geeks call ACID. We get that only with these expensive relational systems which have been around for decades. We have done a lot of these projects on relational . With some good project management discipline – we can deliver the first code drop on the date we planned even if we have to do it in a waterfall and not agile as methodology. But the iterations you want as we learn more – well that might be the thing that is not easy to accomplish. Once we nail down a schema , and we learn months later that we need to change it, it is not very easy to change in short time”

Suit : ” ok got it – not ideal, but then if you tell me there is no other way then we should go with your recommendation. Lets revise our business plan to make it realistic with what technology can do”.

Geek : “ok – looks like we have a plan. I will call a few vendors and get quotes and will work on a project plan”.

THE END !

These kind of conversations happen all the time in enterprises around the world – I have participated in many of these as a suit and as a geek. Neither the suit nor the geek said anything wrong in this fictional conversation above. However what is interesting is what they DID NOT say – and those nuances could have completely changed the climax of this story, and maybe in some cases saved the business a lot of time and money.

Taking MongoDB itself as an example – it is not correct to say “its a modern NoSQL database, and it is not ACID”. MongoDB is ACID within a document . And a document can be quite rich in what data it holds. An order – something that in the relational world is made of a header table, and item table, a schedule lines table, and many master data tables for holding names and addresses and so on – can be held in one document, mapping to one object in the application program. Documents can be nested too . So while MongoDB is not ACID across an entire big sharded cluster – not many apps really need it to do so either. If the application needs multi-step complex transactions – by all means consider a relational DB for persistence, but if not – there is usually a way to make it work nicely in a non-relational DB too (often with nice side benefits like scaling horizontally and keeping data well organized for access using geo tagging, as is the case with MongoDB).

The big questions that were missed in the fictional story above are specific details around business priorities and business logic. Lets consider the simplest case of any order management system – making sure that orders match quantity of finished goods available for sale. It is totally possible that there is only one widget to sell and two customers see it on their screens and want to buy it .

The geek might want the database to make sure only that one customer gets to buy the widget – and the other customer should get an error on his screen. And hence the geek might make a recommendation that a relational DB with fully ACID functionality is the right persistence layer for this scenario.

However – the Suit might think rather differently from the geek, if only someone asked him !

Case 1:

I always want my customers to place an order, even if there is an out of stock possibility. In case I don’t have stock for that customer – I can find it from somewhere else. Or I can make it up to him with a discount for something else. By keeping the customer happy – I have higher chances of getting repeat business. Maybe he was planning to buy this widget and a necklace for his wife – if I did not let him order the widget, he might have probably been annoyed enough to not order the necklace either which is more profitable for me.

What is a big priority for me is to make sure I get this app online at the earliest and keep tweaking it till I get it right. I also want to make sure the user gets extremely good response time on his browser anywhere in the world.

Case 2:

The reconciliation effort of having 2 customers both ordering the last widget is a horrible idea for my business (say I am selling antiques where no replacement exists for a piece)  . My customers actually understand that I sell unique things and that they need some luck and not just money to buy my wares.

That is my priority – I can live with changes that take a little longer to implement, given they are few and far apart.

ENDCASE

What we discussed above is just one of many tradeoffs in an application’s design – albeit a very common one. The devil is in the details. Those important details do not come into light without geeks and suits sitting together and understanding the business priorities and trade offs before choosing a database.

A database is a foundational component of an application. And unlike a decade ago – now there are plenty of choices of databases. If our fictional conversation happened in 80s or 90s, probably we would have picked a relational DB as the answer in either case. And since that was the case – there was not much of a reason for geeks to have any deep conversations with suits to find the best database for the job. That is also why databases are now a $30B plus market – customers did not have much choice and all applications worked on relational databases even if they were not a perfect fit.

Not any more – now a customer can choose whether to use SQL (still plenty of great use cases for SQL exist) , or they can choose a more modern database like MongoDB ( or others – there are other choices too depending on the nuances) for other requirements. But either way – to exercise the bounty of choices, it is imperative that geeks and suits both work together to find the right database.

SAP’s announcements on April 10,2012 – and Vijay says…


SAP very kindly let me participate in their analyst day on April 10th, and a dinner the night before in San Francisco Westin. I particularly enjoyed spending some time with Irfan Khan, who is the CTO of Sybase, and an SVP at SAP. SAP paid for my flights and hotel too. Here are my thoughts coming out of the meeting. As always, these are just my personal opinions and not that of my employer.

GA announcements were made for BW on HANA and BPC 10 (also works on HANA). SAP said they have 80 live customers – although I have not seen a number of how many have switched off their old disk based BW systems, if any. Vishal also showed impressive performance stats on a 100TB 16 node IBM system running HANA. https://twitter.com/#!/vijayasankarv/status/189764752694722561/photo/1
It was also impressive seeing how much performance improvement has been attained on the new Sandy-bridge processors . All of that is commendable – and something SAP should be proud of.

There are a few things I would like SAP to address at the field level. Customers want benchmarks – and SAP should give them some benchmark as part of certifying the hardware. SAP should also prove out the data center readiness of HANA – things like DR/HA, power consumption etc. I told Vishal Sikka that it will be awesome if he can pull the plug literally on HANA system during his SAPPHIRE keynote and show on stage that nothing will happen to data, and that fail over works. He seemed to be in agreement. Same with concurrent users – it is a common question on how performance will vary as concurrent users work on HANA. If these questions are quickly answered, SAP has a good chance of capitalizing on the momentum.
I was also pretty psyched to hear that Vishal had designed some parts of HANA himself – hats off to him. That is true technical leadership.

Steve Lucas explained SAP’s vision on being a database company – and reminded us that it is not just HANA, but also EIM products, ASE, IQ etc. First off – SAP could not have found a better guy than Steve to lead the charge on DB. He is terrific and has a great team behind him. I was not a big fan of SAP’s original message of trying to be the number 2 DB vendor on the planet by revenue. That is not only a rather unrealistic goal given ORACLE, IBM and MS will go all out and protect their turf, it will also weaken important partnerships SAP has with IBM and MS. Vishal played down the number 2 DB vendor message, and instead put this idea forward that SAP just wants to be the fastest growing DB vendor out there. Now that is a credible message, and I am sure the partners will not feel so bad about it. Nevertheless – it will be naive to think that the three other DB vendors will stand still. It will be an interesting play off.

SAP also announced a fund to subsidize HANA implementations which is worth 250 million Euros. The idea is that if a customer buys HANA, SAP will throw in free consulting up to some % of their license fees. I seriously think this is a terrible idea on many levels. Of course I don’t deny that I work for a big SI, and hence I have a serious bias.
1. It will shut off SI partners from HANA for next couple of years. And if they have nothing to gain from HANA, why would they help SAP sell it to clients where they have relationships?
2. If HANA is truly easy and simple – why does it need SAP to implement it? And after SAP walks out of the implementation – who will support HANA ?
3. If SI partners are kept out of HANA work, what will be their confidence in investing in training and IP generation for HANA and everything else SAP comes up with?

Obviously, SAP needs to capitalize on HANA momentum now before ORACLE and IBM come out with something that makes it less of an attraction. So I get why SAP has to go all out now. But I truly believe that taking SI partners along with them would have been a superior strategy. In any case, there is a chance that SAP might in turn “outsource” some of the free consulting work they give to clients to partners. I will be watching this action closely.

SAP has also come up with a VC fund for HANA based start up companies for $155M . This is a brilliant idea – and I hope many people will make good use of it.

Sanjay Poonen spent some time explaining the mobility vision. They are not going to use SUP any more – and stick more to SAP Mobility Platform as the branding. They also announced partnerships on mobility with 3 companies. I got a chance to talk to the CEO of one of these companies called Sencha. I readily admit I like the Sencha partnership more than everything else I heard during the event. If anything, SAP missed a chance to showcase them more prominently. They have a free opensource SDK for HTML5 based mobile and desktop app development. They have an OData connector too for accessing SAP systems. They have a flexible licensing model, and have millions of developers. Now – that does not mean those millions should be claimed as part of SAP ecosystem. But it gives SAP a chance to attract those people to build SAP apps. I would encourage everyone interested in mobility to check these guys out – I am planning to do a hello world on that SDK when I get some free time.

Not a lot of clarity was given on the SAP’s mobile pricing models, sandboxes of SAP systems for mobile developers to make sure their apps work etc – but we should know more from SAPPHIRE.

Here is a parting thought for SAP – at the next teched, could you get Sanjay Poonen on stage to do a bit of keynoting for mobile ? From what I have seen and heard at other events – I am sure Sanjay will surely capture the attention of SAP techies, and he is one of the most articulate people at SAP.

All things said – it was one of the best run SAP events I have been to. Well done, SAP