Geeks , Suits and Their Database Choices

Who chooses the database for an application? how do those decisions get made in a company ? Is it a suit or a geek or both ?

Conventional wisdom says geeks are the best qualified to decide on a database, and suits are better for selecting applications – especially packaged applications. I think the conventional wisdom is still largely true – but would add that suits have a more important role to play in selecting a database today than say a decade ago.

Say, there is a conversation happening between a suit and a geek about budgeting for building a custom order management application . And to get to my point quickly (?) – lets say the last remaining question on the table is which database to buy. Since it is a technical decision, the suit is mostly looking at the geek to start the thought process.

BEGINNING OF THE FICTIONAL STORY

The geek says – “well, it is a transactional application – so unfortunately we cannot consider these modern technologies like MongoDB which are not ACID. We should consider one of the traditional relational databases.”

Suit asks “what does that mean in terms of time and money?”

Geek : “Oh these things are expensive. Think big $$. We probably need to nail down schema design and put some strong change control in place for the project.  Once we start development, schema changes will set us back big time in schedule. Probably best to stick to a waterfall methodology. On the bright side – there are plenty of people who know this technology, and there are many tools and utilities to manage it”

Suit asks ” err…as you know, we don’t exactly know what to expect from customers since it is a brand new market. Can we deploy quickly and then change as we learn more?”

Geek : “I hear you – and if it was not a transactional system, I could have saved you a lot of time and money by choosing something way more agile and less expensive like MongoDB. See for order management transactions, you need this thing we geeks call ACID. We get that only with these expensive relational systems which have been around for decades. We have done a lot of these projects on relational . With some good project management discipline – we can deliver the first code drop on the date we planned even if we have to do it in a waterfall and not agile as methodology. But the iterations you want as we learn more – well that might be the thing that is not easy to accomplish. Once we nail down a schema , and we learn months later that we need to change it, it is not very easy to change in short time”

Suit : ” ok got it – not ideal, but then if you tell me there is no other way then we should go with your recommendation. Lets revise our business plan to make it realistic with what technology can do”.

Geek : “ok – looks like we have a plan. I will call a few vendors and get quotes and will work on a project plan”.

THE END !

These kind of conversations happen all the time in enterprises around the world – I have participated in many of these as a suit and as a geek. Neither the suit nor the geek said anything wrong in this fictional conversation above. However what is interesting is what they DID NOT say – and those nuances could have completely changed the climax of this story, and maybe in some cases saved the business a lot of time and money.

Taking MongoDB itself as an example – it is not correct to say “its a modern NoSQL database, and it is not ACID”. MongoDB is ACID within a document . And a document can be quite rich in what data it holds. An order – something that in the relational world is made of a header table, and item table, a schedule lines table, and many master data tables for holding names and addresses and so on – can be held in one document, mapping to one object in the application program. Documents can be nested too . So while MongoDB is not ACID across an entire big sharded cluster – not many apps really need it to do so either. If the application needs multi-step complex transactions – by all means consider a relational DB for persistence, but if not – there is usually a way to make it work nicely in a non-relational DB too (often with nice side benefits like scaling horizontally and keeping data well organized for access using geo tagging, as is the case with MongoDB).

The big questions that were missed in the fictional story above are specific details around business priorities and business logic. Lets consider the simplest case of any order management system – making sure that orders match quantity of finished goods available for sale. It is totally possible that there is only one widget to sell and two customers see it on their screens and want to buy it .

The geek might want the database to make sure only that one customer gets to buy the widget – and the other customer should get an error on his screen. And hence the geek might make a recommendation that a relational DB with fully ACID functionality is the right persistence layer for this scenario.

However – the Suit might think rather differently from the geek, if only someone asked him !

Case 1:

I always want my customers to place an order, even if there is an out of stock possibility. In case I don’t have stock for that customer – I can find it from somewhere else. Or I can make it up to him with a discount for something else. By keeping the customer happy – I have higher chances of getting repeat business. Maybe he was planning to buy this widget and a necklace for his wife – if I did not let him order the widget, he might have probably been annoyed enough to not order the necklace either which is more profitable for me.

What is a big priority for me is to make sure I get this app online at the earliest and keep tweaking it till I get it right. I also want to make sure the user gets extremely good response time on his browser anywhere in the world.

Case 2:

The reconciliation effort of having 2 customers both ordering the last widget is a horrible idea for my business (say I am selling antiques where no replacement exists for a piece)  . My customers actually understand that I sell unique things and that they need some luck and not just money to buy my wares.

That is my priority – I can live with changes that take a little longer to implement, given they are few and far apart.

ENDCASE

What we discussed above is just one of many tradeoffs in an application’s design – albeit a very common one. The devil is in the details. Those important details do not come into light without geeks and suits sitting together and understanding the business priorities and trade offs before choosing a database.

A database is a foundational component of an application. And unlike a decade ago – now there are plenty of choices of databases. If our fictional conversation happened in 80s or 90s, probably we would have picked a relational DB as the answer in either case. And since that was the case – there was not much of a reason for geeks to have any deep conversations with suits to find the best database for the job. That is also why databases are now a $30B plus market – customers did not have much choice and all applications worked on relational databases even if they were not a perfect fit.

Not any more – now a customer can choose whether to use SQL (still plenty of great use cases for SQL exist) , or they can choose a more modern database like MongoDB ( or others – there are other choices too depending on the nuances) for other requirements. But either way – to exercise the bounty of choices, it is imperative that geeks and suits both work together to find the right database.