Vishal Sikka explained in his SAPPHIRE and TECHED keynotes that the general direction for SAP is to collapse the multiple layers that exist today, and move more and more of them natively into HANA. It sounded like a pretty neat idea to me when I heard it, and I did not think more about it for some time. Timo Elliot then mentioned the example of digital photography getting rid of obsolete layers and revolutionizing the industry. In his opinion, HANA will do the same.
Vishal and Timo are two of the most brilliant minds that I admire in the SAP world. But for some reason, a part of me started thinking – is collapsing layers a good idea generally? My education is in mechanical engineering, and not in computer science. So I started thinking along the lines of machine design to see if I can rationalize this somehow. And I failed.
I started an attempt to discuss with Timo about this on twitter, but it is hard to do it 140 characters at a time. And then my flight to PDX started boarding and I had to abruptly end the twitter session and run to the gate. Since I have some time now in the flight to reflect (and type), here goes.
Layers exist for good design reasons. Layers traditionally have been built for some specific purpose, and over time they get very optimized for that one purpose. As technology improves, and smart people get bright ideas – they can replace a given physical layer with something else that is better. But the layer in design time should still remain in most cases. Why? mostly because there are smarter people in every generation who will improve upon this, and Mr Murphy might show up unannounced.
Taking Timo’s example of digital cameras – the physical storage in films got replaced by memory cards. But there is still some storage layer that did not get replaced. Tomorrow something else will come up that is better than these cards, and this layer will further improve. Now – nothing stops a designer from combining thr storage with the body of the camera or the lens or something else. This might be pretty revolutionary – and might outperform other cameras too, and users might love it in short term. But the question is – with this design, can the next generation of designers add on benefits of advancements in each layer that collapsed? To net it out – does performance outweigh the need to maintain and enhance?
Another example is our family car. I love that car – but have had multiple accidents with it in the past. If the manufacturer of the car had decided to collapse layers – my insurance company would have probably paid off the car after the second time it had an accident, and I would have had to cough up more money to buy another car. It just would not have been easy or cost effective to repair it. I would have been pretty mad at this scenario since all but one accident was the other party’s fault. Fellow Indians who moved to the USA might resonate with this – when I grew up in India, we could get everything on my dad’s Ambassador repaired somehow. When the newer cars came up – the concept of repair was mostly rip and replace. It was not affodable for many of us to do this.
In a stable system, it is comprehensible that some layers could be collapsed. Example : BW has aggregates. With HANA, you can aggregate on the fly. So it makes sense to get rid of this layer once BW gets HANA as the DB. But here is the question in my mind – If and when HANA truly becomes big data friendly (petabytes instead of TB), will it help to have a logical definition of an aggregate that will tune the SQL to get a specific aggregation done more efficiently?
We can argue that most of the data in BW is in TB range, and hence this is not needed at this time. But that brings up another question on innovation – is it innovative if you have to constantly tinker with the design? or should innovation stand the test of time? Timo made an excellent point on twitter – innovation does not come in neat buckets. It is hard to argue with that, and I readily agree. But for innovation to be useful – I believe it needs to be moulded into a form that can work for an extended period of time. At a minimum, this should apply to anything that costs customers in the millions.
Even if I somehow convince myself that collapsing layers make sense in a stable system, since we have a good idea of what each layer does and what will happen in foreseable future – can I extend this and believe that this is applicable to newer systems? I am having a hard time convincing myself of this.
HANA is just about a year old or so. Revisions are coming fast and furious to the HANA software. And HANA has an aggressive roadmap to make it work for BW, ECC and so on in a short period of time. It means work needs to be done on many different things that make up HANA (stabiling the core, enhancing the SQL, scaling out and son on). In this scenario – will it be easier for SAP to develop and enhance HANA if more and more layers are brought natively into HANA ? May be it is – I don’t know. I have not worked in software product development. Hopefully some one from that background can explain to me why this is a better idea.
The closest I can relate to is a custom build engagement that I manage, which uses MSFT technologies. The architects followed a multi layered paradigm to build this. It handles mission critical information, and there is very little tolerance for error. We debated many times internally if it was possible to decrease the number of layers to improve coding speed, and performance. My chief architect held firm that each layer has a job to do and it will serve us well if we stuck to it. We agreed on it – and he was right. As soon as first release was done, we had to start on the next one, and there is just no way we would have kept up with enhancements and performance tuning aspects etc if we did not stick to the multi layered architecture.
Sure we could have gained some performance – but in the overall scheme of things, the solution seemed pretty good. But then – this is a project, and not a product. What works in projects might not be the right approach for product development. So I can’t make any firm conclusions based on this experience.
If I knew in my younger days that my career and interests will go this way, I might have skipped my MBA and done a masters in computer science or gone to a design school. Since I did not do that – I guess I have to learn this the hard way by debating this in public, and probably get electronically beaten up 🙂
And Vijay Says... 
With HANA, I think layers and tiers get consolidated.
Tiers (I mean the platform/machine) consolidation is obvious given the specs of the appliance.
Layers…. rather than saying collapsed (yes, certain layers could be collapsed), I would say , become lean and more specific/cohesive, fully leveraging the capabilities of specific underlying framework.
With more capabilities being added to HANA platform, I won’t be surprised if applications get architected with 1-1 layer per framework.
By framework, I mean the frameworks developed on top of specific capabilities of languages/constructs such as L, R, IMSL, C++ and Next Generation ABAP.
If I have to design an application in HANA, I would still use a layer for UI(HTML5/BSP/WD) , a layer for external service integration(SOAP/ODATA/REST), a layer for process orchestration(ABAP/C++) and a layer for data manipulation/aggregation(R/IMSL/SQLScript) with a common data handling/query mechanism to be shared among these layers.
It just keeps the design cleaner rather than going for a monolith.
And to your original question “Is collapsing layers a good thing?” . I would say yes. Because, the purpose of each layer is to address certain issues while abiding to certain constructs and constrains, specific to that period of time and technology. Those constructs/constraints may change with time, making the existence of the layer redundant.
Whenever I see a disruptive technology, I remember CORBA and how the simple SOAP web services made CORBA redundant. Today we are saying SOAP is heavy and we are looking for lean mechanisms.
LikeLike
layers get added over time to protect the kernel, but once in a while we need to pull out the Ockham razor to cut the excess off. the goal here is to have only two tables, isn’t it?
LikeLike
Yeah we are saying the same thing. My theory would be that PSA is persistance – just a copy of what remains in the ERP, to avoid loading from ERP in future. If ERP is in-memory then that’s not relevant.
We keep corporate memory as the common-denominator. In non-HANA-stored data terms we would probably keep PSA too. The thing with the LSA is that it is flexible in this respect and doesn’t care.
Either way: fewer physical layers, same logical layers.
What HANA does not do is reduce the complexity of merging information from disparate data sources. What it may do is to make it easier, because it’s faster and more flexible. Just not less semantically complex.
LikeLike
So it’s time to put the boxing gloves on.
First, if we used the camera analogy (which I really like) but take it a level deeper – we have replaced an analog pattern of silver crystal particles with a representation of photos on a CCD. What’s really interesting about mid-to-high-end digital cameras is that they save a “digital raw” file which contains the raw information from the CCD. If you reprocess a digital negative today, that you saved from a camera 5 years ago, the you will get a better quality representation – because technology has improved in the software processing. I think this works nicely with your layers concept.
Second, I think that we need to separate physical manifestation of layers with the logical representation of layers. For example the SAP BW storage concept is based on the Layered Scaleable Architecture.
– Acquisition Layer
– Corporate Memory Layer
– Propagation Layer
– Business Transformation Layer
– Flexible Layer
– Dimensional Layer
– Virtual Layer
I’m thinking if you got this far you know about DW and LSA, so my feeling is the following might happen…
– Acquisition Layer becomes part of the ERP, when ERP and DW merge into a single SAP HANA cloud
– Corporate Memory Layer remains. This is needed because it’s the business object layer at which DW information is retained
– Propagation Layer and Business Transformation Layer become logical representations via rule based mechanisms.
– Flexible Layer and Dimensional Layer become one physical layer and one logical layer based upon needs.
– Virtual Layer remains logical
So in doing so we are collapsing physical manifestations of layers – but not the glue and complexity that holds them together. We can’t do that because we built the LSA because we wanted flexibility in the first place.
Which is kinda the point, right 🙂
LikeLike
Hey, you are a comp science major…so guess I shouldn’t put kid gloves 🙂
The future of every layer needs to be questioned, even in BW case.
1. Acquisition layer – why should this become ERP ? As time progresses ERP data might be only a small part of needed information. What is the strategy for multi source?
Which also brings to question why would we have a ERP and DW together scenario, if most of the other data sits elsewhere. Is that the best architecture possible?
2. Corporate memory – why bother with this in a physical sense future? ECC data can be replicated into HANA quickly, or ECC might use HANA as a DB. So why have a layer?
and so on…
Very easily, we can also argue that these layers need to stay as well.
As I mentioned in the blog, and you did too above- physical layer is not the issue here. It is the design time layer – or logical layer that needs to stay.
Data increases in size and complexity faster than technology. So keeping specialized layers at design time seems to be the best insurance to deal with it.
As technology catches up, each layer can get a better physical representation.
Most probably we are saying the same thing any way here 🙂
LikeLike
I have no idea about the layers you’re referring to, but would agree with you on your PoV.
I have no idea how HANA works but to me it sounds like a lender of last resort – 20 years ago us integration folks decided that we couldn’t afford to sacrifice persistency over performance
When I put on my mean hat, this will allow SAP to monolith for a few more years and avoid the Cloud threat that will force them to uncrap their product into neat layers. They eSOA-ed their way out of SOA, and this is their next best thing. SAP isn’t layered, it’s ABAP ll over the place
Having put on my gentle hat, I might argue that SAP wants to simplify their layers – but they don’t need HANA as an excuse to do so, and if so, they should name which layers they want to eradicate and oh, btw, why so
BTW my education is in languages and cultures of Latin America, but I’ll take on a discussion about this any time; only common (enterprise architecture) sense is needed here
LikeLike