How do you measure success and failure of projects?

There are quantitative and qualitative ways to describe the success or failure of a project.

In general, a project is considered succesful when it meets its objectives while staying within an agreed up on budget and time line. This sounds simple enough, except that it is not !

First, who decides if a project is succesful? And if a project passes a series of tests and goes live, is it considered succesful? The common model is to celebrate success at go-live. But going live is just the beginning of the journey. It takes a fair amount of time to judge if the business got sufficient value out the project. Also, at what cost? Did they have to hire more people to support the implementation? Can they make enhancements to the functionality as requirements change, without costing an arm and a leg? Do we attribute this to the original project?

Since I am more familiar with SAP projects than non-SAP projects, let me quote examples from SAP area. In the nineties, SAP was not as rich in functionality as it is today. Many implementations had dirty modifications to source code to make it work according to requirements. These were called succesful projects at the time. 15 years later, these same modifications have caused the customers a lot of trouble in upgrading their systems. So do we still call them succesful?

Second, all stakeholders have to agree to a set of objectives that a project has to meet. Sounds simple enough – but then, it is kind of hard to accomplish too. First, there is the legal issue – if you cannot express it in a language that lawyers agree to, the objective will not be present in the contract. And if it is not in the contract – it is hard to ensure it will be met. Then there is the question of the definition of who a stakeholder is. In large projects, there are many people who are affected, and it is impossible to get them all enlisted to create a list of objectives. So a subset of people is assigned as leads or representatives, and they may or may not know how the new project is going to affect the work of the whole population. 

In one of my first succesful SAP projects, I remember going to the office the day after go-live and finding a large number of trucks backed up from the loading bay all the way into the street. Guess what? no one remembered to capture the requirements of the picking department – so they tried to do it the old way, like they have done all their lives – and then could not update the system with the results. Thankfully, we could solve it with some quick work around – but the point is – you cannot figure out every one’s requirements all the time in big projects.  Voice of the majority does not mean something is right – there are plenty of exceptions in every business process, and they all should be handled.  Consultants swear by “Mutually Exclusive and Collectively Exhaustive” – just that as scope increases, the chance for hitting this paradigm decreases.

Third – how do you agree up on project time line and budget? It is by estimation, which is an inexact science by definition. Based on prior experience, some one thinks that a certain task will take a certain amount of time to finish.  Just as the stock market has taught us that past performance is not a good measure of future (for the short-term),  there is no guarantee that these estimates will hold true across the life of the project. Techniques like PERT make us eliminate bias by using formulae that goes from pessimistic to optimistic. But as all experienced project managers will attest – estimates are best guesses, irrespective of techniques. 

Now – this means estimates have to be revised as projects progress and we have more data. But the question remains – if I estimated $2M today, and then re-estimated after 3 months to find that it actually costs $3M – is the project already a failure? The world outside the project typically looks at projects with this perspective. Once an estimate is made for time and money, then the world expects it to be set in stone, come what may. There is no room for re-estimation.  So most projects get into this situation that they are set up to fail from the beginning.

Finally, no project has unlimited resources – so scope and timeline usually is subservient to available money. This means some one has to prioritize scope. And in this process, some requirements will need to be dropped. Despite the due diligence in this exercise – it is hard to avoid a decrease in value to some part of the customer organization. But when the project finishes and these gaps get exposed – usually it is put as a failure of the project.

Now, I am not the only one to have this knowledge – most customers, consultants and software vendors know this. But why would not this situation change? Just a rhetorical question of course 🙂