Lately, there has been some really interesting presentations and articles on agile methods and how they fit into the big picture of software development. One that was particularly interesting was Scott Ambler’s 2007 IT Project Success Rates Survey (also see the Javapolis presentation).
It presents some information that is different from how the Standish Group defines success in their often refered to CHAOS report. The Standish Group assumes that success is the traditional “on time, on budget and on scope”. In many ways this does not make sense as an estimate of time and budget are made when the least information is available (early in or before a project starts).
In Scott’s survey the following success factor priorities emerged:
- “Schedule: 61.3 percent of respondents said that it is more important to deliver a system when it is ready to be shipped than to deliver it on time.
- Scope: 87.3 percent said that meeting the actual needs of stakeholders is more important than building the system to specification.
- Money: 79.6 percent said that providing the best return on investment (ROI) is more important than delivering a system under budget.
- Quality: 87.3 percent said that delivering high quality is more important than delivering on time and on budget.
- Staff: 75.8 percent said that having a healthy, both mentally and physically, workplace is more important than delivering on time and on budget.”
Comparing Agile with other development methods
Based on these success criteria the survey investigated the success rate of different development methods:
A small step, but apparently Agile methods have the upper hand if the study is correct.
The missing pieces
What I still think is missing is a study that actually measures the outcome of a project after it was put into production and compares this with the desired effects that were envisioned. Neither the Standish Group nor Scott’s study on success factors seem to cover this.
- How quick was the return on the investment (if the goal was to increase revenue and/or reduce cost)?
- Were all of the desired effects realised?
- Were all the developed features used in a way so that they contributed to realising the desired effects? (think “reduce waste” if you are into Lean or “usability” if you are a human factors kind of person).
Measuring the above items would give more insight into the true success rate of a project rather than measuring the highly subjective “do you think is was successful with regard to factor X”? The only problem, I guess, is that few projects detail the desired effects in a way that is measurable in isolation from other factors in the environment (e.g. “the economy”).
The above items would also make it clear that the usability perspecive should be included from the start of a project.