There are a number of surveys available that will give you statistics on how many application development projects succeed or fail, including the most famous of the IT surveys or at least the most quoted, the Chaos Report which is put out annually by the Standish Group. If you look at their survey results, you might decide that the people who are running these projects have no idea what they are doing. After all the 2009 report showed that only 32% of projects were successful, 44% were challenged, and 24% were considered failures. And while those numbers are a definite improvement from when the Standish Group started conducting its survey in 1994, they paint a dismal picture of IT’s ability to manage projects.
That is, until you look at the definition of the categories. The Standish Group defines its categories as follows:
◊ Success: Delivered on time, on budget, with the required features and functions
◊ Challenged: Late, over budget, and/or with less than the required features and
functions
◊ Failure: Cancelled prior to completion or delivered and never used
If you look at the definition for Success, it seems to set an almost impossibly high standard for success, although even that high bar has come down from when they started the survey in 1994 and said that success meant that the software included all the initially specified features and functions. I have other concerns about applying a definition like this to Project Success. Namely:
◊ It ignores other factors that are also key to the perception of success, such as:
● Quality – Is your work really done if you met the target goal, but you’ve got a
large number of bugs to remediate?
● Customer Satisfaction – I don’t know anybody who hasn’t run into the situation
where you delivered exactly what was asked for and the client or the business
decided that it wasn’t what they were expecting.(I’ll talk about how you can
avoid that situation in another post)
● ROI – Return on Investment seems to be one of those forgotten measures of
success for everybody but the finance department.Probably because you can’t
really start measuring it until after the project is done, and by then everybody’s
started to focus on the next project.
◊ It’s simplistic. The Standish Group definition applies a one size fits all definition to
success. And in real life, I’ve found that clients often trade off one factor for
another to create a custom definition of success. For instance, I had a dot-com
client who was racing to be first to market with their product, so schedule, feature
set, and customer satisfaction were the primary factors in determining success for
them. I had another client whose product enhancements were funded out of
revenues, so budget and ROI were the primary factors for them.
I think the bottom line is that there is no single definition of success. It’s going to vary by client and even by project within the client. So if you are the project manager, one of your first goals on your project is to define success for your project. And I do mean define it.Because you’ll find that if you survey your stakeholders, each one of them is going to have a slightly different definition of success. It will be up to you as the project manager to create a definition that everyone can agree on and document it as part of your project charter. You’ll also need to pull it out once in a while to remind your team what the end goal is.
So what are the definitions of success that you’ve used in the past? What are the factors that your business users or clients considered the most important?
– Lisbi Abraham
My reaction after reading the definition of success from the Standish Group as “Delivering on time, on budget, with the required features and functions” wasn’t that this was impossible, but that this should be an expected outcome…with success as going above and beyond this. Given the results of the survey, however, I may be expecting too much. I wonder if expectations now are that projects will be over budget and late?
I concur that success greatly depends on its definition and collectively agreeing upon this before the project begins. Further, check points to gauge this success, as you stated, makes compete sense.
How do you think a project manager should balance gauging success based on measurable results and objective opinions?
I’m not sure who said, “Low expectations are the key to happiness,” but I’m pretty sure whoever it was never built anything worth mentioning. Low expectations are not the key to happiness, but high expectations are absolutely key to success. It seems to me that empirical measures of success (“on time, on budget”), while important, are penultimate. Client satisfaction, though anecdotal, is the ultimate measure of success because client perceptions are the foundation of every consultancy’s reality.
The on time and on budget portions of the definition are actually very reasonable expectations. And businesses should be holding their IT people or consultants accountable to that. Where the definition gets tough is the all required features and functions. Initially, the Standish Group said all initially specified features and functions, but it looks like they backed off of that definition; probably because they recognized that requirements change over time. But even required features and functions is a tough standard to meet for the same reason. That’s because as market conditions change, or the business users develop a better idea of what they want, what is required is going to change. A project’s ability to incorporate all those changes and still make budget and schedule is asking for a lot. What usually happens is that something has to give. Sometimes it is budget, sometimes it is schedule and sometimes it is scope.
I would argue that the Standish Group’s definition of Failure is misleading as well. I’ve been in companies where they’ve cancelled projects prior to completion because the market direction had changed. A cancellation like that wasn’t really considered a failure because it was a recognition of the fact that the purpose of the project was no longer valid and there was no business reason for sinking more money into it.
If you want to get another contract with the client, then absolutely their satisfaction is your ultimate measure of success. But you still need to manage their expectations in order to ensure that satisfaction. Because the client’s expectations are always going to be a moving target. They will be guided by changes in market conditions, new realizations as they are exposed to how their original requirements look in an actual product, etc… If you aren’t careful, those shifting expectations can really hurt your attempts to provide client satisfaction. You need to make sure that you are accommodating their shifting expectations where possible and tempering them when they run into the hard realities of product development.
Jim,
That’s a good point. Cancelling a project under conditions like that could even be considered a successful outcome of good project governance. It’s a recognition of changed circumstances instead of blindly marching along based on the original plan.
So if the client’s expectations are going to be a constantly moving target, then how do you go about managing those expectations?
Thanks for reminding me that I had promised to post a blog entry about how to provide customer satisifaction among shifting expectations.
Just curious, but wouldn’t the project size have a big impact on the success / failure rates of projects?
The project size absolutely has an impact on the success and failure rates of a project. In fact there was a recent survey in Dr. Dobbs (http://www.drdobbs.com/architecture-and-design/226500046) that addresses this very issue. They used more flexible definitions for Project Success than the original Chaos survey but the concepts remained the same. They also looked at how methodology impacted success rates, and here are the survey results for success rates across methodology and team size:
•Ad-hoc projects: 74% for small teams, 58% for medium-sized teams, and 40% for large teams.
•Iterative projects: 80% for small teams, 68% for medium-sized teams, and 55% for large teams.
•Agile projects: 83% for small teams, 70% for medium-sized teams, and 55% for large teams.
•Traditional projects: : 69% for small teams, 61% for medium-sized teams, and 50% for large teams.
For the purposes of your question, we’ll assume that large project = large team size, although that isn’t always the case.
Hi Brian,
You can get the definitions from the original survey at the link I posted above, and I highly recommend reading it for some additional information on the subject. But here are the definitions that were used for the survey:
- Ad-hoc. On ad-hoc software development projects the team does not follow a defined process.
-Iterative. On an iterative software development project the team follows a process which is organized into periods that are often referred to as iterations or time boxes. On any given day of the project team members may be gathering requirements, doing design, writing code, testing, and so on. Rational Unified Process (RUP) is an example of an iterative software process.
- Agile. On an agile software development project the team follows an iterative process which is also lightweight, highly collaborative, self-organizing, and quality focused. An example of an agile process is OpenUP, Scrum, and Extreme Programming (XP).
- Traditional. On a traditional software development project the team follows a staged process where the requirements are first identified, then the architecture/design is defined, then the coding occurs, then testing, then deployment. Traditional processes are often referred to as “waterfall”, “classical”, or simply “serial” processes.
Can you tell me the difference between the project methodologies that you listed? (ie. what’s ad-hoc vs iterative vs Agile vs Traditional?)
Thanks!