The Official
Agile Modeling (AM) Site

Answering the "Where is the Proof That Agile Methods Work" Question

by Scott W. Ambler, Copyright 2002

Adopting and then tailoring a software process to meet your team’s needs is an important and difficult decision, one that is critical to your project’s success.  There is a myriad of choices, ranging from prescriptive processes such as the Rational Unified Process (RUP) and the OPEN process to agile software processes such as eXtreme Programming (XP), SCRUM, and Dynamic System Development Methodology (DSDM).  There is a vast wealth of material written about these methodologies, but how can you tell whether a methodology will work for you?  In particular, how do you determine whether an agile development methodology works within your environment?

If you spend anytime online reading process-oriented resources, such as the XP mailing list hosted by Yahoo! groups or the comp.object newsgroup, you have likely noticed that a common refrain of the detractors of agile methodologies is “where is the proof?”  On the surface, this seems to be a reasonable question because many developers want assurances that these new approaches do in fact work.  However, I feel that these people are barking up the wrong tree for several reasons:

1. Agile software development methodologies are new.  SCRUM and DSDM are among the oldest agile methods, being defined in the mid-1990s.  XP, arguably the most popular of the agile processes, was first described in the late 1990s as a collection of process/organizational patterns and in 2000 published as a book.  The Agile Alliance was loosely formed in the Spring of 2001.  Agile Modeling (AM) was first defined in the Autumn of 2000 and was published in March of 2002 as a book.  

Because of the newness of agile methods there simply hasn’t been sufficient time to prove that they work in a wide variety of situations.  Yes, each methodology has excellent anecdotal evidence that their method works, and more anecdotal evidence is building up every day, but statistical proof provided by a detailed study simply doesn’t exist yet and very likely won’t for several more years.  Some research results do exist, for example the effectiveness of pair programming has been examined in detail, and the issues surrounding iterative and incremental approaches are reasonably well known.  The bottom line is that metrics concerning agile software development methods are slowly being gathered but we’re going to have to wait.

2. I believe that the expectations of people requiring proof aren’t realistic.  Geoffrey Moore, in the book Crossing the Chasm describes five types of profiles of technology adopters: Innovators who pursue new concepts aggressively; early adopters who pursue new concepts very early in their lifecycle; the early majority wait and see before buying into a new concept; the late majority who are concerned about their ability to handle a new concept should they adopt it; and laggards who simply don’t want anything to do with new approaches.  Figure 1 depicts Moore's chasm with respect to agility. 

Figure 1. Crossing the Agile Chasm.

People who fit the innovator or early adopter profile are comfortable with reading a web page or book describing agile techniques, they’ll think about the described concept for a bit and then tailor it for their environment.  This is the spot in the lifecycle that most agile techniques are currently at, and it is these types of organizations that are readily adopting these new techniques.  The early majority will wait for sufficient anecdotal evidence to exist before making the jump to agile software development, and we're starting to see adoption by some of these organizations now.  

Unfortunately, the question “where is the proof” is typically asked by organizations that fit the late majority or even laggard profiles, as Figure 1 indicates.  Because agile techniques clearly aren’t at that stage in their lifecycle yet I believe that this question simply isn’t a fair one at this time.  Note that there’s nothing wrong if your organization fits into either one of these profiles, many organizations are very successful following these business models, the important thing is that you recognize this and act accordingly.

3. We’re spoiled.  previous work within the software metrics field may have set the “proof bar” too high.   In the book Software Assessments, Benchmarks, and Best Practices (Addison Wesley, 2000) Caper Jones of Software Productivity Research presents a wealth of information gathered over decades pertaining to software development techniques and technologies.  In Table 5.4 of that book Jones presents the effectiveness of various development techniques, the most effective one is the reuse of high-quality deliverables with a 350% adjustment factor whereas quality estimating tools provides a 19% adjustment factor and use of formal inspections a 15% adjustment factor.  Table 5.5 goes on to list negative adjustment factors.  For example, crowded office space gives a –27% adjustment factor, no client participation –13% adjustment factor, and geographic separation a –24% adjustment factor.  My point is that current best and worst practices have been examined in detail, but newly proposed techniques such as refactoring and co-location with your customers have not been well studied.  To compound this problem the vast majority of agile methodologists are practitioners who are actively working on software projects and whom presumably have little time to invest in theoretical study of their techniques.  In other words, it may be a long while until we see studies comparable to those of Capers Jones performed regarding agile techniques.

4. It may not be clear what actually needs to be proved regarding agile software development. In Chapter 3 of Agile Software Development Ecosystems (Addison Wesley, 2002) Jim Highsmith observes: “Agile approaches excel in volatile environments in which conformance to plans made months in advance is a poor measure of success. If agility is important, then one of the characteristics we should be measuring is that agility. Traditional measures of success emphasize conformance to predictions (plans). Agility emphasizes responsiveness to change. So there is a conflict because managers and executives say that they want flexibility, but then they still measure success based on conformance to plans. Wider adoption of Agile approaches will be deterred if we try to "prove" Agile approaches are better using "only" traditional measures of success.” 

5. Perhaps the motivations of the people asking for proof aren’t appropriate.  Are they really interested in finding an effective process or are merely looking for a reason to disparage an approach that they aren’t comfortable with?  Are they realistic enough to recognize that no software process is perfect, that there is no silver bullet to be found?  Are they really interested in proof that something works, or simply an assurance of perceived safety?  Which is strange, because if something worked for someone else it doesn’t mean that it will work for them.

6. There is growing evidence from the research community.  Recently books such as Pair Programming Illuminated (Addison Wesley, 2002) and Extreme Programming Perspectives (Addison Wesley, 2002) have published some proof that agile techniques work.  The first book explores pair programming, an XP practice, and confirms the anecdotal evidence that pair programming is in fact an incredibly good idea.  The second book is a collection of papers presented at two agile conferences held in 2002, many of which describe initial research into agile techniques as well as describe case studies.  This book is a very good start at answering the proof question, and I eagerly await similar efforts in the near future (so keep your eye out for them).

7. The anecdotal evidence looks pretty good.  There is currently significant anecdotal evidence that agile software development techniques work, you merely need to spend some time on newsgroups and mailing lists to discover this for yourself.  If anecdotal evidence isn’t sufficient then agile software development processes aren’t for you... yet!   


References and Suggested Reading

See the references page.

Agile Modeling Home Page

Agile Modeling Essays

Crossing the Chasm  XP Perspectives  Pair Programming Illuminated  Agile Software Development Ecosystems  Software Assessments

visits since December 28, 2002.