Answering the "Where is the Proof That Agile
W. Ambler, Copyright 2002
Adopting and then tailoring a
software process to meet your team’s needs is an
important and difficult decision, one that is critical
to your project’s success.
There is a myriad of choices, ranging from
prescriptive processes such as the Rational Unified
Process (RUP) and the OPEN process to agile software
processes such as eXtreme Programming (XP), SCRUM, and
Dynamic System Development Methodology (DSDM).
There is a vast wealth of material written about
these methodologies, but how can you tell whether a
methodology will work for you?
In particular, how do you determine whether an
agile development methodology works within your
If you spend anytime online reading
process-oriented resources, such as the XP mailing list
hosted by Yahoo! groups or the comp.object newsgroup,
you have likely noticed that a common refrain of the
detractors of agile methodologies is “where is the
the surface, this seems to be a reasonable question
because many developers want assurances that these new
approaches do in fact work.
However, I feel that these people are barking up
the wrong tree for several reasons:
1. Agile software development
methodologies are new. SCRUM and DSDM are
among the oldest agile methods, being defined in the
mid-1990s. XP, arguably the most popular of the
agile processes, was first described in the late 1990s
as a collection of process/organizational patterns and
in 2000 published as a book. The Agile
Alliance was loosely formed in the Spring of
2001. Agile Modeling (AM) was first defined in the
Autumn of 2000 and was published in March of 2002 as a book.
Because of the newness of agile
methods there simply hasn’t been sufficient time to
prove that they work in a wide variety of situations.
Yes, each methodology has excellent anecdotal
evidence that their method works, and more anecdotal
evidence is building up every day, but statistical proof
provided by a detailed study simply doesn’t exist yet
and very likely won’t for several more years.
Some research results do exist, for example the
effectiveness of pair programming has been examined in
detail, and the issues surrounding iterative and
incremental approaches are reasonably well known.
The bottom line is that metrics concerning agile
software development methods are slowly being gathered
but we’re going to have to wait.
2. I believe that the
expectations of people requiring proof aren’t
Moore, in the book Crossing
the Chasm describes five types of profiles of
technology adopters: Innovators who pursue new concepts
aggressively; early adopters who pursue new concepts
very early in their lifecycle; the early majority wait
and see before buying into a new concept; the late
majority who are concerned about their ability to handle
a new concept should they adopt it; and laggards who
simply don’t want anything to do with new approaches.
Figure 1 depicts Moore's chasm
with respect to agility.
Crossing the Agile Chasm.
People who fit the innovator or
early adopter profile are comfortable with reading a web
page or book describing agile techniques, they’ll
think about the described concept for a bit and then
tailor it for their environment.
This is the spot in the lifecycle that most agile
techniques are currently at, and it is these types of
organizations that are readily adopting these new
early majority will wait for sufficient anecdotal
evidence to exist before making the jump to agile
software development, and we're starting to see adoption
by some of these organizations now.
Unfortunately, the question “where is the
proof” is typically asked by organizations that fit
the late majority or even laggard profiles, as Figure 1
agile techniques clearly aren’t at that stage in their
lifecycle yet I believe that this question simply
isn’t a fair one at this time.
Note that there’s nothing wrong if your
organization fits into either one of these profiles,
many organizations are very successful following these
business models, the important thing is that you
recognize this and act accordingly.
3. We’re spoiled.
previous work within the software metrics field may have
set the “proof bar” too high. In the book Software
Assessments, Benchmarks, and Best Practices (Addison
Wesley, 2000) Caper Jones of Software Productivity
Research presents a wealth of information gathered over
decades pertaining to software development techniques
and technologies. In
Table 5.4 of that book Jones presents the effectiveness
of various development techniques, the most effective
one is the reuse of high-quality deliverables with a
350% adjustment factor whereas quality estimating tools
provides a 19% adjustment factor and use of formal
inspections a 15% adjustment factor.
Table 5.5 goes on to list negative adjustment
example, crowded office space gives a –27% adjustment
factor, no client participation –13% adjustment
factor, and geographic separation a –24% adjustment
point is that current best and worst practices have been
examined in detail, but newly proposed techniques such
as refactoring and co-location with your customers have
not been well studied.
To compound this problem the vast majority of
agile methodologists are practitioners who are actively
working on software projects and whom presumably have
little time to invest in theoretical study of their
techniques. In other words, it may be a long while until we see studies
comparable to those of Capers Jones performed regarding
4. It may not be clear what
actually needs to be proved regarding agile software
development. In Chapter 3 of Agile
Software Development Ecosystems (Addison Wesley,
2002) Jim Highsmith observes: “Agile approaches excel
in volatile environments in which conformance to plans
made months in advance is a poor measure of success. If
agility is important, then one of the characteristics we
should be measuring is that agility. Traditional
measures of success emphasize conformance to predictions
(plans). Agility emphasizes responsiveness to change. So
there is a conflict because managers and executives say
that they want flexibility, but then they still measure
success based on conformance to plans. Wider adoption of
Agile approaches will be deterred if we try to
"prove" Agile approaches are better using
"only" traditional measures of success.”
5. Perhaps the motivations of
the people asking for proof aren’t appropriate.
Are they really interested in finding an
effective process or are merely looking for a reason to
disparage an approach that they aren’t comfortable
they realistic enough to recognize that no software
process is perfect, that there is no silver bullet to be
they really interested in proof that something works, or
simply an assurance of perceived safety?
Which is strange, because if something worked for
someone else it doesn’t mean that it will work for
6. There is growing evidence from the research
community. Recently books such as Pair
Programming Illuminated (Addison Wesley, 2002) and Extreme
Programming Perspectives (Addison Wesley, 2002) have
published some proof that agile techniques work.
The first book explores pair programming, an XP
practice, and confirms the anecdotal evidence that pair
programming is in fact an incredibly good idea.
The second book is a collection of papers presented at
two agile conferences held in 2002, many of which
describe initial research into agile techniques as well
as describe case studies. This book is a very good
start at answering the proof question, and I eagerly
await similar efforts in the near future (so keep your
eye out for them).
7. The anecdotal evidence looks
pretty good. There is currently significant
anecdotal evidence that agile software development
techniques work, you merely need to spend some time on
newsgroups and mailing lists to discover this for
yourself. If anecdotal evidence isn’t sufficient then agile software
development processes aren’t for you... yet!
See the references
visits since December 28, 2002.