The Daily Defect Count and the Image of a Camel

Count the defects daily – the ones that are part of the project work load. The number goes up and down during the cycle – why and what can you learn? Compare a couple of projects and you have an image, that you can use to visualize what is usually happening in you projects. I usually have the image of a camel…

camel trend

Previously the Testing Planet looked at A Little Track History That Goes a Long Way – how to do a simple tracking of test progress during test execution.  I keep coming back to this technique for daily testcase execution status reporting for both testcases and defects.

One source of information is the number of active defects – how many do we currently have in play to resolve?  The total number of defects is probably not significant, as we can always find one more and one more and the count will always be rising. I see no correlation between the number of testcases and defects found on a day to day basis.

Yesterday’s weather

In the article in Testing Planet 5 we looked at the trend of active defects. IT was based on defect data from eight programs of about seven projects each in the Telco domain. The graph in context looked like this (it’s not the exact numbers of the projects, but sufficiently close).

Defect Trend

Recently I started tracking the active defects in the Life Science project – a seemingly completely different context. I was curious to see what the trends would be.  There had been a hump in the active defects, and it was looking very much like the curve for “project C” as I drew it a long time ago. What happened at 5 days before launch was very interesting – the number of active defects went up – just like the trend I was usually seeing!

I still wonder why there is such an oscillation – is it a matter of too little throughput in fixing defects or a matter of testing after a few days having uncovered the first level of what the system is capable of? Again the actual number is irrelevant. The trend curve – and how you use this to guide your testing is.

Comparing Camels and Dromedary

Dromedar

Like Matt Heusser and Troy Maggenis in Yesterday’s Weather – I could see that a little statistics go a long way. Unless something is completely catastrophic is happening, you usually have about the same weather like yesterday. In context – as there will be seasons and locations like “Christmas sale” and “exploratory testing” where the camel will look completely different.

There are some dangers in comparing two different contexts. All camels have one hump – dromedary have only 1 and (Bactrian) camels have 2 [reference 3]. In both the Telco and Life Science contexts there is a management assumption that test case progress is important to the earned value of the testing activity.

Usually such KPI’s and metrics are second order measurement used to determine how it is going to go (according to plan). It’s a common misunderstanding that we know how the future will be with regards to the test execution. At worst it is the same weather as yesterday; at best it’s better. Either way the explicit knowledge of the past progress can help us to explain the trends in the current context.

Image links

[1] http://en.wikipedia.org/wiki/File:07._Camel_Profile,_near_Silverton,_NSW,_07.07.2007.jpg
[2]  http://upload.wikimedia.org/wikipedia/commons/8/82/2011_Trampeltier_1528.JPG
[3] http://farm7.static.flickr.com/6007/5993667794_705204f259_z.jpg

About the Author

Jesper Lindholt Ottosen is promoting Rapid Software Testing and Context-Driven Testing in Denmark. He is a test Manager of NNIT, previously a senior test manager in CSC and other companies. You can follow him on twitter at @jlottosen and on his blog: Complexity is a Matter of Perspective jlottosen.wordpress.com.

Tags: ,

5 Responses to “The Daily Defect Count and the Image of a Camel”

  1. Jenny MulhollandApril 22, 2014 at 8:17 am #

    One possible link I can think of is to the recent forum discussion on STC about how testers may be more “switched on” to bugs in the product after it has gone live. I wonder whether there could be a similar effect shortly before going live? Perhaps everyone is looking extra hard for problems in the knowledge that there is not much time left to find them. Of perhaps that is the time a tester goes through their notebook and follows up on all those little suspicions that have been niggling at them as they have performed their other testing.

  2. Jakob RavnbakApril 22, 2014 at 8:40 am #

    I’m just wondering: What will your defect tracking look like if you use a weighted defect count?

  3. Jesper Lindholt OttosenApril 22, 2014 at 10:15 am #

    @Jenny: Good wondering :) Thank you for sharing. I haven’t considered tracking when we find the most bugs. Looking at “detected on date” could be a starting point.

    @Jakob: The items tracked are those that need to be done by the deadline. In the original example this was based on a severity filter, but not in the the second. So there is some weight/filtering already in place.

  4. Tom du PréApril 22, 2014 at 12:25 pm #

    There could be a number of reasons for this second hump. Here are some I can think of.

    1. What Jenny Mulholland said above. People start to really focus on finding things before a product goes live.
    2. Developers who have finished developing turn their hand to helping out with testing towards the end of a project.
    3. Some people (mistakenly and foolishly) might leave their regression testing to the very end, so only find regression defects late.
    4. UAT often only starts towards the end of the project (Again, a foolish mistake!)
    5. Code is deployed more production-like environments at this stage, which exposes new defects.
    6. Only at this point in the project has all the code been made available to test. Often the biggest and most complicated things that have taken the longest time to develop get delivered to test last, and these are liable to be buggy.
    7. As developers get closer to their deadlines, they might try to work too fast and make more mistakes. They’re only human.
    8. Despite the above, it is seems very probable to me that the data in your “Average track” line in your chart isn’t significant:
    * You say it’s an average from a fairly small sample of projects
    * The second hump looks like an anomaly on just one data point (1 day) on an otherwise pretty smooth looking downward looking trend. If the projects spanned longer periods, and the second hump itself had a smooth and trend-able curve up and then down again, I would be more convinced.
    * The second hump is a change of just a 2 or 3 points away from where you might expect it to be, which isn’t very much, even when the range of likely values (somewhere between 0 and 11) is pretty narrow.

  5. Jesper Lindholt OttosenApril 23, 2014 at 9:00 pm #

    @all:
    The number of defects tracked is the non-closed bugs being handled. Every day some are being added, retested, rejected, closed, processed and removed – depending on the testers and developers doing their work. I try to think of it, as “it is what it is” a pattern. sometimes the reasons a visible other times not.

    @Tom: I’ll mail you the data :)