When context-driven software testers get together or talk online about testing, a lot of what we’re really talking about is bug hunting. Finding bugs, figuring out whether they matter, and communicating them efficiently to stakeholders are the main skills that I’ve noticed that people in the community care about. We tend to describe our role on teams as “looking out the window” or “holding the flashlight.” Testers, in our view, gather information and then package it for an audience, generally one or more stakeholders.
But I’m not sure that this is what our clients expect of testing, and even what testers outside of the context-driven conversation see their jobs as. In general, in fact, I think they see the tester’s product not as information, but confidence. They’ll say “test it and make sure it works. Even if they’d agree that’s not possible if you asked about it, I think that phrase reveals a lot about people’s basic beliefs about what testing does, and why they want it.
And honestly, they’re not wrong to want that. The value of a bug free product is obvious; the value of information is a a little more ambiguous. I mean, heck, the reason that the CDT community cares so much about information isn’t that we care about it as an end in itself, it’s that we see information as the primary tool for removing bugs, and making good decisions about the software project.
So maybe there’s no conflict here. Maybe confidence is the ultimate product of all that knowledge of that we’re so good at gathering. We focus on getting a good flashlight and deploying it skillfully, and our clients don’t care as they feel like they know enough about what’s out in the darkness.
I think it goes a little deeper than that, though. In fact, my private theory is that this is one of the big reasons it can be so tough to pry people away from test scripts and massive test plan documentation and certifications. None of those things help you find out about the product, but they’re great at creating the illusion of certainty. People who really want confidence will happily take fake but convincing certainty over real but uncomfortable knowledge.
Is this just a terminology problem? Does good information gathering inevitably lead to confidence? Is this a general problem for exploratory testers, or is this just my particular challenge, due to the way I approach testing? I’m still working on the answers to those questions, and a bunch more, but the more testing I do, the more this issue has been gnawing at me.
We don’t always provide an increase in confidence about the system. Testing can lower confidence in the system by showing that it doesn’t work. Maybe instead of testing providing confidence, it’s better to view it as testing making people’s confidence levels more accurate. Testing with good outputs gives people the information they need to accurately assess the nature of the system, which gives them more of the tools they need to have an accurate level of confidence in the system.
LikeLike
Natalie…
You’re right: there’s nothing wrong with wanting anything. I, for example, want a pony. And world peace.
There are some project managers who want confidence. It’s okay for them to want that, too.
If they want confidence, one option would be to hire a fortune teller. I hear that they are very good at providing good news and confidence. But as a tester, that’s not what I offer. I offer an investigation oriented towards discovering facts about the product. In particular, I offer facts about things that threaten the value of the product. Sometimes this undermines confidence in the product, especially when that confidence is unwarranted. However, it does offer the project manager information that she might use to make decisions and take actions that would help her to regain confidence in the product. And she might be confident about the quality of the information that I’ve provided, even though it’s not good news at the moment.
“Is this just a terminology problem? Does good information gathering inevitably lead to confidence?”
This is not just a terminology problem; it’s a problem with the ambiguity in what you’re referring to when you talk about confidence. Confidence in what? Good information gathering might not lead to confidence in *the product*, but it could lead to confidence in *the product manager’s understanding of the product*.
There’s nothing inevitable about any of this. Confidence is not a thing, or a commodity. Confidence is a relationship between some person and some situation. That person may be over- or under-confident in ways that have nothing to do with the information that I provide; she might be neurotic, or in denial; a Cassandra or a Pollyana. In other words, information about the product is what I provide. The confidence that the product manager has about anything is up to her.
LikeLike
“Confidence…in what?”
Whenever I read about “confidence” as it relates to testing, I typically notice something missing: Folks often talk about how testing does or doesn’t provide “confidence”. But they rarely specify “confidence in what”.
For example, from above: “Does good information gathering inevitably lead to confidence?”
I’d ask, “Confidence…in what?”
If the answer was “Confidence in the product”, I’d say, “Maybe. It depends. Testing might possibly lead to confidence in the product.”
But, if the answer was “Confidence in decisions about the product”, I’d say, “Testing almost always leads to confidence in decisions about the product.” That is, I will likely have more confidence in an informed decision (compared to a less informed one).
And so, I like to say, “Testing IS about confidence…in decisions about the product. Testers gather and provide info, which allows others to make more informed…more confident…decisions.”
Michael Bolton and his commenters speak to this idea here: http://www.developsense.com/blog/2013/09/very-short-blog-posts-2-confidence/
“In fact, my private theory is that this is one of the big reasons it can be so tough to pry people away from test scripts and massive test plan documentation and certifications. None of those things help you find out about the product, but they’re great at creating the illusion of certainty. People who really want confidence will happily take fake but convincing certainty over real but uncomfortable knowledge.”
I agree with this theory.
LikeLike
“Confidence…in what?”
Whenever I read about “confidence” as it relates to testing, I typically notice something missing: Folks often talk about how testing does or doesn’t provide “confidence”. But they rarely specify “confidence in what”.
For example, from above: “Does good information gathering inevitably lead to confidence?”
I’d ask, “Confidence…in what?”
If the answer was “Confidence in the product”, I’d say, “Maybe. It depends. Testing might possibly lead to confidence in the product.”
But, if the answer was “Confidence in decisions about the product”, I’d say, “Testing almost always leads to confidence in decisions about the product.” That is, I will likely have more confidence in an informed decision (compared to a less informed one).
And so, I like to say, “Testing IS about confidence…in decisions about the product. Testers gather and provide info, which allows others to make more informed…more confident…decisions.”
Michael Bolton and his commenters speak to this idea here: http://www.developsense.com/blog/2013/09/very-short-blog-posts-2-confidence/
I agree with this theory.
LikeLike
Pingback: Five Blogs – 13 January 2015 | 5blogs
Ahhhh. Yes. “Confidence in what?” is a good question, and one that explains a lot of my own confusion/fuzziness about the topic.
Though some of that is because, to be totally honest, when I think about what clients are asking for in this area, I assume that what they want is kind of fuzzy as well. I can’t shake this feeling a lot of the time that a lot of the time (most of the time?) when people pay for testing, they don’t set out to pay for information. They want a warm fuzzy feeling that the product works and they don’t have to worry, and they think that this thing called “testing” or “QA” is a way to exchange money for those warm fuzzies.
That might be unfair of me, so I’ve been being vague and saying “confidence,” rather than come out and say “I think a lot of people who hire testers are trying to pay for magic pixie dust that will protect their software project from evil.”
LikeLike