Software Development is not Software Testing
One of my responsibilities as a Software Performance Analyst for WebSphere Business Monitor is to create prototypes for medium to large code changes that will improve the performance of the WebSphere Monitor software stack. In developing a recent prototype, I was having trouble making heads or tails of why a particular XML parsing toolkit wasn't working right, so at work on Wednesday my team lead, Kean, and I walked through some of the code together. As we thought about how the parser was behaving in certain circumstances, Kean always seemed to know the next thing to try to figure out how to make the parser work, and as I thought about why I had an epiphany: software development is not software testing.
Unsurprisingly, software development is all about developing software. When a developer adds new functionality to an existing codebase, it's critical that she understands the way the application works before she drops her new code so that she can guarantee that the application will work the way she wants it to after she drops her code. Because understanding a codebase at a low level is requisite to implementing new functionality on that codebase, a developer's most valuable asset is her intuition, and the most important skill she can develop is the ability to wrap her head around large amounts of code in a hurry. In her day-to-day job, it's a developer's job to ask "Why does this software behave this way?"
Again, unsurprisingly, software testing is all about testing software. It's a tester's job to make sure that a piece of software behaves according to a specification. Any behavior that a program exhibits contrary to that specification is what's called a "bug". Testers are generally evaluated by how many bugs they find in a piece of software, so it's easy to see why smart testers get very good at detecting bugs consistently and efficiently. In his day-to-day job, it's a tester's job to ask "How does the software behave when I do this?"
In order to find bugs as consistently and efficiently as possible, it's important that a tester doesn't reason about the code base at the low level a developer does because it colors his thinking and it takes too long. Besides, by the time a tester built a low-level understanding of part of a codebase, some developer probably has already changed it! Instead, it's useful for a tester to view the software he's testing as a black box.
A black box is a model of an abstraction. When you give a black box a set of inputs, the black box "magically" produces a set of outputs. As such, the black box is an abstracted model for the behavior of a system. You don't care how the black box translates its inputs into its outputs; you only care what outputs it produces for a set of inputs.
A good example of a black box is a car. When I'm driving my car and I push the gas pedal, I know the car goes faster. I don't care that the reason it goes faster is that pushing the bas pedal causes the fuel injection system to inject additional fuel and air into the piston chambers, which causes more powerful combustion reactions in the piston chambers, which exerts a stronger force on the pistons, which generates additional torque to the drive shaft, which the transmission then translates into a torque force on wheels, which causes them to turn more quickly. I don't care why pushing the gas pedal makes the car go faster in order to operate a car; I only care that it does.
A tester can develop the understanding of the codebase that he requires quickly because of the way he approaches developing that understanding. When a developer looks at a snippet of code, her mind immediately starts trying to figure out not only what result the code produces, but also why the code produces that result. A tester figures out the what and stops there. This does not reflect laziness on the part of the tester; he stops there because that's all his job requires; anything more would be wasted time.
I realize now that I've been making writing prototypes harder than it needs to be because I've been trying to "be a developer" on a codebase that I really should "be a tester" on. It's not my job to fix or to add new functionality to the WebSphere codebase, it's just my job to make it go faster. I don't make new black boxes, and I don't make existing black boxes give different outputs for the same inputs; I just make existing black boxes give the same outputs for the same inputs faster. To do that, I only need a tester's working, black-box "how"-level understanding of the codebase, not a developer's low-level, "why"-level understanding. I've been asking "why" when I should have been asking "how".
- andy's blog
- Login or register to post comments
i'm not sure i agree...
...but not because you're wrong, only that you're confusing your 'hows' and 'whys.' i agree that your job is the 'how' and not the 'why.' But you mention that testers "don't care how the black box translates its inputs into its outputs; [they] only care what outputs it produces for a set of inputs" yet at the end you realize the question you should be asking is, in fact, 'how' not 'why.'
i'm no software developer (or tester), but I think there is more than a semantic difference between the two: you DO care HOW the black box works, but it's not your job to decide the route taken to make it work (the 'why').
as for the car analogy, in my opinion, you're confusing roles when you say "I don't care why pushing the gas pedal makes the car go faster in order to operate a car; I only care that it does," -- that's the role of the USER.
the role of a tester is not clearly defined in your post - is it a debugger? an optimizer? both? as a tester, you DO need to know, for example, that the gas pedal is connected to a spring-loaded modulator of and ECU which controls injection timing into the piston chambers. if the car isn't giving the expected output - motion - (a bug) or isn't operating at maximum efficiency given its build constraints (non-optimal motion), or is brand new from the design studio (untested), you will need to know those things to work through those issues.
it's NOT the testers job to decide what's the best way to go about doing something. consequently, the 'why' - which is the physics of combustion, mechanical advantage, leverage; the math of gear ratio calculations and injection modulation; or the 4-piston vs 6-piston decision - is the role of the developer/engineer.
The developer makes the car. he doesn't drive it. The tester crashes it and sends it back with a post-it saying what to fix. he doesn't add dual-zone climate control. the user drives it. he doesn't ask questions.
what a tester is
Testers should view the product as a black box whenever they're doing their testing. That's not to say that he can't know how the product works, nor that it isn't sometimes useful for him to know. However, the true value of a tester is his experience in breaking things, and his impartial view of the software. Developers know too much about a piece of software to give it a really good once over.
Part of the problem developers have is the delusion that they understand something. (I'm not throwing stones here, I've had my own delusions of this very kind.) Let's go through an exercise here.
Imagine we're in PHP, and I've told you that I've got a function
array_set_wrapperwhich -- behold, wonder of wonders -- wraps the array set operation. The documentation gives you its implementation:Do you feel a need to test this code? Having seen the implementation, how would you test it? It's pretty obvious this code works. When I see this code, my mind doesn't immediately know how test this code. Honestly, I don't even really feel like I need to test it at all.
But, there actually is a bug in this function. Do you see it yet?
On the other hand, though, if they had simply showed you documentation that says (ignore the fact that there's no such thing as a function prototype in PHP):
...Now how would you test it? Because we haven't broken the abstraction here, I know immediately how to test the thing.
Also, when they give me less information, it makes the bug more obvious. In PHP, array parameters are passed by value by default. This function can't possibly have side effects on my array! The parameter
$aneeds to be passed by reference, or as&$a.I grant you, this is a very simple example, but hopefully this illustrates how breaking code abstraction can make software testing difficult.