Part 2: Object-Oriented Testing
Scenario-Based Test Design
Contents
1. Recapitulation
2. What's Missing?
3. Scenario-Based Test Design
3.1 An example of finding an incorrect specification
4. Effects of Object-Oriented Programming
4.1 Surface structure
4.2 Deep (architectural) structure
5. Summary
1. Recapitulation
In Part 1, I described one type of test design that targets plausible faults. Testing techniques search the code or specification for particular clues of interest, then describe how those clues should be tested. The clues in an object-oriented program are the same as in a conventional program. Divining what to test requires either more work or more bookkeeping, because there are more places where relevant information might live. But the approach is essentially the same.
I am posting these notes so that people can tell me where I'm mistaken, oversimplifying, or whatever. Most of my experience is with fault-based testing, so problems are more likely here.
The concluding part (perhaps parts) will talk about test implementation and test evaluation.
2. What's Missing?
Part 1's fault-based testing misses two main types of bugs:
· incorrect specifications. The product doesn't do what the customer wants. It might do the wrong thing (the calculator implements base 2 logarithms, not the natural logarithms the target market expects) or it might not do some things (such as sensible handling of user input errors).
· interactions among subsystems. These are bugs where some action over there sets up some state that crashes this subsystem over here.
Fault-based testing, which is local in scope and driven more by the product than the customer, finds such problems only by chance. (There are ways to organize testing to increase that chance, but still not enough.) So another type of testing is needed.
3. Scenario-Based Test Design
This new type of testing concentrates on what the customer does, not what the product does. It means capturing the tasks (use cases, if you will) the customer has to perform, then using them and their variants as tests. Of course, this design work is best done before you've implemented the product. It's really an offshoot of a careful attempt at "requirements elicitation" (aka knowing your customer). These scenarios will also tend to flush out interaction bugs. They are more complex and more realistic than fault based tests often are. They tend to exercise multiple subsystems in a single test, exactly because that's what users do. The tests won't find everything, but they will at least cover the higher visibility interaction bugs.
3.1 An example of finding an incorrect specification
Suppose you were designing scenario-based tests for a text editor. Here's a common scenario:
Fix the Final Draft
Background: it's not unusual to print the "final" draft, read it, and discover some annoying errors that weren't obvious from the on-screen image.
1. Print the entire document.
2. Move around in the document, changing certain pages. As each page is changed, it's printed.
3. Sometimes a series of pages is printed.
This scenario describes two things: a test and some user needs. The user needs are pretty obvious: an easy way to print single pages and a range of pages. As far as testing goes, you know you need to test editing after printing (as well as the reverse). You hope to discover that the printing code causes later editing code to break, that the two chunks of code are not properly independent.
No big deal here. So here's another scenario:
Print a New Copy
Background: someone asks you for a fresh copy of the document. You print one.
1. Open the document.
2. Print it.
3. Close the document.
Again, relatively obvious. Except that this document didn't appear out of nowhere. It was created in an earlier task. Does that task affect this one?
In the FrameMaker editor, documents remember how they were last printed. By default, they print the same way next time. After the "Fix the Final Draft" task, just selecting "Print" in the menu and clicking the "Print" button in the dialog box will cause the last corrected page to print again. So, according to FrameMaker, the correct scenario should look like this:
Print a New Copy
1. Open the document.
2. Select "Print" in the menu.
3. Check if you're printing a page range; if so, click to print the entire document.
4. Click on the Print button.
5. Close the document.
But that's surely wrong. Customers will often overlook that check. They'll be annoyed when they trot off to the printer and find one page when they wanted 100. Annoyed customers signal specification bugs.
You might miss this dependency in test design, but you'd likely stumble across it in testing (just like a normal user would). As a tester, you would then have to push back against the probable response: "that's the way it's supposed to work".
4. Effects of Object-Oriented Programming
Object-oriented programming can have two effects. It certainly changes the architecture of the product (its deep structure). It may have some effect on what the customer sees (the surface structure).
4.1 Surface structure
Object-oriented programming arguably encourages a different style of interface. Rather than performing functions, users may be given objects to fool around with in a direct manipulation kind of way. But whatever the interface, the tests are still based on user tasks. Capturing those will continue to involve understanding, watching, and talking with the representative user (and as many non-representative users as are worth considering).
There will surely be some difference in detail. For example, in a conventional system with a "verbish" interface, you might use the list of all commands as a testing checklist. If you had no test scenarios that exercise a command, you perhaps missed some tasks (or the interface has useless commands). In a "nounish" interface, you might use the list of all objects as a testing checklist.
However, remember that a basic principle of testing is that you must trick yourself into seeing the product in a new way. If the product has a direct manipulation interface, you'll test it better if you pretend functions are independent of objects. You'd ask questions like, "Might the user want to use this function - which applies only to the Scanner object - while working with the Printer?" Whatever the interface style, you should use both objects and functions as clues leading to overlooked tasks. (Note: I use the word "clue" carefully. You should not say, "Oh - I don't have any task that uses the 'printer' object", then write a quick and dirty test that prints. You should discover the various ways in which real users would use the printer object to get real work done.)
4.2 Deep (architectural) structure
Test design based on the surface structure will miss things. User tasks will be overlooked. Important variants that should be tested won't be. Particular subsystem interactions won't be probed.
Looking at the deep structure might reveal those oversights. For example, if Blob of Code A depends on Blob of Code Z, but no test seems to exercise A's use of Z, that's a hint. You may have overlooked a user task.
What is the difference due to OO programming? To answer, look at the ways deep structure can be described. I'll use Booch diagrams as an example. In my admittedly limited experience, the different styles of OO design description seem roughly the same for my purposes.
· A class diagram describes relationships between objects. An object of one class may use or contain an object of another class. That's a clue for test design: have you captured (as a test) some task that exercises that relationship? If not, why not? I suspect that only the highest level class diagrams will be useful for scenario-based testing. You might want only to look at the relationships between class categories (groups of related classes).
· The object diagram and interaction diagram give more detail about relationships between objects. My guess is that most such diagrams will have more detail than is useful for scenario testing.
· A class diagram also shows inheritance structure. Inheritance structure is used in fault-based testing, where you ask questions like:
Routine CALLER's only argument is a reference to some base class. What might happen when CALLER is passed a class derived from it? What are the differences in behavior that could affect CALLER?
But you can also use the inheritance structure in scenario-based testing. If some class BackupDestination has subclasses "Floppy" and "ReallyFastStreamingTapeDrive", you should ask what the differences are between floppies and really fast streaming tape drives. How will users use them differently?
· Statecharts (enhanced state machines) are a way of describing many tasks in a compact picture. If there is a state transition exercised by no test, why not? Again, the detailed statecharts for objects buried deep in the system are not likely to be as useful as those for the major, user-visible objects.
5. Summary
Object-oriented programming changes the bookkeeping of test design, not the approach. Bookkeeping changes for scenario-based testing are minor compared to those required for fault-based testing.
Wednesday, February 20, 2008
Object-Oriented Testing-1
Part 1: Object-Oriented Testing
Fault-Based Test Design
July, 1995
Contents
1. Context
2. OO Test Design
3. Fault-Based Testing
3.1. What is testing based on plausible faults?
3.2. What is integration testing?
3.3. What effects does OO programming have on testing new code?
3.4. Which functions must be tested in a subclass? And how much?
3.5. Can tests for a base class be reused for a derived class?
3.6. What other horrible things can happen as a result of sub-classing?
1. Context
Doing testing means doing three things: designing tests, implementing the tests you've designed, and evaluating those tests. Test design is essential to effectiveness. Test implementation is essential to efficiency. Test evaluation improves effectiveness.
More people are weak at design than at implementation, though that's not always obvious (because implementation problems are more immediate and pressing). I tend to concentrate on design.
Before test design, there is test planning, preferably guided by risk analysis. I'm not going to talk about that, or about other aspects of managing testing.
2. OO Test Design
I believe there are two useful approaches to test design: testing based on plausible faults and testing based on plausible usage. In this first part, I'll discuss the first. In Part 2, I'll discuss the second.
3. Fault-Based Testing
Warning: many people think of testing, at least at the code level, as being about exercising all the lines of code or some set of paths through the code. I don't believe path-based testing is particularly useful. That is, the types of testing I'll discuss do what typical path-based techniques do, and more besides. So I ignore paths. If you want to see how path-based testing is affected by OO, this is not the document for you.
3.1. What is testing based on plausible faults?
This type of testing can be based on the specification (user's manuals, etc.) or the code. It works best when based on both. I'll concentrate on the code. (Whatever else may be missing, you've always got the code.)
I'll start with an example. Programmers make off-by-one errors. So when testing a SQRT function that returns errors for negative numbers, we know to try the boundaries: a negative number close to zero, and zero itself. "Zero itself" checks whether the programmer made a mistake like:
if (x > 0) calculate_the_square_root();
instead of the correct:
if (x >= 0) calculate_the_square_root();
As another example, suppose you have a boolean expression:
if (a && !b || c)
Multicondition testing and related techniques probe for certain plausible faults in that expression, such as:
· The "&&" should be an "||"
· A "!" was left out where it was needed
· There should be parentheses around "!b || c"
For each plausible fault, the technique gives you an input that will make the the given, incorrect, expression evaluate wrong. In the expression above, (a=0, b=0, c=0) will make the expression as given evaluate false. If the "&&" should have been a "||", the code has done the wrong thing; it's now taking the wrong path.
Of course, the effectiveness of these techniques depends on their definitions of "plausible fault". If the real faults are all ones that they consider "implausible", these techniques are no better than any random technique. In practice, they work pretty well.
As a side benefit, such techniques will force every line of code to be executed, discovering the large class of faults that invariably fail. Example:
fprintf("Hello, %s!\n"); // two missing arguments
3.2. What is integration testing?
Integration testing looks for plausible faults in function calls. Here are some examples of faults:
· write(fid, buffer, amount);
The code does not check for error returns from write().
· if (-1 == write(fid, buffer, amount)) error_out();
The code assumes a non-negative return means every byte has been written. That's not necessarily the case. It may be that only part of the buffer was written (in which case write() returns a number less than "amount".)
· ptr = strchr(s, '/');
The programmer should have used "strrchr". As written, the code searches for the first slash when it should search for the last.
· if (strncmp(s1, s2, strlen(s1)))...
The programmer should have used "strlen(s2)".
The above plausible faults are of three types: unexpected result, wrong function used, incorrect invocation. What I call integration test design searches for the first and third types. (I won't talk about the second here.) The search depends on an examination of the behaviors of the called function. In the case of a call to write(), that search would lead to these test requirements:
· invoke write so that it succeeds
· invoke so that write returns -1 (also possibly considering each error type)
· invoke so that write succeeds with partial data written
In the case of any call to strncmp(string1, string2, length), it leads to these test requirements:
· invoke strncmp so that it returns <0
· so that strncmp returns 0
· so that strncmp returns >0
· so that strncmp returns 0. It would have returned <0 had it not been for the length argument.
· so that strncmp returns 0. It would have returned >0 had it not been for the length argument.
Integration testing applies to variables as well as to functions. The "behaviors" of the variable are the different kinds of values it can have. You test whether the using code handles each kind of value. (Further discussion omitted.)
IMPORTANT POINT: Integration testing is about finding faults in the calling code, not the called code. The function call is used as a clue, a way to find test requirements that exercise the calling code.
3.3. What effects does OO programming have on testing new code?
There are several ways OO programming could make a difference:
· some types of fault could become less plausible (not worth testing for)
· some types of fault could become more plausible (worth testing now)
· some new types of fault might appear
I'll discuss the last issue first.
When you stare at a chunk of code in an OO program, a few novelties strike you:
· When you invoke a function, it may be hard to tell exactly what code gets exercised. It may be a method belonging to one of many classes.
· It can be hard to tell the exact type/class of a parameter. When the code accesses it, it may get an unexpected value.
But what's the essential difference? Before OO, when you looked at
x = foo(y);
you had to think about the behaviors of a single function. Now you may have to think about the behaviors of base::foo(), of derived::foo(), and so on. For a single call, you need to explore (or at least think about) the union of all distinct behaviors. (This is easier if you follow good OO design practices, which tend to limit the differences in behavior between base and derived classes.)
The testing approach is essentially the same. The difference is one of bookkeeping. (I don't mean to minimize the importance of bookkeeping. As anyone who's stuggled through the OO chapters in my book knows, the bookkeeping for a sloppy OO design is overwhelming.)
The problem of testing function calls in OO is the same as testing code that takes a function as a parameter and then invokes it. Inheritance is a convenient way of producing polymorphic functions. At the call site, what matters is not the inheritance, but the polymorphism. Inheritance does make the search for test requirements more straightforward.
As with functions, the process of testing variable uses doesn't essentially change, but you have to look in more places to decide what needs testing.
Has the plausibility of faults changed? Are some types of fault now more plausible, some less plausible?
OO functions are generally smaller. That means there are more opportunities for integration faults. They become more likely, more plausible. With non-OO programs, people often get away with not designing tests to probe for integration faults. That's less likely to work with OO programs.
It may be that people can now get away with not designing tests to probe for boundary faults, boolean expression faults, and so on. I don't know, but I doubt it.
3.4. Which functions must be tested in a subclass? And how much?
Suppose you have this situation:
· class "base" contains functions inherited() and redefined().
· class "derived" redefines redefined().
Derived::redefined has to be tested afresh: it's new code. Does derived::inherited() have to be retested? If it calls redefined(), and redefined's behavior changed, derived::inherited() may mishandle the new behavior. So it needs new tests even though the code hasn't changed.
Note that derived::inherited() may not have to be completely tested. If there's code in inherited() that does not depend on redefined() that does not call it, nor call any code that indirectly calls it that code need not be retested in the derived class.
Here is an example:
3.5. Can tests for a base class be reused for a derived class?
Base::redefined() and derived::redefined() are two different functions with different specifications and implementations. They would each have a set of test requirements derived from the specification and implementation. Those test requirements probe for plausible faults: integration faults, condition faults, boundary faults, etc.
But the functions are likely to be similar. Their sets of test requirements will overlap. The better the OO design, the greater the overlap. You only need to write new tests for those derived::redefined requirements that are not satisfied by the base::redefined tests.
Notes:
· You have to apply the base::redefined tests to objects of class "derived".
· The test inputs may be appropriate for both classes, but the expected results might differ in the derived class.
Here is an example:
There's a problem here, though. The simpler a test, the more likely it is to be reusable in subclasses. But simple tests tend to find only the faults you specifically target; complex tests are better at finding both those faults and also stumbling across other faults by sheer dumb luck. There's a tradeoff here, one of many between simple and complex tests.
3.6. What other horrible things can happen as a result of subclassing?
Suppose you add the "derived" class. There may be a whole lot of code that looks like
mumble = x->redefined();
...
Before, that code could only be tested against the behaviors of base::redefined, the only version of redefined() that existed. Now you (potentially) have to look at all code that uses redefined(), decide if it could be passed a "derived", then test it with test requirements that describe the difference between the behaviors of base::redefined() and derived::redefined().
Here's an example:
Fault-Based Test Design
July, 1995
Contents
1. Context
2. OO Test Design
3. Fault-Based Testing
3.1. What is testing based on plausible faults?
3.2. What is integration testing?
3.3. What effects does OO programming have on testing new code?
3.4. Which functions must be tested in a subclass? And how much?
3.5. Can tests for a base class be reused for a derived class?
3.6. What other horrible things can happen as a result of sub-classing?
1. Context
Doing testing means doing three things: designing tests, implementing the tests you've designed, and evaluating those tests. Test design is essential to effectiveness. Test implementation is essential to efficiency. Test evaluation improves effectiveness.
More people are weak at design than at implementation, though that's not always obvious (because implementation problems are more immediate and pressing). I tend to concentrate on design.
Before test design, there is test planning, preferably guided by risk analysis. I'm not going to talk about that, or about other aspects of managing testing.
2. OO Test Design
I believe there are two useful approaches to test design: testing based on plausible faults and testing based on plausible usage. In this first part, I'll discuss the first. In Part 2, I'll discuss the second.
3. Fault-Based Testing
Warning: many people think of testing, at least at the code level, as being about exercising all the lines of code or some set of paths through the code. I don't believe path-based testing is particularly useful. That is, the types of testing I'll discuss do what typical path-based techniques do, and more besides. So I ignore paths. If you want to see how path-based testing is affected by OO, this is not the document for you.
3.1. What is testing based on plausible faults?
This type of testing can be based on the specification (user's manuals, etc.) or the code. It works best when based on both. I'll concentrate on the code. (Whatever else may be missing, you've always got the code.)
I'll start with an example. Programmers make off-by-one errors. So when testing a SQRT function that returns errors for negative numbers, we know to try the boundaries: a negative number close to zero, and zero itself. "Zero itself" checks whether the programmer made a mistake like:
if (x > 0) calculate_the_square_root();
instead of the correct:
if (x >= 0) calculate_the_square_root();
As another example, suppose you have a boolean expression:
if (a && !b || c)
Multicondition testing and related techniques probe for certain plausible faults in that expression, such as:
· The "&&" should be an "||"
· A "!" was left out where it was needed
· There should be parentheses around "!b || c"
For each plausible fault, the technique gives you an input that will make the the given, incorrect, expression evaluate wrong. In the expression above, (a=0, b=0, c=0) will make the expression as given evaluate false. If the "&&" should have been a "||", the code has done the wrong thing; it's now taking the wrong path.
Of course, the effectiveness of these techniques depends on their definitions of "plausible fault". If the real faults are all ones that they consider "implausible", these techniques are no better than any random technique. In practice, they work pretty well.
As a side benefit, such techniques will force every line of code to be executed, discovering the large class of faults that invariably fail. Example:
fprintf("Hello, %s!\n"); // two missing arguments
3.2. What is integration testing?
Integration testing looks for plausible faults in function calls. Here are some examples of faults:
· write(fid, buffer, amount);
The code does not check for error returns from write().
· if (-1 == write(fid, buffer, amount)) error_out();
The code assumes a non-negative return means every byte has been written. That's not necessarily the case. It may be that only part of the buffer was written (in which case write() returns a number less than "amount".)
· ptr = strchr(s, '/');
The programmer should have used "strrchr". As written, the code searches for the first slash when it should search for the last.
· if (strncmp(s1, s2, strlen(s1)))...
The programmer should have used "strlen(s2)".
The above plausible faults are of three types: unexpected result, wrong function used, incorrect invocation. What I call integration test design searches for the first and third types. (I won't talk about the second here.) The search depends on an examination of the behaviors of the called function. In the case of a call to write(), that search would lead to these test requirements:
· invoke write so that it succeeds
· invoke so that write returns -1 (also possibly considering each error type)
· invoke so that write succeeds with partial data written
In the case of any call to strncmp(string1, string2, length), it leads to these test requirements:
· invoke strncmp so that it returns <0
· so that strncmp returns 0
· so that strncmp returns >0
· so that strncmp returns 0. It would have returned <0 had it not been for the length argument.
· so that strncmp returns 0. It would have returned >0 had it not been for the length argument.
Integration testing applies to variables as well as to functions. The "behaviors" of the variable are the different kinds of values it can have. You test whether the using code handles each kind of value. (Further discussion omitted.)
IMPORTANT POINT: Integration testing is about finding faults in the calling code, not the called code. The function call is used as a clue, a way to find test requirements that exercise the calling code.
3.3. What effects does OO programming have on testing new code?
There are several ways OO programming could make a difference:
· some types of fault could become less plausible (not worth testing for)
· some types of fault could become more plausible (worth testing now)
· some new types of fault might appear
I'll discuss the last issue first.
When you stare at a chunk of code in an OO program, a few novelties strike you:
· When you invoke a function, it may be hard to tell exactly what code gets exercised. It may be a method belonging to one of many classes.
· It can be hard to tell the exact type/class of a parameter. When the code accesses it, it may get an unexpected value.
But what's the essential difference? Before OO, when you looked at
x = foo(y);
you had to think about the behaviors of a single function. Now you may have to think about the behaviors of base::foo(), of derived::foo(), and so on. For a single call, you need to explore (or at least think about) the union of all distinct behaviors. (This is easier if you follow good OO design practices, which tend to limit the differences in behavior between base and derived classes.)
The testing approach is essentially the same. The difference is one of bookkeeping. (I don't mean to minimize the importance of bookkeeping. As anyone who's stuggled through the OO chapters in my book knows, the bookkeeping for a sloppy OO design is overwhelming.)
The problem of testing function calls in OO is the same as testing code that takes a function as a parameter and then invokes it. Inheritance is a convenient way of producing polymorphic functions. At the call site, what matters is not the inheritance, but the polymorphism. Inheritance does make the search for test requirements more straightforward.
As with functions, the process of testing variable uses doesn't essentially change, but you have to look in more places to decide what needs testing.
Has the plausibility of faults changed? Are some types of fault now more plausible, some less plausible?
OO functions are generally smaller. That means there are more opportunities for integration faults. They become more likely, more plausible. With non-OO programs, people often get away with not designing tests to probe for integration faults. That's less likely to work with OO programs.
It may be that people can now get away with not designing tests to probe for boundary faults, boolean expression faults, and so on. I don't know, but I doubt it.
3.4. Which functions must be tested in a subclass? And how much?
Suppose you have this situation:
· class "base" contains functions inherited() and redefined().
· class "derived" redefines redefined().
Derived::redefined has to be tested afresh: it's new code. Does derived::inherited() have to be retested? If it calls redefined(), and redefined's behavior changed, derived::inherited() may mishandle the new behavior. So it needs new tests even though the code hasn't changed.
Note that derived::inherited() may not have to be completely tested. If there's code in inherited() that does not depend on redefined() that does not call it, nor call any code that indirectly calls it that code need not be retested in the derived class.
Here is an example:
3.5. Can tests for a base class be reused for a derived class?
Base::redefined() and derived::redefined() are two different functions with different specifications and implementations. They would each have a set of test requirements derived from the specification and implementation. Those test requirements probe for plausible faults: integration faults, condition faults, boundary faults, etc.
But the functions are likely to be similar. Their sets of test requirements will overlap. The better the OO design, the greater the overlap. You only need to write new tests for those derived::redefined requirements that are not satisfied by the base::redefined tests.
Notes:
· You have to apply the base::redefined tests to objects of class "derived".
· The test inputs may be appropriate for both classes, but the expected results might differ in the derived class.
Here is an example:
There's a problem here, though. The simpler a test, the more likely it is to be reusable in subclasses. But simple tests tend to find only the faults you specifically target; complex tests are better at finding both those faults and also stumbling across other faults by sheer dumb luck. There's a tradeoff here, one of many between simple and complex tests.
3.6. What other horrible things can happen as a result of subclassing?
Suppose you add the "derived" class. There may be a whole lot of code that looks like
mumble = x->redefined();
...
Before, that code could only be tested against the behaviors of base::redefined, the only version of redefined() that existed. Now you (potentially) have to look at all code that uses redefined(), decide if it could be passed a "derived", then test it with test requirements that describe the difference between the behaviors of base::redefined() and derived::redefined().
Here's an example:
Subscribe to:
Posts (Atom)