Part 2: Object-Oriented Testing
Scenario-Based Test Design
Contents
1. Recapitulation
2. What's Missing?
3. Scenario-Based Test Design
3.1 An example of finding an incorrect specification
4. Effects of Object-Oriented Programming
4.1 Surface structure
4.2 Deep (architectural) structure
5. Summary
1. Recapitulation
In Part 1, I described one type of test design that targets plausible faults. Testing techniques search the code or specification for particular clues of interest, then describe how those clues should be tested. The clues in an object-oriented program are the same as in a conventional program. Divining what to test requires either more work or more bookkeeping, because there are more places where relevant information might live. But the approach is essentially the same.
I am posting these notes so that people can tell me where I'm mistaken, oversimplifying, or whatever. Most of my experience is with fault-based testing, so problems are more likely here.
The concluding part (perhaps parts) will talk about test implementation and test evaluation.
2. What's Missing?
Part 1's fault-based testing misses two main types of bugs:
· incorrect specifications. The product doesn't do what the customer wants. It might do the wrong thing (the calculator implements base 2 logarithms, not the natural logarithms the target market expects) or it might not do some things (such as sensible handling of user input errors).
· interactions among subsystems. These are bugs where some action over there sets up some state that crashes this subsystem over here.
Fault-based testing, which is local in scope and driven more by the product than the customer, finds such problems only by chance. (There are ways to organize testing to increase that chance, but still not enough.) So another type of testing is needed.
3. Scenario-Based Test Design
This new type of testing concentrates on what the customer does, not what the product does. It means capturing the tasks (use cases, if you will) the customer has to perform, then using them and their variants as tests. Of course, this design work is best done before you've implemented the product. It's really an offshoot of a careful attempt at "requirements elicitation" (aka knowing your customer). These scenarios will also tend to flush out interaction bugs. They are more complex and more realistic than fault based tests often are. They tend to exercise multiple subsystems in a single test, exactly because that's what users do. The tests won't find everything, but they will at least cover the higher visibility interaction bugs.
3.1 An example of finding an incorrect specification
Suppose you were designing scenario-based tests for a text editor. Here's a common scenario:
Fix the Final Draft
Background: it's not unusual to print the "final" draft, read it, and discover some annoying errors that weren't obvious from the on-screen image.
1. Print the entire document.
2. Move around in the document, changing certain pages. As each page is changed, it's printed.
3. Sometimes a series of pages is printed.
This scenario describes two things: a test and some user needs. The user needs are pretty obvious: an easy way to print single pages and a range of pages. As far as testing goes, you know you need to test editing after printing (as well as the reverse). You hope to discover that the printing code causes later editing code to break, that the two chunks of code are not properly independent.
No big deal here. So here's another scenario:
Print a New Copy
Background: someone asks you for a fresh copy of the document. You print one.
1. Open the document.
2. Print it.
3. Close the document.
Again, relatively obvious. Except that this document didn't appear out of nowhere. It was created in an earlier task. Does that task affect this one?
In the FrameMaker editor, documents remember how they were last printed. By default, they print the same way next time. After the "Fix the Final Draft" task, just selecting "Print" in the menu and clicking the "Print" button in the dialog box will cause the last corrected page to print again. So, according to FrameMaker, the correct scenario should look like this:
Print a New Copy
1. Open the document.
2. Select "Print" in the menu.
3. Check if you're printing a page range; if so, click to print the entire document.
4. Click on the Print button.
5. Close the document.
But that's surely wrong. Customers will often overlook that check. They'll be annoyed when they trot off to the printer and find one page when they wanted 100. Annoyed customers signal specification bugs.
You might miss this dependency in test design, but you'd likely stumble across it in testing (just like a normal user would). As a tester, you would then have to push back against the probable response: "that's the way it's supposed to work".
4. Effects of Object-Oriented Programming
Object-oriented programming can have two effects. It certainly changes the architecture of the product (its deep structure). It may have some effect on what the customer sees (the surface structure).
4.1 Surface structure
Object-oriented programming arguably encourages a different style of interface. Rather than performing functions, users may be given objects to fool around with in a direct manipulation kind of way. But whatever the interface, the tests are still based on user tasks. Capturing those will continue to involve understanding, watching, and talking with the representative user (and as many non-representative users as are worth considering).
There will surely be some difference in detail. For example, in a conventional system with a "verbish" interface, you might use the list of all commands as a testing checklist. If you had no test scenarios that exercise a command, you perhaps missed some tasks (or the interface has useless commands). In a "nounish" interface, you might use the list of all objects as a testing checklist.
However, remember that a basic principle of testing is that you must trick yourself into seeing the product in a new way. If the product has a direct manipulation interface, you'll test it better if you pretend functions are independent of objects. You'd ask questions like, "Might the user want to use this function - which applies only to the Scanner object - while working with the Printer?" Whatever the interface style, you should use both objects and functions as clues leading to overlooked tasks. (Note: I use the word "clue" carefully. You should not say, "Oh - I don't have any task that uses the 'printer' object", then write a quick and dirty test that prints. You should discover the various ways in which real users would use the printer object to get real work done.)
4.2 Deep (architectural) structure
Test design based on the surface structure will miss things. User tasks will be overlooked. Important variants that should be tested won't be. Particular subsystem interactions won't be probed.
Looking at the deep structure might reveal those oversights. For example, if Blob of Code A depends on Blob of Code Z, but no test seems to exercise A's use of Z, that's a hint. You may have overlooked a user task.
What is the difference due to OO programming? To answer, look at the ways deep structure can be described. I'll use Booch diagrams as an example. In my admittedly limited experience, the different styles of OO design description seem roughly the same for my purposes.
· A class diagram describes relationships between objects. An object of one class may use or contain an object of another class. That's a clue for test design: have you captured (as a test) some task that exercises that relationship? If not, why not? I suspect that only the highest level class diagrams will be useful for scenario-based testing. You might want only to look at the relationships between class categories (groups of related classes).
· The object diagram and interaction diagram give more detail about relationships between objects. My guess is that most such diagrams will have more detail than is useful for scenario testing.
· A class diagram also shows inheritance structure. Inheritance structure is used in fault-based testing, where you ask questions like:
Routine CALLER's only argument is a reference to some base class. What might happen when CALLER is passed a class derived from it? What are the differences in behavior that could affect CALLER?
But you can also use the inheritance structure in scenario-based testing. If some class BackupDestination has subclasses "Floppy" and "ReallyFastStreamingTapeDrive", you should ask what the differences are between floppies and really fast streaming tape drives. How will users use them differently?
· Statecharts (enhanced state machines) are a way of describing many tasks in a compact picture. If there is a state transition exercised by no test, why not? Again, the detailed statecharts for objects buried deep in the system are not likely to be as useful as those for the major, user-visible objects.
5. Summary
Object-oriented programming changes the bookkeeping of test design, not the approach. Bookkeeping changes for scenario-based testing are minor compared to those required for fault-based testing.
Wednesday, February 20, 2008
Object-Oriented Testing-1
Part 1: Object-Oriented Testing
Fault-Based Test Design
July, 1995
Contents
1. Context
2. OO Test Design
3. Fault-Based Testing
3.1. What is testing based on plausible faults?
3.2. What is integration testing?
3.3. What effects does OO programming have on testing new code?
3.4. Which functions must be tested in a subclass? And how much?
3.5. Can tests for a base class be reused for a derived class?
3.6. What other horrible things can happen as a result of sub-classing?
1. Context
Doing testing means doing three things: designing tests, implementing the tests you've designed, and evaluating those tests. Test design is essential to effectiveness. Test implementation is essential to efficiency. Test evaluation improves effectiveness.
More people are weak at design than at implementation, though that's not always obvious (because implementation problems are more immediate and pressing). I tend to concentrate on design.
Before test design, there is test planning, preferably guided by risk analysis. I'm not going to talk about that, or about other aspects of managing testing.
2. OO Test Design
I believe there are two useful approaches to test design: testing based on plausible faults and testing based on plausible usage. In this first part, I'll discuss the first. In Part 2, I'll discuss the second.
3. Fault-Based Testing
Warning: many people think of testing, at least at the code level, as being about exercising all the lines of code or some set of paths through the code. I don't believe path-based testing is particularly useful. That is, the types of testing I'll discuss do what typical path-based techniques do, and more besides. So I ignore paths. If you want to see how path-based testing is affected by OO, this is not the document for you.
3.1. What is testing based on plausible faults?
This type of testing can be based on the specification (user's manuals, etc.) or the code. It works best when based on both. I'll concentrate on the code. (Whatever else may be missing, you've always got the code.)
I'll start with an example. Programmers make off-by-one errors. So when testing a SQRT function that returns errors for negative numbers, we know to try the boundaries: a negative number close to zero, and zero itself. "Zero itself" checks whether the programmer made a mistake like:
if (x > 0) calculate_the_square_root();
instead of the correct:
if (x >= 0) calculate_the_square_root();
As another example, suppose you have a boolean expression:
if (a && !b || c)
Multicondition testing and related techniques probe for certain plausible faults in that expression, such as:
· The "&&" should be an "||"
· A "!" was left out where it was needed
· There should be parentheses around "!b || c"
For each plausible fault, the technique gives you an input that will make the the given, incorrect, expression evaluate wrong. In the expression above, (a=0, b=0, c=0) will make the expression as given evaluate false. If the "&&" should have been a "||", the code has done the wrong thing; it's now taking the wrong path.
Of course, the effectiveness of these techniques depends on their definitions of "plausible fault". If the real faults are all ones that they consider "implausible", these techniques are no better than any random technique. In practice, they work pretty well.
As a side benefit, such techniques will force every line of code to be executed, discovering the large class of faults that invariably fail. Example:
fprintf("Hello, %s!\n"); // two missing arguments
3.2. What is integration testing?
Integration testing looks for plausible faults in function calls. Here are some examples of faults:
· write(fid, buffer, amount);
The code does not check for error returns from write().
· if (-1 == write(fid, buffer, amount)) error_out();
The code assumes a non-negative return means every byte has been written. That's not necessarily the case. It may be that only part of the buffer was written (in which case write() returns a number less than "amount".)
· ptr = strchr(s, '/');
The programmer should have used "strrchr". As written, the code searches for the first slash when it should search for the last.
· if (strncmp(s1, s2, strlen(s1)))...
The programmer should have used "strlen(s2)".
The above plausible faults are of three types: unexpected result, wrong function used, incorrect invocation. What I call integration test design searches for the first and third types. (I won't talk about the second here.) The search depends on an examination of the behaviors of the called function. In the case of a call to write(), that search would lead to these test requirements:
· invoke write so that it succeeds
· invoke so that write returns -1 (also possibly considering each error type)
· invoke so that write succeeds with partial data written
In the case of any call to strncmp(string1, string2, length), it leads to these test requirements:
· invoke strncmp so that it returns <0
· so that strncmp returns 0
· so that strncmp returns >0
· so that strncmp returns 0. It would have returned <0 had it not been for the length argument.
· so that strncmp returns 0. It would have returned >0 had it not been for the length argument.
Integration testing applies to variables as well as to functions. The "behaviors" of the variable are the different kinds of values it can have. You test whether the using code handles each kind of value. (Further discussion omitted.)
IMPORTANT POINT: Integration testing is about finding faults in the calling code, not the called code. The function call is used as a clue, a way to find test requirements that exercise the calling code.
3.3. What effects does OO programming have on testing new code?
There are several ways OO programming could make a difference:
· some types of fault could become less plausible (not worth testing for)
· some types of fault could become more plausible (worth testing now)
· some new types of fault might appear
I'll discuss the last issue first.
When you stare at a chunk of code in an OO program, a few novelties strike you:
· When you invoke a function, it may be hard to tell exactly what code gets exercised. It may be a method belonging to one of many classes.
· It can be hard to tell the exact type/class of a parameter. When the code accesses it, it may get an unexpected value.
But what's the essential difference? Before OO, when you looked at
x = foo(y);
you had to think about the behaviors of a single function. Now you may have to think about the behaviors of base::foo(), of derived::foo(), and so on. For a single call, you need to explore (or at least think about) the union of all distinct behaviors. (This is easier if you follow good OO design practices, which tend to limit the differences in behavior between base and derived classes.)
The testing approach is essentially the same. The difference is one of bookkeeping. (I don't mean to minimize the importance of bookkeeping. As anyone who's stuggled through the OO chapters in my book knows, the bookkeeping for a sloppy OO design is overwhelming.)
The problem of testing function calls in OO is the same as testing code that takes a function as a parameter and then invokes it. Inheritance is a convenient way of producing polymorphic functions. At the call site, what matters is not the inheritance, but the polymorphism. Inheritance does make the search for test requirements more straightforward.
As with functions, the process of testing variable uses doesn't essentially change, but you have to look in more places to decide what needs testing.
Has the plausibility of faults changed? Are some types of fault now more plausible, some less plausible?
OO functions are generally smaller. That means there are more opportunities for integration faults. They become more likely, more plausible. With non-OO programs, people often get away with not designing tests to probe for integration faults. That's less likely to work with OO programs.
It may be that people can now get away with not designing tests to probe for boundary faults, boolean expression faults, and so on. I don't know, but I doubt it.
3.4. Which functions must be tested in a subclass? And how much?
Suppose you have this situation:
· class "base" contains functions inherited() and redefined().
· class "derived" redefines redefined().
Derived::redefined has to be tested afresh: it's new code. Does derived::inherited() have to be retested? If it calls redefined(), and redefined's behavior changed, derived::inherited() may mishandle the new behavior. So it needs new tests even though the code hasn't changed.
Note that derived::inherited() may not have to be completely tested. If there's code in inherited() that does not depend on redefined() that does not call it, nor call any code that indirectly calls it that code need not be retested in the derived class.
Here is an example:
3.5. Can tests for a base class be reused for a derived class?
Base::redefined() and derived::redefined() are two different functions with different specifications and implementations. They would each have a set of test requirements derived from the specification and implementation. Those test requirements probe for plausible faults: integration faults, condition faults, boundary faults, etc.
But the functions are likely to be similar. Their sets of test requirements will overlap. The better the OO design, the greater the overlap. You only need to write new tests for those derived::redefined requirements that are not satisfied by the base::redefined tests.
Notes:
· You have to apply the base::redefined tests to objects of class "derived".
· The test inputs may be appropriate for both classes, but the expected results might differ in the derived class.
Here is an example:
There's a problem here, though. The simpler a test, the more likely it is to be reusable in subclasses. But simple tests tend to find only the faults you specifically target; complex tests are better at finding both those faults and also stumbling across other faults by sheer dumb luck. There's a tradeoff here, one of many between simple and complex tests.
3.6. What other horrible things can happen as a result of subclassing?
Suppose you add the "derived" class. There may be a whole lot of code that looks like
mumble = x->redefined();
...
Before, that code could only be tested against the behaviors of base::redefined, the only version of redefined() that existed. Now you (potentially) have to look at all code that uses redefined(), decide if it could be passed a "derived", then test it with test requirements that describe the difference between the behaviors of base::redefined() and derived::redefined().
Here's an example:
Fault-Based Test Design
July, 1995
Contents
1. Context
2. OO Test Design
3. Fault-Based Testing
3.1. What is testing based on plausible faults?
3.2. What is integration testing?
3.3. What effects does OO programming have on testing new code?
3.4. Which functions must be tested in a subclass? And how much?
3.5. Can tests for a base class be reused for a derived class?
3.6. What other horrible things can happen as a result of sub-classing?
1. Context
Doing testing means doing three things: designing tests, implementing the tests you've designed, and evaluating those tests. Test design is essential to effectiveness. Test implementation is essential to efficiency. Test evaluation improves effectiveness.
More people are weak at design than at implementation, though that's not always obvious (because implementation problems are more immediate and pressing). I tend to concentrate on design.
Before test design, there is test planning, preferably guided by risk analysis. I'm not going to talk about that, or about other aspects of managing testing.
2. OO Test Design
I believe there are two useful approaches to test design: testing based on plausible faults and testing based on plausible usage. In this first part, I'll discuss the first. In Part 2, I'll discuss the second.
3. Fault-Based Testing
Warning: many people think of testing, at least at the code level, as being about exercising all the lines of code or some set of paths through the code. I don't believe path-based testing is particularly useful. That is, the types of testing I'll discuss do what typical path-based techniques do, and more besides. So I ignore paths. If you want to see how path-based testing is affected by OO, this is not the document for you.
3.1. What is testing based on plausible faults?
This type of testing can be based on the specification (user's manuals, etc.) or the code. It works best when based on both. I'll concentrate on the code. (Whatever else may be missing, you've always got the code.)
I'll start with an example. Programmers make off-by-one errors. So when testing a SQRT function that returns errors for negative numbers, we know to try the boundaries: a negative number close to zero, and zero itself. "Zero itself" checks whether the programmer made a mistake like:
if (x > 0) calculate_the_square_root();
instead of the correct:
if (x >= 0) calculate_the_square_root();
As another example, suppose you have a boolean expression:
if (a && !b || c)
Multicondition testing and related techniques probe for certain plausible faults in that expression, such as:
· The "&&" should be an "||"
· A "!" was left out where it was needed
· There should be parentheses around "!b || c"
For each plausible fault, the technique gives you an input that will make the the given, incorrect, expression evaluate wrong. In the expression above, (a=0, b=0, c=0) will make the expression as given evaluate false. If the "&&" should have been a "||", the code has done the wrong thing; it's now taking the wrong path.
Of course, the effectiveness of these techniques depends on their definitions of "plausible fault". If the real faults are all ones that they consider "implausible", these techniques are no better than any random technique. In practice, they work pretty well.
As a side benefit, such techniques will force every line of code to be executed, discovering the large class of faults that invariably fail. Example:
fprintf("Hello, %s!\n"); // two missing arguments
3.2. What is integration testing?
Integration testing looks for plausible faults in function calls. Here are some examples of faults:
· write(fid, buffer, amount);
The code does not check for error returns from write().
· if (-1 == write(fid, buffer, amount)) error_out();
The code assumes a non-negative return means every byte has been written. That's not necessarily the case. It may be that only part of the buffer was written (in which case write() returns a number less than "amount".)
· ptr = strchr(s, '/');
The programmer should have used "strrchr". As written, the code searches for the first slash when it should search for the last.
· if (strncmp(s1, s2, strlen(s1)))...
The programmer should have used "strlen(s2)".
The above plausible faults are of three types: unexpected result, wrong function used, incorrect invocation. What I call integration test design searches for the first and third types. (I won't talk about the second here.) The search depends on an examination of the behaviors of the called function. In the case of a call to write(), that search would lead to these test requirements:
· invoke write so that it succeeds
· invoke so that write returns -1 (also possibly considering each error type)
· invoke so that write succeeds with partial data written
In the case of any call to strncmp(string1, string2, length), it leads to these test requirements:
· invoke strncmp so that it returns <0
· so that strncmp returns 0
· so that strncmp returns >0
· so that strncmp returns 0. It would have returned <0 had it not been for the length argument.
· so that strncmp returns 0. It would have returned >0 had it not been for the length argument.
Integration testing applies to variables as well as to functions. The "behaviors" of the variable are the different kinds of values it can have. You test whether the using code handles each kind of value. (Further discussion omitted.)
IMPORTANT POINT: Integration testing is about finding faults in the calling code, not the called code. The function call is used as a clue, a way to find test requirements that exercise the calling code.
3.3. What effects does OO programming have on testing new code?
There are several ways OO programming could make a difference:
· some types of fault could become less plausible (not worth testing for)
· some types of fault could become more plausible (worth testing now)
· some new types of fault might appear
I'll discuss the last issue first.
When you stare at a chunk of code in an OO program, a few novelties strike you:
· When you invoke a function, it may be hard to tell exactly what code gets exercised. It may be a method belonging to one of many classes.
· It can be hard to tell the exact type/class of a parameter. When the code accesses it, it may get an unexpected value.
But what's the essential difference? Before OO, when you looked at
x = foo(y);
you had to think about the behaviors of a single function. Now you may have to think about the behaviors of base::foo(), of derived::foo(), and so on. For a single call, you need to explore (or at least think about) the union of all distinct behaviors. (This is easier if you follow good OO design practices, which tend to limit the differences in behavior between base and derived classes.)
The testing approach is essentially the same. The difference is one of bookkeeping. (I don't mean to minimize the importance of bookkeeping. As anyone who's stuggled through the OO chapters in my book knows, the bookkeeping for a sloppy OO design is overwhelming.)
The problem of testing function calls in OO is the same as testing code that takes a function as a parameter and then invokes it. Inheritance is a convenient way of producing polymorphic functions. At the call site, what matters is not the inheritance, but the polymorphism. Inheritance does make the search for test requirements more straightforward.
As with functions, the process of testing variable uses doesn't essentially change, but you have to look in more places to decide what needs testing.
Has the plausibility of faults changed? Are some types of fault now more plausible, some less plausible?
OO functions are generally smaller. That means there are more opportunities for integration faults. They become more likely, more plausible. With non-OO programs, people often get away with not designing tests to probe for integration faults. That's less likely to work with OO programs.
It may be that people can now get away with not designing tests to probe for boundary faults, boolean expression faults, and so on. I don't know, but I doubt it.
3.4. Which functions must be tested in a subclass? And how much?
Suppose you have this situation:
· class "base" contains functions inherited() and redefined().
· class "derived" redefines redefined().
Derived::redefined has to be tested afresh: it's new code. Does derived::inherited() have to be retested? If it calls redefined(), and redefined's behavior changed, derived::inherited() may mishandle the new behavior. So it needs new tests even though the code hasn't changed.
Note that derived::inherited() may not have to be completely tested. If there's code in inherited() that does not depend on redefined() that does not call it, nor call any code that indirectly calls it that code need not be retested in the derived class.
Here is an example:
3.5. Can tests for a base class be reused for a derived class?
Base::redefined() and derived::redefined() are two different functions with different specifications and implementations. They would each have a set of test requirements derived from the specification and implementation. Those test requirements probe for plausible faults: integration faults, condition faults, boundary faults, etc.
But the functions are likely to be similar. Their sets of test requirements will overlap. The better the OO design, the greater the overlap. You only need to write new tests for those derived::redefined requirements that are not satisfied by the base::redefined tests.
Notes:
· You have to apply the base::redefined tests to objects of class "derived".
· The test inputs may be appropriate for both classes, but the expected results might differ in the derived class.
Here is an example:
There's a problem here, though. The simpler a test, the more likely it is to be reusable in subclasses. But simple tests tend to find only the faults you specifically target; complex tests are better at finding both those faults and also stumbling across other faults by sheer dumb luck. There's a tradeoff here, one of many between simple and complex tests.
3.6. What other horrible things can happen as a result of subclassing?
Suppose you add the "derived" class. There may be a whole lot of code that looks like
mumble = x->redefined();
...
Before, that code could only be tested against the behaviors of base::redefined, the only version of redefined() that existed. Now you (potentially) have to look at all code that uses redefined(), decide if it could be passed a "derived", then test it with test requirements that describe the difference between the behaviors of base::redefined() and derived::redefined().
Here's an example:
Data-Driven Testing
Data-Driven Testing
Introduction
When deciding to automate the testing procedure of an application one expected benefit is that the automatic tests have the same life-time as the application under test and don't need to be rewritten all the time. Additionally it should be possible to extend existing tests easily along with the application's development.
To develop tests which meet these requirements, it isn't enough to just record a test and use it like this. In most cases it will be necessary to edit the test script to refactor its code and do some other changes. Such a change might be to separate the test data which has been recorded (like the values entered in line edits) from the code by introducing data-driven testing. This will also allow to extend test cases later by just adding new data to the test data without changing the actual test script code.
One requirement for data-driven testing is of course that the test tool in use goes beyond straight event recording and replaying and offers a full-fledged scripting language for test scripts. This is the case for Squish, which currently offers its users to choose between the Tcl and Python scripting languages. In this article we will use Python.
This article will show how to create such a data-driven test and also how to execute an external tool from a test script to take over tasks such as comparing files.
What exactly is data-driven testing?
Before we go on to implement a data-driven test, I'd like to give a short introduction about the idea behind data-driven testing. Basically it means that a test case contains data files which the application under test (AUT) will use in some way during the test. In the simplest case these might be files the application needs to read during the test execution. In such a case, the test framework should offer an API to manage such data files.
A more sophisticated way of data-driven testing is that the testing framework actually understands the contents of the data files and it is possible to iterate over the contents in the test script. This is useful to specify the data which should be entered into input fields, etc. during the test. Instead of hard-coding this data into the test script, it will be stored in records in a data file which can be easily extended with new data. Such data might also be read from a database or any other data source.
Setting up the test
We need to think about what should happen in the test case so we can record the parts which we don't want to code manually. As an example, we will create a test for the textedit example which can be found in the examples directory of Qt. Afterwards we will edit the generated script to refactor its code and make it data-driven.
We won't discuss the details about setting up a test suite here since this is already covered extensively in Squish's manual. The only step I want to stress here is to choose Python as scripting language in the test suite settings of the test suite. We call the test suite suite_textedit and create a test case called tst_simpleinput.
Recording the GUI interactions
In this test case we want to open a file, edit the file, save it and compare the result against an expected output. The input file will be a test data file, the commands to edit the file will also be specified in a test data file along with the expected result. This way the test can be easily extended in the future by just adding more test data. To compare the saved output file against the expected result, we will use the tool diff, which is installed on any Unix system and which is available on Windows as well, e.g. through Cygwin. But first we will record the GUI interactions and then we will go on and modify the generated script.
Already for the recording we need a test data file which we will open in the test case. So we create a file called simple.html in the test case's testdata directory with the following contents:
First paragraph
Bold paragraph
Now when recording, in the record settings dialog which will pop up, we can choose this test data file so it will be copied to the AUT side automatically. Now we record the following test flow:
1. Choose File->Open in the menu
2. Choose the file simple.html in the file dialog by double clicking on it
3. Enter something in the editor (this part of the test will be modified later on anyway, so the entered text doesn't matter)
4. Choose File->Save As in the menu
5. Enter out.html as filename in the file dialog
6. Press Return in the file dialog
7. Choose File->Close in the menu
8. Choose File->Exit in the menu
This will generate a test script like the following:
import time def main(): testData.put("simple.html") time.sleep(0.007) sendEvent("QMoveEvent", ":Richtext Editor", 756, 218, 665, 649) time.sleep(0.047) clickButton(":Richtext Editor.qt_top_dock.File Actions.fileOpen_action_button") time.sleep(0.189) sendEvent("QMoveEvent", ":Richtext Editor.qt_filedlg_gofn", 800, 423, 805, 261) time.sleep(0.1) doubleClickItem(":Richtext Editor.qt_filedlg_gofn.qt_splitter.files and more files.filelistbox"+\ ".qt_viewport", "simple.html", 48, 11, 0, Qt.LeftButton) time.sleep(2.14) type(":Richtext Editor.QTabWidget1.tab pages.QTextEdit2", "Hello") time.sleep(2.876) activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Save As...") time.sleep(0.157) sendEvent("QMoveEvent", ":Richtext Editor.qt_filedlg_gsfn", 800, 423, 788, 325) time.sleep(1.715) type(":Richtext Editor.qt_filedlg_gsfn.name/filter editor", "out.html") time.sleep(0.484) type(":Richtext Editor.qt_filedlg_gsfn.name/filter editor", "") time.sleep(1.654) activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Close") time.sleep(2.876) activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Exit") Setting up the test data
Now we have the basic test script which we can edit to use test data. But before we can do this, we need to set up the test data itself. We will use the following files as an example:
- tests.tsv Specifies the input files and the related expected output files.
- inputwords.tsv Specifies input commands for a test which enters some words in the editor
- expwords.tsv Contains the expected output from the inputwords.tsv test.
- inputpara.tsv Specifies input commands for a test which inserts a new paragraph in the editor
- exppara.tsv Contains the expected output from the inputpara.tsv test.
The contents of tests.tsv looks like this (note that the fields are separates by tabs, not by spaces):
CommandFile ExpectedOutputinputwords.tsv expwords.tsvinputpara.tsv exppara.tsv The first line specifies the column names which will be later used in the test script to access a field in a record. The other lines contain the data. In our case these are the input and expected output files.
We won't show the contents of all files here. But as an example let's look at the test which inserts a new paragraph. Here is the content of the inputpara.tsv file:
ArgumentNew ParagraphThe first line again specifies the column title. The other lines contain the data which should by passed to the Squish type() function which sends key events to a edit widget. In this case first the text New Paragraph will be inserted and then the Return key will be pressed to split the paragraph in two.
The corresponding expected result file exppara.tsv looks like this:
New Paragraph
First paragraph
Bold paragraph
This is the HTML output which the AUT should save when we save the contents of the editor after executing the above actions.
Editing the test script
Now that we have the test data files in place we will go on and edit the recorded test script to use this test data. What we want to achieve is that the test executes the following actions for each record in tests.tsv:
1. Choose File->Open in the menu
2. Choose the file simple.html in the file dialog by double clicking on it
3. Enter the text in the editor as specified in the input data file
4. Choose File->Save As in the menu
5. Enter out.html as filename in the file dialog
6. Press Return in the file dialog
7. Get out.html from the AUT side and compare it against the expected output file
8. Choose File->Close in the menu
Basically we need to put most of the test script into a loop which iterates over all records in tests.tsv. Then we have to implement step 3 to enter the commands as specified in the test data instead of entering some hard-coded data. We also have to implement step 7.
First we put most of the test into the loop. We will use a for loop which loops over all records in the data set tests.tsv and stores the current record in the variable t. In Python we use the following code for this:
for t in testData.dataset("tests.tsv"): In Python the body of the loop is defined by all lines below the loop statement whose indentation is larger than the loop statement's indentation. The first line which has the same indentation as the loop statement again is the first line outside the loop body. So what we will do is to insert the loop statement after the line sendEvent("QMoveEvent", ....). Then we increase the indentation of all lines until the line activateItem(":Richtext Editor.QPopupMenu1", "Close").
The next step is to modify the line type(":Richtext Editor.QTabWidget1.tab pages.QTextEdit2", "Hello") to type in the commands as specified in the current test data record's input file. So we replace this line with the following code:
file = testData.field(t, "CommandFile") for arg in testData.dataset(file): type(":Richtext Editor.QTabWidget1.tab pages.QTextEdit2", testData.field(arg, "Argument")) First we retrieve the name of the input file in the current test data record. We address the field containing the file name of the command file using the column title CommandFile. Then we loop over all commands in the specified input file and pass the commands retrieved from the field labeled Argument to the type() function to insert the text into the text editor.
The last bit is to compare the saved result against the expected result. To do this we will use the command line tool diff. First we will implement a function which runs diff on two files and adds a test result to the result log. We add the following code to the beginning of the script:
def diff(file1, file2): out = commands.getoutput("diff -u " + findFile("testdata", file1) +\ " " + findFile("testdata", file2)) if out == "": test.passes("Diff", file1 + " and " + file2 + " are equal"); else: test.fail("Diff", out) We use the Python module commands to execute the external diff program. Therefore we also need to import the commands module. We call the getoutput function from this module which executes the command specified as string and returns the output from the command. Therefore we put together the command line of diff which has to contain the files which should be compared. We assign the output to the variable out.
The output will contain the differences between the two files. This means if the string is empty, there was no difference. In this case we add a PASS test result with the information about the two files to the test result log. Otherwise we add a FAIL test result with the output generated by diff to the test result log.
Now we only need to call this function from within the test data loop. We add the following code after we pressed Return in the file dialog to save the editor's contents to the file out.html:
testData.get("out.html") diff("out.html", testData.field(t, "ExpectedOutput")); First we need to copy the file out.html from the AUT side to the squishrunner side. We use Squish's testData.get() function for this. Then we call the diff() function we just implemented. For the two files to be compared we pass the out.html file and the file which is specified in the current test data record as expected output in the column ExpectedOutput to the diff() function.
Now we implemented all steps and we can execute the test. By just adding new input and expected output files, and adding them to the tests.tsv file, it is possible to extend this test case. No test script code changes would be necessary. The complete test script will now look like the following:
import timeimport commands # Function which looks for file1 and file2 and invokes diff on them. If the files don't differ# a test pass is added to the test results, a test fail otherwisedef diff(file1, file2): out = commands.getoutput("diff -u " + findFile("testdata", file1) +\ " " + findFile("testdata", file2)) if out == "": test.passes("Diff", file1 + " and " + file2 + " are equal"); else: test.fail("Diff", out) def main(): # copy simple.html to the AUT side testData.put("simple.html") time.sleep(0.007) sendEvent("QMoveEvent", ":Richtext Editor", 756, 218, 665, 649) time.sleep(0.047) # loop over all tests specified in tests.tsv for t in testData.dataset("tests.tsv"): # open the simple.html file clickButton(":Richtext Editor.qt_top_dock.File Actions.fileOpen_action_button") time.sleep(0.189) sendEvent("QMoveEvent", ":Richtext Editor.qt_filedlg_gofn", 800, 423, 805, 261) time.sleep(0.1) doubleClickItem(":Richtext Editor.qt_filedlg_gofn.qt_splitter.files and more files.filelistbox"+\ ".qt_viewport", "simple.html", 48, 11, 0, Qt.LeftButton) time.sleep(2.14) # get the file which specifies the type commands which should be executed file = testData.field(t, "CommandFile") # iterate over all commands in the command file and execute them via type() for arg in testData.dataset(file): type(":Richtext Editor.QTabWidget1.tab pages.QTextEdit2", testData.field(arg, "Argument")) # save the resulting contents to out.html time.sleep(2.876) activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Save As...") time.sleep(0.157) sendEvent("QMoveEvent", ":Richtext Editor.qt_filedlg_gsfn", 800, 423, 788, 325) time.sleep(1.715) type(":Richtext Editor.qt_filedlg_gsfn.name/filter editor", "out.html") time.sleep(0.484) type(":Richtext Editor.qt_filedlg_gsfn.name/filter editor", "") time.sleep(1.654) # copy the saved out.html to the squishrunner side testData.get("out.html") # diff out.html agains the expected result diff("out.html", testData.field(t, "ExpectedOutput")); # close the editor tab activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Close") # exit the AUT time.sleep(2.876) activateItem(":Richtext Editor.automatic menu bar", "File") activateItem(":Richtext Editor.QPopupMenu1", "Exit")Conclusion: This article showed the complete process of creating a data-driven test case. While event capture
and reply eases the first steps of creating a test case, a test tool has to offer much more (a versatile
scriptinglanguage, extension APIs for test data handling, etc.) to allow test engineers to create robust,
maintainable and extensible test cases which will pay off in the long run.
Tuesday, February 5, 2008
Exploratory Testing
Through experience and a lot of trial and error, testers and musicians have developed skills they can't easily explain. Unfortunately, with software testing, there aren't as many obvious avenues for skill development as there are for musicians. Many software testers don't realize that there are learnable exploratory testing skills they can develop to help them become even more valuable to software development teams.When I work in a new organization, it doesn’t take long for me to get introduced to the testers who are highly valued. Often, when I ask testers about the times they did their best work, they apologize for breaking the rules: "I didn’t follow any pre-scripted test cases, and I didn’t really follow the regular testing process." As they describe how they came through for the team on a crucial bug discover, they outline activities that I identify with skilled exploratory testing. Little do they know that this is often precisely why they are so effective as testers. They have developed analysis and investigative skills that give them the confidence to use the most powerful testing tool we have at our disposal: the human mind.Exploratory testing is something that testers naturally engage in, but because it is the opposite of scripted testing, it is misunderstood and sometimes discouraged. In an industry that encourages pre-scripted testing processes, many testers aren’t aware that there are other ways of testing other than writing and following test scripts. Both software testing and music can be interpreted and performed in a variety of ways.
What does skilled exploratory testing look like? Here is scripted testing and exploratory testing in action. In one test effort, I came across a manual test script and its automated counterpart which had been written several releases ago. They were for an application I was unfamiliar with, using technology I was barely acquainted with. I had never run these tests before, so I ran the automated test first to try to learn more about what was being tested. It passed, but the test execution and results logging didn’t provide much information other than "test passed." To me, this is the equivalent of the emails I get that say: "Congratulations! You may already be a winner!" Statements like that on their own, without some sort of corroboration mean very little.
I didn’t learn much from my initial effort: running the automated test didn’t reveal more information about the application or the technology. Since learning is an important part of testing work, I delved more deeply. I moved on to the manual test script, and followed each step. When I got to the end, I checked for the expected results, and sure enough, the actual result I observed matched what was predicted in the script. Time to pass the test and move on, right? I still didn’t understand exactly what was going on with the test and I couldn’t take responsibility for those test results completely on blind faith. That violates my purpose as a tester; if I believed everything worked as advertised, why test at all? Furthermore, experience has taught me that tests can be wrong, particularly as they get out of date. Re-running the scripted tests provided no new information, so it was time leave the scripted tests behind.
One potential landmine in providing quality-related software information is tunnel vision. Scripted tests have a side effect of creating blinders - narrowing your observation space. To widen my observation possibilities, I began to transition from scripted testing to exploratory testing. I began creating new tests by adding variability to the existing manual test, and I was able to get a better idea of what worked and what caused failures. I didn’t want to write these tests down because I wanted to adjust them on the fly so I could quickly learn more. Writing them down would interrupt the flow of discovery, and I wasn’t sure what tests I wanted to repeat later.
I ran another test, and without the scripted tests to limit my observation, noticed something that raised my suspicions: the application became sluggish. Knowing that letting time pass in a system can cause some problems to intensify, I decided to try a different kind of test. I would follow the last part of the manual test script, wait a few minutes, and then thoroughly inspect the system. I ran this new test, and the system felt even more sluggish than before. The application messaging showed me the system was working properly, but the sluggish behavior was a symptom of a larger problem not exposed by the original tests I had performed.
Investigating behind the user interface, I found that the application was silently failing; while it was recording database transactions as completing successfully, it was actually deleting the data. We had actually been losing data ever since I ran the first tests. Even though the tests appeared to pass, the application was failing in a serious manner. If I had relied only on the scripted manual and automated tests, this would have gone undetected, resulting in a catastrophic failure in production. Furthermore, if I had taken the time to write down the tests first, and then execute them, I would most likely have missed this window of opportunity that allowed me to find the source of the problem. Merely running the scripted tests only felt like repeating an idea resolution, and didn’t lead to any interesting discoveries. On the other hand, the interplay between the tension and resolution of exploratory testing ideas quickly led to a very important discovery. Due to results like this, I don’t tend to use many procedural, pre-scripted manual test cases in my own personal work.
So how did I find a problem that was waiting like a time-bomb for a customer to stumble upon? I treated the test scripts for what they were: imperfect sources of information that could severely limit my abilities to observe useful information about the application. Before, during and after test execution, I designed and re-designed tests based on my observations. I also had a bad feeling when I ran the test. I’ve learned to investigate those unsettled feelings rather than suppress them because feelings of tension don’t fit into a script or process; often, this rapid investigation leads to important discoveries. I didn’t let the scripts dictate to me what to test, or what success meant. I had confidence that skilled exploratory testing would confirm or deny the answers supplied by the scripted tests.
Testers who have learned to use their creativity and intelligence when testing come up with ways to manage their testing thought processes. Skilled exploratory testers use mental tricks to help keep their thinking sharp and consistent. Two tricks testers use to kick start their brains are heuristics (problem-solving approaches) and mnemonics (memory aids).I’m sometimes called in as an outside tester to test an application that is nearing completion. If the software product is new, a technique I might use is the "First Time User" heuristic. With very little application information, and using only the information available to the first time user, I begin testing. It’s important for me to know as little as possible about the application at this time, because once I know too much, I can’t really test the way a first-time user would.
As I work through the application, I may find mismatches between the application and the market it is intended for, and the information supplied to users. If I have trouble figuring out the software’s purpose and how to use it, so will end users. This is important usability information that I write down in my notes. If the software isn’t usable, it isn’t going to sell. I invariably find bugs in this first testing session. I explore them, take notes so I can report them later, and design new tests around those bugs. After the session is over, I will have bugs to report and usability questions to ask. I will now have a model developed in my mind for testing this software. My brain will work on this model constantly, even when I’m not testing, and other testing activities will help build this and other models of the software.
Using heuristics and mnemonics helps me be consistent when testing, but I don’t let them rule my testing actions. If I observe something suspicious, I explore it. If something feels wrong, I investigate, and confirm or deny that feeling with defensible facts. It’s common to switch from heuristics and mnemonics to pure free-form improvisation and back again, or to improvise around highly structured tests. Exploratory testing—like improvising—helps me adapt my thinking and my actions based on what the software is telling me. This is a powerful concept. You can seize upon opportunities as soon as you observe them. Furthermore, you can adapt quickly to project risks, and discover and explore new ones. By developing skills to manage your thinking about testing, you no longer have to wait for spontaneous discoveries to appear out of thin air, and not be able to explain why you found a particular problem, or repeat it.
Developing exploratory testing skill puts you in charge of your testing approach. Skilled software testing, like skilled musicianship, is often referred to as "magic", simply because it is misunderstood. Music follows a set of patterns, heuristics and techniques. If you know a handful, you can make music quite readily. Getting there by trial and error takes a lot longer, and you’ll have a hard time explaining how you got there once you arrive. Testing using pure observation and trial and error can be effective, but can be effective much more quickly if there is a system to frame it.
Skilled exploratory testing can be a powerful way of thinking about testing. However, it is often misunderstood, feared and discouraged. When we dictate that all tests must be scripted, we discourage the wonderful tension and resolution in testing, driven by a curious thinker. We limit the possibilities of our software testing leading to new, important discoveries. We also hamper our ability to identify and adapt to emerging project risks. In environments that are dominated by technology, it shouldn’t be surprising that we constantly look to tools and processes for solutions. But tools and processes on their own are stupid things. They still require human intelligence behind them. In the right hands, software tools and processes, much like musical instruments, enable us to realize the results we seek. There are many ways to perform music, and there are many ways to test software. Skilled exploratory testing is another effective thinking tool to add to the testing repertoire.
What does skilled exploratory testing look like? Here is scripted testing and exploratory testing in action. In one test effort, I came across a manual test script and its automated counterpart which had been written several releases ago. They were for an application I was unfamiliar with, using technology I was barely acquainted with. I had never run these tests before, so I ran the automated test first to try to learn more about what was being tested. It passed, but the test execution and results logging didn’t provide much information other than "test passed." To me, this is the equivalent of the emails I get that say: "Congratulations! You may already be a winner!" Statements like that on their own, without some sort of corroboration mean very little.
I didn’t learn much from my initial effort: running the automated test didn’t reveal more information about the application or the technology. Since learning is an important part of testing work, I delved more deeply. I moved on to the manual test script, and followed each step. When I got to the end, I checked for the expected results, and sure enough, the actual result I observed matched what was predicted in the script. Time to pass the test and move on, right? I still didn’t understand exactly what was going on with the test and I couldn’t take responsibility for those test results completely on blind faith. That violates my purpose as a tester; if I believed everything worked as advertised, why test at all? Furthermore, experience has taught me that tests can be wrong, particularly as they get out of date. Re-running the scripted tests provided no new information, so it was time leave the scripted tests behind.
One potential landmine in providing quality-related software information is tunnel vision. Scripted tests have a side effect of creating blinders - narrowing your observation space. To widen my observation possibilities, I began to transition from scripted testing to exploratory testing. I began creating new tests by adding variability to the existing manual test, and I was able to get a better idea of what worked and what caused failures. I didn’t want to write these tests down because I wanted to adjust them on the fly so I could quickly learn more. Writing them down would interrupt the flow of discovery, and I wasn’t sure what tests I wanted to repeat later.
I ran another test, and without the scripted tests to limit my observation, noticed something that raised my suspicions: the application became sluggish. Knowing that letting time pass in a system can cause some problems to intensify, I decided to try a different kind of test. I would follow the last part of the manual test script, wait a few minutes, and then thoroughly inspect the system. I ran this new test, and the system felt even more sluggish than before. The application messaging showed me the system was working properly, but the sluggish behavior was a symptom of a larger problem not exposed by the original tests I had performed.
Investigating behind the user interface, I found that the application was silently failing; while it was recording database transactions as completing successfully, it was actually deleting the data. We had actually been losing data ever since I ran the first tests. Even though the tests appeared to pass, the application was failing in a serious manner. If I had relied only on the scripted manual and automated tests, this would have gone undetected, resulting in a catastrophic failure in production. Furthermore, if I had taken the time to write down the tests first, and then execute them, I would most likely have missed this window of opportunity that allowed me to find the source of the problem. Merely running the scripted tests only felt like repeating an idea resolution, and didn’t lead to any interesting discoveries. On the other hand, the interplay between the tension and resolution of exploratory testing ideas quickly led to a very important discovery. Due to results like this, I don’t tend to use many procedural, pre-scripted manual test cases in my own personal work.
So how did I find a problem that was waiting like a time-bomb for a customer to stumble upon? I treated the test scripts for what they were: imperfect sources of information that could severely limit my abilities to observe useful information about the application. Before, during and after test execution, I designed and re-designed tests based on my observations. I also had a bad feeling when I ran the test. I’ve learned to investigate those unsettled feelings rather than suppress them because feelings of tension don’t fit into a script or process; often, this rapid investigation leads to important discoveries. I didn’t let the scripts dictate to me what to test, or what success meant. I had confidence that skilled exploratory testing would confirm or deny the answers supplied by the scripted tests.
Testers who have learned to use their creativity and intelligence when testing come up with ways to manage their testing thought processes. Skilled exploratory testers use mental tricks to help keep their thinking sharp and consistent. Two tricks testers use to kick start their brains are heuristics (problem-solving approaches) and mnemonics (memory aids).I’m sometimes called in as an outside tester to test an application that is nearing completion. If the software product is new, a technique I might use is the "First Time User" heuristic. With very little application information, and using only the information available to the first time user, I begin testing. It’s important for me to know as little as possible about the application at this time, because once I know too much, I can’t really test the way a first-time user would.
As I work through the application, I may find mismatches between the application and the market it is intended for, and the information supplied to users. If I have trouble figuring out the software’s purpose and how to use it, so will end users. This is important usability information that I write down in my notes. If the software isn’t usable, it isn’t going to sell. I invariably find bugs in this first testing session. I explore them, take notes so I can report them later, and design new tests around those bugs. After the session is over, I will have bugs to report and usability questions to ask. I will now have a model developed in my mind for testing this software. My brain will work on this model constantly, even when I’m not testing, and other testing activities will help build this and other models of the software.
Using heuristics and mnemonics helps me be consistent when testing, but I don’t let them rule my testing actions. If I observe something suspicious, I explore it. If something feels wrong, I investigate, and confirm or deny that feeling with defensible facts. It’s common to switch from heuristics and mnemonics to pure free-form improvisation and back again, or to improvise around highly structured tests. Exploratory testing—like improvising—helps me adapt my thinking and my actions based on what the software is telling me. This is a powerful concept. You can seize upon opportunities as soon as you observe them. Furthermore, you can adapt quickly to project risks, and discover and explore new ones. By developing skills to manage your thinking about testing, you no longer have to wait for spontaneous discoveries to appear out of thin air, and not be able to explain why you found a particular problem, or repeat it.
Developing exploratory testing skill puts you in charge of your testing approach. Skilled software testing, like skilled musicianship, is often referred to as "magic", simply because it is misunderstood. Music follows a set of patterns, heuristics and techniques. If you know a handful, you can make music quite readily. Getting there by trial and error takes a lot longer, and you’ll have a hard time explaining how you got there once you arrive. Testing using pure observation and trial and error can be effective, but can be effective much more quickly if there is a system to frame it.
Skilled exploratory testing can be a powerful way of thinking about testing. However, it is often misunderstood, feared and discouraged. When we dictate that all tests must be scripted, we discourage the wonderful tension and resolution in testing, driven by a curious thinker. We limit the possibilities of our software testing leading to new, important discoveries. We also hamper our ability to identify and adapt to emerging project risks. In environments that are dominated by technology, it shouldn’t be surprising that we constantly look to tools and processes for solutions. But tools and processes on their own are stupid things. They still require human intelligence behind them. In the right hands, software tools and processes, much like musical instruments, enable us to realize the results we seek. There are many ways to perform music, and there are many ways to test software. Skilled exploratory testing is another effective thinking tool to add to the testing repertoire.
Get More of Exploratory Testing:
Bach, James. (2003) Exploratory Testing
Bach, James. (2003) Exploratory Testing
http://www.satisfice.com/articles/et-article.pdf
Kaner, Cem. (2004) The Ongoing Revolution in Software Testing, Presented at Software Test & Performance Conference, December2004, Baltimore, MD
Kaner, Cem. (2004) The Ongoing Revolution in Software Testing, Presented at Software Test & Performance Conference, December2004, Baltimore, MD
http://www.kaner.com/pdfs/TheOngoingRevolution.pdf
Bach, James. (1999, November) Heuristic Risk-Based Testing, Software Testing and Quality Engineering Magazine
Bach, James. (1999, November) Heuristic Risk-Based Testing, Software Testing and Quality Engineering Magazine
Software Testing Concepts
Software testing is the process used to measure the quality of developed computer software. Usually, quality is constrained to such topics as correctness, completeness, security, but can also include more technical requirements as described under the ISO standard ISO 9126, such as capability, reliability, efficiency, portability, maintainability, compatibility and usability.
Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors. Quality is not an absolute; it is value to some person. With that in mind, testing can never completely establish the correctness of arbitrary computer software; testing furnishes a criticism or comparison that compares the state and behavior of the product against a specification. An important point is that software testing should be distinguished from the separate discipline of Software Quality Assurance, which encompasses all business process areas, not just testing.
Today, software has grown in complexity and size. The software product developed by a developer is according to the System Requirement Specification. Every software product has a target audience. For example, a video game software has its audience completely different from banking software. Therefore, when an organization invests large sums in making a software product, it must ensure that the software product must be acceptable to the end users or its target audience. This is where Software Testing comes into play. Software testing is not merely finding defects or bugs in the software, it is the completely dedicated discipline of evaluating the quality of the software.
There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following routine procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are operations the tester attempts to execute with the product, and the product answers with its behavior in reaction to the probing of the tester. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is also used to connote the dynamic analysis of the product—putting the product through its paces. Sometimes one therefore refers to reviews, walk-throughs or inspections as "static testing", whereas actually running the program with a given set of test cases in a given development stage is often referred to as "dynamic testing", to emphasize the fact that formal review processes form part of the overall testing scope.
Subscribe to:
Posts (Atom)