A few jobs back, while Lead Developer for a team at a large company, I recognized that some of the developers on the team were producing fairly shoddy code, and expecting that the “testing team” would discover any problems that the developer’s needed to fix.

This thinking irked me to the core – as it should you. This issue dovetails nicely with my previous post.

I wrote a Standards & Policy document for the team at that time all about Unit Testing, wherein I set forth how and why unit testing should be done.

I was recently reviewing that document for a different purpose, but felt that what I said then was so relevant and important that I’d share it with you now – not for any form of self-adulation, but for your benefit and growth as a developer.

Much of what I said is not original, but culled from best practices, industry standards, etc. – as it should be. Don’t ever waste time reinventing the wheel. I do not take the time in this post to credit each original source, but I will recommend some resources at the end.

What is the purpose of Unit Testing?

To best answer that question, let us start by answering a different question: “What is the role of a developer?” to which the answer is “to consistently deliver 100% error free code which exactly meets requirements, on time, and within budget.”

It therefore goes without saying that it is the developer’s job to test his/her code and ascertain that it is 100% error free. It is NOT the tester’s job to test your code.

Consider this: Name one other job in which it’s acceptable to do substandard work merely because someone else is going to review it for mistakes?

So ultimately, the purpose of Unit Testing is to help you, as the developer, to catch as many bugs as possible, as early as possible, and to ensure that your code functions correctly.

Now that we know the role of the developer, we must look at another question: “what is the role of the Testing Team?” The testing team’s job is to “ship” the software. They verify that all the modules integrate successfully, that the system presents a unified “character” to the user, and that it meets all the requirements – that it does what it was intended to do.

What is the goal of Unit Testing?

The goal of any Unit Testing process should be to outline various methods and ways to uncover as many defects as possible.

It should be obvious that no one approach can universally achieve this goal.

So multiple approaches are outlined which are to be used in unison. Together, these approaches, if consistently applied correctly, should help to uncover the vast majority of the defects in a module.

It is interesting to note that the goal of testing runs in diametric opposition to the goal of the rest of development: The goal is to find errors. A successful test is one that breaks the software. (This is not necessarily to say that a test that does not break the software is unsuccessful. But in such a case, the developer must be very objective in asking “is this code as perfect as I am capable of making it?” before accepting a passing test.) The goal of every other development activity is to prevent errors and keep the software from breaking.

It is also a generally hard activity for developers to perform, because it requires substantially “egoless” developing. Good testing requires you to assume that you’ll find errors in your code. If you assume you won’t, you probably won’t.

If you execute the program hoping that it won’t have any errors, it will be too easy to overlook the errors you find.

In one study, a group of experienced programmers were testing a program with 15 known defects. The average programmer found only 5 of the 15 errors. The best found only 9. The main source of undetected errors was that erroneous output was not examined carefully enough. The errors were visible but the programmers didn’t notice them.

In what way does Unit Testing support the development process?

The process of development tends to be cyclical – Design, Code, Test, Debug, Document, repeat as necessary. Within this cycle, testing is only one aspect, but it is one of the most significant aspects.

Improper or inadequate testing results in frustration among team members; poor software quality; a poor image of the development team from the users point of view; decreased likelihood of meeting deadlines; increased maintenance costs; increased support costs; and ultimately can lead to reduced or lost revenue from decreased customer confidence.

So Unit Testing is the firewall that prevents bad code from making its way into the customer’s hands. It is valuable to note that the approach a developer or team has to testing and the quality of the testing performed makes the difference between good and great software. An average development team can develop world class software by testing well, and taking the opportunity to discover and repair all their mistakes, defects, and oversights before they ever see the light of day.

It cannot be emphasized enough, the importance of Unit Testing. If your unit is properly and adequately tested the effect of that unit on the overall quality of the system cannot help but be good. On the other hand, a single, poorly written, insignificant module that is not properly tested can have a devastating effect on the overall quality of the system.

It should also be understood, however, that proper Unit Testing should not wait until your module is considered complete. In fact, many experienced developers would argue that a module could not be considered complete until it has been properly and adequately tested.

The best point to begin your Unit Testing process is at the beginning of the design stage of your module. In that way, you can begin to consider what impact your design is going to have on the task of Unit Testing, and that may lead you to different design conclusions.

For example, imagine that you have a requirement that, during initial analysis may lend itself to at least 5 different possible design approaches. As you consider these various approaches, you should attempt to eliminate those which lead to either untestable or difficult-to-test implementations.

What is to be tested through the Unit Testing Process?

As a general approach, the following ground needs to be covered:

Test for each relevant requirement to make sure that the requirements have been implemented. Plan the test cases for this step at the requirements-gathering stage or as early as possible – preferably before you begin writing the unit to be tested. Consider testing for common omissions in requirements.

Test for each relevant design concern to make sure that the design has been implemented. Plan the test cases for this step at the design stage or as early as possible – before you begin the detailed coding of the routine to be tested.

Use basis testing to add detailed test cases to those that test the requirements and the design.

Add data-flow tests.

Add the remaining test cases needed to thoroughly exercise the code. At a minimum you should test every line of code. This is true regardless of whether this is a new development project or just a single line change.

Build the test cases along with the product. This can help avoid errors in requirements and design, which tend to be more expensive than coding errors. Plan to test and find defects as early as possible because it’s cheaper to fix defects early.

The Unit Test should cover the following areas:

Does the module meet the requirements?

Does the module correctly implement the design?

Does the module correctly generate the expected output from a given set of input?

Does every line of the module do what it is expected to do?

Has every line of code been executed at least once?

Structured Basis Testing>

The idea behind structured basis testing is fairly simple. You need to test every statement in a program at least once. The easiest way to make sure you’ve gotten all the bases covered is to calculate the number of paths through the program and then develop the minimum number of test cases that will exercise all of those paths.

This is very similar to “code coverage” or “logic coverage” testing, with the difference being that while those methods do indeed test all paths through a program, they don’t include the idea of covering all paths with a minimal set of test cases.

You can compute the minimal number of cases needed for basis testing according to the following method:

Start with 1 for the straight path through the routine.

Add 1 for each of the following keywords: if, while, repeat, for, and, and or.

Add 1 for each case in a case statement. If the case statement doesn’t have a default case, add 1 more.

Here’s an example:

Statement 1; // Count 1 for the routine itself.
Statement 2;
if X < 10 then // Count 2 for the if.
begin
  Statement 3;
end;
Statement 4;

In this instance, you start with one and count the if once to make a total of two. That means that you need to have at least two test cases to cover all the paths through the program. In this example, you’d need to have the following test cases:

Statements controlled by if are executed (X < 10).

Statements controlled by if aren’t executed (X >= 10).

The sample code needs to be a little more realistic to give you an accurate idea of how this kind of testing works. Realism in this case includes code containing defects.

1 {Compute Net Pay}
2
3 TotalWithholdings := 0;
4
5 for ID :=1 to NumEmployees
6 begin
7
8   {Compute social security withholding, if below the maximum.}
9   if ( Employee[ ID ].SSWithheld < MAX_SOCIAL_SECURITY ) then
10   begin
11     SocialSecurity := ComputeSocialSecurity(Employee[ID])
12   end;
13
14   {Set default to no retirement contribution}
15   Retirement := 0;
16
17   {Determine discretionary employee retirement contribution}
18   if ( Employee[ ID ].WantsRetirement ) and
19      ( EligibleForRetirement( Employee[ID] ) ) then
20   begin
21     Retirement := GetRetirement( Employee[ID] )
22   end;
23
24   GrossPay := ComputeGrossPay( Employee[ID] );
25
26   {Determine IRA contribution}
27   IRA := 0;
28   if ( EligibleForIRA( Employee[ID] ) ) then
29   begin
30     IRA := IRAContribution( Employee[ID], Retirement, GrossPay)
31   end;
32
33   {Make Weekly Paycheck}
34   WithHolding := ComputeWithholding( Employee[ID] );
35   NetPay := GrossPay – WithHolding – Retirement – SocialSecurity – IRA;
36   PayEmployee( Employee[ID], NetPay );
37
38   {Add this employee’s paycheck to total for accounting}
39   TotalWithholdings := TotalWithholdings + WithHolding;
40   TotalSocialSecurity := TotalSocialSecurity + SocialSecurity;
41   TotalRetirement := TotalRetirement + Retirement;
42 end;
43
44 SavePayRecords( TotalWithholdings, TotalSocialSecurity, TotalRetirement);

This is a slightly more complicated example. This piece of code is used throught the remainder of this document (referred to as the Withholding Example), and contains a few possible errors.

The following list represents how to count the minimal number of test cases for this example:

Count 1 for the routine itself.

Count 1 for the for in line 5.

Count 1 for the if in line 9.

Count 1 for the if in line 18.

Count 1 for the and in line 18.

Count 1 for the if in line 28.

In this example, you’ll need one initial test case plus one for each of the five keywords, for a total of six. That doesn’t mean that any six test cases will cover all the bases. It means that, at a minimum, six cases are required. Unless the cases are constructed carefully, they almost surely won’t cover all the bases. The trick is to pay attention to the same keywords you used when counting that can be either true or false; make sure you have at least one test case for each true and at least one for each false.

Here is a set of test cases that covers all the above considerations for this example:

1. Nominal Case – All boolean conditions are true.
2. The initial forcondition is false – NumEmployees < 1
3. The first if is false – Employee[ID].SSWithheld >= MAX_SOCIAL_SECURITY
4. The second if is false because the first part of the and is false – Not Employee[ID].WantsRetirement
5. The second if is false because the second part of the and is false – Not EligibleForRetirement(Employee[ID])
6. The third if is false – Not EligibleForIRA(Employee[ID])

If the routine were much more complicated than this, the number of test cases you’d have to use just to cover all the paths would increase pretty quickly. Shorter routines tend to have fewer paths to test. Boolean expressions without a lot of ands and ors have fewer variations to test. Ease of testing is another good reason to keep your routines short and your boolean expressions simple.

With these six test cases, which satisfy the demands of structured basis testing, can you consider the routine to be fully tested? Probably not. This kind of testing only assures that all of the code will be executed. It does not account for variations in data.

Step Through the Code.

Many defects in code often go undiscovered by developers for one simple reason – the code “appears to work.” It is amazing how bad code can slip by the best developers because of one preventable mistake – dependence on bugs to reveal themselves, instead of actually looking for them. There is no substitute for your own critical review of the code as it is executing to determine whether or not it is doing what you intended to tell it to do.

The practice of stepping through your code is considered an industry Best Practice, and the quality of the code produced by the CATS development team will increase measurably if this practice is put into effect. For more information and support of this practice, refer to chapter 4 of “Writing Solid Code: Microsoft’s techniques for developing bug-free C programs,” by Steve Maguire. Write your own joke here, but it should become apparent that if Microsoft consistently followed their own practices, we would all reap the benefits. This book is an excellent source of industry Best Practices.

A common mistake made when implementing this technique, though, is that many paths of execution are missed. That is why this technique is presented after that of structured basis testing – because when applied together, the effectiveness of both is complimentary.

It is also important to take advantage of some of the features Delphi provides, to further increase the effectiveness of this technique. One primary example is the feature that evaluates and returns the value of a variable or property merely by pointing at it within the code inspector during execution. This technique allows you to verify that the values you expect to find are actually present. It assists you in catching many less obvious errors, such as being off-by-one, etc. Don’t just watch the code execute – take steps to comfort yourself that it is really executing your design.

Think of it this way – programming is really just communicating. You communicate your design to the computer. And you can’t do it in your native language, nor in the computers native language – you do it in a third language, which neither you nor the computer is necessarily an expert in. You may have intended to communicate one thing, but the computer interpreted your instructions differently. Or, perhaps less likely but not at all impossible, you communicated exactly what you intended, but the computer executed it differently than it should – perhaps there is a bug in the compiler.

So the only way to effectively ensure that proper and complete communication is taking place is to verify with your own eyes that what you meant for the computer to do, it is actually doing. You’ll be surprised how often it is not.

This also fits in with the goal of Unit Testing, as stated earlier in this document, that “good testing requires you to assume that you’ll find errors in your code.” If you don’t step through your code, or if you don’t inspect your variables to see if they contain the values they should, you are really just assuming that your code is error free, which is contrary to the position you should take.

Data-Flow Testing

Viewing structured basis testing and data-flow testing together gives you another example illustrating that control flow and data flow are equally important in computer programming.

Data-flow testing is based on the idea that data usage is at least as error-prone as control flow.

Data can exist in one of three states:

Defined. – The data has been initialized, but it hasn’t been used yet.

Used. – The data has been used for computation, as an argument to a routine, or for something else.

Killed. – The data was once defined, but is has been undefined in some way. For example, if the data is a pointer, perhaps the pointer has been freed. If it’s a record pointer, maybe the file has been closed and the record pointer is no longer valid.

In addition to the having the above terms, it’s convenient to have terms that describe entering or exiting a routine immediately before or after doing something to a variable.

Entered. – The control flow enters the routine immediately before the variable is acted upon.

Exited. – The control flow leaves the routine immediately after the variable is acted upon.

Fortunately, modern compilers are very good at detecting and warning about many common mistakes. However, to a degree, they are only as good as you let them be – Always turn on all warning, hints, and compiler suggestions. If it complains about something, then you have done something that is at least dangerous or logically incorrect – if not actually wrong.

The key to writing data flow test cases is to exercise all possible “defined – used” paths. Test every combination of defining a variable in one place and using it in another.

Here’s an example:

Example of a Program Whose Data Flow is to be Tested

if ( Condition 1 ) then
  X := a
else
  X := b;

if ( Condition 2 ) then
  Y := X + 1
else
  Y := X – 1;

To cover every path in the program, you need one test case in which Condition 1 is true and one in which it’s false. You also need a test case in which Condition 2 is true and one in which it’s false. This can be handled by two test cases: Case 1 (Condition 1 = True, Condition 2 = True) and Case 2 (Condition 1 = False, Condition 2 = False). Those two test cases are all you need for structured basis testing.

Note, that the above example highlights a situation which can occur (though infrequently), where the “nominal path” through the program – the first test case described under structured basis testing – is equivalent to and covered by case 2 above. In this case you can eliminate the “straight path” test case as redundent.

To cover every defined-used combination, however, you need to add two more cases. You need Case 3 (Condition 1 = True, Condition 2 = False), and Case 4 (Condition 1 = False, Condition 2 = True).

A good way to develop test cases is to start with structured basis testing, which gives you some if not all of the defined-used data flows. Then add the cases you still need to have a complete set of defined-used data-flow test cases.

In the table presented earlier referring to the Withholding Example, structured basis testing provided six test cases for the example routine provided earlier. Data-flow testing of each defined-used pair requires several more test cases, some of which are covered by existing test cases and some of which are not. Here are all the data-flow combinations that add test cases beyond the ones generated by structured basis testing:

7. Define Retirement in line 15 and first use it in line 30. This isn’t necessarily covered by any of the previous test cases.
8. Define Retirement in line 15 and use it first in line 35. This isn’t necessarily covered by any of the previous test cases.
9. Define Retirement in line 21 and use it first in line 35. This isn’t necessarily covered by any of the previous test cases.

Once you run through the process of listing data-flow test cases a few times, you’ll get a sense of which cases are fruitful and which are already covered. When you get stuck, list all the defined-used combinations. That might seem like a lot of work, but it’s guaranteed to show you any cases that you didn’t test for free in the basis-testing approach.

Boundary Analysis

One of the most fruitful areas for testing is boundary conditions – off-by-one errors. Saying Num-1 when you mean Num and saying >= when you mean > are common mistakes.

The idea of boundary analysis is to write test cases that exercise the boundary conditions. If you’re testing for a range of values that are less than Max, you have three possible conditions:

Boundary below Max

Max

Boundary above Max

As shown it takes three cases to ensure that none of the common mistakes has been made.

The Withholding Example code contains a test for Employee[ ID ].SSWithheld > MAX_SOCIAL_SECURITY. According to the principles of boundary analysis, three cases should be examined:

1. Case 1 is defined so that the true boolean condition for Employee[ ID ].SSWithheld < MAX_SOCIAL_SECURITY is the true side of the boundary. Thus, the case1 test case sets Employee[ ID ].SSWithheld to MAX-SOCIAL_SECURITY – 1. This test case was already generated.

3. Case 3 is defined so that the false boolean condition for Employee[ ID ].SSWithheld < MAX_SOCIAL_SECURITY is the false side of the boundary. Thus, the case 3 test case sets Employee[ ID ].SSWithheld to MAX_SOCIAL_SECURITY + 1. This test case was already generated.

10. An additional test case is added for the dead-on case in which Employee[ ID ].SSWithheld = MAX_SOCIAL_SECURITY.

Boundary analysis also applies to minimum and maximum allowable values. In this example, it might be minimum or maximum GrossPay, Retirement, or IRAContribution, but since calculations of those values are outside of the routine, test cases for them aren’t discussed further here. Obviously, though, you should include test cases for these conditions in your unit testing.

A more subtle kind of boundary condition occurs when the boundary involves a combination of variables. For example, if two variables are multiplied together, what happens when both are large positive numbers? Large negative numbers? 0? What if all the strings passed to a routine are uncommonly long? Is adequate precision maintained when performing math on floating-point numbers? Are insignificant digits rounded or truncated?

In the Withholding Example, you might want to see what happens to the variables TotalWithholdings, TotalSocialSecurity, and TotalRetirement when every member of a large group of employees has a large salary – say, a group of programmers at $250,000 each. This calls for another test case:

11. A large group of employees, each of whom has a large salary – for the sake of example (what constitutes “large” depends on the specific system being developed), 1000 employees each with a salary of $250,000, none of whom have had any social security tax withheld and all of whom want retirement withholding.

A test case in the same vein but on the opposite side of the looking glass would be a small group of employees, each of who has a salary of $0.00.

12. A group of 10 employees, each of who has a salary of $0.00

Classes of Bad Data

Aside from guessing that errors show up around boundary conditions, you can guess about and test for several other classes of bad data. Typical bad-data test cases include:

Too little data (or no data)

Too much data

The wrong kind of data (invalid data)

The wrong size of data

Uninitialized data

Some of the test cases you would think of if you followed these suggestions have already been covered. For example, “too little data” is covered by Cases 2 and 12, and it’s hard to come up with anything for “wrong size of data.” This area nonetheless gives rise to a few more cases:

13. An array of 32,768 employees. Tests for too much data. Of course, how much is too much would vary from system to system, but for the sake of the example, assume that this is far too much.
14. A negative salary. Wrong kind of data.
15. A negative number of employees. Wrong kind of data.

Tests of this type help to uncover a number of unknowns, and to ensure proper handling of anticipated problems, such as:

What are the limits, if any, of any data types you’re using?

Are those limits adequate for both current and anticipated future needs?

If you exceed those limits, is it handled gracefully?

Classes of Good Data

When you try to find errors in a program, it’s easy to overlook the fact that the nominal case might contain an error. Usually the nominal cases described in the basis-testing section represent one kind of good data. Here are other kinds of good data that are worth checking:

Nominal cases – middle-of-the-road, expected values

Minimum normal configuration

Maximum normal configuration

Compatibility with old data

Checking each of these kinds of data can reveal errors, depending on the item being tested.

The minimum normal configuration is useful for testing not just one item, but a group of items. In the running example, testing the minimal normal configuration would add the following test case:

16. A group of one employee. To test the minimum normal configuration

The maximum normal configuration is the opposite of the minimum. An example of this would be generating a report that contains the maximum number of cases expected. In the case of the running example, testing the maximum normal configuration depends on the maximum normal number of employees. Assuming it’s 500, you would add the following test case:

17. A group of 500 employees. To test the maximum normal configuration

The last kind of normal data testing, testing for compatibility with old data comes into play when the program or routine is a replacement for an older program or routine. The new routine should produce the same results with old data the old routine did, except in cases in which the old routine was defective. This kind of continuity between versions is the basis for regression testing, the purpose of which is to ensure that corrections and enhancements maintain previous levels of quality without backsliding.

Requirements Testing

Now, we have a reasonable degree of confidence that our code works. We know that every line works, and that every line works as we intended it to work. We know that the program handles all the reasonable and many unreasonable variations in data. We know that it doesn’t “break” anything, if it is an upgrade or replacement. But we do not yet know that it does what it’s supposed to.

This is arguably the most important testing of all. You may have written the most bullet-proof, error-free, super-fast code in the world, but it if fails to achieve it’s intended purpose, it’s usefulness is dubious.

The goal of the requirements testing approach is to establish a test case for each element of the Requirements Traceability Matrix, providing 1-to-1 positive mapping to the RTM, and in the process, assuring that each requirement is both met and tested.

The stated purpose of the RTM is to provide the mechanism that serves as the basis for establishing compliance throughout the life cycle of the contract. If we do not test that the program meets each testable requirement, then we cannot consider the program tested.

Also, although this approach is documented later than other approaches, it should be noted that this test should be designed by the individual(s) performing the analysis and design, while performing the A&D – rather than being deferred to after development is complete. The risk of waiting is the temptation to test the software against what it is observed to do, rather than what it was designed to do, or what it was specified to do.

Unit Test Documentation

The unit test documentation should provide a standard and consistent presentation of the unit test plan for any given unit.

At a minimum, it should contain:

The name of the unit being tested

The version of the application to which the test plan applies

The date of the test plan (updated every time the test plan changes, reflects the date the test plan update is considered complete)

The name of the developer who developed the unit test

The test cases

A column to allow the developer to indicate pass/fail for each case in the test plan

A signature line for the developer to sign-off on the completed, fully passed test plan

A signature line for the test-plan reviewer to sign-off that the test plan meets standards and has passed his/her review.

Conclusion

Testing can never prove the absence of errors – only their presence. If you have tested extensively and found thousands of errors, does it mean that you’ve found all the errors or that you have thousands more to find? An absence of errors could mean ineffective or incomplete test cases as easily as it could mean perfect software.

Unit Testing process at-a-glance

1. Structured Basis Testing
   1. Compute minimal cases
      1. Start with 1 for straight path
      2. Add 1 for each of following keywords: IF, WHILE, REPEAT, FOR, AND, and OR
      3. Add 1 for each case in CASE statement
      4. Add 1 if CASE statement doesn’t have a default case
   2. Add any special cases you feel appropriate
2. Step through the Code
   1. Execute every single line of code with the debugger turned on
   2. Verify that every single line of code gets executed
   3. Force variables to specific values to cause each branch to be executed
   4. Check variables’ and fields’ values along the way
3. Data Flow Test
   1. Turn on all warnings, hints, and compiler suggestions
   2. Test all possible combinations of defining a variable in one place and using it in another
   3. Eliminate redundant test cases
4. Boundary Analysis
   1. Test for off-by-one errors
   2. Test for cases where value is minimum – 1
   3. Test for cases where value is minimum
   4. Test for cases where value is maximum
   5. Test for cases where value is maximum + 1
   6. Test Dates
      1. Test for leap years
         1. Normally Feb. has 28 days
         2. In years evenly divisible by 4, Feb. has 29 days
         3. In years evenly divisible by 100, Feb. has 28 days
         4. In years evenly divisible by 400, Feb. has 29 days.
      2. Test for wrap-around dates
         1. Cross month boundaries
         2. Cross year boundaries
         3. Cross century boundaries
      3. Test for handling of dates beyond built-in date limitations (such as 2079)
      4. Test for Year 2000 handling
   7. Test for field-length issues
5. Bad Data Test
   1. Test for cases where no data is present
   2. Test for cases where bad data is present
   3. Test for cases where too much data is present
      1. If a key field has a built in maximum quantity, test for exceeding that quantity
   4. Test for invalid data
      1. Test for numbers in fields that expect non-numeric
      2. Test for non-numeric in fields that expect numeric
      3. Test for invalid dates
         1. Test for End/Max-type dates are prior to Begin/Min-type dates
      4. Test for Non “Y” or “N” data in fields which expect only “Y” and “N”
      5. Test for no response in fields
      6. Test for blank but not-null field handling
      7. Test for total-fields where sum of individual fields exceed max. length of total field
      8. Test for entering foreign-key values that are not valid in dependency table
6. Good Data Test
   1. Test normal case
   2. Test minimal case
   3. Test maximal case
   4. Test for compatibility with old data
   5. Verify that all necessary foreign-key values are present and supported
7. Requirements Test
   1. Establish a test case for each applicable element in the Requirements Traceability Matrix
8. Unit Test Documentation
   1. Complete Unit Test documentation
   2. Complete review of Unit Test with sign-off from reviewer