1 Test Case Coverage 1 Test Coverage Test Coverage is an important measure of quality for software systems. Test Coverage analysis is the process of: • Finding areas of a program not exercised by a set of test cases, • Creating additional test cases to increase coverage, and • Determining a quantitative measure of code coverage, which is an indirect measure of quality. Also an optional aspect of test coverage analysis is: • Identifying redundant test cases that do not increase coverage. A test coverage analyzer automates this process. Test coverage analysis is sometimes called code coverage analysis. The two terms are synonymous. The academic world more often uses the term "test coverage" while practitioners more often use "code coverage". Test coverage analysis can be used to assure quality of the set of tests, and not the quality of the actual product. Coverage analysis requires access to test program source code and often requires recompiling it with a special command. Code coverage analysis is a structural testing technique (white box testing). Structural testing compares test program behavior against the apparent intention of the source code. This contrasts with functional testing (black-box testing), which compares test program behavior against a requirements specification. Structural testing examines how the program works, taking into account possible pitfalls in the structure and logic. Functional testing examines what the program accomplishes, without regard to how it works internally. 2 Test coverage measures A large variety of coverage measures exist. Here is a description of some fundamental measures and their strengths and weaknesses 3 Procedure-Level Test Coverage Probably the most basic form of test coverage is to measure what procedures were and were not executed during the test suite. This simple statistic is typically available from execution profiling tools, whose job is really to measure performance bottlenecks. If the execution time in some procedures is zero, you need to write new tests that hit those procedures. But this measure of test coverage is so coarse-grained it's not very practical. 4 Line-Level Test Coverage The basic measure of a dedicated test coverage tool is tracking which lines of code are executed, and which are not. This result is often presented in a summary at the procedure, file, or project level giving a percentage of the code that was executed. A large project that achieved 90% code coverage might be considered a well-tested product. Typically the line coverage information is also presented at the source code level, allowing you to see exactly which lines of code were executed and which were not. This, of course, is often the key to writing more tests that will increase coverage: By studying the unexecuted code, you can see exactly what functionality has not been tested. 5 Condition Coverage and Other Measures It's easy to find cases where line coverage doesn't really tell the whole story. For example, consider a block of code that is skipped under certain conditions (e.g., a statement in an if clause). If that code is shown as executed, you don't know whether you have tested the case when it is skipped. You need condition coverage to know. There are many other test coverage measures. However, most available code coverage tools do not provide much beyond basic line coverage. In theory, you should have more. But in practice, if you achieve 95+% line coverage and still have time and budget to commit to further testing improvements, it is an enviable commitment to quality! 6 How Test Coverage Tools Work To monitor execution, test coverage tools generally "instrument" the program by inserting "probes". How and when this instrumentation phase happens can vary greatly between different products. Adding probes to the program will make it bigger and slower. If the test suite is large and time- consuming, the performance factor may be significant. 1 Source-Level Instrumentation Some products add probes at the source level. They analyze the source code as written, and add additional code (such as calls to a code coverage runtime) that will record where the program reached. Such a tool may not actually generate new source files with the additional code. Some products, for example, intercept the compiler after parsing but before code generation to insert the changes they need. One drawback of this technique is the need to modify the build process. A separate version namely, code coverage version in addition to other versions, such as debug (un optimized) and release (optimized) needs to be maintained. Proponents claim this technique can provide higher levels of code coverage measurement (condition coverage, etc.) than other forms of instrumentation. This type of instrumentation is dependent on programming language -- the provider of the tool must explicitly choose which languages to support. But it can be somewhat independent of operating environment (processor, OS, or virtual machine). 2 Executable Instrumentation Probes can also be added to a completed executable file. The tool will analyze the existing executable, and then create a new, instrumented one. This type of instrumentation is independent of programming language. However, it is dependent on operating environment -- the provider of the tool must explicitly choose which processors or virtual machines to support. 3 Runtime Instrumentation Probes need not be added until the program is actually run. The probes exist only in the in- memory copy of the executable file; the file itself is not modified. The same executable file used for product release testing should be used for code coverage. Because the file is not modified in any way, just executing it will not automatically start code coverage (as it would with the other methods of instrumentation). Instead, the code coverage tool must start program execution directly or indirectly. Alternatively, the code coverage tool will add a tiny bit of instrumentation to the executable. This new code will wake up and connect to a waiting coverage tool whenever the program executes. This added code does not affect the size or performance of the executable, and does nothing if the coverage tool is not waiting. Like Executable Instrumentation, Runtime Instrumentation is independent of programming language but dependent on operating environment. 7 Test Coverage Tools at a Glance There are lots of tools available for measuring Test coverage. Company Product OS Lang Bullseye BullseyeCoverage Win32, Unix C/C++ CompuWare DevPartner Win32 C/C++, Java, VB Rational (IBM) PurifyPlus Win32, Unix C/C++, Java, VB Software Research TCAT Win32, Unix C/C++, Java Testwell CTC++ Win32, Unix C/C++ Paterson Technology LiveCoverage Win32 C/C++, VB Coverage analysis is a structural testing technique that helps eliminate gaps in a test suite. It helps most in the absence of a detailed, up-to-date requirements specification. Each project must choose a minimum percent coverage for release criteria based on available testing resources and the importance of preventing post-release failures. Clearly, safety-critical software should have a high goal. We must set a higher coverage goal for unit testing than for system testing since a failure in lower-level code may affect multiple high-level callers. Test . 1 Test Case Coverage 1 Test Coverage Test Coverage is an important measure of quality for software systems. Test Coverage analysis is. of test coverage analysis is: • Identifying redundant test cases that do not increase coverage. A test coverage analyzer automates this process. Test coverage