Christopher Lott, STTI-KL, Univ. Kaiserslautern
lott@informatik.uni-kl.de
Recently I have been packaging a software engineering experiment for replication and publication [1, 2]. The experiment compares the effectiveness and efficiency of three defect-detection techniques, namely code reading by stepwise abstraction, functional testing, and structural testing. This experiment was first described by Basili and Selby in 1987; we extended the goals and hypotheses, developed new code objects, and replicated the experiment twice using student subjects in a controlled context. Packaging the experiment for replication and publication includes making all materials (specifications, source code modules, data collection forms, etc.) available for general use.
As part of packaging the experiment, we developed a characterization framework that simplifies the comparison of experiments, and formulated a number of reuse guidelines for the materials used in our replications. The framework helps compare the results stemming from exact replications (vary only the subjects) and approximate replications (reuse goals and hypotheses but vary other aspects). The reuse guidelines explain assumptions behind the goals, experimental design, and other materials and state some possibilities for tailoring that information for the specific context in which the experiment will be replicated.
Replication of fundamental software engineering experiments will help build the software engineering community's understanding of various defect-detection techniques under varying circumstances, as well as educate the subjects who participate. Like Watts Humphrey's Personal Software Process, an experiment is a risk-free, small project in which students and professionals can gain direct personal experience with various software detect-detection techniques. Such direct, personal experience is a requirement for any person to be convinced of the utility of a candidate technique. The educational benefit of comparing a non-execution-based (i.e., human-based) technique such as code reading with execution-based techniques such as functional and structural testing lies in convincing developers that testing is not the solitary way to detect defects.
In addition to convincing developers of the utility of human-based techniques, the experiment encourages developers to distinguish between assessing reliability (validation) and checking correctness (verification). Verification techniques are a prerequisite for detecting defects early in the software development life-cycle, long before code is available. Once developers have learned the distinction and are convinced of the utility of verification techniques, they are well equipped to apply those techniques to detect defects in requirements and design documents, thus detecting problems when they are still cheap to correct.
I would especially welcome the chance to be involved in a working group at IWSED that would discuss either the replication of experiments, the distinction of verification and validation techniques, or data that reveal differences in the utility of various defect-detection techniques in a given project environment.
1. Erik Kamsties and Christopher M. Lott. An empirical evaluation of three defect-detection techniques. In W. Sch"afer and P. Botella, editors, Proceedings of the 5th European Software Engineering Conference}. Lecture Notes in Computer Science Nr. 989, Springer-Verlag, September 1995.
2. Erik Kamsties and Christopher M. Lott. An empirical evaluation of three defect-detection techniques. Technical Report ISERN 95--02, Department of Computer Science, University of Kaiserslautern, 67653 Kaiserslautern, Germany, May 1995.