Discovering the Fundamental Drivers of Inspection Costs and Benefits.

Software Inspection

Software inspections are in-process technical reviews of software artifacts (e.g., requirements, designs, code, test plans) for the purpose of finding and eliminating defects [1]. Many organizations use a three-step inspection process: Individual analysis, Team analysis, and Repair. First, each reviewer studies the artifact to detect as many defects as possible (Individual analysis). Next, the team of reviewers meets and analyzes the artifact both to aggregate previously identified defects and to discover new ones (Team analysis). All defects are then sent to the artifact’s author for Repair. Under some conditions, the entire inspection may be repeated one or more times.

Although inspection has proven to be effective at finding defects early in product development [2], many defects inevitably go undiscovered. Therefore, several new variations have been attempted. Each of these alternatives, however, makes expensive, but uncontested assumptions: that simple process changes will yield measurable improvements, that inspections must include group meetings, and that an inspection’s effectiveness always justifies its cost. My research suggests that these assumptions may be incorrect. Some of the reasons for this are as follows:

Poor cost-benefit analyses. Typically, new methods are justified on the basis of improved effectiveness, without considering their additional costs. Clearly, in practice, a less effective, but cheaper method might be preferred to a more effective, but costly method. For example, a typical release of Lucent’s 5ESS switch ( =.5M lines of added and changed code per release on a base of 5M lines) can require roughly 1500 code inspections, each with a team of five or more reviewers, over a 25 week coding interval. For many organizations this is too expensive.

Simplistic assessment of costs. While person-effort is the traditional measure of cost, interval (calendar time) has been completely ignored. But in many parts of the industry it is the crucial determinant of product success. Inspections, with their distribution of materials, travel, and meetings, cause delays that significantly lengthen the development interval.

No analysis of internal structure. A few analyses have been done, but they usually compare the performance of one method to that of another. Almost no work has been done to isolate the factors that cause one method to be more or less cost-effective than another.

To address these issues, I built the following taxonomy of potential drivers of inspection costs and benefits.

1. structure (how the steps of the inspection are organized into a process);

2. techniques (how each step is carried out);

3. inputs (reviewer ability and code quality);

4. environment  (interactions with other inspections, project schedule, personal calendars); and

5. technology (tool support).

Based on this taxonomy, I conducted several families of experiments to evaluate and compare the effects of these drivers. This research was sponsored by a National Science Foundation Faculty Early Career Development Award.

Click on the buttons at the bottom of the page to see some of the specific findings on the drivers of inspection costs and benefits. we also have a set of PowerPoint slides (Identifying the Fundamental Drivers of Inspection Costs and Benefits) that summarize our research to date.