The purpose is to determine whether an individual performs differently when reviewing with PBR than when using the usual technique. The dependent variable is the defect rate, in this case, the percentage of true defects found by a single reviewer with respect to the total number of defects in the inspected document.
Documents in different domains are analyzed separately. Thus, in the case of our experimental design, a separate analysis is run for the generic and domain-specific documents.
For each analysis, the corresponding design is a 2 x 2 factorial experiment with repeated measures in blocks of size 2 (Winer, 1991). This analysis involves two factors, or treatments, on which there are repeated measures: the reading technique (RTECH) and the document reviewed (DOC). Both the independent variables are on a nominal scale. The reading technique has two levels: PBR and usual. Also the document reviewed can assume two levels which depend on the problem domain considered: if generic domain, ATM and PG; in the example NASA domain, NASA_A and NASA_B.
Subjects are assigned to one of the two groups, or blocks, and have repeated measures on the two treatments within each block.
These plans use all the treatment conditions required for the complete factorial experiment, but the block size is reduced from four to two; that is, within any block only two treatment combinations appear instead of the four possible treatment combinations. The cost of this reduction in block size is the loss of some information on interactions. Precisely, the interaction RTECH x DOC is totally confounded with the group main effect. This means that the two-factor interaction cannot be estimated separately from the group effect. However, we do not expect that this interaction is important because both the documents are within the same problem domain. In exchange, both of the main effects are completely within-block effects, and thus independent from the subject variability.
We do not assume that there will be equal numbers of participants in the two groups. Thus, in order to perform the analysis of variance for unbalanced design, the GLM procedure is used in the SAS statistical package (SAS, 1989).