Two replications of the experiment at the University of Kaiserslautern

A summary of two runs of the two replications can be found in the summary report.

The Participants

This experiment was replicated twice. The subjects of the replications were graduate students of the Computer Science Department at the University of Kaiserslautern, Germany. They were enrolled in the basic Software Engineering course lasting a semester. This course teaches the basic Software Engineering principles. The course is supplemented by practical exercises. In the first replication 25 students, in the second replication 26 students participated. All were volunteers.

The subjects were assigned to the PBR roles randomly.

The Techniques

PBR was used with the three perspectives described in this lab package: designer, tester, user.

The comparison technique was an Ad-hoc reading approach. i.e., no specific guidelines were given to students how to read the document. It is a non-systematic technique in which reviewers have general and identical responsibilities.

The Documents Used

Only the generic documents were used in both replications.

The Setting

The experiment was conducted in a classroom environment at the University of Kaiserslautern. Participants reviewed the documents during one and a half hour review sessions, during which they were isolated from distractions and interruptions.

Other Changes

• Design Changes
  In contrast to the original experiment, we performed real team meetings in our second replication of the experiment. We found no significant differences between nominal and real team meetings which supports that the nominal team ananlysis is an appropriate technique. A more detail analysis revealed that by perfoming real team meetings one must consider typical meeting effects, i.e., additional defects are not detected and defects detected individually are not reported.
  
• Analysis Changes
  In addition to the analysis provided in the original experiment we performed a meta-analysis for the individual and team analysis to combine the effects of our replications and the original experiments to investige whether there is an effect of adequate size.

Comments from Participants

After the replications, we solicited comments from the participants. Most students suggest to increase the time available for individual defect detection

Summary of Results

The replications corroborate the results of the previous experimens in Maryland. PBR teams were seen to have an improved (statistically significant at the 0.05-level) coverage of defects, for the generic documents. It was also shown that individuals using PBR performed better on the generic documents than Ad-hod reviewers. As this result was only statistically significant for the second replication, we perfomed a meta-analysis of our replications and the original experiments. It turned out that the effect of PBR on individual defect detection effectiveness is statistically significant.

Web Accessibility