Reading Techniques for
Fault Detection
Approach
Generating a family of reading techniques consists of building operational scenarios which require the reader to first create a model of the product, and then answer questions based on analyzing the model with a particular emphasis. Each reading technique in the family can be based upon a different abstraction and question set. The choice of abstraction and the types of questions may depend on the document being read, the problem history of the organization, or the goals of the organization.
Validation Strategy
One of our goals is to replicate the experiments in different environments in order to better understand how to tailor the techniques to different domains, organizations and cultures. So far, replications have been undertaken:
Project Status
The original experiment has been completed, along with some
replications in other contexts (see Validation Strategy).
We have made improvements to PBR based on these experiments and have
run a new study to evaluate the effectiveness of the changes. The
analysis of this study is still ongoing.
Results
Modeling
So far, two different families of reading techniques have been defined
for requirements documents: defect-based reading and perspective-based
reading.
Defect-based reading focuses on a model of the data and functions of the
requirements in a form of state machine notation. The different model views
are based upon focusing on some specific defects classes: data type inconsistency,
incorrect functions, and ambiguity or missing information. The analysis
questions are generated by combining and abstracting a set of questions
that are used in checklists for evaluating the correctness and reliability
of requirements documents.
Perspective-based reading focuses on different product customer perspective,
e.g., reading from the perspective of the software designer, the tester,
the end-user, the maintainer, the hardware engineer. The analysis questions
are generated by focusing predominantly in various requirements type errors,
e.g., incorrect fact, omission, ambiguity, and inconsistency.
Validation
Techniques from both families have been studied experimentally. and
a Lab
Package for the Empirical Investigation of Perspective-Based Reading
was developed. In the Lab package we present our experiences from previous
runs of this experiment, as well as our experimental design and materials,
in order that we may serve as a source of information for other researchers
to replicate the experiment in different environments.
In the defect-based reading study, we evaluated and compared defect-based
reading, ad hoc reading and checklist-based reading, with respect to their
effect on fault detection effectiveness in the context of an inspection
team. The study, a blocked subject-project, was replicated twice using
graduate students at the University of Maryland. Major results were that
(1) the defect-based readers performed better than ad hoc and checklist
readers with an improvement of 35%; (2) the defect-based reading procedures
helped reviewers focus on specific fault classes but were no less effective
at detecting other faults; and (3) checklist reading was no more effective
than ad hoc reading.
In the perspective-based reading study, we evaluated and compared perspective-based
reading and NASA's current reading technique with respect to their effect
on fault detection effectiveness in the context of an inspection team.
Three types of perspective-based reading techniques were defined and studied:
tester-based, designer-based, and user-based. The study, again a blocked
subject-project, was replicated twice in the SEL environment with NASA
professionals. Major results were that (1) perspective-based reading was
effective for generic documents both at the individual level and team level;
and (2) perspective-based reading was effective for the NASA documents
at the team level. It was felt that the techniques could be better tailored
to the NASA style document to improve individual scores.
