Assignment 3

Analysis of Fault Matrix

This assignment will be partitioned into very small sub-assignments with tight deadlines to help the students manage their time effectively throughout the semester.
  • Assignment 3.1: Due date: Feb. 10, 2007: Write a Java program to compute the average number of faults detected by a smoke and comp test case per application. E-mail the Java code and result to the TA. SUBJECT of the e-mail should be "CMSC 433 -- Assignment 3.1". (5% of Assignment 3 grade.)
  • Assignment 3.2: Due date: Mar. 15, 2007: Develop a strategy to analyze the data in the fault matrices and "depth vectors." The goal is to determine whether there is a correlation between testing techniques (comp vs. smoke) and the depths of the faults they detect. Document your strategy using a simple example. Validate the strategy/document by discussing it with the course instructor. Finally, e-mail the document to the TA. SUBJECT of the e-mail should be "CMSC 433 -- Assignment 3.2". (15% of Assignment 3 grade.)
  • Assignment 3.3: Due date: May 5, 2007: Implement the above strategy in Java. Make sure that you use at least four design patterns in your code. Submit the code and a report that justifies the use of particular design patterns. Also remember to write JUnit tests and JavaDoc for all methods/classes. E-mail the document to the TA. SUBJECT of the e-mail should be "CMSC 433 -- Assignment 3.3". (80% of Assignment 3 grade.) Points will be awarded for solutions that are different from those submitted by other students in your class; it is in your interest to work independently this time.

    Description

    We define a fault matrix as a summary of test results. The matrix has N rows, one for each test case, and M columns, one for each fault. A "0" entry in cell (n, m) shows that test case n did not detect fault m; a "1" entry shows otherwise.

    Consider the following four fault matrices for four applications (testing method used is called Smoke; it yields smoke test cases):

  • Smoke_TerpPaint.txt
  • Smoke_TerpPresent.txt
  • Smoke_TerpSpreadSheet.txt
  • Smoke_TerpWord.txt

    Consider the following four fault matrices for the same four applications (testing method used is called Comp; it yields comp test cases):

  • Comp_TerpPaint.txt
  • Comp_TerpPresent.txt
  • Comp_TerpSpreadSheet.txt
  • Comp_TerpWord.txt

    A purely syntactic treatment is used to define "nesting depth" of a detected fault. It is borrowed from the code metric "nesting depth" reported by the ccount tool (http://www.chris-lott.org/resources/cmetrics/ccount.tar.gz). In summary, the depth for each program statement is computed "locally" within the context of its containing method. All program statements contained directly within a method body have a nesting depth of 1. Statements directly within the body of a loop or a conditional statement have a nesting depth of "1 plus the depth of the containing loop or conditional statement." Hence a program statement inside the body of a two nested conditional statements has a nesting depth of 3. The nesting depth for each seeded fault is computed and stored in the following vectors:

  • FaultDepths_TerpPaint.txt
  • FaultDepths_TerpPresent.txt
  • FaultDepths_TerpSpreadSheet.txt
  • FaultDepths_TerpWord.txt