Home Contact

CMSC737: Fundamentals of Software Testing (Fall 2012)

Dept. of Computer Science, University of Maryland, College Park, MD, USA.

PROJECT

SUMMARY

This course has a significantly large and exciting project. Students will examine the practical issues surrounding the complex problem of software testing. Students will work individually to develop models for automated test generation, develop test oracles, evaluate the coverage of test cases, seed artificial faults, and develop coverage and fault matrices. They will study the practical limitations of exhaustive testing and evaluate published test sampling techniques. They will work with non-trivial open-source software and customize their own test harnesses. They will learn, from experience, about the importance of continuous build, integration, and functional regression testing.

PHASES

  • Phase 1; due date: Sep. 17 2012.
    • Goal: Hello World, Code Instrumentation, and Coverage Matrix. Downloading, building (from source), and running the JFCGUITAR test harness on a simple "Hello World" input. Setting up a Jenkins job to do the same automatically. Using cobertura for code instrumentation and coverage matrix computation.
    • Procedure: Consult the main GUITAR page for instructions on how to download, build, and execute the JFCGUITAR test harness. Also consult the cobertura web-page; install it, instrument the application under test, obtain coverage files for each test case, and create ONE coverage matrix (HTML) that shows the coverage of all test cases. (More details have been presented in class.)
    • Deliverables: Each student has been assigned a new Ubuntu machine. Make all your deliverables available on this machine. Deliverables include the complete running Jenkins job. Points will be awarded for the demo.
    • Grading session: During the grading session, I will execute your Jenkins job, which will have to download, build, and execute the test harness on the input. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness. The final output, i.e., the coverage matrix, will be checked.
    • Points: 20.
  • Phase 2; due date: Oct. 1, 2012.
    • Goal: Pick an Application, Test it, and Coverage Matrix.
    • Procedure: Each student has selected a Java Swing application to test. Obtain the SOURCE CODE for the latest version of this application. Follow the instructions above for Phase 1 to obtain the coverage matrix. Note that you need to run ALL Length 2 TEST CASES for this coverage matrix. Also note that the GUITAR test harness must also be built from source code. You cannot use any .class or .jar files for Java or the application under test.
    • Deliverables: Make all your deliverables available on the Ubuntu machine. Deliverables include the complete running Jenkins job. Points will be awarded for the demo.
    • Grading session: During the grading session, I will execute your Jenkins job, which will have to download, build, and execute the test harness on the input. The job should also download (from a code repository) the SOURCE CODE for the application under test and build it. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness and application. The final output, i.e., the coverage matrix, will be checked.
    • Points: 20.
  • Phase 3; due date: Oct. 22, 2012.
    • Goal: Fault Seeding and Fault Matrix.
    • Procedure: Use formal fault seeding techniques to seed 200 artificial faults in the application's source code. The faults must be seeded using a script by Jenkins. In the script, you must indicate the TYPE of fault being seeded.
    • Deliverables: Make all your deliverables available on the Ubuntu machine. Deliverables include the complete running Jenkins job. Points will be awarded for the demo.
    • Grading session: During the grading session, I will execute your Jenkins job, which will have to download, build, and execute the test harness on the input. The job should also download (from a code repository) the SOURCE CODE for the application under test and build it. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness and application. Jenkins will also run the fault seeding script. The final output, i.e., the coverage and fault matrices, will be checked.
    • Points: 20.
  • Phase 4; due date: Nov. 05, 2012.
    • Goal: Fine-tuning GUITAR for Maximum Code Coverage and Best Fault Matrix.
    • Procedure: There are several things you can do to get better performance from GUITAR. First, start your application in a state that gives you the best resulting GUI file (output of ripper). Second, ask GUITAR to ignore certain widgets that might prematurely terminate your application. Third, ask GUITAR to execute certain widgets. Fourth, provide better data for text fields.
    • Deliverables: Make all your deliverables available on the Ubuntu machine. Deliverables include the complete running Jenkins job. Points will be awarded for the demo.
    • Grading session: During the grading session, I will execute your Jenkins job, which will have to download, build, and execute the test harness on the input. The job should also download (from a code repository) the SOURCE CODE for the application under test and build it. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness and application. Jenkins will also run the fault seeding script. The final output, i.e., the coverage and fault matrices, will be checked. Your job should also show plots demonstrating the improvements in coverage and faults.
    • Points: 20.
  • Phase 5; due date: Nov. 26, 2012.
    • Goal: Writing Application-Specific Extensions for Even Better Coverage Matrix and Fault Matrix.
    • Procedure: Because GUITAR cannot interact with all possible types of widgets (the version that we are using in class interacts with all Java Swing library widgets), you need to tell GUITAR how to interact with new custom widgets. Manually examine the .GUI file to determine what widgets are being missed. Then add code (as GUITAR plug-ins) so that GUITAR can interact with these widgets.
    • Deliverables: Make all your deliverables available on the Ubuntu machine. Deliverables include the complete running Jenkins job. Points will be awarded for the demo.
    • Grading session: During the grading session, I will execute your Jenkins job, which will have to download, build, and execute the test harness on the input. The job should also download (from a code repository) the SOURCE CODE for the application under test and build it. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness and application. Jenkins will also run the fault seeding script. The final output, i.e., the coverage and fault matrices, will be checked. Your job should also show plots demonstrating the improvements in coverage and faults.
    • Points: 20.
  • Phase 6; due date: Dec. 3, 2012.
    • Goal: Analysis of Unexecuted Code.
    • Procedure: Report why some parts of the code were not executed, how they may be executed using GUI test cases, how they may be executed using non-GUI test cases, or if they are "dead code".
    • Deliverables: The above report.
    • Grading session: I will read the report.
    • Points: 10.