PROJECT

SUMMARY

This course has a significantly large and exciting project. Students will examine the practical issues surrounding the complex problem of software testing. Students will work in small teams to develop models for automated test generation, develop test oracles, evaluate the coverage of test cases, seed artificial faults, and develop coverage and fault matrices. They will study the practical limitations of exhaustive testing and evaluate published test sampling techniques. They will work with non-trivial open-source software and customize their own test harnesses. They will learn, from experience, about the importance of continuous build, integration, and functional regression testing.

TEAMS

For the first few phases, the class will be divided into the following teams.

JFCGUITAR	SWTGUITAR	UNOGUITAR	iPhoneGUITAR	AndroidGUITAR	WebGUITAR
Rose Kirby	Hyunjong Cho		Youndo Lee	William Goh	Arun Balasubramanian
Ramakrishna Padmanabhan	Weiqing Wu	Bryan Ta	Chris Hill	Jinseong Jeon	Abdul Quamar
		Wikum Dinalankara	Rongjian Lan	Ali Sistani	Ashwin Kumar
			Hua He	Noseong Park	Jayanta Mondal
			Yulu Wang

All teams share the fundamental steps of the project. The only difference is that each team will work on a specific platform. For example, the iPhoneGUITAR team will develop a test harness (prototype provided) for the iPhone platform and use that harness to test open-source apps developed for the iPhone. The JFCGUITAR and SWTGUITAR teams follow a slightly modified schedule to accommodate the different nature of their tasks.

PHASES

• Phase 1; due date: Sep. 7, 2011.
  • Goal: Downloading, building (from source), and running the JFCGUITAR test harness on a simple "Hello World" input. Setting up a Jenkins job to do the same automatically on a new slave machine.
  • Procedure: Consult the main GUITAR page for instructions on how to download, build, and execute the JFCGUITAR test harness.
  • Deliverables: Running Jenkins job. Points will also be awarded for the demo.
  • Grading session: During the grading session, you will have access to a new machine (Windows or Linux) that is connected to the Internet. You will have to download, build, and execute the test harness on the input. You will also need to install Ant, Subversion, Java, C++, and any other tools needed to build and run the harness. Your Jenkins job will be run and tested. It should be able to handle a newly installed slave, i.e., your Jenkins job should check the slave for necessary software and, if needed, install it automatically.
  • Points: 10.
• Phase 2; due date: Oct. 7, 2011.
• due date for SWTGUITAR and JFCGUITAR: Sep. 14, 2011.
  • Goal:Enhancing and integrating your team's test harness into the GUITAR framework. Implementing a "Hello World" example application. Updating wiki for harness. Adding Jenkins jobs to regression test harness on "Hello World".
  • Procedure:A prototype test harness is provided to each team. You will work closely with the course instructor to enhance this prototype and integrate it into the core GUITAR framework.
  • Deliverables:The complete integrated test harness that is capable of implementing the entire GUITAR workflow.
  • Grading session:During the grading session, your Jenkins job will be executed. It will have access to a new slave machine (Windows or Linux) that is connected to the Internet. The job will have to download (from the main GUITAR subversion repository), build, and execute your test harness on a "Hello World" example test application that you will implement.
  • Points:200.
  • Points for SWTGUITAR and JFCGUITAR:20.
• Phase 3; due date: Oct. 22, 2011.
• due date for SWTGUITAR and JFCGUITAR: Oct. 14, 2011.
  • Goal:Selecting test applications, seeding faults, executing the GUITAR "default" workflow on the applications, obtaining coverage and fault matrices. Adding Jenkins jobs to regression test harness on new test applications.
  • Procedure:Select your test applications and get them approved by the instructor. In general, the applications should be open-source, popular, and contain a non-trivial number of lines of code.
  • Deliverables:Scripts and programs to automate the entire 'fault seeding to matrices' workflow.
  • Grading session:Using your own fully automatic script.
  • Points:100.
  • Points for SWTGUITAR and JFCGUITAR:170.
• Phase 4; due date: Nov. 7, 2011.
• due date for SWTGUITAR and JFCGUITAR: Nov. 14, 2011.
  • Goal:Implementing custom widget and event code for the test applications, executing the GUITAR "custom" workflow on the applications, obtaining coverage and fault matrices. Updating Jenkins jobs to regression test harness on test applications with custom widget and event code.
  • Points:100.
  • Points for SWTGUITAR and JFCGUITAR:200.
• Phase 5; due date: Nov. 22, 2011.
• due date for SWTGUITAR and JFCGUITAR: Dec. 7, 2011.
  • Goal:Enhancing the test cases with data to improve code coverage, adding more initial states for the applications, executing the GUITAR "custom" workflow on the applications, obtaining coverage and fault matrices. Only dead code should be left untested. Updating Jenkins jobs to use new data in test cases.
  • Points:100.
  • Points for SWTGUITAR and JFCGUITAR:200.
• Phase 6; due date: Dec. 7, 2011.
  • Goal:Finally packaging the team's test harness. Conducting a small experiment to study test suite reduction, minimization, and prioritization.
  • Points:90.