Used in: Journal paper in progress
(Paper | Full list of publications)
Disclaimer: Scripts on this page are provided for documentation only. Each contains comments describing inputs, outputs, and major algorithms. These scripts make assumptions about the user's file system and, if executed, could have unfortunate consequences, such as deleting your files.
(Items marked with * are not available for download. They may be available on DVD by request.)
Each zipped directory contains src and bin (the uninstrumented application, slightly modified by the experimenter), src-ins and bin-ins (src instrumented with instr and Emma, with _prof/prof.java modified by the experimenter), and additional files used in the experiment. Applications were modified to make them more amenable to GUI ripping and testing---for example, by making JFileChoosers open to a default directory and by removing names of open files from window titles. Applications' configuration files were modified for the same reason---for example, by disabling automatic saves and version checks and by removing expandable menus of previously opened files. A simple input file was created for each application.
Included in application under test.
Mutants were generated with MuJava (and substantial manual effort to deal with GUI classes, which MuJava does not handle). The script pick_mutants documents how the mutants here were selected from all mutants generated for the applications. It shows which application classes, mutant types, and individual mutants were intentionally skipped during mutant selection. Application classes were skipped if their corresponding events were deliberately disabled in the GUI/EFG (see Supporting files); mutant types, if they changed more than one line of source code; and individual mutants, if they made the mutated code difficult to instrument.
The GUI and EFG files generated by GUITAR were modified manually for several reasons: to fix errors that occurred during EFG and test-case generation, to add some missing nodes/edges and remove some incorrect ones, to change absolute file names in JFileChooser events to relative file names, and to permanently disable events that would interfere with the experiment (including events related to printing, events that open other applications or network connections, events that interact with the file system in an uncontrolled way, and events for JMenu widgets, which are superfluous because GUITAR can access JMenuItem widgets directly).
In previous benchmarks, a set of test cases was generated and then replayed. Because of limitations in GUITAR and inherent weaknesses in the EFG model, many of those test cases would not run to completion. In this benchmark, test cases are generated and replayed one at a time to guarantee that each runs to completion. The scripts make_packages, replay_package, and JaymieTestCaseGenerator.java document this process.
In the zip file of test suites, each numbered directory corresponds to a SuiteNum and each file in that directory to a TstNum in the test steps summary file. The script steps documents how the test steps summary was computed.
Some results in the test steps summary are incorrect! See the note in the Report section.
The scripts make_packages and replay_package document how coverage information was collected.
The zip file of test case coverage contains directory cov-cln (coverage for the clean, non-mutated version), which in turn contains numbered directories corresponding to test suites and, within those, numbered files corresponding to test cases and test steps. For example, 798/5.ec is the Emma coverage file for TstNum 5 in SuiteNum 798; 798/5.txt-2 is the Instr output immediately after StepNum 2 in TstNum 5 in SuiteNum 798; and 798/5.txt-14, 798/5.txt-final, and 798/5.txt are Instr output collected after the last step (StepNum 14) in TstNum 5 in SuiteNum 798.
The scripts fault_cov_instances, fault_cov_instances_with_init and fault_cov_with_init document how the fault coverage instances summary and the fault coverage summary were computed.
The script oracle_package documents how coverage (Cov) and fault detection (Det) values were computed, and the script suites documents how the suite summary was computed. For CrosswordSage 0.3.5, fault-detection results for the first 100 suites were checked manually, and no incorrect results were found. In the test-steps summary, only one test case was (wrongly) reported to detect a fault before the faulty line was covered. For Freemind 0.7.1, some incorrect results were found among the first 100 suites, and 13 test cases were reported to detect a fault before the faulty line was covered, so all results were checked for this application. The 80 cases where the fault was wrongly reported as detected were corrected in the suite summary. Note that the test-steps summary was not corrected.
The R script data_set.R documents how all of the data were put together into data sets for analysis.