Download: PDF.
“Designing and comparing automated test oracles for GUI-based software applications” by Qing Xie and Atif M. Memon. ACM Transactions on Software Engineering and Methodology, vol. 16, no. 1, 2007, ACM Press.
Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This paper shows that the test oracle, a mechanism that determines whether a software executed correctly for a test case, also significantly impacts the fault-detection effectiveness and cost of a test case. Graphical user interfaces (GUIs), which have become ubiquitous for interacting with today's software, have created new challenges for test oracle development. Test designers manually “assert" the expected values of specific properties of certain GUI widgets in each test case; during test execution, these assertions are used as test oracles to determine whether the GUI executed correctly. Since a test case for a GUI is a sequence of events, a test designer must decide (1) what to assert, and (2) how frequently to check an assertion, e.g., after each event in the test case or after the entire test case has completed execution. Variations of these two factors significantly impact the fault-detection ability and cost of a GUI test case. A technique to declaratively specify different types of automated GUI test oracles is described. Six instances of test oracles are developed and compared in an experiment on four software systems. The results show that test oracles do affect the fault-detection ability of test cases in different and interesting ways: (1) test cases significantly lose their fault-detection ability when using “weak" test oracles, (2) in many cases, invoking a “thorough" oracle at the end of test case execution yields the best cost-benefit ratio, (3) certain test cases detect faults only if the oracle is invoked during a small “window of opportunity" during test execution, and (4) using thorough and frequently-executing test oracles can make up for not having long test cases.
Download: PDF.
BibTeX entry:
@article{XieMemonTOSEM2006,
author = {Qing Xie and Atif M. Memon},
title = {Designing and comparing automated test oracles for GUI-based
software applications},
journal = {ACM Transactions on Software Engineering and Methodology},
volume = {16},
number = {1},
pages = {4},
publisher = {ACM Press},
address = {New York, NY, USA},
year = {2007},
issn = {1049-331X}
}
Back to Atif Memon's Publications.