Co-located with The IEEE International Conference on Software Testing Verification and Validation

Please look at http://guitar.sourceforge.net for more work on automated GUI testing.

First International Workshop on

TESTing Techniques & Experimentation Benchmarks

for Event-Driven Software (TESTBEDS 2009)

Theme for 2009: GUI-Based Applications

Important Dates

Submission

Organization

Contact: testbeds2009@cs.umd.edu

Workshop Program

09:00 - 09:30	Opening Remarks
09:30 - 10:30	Keynote Are You Sure What Failures Your Tests Produce? by Prof. Lee J. White Abstract: In 2000, the Prof. White reported a new systematic approach for testing GUI systems. In several follow up studies, this approach was evaluated using large commercial GUI systems for both testing and regression testing. An overview of these studies will be presented. Rather than concentrating on the details or virtues of this testing approach, three results that emerged from these GUI software studies will be explored: 1. The use of memory tools uncovered substantial observability problems in that more defects were detected after memory tools were used than before they were used. 2. In GUI systems, it was discovered that defects manifested themselves as different failures (or not at all) for different environments. 3. In the testing of GUI systems, both design tests and implementation tests; design tests are used based upon the design of the GUI system, and how it should work; the tests are derived from this model. Implementation tests are based upon the actual GUI implementation, may include additional behaviors different from the original design; the tests are derived from this model. The design tests are clearly a subset of the implementation tests. In the evaluation of various GUI systems, it was found that there were many more implementation tests (and thus behaviors) in the actual GUIs than the designer had intended. These three observations have very important implications for tests executed by the developer as compared to the identical tests executed on the same GUI system by the user, and this will be thoroughly discussed. There will be discussions as to whether these effects also occur to this extent in the testing of software systems other than GUI systems. Biography: Lee White is currently an Emeritus Professor of Computer Science in the EECS Department at Case Western Reserve University. He received his BSEE degree in 1962 from the University of Cincinnati, and the MSc and PhD in Electrical and Computer Engineering from the University of Michigan in 1963 and 1967. He has served as a Chair of computing departments at the Ohio State University, at the University of Alberta and at CWRU. His research interests are in software engineering, primarily in software testing and in software maintenance. He has consulted for a number of industrial firms, including IBM, General Electric Research Laboratory, Parker-Hannifin, Monsanto Research Laboratory, North American Rockwell and United States Steel. Lee White was one of the early researchers studying the testing of GUI systems. In 2000, he reported a systematic testing approach, and demonstrated its effectiveness on several small GUI systems. In 2001, he reported the results of applying this testing approach to four large commercial GUI systems; also he noted observability problems with the use of memory tools in testing. In 2003, he reported the results of using a modified regression testing approach on a large GUI system change. Again an even larger observability problem was detected using memory tools. In addition, there was evidence that defects manifested themselves as different failures (or not at all) for different environments (here the two software versions).
10:30 - 11:00	Coffee Break
11:00 - 12:30	Session 1 Paper 1: Creating GUI Testing Tools Using Accessibility Technologies Mark Grechanik, Qing Xie, Chen Fu Paper 2: An Extensible Heuristic-Based Framework for GUI Test Case Maintenance Scott McMaster, Atif Memon Paper 3: Structural Feature Extraction for GUI Test Enhancement Christof J. Budnik, Fevzi Belli, Axel Hollmann
12:30 - 14:00	Lunch
14:00 - 15:30	Session 2 Paper 4: Towards Dynamic Adaptive Automated Test Generation for Graphical User Interfaces Xun Yuan, Myra Cohen, Atif Memon Paper 5: An Initial Study of Customer-Reported GUI Defects Brian Robinson, Penelope Brooks Presentation & Discussion: Benchmarking for EDS Testing Atif Memon, Myra Cohen
15:30 - 16:00	Closing Remarks and Coffee

Workshop Overview & Goals

As computers and embedded devices play an increasingly important role aiding end users, researchers, and businesses in today's inter-networked world, several classes of event-driven software (EDS) applications are becoming ubiquitous. Common examples include graphical user interfaces (GUIs), web applications, network protocols, embedded software, software components, and device drivers. An EDS takes internal/external events (e.g., commands, messages) as input (e.g., from users, other applications), changes its state, and sometimes outputs an event sequence. An EDS is typically implemented as a collection of event handlers designed to respond to individual events. Nowadays, EDS is gaining popularity because of the advantages this ``event-handler architecture'' offers to both developers and users. From the developer's point of view, the event handlers may be created and maintained fairly independently; hence, complex system may be built using these loosely coupled pieces of code. In interconnected/distributed systems, event handlers may also be distributed, migrated, and updated independently. From the user's point of view, EDS offers many degrees of usage freedom. For example, in GUIs, users may choose to perform a given task by inputting GUI events (mouse clicks, selections, typing in text-fields) in many different ways in terms of their type, number and execution order.

Quality assurance (QA) is becoming increasingly important for EDS as its correctness may affect the quality of the entire system in which the EDS operates. Software testing is a popular QA technique employed during software development and deployment to help improve its quality. During software testing, test cases are created and executed on the software. One way to test an EDS is to execute each event individually and observe its outcome, thereby testing each event handler in isolation. However, the execution outcome of an event handler may depend on its internal state, the state of other entities (objects, event handlers) and/or the external environment. Its execution may lead to a change in its own state or that of other entities. Moreover, the outcome of an event's execution may vary based on the sequence of preceding events seen thus far. Consequently, in EDS testing, each event needs to be tested in different states. EDS testing therefore may involve generating and executing sequences of events, and checking the correctness of the EDS after each event. Test coverage may not only be evaluated in terms of code, but also in terms of the event-space of the EDS. Regression testing not only requires test selection, but also repairing obsolete test cases. The first major goal of this workshop is to bring together researchers and practitioners to discuss some of these topics.

One of the biggest obstacles to conducting research in the field of EDS testing is the lack of freely available standardized benchmarks containing artifacts (software subjects and their versions, test cases, coverage-adequate test suites, fault matrices, coverage matrices, bug reports, change requests), tools (test-case generators, test-case replayers, fault seeders, regression testers), and processes (how an experimenter may use the tools and artifacts together) [see http://www.cs.umd.edu/~atif/newsite/benchmarks.htm for examples] for experimentation. The second major goal of this workshop is to promote the development of concrete benchmarks for EDS.

To provide focus, the 2009 event will only examine GUI-based applications (no web applications). As this workshop matures, we hope to expand to other types of EDS (e.g., web applications).

[For Web Testing, please consider submitting to this year’s International Workshop on Web Testing: (WebTest 2009)]

Important Dates

· Submission of Full Papers: Friday, 16 January, 2009

· Notification: Friday, 27 February, 2009

· Camera-Ready: Friday, 20 March, 2009

· Workshop: April 4, 2009

Submission

The workshop solicits submission of:

· Full Papers (max 8 pages)

· Position Papers (max 4 pages) [what is a position paper?]

· Demo Papers (max 4 pages) [usually papers describing implementation-level details (e.g., tool, file format, structure) that are of interest to the community]

· Industrial Presentations (slides)

All submissions will be handled through http://www.easychair.org/conferences/?conf=testbeds2009.

Industrial presentations are submitted in the form of presentation slides and will be evaluated by at least two members of the Program Committee for relevance and soundness.

Each paper will be reviewed by at least three referees. Papers should be submitted as PDF files in standard IEEE two-column conference format (Latex , Word). The workshop proceedings will be published on this workshop web-page. Papers accepted for the workshop will appear in the IEEE digital library, providing a lasting archived record of the workshop proceedings.

Organization

General Chair

· Atif M Memon, University of Maryland, USA.

Program Committee

· Fevzi Belli, University of Paderborn, Germany.

· Renee Bryce, Utah State University, USA.

· S.C. Cheung, The Hong Kong University of Science and Technology, Hong Kong.

· Myra Cohen, University of Nebraska – Lincoln, USA.

· Chin-Yu Huang, National Tsing Hua University, Taiwan.

· Scott McMaster, Amazon.com, USA.

· Brian P Robinson, ABB Corporate Research, USA.

· Qing Xie, Accenture Technology Labs, Chicago, USA.

· Xun Yuan, Google, USA.