Workshop on the Evaluation of Software Defect Detection Tools

Sunday, June 12th, 2005
Co-located with PLDI 2005

Workshop co-chairs: Bill Pugh (University of Maryland) and Jim Larus (Microsoft Research)

Program chair: Dawson Engler (Stanford University)

Program Committee: Andy Chou, Manuvir Das, Michael Ernst, Cormac Flanagan, Dan Grossman, Jonathan Pincus, Andreas Zeller

Overview

This workshop is intended to provide a forum for the community of researchers and developers of software defect detection tools to discuss issues related to the development and evaluation of these tools. People working on these tools share many common problems and experiences, which typically do not find their way into academic publications or presentations. This community knowledge is invaluable to other tool developers and can help identify new and promising research directions. This workshop provides an opportunity to discuss topics such as:

• Types of defects that occur in software.
• Categorization of defects by tractability to automated detection
• Defect importance, prioritization and false positives.
  • Significance/consequences of errors found by by automated tools.
  • Defect prioritization and presentation
  • Prevalence of false positives or benign defects
• Experience using specific static or dynamic analysis techniques to identify defects
• Annotations, enhanced type systems or specification languages that aid in finding defects
• Mining information from defect databases and version control histories
• User interface issues for defect detection tools
• Library, project or domain-specific defect patterns or checkers
• Post-deployment defect detection and reporting tools

Proceedings

Copies of the position statements are available (linked below from the schedule). Contact authors for copies of the research presentations. The schedule also contains links to presentations where available.

This is a workshop, and it is the express desire and intent of the conference committee that discussion of research at the workshop not preclude publication of closely related material at conferences or journals. This is one of the reasons that the research presentations are not available from this web site.

Schedule: Sunday, June 12th

Time	What
8:30 am	Discussion on Soundness The Soundness of Bugs is What Matters, Patrice Godefroid, Bell Laboratories, Lucent Technologies (slides) Soundness and its Role in Bug Detection Systems, Yichen Xie, Mayur Naik, Brian Hackett, Alex Aiken, Stanford University
9:15 am	break
9:30 am	Research presentations Locating Matching Method Calls by Mining Revision History Data, Benjamin Livshits, Thomas Zimmermann, Stanford University Evaluating a Lightweight Defect Localization Tool, Valentin Dallmeier, Christian Lindig, Andreas Zeller, Saarland University (slides)
10:30 am	break
10:45 am	Defect Detection at Microsoft - Where the Rubber Meets the Road, Manuvir Das, Center for Software Excellence, Microsoft (slides: pdf; ppt)
11:15 am	Discussion of Deployment and Adoption The Open Source Proving Grounds, Ben Liblit, University of Wisconsin-Madison Issues in deploying SW defect detection tools, David Cok, Eastman Kodak R&D (slides) False Positives Over Time: A Problem in Deploying Static Analysis Tools, Andy Chou, Coverity (slides)
12 noon	lunch
1:00 pm	Research presentations Model Checking x86 Executables with CodeSurfer/x86 and WPDS++, Gogul Balakrishnan, Thomas Reps, Nick Kidd, Akash Lal, Junghee Lim, David Melski, Radu Gruian, Suan Yong, Chi-Hua Chen, Tim Teitelbaum, Univ. of Wisconsin (slides) Empowering Software Debugging Through Architectural Support for Program Rollback, Radu Teodorescu, Josep Torrellas, UIUC Computer Science EXPLODE: A Lightweight, General Approach to Finding Serious Errors in Storage Systems, Junfeng Yang, Paul Twohey, Ben Pfaff, Can Sar, Dawson Engler, Stanford University
2:30 pm	break
2:45 pm	Research presentations Experience from Developing the Dialyzer: A Static Analysis Tool Detecting Defects in Erlang Applications, Kostis Sagonas, Uppsala University Soundness by Static Analysis and False-alarm Removal by Statistical Analysis: Our Airac Experience, Yungbum Jung, Jaehwang Kim, Jaeho Sin, Kwangkeun Yi, Seoul National University
3:45 pm	break
4:00 pm	Discussion of Benchmarking Dynamic Buffer Overflow Detection, Michael Zhivich, Tim Leek, Richard Lippmann, MIT Lincoln Laboratory Using a Diagnostic Corpus of C Programs to Evaluate Buffer Overflow Detection by Static Analysis Tools, Kendra Kratkiewicz, Richard Lippmann, MIT Lincoln Laboratory (slides) BugBench: A Benchmark for Evaluating Bug Detection Tools, Shan Lu, Zhenmin Li, Feng Qin, Lin Tan, Pin Zhou, Yuanyuan Zhou, UIUC (slides: pdf; ppt) Benchmarking Bug Detection Tools. Roger Thornton, Fortify Software (slides) A Call for a Public Bug and Tool Registry, Jeffrey Foster, Univ. of Maryland (slides) Bug Specimens are Important, Jaime Spacco, David Hovemeyer, William Pugh, University Maryland (slides) NIST Software Assurance Metrics and Tool Evaluation (SAMATE) Project, Michael Kass, NIST (slides)
5:00 pm	Discussion of New Ideas Deploying Architectural Support for Software Defect Detection in Future Processors, Yuanyuan Zhou, Josep Torrellas, UIUC (slides: pdf; ppt) Using Historical Information to Improve Bug Finding Techniques, Chadd Williams, Jeffrey Hollingsworth, Univ. of Maryland Locating defects is uncertain, Andreas Zeller, Saarland University (slides: pdf; mov) Is a Bug Avoidable and How Could It Be Found?, Dan Grossman, Univ. of Washington (slides)
5:45 pm	wrap up and discussion of future workshops
6:00 pm	done