Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems

Download: PDF.

“Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems” by Cemal Yilmaz, Arvind S. Krishna. Atif Memon, Adam Porter, Douglas C. Schmidt, Aniruddha Gokhale, and Balachandran Natarajan, in ICSE '05: Proceedings of the 27th international conference on Software engineering, (New York, NY, USA), 2005, pp. 293-302.

Abstract

Developers of highly configurable performance-intensive software systems often use a type of in-house performance-oriented "regression testing" to ensure that their modifications have not adversely affected their software's performance across its large configuration space. Unfortunately, time and resource constraints often limit developers to in-house testing of a small number of configurations and unreliable extrapolation from these results to the entire configuration space, which allows many performance bottlenecks and sources of QoS degradation to escape detection until systems are fielded. To improve performance assessment of evolving systems across large configuration spaces, we have developed a distributed continuous quality assurance (DCQA) process called main effects screening that uses in-the-field resources to execute formally designed experiments to help reduce the configuration space, thereby allowing developers to perform more targeted in-house QA. We have evaluated this process via several feasibility studies on several large, widely-used performance-intensive software systems. Our results indicate that main effects screening can detect key sources of performance degradation in large-scale systems with significantly less effort than conventional techniques.

Download: PDF.

BibTeX entry:

@inproceedings{YilmazICSE2005,
   author = {Cemal Yilmaz and Arvind S. Krishna and Atif Memon and Adam
	Porter and Douglas C. Schmidt and Aniruddha Gokhale and
	Balachandran Natarajan},
   title = {Main effects screening: a distributed continuous quality
	assurance process for monitoring performance degradation in
	evolving software systems},
   booktitle = {ICSE '05: Proceedings of the 27th international conference
	on Software engineering},
   pages = {293--302},
   publisher = {ACM Press},
   address = {New York, NY, USA},
   year = {2005},
   isbn = {1-59593-963-2}
}

Back to Atif Memon's Publications.