Projects of the
Experimental
Software Engineering Group at the University of Maryland
|
Experimental Designs (2)
|
Approach
|
This project has three phases:
- A literature search was performed in order to identify those techniques
that have been used in previous software development projects. We have
identified 12 experimental design, broken down into three broad categories:
historical methods, observational methods, and controlled methods. Roughly
speaking, the historical methods are those most associated with fields such
as archaeology where completed projects are studied, the observational methods
are like astronomy where you have little control over your environment, and the
controlled methods are used heavily in fields such as psychology where
desired effects can be replicated on numerous subjects.
- A literature search of over 600 published software engineering
papers was performed in order to identify how well these 12 methods are
used in practice.
- We are currently studying how well these techniques apply to
industrial environments by evaluating several organizations in order to
understand how well they undertake technology transition tasks.
|
Validation Strategy
|
A literature search in fields other than computer science has been undertaken
to see how well the computer science models relate to other disciplines.
Evaluation studies in other organization is being performed in order to
understand the merits of the various experimental models being proposed.
|
Project Status
| Active - Industrial evaluation underway |
Results
|
An initial classification of 12 experimental designs has been developed.
The initial study of 612 published papers in 1985, 1990, and 1995 shows that:
- About 30% of software engineering papers have no experimental
component and another 30% have a weak form of experimental validation.
- If theoretical papers are deleted from the list of no experimentation
papers, then the percent of papers with no experimentation drops to under 20% --
not far removed from the numbers in physics.
- However, the 30% of papers that were subjectively classified as
assertions (i.e., weak experimental designs) seems high compared to other
fields.
|
