Carolyn Seaman, University of Maryland, Computer Science Department
Establishing a measurement program integrated into the development process is likely to help any organization achieve an in-depth understanding of its specific development issues and thereby lay a solid foundation for process improvement. My position is that the initiation of such a measurement program should always be preceded by a more qualitative study of the development environment and the issues to be addressed by measurement. This serves several purposes. Defining and enacting a measurement program takes a great deal of time and resources. Results may not be available for some time. A pre-study will provide some level of insight quickly. This helps with manager and developer buy-in and motivation. Furthermore, defining efficient and useful measurement procedures first requires a characterization of the place, i.e., organization structures, processes, issues, risks, etc. What is needed is a short term study to obtain early insight, at low cost, into the issues to be addressed by measurement.
I heartily agree with the statement that "the data is only as good as the process definition it is based on". However, this assumes that the data has been collected with reference to some process definition. This is not always the case. My position is that the most meaningful measures of a process are those based straightforwardly on its definition. Furthermore, such measures can be collected more efficiently than measures which are designed without reference to an adequate process definition. In fact, collecting data in a matter that is not "in sync" with the development process will be a struggle, and will result in suspect data.
This means that process definitions must include goals (because meaningful measurement must be relative to some goal) and must identify the relevant actors, artifacts, and tasks. It also means that one organization's relevant measures may not be the same as another's if the two organizations' processes are different. Consequently, the usefulness of "generic" or "universally applicable" measures is inherently limited. Both processes and measures must be defined with common sense and practicality, so that the process definition can in fact facilitate measurement, and so that the resulting data can in fact be used to tune the process.