IWSED-95 Participant Questionnaire
IWSED-95
International Workshop on Software Engineering Data
Utilization of Defect and Cost Data in Process and Risk Management
Version 1.01, Status: Final, Release date of this questionnaire: July 1/July 14, 1995
Co-sponsored by University of Maryland and the IFIP
October 9-11, 1995
University of Maryland, College Park, Maryland, U.S.A.
Further information about the workshop available from following sources:
Email: jkontio@cs.umd.edu
WWW: http://www.cs.umd.edu/projects/SoftEng/ESEG/iwsed/iwsed95/
FTP: ftp.cs.umd.edu/pub/sel/iwsed95
Workshop Chairs
Victor R. Basili Koji Torii
University of Maryland Nara Institute of Science and
Technology
Please, return this questionnaire to the following address by August 11, 1995.
Jyrki Kontio
University of Maryland
Computer Science Department
A.V.Williams Building
College Park, MD 20742
U.S.A.
Phone: 301-405-2743
Fax: 301-405-6707
Email: jkontio@cs.umd.edu
Note: We would prefer email replies to this questionnaire. An electronic version of the questionnaire can be obtained from the author by requesting through email or directly from the following FTP site: ftp.cs.umd.edu/pub/sel/iwsed95.
This paper contains the questionnaire for the IWSED-95 workshop. The theme of the IWSED-95 is the Utilization of Defect and Cost Data in Process and Risk Management.
All IWSED-95 participants will fill out a questionnaire in advance about their software engineering data. This questionnaire includes questions about the participant's organization, its objectives for data collection, what kinds of data have been collected, and how data is used. Based on the returned questionnaires, some participants will be contacted and asked to give presentations on their experiences at the workshop.The questionnaire results will be presented at the IWSED-95 and some of the data will be used in the working groups.
When filling out this questionnaire, we would prefer that you refer to the processes and practices that take place within a single software development organization. We leave it up to you to define what this actually means in your organization, whether it is a single department operating at single site or several units with similar practices. However, if you do not have data available about the whole development organization, you can use this questionnaire to characterize the situation of a single project but be sure to indicate this in question 3.
You may leave some questions unanswered if you do not have the information, the question is not relevant to you, or you cannot release the information due to its confidential nature. In such cases, please indicate the reason for missing answer using the following acronyms:
NAP question not applicable
NAV information not available
NAC information not available due to concerns of confidentiality
We would like to thank you for participating in this survey. Your contribution is valuable in making the IWSED-95 a successful event.
1. What is the level of confidentiality you require from the data you provide ?
[__] the data and the name of my organization can be released to anyone interested
[__] the data and the name of my organization can be released to the IWSED-95 participants
[__] the data can be released but my organization identity must be kept confidential
[__] the data can be released within the IWSED-95 but my organization identity must be kept confidential
[__] the data can only be used in summaries
[__] other, specify: ____________________________________________________________ __________________________________________________________________________ __________________________________________________________________________
2. Estimate the share of software development costs of your organization's total product development costs and the added value that software creates to your customers (answer at least one):
________ % of product development personnel are involved in software development
________ % of product development costs are associated with software
________ % of the added-value characteristics of the product are produced through software
3. Are your responses in this questionnaire based on your organization's process as it is carried out in several projects or do you refer to a single project ?
[__] process of the whole development organization, i.e., to several projects
[__] single project
4. Total number of personnel in the company: ___________________
5. Share of personnel involved in software development in the company: ___________%
6. Total number of personnel in the software development organization that this questionnaire refers to: _______________
7. Number of personnel in the software development organization that this questionnaire refers to: _______________
8. Amount of subcontracted work per year by the software development organization: ____________ person-years
9. Annual revenue stream or budget total: ________________________ (include currency)
10. How many physical locations belong to this organization: ______________
11. What are the size ranges of typical projects in your organization ?
minimum: ________ person years
normal: ________ person years
maximum: ________ person years
12. What are the duration ranges of typical projects in your organization ?
minimum: ________ years
normal: ________ years
maximum: ________ years
13. Characterize the profile of the development projects in your organization, indicating a share of projects mapped to each category:
research prototypes: ________ %
pre-production prototype versions of new system: ________ %
initial deliveries of new systems: ________ %
new versions of existing systems that incorporate new features: ________ %
maintenance of existing systems without incorporating new features: ________ %
14. Does your organization have an external certificate or assessment of the quality or maturity of its software development process, such as ISO-9001 or CMM:
[__] no such assessments or certifications have been done
[__] only informal or internal assessments have been done
[__] an official assessment or certification has been conducted
15. If your organization has used an assessment model, what did you use (check all that apply):
[__] ISO 9001 certification, comments: ____________________________________
[__] CMM (Capability Maturity Model by the Software Engineering Institute) , comments: ________________________________________________________________ ________________________________________________________________
[__] Bootstrap, comments: ______________________________________________ ________________________________________________________________
[__] SPICE (an ISO standard in development) , comments: ____________________ ________________________________________________________________
[__] SPR (Capers Jones' model), comments: ________________________________ ________________________________________________________________
[__] other, please, specify: ______________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________
16. Based either on the results of previous question or on your own opinion, what is the level of process maturity in your organization (check all that apply)
[__] ISO 9001 certified
[__] assessed maturity level: _________
[__] your opinion of the level of maturity: __________________________________ ________________________________________________________________ ________________________________________________________________
17. Does your organization have explicit process definitions, i.e., are your processes documented ?
[__] no
[__] yes
18. What kind of process definitions does your organization have (select the alternative on each line that best describes the situation) ?
< no / some / all > life cycle models and phases are documented
< no / some / all > development methods are documented
< no / some / all > individual techniques and procedures are documented in detail
19. How many explicitly documented, alternative process models do you have for each type of process definition ?
Instruction: An alternative process definition is one where more than one alternative way of accomplishing the same task has been defined. Examples: different life cycle model ("waterfall" and "spiral") and different design methods ("OMT" and "cleanroom").
life cycle models: _____________
development methods: _____________
individual techniques and procedures: _____________
20. What is the size of the smallest, explicitly documented process step in the process definition of your organization ?
Instructions: (1) Review your process documentation. (2) Determine what process steps are explicitly documented, i.e., by defining their inputs, outputs, entry and exit criteria, and their relationships to other processes etc. (3) Determine the typical size and/or duration of such process steps in your organization's projects and (4) enter the values below. Note that mere lists of activities are not considered to be "explicitly documented".
life cycle models: ____________ person months, ___________ calendar months
development methods: ____________ person months, ___________ calendar months
individual techniques: ____________ person months, ___________ calendar months
21. How strongly are the existing process definitions enforced (check one)?
[__] process definitions are guidelines that can be freely modified
[__] modifications can be made as long as they are explicitly documented
[__] main phases and products must be included as such
[__] projects need to comply to the process definition in detail
[__] projects must always comply to the process definition, unless an exception has been formally approved
22. What project management tools are used at your organization (check all that apply) ?
[__] cost estimation tools
[__] project management software
[__] other, please, specify: ______________________________________________
[__] other, please, specify: ______________________________________________
[__] other, please, specify: ______________________________________________
23. What are the main phases or components in your organization's life cycle?
Instruction: name the main phases or cycles in your life cycle model
24. Does your organization have a testing group that is independent from the project team?
[__] yes
[__] no
25. Does your organization have a quality assurance group that is independent from the project team?
[__] yes
[__] no
26. Does your organization have a software process improvement group (a group dedicated to process improvement, such as a "software engineering process group, SEPG," or an "Experience Factory organization") ?
[__] no
[__] yes, please describe: _________________________________________________ ____________________________________________________________________________________________________________________________________ __________________________________________________________________
27. At what phase in the life cycle do you start collecting data ?
Instruction: Using the phases you listed in question 23, enter the number and name of phase you are referring to.
Cost data: ________________________________
Defect data: ________________________________
28. When do you stop collecting data ?
[__] at delivery
[__] _________ months after delivery
[__] other, please, explain: ______________________________________________ ________________________________________________________________ ________________________________________________________________
29. Do you verify reported data?
[__] no
[__] yes
30. How is the process and product data stored (check all that apply):
[__] in an integrated database designed for this purpose
[__] separate databases by the type of data they contain (e.g., defect data and cost data are in separate, non-integrated databases)
[__] each project collects their own data in non-standard form (e.g., in non-compatible spreadsheets)
[__] data is not stored electronically, e.g., collected only in paper form
31. What is the typical problem domain for the projects (check one)?
[__] batch processing
[__] transaction processing
[__] decision support
[__] production/manufacturing systems
[__] embedded/real-time systems
[__] interactive/reactive systems
[__] other, please, explain in more detail: ___________________________________
_________________________________________________________________
32. What is the typical application domain for the organization (check one)?
[__] telecommunications
[__] research
[__] MIS (management information systems)
[__] marketing
[__] legal
[__] administrative (e.g., workflow or computer supported cooperative work)
[__] other, please, explain in more detail: ___________________________________
_________________________________________________________________
_________________________________________________________________
33. Who is the typical customer of your projects (check one) ?
[__] Unique software development, based on a contract between two organizations.
[__] Development or customization of a product that is modified for each customer.
[__] Software development by the cooperative effort of several organizations.
[__] Development for internal use.
[__] Development of commercial, packaged software.
[__] Development of embedded software sold as a part of a product.
[__] other, please, explain in more detail: __________________________________
________________________________________________________________
________________________________________________________________
34. Rank the most important product and process characteristics according to their importance in determining project success from the overall perspective of your organization.
Instruction: Give a priority ranking, "1" being the most important. If the constraint is not relevant, leave it empty.
[__] product quality
[__] product functionality
[__] process productivity (the cost of development per unit of output)
[__] process cycle time (duration of the project)
[__] process consistency and predictability
[__] profit margin (the difference between costs and the revenue you are able to charge)
35. Rank the most important product quality characteristics according to their importance in determining product quality:
Instruction: Give a priority ranking, "1" being the most important. If the constraint is not relevant, leave it empty.
[__] functionality
[__] reliability
[__] usability
[__] efficiency
[__] maintainability
[__] portability
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
36. Rank the most important product and process constraints that typical projects in your organization need to take into account:
Instruction: Give a priority ranking, "1" being the most important. If the constraint is not relevant, leave it empty.
[__] project schedule
[__] project cost
[__] product functionality
[__] product quality
[__] product performance
[__] product memory constraints
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
In this section we will use the IEEE Standard definition of errors, faults and failures. Errors are defects in the human thought process made while trying to understand given information, to solve problems, or to use methods and tools. Faults are concrete manifestations of errors within the software, i.e., one error may cause several faults and various errors may cause identical faults. Failures are departures of the operational software system behavior from user expected requirements. Thus a particular failure may be caused by several faults and some faults may never cause a failure (difference between reliability and correctness).
37. Who use the defect data (check all that apply)?
[__] Project managers
[__] Project personnel
[__] Quality assurance group
[__] software development organization management
[__] Corporate staff
[__] Consultants or researchers
[__] quality assurance or process improvement group
[__] customer
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
38. What kind of quality data do you collect (check all that apply)?
[__] defect data: errors
[__] defect data: faults
[__] defect data: failures
[__] changes made due to defects
[__] usability data
[__] efficiency data
[__] maintainability data
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
39. What defects do you count?
[__] only severe failures, e.g., system down and output errors
[__] all deviations from the specification or requirements
[__] all causes of change
[__] other problems, please, specify: _______________________________________ _________________________________________________________________ _________________________________________________________________
40. How does your organization use the defect and quality data (check all that apply):
[__] to determine the acceptance or delivery of software
[__] to determine project status or to change the development plan
[__] for planning development or estimating development cost
[__] for planning maintenance or estimating maintenance cost
[__] for investigate or evaluate software tools
[__] for software training in your organization
[__] for predicting product quality
[__] for predicting project duration
[__] for identifying and analyzing areas for process improvement
[__] for reporting project progress to customer
[__] for risk management
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
41. How is defect data analyzed (check all that apply)?
[__] intuitively
[__] by simple statistical techniques, such as calculating average or high and low quartiles
[__] by plotting trends in data graphically
[__] by using some organized, structured analysis approach, such as GQM, QFD or SME, please, specify what: ___________________________________________ _________________________________________________________________
[__] by performing common statistical tests on the data
[__] by carefully selecting the kind of statistical tests that are appropriate in each situation (i.e., the organization has an experienced statistician available for the task)
[__] other, please, specify: _______________________________________________ _________________________________________________________________
42. What cost data does your organization collect (check all that apply)?
[__] effort spent
[__] computing resources used
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
43. How frequently is cost data collected at your organization?
Every _________(number) ________ (unit, e.g., days, weeks, months)
44. How is the cost data used (check all that apply)?
[__] controlling project progress (plan vs. actual)
[__] predicting project cost and completion
[__] calibrating cost models
[__] predicting project risk
[__] other, please, specify: _______________________________________________ _________________________________________________________________ _________________________________________________________________
45 What are the dimensions in which cost data can be consolidated, i.e., by how many different factors can you sum up the cost data (check all that apply)?
[__] by time (example: "project costs during the first quarter")
[__] by process elements (e.g., analysis/design/coding/testing; examples: "cost of design process" or "cost of cycle 1 prototype development")
[__] by type of technical development activity (e.g., design/coding/testing; example: "total cost of coding")
[__] by type of communication/cooperation activity (e.g., individual work/one-to-one communication/meetings)
[__] by type of project contribution (e.g., management/project support/technical work)
[__] by origin/modification type (e.g., rework/new development/reuse)
[__] by the tools used (e.g., CASE/programming tools/testing tools)
[__] by product structure (example: "cost of subsystem X")
[__] by type of artifact produced (example: "cost of design specification document")
[__] by functionality of the product (example: "cost of on-line help")
[__] organization unit (example: "cost of department Z in the project")
[__] by personnel class (e.g., senior programmers/programmers/management/QA; example: "cost of management participation)
[__] other, please, specify: _______________________________________________ _________________________________________________________________
[__] other, please, specify: _______________________________________________ _________________________________________________________________
[__] other, please, specify: _______________________________________________ _________________________________________________________________
46. In your estimate, what percentage of the projects in your organization reach the goals set for them in the beginning of projects ? ___________%
47. What do you think is the corresponding success rate of the projects in general in your industry ? _________ %
48. Rank the product and process characteristics according to frequency by which risks affect these characteristics (1 being the most frequent). For instance, if most of your risks threat project duration, mark "process cycle time" as "1".
[__] product quality
[__] product functionality
[__] process productivity (the cost of development per unit of output)
[__] process cycle time (duration of the project)
[__] process consistency and predictability
[__] profit margin (the difference between costs and the revenue you are able to charge)
49. Has you organization tried to use any risk management techniques or approaches:
[__] yes, we have used some techniques extensively
[__] yes, we have had some small experiments/experiences with some techniques
[__] no
(if you answered "no", go directly to question 57)
50. Are projects required to conduct risk management activities (check all that apply):
[__] there are no formal requirements and projects usually do not manage risks
[__] there are no formal requirements but projects usually carry out some risk management activities
[__] risk management is a required topic in project plans or steering group agendas
[__] risk management activities are reviewed and discussed frequently by project management
[__] there are documented techniques, methods or tools available for risk management
[__] use of some documented techniques, methods or tools for risk management are enforced for projects
51. How frequently should risk management activities be carried out in your organization, according to documented or established guidelines ?
[__] there are no documented requirements for frequency of risk management
[__] in the beginning of the project
[__] at each milestone (phase/cycle): typical duration in between: ______ months
[__] at frequent intervals, typical duration in between: ________ months
52. If you use risk management techniques, how long have you been using them: _____ years
53. If you use risk management techniques, how widely are they used currently:
[__] by all projects in our organization
[__] by some projects in our organization, estimate the frequency: ______%
54. Do you monitor how much time, effort and other resources your risk management activities consume ?
[__] no
[__] yes, please state or estimate the cost of risk management activities as a percentage of total project costs: ______%
55. Do you have any evidence or examples of benefits or savings reached by risk management activities:
[__] no
[__] yes, please, specify or give examples: ________________________________________________________________________________________ ________________________________________________________________________________________________________________________________________________________________________________
56. Do you have any evidence or examples of what is the overall impact of risk management techniques compared between projects that use them projects that do not use them:
[__] no
[__] yes, please, specify or give examples: ___________________________________ __________________________________________________________________________________________________________________________________________________________________________________
57. What is your organization's position towards using explicit risk management approaches in the future:
[__] we do not have any plans to use them
[__] we are monitoring and if suitable techniques emerge, we will start using them
[__] we will evaluate some techniques in the near future
[__] we will implement some techniques in the near future
[__] we are using some techniques and plan to continue using them as such
[__] we are using some techniques and will enhance them in the near future
[__] we are using some techniques but will use other techniques or approaches in the near future
58. If your organization does not use any risk management methods now, what are the reasons for it:
[__] lack of knowledge about risk management techniques and practices
[__] techniques seem too costly to use in practice
[__] techniques do not seem to improve the current, informal practices
59. What techniques are used in identification and monitoring of risks (check all that apply):
[__] risk factor checklists (checklists on potential factors that may increase risks)
[__] risk item checklists (checklists on potential risk items, or threats)
[__] risk management checklists (checklists on whether some risk management activities have been carried out)
[__] decision driver analysis
[__] assumption analysis
[__] project decomposition and task criticality analysis
[__] brainstorming sessions to identify risks
[__] do you analyze the causal effects of risk factors, events, or consequences
[__] do you separately analyze risk-prone areas of projects
[__] do utilize product data (e.g. testing results, configuration statistics) to identify potential risks
[__] some tools, what: __________________________________________________
[__] other, please, specify: _______________________________________________
[__] other, please, specify: _______________________________________________
60. What techniques are used in analyzing and prioritizing risks (check all that apply):
[__] calculating and using risk exposure (expected value of risk, i.e., probability * potential loss)
[__] quality factor analysis (analyzing the potential impact on quality characteristics)
[__] performance analysis (e.g., estimating/simulating product performance)
[__] decision trees
[__] calculating and using risk reduction leverage (i.e., the change in risk divided by the cost of risk reducing action )
[__] performing sensitivity analyses with estimation tools (e.g., varying cost model inputs to speculate about risks)
[__] project activity network analysis / critical path analysis
[__] other _____________________________
[__] other _____________________________
[__] other _____________________________
61. Risk resolution: which of the following approaches are commonly known and used as means to reduce risks, once they have been recognized (check all that apply):
[__] revise the goals for the project (e.g., postpone deadline, reduce product functionality)
[__] relax some constraints (e.g., get more resources)
[__] assign resources to take action now to limit the possible negative impact should the risk occur
[__] assign resources to find out more information
[__] take action to reduce the probability of risk
[__] get management or customer approval for the risk
[__] other _____________________________
[__] other _____________________________
[__] other _____________________________
[__] other _____________________________
[__] other _____________________________
62. How do you use project performance data to manage risks:
_____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________
63. How do you use product quality data to manage risks?
_____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________
64. What other data do you use for risk management ?
_____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________ _____________________________________________________________________________