IWSED-95: International Workshop on Software Engineering Data

IWSED-95 Questionnaire Data

By Jyrki Kontio

This report includes a full summary level description of IWSED-95 questionnaire data. The questions from the original questionnaire have been included in this summary but the instructions are not included.

We have indicated responses here in absolute numbers in brackets, unless stated otherwise. Percentage signs (%) indicate frequencies.

The questionnaire has been divided into the following sections

• Organization background
• Software Development Organization and Infrastructure
• Data Collection Procedures
• Software Product and Customer
• Defect Data
• Cost Data
• Risk Management

Level of Confidentiality

1. What is the level of confidentiality you require from the data you provide ?

Question	count	%
the data and the name of my organization can be released to anyone interested	3	14%
the data and the name of my organization can be released to the IWSED-95 participants	6	29%
the data can be released but my organization identity must be kept confidential	5	24%
the data can be released within the IWSED-95 but my organization identity must be kept confidential	4	19%
the data can only be used in summaries	3	14%
Total	21

• One organization indicated that "the data and the name of my organization can be released to anyone interested" on the condition that an explicit note is included that data is from one project.

Organization Background

2. Estimate the share of software development costs of your organization's total product development costs and the added value that software creates to your customers (answer at least one):

Question	<33%	33%<X<67	>=67%
% of product development personnel are involved in software development	5	9	5
% of product development costs are associated with software	2	4	4
% of the added-value characteristics of the product are produced through software	0	1	7

3. Are your responses in this questionnaire based on your organization's process as it is carried out in several projects or do you refer to a single project ?

Question	count	%
process of the whole development organization, i.e., to several projects	16	76%
single project	5	24%

Software Development Organization and Infrastructure

4. Total number of personnel in the company:

Number of employees (X)	count	%
X >= 20,000	8	40%
5,000 <= X < 20,000	4	20%
500 <= X < 5,000	5	25%
X < 500	3	15%
Total	20

5. Share of personnel involved in software development in the company:

Share of SW personnel (X)	count	%
X >= 75%	3	17%
50% <= X < 75%	3	17%
25% <= X < 50%	4	22%
X < 25%	8	44%
Total	18

6. Total number of personnel in the software development organization that this questionnaire refers to:

Total number of personnel in SW organization	count	%
X >= 1,000	3	15%
500 <= X < 1000	2	10%
100 <= X < 500	4	20%
X < 100	10	50%
Total	19

7. Number of personnel in the software development organization that this questionnaire refers to

• Note: this question should have been phrased as "Number of software personnel in the software development organization that this questionnaire refers to" to separate it from the previous question. I..e., the purpose was to find out how many people are involved in the development of software, as opposed to support personnel in the same organization. Because of this mistake the questions 6 and 7 have limited value for analysis.

Number of personnel in SW organization	count	%
X >= 1,000	2	10%
500 <= X < 1000	0	0%
100 <= X < 500	5	25%
X < 100	10	50%
Total	17

8. Amount of subcontracted work per year by the software development organization:

Amount of subcontracted person-years (question 8) / number of SW personnel (question 7) (=X)	count	%
X >= 100%	5	24%
50% <= X < 100%	2	10%
10% <= X < 50%	3	14%
X < 10%	6	29%
Total	16

9. Annual revenue stream or budget total:

• Note: Less than half of respondents replied to this question and the answers are not described here.

10. How many physical locations belong to this organization:

minimum (7 respondents)	1
maximum	76
median	2
average	7

11. What are the size ranges of typical projects in your organization ?
(minimum/normal/maximum in person years)

• Note: Rows correspond to the question in the questionnaire, columns in each row characterize the answers given through minimum, median, average and maximum values given for the question.

	Minimum	Median	Average	Maximum
Minimum	0.1	1	13	100
Normal	1.5	10	32	200
Maximum	5	30	110	600

12. What are the duration ranges of typical projects in your organization?
(minimum/normal/maximum in calendar years)

	Minimum	Median	Average	Maximum
Minimum	0.05	0.3	0.4	1
Normal	0.5	1	1.2	2
Maximum	1.5	2.75	2.6	4

13. Characterize the profile of the development projects in your organization, indicating a share of projects mapped to each category:

	Average share
Research prototypes	5%
Pre-production prototype versions of new system	10%
Initial deliveries of new systems	38%
New versions of existing systems that incorporate new features	44%
Maintenance of existing systems without incorporating new features	25%

14. Does your organization have an external certificate or assessment of the quality or maturity of its software development process, such as ISO-9001 or CMM:

	Count	Percent
no such assessments or certifications have been done	4	21%
only informal or internal assessments have been done	3	16%
an official assessment or certification has been conducted	12	63%
Total	19

15. If your organization has used an assessment model, what did you use (check all that apply):

• Note: the percentage rating here refers to organizations who claimed to have used some assessment model. Some organizations who answered to question 14 as "no assessment done" still used some assessment models for guiding improvements Only one organization had not used any assessment model, i.e., percentage share is based on the sample of 20 respondents. As an organization may have used more than one model, the percentage column adds up to more than 100%.
• Note that ISO certification usually meant general ISO audit, not necessarily a specific audit of the software development.

	Count	Percent
ISO 9001 certification	11	55%
CMM	8	40%
SPICE	1	5%
SPR	1	5%
Other	2	10%

16. Based either on the results of previous question or on your own opinion, what is the level of process maturity in your organization

	Assessed/ audited	Opinion	Total
ISO 9001 certification	9	-	9
CMM level 1	-	4	4
CMM level 2	2	5	7
CMM level 3	3	2	5

17. Does your organization have explicit process definitions, i.e., are your processes documented ?

yes	18
no	3

18. What kind of process definitions does your organization have (select the alternative on each line that best describes the situation) ?

	All	Some	none
life cycle models and phases are documented	14	4	3
development methods are documented	10	9	2
individual techniques and procedures are documented in detail	3	13	5

19. How many explicitly documented, alternative process models do you have for each type of process definition ?

	1	2	>=3
life cycle models	8	5	1
development methods	4	4	5
individual techniques and procedures	2	1	6

20. What is the size of the smallest, explicitly documented process step in the process definition of your organization ?

Effort (person months)	Minimum*	Median	Average	Maximum
Life cycle models	0.01	2	8	50
Development methods	0.01	0.75	0.9	2
Individual techniques	0.03	0.25	0.4	1

* The small minimum numbers refer to a case where process automation tools were used.

Duration (months)	Minimum*	Median	Average	Maximum
Life cycle models	0.01	2	1.8	4
Development methods	0.01	0.5	0.5	1
Individual techniques	0.03	0.2	0.3	1

21. How strongly are the existing process definitions enforced (check one)?

	Count	Percent
Process definitions are guidelines that can be freely modified	4	20%
Modifications can be made as long as they are explicitly documented	4	20%
Main phases and products must be included as such	7	35%
Projects need to comply to the process definition in detail	1	5%
Projects must always comply to the process definition, unless an exception has been formally approved	3	15%
Total	20

22. What project management tools are used at your organization (check all that apply) ?

	Count	Percent*
Project management software	14	67%
Cost estimation tools	7	33%
Spreadsheets or reports	3	14%
Defect estimation tools	3	14%
Other	3	14%

* Percentage figure is based on the whole sample of 21 respondents.

23. What are the main phases or components in your organization's life cycle?

• Note: Responses not included in this summary

24. Does your organization have a testing group that is independent from the project team?

yes	9
no	12

25. Does your organization have a quality assurance group that is independent from the project team?

yes	14
no	7

26. Does your organization have a software process improvement group (a group dedicated to process improvement, such as a "software engineering process group, SEPG," or an "Experience Factory organization") ?

yes	12
no	9

Data Collection Procedures

27. At what phase in the life cycle do you start collecting data ?

• Note: Responses not included in this summary

28. When do you stop collecting data ?

	Count	Percent
At delivery	4	20%
within 6 months of delivery	6	30%
within 6-24 months of delivery	4	20%
Continued as long as system is in use	7	33%
	21

29. Do you verify reported data?

yes	14
no	7

30. How is the process and product data stored (check all that apply):

	Count	Percent*
In an integrated database designed for this purpose	7	33%
Separate databases by the type of data they contain	10	48%
Each project collects their own data in non-standard form	8	38%
Data is not stored electronically	8	38%

* Percentage figure is based on the whole sample of 21 respondents.

Software Product and Customer

31. What is the typical problem domain for the projects (check one)?

	Count	Percent*
Embedded/real-time systems	8	38%
Transaction processing	6	29%
Interactive/reactive systems	3	14%
other (CASE, MIS)	2	10%
Decision support	1	5%
Production/manufacturing systems	1	5%
Batch processing	0	0%

* Percentage figure is based on the whole sample of 21 respondents.

32. What is the typical application domain for the organization (check one)?

	Count	Percent*
Telecommunications	5	24%
Research	0	0%
MIS (management information systems)	3	14%
Marketing	0	0%
Legal	2	10%
Administrative	1	5%
Other	10	48%

* Percentage figure is based on the whole sample of 21 respondents.

33. Who is the typical customer of your projects (check one) ?

	Count	Percent*
Unique software development, based on a contract between two organizations	9	45%
Development of embedded software sold as a part of a product	4	20%
Development or customization of a product that is modified for each customer	3	15%
Development for internal use	2	10%
Development of commercial, packaged software	1	5%
Other	1	5%
Software development by the cooperative effort of several organizations	0	0%

* Percentage figure is based on the whole sample of 20 respondents.

34. Rank the most important product and process characteristics according to their importance in determining project success from the overall perspective of your organization.

• Note: in the following table we have used two methods to summarize importance of process and product characteristics: average and median. Characteristics are presented in decreasing order of importance.

	Median	Average
product quality	2	1.95
product functionality	2	2.47
process cycle time	3	2.47
process productivity	4	3.94
profit margin	4	4.24
process consistency and predictability	5	4.69

35. Rank the most important product quality characteristics according to their importance in determining product quality:

	Median	Average
reliability	1	1.30
functionality	2	2.00
usability	3	3.35
efficiency	3	3.39
maintainability	5	4.12
portability	6	5.71

36. Rank the most important product and process constraints that typical projects in your organization need to take into account:

	Median	Average
project schedule	2	1.90
product quality	2	2.30
product functionality	3	2.74
product performance	3	3.53
project cost	5	4.12
product memory constraints	5	4.47

Defect Data

37. Who use the defect data (check all that apply)?

	Count	Percent*
Project managers	20	95%
Project personnel	16	76%
Quality assurance group	14	67%
Software development organization management	14	67%
Quality assurance or process improvement group	12	57%
Customer	8	38%
Consultants or researchers	6	29%
Corporate staff	5	24%

* Percentage figure is based on the whole sample of 21 respondents.

38. What kind of quality data do you collect (check all that apply)?

	Count	Percent*
Defect data: errors	13	62%
Defect data: faults	19	90%
Defect data: failures	18	86%
Changes made due to defects	15	71%
Usability data	1	5%
Efficiency data	7	33%
Maintainability data	5	24%
Other (e.g., outages, customer satisfaction, requirements changes, trends)	3	14%

* Percentage figure is based on the whole sample of 21 respondents.

39. What defects do you count?

	Count	Percent*
only severe failures, e.g., system down and output errors	2	10%
all deviations from the specification or requirements	13	62%
all causes of change	10	48%
other problems	2	10%

* Percentage figure is based on the whole sample of 21 respondents.

40. How does your organization use the defect and quality data (check all that apply):

	Count	Percent*
to determine the acceptance or delivery of software	18	86%
to determine project status or to change the development plan	15	71%
for identifying and analyzing areas for process improvement	14	67%
for predicting product quality	12	57%
for reporting project progress to customer	11	52%
for predicting project duration	8	38%
for planning development or estimating development cost	5	24%
for planning maintenance or estimating maintenance cost	5	24%
for software training in your organization	5	24%
for risk management	5	24%
for investigate or evaluate software tools	1	5%

* Percentage figure is based on the whole sample of 21 respondents.

41. How is defect data analyzed (check all that apply)?

	Count	Percent*
by simple statistical techniques, such as calculating average or high and low quartiles	12	57%
by plotting trends in data graphically	12	57%
intuitively	11	52%
by using some organized, structured analysis approach, (GQM = 3, fault prediction models = 1)	6	29%
by performing common statistical tests on the data	4	19%
by carefully selecting the kind of statistical tests that are appropriate in each situation (i.e., the organization has an experienced statistician available for the task)	4	19%

* Percentage figure is based on the whole sample of 21 respondents.

Cost Data

42. What cost data does your organization collect (check all that apply)?

	Count	Percent*
effort spent	21	100%
computing resources used	6	29%
other (e.g., hardware purchase)	1	5%

* Percentage figure is based on the whole sample of 21 respondents.

43. How frequently is cost data collected at your organization?

	Count	Percent*
Once an hour	1	5%
Once a day	3	16%
Once a week	6	32%
Once in two weeks	1	5%
Once a month	7	37%
Once a year	1	5%

* Percentage figure is based on the whole sample of 19 respondents.

44. How is the cost data used (check all that apply)?

	Count	Percent*
controlling project progress (plan vs. actual)	16	76%
predicting project cost and completion	13	62%
calibrating cost models	7	33%
predicting project risk	4	19%
other (e.g., prediction, characterization, evaluation of work)	5	24%

* Percentage figure is based on the whole sample of 21 respondents.

45 What are the dimensions in which cost data can be consolidated, i.e., by how many different factors can you sum up the cost data (check all that apply)?

	Count	Percent*
by time (example: "project costs during the first quarter")	18	86%
by process elements (e.g., analysis/design/coding/testing; examples: "cost of design process" or "cost of cycle 1 prototype development")	18	86%
organization unit (example: "cost of department Z in the project")	15	71%
by type of technical development activity (e.g., design/coding/testing; example: "total cost of coding")	13	62%
by product structure (example: "cost of subsystem X")	11	52%
by type of project contribution (e.g., management/project support/technical work)	8	38%
by personnel class (e.g., senior programmers/programmers/management/QA; example: "cost of management participation)	8	38%
by origin/modification type (e.g., rework/new development/reuse)	7	33%
by functionality of the product (example: "cost of on-line help")	4	19%
by type of communication/cooperation activity (e.g., individual work/one-to-one communication/meetings)	3	14%
by type of artifact produced (example: "cost of design specification document")	2	10%
by the tools used (e.g., CASE/programming tools/testing tools)	1	5%
other	1	5%

* Percentage figure is based on the whole sample of 21 respondents.

Risk Management

Project success in risk management

46. In your estimate, what percentage of the projects in your organization reach the goals set for them in the beginning of projects ?

47. What do you think is the corresponding success rate of the projects in general in your industry ?

• Questions 46 and 47 tried to estimate how well the respondents perceive their organization to perform, compared to industry averages.

Ratio between own estimate and industry estimate (question 46 / question 47)	Count	Percent*
Better than industry average	6	55%
About the same as industry average	2	18%
Worse than industry average	3	27%

* Percentage figure is based on a sample of 11 respondents.

48. Rank the product and process characteristics according to frequency by which risks affect these characteristics (1 being the most frequent). For instance, if most of your risks threat project duration, mark "process cycle time" as "1".

	Average	Median
process cycle time	1.82	1
product quality	2.47	2
product functionality	3.29	3
process productivity	3.87	4
profit margin	3.92	3.5
process consistency and predictability	4.17	4.5

Risk management infrastructure

49. Has you organization tried to use any risk management techniques or approaches:

	Count	Percent
yes, we have used some techniques extensively	4	21%
yes, we have had some small experiments/experiences with some techniques	8	42%
no	7	37%
Total	19

50. Are projects required to conduct risk management activities (check all that apply):

	Count	Percent*
There are no formal requirements but projects usually carry out some risk management activities	2	17%
Risk management is a required topic in project plans or steering group agendas	7	58%
Risk management activities are reviewed and discussed frequently by project management	6	50%
There are documented techniques, methods or tools available for risk management	5	42%
Use of some documented techniques, methods or tools for risk management are enforced for projects	3	25%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

51. How frequently should risk management activities be carried out in your organization, according to documented or established guidelines ?

	Count	Percent*
There are no documented requirements for frequency of risk management	2	17%
Only in the beginning of the project	1	8%
At each milestone (interval between 1-3 months)	5	42%
At frequent intervals (interval between 1-3 months)	4	33%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

52. If you use risk management techniques, how long have you been using them?

	Count	Percent
Year or less	2	29%
Over a year	5	71%
Total	7

53. If you use risk management techniques, how widely are they used currently:

Frequency = X	Count	Percent
X=100%	3	33%
50% <= X < 100%	3	33%
0% <X < 50%	3	33%
Total	9

54. Do you monitor how much time, effort and other resources your risk management activities consume ?

	Count	Percent*
no	10	83%
yes (range between 2% - 5%)	2	17%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

55. Do you have any evidence or examples of benefits or savings reached by risk management activities:

	Count	Percent*
no	10	83%
yes (examples: better predictability, better quality)	2	17%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

56. Do you have any evidence or examples of what is the overall impact of risk management techniques compared between projects that use them projects that do not use them:

All respondents replied "no".

57. What is your organization's position towards using explicit risk management approaches in the future:

	Count	Percent
we do not have any plans to use them	5	28%
we are monitoring and if suitable techniques emerge, we will start using them	2	11%
we will evaluate some techniques in the near future	2	11%
we will implement some techniques in the near future	0	0%
we are using some techniques and plan to continue using them as such	1	6%
we are using some techniques and will enhance them in the near future	6	33%
we are using some techniques but will use other techniques or approaches in the near future	2	11%
Total	18

58. If your organization does not use any risk management methods now, what are the reasons for it:

	Count	Percent*
lack of knowledge about risk management techniques and practices	6	75%
techniques seem too costly to use in practice	1	13%
techniques do not seem to improve the current, informal practices	1	13%
	8

Risk management techniques

59. What techniques are used in identification and monitoring of risks (check all that apply):

	Count	Percent*
utilize product data (e.g. testing results, configuration statistics) to identify potential risks	10	83%
brainstorming sessions to identify risks	9	75%
risk item checklists (checklists on potential risk items, or threats)	7	58%
risk management checklists (checklists on whether some risk management activities have been carried out)	6	50%
risk factor checklists (checklists on potential factors that may increase risks)	5	42%
project decomposition and task criticality analysis	4	33%
some tools	4	33%
analyze the causal effects of risk factors, events, or consequences	3	25%
separately analyze risk-prone areas of projects	2	17%
assumption analysis	1	8%
decision driver analysis	0	0%
other	1	8%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

60. What techniques are used in analyzing and prioritizing risks (check all that apply):

	Count	Percent*
project activity network analysis / critical path analysis	7	58%
performance analysis (e.g., estimating/simulating product performance)	5	42%
calculating and using risk exposure (expected value of risk, i.e., probability * potential loss)	3	25%
quality factor analysis (analyzing the potential impact on quality characteristics)	3	25%
performing sensitivity analyses with estimation tools (e.g., varying cost model inputs to speculate about risks)	2	17%
other (e.g., contingency planning, case analysis, FPA)	4	33%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

61. Risk resolution: which of the following approaches are commonly known and used as means to reduce risks, once they have been recognized (check all that apply):

	Count	Percent*
revise the goals for the project (e.g., postpone deadline, reduce product functionality)	11	92%
relax some constraints (e.g., get more resources)	11	92%
get management or customer approval for the risk	10	83%
take action to reduce the probability of risk	9	75%
assign resources to take action now to limit the possible negative impact should the risk occur	8	67%
assign resources to find out more information	5	42%

* Percentage figure is based on the whole sample of 12 respondents who indicated the use of risk management techniques (question 49).

Use of risk data

62. How do you use project performance data to manage risks:

Following is a list of answers given:

• "Effort data in ongoing projects used for future projects"
• "Performance data is used to see whether we are meeting traffic and load conditions needed to satisfy our customers. Schedule and content are monitored through project meetings; recovery plans are put in place as needed."
• "We grasp the progress of each works in every phases by collecting data, such as rate of programs coded, rate of test scripts checked."
• "Using these data, we find fundamental problem of delay of each works and do something to solve or prevent problems. (For example, reinforce of working structure or resources, reduction of development) To decide whether the development is feasible or not."
• "When enough data are collected, they will be used for resource allocation."
• "Past records are referenced to judge subcontractors' ability. Project Manager are aware of the achievement of engineers in the past projects. Based on the observation, they decide assignments to the subcontractors, which part in a projects and how much work."
• "Data are referenced to assign personnel and to postpone schedule. But not be used for calculation, or for input a particular estimation method, because project performance data are too roughly measured to use estimate a detail schedule of work."
• "Reallocation of resources, Eliminate risk factor"
• "Reallocate resources based on Actual vs. Plan"
• "Detect occurrence of problem."

63. How do you use product quality data to manage risks?

Following is a list of answers given:

• "We use defect containment data in process defect data to see how our project is doing in regard to quality goals. This data is also used for release criteria; Reliability is part of this. This data is also used to guide future development."
• "(1) Judgment material if shipment is admittable or not, (2) Judgment material if acceptance of software from vender, (3) Analyze defect data by categorization into error or faults, and improve quality of products and process by removing the origin of error."
• "By this activity we avoid risks after shipment or similar development in the future."
• "To decide how long will it take until delivery."
• "They will be used for judging if the product satisfies the customer's acceptance criteria"
• "Quality data is used to consider how much work each subcontractor can be in charge of, since excess assignment lower quality of the work."
• "We use them to set up the next project's quality objectives. Also we use them to identify the low quality process and to take actions, i.e., increase review effort."
• "Identify sub-systems and functions, or programs and modules which are poor status in quality data. Recommend focusing reviews or testing,. Assign personnel and to postpone schedule."
• "Early identification of risk areas"
• "If defect data is higher than expected, assign tiger team to analyze, If Field-Detected-Defects might threaten patient safety, function is disabled."
• "Only monitor whether it exceed estimated/planned one."

64. What other data do you use for risk management ?

Following is a list of answers given:

• "Commitment alterations are tracked at high levels. Issue lists are created and used at project reviews."
• "(1) We correct technical trouble information related to software package and hardware of other vender mainly used in Client Server System, workstation/personal computer. We publish this information as "CSS-QNS". All the employee in our division can prepare to avoid troubles in case they use the same product as recorded in the article.
  (2) To secure efficiency of software is one of the important subject, so many case it affect on project's cost and duration. In ordinary case we collect efficiency data such as dynamic step of code, CPU utilization rate and resource utilization rate, response time and so on. We estimate these data in design phase of development and verify the generally in system test phases. In case efficiency of system (software) is not enough, we tune up this by some methods such as adding up resource, reduction of step count by re-design structure of module, grade up of devices."
• "Number of engineers. Skill level of engineers. Information about a subcontractor's subcontractor. Generally, the personal principle is predominant rule. Careful assignment is essential to avoid confusion based on the information who have what kind of skill and ability."
• "Subcontractors (initial assessment, experiences) Tools (experience database)"
• "Code production profile"
• "Engineering analyses + assessments, Product evaluations"
• "We are now applying FPA (Function Point Analysis) so as to monitor whether product size is differed from initially estimated/planned one. Since surveyed organization develops only in-house use product, requirement/specification may be changed frequently without correcting budget/duration. From our observation, almost risk (delivery delay, low quality etc.) are caused by mistake of product sizing (i.e., productivity is very stable). "

Go to IWSED-95 home page

Updated 06-Mar-1996 by Jyrki Kontio

Web Accessibility