Reported by Jyrki Kontio, H. Dieter Rombach, Helena Englund, and Alfred Bröckers
H. Dieter Rombach, University of Kaiserslautern, co-chair
Jyrki Kontio, University of Maryland, co-chair
The risk management and project management working groups started their work in a joint session, for which we have written a separate report (joint session in project and risk management). The joint session concluded in initial definitions for project management and risk management.
Project management was defined as an organizational function that plans and executes projects. Project management (as an organizational function) contains several activities, such as planning, monitoring, controlling, resource allocation, estimating, and task assignment. These different activities may employ different technologies, such as cost and schedule estimation, use of PERT charts, metrics databases - and risk management.
While we defined project management as an organizational function, we defined risk management as a technology, i.e., a set of methods, techniques and tools that can be used for managing risks. Risk was defined as "any action that occurs and has a major impact on performance, cost and/or schedule". The initial, and somewhat confusing, overlap of the terms projects management and risk management was thus easier to resolve.
The initial agenda for the working group had been defined by a set of issues that had been proposed before the workshop:
Based on the presentations given on the first day, Dieter Rombach presented a revised list of issues for discussion and the group discussed and prioritized this list. The following is a list of the issues selected:
Some other issues were also raised in Rombach's list but due to lack of time they were not addressed explicitly during the discussion:
The state-of-the-art review was initiated by Jyrki Kontio's briefing on his understanding of the risk management practices in various organizations. His briefing was based on a risk management practices survey [Ropponen 1993], IWSED-95 questionnaire data, and some case studies and observations made in several organizations. Surprisingly, the software engineering literature contains few reports on practical, empirical experiences on the use of risk management. In addition to the survey mentioned earlier [Ropponen 1993], the SEI conference series has reported some experiences [SEI92-93], and the practices used in the U.S. defense sector have been reported in detail [Boehm 1989, Boehm 1991].
It seems that although the U.S. defense sector has applied carefully documented risk management approaches for a couple of decades, these practices have not spread widely in the commercial sector. Ropponen's survey found that only about 20% of the surveyed project managers used risk management methods [Ropponen 1993]. The IWSED-95 data indicated that risk management is not commonly exercised: only 20% said they use risk management techniques "extensively", 35% had had some experiments in risk management and 30% reported that they did not apply any methods for risk management.
It is important to point out that the low risk management method use figures above do not mean that organizations do not manage risks. Coming back to the definitions we introduced earlier, risk management is always a part of the project management organizational function. What the numbers indicate few organizations have recognized risk management as an area that would require the use of specific approaches (the risk management technology). In practice, many organizations apply some risk management methods informally. However, as the risk management methods do not seem to receive much attention, these informal methods are likely to be rather simple.
It is interesting to note that according to IWSED-95 data, all organizations that did not plan to implement any risk management methods had had no experience in using any risk management method or tool. On the other hand, almost all who had experimented or used risk management methods planned to continue using them or planned to introduce some new or alternative methods. This suggests that exposure to risk management technology increases its usage, i.e., organizations seem to consider risk management as a "smart investment" once they know what it can do.
Kontio also categorized risk management practices in organizations:
The standard situation in the industry seems to be that "nothing systematically applied". Responsibility for risk management is left to each individual.
Recently many organizations have taken some initial steps in requiring some kind of risk analysis. This may have been fueled by software quality related standards or improvement models (such as ISO 9001 and CMM). It seems that even when risk management is formally required, it is not necessarily strongly enforced in practice. Also, the fact that risk management is, e.g., a required topic in project management plan does not guarantee that it is done well.
Some organizations have gone further and specified how risk management should be done. They may provide some guidance for identification of risks and they may have some guidelines on how to document and risks and estimate their seriousness. Checklists are commonly used for risk identification, there are several tools, both commercial and internally developed, in use that allow quantification and calculation of risk.
Mitsuhiro Takahashi pointed out that in Japan risk management is often carried out informally through a series of meetings. Risks are identified, discussed and resolved as an integral part of project management. This indicates that risk management is viewed from the perspective of organizational functions. The meeting-driven working style may, in fact, be quite effective in recognizing potential risks: organizing brainstorming meetings is often recommended as a technique for risk identification.
It was also pointed out during the discussion that few organizations collect data about risks (Basili & Marciniak). Without historical data it is very difficult to build any kind of estimation models. Furthermore, estimation of probabilities is especially difficult even when historical data is available: "you can't drive a car by looking at the rearview mirror when you are going to new places" (Kontio). As some risks are not associated with past data, historical data cannot help in predicting them.
During the early part of the workshop we had pointed out that most of the risk management literature has concentrated on quantification of risk, sometimes without the necessary empirical basis. While the quantification of risks is an ultimate goal, any attempt in risk quantification should be carefully made so that it does not result in inaccurate, meaningless numbers. Therefore, both qualitative and quantitative approaches in risk management are needed and one of the challenges in the risk management research community is to conduct balanced and coordinated research in these areas.
In this session Bröckers reviewed some quantitative risk analysis approaches, such as PERT, RISCOMO and macro-simulation. These approaches sometimes produce questionable analysis results, do not support risk identification and do not help in planning of risk mitigation.
In his recent work Bröckers has developed a process model-based risk analysis model. Many risks are affected by a limited number of factors (or "causes") that affect product and process quality. Much of the information about these factors can be observed or recorded during process enactment. His approach is to define organization-specific simulation models that allow the modeling of these factors into the quality and process models of the organization. It is possible that risks can be estimated by stochastic simulation using these models.
The benefit of this approach is that it utilizes empirical data, uses organization-specific process knowledge, and allows aggregation and simulation of this data to estimate the impact on the product and process. So far Bröckers has developed guidelines for building descriptive process models and stochastic quality models and there exists a prototype of the simulator.
A major question that was repeatedly discussed in the working group was the difficulties involved in quantifying the possible benefits of risk management. From the researcher's perspective this question can also be rephrased as the difficulty of validating a risk management method or a tool.
The fundamental problem with validating risk management benefits is that risk management deals with stochastic, low probability events, yet we have very few data points available for analysis. In other words, one cannot conclude that when a low probability risk did not happen it was because some risk prevention action took place.
Helena Englund described the approach that was used at a University of Maryland case study. In that case study it was assumed that one of the main characteristics of any risk management method is the level of confidentiality the decision makers have on it. Instead of trying to accomplish the (impossible) task of comparing how a method can predict true probabilities of risks it is easier to measure decision makers' confidence in a risk management method.
Rombach pointed out that experimentation is one way of obtaining information about possible risks before introducing a new method or technology into real-world projects. Software product engineers deal with high-risk requirements via prototyping; likewise process engineers should deal with high-risk methods and technologies through laboratory experiments.
One under-utilized approach to collect information about risks and risk management actions is the analysis of past projects, i.e., post mortem analysis of projects. This approach might be especially useful for evaluating risk management methods and tools. The benefits of post mortem analysis are that a project does not change its behavior because of the method that is used. If project participants are available and willing to re-enact the project, this may be one way of validating risk management approaches. Of course, the approach also has major disadvantages: participants may be biased by hindsight as they know what happened, their perception of things will have changed, and they may react differently knowing that they are not making real decisions.
Another approach is based on laboratory experiments where two risk management methods are applied by two separate groups in a limited number of sessions. Confidence and feedback on the methods is collected by a comprehensive questionnaire. At the end the groups join and try to synthesize a correct view of the risk situation in the experiment. This synthesized view is taken as the ideal reference point and the method that independently came closer to it is assumed to have been more effective. The information obtained before groups met can be used to provide further insights and to block out other factors.
The working group attempted to relate different risk management
approaches to each other. A two dimensional graph was used for
this purpose. The vertical axis represents different risk management
activities, i.e., identification, analysis and action. On the
horizontal axis we had a classification of different types of
knowledge that may be available for risk management: general knowledge,
implicit in-house knowledge, project specific explicit knowledge,
and generalized knowledge. One could argue that the confidence
in risk management information increases towards the right. Consequently,
an organization should try to have as much of high confidence
information for its risk management decisions as possible.
Figure 1: Risk management context
To do this, general knowledge should be made available in the organization, implicit knowledge should be formalized (both by formalizing experience reports and collecting relevant measurement data), and project specific knowledge should be generalized to the extent possible. The working group identified this as a general trend that organizations should try to follow.
More powerful analysis techniques can be used once risk-related knowledge is can be expressed more formally. While qualitative methods are applicable all across the vertical axis range, quantitative baseline analysis (i.e., comparing current performance against a baseline) and impact analysis (e.g., simulation) can be used for more powerful analysis. However, it is clear that some relevant risk management information will always be in the left of our vertical axis, i.e., some of it will have low confidence levels.
In summary, we concluded that
Updated 06-Mar-96 by Jyrki Kontio