Risk management in software projects became a cardinal issue since most software projects do not stand the customers’ anticipation regarding the quality, overall costs, and timetables. In fact, according to our experience it is almost impossible to reach the expectations with all the three parameters together. In many projects it is the matter of the short blanket which is pulled up or down without being able to cover all the body together. In most instances the area which suffers the most is the software quality because the customer can supervise more easily the timetable and the budget.
A proper risk management can help bringing software projects to a happy ending and with a good quality, help decide as to time and budget constraints, point out those project components which are more sensitive and decide to what detail level the tests should be planned and executed.
Risk management falls within the boundaries of quality assurance and therefore this task is indeed expected to be looked at and managed by the QA manager.
Risk management deals with two groups of risk types:
- Project Risks
- Functional Risks
Project risks
Project risks are in fact project managerial risks, and are not cut out directly from the required system’s functions as the functional risks, but rather from the resources requirements, work plan, time table and other external issues which might endanger the project.
An example for a project risk can relate to the situation where none of the developers is trained enough to use the developing tool in an optimal manner.
Functional risks
Functional risks are based on the system functions as defined in the system’s specifications. These relates to the specifications in the STP document which lay down the foundations for the system breakdown into testing components. Each of the functions is regarded as a quantifiable risk which needs to be analysed. This work is done within the STP framework (See STP guideline).
When dealing with functional risks, what in fact is being measured is the sensitivity of each function or testing component:
By performing functional risk analysis the tester/developer can point out those functions / testing components within the system which are more sensitive to faulty performance and/or liable to cause damage to the system.
There are few common actions derived from this assessment:
- To give more attention to these specific components and their functionality
- To consider changes in the component’s design
- To provide a more detailed level of testing to ensure correct “behaviour”.
Example for a functional risk can relate to a line in the system’s specification regarding the complication of communicating data from one computer to another hardware component, or – the complexity of a specific calculation which constitute the overall function of the system.
Project Risks Assessment
Risk Identification
There are several types of risks under the project management risk groups. The risk types differ between projects and companies as well as the development phase within the same project. Following are risks types which were identified as general risks to software projects. (It should be noted that not all these risks types are applicable to all projects at every development phase):
- Methodological Risks
- Technological Risks
- Project Management Risks
- User Problems Risks
- System Development Risks
- System Implementation Risks
- Security and Recovery Risks
- System Maintenance Risks
The following list of typical risk areas may vary between projects. Some risk areas are more typical to the preliminary stages of a development project, some for the final stages, etc. It is nevertheless very important to allocate the proper risks to our project. In order to achieve this goal, the following risk areas may be used as a basic guideline.
It should be noted that certain projects could come up with additional risk areas and that even in the same project risk areas vary between the time of the project beginning and its end.
After the risk areas were identified, they can be calculated and assessed.
Following are the typical risk areas identified in projects:
Methodological Risks
Risks related to the development methodology. Possible risk regarding this issue, are:
- Implementation of new development standards
- Using development methodology which is un-fit to the organisation
- Over-head in the implementation of the methodology
- No control over the implementation of the methodology
Possible action items regarding these risks, are:
- To set up standard procedure for implementing a new methodology
- To educate users with the methodology
- To perform a periodic status check on the implementation
Technological risks
These risks relate to new or unfamiliar technology which must be used in the project. Possible risks regarding this issue might be:
- Lack of professional support.
- “Bugs” in untested technology.
- Lack of knowledge of this technology in project team.
- Technological complications
- Low quality technological performance
- Integration risks
- 3rd party products
Possible action items regarding these risks, are:
- To ensure that the supplier has the required knowledge for support purposes. Recommend a special support contract, etc.
- Demand copies of product’s internal testing documents.
- Arrange for training courses.
These risks relate to the human resource/s which perform the project management tasks and duties. Possible risks regarding project management, are:
- Lack of experience of the project manager in the project’s field of business.
- Lack of assurance that key personal would not leave the project before delivery time.
- Lack of management experience.
- The site of the teams
- Tight schedule
Possible answers to risks in this area, are:
- To arrange courses or seminars regarding the business.
- To ensure in the contracts between the parties that no key personal can be moved before the end of the project.
- Replacement or support to be given to the inexperienced managers.
- Place the developing team close enough to the management
User / customer problems risks
User or customer problems risks relate to problems generated by the users or by their communication lines to the development team. Possible risks regarding this issue, are:
- The danger of getting continuos requests for changes from the customer.
- Having a customer representative which is not capable of handling the liaison.
- Insufficient co-operation with the customer’s staff.
- Unconcerned user
- User who is too involved
- Internal politics within the organisation
Possible answers to risks in this area, are:
- To reach a clear and detailed summary of required functions and features signed by the customer.
- Ask for the replacement or support to be given to the inexperienced persons.
- Calling a meeting with the customer to highlight the necessity of co-operation.
The development risks relate to the work of the development team during the course of the project. Possible risks regarding this issue, are:
- Lack of the necessary equipment to start the project.
- Insufficient space allocated to the project team.
- The possibility of a big staff turnover rate due to the distance of the development site.
- Inexperienced developing team
- Tight schedule which leads to too many overtime hours / reduce quality
Possible action items for such risks, are:
- Issue of an urgent order for the necessary items with a secondary plan to borrow other items if needed.
- Allocation of another office for part of the team.
- Checking the commitment of each team member for the project.
- System complexity
The implementation risks relate to the operation in which the new system is installed in the customer’s premises, courses are given, and the new system is activated for testing than parallel run and last as a production system.
Possible risks regarding this issue, are:
- Lack of the necessary power supply to the installation site.
- The uncertainty regarding the installation of certain components.
Possible action items for such risks, are:
- Liaison with the local electricity supplier regarding the problem and at the same time, find out the cost of an equivalent power supply generator.
- Verification that the necessary components are to be shipped as soon as possible.
System Maintenance Risks
System maintenance risks relates usually to the support period in which the new system is supported. Possible risk regarding this issue, is:
- Lack of qualified operators.
Possible action items for these risks, is:
- Recruitment /training.
Calculating the risks’ levels
The levels are calculated per risk type, as follows:
Criticality level:
Each risk is allocated with a criticality level which may vary between 1 for the least critical
risks and 5 for the most critical risks. Criticality applies to the effect of the said risk on the
success of the project.
Example: The fact that the developers are not experienced enough to develop the new
system with the selected tool, could be defined as very critical for the success of the project.
In such a case the criticality level given would be 5. This criticality level would be reduced if
the developers would be given a course on that tool ahead of time.
Probability level:
Each risk is allocated with probability level between 0% and 100%. This is to specify simply
was it currently the chance of the risk to come true.
Example: If currently it is speculated that the developers will not be trained, we may allocate
a probability rate of 80% or 90%. If the next day a final decision would be to allow few of
them the necessary training, the probability would be fixed at 40%.
Risk Level:
The weight is simply the multiplication of the two above factors. CRITICALITY X
PROBABILITY.
Example: Criticality of 5 points X Probability of 80% = 5.0 X 0.8 = 4.0
Action items are the most important issue under the risk management title as this is basically the most meaningful output of this effort. Without the action items the results would be filed away as another “shelf” product.
Action items are the results of the risk management process in the format of a “TO DO” list with a description, name of person in charge, etc.:
- Description – a description of the problem
- Responsibility – name of the person in charge
- Next reporting date – expected feedback date
- Report to (optional) – whom is to get the feedback
The risk committee considered the following risk area:
(1) The developers are not trained to use the developing tool.
The committee decided that the first item has a criticality level of “3” as it does endanger
somewhat the project’s success to stand the timetable. The chances for the developers to
come through were estimated as 50:50, therefore the probability was fixed at 50%.
Calculation & Evaluation:
| Item | Description | (a)
Criticality (1 – 5) |
(b) Probability
0%-100% |
(c)
Level a X b |
Action item
number |
Responsible
person |
Deadline |
| (1) | Dev. Training | 3 | 50 | 150 | (1) | Jimmy | 30 June |
Action items:
(1) The development manager is to decide about a training course
Example Summary: Following are the results from the above example in a detailed form of the description.
| Item | Description | Level | Action item | Responsible | Date |
| (1) | Dev. Training
The developers are not trained in using the dev. tool |
150 | Dev. Mgr is to decide about training course | Jimmy | 30 June |
Risk minimisation
In order to minimise risks as far as possible, they must be detected in advance when the problem is still “small”. This can be done if the alert for new problems identified is made immediately and passed to the QC manager in addition to the functional manager in charge of the specific issue.
The earlier a problem is detected the easier it is to handle it.
Risks can be minimised by early alert.
In most cases, risk management contribution to projects is not necessarily by helping to find unknown problems but rather by helping in sharing problems which were traditionally kept in the boundaries of the department or team which detected it but believed they can handle it by themselves. These problems were usually brought out to day light when it was too late.
Ongoing risk management
Risk management does not replace the project management but rather helps project management and management in general to keep an open eye over the project as a “watch-dog”.
Ideally, the QA manager is to open every session of the risk committee and the steering committee with the short list of risks and problems which are on the agenda, the recent updates regarding them and when is the next dead line for each of them.
Functional Risks
There are two different methods in calculating and follow-up procedure of functional Risk Assessment:
- Method A: which defines the Sensitivity Level of each system Function, to be evaluated against the test results.
- Method B: in which the Attention Level is defined, and a specific number of tests will determine the Risk Level.
Functional Risks Assessment – Method A
Risk Identification
There two types of risks under the functional management risk group:
- Þ Function Level Risks
- Þ Overall System Level Risks
The Function level risk differ between projects because they are based on the system requirements, but some of the overall system level risks might be similar, in definition, to many systems.
As explained previously, this process is more a sensitivity assessment rather than risk assessment: the functional topics are measured for their sensitivity to faulty performance and their risk to the system .
The definition of the functions for this assessment risks are derived from the system requirements, and from the understanding of how the system should work:
Functional Level Risks:
Each function defined in every component of the system can be described as a functional
risk, for example:
- Þ Specific calculation
- Þ GUI
- Þ Data transfer
- Þ File update
- Þ Report printout
Overall System Level Risks:
In additional, there are some functional risks on the system level, such as:
- Þ Concurrent activities
- Þ Security
- Þ Failure & Recovery
- Þ Volume
- Þ Stress
When a functional topic is found to be more sensitive, it means that a problem might occur more easily when performing this function and/or that the damage to the system, due to that problem, can be great.
The possible action items for such cases can be:
- Recommendation for a change in the system’s design
- A special attention in the development and testing of that function, to minimise the chance of such problem to occur.
Calculating the sensitivity levels
Following is a general description as to how to build and calculate the sensitivity level for each functional risk:
- The sensitivity level will be defined by two major factors:
- Þ Probability – The chance that a problem might occur while the function is executed
- Þ Severity – The damage this function might cause to the system if a problem will
occur.
- Each of the above factors is actually a combination of various related parameters as fit to the specific system. For example:
- Þ Complexity – Probability
- Þ Usage frequency – Probability
- Þ Number of interfaces – Severity
- Þ I/O performance – Severity
- Þ Significance – Severity
- After defining the parameters which will be part of the risk calculation, each risk parameter will be assigned a weight value which will reflects its part/importance in the overall picture of all the parameters. This value is represented as a percentage, where the weight values of all the defined parameters will total 100%.
For example: Complexity = 0.3
Frequency = 0.2
Interfaces = 0.2
I/Os = 0.2
Significance = 0.1
- Each parameter, for every function, will also be given a value within a pre-defined range (i.e., 1 – 5). This value will represent the parameter for the specific function.
For example: GUI: Complexity = 2
Frequency = 5
Interfaces = 1
I/Os = 2
Significance = 3
- The calculation of the sensitivity level for each function will be as follows:
Probability factor = Sum of all probability parameters (weight * Value)
Severity factor = Sum of all severity parameters (weight * Value)
Sensitivity Level = Probability * Severity
For example: Probability Factor = 2 * 0.3 + 5 * 0.2 = 1.6
Severity Factor = 1 * 0.2 + 2 * 0.2 + 3 * 0.1 = 0.9
Sensitivity Level = 1.6 * 0.9 = 1.44
This calculation is to be performed for each function defined, for the system and the overall project.
Action items should be added to each calculated Risk Level: to be taken in order to prevent or minimise the risk.
Note: If not all the parameters range values are the same (for example: Complexity range value = 1-5, Interfaces range value = 1-20) there must be a factor value applied on each parameter to adjust it.
Setting Up Action List
Action items under this risk group do not vary that much, but they are not less important for that. The action list will provide the user with a way to minimise expected problems ahead of time.
The Action List for Functional Risk Assessment will generally include a list of the most sensitive functions either to be looked at again from the code & design points, or to arrange for more detailed testing for those functions.
Example
Calculation & Evaluation:
| Function | (a)
Complexity W=0.3 |
(b)
Frequency W=0.2 |
(c)
Probability |
(d)
I/Os W=0.4 |
(e)
Significance W=0.1 |
(f)
Severity |
Level
of Sens. |
| (1 – 5) | (1 – 5) | a*W+b*W | (1 – 5) | (1 – 5) | d*W+e*W | c * f | |
| GUI | 2 | 5 | 1.6 | 1 | 3 | 0.7 | 1.12 |
| Calc. | 4 | 4 | 2 | 2 | 5 | 1.3 | 3.3 |
| Interface | 3 | 3 | 1.5 | 4 | 2 | 1.8 | 3.3 |
Action items:
| GUI | None |
| Calc | Detailed level testing |
| Interface | Re-examine design |
Note:
If the risk parameter values are not all ranged between the same values, a factor should be assigned to all, to reconcile the differences.
Risk minimisation
In order to minimise risks as far as possible, they must be detected in advance before it’s too late or too expansive to make changes.
This can be done if the actions are set ahead of time.
Risks can be minimised by early alert.
Ongoing risk management
Risk assessment on the functional issues helps project developers and testers to ensure that the final product will “behave” in the expected manner.
The above risk assessment process can be done periodically, throughout the testing life cycle of a software system, but mostly – is should be compared to the test results in order to validate the correct behaviour of the sensitive functions.
Functional Risks Assessment – Method B
General
This method represents a mechanism used to define the risk that a serious problem in a function or component of the system will not be detected during the testing life cycle.
The method provides the customer and the testers with a tool to help analyse the level of testing required and the expected status of the overall system after the testing is
done.
The mechanism is divided into three stages:
The Target stage
This stage is done when planning the tests (STP Stage). Each component of the system is assigned an “Attention Level”.
The “Attention Level” value represents a way to measure the attention required by each function/component of the system in order to lower the risk of not detecting a problem.
First Check Base
During the tests definition (TRDs) we can calculate the first Risk Level. At that point the potential risk level can be calculated according to the number of tests defined for each function/component.
The calculation assumes that the more tests defined the less the risk of not detecting a problem.
The Risk Level value is calculated for each one of the functions/components by dividing the Attention Level by the number of Touch Points defined.
At this time, an analysis is required:
The goal is to have all risk level values for all functions /components set at the same value, and to aim at a low value.
Note: In order to lower the Risk Level value of a certain function / component, the number of Touch Points should be increased.
Final Check Base
This stage is like the previous one, but it relates only to those Touch Points which had been actually executed.
This stage is especially important in cases where not all the defined tests are actually executed.
As in the previous stage, also here it's important to raise the number of Touch Points executed to lower the Risk Level of a certain function / component.
Also here, the aim is to end up with an equal risk level for all functions/components, optimally as low as possible.
The method used to ‘“Measure” the level of testing required for each function/component of the system, is based on three major factors:
Structural Complexity
Definition: Value which represents the complexity of the topic / component, by a value in the range of 1 – 10.
When estimating the Complexity value, two factors should be considered: the number of activities or passages which are included in the function/component and the number if
interfaces involved in its design.
Damage Level
Definition: The damage this topic/component might cause to the system if a problem will occur, estimated by a value of 1 – 10.
This factor is measured by different parameters, depending on the nature of the tested system.
Touch Points
Definition: The number of tests defined or executed, which are related in some way to the function/component in question.
This value will determine the Risk Level at the test definition and test execution phases.
Cost vs. Benefits
The consideration weather risk management does contribute to the project more than its costs was raised in the past. We can categorically state that a risk management which is handled properly does worth the efforts involved. Risk management must be proportional to the project it is serving. Smaller projects need less investment in time and personal than bigger projects. The bigger projects as well need to calculate carefully their budgets regarding risk management with the consideration that the number of risks and problems must increase as the project goes forward.
The benefits are:
Management regularly receives a summary of all the existing problems in the project with their estimated weights.
- Management can follow up the history of a problem from the beginning and its risk weight trend.
- The chance of a problem to stay hidden with this method are slimmer and managers will be reluctant to keep problems for themselves to solve quietly without sharing them.
- Management gets a “second opinion” about the problems which are on the agenda.
