Keywords

1 Introduction

In many cases, innovation projects have been postponed or canceled due to incorrect risk evaluations, leading to financial and/or opportunity losses which creates negative impacts on organizations and societies. In this sense, it is important to select innovation projects using reliable risk analysis tools and techniques. Risk assessment is becoming critical to organizations especially in the innovation area; however, important risk factors are being systematically omitted because quantitative data is usually unavailable in innovation projects, which makes quantitative risk assessment tools inappropriate to deal with such risk scenarios. The alternative is to use qualitative risk assessment tools which are becoming important in risk related activities. Despite the importance of qualitative risk analysis in modern economic activities, there are very few qualitative risk models available in literature. A literature survey shows that further developments are needed to extend qualitative risk models to incorporate risk assessment and management in projects. In this approach developments are needed in several areas, such as risk integration in project development and implementation, risk modulation of needs and wants, risk of underperformance in project phases, risk of combined project phases, and risk in quality control. The risk evaluation in these areas is of utmost importance, especially in the selection of innovation projects which is a decision-making activity that has several challenges in practice. The objective is to choose the best alternative among possible alternatives to obtain maximum outcomes with lower resources allocation. This can be a hard task when available information is scarce and imprecise, therefore, uncertainty is a common factor in decision making that strongly challenges decision makers. In this sense, qualitative models to evaluate aggregated risk in innovative projects are important tools that need to be developed and improved for innovation activities. In this paper, a model to evaluate qualitative risk of projects is proposed. The model evaluates qualitative aggregated risk considering a generic framework easily applicable to a wide range of projects. It will allow to prioritize innovation projects according to their risk feature that supports decision making activities.

2 Contribution to Life Improvement

Risk evaluation is an important procedure that supports decision making activities. It allows to set risk scenarios with risk variables that model scenario behaviors (outputs) accordingly to resources and inputs. Consequently, decisions can be monitored regarding their influence on a given risk scenario, which can be useful to optimize results. For instance, in mechanical design, risk assessment can be used to improve reliability of new products and services during their design and development phases. In this case, risk can be evaluated based on a set of decisions steps that range from user’s requirements identification to acceptance testing, each one of these decisions can increase or decrease the risk of obtaining unwanted results in design. This subject becomes extremely important in cases where human lives can be jeopardizing due to a poor design. There are many cases where human lives were lost due unreliable decisions in design activities. One example is the space shuttle Challenger disaster in 1986 where 7 crew members were killed due to an o-ring seal that was not designed to handle the low temperatures experienced in this launch. Later, in 2003 the space shuttle Columbia disintegrated upon reentering Earth’s atmosphere killing 7 people. The risk-management scenario failed to recognize the implications of a debris impact on the Columbia left wing leading to the catastrophic accident. Other example is the Chinese bullet train crash in 2011 that killed 40 people due to a lightning strike. The accident report pointed out design flaws in control equipment that lead to an improper handling of the lightning strike. In literature one can find many other examples like the ones mentioned above that lead to conclude that risk management in association with design activities is of utmost importance to avoid catastrophic accidents with loss of human lives. The most used tools in risk assessment and management are strongly based on quantitative data, however, in many cases, like the aforementioned ones, information required to use these tools is scarce or none, i.e. the typical tools in risk assessment are data based which are useless in cases where information is scarce, because they are statistically based which in turn requires access to events log which is a condition difficult to fulfill in innovation projects. The alternative to overcome this issue is to use qualitative risk assessment tools making use of expert’s experience to evaluate risk scenarios and respective failure modes. Therefore, qualitative risk assessment of innovative projects actively contributes to the life improvement of societies by promoting reliability in innovation.

3 State of the Art

The most common way to rank projects according to their risk is using Multi-Criteria Decision Making (MCDM) methods which in some cases are complex to implement due to their formulation or due to the limitations to get reliable quantitative data [1]. Very few of them can deal with uncertainty and human judgment; examples of these models are the Multi-Attribute Utility Theory (MAUT), Fuzzy Set Theory, ELECTRE, PROMETHEE, and DEMATEL [2]. Multi-Attribute Utility Theory [3] is an improvement of the Multi-Attribute Value Theory and considers risk and uncertainty to support multi criteria decision making of real-world problems. These methods need a large amount of data which can be a shortcoming for applications with scarce information available. Moreover, it requires an accurate identification of the decision makers’ preferences which can be difficult to get since the decision makers weights may change during the project design and implementation. The strong feature of these models is the capability to account with uncertainty which is the reason of their popularity. Fuzzy Set Theory [4] also allows to consider imprecise and insufficient information, it is widely applied in a wide range of applications including MCDM. Fuzzy logic models comprise a set of input and output functions to model a given system. It is based on predefined if-then rules to define the relation between inputs and outputs. It depends on the user expertise and experience to adequately select or create membership functions which for a non-fuzzy logic user can be quite difficult to develop being a drawback. Fuzzy logic has been used in risk assessment in decision making activities and risk management in several areas of application ranging from engineering and economics to medical and management [5, 6]. ELECTRE is an outranking method developed in the 1960s by Bernard Roy to solve problems related to the weighted sum technique [7]. This method has several versions, at least 6 versions, and the main applications are on choosing, ranking and sorting. The main advantage is the ability to consider imprecise information; disadvantages are the high complexity of results interpretation, and the outranking procedure do not allow the comparison between alternatives [8]. PROMETHEE stands for Preference Ranking Organization METhod for Enrichment Evaluation and is also a MCDA outranking method. There are two versions of this method, the first one is the PROMETHEE I which performs partial rankings, and the second one is the PROMETHEE II which makes a complete ranking [9]. The method advantages are the ease of use, the capability to deal with quantitative and qualitative data and criteria. Disadvantages are the strong computational requirements, rank reversal [10], an unstructured decision problem which makes difficult decision makers to have a complete overview over the problem. The original method does not have a procedure to weight criteria; however, more recently pair-wise comparison procedures have been included into PROMETHEE analysis to overcome this issue [11, 12]. DEMATEL stands for DEcision MAking Trial and Evaluation Laboratory, it is a MCDM technique developed by the Battelle Geneva Research Centre to structure causal relationships [13]. This method considers the interdependence between factors and can be used to analyze complex problems being strongly used in decision making activities under imprecise and uncertain information [14]. The aforementioned models have been used to evaluate risk of projects, most of them requires quantitative data to deal with uncertainty and the ones that do not require quantitative data are complex to implement. These models were developed to generic MCDM applications and were not specific developed to risk evaluation, in this sense the development of specific models to evaluate qualitative risk is of utmost importance to support decision making activities in situation of scarce information.

4 Research Contribution

In this section, a fuzzy logic model is proposed based in the v-model framework to evaluate aggregated risk of projects under qualitative approaches. This framework is applicable to a wide range of projects which makes the developed model also applicable to an extensive number of risk scenarios, which is a feature very appreciated by the end user which normally pursuits a universal tool to deal with different scenarios. The v-model framework was initially developed to characterize the life cycle and management activities of projects and graphically summarizes the main phases usually found in any project, where the project phases activities and results are described and dynamically analyzed. The objective is to improve quality, reduce production and development costs, and improve communication between the project stakeholders and project developers.

The v-model was independently developed in the 1980s by the German government and by the Hughes Aircraft Company, an USA company. Nowadays, it is widely used in commercial and defense programs being the v-model the official project management methodology of these two countries.

There are three main versions, namely, the German v-model which is similar to PRINCE2, a UK government project management methodology, the General testing v-model framework commonly used in software development, and the US government standard which is more focused on narrower systems development.

The main advantage of this framework is the users’ participation in the model implementation and maintenance. This feature brings a strong insight to the v-model users which promotes quality and flexibility when improvements or changes to meet requirements are needed. Moreover, each project phases, inherent activities and objectives are described in detail during the v-model implementation, which promotes focus and assistance during the project implementation and update.

The mind-set required to use the v-model framework is particularly suitable to perform risk analysis because it promotes an in-deep knowledge of each project phase, considering internal and external factors as well has the interconnection between the project phases, due to this reason the v-model was chosen among many others such as the waterfall or spiral model. The v-model framework is graphically represented by a v comprising seven phases sorted into two branches as depicted in Fig. 1. The left branch comprises three phases, namely, the user requirements, functional requirements, and design specifications at both high and low levels. The right branch comprises the testing project phases where verification and validation activities are performed together with the project phases. The v-model implementation starts at the v top-left (requirements identification) then proceeds down towards the implementation phase and then up reaching the top-right (system testing), highlighting different phases with different levels, including verification and testing.

Fig. 1.
figure 1

The v-model 7 phases.

Full size image

The v-model framework starts with the requirements identification activities (phase A) to identify the user’s needs. Next, follows the high-level design (phase B) where conceptual design and integration tests are defined. Phase C, detailed design, is a low-level design activity that produces all technical components and specification documents needed to implement the project, in this phase the unit tests are also defined. Phase D, implementation, covers the project implementation activities based on the outcomes achieved in phases A, B and C. Next, in phase E, the tests defined in phase C are implemented. The results are correlated with the specifications established in phase C, if the specifications are not met then a redesign is needed. Phase F, integration testing, tests the conceptual design in order to verify the system function across all components. Phase G, system testing, correlates the product performance with the users’ requirements, this phase also may include acceptance tests performed in live environment.

Qualitative Risk Assessment Model

Figure 2 shows the conceptual idea of the qualitative risk assessment model implemented to evaluate aggregated risk of projects using the seven v-model phases and fuzzy logic functions. In the most-left region, it is depicted the v-model seven phases, ranging from A to G. For each phase, it is identified the three most critical basic events that may lead to the phase hazard. For each one of these basic events it is evaluated the respective qualitative risk in respect to the phase hazard.

Fig. 2.
figure 2

Qualitative risk assessment model framework

Full size image

Next, the phase risk in respect to the phase hazard is computed using a fuzzy logic function (MISO), this function has as input the qualitative risk values of the phase basic events and has as output the phase risk. This approach will allow to enter with more than one basic event in phase risk assessment, which is a contribution, since the common practice is to consider only the basic event that has the highest qualitative risk, however, this approach underestimates the phase risk.

To illustrate this subject, Fig. 3 shows two different scenarios, (a) and (b), for a given project phase, where it is represented the risk of three basic events for each scenario. Both cases have the same maximum risk, i.e., 0.7, however in case 3 (Fig. 2b) the risk of basic events 2 and 3 is much higher than in case 1 (Fig. 3a), in this sense, and considering that a basic event occurrence has a probabilistic behavior, neglecting the contribution of secondary basic events by considering only the maximum risk value found underestimates the overall risk. In this sense, the proposed approach overcomes this issue by entering with more than one basic event in risk assessment.

Fig. 3.
figure 3

Basic events risk (a) risk scenario 1, (b) risk scenario 2.

Full size image

After evaluating the risk of single phases, the aggregated risk between the v-model phases is computed in two stages, first, the aggregated risk of phases within the same level (AG, BF, CE) is evaluated using a TISO type of fuzzy functions. The inputs are the risk obtained for single phases and the output is the risk of combined phases, namely, AG, BF and CE. Then, the project aggregated risk is computed considering the risk calculated for the aggregated phases AG, BF, CE, and phase D, using a four variable fuzzy logic function (MISO), where the inputs are the risk obtained for the three combined phases (AG, BF and CE) plus the risk of the single phase D.

Input Membership Functions

To evaluate the risk of single and combined phases described in previous subsection it was used triangular membership functions for input as presented in Fig. 4; vertical axis represents the degree of membership and the horizontal axis represents the risk. Triangular membership functions are an effective way to modulate fuzzy inputs in decision problems and are easy to implement [1].

Fig. 4.
figure 4

Typical triangular membership functions.

Full size image

Output Membership Functions

Trapezoidal membership functions for output were developed and modelled using the risk isosurface function (RI) [15] to evaluate the risk of single and combined phases where the order of importance of each input is considered.

The risk isosurface function is a FMEA based risk assessment function to evaluate qualitative risk and was developed to overcome the shortcomings found in the original risk priority number (RPN), namely, the non-injectivity and non-surjectivity which may lead to incoherent results. Equation (1) shows the RI function for n risk variables.

$$ RI\left( {x_{1} ,x_{2} , \ldots ,x_{n} } \right)_{{x_{1} > x_{2} > \ldots > x_{n} }} = \sum\limits_{i = 1}^{n} {\left( {x_{i} - 1} \right)} \cdot \mu^{n - i} + 1 $$
(1)

where \( x_{1} ,x_{2} , \ldots ,x_{n} \) are risk variables, \( \mu \) is the scale rating maximum value, and \( x_{1} > x_{2} > \ldots > x_{n} \) is the order of importance between risk variables. Equation (2) shows an example of Eq. (1) set for a four variables risk space.

$$ RI(x_{1} ,x_{2} ,x_{3} ,x_{4} )_{{x_{1} > x_{2} > x_{3} > x_{4} }} = \left( {x_{1} - 1} \right) \cdot \mu^{3} + \left( {x_{2} - 1} \right) \cdot \mu^{2} + \left( {x_{3} - 1} \right) \cdot \mu + \left( {x_{4} - 1} \right) + 1 $$
(2)

To introduce the new method developed to create the output membership functions, the data described in Table 1 is used as an illustrative example. In this Table, two inputs A and B are considered; each one of them has three membership functions Low, Average, and High which can be represented in a fuzzy set score as 1, 2, and 3 respectively. These fuzzy set scores can also represent the weight of each membership function, therefore, Low weights 1, it has the lowest weight and High weights 3, which means that has the highest weight. The number of permutations with repetition is given by \( n^{k} \), where \( n \) is the number of elements in a set and \( k \) is the number of elements in the subset. Therefore, in this case \( 3^{2} \) yields 9 permutations as shown in Table 1.

Table 1. Example of rules and output definition for two inputs – single output function.
Full size table

In the first column we have the rule number from 1 to 9, at columns 2 to 5 we have the linguistic and fuzzy set scores for inputs A and B. The number of lines is equal to the number of permutations. In case of three inputs we have \( 3^{3} \) permutations and so on, also the number of rules is given by the number of unique permutations. These permutations are not repeated; therefore, they are unique combinations that can be related to their inherent risk. This can be made using the risk isosurface function set for the number of inputs of the fuzzy logic function considering their order of importance.

Therefore, firstly it is necessary identify the inputs order of importance which sets the position of each input value (fuzzy set score) in the risk isosurface function. For example, suppose that input B is more important than input A the risk isosurface function is set as follows in Eq. (3)

$$ RI(A,B)_{B > A} = \frac{3 \cdot B + A - 3}{9} $$
(3)

where 3 is the number of possible scores in the fuzzy set, namely 1, 2 and 3, and 9 is the RI maximum value for the fuzzy set values considered.

Column 6 of Table 1 shows the RI results for each permutation obtained for this example. Considering that, inputs A and B represent inputs of risk therefore the output evaluated by the RI function represents also a measure of risk which is a aggregated risk between inputs A and B. Correlating the fuzzy sets scores and their permutations with the RI results it can be concluded that there is a conformity between the permutations and the risk level obtained with RI. For example, comparing rules 2 and 4 it can be concluded that the order of importance considered, B > A, allows to differentiate these two rules, also the highest measure of risk in input (rule 6) has as output the highest RI value, the opposite occurs for rule 1 which has the lowest input risk. In the last two columns of Table 1 it is shown the values to define the output membership functions for each rule.

Regarding this approach we use a trapezoidal membership function for each permutation in order to define the output of the fuzzy logic function. The implementation of these membership functions usually requires four points to create the trapezoidal shape as shown in Fig. 5 where the output membership function graph for the example of Table 1 is shown as well as the points \( x_{1} \) to \( x_{4} \) of membership function number 3. In this sense, the coordinates of these points are given for each permutation as \( x_{1i} (\alpha_{i} ,0) \); \( x_{2i} (\alpha_{i} ,1) \); \( x_{3i} (\beta_{i} ,1) \); and \( x_{4i} (\beta_{i} ,0) \), where \( \alpha \) and \( \beta \) are the values of columns 7 and 8 of Table 1, and \( i \) represents the permutation number which ranges from 1 to \( n^{k} \).

Fig. 5.
figure 5

Example of output membership functions.

Full size image

The \( \alpha \) values are obtained by shifting down 1 row the RI results and the \( \beta \) values are equal to the respective RI values. In this way, there is no overlap between membership functions which allows to create an injective and surjective risk function, therefore each permutation has their unique measure of risk. Since each permutation has their own numbered membership function and since each rule definition is based on the fuzzy set scores permutation, thus the rule number is equal to the number of the membership function. In this approach there is no overlap between output membership functions which allows to differentiate the risk measure of each fuzzy set permutation. The methods described in this example can be applicable to define fuzzy logic output membership functions for functions with more than two inputs, to do that it is necessary define the order of importance of each input and them set the risk isosurface function to the number of inputs to obtain the RI results for each possible permutation. Figure 6 shows the output graph of the combined risk for inputs A and B of Table 1 example. The surface shape is in accordance with the risk inference performed based on the inputs and outputs considered showing a continuous range from 0 to 1.

Fig. 6.
figure 6

Example output plot for inputs A and B.

Full size image

For simplicity, the example considered to introduce this method covers only two risk inputs and one output, however, this method can be extrapolated for a higher number of inputs.

5 Discussion of Results

The proposed model allows to qualitatively evaluate aggregated risk by incorporating the contribution of different risk sources to get an overall risk. In literature it can be found several quantitative models to deal with this subject such as fault tree based models, event tree based models or even MCDM models as discussed in the state of the art section, but at the best authors’ knowledge, qualitative models to evaluate aggregated risk are not available in literature. In this sense the proposed model contributes to the body of knowledge in qualitative risk assessment.

In fact, there are several qualitative models to evaluate risk, but none of them evaluates aggregated risk. Failure modes and effect analysis (FMEA) is the most known qualitative risk model and the most used in practice. This model is easy to learn and use but has several shortcomings related to the way in which the risk is evaluated, due to that FMEA only prioritizes the risk of basic events and do not evaluates the respective risk of each basic event which is a drawback. Despite that it is widely disseminated in industry due to its ease of use. On the other hand, the Risk Isosurface function overcomes the limitations found in FMEA allowing the evaluation of qualitative risk with a difficulty level very similar to the one found in FMEA.

Due to that, the RI function was used to develop the output membership functions and inference rules used in the aggregated risk model. The method proposed to implement the inference rules and output membership functions guarantees that the fuzzy logic model is totally defined in its domain which is a plus since the quality of the output results is strongly related to the uniformity of the inference rules. Moreover, the risk surface depicted in Fig. 6 shows that the risk variation is in accordance with the variation of inputs which indicates that an increase of inputs levels will increase the output level, this shows that the fuzzy logic model is injective and surjective which are properties that guarantee a unique relation between inputs and outputs.

The model limitations are related with the quality of the qualitative data. The better the qualitative data the better the model estimates. This limitation is shared by all qualitative models and is a drawback pointed out by quantitative models’ developers. However, there is several strategies to overcome this limitation such as the selection of experienced experts or the use of Delphi surveys. The use of qualitative models is reserved for cases where quantitative data is not available therefore the comparison between quantitative and qualitative models must be avoided since they are applicable to different situations.

6 Conclusions and Further Work

In this work, a risk assessment model was proposed to evaluate aggregate risk of projects under a qualitative approach. The developed model is particularly suitable to evaluate the risk of innovative projects where the absence of probabilistic data does not allow the use of typical risk assessment tools which are typically based on quantitative data. The proposed model was developed using the v-model as a generic framework to evaluate the aggregated risk of projects. For each one of the seven v-model phases it is qualitatively identified three basic events that have the highest risk using qualitative risk models, such as FMEA or RPI. Then the phase risk is evaluated using a fuzzy logic model where the inputs are the risk of each basic event and the output is the respective risk phase. Next, the aggregated risk of project phases is evaluated using the same approach where the fuzzy logic inputs are the risk of each phase and the output is the aggregated risk of the respective phases. A new method to implement the output membership functions and respective inference rules was proposed considering the relative weights between input variables. The developed model contributes to the body of knowledge in qualitative risk assessment of projects considering the causal effect between project phases which allows to evaluate the aggregated risk of projects being suitable to assist in decision making. Further works are needed to implement and adapt the developed model to a real time aggregated risk evaluation approach which will allow to dynamically monitor the project risk during its implementation considering external and internal risk variables as well as their aggregation according to the project phases as suggested in [16]. This has numerous advantages such as the capability to forecast in real time the project cost, forecast the end date or even identify critical phases that need to be closely monitored allowing mitigation initiatives if needed.