Safety Performance Evaluation for Civil Aviation Maintenance Department

Abstract

The MMEM model and the core risk analysis method are applied to construct the safety performance index system of the civil aviation maintenance department, and four categories of the safety performance indicators are set up, including maintenance error, aircraft condition, safety foundation and safety management. In order to take into account the different effects of subjective and objective weights, the comprehensive evaluation model based on the subjective and objective combination weight method for the safety performance of the airline maintenance department is established. The analytic hierarchy process and entropy method are used to calculate the subjective weight and objective weight of the index respectively, and the optimization theory is used to calculate the combination weight of the index. The data from 2011 to 2016 in a maintenance department of an airline is used as an example of application, and the results of the evaluation are analyzed. The results show that the proposed safety performance evaluation model can quantitatively reflect the safety state of the maintenance department and the evaluation results are reasonable.

1 Introduction

Aircraft maintenance is one of the important guarantees for the safe operation of the aircraft. According to the statistics of the International Air Transport Association, the proportion of aviation accidents caused by aviation maintenance errors has risen from 20% in the early twentieth Century to 80% in 1990s [1]. The maintenance department in the airline undertakes most of the maintenance work and activities in Chinese aviation transportation, and it is the basic unit of airline safety management. By evaluating safety performance of maintenance department, the airline can understand the maintenance department’s overall safety situation, and discover the safety problems in time, so as to take corresponding countermeasures. Besides, the evaluation results can be used as the basis of incentive system, and are conducive to the safety performance oriented safety culture, thus improving the airline safety level.

Based on MMEM model and core risk analysis, a comprehensive safety performance index system of maintenance department is developed in this paper. And based on the subjective and objective combination weight method, a comprehensive evaluation model of safety performance is established. The subjective weights and objective weights of the indexes are calculated by means of analytic hierarchy process and entropy method respectively, and the combination weights of the indexes are calculated by the optimization theory. The data of maintenance department of an airline from 2014 to 2016 is used as an example of application to verify the feasibility and applicability of the evaluation model.

2 Safety Performance Index System Based on MMEM Model and Core Risk Analysis

To evaluate the level of safety performance accurately, we need to establish a comprehensive and scientific index system. With the development of safety management theory, it is difficult to fully reflect the safety status only by accident, incident of this kind of high consequence indexes. A systematic and scientific index system needs to be set up to assess the safety of the organization, and to expose problems in the process of operation and management while reflecting the safety results. Therefore, the safety performance indicators established in this paper are not only limited to high consequence indicators, but also include low consequence indicators including aircraft condition, safety foundation and safety management.

2.1 The Principle for Setting Safety Performance Indexes

MMEM model is a relatively independent and closely related system composed of four components of Man, Machine, Environment and Management, and is expressed as the mutual influence and association between the four principal components of “ Man - Machine - Environment - Management ” [2]. The theory of MMEM is developed on the basis of the traditional theory of Man, Machine and Environment. It compensates for the shortcomings of traditional analysis methods, and puts forward a “management” factor that people once ignored. Modern safety management emphasizes the role of management especially. Excellent management can not only make up for the safety defects in the process, but also can effectively control the cost [3].

As a safety system, civil aircraft maintenance system is also composed of four subsystems: human, machine, environment and management. Essentially, it is also a MMEM system. According to the MMEM system theory, we establish various elements that involve safety performance from four aspects, and we can more comprehensively analyze all the factors that affect the safety status and avoid omissions. In order to accurately reflect the maintenance department’s safety status, it is necessary to take full account of its relevant historical data. Through the analysis of historical data, we find the core risk of maintenance department, and derive the safety performance indicators that affect core risk from four aspects of people, machine, environment and management.

According to this idea, index types can be divided into four categories. As shown in Fig. 1.

Fig. 1.

The idea of establishing safety performance indexes

The first category is the maintenance error. These indicators correspond to the subsystem of man, and are the indicators of high consequences. They are used to assess the risk of human error in the maintenance department.

The second category is the aircraft condition. This kind of index is corresponding to the subsystem of machine, and is a low consequence index, which is used to evaluate the technical fault status of the flying fleet.

The third category is safety foundation. This type of index corresponds to the subsystem of environment to evaluate the personnel composition and maintenance quality of the maintenance department.

The fourth category is safety management. This type of index corresponds to the subsystem of management to assess the development of safety management.

2.2 Methods for Setting Safety Performance Indexes

Setting of maintenance error indexes.

According to the standards of the unsafe events of the CAAC and the airlines, combined with the results of historical data analysis, it is possible to select the type of events related to the maintenance department as maintenance error indexes. In accordance with the event rank, it can be divided into accidents, accidents, errors, and other unsafe event etc. Table 1 is an example of part of the maintenance error indexes.

Table 1. Maintenance error indexes samples

Setting of aircraft condition indexes.

According to the historical data of the airline, the core risks associated with the maintenance department can be combed. Then the Reason model and the SHEL model are used to analyze the unsafe status that may lead to the occurrence of core risks, as shown in Fig. 2. The unsafe status related to the aircraft can be set as aircraft condition indexes, and Table 2 is an example.

Fig. 2.

Methods for setting aircraft condition indexes

Table 2. Aircraft station indexes samples

Setting of safety foundation and safety management indexes.

These indicators are used to assess the personnel composition, maintenance quality and safety management of the maintenance department. For these indicators, the brainstorming method can be used in combination with the safety management system element method. Table 3 is a part of the example of these two types of indicators.

Table 3. Safety foundation and safety management indexes samples

Table 4. Judgment matrix and weights of second grade indexes

3 Safety Performance Evaluation Model Based on the Subjective and Objective Combination Weight Method

The determination of weights plays a very important role in the comprehensive evaluation. And the difference in the method of comprehensive evaluation is also mainly reflected in the difference of the method of weight calculation. There are various kinds of weight calculation methods at present [4, 5], but in general, it can be divided into two categories: subjective method and objective method.

These two kinds of methods have their own advantages and disadvantages. Subjective weight calculation method is greatly influenced by the subjective impression of the evaluator, and the result often cannot reflect the real situation objectively. Objective method, though more objectivity in most cases, is sometimes contrary to the actual importance of each index.

Combination weight method is used to determine index weight, that is to say, the weights calculated by two or more than two weighting methods are combined, and the optimal theory is applied to build the model to get the combination weight. This method can take into account the influence of subjective and objective, and the result is in good agreement with the actual situation, so it has good practicability.

In this paper, a comprehensive evaluation model of maintenance department safety performance is established. The weight is determined by the combination weight calculation model of AHP and entropy method, considering the influence of subjective and objective factors.

3.1 Establishment Scheme of Safety Performance Evaluation Model

This paper establishes four kinds of safety performance indicators, which are different in data characteristics and attributes. Different safety performance indicators calculation models are needed. The overall scheme of safety performance evaluation model is shown in Fig. 3.

Fig. 3.

Establishment scheme of safety performance evaluation model

According to the principle of risk evaluation and the Hayne rule, the risk value calculation model is set up for the indexes of maintenance error and aircraft condition. For the indexes of safety foundation and safety management, the scores of various performance indicators are calculated according to the airlines’ assessment methods and sorting methods.

After obtaining these four kinds of performance indicators, we use AHP and entropy method to calculate the subjective weight and objective weight respectively, and then use the optimization theory to establish the model and get the combined weight of indicators. Finally, the weight vector is linearly weighted with the value vector of the evaluation index, and the comprehensive evaluation results can be obtained.

3.2 Calculation Models for Different Types of Indexes

Risk value calculation for maintenance error and aircraft condition indexes.

According to the risk assessment theory, the risk value calculation model needs to be integrated with the possibility and severity of the unsafe events. For the risk of maintenance error and aircraft condition indicators, probability is the frequency of unsafe events corresponding to the index. Severity requires a unified quantitative standard, and the severity of the unsafe events of different levels is assigned according to the Hayne rule.

On the basis of calculating methods of the possibility and the severity of each index, it can be obtained the risk value of the maintenance error index of the airline maintenance department R in a given period of evaluation.

$$ {\text{R }} = \frac{{\sum {\left( {\text{severity of each index corrending to unsafe event}} \right)} }}{\text{flight movements}} $$

(1)

Calculation for safety foundation and safety management indexes.

These two types of indicators are process indicators related to management. Take the safety management index as an example, in the internal assessment of the airline, there is a clear definition of the score for each index. The weight of different indexes can be calculated by the sorting method, then the index value of the safety management is obtained by the weighted arithmetic average method.

3.3 Comprehensive Evaluation Model

In this paper, a comprehensive evaluation model of maintenance department safety performance is established. The weight of indicators is determined by the combined weight calculation model of AHP and entropy method, considering the influence of subjective and objective factors.

For the convenience of expression, it is assumed that there are $m$ evaluation objects (${\text{i = 1,2,}} \ldots {\text{m}}$) and $n$ evaluation indicators (${\text{j = 1,2,}} \ldots {\text{n}}$), ${\text{x}}_{\text{ij}}$ represents the value of the ${\text{i}}$th evaluation object on the jth index. $w_{j}^{1}$ represents the subjective weight of the jth index, $w_{j}^{2}$ represents the objective weight of the jth index, and $w_{j}^{o}$ represents the combination weight of the jth index.

Determination of subjective weight by AHP [6,7,8].

On the basis of determining the indicators, pairwise comparisons are made by experts to the importance of each indicator based on their years of experience. Then the weight of each evaluation index is calculated by using the comparison judgment matrix and the hierarchy order. And 贴 the random consistency ratio is used to calculate the consistency of comparison judgment matrix.

Determination of objective weight by entropy method [9].

Entropy is the measure of the degree of disorder of the system, and the information is interpreted as the expression of the degree of disorder of the system, which is expressed as the variability of a certain index of the system.

That is, the smaller the variation degree of a certain index of a system is, the smaller the information content it contains, the greater its entropy value, the smaller the corresponding weight of the index in the multi index comprehensive evaluation system. Conversely, the greater the variability of a certain index of a system is, the larger the amount of information it contains and the smaller its entropy, the greater the weight of the index in multi index comprehensive evaluation system.

According to the above principle, the steps of determining the coefficient of evaluation index by entropy method are as follows.

• To deal with the indicators so that they are in the same trend.

• Using the normalization method, the proportion of the index value of the $i$th evaluation object under the jth index is calculated.

  $$ {\text{p}}_{\text{ij}} = \frac{{{\text{x}}_{\text{ij}} }}{{\sum\nolimits_{{{\text{i}} = 1}}^{\text{m}} {{\text{x}}_{\text{ij}} } }}\quad {\text{j = 1,2,}} \ldots {\text{n}} $$

  (2)

  It is assumed that ${\text{x}}_{\text{ij}} > 0$, if the hypothesis is not satisfied, a suitable method can be used to transform the data.

• Calculating the entropy of the jth index.

  $$ {\text{h}}_{\text{j}} = - \frac{1}{{\ln {\text{m}}}}\sum\nolimits_{{{\text{i}} = 1}}^{\text{m}} {{\text{p}}_{\text{ij}} } \ln {\text{p}}_{\text{ij}} ,\,\,0 \le {\text{h}}_{\text{j}} \le 1 $$

  (3)

  For a given jth index, the smaller the difference between $x_{ij}$, the greater the $h_{j}$.

• Calculating the difference coefficient of the jth index.

  $$ {\text{g}}_{\text{j}} = 1 - {\text{h}}_{\text{j}} $$

  (4)

  The greater the difference coefficient, the more important the index is.

• Calculating the objective weight of each index.

  $$ {\text{w}}_{\text{j}}^{2} = \frac{{{\text{g}}_{\text{j}} }}{{\sum\nolimits_{{{\text{j}} = 1}}^{\text{n}} {{\text{g}}_{\text{j}} } }} $$

  (5)

Combination weight method for calculating subjective and objective combination weight [10].

In order to make the results of comprehensive evaluation more scientific, a reasonable approach is to combine the weight coefficients obtained by different weighting methods according to certain methods. Through combination weighting, the results can not only embody subjective information, but also embody objective information.

It is assumed that there are ${\text{n}}$ items and ${\text{s}}$ weighting methods for a multi index comprehensive evaluation problem, then the weight vector obtained by the ${\text{kth}}$ method is expressed as ${\text{W}}^{\text{k}} = \left( {{\text{w}}_{1}^{\text{k}} ,{\text{w}}_{2}^{\text{k}} , \ldots {\text{w}}_{\text{n}}^{\text{k}} } \right),$ ${\text{k = 1,2,}} \ldots {\text{s}}$, and $\sum\nolimits_{{{\text{j}} = 1}}^{\text{n}} {{\text{w}}_{\text{j}}^{\text{k}} = 1}$. The combined weight vector is expressed as ${\text{W}}^{\text{o}} = \left( {{\text{w}}_{1}^{\text{o}} ,{\text{w}}_{2}^{\text{o}} , \ldots {\text{w}}_{\text{n}}^{\text{o}} } \right)$, and $\sum\nolimits_{{{\text{j}} = 1}}^{\text{n}} {{\text{w}}_{\text{j}}^{\text{o}} = 1}$.

The combined weight of the jth index can be expressed as a linear combination of the known $s$ weights, that is, ${{\rm w}}_{{\rm j}}^{{\rm o}} = \sum\nolimits_{{{{\rm k}} = 1}}^{{\rm s}} {\uptheta_{{\rm k}} {{\rm w}}_{{\rm j}}^{{\rm k}} }$ ${\text{j = 1,2,}} \ldots {\text{n}}$.

${\text{w}}_{\text{j}}^{\text{k}}$ is known, and the combination vector can be obtained by finding the coefficient vector of the weight.

The optimization model is set up, the goal is to minimize the square sum of the deviation of the combination weight and the known ${\text{s}}$ weight. According to the differential property of the matrix, the first derivative condition of its optimization can be obtained, which corresponds to the linear equation set below.

$$ \left| {\begin{array}{*{20}c} {\begin{array}{*{20}c} {{\text{W}}^{1} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} } & {{\text{W}}^{1} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} } \\ {{\text{W}}^{2} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} } & {{\text{W}}^{2} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} } \\ \end{array} } & {\begin{array}{*{20}c} \ldots & {{\text{W}}^{1} \cdot \left( {{\text{W}}^{\text{s}} } \right)^{\text{T}} } \\ \ldots & {{\text{W}}^{2} \cdot \left( {{\text{W}}^{\text{s}} } \right)^{\text{T}} } \\ \end{array} } \\ {\begin{array}{*{20}c} \ldots & \ldots \\ {{\text{W}}^{\text{s}} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} } & {{\text{W}}^{\text{s}} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} } \\ \end{array} } & {\begin{array}{*{20}c} \ldots & \ldots \\ \ldots & {{\text{W}}^{\text{s}} \cdot \left( {{\text{W}}^{\text{s}} } \right)^{\text{T}} } \\ \end{array} } \\ \end{array} } \right|\left| {\begin{array}{*{20}c} {\uptheta_{1} } \\ {\begin{array}{*{20}c} {\uptheta_{2} } \\ \ldots \\ {\uptheta_{\text{s}} } \\ \end{array} } \\ \end{array} } \right| = \left| {\begin{array}{*{20}c} {{\text{W}}^{1} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} } \\ {\begin{array}{*{20}c} {{\text{W}}^{2} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} } \\ \ldots \\ {{\text{W}}^{\text{s}} \cdot \left( {{\text{W}}^{\text{s}} } \right)^{\text{T}} } \\ \end{array} } \\ \end{array} } \right| $$

(6)

To solve the above formula, we can get the solution of $\left( {\uptheta_{1} ,\uptheta_{2} , \ldots\uptheta_{\text{s}} } \right)$, and then the combined weight is obtained.

Calculation of comprehensive evaluation results.

The weighted sum of the weight vector and the index value vector can be calculated, and the comprehensive evaluation result $\text{Y}_{i}$ can be obtained.

$$ {\text{Y}}_{i} = \sum\nolimits_{j = 1}^{n} {w_{j}^{o} {\text{x}}_{\text{ij}} } \quad ({\text{i = 1,2,}} \ldots {\text{m}}) $$

(7)

4 Application Example

Taking 2011–2016 years’ data of an airline’s maintenance department as an example, the safety performance evaluation model established is applied to evaluate the performance for 6 years.

4.1 Calculation of Subjective Weight by AHP

Using the AHP described in Sect 3.3, the judgment matrix and weight value of the index are calculated, and the consistency judgment is carried out. The consistency ratio of judgment matrix is less than 0.1, which satisfies the requirement of consistency.

4.2 Calculation of Objective Weight by Entropy Method

According to the method described in Sect. 3.2, the evaluation results of four categories of indexes of the maintenance department from 2011 to 2016 are obtained (see Table 5).

Table 5. Evaluation results of four types of indexes

According to the formula (2), the proportion of the index value of the evaluation object is calculated, as shown in Table 6.

Table 6. Proportion of evaluation object index value

According to the formula (3) and the formula (4), the entropy value and the difference coefficient of each index are calculated, and the objective weight of the index is calculated according to the formula (5), as listed in Table 7.

Table 7. Entropy, difference coefficient and objective weights

4.3 Calculation of Combination Weight

According to the algorithm described in the formula (6), for this example, there is:

$$ \left\{ {\begin{array}{*{20}c} {{\text{W}}^{1} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} \cdot\uptheta_{1} + {\text{W}}^{1} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} \cdot\uptheta_{2} = {\text{W}}^{1} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} } \\ {{\text{W}}^{2} \cdot \left( {{\text{W}}^{1} } \right)^{\text{T}} \cdot\uptheta_{1} + {\text{W}}^{2} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} \cdot\uptheta_{2} = {\text{W}}^{2} \cdot \left( {{\text{W}}^{2} } \right)^{\text{T}} } \\ \end{array} } \right. $$

(8)

The ${\text{W}}^{1}$ represents the subjective weight, and the ${\text{W}}^{2}$ represents the objective weight. The subjective weight and objective weight calculated in Sects 4.1 and 4.2 are replaced by the formula (8), and the equations can be obtained.

$$ \left\{ {\begin{array}{*{20}c} {0.3184\theta_{1} + 0.2344\theta_{2} = 0.3184} \\ {0.2344\theta_{1} + 0.4330\theta_{2} = 0.4330} \\ \end{array} } \right. $$

The solutions are $\uptheta_{1} = 0.4386,\uptheta_{2} = 0.7626$.

Because $\uptheta_{1}$ and $\uptheta_{2}$ are also weighted vectors, we require that $\uptheta_{1} +\uptheta_{2} = 1$. The normalization of $\uptheta_{1}$ and $\uptheta_{2}$ can be obtained.

$$ \uptheta_{1}^{{\prime }} = 0.3652,\uptheta_{2}^{{\prime }} = 0.6348 $$

The optimal combination weight vector can be calculated by the linear weighted sum of the subjective weight and the objective weight, as listed in Table 8.

Table 8. Combination weights

4.4 Results and Analysis of Comprehensive Evaluation

According to the combination weights, the weighted sum of the index is calculated, and the comprehensive evaluation results of the maintenance department from 2011 to 2016 can be obtained, as shown in Fig. 4.

Fig. 4.

Results of safety performance evaluation

From the comprehensive evaluation results of combination weight method, we can see that safety performance level is related to maintenance error, aircraft condition, safety foundation and safety management, and these factors affect and restrict each other. Safety foundation and aircraft condition are the basis and guarantee of high level safety performance, and the level of safety performance depends on the effective development of safety management.

The safety performance not only depends on the maintenance error, but also depends on the aircraft condition, the safety foundation and the safety management. For example, in 2016, although the evaluation of the maintenance error category is higher, the aircraft condition and safety foundation are poor, and the final results of the safety performance evaluation are lower than that of other years.

Compared with Tables 4, 7 and 8, the weight of the index obtained by the combination weight method is between the weights obtained by the AHP process and the entropy method. The proportion of the two in the comprehensive evaluation is determined, and their functions and effects are coordinated and balanced, and the best combination is achieved, which overcomes the one-sidedness of the single weight and makes the comprehensive evaluation more reasonable and scientific.

5 Conclusion

Safety performance evaluation index system of maintenance department of airlines was established, and the comprehensive evaluation model based on the combination of the subjective and objective weights is set up. With the historical data of 6 years, the comprehensive evaluation of the safety performance level of the airline maintenance department has been carried out.

The safety performance index system based on MMEM and core risk analysis can reflect the safety status of maintenance department from four dimensions of maintenance error, aircraft condition, safety foundation and safety management, and the index system is more comprehensive.

The established safety performance evaluation model can not only compare the performance level of each year from four dimensions, but also draw the overall level of safety performance every year, which is conducive to further analysis of safety performance level and reasons.

From the result of the evaluation, the level of safety performance should not only pay attention to the results, but also should pay more attention to process management. Process management affects the overall evaluation of safety performance.