Sample Size for Assessing Agreement between Two Methods of Measurement by Bland−Altman Method

1 Introduction

The original article by Bland and Altman [1] which proposed the method of agreement analysis has received more than 30,000 citations in the biomedical literature and has increased in usage in recent years. Nature asked Thomson Reuters, which now owns the SCI, to list the 100 most highly cited papers of all time. The third most frequently cited statistics paper (number 29) is a 1986 publication by British statisticians Martin Bland and Douglas Altman who proposed a technique – now known as the Bland–Altman plot [2]. It has been stressed that estimates of limits of agreement (LOAs) should be accompanied with confidence intervals. However, they found that “confidence intervals are seldom quoted” in reports of method comparison studies [3–5]. A reporting standards for Bland–Altman agreement analysis in laboratory research found that the CI limits of LOAs were reported in only 6 % of 50 studies published later than 2012 [6]. Many researchers forget that Bland and Altman presented the limits of agreement as a reference interval only. The LOAs do not guarantee coverage on the range of potential differences between the two measurements and cannot be used directly for statistical inference. Just as Hamilton and Stamey pointed out, the Bland–Altman limits of agreement by themselves provide only a reference interval and should never be used as the determining factor to conclude agreement between two methods [7]. They informed that future researchers should take this variability into account and always provide confidence intervals when using the LOAs approach. Like any other clinical trials, it is essential to determine the sample size for method comparison studies [8]. Although Bland has given the sample size for a study of agreement between two methods of measurement which were available from his website [9], the sample size was determined without considering the power of the statistical procedure and could not guarantee the power of test. Sample size calculations were performed in only 30 % of publications reviewed [6]. Bland and Altman hoped to find time to publish some of these, for example, on sample size estimation for measurement method agreement studies, but up to now it is still not solved satisfactorily [3].

In this paper we propose a method to calculate sample size for Bland–Altman method, and Monte-Carlo simulations are used to validate the correctness of the method. In Section 2, we introduce the assumptions and theory of Bland–Altman method. Section 3 focuses on the derivation of the new method of sample size estimation for Bland–Altman method. In Section 4, Monte-Carlo simulation is used to obtain the corresponding powers. The results of Monte-Carlo simulation showed that the achieved powers could coincide with the pre-determined level of powers, thus validating the correctness of the method. In Section 5, we show a clinical worked example from a set of measured data of free prostate specific antigen (FPSA), which is often used to evaluate the presence of prostate cancer and other prostate disorders. We give concluding remarks in Section 6.

2 Method

3 Sample size formulae

Considering the confidence interval estimation of LOAs has symmetry of μ and –μ (μ ≥ 0) and the sample size estimations of these two situations should be the same, we just discuss the situation when μ ≥ 0. According to the statistical inference principle of Bland–Altman limits of agreement, we can separate total type I error (α) into two parts which are both α/2. Similarly, we can separate total type II error (β) into two parts. One is the first type II error (β1) of the upper limit value of LOAs and the other is the second type II error (β2) of the lower limit value of LOAs (Figure 1.) [13].

Figure 1:

Decomposition diagram of type I error (α) and type II error (β) for Bland and Altman method. Total type I error (α) consisted of two parts which are both α/2. Similarly, total type II error (β) consisted of two parts. One is the first type II error (β₁) of the upper limit value of LOAs and the other is the second type II error (β₂) of the lower limit value of LOAs.

We can get a direct estimate of β1 and β2:

where Φ(∙) is defined as the cumulative density function of standard normal distribution N(0,1), seLOAs is the standard error of the lower limit or upper limit of LOAs, seLOAs=SD1n+z1−γ/222(n−1), δ is the maximum allowable difference that can be accepted clinically, it needs to be defined in advance.

According to the statistical distribution theory, it is best to calculate the type II error (β) under the assumption of a non-central t-distribution [14], that is:

where probt∙,n−1,τ1 denotes the cumulative distribution function of a Student’s non-central t-distribution with n−1 degrees of freedom and non-centrality parameter τ1.

The non-centrality parameters τ1 and τ2 are non-central parameters defined as

We can get an estimate of the power:

When μ=0, the sample size calculation can be written as follows:

where tinv(1−β/2,n−1,t1−α/2,n−1) is defined as the inverse of a Student’s non-central t-distribution.

In eq. (6), tinv(1−β/2,n−1,t1−α/2,n−1) is related to sample size (n), we need to use iterative method to calculate sample size. Firstly we replace non-central t-distribution quantile with standard normal distribution quantile to obtain the initial value (n0), and then iterate step-by-step until n reaches a stable value.

When μ>0, we firstly calculate by eq. (6) to obtain an initial value (n1), then calculate by eq. (5) to achieve the power. If the estimated power is close enough to the pre-specified power then n is the sample size that we want to estimate. Otherwise we make n1 equal to n1+1 and judge whether the estimated power is close enough the pre-specified power. Repeat the procedure above until be closest to but greater than the pre-specified power. Table 1 summaries reasonable estimates of the sample size using eqs (5) and (6) for various standardized difference limits (μ/σ), different standardized agreement limits (δ/σ), and different type II error (β) assuming that the data are well-behaved.

Table 1 can be a reference for clinical researchers to estimate the sample size in the agreement assessment trial between two methods of measurement.

4 Simulation

Monte-Carlo simulation studies with 10,000 replicates were performed to examine the validity and correctness of the proposed formulae for estimating sample size by calculating empirical powers. Simulation data were generated on the basis of normal distribution by considering typical situations under different parameter settings. If the achieved power is very close to the pre-specified power, then it could be proved that our formulae can estimate the reasonable sample size.

As claimed above, simulating corresponding powers is easy: Firstly, we define the mean of differences (μ), the standard deviation of differences (σ) and the pre-defined clinical agreement limit (δ). And then we estimate the sample size with eqs (5) and (6), and calculate the 95 % confidence interval for the 95 % LOAs. If they lie within the pre-defined clinical agreement limits (−δ,δ), then we draw a conclusion that the two methods agree, otherwise disagree. We repeat the procedure above 10,000 times and compute the times (t) that draw agreement conclusion. The value of t10000×100% is the achieved power. Table 2 presents the achieved power for different parameter settings corresponding to the Table 1.

Table 2 indicates that the achieved powers are generally close to the pre-specified power of 80 % or 90 %. It shows that the formulae give reasonable estimates of the sample size using eqs (5) and (6) for various parameter settings.

Bland has given the sample size for a study of agreement between two methods of measurement which were available from his website [9]. In the 1986 Lancet paper they gave a formula for the confidence interval for the 95 % limits of agreement. The standard error of the 95 % limit of agreement is approximately root 3s2/n, where s is the standard deviation of the differences between measurements by the two methods and n is the sample size. The confidence interval is the estimate of the limit, dˉ plus or minus 1.96s, plus or minus 1.96 standard errors, then the sample size can be worked out.

We set α=0.05, β=0.20, μ=−0.4 0.4, σ2=1, δ=2.7, and pre-specified power = 80 %. Figure 2 shows the sample sizes and powers of B-A method and new method under different parameter settings. With the Bland–Altman method, the sample size is calculated without considering the power of the statistical procedure, and so the probability of obtaining the required width is only 0.50. With the new method, the achieved power is generally close to the pre-specified power of 80 %.

Figure 2:

Comparisons of sample size and power (%) between Bland–Altman method and New method under different parameter settings. The parameter settings are α = 0.05, β = 0.20, μ = −0.4 ~ 0.4, σ = 1, δ = 2.7.

The number of subjects required in the Bland–Altman method proposed by Bland is determined on the basis of the expected width of a confidence interval. It fails to explicitly consider the probability of achieving the desired interval width and may thus provide sample sizes that are too small to have enough power. However, the new method is more appropriate, because it can ensure an adequate probability of achieving the desired precision.

5 Example

We show a clinical worked example from a set of measured data of free prostate specific antigen (FPSA), which is often used to evaluate the presence of prostate cancer and other prostate disorders. AIA-1800 and I2000 methods were used to measure the FPSA. In the process of measurement, a same random sequence of sample was used in the two instruments [15]. Through a pre-experiment we get the mean and standard deviation of differences between AIA-1800 and I2000 methods are 0.001167 mmol/l and 0.001129 mmol/l respectively. Defining α=0.05, β=0.20, (−δ,δ)=(−0.004,0.004)mmol/l, we can calculate that a sample size of 83 would be needed to provide 80 % power to assess agreement between two methods of measurement. Monte-Carlo simulation is used to obtain the corresponding power which is equal to 80.51 %, closely to the pre-defined power (80 %).

6 Conclusion

Based on the statistical inference principle and mathematical distribution theory, we have derived the calculating formula of sample size for Bland–Altman method under different parameter settings. For the sake of convenience, we have given a set of table which can be easily find out the sample size for different standardized difference limits (μ/σ), standardized agreement limits (δ/σ), and type error (β) under two-sided α=0.05. Both α and β should be considered to have sample size large enough to ensure that the half width of a 100(1−α)% confidence interval is no larger than a pre-specified width with a pre-specified assurance probability 100(1−β)%. We carried out Monte-Caro simulation studies to validate the correctness of the proposed method. The simulation results reveal that the achieved powers could coincide with the pre-determined level of powers, thus validating the correctness of the formulae.

It is important to be aware of the pre-specified clinically acceptable agreement limits. As with equivalence or non-inferiority clinical trials, the clinical agreement limits need to be determined in advance by clinical researchers and biostatistician. Defining these agreement limits may be a difficult aspect in designing the measurement comparison studies, because they depend upon not only the clinical scenario but also other variables. Nevertheless, an attempt must be made to define them; a Delphi survey (opinion from experts) may be used to design the study. This survey is a group facilitation technique, which is an iterative multistage process designed to transform an opinion into group consensus [16].

Lin et al. [17] had discussed some issues about sample size using the tolerance interval; however, there are some deficiencies. First of all, in Lin’s study, the hypothesis of the sample size calculating method is provided just under μ=0, which is not considered for μ≠0. In fact, the two measurements may not be perfectly consistent (μ≠0), but we still believe their consistency as the population difference within a certain acceptable range (δ). In Lin’s paper, it can be seen that the simulated power is consistently less than the pre-specified power for all design specifications, so the sample sizes are much under-estimated.

Hahn and Meeker [18] defined a tolerance interval that is an interval that one can claim to contain at least a specified proportion, p, of the population with a specified degree of confidence, 100(1–α)% and provided the sample size estimation about the tolerance interval. We can see that their sample size estimation is based on the desired precision without considering type II error (β) or power, and just addresses the frequently asked question “How large a sample do I need to obtain a confidence interval?” Although our confidence interval of LOAs is similar to their tolerance interval, the theories and the procedures of sample size estimation are totally different. Our method of sample size estimation is derived not only on the pre-determined level of α but also on the β.

Recently, studies of the agreement between two instruments or clinical tests have proliferated in ophthalmic literature. McAlinden et al. used a method of sample size calculation for agreement studies on the basis of method proposed by Bland [19]. The sample size was calculated without considering the power of the statistical procedure, so the probability of obtaining the required width was only 0.50 [20]. During the study design stage, considering the power in sample size calculations could lead to expected conclusions under the predetermined power level. Cesana et al. provided another sample size estimation required for demonstrating a Pearson correlation coefficient between the differences and the means of the measures [20], and we think this method is unreasonable. Actually, the correlation coefficient given by Cesana reflected the proportional bias. As we know, one of the assumptions about the application of the Bland–Altman method is no proportional bias. Without fulfillment of the assumption, this method would not be applicable.

There are some limitations to this study. Our sample size formulae are just appropriate for the data which are well-behaved. If the data behavior is not very well, such as non-normality or non-constant variance of the differences (heteroscedasticity) and proportional bias, the formulae are not suitable to solve the problem of estimating sample size.