A latent variable approach for modeling recall-based time-to-event data with Weibull distribution

Abstract

The ability of individuals to recall events is influenced by the time interval between the monitoring time and the occurrence of the event. In this article, we introduce a non-recall probability function that incorporates this information into our modeling framework. We model the time-to-event using the Weibull distribution and adopt a latent variable approach to handle situations where recall is not possible. In the classical framework, we obtain point estimators using expectation-maximization algorithm and construct the observed Fisher information matrix using missing information principle. Within the Bayesian paradigm, we derive point estimators under suitable choice of priors and calculate highest posterior density intervals using Markov Chain Monte Carlo samples. To assess the performance of the proposed estimators, we conduct an extensive simulation study. Additionally, we utilize age at menarche and breastfeeding datasets as examples to illustrate the effectiveness of the proposed methodology.

Appendix

1.1 Expectations used in E Step of E-M algorithm

While applying the E-M algorithm in Sect. 2.1, we introduced three latent variables. The conditional densities of these introduced latent variables can be obtained by utilizing the concept of truncation. In this context, we define the conditional densities of these latent variables in a general form, applicable to arbitrary parameter values. The conditional density of non-recall latent variate $t^*_{l}$ is given by

$$\begin{aligned} f_l\big (t^*_{l} | t^*_{l}<s_i,\varrho ,\varsigma \big )= \frac{f(t^*_{l};\varrho ,\varsigma )}{1-\bar{F}(s_i;\varrho ,\varsigma )} =\frac{\varrho \varsigma {t^*_{l}}^{\varrho -1} \exp \left\{ -\varsigma {t^*_{l}}^{\varrho }\right\} }{1-\exp \left\{ -\varsigma s_i^{\varrho }\right\} }. \end{aligned}$$

(25)

The conditional density of a right-censored latent variate $t^*_{r}$ is given by

$$\begin{aligned} f_r\Big (t^*_{r} | t^*_{r}>s_i,\varrho ,\varsigma \Big )= \frac{f(t^*_{r};\varrho ,\varsigma )}{\bar{F}(s_i;\varrho ,\varsigma )} =\frac{\varrho \varsigma {t^*_{r}}^{\varrho -1} \exp \left\{ -\varsigma {t^*_{r}}^{\varrho }\right\} }{\exp \left\{ -\varsigma s_i^{\varrho }\right\} }. \end{aligned}$$

(26)

Further, the conditional density of random variate $w^*_{l}$ is given by

$$\begin{aligned} h_w\Big (w^*_{l} | w^*_{l}<(s_i-t^*_{l}),\omega \Big )= \frac{h(w^*_{l};\omega )}{1-\bar{H}(s_i-t^*_{l};\omega )} =\frac{\omega \exp \left\{ -\omega w^*_{l}\right\} }{1-\exp \left\{ -\omega (s_i-t^*_{l})\right\} }. \end{aligned}$$

(27)

Here, the symbols $\varrho$ and $\varsigma$ represent the parameters of the Weibull distribution, while $\omega$ denotes the parameter of the exponential distribution.

During the $(k+1)^{th}$ iteration in the M step of the E-M algorithm, the expectation terms of the latent variables can be computed using the conditional densities defined in Equation (25) to (27), with the obtained parameter values from the k-th step. The expectation terms utilized in the E-M algorithm and the construction of the Fisher information matrix are provided below:

$$\begin{aligned} \xi _1\Big (t_{li}^*;\alpha ,\beta \Big )&= E\Big [\ln (t_{li}^*) | t_{li}^*<s_i,\alpha ,\beta \Big ] =\frac{\int _{0}^{s_i} \ln (u) \alpha \beta u^{\alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{1-\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{I_1(s_i)-\ln (\beta ) \Big ( 1-\exp \{-\beta s_i^{\alpha }\}\Big )}{\alpha \Big (1-\exp \{-\beta s_i^{\alpha }\}\Big )}, \nonumber \\ \xi _2\Big (t_{li}^*;\alpha ,\beta \Big )&= E\Big [{t_{li}^*}^{\alpha } | t_{li}^*<s_i,\alpha ,\beta \Big ]=\frac{\int _{0}^{s_i} u^{2 \alpha -1} \alpha \beta \exp \left\{ -\beta u^{\alpha -1}\right\} du}{1-\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{1-(1+\beta s_i^{\alpha }) \exp \{-\beta s_i^{\alpha }\}}{\beta \Big (1-\exp \{-\beta s_i^{\alpha }\}\Big )}, \nonumber \\ \xi _3\Big (t_{ri}^*;\alpha ,\beta \Big )&= E\Big [\ln (t_{ri}^*) | t_{ri}^*>s_i,\alpha ,\beta \Big ]=\frac{\int _{s_i}^{\infty } \ln (u) \alpha \beta u^{\alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&= \frac{I_2(s_i)-\ln (\beta ) \Big (\exp \{-\beta s_i^{\alpha }\}\Big )}{\alpha \exp \{-\beta s_i^{\alpha }\}}, \nonumber \\ \xi _4\Big (t_{ri}^*;\alpha ,\beta \Big )&= E\Big [{t_{ri}^*}^{\alpha } | t_{ri}^*>s_i,\alpha ,\beta \Big ]=\frac{\int _{s_i}^{\infty } u^{2 \alpha -1} \alpha \beta \exp \left\{ -\beta u^{\alpha -1}\right\} du}{\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{1}{\beta }\Big (1+\beta s_i^{\alpha }\Big ), \nonumber \\ \xi _5\Big (w_{li}^*;\lambda \Big )&= E\Big [w_{li}^* | w_{li}^*<(s_i-t_{li}^*),\lambda \Big ]=\frac{\int _{0}^{s_i-t_{li}^*}u \lambda \exp \left\{ -\lambda u\right\} du}{1-\exp \{-\lambda (s_i-t_{li}^{*})\}} \nonumber \\&=\frac{1}{\lambda } \left[ \frac{1-\Big (1+\lambda (s_i-t_{li}^{*})\Big )\exp \{-\lambda (s_i-t_{li}^*)\}}{1-\exp \{-\lambda (s_i-t_{li}^{*})\}} \right] , \nonumber \\ \xi _6\Big (t_{li}^*;\alpha ,\beta \Big )&=E\Big [{t_{li}^*}^\alpha \ln (t_{li}^*) | t_{li}^*<s_i,\alpha ,\beta \Big ]=\frac{\int _{0}^{s_i} \ln (u) \alpha \beta u^{2 \alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{1-\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{ I_3(s_i)-\ln (\beta )\Big (1-(1+\beta s_i^{\alpha }) \exp \{-\beta s_i^{\alpha }\}\Big )}{\alpha \beta \Big (1-\exp \{-\beta s_i^{\alpha }\}\Big )}, \nonumber \\ \xi _7\Big (t_{li}^*;\alpha ,\beta \Big )&=E\Big [{t_{li}^*}^\alpha (\ln (t_{li}^*))^2 | t_{li}^*<s_i,\alpha ,\beta \Big ]=\frac{\int _{0}^{s_i} \Big (\ln (u)\Big )^2 \alpha \beta u^{2 \alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{1-\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&= \frac{ I_4(s_i)+\Big (\ln (\beta )\Big )^2\Big (1-(1+\beta s_i^{\alpha }) \exp \{-\beta s_i^{\alpha }\}\Big ) -2\ln (\beta ) I_3(s_i)}{\alpha \beta ^2 \Big (1-\exp \{-\beta s_i^{\alpha }\}\Big )}, \nonumber \\ \xi _8\Big (t_{ri}^*;\alpha ,\beta \Big )&=E\Big [{t_{ri}^*}^\alpha \ln (t_{ri}^*) | t_{ri}^*>s_i,\alpha ,\beta \Big ]=\frac{\int _{s_i}^{\infty } \ln (u) \alpha \beta u^{2 \alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{ I_5(s_i)-\ln (\beta )\Big ((1+\beta s_i^{\alpha }) \exp \{-\beta s_i^{\alpha }\}\Big )}{\alpha \beta \exp \{-\beta s_i^{\alpha }\}}, \nonumber \\ \xi _9\Big (t_{ri}^*;\alpha ,\beta \Big )&=E\Big [{t_{ri}^*}^\alpha \Big (\ln (t_{ri}^*)\Big )^2 | t_{ri}^*>s_i,\alpha ,\beta \Big ]=\frac{\int _{s_i}^{\infty } \Big (\ln (u)\Big )^2 \alpha \beta u^{2 \alpha -1} \exp \left\{ -\beta u^{\alpha -1}\right\} du}{\exp \{-\beta s_i^{\alpha }\}} \nonumber \\&=\frac{ I_6(s_i)+\Big (\ln (\beta )\Big )^2\Big ((1+\beta s_i^{\alpha }) \exp \{-\beta s_i^{\alpha }\}\Big )-2\ln (\beta ) I_5(s_i)}{\alpha \beta ^2\exp \{-\beta s_i^{\alpha }\}}, \end{aligned}$$

where,

$$\begin{aligned}&I_1(z)=\int _{0}^{\beta z^{\alpha }}\ln (z) \exp \{-z\}dz, \quad I_2(z)=\int _{\beta z^{\alpha }}^{\infty }\ln (z) \exp \{-z\}dz, \\&I_3(z)=\int _{0}^{\beta z^{\alpha }}z \ln (z) \exp \{-z\}dz, \quad I_4(z)=\int _{0}^{\beta z^{\alpha }}z \Big (\ln (z)\Big )^2 \exp \{-z\}dz, \\&I_5(z)=\int _{\beta z^{\alpha }}^{\infty }z \ln (z) \exp \{-z\}dz, \quad I_6(z)=\int _{\beta z^{\alpha }}^{\infty }z \Big (\ln (z)\Big )^2 \exp \{-z\}dz. \end{aligned}$$

The expectation terms provided here are calculated using Monte Carlo integration techniques during simulation.

1.2 Simulation tables

See Tables 4, 5, 6, 7.

Table 4 Mean square error (MSE) and absolute bias (AB) for ML and Bayes methods under uniform monitoring points for varying sample sizes n

Table 5 Mean square error (MSE) and absolute bias (AB) for ML and Bayes methods under exponential monitoring points for varying sample sizes n

Table 6 Average length (AL), shape and coverage probability (CP) for ML and Bayes methods under uniform monitoring points for varying sample sizes n

Table 7 Average length (AL), shape and coverage probability (CP) for ML and Bayes methods under exponential monitoring points for varying sample sizes n

1.3 MCMC convergence diagnostics

See Figs. 2, 3, 4, 5, 6, 7, 8, 9, 10.

Fig. 2

MCMC trace plots, ACF and densities of the posterior samples under gamma prior for menarche data

Fig. 3

MCMC trace plots, ACF and densities of the posterior samples under gamma prior$^1$ for menarche data

Fig. 4

MCMC trace plots, ACF and densities of the posterior samples under log-normal prior for menarche data

Fig. 5

MCMC trace plots, ACF and densities of the posterior samples under gamma prior for breastfeeding data at Level I

Fig. 6

MCMC trace plots, ACF and densities of the posterior samples under gamma prior$^1$ for breastfeeding data at Level I

Fig. 7

MCMC trace plots, ACF and densities of the posterior samples under gamma prior for breastfeeding data at Level II

Fig. 8

MCMC trace plots, ACF and densities of the posterior samples under gamma prior$^1$ for breastfeeding data at Level II

Fig. 9

MCMC trace plots, ACF and densities of the posterior samples under gamma prior for breastfeeding data at Level III

Fig. 10

MCMC trace plots, ACF and densities of the posterior samples under gamma prior$^1$ for breastfeeding data at Level III