Random forest based quantile-oriented sensitivity analysis indices estimation

Abstract

We propose a random forest based estimation procedure for Quantile-Oriented Sensitivity Analysis—QOSA. In order to be efficient, a cross-validation step on the leaf size of trees is required. Our full estimation procedure is tested on both simulated data and a real dataset. Our estimators use either the bootstrap samples or the original sample in the estimation. Also, they are either based on a quantile plug-in procedure (the R-estimators) or on a direct minimization (the Q-estimators). This leads to 8 different estimators which are compared on simulations. From these simulations, it seems that the estimation method based on a direct minimization is better than the one plugging the quantile. This is a significant result because the method with direct minimization requires only one sample and could therefore be preferred.

Appendix

1.1 Bootstrap version of the estimators of the O term

1.1.1 Quantile estimation with a weighted approach

Another estimator of the C_CDF can be achieved by replacing the weights $w_{n,j}^o \left( x_i \right)$ based on the original dataset of the forest by those using the bootstrap samples $w_{n,j}^b \left( x_i \right)$ provided in Eq. (4). That gives the following estimator which has been proposed in Elie-Dit-Cosaque and Maume-Deschamps (2022b),

$$\begin{aligned} F_{k,n}^b \left( \left. y \right| X_i = x_i \right) = \sum _{j=1}^{n} w_{n,j}^b \left( x_i \right) \mathbbm {1}_{ \{Y^j \leqslant y\}} \ . \end{aligned}$$

The conditional quantiles are then estimated by plugging $F_{k,n}^b \left( \left. y \right| X_i = x_i \right)$ instead of $F \left( \left. y \right| X_i = x_i \right)$. Accordingly, the associated estimator of $\mathbb {E}\left[ \psi _\alpha \left( Y, q^\alpha \left( \left. Y \right| X_i \right) \right) \right]$ based on these weights is denoted $\widehat{R}_i^{1, b}$.

1.1.2 Quantile estimation within a leaf

For the $\ell$-th tree, the estimator $\widehat{q}_\ell ^{b, \alpha } \left( \left. Y \right| X_i=x_i \right)$ of $q^\alpha \left( \left. Y \right| X_i = x_i \right)$ is obtained with the bootstrap observations falling into $A_n(x_i;\Theta _\ell , \mathcal {D}_n^i)$ as follows

$$\begin{aligned} \begin{aligned} \widehat{q}_\ell ^{b, \alpha } \left( \left. Y \right| X_i=x_i \right)&= \inf _{p=1,\ldots ,n} \left\{ \right. Y^{p},\ \left( X_i^p, Y^p \right) \in \mathcal {D}_n^{i \star }(\Theta _\ell ) \text { and } X_i^p \in A_n(x_i; \Theta _\ell , \mathcal {D}_n^i): \\&\left. \sum _{j=1}^n \dfrac{B_{j} \left( \Theta _\ell ^1,\mathcal {D}_n^i \right) \cdot \mathbbm {1}_{\left\{ X_i^j\in A_n(x_i;\Theta _\ell ,\mathcal {D}_n^i) \right\} } \cdot \mathbbm {1}_{\left\{ Y^j \leqslant Y^p \right\} }}{N_n^b(x_i;\Theta _\ell ,\mathcal {D}_n^i)} \geqslant \alpha \right\} \ . \end{aligned} \end{aligned}$$

That gives us the following random forest estimate of the conditional quantile

$$\begin{aligned} \widehat{q}^{b, \alpha } \left( \left. Y \right| X_i=x_i \right) = \dfrac{1}{k} \sum _{\ell =1}^{k} \widehat{q}_\ell ^{o, \alpha } \left( \left. Y \right| X_i=x_i \right) \ . \end{aligned}$$

Hence, we propose the estimator $\widehat{R}_i^{2, b}$ of $\mathbb {E}\left[ \psi _\alpha \left( Y, q^\alpha \left( \left. Y \right| X_i \right) \right) \right]$ using the bootstrap samples.

1.1.3 Minimum estimation with a weighted approach

Another estimator is obtained by replacing weigths $w_{n,j}^o \left( x_i \right)$ with the $w_{n,j}^b \left( x_i \right)$ version presented in Eq. (4) using the bootstrap samples. The obtained estimator of the O term is denoted by $\widehat{Q}_i^{1, b}$.

1.1.4 Minimum estimation within a leaf

For the $\ell$-th tree, let $N_n^b(m;\Theta _\ell ,\mathcal {D}_n^i)$ be the number of observations of the bootstrap sample $\mathcal {D}_n^{i \star } \left( \Theta _\ell \right)$ falling into the m-th leaf node and $N_{leaves}^\ell$ be the number of leaves in the $\ell$-th tree. We define the following tree estimator for the O term

$$\begin{aligned}&\dfrac{1}{N_{leaves}^\ell } \sum _{m=1}^{N_{leaves}^\ell } \Bigg ( \min \left\{ p=1,\ldots ,n,\ \left( X_i^p, Y^p \right) \in \mathcal {D}_n^{i \star }(\Theta _\ell ) \text { and } X_i^p \in A_n \left( m; \Theta _\ell , \mathcal {D}_n^i \right) \right\} \\&\quad \left. \sum _{j=1}^n \dfrac{B_{j} \left( \Theta _\ell ^1,\mathcal {D}_n^i \right) \cdot \psi _\alpha \left( Y^j, Y^{p} \right) \cdot \mathbbm {1}_{\left\{ \left( X_i^j, Y^j \right) \in \mathcal {D}_n^{i \star }(\Theta _\ell ),\ X_i^j \in A_n \left( m; \Theta _\ell , \mathcal {D}_n^i \right) \right\} }}{N_n^b(m;\Theta _\ell ,\mathcal {D}_n^i)} \right) \ . \end{aligned}$$

The approximations of the k randomized trees are then averaged to obtain the following random forest estimate

$$\begin{aligned} \widehat{Q}_i^{2, b}&= \dfrac{1}{k} \sum _{\ell =1}^k \left[ \dfrac{1}{N_{leaves}^\ell } \sum _{m=1}^{N_{leaves}^\ell } \Bigg ( \min \right. \left\{ p=1,\ldots ,n,\ \left( X_i^p, Y^p \right) \in \mathcal {D}_n^{i \star }(\Theta _\ell ) \text { and }\right. \\&\left. \quad X_i^p \in A_n \left( m; \Theta _\ell , \mathcal {D}_n^i \right) \right\} \\&\quad \left. \left. \sum _{j=1}^n \dfrac{B_{j} \left( \Theta _\ell ^1,\mathcal {D}_n^i \right) \cdot \psi _\alpha \left( Y^j, Y^{p} \right) \cdot \mathbbm {1}_{\left\{ \left( X_i^j, Y^j \right) \in \mathcal {D}_n^{i \star }(\Theta _\ell ),\ X_i^j \in A_n \left( m; \Theta _\ell , \mathcal {D}_n^i \right) \right\} }}{N_n^b(m;\Theta _\ell ,\mathcal {D}_n^i)} \right) \right] \ . \end{aligned}$$

1.1.5 Minimum estimation with a weighted approach and complete trees

By using the weights $w_{n,j}^b \left( \textbf{x}\right)$ instead of $w_{n,j}^o \left( \textbf{x}\right)$, we may define the estimator $\widehat{Q}_i^{3, b}$.

1.2 Algorithms for estimating the first-order QOSA index

Algorithm 2

QOSA index estimators plugging the quantile

Algorithm 3

QOSA index estimators with the weighted minimum approach

Algorithm 4

QOSA index estimators computing the minimum in leaves

Algorithm 5

QOSA index estimators with the weighted minimum and fully grown trees