A Comparison of Covariate Shift Detection Methods on Medical Datasets

Abstract

The performance of machine learning models is known to deteriorate on datasets drawn from a different distribution from the one used in model building. In a supervised setting, this deterioration can be assessed by a decrease in the evaluation metrics of the models. When no gold standard information is available, and thus evaluation metrics cannot be determined, one may directly address the problem of detecting whether two datasets differ in distribution. Methods for assessing the difference of distribution from their samples are known as covariate shift detection algorithms.

We investigate the ability of the maximum mean discrepancy method, of univariate tests, and of a domain classifier trained to distinguish two datasets, to detect covariate shift in two datasets: one collected for predicting stroke, and one collected for predicting acute myocardial infarction. For this, we artificially perturb parts of the datasets, and check how well these modified datasets can be distinguished from the remaining portions of the original datasets. We observe that univariate tests compare favorably with the other two methods, that changes can be detected more easily in large datasets, that smaller changes are more difficult to detect than larger changes, and that dimensionality reduction is detrimental to detecting covariate shift.