Translational biomarker discovery in clinical metabolomics: an introductory tutorial - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Apr;9(2):280-299.
doi: 10.1007/s11306-012-0482-9. Epub 2012 Dec 4.

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Affiliations

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Jianguo Xia et al. Metabolomics. 2013 Apr.

Abstract

Metabolomics is increasingly being applied towards the identification of biomarkers for disease diagnosis, prognosis and risk prediction. Unfortunately among the many published metabolomic studies focusing on biomarker discovery, there is very little consistency and relatively little rigor in how researchers select, assess or report their candidate biomarkers. In particular, few studies report any measure of sensitivity, specificity, or provide receiver operator characteristic (ROC) curves with associated confidence intervals. Even fewer studies explicitly describe or release the biomarker model used to generate their ROC curves. This is surprising given that for biomarker studies in most other biomedical fields, ROC curve analysis is generally considered the standard method for performance assessment. Because the ultimate goal of biomarker discovery is the translation of those biomarkers to clinical practice, it is clear that the metabolomics community needs to start "speaking the same language" in terms of biomarker analysis and reporting-especially if it wants to see metabolite markers being routinely used in the clinic. In this tutorial, we will first introduce the concept of ROC curves and describe their use in single biomarker analysis for clinical chemistry. This includes the construction of ROC curves, understanding the meaning of area under ROC curves (AUC) and partial AUC, as well as the calculation of confidence intervals. The second part of the tutorial focuses on biomarker analyses within the context of metabolomics. This section describes different statistical and machine learning strategies that can be used to create multi-metabolite biomarker models and explains how these models can be assessed using ROC curves. In the third part of the tutorial we discuss common issues and potential pitfalls associated with different analysis methods and provide readers with a list of nine recommendations for biomarker analysis and reporting. To help readers test, visualize and explore the concepts presented in this tutorial, we also introduce a web-based tool called ROCCET (ROC Curve Explorer & Tester, http://www.roccet.ca). ROCCET was originally developed as a teaching aid but it can also serve as a training and testing resource to assist metabolomics researchers build biomarker models and conduct a range of common ROC curve analyses for biomarker studies.

Keywords: AUC; Biomarker analysis; Biomarker validation and reporting; Bootstrapping; Confidence intervals; Cross validation; Optimal threshold; ROC curve; Sample size.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
PubMed search results using key words “metabolomics” and “biomarker” from year 2001 to 2011
Fig. 2
Fig. 2
Illustration of TP, TN, FP, and FN with hypothetical biomarker test data. The distributions of true outcomes are given by the two Gaussian curves with positive cases on the right side and negative cases on the left. The cut-off level is indicated by the dashed line. Due to the overlap between the biomarker concentrations of the two populations, the cut-off level will misclassify the left-hand side of the positive cases and the right-hand side of the negative cases. TP true positives, TN true negatives, FP false positives, FN false negatives
Fig. 3
Fig. 3
Empirical ROC curve and optimal cut-off. After obtaining a list of sensitivity and specificity values from all possible cut-offs, one should plot all pairs of sensitivity and 1-specificity values as empty circles, and then connect each neighboring circles with line segments to generate empirical ROC curves. The optimal cut-off (solid circle in magenta) can be identified as the point with by minimal d the distance from a cut-off to the solid grey circle (0, 1), or the point with maximal vertical distance from the diagonal line, also known as the Youden index J (Color figure online)
Fig. 4
Fig. 4
Performance comparison using partial AUC. The AUC of Test A and Test B are about the same. However, Test B is superior to Test A at regions of high specificity (0.8, 1). Therefore, using the partial AUC will be more appropriate in this case
Fig. 5
Fig. 5
Using a bootstrapping approach to compute the 95 % confidence interval (CI) for a single cut-off or for the complete ROC curve
Fig. 6
Fig. 6
Comparison of different models based on ROC curves. Six biomarker models were created using a linear SVM with different numbers of features. ROC curves were generated using the predicted class probabilities from repeated cross validation for each model. The legend shows the feature numbers and the AUCs of the six models
Fig. 7
Fig. 7
Difference between ranking and classification. The two scatter plots show the predicted class probabilities for 50 new samples by two biomarker models. Both models are able to rank all new samples correctly. Therefore, they both will have the same AUC (1.0) but exhibit different error rates (3/50 and 1/50 respectively) due to their different decision boundaries, which were determined during the model creation process

Similar articles

Cited by

References

    1. Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences of the United States of America. 2002;99(10):6562–6566. doi: 10.1073/pnas.102102699. - DOI - PMC - PubMed
    1. Arkin CF, Wachtel MS. How many patients are necessary to assess test performance? JAMA: The Journal of the American Medical Association. 1990;263(2):275–278. doi: 10.1001/jama.1990.03440020109043. - DOI - PubMed
    1. Atkinson AJ, Colburn WA, DeGruttola VG, DeMets DL, Downing GJ, Hoth DF, et al. Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework*. Clinical Pharmacology and Therapeutics. 2001;69(3):89–95. doi: 10.1067/mcp.2001.113989. - DOI - PubMed
    1. Bahado-Singh RO, Akolekar R, Mandal R, Dong E, Xia J, Kruger M, et al. Metabolomics and first-trimester prediction of early-onset preeclampsia. Journal of Maternal, Fetal and Neonatal Medicine. 2012;25(10):1840–1847. doi: 10.3109/14767058.2012.680254. - DOI - PubMed
    1. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology. 1975;12(4):387–415. doi: 10.1016/0022-2496(75)90001-2. - DOI

LinkOut - more resources