Abstract
The Receiver Operating Characteristic (ROC) analysis is a technique that is applied in medical diagnostic testing, especially for discriminating between two health status of a patient: normal (negative) and abnormal (positive). Its ability to compare the performance of different diagnostic systems based on an empirical estimation of the area under the ROC curve (AUC) has made this technique very attractive. Thus, it is important to select an appropriate software program to carry out this comparison. However, this selection has been a difficult task, considering the operational features available in each program. In this work, we aimed to demonstrate how three of the software programs available on the market allow comparing different systems based on AUC indicator, and which tests they use. The features, functionality and performance of the three software programs were evaluated, as well as advantages and disadvantages associated with their use. For illustrative purposes, we used one dataset of the Clinical Risk Index for Babies (CRIB) from Neonatal Intensive Care Units (NICUs) in Portugal.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bamber, D.: The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psycology 4, 387–415 (1975)
Braga, A.C., Costa, L., Oliveira, P.: An alternative method for global and partial comparison of two diagnostic system based on ROC curves. J. Stat. Comput. Simul. 83, 307–325 (2013)
Frade, H., Braga, A.C.: Comp2ROC: R package to compare two ROC curves. In: Mohamad, M.S., et al. (eds.) 7th International Conference on PACBB. AISC, vol. 222, pp. 127–135. Springer, Heidelberg (2013)
Braga, A.C.: Compare two ROC curves that intersect. Repository CRAN (2015)
Coelho, S., Braga, A.C.: Performance evaluation of two software for analysis through ROC curves: Comp2ROC vs SPSS. In: Gervasi, O., Murgante, B., Misra, S., Gavrilova, M.L., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2015. LNCS, vol. 9156, pp. 144–156. Springer, Heidelberg (2015)
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837–845 (1988)
Dorfman Jr., D.D., Alf Jr., E.: Maximum-likelihood estimation of parameters of signal-detection of confidence intervals-Rating-method data. J. Math. Psychol. 6, 487–496 (1969)
Egan, J.P.: Signal Detection Theory and ROC Analysis. Academic Press, New York (1975)
Fletcher, R.H., Fletcher, S.W., Wagner, E.H.: Epidemiologia Clínica: Aspetos Fundamentales. Elsevier Espanha, Barcelona (2002)
Green, D.M., Swets, J.A.: Signal Detection Theory and Psychophysics (rev.ed). Robert E. Krieger Publishing Company, Huntington (1974)
Hanley, J.A., McNeill, B.J.: The meaning and use of the area under receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982)
Hanley, J.A., McNeill, B.J.: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 3, 839–843 (1983)
Lusted, L.B.: Logical analysis in roentgen diagnosis. Radiology 74, 178–193 (1960)
Martinez, E.Z., Louzada-Neto, F., Pereira, B.B.: A curva ROC para testes diagnósticos. Cadernos Saúde Coletiva 11, 7–31 (2003)
Metz, C.E.: Basic principles of ROC analysis. Semin. Nucl. Med. 8, 283–298 (1978)
Metz, C.E.: Statistical Analysis of ROC Data in Evaluating Diagnostic Performance. Multiple Regression Analysis: Applications in the Health Sciences. American Institute of Physics (1986)
Mourão, M.F., Braga, A.C.: Evaluation of the CRIB as an indicator of the performance of neonatal intensive care units using the software ROCNPA. In: Computational Science and its Applications, pp. 151–154 (2012)
Mourão, M.F., Braga, A.C., Almeida, A., Mimoso, G., Oliveira, P.N.: Adjusting covariates in CRIB score index using ROC regression analysis. In: Gervasi, O., Murgante, B., Misra, S., Gavrilova, M.L., Rocha, A.M.A.C., Torre, C., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2015. LNCS, vol. 9156, pp. 157–171. Springer, Heidelberg (2015)
Swets, J.A., Pickett, R.M.: Signal Detection Theory and ROC Analysis in Psychology and Diagnostic: Collected Papers. Lawrence Erlbaum Associates, Inc., Mahwah (1996)
Acknowledgments
The authors would like to thank the availability of the data by the RNMBP team.
This work has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT - (Fundação para a Ciência e Tecnologia) within the Project Scope: UID/CEC/00319/2013.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Mourão, M.F., Braga, A.C. (2016). Strengths and Weaknesses of Three Software Programs for the Comparison of Systems Based on ROC Curves. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2016. ICCSA 2016. Lecture Notes in Computer Science(), vol 9786. Springer, Cham. https://doi.org/10.1007/978-3-319-42085-1_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-42085-1_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-42084-4
Online ISBN: 978-3-319-42085-1
eBook Packages: Computer ScienceComputer Science (R0)