Measuring the accuracy of diagnostic systems.

@article{Swets1988MeasuringTA,
  title={Measuring the accuracy of diagnostic systems.},
  author={John A. Swets},
  journal={Science},
  year={1988},
  volume={240 4857},
  pages={
          1285-93
        },
  url={https://api.semanticscholar.org/CorpusID:10006606}
}
For diagnostic systems used to distinguish between two classes of events, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy.
View on PubMed
wixtedlab.ucsd.edu
9,363 Citations
Highly Influential Citations
1,749
Background Citations
1,965
Methods Citations
2,278
Results Citations
101

9,363 Citations

ROC Analysis in the Evaluation of Intelligent Medical Systems

A new method for generating the probability density functions and confidence bounds for ROC points which is robust and accurate for any sample size is presented, particularly suited for evaluating the performance of systems where sample sizes are small.

Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.

Receiver-operating characteristic (ROC) plots provide a pure index of accuracy by demonstrating the limits of a test's ability to discriminate between alternative states of health over the complete spectrum of operating conditions.

Classification of diagnostic tests.

Assessing Test Accuracy and its Clinical Consequences: A Primer for Receiver Operating Characteristic Curve Analysis

These are the basic features necessary for the acceptable performance of ROC curve analysis, which are not dealt with in any detail in current textbooks addressed to the needs of clinical chemists.

Evaluation of intelligent medical systems

This work establishes an important new method for generating pdf's, and provides an accurate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent medical systems.

Comparing and Optimizing Diagnostic Tests

Though it is not appropriate for all clinical situations, the ROC analysis method can be used for most diagnostic tests whenever information is desired for its own sake or when reducing uncertainty is the goal of testing.

Advances in statistical methodology for the evaluation of diagnostic and laboratory tests.

Two new simultaneous confidence regions associated with the ROC curve are presented, one based on Kolmogorov-Smirnov confidence bands for distribution functions and the other based on bootstrapping.

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

A novel, robust and accurate method is proposed, derived from first principles, which calculates the probability density function for each point on a ROC curve for any given sample size, and provides an accurate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent medical systems.

Receiver operating characteristic studies and measurement errors.

Monte Carlo simulation indicates that, for several families of distributions, the bias-corrected estimators have smaller bias and comparable MSE to the usual estimator.

Relation of Observer Agreement to Accuracy According to a Two-receiver Signal Detection Model of Diagnosis

A model of test interpretation is defined which accounts for disagreement when observers use identical diagnostic strategies, and determines an optimal diagnostic strategy with advantages over maximum likelihood and Bayes diagnostic strategies.
...

38 References

Indices of discrimination or diagnostic accuracy: their ROCs and implied models.

It is suggested that candidate indices be evaluated by calculating their relative operating characteristics (ROCs) so that the form of an index's ROC identifies the model of the discrimination process that is implied by the index, and that theoretical form can be compared with the forms of empirical ROCs.

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests.

To determine why many diagnostic tests have proved to be valueless after optimistic introduction into medical practice, we reviewed a series of investigations and identified two major problems that…

ROC analysis applied to the evaluation of medical imaging techniques.

This paper presents a brief description of the ROC, and shows how it provides a measure of diagnostic accuracy that is free of judgmental bias.

ROC Methodology in Radiologic Imaging

This article develops ROC concepts in an intuitive way by identifying the fundamental issues that motivate ROC analysis and practical techniques for ROC data collection and data analysis.

On the scientific evaluation of diagnostic procedures.

It is realized that often the process contains some element of uncertainty so that diagnosis might be defined as “the procedure of reaching the most probable conclusion based on the facts at hand.”

An Evaluation of Polygraphers' Judgments: A Review From a Decision Theoretic Perspective

Hebrew University, Jerusalem, IsraelA critical discussion of the theory, practice, and evaluation process of the ControlQuestions Technique (CQT) is undertaken. Sources of possible contaminationin…

Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance.

Measuring both accuracy and the decision criterion appropriately enables one to delineate which of these dependent variables is affected by which independent variables in psychological studies, and to assess the efficacy as well as the accuracy of diagnostic systems.

The Effect of Verification on the Assessment of Imaging Techniques

To assess relative accuracies of three different radiographic techniques, chest radiographs were taken of 66 patients with each of the three techniques, and the films were evaluated by six radiologists, finding that any one of the techniques could be shown to be better than the other, depending on how the truth was defined.

Construction of Receiver Operating Characteristic Curves when Disease Verification Is Subject to Selection Bias

A method to provide an unbiased estimate of the ROC curve under these sample conditions is proposed, which requires information on the probability distribution of test results in the population from which the verified sample is drawn.

Assessment of diagnostic technologies.

A general protocol for rigorous evaluation of diagnostic systems in medicine was applied successfully in a comparative study of two radiologic techniques, and computed tomography was found to be substantially more accurate than radionuclide scanning.