Can a Machine Learn from Radiologists' Visual Search Behaviour and Their Interpretation of Mammograms-a Deep-Learning Study - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Oct;32(5):746-760.
doi: 10.1007/s10278-018-00174-z.

Can a Machine Learn from Radiologists' Visual Search Behaviour and Their Interpretation of Mammograms-a Deep-Learning Study

Suneeta Mall  1 , Patrick C Brennan  2 , Claudia Mello-Thoms  2   3
Affiliations

Can a Machine Learn from Radiologists' Visual Search Behaviour and Their Interpretation of Mammograms-a Deep-Learning Study

Suneeta Mall et al. J Digit Imaging. 2019 Oct.

Abstract

Visual search behaviour and the interpretation of mammograms have been studied for errors in breast cancer detection. We aim to ascertain whether machine-learning models can learn about radiologists' attentional level and the interpretation of mammograms. We seek to determine whether these models are practical and feasible for use in training and teaching programmes. Eight radiologists of varying experience levels in reading mammograms reviewed 120 two-view digital mammography cases (59 cancers). Their search behaviour and decisions were captured using a head-mounted eye-tracking device and software allowing them to record their decisions. This information from radiologists was used to build an ensembled machine-learning model using top-down hierarchical deep convolution neural network. Separately, a model to determine type of missed cancer (search, perception or decision-making) was also built. Analysis and comparison of variants of these models using different convolution networks with and without transfer learning were also performed. Our ensembled deep-learning network architecture can be trained to learn about radiologists' attentional level and decisions. High accuracy (95%, p value ≅ 0 [better than dumb/random model]) and high agreement between true and predicted values (kappa = 0.83) in such modelling can be achieved. Transfer learning techniques improve by < 10% with the performance of this model. We also show that spatial convolution neural networks are insufficient in determining the type of missed cancers. Ensembled hierarchical deep convolution machine-learning models are plausible in modelling radiologists' attentional level and their interpretation of mammograms. However, deep convolution networks fail to characterise the type of false-negative decisions.

Keywords: Breast cancer; Deep learning; Machine learning; Mammography; Visual search.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Foveal areas (FAs) are the breast areas measuring 2.5° radial angle consisting of at least 3 temporally sequential fixations. These are highlighted with white circles. Red star indicates true malignancy, and blue square mark indicates location where a radiologist reported a malignant finding. Green points and dotted lines represent the temporal visual search behaviour (fixation points and the temporal sequencing amidst these points). The FAs (total 2) containing blue star in this figure on the right view have been classified as True Positive (TP) as the true cancer lies within the FA
Fig. 2
Fig. 2
Peripheral areas (PAs) are breast areas within 2.5° radial angle from a location where a decision was made by radiologists, consisting of less than 3 temporally sequential fixations. In this figure, the area shown in red circle is an example of PA. PA, in this example, is TP. For details of the figure annotations, please refer to Fig. 1 legend
Fig. 3
Fig. 3
Never fixated areas (NFAs) are breast areas that did not receive any fixations by any of the 8 radiologists. This figure overlays visual search behaviour of all radiologists for the case indicating areas that did not receive any attention by any of the radiologists. Example of the NFA is shown in orange circle. For details of the figure annotations, please refer to Fig. 1 legend
Fig. 4
Fig. 4
Pictorial representation of high-level architecture of deep convolution neural network
Fig. 5
Fig. 5
Network architecture of iALD models. Nodes with symbol “formula image” are the ConvNet nodes
Fig. 6
Fig. 6
Network architecture of MC models. Nodes with symbol “formula image” are the ConvNet nodes
Fig. 7
Fig. 7
Two examples of each attentional level area type (FA, PA and NFA) showing both the original area image obtained from grey-scale mammographic image and the colour-converted image (obtained by applying normalization and colour conversion using lookup table approach)
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

References

    1. AIHW: Cancer in Australia 2017," in Cancer series no. 101. Cat. No. CAN 100. Canberra: AIHW, 2017
    1. (May, 2018). Australian Institute of Health and Welfare 2017. Australian Cancer Incidence and Mortality (ACIM) books: Breast Cancer. Available: https://www.aihw.gov.au/reports/cancer/acim-books
    1. S. I. Ferlay J, Ervik M, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray, F. (2014, 16/1/2015). GLOBOCAN 2012 v1.1, Cancer Incidence and Mortality Worldwide: IARC Cancer Base No. 11 [Internet]. Available: http://globocan.iarc.fr - PubMed
    1. S. Mall, S. Lewis, P. Brennan, J. Noakes, and C. Mello-Thoms, "The role of digital breast tomosynthesis in the breast assessment clinic: a review, " Journal of Medical Radiation Sciences, pp. n/a-n/a, 2017. - PMC - PubMed
    1. Nelson HD, Fu R, Cantor A, Pappas M, Daeges M, Humphrey L. Effectiveness of breast cancer screening: systematic review and meta-analysis to update the 2009 U.S. preventive services task force recommendation. Annals of Internal Medicine. 2016;164(4):244. doi: 10.7326/M15-0969. - DOI - PubMed