A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Jan:51:101-115.
doi: 10.1016/j.media.2018.10.010. Epub 2018 Oct 28.

A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning

Naji Khosravan  1 , Haydar Celik  2 , Baris Turkbey  2 , Elizabeth C Jones  2 , Bradford Wood  2 , Ulas Bagci  3
Affiliations

A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning

Naji Khosravan et al. Med Image Anal. 2019 Jan.

Abstract

Computer aided diagnosis (CAD) tools help radiologists to reduce diagnostic errors such as missing tumors and misdiagnosis. Vision researchers have been analyzing behaviors of radiologists during screening to understand how and why they miss tumors or misdiagnose. In this regard, eye-trackers have been instrumental in understanding visual search processes of radiologists. However, most relevant studies in this aspect are not compatible with realistic radiology reading rooms. In this study, we aim to develop a paradigm shifting CAD system, called collaborative CAD (C-CAD), that unifies CAD and eye-tracking systems in realistic radiology room settings. We first developed an eye-tracking interface providing radiologists with a real radiology reading room experience. Second, we propose a novel algorithm that unifies eye-tracking data and a CAD system. Specifically, we present a new graph based clustering and sparsification algorithm to transform eye-tracking data (gaze) into a graph model to interpret gaze patterns quantitatively and qualitatively. The proposed C-CAD collaborates with radiologists via eye-tracking technology and helps them to improve their diagnostic decisions. The C-CAD uses radiologists' search efficiency by processing their gaze patterns. Furthermore, the C-CAD incorporates a deep learning algorithm in a newly designed multi-task learning platform to segment and diagnose suspicious areas simultaneously. The proposed C-CAD system has been tested in a lung cancer screening experiment with multiple radiologists, reading low dose chest CTs. Promising results support the efficiency, accuracy and applicability of the proposed C-CAD system in a real radiology room setting. We have also shown that our framework is generalizable to more complex applications such as prostate cancer screening with multi-parametric magnetic resonance imaging (mp-MRI).

Keywords: Attention; Eye-tracking; Graph sparsification; Lung cancer screening; Multi-task deep learning; Prostate cancer screening.

PubMed Disclaimer

Figures

Figure 1:
Figure 1:
To extract radiologist ROIs the dense eye-tracking data goes through a clustering and sparsification algorithm. After extracting ROIs a 3D multi-task CNN is used to perform FP removal and segmentation of lesions inside the ROIs, jointly.
Figure 2:
Figure 2:
A representation of the Eye-Tracking system in a realistic radiology setting is illustrated. Eye-Tracking system, the connection to the workstation, and the C-CAD system are integrated into the PACS (MIPAV) system directly as shown on the left. Screening experiments in normal light (a,c) and dark (b,d) radiology rooms for single (a,b) and multi-screen (c,d) experiments are shown on the right.
Figure 3:
Figure 3:
Eye-tracking data recorded from lung cancer screening. Low-dose CT is used in a single screen. Gaze patterns (right), heat maps of gaze patterns (middle), and coverage area of the gaze patterns (left) are illustrated.
Figure 4:
Figure 4:
Eye-tracking data recorded from prostate cancer screening. Multi-Parametric MRI is used in four screens (left upper: T2-weighted (T2w), right upper: apparent diffusion coefficient (ADC) map, left lower: diffusion weighted imaging (DWI), right lower: dynamic contrast enhanced (DCE) maps). Gaze patterns across different screens and the paths are illustrated for an example screening task. Gaze patterns (right), heat maps of gaze patterns (middle), and coverage area of the gaze patterns (left) are illustrated.
Figure 5:
Figure 5:
(a) 3D Graph representation of visual search patterns from a lung cancer screening experiment. (b) Clustering helps to group gaze points to define attention regions. Colors indicate different clusters.
Figure 6:
Figure 6:
Results of applying proposed graph sparsification method on a 2D dense synthetic data. Edge ratio is the ratio of edges after applying the method to the original graph.
Figure 7:
Figure 7:
The 3D deep multi-task CNN architecture. The size of all convolutions are 3×3×3 with a stride of 1 in each dimension. The downsampling and upsampling operators are performed only in the xy plane, and do not affect temporal information. All convolution and layers are 3D. The network has 14 shared layers, 2 FP removal specific layers and one segmentation specific layer.
Figure 8:
Figure 8:
Sparsification results from synthetic data experiments. The number of graph nodes are reduced from 5000 to 196 in the clustering step, and the number of edges (after clustering) are reduced from 4269 to 524 in the graph sparsification step.
Figure 9:
Figure 9:
Lung cancer screening experiments with CT data. First column: dense gaze patterns. Second column: attention based clustering. Third column: nodes in clusters are reduced. Fourth column: sparse graph after further reducing edges.
Figure 10:
Figure 10:
Lung cancer screening experiment with CT data. Dense and sparse gaze points on 3D lung surface as well as time analysis. Number of nodes in the largest cluster (N), corresponding time spent by radiologist on that cluster (Tc) and overall screening time (T), with the eye-tracker frequency being 60Hz, for each radiologist is computed.
Figure 11:
Figure 11:
Quantitative parameters to compare graph topology between already clustered data and sparsified data with respect to the preserved edge ratio. R# indicates a particular radiologist (blue, green, red). (Lung cancer screening experiment)
Figure 12:
Figure 12:
Quantitative parameters to compare graph topology between already clustered and sparsified data with respect to the preserved edge ratio. (Synthetic data experiment)
Figure 13:
Figure 13:
Inter-observer variation of MSE for 2 radiologists on 4 different scans.
Figure 14:
Figure 14:
Left: Comparison of accuracy on the test set over training epochs. An increase from 95% to 97% is observed. Middle: Comparison of DSC on test set is shown. An increase from 87% to 91% is observed in the network’s trained state. Right: FROC analysis and its comparison with the state of art for false positive reduction.
Figure 15:
Figure 15:
Prostate cancer screening experiments with multi-parametric MRI. Left: four MRI modalities and corresponding dense gaze patterns. Right: Clustered and sparsified gaze patterns corresponding to each modality. First column: clustered dense gaze patterns. Second column: attention based clustering. Third column: sparse graph after further reducing edges.
Figure 16:
Figure 16:
Prostate screening experiment quantitative results.
Figure 17:
Figure 17:
Variation of MSE on different prostate images per modality.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Al Mohammad B, Brennan P, Mello-Thoms C, 2017. A review of lung cancer screening and the role of computer-aided detection. Clinical Radiology 72, 433–442. - PubMed
    1. Al-Moteri MO, Symmons M, Plummer V, Cooper S, 2017. Eye tracking to investigate cue processing in medical decision-making: A scoping review. Computers in Human Behavior 66, 52–66.
    1. Badrinarayanan V, Kendall A, Cipolla R, 2017. Segnet: A deep convolutional encoder-decoder architecture for scene segmentation. IEEE transactions on pattern analysis and machine intelligence. - PubMed
    1. Buty M, Xu Z, Gao M, Bagci U, Wu A, Mollura DJ, 2016. Characterization of lung nodule malignancy using hybrid shape and appearance features, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer; pp. 662–670.
    1. Caroline C, 2014. Lung cancer screening with low dose ct. Radiologic clinics of North America 52, 27. - PMC - PubMed