Ipsilateral Lesion Detection Refinement for Tomosynthesis

1). Candidate Detection Module:

Single-shot object detection methods such as You Only Look Once or YOLO [31] and RetinaNet [37] can be used as fast candidate detection methods. Similar to our pilot study [30], a YOLO v2 model [38] processed the stack of DBT slice images to propose initial lesion candidates. A fully convolutional design with separate detection score and bounding-box prediction heads allowed the processing of images with different sizes and aspect ratios. For each proposed lesion candidate, the model predicted a candidate generation score $P_{cg}^i$ as well as bounding box coordinates $x$, $y$, $w$, $h$. For each DBT slice, lesion candidates with significant overlap were removed through Non-Maximum-Suppression (NMS) with an Intersection-Over-Union (IoU) threshold of 0.2. The model was trained using the original YOLO loss function [31] as in Eq. 1.

2). Patch Classification Module:

Traditionally, cascaded patch classifiers were used to refine detected lesion candidates [4]. Instead, recent deep-learning based object detection methods such as RetinaNet [37] and Mask-RCNN [33] advocated for more multi-task and end-to-end trained pipelines. However, our workflow demonstrated substantial improvement in performance by applying a simple cascaded patch classifier on top of a well-tuned candidate detection model. Lesion patches were generated by cropping a fixed $400\times 400\times 3$ patch centered on the predicted $x$, $y$, and $z$ location and fed to the patch classifier. We attributed the observed improvement to the significant imbalance of positive and negative samples in the candidate detection stage. The single view detection stage’s case-level LROC and ROI-level FROC performances are shown in Fig.6a and 6b.

The patch classifier was trained using the sigmoid cross-entropy loss as in Eq.2, where $N$ is the number of samples in a mini-batch. All proposed patches within a volume that has a patch classification score $P_{pc}^i>0.01$ are projected onto the same plane for the Volumetric Non-Maximum-Suppression (Vol-NMS) operation. The predicted x, y, w, and h from the YOLO detector and the corresponding patch classification score are used to compute the Vol-NMS output with an IoU threshold of 0.4. Surviving patches $P_{single}^i$ are used as the final output of the single-view detection stage.

1). Matching Architecture:

Several studies have shown the Siamese network can be used to re-identify images of the same object regardless of differences in lighting, angle, or image quality [34], [35], [39]. Therefore, we applied a Siamese network to re-identify the $ith$ and $jth$ lesion candidates in corresponding ipsilateral views. A generic feature extraction backbone created a $12\times 12\times 1280$ latent feature vector $f$ for each lesion candidate. To aid the matching process, a datum line is drawn from the pectoral muscle line to measure the candidate-to-pectoral-muscle distance ($d_{pec}^{ij}$) and candidate-to-nipple distance ($d_{nip}^{ij}$). The difference in the two distances $\text{Δ}{d}_{pec}^{ij}$ and $\text{Δ}{d}_{nip}^{ij}$ were embedded and concatenated to the latent features after global average pooling. Element-wise mean-square-error of the extracted feature was inputted to two fully connected ($2fc$) layers with 128 and 64 elements respectively to compute the matching probability $P_{Match}$.

The Siamese network $G$ was trained using sigmoid cross entropy loss as in Eq.4. During training, the label of $kth$ lesion candidate pair in the mini-batch $y_{Match}^k$ was set to 1 only if the two candidates were from the same screening exam and had the same lesion ID, otherwise, it was set to 0.

During inference, a greedy matching operation ranked all ipsilateral pairs based on the predicted matching probability $P_{Match}^{ij}$. Starting from the top-ranking ipsilateral pairs, the final pair relation was established only if both lesion candidates are not yet matched. This design is intended to mimic the intuition that each detection should have a unique ipsilateral pair in the corresponding view. This step is critical to allow the following refinement step to further separate cancer vs noncancer lesion candidates.

2). Ipsilateral Lesion Refinement Module:

Based on the ipsilateral matching result, the refinement module modified each single-view lesion detection score. Analogous to the way radiologists perform ipsilateral matching, lesions correlated through ipsilateral views were marked as more suspicious. To derive the $ith$ modifier value $P_{modifier}^i$ for each lesion patch, a set of lesion-specific weighting factors $\alpha$, $\beta$ and $\gamma$ were predicted to correlate the $P_{Match}$ with the optimum modifier value as in Eq.5. These weights respectively reinforced ($\alpha$) or weakened ($\beta$) the predicted $P_{Match}$ with a bias term. Then the multi-view detection score was the sum of the modifier and the single-view detection score as in Eq.6. Concrete examples of refinement processes are shown in Fig 4.

Each of the $\alpha$, $\beta$ and $\gamma$ values were predicted by independent $2fc$ regressors with a linear output activation function. Each regressor was given the 1280 length feature vector extracted from the single-view stage patch classifier. The continuous nature of matching probability $P_{Match}$ and single-view lesions score $P_{single}$ make the task an underlying regression problem. The three regressors were trained using MSE loss formulation as in Eq.7. During training, the refined scores $P_{refined}^i$ were clipped from −∞ to 1 if the patch was labeled as positive, and 0 to ∞ if the patch was labeled as negative.

3). Relation Block Patch Classifier Baseline:

The relation block learns the statistical dependency of various objects end-to-end without explicit labeling of object relations [13]. To provide a controlled comparison, we replaced the ipsilateral lesion matching and lesion refinement module in IMR-Net with a cascaded patch classifier that utilized the generic relation block module [11], [27]. The top $N$ lesion detection candidates from each ipsilateral view were paired. The exhaustive relation features of each candidate with all available candidates in the other view were computed through the relation block. Relation features were then applied to each lesion’s latent features through addition. A new $2fc$ regressor then computed the lesion score as in our patch classifier. This model was trained using sigmoid cross-entropy loss as in Eq. 2.

1). Candidate Detection:

The candidate detection model was designed to remove obvious normal tissues from the candidate pool while maintaining cancer sensitivity. A lesion was defined as positive if it was ±3 slices in the $z$ direction with respect to the central slice of the reference standard and having an Intersection-over-Union (IoU) larger than 0.2. On the fly during training, DBT slice images were randomly augmented, and slices were randomly cropped into $1200\times 1600\times 3$ patches. Benign cases were not used as negatives in the training to avoid degrading sensitivity. Only patches with scores larger than 0.4 for Hologic and 0.8 for GE were passed into the patch classifier, thus yielding an average of 100 false positives per view (FPPI) (prior to $z$ direction candidate merging), and ROI-level sensitivities were 98% for Hologic and 93% for GE on the validation dataset.

2). Patch Classifier:

For classification, $400\times 400\times 3$ patches were generated from the candidate detection results. During training, standard random augmentations were again performed on the fly. A patch was labeled as positive if it was ±1 slices away from the reference standard annotation and had an IoU larger than 0.2. During inference, only patches with classification scores larger than 0.05 were merged in the z-direction and passed into the ipsilateral processing stages. This yielded an average FFPI of 5.6 and 5.1, with an ROI-level sensitivity of 96% and 92% respectively for Hologic and GE on the validation dataset.

3). Patch Matching:

Surviving patches with a classification score larger than 0.2 for both Hologic and GE were passed into the matching model. The same random augmentation as the patch classifier training was also applied, but the random cropping and scaling factors for each ipsilateral pair were synchronized to learn the relative size relation. In object re-identification, the sampling of positive and negative pairs is critical. The following possible combinations of ipsilateral pairs for true-positive (TP) and false-positive (FP) patches were randomly sampled in equal ratios during training:

1. TP-TP positive pairs from the same cancer case.

2. TP-TP negative pairs from two different cancer cases.

3. TP-FP negative pairs from a cancer and a normal case.

4. FP-FP negative pairs from two different normal cases.

During training, only the first combination above was labeled as positive, while the others were all negatives that were intentionally defined to reduce any accidental pairing. During inference, exhaustive ipsilateral pairs were formed regardless of the TP or FP label.

For each batch, there were 16 positive pairs and 48 negative pairs for a batch size of 64. The entire model remained trainable. The best model iteration was selected based on batch-level classification AUC, which reached 0.95 and 0.92 respectively for Hologic and GE testing datasets.

4). Ipsilateral Refinement:

Next, the detection was refined using the ipsilateral modifiers. This stage was trained on the lesion detection pool that survived the classification stage and NMS operation. Additionally, ipsilateral pairs were excluded when the difference in lesion-to-nipple distance was larger than 5 cm. For each lesion candidate, the matching probability was set to 0 if no valid ipsilateral detection was found.

The trainable components of the refinement module were three independent $2fc$ regressor heads. To train the three regressors, we first replicated the patch classifier’s data pipeline, model architecture, and trained weights while attaching the three randomly initialized regressor heads. Only the newly initialized regressors remained trainable. The same augmentation during the patch classifier training was performed to prevent over-fitting. The development dataset also contained a small percentage of cases with missing ipsilateral views, for which the ipsilateral modifier was set to 0. During inference, extracted latent feature $f$ from the patch classifier model was fed to the trained $2fc$ regressor heads to obtain the three weighing factors.

5). Relation Block Patch Classifier Baseline:

To compare against our proposed method, the matching and refinement models were replaced with the well-known relation block patch classifier, which can theoretically play the same role. Specifically, all lesion candidates surviving the single-view detection stage were passed into the relation block patch classifier. Each batch contains 16 sets of ipsilateral pairs. For each batch, the top 8 lesion candidates in each view were listed. All exhaustive ipsilateral relations were computed through the relation block. The key, value, and query transformation functions [13] were implemented as three independent $1\times 1$ convolution layers; each took a feature dimension of $N\times 1280$ and output a transformed feature dimension of $N\times 320$. The lesion-to-nipple distance and lesion-to-pectoral-muscle distance were embedded and passed into the relation block as geometric features.

1). Comparison against SOTA Object Detectors:

First, the proposed single-view detection module was compared against other state-of-the-art (SOTA) detectors trained on the same dataset. The results are shown in Fig. 5. All stand-alone detection models performed inferior to our cascaded setup (p < 0.0001), which has a detection model followed by a patch classifier. This result shows the patch classifier significantly improves the separability of cancer vs noncancer candidates on top of the selected detection model. On the other hand, when YOLO is replaced with a Mask-RCNN model, the overall detection performance remained nearly identical. This indicates the performance improvement of Mask-RCNN is redundant to that of cascading a patch classifier, yielding the more complicated architecture design of Mask-RCNN unnecessary. Moreover, the focal-loss used in Retina-Net training degrades the detection performance. We hypothesize this is due to the focal loss emphasizing more difficult false-positive detection, further skewed cancer vs noncancer sample imbalance.

2). Comparison against Single-View Baseline:

First, IMR-Net was compared against the single-view baseline described in section IV with location-based evaluation metrics. The results are shown in Fig. 6. For both vendors, IMR-Net outperformed the single-view baseline substantially as shown in the AUC for case-level ROC curve. First, performance was measured as the partial area under the ROC curve (pAUC) in the high sensitivity range of 90%−100%, which is clinically more relevant than examining the entire ROC curve. In nonparametric testing, we compared the pAUC of IMR-Net to that of the single-view baseline, resulting in statistically significant improvement of the IMR-Net method (p < 0.0001) for both vendors. Second, at the operating points where the model is most likely to be deployed in the clinical workflow, the Hologic model at case sensitivity of 92.5% gained 10% in specificity (p < 0.0001); the GE model at case sensitivity of 88% gained 7% in specificity (p < 0.0001).

Qualitatively, the effect of ipsilateral pairing is illustrated in the concrete examples shown in Fig 4. For the cancer case, the $\alpha$ term increased the lesion score due to the strong ipsilateral match. For the noncancer FP, the $\beta$ term reduced the score. The overall trends are visualized in Fig 9 by comparing each ipsilateral refined score versus the single-view baseline. In these figures, the identity diagonal represents no change from ipsilateral refinement. Scores for most cancer and benign cases were shifted above the diagonal, which was expected since the matching of these actionable lesions across the two views led to greater confidence. Conversely, normal FPs were shifted below the diagonal. The matching effect or distance from the diagonal was stronger for Hologic than GE, which led to the difference in performances shown previously in Fig. 6. These differences may be driven by the greater z-resolution and thus higher inter-slice variation in the GE DBT images. Thus, cancer related imaging features might be spread out onto more than 3 DBT slices, causing a reduction in detection sensitivity in our current model architecture. Future work may address these issues with volumetric feature aggregation techniques and more robust lesion matching model designs.

3). Comparison against Relation Block:

We next constructed a relation block variation of the ipsilateral detection pipeline, with results shown in Fig 7. Despite its greater flexibility in relation modeling, the relation block variation was inferior to IMR-Net. The superiority of IMR-Net may be attributed to two factors. First, the ipsilateral matching module was trained in strong supervision with lesion pair labels. In contrast, the relation block could only perform relation learning through back-propagating information from the lesion labels (cancer vs. non-cancer) by weak supervision. Second, the relation block training required a mini-batch configuration that inherently emphasized more difficult lesion candidates due to the top $N$ detection sampling. We observed just training a generic single-view cascaded classifier with this batch configuration results in a performance trade-off between high versus low sensitivity regions on the LROC curve.

Additionally, the relation block module showed an FROC gain compared to its own single-view baseline, but that gain did not transfer to the case LROC. Clinically, a screening case can only be scored as negative if the CAD system suppressed all FP detection on all views. The FROC performance gain observed came from reducing the amount of FP detection, but this would affect only the CAD user’s interactive experience rather than case-level cancer detection performance. This experiment showed the relation block module reduced the average amount of FP CAD user need to dismiss but at the same time increased the number of cases with FP detection. This indicates ROI-Level FROC alone is not sufficient to evaluate CAD system performances.

4). Comparisons to 2D Mammography Studies:

Screening mammography and DBT exams have the same exam structure with two CC and two MLO views for each breast. The difference between mammography and DBT CAD design is mainly in two folds. First, the volumetric nature of DBT images requires the slice detection framework to merge redundant detection in different depth locations. Second, processing a large number of slices in tomosynthesis is more computationally intense and requires model architecture optimization for efficient field deployment. However, the implementation of the multi-view reading strategies should generalize to both modalities.

In Fig 8 we compared our proposed framework against other existing mammography mass detection methods [32], [41]–[44] on the CBIS-DDSM dataset using cancer lesion detection FROC. Consistent with earlier studies, only the mass subset of CBIS-DDSM dataset was used for training, as well as the metric for performance comparison. Each module is trained from scratch using only CBIS-DDSM dataset. As seen with the DBT results above, the 2D mammography performance also improved from the single-view to the ipsilateral model. Additionally, our simple models match or outperform other published operating points.