$\hbox {C}^3$Fusion: Consistent Contrastive Colon Fusion, Towards Deep SLAM in Colonoscopy

Abstract

3D colon reconstruction from Optical Colonoscopy (OC) to detect non-examined surfaces remains an unsolved problem. The challenges arise from the nature of optical colonoscopy data, characterized by highly reflective low-texture surfaces, drastic illumination changes and frequent tracking loss. Recent methods demonstrate compelling results, but suffer from: (1) frangible frame-to-frame (or frame-to-model) pose estimation resulting in many tracking failures; or (2) rely on point-based representations at the cost of scan quality. In this paper, we propose a novel reconstruction framework that addresses these issues end to end, which result in both quantitatively and qualitatively accurate and robust 3D colon reconstruction. Our SLAM approach, which employs correspondences based on contrastive deep features, and deep consistent depth maps, estimates globally optimized poses, is able to recover from frequent tracking failures, and estimates a global consistent 3D model; all within a single framework. We perform an extensive experimental evaluation on multiple synthetic and real colonoscopy videos, showing high-quality results and comparisons against relevant baselines.

A. Appendix

1.1 A.1 Synthetic Data Generation

For the purpose of reproducibility, we state the parameters that were used to build each synthetic sequence using the synthetic colonoscopy simulator [41]. The parameters are summarised in Table. 4, where RP stands for Random Path. It is worth mentioning that the user does not have the ability to set the seed of the random number generator for the random path chosen.

1.2 A.2 Depth Training and Implementation Details

We use AdamW optimizer [23], with $\beta _1 = 0.9$, $\beta _2 = 0.999$. We train the synthetic and the Colon10k models for 40 epochs, with a batch size of 16 on a 24GB Nvidia 3090 RTX. The initial learning rate is $10^{-4}$; we reduce it by half on each of the 16th, 24th and 32nd epochs. As for the 3D colon print model, we train for 200 epochs; reduce the learning rate by half on each of the 80th, 120th and 170th epochs. We center-crop the synthetic images to $400\times 400$ to remove vignetting effects. The Colon10k images are provided in an un-distorted and center-cropped version of 270$\,\times \,$216 pixels. Finally, the cropped image is scaled to 224$\,\times \,$224 before feeding to the network. For the 3D colon print, we employ test time training due to the scarcity of the data and the fact that the training process is completely self-supervised.

To generate the specular reflection mask for each frame, we convert the input frames to YUV color-space and apply a threshold of 90% on the Y channel and dilate the resulting binary mask with a kernel of 13 pixels.

We use MMLab’s [5] implementation of ResNet [16], deformable convolutions and FPN. All ResNet encoders and the FPN were pre-trained on ImageNet [34]. We use ResNet50 for the depth encoder. For the pose encoder and FPN, we use ResNet18. Deformable convolution layers are applied in the depth encoder stages of conv3, conv4 and conv5. We set $\lambda _{ph-extra}=0.1$ and $\lambda _{dc}=0.1$, $\tau =0.01$.

1.3 A.3 Correspondence Matching Qualitative Results

Matching examples of ContraFeat, SIFT [24] and SuperPoint [11] are shown in Fig. 8. ContraFeat incline to produce more correct matches and spread out evenly throughout the image, and is less susceptible against drastic illumination changes.

Fig. 8.

Matching qualitative comparison on the synthetic data. Correct matches are lines and mismatches are lines. Mismatches defined when correspondence re-projection error is greater than 1$\%$ of colons diameter. (Color figure online)

1.4 A.4 Comparison of the Estimated Trajectories and Ground Truth Trajectories

Fig 9 compares the estimated trajectory and ground truth trajectory on the 3D colon print between DSO [12], our framework using SuperPoint [11] and our proposed method. The pose estimation from the network is of arbitrary scale. Therefore, we first align the two trajectories using similarity transform [18] following with first-frame alignment for better visualization and comparison. Note that the estimated trajectories by our framework is more accurate with loops of similar shape as compared to the ground truth trajectory.

Fig. 9.

Comparison of the estimated trajectories and ground truth trajectories on the 3D colon print sequence.

1.5 A.5 Extra Qualitative Depth-Map Predictions Results

In Fig. 10 and Fig. 11 we show extra depth-map predictions of the 3D colon print and Colon10K [25].

1.6 A.6 Extra Qualitative 3D Reconstruction Results on Colon10K

In Fig. 12 and Fig. 13 we show extra points-of-view of the 3D reconstructions by our proposed framework on Colon10K [25] and 3D colon print data.

Table 4. Synthetic data creation parameters.

1.7 A.7 SuperPoint Training

SuperPoint [11] was trained using [17] Pytorch implementation with their suggested improvements that enable end to end training using a softargmax at the detector head and a sparse descriptor loss that allows an efficient training. Photo-metric augmentations were adapted to the colon data-set by lowering the contrast, blur and noise levels to values that enabled the extraction of features even from deeper shadowed areas of the colon. the network was trained for about 100 epochs, with a batch size of 10 and learning rate of 0.0003. The best checkpoint was chosen based on validation set precision and recall.

Fig. 10.

Extra depth map prediction results on 3D colon print.

Fig. 11.

Extra depth map prediction results on Colon10K [25]. Left image exhibits a highly specular area with strong motion blur. The middle image exhibits strong illumination differences. The right image exhibits low texture images. In all three examples, our depth network produces detailed and artifact-free depth maps.

Fig. 12.

Extra reconstruction qualitative results on Colon10K [25] and 3D colon print.

Fig. 13.

Extra points-of-view of the 3D reconstruction results on Colon10K data-set. proposed framework (top), mesh reconstructed from depth and pose predictions by Godard et al. [13] (bottom).

Fig. 14.

Captured video and re-rendered reconstruction model similarity.

1.8 A.8 Supplementary Video Results

In the supplementary video, labeled as rgb_tex_geo.mp4, we show the fully endoscopic investigation of the 3D colon print while comparing the resemblance between the reconstructed model and the captured RGB images. This is accomplished by re-rendering the reconstructed model using the camera intrinsics, camera predicted pose and framework’s output mesh. In the video rgb_tex_geo.mp4 we visualise the captured video (Left) next to the re-rendered reconstructed model with texture (right). An example can be seen in Fig. 14. An additional camera fly-through video is available, labeled as fly_through.mkv, showing the final reconstruction of the 3D colon print.