Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery

Abstract

Purpose

Multi- and cross-modal learning consolidates information from multiple data sources which may offer a holistic representation of complex scenarios. Cross-modal learning is particularly interesting, because synchronized data streams are immediately useful as self-supervisory signals. The prospect of achieving self-supervised continual learning in surgical robotics is exciting as it may enable lifelong learning that adapts to different surgeons and cases, ultimately leading to a more general machine understanding of surgical processes.

Methods

We present a learning paradigm using synchronous video and kinematics from robot-mediated surgery. Our approach relies on an encoder–decoder network that maps optical flow to the corresponding kinematics sequence. Clustering on the latent representations reveals meaningful groupings for surgeon gesture and skill level. We demonstrate the generalizability of the representations on the JIGSAWS dataset by classifying skill and gestures on tasks not used for training.

Results

For tasks seen in training, we report a 59 to 70% accuracy in surgical gestures classification. On tasks beyond the training setup, we note a 45 to 65% accuracy. Qualitatively, we find that unseen gestures form clusters in the latent space of novice actions, which may enable the automatic identification of novel interactions in a lifelong learning scenario.

Conclusion

From predicting the synchronous kinematics sequence, optical flow representations of surgical scenes emerge that separate well even for new tasks that the model had not seen before. While the representations are useful immediately for a variety of tasks, the self-supervised learning paradigm may enable research in lifelong and user-specific learning.

Appendix

To test whether our method can generalize to new and clinically more relevant scenarios, we also collect a dataset on the da Vinci Research Kit [11] with a hydrogel hysterectomy phantom (University of Rochester Medical Center, constructed similarly to the phantom presented in [22]). The procedure was performed by a gynecology fellow and we annotate one section of the procedure that contains suturing with the gestures that correspond to those in JIGSAWS. As JIGSAWS does not give the calibration matrix between the tools and the camera, the kinematics would be misaligned between the two datasets. Therefore, we limit our investigations to test time inference without any retraining to study whether our method can recognize gestures in this setting.

Observations We chose to densely annotate the video instead of only predicting at the beginning of a gesture to account for possible misalignment in start times of gestures so we report one gesture every 50 frames. Given the numerous differences in how the data was collected and annotated, we report only qualitative observations. The gestures that transferred best were G3—“pushing needle through tissue” and G14—“reaching for suture with right hand”. The former action involves moving tissue, which causes denser optical flow than just moving the instruments, potentially making it easier to recognize in a new scene. The latter solely involves movement of the right instrument (compared to a potentially similar gesture “pulling suture with right hand” which involves moving the suture as well), which may also aid its transfer. Figure 5 shows a sequence of frames that were correctly labeled as “pushing needle through tissue” while Fig. 6 shows gestures that were incorrectly labeled the same.