Abstract
Surgical instrument segmentation is an essential task in the domain of computer-assisted surgical system. It is critical to increase the context-awareness of surgeons during the operation. We propose a new model based on the U-Net architecture for surgical instrument segmentation, which aggregates multi-scale feature maps and has cascaded dilated convolution layers. The model adopts dense upsampling convolution instead of deconvolution for upsampling. We set the side loss function on each side-output layer. The loss function includes an output loss function and all side loss functions to supervise the training of each layer. To validate our model, we compare our proposed model with advanced architecture U-Net in the dataset consisting of laparoscopy images from multiple surgical operations. Experiment results demonstrate that our model achieves good performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bouget D, Allan M, Stoyanov D, Jannin P: Vision-based and marker-less surgical tool detection and tracking: A review of the literature. Med Image Anal 35:633–654, 2017. https://doi.org/10.1016/j.media.2016.09.003
Baumhauer M, Feuerstein M, Meinzer H-P, Rassweiler J: Navigation in endoscopic soft tissue surgery: Perspectives and limitations. J Endourol 22:751–766, 2008. https://doi.org/10.1089/end.2007.9827
Lalys F, Jannin P: Surgical process modelling: A review. Int J Comput Assist Radiol Surg 9:495–511, 2014. https://doi.org/10.1007/s11548-013-0940-5
Doignon C, Nageotte F, De Mathelin M: Detection of grey regions in color images: Application to the segmentation of a surgical instrument in robotized laparoscopy, in: 2004 IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IEEE Cat. No.04CH37566), IEEE, n.d.: pp. 3394–3399. https://doi.org/10.1109/IROS.2004.1389941
Bouget D, Benenson R, Omran M, Riffaud L, Schiele B, Jannin P: Detecting surgical tools by modelling local appearance and global shape. IEEE Trans Med Imaging 34:2603–2617, 2015. https://doi.org/10.1109/TMI.2015.2450831
Allan M, Ourselin S, Thompson S, Hawkes DJ, Kelly J, Stoyanov D: Toward detection and localization of instruments in minimally invasive surgery. IEEE Trans Biomed Eng 60:1050–1058, 2013. https://doi.org/10.1109/TBME.2012.2229278
Sznitman R, Ali K, Richa R, Taylor RH, Hager GD, Fua P: Data-driven visual tracking in retinal microsurgery. Med Image Comput Comput Assist Intervent 2012:568–575, 2012. https://doi.org/10.1007/978-3-642-33418-4_70
Pan SJ, Yang Q: A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359, 2010. https://doi.org/10.1109/TKDE.2009.191
Long J, Shelhamer E, Darrell T: Fully convolutional networks for semantic segmentation, in: 2015 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE, 2015: pp. 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965
Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition, (2014). http://arxiv.org/abs/1409.1556 (accessed September 28, 2018)
He K, Zhang X, RenS, Sun J: Deep residual learning for image recognition, in: 2016 IEEE Conf. Comput. Vis. Pattern Recognit., IEEE, 2016: pp. 770–778. https://doi.org/10.1109/CVPR.2016.90
Attia M, HossnyM, Nahavandi S, Asadi H: Surgical tool segmentation using a hybrid deep CNN-RNN auto encoder-decoder, in: 2017 IEEE Int. Conf. Syst. Man, Cybern., IEEE, 2017: pp. 3373–3378. https://doi.org/10.1109/SMC.2017.8123151
Garcia-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S: ToolNet: Holistically-nested real-time segmentation of robotic surgical tools, in: 2017 IEEE/RSJ Int. Conf. Intell. Robot. Syst., IEEE, 2017: pp. 5717–5722. https://doi.org/10.1109/IROS.2017.8206462
García-Peraza-Herrera LC, Li W, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S: Real-time segmentation of non-rigid surgical tools based on deep learning and tracking, in: International Workshop on Computer-Assisted and Robotic Endoscopy (CARE), 2017: pp. 84–95. https://doi.org/10.1007/978-3-319-54057-3_8
Ronneberger O, Fischer P, Brox T: U-Net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, 2015: pp. 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Allan M, Shvets A, Kurmann T, Zhang Z, Duggal R, Su Y-H, Rieke N, Laina I, Kalavakonda N, Bodenstedt S, Herrera L, Li W, Iglovikov V, Luo H, Yang J, Stoyanov D, Maier-Hein L, Speidel S, Azizian M: 2017 Robotic Instrument Segmentation Challenge, 2019. http://arxiv.org/abs/1902.06426 (accessed 23 Feb 2019)
Yu F, Koltun V: Multi-Scale Context Aggregation by Dilated Convolutions, Eprint ArXiv:1511.07122. (2015). http://arxiv.org/abs/1511.07122 (accessed 28 Sept 2018)
Xie S, Tu Z: Holistically-nested edge detection. Int J Comput Vis 125:3–18, 2017. https://doi.org/10.1007/s11263-017-1004-z
Zeiler MD, Krishnan D, Taylor GW, Fergus R: Deconvolutional networks, in: 2010 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., IEEE, 2010: pp. 2528–2535. https://doi.org/10.1109/CVPR.2010.5539957
Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, Cottrell G: Understanding convolution for semantic segmentation, in: 2018 IEEE Winter Conf. Appl. Comput. Vis., IEEE, 2018: pp. 1451–1460. https://doi.org/10.1109/WACV.2018.00163
Chang H-H, Zhuang AH, Valentino DJ, Chu W-C: Performance measure characterization for evaluating neuroimage segmentation algorithms. Neuroimage. 47:122–135, 2009. https://doi.org/10.1016/j.neuroimage.2009.03.068
Schaefer S, McPhail T, Warren J: Image deformation using moving least squares. ACM Trans Graph 25:533, 2006. https://doi.org/10.1145/1141911.1141920
Kingma DP, Ba J: Adam: A method for stochastic Optimization, (2014). http://arxiv.org/abs/1412.6980 (accessed 4 Oct 2018)
Shvets A, Rakhlin A, Kalinin A, Iglovikov V: Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning, (2018). https://arxiv.org/abs/1803.01207
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Yu, L., Wang, P., Yu, X. et al. A Holistically-Nested U-Net: Surgical Instrument Segmentation Based on Convolutional Neural Network. J Digit Imaging 33, 341–347 (2020). https://doi.org/10.1007/s10278-019-00277-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-019-00277-1