Abstract
Image segmentation is an essential component in medical image analysis. The case of 3D images such as MRI is particularly challenging and time consuming. Interactive or semi-automatic methods are thus highly desirable. While deep learning outperforms classical methods in automatic segmentation, its use in interactive frameworks is still limited. The main reason is that most neural networks do not lend themselves well to the required user interaction loop. We propose a general deep learning-based interactive framework for image segmentation, which embeds a base network in a user interaction loop with a user feedback memory. We propose to model the memory explicitly as a sequence of consecutive framework states, from which the features can be learned. A major difficulty is related to training, as the network inputs include the user feedback and thus depend on the network’s previous output. We propose to introduce a virtual user in the training process, modelled by simulating the user feedback from the current segmentation. We demonstrate our framework on the task of female pelvis MRI segmentation, using a new dataset. We evaluate our framework against existing work with the standard metrics and conduct a user evaluation. Our framework outperforms existing systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amrehn, M., et al.: UI-Net: interactive artificial neural networks for iterative image segmentation based on a user model. In: Eurographics Workshop on Visual Computing for Biology and Medicine. The Eurographics Association (2017)
Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 1, pp. 105–112 (2001)
Collins, T., et al.: Augmented reality guided laparoscopic surgery of the uterus. IEEE Trans. Med. Imaging 40(1), 371–380 (2021)
Criminisi, A., Sharp, T., Blake, A.: GeoS: geodesic image segmentation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 99–112. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88682-2_9
Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Diaz-Pinto, A., et al.: Monai label: a framework for AI-assisted interactive labeling of 3D medical images. ArXiv abs/2203.12362 (2022)
Futrega, M., Milesi, A., Marcinkiewicz, M., Ribalta, P.: Optimized U-Net for brain tumor segmentation. ArXiv abs/2110.03352 (2021)
Goch, C.J., Metzger, J., Nolden, M.: Abstract: medical research data management using MITK and XNAT. In: Bildverarbeitung für die Medizin 2017. I, pp. 305–305. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54345-0_68
Grady, L.: Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1768–1783 (2006)
Guo, Y., Gao, Y., Shen, D.: Deformable MR prostate segmentation via deep feature learning and sparse patch matching. IEEE Trans. Med. Imaging 35(4), 1077–1089 (2016)
Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Kikinis, R., Pieper, S.D., Vosburgh, K.G.: 3D slicer: a platform for subject-specific image analysis, visualization, and clinical support. In: Jolesz, F.A. (ed.) Intraoperative Imaging and Image-Guided Therapy, pp. 277–289. Springer, New York (2014). https://doi.org/10.1007/978-1-4614-7657-3_19
Kline, T.L., et al.: Performance of an artificial multi-observer deep neural network for fully automated segmentation of polycystic kidneys. J. Digit. Imaging 30(4), 442–448 (2017)
Le’Clerc Arrastia, J., et al.: Deeply supervised UNet for semantic segmentation to assist dermatopathological assessment of basal cell carcinoma. J. Imaging 7(4), 71 (2021)
Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., Liu, H.: Expectation-maximization attention networks for semantic segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9166–9175 (2019)
Liao, X., et al.: Iteratively-refined interactive 3d medical image segmentation with multi-agent reinforcement learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9391–9399 (2020)
Lin, T.Y., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999–3007 (2017)
Minaee, S., Boykov, Y.Y., Porikli, F., Plaza, A.J., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Sakinis, T., et al.: Interactive segmentation of medical images through fully convolutional neural networks. ArXiv abs/1903.08205 (2019)
Shan, F., et al.: Lung infection quantification of COVID-19 in CT images with deep learning. ArXiv (2020)
Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V.: U-net and its variants for medical image segmentation: a review of theory and applications. IEEE Access 9, 82031–82057 (2021)
Siddiquee, M.M.R., Myronenko, A.: Redundancy reduction in semantic segmentation of 3D brain tumor MRIS. ArXiv abs/2111.00742 (2021)
Tustison, N.J., et al.: N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29(6), 1310–1320 (2010)
Wang, G., et al.: Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans. Med. Imaging 37, 1562–1573 (2018)
Wang, G., et al.: Deepigeos: a deep interactive geodesic framework for medical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1559–1572 (2019)
Zhou, B., Chen, L., Wang, Z.: Interactive deep editing framework for medical image segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 329–337. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_37
Zhou, T., Li, L., Bredell, G., Li, J., Konukoglu, E.: Quality-aware memory network for interactive volumetric image segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 560–570. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_52
Zhu, H., Meng, F., Cai, J., Lu, S.: Beyond pixels: a comprehensive survey from bottom-up to semantic image segmentation and cosegmentation. J. Vis. Commun. Image Represent. 34, 12–27 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mikhailov, I., Chauveau, B., Bourdel, N., Bartoli, A. (2022). A Deep Learning-Based Interactive Medical Image Segmentation Framework. In: Wu, S., Shabestari, B., Xing, L. (eds) Applications of Medical Artificial Intelligence. AMAI 2022. Lecture Notes in Computer Science, vol 13540. Springer, Cham. https://doi.org/10.1007/978-3-031-17721-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-17721-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-17720-0
Online ISBN: 978-3-031-17721-7
eBook Packages: Computer ScienceComputer Science (R0)
