Abstract
3D brain tumor segmentation is an indispensable part of computer-aided detection systems, which is a very time-consuming and error-prone task. In recent years, many deep learning methods, such as U-net and DeepMedic, have been developed to reduce the burden on physicians and improve segmentation accuracy. Particularly with the transformer, an attention-based approach, the segmentation accuracy has been improved. However, transformer-type methods are computationally heavy and require workstations accelerated with graphic process units for fast inference. In this paper, we fuse attention block and convolutional neural network (CNN) with transformer connection to improve segmentation. Meantime, we compress the model and preserve the accuracy with the knowledge distillation technique. We investigated the knowledge distilling efficiency between different architectures. We found the knowledge from the attention-CNN fused model could improve the accuracy of the compressed U-net model. With the knowledge distillation, the compressed U-net architecture could achieve comparable accuracy with around only 11% parameters of the original U-net architecture. As for the compressed attention-CNN fused model, the knowledge from the feature attention combining the knowledge from soft target work better than soft target only. This work would help researchers to compress their model architecture properly.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4(1), 1–13 (2017)
Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)
Chen, W., Liu, B., Peng, S., Sun, J., Qiao, X.: S3D-UNet: separable 3D U-Net for brain tumor segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 358–368. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_32
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Islam, M., Vibashan, V.S., Jose, V.J.M., Wijethilake, N., Utkarsh, U., Ren, H.: Brain tumor segmentation and survival prediction using 3D attention UNet. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11992, pp. 262–272. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46640-4_25
Li, C., Chen, M., Zhang, J., Liu, H.: Cardiac MRI segmentation with focal loss constrained deep residual networks. Phys. Med. Biol. 66(13), 135012 (2021)
Li, K., Yu, L., Wang, S., Heng, P.A.: Towards cross-modality medical image segmentation with online mutual knowledge distillation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 775–783 (2020)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Pravitasari, A.A., et al.: Unet-vgg16 with transfer learning for MRI-based brain tumor segmentation. Telkomnika 18(3), 1310–1318 (2020)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Tseng, K.K., Zhang, R., Chen, C.M., Hassan, M.M.: DNetUnet: a semi-supervised CNN of medical image segmentation for super-computing AI service. J. Supercomput. 77(4), 3594–3615 (2021)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, W., Chen, C., Ding, M., Yu, H., Zha, S., Li, J.: TransBTS: multimodal brain tumor segmentation using transformer. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12901, pp. 109–119. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87193-2_11
Xu, F., Ma, H., Sun, J., Wu, R., Liu, X., Kong, Y.: LSTM multi-modal UNet for brain tumor segmentation. In: 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC), pp. 236–240. IEEE (2019)
Xu, P., et al.: Efficient knowledge distillation for liver CT segmentation using growing assistant network. Phys. Med. Biol. 66(23), 235005 (2021)
Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: UNet++: a nested U-Net architecture for medical image segmentation. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 3–11. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00889-5_1
Acknowledgement
This work was supported in part by the National Key Research and Development Program of China (No: 2020AAA0109502), and by the Talent Program of Zhejiang Province (2021R51004).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, P., Kim, K., Liu, H., Li, Q. (2022). Attention-Fused CNN Model Compression with Knowledge Distillation for Brain Tumor Segmentation. In: Yang, G., Aviles-Rivero, A., Roberts, M., Schönlieb, CB. (eds) Medical Image Understanding and Analysis. MIUA 2022. Lecture Notes in Computer Science, vol 13413. Springer, Cham. https://doi.org/10.1007/978-3-031-12053-4_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-12053-4_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-12052-7
Online ISBN: 978-3-031-12053-4
eBook Packages: Computer ScienceComputer Science (R0)