Abstract
Brain lesion segmentation plays an essential role in neurological research and diagnosis. As brain lesions can be caused by various pathological alterations, different types of brain lesions tend to manifest with different characteristics on different imaging modalities.Due to this complexity, brain lesion segmentation methods are often developed in a task-specific manner. A specific segmentation model is developed for a particular lesion type and imaging modality.However, the use of task-specific models requires predetermination of the lesion type and imaging modality, which complicates their deployment in real-world scenarios.In this work, we propose a universal foundation model for 3D brain lesion segmentation, which can automatically segment different types of brain lesions for input data of various imaging modalities. We formulate a novel Mixture of Modality Experts (MoME) framework with multiple expert networks attending to different imaging modalities. A hierarchical gating network combines the expert predictions and fosters expertise collaboration. Furthermore, we introduce a curriculum learning strategy during training to avoid the degeneration of each expert network and preserve their specialisation. We evaluated the proposed method on nine brain lesion datasets, encompassing five imaging modalities and eight lesion types.The results show that our model outperforms state-of-the-art universal models and provides promising generalisation to unseen datasets.
Work conducted as a visiting PhD student at Imperial College London, under the joint supervision of corresponding authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Avants, B.B., Tustison, N.J., Song, G., et al.: A reproducible evaluation of ANTs similarity metric performance in brain image registration. NeuroImage 54(3) (2011)
Baid, U., Ghodasara, S., Bilello, M., et al.: The RSNA-ASNR-MICCAI BraTS 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021)
Basaran, B.D., Zhang, W., Qiao, M., et al.: LesionMix: A lesion-level data augmentation method for medical image segmentation. arXiv preprint arXiv:2308.09026 (2023)
Butoi, V.I., Gonzalez Ortiz, J.J., Ma, T., et al.: UniverSeg: Universal medical image segmentation. International Conference on Computer Vision (2023)
Chen, T., Chen, X., Du, X., et al.: AdaMV-MoE: Adaptive multi-task vision mixture-of-experts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
Chen, Z., Shen, Y., Ding, M., et al.: Mod-Squad: Designing mixtures of experts as modular multi-task learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
Chi, Z., Dong, L., Huang, S., et al.: On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems 35 (2022)
Commowick, O., Istace, A., Kain, M., et al.: Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure. Scientific Reports 8(1) (2018)
Czolbe, S., Dalca, A.V.: Neuralizer: General neuroimage analysis without re-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
Gao, Y., Li, Z., Liu, D., et al.: Training like a medical resident: Universal medical image segmentation via context prior learning. arXiv preprint arXiv:2306.02416 (2023)
Gong, S., Zhong, Y., Ma, W., et al.: 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable medical image segmentation. arXiv preprint arXiv:2306.13465 (2023)
Hernandez Petzsche, M.R., de la Rosa, E., Hanning, U., et al.: ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset. Scientific Data 9(1) (2022)
Isensee, F., Jaeger, P.F., Kohl, S.A., et al.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2) (2021)
Kamnitsas, K., Ledig, C., Newcombe, V.F., et al.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36 (2017)
Kirillov, A., Mintun, E., Ravi, N., et al.: Segment anything. ICCV (2023)
Kuijf, H.J., Biesbroek, J.M., De Bresser, J., et al.: Standardized assessment of automatic segmentation of white matter hyperintensities and results of the WMH segmentation challenge. IEEE Transactions on Medical Imaging 38(11) (2019)
Lee, C.Y., Xie, S., Gallagher, P., et al.: Deeply-supervised nets. In: Artificial Intelligence and Statistics (2015)
Liew, S.L., Anglin, J.M., Banks, N.W., et al.: A large, open source dataset of stroke anatomical brain images and manual lesion segmentations. Scientific Data 5(1) (2018)
Liu, J., Zhang, Y., Chen, J.N., et al.: Clip-driven universal model for organ segmentation and tumor detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
Ma, J., He, Y., Li, F., et al.: Segment anything in medical images. Nature Communications 15(1) (2024)
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9(11) (2008)
Marcus, D.S., Fotenos, A.F., Csernansky, J.G., et al.: Open access series of imaging studies: longitudinal MRI data in nondemented and demented older adults. Journal of Cognitive Neuroscience 22(12) (2010)
Ou, Y., Yuan, Y., Huang, X., et al.: Patcher: Patch transformers with mixture of experts for precise medical image segmentation. In: MICCAI (2022)
Puigcerver, J., Riquelme, C., Mustafa, B., et al.: From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951 (2023)
Rajbhandari, S., Li, C., Yao, Z., et al.: DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In: International Conference on Machine Learning (2022)
Schmidt, G.P., Wintersperger, B., Graser, A., et al.: High-resolution whole-body magnetic resonance imaging applications at 1.5 and 3 Tesla: a comparative study. Investigative Radiology 42(6) (2007)
Shah, A.H., Snelling, B., Bregy, A., et al.: Discriminating radiation necrosis from tumor progression in gliomas: a systematic review what is the best imaging modality? Journal of Neuro-Oncology 112 (2013)
Ulrich, C., Isensee, F., Wald, T., et al.: MultiTalent: A multi-dataset approach to medical image segmentation. In: MICCAI (2023)
Wang, H., Guo, S., Ye, J., et al.: SAM-Med3D. arXiv preprint arXiv:2310.15161 (2023)
Wasserthal, J., Breit, H.C., Meyer, M.T., et al.: TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images. Radiology: Artificial Intelligence 5(5) (2023)
Wood, D.A., Kafiabadi, S., Al Busaidi, A., et al.: Deep learning models for triaging hospital head MRI examinations. Medical Image Analysis 78 (2022)
Wu, O., Christensen, S., Hjort, N., et al.: Characterizing physiological heterogeneity of infarction risk in acute human ischaemic stroke using MRI. Brain 129(9) (2006)
Zhang, S., Metaxas, D.: On the challenges and perspectives of foundation models for medical image analysis. Medical Image Analysis 91 (2024)
Zhang, X., Liu, C., Ou, N., et al.: CarveMix: a simple data augmentation method for brain lesion segmentation. NeuroImage 271 (2023)
Zhang, Y., Cai, R., Chen, T., et al.: Robust mixture-of-expert training for convolutional neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
Zhou, Y., Chia, M.A., Wagner, S.K., et al.: A foundation model for generalizable disease detection from retinal images. Nature 622(7981) (2023)
Zhu, J., Zhu, X., Wang, W., et al.: Uni-perceiver-MoE: Learning sparse generalist models with conditional MoEs. Advances in Neural Information Processing Systems 35 (2022)
Acknowledgements
C. Ye is supported by the Beijing Municipal Natural Science Foundation (7242273) and the Fundamental Research Funds for the Central Universities (2024CX06040). W. Bai is co-funded by EPSRC DeepGeM Grant (EP/W01842X/1) and NIHR Imperial Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors declare no competing interests.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, X. et al. (2024). A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15012. Springer, Cham. https://doi.org/10.1007/978-3-031-72390-2_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-72390-2_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72389-6
Online ISBN: 978-3-031-72390-2
eBook Packages: Computer ScienceComputer Science (R0)
