Abstract
Representation learning has become increasingly important, especially as powerful models have shifted towards learning latent representations before fine-tuning for downstream tasks. This approach is particularly valuable in leveraging the structural information within brain anatomy. However, a common limitation of recent models developed for MRIs is their tendency to ignore or remove geometric information, such as translation and rotation, thereby creating invariance with respect to geometric operations. We contend that incorporating knowledge about these geometric transformations into the model can significantly enhance its ability to learn more detailed anatomical information within brain structures. As a result, we propose a novel method for encoding 3D MRIs that enforces equivariance with respect to all rotations in 3D space, in other words, SO(3)-equivariance (SOE). By explicitly modeling this geometric equivariance in the representation space, we ensure that any rotational operation applied to the input image space is also reflected in the embedding representation space. This approach requires moving beyond traditional representation learning methods, as we need a representation vector space that allows for the application of the same SO(3) operation in that space. To facilitate this, we leverage the concept of vector neurons. The representation space formed by our method, SOE, captures the brain’s structural and anatomical information more effectively. We evaluate SOE pretrained on the structural MRIs of two public data sets with respect to the downstream task of predicting age and diagnosing Alzheimer’s Disease from T1-weighted brain scans of the ADNI data set. We demonstrate that our approach not only outperforms other methods but is also robust against various degrees of rotation along different axes. The code is available at https://github.com/shizhehe/SOE-representation-learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brown, S.A., et al.: The national consortium on alcohol and neurodevelopment in adolescence (NCANDA): a multisite study of adolescent development and substance use. J. Stud. Alcohol Drugs 76(6), 895–908 (2015)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations (2020)
Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., Guibas, L.: Vector neurons: a general framework for so(3)-equivariant networks. arXiv preprint arXiv:2104.12229 (2021)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. CoRR abs/2010.11929 (2020). https://arxiv.org/abs/2010.11929
Esfahani, E.E., Hosseini, A.: Compressed MRI reconstruction exploiting a rotation-invariant total variation discretization. Magn. Reson. Imaging 71, 80–92 (2020)
Esteves, C., Allen-Blanchette, C., Makadia, A., Daniilidis, K.: Learning so(3) equivariant representations with spherical CNNs. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Grill, J.B., et al.: Bootstrap your own latent: a new approach to self-supervised learning (2020)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners (2021)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.B.: Momentum contrast for unsupervised visual representation learning. CoRR abs/1911.05722 (2019). http://arxiv.org/abs/1911.05722
Huang, S.C., Pareek, A., Jensen, M., Lungren, M.P., Yeung, S., Chaudhari, A.S.: Self-supervised learning for medical image classification: a systematic review and implementation guidelines. NPJ Digit. Med. 6(1), 74 (2023)
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial examples are not bugs, they are features (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes (2022)
Kwon, S., Choi, J.Y., Ryu, E.K.: Rotation and translation invariant representation learning with implicit neural representations. arXiv preprint arXiv:2304.13995 (2023)
Lang, D.M., Schwartz, E., Bercea, C.I., Giryes, R., Schnabel, J.A.: 3D masked autoencoders with application to anomaly detection in non-contrast enhanced breast MRI. arXiv preprint arXiv:2303.05861 (2023)
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. CoRR abs/2103.14030 (2021). https://arxiv.org/abs/2103.14030
Mueller, S.G., et al.: The Alzheimer’s disease neuroimaging initiative. Neuroimaging Clin. N. Am. 15(4), 869–877 (2005). https://doi.org/10.1016/j.nic.2005.09.008, https://www.sciencedirect.com/science/article/pii/S1052514905001024, alzheimer’s Disease: 100 Years of Progress
van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding (2019)
Ouyang, J., Zhao, Q., Adeli, E., Zaharchuk, G., Pohl, K.M.: Self-supervised learning of neighborhood embedding for longitudinal mri. Med. Image Anal. 82, 102571 (2022). https://doi.org/10.1016/j.media.2022.102571, https://www.sciencedirect.com/science/article/pii/S1361841522002122
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
Pohl, K.M., et al.: The NCANDA_PUBLIC_6Y_REDCAP_V01 data release of the national consortium on alcohol and neurodevelopment in adolescence (NCANDA) (2021). https://dx.doi.org/10.7303/syn25606546
Sriram, A., Gaidon, A., Wu, J., Niebles, J.C., Fei-Fei, L., Adeli, E.: Home: homography-equivariant video representation learning (2023)
Weiler, M., Hamprecht, F.A., Storath, M.: Learning steerable filters for rotation equivariant CNNs. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 849–858 (2018). https://doi.org/10.1109/CVPR.2018.00095
Acknowledgements
This work was supported by the HAI-Google Research Award, U.S. National Institute (AG073104), the 2024 HAI Hoffman-Yee Grant, and the HAI-Google Cloud Credits Award.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
He, S., Paschali, M., Ouyang, J., Masood, A., Chaudhari, A., Adeli, E. (2025). SOE: SO(3)-Equivariant 3D MRI Encoding. In: Bathula, D.R., et al. Machine Learning in Clinical Neuroimaging. MLCN 2024. Lecture Notes in Computer Science, vol 15266. Springer, Cham. https://doi.org/10.1007/978-3-031-78761-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-78761-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78760-7
Online ISBN: 978-3-031-78761-4
eBook Packages: Computer ScienceComputer Science (R0)
