Abstract
Neuroscience faces challenges in reliability due to limited statistical power, reproducibility issues, and inconsistent terminology. To address these challenges, we introduce NeuroConText, the first brain meta-analysis model that uses a contrastive approach to enhance the association between text data and brain activation coordinates reported in 20K neuroscientific articles from PubMed Central. NeuroConText integrates the capabilities of recent large language models (LLMs) rather than traditional bag-of-words methods, to better capture the text semantic, and improve the association with brain activation. It is adapted to processing neuroscientific text regardless of length and generalizes well across various textual content-titles, abstracts, and full-body. Our experiments show NeuroConText significantly outperforms state-of-the-art methods with a threefold increase in linking text to brain activations in terms of recall@10. NeuroConText also allows decoding brain images from latent text representations, successfully maintaining the quality of brain image reconstruction compared to the state-of-the-art.
R. Meudec, F. Ghayem—Equally contributed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Black, S., Leo, G., Wang, P., Leahy, C., Biderman, S.: Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. Zenodo 1.0 (2021)
Button, K.S., Ioannidis, J.P., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S., Munafò, M.R.: Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience 14(5), 365–376 (2013)
Carp, J.: The secret lives of experiments: methods reporting in the fmri literature. Neuroimage 63(1), 289–300 (2012)
Dadi, K., Varoquaux, G., Machlouzarides-Shalit, A., Gorgolewski, K.J., Wassermann, D., Thirion, B., Mensch, A.: Fine-grain atlases of functional modes for fmri analysis. NeuroImage 221, 117126 (2020)
Dice, L.R.: Measures of the Amount of Ecologic Association Between Species. Ecology 26(3), 297–302 (1945). https://doi.org/10.2307/1932409, https://www.jstor.org/stable/1932409
Dockès, J., Poldrack, R.A., Primet, R., Gözükan, H., Yarkoni, T., Suchanek, F., Thirion, B., Varoquaux, G.: Neuroquery, comprehensive meta-analysis of human brain mapping. Elife 9, e53385 (2020)
Fox, P.T., Parsons, L.M., Lancaster, J.L.: Beyond the single study: function/location metanalysis in cognitive neuroimaging. Current opinion in neurobiology 8(2), 178–187 (1998)
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al.: The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)
Horn, J.D.V., Grafton, S.T., Rockmore, D., Gazzaniga, M.S.: Sharing neuroimaging studies of human cognition. Nature neuroscience 7(5), 473–481 (2004)
Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Laird, A.R., Lancaster, J.J., Fox, P.T.: Brainmap: the social evolution of a human brain mapping database. Neuroinformatics 3, 65–77 (2005)
Menuet, R., Meudec, R., Dockès, J., Varoquaux, G., Thirion, B.: Comprehensive decoding mental processes from web repositories of functional brain images. Sci. Rep. 12(1), 7050 (Apr 2022)
Minzenberg, M.J., Laird, A.R., Thelen, S., Carter, C.S., Glahn, D.C.: Meta-analysis of 41 functional neuroimaging studies of executive function in schizophrenia. Archives of general psychiatry 66(8), 811–822 (2009)
Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.: Predicting human brain activity associated with the meanings of nouns. science 320(5880), 1191–1195 (2008)
Ngo, G.H., Nguyen, M., Chen, N.F., Sabuncu, M.R.: Text2brain: Synthesis of brain activation maps from free-form text query. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VII 24. pp. 605–614. Springer (2021)
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Pinho, A.L., Amadon, A., Ruest, T., Fabre, M., Dohmatob, E., Denghien, I., Ginisty, C., Becuwe-Desmidt, S., Roger, S., Laurier, L., Joly-Testault, V., Médiouni-Cloarec, G., Doublé, C., Martins, B., Pinel, P., Eger, E., Varoquaux, G., Pallier, C., Dehaene, S., Hertz-Pannier, L., Thirion, B.: Individual Brain Charting, a high-resolution fMRI dataset for cognitive mapping. Scientific Data 5, 180105 (Jun 2018).https://doi.org/10.1038/sdata.2018.105
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Information processing & management 24(5), 513–523 (1988)
Thomas, A.W., Ré, C., Poldrack, R.A.: Self-supervised learning of brain dynamics from broad neuroimaging data. arXiv preprint arXiv:2206.11417 (2022)
Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. pp. 776–794. Springer (2020)
Yarkoni, T., Poldrack, R.A., Nichols, T.E., Van Essen, D.C., Wager, T.D.: Large-scale automated synthesis of human functional neuroimaging data. Nature methods 8(8), 665–670 (2011)
Acknowledgement
This work is supported by the KARAIB AI chair (ANR-20-CHIA-0025-01), the ANR-22-PESN-0012 France 2030 program, and the HORIZON-INFRA-2022-SERV-B-01 EBRAINS 2.0 infrastructure project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Meudec, R., Ghayem, F., Dockès, J., Wassermann, D., Thirion, B. (2024). NeuroConText: Contrastive Text-to-Brain Mapping for Neuroscientific Literature. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15003. Springer, Cham. https://doi.org/10.1007/978-3-031-72384-1_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-72384-1_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72383-4
Online ISBN: 978-3-031-72384-1
eBook Packages: Computer ScienceComputer Science (R0)
