Abstract
The brain is capable of integrating information in multiple sensory channels in a Bayesian optimal way. Based on a decentralized network model inspired by electrophysiological recordings, we consider the structural pre-requisites for optimal multisensory integration. In this architecture, same-channel feedforward and recurrent links encode the unisensory likelihoods, whereas reciprocal couplings connecting the different modules are shaped by the correlation in the joint prior probabilities. Moreover, the statistical relationship between the difference in the optimal network structures and the difference in the priors and the likelihoods clearly shows that the network can encode multisensory information in a distributed manner. Our results generate testable predictions for future experiments and are likely to be applicable to other artificial systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alais, D., Burr, D.: No direction-specific bimodal facilitation for audiovisual motion detection. Cogn. Brain Res. 19(2), 185–194 (2004)
Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)
Gu, Y., Angelaki, D.E., DeAngelis, G.C.: Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11(10), 1201–1210 (2008)
Chen, A., DeAngelis, G.C., Angelaki, D.E.: Macaque parieto-insular vestibular cortex: responses to self-motion and optic flow. J. Neurosci. 30(8), 3022–3042 (2010)
Chen, A., DeAngelis, G.C., Angelaki, D.E.: Functional specializations of the ventral intraparietal area for multisensory heading discrimination. J. Neurosci. 33(8), 3567–3581 (2013)
Chen, A., Gu, Y., Liu, S., DeAngelis, G.C., Angelaki, D.E.: Evidence for a causal contribution of macaque vestibular, but not intraparietal, cortex to heading perception. J. Neurosci. 36(13), 3789–3798 (2016)
Zhang, W.H., Chen, A., Rasch, M.J., Wu, S.: Decentralized multisensory information integration in neural systems. J. Neurosci. 36(2), 532–547 (2016)
Zhang, W.H., Wang, H., Wong, K.Y.M., Wu, S.: “Congruent” and “opposite” neurons: sisters for multisensory integration and segregation. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 3180–3188. Curran Associates, Inc. (2016)
Wang, H., Zhang, W.-H., Wong, K.Y.M., Wu, S.: How the prior information shapes neural networks for optimal multisensory integration. In: Cong, F., Leung, A., Wei, Q. (eds.) ISNN 2017. LNCS, vol. 10262, pp. 128–136. Springer, Cham (2017). doi:10.1007/978-3-319-59081-3_16
Chen, A., Gu, Y., Liu, S., DeAngelis, G.C., Angelaki, D.E.: Evidence for a causal contribution of macaque vestibular, but not intraparietal, cortex to heading perception. In: The 16th Japan-China-Korea Joint Workshop on Neurobiology and Neuroinformatics (NBNI 2016), Hong Kong (2016)
Murray, R.F., Morgenstern, Y.: Cue combination on the circle and the sphere. J. Vis. 10(11), 15 (2010)
Girshick, A.R., Landy, M.S., Simoncelli, E.P.: Cardinal rules: visual orientation perception reflects knowledge of environmental statistics. Nat. Neurosci. 14(7), 926–932 (2011)
Ghazanfar, A.A., Schroeder, C.E.: Is neocortex essentially multisensory? Trends Cogn. Sci. 10(6), 278–285 (2006)
Sato, Y., Toyoizumi, T., Aihara, K.: Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural Comput. 19(12), 3335–3355 (2007)
Shams, L., Ma, W.J., Beierholm, U.: Sound-induced flash illusion as an optimal percept. NeuroReport 16(17), 1923–1927 (2005)
Shams, L., Beierholm, U.R.: Causal inference in perception. Trends Cogn. Sci. 14(9), 425–432 (2010)
Ma, W.J., Beck, J.M., Latham, P.E., Pouget, A.: Bayesian inference with probabilistic population codes. Nat. Neurosci. 9(11), 1432–1438 (2006)
Fung, C.C.A., Wong, K.Y.M., Wu, S.: A moving bump in a continuous manifold: a comprehensive study of the tracking dynamics of continuous attractor neural networks. Neural Comput. 22(3), 752–792 (2010)
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of the 30th International Conference on Machine Learning, vol. 28 (2013)
Acknowledgments
This work is supported by the Research Grants Council of Hong Kong (N\(\_\)HKUST606/12, 605813 and 16322616) and National Basic Research Program of China (2014CB846101) and the Natural Science Foundation of China (31261160495).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Wang, H., Zhang, WH., Wong, K.Y.M., Wu, S. (2017). Encoding Multisensory Information in Modular Neural Networks. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10637. Springer, Cham. https://doi.org/10.1007/978-3-319-70093-9_70
Download citation
DOI: https://doi.org/10.1007/978-3-319-70093-9_70
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70092-2
Online ISBN: 978-3-319-70093-9
eBook Packages: Computer ScienceComputer Science (R0)