Abstract
We subseribe to the idea that the brain employs generative networks. In turn, we conclude that channel capacity constraints form the main obstacle for effective information transfer in the brain. Robust and fast information flow processing methods warranting efficient information transfer, e.g. grouping of inputs and information maximization principles need to be applied. For this reason, indepent component analyses on groups of patterns were conducted using (a) model labyrinth, (b) movies on highway traffic and (c) mixed acoustical signals. We found that in all cases ‘familiar’ inputs give rise to cumulated firing histograms close to exponential distributions, whereas ‘novel’ information are better deseribed by broad, sometimes truncated Gaussian distributions. It can be shown that upon minimization of mutual information between processing channels, noise can reveal itself locally. Therefore, we conjecture that novelty - as opposed to noise - can be recognized by means of the statistics of neuronal firing in brain areas.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Attneave, F. (1954). Some informational aspects of visual perception. Psychological Review, 61, 183–193.
Baddeley, R. J., Abbott, L. F., Booth, M., Sengpiel, F., Freeman, T., Wakeman, E. A. & Rolls, E. T. (1997). Responses of neurons in primary and inferior temporal visual cortices to natural scenes. Proc. Roy. Soc. London B 264, 1775–1783.
Barlow, H. B. (1961). In: Sensory communication. Rosenblith, W. A. (ed.). Cambridge, MA: MIT Press.
Bell, A. J. & Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7, 1129–1159.
Bell, A. J. & Sejnowski, T. J. (1997). The ‘independent components’ of natural scenes are edge filters. Vision Research 37, 3327–3389.
Charles, D. & Fyfe, C. (1998). Modelling multiple cause structure using rectification constraints. Neural Systems, 9,167–182.
Chrobak, J. J., Lörincz, A. & Buzsaki G. (2000). Physiological patterns in the hipocampus-entorhinal cortex system. Hippocampus, 10,457–465.
Cover, T. & Thomas, J. (1991). Elements of information theory. New York, USA: John Wiley and Sons.
Dong, D. W. & Atick, J. J. (1995). Temporal decorrelation: a theory of lagged and nonlagged responses in the lateral geniculate nucleus. Network, 6, 159–178
Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Otical Society of America, A 4, 2379–2394.
Hatteren, J. H. & Ruderman, D. L. (1998). Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proc. Roy. Soc. London, B265, 2315–2320.
Haykin, S. (1999). Neural Networks: A comprehensive foundation. New Jersey, USA: Prentice Hall.
Henze, D. A., Cameron, W.E. & Barrionuevo, G. (1996). Dendritic morphology and its effects on the amplitude and rise-time of synaptic signals in hippocampal CA3 pyramidal cells. J. Comp. Neurology, 369, 331–344.
Hinton, G. E. & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society B, 352, 1177–1190.
Hinton, G. E. & Zemel, R. S. (1994). Autoencoders, minimum description length, and Heimholtz free energy. In: J. D. Cowan, G. Tesauro, & J. Alspector (ed), Advances in neural information processing systems, 6, 3–10. San Mateo, CA: Morgan Kaufmann.
Horn, B.K.P. (1977). Understanding image intensities. Artifical Intelligence, 8, 201–231.
Hvärinen, A., Hoyer, P., & Oja, E. (1999) Sparse code shrinkage: Denoising by maximum likelihood extimation. In: Kearns, M., Solla, S. A., & Cohn, D. (ed), Advances in Neural Information Processing Systems, 12, MIT Press, Cambridge MA.
Hvärinen, A. & Oja, E. (1997). A fast fixed-point algorithm for independent component analysis. Neural Computation, 9, 1483–1492.
Hvärinen, A. & Oja, E. (2000). Independent component analysis: Aigorithms and applications. Neural Networks, 13(4–5):411430.
Jaffe, D. B & Carnevale, N. T. (1999). Passive normalization of synaptic integration influenced by dendritic architecture. Journal of Neurophysiolog, 82, 3268–3285.
Jutten, C. & Herault, J. (1991). Blind separation of sources, Part I: An adaptive algorithm based on neuromimetic architecture. Signal Processing, 24, 1–10.
Karhunen, J. Oja, E. Wang, L. Vigario, R. & Joutsensalo, J. (1997). A class of neural networks for independent component analysis. IEEE Transactions on Neural Networks, 8, 486–504.
Koch, C. & Poggio, T. (1999). Predicting the visual world: Silence is golden. Nature Neuroscience, 2, 9–10.
Lisman, J. E. (1999). Relating hippocampal circuitry to function: Recall of memory sequences by reciprocal dentate-CA3 interactions. Neuron, 22, 233–242.
Lisman, J. E. & Idiart, M. A. P. (1995). A mechanism for storing 7+/-2 short-term memories in oscillatory subcycles. Science, 267, 1512–1514.
Lörincz, A. (1997). Towards a unified model of cortical computation II: From contral architecture to a model of consciousness. Neural Network World, 7, 137–152.
Lörincz, A. (1998). Forming independent components via temporal locking of reconstruction architectures: A functional model of the hippocampus. Biological Cybernetics, 79,263–275.
Lörincz, A. & Buzsáki, G. (1999). Computational model of the entorhinal-hippocampal region derived from a single principle. In: Proceedings of IJCNN, Washington, July 9–16, JCNN2136.PDF IEEE Catalog Number: 99CH36339C, ISBN: 0–7803–5532–6
Lörincz, A.&Buzsáki, Gy. (2000). Two-phase computational model training long-term memories in the entorhinal-hippocampal region. In: Scharfman, H. E., Witter, M. P., Schwarcz, R. (ed), The parahippocampal region: Implications for neurological and psychiatrie diseases, 911, New York Academy of Sciences, 83–111.
Olshausen, B. A. & Field, D. J. (1996). Emergence of simple-cell receptive field properties by leaming a sparse code for natural images. Nature, 381, 607–609.
Parra, L., Deco, G. & Miesbach, S. (1995). Statistical independence and novelty detection with information preserving nonlinear maps. Neural Computation, 8, 260–269.
Rao, R. P. N. & Ballard, D.H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2, 79–87.
Rolls, E. T. & Treves, A. (1998). Neural networks and brain function. Oxford, UK: Oxford University Press.
Searle, J. R. (1992). The rediscovery of mind. Cambridge, MA: Bradford Books, MIT Press.
Shannon, C. E. (1948). A mathematical theory of communication. AT&T Bell Labs. Tech. J., 27,397–423.
Thorpe, S. Fize, D. & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381, 520–522.
Treves, A., Panzeri, S., Rolls, E.T., Booth, M. & Wakeman, E. A. (1999). Firing rate distributions and efficiency of information transmission of inferior temporal cortex neurons to natural visual stimuli. Neural Computation, 11, 601–631.
Wimbauer, S., Wenish, O. G., Miller, K. D. & van Hemmen, J. L. (1997). Development of spatio-temporal receptive fields of simple cells: I. Model formulation. Biological Cybernetics, 77, 456–461.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag London
About this paper
Cite this paper
Lőrincz, A., Szatmáry, B., Szirtes, G., Takács, B. (2001). Recognition of Novelty Made Easy: Constraints of Channel Capacity on Generative Networks. In: French, R.M., Sougné, J.P. (eds) Connectionist Models of Learning, Development and Evolution. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-0281-6_8
Download citation
DOI: https://doi.org/10.1007/978-1-4471-0281-6_8
Publisher Name: Springer, London
Print ISBN: 978-1-85233-354-6
Online ISBN: 978-1-4471-0281-6
eBook Packages: Springer Book Archive