乐胖代购免代理版

{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T08:31:19Z","timestamp":1725957079789},"reference-count":72,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T00:00:00Z","timestamp":1614816000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["01GQ1343"],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009318","name":"Helmholtz Association","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100009318","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["720270"],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"Reinforcement learning is a paradigm that can account for how organisms learn to adapt their behavior in complex environments with sparse rewards. To partition an environment into discrete states, implementations in spiking neuronal networks typically rely on input architectures involving place cells or receptive fields specified ad hoc<\/jats:italic> by the researcher. This is problematic as a model for how an organism can learn appropriate behavioral sequences in unknown environments, as it fails to account for the unsupervised and self-organized nature of the required representations. Additionally, this approach presupposes knowledge on the part of the researcher on how the environment should be partitioned and represented and scales poorly with the size or complexity of the environment. To address these issues and gain insights into how the brain generates its own task-relevant mappings, we propose a learning architecture that combines unsupervised learning on the input projections with biologically motivated clustered connectivity within the representation layer. This combination allows input features to be mapped to clusters; thus the network self-organizes to produce clearly distinguishable activity patterns that can serve as the basis for reinforcement learning on the output projections. On the basis of the MNIST and Mountain Car tasks, we show that our proposed model performs better than either a comparable unclustered network or a clustered network with static input projections. We conclude that the combination of unsupervised learning and clustered connectivity provides a generic representational substrate suitable for further computation.<\/jats:p>","DOI":"10.3389\/fncom.2021.543872","type":"journal-article","created":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T08:22:12Z","timestamp":1614846132000},"update-policy":"http:\/\/dx.doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Unsupervised Learning and Clustered Connectivity Enhance Reinforcement Learning in Spiking Neural Networks"],"prefix":"10.3389","volume":"15","author":[{"given":"Philipp","family":"Weidel","sequence":"first","affiliation":[]},{"given":"Renato","family":"Duarte","sequence":"additional","affiliation":[]},{"given":"Abigail","family":"Morrison","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2021,3,4]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"59","DOI":"10.3389\/fncir.2020.541728","article-title":"The interplay of synaptic plasticity and scaling enables self-organized formation and allocation of multiple memory representations","volume":"14","author":"Auth","year":"2020","journal-title":"Front. Neural Circ"},{"key":"B2","article-title":"Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets","volume-title":"arXiv preprint arXiv:1901.09049","author":"Bellec","year":"2019"},{"key":"B3","article-title":"OpenAI Gym","author":"Brockman","year":"2016","journal-title":"arXiv preprint arXiv:1606.01540"},{"key":"B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCNN.2013.6706961","article-title":"\u201cBiologically plausible models of homeostasis and STDP: stability and learning in spiking neural networks,\u201d","volume-title":"The 2013 International Joint Conference on Neural Networks (IJCNN)","author":"Carlson","year":"2013"},{"key":"B5","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1038\/nn.2501","article-title":"Stimulus onset quenches neural variability: A widespread cortical phenomenon","volume":"13","author":"Churchland","year":"2010","journal-title":"Nat. Neurosci"},{"key":"B6","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1016\/0165-1684(94)90029-9","article-title":"Independent component analysis, a new concept?","volume":"36","author":"Comon","year":"1994","journal-title":"Signal Process"},{"key":"B7","doi-asserted-by":"publisher","first-page":"99","DOI":"10.3389\/fncom.2015.00099","article-title":"Unsupervised learning of digit recognition using spike-timing-dependent plasticity","volume":"9","author":"Diehl","year":"2015","journal-title":"Front. Comput. Neurosci"},{"key":"B8","doi-asserted-by":"publisher","first-page":"7315","DOI":"10.1523\/JNEUROSCI.0874-15.2015","article-title":"Expansion and state-dependent variability along sensory processing streams","volume":"35","author":"Duarte","year":"2015","journal-title":"J. Neurosci"},{"key":"B9","first-page":"1","article-title":"\u201cEncoding symbolic sequences with spiking neural reservoirs,\u201d","volume-title":"Proceedings of the International Joint Conference on Neural Networks","author":"Duarte","year":"2018"},{"key":"B10","doi-asserted-by":"publisher","first-page":"124","DOI":"10.3389\/fncom.2014.00124","article-title":"Dynamic stability of sequential stimulus representations in adapting neuronal networks","volume":"8","author":"Duarte","year":"2014","journal-title":"Front. Comput. Neurosci"},{"key":"B11","doi-asserted-by":"publisher","first-page":"85","DOI":"10.3389\/fncir.2015.00085","article-title":"Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules","volume":"9","author":"Fr\u00e9maux","year":"2016","journal-title":"Front. Neural Circ"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e1003024","DOI":"10.1371\/journal.pcbi.1003024","article-title":"Reinforcement learning using a continuous time actor-critic framework with spiking neurons","volume":"9","author":"Fr\u00e9maux","year":"2013","journal-title":"PLoS Comput. Biol"},{"key":"B13","doi-asserted-by":"publisher","first-page":"1450002","DOI":"10.1142\/S0129065714500026","article-title":"Code-specific learning rules improve action selection by populations of spiking neurons","volume":"24","author":"Friedrich","year":"2014","journal-title":"Int. J. Neural Syst"},{"key":"B14","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1016\/j.conb.2016.01.010","article-title":"Why neurons mix: High dimensionality for higher cognition","volume":"37","author":"Fusi","year":"2016","journal-title":"Curr. Opin. Neurobiol"},{"key":"B15","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1162\/neco.1989.1.4.502","article-title":"Random neural networks with negative and positive signals and product form solution","volume":"1","author":"Gelenbe","year":"1989","journal-title":"Neural Comput"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1430","DOI":"10.4249\/scholarpedia.1430","article-title":"NEST (NEural Simulation Tool)","volume":"2","author":"Gewaltig","year":"2007","journal-title":"Scholarpedia"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.7554\/eLife.29226","article-title":"Cortical response states for enhanced sensory discrimination","volume":"6","author":"Gutnisky","year":"2017","journal-title":"eLife"},{"key":"B18","first-page":"773","article-title":"Homeostatic plasticity in Bayesian spiking networks as Expectation Maximization with posterior constraints","volume":"1","author":"Habenschuss","year":"2012","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B19","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1093\/cercor\/bhj132","article-title":"A statistical analysis of information-processing properties of lamina-specific cortical microcircuit models","volume":"17","author":"Haeusler","year":"2007","journal-title":"Cereb. Cortex"},{"key":"B20","volume-title":"Introduction to the Theory of Neural Computation","author":"Hertz","year":"1991"},{"key":"B21","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1016\/S0893-6080(00)00026-5","article-title":"Independent component analysis: algorithms and applications","volume":"13","author":"Hyv\u00e4rinen","year":"2000","journal-title":"Neural Netw"},{"key":"B22","volume-title":"The \u201cEcho State\u201d Approach to Analysing and Training Recurrent Neural Networks","author":"Jaeger","year":"2001"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1003","DOI":"10.1016\/j.neuron.2017.02.019","article-title":"Navigating the neural space in search of the neural code","volume":"93","author":"Jazayeri","year":"2017","journal-title":"Neuron"},{"key":"B24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCNN.2012.6252834","article-title":"\u201cLearning from positive and negative rewards in a spiking neural network model of basal ganglia,\u201d","volume-title":"The 2012 International Joint Conference on Neural Networks (IJCNN)","author":"Jitsev","year":"2012"},{"key":"B25","doi-asserted-by":"publisher","first-page":"8511","DOI":"10.1523\/JNEUROSCI.2078-16.2017","article-title":"Feedback inhibition shapes emergent computational properties of cortical microcircuit motifs","volume":"37","author":"Jonke","year":"2017","journal-title":"J. Neurosci"},{"key":"B26","article-title":"Closing the loop between neural network simulators and the OpenAI Gym","volume-title":"arXiv preprint arXiv:1709.05650","author":"Jordan","year":"2017"},{"key":"B27","doi-asserted-by":"publisher","first-page":"A132","DOI":"10.17815\/jlsrf-4-121-1","article-title":"JURECA: Modular supercomputer at J\u00fclich Supercomputing Centre","volume":"4","author":"Krause","year":"2018","journal-title":"Journal of large-scale research facilities"},{"key":"B28","doi-asserted-by":"publisher","first-page":"R231","DOI":"10.1016\/j.cub.2019.02.034","article-title":"Neural network models and deep learning","volume":"29","author":"Kriegeskorte","year":"2019","journal-title":"Curr. Biol"},{"key":"B29","doi-asserted-by":"publisher","first-page":"7723","DOI":"10.1073\/pnas.1820458116","article-title":"Unsupervised learning by competing hidden units","volume":"116","author":"Krotov","year":"2019","journal-title":"Proc. Natl. Acad. Sci. U.S.A"},{"key":"B30","doi-asserted-by":"publisher","first-page":"47","DOI":"10.3389\/fnsys.2018.00047","article-title":"The interplay between cortical state and perceptual learning: a focused review","volume":"12","author":"Kwon","year":"2018","journal-title":"Front. Syst. Neurosci"},{"key":"B31","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B32","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.conb.2019.01.011","article-title":"Backpropagation through time and the brain","volume":"55","author":"Lillicrap","year":"2019","journal-title":"Curr. Opin. Neurobiol"},{"key":"B33","volume-title":"NEST 2.16.10.","author":"Linssen","year":"2018"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1498","DOI":"10.1038\/nn.3220","article-title":"Slow dynamics and high variability in balanced cortical networks with clustered connections","volume":"15","author":"Litwin-Kumar","year":"2012","journal-title":"Nat. Neurosci"},{"key":"B35","doi-asserted-by":"publisher","first-page":"5319","DOI":"10.1038\/ncomms6319","article-title":"Formation and maintenance of neuronal assemblies through synaptic plasticity","volume":"5","author":"Litwin-Kumar","year":"2014","journal-title":"Nat. Commun"},{"key":"B36","doi-asserted-by":"crossref","first-page":"1150","DOI":"10.1109\/ICCV.1999.790410","article-title":"\u201cObject recognition from local scale-invariant features,\u201d","volume-title":"The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999","author":"Lowe","year":"1999"},{"key":"B37","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/j.cosrev.2009.03.005","article-title":"Reservoir computing approaches to recurrent neural network training","volume":"3","author":"Luko\u0161evi\u010dius","year":"2009","journal-title":"Comput. Sci. Rev"},{"key":"B38","first-page":"213","article-title":"\u201cA model for real-time computation in generic neural microcircuits,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Maass","year":"2003"},{"key":"B39","doi-asserted-by":"publisher","first-page":"2531","DOI":"10.1162\/089976602760407955","article-title":"Real-time computing without stable states: a new framework for neural computation based on perturbations","volume":"14","author":"Maass","year":"2002","journal-title":"Neural Comput"},{"key":"B40","doi-asserted-by":"publisher","first-page":"94","DOI":"10.3389\/fncom.2016.00094","article-title":"Towards an integration of deep learning and neuroscience","volume":"10","author":"Marblestone","year":"2016","journal-title":"Front. Comput. Neurosci"},{"key":"B41","article-title":"Deep learning: a critical appraisal","volume-title":"arXiv preprint arXiv:1801.00631","author":"Marcus","year":"2018"},{"key":"B42","doi-asserted-by":"publisher","first-page":"11","DOI":"10.3389\/fnsys.2016.00011","article-title":"Stimuli reduce the dimensionality of cortical activity","volume":"10","author":"Mazzucato","year":"2016","journal-title":"Front. Syst. Neurosci"},{"key":"B43","first-page":"41","article-title":"\u201cFisher discriminant analysis with kernels,\u201d","volume-title":"Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop","author":"Mika","year":"1999"},{"key":"B44","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"B45","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1007\/s11633-017-1093-8","article-title":"Why deep neural nets cannot ever match biological intelligence and what to do about it?","volume":"14","author":"Nikoli\u0107","year":"2017","journal-title":"Int. J. Automat. Comput"},{"key":"B46","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1007\/BF00275687","article-title":"Simplified neuron model as a principal component analyzer","volume":"15","author":"Oja","year":"1982","journal-title":"J. Math. Biol"},{"key":"B47","doi-asserted-by":"publisher","first-page":"5419","DOI":"10.1073\/pnas.1016051108","article-title":"A synaptic organizing principle for cortical neuronal groups","volume":"108","author":"Perin","year":"2011","journal-title":"Proc. Natl. Acad. Sci. U.S.A"},{"key":"B48","doi-asserted-by":"publisher","first-page":"2694","DOI":"10.1162\/neco.2007.19.10.2694","article-title":"Learning with \u201crelevance\u201d: using a third factor to stabilize Hebbian learning","volume":"19","author":"Porr","year":"2007","journal-title":"Neural Comput"},{"key":"B49","doi-asserted-by":"publisher","first-page":"e1001133","DOI":"10.1371\/journal.pcbi.1001133","article-title":"An imperfect dopaminergic error signal can drive temporal-difference learning","volume":"7","author":"Potjans","year":"2011","journal-title":"PLoS Comput. Biol"},{"key":"B50","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1162\/neco.2008.08-07-593","article-title":"A spiking neural network model of an actor-critic learning agent","volume":"21","author":"Potjans","year":"2009","journal-title":"Neural Comput"},{"key":"B51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5402\/2012\/847305","article-title":"Neural network implementations for PCA and its extensions","volume":"2012","author":"Qiu","year":"2012","journal-title":"ISRN Artif. Intell"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1109\/TNANO.2013.2250995","article-title":"\u201cImmunity to device variations in a spiking neural network with memristive nanodevices,\u201d","author":"Querlioz","year":"2013","journal-title":"IEEE"},{"key":"B53","doi-asserted-by":"publisher","first-page":"1005","DOI":"10.1016\/j.neuron.2018.05.020","article-title":"Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics","volume":"98","author":"Remington","year":"2018","journal-title":"Neuron"},{"key":"B54","doi-asserted-by":"publisher","first-page":"1761","DOI":"10.1038\/s41593-019-0520-2","article-title":"A deep learning framework for neuroscience","volume":"22","author":"Richards","year":"2019","journal-title":"Nat. Neurosci"},{"key":"B55","doi-asserted-by":"publisher","first-page":"24","DOI":"10.3389\/fncom.2010.00024","article-title":"Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses","volume":"4","author":"Rigotti","year":"2010","journal-title":"Front. Comput. Neurosci"},{"key":"B56","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1007\/s00422-017-0737-7","article-title":"Winnerless competition in clustered balanced networks: inhibitory assemblies do the trick","volume":"112","author":"Rost","year":"2018","journal-title":"Biol. Cybernet"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.1101\/2020.02.27.968339","article-title":"Spiking neural network model of motor cortex with joint excitatory and inhibitory clusters reflects task uncertainty, reaction times, and variability dynamics","author":"Rostami","year":"2020","journal-title":"bioRxiv [Preprint]"},{"key":"B58","first-page":"8735","article-title":"\u201cDendritic cortical microcircuits approximate the backpropagation algorithm,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Sacramento","year":"2018"},{"key":"B59","first-page":"471","article-title":"\u201cAn overview of reservoir computing: theory, applications and implementations,\u201d","volume-title":"Proceedings of the 15th European Symposium on Artificial Neural Networks","author":"Schrauwen","year":"2007"},{"key":"B60","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1038\/nrn.2015.26","article-title":"Dopamine reward prediction-error signalling: a two-component response","volume":"17","author":"Schultz","year":"2016","journal-title":"Nat. Rev. Neurosci"},{"key":"B61","doi-asserted-by":"publisher","first-page":"e68","DOI":"10.1371\/journal.pbio.0030068","article-title":"Highly nonrandom features of synaptic connectivity in local cortical circuits","volume":"3","author":"Song","year":"2005","journal-title":"PLoS Biol"},{"key":"B62","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"B63","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1016\/j.conb.2016.01.014","article-title":"Toward the neural implementation of structure learning","volume":"37","author":"Tervo","year":"2016","journal-title":"Curr. Opin. Neurobiol"},{"key":"B64","doi-asserted-by":"publisher","first-page":"e1003307","DOI":"10.1371\/journal.pcbi.1003307","article-title":"Synaptic scaling enables dynamically distinct short- and long-term memory formation","volume":"9","author":"Tetzlaff","year":"2013","journal-title":"PLoS Comput. Biol"},{"key":"B65","doi-asserted-by":"publisher","first-page":"47","DOI":"10.3389\/fncom.2011.00047","article-title":"Synaptic scaling in combination with many generic plasticity mechanisms stabilizes circuit connectivity","volume":"5","author":"Tetzlaff","year":"2011","journal-title":"Front. Comput. Neurosci"},{"key":"B66","doi-asserted-by":"publisher","first-page":"36","DOI":"10.3389\/fncom.2012.00036","article-title":"Analysis of synaptic scaling in combination with Hebbian plasticity in several simple networks","volume":"6","author":"Tetzlaff","year":"2012","journal-title":"Front. Comput. Neurosci"},{"key":"B67","doi-asserted-by":"crossref","DOI":"10.1109\/IJCNN.2010.5596492","article-title":"\u201cMemory versus non-linearity in reservoirs,\u201d","volume-title":"Proceedings of the International Joint Conference on Neural Networks","author":"Verstraeten","year":"2010"},{"key":"B68","doi-asserted-by":"publisher","first-page":"31","DOI":"10.3389\/fninf.2016.00031","article-title":"Closed loop interactions between spiking neural network and robotic simulators based on MUSIC and ROS","volume":"10","author":"Weidel","year":"2016","journal-title":"Front. Neuroinform"},{"key":"B69","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1016\/j.tics.2018.12.005","article-title":"Theories of error back-propagation in the brain","volume":"23","author":"Whittington","year":"2019","journal-title":"Trends Cogn. Sci"},{"key":"B70","doi-asserted-by":"publisher","first-page":"138","DOI":"10.3389\/fncom.2015.00138","article-title":"Models of metaplasticity: a review of concepts","volume":"9","author":"Yger","year":"2015","journal-title":"Front. Comput. Neurosci"},{"key":"B71","doi-asserted-by":"publisher","first-page":"79","DOI":"10.3389\/fncom.2019.00079","article-title":"Passing the message: representation transfer in modular balanced networks","volume":"13","author":"Zajzon","year":"2019","journal-title":"Front. Comput. Neurosci"},{"key":"B72","doi-asserted-by":"publisher","first-page":"6922","DOI":"10.1038\/ncomms7922","article-title":"Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks","volume":"6","author":"Zenke","year":"2015","journal-title":"Nat. Commun"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2021.543872\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,4]],"date-time":"2021-03-04T08:22:36Z","timestamp":1614846156000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2021.543872\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,4]]},"references-count":72,"alternative-id":["10.3389\/fncom.2021.543872"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2021.543872","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.03.17.995563","asserted-by":"object"}]},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,4]]}}}