{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T05:56:20Z","timestamp":1719294980662},"reference-count":52,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2019,7,8]],"date-time":"2019-07-08T00:00:00Z","timestamp":1562544000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis<\/jats:italic> is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined.<\/jats:p>","DOI":"10.7717\/peerj-cs.205","type":"journal-article","created":{"date-parts":[[2019,7,8]],"date-time":"2019-07-08T03:49:30Z","timestamp":1562557770000},"page":"e205","source":"Crossref","is-referenced-by-count":3,"title":["Sample-level sound synthesis with recurrent neural networks and conceptors"],"prefix":"10.7717","volume":"5","author":[{"given":"Chris","family":"Kiefer","sequence":"first","affiliation":[{"name":"Experimental Music Technologies Lab, Department of Music, University of Sussex, Brighton, United Kingdom"}]}],"member":"4443","published-online":{"date-parts":[[2019,7,8]]},"reference":[{"key":"10.7717\/peerj-cs.205\/ref-1","article-title":"Synthesizing audio with generative adversarial networks","author":"Donahue","year":"2018"},{"key":"10.7717\/peerj-cs.205\/ref-2","doi-asserted-by":"publisher","first-page":"22381","DOI":"10.1038\/srep22381","article-title":"Fully analogue photonic reservoir computer","volume":"6","author":"Duport","year":"2016","journal-title":"Scientific Reports"},{"issue":"3","key":"10.7717\/peerj-cs.205\/ref-3","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1162\/ARTL_a_00169","article-title":"You pretty little flocker: exploring the aesthetic state space of creative ecosystems","volume":"21","author":"Eldridge","year":"2015","journal-title":"Artificial Life"},{"key":"10.7717\/peerj-cs.205\/ref-4","article-title":"Gansynth: adversarial neural audio synthesis","author":"Engel","year":"2019"},{"key":"10.7717\/peerj-cs.205\/ref-5","article-title":"Neural audio synthesis of musical notes with WaveNet autoencoders","author":"Engel","year":"2017"},{"key":"10.7717\/peerj-cs.205\/ref-6","first-page":"588","article-title":"Pattern recognition in a bucket","author":"Fernando","year":"2003"},{"key":"10.7717\/peerj-cs.205\/ref-7","article-title":"Real-time human interaction with supervised learning algorithms for music composition and performance","author":"Fiebrink","year":"2011","journal-title":"PhD thesis"},{"issue":"4044","key":"10.7717\/peerj-cs.205\/ref-8","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1038\/159591a0","article-title":"Acoustical quanta and the theory of hearing","volume":"159","author":"Gabor","year":"1947","journal-title":"Nature"},{"key":"10.7717\/peerj-cs.205\/ref-9","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.neucom.2017.11.073","article-title":"Local lyapunov exponents of deep echo state networks","volume":"298","author":"Gallicchio","year":"2018","journal-title":"Neurocomputing"},{"key":"10.7717\/peerj-cs.205\/ref-10","first-page":"131052","article-title":"Encoding and decoding dynamic sensory signals with recurrent neural networks: an application of conceptors to birdsongs","author":"Gast","year":"2017","journal-title":"bioRxiv"},{"key":"10.7717\/peerj-cs.205\/ref-11","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/978-981-287-618-8_18","article-title":"Creating music and texts with flow machines","volume-title":"Multidisciplinary contributions to the science of creative thinking","author":"Ghedini","year":"2016"},{"key":"10.7717\/peerj-cs.205\/ref-12","first-page":"339","article-title":"Learning features from music audio with deep belief networks","author":"Hamel","year":"2010"},{"key":"10.7717\/peerj-cs.205\/ref-13","first-page":"126","article-title":"The microbial genetic algorithm","author":"Harvey","year":"2009"},{"issue":"23","key":"10.7717\/peerj-cs.205\/ref-14","first-page":"244","article-title":"Echo state networks with filter neurons and a delay and sum readout","volume":"2","author":"Holzmann","year":"2009","journal-title":"Neural Networks"},{"key":"10.7717\/peerj-cs.205\/ref-15","article-title":"Reservoir computing: a powerful black-box framework for nonlinear audio processing","author":"Holzmann","year":"2009"},{"key":"10.7717\/peerj-cs.205\/ref-16","first-page":"108","article-title":"Exploring continuous time recurrent neural networks through novelty search","author":"Ianigro","year":"2018"},{"key":"10.7717\/peerj-cs.205\/ref-17","article-title":"Short term memory in echo state networks","author":"Jaeger","year":"2002","journal-title":"Technical report"},{"key":"10.7717\/peerj-cs.205\/ref-18","article-title":"A tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the \u201cecho state network\u201d approach","author":"Jaeger","year":"2005","journal-title":"Technical report"},{"key":"10.7717\/peerj-cs.205\/ref-19","article-title":"The \u201cecho state\u201d approach to analysing and training recurrent neural networks-with an erratum note","author":"Jaeger","year":"2010","journal-title":"Technical Report 148"},{"key":"10.7717\/peerj-cs.205\/ref-20","article-title":"Conceptors: an easy introduction","author":"Jaeger","year":"2014"},{"key":"10.7717\/peerj-cs.205\/ref-21","article-title":"Controlling recurrent neural networks by conceptors","author":"Jaeger","year":"2014"},{"issue":"13","key":"10.7717\/peerj-cs.205\/ref-22","first-page":"1","article-title":"Using conceptors to manage neural long-term memories for temporal patterns","volume":"18","author":"Jaeger","year":"2017","journal-title":"Journal of Machine Learning Research"},{"key":"10.7717\/peerj-cs.205\/ref-23","first-page":"310","article-title":"Can\u2019t get you out of my head: a connectionist model of cyclic rehearsal","author":"Jaeger","year":"2006"},{"key":"10.7717\/peerj-cs.205\/ref-24","article-title":"Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control","author":"Jaques","year":"2016"},{"issue":"4","key":"10.7717\/peerj-cs.205\/ref-25","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1109\/TASL.2008.2012314","article-title":"Quantitative analysis of a common audio similarity measure","volume":"17","author":"Jensen","year":"2009","journal-title":"IEEE Transactions on Audio, Speech, and Language Processing"},{"key":"10.7717\/peerj-cs.205\/ref-26","doi-asserted-by":"crossref","DOI":"10.1109\/ALIFE.2007.367795","article-title":"Is there a liquid state machine in the bacterium Escherichia Coli?","author":"Jones","year":"2007"},{"issue":"3","key":"10.7717\/peerj-cs.205\/ref-27","doi-asserted-by":"publisher","first-page":"315","DOI":"10.1007\/s12559-017-9457-5","article-title":"Real-time audio processing with a cascade of discrete-time delay line-based reservoir computers","volume":"9","author":"Keuninckx","year":"2017","journal-title":"Cognitive Computation"},{"key":"10.7717\/peerj-cs.205\/ref-28","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ins.2013.04.014","article-title":"Very short time environmental sound classification based on spectrogram pattern matching","volume":"243","author":"Khunarsal","year":"2013","journal-title":"Information Sciences"},{"key":"10.7717\/peerj-cs.205\/ref-29","article-title":"Musical instrument mapping design with echo state networks","author":"Kiefer","year":"2014"},{"key":"10.7717\/peerj-cs.205\/ref-30","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1007\/978-3-642-35289-8_36","article-title":"A practical guide to applying echo state networks","volume-title":"Neural networks: tricks of the trade","author":"Luko\u0161evi\u010dius","year":"2012"},{"issue":"11","key":"10.7717\/peerj-cs.205\/ref-31","doi-asserted-by":"crossref","first-page":"2531","DOI":"10.1162\/089976602760407955","article-title":"Real-time computing without stable states: a new framework for neural computation based on perturbations","volume":"14","author":"Maass","year":"2002","journal-title":"Neural Computation"},{"key":"10.7717\/peerj-cs.205\/ref-32","doi-asserted-by":"publisher","DOI":"10.25377\/sussex.7764845.v1","article-title":"Dataset of sounds used with the Ixi Lang live coding environment","author":"Magnusson","year":"2019"},{"key":"10.7717\/peerj-cs.205\/ref-33","first-page":"40","article-title":"Echo state networks and self-prediction","author":"Mayer","year":"2004","journal-title":"Springer"},{"key":"10.7717\/peerj-cs.205\/ref-34","article-title":"Generative algorithms for making music: emergence, evolution, and ecosystems","volume-title":"The Oxford handbook of computer music","author":"McCormack","year":"2011"},{"key":"10.7717\/peerj-cs.205\/ref-35","article-title":"SampleRNN: an unconditional end-to-end neural audio generation model","author":"Mehri","year":"2016"},{"key":"10.7717\/peerj-cs.205\/ref-36","article-title":"Nonlinear dynamics in musical interactions","author":"Mudd","year":"2017","journal-title":"PhD thesis"},{"key":"10.7717\/peerj-cs.205\/ref-37","article-title":"Wavenet: a generative model for raw audio","author":"Oord","year":"2016"},{"key":"10.7717\/peerj-cs.205\/ref-38","first-page":"7","article-title":"On the evaluation of perceptual similarity measures for music","author":"Pampalk","year":"2003"},{"issue":"2","key":"10.7717\/peerj-cs.205\/ref-39","doi-asserted-by":"crossref","first-page":"61","DOI":"10.2307\/3680222","article-title":"Automated granular synthesis of sound","volume":"2","author":"Roads","year":"1978","journal-title":"Computer Music Journal"},{"key":"10.7717\/peerj-cs.205\/ref-40","volume-title":"Microsound","author":"Roads","year":"2004"},{"issue":"2","key":"10.7717\/peerj-cs.205\/ref-41","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1162\/COMJ_a_00176","article-title":"Feedback systems: an analytical framework","volume":"37","author":"Sanfilippo","year":"2013","journal-title":"Computer Music Journal"},{"key":"10.7717\/peerj-cs.205\/ref-42","first-page":"471","article-title":"An overview of reservoir computing: theory, applications and implementations","author":"Schrauwen","year":"2007"},{"issue":"1","key":"10.7717\/peerj-cs.205\/ref-43","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1080\/09298210600696857","article-title":"Concatenative sound synthesis: the early years","volume":"35","author":"Schwarz","year":"2006","journal-title":"Journal of New Music Research"},{"key":"10.7717\/peerj-cs.205\/ref-44","first-page":"1001","article-title":"Automatic audio morphing","author":"Slaney","year":"1996"},{"issue":"5","key":"10.7717\/peerj-cs.205\/ref-45","doi-asserted-by":"publisher","first-page":"e37372","DOI":"10.1371\/journal.pone.0037372","article-title":"Transferring learning from external to internal weights in echo-state networks with sparse connectivity","volume":"7","author":"Sussillo","year":"2012","journal-title":"PLOS ONE"},{"key":"10.7717\/peerj-cs.205\/ref-46","article-title":"Groovy neural networks","author":"Tidemann","year":"2008"},{"key":"10.7717\/peerj-cs.205\/ref-47","article-title":"Real-time granular synthesis with the DMX-1000","author":"Truax","year":"1986"},{"issue":"2","key":"10.7717\/peerj-cs.205\/ref-48","doi-asserted-by":"crossref","first-page":"38","DOI":"10.2307\/3680442","article-title":"Discovering inner complexity: time shifting and transposition with a real-time granulation technique","volume":"18","author":"Truax","year":"1994","journal-title":"Computer Music Journal"},{"issue":"10","key":"10.7717\/peerj-cs.205\/ref-49","doi-asserted-by":"publisher","first-page":"1958","DOI":"10.1016\/j.neucom.2010.01.016","article-title":"A comparative study of Reservoir Computing strategies for monthly time series prediction","volume":"73","author":"Wyffels","year":"2010","journal-title":"Neurocomputing"},{"key":"10.7717\/peerj-cs.205\/ref-50","article-title":"Real-valued parametric conditioning of an RNN for interactive sound synthesis","author":"Wyse","year":"2018"},{"key":"10.7717\/peerj-cs.205\/ref-51","volume-title":"Formalized music","author":"Xenakis","year":"1971"},{"issue":"2","key":"10.7717\/peerj-cs.205\/ref-52","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1109\/TETCI.2017.2783885","article-title":"Automatic programming of VST sound synthesizers using deep networks and other techniques","volume":"2","author":"Yee-King","year":"2018","journal-title":"IEEE Transactions on Emerging Topics in Computational Intelligence"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-205.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-205.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-205.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-205.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,7,8]],"date-time":"2019-07-08T03:49:51Z","timestamp":1562557791000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-205"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,8]]},"references-count":52,"alternative-id":["10.7717\/peerj-cs.205"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.205","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.2\/reviews\/3","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.1\/reviews\/1","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.2\/reviews\/1","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.1\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.2\/reviews\/2","asserted-by":"object"},{"id-type":"doi","id":"10.7287\/peerj-cs.205v0.1\/reviews\/3","asserted-by":"object"}]},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,8]]},"article-number":"e205"}}