Abstract
Deep learning architecture has shown remarkable performance in machine learning and AI applications. However, training a spiking Deep Convolutional Neural Network (DCNN) while incorporating traditional CNN properties remains an open problem for researchers. This paper explores a novel spiking DCNN consisting of a convolutional/pooling layer followed by a fully connected SNN trained in a greedy layer-wise manner. The feature extraction of images is done by the spiking DCNN component of the proposed architecture. And in achieving the feature extraction, we leveraged on the SAILnet to train the original MNIST data. To serve as input to the convolution layer, we process the raw MNIST data with bilateral filter to get the filtered image. The convolution kernel trained in the previous step is used to calculate the filtered image’s feature map, and carry out the maximum pooling operation on the characteristic map. We use BP-STDP to train the fully connected SNN for prediction. To avoid over fitting and to further improve the convergence speed of the network, a dynamic dropout is added when the accuracy of the training sets reaches 97% to prevent co-adaptation of neurons. In addition, the learning rate is automatically adjusted in training, which ensures an effective way to speed up training and slow down the rising speed of the training accuracy at each epoch. Our model is evaluated on the MNIST digit and Cactus3 shape datasets, with the recognition performance on test datasets being 96.16% and 97.92% respectively. The level of performance shows that our model is capable of extracting independent and prominent features in images using spikes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Tavanaei, A., Ghodrati, M., Kheradpisheh, S.R., Masquelier, T., Maida, A.: Deep learning in spiking neural networks. Neural Netw. 111, 47–63 (2018)
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)
Krizhevsky, A., Sutskever, I., Hintom, G.E.: ImageNet classification with deep convolutional neural networks (2012)
Rawat, W., Wang, Z.: Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29(9), 1–98 (2017)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1717–1724. IEEE (2014)
Abdel-Hamid, O., Deng, L., Yu, D.: Exploring convolutional neural network structures and optimization techniques for speech recognition. In: Proceedings Annual Conference of the International Speech Communication Association, INTERSPEECH, pp. 3366–3370, August 2013
Sainath, T.N., Mohamed, A.R., Kingsbury, B., Ramabhadran, B.: Deep convolutional neural networks for LVCSR. In: ICASSP, IEEE International Conference on Acoustics, Speech, and Signal Processing - Proceedings, pp. 8614–8618 (2013)
Tavanaei, A., Maida, A.S., Kaniymattam, A., Loganantharaj, R.: Towards recognition of protein function based on its structure using deep convolutional networks. In: Proceedings - 2016 IEEE International Conference on Bioinformatics and Biomedicine, pp. 145–149 (2017)
Zeng, H., Edwards, M.D., Liu, G., Gifford, D.K.: Convolutional neural network architectures for predicting DNA-protein binding. Bioinformatics 32(12), i121–i127 (2016)
Zhang, Y., Qiu, Z., Yao, T., Liu, D., Mei, T.: Fully convolutional adaptation networks for semantic segmentation. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 6810–6818 (2018)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Bengio, Y., Mesnard, T., Fischer, A., Zhang, S., Wu, Y.: STDP-compatible approximation of backpropagation in an energy-based model. Neural Comput. 577, 555–577 (2017)
O’Connor, P., Welling, M.: Deep spiking networks. In: Proceedings of the 33rd International Conference on Machine Learning (2016)
Masquelier, T., Thorpe, S.J.: Unsupervised learning of visual features through spike timing dependent plasticity. PLoS Comput. Biol. 3(2), 0247–0257 (2007)
Kheradpisheh, S.R., Ganjtabesh, M., Masquelier, T.: Bio-inspired unsupervised learning of visual features leads to robust invariant object recognition. Neurocomputing 205, 382–392 (2016)
Tavanaei, A., Masquelier, T., Maida, A.S.: Acquisition of visual features through probabilistic spike-timing-dependent plasticity. In: Proceedings International Joint Conference on Neural Networks, pp. 307–314 (2016)
Kheradpisheh, S.R., Ganjtabesh, M., Thorpe, S.J., Masquelier, T.: STDP-based spiking deep convolutional neural networks for object recognition. Neural Netw. 99, 56–67 (2018)
Thiele, J.C., Bichler, O., Dupret, A.: Event-based, timescale invariant unsupervised online deep learning with STDP. Front. Comput. Neurosci. 12, 1–13 (2018)
Tavanaei, A., Maida, A.: BP-STDP: approximating backpropagation using spike timing dependent plasticity. Neurocomputing 330, 39–47 (2019)
Tavanaei, A., Maida, A.S.: Multi-layer unsupervised learning in a spiking convolutional neural network. In: Proceedings International Joint Conference on Neural Networks, May 2017, pp. 2023–2030 (2017)
Tavanaei, A., Maida, A.S.: Bio-inspired spiking convolutional neural network using layer-wise sparse coding and STDP learning (2016)
Foldiak, P.: Forming sparse representation by local anti-Hebbian learning. Biol. Cybern. 170, 165–170 (1990)
Zylberberg, J., Murphy, J.T., DeWeese, M.R.: A sparse coding model with Synaptically local plasticity and spiking neurons can account for the diverse shapes of V1 simple cell receptive fields. PLoS Comput. Biol. 7(10) (2011)
Markram, H., Gerstner, W., Sjöström, P.J.: Spike-timing-dependent plasticity: a comprehensive overview. Front. Synaptic Neurosci. 4, 2010–2012 (2012)
Tavanaei, A., Maida, A.S.: Training a hidden Markov model with a Bayesian spiking neural network. J. Sig. Process. Syst. 90(2), 211–220 (2016). https://doi.org/10.1007/s11265-016-1153-2
Tavanaei, A., Kirby, Z., Maida, A.S.: Training Spiking ConvNets by STDP and gradient descent. In: Proceedings International Joint Conference on Neural Networks, July 2018, pp. 1–8 (2018)
Diehl, P.U., Cook, M.: Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci. 9, 1–9 (2015)
Lee, C., Srinivasan, G., Panda, P., Roy, K.: Deep spiking convolutional neural network trained with unsupervised spike timing dependent plasticity. IEEE Trans. Cogn. Dev. Syst. 11(3), 384–394 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Turkson, R.E., Qu, H., Wang, Y., Eghan, M.J. (2020). Unsupervised Multi-layer Spiking Convolutional Neural Network Using Layer-Wise Sparse Coding. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12534. Springer, Cham. https://doi.org/10.1007/978-3-030-63836-8_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-63836-8_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63835-1
Online ISBN: 978-3-030-63836-8
eBook Packages: Computer ScienceComputer Science (R0)