Abstract
This paper presents an approach for substantial reduction of the training and operating phases of Self-Organizing Maps in tasks of 2-D projection of multi-dimensional symbolic data for natural language processing such as language classification, topic extraction, and ontology development. The conventional approach for this type of problem is to use n-gram statistics as a fixed size representation for input of Self-Organizing Maps. The performance bottleneck with n-gram statistics is that the size of representation and as a result the computation time of Self-Organizing Maps grows exponentially with the size of n-grams. The presented approach is based on distributed representations of structured data using principles of hyperdimensional computing. The experiments performed on the European languages recognition task demonstrate that Self-Organizing Maps trained with distributed representations require less computations than the conventional n-gram statistics while well preserving the overall performance of Self-Organizing Maps.
This work was supported by the Swedish Research Council (VR, grant 2015-04677) and the Swedish Foundation for International Cooperation in Research and Higher Education (grant IB2018-7482) for its Initiation Grant for Internationalisation, which allowed conducting the study.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Available online at http://www.statmt.org/europarl/.
References
Alahakoon, D., Halgamuge, S., Srinivasan, B.: Dynamic self-organizing maps with controlled growth for knowledge discovery. IEEE Trans. Neural Netw. 11(3), 601–614 (2000)
Appiah, K., Hunter, A., Dickinson, P., Meng, H.: Implementation and applications of tri-state self-organizing maps on FPGA. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1150–1160 (2012)
Bandaragoda, T.R., De Silva, D., Alahakoon, D.: Automatic event detection in microblogs using incremental machine learning. J. Assoc. Inform. Sci. Technol. 68(10), 2394–2411 (2017)
Bishop, C.M., Svensén, M., Williams, C.K.: GTM: the generative topographic mapping. Neural Comput. 10(1), 215–234 (1998)
De Silva, D., Alahakoon, D.: Incremental knowledge acquisition and self learning from text. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2010)
De Silva, D., Alahakoon, D., Yu, X.: A data fusion technique for smart home energy management and analysis. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4594–4600 (2016)
De Silva, D., et al.: Machine learning to support social media empowered patients in cancer care and cancer treatment decisions. PLoS One 13(10), 1–10 (2018)
De Silva, D., Yu, X., Alahakoon, D., Holmes, G.: A data mining framework for electricity consumption analysis from meter data. IEEE Trans. Industr. Inf. 7(3), 399–407 (2011)
Dittenbach, M., Merkl, D., Rauber, A.: The growing hierarchical self-organizing map. In: International Joint Conference on Neural Networks (IJCNN), vol. 6, pp. 15–19 (2000)
Eliasmith, C.: How to Build a Brain. Oxford University Press, Oxford (2013)
Frady, E.P., Kleyko, D., Sommer, F.T.: A theory of sequence indexing and working memory in recurrent neural networks. Neural Comput. 30, 1449–1513 (2018)
Hazan, H., Saunders, D.J., Sanghavi, D.T., Siegelmann, H.T., Kozma, R.: Unsupervised learning with self-organizing spiking neural networks. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6 (2018)
Hinton, G., McClelland, J., Rumelhart, D.: Distributed representations. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing. Explorations in the Microstructure of Cognition - Volume 1 Foundations, pp. 77–109. MIT Press, Cambridge (1986)
Jayaratne, M., Alahakoon, D., De Silva, D., Yu, X.: Bio-inspired multisensory fusion for autonomous robots. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3090–3095 (2018)
Joshi, A., Halseth, J.T., Kanerva, P.: Language geometry using random indexing. In: de Barros, J.A., Coecke, B., Pothos, E. (eds.) QI 2016. LNCS, vol. 10106, pp. 265–274. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52289-0_21
Kanerva, P.: Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1(2), 139–159 (2009)
Kaski, S., Honkela, T., Lagus, K., Kohonen, T.: WEBSOM-self-organizing maps of document collections1. Neurocomputing 21(1–3), 101–117 (1998)
Kleyko, D., Osipov, E.: Brain-like classifier of temporal patterns. In: International Conference on Computer and Information Sciences (ICCOINS), pp. 104–113 (2014)
Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Alahakoon, D.: Integer self-organizing maps for digital hardware. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Kleyko, D., Osipov, E., Gayler, R.: Recognizing permuted words with vector symbolic architectures: a cambridge test for machines. Proc. Comput. Sci. 88, 169–175 (2016)
Kleyko, D., Osipov, E., Papakonstantinou, N., Vyatkin, V.: Hyperdimensional computing in industrial systems: the use-case of distributed fault isolation in a power plant. IEEE Access 6, 30766–30777 (2018)
Kleyko, D., Osipov, E., Wiklund, U.: A hyperdimensional computing framework for analysis of cardiorespiratory synchronization during paced deep breathing. IEEE Access 7, 34403–34415 (2019)
Kleyko, D., Rahimi, A., Gayler, R., Osipov, E.: Autoscaling bloom filter: controlling trade-off between true and false positives. Neural Comput. Appl. 1–10 (2019)
Kleyko, D., Rahimi, A., Rachkovskij, D., Osipov, E., Rabaey, J.: Classification and recall with binary hyperdimensional computing: trade-offs in choice of density and mapping characteristic. IEEE Trans. Neural Netw. Learn. Syst. 29(12), 5880–5898 (2018)
Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences. Springer, Heidelberg (2001). https://doi.org/10.1007/978-3-642-56927-2
Kohonen, T., Somervuo, P.: Self-organizing maps of symbol strings. Neurocomputing 21(1–3), 19–30 (1998)
Kusiak, A.: Smart manufacturing must embrace big data. Nat. News 544(7648), 23 (2017)
Nallaperuma, D., De Silva, D., Alahakoon, D., Yu, X.: Intelligent detection of driver behavior changes for effective coordination between autonomous and human driven vehicles. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3120–3125 (2018)
Nawaratne, R., Bandaragoda, T., Adikari, A., Alahakoon, D., De Silva, D., Yu, X.: Incremental knowledge acquisition and self-learning for autonomous video surveillance. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4790–4795 (2017)
Osipov, E., Kleyko, D., Legalov, A.: Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3276–3281 (2017)
Plate, T.A.: Holographic Reduced Representations: Distributed Representation for Cognitive Structures. Center for the Study of Language and Information (CSLI), Stanford (2003)
Quasto, U., Richter, M., Biemann, C.: Corpus portal for search in monolingual corpora. In: Fifth International Conference on Language Resources and Evaluation (LREC), pp. 1799–1802 (2006)
Rachkovskij, D.A.: Representation and processing of structures with binary sparse distributed codes. IEEE Trans. Knowl. Data Eng. 3(2), 261–276 (2001)
Rahimi, A., et al.: High-dimensional Computing as a Nanoscalable Paradigm. IEEE Trans. Circ. Syst. I: Regul. Pap. 64(9), 2508–2521 (2017)
Rahimi, A., Kanerva, P., Benini, L., Rabaey, J.M.: Efficient biosignal processing using hyperdimensional computing: network templates for combined learning and classification of ExG signals. Proc. IEEE 107(1), 123–143 (2019)
Rahimi, A., Kanerva, P., Rabaey, J.: A robust and energy efficient classifier using brain-inspired hyperdimensional computing. In: IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 64–69 (2016)
Rasanen, O., Kakouros, S.: Modeling dependencies in multiple parallel data streams with hyperdimensional computing. IEEE Signal Process. Lett. 21(7), 899–903 (2014)
Recchia, G., Sahlgren, M., Kanerva, P., Jones, M.: Encoding sequential information in semantic space models: comparing holographic reduced representation and random permutation. Computat. Intell. Neurosci. 1–18 (2015)
Santana, A., Morais, A., Quiles, M.: An alternative approach for binary and categorical self-organizing maps. In: International Joint Conference on Neural Networks (IJCNN), pp. 2604–2610 (2017)
Shah-Hosseini, H., Safabakhsh, R.: TASOM: a new time adaptive self-organizing map. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 33(2), 271–282 (2003)
Vesanto, J., Alhoniemi, E.: Clustering of the self-organizing map. IEEE Trans. Neural Netw. 11(3), 586–600 (2000)
Zolotukhin, M., Hamalainen, T., Juvonen, A.: Online anomaly detection by using n-gram model and growing hierarchical self-organizing maps. In: International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 47–52 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Vyatkin, V., Alahakoon, D. (2019). Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing. In: Bjørner, N., Virbitskaite, I., Voronkov, A. (eds) Perspectives of System Informatics. PSI 2019. Lecture Notes in Computer Science(), vol 11964. Springer, Cham. https://doi.org/10.1007/978-3-030-37487-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-37487-7_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37486-0
Online ISBN: 978-3-030-37487-7
eBook Packages: Computer ScienceComputer Science (R0)