Abstract
With the development of the telecommunication network, more and more devices are used in the network, which has been a burden for the network operation and maintenance. At the same time, network devices generate large amounts of log data every day, recording the activities of each device in detail. As a result, the log can reflect the performance of network state, and sometimes, we can predict the occurrence of network failure based on the log. However, since the log has such features: big volume, multi-source heterogeneous and difficult to understand, people have not reasonably used it to analyze and predict network failure. Therefore, we propose a method for structuring a large number of device logs in the short term, and use the data generated from a real communication device network to verify the effect. Besides, we compare our method with the traditional log parsers, such as regular expressions, LogSig, etc. to demonstrate the efficient processing performance and accurate pattern extraction analysis for massive network device logs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
He, P., et al.: An evaluation study on log parsing and its use in log mining. In: IEEE/IFIP International Conference on Dependable Systems and Networks. IEEE Computer Society, pp. 654–661 (2016)
Fu, Q., Lou, J., Wang, Y., Li, J.: Execution anomaly detection in distributed systems through unstructured log analysis. In: Proceedings of International Conference on Data Mining, ICDM 2009 (2009)
Makanju, A., Zincir-Heywood, A., Milios, E.: Clustering event logs using iterative partitioning. In: Proceedings of International Conference on Knowledge Discovery and Data Mining, KDD 2009 (2009)
Tang, L., Li, T., Perng, C.: LogSig: generating system events from raw textual logs. In: Proceedings of ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 785–794 (2011)
Kimura, T., lshibashi, K., Mori, T., Shiomoto, K.: Spatio-temporal factorization of log data for understanding network events. In: INFOCOM 2014 Proceedings. IEEE (2014)
Juneja, P., Kundra, D., Sureka, A.: Anvaya: an algorithm and case-study on improving the goodness of software process models generated by mining event-log data in issue tracking systems. Support. Care Cancer 6(6), 539–541 (2015)
Hamooni, H., Debnath, B., Xu, J., et al.: LogMine: fast pattern recognition for log analytics. In: CIKM (2016)
Bengio, Y., et al.: A neural probabilistic language model. J. Mach. Learn. Res. 3(6), 113 (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Li, L., Man, Y., Chen, M. (2018). A Method of Large - Scale Log Pattern Mining. In: Zu, Q., Hu, B. (eds) Human Centered Computing. HCC 2017. Lecture Notes in Computer Science(), vol 10745. Springer, Cham. https://doi.org/10.1007/978-3-319-74521-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-74521-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-74520-6
Online ISBN: 978-3-319-74521-3
eBook Packages: Computer ScienceComputer Science (R0)