Abstract
Handwritten mathematical expression recognition (HMER) is a challenging task due to its complex two-dimensional structure of mathematical expressions and the high similarity between handwritten texts. Most existing encoder-decoder approaches for HMER mainly depend on local visual features but are seldom studied in explicit global semantic information. Besides, existing works for HMER primarily focus on local information. However, this obtained information is difficult to transmit between distant locations. In this paper, we propose a semantic-aware non-local network to tackle the above problems for HMER. Specifically, we propose to adopt the non-local network to capture long-term dependencies while integrating local and non-local features. Moreover, we customized the FastText language model to our backbone to learn the semantic-aware information. The experimental results illustrate that our design consistently outperforms the state-of-the-art methods on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016 datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
He, F., Tan, J., Bi, N.: Handwritten mathematical expression recognition: a survey. In: Lu, Y., Vincent, N., Yuen, P.C., Zheng, W.-S., Cheriet, F., Suen, C.Y. (eds.) ICPRAI 2020. LNCS, vol. 12068, pp. 55–66. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59830-3_5
Mouchere, H., et al.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: International Conference on Frontiers in Handwriting Recognition. IEEE (2014)
Mouchère, H., et al.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)
Wang, D.-H., et al.: ICFHR 2020 competition on offline recognition and spotting of handwritten mathematical expressions-OffRaSHME. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: towards accurate text recognition in natural images. In: ICCV 2017, pp. 5086–5094 (2017)
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: attentive language models beyond a fixed-length context. In: ACL (1), pp. 2978–2988 (2019)
Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)
Gordo, A., et al.: LEWIS: latent embeddings for word images and their semantics. In: IEEE International Conference on Computer Vision. IEEE (2015)
Wilkinson, T., Brun, A.: Semantic and verbatim word spotting using deep neural networks. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)
Wang, Q.-F., Yin, F., Liu, C.-L.: Handwritten Chinese text recognition by integrating multiple contexts. IEEE Trans. Pattern Anal. Mach. Intell. 34(8), 1469–1481 (2011)
Wang, X., et al.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Bojanowski, P., et al.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436–459. ACM (1967)
Chan, K.F., Yeung, D.Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Pattern Recogn. 34(8), 1671–1684 (2001)
Lavirotte, S., Pottier, L.: Mathematical formula recognition using graph grammar. Proc. SPIE Int. Soc. Opt. Eng. 3305, 44–52 (2016)
Yamamoto, R., et al.: On-line recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Proceedings of International Workshop on Frontiers in Handwriting Recognition, pp. 249–254, October 2006
Maclean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recogn. 16(2), 139–163 (2013)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. Computer Science (2014)
Bahdanau, D., et al.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2016)
Zhang, J., et al.: Radical analysis network for zero-shot learning in printed Chinese character recognition. In: 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE (2018)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. Computer Science, pp. 2048–2057 (2015)
Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE (2018)
Zhang, J., et al.: Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR) (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)
Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Zhang, J., et al.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning. PMLR (2020)
Wu, J.-W., Yin, F., Zhang, YM., Zhang, X.-Y., Liu, C.-L.: Image-to-markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS, vol. 11051, pp. 18–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_2
Wu, J.-W., et al.: Handwritten mathematical expression recognition via paired adversarial learning. Int. J. Comput. Vis. 128(10), 2386–2401 (2020)
Li, Z., et al.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 570–584. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_37
Truong, T.-N., et al.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)
Truong, T.-N., Nguyen, C.T., Nakagawa, M.: Syntactic data generation for handwritten mathematical expression recognition. Pattern Recogn. Lett. 153, 83–91 (2022)
Bian, X., et al.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. arXiv e-prints (2021)
Yuan, Y., et al.: Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601 (2022)
Liu, Y.-L., et al.: A robust and fast non-local means algorithm for image denoising. J. Comput. Sci. Technol. 23(2), 270–279 (2008)
Acknowledgement
This work is supported by Industry-University Cooperation Project of Fujian Science and Technology Department (No. 2021H6035), and the Science and Technology Planning Project of Fujian Province (No. 2021J011191, 2020H0023, 2020Y9064), and the Joint Funds of 5th Round of Health and Education Research Program of Fujian Province (No. 2019-WJ-41).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, XH., Wang, DH., Du, X., Zhu, S. (2022). Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13536. Springer, Cham. https://doi.org/10.1007/978-3-031-18913-5_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-18913-5_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18912-8
Online ISBN: 978-3-031-18913-5
eBook Packages: Computer ScienceComputer Science (R0)