Abstract
Most text recognition methods are trained on large amounts of labeled data. Although text images are easily accessible, labeling them is costly. Thus how to utilize the unlabeled data is worth studying. In this paper, we propose a MUltiple Granularity Semi-supervised (MUGS) method using both labeled and unlabeled data for text recognition. Inspired by the hierarchical structure (sentence-word-character) of text, we apply semi-supervised learning at both word-level and character-level. Specifically, a Dynamic Aggregated Self-training (DAS) framework is introduced to generate pseudo-labels from unlabeled data at word-level. To ensure the quality and stability of the pseudo-labeling procedure, the pseudo-labels are aggregated from one dynamic model queue which keeps updating in the whole semi-supervised training process. At the character-level, a novel module named WTC (Word To Character) that can convert sequential features to character representations is invented. Next, contrastive learning is applied to these character representations for better fine-grained visual modeling. The characters from various images that share the same classes are pulled together and the ones in different classes are set far apart in the representation space. With the combination of supervisions in different granularities, more information can be exploited from the unlabeled data. The effectiveness and robustness of the model are enhanced by a large margin. Comprehensive experiments on several public benchmarks validate our method and competitive performance is achieved with much fewer labeled data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aberdam, A., et al.: Sequence-to-sequence contrastive learning for text recognition. In: CVPR (2021)
Alex, G., Santiago, F., Faustino, G., Jürgen, S.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: ICML (2006)
Anhar, R., Palaiahankote, S., Seng, C.C., Lim, T.C.: A robust arbitrary text detection system for natural scene images. Expert Syst. Appl. 41, 8027–8048 (2014)
Baek, J., Matsui, Y., Aizawa, K.: What if we only use real datasets for scene text recognition? toward scene text recognition with fewer labels. In: CVPR (2021)
Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: Aster: an attentional scene text recognizer with flexible rectification. TPAMI 41, 2035–2048 (2018)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. TPAMI 39, 2298–2304 (2016)
Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. TPAMI 39, 2298–2304 (2017)
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: a holistic approach to semi-supervised learning. In: NIPS (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML (2020)
Chen-Yu, L., Simon, O.: Recursive recurrent nets with attention modeling for ocr in the wild. In: CVPR (2016)
Ch’ng, C.K., Chan, C.S., Liu, C.: Total-text: towards orientation robustness in scene text detection. IJDAR 23, 31–52 (2020)
Yao, C., Bai, X., Shi, B., Liu, W.: Strokelets: a learned multi-scale representation for scene text recognition. In: CVPR (2014)
Yao, C., Bai, X., Liu, W.: A unified framework for multioriented text detection and recognition. TIP 23, 4737–4749 (2014)
Yu, D., et al.: Towards accurate scene text recognition with semantic reasoning networks. In: CVPR (2020)
Dong-Dong, C., Wei, W., Wei, G., ZhiHua, Z.: Tri-net for semi-supervised deep learning. In: IJCAI (2018)
Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: CVPR (2016)
Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: CVPR (2006)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR (2020)
Li, H., Wang, P., Shen, C., Zhang, G.: Show, attend and read: a simple and strong baseline for irregular text recognition. In: AAAI (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Karatzas, D., et al.: Icdar 2015 competition on robust reading. In: ICDAR (2015)
Karatzas, D., et al.: Icdar 2013 robust reading competition. In: ICDAR (2013)
Kim, S., Kim, D., Cho, M., Kwak, S.: Proxy anchor loss for deep metric learning. In: CVPR (2020)
Lee, D.H.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: ICML (2013)
Litman, R., Anschel, O., Tsiper, S., Litman, R., Mazor, S., Manmatha, R.: Scatter: selective context attentional scene text recognizer. In: CVPR (2020)
Lucas, S.M., et al.: ICDAR 2003 robust reading competitions: entries, results and future directions. In: IJDAR (2005)
Luo, C., Jin, L., Chen, J.: Siman: exploring self-supervised representation learning of scene text via similarity-aware normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1039–1048 (2022)
Lyu, P., Yang, Z., Leng, X., Wu, X., Li, R., Shen, X.: 2D attentional irregular scene text recognizer (2019). CoRR abs/1906.05708. http://arxiv.org/abs/1906.05708
Max, J., Karen, S., Andrea, V., Andrew, Z.: Synthetic data and artificial neural networks for natural scene text recognition (2014). CoRR. http://arxiv.org/abs/1406.2227
Mishra, A., Alahari, K., Jawahar, C.: Scene text recognition using higher order language priors. In: BMVC (2012)
Neumann, L., Matas, J.: Real-time scene text localization and recognition. In: CVPR (2012)
Song, Q., Jiang, Q., Li, N., Zhang, R., Wei, X.: Reads: a rectified attentional double supervised network for scene text recognition. In: ICPR (2020)
Jiang, Q., Song, Q., Li, N., Zhang, R., Wei, X.: Heterogeneous network based semi-supervised learning for scene text recognition. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 64–78. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_5
Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. In: NIPS (2020)
Phan, T.Q., Shivakumara, P., Tian, S., Tan, C.L.: Recognizing text with perspective distortion in natural scenes. In: ICCV (2013)
Fang, S., Xie, H., Wang, Y., Mao, Z., Zhang, Y.: Read like humans: autonomous, bidirectional and iterative language modeling for scene text recognition. In: CVPR (2021)
Qiao, S., Shen, W., Zhang, Z., Wang, B., Yuille, A.: Deep co-training for semi-supervised image recognition. In: ECCV (2018)
Sohn, K., et al.: Fixmatch: simplifying semi-supervised learning with consistency and confidence. In: NIPS (2020)
Triguero, I., García, S., Herrera, F.: Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowl. Inf. Syst. 42, 245–284 (2015)
Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)
Wang, K., Babenko, B., Belongie, S.: End-to-end scene text recognition. In: ICCV (2011)
Yalniz, I.Z., Jégou, H., Chen, K., Paluri, M., Mahajan, D.: Billion-scale semi-supervised learning for image classification (2019). CoRR abs/1905.00546, http://arxiv.org/abs/1905.00546
Yang, M., et al.: Reading and writing: discriminative and generative modeling for self-supervised text recognition. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 4214–4223 (2022)
Sun, Y., et al.: Circle loss: a unified perspective of pair similarity optimization. In: CVPR (2020)
Gao, Y., Chen, Y., Wang, J., Lu, H.: Semi-supervised scene text recognition. TIP 30, 3005–3016 (2021)
Gao, Y., Huang, Z., Dai, Y., Xu, C., Chen, K., Tuo, J.: Double supervised network with attention mechanism for scene text recognition (2018). CoRR abs/1808.00677, http://arxiv.org/abs/1808.00677
Zeiler, M.D.: Adadelta: an adaptive learning rate method. Comput. Sci. (2012)
Zhang, X., Zhu, B., Yao, X., Sun, Q., Li, R., Yu, B.: Context-based contrastive learning for scene text recognition. In: AAAI (2022)
Zheng, C., et al.: Pushing the performance limit of scene text recognizer without human annotation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14116–14125 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Song, Q., Jiang, Q., Wang, L., Zhao, L., Zhang, R. (2023). MUGS: A Multiple Granularity Semi-supervised Method for Text Recognition. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14191. Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-41734-4_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41733-7
Online ISBN: 978-3-031-41734-4
eBook Packages: Computer ScienceComputer Science (R0)
