Abstract
Along with the innovation and development of society, millions of documents are generated daily, and new types of documents related to new activities and services appear regularly. In the workflow for processing these documents, the first step is to classify the received documents to assign them to the relevant departments or staff. Therefore, two major problems arise: 1) the document classification algorithm is required to not only learn new categories efficiently but also have low error rate on them; 2) document classification needs to have the ability to detect when new classes appear. To address the first problem, we propose a class incremental learning method combined with ambiguity rejection to reduce the error rate of document classification. In this method, the ambiguity rejection module uses the classifier’s probabilities and statistical analysis to optimize per-class thresholds for filtering out (rejecting) uncertain results, thereby increasing the accuracy of non-filtered documents. Our method is thoroughly evaluated on a public business document dataset and on our private administrative document dataset including 36 categories of more than 23 thousand images. On our private dataset, the method increases the classification accuracy of 0.82%, from 98.28% to 99.10%, while the rejection rate is only 3.42%. Additionally, with incremental learning scenario, it can achieve error and rejection rates of 1% and 8.98%, respectively. This result has great significance, can also be applied to similar document datasets, and especially has great potential for real-world applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Neural Information Processing Systems, vol. 25, pp. 1097–1105 (2012). https://doi.org/10.1145/3065386
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv arXiv:cs.CL/1810.04805 (2019)
Le, H., et al.: FlauBERT: unsupervised language model pre-training for French. In: The 12th Language Resources and Evaluation Conference. LREC 2020, pp. 2479–2490 (2020)
Martin, L., et al.: CamemBERT: a tasty French language model. In: The 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, pp. 7203–7219 (2020)
d’Andecy, V.P., Joseph, A., Ogier, J.M.: InDUS: incremental document understanding system focus on document classification. In: Proceedings of 13th IAPR International Workshop on Document Analysis Systems, DAS 2018, pp. 239–244 (2018). https://doi.org/10.1109/DAS.2018.53
Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: DocFormer: end-to-end transformer for document understanding. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), ICCV 2021, pp. 973–983 (2021). https://doi.org/10.1109/ICCV48922.2021.00103
Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3: pre-training for document AI with unified text and image masking. arXiv arXiv:abs/2204.08387 (2022)
Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding. arXiv arXiv:abs/2012.14740 (2021)
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1192–1200 (2020). https://doi.org/10.1145/3394486.3403172
Li, P., et al.: SelfDoc: self-supervised document representation learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5648–5656 (2021). https://doi.org/10.1109/CVPR46437.2021.00560
Bakkali, S., Ming, Z., Coustaty, M., Rusiñol, M.: EAML: ensemble self-attention-based mutual learning network for document image classification. Int. J. Doc. Anal. Recogn. (IJDAR) 24, 251–268 (2021). https://doi.org/10.1007/s10032-021-00378-0
Mahamoud, I.S., Voerman, J., Coustaty, M., Joseph, A., d’Andecy, V.P., Ogier, J.M.: Multimodal attention-based learning for imbalanced corporate documents classification. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) Document Analysis and Recognition - ICDAR 2021. ICDAR 2021. LNCS, vol. 12823, pp. 223–237. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86334-0_15
Voerman, J., Mahamoud, I.S., Joseph, A., Coustaty, M., d’Andecy, V.P., Ogier, J.M.: Toward an incremental classification process of document stream using a cascade of systems. In: Barney Smith, E.H., Pal, U. (eds.) Document Analysis and Recognition - ICDAR 2021 Workshops. ICDAR 2021. LNCS, vol. 12917, pp. 240–254. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86159-9_16
Bakkali, S., Ming, Z., Coustaty, M., Rusiñol, M.: Visual and textual deep feature fusion for document image classification. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2020), pp. 2394–2403 (2020). https://doi.org/10.1109/CVPRW50498.2020.00289
Harley, A., Ufkes, A., Derpanis, K.: Evaluation of deep convolutional nets for document image classification and retrieval. In: 13th International Conference on Document Analysis and Recognition (ICDAR), ICDAR 2015, pp. 991–995 (2015). https://doi.org/10.1109/ICDAR.2015.7333910
Afzal, M.Z., Kölsch, A., Ahmed, S., Liwicki, M.: Cutting the error by half: investigation of very deep CNN and advanced training strategies for document image classification. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 883–888 (2017). https://doi.org/10.1109/ICDAR.2017.149
Das, A., Roy, S., Bhattacharya, U., Parui, S.K.: Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks. In: 24th International Conference on Pattern Recognition (ICPR), pp. 3180–3185. IEEE (2018). https://doi.org/10.1109/ICPR.2018.8545630
Shu, L., Xu, H., Liu, B.: DOC: deep open classification of text documents. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, pp. 2911–2916 (2017). https://doi.org/10.18653/v1/D17-1314
Mathew, M., Karatzas, D., Jawahar, C.V.: DocVQA: a dataset for VQA on document images. In: WACV (2021)
Zhong, X., Tang, J., Yepes, A.J.: PubLayNet: largest dataset ever for document layout analysis. In: International Conference on Document Analysis and Recognition (ICDAR), ICDAR 2019 (2019). https://doi.org/10.1109/ICDAR.2019.00166
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542 (2017). https://doi.org/10.1109/CVPR.2017.587
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop (2015)
Van de Ven, G.M., Tuytelaars, T., Tolias, A.S.: Three types of incremental learning. Nat. Mach. Intell. 4, 1185–1197 (2022). https://doi.org/10.1038/s42256-022-00568-3
Hou, S., Pan, X., Change, C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
Wu, Y., et al.: Large scale incremental learning. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
Liu, Y., Su, Y., Liu, A.A., Schiele, B., Sun, Q.: Mnemonics training: multi-class incremental learning without forgetting. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
Mittal, S., Galesso, S., Brox, T.: Essentials for class incremental learning. In: IEEE Conference on Computer Vision and Pattern Recognition (2021)
Chow, C.K.: On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16, 41–46 (1970)
Pietraszek, T.: Optimizing abstaining classifiers using ROC analysis. In: ICML (2005)
Pham, T.C., Luong, C.M., Hoang, V.D., Doucet, A.: AI outperformed every dermatologist in dermoscopic melanoma diagnosis, using an optimized deep-CNN architecture with custom mini-batch logic and loss function. Sci. Rep. 11, 17485 (2021). https://doi.org/10.1038/s41598-021-96707-8
Fumera, G., Roli, F., Giacinto, G.: Multiple reject thresholds for improving classification reliability. In: Ferri, F.J., Iñesta, J.M., Amin, A., Pudil, P. (eds.) SSPR /SPR 2000. LNCS, vol. 1876, pp. 863–871. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-44522-6_89
Motlagh, N.K., Davis, J., Anderson, T., Gwinnup, J.: Learning when to say “I Don’t Know”. arXiv arXiv:2209.04944 (2022)
Franc, V., Prusa, D., Voracek, V.: Optimal strategies for reject option classifiers. J. Mach. Learn. Res. 24, 1–49 (2023)
Acknowledgement
This work was supported by the French government in the framework of the France Relance program and by the YOOZ company under the grant number ANR-21-PRRD-0010-01. We also would like to thank Guénael Manic, Mohamed Saadi, Jonathan Ouellet and Jérôme Lacour from YOOZ for their support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pham, TC., Coustaty, M., Joseph, A., Poulain d’Andecy, V., Visani, M., Sidere, N. (2023). Incremental Learning and Ambiguity Rejection for Document Classification. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14191. Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-41734-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41733-7
Online ISBN: 978-3-031-41734-4
eBook Packages: Computer ScienceComputer Science (R0)
