An augmented reality for an arabic text reading and visualization assistant for the visually impaired | Multimedia Tools and Applications Skip to main content

Advertisement

Springer Nature Link
Log in

An augmented reality for an arabic text reading and visualization assistant for the visually impaired

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Text, as one of humanity’s most influential innovations, has played an important role in shaping our lives. Reading a text is a difficult task due to several reasons factors, such as luminosity, text orientation, writing style, and very light colors. However, the visually impaired, on the other hand, have difficulty reading a text in all of these situations. In particular, a handwritten text is more difficult to read than a digital text due to the different forms and styles of the handwriting of different writers or, sometimes, of the same writer. Therefore, they would benefit from a device or a system to help them to solve this problem and improve their quality of life. Arabic language recognition and identification is a very difficult task because of diacritics such as consonant score, tashkil, and others. In this context, we propose a recognition and identification system for Arabic Handwritten Texts with Diacritics (AHTD) based on deep learning by using the convolutional neural network. Text images are trained, tested, and validated with our Arabic Handwritten Texts with a Diacritical Dataset (AHT2D). Then, the recognized text is enhanced with augmented reality technology and produced as a 2D image. Finally, the recognized text is converted into an audio output using AR technology. Voice output and visual output are given to the visually impaired user. The experimental results show that the proposed system is robust, with an accuracy rate of 95%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Algorithm 1
Fig. 9
Fig. 10
Algorithm 2
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data Availability

The dataset which is used in this paper is public available.

References

  1. Abbes̀ R, Dichy J (2008) Extraction automatique de fréquences lexicales en arabe et analyse d’un corpus journalistique avec le logiciel araconc et la base de connaissances diinar. 1. Serge Heiden & Bénédicte Pincemain, Proceedings of JADT, pp 12–14

  2. Abuzaraida MA, Elmehrek M, Elsomadi E (2021) Online handwriting arabic recognition system using k-nearest neighbors classiffier and dct features. International Journal of Electrical & Computer Engineering (2088-8708) 11(4)

  3. Almansari OA, Hashim NNWN (2019) Recognition of isolated handwritten arabic characters. In: 2019 7th International conference on Mechatronics engineering (ICOM), pp 1–5. IEEE

  4. Almisreb AA, Turaev S, Saleh MA, Al Junid SAM et al (2022) Arabic handwriting classification using deep transfer learning techniques. Pertanika Journal of Science & Technology, vol 30(1)

  5. Alrobah N, Albahli S (2021) A hybrid deep model for recognizing arabic handwritten characters. IEEE Access

  6. Andriyandi AP, Darmalaksana W, adillah Maylawati D, Irwansyah FS, Mantoro T, Ramdhani MA (2020) Augmented reality using features accelerated segment test for learning tajweed. Telkomnika (Telecommunication Comput Electron Control 18(1):208–216. https://doi.org/10.12928/TELKOMNIKA.V18I1.14750

    Article  Google Scholar 

  7. Ardian Z, Santoso PI, Hantono BS (2018) Argot: Text-based detection systems in real time using augmented reality for media translator aceh-indonesia with android-based smartphones. In: Journal of physics: conference series, vol 1019, pp 012074. IOP Publishing

  8. Balhara S, Gupta N, Alkhayyat A, Bharti I, Malik RQ, Mahmood SN, Abedi F (2022) A survey on deep reinforcement learning architectures, applications and emerging trends. IET Communications

  9. Busaeed S, Mehmood R, Katib I (2022) Requirements, challenges and use of digital devices and apps for blind and visually impaired

  10. Butt H, Raza MR, Ramzan MJ, Ali MJ, Haris M (2021) Attention-based cnn-rnn arabic text recognition from natural scene images. Forecasting 3 (3):520–540

    Article  Google Scholar 

  11. Callaos N (2022) Intellectual development via trans-disciplinary communication

  12. Chen L, Chen P, Lin Z (2020) Artificial intelligence in education: a review. Ieee Access 8:75264–75278

    Article  Google Scholar 

  13. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, pp 319–340

  14. Eltay M, Zidouri A, Ahmad I (2020) Exploring deep learning approaches to recognize handwritten arabic texts. IEEE Access 8:89882–89898

    Article  Google Scholar 

  15. Eltay M, Zidouri A, Ahmad I, Elarian Y (2022) Generative adversarial network based adaptive data augmentation for handwritten arabic text recognition. PeerJ Computer Science 8:861

    Article  Google Scholar 

  16. Ge Y (2019) A survey on big data in the age of artificial intelligence. In: 2019 6th International conference on information, cybernetics, and computational social systems (ICCSS), pp 72–77. IEEE

  17. Ghosh M, Mukherjee H, Obaidullah SM, Santosh K, Das N, Roy K (2021) Lwsinet: a deep learning-based approach towards video script identification. Multimed Tools Appl, pp 1–34

  18. Hamdi Y, Boubaker H, Alimi AM (2021) Data augmentation using geometric, frequency, and beta modeling approaches for improving multi-lingual online handwriting recognition. International Journal on Document Analysis and Recognition (IJDAR), pp 1–16

  19. He W, Zhang X-Y, Yin F, Liu C-L (2017) Deep direct regression for multi-oriented scene text detection. In: Proceedings of the IEEE international conference on computer vision, pp 745–753

  20. Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with elms for big data

  21. Lei L, Tan Y, Zheng K, Liu S, Zhang K, Shen X (2020) Deep reinforcement learning for autonomous internet of things: model, applications and challenges. IEEE Communications Surveys & Tutorials 22(3):1722–1760

    Article  Google Scholar 

  22. Mohammed MJ, Tariq SM, Ayad H (2021) Isolated arabic handwritten words recognition using ehd and hog methods. Indonesian Journal of Electrical Engineering and Computer Science 22(2):193–200

    Article  Google Scholar 

  23. Mori B, Gioventù C (2020) An augmented reality (ar) experience for lorenzo lotto. In: Virtual and augmented reality in education, art, and museums, pp 324–332. IGI Global

  24. Mostafa A, Elsayed A, Ahmed M, Mohamed R, Adel M, Ashraf Y (2020) Smart educational game based on augmented reality. Technical report, EasyChair

  25. Mostafa A, Mohamed O, Ashraf A, Elbehery A, Jamal S, Khoriba G, Ghoneim AS (2021) Ocformer: a transformer-based model for arabic handwritten text recognition. In: 2021 International mobile, intelligent, and ubiquitous computing conference (MIUCC), pp 182–186. IEEE

  26. Muaad AY, Al-antari MA, Lee S, Davanagere HJ (2021) A novel deep learning arcar system for arabic text recognition with character-level representation. In: Computer sciences & mathematics forum, vol 2, p 14. MDPI

  27. Muaad AY, Jayappa H, Al-antari MA, Lee S (2021) Arcar: a novel deep learning computer-aided recognition for character-level arabic text representation and recognition. Algorithms 14(7):216

    Article  Google Scholar 

  28. Ouali I, Ghozzi F, Taktak R, Sassi MSH (2019) Ontology alignment using stable matching. Procedia Computer Science 159:746–755

    Article  Google Scholar 

  29. Ouali I, Hadj Sassi MS, Ben Halima M, Wali A (2021) Architecture for real-time visualizing arabic words with diacritics using augmented reality for visually impaired people. In: International conference on advanced information networking and applications, pp 285–296. Springer

  30. Ouali I, Halima MB, Ali W (2022) Augmented reality for scene text recognition, visualization and reading to assist visually impaired people. Procedia Computer Science 176:158–167

    Article  Google Scholar 

  31. Ouali I, Halima MB, Wali A (2020) A new architecture based ar for detection and recognition of objects and text to enhance navigation of visually impaired people. Procedia Computer Science 176:602–611

    Article  Google Scholar 

  32. Ouali I, Halima MB, Wali A (2022) Text detection and recognition using augmented reality and deep learning. In: International conference on advanced information networking and applications, pp 13–23. Springer

  33. Ouali I, Halima MB, Wali A (2022) Real-time application for recognition and visualization of arabic words with vowels based dl and ar. In: 2022 18th International wireless communications & mobile computing conference (IWCMC), pp 678–683. IEEE

  34. Pechwitz M, El Abed H, Märgner V (2012) Handwritten arabic word recognition using the ifn/enit-database. In: Guide to OCR for Arabic scripts, pp 169–213. Springer

  35. Pei Y, Wu Y, Wang S, Wang F, Jiang H, Xu S, Zhou J (2019) Wa vis: A web-based augmented reality text data visual analysis tool. In: 2019 International conference on virtual reality and visualization (ICVRV), pp 11–17. IEEE

  36. Peng F, Zhai J (2017) A mobile augmented reality system for exhibition hall based on vuforia. In: 2017 2nd International conference on image, vision and computing (ICIVC), pp 1049–1052. IEEE

  37. Safabakhsh R, Adibi P (2005) Nastaaligh handwritten word recognition using a continuous-density variable-duration hmm. Arab J Sci Eng 30(1):95–120

    Google Scholar 

  38. Selmi Z, Halima MB, Alimi AM (2017) Deep learning system for automatic license plate detection and recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 1, pp 1132–1138. IEEE

  39. Selmi Z, Halima MB, Wali A, Alimi AM (2017) A framework of text detection and recognition from natural images for mobile device. In: Ninth international conference on machine vision (ICMV 2016), vol 10341, pp 1034127. International Society for Optics and Photonics

  40. Sheehan S, Luz S, Masoodian M (2021) Temotopic: temporal mosaic visualisation of topic distribution, keywords, and context. In: Proceedings of the EACL Hackashop on news media content analysis and automated report generation, pp 56–61

  41. Syahidi AA, Tolle H, Supianto AA, Arai K (2018) Bandoar: Real-time text based detection system using augmented reality for media translator banjar language to indonesian with smartphone. In: 2018 IEEE 5th international conference on engineering technologies and applied sciences (ICETAS), pp 1–6. IEEE

  42. Turki H, Halima MB, Alimi AM (2016) Text detection in natural scene images using two masks filtering. In: 2016 IEEE/ACS 13th international conference of computer systems and applications (AICCSA), pp 1–6. IEEE

  43. Turki H, Halima MB, Alimi AM (2017) Text detection based on mser and cnn features. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 1, pp 949–954. IEEE

  44. Turki H, Halima MB, Alimi AM (2017) Text detection based on mser and cnn features. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 1, pp 949–954. IEEE

  45. Yan R, Peng L, Bin G, Wang S, Cheng Y (2017) Residual recurrent neural network with sparse training for offline arabic handwriting recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), vol 1, pp 1031–1037. IEEE

  46. Zayene O, Hennebert J, Touj SM, Ingold R, Amara NEB (2015) A dataset for arabic text detection, tracking and recognition in news videos-activ. In: 2015 13th International conference on document analysis and recognition (ICDAR), pp 996–1000. IEEE

Download references

Acknowledgements

All the authors are deeply grateful to the editors and reviewers for their handling of this manuscript.

Author information

Authors and Affiliations

  1. REGIM: REsearch Groups on Intelligent Machines, National Engineering School of Sfax (ENIS), University of Sfax, Sfax, Tunisia

    Imene Ouali, Mohamed Ben Halima & Ali Wali

Authors
  1. Imene Ouali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Ben Halima
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ali Wali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Imene Ouali.

Ethics declarations

Conflict of Interests

We declare that we have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ouali, I., Halima, M.B. & Wali, A. An augmented reality for an arabic text reading and visualization assistant for the visually impaired. Multimed Tools Appl 82, 43569–43597 (2023). https://doi.org/10.1007/s11042-023-14880-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14880-6

Keywords

Search

Navigation