Multimodal Sign Language Recognition System: Integrating Image Processing and Deep Learning for Enhanced Communication Accessibility

doi:10.23940/ijpe.24.05.p2.271281

Abstract

Abstract: Communication for individuals who are hearing- and speech-impaired, commonly referred to as the deaf and mute community, heavily relies on sign language as their primary mode of expression. This study presents a novel framework leveraging image processing techniques for the detection and recognition of sign language gestures. The developed software offers promising avenues for enhancing comprehension of Sign Language, with potential applications in educational settings, public spaces, and interpersonal interactions. The proposed method streamlines the recognition process of sign language, employing deep learning algorithms for the accurate prediction of signs. The system operates by processing input images containing signs through a convolutional neural network, encompassing stages such as pre-processing, feature extraction, model training, testing, and sign-to-text conversion. Crucially, the system's output provides text-based descriptions of the sign in the input image and notably integrates voice output for enhanced accessibility and communication. This multifaceted approach contributes towards bridging communication barriers between individuals with disabilities and those without, promoting inclusivity and understanding in diverse social contexts.

Key words: framework, sign language, convolutional neural network, gestures

Mukta Jagdish and Valliappan Raju. Multimodal Sign Language Recognition System: Integrating Image Processing and Deep Learning for Enhanced Communication Accessibility [J]. Int J Performability Eng, 2024, 20(5): 271-281.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Nguyen, H.B. and Do, H.N., 2019, April. Deep learning for american sign language fingerspelling recognition system. In 2019 26th International Conference on Telecommunications (ICT)(pp. 314-318). IEEE.
[2] Hema B.N., Anjum S., Hani U., Vanaja P. and Akshatha M., 2019. Sign Language and Gesture Recognition for Deaf and Dumb People. International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056,6(03).
[3] Rajamohan A., Hemavathy R. and Dhanalakshmi M., 2013. Deaf-mute communication interpreter. International Journal of Scientific Engineering and Technology,2(5), pp.336-341.
[4] Dong C., Leu M.C. and Yin Z., 2015. American sign language alphabet recognition using microsoft kinect. InProceedings of the IEEE conference on computer vision and pattern recognition workshops(pp. 44-52).
[5] Panwar, M. and Mehra, P.S., 2011, November. Hand gesture recognition for human computer interaction. In 2011 International Conference on Image Information Processing(pp. 1-7). IEEE.
[6] Oniga S., Tisan A., Mic D., Buchman A. and Vida-Ratiu A., 2007, May. Hand postures recognition system using artificial neural networks implemented in FPGA. In 2007 30th International Spring Seminar on Electronics Technology (ISSE)(pp. 507-512). IEEE.
[7] Mahmoudi, F. and Parviz, M., 2006, July. Visual hand tracking algorithms. In Geometric Modeling and Imaging--New Trends (GMAI'06)(pp. 228-232). IEEE.
[8] Bao P.T., Binh N.T. and Khoa T.D., 2009, August. A new approach to hand tracking and gesture recognition by a new feature type and HMM. In 2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery (Vol. 4, pp. 3-6). IEEE.
[9] Yun, L. and Peng, Z., 2009, October. An automatic hand gesture recognition system based on Viola-Jones method and SVMs. In 2009 Second international workshop on computer science and engineering (Vol. 2, pp. 72-76). IEEE.
[10] Ghotkar A.S., Khatal R., Khupase S., Asati S. and Hadap M., 2012, January. Hand gesture recognition for indian sign language. In 2012 International Conference on Computer Communication and Informatics(pp. 1-4). IEEE.
[11] Gunawan H., Thiracitta N. and Nugroho A., 2018, September. Sign language recognition using modified convolutional neural network model. In 2018 Indonesian Association for Pattern Recognition International Conference (INAPR)(pp. 1-5). IEEE.
[12] Gupta B., Shukla P. and Mittal A., 2016, January. K-nearest correlated neighbor classification for Indian sign language gesture recognition using feature fusion. In 2016 International conference on computer communication and informatics (ICCCI)(pp. 1-5). IEEE.
[13] Mariappan, H.M. and Gomathi, V., 2019, February. Real-time recognition of Indian sign language. In 2019 international conference on computational intelligence in data science (ICCIDS)(pp. 1-6). IEEE.

[1]	Rohit Chandra Joshi, Aayush Juyal, Abhijeet Mishra, Avni Verma, and Kanika Singla. Deep Learning-Based Face Emotion Recognition: A Comparative Study [J]. Int J Performability Eng, 2024, 20(1): 1-9.
[2]	C. Rohith Bhat and Madhusundar Nelson. Artificial Intelligence Based Credit Card Fraud Detection for Online Transactions Optimized with Sparrow Search Algorithm [J]. Int J Performability Eng, 2023, 19(9): 624-632.
[3]	Neha Kohli and Tapas Kumar. Envisaging Alzheimer’s Disease Stage through Fuzzy Rank-Based Ensemble of Transfer Learning Models [J]. Int J Performability Eng, 2023, 19(6): 397-406.
[4]	Manvi Khatri and Ajay Sharma. Deep Learning Approach based on Iris, Face, and Palmprint Fusion for Multimodal Biometric Recognition System [J]. Int J Performability Eng, 2023, 19(6): 407-416.
[5]	Pavneet Singh, Jigyasa Chopra, Amandeep Singh, Nikita Nijhawan, and Kritika. Deep Learning Innovations for Enhanced Drusen Detection in Retinal Images [J]. Int J Performability Eng, 2023, 19(12): 779-787.
[6]	Saumya Kumar, Puneet Goswami, and Shivani Batra. Enriched Diagnosis of Osteoporosis using Deep Learning Models [J]. Int J Performability Eng, 2023, 19(12): 824-833.
[7]	Sonika Jindal, Monika Sachdeva, and Alok Kumar Singh Kushwaha. Human Activity Recognition using Ensemble Convolutional Neural Networks and Long Short-Term Memory [J]. Int J Performability Eng, 2022, 18(9): 660-667.
[8]	Roop Preet Kaur, Anshu Sharma, Inderpal Singh, and Rahul Malhotra. Deep Learning-Based Pneumonia Recognition from Chest X-Ray Images [J]. Int J Performability Eng, 2022, 18(5): 380-386.
[9]	Lei Su, Fan Yang, Yu Shen, and Zhichun Yang. Two-Level Assessment Method for Electrical Fire Risk of High-Rise Buildings based on Interval TOPSIS Method [J]. Int J Performability Eng, 2022, 18(3): 167-175.
[10]	Revati M. Wahul, Archana P. Kale, and Abhishek D. Patange. Health Monitoring of Turning Tool through Vibration Signals Processed using Convolutional Neural Network Architecture [J]. Int J Performability Eng, 2022, 18(10): 730-740.
[11]	Angel Arul Jothi J and Razia Sulthana A. A Review on the Literature of Fashion Recommender System using Deep Learning [J]. Int J Performability Eng, 2021, 17(8): 695-702.
[12]	Chhabra Megha, Shukla Manoj Kumar, and Ravulakolluc Kiran Kumar. Intelligent Optimization of Latent Fingerprint Image Segmentation using Stacked Convolutional Autoencoder [J]. Int J Performability Eng, 2021, 17(4): 379-393.
[13]	Sofiane Maza and Oussama Megouas. Framework for Trustworthiness in Software Development [J]. Int J Performability Eng, 2021, 17(2): 241-252.
[14]	Abdul Ghafoor Etemad, Ali Imam Abidi, and Megha Chhabra. Fine-Tuned T5 for Abstractive Summarization [J]. Int J Performability Eng, 2021, 17(10): 900-906.
[15]	Zhifeng Zhang, Xiao Cui, Pu Li, Jintao Jiang, and Xiaohui Ji. Hyperspectral Data Analysis based on Integrated Deep Learning [J]. Int J Performability Eng, 2020, 16(8): 1225-1234.