Gesture Recognition of Filipino Sign Language Using Convolutional and Long-Short Term Memory Neural Network

Abstract

Sign language is a form of communication prominently used by the deaf-mute community to convey their ideas and thoughts. In the Philippines, local signers use Filipino Sign Language (FSL) derived from the well-known American Sign Language (ASL). Despite the recent formalization of FSL as the country’s official sign language, there is still minimal familiarity among the public. That said, Sign Language Recognition (SLR) systems integrated with machine learning applications have been developed to understand FSL better. However, the prevalent limitations of most of these systems are that it only involves static signs and asynchronous recognition. This study aimed to take this solution further and overcome existing limitations by developing a model capable of recognizing FSL gestures in real-time usable for applications such as in government service centers. To this end, the study proposes the deep learning algorithm Convolutional and Long Short-Term Memory Neural Networks in system capturing of real-time signs from a signer. The proponents considered 15 signs related to common greetings and business transactions. A total of 450 video recordings were collected for the signs with each having an equal number of samples. The collected data underwent cleaning, preprocessing, and augmentation before training. The proposed model’s performance was analyzed with the following classification metrics: Accuracy, Precision, Recall, and F1-Score, and was able to achieve 95% accuracy and a macro-average of 0.95 precision, 0.95 Recall, and 0.95 F1-Score. Furthermore, the model had a comparable accuracy and loss between validation and test data—a 95.18% accuracy and 0.13629 loss on validation while 95.93% accuracy and loss of 0.1478 on the test. With that said, the proposed model was well-fit for classifying the 15 signs that involve upper body movements.