Abstract
In this paper, we present a framework for recognition of static finger spelling in Japanese Sign Language on RGB images. The finger spelled signs were recognized by an ensemble consisting of a ResNet-based convolutional neural network and two ResNet quaternion convolutional neural networks. A 3D articulated hand model has been used to generate synthetic finger spellings and to extend a dataset consisting of real hand gestures. Twelve different gesture realizations were prepared for each of 41 signs. Ten images have been rendered for each realization through interpolations between the starting and end poses. Experimental results demonstrate that owing to sufficient amount of training data a high recognition rate can be attained on images from a single RGB camera. Results achieved by the ResNet quaternion convolutional neural network are better than results obtained by the ResNet CNN. The best recognition results were achieved by the ensemble. The JSL-rend dataset is available for download.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sagayam, M., Hemanth, J.: Hand posture and gesture recognition techniques for virtual reality applications: a survey. Virtual Reality 21(2), 91–107 (2017)
Chen, F., Zhong, Q., Cannella, F., Sekiyama, K., Fukuda, T.: Hand gesture modeling and recognition for human and robot interactive assembly using Hidden Markov Models. Int. J. Adv. Rob. Syst. 12(4), 48 (2015)
Raj, M.D., Gogul, I., Thangaraja, M., Kumar, V.: Static gesture recognition based precise positioning of 5-DOF robotic arm using FPGA. In: Trends in Industrial Measurement and Automation (TIMA), pp. 1–6 (2017)
Liu, H., Wang, L.: Gesture recognition for human-robot collaboration: a review. Int. J. Ind. Ergon. 68, 355–367 (2018)
Patil, S., et al.: GesturePod: programmable gesture recognition for augmenting assistive devices, Technical report, Microsoft, May 2018
Rautaray, S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015)
Al-Shamayleh, A.S., Ahmad, R., Abushariah, M., Alam, K.A., Jomhari, N.: A systematic literature review on vision based gesture recognition techniques. Multimedia Tools Appl. 77(21), 28121–28184 (2018)
Ohn-Bar, E., Trivedi, M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014)
Pisharady, P., Saerbeck, M.: Recent methods and databases in vision-based hand gesture recognition. Comput. Vis. Image Underst. 141, 152–165 (2015)
Oyedotun, O., Khashman, A.: Deep learning in vision-based static hand gesture recognition. Neural Comput. Appl., 1–11 (2016)
Tompson, J., Stein, M., LeCun, Y., Perlin, K.: Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph. 33(5) (2014)
Nagi, J., Ducatelle, F., et al.: Max-pooling convolutional neural networks for vision-based hand gesture recognition. In: IEEE ICSIP, pp. 342–347 (2011)
Barros, P., Magg, S., Weber, C., Wermter, S.: A multichannel convolutional neural network for hand posture recognition. In: Wermter, S., et al. (eds.) ICANN 2014. LNCS, vol. 8681, pp. 403–410. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11179-7_51
Koller, O., Ney, H., Bowden, R.: Deep hand: how to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3793–3802 (2016)
Tabata, Y., Kuroda, T.: Finger spelling recognition using distinctive features of hand shape. In: International Conference on Disability, Virtual Reality and Associated Technologies with Art Abilitation, pp. 287–292 (2008)
Kane, L., Khanna, P.: A framework for live and cross platform fingerspelling recognition using modified shape matrix variants on depth silhouettes. Comput. Vis. Image Underst. 141, 138–151 (2015)
Kwolek, B., Sako, S.: Learning siamese features for finger spelling recognition. In: Blanc-Talon, J., Penne, R., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2017. LNCS, vol. 10617, pp. 225–236. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70353-4_20
Rosalina, L.Y., Hadisukmana, N., Wahyu, R.B., Roestam, R., Wahyu, Y.: Implementation of real-time static hand gesture recognition using artificial neural network. In: CAIPT, pp. 1–6 (2017)
Asad, M., Slabaugh, G.: SPORE: staged probabilistic regression for hand orientation inference. Comput. Vis. Image Underst. 161, 114–129 (2017)
Dawod, A.Y., Nordin, M.J., Abdullah, J.: Static fingerspelling recognition based on boundary tracing algorithm and chain code. In: International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, pp. 104–109. ACM (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Parcollet, T., et al.: Quaternion convolutional neural networks for end-to-end automatic speech recognition. In: Interspeech, ISCA, pp. 22–26 (2018)
Popa, C.A.: Learning algorithms for quaternion-valued neural networks. Neural Process. Lett. 47(3), 949–973 (2018)
Nitta, T.: A quaternary version of the back-propagation algorithm. In: Proceedings of International Conference on Neural Networks, vol. 5, pp. 2753–2756 (1995)
Zhu, X., Xu, Y., Xu, H., Chen, C.: Quaternion convolutional neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 645–661. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_39
Opitz, D., Maclin, R.: Popular ensemble methods: an empirical study. J. Artif. Int. Res. 11(1), 169–198 (1999)
Acknowledgments
This work was supported by Polish National Science Center (NCN) under a research grant 2017/27/B/ST6/01743 and JSPS KAKENHI under a grant 17H06114.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Nguen, N.T., Sako, S., Kwolek, B. (2019). Deep CNN-Based Recognition of JSL Finger Spelling. In: Pérez García, H., Sánchez González, L., Castejón Limas, M., Quintián Pardo, H., Corchado Rodríguez, E. (eds) Hybrid Artificial Intelligent Systems. HAIS 2019. Lecture Notes in Computer Science(), vol 11734. Springer, Cham. https://doi.org/10.1007/978-3-030-29859-3_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-29859-3_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29858-6
Online ISBN: 978-3-030-29859-3
eBook Packages: Computer ScienceComputer Science (R0)