Abstract
Recently hand gesture recognition via millimeter-wave radar has attracted a lot of research attention for human-computer interaction. Encouraged by the ability of deep learning models in successfully tackling hand gesture recognition tasks, we propose a deep neural network (DNN) model namely, Res3DTENet that aims to classify dynamic hand gestures using the radio frequency (RF) signals. We propose a scheme that improves the convolutional process of 3DCNNs with residual skip connection (Res3D) to emphasize local-global information and enriches the intra-frame spatio-temporal feature representation. A multi-head attention transformer encoder (TE) network has been trained over the spatio-temporal features to refine the inter-frame temporal dependencies of range-Doppler sequences. The experiments are carried out on the publicly available Soli hand gesture data set. Based on our extensive experiments, we show that the proposed network achieves improved gesture recognition accuracy than the state-of-the-art hand gesture recognition methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amin, M.G., Zeng, Z., Shan, T.: Hand gesture recognition based on radar micro-doppler signature envelopes. In: IEEE Radar Conference (RadarConf), pp. 1–6 (2019)
Berenguer, A.D., Oveneke, M.C., Alioscha-Perez, M., Bourdoux, A., Sahli, H., et al.: Gesturevlad: combining unsupervised features representation and spatio-temporal aggregation for doppler-radar gesture recognition. IEEE Access 7, 137122–137135 (2019)
Chen, K.S.: Principles of Synthetic Aperture Radar Imaging: A System Simulation Approach, vol. 2. CRC Press, Boca Raton (2016)
Choi, J.W., Ryu, S.J., Kim, J.H.: Short-range radar based real-time hand gesture recognition using LSTM encoder. IEEE Access 7, 33610–33618 (2019)
Hazra, S., Santra, A.: Short-range radar-based gesture recognition system using 3D CNN with triplet loss. IEEE Access 7, 125623–125633 (2019)
Lien, J., et al.: Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Trans. Graph. (TOG) 35, 1–19 (2016)
Park, J., Cho, S.H.: IR-UWB radar sensor for human gesture recognition by using machine learning. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pp. 1246–1249. IEEE (2016)
Sakamoto, T., Gao, X., Yavari, E., Rahman, A., Boric-Lubecke, O., Lubecke, V.M.: Radar-based hand gesture recognition using IQ echo plot and convolutional neural network. In: IEEE Conference on Antenna Measurements & Applications (CAMA), pp. 393–397 (2017)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Wang, S., Song, J., Lien, J., Poupyrev, I., Hilliges, O.: Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST), pp. 851–860 (2016)
Wang, Y., Wang, S., Zhou, M., Jiang, Q., Tian, Z.: TS-I3D based hand gesture recognition method with radar sensor. IEEE Access 7, 22902–22913 (2019)
Xia, Z., Luomei, Y., Zhou, C., Xu, F.: Multidimensional feature representation and learning for robust hand-gesture recognition on commercial millimeter-wave radar. IEEE Trans. Geosci. Remote Sens. 1–16 (2020). https://doi.org/10.1109/TGRS.2020.3010880
Zhang, J., Tao, J., Shi, Z.: Doppler-radar based hand gesture recognition system using convolutional neural networks. In: International Conference in Communications, Signal Processing, and Systems, pp. 1096–1113 (2017)
Zhang, Z., Tian, Z., Zhang, Y., Zhou, M., Wang, B.: u-DeepHand: FMCW radar-based unsupervised hand gesture feature learning using deep convolutional auto-encoder network. IEEE Sens. J. 19(16), 6811–6821 (2019)
Zhang, Z., Tian, Z., Zhou, M.: Latern: dynamic continuous hand gesture recognition using FMCW radar sensor. IEEE Sens. J. 18(8), 3278–3289 (2018)
Acknowledgement
This research was partially supported by the Semiconductor Research Corporation (SRC) Grant 2019-IR-2924.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Jaswal, G., Srirangarajan, S., Roy, S.D. (2021). Range-Doppler Hand Gesture Recognition Using Deep Residual-3DCNN with Transformer Network. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12666. Springer, Cham. https://doi.org/10.1007/978-3-030-68780-9_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-68780-9_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68779-3
Online ISBN: 978-3-030-68780-9
eBook Packages: Computer ScienceComputer Science (R0)
