Abstract
Recently, object detection technologies applied in rehabilitation systems are mainly based on the ready-made technology of CNNs. This paper proposes an DETR-based detector which is an end-to-end object detector with spatiotemporal context learning for machine-assisted rehabilitation. To improve the performance of small object detection, first, the multi-level features of the RepVGG are fused with the SE attention mechanism to build a SEFP-RepVGG. To make the encoder-decoder structure more suitable, next, the value of the encoder is generated by using feature maps with more detailed information than key/query. To reduce computation, Patch Merging is finally imported to modify the feature map scale of the input encoder. The proposed detector has higher real-time performance than DETR and obtains the competitive detection accuracy on the ImageNet VID benchmark. Some typical samples from the NTU RGB-D 60 dataset are selected to build a new limb-detection dataset for further evaluation. The results show the effectiveness of the proposed detector in the rehabilitation scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Kadu, A., Singh, M.: Comparative analysis of e-health care telemedicine system based on internet of medical things and artificial intelligence. In: 2nd International Conference on Smart Electronics and Communication (ICOSEC), pp. 1768–1775 (2021). https://doi.org/10.1109/ICOSEC51865.2021.9591941
Debnath, B., O’Brien, M., Yamaguchi, M., Behera, A.: A review of computer vision-based approaches for physical rehabilitation and assessment. Multimedia Syst. 28(1), 209–239 (2021). https://doi.org/10.1007/s00530-021-00815-4
Chae, S.H., Kim, Y., Lee, K.S., Park, H.S.: Development and clinical evaluation of a web-based upper limb home rehabilitation system using a smartwatch and machine learning model for chronic stroke survivors: prospective comparative study. JMIR Mhealth Uhealth 8(7), e17216 (2020). https://doi.org/10.2196/17216
Chiang, A.T., Chen, Q., Wang, Y., Fu, M.R.: Kinect-based in-home exercise system for lymphatic health and lymphedema intervention. IEEE J. Transl. Eng. Health Med. 6, 1–13 (2018). https://doi.org/10.1109/JTEHM.2018.2859992
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031
Redmon, J.,, Farhadi, A.: YOLOv3: an incremental improvement. eprint arXiv:1804.02767 (2018). https://doi.org/10.48550/arXiv.1804.02767
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: RepVGG: making VGG-style ConvNets great again. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13728–13737. IEEE, New York (2021). https://doi.org/10.1109/CVPR46437.2021.01352
Zhang, X., Gao, Y., Ye, F., Liu, Q., Zhang, K.: An approach to improve SSD through skip connection of multiscale feature maps. Comput. Intell. Neurosci. 2020, 13 (2020). https://doi.org/10.1155/2020/2936920
Yao, Z., Ai, J., Li, B., Zhang, C.: Efficient DETR: improving end-to-end object detector with dense prior. eprint arXiv: 2104.01318 (2021). https://doi.org/10.48550/arxiv.2104.01318
Li, S., et al.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. eprint arXiv: 1907.00235 (2019). https://doi.org/10.48550/arxiv.1907.00235
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002. IEEE, New York (2021). https://doi.org/10.48550/arxiv.2103.14030
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Shahroudy, A., Liu, J., Ng, T.-T., Wang, G.: NTU RGB+D: a large scale dataset for 3D human activity analysis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1010–1019. IEEE, New York (2016). https://doi.org/10.1109/CVPR.2016.115
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. eprint arXiv: 2010.04159 (2020). https://doi.org/10.48550/arxiv.2010.04159
Meng, D., et al.: Conditional DETR for fast training convergence. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3631–3640. IEEE, New York (2021). https://doi.org/10.1109/ICCV48922.2021.00363
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995. IEEE, New York (2017). https://doi.org/10.1109/CVPR.2017.634
Howard, A., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. eprint arXiv: 1704.04861 (2017). https://doi.org/10.48550/arxiv.1704.04861
Child, R., Gray, S., Radford, A., Sutskever, I: Generating long sequences with sparse transformers. eprint arXiv: 1904.10509 (2019). https://doi.org/10.48550/arxiv.1904.10509
Zhou, N.: Research on video object detection based on temporal characteristics. J. China Acad. Electron. Inf. 16(02), 157–164 (2021)
Chai, Y.: Patchwork: a patch-wise attention network for efficient object detection and segmentation in video streams. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3414–3423. IEEE, New York (2019). https://doi.org/10.1109/ICCV.2019.00351
Tan, M., Pang, R., Le, Q.V.: EfficientDet: scalable and efficient object detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10778–10787. IEEE, New York (2019). https://doi.org/10.1109/CVPR42600.2020.0107
Chen, X., Yu, J., Wu, Z.: Temporally identity-aware SSD with attentional LSTM. IEEE Trans. Cybern. 50(6), 2674–2686 (2020). https://doi.org/10.1109/TCYB.2019.2894261
Kang, K., et al.: Object detection in videos with tubelet proposal networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 889–897. IEEE, New York (2017). https://doi.org/10.1109/CVPR.2017.101
Acknowledgments
The authors would like to acknowledge the support from the AiBle project co-financed by the European Regional Development Fund, National Key R&D Program of China (Grant No. 2018YFB1304600), CAS Interdisciplinary Innovation Team (Grant No. JCTD-2018-11), Liaoning Province Higher Education Innovative Talents Program Support Project (Grant No. LR2019058), and National Natural Science Foundation of China (grant No. 52075530, 51575412, and 62006204). LiaoNing Province Joint Open Fund for Key Scientific and Technological Innovation Bases (Grant No. 2021-KF-12-05).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Gao, H., Ma, T., Yu, J. (2022). An End-to-End Object Detector with Spatiotemporal Context Learning for Machine-Assisted Rehabilitation. In: Liu, H., et al. Intelligent Robotics and Applications. ICIRA 2022. Lecture Notes in Computer Science(), vol 13455. Springer, Cham. https://doi.org/10.1007/978-3-031-13844-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-13844-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-13843-0
Online ISBN: 978-3-031-13844-7
eBook Packages: Computer ScienceComputer Science (R0)