Integration of a BCI with a Hand Tracking System and a Motorized Robotic Arm to Improve Decoding of Brain Signals Related to Hand and Finger Movements | SpringerLink
Skip to main content
Springer Nature Link
Log in

Integration of a BCI with a Hand Tracking System and a Motorized Robotic Arm to Improve Decoding of Brain Signals Related to Hand and Finger Movements

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13017))

Included in the following conference series:

Abstract

Nowadays, most of the functional relationships between brain signals and the corresponding complex hand/fingers movements (cause, effect, feedback) are not yet completely understood. In the last years it has been assisted to important advances in Brain Computer Interfaces (BCI), computer vision (CV)-based tracking systems and Robots, especially due to: hardware improvements and miniaturization; increasing pursuit of intelligent and real time tracking systems; fast design, prototyping and production of consumer robots facilitated by 3D printing technologies.

We present an integrated system composed by a BCI, a CV-based hand tracking system and a motorized robotic arm reproducing the hand and the forearm of a person (scale 1:1). The proposal is to synchronize and to monitor the brain activity during complex hand and fingers movements (interpreted and reproduced in real time on a numerical hand model by the tracking system). Further, we aim at recognizing the brain signals which give rise to specific movements and, finally, at using them for producing the corresponding movements on the robotic arm. Different scenarios and potential use-cases are reported and their usefulness discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9380
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11725
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ameur, S., Ben Khalifa, A., Bouhlel, M.S.: A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with leap motion. Entertain. Comput. 35, 1–10 (2020)

    Article  Google Scholar 

  2. Boostani, R., Moradi, M.H.: Evaluation of the forearm EMG signal features for the control of a prosthetic hand. Physiol. Meas. 24(2), 309–319 (2003). https://doi.org/10.1088/0967-3334/24/2/307

    Article  Google Scholar 

  3. Breitwieser, C., Kreilinger, A., Neuper, C., Müller-Putz, G.: The TOBI hybrid BCI-the data acquisition module. In: Proceedings of the First TOBI Workshop, vol. 58 (2010)

    Google Scholar 

  4. Carrieri, M., et al.: Prefrontal cortex activation upon a demanding virtual hand-controlled task: a new frontier for neuroergonomics. Front. Hum. Neurosci. 10(53), 1–13 (2016)

    Google Scholar 

  5. Chatzis, T., Stergioulas, A., Konstantinidis, D., Dimitropoulos, K., Daras, P.: A comprehensive study on deep learning-based 3D hand pose estimation methods. Appl. Sci. 10(19), 6850 (2020). https://doi.org/10.3390/app10196850

    Article  Google Scholar 

  6. Coyle, S.M., Ward, T.E., Markham, C.M.: Brain-computer interface using a simplified functional near-infrared spectroscopy system. J. Neural Eng. 4(3), 219–226 (2007). https://doi.org/10.1088/1741-2560/4/3/007

    Article  Google Scholar 

  7. Devaraja, R.R., Maskeliūnas, R., Damaševičius, R.: Design and evaluation of anthropomorphic robotic hand for object grasping and shape recognition. Computers 10(1), 1 (2020). https://doi.org/10.3390/computers10010001

    Article  Google Scholar 

  8. Di Giamberardino, P., Iacoviello, D., Placidi, G., Polsinelli, M., Spezialetti, M.: A brain computer interface by EEG signals from self-induced emotions. In: Tavares, J.M.R.S., Natal Jorge, R.M. (eds.) ECCOMAS 2017. LNCVB, vol. 27, pp. 713–721. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-68195-5_77

    Chapter  Google Scholar 

  9. Erden, F., Çetin, A.E.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014)

    Article  Google Scholar 

  10. Franchi, D., Maurizi, A., Placidi, G.: Characterization of a SimMechanics model for a virtual glove rehabilitation system. In: Barneva, R.P., Brimkov, V.E., Hauptman, H.A., Natal Jorge, R.M., Tavares, J.M.R.S. (eds.) CompIMAGE 2010. LNCS, vol. 6026, pp. 141–150. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12712-0_13

    Chapter  Google Scholar 

  11. Halsband, U., Lange, R.K.: Motor learning in man: a review of functional and clinical studies. J. Physiol.-Paris 99(4–6), 414–424 (2006). https://doi.org/10.1016/j.jphysparis.2006.03.007

    Article  Google Scholar 

  12. Hoshi, E.: Cortico-basal ganglia networks subserving goal-directed behavior mediated by conditional visuo-goal association. Front. Neural Circ. 7, 158 (2013). https://doi.org/10.3389/fncir.2013.00158

    Article  Google Scholar 

  13. Iacoviello, D., Pagnani, N., Petracca, A., Spezialetti, M., Placidi, G.: A poll oriented classifier for affective brain computer interfaces. In: Proceedings of the 3rd International Congress on Neurotechnology, Electronics and Informatics - Volume 1: NEUROTECHNIX, pp. 41–48 (2015)

    Google Scholar 

  14. Iacoviello, D., Petracca, A., Spezialetti, M., Placidi, G.: A classification algorithm for electroencephalography signals by self-induced emotional stimuli. IEEE Trans. Cybern. 46(12), 3171–3180 (2016)

    Article  Google Scholar 

  15. Kiselev, V., Khlamov, M., Chuvilin, K.: Hand gesture recognition with multiple leap motion devices. In: 2019 24th Conference of Open Innovations Association (FRUCT), pp. 163–169. IEEE (2019)

    Google Scholar 

  16. Li, T., Xue, T., Wang, B., Zhang, J.: Decoding voluntary movement of single hand based on analysis of brain connectivity by using EEG signals. Front. Hum. Neurosci. 12, 381 (2018). https://doi.org/10.3389/fnhum.2018.00381

    Article  Google Scholar 

  17. Liao, K., Xiao, R., Gonzalez, J., Ding, L.: Decoding individual finger movements from one hand using human EEG signals. PLoS ONE 9(1), e85192 (2014). https://doi.org/10.1371/journal.pone.0085192

    Article  Google Scholar 

  18. Marin, G., Dominio, F., Zanuttigh, P.: Hand gesture recognition with jointly calibrated leap motion and depth sensor. Multimedia Tools Appl. 75(22), 14991–15015 (2016)

    Article  Google Scholar 

  19. Mick, S., et al.: Reachy, a 3D-printed human-like robotic arm as a testbed for human-robot control strategies. Front. Neurorobotics 13, 65 (2019). https://doi.org/10.3389/fnbot.2019.00065

    Article  Google Scholar 

  20. Miller, K.J., Schalk, G., Fetz, E.E., den Nijs, M., Ojemann, J.G., Rao, R.P.: Cortical activity during motor execution, motor imagery, and imagery-based online feedback. Proc. Natl. Acad. Sci. 107(9), 4430–4435 (2010). https://doi.org/10.1073/pnas.0913697107

    Article  Google Scholar 

  21. Moro, S.B., et al.: A novel semi-immersive virtual reality visuo-motor task activates ventrolateral prefrontal cortex: a functional near-infrared spectroscopy study. J. Neural Eng. 13(3), 1–14 (2016)

    Article  Google Scholar 

  22. Müller-Putz, G.R., Schwarz, A., Pereira, J., Ofner, P.: From classic motor imagery to complex movement intention decoding. In: Progress in Brain Research, pp. 39–70. Elsevier (2016). https://doi.org/10.1016/bs.pbr.2016.04.017

  23. Petracca, A., et al.: A virtual ball task driven by forearm movements for neuro-rehabilitation. In: 2015 International Conference on Virtual Rehabilitation (ICVR). pp. 162–163 (2015). https://doi.org/10.1109/ICVR.2015.7358600

  24. Placidi, G.: A smart virtual glove for the hand telerehabilitation. Comput. Biol. Med. 37(8), 1100–1107 (2007)

    Article  Google Scholar 

  25. Placidi, G., Avola, D., Cinque, L., Polsinelli, M., Theodoridou, E., Tavares, J.M.R.S.: Data integration by two-sensors in a LEAP-based virtual glove for human-system interaction. Multimedia Tools Appl. 80(12), 18263–18277 (2021). https://doi.org/10.1007/s11042-020-10296-8

    Article  Google Scholar 

  26. Placidi, G., Avola, D., Iacoviello, D., Cinque, L.: Overall design and implementation of the virtual glove. Comput. Biol. Med. 43(11), 1927–1940 (2013)

    Article  Google Scholar 

  27. Placidi, G., Avola, D., Petracca, A., Sgallari, F., Spezialetti, M.: Basis for the implementation of an EEG-based single-trial binary brain computer interface through the disgust produced by remembering unpleasant odors. Neurocomputing 160((C)), 308–318 (2015)

    Article  Google Scholar 

  28. Placidi, G., Cinque, L., Petracca, A., Polsinelli, M., Spezialetti, M.: A virtual glove system for the hand rehabilitation based on two orthogonal leap motion controllers. In: Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, pp. 184–192. INSTICC, SciTePress (2017)

    Google Scholar 

  29. Placidi, G., Cinque, L., Polsinelli, M.: A fast and scalable framework for automated artifact recognition from EEG signals represented in scalp topographies of independent components. Comput. Biol. Med. 132, 104347 (2021). https://doi.org/10.1016/j.compbiomed.2021.104347

    Article  Google Scholar 

  30. Placidi, G., Cinque, L., Polsinelli, M., Spezialetti, M.: Measurements by a leap-based virtual glove for the hand rehabilitation. Sensors 18(3), 1–13 (2018)

    Article  Google Scholar 

  31. Placidi, G., Giamberardino, P.D., Petracca, A., Spezialetti, M., Iacoviello, D.: Classification of emotional signals from the DEAP dataset. In: Proceedings of the 4th International Congress on Neurotechnology, Electronics and Informatics. SCITEPRESS - Science and Technology Publications (2016). https://doi.org/10.5220/0006043400150021

  32. Placidi, G., Petracca, A., Spezialetti, M., Iacoviello, D.: Classification strategies for a single-trial binary brain computer interface based on remembering unpleasant odors. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 7019–7022 (2015)

    Google Scholar 

  33. Placidi, G., Petracca, A., Spezialetti, M., Iacoviello, D.: A modular framework for EEG web based binary brain computer interfaces to recover communication abilities in impaired people. J. Med. Syst. 40(1), 34 (2016)

    Article  Google Scholar 

  34. Shen, H., Yang, X., Hu, H., Mou, Q., Lou, Y.: Hand trajectory extraction of human assembly based on multi-leap motions. In: 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 193–198 (2019)

    Google Scholar 

  35. Spezialetti, M., Cinque, L., Tavares, J.M.R., Placidi, G.: Towards EEG-based BCI driven by emotions for addressing BCI-illiteracy: a meta-analytic review. Behav. Inf. Technol. 37(8), 855–871 (2018). https://doi.org/10.1080/0144929x.2018.1485745

    Article  Google Scholar 

  36. Kin Tam, W., Wu, T., Zhao, Q., Keefer, E., Yang, Z.: Human motor decoding from neural signals a review. BMC Biomed. Eng. 1(1), 22 (2019). https://doi.org/10.1186/s42490-019-0022-z

    Article  Google Scholar 

  37. Townsend, G., Graimann, B., Pfurtscheller, G.: Continuous EEG classification during motor imagery–simulation of an asynchronous BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 12(2), 258–265 (2004). https://doi.org/10.1109/tnsre.2004.827220

    Article  Google Scholar 

  38. Vourvopoulos, A., Bermúdez i Badia, S.: Motor priming in virtual reality can augment motor-imagery training efficacy in restorative brain-computer interaction: a within-subject analysis. J. NeuroEngineering Rehabil. 13(1), 69 (2016). https://doi.org/10.1186/s12984-016-0173-2

    Article  Google Scholar 

  39. Wolpaw, J.R., McFarland, D.J.: Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. 101(51), 17849–17854 (2004). https://doi.org/10.1073/pnas.0403504101

    Article  Google Scholar 

  40. Yang, L., Chen, J., Zhu, W.: Dynamic hand gesture recognition based on a leap motion controller and two-layer bidirectional recurrent neural network. Sensors 20, 2106–2123 (2020)

    Article  Google Scholar 

  41. Yoo, S.S., et al.: Brain-computer interface using fMRI: spatial navigation by thoughts. NeuroReport 15(10), 1591–1595 (2004). https://doi.org/10.1097/01.wnr.0000133296.39160.fe

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. A2VI-Lab, c/o Department of MeSVA, University of L’Aquila, Via Vetoio Coppito, 67100, L’Aquila, Italy

    Giuseppe Placidi & Matteo Polsinelli

  2. Department of DISIM, University of L’Aquila, Via Vetoio Coppito, 67100, L’Aquila, Italy

    Giovanni De Gasperis, Filippo Mignosi & Matteo Spezialetti

Authors
  1. Giuseppe Placidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giovanni De Gasperis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Filippo Mignosi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matteo Polsinelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matteo Spezialetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giuseppe Placidi .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. University of Texas at Arlington, Arlington, TX, USA

    Vassilis Athitsos

  3. University of South Carolina, Columbia, SC, USA

    Tong Yan

  4. City University of Hong Kong, Kowloon, Hong Kong

    Manfred Lau

  5. School of Engineering and Computing, University of Durham, Durham, Durham, UK

    Frederick Li

  6. Airbnb, New York, NY, USA

    Conglei Shi

  7. Peking University, Beijing, China

    Xiaoru Yuan

  8. Purdue University, West Lafayette, IN, USA

    Christos Mousas

  9. IST, School of Modeling, Simulation, and Training, Orlando, FL, USA

    Gerd Bruder

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Placidi, G., De Gasperis, G., Mignosi, F., Polsinelli, M., Spezialetti, M. (2021). Integration of a BCI with a Hand Tracking System and a Motorized Robotic Arm to Improve Decoding of Brain Signals Related to Hand and Finger Movements. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2021. Lecture Notes in Computer Science(), vol 13017. Springer, Cham. https://doi.org/10.1007/978-3-030-90439-5_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90439-5_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90438-8

  • Online ISBN: 978-3-030-90439-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation