Emerging Frontiers in Human–Robot Interaction | Journal of Intelligent & Robotic Systems
Skip to main content

Advertisement

Springer Nature Link
Log in
Menu
Find a journal Publish with us Track your research
Search
Cart
  1. Home
  2. Journal of Intelligent & Robotic Systems
  3. Article

Emerging Frontiers in Human–Robot Interaction

  • Review Paper
  • Open access
  • Published: 18 March 2024
  • Volume 110, article number 45, (2024)
  • Cite this article
Download PDF

You have full access to this open access article

Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript
Emerging Frontiers in Human–Robot Interaction
Download PDF
  • Farshad Safavi1,
  • Parthan Olikkal1,
  • Dingyi Pei1,
  • Sadia Kamal1,
  • Helen Meyerson1,
  • Varsha Penumalee1 &
  • …
  • Ramana Vinjamuri  ORCID: orcid.org/0000-0003-1650-55241 

  • 1778 Accesses

  • 6 Citations

  • Explore all metrics

Abstract

Effective interactions between humans and robots are vital to achieving shared tasks in collaborative processes. Robots can utilize diverse communication channels to interact with humans, such as hearing, speech, sight, touch, and learning. Our focus, amidst the various means of interactions between humans and robots, is on three emerging frontiers that significantly impact the future directions of human–robot interaction (HRI): (i) human–robot collaboration inspired by human–human collaboration, (ii) brain-computer interfaces, and (iii) emotional intelligent perception. First, we explore advanced techniques for human–robot collaboration, covering a range of methods from compliance and performance-based approaches to synergistic and learning-based strategies, including learning from demonstration, active learning, and learning from complex tasks. Then, we examine innovative uses of brain-computer interfaces for enhancing HRI, with a focus on applications in rehabilitation, communication, brain state and emotion recognition. Finally, we investigate the emotional intelligence in robotics, focusing on translating human emotions to robots via facial expressions, body gestures, and eye-tracking for fluid, natural interactions. Recent developments in these emerging frontiers and their impact on HRI were detailed and discussed. We highlight contemporary trends and emerging advancements in the field. Ultimately, this paper underscores the necessity of a multimodal approach in developing systems capable of adaptive behavior and effective interaction between humans and robots, thus offering a thorough understanding of the diverse modalities essential for maximizing the potential of HRI.

Article PDF

Download to read the full article text

Similar content being viewed by others

New Horizons in Human–Robot Interaction: Synergy, Cognition, and Emotion

Chapter © 2024

Social Robotics

Chapter © 2016

Emotionally Driven Robot Control Architecture for Human-Robot Interaction

Chapter © 2014

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.
  • Artificial Intelligence
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Code Availability

Not applicable.

References

  1. Goodrich, M.A., Schultz, A.C.: Human–Robot Interaction: A Survey. Foundations and Trends® in Human–Computer Interaction 1(3), 203–275 (2008). https://doi.org/10.1561/1100000005

  2. Vansteensel, M.J., Jarosiewicz, B.: Chapter 7 - brain-computer interfaces for communicationacurrent affiliation: Neuropace, inc., mountain view, ca, united states. Handbook of Clinical Neurology, vol. 168, pp. 67–85. Elsevier (2020). https://doi.org/10.1016/B978-0-444-63934-9.00007-X . https://www.sciencedirect.com/science/article/pii/B978044463934900007X

  3. Mehrabian, A.: Communication without words. (1968)

  4. Kaulard, K., Cunningham, D., Bu¨lthoff, H., Wallraven, C.: The MPI Facial Expression Database — A Validated Database of Emotional and Conversational Facial Expressions. PloS one 7, 32321 (2012) https://doi.org/10.1371/journal.pone.0032321

  5. Bhushan, K.: Mitra: The ’made in india’ robot that stole the show at ges hyderabad. Hindustan Times (2017)

  6. Marcos-Pablos, S., Garc´ıa-Pen˜alvo, F.: Emotional Intelligence in Robotics: A Scoping Review, pp. 66–75 (2022). https://doi.org/10.1007/978-3-030-87687-6 7

  7. Tariq, M., Trivailo, P.M., Simic, M.: Eeg-based bci control schemes for lower-limb assistive-robots. Front. Hum. Neurosci. 12, 312 (2018)

    Article  Google Scholar 

  8. Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009). https://doi.org/10.1016/j.robot.2008.10.024

    Article  Google Scholar 

  9. Billard, A., Calinon, S., Dillmann, R., Schaal, S.: Robot Programming by Demonstration. In: Springer Handbook of Robotics, pp. 1371–1394. Springer, Berlin, Heidelberg (2008). https://doi.org/10.1007/978-3-540-30301-560

  10. Amor, H.B., Vogt, D., Ewerton, M., Berger, E., Jung, B., Peters, J.: Learning responsive robot behavior by imitation. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3257–3264 (2013). https://doi.org/10.1109/IROS.2013.6696819

  11. Kosuge, K., Yoshida, H., Fukuda, T.: Dynamic control for robot-human collaboration. In: Proceedings of 1993 2nd IEEE International Workshop on Robot and Human Communication, pp. 398–401 (1993). https://doi.org/10.1109/ROMAN.1993.367685

  12. Kosuge, K., Kazamura, N.: Control of a robot handling an object in cooperation with a human. In: Proceedings 6th IEEE International Workshop on Robot and Human Communication. RO-MAN’97 SENDAI, pp. 142–147 (1997). https://doi.org/10.1109/ROMAN.1997.646971

  13. Ikeura, R., Inooka, H.: Variable impedance control of a robot for cooperation with a human. In: Proceedings of 1995 IEEE International Conference on Robotics and Automation, vol. 3, pp. 3097–31023 (1995). 1109/ROBOT.1995.525725

  14. Duchaine, V., Gosselin, C.M.: General Model of Human-Robot Cooperation Using a Novel Velocity Based Variable Impedance Control. In: Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07), pp. 446–451 (2007). https://doi.org/10.1109/WHC.2007.59

  15. Bussy, A., Gergondet, P., Kheddar, A., Keith, F., Crosnier, A.: Proactive behavior of a humanoid robot in a haptic transportation task with a human partner. In: Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, pp. 962–967 (2012). https://doi.org/10.1109/ROMAN.2012.6343874

  16. Agravante, D.J., Cherubini, A., Bussy, A., Kheddar, A.: Human-humanoid joint haptic table carrying task with height stabilization using vision. In: IEEE International Conference on Intelligent Robots and Systems, pp. 4609–4614 (2013). https://doi.org/10.1109/IROS.2013.6697019

  17. Corteville, B., Aertbelien, E., Bruyninckx, H., De Schutter, J., Van Brussel, H.: Human-inspired robot assistant for fast point-to-point movements. Technical report (2007)

  18. Maeda, Y., Hara, T., Arai, T.: Human-robot cooperative manipulation with motion estimation. IEEE Int. Conf. Intell. Robots. Syst. 4, 2240–2245 (2001). https://doi.org/10.1109/IROS.2001.976403

    Article  Google Scholar 

  19. Tsumugiwa, T., Yokogawa, R., Hara, K.: Variable impedance control based on estimation of human arm stiffness for human-robot cooperative calligraphic task. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), vol. 1, pp. 644–6501 (2002). https://doi.org/10.1109/ROBOT.2002.1013431

  20. Yang, C., Ganesh, G., Haddadin, S., Parusel, S., Albu-Schaeffer, A., Burdet, E.: Human-Like Adaptation of Force and Impedance in Stable and Unstable Interactions. IEEE Trans. Robot. 27(5), 918–930 (2011). https://doi.org/10.1109/TRO.2011.2158251

    Article  Google Scholar 

  21. Calinon, S., Evrard, P., Gribovskaya, E., Billard, A., Kheddar, A.: Learning collaborative manipulation tasks by demonstration using a haptic interface. In: 2009 International Conference on Advanced Robotics, pp. 1–6 (2009)

  22. Mainprice, J., Berenson, D.: Human-robot collaborative manipulation planning using early prediction of human motion. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 299–306 (2013). https://doi.org/10.1109/IROS.2013.6696368

  23. Ben Amor, H., Neumann, G., Kamthe, S., Kroemer, O., Peters, J.: In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 2831–2837. Institute of Electrical and Electronics Engineers Inc. (2014). https://doi.org/10.1109/ICRA.2014.6907265

  24. Sim˜ao, M., Mendes, N., Gibaru, O., Neto, P.: A Review on electromyography decoding and pattern recognition for human-machine interaction. IEEE Access 7, 39564–39582 (2019) https://doi.org/10.1109/ACCESS.2019.2906584

  25. Tao, Y., Huang, Y., Zheng, J., Chen, J., Zhang, Z., Guo, Y., Li, P.: MultiChannel sEMG based human lower limb motion intention recognition method. In: 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 1037–1042 (2019). https://doi.org/10.1109/AIM.2019.8868529

  26. Fang, B., Zhou, Q., Sun, F., Shan, J., Wang, M., Xiang, C., Zhang, Q.: Gait neural network for human-exoskeleton interaction. Front. Neurorobot. 14 (2020). https://doi.org/10.3389/fnbot.2020.00058

  27. Olikkal, P., Pei, D., Adali, T., Banerjee, N., Vinjamuri, R.: Data Fusion-based musculoskeletal synergies in the grasping hand. Sensors 22(19) (2022). https://doi.org/10.3390/s22197417

  28. Olikkal, P., Pei, D., Adali, T., Banerjee, N., Vinjamuri, R.: Musculoskeletal synergies in the grasping hand. In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3649– 3652 (2022). https://doi.org/10.1109/EMBC48229.2022.9871023

  29. Scano, A., Mira, R.M., D’Avella, A.: Mixed matrix factorization: a novel algorithm for the extraction of kinematic-muscular synergies. J. Neurophysiol. 127(2), 529–547 (2022). https://doi.org/10.1152/jn.00379.2021

    Article  Google Scholar 

  30. Burns, M.K., Orden, K.V., Patel, V., Vinjamuri, R.: Towards a wearable hand exoskeleton with embedded synergies. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 213–216 (2017). https://doi.org/10.1109/EMBC.2017.8036800

  31. Burns, M.K., Pei, D., Vinjamuri, R.: Myoelectric control of a soft hand exoskeleton using kinematic synergies. IEEE Trans. Biomed. Circuits Syst. 13(6), 1351–1361 (2019). https://doi.org/10.1109/TBCAS.2019.2950145

    Article  Google Scholar 

  32. Flash, T., Hogans3, N.: The coordination of arm movements: an experimentally confirmed mathematical model’. Tech. Rep. 7 (1985)

  33. Gribovskaya, E., Kheddar, A., Billard, A.: Motion learning and adaptive impedance for robot control during physical interaction with humans. In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 4326–4332 (2011). https://doi.org/10.1109/ICRA.2011.5980070

  34. Mussa-Ivaldi, F.A.: Modular features of motor control and learning. Curr. Opin. Neurobiol. 9(6), 713–717 (1999). https://doi.org/10.1016/S0959-4388(99)00029-X

    Article  Google Scholar 

  35. Flash, T., Hochner, B.: Motor primitives in vertebrates and invertebrates. Curr. Opin. Neurobiol. 15(6), 660–666 (2005). https://doi.org/10.1016/j.conb.2005.10.011

    Article  Google Scholar 

  36. Schaal, S., Mohajerian, P., Ijspeert, A.J.: Dynamics systems vs. optimal control - a unifying view. Progress in brain research 165, 425–445 (2007) https://doi.org/10.1016/S0079-6123(06)65027-9

  37. Maeda, G., Ewerton, M., Lioutikov, R., Ben Amor, H., Peters, J., Neumann, G.: In: IEEE-RAS International Conference on Humanoid Robots, vol. 2015February, pp. 527–534. IEEE Computer Society (2015). https://doi.org/10.1109/HUMANOIDS.2014.7041413

  38. Paraschos, A., Daniel, C., Peters, J.R., Neumann, G.: In: Burges, C.J., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 26. Curran Associates, Inc. (2013). https://proceedings.neurips.cc/paper/2013/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf

  39. Rozo, L., Calinon, S., Caldwell, D.G., Jim´enez, P., Torras, C.: Learning Physical Collaborative Robot Behaviors From Human Demonstrations. IEEE Trans. Robot. 32(3), 513–527 (2016) https://doi.org/10.1109/TRO.2016.2540623

  40. Novak, D., Riener, R.: A survey of sensor fusion methods in wearable robotics. Robot. Auton. Syst. 73, 155–170 (2015). https://doi.org/10.1016/j.robot.2014.08.012

    Article  Google Scholar 

  41. Bernstein, N.: The co-ordination and regulation of movements. The coordination and regulation of movements (1966)

  42. Breazeal, C., Brooks, A.G., Gray, J., Hoffman, G., Kidd, C.D., Lee, H., Lieberman, J., Lockerd, A.L., Mulanda, D.: HUMANOID ROBOTS AS COOPERATIVE PARTNERS FOR PEOPLE. (2004)

  43. Calinon, S., Billard, A.: Teaching a humanoid robot to recognize and reproduce social cues. In: ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication, pp. 346–351 (2006). https://doi.org/10.1109/ROMAN.2006.314458

  44. Misra, D.K., Sung, J., Lee, K., Saxena, A.: Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions. Int. J. Robot. Res. 35(1–3), 281–300 (2016). https://doi.org/10.1177/0278364915602060

    Article  Google Scholar 

  45. Ravichandar, H.C., Polydoros, A.S., Chernova, S., Billard, A.: Recent advances in robot learning from demonstration. (2020)

  46. Calinon, S., Guenter, F., Billard, A.: On Learning, Representing, and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 37(2), 286–298 (2007). https://doi.org/10.1109/TSMCB.2006.886952

  47. Maeda, G.J., Neumann, G., Ewerton, M., Lioutikov, R., Kroemer, O., Peters, J.: Probabilistic movement primitives for coordination of multiple human–robot collaborative tasks. Auton. Robots 41(3), 593–612 (2017). https://doi.org/10.1007/s10514-016-9556-2

    Article  Google Scholar 

  48. Peters, R.A., Campbell, C.L., Bluethmann, W.J., Huber, E.: Robonaut task learning through teleoperation. In: 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422), vol. 2, pp. 2806–28112 (2003). https://doi.org/10.1109/ROBOT.2003.1242017

  49. Abbeel, P., Coates, A., Ng, A.: Autonomous helicopter aerobatics through apprenticeship learning I. J Robot. Res. 29, 1608–1639 (2010). https://doi.org/10.1177/0278364910371999

    Article  Google Scholar 

  50. Mohseni-Kabir, A., Rich, C., Chernova, S., Sidner, C.L., Miller, D.: Interactive hierarchical task learning from a single demonstration. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction. HRI ’15, pp. 205–212. Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2696454.2696474

  51. Whitney, D., Rosen, E., Phillips, E., Konidaris, G.D., Tellex, S.: Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality. In: International Symposium of Robotics Research (2017)

  52. Dillmann, R.: Teaching and learning of robot tasks via observation of human performance. Robot. Auton. Syst. 47(2), 109–116 (2004). https://doi.org/10.1016/j.robot.2004.03.005

    Article  MathSciNet  Google Scholar 

  53. Vogt, D., Stepputtis, S., Grehl, S., Jung, B., Ben Amor, H.: A system for learning continuous human-robot interactions from human-human demonstrations, pp. 2882–2889 (2017). https://doi.org/10.1109/ICRA.2017.7989334

  54. Kaiser, J., Melbaum, S., Tieck, J.C.V., Roennau, A., Butz, M.V., Dillmann, R.: Learning to reproduce visually similar movements by minimizing event-based prediction error. In: 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), pp. 260–267 (2018). https://doi.org/10.1109/BIOROB.2018.8487959

  55. Cakmak, M., Thomaz, A.L.: Designing robot learners that ask good questions. In: 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 17–24 (2012). https://doi.org/10.1145/2157689.2157693

  56. Argall, B., Browning, B., Veloso, M.: Learning by demonstration with critique from a human teacher. In: 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 57–64 (2007). https://doi.org/10.1145/1228716.1228725

  57. Niekum, S., Osentoski, S., Konidaris, G., Barto, A.G.: Learning and generalization of complex tasks from unstructured demonstrations. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5239–5246 (2012). https://doi.org/10.1109/IROS.2012.6386006

  58. Yang, C., Zeng, C., Cong, Y., Wang, N., Wang, M.: A Learning framework of adaptive manipulative skills from human to robot. IEEE Trans. Industr. Inf. 15(2), 1153–1161 (2019). https://doi.org/10.1109/TII.2018.2826064

    Article  Google Scholar 

  59. Figueroa, N., Ureche, A.L.P., Billard, A. (2016) Learning complex sequential tasks from demonstration: A pizza dough rolling case study. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 611–612. https://doi.org/10.1109/HRI.2016.7451881

  60. Chernova, S., Thomaz, A.L.: Robot Learning from Human Teachers, (2014)

  61. Lee, J.: A survey of robot learning from demonstrations for Human-Robot Collaboration. ArXiv abs/1710.08789 (2017)

  62. Vidal, J.J.: Realtime Detection of Brain Events in EEG. Proc. IEEE 65(5), 633–641 (1977). https://doi.org/10.1109/PROC.1977.10542

    Article  Google Scholar 

  63. Chapin, J.K., Moxon, K.A., Markowitz, R.S., Nicolelis, M.A.L.: Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat. Neurosci. 2(7), 664–670 (1999). https://doi.org/10.1038/10223

    Article  Google Scholar 

  64. Fetz, E.E.: Real-time control of a robotic arm by neuronal ensembles. Nat. Neurosci. 2(7), 583–584 (1999). https://doi.org/10.1038/10131

    Article  Google Scholar 

  65. Birbaumer, N., Ku¨bler, A., Ghanayim, N., Hinterberger, T., Perelmouter, J., Kaiser, J., Iversen, I., Kotchoubey, B., Neumann, N., Flor, H.: The thought translation device (TTD) for completely paralyzed patients. IEEE Transactions on Rehabilitation Engineering 8(2), 190–193 (2000) https://doi.org/10.1109/86.847812

  66. Taylor, D.M., Helms Tillery, S.I., Schwartz, A.B.: Direct cortical control of 3D neuroprosthetic devices. Technical report (2002). https://www.science.org

  67. Velliste, M., Perel, S., Spalding, M.C., Whitford, A.S., Schwartz, A.B.: Cortical control of a prosthetic arm for self-feeding. Nature 453(7198), 1098–1101 (2008). https://doi.org/10.1038/nature06996

    Article  Google Scholar 

  68. Inoue, S., Akiyama, Y., Izumi, Y., Nishijima, S.: The development of BCI using alpha waves for controlling the robot arm. In: IEICE Transactions on Communications, vol. E91-B, pp. 2125–2132 (2008). https://doi.org/10.1093/ietcom/e91-b.7.2125

  69. Tonin, L., Leeb, R., Tavella, M., Perdikis, S., Millan, J.R.: The role of sharedcontrol in Bel-based telepresence. In: 2010 IEEE International Conference on Systems, pp. 1462–1466 (2010)

  70. Flesher, S.N., Collinger, J.L., Foldes, S.T., Weiss, J.M., Downey, J.E., TylerKabara, E.C., Bensmaia, S.J., Schwartz, A.B., Boninger, M.L., Gaunt, R.A.: Intracortical microstimulation of human somatosensory cortex. Sci. Trans. Med. 8(361), 361–141361141 (2016)

    Article  Google Scholar 

  71. Belkacem, A.N., Jamil, N., Palmer, J.A., Ouhbi, S., Chen, C.: Brain computer interfaces for improving the quality of life of older adults and elderly patients. Front. Media. S.A. (2020). https://doi.org/10.3389/fnins.2020.00692

  72. M.Bhuvaneshwari, MaryKanaga, E.G., ThomasGeorge, J.A., KumudhaRaimond, S.ThomasGeorge: A comprehensive review on deep learning techniques for a BCI-based communication system. In: Demystifying Big Data, Machine Learning, and Deep Learning for Healthcare Analytics, pp. 131–157 (2021)

  73. Baniqued, P.D.E., Stanyer, E.C., Awais, M., Alazmani, A., Jackson, A.E., MonWilliams, M.A., Mushtaq, F., Holt, R.J.: Brain–computer interface robotics for hand rehabilitation after stroke: a systematic review. BioMed Central Ltd (2021). https://doi.org/10.1186/s12984-021-00820-8

    Article  Google Scholar 

  74. Looned, R., Webb, J., Xiao, Z.G., Menon, C.: Assisting drinking with an affordable BCI-controlled wearable robot and electrical stimulation: A preliminary investigation. J. NeuroEng. Rehab. 11(1) (2014) https://doi.org/10.1186/1743-0003-11-51

  75. Lopez-Larraz, E., Trincado-Alonso, F., Rajasekaran, V., Perez-Nombela, S., DelAma, A.J., Aranda, J., Minguez, J., Gil-Agudo, A., Montesano, L.: Control of an ambulatory exoskeleton with a brain-machine interface for spinal cord injury gait rehabilitation. Front. Neurosci. 10(AUG), 359 (2016). https://doi.org/10.3389/fnins.2016.00359

    Article  Google Scholar 

  76. Garc´ıa-Cossio, E., Severens, M., Nienhuis, B., Duysens, J., Desain, P., Keijsers, N., Farquhar, J.: Decoding sensorimotor rhythms during robotic-assisted treadmill walking for brain computer interface (BCI) applications. PLoS ONE 10(12) (2015) https://doi.org/10.1371/journal.pone.0137910

  77. King, C.E., Wang, P.T., Chui, L.A., Do, A.H., Nenadic, Z.: Operation of a braincomputer interface walking simulator for individuals with spinal cord injury. J. Neuro Eng. Rehab. 10(1) (2013). https://doi.org/10.1186/1743-0003-10-77

  78. Frisoli, A., Loconsole, C., Leonardis, D., Bann`o, F., Barsotti, M., Chisari, C., Bergamasco, M.: A new gaze-BCI-driven control of an upper limb exoskeleton for rehabilitation in real-world tasks. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews 42(6), 1169–1179 (2012). https://doi.org/10.1109/TSMCC.2012.2226444

  79. Carino-Escobar, R.I., Carrillo-Mora, P., Vald´es-Cristerna, R., RodriguezBarragan, M.A., Hernandez-Arenas, C., Quinzan˜os-Fresnedo, J., GaliciaAlvarado, M.A., Cantillo-Negrete, J.: Longitudinal analysis of stroke patients’ brain rhythms during an intervention with a brain-computer interface. Neural Plasticity 2019 (2019). https://doi.org/10.1155/2019/7084618

  80. Collinger, J.L., Gaunt, R.A., Schwartz, A.B.: Progress towards restoring upper limb movement and sensation through intracortical brain-computer interfaces. Current Opinion in Biomedical Engineering 8, 84–92 (2018) https://doi.org/10.1016/j.cobme.2018.11.005

  81. Flesher, S.N., Downey, J.E., Wiess, J.M., Hughes, C.L., Herrera, A.J., Tylerkabara, E.C., Boninger, M.L., Collinger, J.L., Gaunt, R.A.: A brain-computer interface that evokes tactile sensations improves robotic arm control. Science 836(6544), 831–836 (2021)

    Article  Google Scholar 

  82. Alimardani, M., Nishio, S., Ishiguro, H.: Removal of proprioception by BCI raises a stronger body ownership illusion in control of a humanlike robot. Scientific Reports 6 (2016). https://doi.org/10.1038/srep33514

  83. Nurseitov, D., Serekov, A., Shintemirov, A., Abibullaev, B.: Design and Evaluation of a P300-ERP based BCI System for Real-Time Control of a Mobile Robot. In: 5th International Winter Conference on Brain-Computer Interface, BCI 2017, pp. 115–120 (2017). https://doi.org/10.1109/IWW-BCI.2017.7858177

  84. Batres-Mendoza, P., Guerra-Hernandez, E.I., Espinal, A., Perez-Careta, E., Rostro-Gonzalez, H.: Biologically-Inspired Legged Robot Locomotion Controlled with a BCI by Means of Cognitive Monitoring. IEEE Access 9, 35766–35777 (2021). https://doi.org/10.1109/ACCESS.2021.3062329

    Article  Google Scholar 

  85. Bell, C.J., Shenoy, P., Chalodhorn, R., Rao, R.P.N.: Control of a humanoid robot by a noninvasive brain-computer interface in humans. J. Neural Eng. 5(2), 214–220 (2008). https://doi.org/10.1088/1741-2560/5/2/012

    Article  Google Scholar 

  86. Chae, Y., Jeong, J., Jo, S.: Toward brain-actuated humanoid robots: Asynchronous direct control using an EEG-Based BCI. IEEE Trans. Rob. 28(5), 1131–1144 (2012). https://doi.org/10.1109/TRO.2012.2201310

    Article  Google Scholar 

  87. Kubacki, A., Jakubowski, A.: Controlling the industrial robot model with the hybrid BCI based on EOG and eye tracking, vol. 2029 (2018). https://doi.org/10.1063/1.5066494

  88. Farmaki, C., Zacharioudakis, N., Pediaditis, M., Krana, M., Sakkalis, V.: Application of dry EEG electrodes on low-cost SSVEP-based BCI for robot navigation. In: IST 2022 - IEEE International Conference on Imaging Systems and Techniques, Proceedings (2022). https://doi.org/10.1109/IST55454.2022.9827672

  89. Soroush, P.Z., Shamsollahi, M.B.: A non-user-based BCI application for robot control. 2018 IEEE EMBS Conference on Biomedical Engineering and Sciences, IECBES 2018 - Proceedings, 36–41 (2019) https://doi.org/10.1109/IECBES.2018.8626701

  90. Saduanov, B., Alizadeh, T., An, J., Abibullaev, B.: Trained by demonstration humanoid robot controlled via a BCI system for telepresence. In: 2018 6th International Conference on Brain-Computer Interface, BCI 2018, vol. 2018-January, pp. 1–4 (2018). https://doi.org/10.1109/IWW-BCI.2018.8311508

  91. Brownlee, A., Bruening, L.M.: Methods of communication at end of life for the person with amyotrophic lateral sclerosis. Topics in Language Disorders 32(2) (2012)

  92. Kennedy, P., Andreasen, D., Bartels, J., Ehirim, P., Mao, H., Velliste, M., Wichmann, T., Wright, J.: Making the lifetime connection between brain and machine for restoring and enhancing function. In: Progress in Brain Research vol. 194, pp. 1–25 (2011). https://doi.org/10.1016/B978-0-444-53815-4.00020-0

  93. Pandarinath, C., Nuyujukian, P., Blabe, C.H., Sorice, B.L., Saab, J., Willett, F.R., Hochberg, L.R., Shenoy, K.V., Henderson, J.M.: High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6 (2017) https://doi.org/10.7554/eLife.18554

  94. Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.P., Gao, S.: Highspeed spelling with a noninvasive brain-computer interface. Proc. Natl. Acad. Sci. U.S.A. 112(44), 6058–6067 (2015). https://doi.org/10.1073/pnas.1508080112

    Article  Google Scholar 

  95. Han, J., Xu, M., Wang, Y., Tang, J., Liu, M., An, X., Jung, T.P., Ming, D.: ’Write’ but not ’spell’ Chinese characters with a BCI-controlled robot. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, vol. 2020-July, pp. 4741–4744 (2020). https://doi.org/10.1109/EMBC44109.2020.9175275

  96. Velasco-Alvarez, F., Fernandez-Rodriguez, A., Vizcaıno-Martın, F.J., Diaz-Estrella, A., Ron-Angevin, R.: Brain–computer interface (BCI) control of a virtual assistant in a smartphone to manage messaging applications. Sensors 21(11), 3716 (2021). https://doi.org/10.3390/s21113716

    Article  Google Scholar 

  97. Hung, S.C.Y., Tu, C.-H., Wu, C.-E., Chen, C.-S.C.-H., Chan, Y.-M., Chen, C.S.C.-H.: Compacting, Picking and growing for unforgetting continual learning. CoRR abs/1910.0 (2019)

  98. Hung, S.C.Y., Lee, J.-H., Wan, T.S.T., Chen, C.-S.C.-H., Chan, Y.-M., Chen, C.S.C.-H.: Increasingly packing multiple facial-informatics modules in a unified deep-learning model via lifelong learning. ICMR ’19, pp. 339–343. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3323873.3325053

  99. Peng, B.: Emotional state analysis model of humanoid robot in humancomputer interaction process. J. Robot. 2022 (2022) https://doi.org/10.1155/2022/8951671

  100. Lim, C.G., Lee, C.Y., Kim, Y.M.: A performance analysis of user’s intention classification from EEG signal by a computational intelligence in BCI. In: ACM International Conference Proceeding Series, pp. 174–179 (2018). https://doi.org/10.1145/3184066.3184092

  101. Curran, E.A., Stokes, M.J.: Learning to control brain activity: A review of the production and control of EEG components for driving brain-computer interface (BCI) systems. Academic Press Inc. (2003). https://doi.org/10.1016/S0278-2626(03)00036-8

  102. Foong, R., Tang, N., Chew, E., Chua, K.S.G., Ang, K.K., Quek, C., Guan, C., Phua, K.S., Kuah, C.W.K., Deshmukh, V.A., Yam, L.H.L., Rajeswaran, D.K.: Assessment of the efficacy of EEG-Based MI-BCI with Visual Feedback and EEG Correlates of Mental Fatigue for Upper-Limb Stroke Rehabilitation. IEEE Trans. Biomed. Eng. 67(3), 786–795 (2020). https://doi.org/10.1109/TBME.2019.2921198

    Article  Google Scholar 

  103. Wang, M., Zhang, S., Lv, Y., Lu, H.: Anxiety Level Detection Using BCI of Miner’s Smart Helmet. Mob. Netw. Appl. 23(2), 336–343 (2018). https://doi.org/10.1007/s11036-017-0935-5

    Article  Google Scholar 

  104. Wang, F., Zhang, X., Fu, R., Sun, G.: Study of the home-auxiliary robot based on BCI. Sensors (Switzerland) 18(6), 1779 (2018). https://doi.org/10.3390/s18061779

    Article  Google Scholar 

  105. Egziabher, T.B.G., Edwards, S.: Human Robot Interaction-an Introduction 53, 1689–1699 (2013)

    Google Scholar 

  106. Esfahani, E.T., Sundararajan, V.: Using brain-computer interfaces to detect human satisfaction in human-robot interaction (2011). https://doi.org/10.1142/S0219843611002356

    Article  Google Scholar 

  107. Roshdy, A., Karar, A.S., Al-Sabi, A., Barakeh, Z.A., El-Sayed, F., Alkork, S., Beyrouthy, T., Nait-Ali, A.: Towards human brain image mapping for emotion digitization in robotics. BioSMART 2019 - Proceedings: 3rd International Conference on Bio-Engineering for Smart Technologies (2019). 10. 1109/BIOSMART.2019.8734244

  108. Staffa, M., Rossi, S.: Enhancing affective robotics via human internal state monitoring, pp. 884–890 (2022). https://doi.org/10.1109/ro-man53752.2022. 9900762

  109. Bryan, M., Green, J., Chung, M., Chang, L., Scherer, R., Smith, J., Rao, R.P.N.: An adaptive brain-computer interface for humanoid robot control. In: IEEERAS International Conference on Humanoid Robots, pp. 199–204 (2011). https://doi.org/10.1109/Humanoids.2011.6100901

  110. He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., Pan, J.: Advances in multimodal emotion recognition based on brain–computer interfaces. MDPI AG (2020). https://doi.org/10.3390/brainsci10100687

    Article  Google Scholar 

  111. Kragel, P.A., LaBar, K.S.: Decoding the Nature of Emotion in the Brain. Trends Cogn. Sci. 20(6), 444–455 (2016). https://doi.org/10.1016/j.tics.2016.03.011

    Article  Google Scholar 

  112. Saarimaki, H., Ejtehadian, L.F., Glerean, E., Jaaskelainen, I.P., Vuilleumier, P., Sams, M., Nummenmaa, L.: Distributed affective space represents multiple emotion categories across the human brain. Soc. Cognit. Affect. Neurosci. 13(5), 471–482 (2018). https://doi.org/10.1093/scan/nsy018

    Article  Google Scholar 

  113. Siegel, E.H., Sands, M.K., Noortgate, W.V., Condon, P., Chang, Y., Dy, J.G., Quigley, K.S., Barrett, L.F.: Emotion fingerprints or emotion populations? A meta-analytic investigation of autonomic features of emotion categories. Psychol. Bull. 144, 343–393 (2018)

    Article  Google Scholar 

  114. Cai, W., Wei, Z.: PiiGAN: generative adversarial networks for pluralistic image inpainting. IEEE Access 8, 48451–48463 (2020)

    Article  Google Scholar 

  115. Zhang, L., Sun, L., Yu, L., Dong, X., Chen, J., Cai, W., Wang, C., Ning, X.: ARFace: attention-aware and regularization for face recognition with reinforcement learning. IEEE Transactions on Biometrics, Behavior, and Identity Science 4(1), 30–42 (2021)

    Article  Google Scholar 

  116. D’Aniello, B., Semin, G.R., Alterisio, A., Aria, M., Scandurra, A.: Interspecies transmission of emotional information via chemosignals: from humans to dogs (Canis lupus familiaris). Anim. Cogn. 21(1), 67–78 (2018)

    Article  Google Scholar 

  117. Maydych, V., Claus, M., Watzl, C., Kleinsorge, T.: Attention to emotional information is associated with cytokine responses to psychological stress. Front. Neurosci. 12, 687 (2018)

    Article  Google Scholar 

  118. Suslow, T., Husslack, A., Kersting, A., Bodenschatz, C.M.: Attentional biases to emotional information in clinical depression: a systematic and meta-analytic review of eye tracking findings. J. Affect. Disord. 274, 632–642 (2020)

    Article  Google Scholar 

  119. Biehl, V.: Matsumoto and Ekman’s Japanese and Caucasian Facial Expressions of Emotion (JACFEE): Reliability Data and Cross-National Differences. J. Nonverb. behav. 21(1), 21 (1997)

    Article  Google Scholar 

  120. Tomkins, S.S., McCarter, R.: What and Where are the Primary Affects? Some Evidence for a Theory. Percept. Motor. Skills. 18(1), 119–158 (1964). https://doi.org/10.2466/pms.1964.18.1.119

    Article  Google Scholar 

  121. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Person. Soc. Psycho. 17(2), 124–129 (1971). https://doi.org/10.1037/h0030377

    Article  Google Scholar 

  122. Ekman, P.: Pictures of Facial Affect. (1976)

  123. Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992). https://doi.org/10.1080/02699939208411068

    Article  Google Scholar 

  124. Johnson-Laird, P.N., Oatley, K.: Basic emotions, rationality, and folk theory. Cogn. Emot. 6, 201–223 (1992)

    Article  Google Scholar 

  125. Russell, J.: A Circumplex Model of Affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980). https://doi.org/10.1037/h0077714

    Article  Google Scholar 

  126. Russell, J., Bullock, M.: Multidimensional scaling of emotional facial expressions. similarity from preschoolers to adults. J. Person. Soc. Psychol. 48, 1290–1298 (1985). https://doi.org/10.1037/0022-3514.48.5.1290

    Article  Google Scholar 

  127. Anderson, A.K., Christoff, K., Stappen, I., Panitz, D., Ghahremani, D.G., Glover, G., Gabrieli, J.D.E., Sobel, N.: Dissociated neural representations of intensity and valence in human olfaction. Nat. Neurosci. 6(2), 196–202 (2003). https://doi.org/10.1038/nn1001

    Article  Google Scholar 

  128. Russell, J.: Core Affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003). https://doi.org/10.1037/0033-295X.110.1.145

    Article  Google Scholar 

  129. Panayiotou, G.: Emotional dimensions reflected in ratings of affective scripts. Person. Indiv. Diff. 44, 1795–1806 (2008). https://doi.org/10.1016/j.paid.2008.02.006

    Article  Google Scholar 

  130. Ren, F., Huang, Z.: Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEEJ Trans. Electr. Electron. Eng. 10(6), 713–722 (2015). https://doi.org/10.1002/tee.22151

    Article  Google Scholar 

  131. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going Deeper in Facial Expression Recognition using Deep Neural Networks. CoRR abs/1511.0 (2015)

  132. Goodfellow, I.J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., Bengio, Y.: Challenges in representation learning: A report on three machine learning contests. Neural Netw. 64, 59–63 (2015). https://doi.org/10.1016/j.neunet.2014.09.005

    Article  Google Scholar 

  133. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2019). https://doi.org/10.1109/taffc.2017.2740923

    Article  Google Scholar 

  134. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010). https://doi.org/10.1109/CVPRW.2010.5543262

  135. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2106–2112 (2011). https://doi.org/10.1109/ICCVW.2011.6130508

  136. Susskind, J.M., Anderson, A.K., Hinton, G.E.: The toronto face database. Department of Computer Science, University of Toronto, Toronto, ON, Canada, Tech. Rep 3, 29 (2010)

  137. Georgescu, M.-I., Ionescu, R.T., Popescu, M.: Local Learning with Deep and Handcrafted Features for Facial Expression Recognition. CoRR abs/1804.1 (2018)

  138. Savchenko, A.V., Savchenko, L.V., Makarov, I.: Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transactions on Affective Computing, 1–12 (2022). https://doi.org/10.1109/TAFFC.2022.3188390

  139. Kervadec, C., Vielzeuf, V., Pateux, S., Lechervy, A., Jurie, F.: CAKE: Compact and Accurate K-dimensional representation of Emotion. CoRR abs/1807.1 (2018)

  140. Li, S., Deng, W.: Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing 13(3), 1195–1215 (2022) https://doi.org/10.1109/TAFFC.2020.2981446arXiv:1804.08348

  141. Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. In: 2014 Canadian Conference on Computer and Robot Vision, pp. 98–103 (2014). https://doi.org/10.1109/CRV.2014.21

  142. Pons, G., Masip, D.: Multitask, Multilabel, and multidomain learning with convolutional networks for emotion recognition. IEEE Trans. Cybern. 52(6), 4764–4771 (2022). https://doi.org/10.1109/TCYB.2020.3036935

    Article  Google Scholar 

  143. Kollias, D., Sharmanska, V., Zafeiriou, S.: Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study. CoRR abs/2105.0 (2021)

  144. Pourmirzaei, M., Esmaili, F., Montazer, G.A.: Using Self-Supervised CoTraining to Improve Facial Representation. CoRR abs/2105.0 (2021)

  145. Antoniadis, P., Filntisis, P.P., Maragos, P.: Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition. CoRR abs/2106.0 (2021)

  146. Wen, Z., Lin, W., Wang, T., Xu, G.: Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition. CoRR abs/2109.0 (2021)

  147. Savchenko, A.V.: Facial expression and attributes recognition based on multitask learning of lightweight neural networks. CoRR abs/2103.1 (2021)

  148. Chen, Y., Wang, J., Chen, S., Shi, Z., Cai, J.: Facial Motion Prior Networks for Facial Expression Recognition. CoRR abs/1902.0 (2019)

  149. Ryumina, E., Dresvyanskiy, D., Karpov, A.: In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study. Neurocomputing 514, 435–450 (2022). https://doi.org/10.1016/j.neucom.2022.10.013

    Article  Google Scholar 

  150. Siqueira, H., Magg, S., Wermter, S.: Efficient facial feature learning with wide ensemble-based convolutional neural networks. CoRR abs/2001.0 (2020)

  151. Safavi, F., Rahnemoonfar, M.: Comparative study of real-time semantic segmentation networks in aerial images during flooding events. IEEE J. Sel. Topics Appl. Earth Observat. Remote Sensing 16, 15–31 (2023). https://doi.org/10.1109/JSTARS.2022.3219724

    Article  Google Scholar 

  152. Safavi, F., Chowdhury, T., Rahnemoonfar, M.: Comparative study between realtime and non-real-time segmentation models on flooding events. In: 2021 IEEE International Conference on Big Data (Big Data), pp. 4199–4207 (2021). https://doi.org/10.1109/BigData52589.2021.9671314

  153. Rahnemoonfar, M., Safavi, F.: Efficient large-scale damage assessment after natural disasters with uavs and deep learning. In: IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium, pp. 1668–1671. (2023). https://doi.org/10.1109/IGARSS52108.2023.10281912

  154. Rahnemoonfar, M., Safavi, F.: Real-time Aerial Pixel-wise Scene Understanding after Natural Disasters. In: AGU Fall Meeting Abstracts, vol. 2021, pp. 35–16 (2021)

  155. Safavi, F., Patel, K., Vinjamuri, R.K.: Towards efficient deep learning models for facial expression recognition using transformers. In: 2023 IEEE 19th International Conference on Body Sensor Networks (BSN), pp. 1–4 (2023). https://doi.org/10.1109/BSN58485.2023.10331041

  156. Ciregan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3642–3649 (2012). https://doi.org/10.1109/CVPR.2012.6248110

  157. Pham, L., Vu, T.H., Tran, T.A.: Facial Expression Recognition Using Residual Masking Network. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4513–4519 (2021). https://doi.org/10.1109/ICPR48806.2021. 9411919

  158. Simonyan, K., Zisserman, A.: Two-Stream Convolutional Networks for Action Recognition in Videos. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27 (2014). https://proceedings.neurips.cc/paper/2014/file/ 00ec53c4682d36f5c4359f4ae7bd7ba1-Paper.pdf

  159. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017). https://doi.org/10.1109/TIP.2017.2689999

    Article  MathSciNet  Google Scholar 

  160. Valstar, M., Gratch, J., Schuller, B., Ringeval, F., Lalanne, D., Torres Torres, M., Scherer, S., Stratou, G., Cowie, R., Pantic, M.: AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. AVEC ’16, pp. 3– 10. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2988257.2988258

  161. Ringeval, F., Schuller, B., Valstar, M., Gratch, J., Cowie, R., Scherer, S., Mozgai, S., Cummins, N., Schmitt, M., Pantic, M.: AVEC 2017: Real-life depression, and affect recognition workshop and challenge. In: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. AVEC ’17, pp. 3–9. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3133944.3133953

  162. Schoneveld, L., Othmani, A., Abdelkawy, H.: Leveraging recent advances in deep learning for audio-visual emotion recognition. CoRR abs/2103.0 (2021)

  163. Corneanu, C.A., Sim´on, M.O., Cohn, J.F., Guerrero, S.E.: Survey on RGB, 3D, Thermal, and multimodal approaches for facial expression recognition: History, Trends, and Affect-Related Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8), 1548–1568 (2016) https://doi.org/10.1109/TPAMI.2016.2515606

  164. Vo, T.-H., Lee, G.-S., Yang, H.-J., Kim, S.-H.: Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8, 131988–132001 (2020). https://doi.org/10.1109/ACCESS.2020.3010018

    Article  Google Scholar 

  165. Wang, K., Peng, X., Yang, J., Meng, D., Qiao, Y.: Region attention networks for pose and occlusion robust facial expression recognition. CoRR abs/1905.0 (2019)

  166. Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2402–2411 (2021)

  167. Dael, N., Mortillaro, M., Scherer, K.R.: Emotion expression in body action and posture. Emotion 12(5), 1085–1101 (2012)

    Article  Google Scholar 

  168. Aviezer, H., Trope, Y., Todorov, A.: Body Cues, Not Facial Expressions, Discriminate between intense positive and negative emotions. Science 338(6111), 1225–1229 (2012). https://doi.org/10.1126/science.1224313

    Article  Google Scholar 

  169. Noroozi, F., Corneanu, C.A., Kaminska, D., Sapinski, T., Escalera, S., Anbarjafari, G.: Survey on emotional body gesture recognition. CoRR abs/1801.0 (2018)

  170. Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: ACII (2007)

  171. Marchant, L., Mcgrew, W., Eibl-Eibesfeldt, I.: Is Human Handedness Universal? Ethological analyses from three traditional cultures. Ethology 101, 239–258 (2010). https://doi.org/10.1111/j.1439-0310.1995.tb00362.x

    Article  Google Scholar 

  172. Saha, S., Datta, S., Konar, A., Janarthanan, R.: A study on emotion recognition from body gestures using Kinect sensor. 2014 International Conference on Communication and Signal Processing, 56–60 (2014)

  173. Kaliouby, R.E., Robinson, P.: Generalization of a vision-based computational model of mind-reading. In: ACII (2005)

  174. Sapinski, T., Kaminska, D., Pelikant, A., Anbarjafari, G.: Emotion recognition from skeletal movements. Entropy 21(7), 646 (2019). https://doi.org/10.3390/e21070646

    Article  Google Scholar 

  175. Glowinski, D., Mortillaro, M., Scherer, K., Dael, N., Volpe, G., Camurri, A.: Towards a minimal representation of affective gestures (Extended abstract). In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 498–504 (2015). https://doi.org/10.1109/ACII.2015.7344616

  176. Huang, Y., Wen, H., Qing, L., Jin, R., Xiao, L.: Emotion recognition based on body and context fusion in the wild. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 3602–3610 (2021). https://doi.org/10.1109/ICCVW54120.2021.00403

  177. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion Recognition in Context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  178. Inthiam, J., Mowshowitz, A., and Eiji Hayashi: Mood perception model for social robot based on facial and bodily expression using a hidden markov model. Journal of Robotics and Mechatronics 31(4), 629–638 (2019) https:// doi.org/https://doi.org/10.20965/jrm.2019.p0629

  179. Yang, Z., Narayanan, S.S.: Analysis of emotional effect on speech-body gesture interplay. In: Interspeech (2014)

  180. Vu, H.A., Yamazaki, Y., Dong, F., Hirota, K.: Emotion recognition based on human gesture and speech information using RT middleware. In: 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011), pp. 787–791 (2011). https://doi.org/10.1109/FUZZY.2011.6007557

  181. Gunes, H., Piccardi, M.: Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 30(4), 1334–1345 (2007). https://doi.org/10.1016/j.jnca.2006.09.007

    Article  Google Scholar 

  182. Psaltis, A., Kaza, K., Stefanidis, K., Thermos, S., Apostolakis, K.C., Dimitropoulos, K., Daras, P.: Multimodal affective state recognition in serious games applications. IST 2016 - 2016 IEEE International Conference on Imaging Systems and Techniques, Proceedings, 435–439 (2016) https://doi.org/10.1109/IST.2016.7738265

  183. Kessous, L., Castellano, G., Caridakis, G.: Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J. Multimod. User Interf. 3(1), 33–48 (2010). https://doi.org/10.1007/s12193-009-0025-5

    Article  Google Scholar 

  184. Lim, J.Z., Mountstephens, J., Teo, J.: Emotion recognition using eye-tracking: Taxonomy, review and current challenges. Sensors (Switzerland) 20(8), 1–21 (2020). https://doi.org/10.3390/s20082384

    Article  Google Scholar 

  185. Gilzenrat, M.S., Nieuwenhuis, S., Jepma, M., Cohen, J.D.: Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cogn. Affect. Behav. Neurosci. 10, 252–269 (2010)

    Article  Google Scholar 

  186. Zheng, W.-L., Dong, B.-N., Lu, B.-L.: Multimodal emotion recognition using EEG and eye tracking data. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5040–5043 (2014). https://doi.org/10.1109/EMBC.2014.6944757

  187. Aracena, C., Basterrech, S., Sn´ael, V., Vel´asquez, J.: Neural networks for emotion recognition based on eye tracking data. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 2632–2637 (2015). https://doi.org/10.1109/SMC.2015.460

  188. Raudonis, V., Dervinis, G., Vilkauskas, A., Paulauskaite-Taraseviciene, A., Kersulyte-Raudone, G.: Evaluation of human emotion from eye motions. Int. J. Adv. Comput. Sci. Appl 4(8), 79–84 (2013)

    Google Scholar 

  189. Alhargan, A., Cooke, N., Binjammaz, T.: Affect recognition in an interactive gaming environment using eye tracking. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 285–291 (2017). https://doi.org/10.1109/ACII.2017.8273614

  190. Sawers, A., Ting, L.H.: Perspectives on human-human sensorimotor interactions for the design of rehabilitation robots. J. Neuroeng. Rehabil. 11(1), 1–13 (2014). https://doi.org/10.1186/1743-0003-11-142

    Article  Google Scholar 

  191. Zhang, J., Wang, B., Zhang, C., Xiao, Y., Wang, M.Y.: An EEG/EMG/EOGbased multimodal human-machine interface to real-time control of a soft robot hand. Front. Neurorobot. 13(7) (2019)

  192. Kaur, A.: Wheelchair control for disabled patients using EMG/EOG based human machine interface: a review. J. Med. Eng. Technol. 45(1), 61–74 (2021)

    Article  Google Scholar 

  193. Xu, B., Li, W., Liu, D., Zhang, K., Miao, M., Xu, G., Song, A.: Continuous hybrid BCI Control for robotic arm using noninvasive electroencephalogram, Computer vision, and eye tracking. Mathematics 10(4), 618 (2022)

    Article  Google Scholar 

Download references

Funding

Research supported by National Science Foundation (CAREER Award HCC-2053498).

Author information

Authors and Affiliations

  1. Sensorimotor Control Laboratory (Vinjamuri Lab), Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, University of Maryland Baltimore County, Baltimore, MD, 21250, USA

    Farshad Safavi, Parthan Olikkal, Dingyi Pei, Sadia Kamal, Helen Meyerson, Varsha Penumalee & Ramana Vinjamuri

Authors
  1. Farshad Safavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parthan Olikkal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dingyi Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sadia Kamal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Helen Meyerson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Varsha Penumalee
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ramana Vinjamuri
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Farshad Safavi helped organize the research paper, collected, and analyzed research studies on emotional intelligence perception, and participated in writing and revising the manuscript. Parthan Olikkal collected and analyzed research studies on human robot collaboration and participated in writing the manuscript. Dingyi Pei collected and analyzed research studies on brain-computing interface and participated in writing the manuscript. Sadia Kamal, Helen Meyerson and Varsha Penumalee contributed toward revising the manuscript. Ramana Vinjamuri directed the review studies, developed the framework, and participated in writing and revising the manuscript.

Corresponding author

Correspondence to Ramana Vinjamuri.

Ethics declarations

Ethical Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Conflicts of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Safavi, F., Olikkal, P., Pei, D. et al. Emerging Frontiers in Human–Robot Interaction. J Intell Robot Syst 110, 45 (2024). https://doi.org/10.1007/s10846-024-02074-7

Download citation

  • Received: 01 August 2023

  • Accepted: 20 February 2024

  • Published: 18 March 2024

  • DOI: https://doi.org/10.1007/s10846-024-02074-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Human–Robot Interaction
  • Human–Robot Collaboration
  • Brain-Computer Interface
  • Emotional Intelligent Perception
  • Computer Vision
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Advertisement

Search

Navigation

  • Find a journal
  • Publish with us
  • Track your research

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Journal finder
  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support
  • Cancel contracts here

8.209.245.224

Not affiliated

Springer Nature

© 2024 Springer Nature