Data-Driven Generation of Eyes and Head Movements of a Social Robot in Multiparty Conversation | SpringerLink
Skip to main content
Springer Nature Link
Log in

Data-Driven Generation of Eyes and Head Movements of a Social Robot in Multiparty Conversation

  • Conference paper
  • First Online:
Social Robotics (ICSR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14453 ))

Included in the following conference series:

  • 757 Accesses

Abstract

Given the importance of gaze in Human-Robot Interactions (HRI), many gaze control models have been developed. However, these models are mostly built for dyadic face-to-face interaction. Gaze control models for multiparty interaction are more scarce. We here propose and evaluate data-driven gaze control models for a robot game animator in a three-party interaction. More precisely, we used Long Short-Term Memory networks to predict gaze target and context-aware head movements given robot’s communication intents and observed activities of its human partners. After comparing objective performance of our data-driven model with a baseline and ground truth data, an online audiovisual perception study was conducted to compare the acceptability of these control models in comparison with low-anchor incongruent speech and gaze sequences driving the Furhat robot. The results show that our data-driven prediction of gaze targets is viable, but that third-party raters are not so sensitive to controls with congruent head movements.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 8464
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 10581
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.prolific.co/.

References

  1. Admoni, H., Scassellati, B.: Social eye gaze in human-robot interaction: a review. J. Hum.-Robot Interact. Steering Committee 6(1), 25–63 (2017)

    Article  Google Scholar 

  2. Al Moubayed, S., Beskow, J., Skantze, G., Granström, B.: Furhat: a back-projected human-like robot head for multiparty human-machine interaction. Int. J. Humanoid Rob. 2021, 1–11 (2013)

    Google Scholar 

  3. Aliasghari, P., Taheri, A., Meghdari, A.F., Maghsoodi, E.: Implementing a gaze control system on a social robot in multi-person interactions. SN Appl. Sci. 2, 1–13 (2020)

    Article  Google Scholar 

  4. Birdwhistell, R.L.: Background to kinesics. ETC Rev. General Semant. 13(1), 10–18 (1955)

    Google Scholar 

  5. Cambuzat, R., Elisei, F., Bailly, G., Simonin, O., Spalanzani, A.: Immersive teleoperation of the eye gaze of social robots assessing gaze-contingent control of vergence, yaw and pitch of robotic eyes. In: ISR 2018–50th International Symposium on Robotics, pp. 232–239. VDE, Munich (2018)

    Google Scholar 

  6. Correia, F., Campos, J., Melo, F., Paiva, A.: Robotic gaze responsiveness in multiparty teamwork. Int. J. Soc. Robot. 15, 27–36 (2022)

    Article  Google Scholar 

  7. Freedman, E., Sparks, D.: Coordination of the eyes and head: movement kinematics. Exp. Brain Res. 131, 22–32 (2000)

    Article  Google Scholar 

  8. Fuller, J.H.: Comparison of Head Movement Strategies among Mammals. In: The Head-Neck Sensory Motor System. Oxford University Press (1992)

    Google Scholar 

  9. Fuller, J.H.: Head movement propensity. Exp. Brain Res. 92, 152–164 (2004)

    Google Scholar 

  10. Gillet, S., Cumbal, R., Pereira, A., Lopes, J., Engwall, O., Leite, I.: Robot gaze can mediate participation imbalance in groups with different skill levels. In: Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, pp. 303–311. Association for Computing Machinery, New York (2021)

    Google Scholar 

  11. Haefflinger, L., Elisei, F., Gerber, S., Bouchot, B., Vigne, J.P., Bailly, G.: On the benefit of independent control of head and eye movements of a social robot for multiparty human-robot interaction. In: Kurosu, M., Hashizume, A. (eds.) Human-Computer Interaction. LNCS, vol. 14011, pp. 450–466. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-35596-7_29

    Chapter  Google Scholar 

  12. Huang, H.H., Kimura, S., Kuwabara, K., Nishida, T.: Generation of head movements of a robot using multimodal features of peer participants in group discussion conversation. Multimodal Technol. Interact. 4(2), 15 (2020)

    Article  Google Scholar 

  13. Ishii, R., Otsuka, K., Kumano, S., Yamato, J.: Predicting who will be the next speaker and when in multi-party meetings. NTT Tech. Rev. 13, 07 (2015)

    Google Scholar 

  14. Itti, L., Dhavale, N., Pighin, F.: Photorealistic attention-based gaze animation. In: 2006 IEEE International Conference on Multimedia and Expo, pp. 521–524 (2006)

    Google Scholar 

  15. Jonell, P., Yoon, Y., Wolfert, P., Kucherenko, T., Henter, G.E.: HEMVIP: human evaluation of multiple videos in parallel. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 707–711. Association for Computing Machinery, New York (2021)

    Google Scholar 

  16. Kendon, A.: Some functions of gaze-direction in social interaction. Acta Physiol. (Oxf) 26(1), 22–63 (1967)

    Google Scholar 

  17. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2014)

    Google Scholar 

  18. Kuno, Y., Sadazuka, K., Kawashima, M., Yamazaki, K., Yamazaki, A., Kuzuoka, H.: Museum guide robot based on sociological interaction analysis. In: CHI 2007, pp. 1191–1194. Association for Computing Machinery, New York (2007)

    Google Scholar 

  19. Metta, G., et al.: The iCub humanoid robot: an open-systems platform for research in cognitive development. Neural Netw. 23(8), 1125–1134 (2010)

    Article  Google Scholar 

  20. Mishra, C., Skantze, G.: Knowing where to look: a planning-based architecture to automate the gaze behavior of social robots. In: 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1201–1208. IEEE Press (2022)

    Google Scholar 

  21. Mutlu, B., Kanda, T., Forlizzi, J., Hodgins, J., Ishiguro, H.: Conversational gaze mechanisms for humanlike robots. ACM Trans. Interact. Intell. Syst. 1(2), 1–33 (2012)

    Article  Google Scholar 

  22. Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., Hagita, N.: Footing in human-robot conversations: How robots might shape participant roles using gaze cues. In: Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, pp. 61–68. Association for Computing Machinery (2009)

    Google Scholar 

  23. Nakano, Y.I., Yoshino, T., Yatsushiro, M., Takase, Y.: Generating robot gaze on the basis of participation roles and dominance estimation in multiparty interaction. ACM Trans. Interact. Intell. Syst. 5(4), 1–23 (2015)

    Article  Google Scholar 

  24. Nguyen, D.-C., Bailly, G., Elisei, F.: Comparing cascaded LSTM architectures for generating head motion from speech in task-oriented dialogs. In: Kurosu, M. (ed.) HCI 2018. LNCS, vol. 10903, pp. 164–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91250-9_13

    Chapter  Google Scholar 

  25. Pereira, A., Oertel, C., Fermoselle, L., Mendelson, J., Gustafson, J.: Effects of different interaction contexts when evaluating gaze models in HRI. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pp. 131–139. Association for Computing Machinery, New York (2020)

    Google Scholar 

  26. Prévot, L., Elisei, F., Bailly, G.: The robotrio corpus (2020). https://hdl.handle.net/11403/robotrio/v1, ORTOLANG (Open Resources and TOols for LANGuage) - www.ortolang.fr

  27. Sacks, H., Schegloff, E., Jefferson, G.: A simple systematic for the organisation of turn taking in conversation. Language 50, 696–735 (1974)

    Article  Google Scholar 

  28. Shintani, T., Ishi, C.T., Ishiguro, H.: Analysis of role-based gaze behaviors and gaze aversions, and implementation of robot’s gaze control for multi-party dialogue. In: HAI 2021, pp. 332–336. Association for Computing Machinery (2021)

    Google Scholar 

  29. Sidner, C.L., Kidd, C.D., Lee, C., Lesh, N.: Where to look: a study of human-robot engagement. In: Proceedings of the 9th International Conference on Intelligent User Interfaces, IUI 2004, pp. 78–84. Association for Computing Machinery, New York (2004)

    Google Scholar 

  30. Skantze, G., Johansson, M., Beskow, J.: Exploring turn-taking cues in multi-party human-robot discussions about objects. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 67–74. Association for Computing Machinery, New York (2015)

    Google Scholar 

  31. Stefanov, K., Salvi, G., Kontogiorgos, D., Kjellström, H., Beskow, J.: Modeling of human visual attention in multiparty open-world dialogues. J. Hum.-Robot Interact. 8(2), 1–21 (2019)

    Google Scholar 

  32. Vertegaal, R., Slagter, R., van der Veer, G., Nijholt, A.: Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 301–308. Association for Computing Machinery, New York (2001)

    Google Scholar 

  33. Zangemeister, W., Stark, L.: Types of gaze movement: variable interactions of eye and head movements. Exp. Neurol. 77(3), 563–577 (1982)

    Article  Google Scholar 

  34. Zaraki, A., Mazzei, D., Giuliani, M., de rossi, D.: Designing and evaluating a social gaze-control system for a humanoid robot. IEEE Trans. Syst. Man Cybernet. A Syst. Hum. 44, 157–168 (2014)

    Google Scholar 

Download references

Acknowledgements

The RoboTrio corpus was supported by CNRS through a S2IH PEPS funding. This research is supported by the ANR 19-P3IA-0003 MIAI. The first author is financed by a CIFRE PhD granted by ANRT (2021/0836).

Author information

Authors and Affiliations

  1. GIPSA-Lab, Grenoble-Alpes Univ., Grenoble, France

    Léa Haefflinger, Frédéric Elisei & Gérard Bailly

  2. Atos, 38130, Échirolles, France

    Léa Haefflinger, Béatrice Bouchot & Brice Varini

Authors
  1. Léa Haefflinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frédéric Elisei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Béatrice Bouchot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brice Varini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gérard Bailly
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Léa Haefflinger .

Editor information

Editors and Affiliations

  1. Qatar University, Doha, Qatar

    Abdulaziz Al Ali

  2. Qatar University, Doha, Qatar

    John-John Cabibihan

  3. Qatar University, Doha, Qatar

    Nader Meskin

  4. University of Naples Federico II, Napoli, Italy

    Silvia Rossi

  5. Qingdao University, Qingdao, China

    Wanyue Jiang

  6. The University of Alabama, Tuscaloosa, AL, USA

    Hongsheng He

  7. National University of Singapore, Queenstown, Singapore

    Shuzhi Sam Ge

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Haefflinger, L., Elisei, F., Bouchot, B., Varini, B., Bailly, G. (2024). Data-Driven Generation of Eyes and Head Movements of a Social Robot in Multiparty Conversation. In: Ali, A.A., et al. Social Robotics. ICSR 2023. Lecture Notes in Computer Science(), vol 14453 . Springer, Singapore. https://doi.org/10.1007/978-981-99-8715-3_17

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8715-3_17

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8714-6

  • Online ISBN: 978-981-99-8715-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics