A social robot as your reading companion: exploring the relationships between gaze patterns and knowledge gains | Journal on Multimodal User Interfaces
Skip to main content
Springer Nature Link
Log in

A social robot as your reading companion: exploring the relationships between gaze patterns and knowledge gains

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Intelligent tutoring systems have been widely used in educational activities over the past 20 years. With significantly less effort put into writing or reading assistance, the majority of intelligent tutoring systems focus on mathematics or problem solving activities. However, with the development of E-reading-centered E-education, how to improve students’ learning performance during reading has become increasingly important. Therefore, in this paper, we take a first step in the direction of an adaptive intelligent tutoring system by investigating how different reading strategies relate to knowledge gain based on gaze features and how an embodied social robot affects gaze patterns and reading strategies. The findings showed that different knowledge gains have significant differences in scanning methods and reading depth, and that the feedback given by social robots significantly affects participants’ gaze patterns during the whole reading process. To automatically differentiate between two levels of knowledge gain, several prediction experiments based on various reading strategy-related gaze features were carried out. The results demonstrate that saccades are the best predictors of knowledge gain, with the best model having an average accuracy of 74.2%. Finally, real-time simulation experiments were conducted with sixty participants using the leave-one-out method, and an accurate prediction of the level of knowledge gain of 71. 5% was achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and materials

The eye gaze data that support the findings of this study are available from the Interactive Intelligence group of TU delft. The data may be available upon request but not for all due to relevant data protection laws.

Notes

  1. https://pupil-labs.com/.

References

  1. Abidin AZ (2020) Students reading comprehension through scanning technique. J Asian Multicult Res Educ Study 1(1):28–35. https://doi.org/10.47616/jamres.v1i1.13

    Article  Google Scholar 

  2. Almasri A, Ahmed A, Almasri N et al (2019) Intelligent tutoring systems survey for the period 2000–2018. Int J Acad Eng Res

  3. Anderson JR, Boyle CF, Reiser BJ (1985) Intelligent tutoring systems. Science 228(4698):456–462. https://doi.org/10.1126/science.228.4698.456

    Article  ADS  CAS  PubMed  Google Scholar 

  4. Asmawati A (2015) The effectiveness of skimming–scanning strategy in improving students’ reading comprehension at the second grade of SMK Darussalam Makassar. Engl Teach Learn Res J ETERNAL 1(1):69–83. https://doi.org/10.24252/Eternal.V11.2015.A9

    Article  Google Scholar 

  5. Bainbridge WA, Hart JW, Kim ES et al (2011) The benefits of interactions with physically present robots over video-displayed agents. Int J Soc Robot 3(1):41–52. https://doi.org/10.1007/s12369-010-0082-7

    Article  Google Scholar 

  6. Bamkin M, Goulding A, Maynard S (2013) The children sat and listened: storytelling on children’s mobile libraries. N Rev Child Lit Librariansh 19(1):47–78. https://doi.org/10.1080/13614541.2013.755023

    Article  Google Scholar 

  7. Belpaeme T, Kennedy J, Ramachandran A et al (2018) Social robots for education: a review. Sci Robot 3(21):eaat5954. https://doi.org/10.1126/scirobotics.aat5954

    Article  PubMed  Google Scholar 

  8. Bickmore T, Kimani E, Shamekhi A et al (2021) Virtual agents as supporting media for scientific presentations. J Multimodal User Interfaces 15:131–146. https://doi.org/10.1007/s12193-020-00350-y

    Article  Google Scholar 

  9. Biedert R, Hees J, Dengel A et al (2012) A robust realtime reading-skimming classifier. In: Proceedings of the symposium on eye tracking research and applications, pp 123–130. https://doi.org/10.1145/2168556.2168575

  10. Biswas G, Leelawong K, Schwartz D et al (2005) Learning by teaching: a new agent paradigm for educational software. Appl Artif Intell 19(3–4):363–392. https://doi.org/10.1080/08839510590910200

    Article  Google Scholar 

  11. Bixler R, D’Mello S (2016) Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Model User Adapt Int 26(1):33–68. https://doi.org/10.1007/s11257-015-9167-1

    Article  Google Scholar 

  12. Block E (1986) The comprehension strategies of second language readers. TESOL Q 20(3):463–494. https://doi.org/10.2307/3586295

    Article  MathSciNet  Google Scholar 

  13. Blok H, Oostdam R, Otter ME et al (2002) Computer-assisted instruction in support of beginning reading instruction: a review. Rev Educ Res 72(1):101–130. https://doi.org/10.3102/003465430720011

    Article  Google Scholar 

  14. Brysbaert M, Mitchell DC (1996) Modifier attachment in sentence parsing: evidence from Dutch. Q J Exp Psychol Sect A 49(3):664–695. https://doi.org/10.1080/713755636

    Article  Google Scholar 

  15. Coppi AE, Oertel C, Cattaneo A (2021) Effects of experts’ annotations on fashion designers apprentices’ gaze patterns and verbalisations. Vocat Learn 14(3):511–531. https://doi.org/10.1007/s12186-021-09270-8

    Article  Google Scholar 

  16. Crossley SA, Varner LK, Roscoe RD et al (2013) Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. In: International conference on artificial intelligence in education. Springer, pp 269–278. https://doi.org/10.1007/978-3-642-39112-5_28

  17. Davison DP, Wijnen FM, Charisi V et al (2021) Words of encouragement: how praise delivered by a social robot changes children’s mindset for learning. J Multimodal User Interfaces 15:61–76. https://doi.org/10.1007/s12193-020-00353-9

    Article  Google Scholar 

  18. Bixler ER, D’Mello KS (2021) Crossed eyes: domain adaptation for gaze-based mind wandering models. Association for Computing Machinery, New York, NY, USA, ETRA ’21 full papers. https://doi.org/10.1145/3448017.3457386

  19. Faber M, Bixler R, D’Mello SK (2018) An automated behavioral measure of mind wandering during computerized reading. Behav Res Methods 50(1):134–150. https://doi.org/10.3758/s13428-017-0857-y

    Article  ADS  PubMed  Google Scholar 

  20. Feng S, D’Mello S, Graesser AC (2013) Mind wandering while reading easy and difficult texts. Psychon Bull Rev 20(3):586–592. https://doi.org/10.3758/s13423-012-0367-y

    Article  PubMed  Google Scholar 

  21. Follmer DJ (2018) Executive function and reading comprehension: a meta-analytic review. Educ Psychol 53(1):42–60. https://doi.org/10.1080/00461520.2017.1309295

    Article  Google Scholar 

  22. Garner R (1987) Metacognition and reading comprehension. Ablex Publishing, New York

    Google Scholar 

  23. Gernsbacher MA, McKinney VM (1999) Comprehension: a paradigm for cognition. Am Sci 87(6):568

    Google Scholar 

  24. Van der Gijp A, Ravesloot C, Jarodzka H et al (2017) How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Adv Health Sci Educ 22(3):765–787. https://doi.org/10.1007/s10459-016-9698-1

    Article  Google Scholar 

  25. Golinkoff RM (1975) A comparison of reading comprehension processes in good and poor comprehenders. Read Res Q 11:623–659. https://doi.org/10.2307/747459

    Article  Google Scholar 

  26. Gordon G, Breazeal C (2015) Bayesian active learning-based robot tutor for children’s word-reading skills. In: Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v29i1.9376

  27. Guthrie JT, Wigfield A, Barbosa P et al (2004) Increasing reading comprehension and engagement through concept-oriented reading instruction. J Educ Psychol 96(3):403. https://doi.org/10.1037/0022-0663.96.3.403

    Article  Google Scholar 

  28. Harmer J (2001) The practice of English language teaching. London/New York, pp 401–405

  29. Hedge T (2001) Teaching and learning in the language classroom, vol 106. Oxford University Press, Oxford

    Google Scholar 

  30. Holmqvist K, Nyström M, Andersson R et al (2011) Eye tracking: a comprehensive guide to methods and measures. Oxford University Press, Oxford

    Google Scholar 

  31. Huang X, Craig SD, Xie J et al (2016) Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learn Ind Differ 47:258–265. https://doi.org/10.1016/j.lindif.2016.01.012

    Article  Google Scholar 

  32. Hutt S, Krasich K, Mills C et al (2019) Automated gaze-based mind wandering detection during computerized learning in classrooms. User Model User Adapt Int 29(4):821–867. https://doi.org/10.1007/s11257-019-09228-5

    Article  Google Scholar 

  33. Hyönä J, Lorch Jr RF, Rinck M (2003) Eye movement measures to study global text processing. In: The mind’s eye. Elsevier, pp 313–334. https://doi.org/10.1016/B978-044451020-4/50018-9

  34. Ince G, Yorganci R, Ozkul A et al (2021) An audiovisual interface-based drumming system for multimodal human-robot interaction. J Multimodal User Interfaces 15:413–428. https://doi.org/10.1007/s12193-020-00352-w

    Article  Google Scholar 

  35. Kaakinen JK, Hyona J (2005) Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking, and recall. Discourse Process 40(3):239–257. https://doi.org/10.1207/s15326950dp4003_4

    Article  Google Scholar 

  36. Kaakinen JK, Hyönä J (2007) Perspective effects in repeated reading: an eye movement study. Mem Cogn 35(6):1323–1336. https://doi.org/10.3758/BF03193604

    Article  Google Scholar 

  37. Kaakinen JK, Hyönä J (2010) Task effects on eye movements during reading. J Exp Psychol Learn Mem Cogn 36(6):1561. https://doi.org/10.1037/a0020693

    Article  PubMed  Google Scholar 

  38. Kaakinen JK, Hyönä J, Keenan JM (2002) Perspective effects on online text processing. Discourse Process 33(2):159–173. https://doi.org/10.1207/S15326950DP3302_03

    Article  Google Scholar 

  39. Kaakinen JK, Hyönä J, Keenan JM (2003) How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. J Exp Psychol Learn Mem Cogn 29(3):447–457. https://doi.org/10.1037/0278-7393.29.3.447

    Article  PubMed  Google Scholar 

  40. Kaakinen JK, Olkoniemi H, Kinnari T et al (2014) Processing of written irony: an eye movement study. Discourse Process 51(4):287–311. https://doi.org/10.1080/0163853X.2013.870024

    Article  Google Scholar 

  41. Kassner M, Patera W, Bulling A (2014) Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In: Adjunct proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, New York, NY, USA, UbiComp ’14 Adjunct, pp 1151–1160. https://doi.org/10.1145/2638728.2641695

  42. Kennedy J, Baxter P, Senft E et al (2016) Social robot tutoring for child second language learning. In: 2016 11th ACM/IEEE international conference on human-robot interaction (HRI), pp 231–238. https://doi.org/10.1109/HRI.2016.7451757

  43. Khachatryan GA, Romashov AV, Khachatryan AR et al (2014) Reasoning mind Genie 2: an intelligent tutoring system as a vehicle for international transfer of instructional methods in mathematics. Int J Artif Intell Educ 24(3):333–382. https://doi.org/10.1007/s40593-014-0019-7

    Article  Google Scholar 

  44. Kidd CD, Breazeal C (2007) A robotic weight loss coach. In: Proceedings of the national conference on artificial intelligence. London, AAAI Press, MIT Press, Menlo Park, Cambridge, p 1985

  45. Kontogiorgos D, Pereira A, Gustafson J (2021) Grounding behaviours with conversational interfaces: effects of embodiment and failures. J Multimodal User Interfaces 15:239–254. https://doi.org/10.1007/s12193-021-00366-y

    Article  Google Scholar 

  46. Lee Y, Chen H, Zhao G et al (2022) Wedar: webcam-based attention analysis via attention regulator behavior recognition with a novel e-reading dataset. In: Proceedings of the 2022 international conference on multimodal interaction. Association for Computing Machinery, New York, NY, USA, ICMI ’22, pp 319–328. https://doi.org/10.1145/3536221.3556619

  47. Leite I, Martinho C, Paiva A (2013) Social robots for long-term interaction: a survey. Int J Soc Robot 5(2):291–308. https://doi.org/10.1007/s12369-013-0178-y

    Article  Google Scholar 

  48. Leite I, Pereira A, Mascarenhas S et al (2013) The influence of empathy in human-robot relations. Int J Hum Comput Stud 71(3):250–260. https://doi.org/10.1016/j.ijhcs.2012.09.005

    Article  Google Scholar 

  49. Leyzberg D, Spaulding S, Toneva M et al (2012) The physical presence of a robot tutor increases cognitive learning gains. In: Proceedings of the annual meeting of the cognitive science society. https://escholarship.org/uc/item/7ck0p200

  50. Li J (2015) The benefit of being physically present: a survey of experimental works comparing copresent robots, telepresent robots and virtual agents. Int J Hum Comput Stud 77:23–37. https://doi.org/10.1016/j.ijhcs.2015.01.001

    Article  ADS  Google Scholar 

  51. Lin W, Yueh HP, Wu HY et al (2014) Developing a service robot for a children’s library: a design-based research approach. J Am Soc Inf Sci 65(2):290–301. https://doi.org/10.1002/asi.22975

    Article  Google Scholar 

  52. Michaelis JE, Mutlu B (2018) Reading socially: transforming the in-home reading experience with a learning-companion robot. Sci Robot 3(21):eaat5999. https://doi.org/10.1126/scirobotics.aat5999

    Article  PubMed  Google Scholar 

  53. Mousavinasab E, Zarifsanaiey N, NiakanKalhori RS et al (2018) Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interact Learn Environ 29(1):142–163. https://doi.org/10.1080/10494820.2018.1558257

    Article  Google Scholar 

  54. Murray WS (2000) Commentary on section 4. Sentence processing: issues and measures. In: Kennedy A, Radach R, Heller D et al (eds) Reading as a perceptual process. North-Holland/Elsevier Science Publishers, pp 649–664. https://doi.org/10.1016/B978-008043642-5/50030-9

  55. National Reading Panel (US) NIoCH, (US) HD (2000) Teaching children to read: an evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups. National Institute of Child Health and Human Development, National Institutes of Health

  56. Oertel C, Coppi A, Olsen JK, et al (2019) On the use of gaze as a measure for performance in a visual exploration task. In: European conference on technology enhanced learning. Springer, pp 386–395. https://doi.org/10.1007/978-3-030-29736-7_29

  57. Olson E (2011) AprilTag: a robust and flexible visual fiducial system. In: 2011 IEEE international conference on robotics and automation. IEEE, pp 3400–3407. https://doi.org/10.1109/ICRA.2011.5979561

  58. Paris SG, Cross DR, Lipson MY (1984) Informed strategies for learning: a program to improve children’s reading awareness and comprehension. J Educ Psychol 76(6):1239

    Article  Google Scholar 

  59. Paris SG, Wasik B, Turner JC (1991) The development of strategic readers. In: Barr R, Kamil ML, Mosenthal PB et al (eds) Handbook of reading research, vol 2. Lawrence Erlbaum Associates Inc, Mahwah, pp 609–640

    Google Scholar 

  60. Pernice K, Whitenton K, Nielsen J et al (2014) How people read online: the eyetracking evidence. Nielsen Norman Group, Fremont

    Google Scholar 

  61. Pollard-Durodola SD, Gonzalez JE, Simmons DC et al (2011) The effects of an intensive shared book-reading intervention for preschool children at risk for vocabulary delay. Except Child 77(2):161–183. https://doi.org/10.1177/001440291107700202

    Article  Google Scholar 

  62. Pourhosein Gilakjani A, Sabouri NB (2016) How can students improve their reading comprehension skill. J Stud Educ 6(2):229–240. https://doi.org/10.5296/jse.v6i2.9201

    Article  Google Scholar 

  63. Pressley M, El-Dinary PB, Brown R (1992) Skilled and not-so-skilled reading: Good information processing and not-so-good information processing. In: Pressley M, Harris KR, Guthrie JT (eds) Promoting academic competence and literacy in school. Academic Press, pp 91–127

  64. Rajendran R, Kumar A, Carter KE et al (2018) Predicting learning by analyzing eye-gaze data of reading behavior. Int Educ Data Min Soc 16–20. https://api.semanticscholar.org/CorpusID:52173770

  65. Raney GE, Campbell SJ, Bovee JC (2014) Using eye movements to evaluate the cognitive processes involved in text comprehension. J Vis Exp JoVE 83:e50,780. https://doi.org/10.3791/50780

    Article  Google Scholar 

  66. Reichle ED, Pollatsek A, Fisher DL et al (1998) Toward a model of eye movement control in reading. Psychol Rev 105(1):125–157. https://doi.org/10.1037/0033-295X.105.1.125

    Article  CAS  PubMed  Google Scholar 

  67. Reynolds RE (2000) Attentional resource emancipation: toward understanding the interaction of word identification and comprehension processes in reading. Sci Stud Read 4(3):169–195. https://doi.org/10.1207/S1532799XSSR0403_1

    Article  Google Scholar 

  68. Rumpf C, Boronczyk F, Breuer C (2020) Predicting consumer gaze hits: a simulation model of visual attention to dynamic marketing stimuli. J Bus Res 111:208–217. https://doi.org/10.1016/j.jbusres.2019.03.034

    Article  Google Scholar 

  69. Savitzky A, Golay MJ (1964) Smoothing and differentiation of data by simplified least squares procedures. Anal Chem 36(8):1627–1639

    Article  ADS  CAS  Google Scholar 

  70. Sevcenko N, Appel T, Ninaus M et al (2023) Theory-based approach for assessing cognitive load during time-critical resource-managing human-computer interactions: an eye-tracking study. J Multimodal User Interfaces 17(1):1–19. https://doi.org/10.1007/s12193-022-00398-y

    Article  Google Scholar 

  71. Shrestha S, Lenz K, Chaparro B et al (2007) “f” pattern scanning of text and images in web pages. In: Proceedings of the human factors and ergonomics society annual meeting. SAGE Publications, Los Angeles, pp 1200–1204. https://doi.org/10.1177/154193120705101831

  72. Steenbergen-Hu S, Cooper H (2013) A meta-analysis of the effectiveness of intelligent tutoring systems on k-12 students’ mathematical learning. J Educ Psychol 105(4):970–987. https://doi.org/10.1037/a0032447

    Article  Google Scholar 

  73. Sugimoto M (2011) A mobile mixed-reality environment for children’s storytelling using a handheld projector and a robot. IEEE Trans Learn Technol 4(3):249–260. https://doi.org/10.1109/TLT.2011.13

    Article  Google Scholar 

  74. Van Dijk TA, Kintsch W et al (1983) Strategies of discourse comprehension. Psychology 6(6):12

    Google Scholar 

  75. Wang YH, Young SSC, Jang JSR (2013) Using tangible companions for enhancing learning English conversation. J Educ Technol Soc 16(2):296–309

    CAS  Google Scholar 

  76. Weinfurt KP (1995) Multivariate analysis of variance. In: Grimm LG, Yarnold PR (eds) Reading and understanding multivariate statistics. American Psychological Association, pp 245–276

    Google Scholar 

  77. Weisstein EW (2004) Bonferroni correction. https://mathworld.wolfram.com/BonferroniCorrection.html

  78. Wijekumar K, Meyer BJ, Lei P et al (2020) Supplementing teacher knowledge using web-based intelligent tutoring system for the text structure strategy to improve content area reading comprehension with fourth-and fifth-grade struggling readers. Dyslexia 26(2):120–136. https://doi.org/10.1002/dys.1634

  79. Yadollahi E, Johal W, Paiva A et al (2018) When deictic gestures in a robot can harm child-robot collaboration. In: Proceedings of the 17th ACM conference on interaction design and children, pp 195–206. https://doi.org/10.1145/3202185.3202743

  80. Yang W, Dai W, Gao L (2012) Intensive reading and necessity to integrate learning strategies. Engl Lang Lit 2(1):55–63. https://doi.org/10.5539/ells.v2n1p112

    Article  CAS  Google Scholar 

  81. Yueh HP, Lin W, Wang SC et al (2020) Reading with robot and human companions in library literacy activities: a comparison study. Br J Edu Technol 51(5):1884–1900. https://doi.org/10.1111/bjet.13016

    Article  Google Scholar 

  82. Zhao Q, Yuan X, Tu D et al (2015) Eye moving behaviors identification for gaze tracking interaction. J Multimodal User Interfaces 9:89–104. https://doi.org/10.1007/s12193-014-0171-2

  83. Zwaan RA, Singer M (2003) Text comprehension. In: Graesser AC, Gernsbacher MA, Goldman SR (eds) Handbook of discourse processes. Lawrence Erlbaum Associates Publishers, Mahwah, pp 83–121

    Google Scholar 

Download references

Acknowledgements

The data collection was supported by the Interactive Intelligence Group and the Web Information Systems Group at TU Delft. Additionally, the authors would like to thank Yoon Lee, Catharine Oertel, Marcus Specht, and Jennifer Olsen for their help with the work.

Funding

The first author is funded by the China Scholarship Council (CSC) (No. 202006120103) from the Ministry of Education of the P.R. China. This work was supported by the National Natural Science Foundation of China under Grant 61876054. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001, People’s Republic of China

    Xuan Liu & Qiang Wang

  2. Department of Control Science and Engineering, Harbin Institute of Technology, Weihai, 150001, People’s Republic of China

    Jiachen Ma

Authors
  1. Xuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiachen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

XL designed and performed the experiments, derived the models and analysed the data. JM, QW, and XL interpreted the data analysis results and wrote the manuscript together. All authors of this paper have read and approved the final version submitted.

Corresponding author

Correspondence to Qiang Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Questionnaire

Appendix A: Questionnaire

See Tables 4, 5 and .

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Ma, J. & Wang, Q. A social robot as your reading companion: exploring the relationships between gaze patterns and knowledge gains. J Multimodal User Interfaces 18, 21–41 (2024). https://doi.org/10.1007/s12193-023-00418-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-023-00418-5

Keywords

Search

Navigation