Mining of Keystroke and Mouse Dynamics to Increase the Engagement of Students with Programming Assignments | SpringerLink
Skip to main content
Springer Nature Link
Log in

Mining of Keystroke and Mouse Dynamics to Increase the Engagement of Students with Programming Assignments

  • Conference paper
  • First Online:
Computational Intelligence (IJCCI 2017)

Part of the book series: Studies in Computational Intelligence ((SCI,volume 829))

Included in the following conference series:

Abstract

The aim of the experiments described in this paper is to evaluate the use of keyboard and mouse dynamics as an appropriate non-obtrusive sensory input for an system that is sensitive to the affective state of its user. Our motivation for starting this research line has been the lack of tools and methodologies for taking into account this affective state in learning environments. In an ideal situation, when instructors have to choose from a collection of programming assignments, they should consider the studentś affective state and skills in order to select a learning resource with the appropriate level of difficulty. However, neither the data or the ability to process it are present in current learning management systems. This work tries to address this problem, by focused on the capture and pre-processing of data that is going to be fed to several machine learning techniques with the objective of classifying the affective states of users with different levels of expertise when learning a new programming language. We capture student data from a web-based platform, a learning management system where students interact with programming exercises. We introduce in this paper a series of pre-processing techniques that are able to convert data from keyboard and mouse dynamics captured from students as they were coding basic Python programming assignments into feature vector, that have been later used for the classification into five affective states: boredom, frustration, distraction, relaxation and engagement. The following classification algorithms have been evaluated: k-nearest neighbors,feed forward neural networks, naïve Bayes classifier, J-48 tree induction algorithm, deep learning, random forest, gradient boosted trees and naïve Bayes Kernel). The best accuracy was around 78% and was achieved by the tree induction algorithms. Results show that data gathered from ready-available, non-obtrusive sensors can be used successfully as another input to hybrid classification models in order to predict an individualś affective states, and these in turn can be used to make students more engaged and thus learn more efficiently.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
JPY 14299
Price includes VAT (Japan)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Robins, A., Rountree, J., Rountree, N.: Learning and teaching programming: a review and discussion. Comput. Sci. Educ. 13, 137–172 (2003)

    Article  Google Scholar 

  2. Lahtinen, E., Ala-Mutka, K., Järvinen, H.M.: A study of the difficulties of novice programmers. In: ACM SIGCSE Bulletin, vol. 37, pp. 14–18. ACM, New York (2005)

    Google Scholar 

  3. Munson, J.P., Zitovsky, J.P.: Models for early identification of struggling novice programmers. In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, pp. 699–704. ACM, New York (2018)

    Google Scholar 

  4. Bennedsen, J., Caspersen, M.E.: Failure rates in introductory programming. ACM SIGCSE Bull. 39, 32–36 (2007)

    Article  Google Scholar 

  5. Jenkins, T.: The motivation of students of programming. In: ACM SIGCSE Bulletin, vol. 33, pp. 53–56. ACM, New York (2001)

    Google Scholar 

  6. Dijkstra, E.W., et al.: On the cruelty of really teaching computing science. Commun. ACM 32, 1398–1404 (1989)

    Google Scholar 

  7. Jenkins, T.: On the difficulty of learning to program. In: Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences, vol. 4, pp. 53–58 (2002). Citeseer

    Google Scholar 

  8. Kort, B., Reilly, R., Picard, R.W.: An affective model of interplay between emotions and learning: reengineering educational pedagogy – building a learning companion. In: ICALT, vol. 1, pp. 43–47 (2001)

    Google Scholar 

  9. Rossin, D., Ro, Y.K., Klein, B.D., Guo, Y.M.: The effects of flow on learning outcomes in an online information management course. J. Inf. Syst. Educ. 20, 87 (2009)

    Google Scholar 

  10. Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V., Lim, S.A.L., Pascua, S.A., Sugay, J.O., Tabanao, E.S.: Affective and behavioral predictors of novice programmer achievement. In: ACM SIGCSE Bulletin, vol. 41, pp. 156–160. ACM, New York (2009)

    Google Scholar 

  11. Bosch, N., D’Mello, S., Mills, C.: What emotions do novices experience during their first computer programming learning session? In: International Conference on Artificial Intelligence in Education, pp. 11–20. Springer, Berlin (2013)

    Google Scholar 

  12. Khan, I.A., Hierons, R.M., Brinkman, W.P.: Mood independent programming. In: Proceedings of the 14th European Conference on Cognitive Ergonomics: Invent! Explore!, pp. 269–272. ACM, New York (2007)

    Google Scholar 

  13. Csikszentmihalyi, M.: Flow: The Psychology of Optimal Performance. Cambridge University Press, Cambridge (1990)

    Google Scholar 

  14. Bangert-Drowns, R.L., Pyke, C.: Teacher ratings of student engagement with educational software: an exploratory study. Educ. Technol. Res. Dev. 50, 23–37 (2002)

    Article  Google Scholar 

  15. Gottardo, E., Pimentel, A.R.: Affective human-computer interaction in educational software: a hybrid model for emotion inference. In: Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, p. 54. ACM, New York (2017)

    Google Scholar 

  16. Hundhausen, C., Olivares, D., Carter, A.: IDE-based learning analytics for computing education: a process model, critical review, and research agenda. ACM Trans. Comput. Educ. (TOCE) 17, 11 (2017)

    Google Scholar 

  17. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1175–1191 (2001)

    Article  Google Scholar 

  18. Bosch, N., DMello, S.: The affective experience of novice computer programmers. Int. J. Artif. Intell. Educ. 27, 181–206 (2017)

    Google Scholar 

  19. Asensio Montiel, J.J., García Sánchez, P., Mora García, A.M., Fernández Ares, A.J., Merelo Guervós, J.J., Castillo Valdivieso, P.Á.: Progamer: aprendiendo a programar usando videojuegos como metáfora para visualización de código. ReVisión 7 (2014)

    Google Scholar 

  20. Kula, I., Branaghan, R.J., Atkinson, R.K., Roscoe, R.D.: Assessing user experience via biometric sensor affect detection. In: End-User Considerations in Educational Technology Design. IGI Global (2017)

    Google Scholar 

  21. Swansi, V., Herradura, T., Suarez, M.T.: (Analyzing Novice Programmers EEG Signals using Unsupervised Algorithms)

    Google Scholar 

  22. Gonzalez-Sanchez, J., Baydogan, M., Chavez-Echeagaray, M.E., Atkinson, R.K., Burleson, W.: Affect measurement: a roadmap through approaches, technologies, and data analysis. In: Emotions and Affect in Human Factors and Human-Computer Interaction, pp. 255–288. Elsevier (2017)

    Google Scholar 

  23. Zhai, J., Barreto, A.: Stress detection in computer users through non-invasive monitoring of physiological signals. Blood 5 (2008)

    Google Scholar 

  24. Sidney, K.D., Craig, S.D., Gholson, B., Franklin, S., Picard, R., Graesser, A.C.: Integrating affect sensors in an intelligent tutoring system. In: Affective Interactions: The Computer in the Affective Loop Workshop at, pp. 7–13 (2005)

    Google Scholar 

  25. Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. AIED, vol. 200, pp. 17–24 (2009)

    Google Scholar 

  26. Valdez, M.G., Aguila, A.H., Merelo, J.J., Soto, A.M.: Enhancing student engagement via reduction of frustration with programming assignments using machine learning. In: Proceedings of the 9th International Joint Conference on Computational Intelligence, IJCCI, INSTICC, vol. 1, pp. 297–304. SciTePress (2017)

    Google Scholar 

  27. Zimmermann, P., Guttormsen, S., Danuser, B., Gomez, P.: Affective computing - a rationale for measuring mood with mouse and keyboard. Int. J. Occup. Saf. Ergon. 9, 539–551 (2003)

    Article  Google Scholar 

  28. Bixler, R., D’Mello, S.: Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces, pp. 225–234. ACM, New York (2013)

    Google Scholar 

  29. Candel, A., Parmar, V., LeDell, E., Arora, A.: Deep learning with h2o. Technical report, H2O.ai Inc (2016). http://docs.h2o.ai/h2o/latest-stable/h2o-docs/booklets/DeepLearningBooklet.pdf

  30. Kubey, R., Larson, R., Csikszentmihalyi, M.: Experience sampling method applications to communication research questions. J. Commun. 46, 99–120 (1996)

    Article  Google Scholar 

  31. Yang, S.J.: Context aware ubiquitous learning environments for peer-to-peer collaborative learning. Educ. Technol. Soc. 9, 188–201 (2006)

    Google Scholar 

  32. Dillenbourg, P., Schneider, D., Synteta, P.: Virtual learning environments. In: 3rd Hellenic Conference “Information & Communication Technologies in Education”, Kastaniotis Editions, Greece, pp. 3–18 (2002)

    Google Scholar 

  33. Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 677–682. ACM, New York (2005)

    Google Scholar 

  34. Elliott, C., Rickel, J., Lester, J.: Lifelike pedagogical agents and affective computing: an exploratory synthesis. In: Artificial Intelligence Today, pp. 195–211. Springer, Berlin (1999)

    Google Scholar 

  35. D’Mello, S., Jackson, T., Craig, S., Morgan, B., Chipman, P., White, H., Person, N., Kort, B., el Kaliouby, R., Picard, R., et al.: Autotutor detects and responds to learners affective and cognitive states. In: Workshop on Emotional and Cognitive Issues at the International Conference on Intelligent Tutoring Systems, pp. 306–308 (2008)

    Google Scholar 

  36. McQuiggan, S.W., Robison, J.L., Lester, J.C.: Affective transitions in narrative-centered learning environments. Educ. Technol. Soc. 13, 40–53 (2010)

    Google Scholar 

  37. Shen, L., Wang, M., Shen, R.: Affective e-learning: using “emotional” data to improve learning in pervasive learning environment. Educ. Technol. Soc. 12, 176–189 (2009)

    Google Scholar 

  38. Vizer, L.M., Zhou, L., Sears, A.: Automated stress detection using keystroke and linguistic features: an exploratory study. Int. J. Hum.-Comput. Stud. 67, 870–886 (2009)

    Article  Google Scholar 

  39. Busjahn, T., Schulte, C., Sharif, B., Begel, A., Hansen, M., Bednarik, R., Orlov, P., Ihantola, P., Shchekotova, G., Antropova, M., et al.: Eye tracking in computing education. In: Proceedings of the Tenth Annual Conference on International Computing Education Research, pp. 3–10. ACM, New York (2014)

    Google Scholar 

  40. Turner, R., Falcone, M., Sharif, B., Lazar, A.: An eye-tracking study assessing the comprehension of C++ and python source code. In: Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 231–234. ACM, New York (2014)

    Google Scholar 

  41. Blikstein, P.: Using learning analytics to assess students’ behavior in open-ended programming tasks. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge, pp. 110–116. ACM, New York (2011)

    Google Scholar 

  42. Gunetti, D., Picardi, C.: Keystroke analysis of free text. ACM Trans. Inf. Syst. Secur. (TISSEC) 8, 312–347 (2005)

    Article  MATH  Google Scholar 

  43. Janakiraman, R., Sim, T.: Keystroke dynamics in a general setting. In: International Conference on Biometrics, pp. 584–593. Springer, Berlin (2007)

    Google Scholar 

  44. Epp, C., Lippold, M., Mandryk, R.L.: Identifying emotional states using keystroke dynamics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 715–724. ACM, New York (2011)

    Google Scholar 

  45. Salmeron-Majadas, S., Santos, O.C., Boticario, J.G.: Exploring indicators from keyboard and mouse interactions to predict the user affective state. In: Educational Data Mining (2014)

    Google Scholar 

  46. Bakhtiyari, K., Husain, H.: Fuzzy model on human emotions recognition (2014). arXiv:1407.1474

  47. Lim, Y.M.: Detecting and modelling stress levels in e-learning environment users (2017)

    Google Scholar 

  48. Kołakowska, A.: A review of emotion recognition methods based on keystroke dynamics and mouse movements. In: 2013 6th International Conference on Human System Interactions (HSI), pp. 548–555. IEEE (2013)

    Google Scholar 

  49. Carneiro, D., Novais, P.: Quantifying the effects of external factors on individual performance. Futur. Gener. Comput. Syst. 66, 171–186 (2017)

    Article  Google Scholar 

  50. Kołakowska, A.: Usefulness of keystroke dynamics features in user authentication and emotion recognition, pp. 42–52. Springer International Publishing (2018)

    Google Scholar 

  51. Wrobel, M.R.: Applicability of emotion recognition and induction methods to study the behavior of programmers. Appl. Sci. 8, 323 (2018)

    Article  Google Scholar 

  52. Sim, T., Janakiraman, R.: Are digraphs good for free-text keystroke dynamics? In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–6. IEEE (2007)

    Google Scholar 

  53. Tan, P.N., et al.: Introduction to Data Mining. Pearson Education India (2006)

    Google Scholar 

Download references

Acknowledgements

This work has been supported in part by: de Ministerio español de Economía y Competitividad under project TIN2014-56494-C4-3-P (UGR-EPHEMECH), DeepBio (TIN2017-85727-C4-2-P) and by CONACYT PEI Project No. 220590.

Author information

Authors and Affiliations

  1. Department of Graduate Studies, Instituto Tecnologico de Tijuana, Tijuana, Mexico

    Mario Garcia Valdez, Amaury Hernandez Aguila & Alejandra Mancilla Soto

  2. Geneura Team and CITIC, University of Granada, Granada, Spain

    Juan-J. Merelo

Authors
  1. Mario Garcia Valdez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan-J. Merelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amaury Hernandez Aguila
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alejandra Mancilla Soto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mario Garcia Valdez .

Editor information

Editors and Affiliations

  1. IUT Sénart, Université Paris-Est Créteil (UPEC), Créteil, France

    Christophe Sabourin

  2. Department of Computer Architecture and Technology, University of Granada, Granada, Spain

    Juan Julian Merelo

  3. Université Paris-Est Créteil (UPEC), Créteil, France

    Kurosh Madani

  4. University of Reading, Reading, UK

    Kevin Warwick

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Valdez, M.G., Merelo, JJ., Aguila, A.H., Soto, A.M. (2019). Mining of Keystroke and Mouse Dynamics to Increase the Engagement of Students with Programming Assignments. In: Sabourin, C., Merelo, J.J., Madani, K., Warwick, K. (eds) Computational Intelligence. IJCCI 2017. Studies in Computational Intelligence, vol 829. Springer, Cham. https://doi.org/10.1007/978-3-030-16469-0_3

Download citation

Publish with us

Policies and ethics

Search

Navigation