Abstract
This paper aims to investigate students’ behavioral engagement (On-Task vs. Off-Task) in authentic classrooms. We propose a two-phased approach for automatic engagement detection: In Phase 1, contextual logs are utilized to assess active usage of the content platform. If there is active use, the appearance information is utilized in Phase 2 to infer behavioral engagement. Through authentic classroom pilots, we collected around 170 hours of in-the-wild data from 28 students in two different classrooms using two different content platforms (one for Math and one for English as a Second Language (ESL)). Our data collection application captured appearance data from a 3D camera and context data from uniform resource locator (URL) logs. We experimented with two test cases: (1) Cross-classroom, where trained models were tested on a different classroom’s data; (2) Cross-platform, where the data collected in different subject areas (Math or ESL) were utilized in training and testing, respectively. For the first case, the behavioral engagement was detected with an F1-score of 77%, using only appearance. Incorporating the contextual information improved the overall performance to 82%. For the second case, even though the subject areas and content platforms changed, the proposed appearance classifier still achieved 72% accuracy (compared to 77%). Our experiments proved that the accuracy of the proposed model is not adversely impacted considering different set of students or different subject areas.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Training set sizes differ for each student, as leave-one-subject-out approach is utilized in model training.
References
Madden, M., Lenhart, A., Duggan, M., Cortesi, S., Gasser, U.: Teens and Technology. Pew Internet & American Life Project, Washington, DC (2013)
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. Int. J. Learn. Technol. 4(3–4), 129–164 (2009)
D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: a gaze-reactive intelligent tutoring system. Int. J. Hum. Comput. Stud. 70(5), 377–398 (2012)
Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept, state of the evidence. Rev. Educ. Res. 74(1), 59–109 (2004)
Bosch, N., D’Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., Zhao, W.: Automatic detection of learning-centered affective states in the wild. In: International Conference on Intelligent User Interfaces (2015)
Chen, J., Luo, N., Liu, Y., Liu, L., Zhang, K., Kolodziej, J.: A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1–2), 215–233 (2016)
Alyuz, N., Okur, E., Oktay, E., Genc, U., Aslan, S., Mete, S.E., Stanhill, D., Arnrich, B., Esme, A.A.: Towards an emotional engagement model: can affective states of a learner be automatically detected in a 1:1 learning scenario. In: International Conference on User Modeling, Adaptation, and Personalization - Workshop on Personalization Approaches in Learning Environments (2016)
Alyuz, N., Okur, E., Oktay, E., Genc, U., Aslan, S., Mete, S.E., Arnrich, B., Esme, A.A.: Semi-supervised model personalization for improved detection of learner’s emotional engagement. In: International Conference on Multimodal Interaction (2016)
Reinhard Pekrun, L.L.-G.: Academic emotions and student engagement. In: Handbook of Research on Student Engagement, pp. 259–282. Springer, New York (2012)
Rodrigo, M.M.T., Baker, R.S.J.D., Rossi, L.: Student off-task behavior in computer-based learning in the Philippines: comparison to prior research in the USA. Teachers Coll. Rec. 115(10), 1–27 (2013)
Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Embodied affect in tutorial dialogue: student gesture and posture. In: International Conference on Artificial Intelligence in Education (2013)
Paquette, L., Rowe, J., Baker, R., Mott, B., Lester, J., DeFalco, J., Brawner, K., Sottilare, R., Georgoulas, V.: Sensor-free or sensor-full: a comparison of data modalities in multi-channel affect detection. In: International Conference on Educational Data Mining (2015)
Miller, W.L., Baker, R.S., Labrum, M.J., Petsche, K., Liu, Y.H., Wagner, A.Z.: Automated detection of proactive remediation by teachers in Reasoning Mind classrooms. In: International Conference on Learning Analytics and Knowledge (2015)
Fancsali, S.E.: Data-Driven causal modeling of “gaming the system” and off-task behavior in cognitive tutor algebra. In: NIPS Workshop on Data Driven Education (2013)
Gobert, J.D., Baker, R.S., Wixon, M.B.: Operationalizing and detecting disengagement within online science microworlds. Educ. Psychol. 50(1), 43–57 (2015)
Mohammadi-Aragh, M.J., Williams, C.B.: Student attention in unstructured-use, computer-infused classrooms. In: American Society of Engineering Education Annual Conference (2013)
Su, Y.N., Hsu, C.C., Chen, H.C., Huang, K.K., Huang, Y.M.: Developing a sensor-based learning concentration detection system. Eng. Comput. 31(2), 216–230 (2014)
Aslan, S., Cataltepe, Z., Diner, I., Dundar, O., Esme, A.A., Ferens, R., Kamhi, G., Oktay, E., Soysal, C., Yener, M.: Learner engagement measurement and classification in 1:1 learning. In: International Conference on Machine Learning and Applications (2014)
Intel Corporation: Intel RealSense SDK: Design Guidelines (2014). https://software.intel.com/sites/default/files/managed/27/50/Intel%20RealSense%20SDK%20Design%20Guidelines%20F200%20v2.pdf. Accessed 2016
Chen, C., Liaw, A., Breiman, L.: Using Random Forest to Learn Imbalanced Data. University of California, Berkeley (2004)
Aslan, S., Mete, S.E., Okur, E., Oktay, E., Alyuz, N., Genc, U., Stanhill, D., Esme, A.A.: Human expert labeling process (HELP): towards a reliable higher-order user state labeling by human experts. In: International Conference on Intelligent Tutoring Systems - Workshops (2016)
Krippendorff, K.: Reliability in content analysis. Hum. Commun. Res. 30(3), 411–433 (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Okur, E., Alyuz, N., Aslan, S., Genc, U., Tanriover, C., Arslan Esme, A. (2017). Behavioral Engagement Detection of Students in the Wild. In: André, E., Baker, R., Hu, X., Rodrigo, M., du Boulay, B. (eds) Artificial Intelligence in Education. AIED 2017. Lecture Notes in Computer Science(), vol 10331. Springer, Cham. https://doi.org/10.1007/978-3-319-61425-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-61425-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61424-3
Online ISBN: 978-3-319-61425-0
eBook Packages: Computer ScienceComputer Science (R0)