EmAP-ML: A Protocol of Emotions and Behaviors Annotation for Machine Learning Labels

Abstract

The detection of students’ emotions in computer-based learning environments is a complex task. Although emotions can be detected from sensors, a less intrusive method is to train supervised machine learning algorithms for the emotions prediction based on the log of students’ actions on the system. For these algorithms to work as expected, they need to be trained with a large amount of reliable ground truth labels. Generally, labels are generated by students themselves or by coders monitoring students, watching videos from the students, or reviewing logs of students’ actions. Younger learners (i.e., children) are unable to label their emotions properly. Still, it is difficult for a coder to identify students’ emotions only from their face since the emotional facial expression is generally subtle in a learning setting. This article describes EmAP-ML (Emotions Annotation Protocol for Machine Learning), a protocol for coders to annotate students’ learning emotions and behaviors based on video records, which contains facial expressions, ambient audio, and computer screen. The screen and ambient audio records allow coders to infer students’ appraisal (an evaluation that elicits an emotion) to identify emotions even when the facial expression is subtle. This protocol was evaluated by two coders who annotated videos obtained from 55 students while using a tutoring system, having achieved an agreement coefficient of 0.62, measured through Cohen’s Kappa statistics.