Abstract
In this paper, we present an approach for predicting users’ level of engagement from nonverbal cues within a game environment. We use a data corpus collected from 28 participants (152 minutes of video recording) playing the popular platform game Super Mario Bros. The richness of the corpus allows extraction of several visual and facial expression features that were utilised as indicators of players’ affects as captured by players’ self-reports. Neuroevolution preference learning is used to construct accurate models of player experience that approximate the relationship between extracted features and reported engagement. The method is supported by a feature selection technique for choosing the relevant subset of features. Different setup settings were implemented to analyse the impact of the type of the features and the position of the extraction window on the modelling accuracy. The results obtained show that highly accurate models can be constructed (with accuracies up to 96.82%) and that players’ nonverbal behaviour towards the end of the game is the most correlated with engagement. The framework presented is part of a bigger picture where the generated models are utilised to tailor content generation to a player’s particular needs and playing characteristics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Abadi, M.K., Staiano, J., Cappelletti, A., Zancanaro, M., Sebe, N.: Multimodal engagement classification for affective cinema. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 411–416. IEEE (2013)
Asteriadis, S., Tzouveli, P., Karpouzis, K., Kollias, S.: Estimation of behavioral user state based on eye gaze and head poseapplication in an e-learning environment. Multimedia Tools and Applications 41(3), 469–493 (2009)
Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., Karpouzis, K.: Multimodal emotion recognition from expressive faces, body gestures and speech. In: Artificial Intelligence and Innovations 2007: From Theory to Applications, pp. 375–388. Springer (2007)
Castellano, G., Pereira, A., Leite, I., Paiva, A., McOwan, P.W.: Detecting user engagement with a robot companion using task and social interaction-based features. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 119–126. ACM (2009)
Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19(3), 267–303 (2009)
Mello, D., Craig, S.K., Graesser, S.D., Multi-method, A.C.: assessment of affective experience and expression during deep learning. Int. J. Learn. Technol. 4(3/4), 165–187 (2009)
Fürnkranz, J., Hüllermeier, E.: Pairwise preference learning and ranking. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) ECML 2003. LNCS (LNAI), vol. 2837, pp. 145–156. Springer, Heidelberg (2003)
Grafsgaard, J.F., Boyer, K.E., Lester, J.C.: Toward a machine learning framework for understanding affective tutorial interaction. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 52–58. Springer, Heidelberg (2012)
Ioannou, S., Caridakis, G., Karpouzis, K., Kollias, S.: Robust feature detection for facial expression recognition. Journal on Image and Video Processing (2) (2007)
Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies 66(9), 641–661 (2008)
Kapoor, A., Burleson, W., Picard, R.: Automatic prediction of frustration. International Journal of Human-Computer Studies (8), 724–736 (2007)
Karpouzis, K., Shaker, N., Yannakakis, G.N., Asteriadis, S.: The platformer experience dataset. IEEE Transactions on Computational Intelligence and AI in Games (2014)
Küblbeck, C., Ernst, A.: Face detection and tracking in video sequences using the modifiedcensus transformation. Image and Vision Computing 24(6), 564–572 (2006)
Lisetti, C.L., Nasoz, F.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing 2004, 1672–1687 (2004)
Mandryk, R., Inkpen, K., Calvert, T.: Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology (2), 141–158 (2006)
Martinez, H., Yannakakis, G.: Mining multimodal sequential patterns: A case study on affect detection. In: Proceedings of the 13th International Conference in Multimodal Interaction, ICMI 2011, Alicante. ACM Press (November 2011)
McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., Graesser, A.: Facial features for affective state detection in learning environments. In: Proceedings of the 29th Annual Cognitive Science Society, pp. 467–472 (2007)
Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience for content creation. IEEE Transactions on Computational Intelligence and AI in Games 2(1), 54–67 (2010)
Peters, C., Asteriadis, S., Karpouzis, K., de Sevin, E.: Towards a real-time gaze-based shared attention for a virtual agent. In: Workshop on Affective Interaction in Natural Environments (AFFINE), ACM International Conference on Multimodal Interfaces, ICMI 2008 (2008)
Shaker, N., Yannakakis, G.N., Togelius, J.: Towards Automatic Personalized Content Generation for Platform Games. In: Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE (2010)
Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: A game-based corpus for analysing the interplay between game context and player experience. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part II. LNCS, vol. 6975, pp. 547–556. Springer, Heidelberg (2011)
Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: Fusing visual and behavioral cues for modeling user experience in games. IEEE Transactions on Cybernetics 43(6), 1519–1531 (2013)
Shaker, N., Togelius, J., Yannakakis, G.N., Weber, B., Shimizu, T., Hashiyama, T., Sorenson, N., Pasquier, P., Mawhorter, P., Takahashi, G., Smith, G., Baumgarten, R.: The 2010 Mario AI championship: Level generation track. IEEE Transactions on Computational Intelligence and Games (2011)
Shaker, N., Yannakakis, G.N., Togelius, J.: Crowd-sourcing the aesthetics of platform games. IEEE Transactions on Computational Intelligence and AI in Games (2013)
Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing 3(1), 42–55 (2012)
Tognetti, S., Garbarino, M., Bonanno, A.T., Matteucci, M., Bonarini, A.: Enjoyment recognition from physiological data in a car racing game. In: Proceedings of the 3rd international Workshop on Affective Interaction in Natural Environments, pp. 3–8. ACM (2010)
Yannakakis, G.N., Togelius, J.: Experience-Driven Procedural Content Generation. IEEE Transactions on Affective Computing (2011)
Yannakakis, G.N., Hallam, J.: Entertainment modeling through physiology in physical play. Int. J. Hum.-Comput. Stud. 66, 741–755 (2008)
Yannakakis, G.N., Maragoudakis, M., Hallam, J.: Preference learning for cognitive modeling: a case study on entertainment preferences. Trans. Sys. Man Cyber. Part A 39, 1165–1175 (2009)
Yannakakis, G.N., Maragoudakis, M., Hallam, J.: Preference learning for cognitive modeling: a case study on entertainment preferences. Trans. Sys. Man Cyber. Part A 39, 1165–1175 (November 2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Shaker, N., Shaker, M. (2014). Towards Understanding the Nonverbal Signatures of Engagement in Super Mario Bros. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, GJ. (eds) User Modeling, Adaptation, and Personalization. UMAP 2014. Lecture Notes in Computer Science, vol 8538. Springer, Cham. https://doi.org/10.1007/978-3-319-08786-3_38
Download citation
DOI: https://doi.org/10.1007/978-3-319-08786-3_38
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08785-6
Online ISBN: 978-3-319-08786-3
eBook Packages: Computer ScienceComputer Science (R0)