Abstract
Robots that can recognize emotions can improve humans’ mental health by providing empathy and social communication. Emotion recognition by robots is challenging because unlike in human-computer environments, facial information is not always available. Instead, our method proposes using speech and gait analysis to recognize human emotion. Previous research suggests that the dynamics of emotional human speech also underlie emotional gait (walking). We investigate the possibility of combining these two modalities via perceptually common parameters: Speed, Intensity, irRegularity, and Extent (SIRE). We map low-level features to this 4D cross-modal emotion space and train a Gaussian Mixture Model using independent samples from both voice and gait. Our results show that a single, modality-mixed trained model can perform emotion recognition for both modalities. Most interestingly, recognition of emotion in gait using a model trained uniquely on speech data gives comparable results to a model trained on gait data alone, providing evidence for a common underlying model for emotion across modalities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Clynes, M.: Sentics: The Touch of the Emotions. Prism Press, UK (1989)
Pollick, F.E., Paterson, H.M., Bruderlin, A., Sanford, A.J.: Perceiving affect from arm movement. J. Personal. 82, 51–61 (2001)
Van Bezooijen, R., Van Otto, S.A., Heenan, T.A.: Recognition of vocal dimensions of emotion: A three-nation study to identify universal characteristics. J. Cross-Cultural Psych. 14, 387–406 (1983)
Juslin, P.N., Laukka, P.: Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129(5), 770–814 (2003)
Spencer, H.: The origin and function of music. Fraser’s Magazine 56, 396–408 (1857)
Scherer, K.H.: Vocal affect expression: A review and a model for future research. Psychol. Bull. 99, 143–165 (1986)
Snowdon, C.T.: Expression of emotion in non-human animals. In: Davidson, R.J., Sherer, K.H., Goldsmith, H.H. (eds.) Handbook of affective sciences, pp. 457–480. Oxford University Press, London (2003)
Breazeal, C.: Designing sociable robots, 1st edn. The MIT Press, Cambridge (2004)
Lim, A., Ogata, T., Okuno, H.G.: Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music. EURASIP J. Audio, Speech, and Music Proc. 2012(3) (2012)
Lim, A., Ogata, T., Okuno, H.G.: Converting emotional voice to motion for robot telepresence. In: Humanoids, Bled, pp. 472–479 (2011)
Cowie, R., et al.: Emotion recognition in human-computer interaction. IEEE Signal Proc. Magazine 18(1), 32–80 (2001)
Fernandez, R., Picard, R.W.: Classical and Novel Discriminant Features for Affect Recognition from Speech. In: INTERSPEECH, pp. 4–8 (2005)
Mion, L., De Poli, G.: Score-independent audio features for description of music expression. IEEE Trans. Audio Speech Lang. Process. 16(2), 458–466 (2008)
Livingstone, S.R., Brown, A.R., Muhlberger, R., Thompson, W.F.: Modifying score and performance changing musical emotion: a computational rule system for modifying score and performance. Comput. Music J. 34(1), 41–65 (2010)
Amaya, K., Bruderlin, A., Calvert, T.: Emotion from motion. Graph. In: Interface, pp. 222–229 (1996)
Pelachaud, C.: Studies on gesture expressivity for a virtual agent. Speech Commun. 51(7), 630–639 (2009)
Camurri, A., Volpe, G.: Communicating expressiveness and affect in multimodal interactive systems. Multimedia 12(1), 43–53 (2005)
Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., Amir, N., Karpouzis, K.: The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 488–500. Springer, Heidelberg (2007)
Roether, C.L., Omlor, L., Christensen, A., Giese, M.A.: Critical features for the perception of emotion from gait. J. Vision 9(6), 15, 1–32 (2009)
Montepare, J.M., Goldstein, S.B.: The identification of emotions from gait information. J. Nonverbal Behav. 11(1), 33–42 (1987)
Janssen, D., et al.: Recognition of emotions in gait patterns by means of artificial neural nets. J. Nonverbal Behav. 32, 79–92 (2008)
Unuma, M., Anjyo, K., Takeuchi, R.: Fourier principles for emotion-based human figure animation. In: SIGGRAPH, Los Angeles, pp. 91–96 (1995)
Montepare, J., Koff, E., Zaichik, D., Albert, M.: The use of body movements and gestures as cues to emotions in younger and older adults. J. Nonverbal Behav. 23(2), 133–152 (1999)
Karg, M., Kuhnlenz, K., Buss, M.: Recognition of affect based on gait patterns. IEEE Trans. Sys., Man, Cyber. 40(4), 1050–1061 (2010)
Ma, Y., Paterson, H.M., Pollick, F.E.: A motion-capture library for the study of identity, gender, and emotion perception from biological motion. Behav. Res. Meth., Inst., & Comp. 38, 134–141 (2006)
Bernhardt, D.: Detecting emotions from everyday body movements. Presenccia PhD Sym., Barcelona (2007)
Pedregosa, F., et al.: Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Parrot, W.G.: Emotions in social psychology. Philadelphia Press, Philadelphia (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lim, A., Okuno, H.G. (2012). Using Speech Data to Recognize Emotion in Human Gait. In: Salah, A.A., Ruiz-del-Solar, J., Meriçli, Ç., Oudeyer, PY. (eds) Human Behavior Understanding. HBU 2012. Lecture Notes in Computer Science, vol 7559. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34014-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-34014-7_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34013-0
Online ISBN: 978-3-642-34014-7
eBook Packages: Computer ScienceComputer Science (R0)