Recognition of Emotions in Gait Patterns by Means of Artificial Neural Nets

Abstract

This paper describes an application of emotion recognition in human gait by means of kinetic and kinematic data using artificial neural nets. Two experiments were undertaken, one attempting to identify participants’ emotional states from gait patterns, and the second analyzing effects on gait patterns of listening to music while walking. In the first experiment gait was analyzed as participants attempted to simulate four distinct emotional states (normal, happy, sad, angry). In the second experiment, participants were asked to listen to different types of music (excitatory, calming, no music) before and during gait analysis. Derived data were fed into different types of artificial neural nets. Results showed not only a clear distinction between individuals, but also revealed clear indications of emotion recognition in nets.

Appendix

The Nets

The processing was implemented using Matlab R2006a, the Neural Network Toolbox V5.0 and the SOM Toolbox (Vesanto et al. 2000). Before the data were fed into the networks, a further normalization to the interval [−1 to 1] was completed in order to prepare the data for the nets. For the kinetic data, the MLP consisted of three layers with 200 neurons (50 x-, 50 y-, and 100 z-data-points) in the first layer, about (n+c)/2 neurons in the hidden layer, where n is the number of input neurons and c is the number of desired classes and also the number of output neurons. The MLPs for the kinematic data were built analogically except for the first layer which contained 168 neurons (21 data points · 8 angles and angular velocities). As an activation function, the tangens hyperbolicus was chosen in all layers. The nets were initialized with the Nguyen–Widrow function (Nguyen and Widrow 1990) and trained with the scaled conjugate gradient algorithm (Møller 1993). Training lengths were set to 500 epochs in general and 600 epochs for the classification with all data from 38 participants respectively. Recognition rates were calculated counting the misclassifications and expressing them as a percentage using cross-validation.

The SOM algorithms were used in two different ways. The normal SOM was used to classify the data as usual; the 2SOM implied two connected SOMs (see Fig. 4). Within this architecture, the first SOM (SOM A) served as data reduction; the second SOM (SOM B) took note of classification tasks. To achieve this, the data vectors were presented as feature-vectors. One single vector then included the values of all angles and angular velocities for a particular point in time, what Bauer and Schöllhorn (1997) call a more ‘coordination-oriented’ approach. In this way, the process information of the movement was represented by the trajectory built by the successively activated neurons in a two-dimensional space. The trajectory of activated neurons was converted to a new data vector by using the x- and y-coordinates from each neuron as new data points. Finally, the second SOM classified the actual measurement. All used SOMs were two-dimensional maps with a hexagonal lattice and a rectangular shape, as recommended by Kohonen et al. (1995). A comparatively small lattice of 5 × 3 neurons was preferred for the data. As a training method batchtrain was chosen. The learning rate was initially set to 0.5 for the rough training phase and then reduced to 0.05 for the fine tuning phase. Training length was set to 10 epochs in rough training and 40 epochs in fine tuning. Further training was not necessary and yielded no better results. The calculation of recognition rates was achieved as follows: During repeated presentation of training data, the distances from each classified gait pattern to the ‘emotion clusters’ (not the clustering on the SOM, but the collection of gait patterns from one simulated emotion) of all emotions were calculated. If the distance to the emotion-cluster, to which the gait pattern belonged to, was not the shortest, the classification was counted as a misclassification.