DeepConnection: classifying momentary relationship state from images of romantic couples

Abstract

Detecting momentary relationship state and quality in romantic couples is an important endeavor for relationship research, couple therapy, and of course couples themselves. Yet current methods to achieve this are intrusive, asynchronous, plagued by ceiling effects, and only assess subjective responses to questionnaires while trying to capture the objective state of a relationship. According to social appraisal theory, human beings rely on emotional responses to assess interpersonal situations, a key element for relationship functioning in couples. Using couples is particularly advantageous as strong emotional reactions are triggered in romantic relationships. Here, we employ deep learning methods to assess the momentary relationship state of romantic couples from predominantly stock images via facial and bodily emotion expression and other features. Our new model, DeepConnection, comprises pre-trained residual neural networks, spatial pyramid pooling layers, and power mean transformations to extract relevant features from images for binary classification. With this, we achieved an average accuracy of nearly 97% on a separate validation dataset. We also engaged in model interpretation using Gradient-weighted Class Activation Mapping (Grad-CAM) to identify which features allow DeepConnection to detect binarized momentary relationship state. To demonstrate generalizability and robustness, we used DeepConnection to analyze videos of couples exhibiting a range of different postures and facial expressions. Here, we achieved an average accuracy of about 85% with a trained DeepConnection model. The work presented here could inform couples, advance relationship research, and find application in couple therapy to assist the therapist.

Appendix

See below Table3; Figs. 4, 5, 6, 7, 8 and 9.

Table 3 Performance metrics of alternative deep learning models used to classify momentary relationship state of romantic couples

Fig. 4

Interpretation of ResNet-34 model predictions using Grad-CAM. Note Four representative images of couples not used in training (two labeled as happy (a–b) and two as unhappy (c–d) were used as inputs a trained ResNet-34 model. Gradients at the last convolutional layer were used to generate class-specific saliency heatmaps and layered on top of the original images together with the predicted class label and the model confidence in percent. Heatmaps of predicted class labels are always depicted as the left-hand image irrespective of the true label

Fig. 5

Interpretation of SPP model predictions using Grad-CAM. Note Four representative images of couples not used in training (two labeled as happy (a–b) and two as unhappy (c–d)) were used as inputs for a trained spatial pyramid pooling (SPP) model. Gradients at the last convolutional layer were used to generate class-specific saliency heatmaps and layered on top of the original images together with the predicted class label and the model confidence in percent. Heatmaps of predicted class labels are always depicted as the left-hand image irrespective of the true label

Fig. 6

Interpretation of DeepConnection model predictions using Grad-CAM. Note Four representative images of couples not used in training (two labeled as happy (a–b) and two as unhappy (c–d)) were used as inputs for the trained DeepConnection model. Gradients at the last convolutional layer were used to generate class-specific saliency heatmaps and layered on top of the original images together with the predicted class label and the model confidence in percent. Heatmaps of predicted class labels are always depicted as the left-hand image irrespective of the true label

Fig. 7

Activations and weights of the last fully connected layer of the SPP model for selected images. Note Four representative images of couples not used in training (two labeled as happy (a–b) and two as unhappy (c–d)) were used as inputs for the trained spatial pyramid pooling (SPP) model. Plotted points represent the weights of the last fully connected layer of size 100 × 2 in the trained SPP model which, together with their inputs, determine the class probabilities. Points were colored from yellow to red by the magnitude of activations from the presented image directly prior to the last fully connected layer. To determine class probabilities, activations will be multiplied by class weights and summed

Fig. 8

Activations and weights of the last fully connected layer of the DeepConnection model for selected images. Note Four representative images of couples not used in training (two labeled as happy (a–b) and two as unhappy (c–d)) were used as inputs for the trained DeepConnection model. Plotted points represent the weights of the last fully connected layer of size 100 × 2 in the trained DeepConnection model which, together with their inputs, determine the class probabilities. Points were colored from yellow to red by the magnitude of activations from the presented image directly prior to the last fully connected layer. To determine class probabilities, activations will be multiplied by class weights and summed

Fig. 9

Training the DeepConnection model is accelerated by multi-sample dropout. Note. Exchanging the dropout layer immediately prior to the last fully connected layer in DeepConnection with a multi-sample dropout layer (using eight dropout channels) enabled faster training convergence, both with regard to the number of epochs (a) as well as to the total amount of time required (b). Data shown was gathered from five training runs per dropout method and is plotted as mean ± standard deviation. The calculation of p-values was done using a two-tailed Student’s t-test