Towards a conceptual framework of research on social signal processing | Journal on Multimodal User Interfaces Skip to main content

Advertisement

Springer Nature Link
Log in

Towards a conceptual framework of research on social signal processing

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Social Signal Processing (SSP) as an emerging research area can draw on material from many disciplines, but it needs effective ways to organise the material. We propose a framework that integrates concepts, drawn primarily from psychology, but with input from other disciplines, in a way that indicates how they relate to SSP. We identify seven core constructs: state; indicators that convey info about it; process by which indicators are generated; types of inference involved in identifying states from indicators; perceptual issues, including recovery of indicators from flux of activity and accuracy of impressions about the state; the role of macro-context; and the different levels at which analysis may be couched (the individual, the dyad, the group, the organisation). These may or may not be reflected in the structure of an SSP system, but the natural subtlety and context-sensitivity of human communication makes it important that people designing systems should consider how they relate to its task, and decide how best to take account of them. The analysis works from simple models which consider only states and indicators to models which embrace all seven constructs. At each stage it points to the literatures which discuss the relevant issues. It is fully acknowledged that attempting such a synthesis raises many problems (not least of terminology), and that alternative frameworks deal at least as well with subsets of the issues. It remains to be seen whether they could be extended to cover a comparable range.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Cowie R (2009) Perception of emotion: towards a realistic understanding of the task. Philos R Soc B 364:3515–3525

    Google Scholar 

  2. Brunet P, McKeown G, Cowie R, Donnan H, Douglas-Cowie E (2009) Social signal processing: what are the relevant variables? and in what ways do they relate? In: Proceedings ACII 2009, vol 2 IEEE Catalog Number: CFP0964H-USB ISBN: 978-1-4244-4799-2, Library of Congress: 2009905375, pp 77–82

  3. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor J (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18:32–80

    Google Scholar 

  4. Cowie R (2010) Describing the forms of emotional colouring that pervade everyday life. In: Goldie P (ed) Oxford handbook of philosophy of emotion. Oxford University Press, London, pp 63–94

    Google Scholar 

  5. Castelfranchi C (1995) Commitment: individual intentions to groups and organization. In: Proc 1st internat conf on multiagent systems AAAI, pp 41–48

    Google Scholar 

  6. Paglieri F, Castelfranchi C (2008) Changing the mind: mindreading, intentional action and consciousness. Sist Intell 20:489–520

    Google Scholar 

  7. Pezzulo G, Castelfranchi C (2009) Intentional action: from anticipation to goal-directed behavior. Psychol Res 73:437–440

    Google Scholar 

  8. O’Connell DC, Kowal S, Kaltenbacher E (1990) Turn-taking: A critical analysis of the research tradition. J Psycholinguist Res 19:345–373

    Google Scholar 

  9. de Kok I, Heylen D (2009) Multimodal end-of-turn prediction in multi-party meetings. In: Proc of the 2009 internat conf on multimodal interfaces, Cambridge, Massachusetts, USA, 02–04 November 2009. doi:10.1145/1647314.1647332

    Google Scholar 

  10. Eysenck HJ, Eysenck SB (1971) The orthogonality of psychoticism and neuroticism: A factor study. Percept Mot Skills 33:461–462

    Google Scholar 

  11. McCrae RR, Costa PT (1990) Personality in adulthood. The Guilford Press, New York

    Google Scholar 

  12. Douglas-Cowie E (1984) The sociolinguistic situation in Northern Ireland. In: Trudgill P (ed) Language in the British Isles. Cambridge University Press, Cambridge, pp 533–545

    Google Scholar 

  13. Giles H (1973) Communicative effectiveness as a function of accented speech. J Soc Clin Psychol 10:187–188

    Google Scholar 

  14. Labov W (1966) The social stratification of English in New York City. New York Center for Applied Linguistics Washington DC

  15. Trudgill P (1983) On dialect: social and geographical perspectives. University Press, New York

    Google Scholar 

  16. Crano WD, Prislin R (2006) Attitudes and persuasion. Annu Rev Psychol 57:345–374

    Google Scholar 

  17. Petty RE, Briñol P (2008) Persuasion: from single to multiple to metacognitive processes. Perspect Psychol Sci 3:137–147

    Google Scholar 

  18. Connor U, Lauer J (1985) Understanding persuasive essay writing: Linguistic/rhetorical approach. Text 5:309–326

    Google Scholar 

  19. Pornpitakpan C (2004) The persuasiveness of source credibility: A critical review of five decades’ evidence. J Appl Soc Psychol 34:243–281

    Google Scholar 

  20. Burgoon JK, Birk T, Pfau M (1990) Nonverbal behaviors, persuasion, and credibility. Hum Commun Res 17:140–169

    Google Scholar 

  21. Burgoon JK, Dunbar NE (2006) Nonverbal expressions of dominance and power in human relationships. In: Patterson ML (ed) The sage handbook of nonverbal communication. Sage Publications Inc, Thousand Oaks, pp 279–297

    Google Scholar 

  22. Mast M (2002) Dominance as expresses and inferred through speaking time: A meta-analysis. Hum Commun Res 28:420–450

    Google Scholar 

  23. Ostrov JM, Collins WA (2007) Social dominance in romantic relationships: A prospective longitudinal study of non-verbal processes. Soc Dev 16:580–595

    Google Scholar 

  24. Ekman P (1980) The face of man: expressions of universal emotions in a New Guinea village. Garland STPM Press, New York

    Google Scholar 

  25. Ekman P (1992) An argument for basic emotions. Cognition Emotion 6:169-200

    Google Scholar 

  26. Mowrer OH (1960) Learning theory and behavior. Wiley, New York

    Google Scholar 

  27. Izard CE (1977) Human emotions. Plenum Press, New York

    Google Scholar 

  28. Calder AJ, Young AW, Perrett DI, Etcoff NL (1996) Categorical perception of morphed facial expressions. Vis Cogn 3:81–117

    Google Scholar 

  29. Durand K, Gallay M, Seigneuric A, Robichon F, Baudouin J (2007) The development of facial emotion recognition: the role of configural information. J Exp Child Psychol 97:14–27

    Google Scholar 

  30. Ekman P (1993) Facial expression and emotion. Am Psychol 48:384–392

    Google Scholar 

  31. Gao X, Maurer D (2009) Influence of intensity on children’s sensitivity to happy, sad, and fearful facial expressions. J Exp Child Psychol

  32. Scherer KR (1999) Appraisal theories. In: Dalgleish T, Power M (eds) Handbook of cognition and emotion. Wiley, Chichester, pp 637–663

    Google Scholar 

  33. Frijda NH, Kuipers P, ter Schure E (1989) Relations among emotion, appraisal, and emotional action readiness. J Pers Soc Psychol 57:212–228

    Google Scholar 

  34. Ortony A, Clore GL, Collins A (1988) The Cognitive Structure of Emotions. Cambridge University Press, Cambridge

    Google Scholar 

  35. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161–1178

    Google Scholar 

  36. Bimler D, Kirkland J (1997) Multidimensional scaling of hierarchical sorting data applied to facial expressions. Scand J Psychol 38:349–357

    Google Scholar 

  37. Katsikitis M (1997) The classification of facial expressions of emotion: A multidimensional-scaling approach. Perception 26:613–626

    Google Scholar 

  38. Schlosberg H (1952) The description of facial expressions in terms of 2 dimensions. J Exp Psychol 44:229–237

    Google Scholar 

  39. Schlosberg H (1954) Three dimensions of emotions. Psychol Rev 61:81–88

    Google Scholar 

  40. Green RS, Cliff N (1975) Multidimensional comparisons of structures of vocally and facially expressed emotion. Percept Psychophys 17:429–438

    Google Scholar 

  41. Laukka P, Juslin PN, Bresin R (2005) A dimensional approach to vocal expression of emotion. Cogn Emot 19:633–653

    Google Scholar 

  42. Sweeney K, Whissell C (1984) A dictionary of affect in language: I. Establishment and preliminary validation. Percept Mot Skills 59:695–698

    Google Scholar 

  43. Whissell C (1999) Phonosymbolism and the emotional nature of sounds: evidence of the preferential use of particular phonemes in texts of differing emotional tone. Percept Mot Skills 89:19–48

    Google Scholar 

  44. Biber D (2006) Stance in spoken and written university registers. J Engl Acad Purp 5:97–116

    Google Scholar 

  45. Clift R (2006) Indexing stance: reported speech as an interactional evidential. J Socioling 10:569–595

    Google Scholar 

  46. Lempert M (2009) On ‘flip-flopping’: branded stance-taking in US electoral politics. J Socioling 13:223–248

    Google Scholar 

  47. Schmitt BH (1987) The ecological approach to social perception: A conceptual critique. J Theory Soc Behav 17:265

    Google Scholar 

  48. Runeson S, Juslin P, Olsson H (2000) Visual perception of dynamic properties: cue heuristics versus direct-perceptual competence. Psychol Rev 107:525–555

    Google Scholar 

  49. Rockwell P (2007) Vocal features of conversational sarcasm: A comparison of methods. J Psycholinguist Res 36:361–369

    Google Scholar 

  50. Batliner A, Steidl S, Schuller B, Seppi D, Vogt T, Wagner J, Devillers L, Vidrascu L, Aharonson V, Kessous L, Ami N (2011) Whodunnit—searching for the most important feature types signalling emotion-related user states in speech. Comput Speech Lang 25:4–28

    Google Scholar 

  51. Young AW, Rowland D, Calder AT, Etcoff NL, Seth A, Perrett DT (1997) Facial expressions megamix: tests of dimensional and category accounts of emotion recognition. Cognition 63:271–313

    Google Scholar 

  52. Reichenbach L, Masters J (1983) Children’s use of expressive and contextual cues in judgements of emotion. Child Dev 54:993–1004

    Google Scholar 

  53. Cowie R, Douglas-Cowie E, McKeown G, Gibney C (2009) The challenges of dealing with distributed signs of emotion: theory and empirical evidence. In: Proceedings ACII 2009, vol 1. IEEE Press, New York, pp 351–356

    Google Scholar 

  54. Scherer KR, Ellring H (2007) Multimodal expression of emotion: affect programs or componential appraisal patterns? Emotion 7:158–171

    Google Scholar 

  55. Ambadar Z, Cohn JF, Reed LI (2009) All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. J Nonverbal Behav 33:17–34

    Google Scholar 

  56. Baron-Cohen S, Wheelwright S, Jolliffe T (1997) Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger syndrome. Vis Cogn 4:311–331

    Google Scholar 

  57. Tasker SL, Schmidt LA (2008) The ‘dual usage problem’ in the explanations of ‘joint attention’ and children’s socioemotional development: A reconceptualization. Dev Rev 28:263–288

    Google Scholar 

  58. Rowlands M (2006) Body language: representation in action. MIT Press, Cambridge

    Google Scholar 

  59. Hansen JE, Schuldt WJ (1982) Physical distance, sex, and intimacy in self-disclosure. Psychol Rep 51:3–6

    Google Scholar 

  60. Kendon A (1970) Movement coordination in social interaction: some examples described. Acta Psychol 32:101–125

    Google Scholar 

  61. de Jong PJ (1999) Communicative and remedial effects of social blusing. J Nonverbal Behav 23:197–217

    Google Scholar 

  62. Cornelius RR (1996) The science of emotion: Research and tradition in the psychology of emotions. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  63. Ekman P, Friesen WV (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6:238–252

    Google Scholar 

  64. Krumhuber EG, Manstead ASR (2009) Can duchenne smiles be feigned? New evidence on felt and false smiles. Emotion 9:807–820

    Google Scholar 

  65. Berenbaum H, Rotter A (1992) The relationship between spontaneous facial expressions of emotion and voluntary control of facial muscles. J Nonverbal Behav 16:179–190

    Google Scholar 

  66. Schmidt KL, Cohn JF, Tian Y (2003) Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol 65:49–66

    Google Scholar 

  67. Gazzaniga MS, Smylie CS (1990) Hemispheric mechanisms controlling voluntary and spontaneous facial expressions. J Cogn Neurosci 2:239–245

    Google Scholar 

  68. Korb S, Grandjean D, Scherer K (2008) Motor commands of facial expressions: the bereitschaftspotential of posed smiles. Brain Topogr 20:232–238

    Google Scholar 

  69. Ekman P, Friesen WV, O’Sullivan M (1988) Smiles when lying. J Pers Soc Psychol 54:414–420

    Google Scholar 

  70. Harris CR, Alvarado N (2005) Facial expressions, smile types, and self-report during humour, tickle, and pain. Cogn Emot 19:655–669

    Google Scholar 

  71. Poole GD, Craig KD (1992) Judgments of genuine, suppressed, and faked facial expressions of pain. J Pers Soc Psychol 63:797–805

    Google Scholar 

  72. Sumner P, Husain M (2008) At the edge of consciousness: automatic motor activation and voluntary control. Neuroscientist 14:474–486

    Google Scholar 

  73. Psillos S (2002) Simply the best: A case for abduction. In: Sadri F, Kakas A (eds) Computational logic: From logic programming into the future. LNAI, vol 2408. Springer, Berlin, pp 605–625

    Google Scholar 

  74. Psillos S (2004) Inference to the best explanation and bayesianism. In: Stadler F (ed) Induction and deduction in the sciences. Kluwer, Dordrecht, pp 83–91

    Google Scholar 

  75. Hahn S, Carlson C, Singer S, Gronlund SD (2006) Aging and visual search: automatic and controlled attentional bias to threat faces. Acta Psychol 123:312–336

    Google Scholar 

  76. Lamme VAF (2003) Why visual attention and awareness are different. Trends Cogn Sci 7:12–18

    Google Scholar 

  77. Silva PS, Ribeiro-do-Valle LE (2008) Evidence for divided automatic attention. Braz J Med Biol Res 41:159–169

    Google Scholar 

  78. Turatto M, Benso F, Facoetti A, Galfano G, Mascetti GG, Umiltà C (2000) Automatic and voluntary focusing of attention. Percept Psychophys 62:935–952

    Google Scholar 

  79. Yantis S, Jonides J (1990) Abrupt visual onsets and selective attention: voluntary versus automatic allocation. J Exp Psychol Hum 16:121–134

    Google Scholar 

  80. Ceci SJ (1983) Automatic and purposive semantic processing characteristics of normal and language/learning-disabled children. Dev Psychol 19:427–439

    Google Scholar 

  81. Fischler I, Bloom PA (1979) Automatic and attentional processes in the effects of sentence contexts on word recognition. J Verbal Learn Verbal Behav 18:1–20

    Google Scholar 

  82. Muller-Gass A, Roye A, Kirmse U, Saupe K, Jacobsen T, Schröger E (2007) Automatic detection of lexical change: an auditory event-related potential study. NeuroReport 18:1747–1751

    Google Scholar 

  83. Kuyper P (1972) The cocktail party effect. Audiology 11:277–282

    Google Scholar 

  84. Morey RA, Mitchell TV, Inan S, Lieberman JA, Belger A (2008) Neural correlates of automatic and controlled auditory processing in schizophrenia. J Neuropsychiatry Clin Neurosci 20:419–430

    Google Scholar 

  85. Müller D, Schröger E (2007) Temporal grouping affects the automatic processing of deviant sounds. Biol Psychol 74:358–364

    Google Scholar 

  86. Fox E, Lester V, Russo R, Bowles RJ, Pichler A, Dutton K (2000) Facial expressions of emotion: are angry faces detected more efficiently? Cogn Emot 14:61–92

    Google Scholar 

  87. Rothermund K, Wentura D, Bak PM (2001) Automatic attention to stimuli signalling chances and dangers: moderating effects of positive and negative goal and action contexts. Cogn Emot 15:231–248

    Google Scholar 

  88. Murphy G (1947) Personality: a biosocial approach to origins and structure. Harper & Brothers, New York

    Google Scholar 

  89. Mischel W, Ebbesen EB, Zeiss AR (1973) Selective attention to the self: situational and dispositional determinants. J Pers Soc Psychol 27:129–142

    Google Scholar 

  90. Wright JC, Mischel W (1987) A conditional approach to dispositional constructs: the local predictability of social behaviour. J Pers Soc Psychol 55:454–469

    Google Scholar 

  91. Allport GW (1927) Concepts of trait and personality. Psychol Bull 24:284–292

    Google Scholar 

  92. Allport GW (1931) What is a trait of personality? J Abnorm Soc Psychol 25:368–372

    Google Scholar 

  93. Cattell RB (1946) Description and measurement of personality. World book Co., Yonkers-on-Hudson

    Google Scholar 

  94. Bem DJ, Allen A (1974) On predicting some of the people some of the time: the search for cross-situational consistencies in behavior. Psychol Rev 81:506–520

    Google Scholar 

  95. Janssen J, Erkens G, Kirschner PA, Kanselaar G (2009) Influence of group member familiarity on online collaborative learning. Comput Hum Behav 25:161–170

    Google Scholar 

  96. Thonus T (2008) Acquaintanceship, familiarity, and coordinated laughter in writing tutorials. Linguist Educ 19:333–350

    Google Scholar 

  97. Fridlund AJ (1991) The sociality of solitary smiles: effects of an implicit audience. J Pers Soc Psychol 60:229–240

    Google Scholar 

  98. Cook WL, Kenny DA (2005) The actor-partner interdependence model: A model of directional effects in developmental studies. Int J Behav Dev 29:101–109

    Google Scholar 

  99. Janis I (1972) Victims of groupthink. Houghton-Mifflin, Boston

    Google Scholar 

  100. Drury J, Cocking C, Reicher S (2009) Everyone for themselves? A comparative study of crowd solidarity among emergency survivors. Br J Soc Psychol 48:487–506

    Google Scholar 

  101. Yamagishi T, Mifune N (2009) Social exchange and solidarity: In-group love or out-group hate? Evol Hum Behav 30:229–237

    Google Scholar 

  102. Brotheridge CM (2008) Structuring hierarchical deference and solidarity in the context of changing communication media. Inf Commun Soc 11:1132–1154

    Google Scholar 

  103. Diefendorff J, Morehart J, Gabriel A (2010) The influence of power and solidarity on emotional display rules at work. Motiv Emot 34:120–132

    Google Scholar 

  104. Moore AJ (1997) The evolution of social signals: morphological, functional, and genetic integration of sex pheromone in Nauphoeta cinerea. Evolution 51:1920–1928

    Google Scholar 

  105. Mehu M, Little AC, Dunbar RIM (2008) Sex differences in the impact of smiling on social judgements: an evolutionary approach. J Soc Evol Cult Psychol 2:103–121

    Google Scholar 

  106. Poggi I, D’Errico F (2010) Cognitive modeling of human social signals. In: Proceedings of the 2nd international workshop on Social signal processing, pp 21–26

    Google Scholar 

  107. Thelwall M, Wilkinson D, Uppal S (2010) Data mining emotion in social network communication: gender differences in MySpace. J Am Soc Inf Sci Technol 61:190–199

    Google Scholar 

  108. Hodge R, Kress G (1988) Social semiotics. Polity, Cambridge

    Google Scholar 

Download references

Acknowledgement

This study was supported by FP7 IST project SSPnet.

Author information

Authors and Affiliations

  1. School of Psychology, Queen’s University Belfast University Road, Belfast, UK

    Paul M. Brunet & Roderick Cowie

Authors
  1. Paul M. Brunet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roderick Cowie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paul M. Brunet.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Brunet, P.M., Cowie, R. Towards a conceptual framework of research on social signal processing. J Multimodal User Interfaces 6, 101–115 (2012). https://doi.org/10.1007/s12193-012-0092-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-012-0092-x

Keywords

Search

Navigation