A survey on emotional visualization and visual analysis | Journal of Visualization Skip to main content
Springer Nature Link
Log in

A survey on emotional visualization and visual analysis

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

In the past 10 years, how to visualize human emotions in communication has become an important topic. For providing personalized customer service for enterprises from self-reflection in psychology to opinion mining, emotional visualization uses coded emotional computing results to make various basic charts, and some novel visual analysis systems for all-round analysis which intuitively reveal personal views and emotional styles. Emotion visualization uses coded emotion computing results to reflect the emotion analysis tasks, such as self-reflection in psychology or social media opinion mining results. With the help of various basic charts, infographics, and some novel visual analysis systems, it makes all directions’ analysis and intuitively reveals personal opinions and emotional styles. At present, emotional visualization has developed to use different platforms or multiple platforms to analyze various complex data, including text, sound, image, video, physiological signal or any mixed data. In this paper, we discuss a total of 75 approaches from four different categories: data source type, emotional computing, visual coding and visualization and visual analysis tasks, and 15 subcategories, including visual works mentioned in published paper and interactive visual works published on the Internet. Then, we discuss the further research approaches of emotional visualization and the prospects of emotional visualization under multidimensional data collaboration. We expect that this survey can help researchers interested in emotional visualization of varied data to find a more suitable visualization method for their data and projects.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Abboud R, Tekli J (2018) Muse prototype for music sentiment expression. In: 2018 IEEE international conference on cognitive computing (ICCC), IEEE, pp 106–109

  • Adiletta MJ, Thomas O (2020) An artistic visualization of music modeling a synesthetic experience. arXiv preprint arXiv:2012.08034

  • Alhamid MF, Alsahli S, Rawashdeh M et al (2017) Detection and visualization of arabic emotions on social emotion map. In: 2017 IEEE international symposium on multimedia (ISM), IEEE, pp 378–381

  • Aljanaki A, Yang YH, Soleymani M (2017) Developing a benchmark for emotional analysis of music. PloS One 12(3):e0173

    Article  Google Scholar 

  • Almahmoud J, Kikkeri K (2020) Speech-based emotion recognition using neural networks and information visualization. arXiv preprint arXiv:2010.15229

  • Alper B, Yang H, Haber E et al (2011) Opinionblocks: visualizing consumer reviews. In: IEEE VisWeek 2011 workshop on interactive visual text analytics for decision making

  • Arellano D, Varona J, Perales FJ (2008) Generation and visualization of emotional states in virtual characters. Comput Animat Virt Worlds 19(3–4):259–270

    Article  Google Scholar 

  • Azcarate A, Hageloh F, Van de Sande K et al (2005) Automatic facial emotion recognition. Universiteit van Amsterdam, Amsterdam, pp 1–6

    Google Scholar 

  • Azevedo R, Taub M, Mudrick NV et al (2017) Using data visualizations to foster emotion regulation during self-regulated learning with advanced learning technologies. In: Informational environments. Springer, pp 225–247

  • Baum D, Rauber A (2006) Emotional descriptors for map-based access to music libraries. In: International conference on Asian digital libraries, Springer, pp 370–379

  • Boumaiza A (2015) A survey on sentiment analysis and visualization. J Emerg Technol Web Intell, 7(1)

  • Braşoveanu AM, Hubmann-Haidvogel A, Scharl A (2012) Interactive visualization of emerging topics in multiple social media streams. In: Proceedings of the international working conference on advanced visual interfaces, pp 530–533

  • Bresciani S (2009) The risks of visualization: a classification of disadvantages associated with graphic representations of information. In: In. UVK Verlagsgesellschaft GmbH, pp 165–178

  • Calderon F, Chang CH, Argueta C et al (2015) Analyzing event opinion transition through summarized emotion visualization. In: Proceedings of the 2015 IEEE/ACM international conference on advances in social networks analysis and mining 2015, pp 749–752

  • Cernea D, Kerren A (2015) A survey of technologies on the rise for emotion-enhanced interaction. Comput Rev 31:70–86

    Google Scholar 

  • Cernea D, Kerren A, Ebert A (2011) Detecting insight and emotion in visualization applications with a commercial eeg headset. In: Proceedings of SIGRAD 2011. Evaluations of graphics and visualization-efficiency; usefulness; accessibility; usability; November 17–18; 2011; KTH; Stockholm; Sweden, Linköping University Electronic Press, 065, pp 53–60

  • Cernea D, Ebert A, Kerren A (2013) A study of emotion-triggered adaptation methods for interactive visualization. In: UMAP Workshops, Citeseer

  • Cernea D, Weber C, Ebert A et al (2013) Emotion scents: a method of representing user emotions on gui widgets. In: Visualization and data analysis 2013, International Society for Optics and Photonics, p 86540F

  • Cernea D, Weber C, Ebert A et al (2015) Emotion-prints: Interaction-driven emotion visualization on multi-touch interfaces. In: Visualization and data analysis 2015, International Society for Optics and Photonics, p 93970A

  • Chen C, Ibekwe-SanJuan F, SanJuan E et al (2006) Visual analysis of conflicting opinions. In: 2006 IEEE symposium on visual analytics science and technology, IEEE, pp 59–66

  • Chen CH, Weng MF, Jeng SK et al (2008) Emotion-based music visualization using photos. In: International conference on multimedia modeling, Springer, pp 358–368

  • Chen NC, Feldman LB, Kroll JF et al (2014) Emoticons and linguistic alignment: how visual analytics can elicit storytelling. In: 2014 IEEE conference on visual analytics science and technology (VAST), IEEE, pp 237–238

  • Das A, Bandyopadhyay S, Gambäck B (2012) Sentiment analysis: What is the end user’s requirement? In: Proceedings of the 2nd International conference on web intelligence, mining and semantics, pp 1–10

  • da Silva Franco RY, do Amor Divino Lima RS, Paixão M et al (2019) Uxmood-a sentiment analysis and information visualization tool to support the evaluation of usability and user experience. Information 10(12):366

    Article  Google Scholar 

  • Derick L, Sedrakyan G, Munoz-Merino PJ et al (2017) Evaluating emotion visualizations using AffectVis, an affect-aware dashboard for students. J Res Innov Teach Learn 10(2):107–125

    Article  Google Scholar 

  • Diakopoulos N, Naaman M, Kivran-Swaine F (2010) Diamonds in the rough: social media visual analytics for journalistic inquiry. In: 2010 IEEE symposium on visual analytics science and technology, IEEE, pp 115–122

  • DiPaola S, Arya A (2006) Emotional remapping of music to facial animation. In: Proceedings of the 2006 ACM SIGGRAPH symposium on Videogames, pp 143–149

  • Du M, Yuan X (2020) A survey of competitive sports data visualization and visual analysis. J Vis 24(1):47–67

    Article  Google Scholar 

  • Gaind B, Syal V, Padgalwar S (2019) Emotion detection and analysis on social media. arXiv preprint arXiv:1901.08458

  • Gali G, Oliver S, Chevalier F et al (2012) Visualizing sentiments in business-customer relations with metaphors. In: CHI’12 extended abstracts on human factors in computing systems, pp 1493–1498

  • Gobron S, Ahn J, Paltoglou G et al (2010) From sentence to emotion: a real-time three-dimensional graphics metaphor of emotions extracted from text. Vis Comput 26(6):505–519

    Article  Google Scholar 

  • Goto M, Goto T (2005) Musicream: new music playback interface for streaming, sticking, sorting, and recalling musical pieces. In: ISMIR, Citeseer, pp 404–411

  • Grekow J (2011) Emotion based music visualization system. In: International symposium on methodologies for intelligent systems, Springer, pp 523–532

  • Guthier B, Alharthi R, Abaalkhail R et al (2014) Detection and visualization of emotions in an affect-aware city. In: Proceedings of the 1st international workshop on emerging multimedia applications and services for smart cities, pp 23–28

  • Guzman E (2013) Visualizing emotions in software development projects. In: 2013 First IEEE working conference on software visualization (VISSOFT), IEEE, pp 1–4

  • Ha H, Kim Gn, Hwang W et al (2014) Cosmovis: analyzing semantic network of sentiment words in movie reviews. In: 2014 IEEE 4th symposium on large data analysis and visualization (LDAV), IEEE, pp 113–114

  • Ha H, Han H, Mun S et al (2019) An improved study of multilevel semantic network visualization for analyzing sentiment word of movie review data. Appl Sci 9(12):2419

    Article  Google Scholar 

  • Hanser E, Mc Kevitt P, Lunney T et al (2010) Newsviz: emotional visualization of news stories. In: Proceedings of the NAACL HLT 2010 Workshop on computational approaches to analysis and generation of emotion in text, pp 125–130

  • Haro M, Xambó A, Fuhrmann F et al (2010) The musical avatar: a visualization of musical preferences by means of audio content description. In: Proceedings of the 5th audio mostly conference: a conference on interaction with sound, pp 1–8

  • Hennig P, Berger P, Meinel C et al (2014) Exploring emotions over time within the blogosphere. In: 2014 international conference on data science and advanced analytics (DSAA), IEEE, pp 587–592

  • Hennig P, Berger P, Brehm M et al (2015) Hot spot detection-an interactive cluster heat map for sentiment analysis. In: 2015 IEEE international conference on data science and advanced analytics (DSAA), IEEE, pp 1–9

  • Hilliges O, Holzer P, Klüber R et al (2006) Audioradar: A metaphorical visualization for the navigation of large music collections. In: International symposium on smart graphics, Springer, pp 82–92

  • Hupont I, Baldassarri S, Cerezo E et al (2013) Advanced human affect visualization. In: Proceedings of the 2013 IEEE international conference on systems, man, and cybernetics

  • Jänicke S (2020) Teaching on the intersection of visualization and digital humanities. In: VISIGRAPP (3: IVAPP), pp 100–109

  • Jeong WU, Kim SH (2019) Synesthesia visualization of music waveform:’kinetic lighting for music visualization’. Int J Asia Digital Art Des Assoc 23(2):22–27

    Google Scholar 

  • Jia F, Chen CC (2020) Emotional characteristics and time series analysis of internet public opinion participants based on emotional feature words. Int J Adv Robot Syst 17(1):1729881420904

    Article  MathSciNet  Google Scholar 

  • Jiayu W, Zhiyong F, Zhiyuan L et al (2013) Creating reflections in public emotion visualization: prototype exploration on traffic theme. In: Proceedings of the 9th ACM conference on creativity & cognition, pp 357–361

  • Jin H, Wang X, Lian Y et al (2019) Emotion information visualization through learning of 3D morphable face model. Vis Comput 35(4):535–548

    Article  Google Scholar 

  • Kaklauskas A, Abraham A, Dzemyda G et al (2020) Emotional, affective and biometrical states analytics of a built environment. Eng Appl Art Intell 91(103):621

    Google Scholar 

  • Kempter R, Sintsova V, Musat C et al (2014) Emotionwatch: Visualizing fine-grained emotions in event-related tweets. In: Proceedings of the international AAAI conference on web and social media

  • Kerren A, Cernea D, Pohl M (2016) Workshop on emotion and visualization: emovis 2016. In: Companion Publication of the 21st international conference on intelligent user interfaces, pp 1–2

  • Khulusi R, Kusnick J, Meinecke C et al (2020) A survey on visualizations for musical data. In: Computer graphics forum, Wiley Online Library, pp 82–110

  • Kim E, Klinger R (2018) A survey on sentiment and emotion analysis for computational literary studies. arXiv preprint arXiv:1808.03137

  • Kim HR (2020) Development of the artwork using music visualization based on sentiment analysis of lyrics. J Korea Contents Assoc 20(10):89–99

    Google Scholar 

  • Kucher K, Schamp-Bjerede T, Kerren A et al (2016) Visual analysis of online social media to open up the investigation of stance phenomena. Inform Vis 15(2):93–116

    Article  Google Scholar 

  • Kuksenok K, Brooks M, Robinson JJ et al (2012) Automating large-scale annotation for analysis of social media content. In: IEEE workshop on interactive visual text analytics for analysis of social media

  • Lee Y, Fathia RN (2016) Interactive music visualization for music player using processing. In: 2016 22nd international conference on virtual system & multimedia (VSMM), IEEE, pp 1–4

  • Li M, Guntuku S, Jakhetiya V et al (2019) Exploring (dis-) similarities in emoji-emotion association on twitter and weibo. In: Companion proceedings of the 2019 world wide web conference, pp 461–467

  • Lu Y, Hu X, Wang F et al (2015) Visualizing social media sentiment in disaster scenarios. In: Proceedings of the 24th international conference on world wide web, pp 1211–1215

  • Lyu Z, Li J, Wang B (2021) Aiive: interactive visualization and sonification of neural networks in virtual reality. arXiv e-prints

  • Mehrabian A (1997) Comparison of the pad and panas as models for describing emotions and for differentiating anxiety from depression. J Psychopathol Behav Assess 19(4):331–357

    Article  Google Scholar 

  • Oliveira VADJ, Stoiber C, grüblbauer J et al (2020) Sambavis: design study of a visual analytics tool for the music industry powered by youtube comments. In: Eurovis 2020

  • Paraskevopoulos G, Tzinis E, Ellinas N et al (2021) Unsupervised low-rank representations for speech emotion recognition. arXiv preprint arXiv:2104.07072

  • Passalis N, Doropoulos S (2021) deepsing: generating sentiment-aware visual stories using cross-modal music translation. Expert Syst Appl 164(114):059

    Google Scholar 

  • Pesek M, Strle G, Kavčič A et al (2017) The moodo dataset: integrating user context with emotional and color perception of music for affective music information retrieval. J New Music Res 46(3):246–260

    Article  Google Scholar 

  • Pinilla A, Garcia J, Raffe W et al (2020) Emotion visualization in virtual reality: an integrative review. arXiv preprint arXiv:2012.08849

  • Pion-Tonachini L, Hsu SH, Makeig S et al (2015) Real-time eeg source-mapping toolbox (rest): online ica and source localization. In: 2015 37th annual international conference of the ieee engineering in medicine and biology society (EMBC), IEEE, pp 4114–4117

  • Plutchik R (2001) The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci 89(4):344–350

    Article  Google Scholar 

  • Prasojo RE, Darari F, Kacimi M (2015) Orcaestra: organizing news comments using aspect, entity and sentiment extraction. Poster Abstracts of IEEE VIS

  • Qi L (2018) Data visualization as creative art practice. Vis Commun 17(3):147035721876

    Google Scholar 

  • Ren F, Liu N (2018) Emotion computing using word mover’s distance features based on ren_cecps. PloS One 13(4):e0194

    Article  Google Scholar 

  • Robitaille P, McGuffin MJ (2019) Increased affect-arousal in vr can be detected from faster body motion with increased heart rate. In: Proceedings of the ACM SIGGRAPH symposium on interactive 3D graphics and games, pp 1–6

  • Russell JA (1989) Measures of emotion. In: The measurement of emotions. Elsevier, p 83–111

  • Scharl A, Hubmann-Haidvogel A, Jones A et al (2016) Analyzing the public discourse on works of fiction-detection and visualization of emotion in online coverage about hbo’s game of thrones. Inform Process Manag 52(1):129–138

    Article  Google Scholar 

  • Seo YS, Huh JH (2019) Automatic emotion-based music classification for supporting intelligent IoT applications. Electronics 8(2):164

    Article  Google Scholar 

  • Siti Sendari IAEZ, Dian Candra Lestari HPH (2020) Opinion analysis for emotional classification on emoji tweets using the naïve bayes algorithm. Knowl Eng Data Sci 3(1):50–59

    Article  Google Scholar 

  • Sung CY, Huang XY, Shen Y et al (2016) Topin: a visual analysis tool for time-anchored comments in online educational videos. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, pp 2185–2191

  • Tajadura-Jiménez A, Väljamäe A, Asutay E et al (2010) Embodied auditory perception: the emotional impact of approaching and receding sound sources. Emotion 10(2):216

    Article  Google Scholar 

  • Topal K, Ozsoyoglu G (2016) Movie review analysis: Emotion analysis of imdb movie reviews. In: 2016 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), IEEE, pp 1170–1176

  • Torkildson MK, Starbird K, Aragon C (2014) Analysis and visualization of sentiment and emotion on crisis tweets. In: International conference on cooperative design, visualization and engineering, Springer, pp 64–67

  • van Gulik R, Vignoli F, van de Wetering H (2004) Mapping music in the palm of your hand, explore and discover your collection. In: Proceedings of the 5th international conference on music information retrieval, Queen Mary, University of London London

  • Vryzas N, Liatsou A, Kotsakis R et al (2017) Augmenting drama: A speech emotion-controlled stage lighting framework. In: Proceedings of the 12th international audio mostly conference on augmented and participatory sound and music experiences, pp 1–7

  • Wang Y, Segal A, Klatzky RL et al (2019) An emotional response to the value of visualization. IEEE Comput Gr Appl 39(5):8–17

    Article  Google Scholar 

  • Wu Y, Liu S, Yan K et al (2014) Opinionflow: visual analysis of opinion diffusion on social media. IEEE Trans Vis Comput Gr 20(12):1763–1772

    Article  Google Scholar 

  • Yang Z, Zhang Y, Luo J (2019) Human-centered emotion recognition in animated gifs. In: 2019 IEEE international conference on multimedia and expo (ICME), IEEE, pp 1090–1095

  • Zeng H, Wang X, Wu A et al (2019) Emoco: visual analysis of emotion coherence in presentation videos. IEEE Trans Vis Comput Gr 26(1):927–937

    Google Scholar 

  • Zhang S, Huang Q, Jiang S et al (2010) Affective visualization and retrieval for music video. IEEE Trans Multimed 12(6):510–522

    Article  Google Scholar 

  • Zhao J, Gou L, Wang F et al (2014) Pearl: an interactive visual analytic tool for understanding personal emotion style derived from social media. In: 2014 IEEE conference on visual analytics science and technology (VAST), IEEE, pp 203–212

Download references

Author information

Author notes

  1. Jiaqi Wang and Tianyi Gui have contributed equally to this work.

Authors and Affiliations

  1. School of New Media, Beijing Institute of Graphic Communication, Beijing, China

    Jiaqi Wang, Tianyi Gui, Mingzhi Cheng, Xuan Wu, Ruolin Ruan & Meng Du

  2. International New Media Institution of Industry-Education-Research Cooperation, Beijing, China

    Jiaqi Wang, Tianyi Gui, Mingzhi Cheng, Xuan Wu, Ruolin Ruan & Meng Du

  3. The Faculty of Science, University of Auckland, Auckland, New Zealand

    Meng Du

Authors
  1. Jiaqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianyi Gui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingzhi Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruolin Ruan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Meng Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meng Du.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Gui, T., Cheng, M. et al. A survey on emotional visualization and visual analysis. J Vis 26, 177–198 (2023). https://doi.org/10.1007/s12650-022-00872-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-022-00872-5

Keywords

Search

Navigation