A lightweight attention-based network for micro-expression recognition | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

A lightweight attention-based network for micro-expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Micro-expression has emerged to be a feasible strategy in affective estimation due to its great reliability in emotion detection. Recent years have witnessed that deep learning methods were successfully applied to the micro-expression recognition field. In visual data, micro-expression only exists in regions such as eyebrows and mouth, etc, which leads to its imbalanced distribution. Therefore, it’s difficult for networks to distinguish the above micro-expression description with weak intensity when extracting feature maps. To tackle such issues, we propose a novel lightweight attention model, LAM, to improve the network recognition performance. LAM is enabled to calculate the correlation between the feature maps (channel dimension) and the correlation within the feature maps (spatial dimension), thus helping the network to focus on micro-expression information. Additionally, cooperated with residual block at various scales in Resnet, LAM can adaptively compute and update the feature maps in each network layer. Technically, coping with small datasets, we build LAM without adding obvious parameters, while a straightforward but efficient strategy that transfer facial expression knowledge is utilized together. Extensive experimental evaluations on two benchmarks (CASME II and SAMM) and post-hoc feature visualizations demonstrate the effectiveness of our proposed network with LAM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The datasets (CASME II and SAMM) used in this work are publicly available.

References

  1. Bhall RK, Sharma V (2015) A review: detection and analysis of facial micro-expression. Int J Educ Sci Res Rev (IJESRR) 2(1)

  2. Porter S, Brinke LT (2008) Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychol Sci 19(5):508–514

    Article  PubMed  Google Scholar 

  3. Russell TA, Chu E, Phillips ML (2006) A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br J Clin Psychol 45:579–583

    Article  PubMed  Google Scholar 

  4. Weinberger S (2010) Airport security: intent to deceive. Nature 412–415

  5. Ekman P (2007) Emotions revealed: recognizing faces and feelings to improve communication and emotional Life. Macmillan

    Google Scholar 

  6. Ojala T, Pietikainen M, Harwood D (1994) Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. Proceedings of the 12th international conference on pattern pecognition, pp 582–585

  7. Sun D, Roth S, Black MJ (2014) A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int J Comput Vis 106(2):115–137

    Article  Google Scholar 

  8. Jakkula V (2006) Tutorial on support vector machine (svm). School of EECS, Washington State University 37(2.5):3

  9. Wang SJ, Chen HL, Yan WJ et al (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43

    Article  Google Scholar 

  10. Pfister T, Li X B, Zhao GY et al (2011) Recognising spontaneous facial micro-expressions[C]. In Proceedings of the 2011 IEEE international conference on computer vision. Barcelona, Spain, pp 1449–1456

  11. Wei J, Lu G, Yan J et al (2022) Micro-expression recognition using local binary pattern from five intersecting planes[J]. Multimed Tools Appl 5(9):1–26

    Google Scholar 

  12. Wang YD, See J, Phan PCW et al (2014) LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition[C]. In Proceedings of the 12th Asian conference on computer vision. Singapore, Singapore, pp 21–23

  13. Huang XH, Wang SJ, Zhao GY et al (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection[C]. In Proceedings of the 2015 IEEE international conference on Computer vision workshops. Santiago, Chile, pp 1–9

  14. Huang X, Wang SJ, Liu X et al (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[J]. IEEE Trans Affect Comput 10(1):32–47

    Article  Google Scholar 

  15. Wei J, Lu G, Yan J (2021) A comparative study on movement feature in different directions for micro-expression recognition[J]. Neurocomputing 449(1):159–171

    Article  Google Scholar 

  16. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision[C]. In Proceedings of the 7th international joint conference on artificial intelligence, vol 2. Vancouver, Canada, pp 674–679

  17. Xu F, Zhang J, Wang JZ (2017) Micro-expression identification and categorization using a facial dynamics map[J]. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  18. Liu YJ, Zhang JK, Yan WJ et al (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  19. Xia B, Wang W, Wang S et al (2020) Learning from macro-expression: a micro-expression recognition framework[C]. In Proceedings of the 28th ACM international conference on multimedia. Montreal, Canada, pp 2936–2944

  20. Wang C, Peng M, Bi T et al (2020) Micro-attention for micro-expression recognition[J]. Neurocomputing 410(7):354–362

    Google Scholar 

  21. Tang H, Chai L, Lu W (2022) Transferring dual stochastic graph convolutional network for facial micro-expression recognition[J]. Image Video Process 34(1):2–6

    Article  Google Scholar 

  22. Zhang J, Yan B, Du X et al (2022) Motion magnification multi-feature relation network for facial microexpression recognition[J]. Complex Intell Syst 2(4):1–14

    Google Scholar 

  23. Khor HQ, See J, Phan RCW et al (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]. In Proceedings of the 13th IEEE international conference on automatic face and gesture recognition. Xi’an, China, pp 667–674

  24. Pérez JS, Meinhardt LE, Facciolo G (2013) TV-L1 optical flow estimation[J]. Image Processing on Line 1(3):137–150

    Article  Google Scholar 

  25. Itti L, Koch C (2001) Computational modelling of visual attention. Nat Rev Neurosci 2(3):194

    Article  CAS  PubMed  Google Scholar 

  26. Liang Z, He J, Sun Y (2020) A three-dimensional convolution neural network evolution method for automatic recognition of micro expression. Comput Sci 47(8):227–232

    Google Scholar 

  27. Liu Y, Du H, Zheng L et al (2019) A neural micro-expression recognizer. 2019 14th IEEE international conference on automatic face & gesture recognition, pp 23–30

  28. Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) Imagenet: a large-scale hierarchical image database. In Proceedings of the 2009 IEEE conference on computer vision and pattern recognition. Miami, Florida, USA, pp 248–255

  29. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE computer vision and pattern recognition workshops, San Francisco, CA, USA, pp 94–101

  30. Lyons, Michael, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In Proceedings of the 3rd international conference on automatic face and gesture recognition. Nara, Japan, pp 200–205

  31. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH, Fu X (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PloS one 9(1)

  32. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) Samm: a spontaneous micro-facial movement dataset. IEEE Trans Affective Comput 9(1):116–129

    Article  Google Scholar 

  33. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 6:681–685

    Article  Google Scholar 

  34. Oh TH, Jaroensri R, Kim C, Elgharib M, Durand FE, Freeman WT, Matus-ik W (2018) Learning-based video motion magnification. Proceedings of the European conference on computer Vision, pp 633–648

  35. Merghani W, Davison A, Yap M (2018) Facial micro-expressions grand challenge 2018: evaluating spatiotemporal features for classification of objective classes. In Proceedings of the 13th International conference on automatic face & gesture recognition. Xi’an, China, pp 662–666

  36. Khor HQ, See J, Liong T et al (2019) Dual-stream shallow networks for facial micro-expression recognition[C]. In Proceedings of the 2019 IEEE international conference on image processing. Taipei, Taiwan, pp 36–40

  37. Lei L, Chen T, Li SG et al (2021) Micro-expression recognition based on facial graph representation learning and facial action unit fusion[C]. In Proceedings of the 2021 IEEE/CVF conference on computer vision and pattern recognition workshops. Tennessee, USA, pp 76–79

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 71774159 and 61977061, the Fund of State Key Laboratory of NBC Protection for Civilian under Grants SKLNBC2020-23. Dr. Dashuai Hao and Dr. Chen Zhang contribute equally to this work.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, No.1 Daxue Road, Xuzhou, 221116, Jiangsu, China

    Dashuai Hao, Chen Zhang, Guan Yuan, Qiuyan Yan & Xiaobao Zhuang

  2. State Key Laboratory of NBC Protection for Civilian, 100038, Beijing, China

    Mu Zhu

  3. Digitization of Mine, Engineering Research Center of Ministry of Education, Xuzhou, 221116, Jiangsu, China

    Guan Yuan

Authors
  1. Dashuai Hao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mu Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Chen Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Guan Yuan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Qiuyan Yan
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Xiaobao Zhuang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Mu Zhu or Guan Yuan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hao, D., Zhu, M., Zhang, C. et al. A lightweight attention-based network for micro-expression recognition. Multimed Tools Appl 83, 29239–29260 (2024). https://doi.org/10.1007/s11042-023-16616-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16616-y

Keywords