CC-CNN: A cross connected convolutional neural network using feature level fusion for facial expression recognition | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

CC-CNN: A cross connected convolutional neural network using feature level fusion for facial expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Facial expressions are an important form of non-verbal communication as they directly reflect the internal emotions of a person. The primary task of automated Facial Expression Recognition (FER) systems lies in extracting salient features related to facial expressions. In this paper, a Cross Connected Convolutional Neural Network (CC-CNN) has been proposed for extracting the facial features. The proposed CC-CNN model contains two levels of input for extracting the features related to facial expressions. Cyclopentane Feature Descriptor (CyFD), inspired by cyclopentane’s structure, has been proposed to extract significant features. The feature response map generated by the CyFD method has been given as input to the first level, and in the second level, the features have been extracted directly from the facial image. The input images from both levels are passed through a series of convolutional layers with cross connections for extracting the fused (local and global) features related to facial expressions. Finally, towards the end, the CC-CNN method works by fusing all the features extracted from both levels. To validate the efficiency of the proposed CC-CNN method, the experiments have been performed on benchmark FER datasets such as CK+, MUG, RAF, FER2013 and FERG. The comparison results from the experimental analysis revealed that the proposed model outperformed the recent FER methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Availability of data and materials

The datasets used in this work are available in the below links. CK+: https://www.pitt.edu/emotion/ck-spread.htm MUG: https://mug.ee.auth.gr/fed/ FER2013: https://www.kaggle.com/msambare/fer2013 RAF: http://www.whdeng.cn/raf/model1.html#dataset FERG: http://grail.cs.washington.edu/projects/deepexpr/ferg-2d-db.html

References

  1. Agarwal S, Santra B, Mukherjee DP (2018) Anubhav: recognizing emotions through facial expression. Vis Comput 34(2):177–191

    Article  Google Scholar 

  2. Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pages 1–4. IEEE

  3. Allaert B, Bilasco IM, Djeraba C (2019) Micro and macro facial expression recognition using advanced local motion patterns. IEEE Trans Affect Comput

  4. Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. In Asian conference on computer vision, pages 136–153. Springer

  5. Barua PD, Dogan S, Tuncer T, Baygin M, Acharya UR (2021) Novel automated pd detection system using aspirin pattern with eeg signals. Comput Biol Med 137:104841

  6. Bisogni C, Castiglione A, Hossain S, Narducci F, Umer S (2022) Impact of deep learning approaches on facial expression recognition in healthcare industries. IEEE Trans Ind Inform

  7. Cai J, Meng Z, Khan AS, Li Z, O’Reilly J, Tong Y (2018) Probabilistic attribute tree in convolutional neural networks for facial expression recognition. arXiv preprint arXiv:1812.07067

  8. Cotter SF (2020) Mobiexpressnet: A deep learning network for face expression recognition on smart phones. In 2020 IEEE International Conference on Consumer Electronics (ICCE), pages 1–4. IEEE

  9. Dauda A, Bhoi N (2014) Facial expression recognition using pca & distance classifier. Int J Sci Eng Res 5

  10. Doulamis N (2018) Adaptable deep learning structures for object labeling/tracking under dynamic visual environments. Multimed Tools Appl 77(8):9651–9689

    Article  Google Scholar 

  11. Dubey SR (2019) Local directional relation pattern for unconstrained and robust face retrieval. Multimed Tools Appl 78(19):28063–28088

    Article  Google Scholar 

  12. Ekman P, Friesen WV (2003) Unmasking the face: A guide to recognizing emotions from facial clues, volume 10. Ishk

  13. Ekweariri AN, Yurtkan K (2017) Facial expression recognition using enhanced local binary patterns. In 2017 9th international conference on Computational Intelligence and Communication Networks (CICN), pages 43–47. IEEE

  14. Fan X, Jiang M, Shahid AR, Yan H (2022) Hierarchical scale convolutional neural network for facial expression recognition. Cogn Neurodyn, pages 1–12

  15. Fan Y, Lam JCK, Li VOK (2018) Multi-region ensemble convolutional neural network for facial expression recognition. In International Conference on Artificial Neural Networks, pages 84–94. Springer

  16. Fan Y, Li V, Lam JCK (2020) Facial expression recognition with deeply-supervised attention network. IEEE Trans Affect Comput

  17. Gan Y, Chen J, Yang Z, Xu L (2020) Multiple attention network for facial expression recognition. IEEE Access 8:7383–7393

    Article  Google Scholar 

  18. Ge J, Cui X, Xiao K, Zou C, Chen Y, Wei R (2019) Bnrelu: combine batch normalization and rectified linear unit to reduce hardware overhead. In 2019 IEEE 13th International Conference on ASIC (ASICON), pages 1–4. IEEE

  19. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH et al (2013) Challenges in representation learning: A report on three machine learning contests. In International conference on neural information processing, pages 117–124. Springer

  20. Happy SL, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  21. Hazourli AR, Djeghri A, Salam H, Othmani A (2021) Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display. Multimed Tools Appl 80:13639–13662

    Article  Google Scholar 

  22. Huo H, Yu Y, Liu Z (2022) Facial expression recognition based on improved depthwise separable convolutional network. Multimed Tools Appl, pages 1–18

  23. Hussain M, Bird JJ, Faria DR (2018) A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence, pages 191–202. Springer

  24. Hwooi SKW, Othmani A, Sabri AQM (2022) Deep learning-based approach for continuous affect prediction from facial expression images in valence-arousal space. IEEE Access 10:96053–96065

    Article  Google Scholar 

  25. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR

  26. Jabid T, Kabir MH, Chae O (2010) Gender classification using local directional pattern (ldp). In 2010 20th International conference on pattern recognition, pages 2162–2165. IEEE

  27. Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74

    Article  ADS  Google Scholar 

  28. Kartheek MN,Madhuri R, Prasad MVNK, Bhukya R (2021) Knight tour patterns: Novel handcrafted feature descriptors for facial expression recognition. In International Conference on Computer Analysis of Images and Patterns, pages 210–219. Springer

  29. Kartheek MN, Prasad MV, Bhukya R (2021) Radial mesh pattern: a handcrafted feature descriptor for facial expression recognition. J Ambient Intell Humaniz Comput, pages 1–13

  30. Khalid M, Baber J, Kasi MK, Bakhtyar M, Devi V, Sheikh N (2020) Empirical evaluation of activation functions in deep convolution neural network for facial expression recognition. In 2020 43rd International Conference on Telecommunications and Signal Processing (TSP), pages 204–207. IEEE

  31. Kola DGR, Samayamantula SK (2021) Facial expression recognition using singular values and wavelet-based lgc-hd operator. IET Biom 10(2):207–218

    Article  Google Scholar 

  32. Kola DGR, Samayamantula SK (2021) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 80(2):2243–2262

    Article  Google Scholar 

  33. Kong Y, Zhang S, Zhang K, Ni Q, Han J (2022) Real-time facial expression recognition based on iterative transfer learning and efficient attention network. IET Image Process

  34. Kumari J, Rajesh R, Pooja KM (2015) Facial expression recognition: A survey. Proc Comput Sci 58:486–491

    Article  Google Scholar 

  35. Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2852–2861

  36. Li Z, Han S, Khan AS, Cai J, Meng Z, O’Reilly J, Tong Y (2019) Pooling map adaptation in convolutional neural network for facial expression recognition. In 2019 IEEE international conference on multimedia and expo (ICME), pages 1108–1113. IEEE

  37. Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1805–1812

  38. Lopes AT, De Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Article  ADS  Google Scholar 

  39. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops, pages 94–101. IEEE

  40. Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recognit 47(3):1282–1293

    Article  ADS  Google Scholar 

  41. Makhmudkhujaev Md F, Iqbal MT, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turk J Electr Eng Comput Sci 27(1):516–531

    Article  Google Scholar 

  42. Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kranthi Kumar D (2019) Regional adaptive affinitive patterns (radap) with logical operators for facial expression recognition. IET Image Process 13(5):850–861

    Article  Google Scholar 

  43. Miao Si, Haoyu Xu, Han Zhenqi, Zhu Yongxin (2019) Recognizing facial expressions using a shallow convolutional neural network. IEEE Access 7:78000–78011

    Article  Google Scholar 

  44. Mohan K, Ayan S, Anis Y, Ondrej K (2022) Flepnet: Feature level ensemble parallel network for facial expression recognition. IEEE Trans Affect Comput 13(4):2058–2070

    Article  Google Scholar 

  45. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In 2016 IEEE Winter conference on applications of computer vision (WACV), pages 1–10. IEEE

  46. Nan Y, Ju J, Hua Q, Zhang H, Wang B (2022) A-mobilenet: An approach of facial expression recognition. Alex Eng J 61(6):4435–4444

    Article  Google Scholar 

  47. Nan F, Jing W, Tian F, Zhang J, Chao KM, Hong Z, Zheng Q (2022) Feature super-resolution based facial expression recognition for multi-scale low-resolution images. Knowl Based Syst 236:107678

  48. Ouyang W, Zeng X, Wang X, Qiu S, Luo P, Tian Y, Yang S, Wang Z, Li H, Li H et al (2016) Deepid-net: Object detection with deformable part based convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 39(7):1320–1334

    Article  PubMed  Google Scholar 

  49. Pham TT, Kim S, Lu Y, Jung SW, Won CS (2019) Facial action units-based image retrieval for facial expression recognition. IEEE Access 7:5200–5207

    Article  Google Scholar 

  50. Ping J, Gang L, Quan W, Wu J (2020) Accurate and reliable facial expression recognition using advanced softmax loss with fixed weights. IEEE Signal Process Lett 27:725–729

    Article  ADS  Google Scholar 

  51. Putro MD, Nguyen DL, Jo KH (2020) A dual attention module for real-time facial expression recognition. In IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, pages 411–416. IEEE

  52. Rajan S, Chenniappan P, Devaraj S, Madian N (2020) Novel deep learning model for facial expression recognition based on maximum boosted cnn and lstm. IET Image Process 14(7):1373–1381

    Article  Google Scholar 

  53. Reddy AH, Kolli K, Kiran YL (2022) Deep cross feature adaptive network for facial emotion classification. Signal Image Video Process 16(2):369–376

    Article  Google Scholar 

  54. Rivera AR, Castillo JR, Chae OO (2012) Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  55. Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recognit Lett 51:94–100

    Article  ADS  Google Scholar 

  56. Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018

    Article  ADS  MathSciNet  Google Scholar 

  57. Schoneveld L, Othmani A, Abdelkawy H (2021) Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recognit Lett 146:1–7

    Article  ADS  Google Scholar 

  58. Schoneveld L, Othmani A (2021) Towards a general deep feature extractor for facial expression recognition. In 2021 IEEE International Conference on Image Processing (ICIP), pages 2339–2342. IEEE

  59. Sen D, Datta S, Balasubramanian R (2019) Facial emotion classification using concatenated geometric and textural features. Multimed Tools Appl 78(8):10287–10323

    Article  Google Scholar 

  60. Shabat AMM, Tapamo JR (2018) Angled local directional pattern for texture analysis with an application to facial expression recognition. IET Comput Vis 12(5):603–608

    Article  Google Scholar 

  61. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  62. Shengtao G, Chao X, Bo F (2019) Facial expression recognition based on global and local feature fusion with cnns. In 2019 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), pages 1–5. IEEE

  63. Shervin M, Mehdi M, Amirali A (2021) Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors 21(9):3046

    Article  Google Scholar 

  64. Shi C, Tan C, Wang L (2021) A facial expression recognition method based on a multibranch cross-connection convolutional neural network. IEEE Access 9:39255–39274

    Article  Google Scholar 

  65. Siqueira H, Magg S, Wermter S (2020) Efficient facial feature learning with wide ensemble-based convolutional neural networks. In Proceedings of the AAAI conference on artificial intelligence 34:5800–5809

  66. Soman G, Vivek MV, Judy MV, Papageorgiou E, Gerogiannis VC (2022) Precision-based weighted blending distributed ensemble model for emotion classification. Algorithms 15(2):55

    Article  Google Scholar 

  67. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9

  68. Tang Y, Zhang X, Hu X, Wang S, Wang H (2020) Facial expression recognition using frequency neural network. IEEE Trans Image Process 30:444–457

    Article  ADS  PubMed  Google Scholar 

  69. Tuncer T, Dogan S, Abdar M, Pławiak P (2020) A novel facial image recognition method based on perceptual hash using quintet triple binary pattern. Multimed Tools Appl 79(39):29573–29593

    Article  Google Scholar 

  70. Verma M, Saxena P, Vipparthi SK, Singh G (2022) Cross-centroid ripple pattern for facial expression recognition. arXiv preprint arXiv:2201.05958

  71. Wang X, Huang J, Zhu J, Yang M, Yang F (2018) Facial expression recognition with deep learning. In Proceedings of the 10th International Conference on Internet Multimedia Computing and Service, pages 1–4

  72. Wang F, Shen L (2019) Facial expression recognition: Residue learning using svm. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pages 1675–1680. IEEE

  73. Wang Y, Wu J, Hoashi K (2019) Lightweight deep convolutional neural networks for facial expression recognition. In 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), pages 1–6. IEEE

  74. Xie S, Hu H, Wu Y (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recognit 92:177–191

    Article  ADS  Google Scholar 

  75. Yang J, Lv Z, Kuang K, Yang S, Xiao L, Tang Q (2022) Rasn: Using attention and sharing affinity features to address sample imbalance in facial expression recognition. IEEE Access 10:103264–103274

    Article  Google Scholar 

  76. Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pages 866–871. IEEE

  77. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In Proceedings of the 2015 ACM on international conference on multimodal interaction, pages 435–442

  78. Zhao G, Yang H, Yu M (2020) Expression recognition method based on a lightweight convolutional neural network. IEEE Access 8:38528–38537

    Article  Google Scholar 

  79. Zhou N, Liang R, Shi W (2020) A lightweight convolutional neural network for real-time facial expression detection. IEEE Access 9:5573–5584

Download references

Author information

Authors and Affiliations

  1. School of Computer and Information Sciences, University of Hyderabad, Hyderabad, 500046, Telangana, India

    Kadimi Naveen Kumar Tataji

  2. School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, 632014, Tamil Nadu, India

    Mukku Nisanth Kartheek

  3. Institute for Development and Research in Banking Technology, Masab Tank, Hyderabad, 500057, Telangana, India

    Munaga V. N. K. Prasad

Authors
  1. Kadimi Naveen Kumar Tataji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mukku Nisanth Kartheek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Munaga V. N. K. Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mukku Nisanth Kartheek.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar Tataji, K.N., Kartheek, M.N. & Prasad, M.V.N.K. CC-CNN: A cross connected convolutional neural network using feature level fusion for facial expression recognition. Multimed Tools Appl 83, 27619–27645 (2024). https://doi.org/10.1007/s11042-023-16433-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16433-3

Keywords

Search

Navigation