Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition | Mobile Networks and Applications Skip to main content
Springer Nature Link
Log in

Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

Human behavior analysis from big multimedia data has become a trending research area with applications to various domains such as surveillance, medical, sports, and entertainment. Facial expression analysis is one of the most prominent clues to determine the behavior of an individual, however, it is very challenging due to variations in face poses, illuminations, and different facial tones. In this paper, we analyze human behavior using facial expressions by considering some famous TV-series videos. Firstly, we detect faces using Viola-jones algorithm followed by tracking through Kanade-Lucas-Tomasi (KLT) algorithm. Secondly, we use histogram of oriented gradients (HOG) features with support vector machine (SVM) classifier for facial recognition. Next, we recognize facial expressions using the proposed light-weight convolutional neural network (CNN). We utilize data augmentation techniques to overcome the issue of appearance of faces from different views and lightening conditions in video data. Finally, we predict human behaviors using an occurrence matrix acquired from facial recognition and expressions. The subjective and objective experimental evaluations prove better performance for both facial expression recognition and human behavior understanding.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Cornejo JYR, Pedrini H, Flórez-Revuelta F (2015) Facial Expression Recognition with Occlusions based on Geometric Representation. In: Iberoamerican Congress on Pattern Recognition, Springer

  2. Lu H, Wang M, Sangaiah AK (2018) Human Emotion Recognition Using an EEG Cloud Computing Platform. Mobile Networks and Applications: 1–10

  3. Khan SA, Hussain A, Usman M (2016) Facial expression recognition on real world face images using intelligent techniques: A survey. Optik-International Journal for Light and Electron Optics 127(15):6195–6203

    Article  Google Scholar 

  4. Dartmann G, Song H, Schmeink A (2019) Big data analytics for cyber-physical systems: machine learning for the internet of things. Elsevier, Amsterdam

    Google Scholar 

  5. Chen M, Mao S, Liu Y (2014) Big data: A survey. Mobile Netw Appl 19(2):171–209

    Article  Google Scholar 

  6. Sajjad M et al (2019) Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services. Inf Sci 479:416–431

    Article  Google Scholar 

  7. Lv Y, Feng Z, Xu C (2014) Facial expression recognition via deep learning. In: Smart Computing (SMARTCOMP), 2014 International Conference on. IEEE

  8. Fridlund AJ (2014) Human facial expression: An evolutionary view. Academic Press, Cambridge

    Google Scholar 

  9. Hossain MS et al (2016) Audio-visual emotion recognition using big data towards 5G. Mobile Netw Appl 21(5):753–763

    Article  Google Scholar 

  10. Bartlett MS et al (2005) Recognizing facial expression: machine learning and application to spontaneous behavior. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, IEEE

  11. Tian Y-L (2004) Evaluation of face resolution for expression analysis. In: Computer Vision and Pattern Recognition Workshop, 2004. CVPRW'04. Conference on, IEEE

  12. Padgett C, Cottrell GW (1997) Representing face images for emotion classification. In: Advances in neural information processing systems

  13. Cohen I et al (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1):160–187

    Article  Google Scholar 

  14. Moussaïd M et al (2010) The walking behaviour of pedestrian social groups and its impact on crowd dynamics. PLoS One 5(4):e10047

    Article  MathSciNet  Google Scholar 

  15. Robertson N, Reid I, Brady M (2008) Automatic human behaviour recognition and explanation for CCTV video surveillance. Secur J 21(3):173–188

    Article  Google Scholar 

  16. Chua S-L, Marsland S, Guesgen HW (2009) Behaviour recognition from sensory streams in smart environments. In: Australasian Joint Conference on Artificial Intelligence, Springer

  17. Sadilek A, Kautz H (2012) Location-based reasoning about complex multi-agent behavior. J Artif Intell Res 43:87–133

    Article  MathSciNet  MATH  Google Scholar 

  18. Baxter RH, Robertson NM, Lane DM (2015) Human behaviour recognition in data-scarce domains. Pattern Recogn 48(8):2377–2393

    Article  Google Scholar 

  19. Uddin, M.Z., et al. (2017) A facial expression recognition system using robust face features from depth videos and deep learning. Comput Electr Eng

  20. Lopes AT et al (2017) Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  21. Al-Shabi M, Cheah WP, Connie T (2016) Facial expression recognition using a hybrid CNN-SIFT aggregator. arXiv preprint arXiv:1608.02833

  22. Chang K-Y, Chen C-S, Hung Y-P (2013) Intensity rank estimation of facial expressions based on a single image. in Systems, Man, and Cybernetics (SMC), 2013 IEEE International Conference on. IEEE

  23. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM

  24. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  25. Bouguet J-Y (2001) Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel Corporation 5(1–10):4

    Google Scholar 

  26. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: international Conference on computer vision & Pattern Recognition (CVPR'05). IEEE Computer Society

  27. Shokrani S, Moallem P, Habibi M (2014) Facial emotion recognition method based on Pyramid Histogram of Oriented Gradient over three direction of head. In: Computer and Knowledge Engineering (ICCKE), 2014 4th International eConference on. IEEE

  28. Ullah A et al (2019) Action recognition using optimized deep autoencoder and CNN for surveillance data streams of non-stationary environments. Futur Gener Comput Syst 96:386–397

    Article  Google Scholar 

  29. Muhammad K et al (2018) Convolutional Neural Networks Based Fire Detection in Surveillance Videos. IEEE Access 6:18174–18183

    Article  MathSciNet  Google Scholar 

  30. Al-Turjman F (2019) 5G-enabled devices and smart-spaces in social-IoT: an overview. Futur Gener Comput Syst 92:732–744

    Article  Google Scholar 

  31. Lv Z et al (2017) Next-generation big data analytics: State of the art, challenges, and future research topics. IEEE Transactions on Industrial Informatics 13(4):1891–1899

    Article  Google Scholar 

  32. Sajjad M et al (2019) Multi-grade brain tumor classification using deep CNN with extensive data augmentation. J Comput Sci 30:174–182

    Article  Google Scholar 

  33. Hussain T et al. (2019) Cloud-Assisted Multi-View Video Summarization using CNN and Bi-Directional LSTM. IEEE Transactions on Industrial Informatics : 1

  34. Muhammad K, Hussain T, Baik SW (2018) Efficient CNN based summarization of surveillance videos for resource-constrained devices. Pattern Recogn Lett

  35. Sajjad M et al (2018) Integrating salient colors with rotational invariant texture features for image representation in retrieval systems. Multimed Tools Appl 77(4):4769–4789

    Article  MathSciNet  Google Scholar 

  36. Ullah A et al. (2018) Activity Recognition using Temporal Optical Flow Convolutional Features and Multi-Layer LSTM. IEEE Trans Ind Electron

  37. Mehmood I et al. (2019) Efficient Image Recognition and Retrieval on IoT-Assisted Energy-Constrained Platforms from Big Data Repositories. IEEE Internet Things J

  38. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems

  39. Calvo MG, Lundqvist D (2008) Facial expressions of emotion (KDEF): Identification under different display-duration conditions. Behav Res Methods 40(1):109–115

    Article  Google Scholar 

  40. Liew CF, Yairi T (2013) A comparison study of feature spaces and classification methods for facial expression recognition. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO)

  41. Zhou Y, Shi BE (2017) Action unit selective feature maps in deep networks for facial expression recognition. In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE

  42. Savoiu A, Wong J. Recognizing facial expressions using deep learning

  43. Rao Q et al (2015) Multi-pose facial expression recognition based on SURF boosting. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII)

  44. Chen J et al (2014) Facial expression recognition based on facial components detection and hog features. In: International Workshops on Electrical and Computer Engineering Subfields

Download references

Author information

Authors and Affiliations

  1. Digital Image Processing Laboratory, Department of Computer Science, Islamia College Peshawar, Peshawar, Pakistan

    Muhammad Sajjad & Sana Zahir

  2. Intelligent Media Laboratory, Digital Contents Research Institute, Sejong University, Seoul, Republic of Korea

    Amin Ullah

  3. Department of Software, Sejong University, Seoul, Republic of Korea

    Khan Muhammad

  4. Department of Computer Science, University of Memphis, Memphis, TN, USA

    Zahid Akhtar

Authors
  1. Muhammad Sajjad
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Sana Zahir
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Amin Ullah
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zahid Akhtar
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Khan Muhammad
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Khan Muhammad.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sajjad, M., Zahir, S., Ullah, A. et al. Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition. Mobile Netw Appl 25, 1611–1621 (2020). https://doi.org/10.1007/s11036-019-01366-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-019-01366-9

Keywords