A Comprehensive Framework for Facial Emotion Detection using Deep Learning

doi:10.23940/ijpe.24.08.p3.487497

Abstract

Abstract: Because people's intentions, feelings, and emotions are reflected in their facial expressions, researchers have been drawn to create self-acting autonomous facial expression detection systems. The facial emotion expression system is difficult due to the model's complexity, the small number of training data, and the minute micro facial muscle movements, despite the advancements in deep learning frameworks for automatic facial expression detection. This study suggests a deep learning framework consisting of Fully convolutional network called FCN-Long Short-Term Memory (LSTM), to detect behavior, mood, and facial activity using fine-grained facial action unit recognition with an ERS model. Based on these distinct patterns, the framework may be used to infer an individual's emotional state. The FCN helps in extracting the useful features, which then helps to increase the accuracy of the FCN-LSTM model. The LSTM is used to recognize the facial expression and present the state of the individual based on the expression by using the features that are extracted by FCN. This combination is helpful in generating accurate results for facial emotions recognition (FER). The model is trained and tested on Emotions (Emo-DB) dataset. The accuracy is observed as 94.67%, which is higher when compared to existing deep learning models.

Key words: accuracy, CNN, deep learning, facial emotions recognition (FER)

Nilesh Shelke, Deepali Sale, Sagar Shinde, Atul Kathole, and Rachna Somkunwar. A Comprehensive Framework for Facial Emotion Detection using Deep Learning [J]. Int J Performability Eng, 2024, 20(8): 487-497.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Khan A.R.,2022. Facial emotion recognition using conventional machine learning and deep learning methods: current achievements, analysis and remaining challenges.Information, 13(6), p.268.
[2] Moret-Tatay C., Wester A.G. and Gamermann D., 2020. To Google or not: Differences on how online searches predict names and faces.Mathematics, 8(11), p.1964.
[3] Meethongjan K., Dzulkifli M., Rehman A., Altameem A. and Saba T., 2013. An intelligent fused approach for face recognition. Journal of Intelligent Systems,22(2), pp.197-212.
[4] Hossain S., Umer S., Rout R.K. and Tanveer M., 2023. Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling. Applied Soft Computing, 134, p.109997.
[5] Yar H., Hussain T., Khan Z.A., Koundal D., Lee M.Y. and Baik S.W., 2021. [Retracted] Vision Sensor‐Based Real‐Time Fire Detection in Resource‐Constrained IoT Environments.Computational intelligence and neuroscience, 2021(1), p.5195508.
[6] Joudaki, S. and Rehman, A., 2022. Dynamic hand gesture recognition of sign language using geometric features learning. International Journal of Computational Vision and Robotics,12(1), pp.1-16.
[7] Alkawaz M.H., Mohamad D., Basori A.H. and Saba T., 2015. Blend shape interpolation and FACS for realistic avatar.3D Research, 6, pp.1-10.
[8] Rouast P.V., Adam M.T. and Chiong R., 2019. Deep learning for human affect recognition: Insights and new developments. IEEE Transactions on Affective Computing,12(2), pp.524-543.
[9] Abbas N., Saba T., Mohamad D., Rehman A., Almazyad A.S. and Al-Ghamdi J.S., 2018. Machine aided malaria parasitemia detection in Giemsa-stained thin blood smears.Neural Computing and Applications, 29, pp.803-818.
[10] Iqbal S., Ghani Khan M.U., Saba T., Mehmood Z., Javaid N., Rehman A. and Abbasi R., 2019. Deep learning model integrating features and novel classifiers fusion for brain tumor segmentation. Microscopy research and technique,82(8), pp.1302-1315.
[11] Khan M.A., Kadry S., Zhang Y.D., Akram T., Sharif M., Rehman A. and Saba T., 2021. Prediction of COVID-19-pneumonia based on selected deep features and one class kernel extreme learning machine. Computers & Electrical Engineering, 90, p.106960.
[12] Khan M.A., Sharif M.I., Raza M., Anjum A., Saba T. and Shad S.A., 2022. Skin lesion segmentation and classification: A unified framework of deep neural network features fusion and selection.Expert Systems, 39(7), p.e12497.
[13] Nawaz M., Mehmood Z., Nazir T., Naqvi R.A., Rehman A., Iqbal M. and Saba T., 2022. Skin cancer detection from dermoscopic images using deep learning and fuzzy k‐means clustering. Microscopy research and technique,85(1), pp.339-351.
[14] Saba T., Haseeb K., Ahmed I. and Rehman A., 2020. Secure and energy-efficient framework using Internet of Medical Things for e-healthcare. Journal of Infection and Public Health,13(10), pp.1567-1575.
[15] Mollahosseini A., Chan D. and Mahoor M.H., 2016, March. Going deeper in facial expression recognition using deep neural networks. In 2016 IEEE Winter conference on applications of computer vision (WACV)(pp. 1-10). IEEE.
[16] Lopes A.T., De Aguiar E., De Souza A.F. and Oliveira-Santos T., 2017. Facial expression recognition with convolutional neural networks: coping with few data and the training sample order.Pattern recognition, 61, pp.610-628.
[17] Mohammadpour M., Khaliliardali H., Hashemi S.M.R. and AlyanNezhadi M.M., 2017, December. Facial emotion recognition using deep convolutional networks. In 2017 IEEE 4th international conference on knowledge-based engineering and innovation (KBEI)(pp. 0017-0021). IEEE.
[18] Agrawal, A. and Mittal, N., 2020. Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer,36(2), pp.405-412.
[19] Cai J., Chang O., Tang X.L., Xue C. and Wei C., 2018, July. Facial expression recognition method based on sparse batch normalization CNN. In 2018 37th Chinese control conference (CCC)(pp. 9608-9613). IEEE.
[20] Li Y., Zeng J., Shan S. and Chen X., 2018. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Transactions on Image Processing,28(5), pp.2439-2450.
[21] Yolcu G., Oztel I., Kazan S., Oz C., Palaniappan K., Lever T.E. and Bunyak F., 2019. Facial expression recognition for monitoring neurological disorders based on convolutional neural network.Multimedia Tools and Applications, 78, pp.31581-31603.
[22] Kim D.H., Baddar W.J., Jang J. and Ro Y.M., 2017. Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Transactions on Affective Computing,10(2), pp.223-236.
[23] Yu Z., Liu G., Liu Q. and Deng J., 2018. Spatio-temporal convolutional features with nested LSTM for facial expression recognition.Neurocomputing, 317, pp.50-57.
[24] Liang D., Liang H., Yu Z. and Zhang Y., 2020. Deep convolutional BiLSTM fusion network for facial expression recognition.The Visual Computer, 36, pp.499-508.
[25] Matsugu M., Mori K., Mitari Y. and Kaneda Y., 2003. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural networks,16(5-6), pp.555-559.
[26] Li Z., Imai J.I. and Kaneko M., 2009, October. Facial-component-based bag of words and phog descriptor for facial expression recognition. In 2009 IEEE International Conference on Systems, Man and Cybernetics(pp. 1353-1358). IEEE.
[27] Zhang, L. and Tjondronegoro, D., 2011. Facial expression recognition using facial movement features. IEEE transactions on affective computing,2(4), pp.219-229.
[28] Yu, Z. and Zhang, C., 2015, November. Image based static facial expression recognition with multiple deep network learning. InProceedings of the 2015 ACM on international conference on multimodal interaction(pp. 435-442).
[29] Dabhi M., Shah M., Bharti P., Puvar P. and Prajapati B., 2020. Unusual activity detection in crowd using deep learning. J Emerg Technol Innov Res (JETIR),7(6), pp.470-476.
[30] Gawande U., Hajari K. and Golhar Y., 2023. Real-time deep learning approach for pedestrian detection and suspicious activity recognition.Procedia Computer Science, 218, pp.2438-2447.
[31] Buttar A.M., Bano M., Akbar M.A., Alabrah A. and Gumaei A.H., 2023. Toward trustworthy human suspicious activity detection from surveillance videos using deep learning.Soft Computing, pp.1-13.
[32] Rehman A., Saba T., Khan M.Z., Damaševičius R. and Bahaj S.A., 2022. Internet‐of‐Things‐Based Suspicious Activity Recognition Using Multimodalities of Computer Vision for Smart City Security.Security and communication Networks, 2022(1), p.8383461.
[33] Quadri, S.A. and Katakdhond, K.S., 2022. Suspicious Activity Detection Using Convolution Neural Network.Journal of Pharmaceutical Negative Results, pp.1235-1245.
[34] Tripathi R.K., Jalal A.S. and Agrawal S.C., 2018. Suspicious human activity recognition: a review.Artificial Intelligence Review, 50, pp.283-339.
[35] Kamthe, U.M. and Patil, C.G., 2018, August. Suspicious activity recognition in video surveillance system. In 2018 Fourth international conference on computing communication control and automation (ICCUBEA)(pp. 1-6). IEEE.
[36] Schmidhuber, J. and Hochreiter, S., 1997. Long short-term memory. Neural Comput,9(8), pp.1735-1780.
[37] Hung, B.T. and Tien, L.M., 2021. Facial expression recognition with CNN-LSTM. In Research in Intelligent and Computing in Engineering: Select Proceedings of RICE 2020(pp. 549-560). Springer Singapore.