Abstract
Facial expression recognition is an intriguing research area that has been explored and utilized in a wide range of applications such as health, security, and human-computer interactions. The ability to recognize facial expressions accurately is crucial for human-computer interactions. However, most of the facial expression analysis techniques have so far paid little or no concern to users’ data privacy. To overcome this concern, in this paper, we incorporated Federated Learning (FL) as a privacy-preserving machine learning approach in the field of facial expression recognition to develop a shared model without exposing personal information. The individual models are trained on the different client devices where the data is stored. In this work, a lightweight Convolutional Neural Network (CNN) model called the MobileNet architecture is utilised to detect expressions from facial images. To evaluate the model, two publicly available datasets are used and several experiments are conducted. The result shows that the proposed privacy-preserving Federated-MobileNet approach could recognize facial expressions with considerable accuracy compared to the general approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Code Availability
The code and datasets are available at https://github.com/tapu1996/AII-2022-Federated-Learning-96104.
References
Ahmed, S., Hossain, M.F., Nur, S.B., Shamim Kaiser, M., Mahmud, M.: Toward machine learning-based psychological assessment of autism spectrum disorders in school and community. In: Kaiser, M.S., Bandyopadhyay, A., Ray, K., Singh, R., Nagar, V. (eds.) Proceedings of Trends in Electronics and Health Informatics. LNNS, vol. 376, pp. 139–149. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-8826-3_13
Niamat Ullah Akhund, T.M., Mahi, M.J.N., Hasnat Tanvir, A.N.M., Mahmud, M., Kaiser, M.S.: ADEPTNESS: Alzheimer’s disease patient management system using pervasive sensors - early prototype and preliminary results. In: Wang, S., et al. (eds.) BI 2018. LNCS (LNAI), vol. 11309, pp. 413–422. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05587-5_39
Akter, T., Ali, M.H., Satu, M.S., Khan, M.I., Mahmud, M.: Towards autism subtype detection through identification of discriminatory factors using machine learning. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 401–410. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_36
Al Banna, M.H., Ghosh, T., Taher, K.A., Kaiser, M.S., Mahmud, M.: A monitoring system for patients of autism spectrum disorder using artificial intelligence. In: Mahmud, M., Vassanelli, S., Kaiser, M.S., Zhong, N. (eds.) BI 2020. LNCS (LNAI), vol. 12241, pp. 251–262. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59277-6_23
Al Mamun, S., Kaiser, M.S., Mahmud, M.: An artificial intelligence based approach towards inclusive healthcare provisioning in society 5.0: a perspective on brain disorder. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 157–169. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_15
Biswas, M., Kaiser, M.S., Mahmud, M., Al Mamun, S., Hossain, M.S., Rahman, M.A.: An XAI based autism detection: the context behind the detection. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 448–459. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_40
Biswas, M., et al.: Indoor navigation support system for patients with neurodegenerative diseases. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 411–422. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_37
Chatterjee, A., Gupta, U., Chinnakotla, M.K., Srikanth, R., Galley, M., Agrawal, P.: Understanding emotions in text using deep learning and big data. Comput. Hum. Behav. 93, 309–317 (2019)
Chen, J., Hu, B., Moore, P., Zhang, X., Ma, X.: Electroencephalogram-based emotion assessment system using ontology and data mining techniques. Appl. Soft Comput. 30, 663–674 (2015)
Chhikara, P., Singh, P., Tekchandani, R., Kumar, N., Guizani, M.: Federated learning meets human emotions: a decentralized framework for human-computer interaction for IoT applications. IEEE Internet Things J. 8(8), 6949–6962 (2020)
Courville, P., Goodfellow, A., Mirza, I., Bengio, Y.: Fer-2013 face database. Universit de Montreal, Montréal (2013)
Gharaee, Z., Gärdenfors, P., Johnsson, M.: First and second order dynamics in a hierarchical SOM system for action recognition. Appl. Soft Comput. 59, 574–585 (2017)
Ghosh, T., et al.: An attention-based mood controlling framework for social media users. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) BI 2021. LNCS (LNAI), vol. 12960, pp. 245–256. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_23
Ghosh, T., et al.: Artificial intelligence and internet of things in screening and management of autism spectrum disorder. Sustain. Cities Soc. 74, 103189 (2021)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Huang, K.Y., Wu, C.H., Su, M.H., Kuo, Y.T.: Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Trans. Affect. Comput. 11(3), 393–404 (2018)
Huang, Z., Epps, J., Joachim, D.: Speech landmark bigrams for depression detection from naturalistic smartphone speech. In: Proceedings of ICASSP, pp. 5856–5860 (2019)
Jain, D.K., Shamsolmoali, P., Sehdev, P.: Extended deep neural network for facial emotion recognition. Pattern Recognit. Lett. 120, 69–74 (2019)
Jesmin, S., Kaiser, M.S., Mahmud, M.: Towards artificial intelligence driven stress monitoring for mental wellbeing tracking during COVID-19. In: Proceedings of WI-IAT, pp. 845–851 (2020)
Khaireddin, Y., Chen, Z.: Facial emotion recognition: state of the art performance on fer2013. arXiv preprint arXiv:2105.03588 (2021)
Kulkarni, S.S., Reddy, N.P., Hariharan, S.: Facial expression (mood) recognition from facial images using committee neural networks. Biomed. Eng. Online 8(1), 1–12 (2009)
Kwon, S., et al.: Att-Net: enhanced emotion recognition system using lightweight self-attention module. Appl. Soft Comput. 102, 107101 (2021)
Latif, S., Asim, M., Rana, R., Khalifa, S., Jurdak, R., Schuller, B.W.: Augmenting generative adversarial networks for speech emotion recognition. arXiv preprint arXiv:2005.08447 (2020)
Lee, H.C., Wu, C.Y., Lin, T.M.: Facial expression recognition using image processing techniques and neural networks. In: Pan, J.S., Yang, C.N., Lin, C.C. (eds.) Advances in Intelligent Systems and Applications. Smart Innovation, Systems and Technologies, vol. 21, pp. 259–267. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35473-1_26
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Proceeedings of IEEE CVPR, pp. 94–101 (2010)
Mahmud, M., et al.: Towards explainable and privacy-preserving artificial intelligence for personalisation in autism spectrum disorder. In: Antona, M., Stephanidis, C. (eds.) HCII 2022. LNCS, vol. 13309, pp. 356–370. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05039-8_26
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics , pp. 1273–1282 (2017)
Mehendale, N.: Facial emotion recognition using convolutional neural networks (FERC). SN Appl. Sci. 2(3), 1–8 (2020)
Merler, M., et al.: Automatic curation of sports highlights using multimodal excitement features. IEEE Trans. Multimedia 21(5), 1147–1160 (2018)
Al Nahian, M.J., Ghosh, T., Uddin, M.N., Islam, M.M., Mahmud, M., Kaiser, M.S.: Towards artificial intelligence driven emotion aware fall monitoring framework suitable for elderly people with neurological disorder. In: Mahmud, M., Vassanelli, S., Kaiser, M.S., Zhong, N. (eds.) BI 2020. LNCS (LNAI), vol. 12241, pp. 275–286. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59277-6_25
Nan, Y., Ju, J., Hua, Q., Zhang, H., Wang, B.: A-mobilenet: an approach of facial expression recognition. Alex. Eng. J. 61(6), 4435–4444 (2022)
Rana, R., Latif, S., Gururajan, R., Gray, A., Mackenzie, G., Humphris, G., Dunn, J.: Automated screening for distress: a perspective for the future. Eur. J. Cancer Care 28(4), e13033 (2019)
Schoneveld, L., Othmani, A., Abdelkawy, H.: Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recognit. Lett. 146, 1–7 (2021)
Shaffi, N., et al.: Triplet-loss based siamese convolutional neural network for 4-way classification of alzheimerÕs disease. In: Mahmud, M., He, J., Vassanelli, S., van Zundert, A., Zhong, N. (eds.) BI. LNCS, vol. 13406, pp. 277–287. Springer, Cham (2022)
Smith, V., Chiang, C.K., Sanjabi, M., Talwalkar, A.S.: Federated multi-task learning. In: Advance in Neural Information Processing System, vol. 30 (2017)
Sumi, A.I., Zohora, M.F., Mahjabeen, M., Faria, T.J., Mahmud, M., Kaiser, M.S.: fASSERT: A fuzzy assistive system for children with autism using internet of things. In: Wang, S., et al. (eds.) BI 2018. LNCS (LNAI), vol. 11309, pp. 403–412. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05587-5_38
Tsouvalas, V., Ozcelebi, T., Meratnia, N.: Privacy-preserving speech emotion recognition through semi-supervised federated learning. arXiv preprint arXiv:2202.02611 (2022)
Vögel, H.J., et al.: Emotion-awareness for intelligent vehicle assistants: A research agenda. In: Proceedings of SEFAIAS, pp. 11–15 (2018)
Wadhera, T., Mahmud, M.: Brain networks in autism spectrum disorder, epilepsy and their relationship: a machine learning approach. In: Chen, T., Carter, J., Mahmud, M., Khuman, A.S. (eds.) Artificial Intelligence in Healthcare. Brain Informatics and Health, pp. 125–142. Springer, Singapore (2022). https://doi.org/10.1007/978-981-19-5272-2_6
Wadhera, T., Mahmud, M.: Influences of social learning in individual perception and decision making in people with autism: a computational approach. In: Mahmud, M., He, J., Vassanelli, S., van Zundert, A., Zhong, N. (eds.) BI 2022. LNCS, vol. 13406, pp. 50–61. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-15037-1_5
Yang, J., Liu, J., Han, R., Wu, J.: Transferable face image privacy protection based on federated learning and ensemble models. Complex Intell. Syst. 7(5), 2299–2315 (2021). https://doi.org/10.1007/s40747-021-00399-6
Yurochkin, M., et al.: Bayesian nonparametric federated learning of neural networks. In: Proceedings of ICML, pp. 7252–7261 (2019)
Zhang, F., Li, Q., Ren, Y., Xu, H., Song, Y., Liu, S.: An expression recognition method on robots based on mobilenet v2-ssd. In: Proceedings of ICSAI, pp. 118–122 (2019)
Acknowledgements
The authors extend their sincere gratitude to Prof Hongbo Liu from the Dalian University of Technology, China for the useful discussions. Dr Mufti Mahmud is supported by the AI-TOP (2020-1-UK01-KA201-079167) and DIVERSASIA (618615-EPP-1-2020-1-UKEPPKA2-CBHEJP) projects funded by the European Commission under the Erasmus+ programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ghosh, T. et al. (2022). A Privacy-Preserving Federated-MobileNet for Facial Expression Detection from Images. In: Mahmud, M., Ieracitano, C., Kaiser, M.S., Mammone, N., Morabito, F.C. (eds) Applied Intelligence and Informatics. AII 2022. Communications in Computer and Information Science, vol 1724. Springer, Cham. https://doi.org/10.1007/978-3-031-24801-6_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-24801-6_20
Publisher Name: Springer, Cham
Online ISBN: 978-3-031-24801-6
eBook Packages: Computer ScienceComputer Science (R0)