Abstract
In this paper, we present a novel solution for real-time emotion analysis in video conferencing, aiming to enhance therapies for both patients and therapists. Building upon the SenseCare project, we extend its capabilities to include a video conferencing platform with scalable real-time emotion analysis using widely available software and frameworks avoiding critical vendor locks. Our architecture focuses on ease of adaptation and further development, connecting a WebRTC conferencing platform to a scalable Kubernetes backend for emotion analysis. Emphasizing low latency, we implement the producer-consumer pattern and utilize a message broker. For emotion analysis, we use convolution neural networks. We propose a methodology for identifying an optimal batch size that maximizes backend efficiency while maintaining low latency. Our approach exhibits scalability, allowing for seamless adaptation during periods of high system utilization. Our findings demonstrate the feasibility of employing CNNs for sub-second emotion analysis on an affordable Kubernetes cluster, enabling multiple users to effectively engage in the system as patients and therapists.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Mobile Vs. Desktop Internet Usage (2023). https://www.broadbandsearch.net/blog/mobile-desktop-internet-usage-statistics
WebRTC (2021). https://webrtc.org/
eduMEET - web-based videoconferencing platform (2023). https://edumeet.org/
Keycloak (2023). https://www.keycloak.org/
MediaPipe (2023). https://google.github.io/mediapipe/
Messaging that just works - RabbitMQ (2023). https://www.rabbitmq.com/
TensorFlow (2023). https://www.tensorflow.org/
Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., Grundmann, M.: BlazeFace: sub-millisecond neural face detection on mobile GPUs (2019). https://doi.org/10.48550/arXiv.1907.05047. http://arxiv.org/abs/1907.05047. arXiv:1907.05047 [cs]
Bond, R.R.: SenseCare: using affective computing to manage and care for the emotional wellbeing of older people. In: Giokas, K., Bokor, L., Hopfgartner, F. (eds.) eHealth 360\(^\circ \). LNICST, vol. 181, pp. 352–356. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49655-9_42
Donovan, R., et al.: SenseCare: using automatic emotional analysis to provide effective tools for supporting, pp. 2682–2687 (2018). https://doi.org/10.1109/BIBM.2018.8621250
Franzen, M., Gresser, M.S., Müller, T., Mauser, P.D.S.: Developing emotion recognition for video conference software to support people with autism. arXiv:2101.10785 [cs] (2021)
Glueckauf, R.L., et al.: Survey of psychologists’ telebehavioral health practices: technology use, ethical issues, and training needs. Prof. Psychol. Res. Pract. 49(3), 205 (2018). https://doi.org/10.1037/pro0000188. https://psycnet.apa.org/fulltext/2018-28691-004.pdf. publisher: US: American Psychological Association
Grozev, B.: Efficient and scalable video conferences with selective forwarding units and WebRTC. Ph.D. thesis, Université de Strasbourg (2019). https://doi.org/10.13140/RG.2.2.11791.20645
Healy, M., Donovan, R., Walsh, P., Zheng, H.: A machine learning emotion detection platform to support affective well being. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 2694–2700 (2018). https://doi.org/10.1109/BIBM.2018.8621562
Howard, A., et al.: Searching for MobileNetV3 (2019). http://arxiv.org/abs/1905.02244. arXiv:1905.02244 [cs]
LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 253–256. IEEE, Paris, France (2010). https://doi.org/10.1109/ISCAS.2010.5537907. http://ieeexplore.ieee.org/document/5537907/
Maier, D.: SourceCode Paper-Realtime-Emotionrecognition. https://github.com/DennisMaier/Paper-Realtime-Emotionrecognition
Sander, J., et al.: Online therapy: an added value for inpatient routine care? Perspectives from mental health care professionals. Euro. Arch. Psychiatry Clin. Neurosci. 272, 107–118 (2021). https://doi.org/10.1007/s00406-021-01251-1
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Maier, D., Hemmje, M., Kikic, Z., Wefers, F. (2024). Real-Time Emotion Recognition in Online Video Conferences for Medical Consultations. In: Nicosia, G., Ojha, V., La Malfa, E., La Malfa, G., Pardalos, P.M., Umeton, R. (eds) Machine Learning, Optimization, and Data Science. LOD 2023. Lecture Notes in Computer Science, vol 14505. Springer, Cham. https://doi.org/10.1007/978-3-031-53969-5_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-53969-5_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53968-8
Online ISBN: 978-3-031-53969-5
eBook Packages: Computer ScienceComputer Science (R0)