{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,17]],"date-time":"2024-08-17T09:54:45Z","timestamp":1723888485775},"reference-count":49,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2020,6,20]],"date-time":"2020-06-20T00:00:00Z","timestamp":1592611200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2019R1F1A1057984"],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"Emotion recognition plays an important role in the field of human\u2013computer interaction (HCI). An electroencephalogram (EEG) is widely used to estimate human emotion owing to its convenience and mobility. Deep neural network (DNN) approaches using an EEG for emotion recognition have recently shown remarkable improvement in terms of their recognition accuracy. However, most studies in this field still require a separate process for extracting handcrafted features despite the ability of a DNN to extract meaningful features by itself. In this paper, we propose a novel method for recognizing an emotion based on the use of three-dimensional convolutional neural networks (3D CNNs), with an efficient representation of the spatio-temporal representations of EEG signals. First, we spatially reconstruct raw EEG signals represented as stacks of one-dimensional (1D) time series data to two-dimensional (2D) EEG frames according to the original electrode position. We then represent a 3D EEG stream by concatenating the 2D EEG frames to the time axis. These 3D reconstructions of the raw EEG signals can be efficiently combined with 3D CNNs, which have shown a remarkable feature representation from spatio-temporal data. Herein, we demonstrate the accuracy of the emotional classification of the proposed method through extensive experiments on the DEAP (a Dataset for Emotion Analysis using EEG, Physiological, and video signals) dataset. Experimental results show that the proposed method achieves a classification accuracy of 99.11%, 99.74%, and 99.73% in the binary classification of valence and arousal, and, in four-class classification, respectively. We investigate the spatio-temporal effectiveness of the proposed method by comparing it to several types of input methods with 2D\/3D CNN. We then verify the best performing shape of both the kernel and input data experimentally. We verify that an efficient representation of an EEG and a network that fully takes advantage of the data characteristics can outperform methods that apply handcrafted features.<\/jats:p>","DOI":"10.3390\/s20123491","type":"journal-article","created":{"date-parts":[[2020,6,23]],"date-time":"2020-06-23T13:05:33Z","timestamp":1592917533000},"page":"3491","source":"Crossref","is-referenced-by-count":43,"title":["Spatio-Temporal Representation of an Electoencephalogram for Emotion Recognition Using a Three-Dimensional Convolutional Neural Network"],"prefix":"10.3390","volume":"20","author":[{"given":"Jungchan","family":"Cho","sequence":"first","affiliation":[{"name":"Department of Software, Gachon University, Seongnam 1342, Korea"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-3241-8455","authenticated-orcid":false,"given":"Hyoseok","family":"Hwang","sequence":"additional","affiliation":[{"name":"Department of Software, Gachon University, Seongnam 1342, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1007\/s00779-017-1072-7","article-title":"Towards emotion recognition for virtual environments: An evaluation of eeg features on benchmark dataset","volume":"21","author":"Menezes","year":"2017","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"De Nadai, S., D\u2019Inc\u00e0, M., Parodi, F., Benza, M., Trotta, A., Zero, E., Zero, L., and Sacile, R. (2016, January 12\u201316). Enhancing safety of transport by road by on-line monitoring of driver emotions. Proceedings of the 11th System of Systems Engineering Conference (SoSE), Kongsberg, Norway.","DOI":"10.1109\/SYSOSE.2016.7542941"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wang, F., Zhong, S.h., Peng, J., Jiang, J., and Liu, Y. (2018). Data augmentation for eeg-based emotion recognition with deep convolutional neural networks. International Conference on Multimedia Modeling, Springer.","DOI":"10.1007\/978-3-319-73600-6_8"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Guo, R., Li, S., He, L., Gao, W., Qi, H., and Owens, G. (2013, January 5\u20138). Pervasive and unobtrusive emotion sensing for human mental health. Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops, Venice, Italy.","DOI":"10.4108\/icst.pervasivehealth.2013.252133"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"99","DOI":"10.5334\/pb-46-1-2-99","article-title":"Psychopathy and physiological detection of concealed information: A review","volume":"46","author":"Verschuere","year":"2006","journal-title":"Psychol. Belg."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Duan, R.N., Zhu, J.Y., and Lu, B.L. (2013, January 6\u20138). Differential entropy feature for EEG-based emotion classification. Proceedings of the 6th International IEEE\/EMBS Conference on Neural Engineering (NER), San Diego, CA, USA.","DOI":"10.1109\/NER.2013.6695876"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"8375","DOI":"10.1109\/ACCESS.2016.2628407","article-title":"Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation","volume":"4","author":"Zhang","year":"2016","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tao, F., Liu, G., and Zhao, Q. (2018, January 20\u201322). An ensemble framework of voice-based emotion recognition system for films and TV programs. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Beijing, China.","DOI":"10.1109\/ICASSP.2018.8461617"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yang, Y., Wu, Q., Fu, Y., and Chen, X. (2018). Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition. International Conference on Neural Information Processing, Springer.","DOI":"10.1007\/978-3-030-04239-4_39"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"051001","DOI":"10.1088\/1741-2552\/ab260c","article-title":"Deep learning-based electroencephalography analysis: A systematic review","volume":"16","author":"Roy","year":"2019","journal-title":"J. Neural Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1337","DOI":"10.1007\/s11517-011-0828-x","article-title":"A comparison of univariate, vector, bilinear autoregressive, and band power features for brain\u2013computer interfaces","volume":"49","author":"Brunner","year":"2011","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Li, X., Song, D., Zhang, P., Yu, G., Hou, Y., and Hu, B. (2016, January 15\u201318). Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Shenzhen, China.","DOI":"10.1109\/BIBM.2016.7822545"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kwon, Y.H., Shin, S.B., and Kim, S.D. (2018). Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system. Sensors, 18.","DOI":"10.3390\/s18051383"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, A., Yang, B., and Huang, L. (2008, January 27\u201330). Feature extraction of EEG signals using power spectral entropy. Proceedings of the International Conference on BioMedical Engineering and Informatics, Sanya, China.","DOI":"10.1109\/BMEI.2008.254"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1109\/TAFFC.2014.2339834","article-title":"Feature extraction and selection for emotion recognition from EEG","volume":"5","author":"Jenke","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2230","DOI":"10.1016\/j.compbiomed.2013.10.017","article-title":"EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm","volume":"43","author":"Yoon","year":"2013","journal-title":"Comput. Biol. Med."},{"key":"ref_17","unstructured":"Bajaj, V., and Pachori, R.B. (June, January 30). Human emotion classification from EEG signals using multiwavelet transform. Proceedings of the International Conference on Medical Biometrics, Shenzhen, China."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chen, D.W., Miao, R., Yang, W.Q., Liang, Y., Chen, H.H., Huang, L., Deng, C.J., and Han, N. (2019). A feature extraction method based on differential entropy and linear discriminant analysis for emotion recognition. Sensors, 19.","DOI":"10.3390\/s19071631"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"44317","DOI":"10.1109\/ACCESS.2019.2908285","article-title":"Accurate EEG-based emotion recognition on combined features using deep convolutional neural networks","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_20","unstructured":"Tripathi, S., Acharya, S., Sharma, R.D., Mittal, S., and Bhattacharya, S. (2017, January 6\u20139). Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset. Proceedings of the Twenty-Ninth IAAI Conference, San Francisco, CA, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yang, H., Han, J., and Min, K. (2019). A Multi-Column CNN Model for Emotion Recognition from EEG Signals. Sensors, 19.","DOI":"10.3390\/s19214736"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shao, H.M., Wang, J.G., Wang, Y., Yao, Y., and Liu, J. (2019, January 24\u201327). EEG-Based Emotion Recognition with Deep Convolution Neural Network. Proceedings of the IEEE 8th Data Driven Control and Learning Systems Conference (DDCLS), Dali, China.","DOI":"10.1109\/DDCLS.2019.8908880"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Cho, J., and Lee, M. (2019). Building a Compact Convolutional Neural Network for Embedded Intelligent Sensor Systems Using Group Sparsity and Knowledge Distillation. Sensors, 19.","DOI":"10.3390\/s19194307"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lin, W., Li, C., and Sun, S. (2017). Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal. International Conference on Image and Graphics, Springer.","DOI":"10.1007\/978-3-319-71589-6_33"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Li, Z., Tian, X., Shu, L., Xu, X., and Hu, B. (2017). Emotion recognition from eeg using rasm and lstm. International Conference on Internet Multimedia Computing and Service, Springer.","DOI":"10.1007\/978-981-10-8530-7_30"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, Y., Huang, J., Zhou, H., and Zhong, N. (2017). Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Appl. Sci., 7.","DOI":"10.3390\/app7101060"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chao, H., Dong, L., Liu, Y., and Lu, B. (2019). Emotion recognition from multiband EEG signals using CapsNet. Sensors, 19.","DOI":"10.3390\/s19092212"},{"key":"ref_28","first-page":"355","article-title":"Emotion recognition based on EEG using LSTM recurrent neural network","volume":"8","author":"Alhagry","year":"2017","journal-title":"Emotion"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Yang, Y., Wu, Q., Qiu, M., Wang, Y., and Chen, X. (2018, January 8\u201313). Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.","DOI":"10.1109\/IJCNN.2018.8489331"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11\u201318). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Las Condes, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Qiu, Z., Yao, T., and Mei, T. (2017, January 22\u201329). Learning spatio-temporal representation with pseudo-3d residual networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.590"},{"key":"ref_32","first-page":"329","article-title":"EEG-based emotion recognition using 3D convolutional neural networks","volume":"9","author":"Salama","year":"2018","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"46007","DOI":"10.1109\/ACCESS.2020.2978163","article-title":"EEG-Based Emotion Classification Using Spiking Neural Networks","volume":"8","author":"Luo","year":"2020","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/TAMD.2015.2431497","article-title":"Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks","volume":"7","author":"Zheng","year":"2015","journal-title":"IEEE Trans. Autonom. Ment. Dev."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Cimtay, Y., and Ekmekcioglu, E. (2020). Investigating the Use of Pretrained Convolutional Neural Network on Cross-Subject and Cross-Dataset EEG Emotion Recognition. Sensors, 20.","DOI":"10.3390\/s20072034"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"Deap: A database for emotion analysis; using physiological signals","volume":"3","author":"Koelstra","year":"2011","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1017\/S0954579405050340","article-title":"The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology","volume":"17","author":"Posner","year":"2005","journal-title":"Dev. Psychopathol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1037\/0003-066X.50.5.372","article-title":"The emotion probe: Studies of motivation and attention","volume":"50","author":"Lang","year":"1995","journal-title":"Am. Psychol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/s10723-018-9462-2","article-title":"Enhancing text using emotion detected from EEG signals","volume":"17","author":"Gupta","year":"2019","journal-title":"J. Grid Comput."},{"key":"ref_41","unstructured":"Haykin, S. (1994). Neural Networks: A Comprehensive Foundation, Prentice Hall PTR."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.neucom.2013.06.046","article-title":"Emotional state classification from EEG data using machine learning approach","volume":"129","author":"Wang","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hara, K., Kataoka, H., and Satoh, Y. (2018, January 18\u201323). Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00685"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1510","DOI":"10.1109\/TPAMI.2017.2712608","article-title":"Long-term temporal convolutions for action recognition","volume":"40","author":"Varol","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18\u201323). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00675"},{"key":"ref_46","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23\u201328). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.223"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"37","DOI":"10.3389\/fnbot.2019.00037","article-title":"SAE+ LSTM: A New framework for emotion recognition from multi-channel EEG","volume":"13","author":"Xing","year":"2019","journal-title":"Front. Neurorobot."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/12\/3491\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,30]],"date-time":"2024-06-30T05:40:39Z","timestamp":1719726039000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/12\/3491"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,20]]},"references-count":49,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["s20123491"],"URL":"https:\/\/doi.org\/10.3390\/s20123491","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,20]]}}}