Abstract
The development of brain-computer interface (BCI) systems has received considerable attention from neuroscientists in recent years. BCIs can serve as a means of communication and for the restoration of motor function for patients with motor disorders. An essential part of the design of a BCI is correctly classifying the brain signals, historically collected using electroencephalography (EEG). However, recent studies have shown more robust classification results when EEG is combined with other neuroimaging methods such as fNIRS. Conventional classification methods need a priori feature preprocessing to train the model; such feature selection is a difficult and heavily studied problem. By using deep neural networks (DNN), in which recordings can be fed directly to the algorithm for training, we avoid the need for feature selection. In this study, the capabilities of DNNs in the classification of the hybrid EEG-fNIRS recordings of motor imagery (MI) and mental workload (MWL) tasks are investigated. A five-layer fully connected network is used for classification. This study makes use of two open-source meta-datasets collected at the Technische Universitat Berlin. The first dataset includes brain activity recordings of 26 healthy participants during three cognitive tasks: (1) n-back (0-, 2- and 3-back), (2) discrimination/selection response task (DSR) and (3) word generation (WG) tasks. The second dataset, motor imagery, consists of left and right-hand motor imagery tasks, each for 29 healthy participants. Our results show that classification accuracy is considerably higher for multimodal recordings when compared to EEG or fNIRS recordings alone. The proposed algorithm improves classification performance relative to a conventional support vector machine (SVM), reaching 90% average accuracy for both tasks, 8% higher than SVM performance. These results demonstrate the feasibility of achieving strong classification performance using multimodal BCI and deep learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Hong, K.-S. Naseer, N., Kim, Y.-H.: Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neurosci. Lett. 587, 87–92 (2015)
Ayaz, H., Dehais, F.: Neuroergonomics: The Brain at Work and Everyday Life, 1st edn. Elsevier, Academic Press, Cambridge (2019)
Ahn, S., Jun, S.C.: Multi-modal integration of EEG-fNIRS for brain-computer interfaces - current limitations and future directions. Front. Hum. Neurosci. 11, 503 (2017). https://doi.org/10.3389/fnhum.2017.00503
Liu, Y., Ayaz, H., Shewokis, P.A.: Mental workload classification with concurrent electroencephalography and functional near-infrared spectroscopy. Brain Comput. Interfaces 4, 1–11 (2017). https://doi.org/10.1080/2326263x.2017.1304020
Dehais, F., Duprès, A., Di Flumeri, G., Verdière, K.J., Borghini, G., Babiloni, F., Roy, R.N.: Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI. In: IEEE SMC (2018)
Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., Ball, T.: Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 38(11), 5391–5420 (2017)
Wang, Z., Lyu, S., Schalk, G., Ji, Q.: Deep feature learning using target priors with applications in ECOG signal decoding for BCI. In: IJCAI, pp. 1785–1791 (2013)
Hajinoroozi, M., Mao, Z., Jung, T.-P., Lin, C.-T., Huang, Y.: EEG-based prediction of driver’s cognitive performance by deep convolutional neural network. Sig. Process. Image Commun. 47, 549–555 (2016)
An, X., Kuang, D., Guo, X., Zhao, Y., He, L.: A deep learning method for classification of EEG data based on motor imagery. In: Huang, D.S., Han, K., Gromiha, M. (eds.) Intelligent Computing in Bioinformatics. Lecture Notes in Computer Science, vol. 8590 (2014)
Hong, K.-S., Naseer, N., Kim, Y.-H.: Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neurosci. Lett. 587, 87–92 (2015)
Khan, M.J., Hong, M.J., Hong, K.-S.: Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front. Hum. Neurosci. 8, 244 (2014)
Coyle, S.M., Ward, T.E., Markham, C.M.: Brain computer interface using a simplified functional near-infrared spectroscopy system. J. Neural Eng. 4(3), 219 (2007)
Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Muller, K.-R., Blankertz, B.: Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage 59(1), 519–529 (2012)
Abibullaev, B., An, J., Moon, J.-I.: Neural network classification of brain hemodynamic responses from four mental tasks. Int. J. Optomechatronics 5(4), 340–359 (2011)
Hennrich, J., Her, C., Heger, D., Schultz, T.: Investigating deep learning for fNIRS based BCI. In: EMBC, pp. 2844–2847 (2015)
Nguyen, H.T., Ngo, C.Q., Truong Quang Dang, K., Vo, V.T.: Temporal hemodynamic classification of two hands tapping using functional near-infrared spectroscopy. Front. Hum. Neurosci. 7, 516 (2013)
Trakoolwilaiwan, T., Behboodi, B., Lee, J., Kim, K., Choi, J.-W.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain-computer interface: three-class classification of rest, right-, and left-Hand motor execution. Neurophotonics 5 (2007)
Croce, P., Zappasodi, F., Merla, A., Chiarelli, M.: Exploiting neurovascular coupling: a Bayesian sequential Monte Carlo approach applied to simulated EEG fNIRS data. J. Neural Eng. 14(4) (2017)
Chiarelli, A.M., Zappasodi, F., Di Pompeo, F., Merla, A.: Simultaneous functional near-infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: a review. Neurophotonics 4(4) (2017)
Ma, L., Zhang, L., Wang, L., Xu, M., Qi, H., Wan, B., Ming, D., Hu, Y.: A hybrid brain-computer interface combining the EEG and NIRS. In: 2012 IEEE International Conference Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS), pp. 159–162 (2012)
Lee, M.-H., Fazli, S., Mehnert, J., Lee, S.-W.: Hybrid brain-computer interface based on EEG and NIRS modalities. In: 2014 International Winter Workshop on Brain-Computer Interface (BCI), (2014)
Buccino, A.P., Keles, H.O., Omurtag, A.: Hybrid EEG-fNIRS asynchronous brain computer interface for multiple motor tasks. PloS ONE 11(1) (2016)
Jirayucharoensak, S., Pan-Ngum, S., Israsena, P.: EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci. World J. (2014)
Shin, J., Von Luhmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Muller, K.-R.: Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. In: Generic Research Data (2018)
Shin, J., Von Lhmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Muller, K.-R.: Simultaneous aquisition of EEG and NIRS during cognitive tasks for an open access dataset. In: Scientific Data, vol. 5 (2018)
Pfurtscheller, G.: Functional brain imaging based on ERD/ERS. Vis. Res. 41(10–11), 1257–1260 (2001)
Pourshafi, A., Saniei, M., Saeedian, A., Saadati, M.: Optimal reactive power compensation in a deregulated distribution network. In: 44th International Universities Power Engineering Conference (UPEC), pp. 1–6 (2009)
Neuper, C., Pfurtscheller, G.: Event-related dynamics of cortical rhythms: frequency-specific features and functional correlates. Int. J. Psychophysiol. 43(1), 41–58 (2001)
Saadati, M., Nelson, J.K.: Multiple transmitter localization using clustering by likelihood of transmitter proximity. In: 51st Asilomar Conference on Signals, Systems, and Computers, pp. 1769–1773 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Saadati, M., Nelson, J., Ayaz, H. (2020). Multimodal fNIRS-EEG Classification Using Deep Learning Algorithms for Brain-Computer Interfaces Purposes. In: Ayaz, H. (eds) Advances in Neuroergonomics and Cognitive Engineering. AHFE 2019. Advances in Intelligent Systems and Computing, vol 953. Springer, Cham. https://doi.org/10.1007/978-3-030-20473-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-20473-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20472-3
Online ISBN: 978-3-030-20473-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)