Self-Supervised Learning for Near-Wild Cognitive Workload Estimation | Journal of Medical Systems Skip to main content
Springer Nature Link
Log in

Self-Supervised Learning for Near-Wild Cognitive Workload Estimation

  • Research
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Feedback on cognitive workload may reduce decision-making mistakes. Machine learning-based models can produce feedback from physiological data such as electroencephalography (EEG) and electrocardiography (ECG). Supervised machine learning requires large training data sets that are (1) relevant and decontaminated and (2) carefully labeled for accurate approximation, a costly and tedious procedure. Commercial over-the-counter devices are low-cost resolutions for the real-time collection of physiological modalities. However, they produce significant artifacts when employed outside of laboratory settings, compromising machine learning accuracies. Additionally, the physiological modalities that most successfully machine-approximate cognitive workload in everyday settings are unknown. To address these challenges, a first-ever hybrid implementation of feature selection and self-supervised machine learning techniques is introduced. This model is employed on data collected outside controlled laboratory settings to (1) identify relevant physiological modalities to machine approximate six levels of cognitive-physical workloads from a seven-modality repository and (2) postulate limited labeling experiments and machine approximate mental-physical workloads using self-supervised learning techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

No datasets were generated or analysed during the current study.

References

  1. Jiang, L., Eickhoff, S.B., Genon, S., Wang, G., Yi, C., He, R., Huang, X., Yao, D., Dong, D., Li, F., Xu, P.: Multimodal covariance network reflects individual cognitive flexibility. International Journal of Neural Systems 34(4), 2450018–17 (2024)

    Article  PubMed  Google Scholar 

  2. Dawson, N.V.: Physician judgment in clinical settings: methodological influences and cognitive performance. Clinical Chemistry 39(7), 1468–1480 (1993)

    Article  CAS  PubMed  Google Scholar 

  3. Horvat, M., Tement, S.: Self-reported cognitive difficulties and cognitive functioning in relation to emotional exhaustion: Evidence from two studies. Stress and Health 36(3), 350–364 (2020)

    Article  PubMed  Google Scholar 

  4. McMorris, T.: Cognitive fatigue effects on physical performance: The role of interoception. Sports Medicine 50(10), 1703–1708 (2020)

    Article  PubMed  Google Scholar 

  5. Xiao, L.X., Zeng, J., Chen, C., Chi, H.-L., Shen, G.Q.: Smart work package learning for decentralized facial fatigue monitoring. Computer-Aided Civil and Infrastructure Engineering 38(6), 799–817 (2023)

    Article  Google Scholar 

  6. Mohsenvand, M.N., Izadi, M.R., Maes, P.: Contrastive representation learning for electroencephalogram classification. In: Machine Learning for Health, pp. 238–253 (2020). PMLR

  7. Belletier, C., Charkhabi, M., Andrade Silva, G., Ametepe, K., Lutz, M., Izaute, M.: Wearable cognitive assistants in a factory setting: a critical review of a promising way of enhancing cognitive performance and well-being. Cognition, Technology & Work 23(1), 103–116 (2021)

    Article  Google Scholar 

  8. Sankari, Z., Adeli, H.: Heartsaver: A mobile cardiac monitoring system for autodetection of atrial fibrillation, myocardial infarction, and atrio-ventricular block. Computers in Biology and Medicine 41(4), 211–220 (2011)

    Article  PubMed  Google Scholar 

  9. Graña, M., Aguilar-Moreno, M., De Lope Asiain, J., Araquistain, I.B., Garmendia, X.: Improved activity recognition combining inertial motion sensors and electroencephalogram signals. International Journal of Neural Systems 30(10), 2050053 (2020)

  10. Teran-Pineda, D., Thurnhofer-Hemsi, K., Dominguez, E.: Human gait activity recognition using multimodal sensors. International Journal of Neural Systems 33(11), 2350058–15 (2023)

    Article  PubMed  Google Scholar 

  11. Hoilett, O.S., Twibell, A.M., Srivastava, R., Linnes, J.C.: Kick ll: A smartwatch for monitoring respiration and heart rate using photoplethysmography. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3821–3824 (2018). IEEE

  12. Qaisar, S.M.: A computationally efficient eeg signals segmentation and de-noising based on an adaptive rate acquisition and processing. In: 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP), pp. 182–186 (2018). IEEE

  13. Ranjan, R., Sahana, B.C., Bhandari, A.K.: Ocular artifact elimination from electroencephalography signals: A systematic review. Biocybernetics and Biomedical Engineering 41(3), 960–996 (2021)

    Article  Google Scholar 

  14. Varone, G., Hussain, Z., Sheikh, Z., Howard, A., Boulila, W., Mahmud, M., Howard, N., Morabito, F.C., Hussain, A.: Real-time artifacts reduction during tms-eeg co-registration: a comprehensive review on technologies and procedures. Sensors 21(2), 637 (2021)

    Article  PubMed  PubMed Central  Google Scholar 

  15. Karakullukcu, N., Yilmaz, B.: Detection of movement intention in eeg-based brain-computer interfaces using fourier-based synchrosqueezing transform. International Journal of Neural Systems 32(1), 2150059 (2022). 15 pages

  16. Adeli, H., Hung, S.-L.: An adaptive conjugate gradient learning algorithm for efficient training of neural networks. Applied Mathematics and Computation 62(1), 81–102 (1994)

    Article  Google Scholar 

  17. Hung, S.-L., Adeli, H.: A parallel genetic/neural network learning algorithm for mimd shared memory machines. IEEE Transactions on Neural Networks 5(6), 900–909 (1994)

    Article  CAS  PubMed  Google Scholar 

  18. Hassanpour, A., Moradikia, M., Adeli, H., Khayami, S.R., Shamsinejadbabaki, P.: A novel end-to-end deep learning scheme for classifying multi-class motor imagery electroencephalography signals. Expert Systems 36(6), 12494 (2019)

    Article  Google Scholar 

  19. Burns, A., Adeli, H., Buford, J.A.: Brain–computer interface after nervous system injury. The Neuroscientist textbf20(6), 639–651 (2014)

  20. Acharya, U.R., Oh, S.L., Hagiwara, Y., Tan, J.H., Adeli, H.: Deep convolutional neural network for the automated detection and diagnosis of seizure using eeg signals. Computers in Biology and Medicine 100, 270–278 (2018)

    Article  PubMed  Google Scholar 

  21. Jiang, X., Bian, G.-B., Tian, Z.: Removal of artifacts from eeg signals: a review. Sensors 19(5), 987 (2019)

    Article  PubMed  PubMed Central  Google Scholar 

  22. Fridman, L., Reimer, B., Mehler, B., Freeman, W.T.: Cognitive load estimation in the wild. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–9 (2018)

  23. Burns, A., Adeli, H.: Wearable technology for patients with brain and spinal cord injuries. Reviews in the Neurosciences 28(8), 913–920 (2017)

    Article  PubMed  Google Scholar 

  24. Albuquerque, I., Tiwari, A., Parent, M., Cassani, R., Gagnon, J.-F., Lafond, D., Tremblay, S., Falk, T.H.: Wauc: a multi-modal database for mental workload assessment under physical activity. Frontiers in Neuroscience 14, 549524 (2020)

    Article  PubMed  PubMed Central  Google Scholar 

  25. Xu, F., Yan, Y., Zhu, J., Chen, X., Gao, L., Liu, Y., Shi, W., Lou, Y., Wang, W., Leng, J., Zhang, Y.: Self-supervised eeg representation learning with contrastive predictive coding for post-stroke. International Journal of Neural Systems 33(12), 2350066–16 (2023)

    Article  PubMed  Google Scholar 

  26. Martinez-Murcia, F.J., Arco, J.E., Jiménez-Mesa, C., Segovia, F., Illán, I.A., Ramírez, J., Görriz, J.M.: Bridging imaging and clinical scores in parkinson’s disease progression via multimodal self-supervised deep learning. International Journal of Neural Systems 34(8), 2450043–16 (2024)

    Article  PubMed  Google Scholar 

  27. Shen, J., Yan, W., Qin, S., Zheng, X.: A self-supervised monocular depth estimation model with scale recovery and transfer learning for construction scene analysis. Computer-Aided Civil and Infrastructure Engineering 38(9), 1142–1161 (2023)

    Article  Google Scholar 

  28. Huang, J., Yang, X., Zhou, F., Li, X., Zhou, B., Lu, S., Ivashov, S., Giannakis, I., Kong, F., Slob, E.: A deep learning framework based on improved self-supervised learning for ground penetrating radar tunnel lining inspection. Computer-Aided Civil and Infrastructure Engineering 39(6), 814–833 (2024)

    Article  Google Scholar 

  29. George, S.H., Rafiei, M.H., Gauthier, L., Borstad, A., Buford, J.A., Adeli, H.: Computer-aided prediction of extent of motor recovery following constraintinduced movement therapy in chronic stroke. Behavioural Brain Research 329, 191–199 (2017)

    Article  PubMed  Google Scholar 

  30. George, S.H., Rafiei, M.H., Borstad, A., Adeli, H., Gauthier, L.V.: Gross motor ability predicts response to upper extremity rehabilitation in chronic stroke. Behavioural Brain Research 333, 314–322 (2017)

    Article  PubMed  PubMed Central  Google Scholar 

  31. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607 (2020). PMLR

  32. Liang, M., Chang, Z., He, S., Chen, Y., Gan, Y., Schlangen, E., Šavija, B.: Predicting early-age stress evolution in restrained concrete by thermo-chemomechanical model and active ensemble learning. Computer-Aided Civil and Infrastructure Engineering 37(14), 1809–1833 (2022)

    Article  Google Scholar 

  33. Castillo-Barnes, D., Martinez-Murcia, F.J., Jiménez-Mesa, C., Arco, J.E., SalasGonzalez, D., Ramírez, J., Görriz, J.M.: Non-linear weighting ensemble learning model to diagnose parkinson’s disease using multimodal data. International Journal of Neural Systems 33(8), 2350041 (2023). (20 pages)

  34. Pak, H., Leach, S., Yoon, S.H., Paal, S.G.: A knowledge transfer enhanced ensemble approach to predict the shear capacity of reinforced concrete deep beams without stirrups. Computer-Aided Civil and Infrastructure Engineering 38(11), 1520–1535 (2023)

    Article  Google Scholar 

  35. Shaffi, N., Subramanian, K., Vimbi, V., Hajamohideen, F., Abdesselam, A., Mahmud, M.: Performance evaluation of deep, shallow, and ensemble machine learning methods for the automated classification of alzheimer’s disease. International Journal of Neural Systems 34(7), 2450029–17 (2024)

    Article  PubMed  Google Scholar 

  36. Rafiei, M.H., Gauthier, L.V., Adeli, H., Takabi, D.: Self-supervised learning for electroencephalography. IEEE Transactions on Neural Networks and Learning Systems (2022)

  37. Hart, S.G., Staveland, L.E.: Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in Human Psychology: Human Mental Workload (1988)

  38. Rafiei, M.H., Kelly, K.M., Borstad, A.L., Adeli, H., Gauthier, L.V.: Predicting improved daily use of the more affected arm poststroke following constraintinduced movement therapy. Physical Therapy 99(12), 1667–1678 (2019)

    Article  PubMed  PubMed Central  Google Scholar 

  39. Mohammadshirazi, A., Kalkhorani, V.A., Humes, J., Speno, B., Rike, J., Ramnath, R., Clark, J.D.: Predicting airborne pollutant concentrations and events in a commercial building using low-cost pollutant sensors and machine learning: a case study. Building and Environment 213, 108833 (2022)

    Article  Google Scholar 

  40. Abualigah, L.: Group search optimizer: a nature-inspired meta-heuristic optimization algorithm with its results, variants, and applications. Neural Computing and Applications 33(7), 2949–2972 (2021)

    Article  Google Scholar 

  41. Wang, J., Khishe, M., Kaveh, M., Mohammadi, H.: Binary chimp optimization algorithm (bchoa): A new binary meta-heuristic for solving optimization problems. Cognitive Computation 13(5), 1297–1316 (2021)

    Article  Google Scholar 

  42. Glover, F., Kochenberger, G., Xie, W., Luo, J.: Diversification methods for zeroone optimization. Journal of Heuristics 25(4), 643–671 (2019)

    Article  Google Scholar 

  43. Rafiei, M.H., Adeli, H.: A new neural dynamic classification algorithm. IEEE Transactions on Neural Networks and Learning Systems 28(12), 3074–3083 (2017)

    Article  PubMed  Google Scholar 

  44. Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(8), 1798–1828 (2013)

    Article  PubMed  Google Scholar 

  45. Ganjali, M., Mehridehnavi, A., Rakhshani, S., Khorasani, A.: Unsupervised neural manifold alignment for stable decoding of movement from cortical signals. International Journal of Neural Systems 34(1), 2450006–16 (2024)

    Article  PubMed  Google Scholar 

  46. Cui, J., Xiao, J., Hou, Y., Wu, X., Zhou, J., Peng, X., Wang, Y.: Unsupervised domain adaptive dose prediction via cross-attention transformer and targetspecific knowledge preservation. International Journal of Neural Systems 33(11), 2350057–18 (2023)

    Article  PubMed  Google Scholar 

  47. Gao, Y., Zhai, P., Mosalam, K.M.: Balanced semisupervised generative adversarial network for damage assessment from low-data imbalanced-class regime. Computer-Aided Civil and Infrastructure Engineering 36(9), 1094–1113 (2021)

    Article  Google Scholar 

  48. Hua, Y., Shu, X., Wang, Z., Zhang, L.: Uncertainty-guided voxel-level supervised contrastive learning for semi-supervised medical image segmentation. International Journal of Neural Systems 32(04), 2250016 (2022)

    Article  PubMed  Google Scholar 

  49. Zakeri, B., Monsefi, A.K., Samsam, S., Monsefi, B.K.: Weakly supervised learning technique for solving partial differential equations; case study of 1-d reactiondiffusion equation. In: International Congress on High-Performance Computing and Big Data Analysis, pp. 367–377 (2019). Springer

  50. Wijnands, J.S., Zhao, H., Nice, K.A., Thompson, J., Scully, K., Guo, J., Stevenson, M.: Identifying safe intersection design through unsupervised feature extraction from satellite imagery. Computer-Aided Civil and Infrastructure Engineering 36(3), 346–361 (2021)

    Article  Google Scholar 

  51. Banville, H., Chehab, O., Hyvärinen, A., Engemann, D.-A., Gramfort, A.: Uncovering the structure of clinical eeg signals with self-supervised learning. Journal of Neural Engineering 18(4), 046020 (2021)

    Article  Google Scholar 

  52. Ohri, K., Kumar, M.: Review on self-supervised image recognition using deep neural networks. Knowledge-Based Systems 224, 107090 (2021)

    Article  Google Scholar 

  53. Sun, W., Tagliasacchi, A., Deng, B., Sabour, S., Yazdani, S., Hinton, G.E., Yi, K.M.: Canonical capsules: Self-supervised capsules in canonical pose. Advances in Neural Information Processing Systems 34, 24993–25005 (2021)

    Google Scholar 

  54. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). IEEE

  55. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

  56. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  CAS  PubMed  Google Scholar 

  57. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345–1359 (2009)

    Article  Google Scholar 

  58. Yu, Z., Albera, L., Le Bouquin Jeannes, R., Kachenoura, A., Karfoul, A., Yang, C., Shu, H.: Epileptic seizure prediction using deep neural networks via transfer learning and multi-feature fusion. International Journal of Neural Systems, 2250032–2250032 (2022)

  59. Rafiei, M.H., Adeli, H.: A novel unsupervised deep learning model for global and local health condition assessment of structures. Engineering Structures 156, 598–607 (2018)

    Article  Google Scholar 

  60. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1), 1929–1958 (2014)

    Google Scholar 

  61. Krogh, A., Hertz, J.: A simple weight decay can improve generalization. Advances in Neural Information Processing Systems 4 (1991)

  62. Rafiei, M.H., Gu, Y., El-Awady, J.A.: Machine learning of dislocation-induced stress fields and interaction forces. JOM 72(12), 4380–4392 (2020)

  63. Wang, J., Sun, M., Huang, W.: Unsupervised eeg-based seizure anomaly detection with denoising diffusion probabilistic models. International Journal of Neural Systems 34(9), 2450047–14 (2024)

    Article  PubMed  Google Scholar 

  64. Wang, S., Feng, H., Lv, H., Nie, C., Feng, W., Peng, H., Zhang, L., Zhao, Y.: Cross-subject seizure detection via unsupervised domain-adaptation. International Journal of Neural Systems 34(10), 16–2450055 (2024)

  65. Fernandez-Rodríguez, J.D., Palomo, E.J., Ortiz-de-Lazcano-Lobato, J.M., Ramos-Jimenez, G., Lopez-Rubio, E.: Dynamic learning rates for continual unsupervised learning. Integrated Computer-Aided Engineering 30(3), 257–273 (2023)

    Article  Google Scholar 

  66. Giglioni, V., Venanzi, I., Poggioni, V., Milani, A., Ubertini, F.: Autoencoders for unsupervised real-time bridge health assessment. Computer-Aided Civil and Infrastructure Engineering 38(8), 959–974 (2023)

    Article  Google Scholar 

Download references

Acknowledgements

This research was not funded by any organization.

Author information

Author notes

  1. Lynne V. Gauthier, Hojjat Adeli and Daniel Takabi are contributed equally to this work.

Authors and Affiliations

  1. Whiting School of Engineering, Johns Hopkins University, 21218, Baltimore, MD, USA

    Mohammad H. Rafiei

  2. Department of Physical Therapy and Kinesiology, University of Massachusetts Lowell, 01854, Lowell, MA, USA

    Lynne V. Gauthier

  3. Departments of Biomedical Informatics and Neuroscience, The Ohio State University, 43210, Columbus, OH, USA

    Hojjat Adeli

  4. School of Cybersecurity, Old Dominion University, 23529, Norfolk, VA, USA

    Daniel Takabi

Authors
  1. Mohammad H. Rafiei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lynne V. Gauthier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hojjat Adeli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Takabi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MHR Developed the methodology, performed the numerical experimentation, obtained results, and wrote the first draft. LVG Provided suggestions to improve the methodology and presentation of the paper. HA Provided suggestions to improve the methodology and presentation of the paper. DT Provided suggestions to improve the methodology and presentation of the paper.

Corresponding author

Correspondence to Hojjat Adeli.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rafiei, M.H., Gauthier, L.V., Adeli, H. et al. Self-Supervised Learning for Near-Wild Cognitive Workload Estimation. J Med Syst 48, 107 (2024). https://doi.org/10.1007/s10916-024-02122-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-024-02122-7

Keywords

Search

Navigation