Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Oct 23:13:375.
doi: 10.3389/fnhum.2019.00375. eCollection 2019.

Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures

Affiliations

Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures

Erdinç İşbilir et al. Front Hum Neurosci. .

Abstract

Recent advances in neuroimaging technologies have rendered multimodal analysis of operators' cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts' neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices' response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field.

Keywords: cognitive workload; eye tracking; fNIRS; human machine interface; neuroergonomics.

PubMed Disclaimer

Figures

Figure 1
Figure 1
The functional near-infrared spectroscopy (fNIRS) probe with four light sources and 10 detectors, and the corresponding 16 optodes mapped over the prefrontal cortex (PFC).
Figure 2
Figure 2
The Pupil Labs mobile eye tracking system.
Figure 3
Figure 3
The installation of the fNIRS sensor and the mobile eye tracker for multimodal recording with minimal IR interference.
Figure 4
Figure 4
A wirediagram of the graphical user interface used by the participants during the experiment. Error messages were popped up as a separate window in the middle of the screen in the event of exceptions.
Figure 5
Figure 5
Task completion times for the novice and expert groups for task 1 (left) and task 2 (right). The whiskers indicate standard error. Significant differences are indicated with an asterisk.
Figure 6
Figure 6
Average HbT changes observed at 16 optodes during the second task before and after the prompted error message.
Figure 7
Figure 7
BSpline interpolated F-ratios that exceeded the significance threshold obtained for each optode for the contrast between mean HbT changes during post- and pre-error message. The significant responses in both groups were predominantly localized over the left dorsolateral prefrontal cortex (dlPFC) together with a narrower region within right dlPFC.
Figure 8
Figure 8
BSpline interpolated F-ratios that exceeded the significance threshold obtained for each optode for the interaction of expertise level and pre-/post-episode on the mean HbT changes. The experts were most distinguished from novices with respect to their HbT response in the right dlPFC when their pre- and post-error average HbT responses were contrasted.
Figure 9
Figure 9
Bar charts showing the average fixation durations (left) and saccadic amplitudes (right) observed for the novice and expert operators during the first task.
Figure 10
Figure 10
Bar charts showing the average fixation durations observed for the novice and expert operators during the second task.
Figure 11
Figure 11
The histogram for the single discriminant function that separate novices (blue) and experts (red).

Similar articles

Cited by

References

    1. Aghajani H., Garbey M., Omurtag A. (2017). Measuring mental workload with EEG+fNIRS. Front. Hum. Neurosci. 11:359. 10.3389/fnhum.2017.00359 - DOI - PMC - PubMed
    1. Ayaz H. (2010). Functional Near Infrared Spectroscopy Based Brain Computer Interface. Philadelphia, PA: Drexel University.
    1. Ayaz H., Izzetoglu M., Platek S. M., Bunce S., Izzetoglu K., Pourrezaei K., et al. (2006). “Registering fNIR data to brain surface image using MRI templates,” in Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society (New York, NY: IEEE; ), 2671–2674. - PubMed
    1. Ayaz H., Onaral B., Izzetoglu K., Shewokis P. A., McKendrick R., Parasuraman R. (2013). Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: empirical examples and a technological development. Front. Hum. Neurosci. 7:871. 10.3389/fnhum.2013.00871 - DOI - PMC - PubMed
    1. Ayaz H., Shewokis P. A., Bunce S., Izzetoglu K., Willems B., Onaral B. (2012). Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36–47. 10.1016/j.neuroimage.2011.06.023 - DOI - PubMed

LinkOut - more resources