Keywords

1 Introduction

Introduction of cyber as a domain of operations [1] places enhanced metacognitive demands on individuals as task characteristics require effective coordination between multiple agents and asset types (human, technical and intangible). Human factors in cyber defence are getting increased attention from research communities (see. e.g. [2,3,4,5]), however there are currently no available performance measures to evaluate human performance in cyber operations [6, 7]. Consequently no common best practice or guidelines are found in the area of education and training for cyber defence individuals and teams [8,9,10].

1.1 Hybrid Space

This contribution aims to address the issue of performance measurement by utilising the versatility of the Hybrid Space framework (HS) (Fig. 1a; [11]). The HS frames the interconnection between cyber-physical (x-axis) and the strategic-tactical (y-axis) dimensions. The adaptability of the HS framework allowed researchers to visualize the location of a cognitive focus, at a given point-in-time, based upon the subjects’ self-report. Understanding the processes and actions required to enhance performance in a hybrid environment may rely on metacognitive skills in human computer interaction and consequently how these skills aid cognitive agility in the HS.

Fig. 1.
figure 1

(a) The Hybrid Space. (b) Exemplary visualization of cognitive agility

Full size image

1.2 Metacognition

Metacognition refers to ‘thinking about thinking’ and includes the components knowledge of one’s abilities, situational awareness, and behavioral regulation strategies [12]. Individuals with high metacognitive skills have more accurate and confident judgment of their own performance in relation to task demands and are better able to accurately describe their strengths, weaknesses, and their potential to improve. Metacognition is considered as having two dimensions: metacognitive awareness and metacognitive regulation [12].

Metacognitive Awareness and Regulation:

Metacognitive awareness refers to what learners know about learning and includes knowledge of one’s own cognitive abilities (e.g. ‘I have trouble remembering dates in history’) and knowledge of the particular tasks (e.g. ‘The ideas in this chapter that I’m going to read are complex’). High metacognitive awareness of own cognitive processes (planning, monitoring, evaluations) can facilitate accurate judgment of performance levels [13]. Metacognitive regulation refers to how people monitor their learning and control their cognitive processes while learning, for example, realising that a particular strategy is not efficient in reaching one’s goals and being able to change to more efficient strategies. In a cyber operations context this can mean recognising a potential threat in cyberspace as exceeding individual technical abilities and consider activating additional personal or technical resources in the physical domain. Developing a metacognitive understanding of own behavior can be understood as becoming aware that the outcomes of previous actions were taken under immense time pressure to serve short-term goals serving primarily tactical purposes, then re-adjust previous short-term goals to focus on more strategic and long term goals. The ability to be metacognitively aware of own performance without underestimation of own capacities or inappropriate over-confidence is considered a relatively stable personality trait that can be quantified and made subject to training and improvement [11].

1.3 State of Art

While much research is found in the field of metacognition, critical thinking and learning (see [14] for full review), there is a growing interest focusing on metacognitive strategies within the cyber security domain.

The ability of decision-makers and problem-solvers to maintain cyberspace operations can be affected by changes in the information environment [15]. To mitigate this, cognitive constructs should be included into future simulations and modeling of cyber attacks to increase the understanding of the effects of the attacks [15].

Problem solving involves several cognitive processes. Problems that require high-order cognitive processes can be characterized into well-, semi-, ill-, and severely ill-structured problem [16]. Critical thinking and metacognition are needed when semi-, ill-, and severely ill structured problems are solved successfully. In the extended paper of their original study, [17] the authors discuss further the need to develop learning tools that elicit sense-making and metacognitive processes. Empowering novices’ sense-making skills and metacognitive processes would accelerate their path to become expert.

The above research can also be applied when attempting to improve human performance in Cyberspace operations. The challenges presented via cyber come in many known and still to be discovered forms. Typically they can vary from well-structured to severely ill-structured.

What is a ‘good’ performance in cyber tasks is still under discussion [6, 7]. Earlier research shows that measuring performance in cyber security training scenarios requires more than simply ‘capture-the-flag’ type competitions [18, 19]. Increasingly the importance of developing human abilities to grasp threat complexities, understand and minimize consequences and communication are taken into consideration when performance is evaluated [20].

Good situational awareness is a factor that increases the probability for good performance, though it does not guarantee it, it is a good starting point [21]. This emphasizes the benefit of education and training methods that build on cognitive processes capable of improving human capacities to gain and maintain situation awareness. For example, an experiment that contains four elements for measuring performance that are measured at random times: mission, cyber solution, metrics and game, can support prospects of improved cyber situational awareness [22]. The following two examples are techniques capable of measuring situational awareness in cyber training scenarios.

SAGAT [23] protocol contains several questions on three different situational awareness levels: perception, comprehension, and projection. Answers are compared to the selected variables and the more accurate the answers are the higher-level awareness a person has. The results are provided on each situational awareness level.

QUASA [24] is a quantitative, combined probe and self-evaluation technique, where a person is required to answer ‘probe statements’ by either agreeing or disagreeing. After agreeing or disagreeing, the subject evaluates to what degree of confidence the prior assessment was made and states which of the other teams (participating the same experiment) will most likely give a correct answer to this probe.

In our study, the HS framework was used as a tool to measure cognitive focus movements, aka. cognitive agility, with the aim of relating these movements to metacognitive strategies. Focusing on these strategies during pre-training and educational programmes could support long-term development and application of high order cognitive skills during cyber defence scenarios, resulting in improved performance.

1.4 Hypothesis

Based on these previous findings and our assumptions, we hypothesized that higher metacognitive awareness would be positively associated with more movement in the HS. We further expected stronger metacognitive regulatory skills to be positively related to HS movements.

Focusing on metacognitive strategies as means of evaluating cognitive performance gave us the opportunity to validate the above core assumptions on which the HS is based. We tested how metacognitive strategies, as measured by meta-cognitive awareness and self-regulatory processes influence HS movement.

2 Method

This study operationalized and quantified cyber operators’ subjective movements in the HS as a function of metacognitive abilities in individual officers during a cyber defense exercise (CDX).

2.1 Cyber Defense Exercise

Data was collected during the Norwegian Defense Cyber Academy’s (NDCA) annual Cyber Defense Exercise (CDX). This is an arena that facilitates the opportunity for fourth year cyber engineer students to train in tactics, techniques and procedures for handling various types of cyber attacks. The exercise contributes to improving appreciation for the human and technical competences necessary to establish, manage and defend a military information infrastructure under simulated operational conditions. The students worked in four teams of 9 or 10 members (of a total of 37 students, 31 participated in the study), took decisions and acted in order to strengthen operational freedom, mission assurance and control in the cyber domain. The four teams participating in the exercise worked independently from each other but not against each other. Success was given in the form of expert feedback to the decisions and actions taken during the exercise. Intrusions were initiated by an affiliated agency engaged to help the NDCA with their education.

2.2 Measurements and Metrics

The teams were composed of 31 cyber officer cadets (M age = 22.7 years, SD = 0.71) resembling a complete cohort enlisted in the NDCA. The exercise lasted four days and data was collected on the third day.

The HS is mapped on a Cartesian plane visualizing the cyber-physical and tactical-strategic dimensions. Participants were asked to simultaneously mark their cognitive location within the HS [11] (see Fig. 1b) each hour from 08:00 to 20:00. In addition, students noted their current task at each position, to give context to further analysis (Fig. 2).

Fig. 2.
figure 2

Hybrid space movements

Full size image

Hybrid Space Operationalization:

Movement in the HS is operationalized through four constructs and represents the dependent variables in the study. Four dependent variables were created:

  • HSDT: distance traveled in the Cartesian Plane measured by Euclidian distance

  • HSQC: Number of quadrant changes

  • HSxM: Movement along the cyber-physical domain (x-axis)

  • HSyM: Movement along the strategic-tactical domain (y-axis)

To measure metacognitive awareness, the Metacognitive Awareness Inventory [25] was used. It is a self-report scale comprising of 52 items that includes several subscales assessing knowledge of cognition (declarative knowledge, procedural knowledge, conditional knowledge) and regulation of knowledge (planning, information management strategies, monitoring, debugging strategies and evaluation). Items are assessed on bipolar responses (true/false) and then ratios are computed from the subscales. Sample items include ‘I find myself using helpful learning strategies automatically’ (procedural knowledge) and ‘I ask myself if I have considered all options when solving a problem’ (comprehension monitoring). The test shows high reliability on all subscales (Cronbach’s α = .90).

To measure metacognitive regulation, the Self-Regulation Questionnaire [26] was used. The SRQ-63 is a 5-point Likert self-report scale ranging from strongly disagree to strongly agree. The scale has 7 subscales that consist of receiving, evaluating, triggering, searching, formulating, implementing, and assessing. Sample items include; ‘I usually keep track of my progress toward my goals’ and ‘I have sought out advice or information about changing’. The test shows high reliability (test-retest: r = .94, p < .0001; α = .91).

2.3 Statistical Analysis

To test the hypothesis multiple regression analyses on each of the HS dependent variables was performed. Metacognitive awareness and self-regulatory scores were entered as the independent variables and HS operationalizations were entered as the dependent variables. Alpha (α) significance levels are set to .05.

3 Results

To test the hypothesis, four regression analyses were performed where each of the Cognitive Agility Indices were entered as dependent variables and MCAI and SRQ correlates was entered as independent variables. Results are shown in Table 1.

Table 1. Descriptive Statistics (N = 31)
Full size table

Hybrid Space distance travelled (HSDT): Metacognitive awareness (debugging b = −.235, t = −1.317, p = .199) and total scores on self-regulation (b = .293, t = 1.644, p = .111) explained 17.8% of the distance moved in the Hybrid space (F = 3.040, p = .032, R 2 = .178, Adjusted R 2 = .120).

Quadrant changes (HSQC): Self-regulation predicted quadrant changes (F = 3.407, p = .023, R 2 = .345, Adjusted R 2 = .243) but only evaluating (b = .259, t = 1,484, p = .075), triggering b = .347, t = 1.964, p = .030), searching (b = .198, t = 1.122, p = .136) and implementing (b = −.229, t = −1.323, p = .099) were the only factors that contributed in explaining the variance.

Only self-regulation (evaluation r = .371, F = 4.640, p = .040, R 2 = .138, Adjusted R 2 = .108,) was positively associated with cognitive movements relative to the x-axis.

Neither metacognitive awareness nor self-regulatory processes could explain cognitive movements relative to the y-axis.

4 Discussion

Metacognition could predict movement in the HS, but not for y-axis movements. Y-axis movements may be dependent on fundamental cognition (i.e. rumination, worry, and self-efficacy) that may better explain vertical maneuvering. Metacognitions did have positive relationships to all other HS operationalizations but it was specific processes that could predict changes. Total distance travelled in the Hybrid Space (HSDT) was predicted by metacognitive debugging strategies, defined as a regulation of cognition used to correct comprehension and performance errors, and self-regulation. Evaluative metacognitive regulatory behaviors predicted x-axis movements, and along with triggering behaviors, searching for solutions, and implementing new strategies was associated with more quadrant changes.

This empirical contribution supports implementing the OLB pedagogic model as pathway to improved communication performance in socio-technical cyber-physical environments [27]. Formal teaching programmes designed to build metacognitive competence could facilitate cyber cadets’ use of intrinsic cognitive agility, as planned action. By measuring these performance metrics during cyberspace education and training scenarios - as cadets make planned conscious cross-quadrant maneuvers in the HS - the results can be collated into Cognitive Agility Index. It is hoped that this work will lead to the first science based performance indicator scale for cyber teams and individuals.

5 Conclusion

This research demonstrate that metacognition - measured as movements in the Hybrid Space framework - can be a useful method of evaluating individual cognitive performance in cyberspace operations.

The findings indicate impulsive cognitive movement due to the sample group acting without conscious thought. This is typical for people who have not undergone formal educational programmes of metacognitive learning. The participants reported movements could be defined as representing an intrinsic cognitive agility index. Given that metacognitive ability is trainable, the results of this experiment clearly indicate techniques to improve performance in a complex environment: such as that presented by the Hybrid Space framework.