Keywords

1 Introduction

Recent reports have indicated a shortage of information security staff, particularly in small and medium-sized enterprises (SMEs), which can lead to a vulnerability for supply chain attacks.

According to the “Fundamental Research for Education of Information Security Human Resources” published by the Information-technology Promotion Agency, Japan (IPA) in 2012 and its continual research published in 2014 [1], the shortage of human resources in the cybersecurity field is estimated to be approximately 81,000 people, and 61,000 people are working for companies that do not have a human resource department in this area.. In addition, the Institute of Information Security stated in its “Report on questionnaire survey on information security incidents” [2] that approximately 25% of the SMEs in Japan do not have human resources in the field of cybersecurity and that approximately 41% have only one concurrent cybersecurity information staff member. Therefore, approximately 67% of the SMEs have limited or no human resources in this field. Moreover, there have been some reports of advanced persistent threats in which attackers have attempted to attack related groups or companies that are less secure to initial attacks. One such example was reported by IPA, Japan [3]. Thus, it is important to encourage less secure communities such as SMEs to take cybersecurity more seriously.

To improve this situation, a working group for the development of human resources for cybersecurity was organized in the Cyber ​​Security Center in Japan, which published its “Measures for Developing Cybersecurity Human Resources Inter-Group Working Report [4]” in 2018. In this report, it was reported that we require not only specialists to work in the domain of cybersecurity or general IT operations, but also experts who understand security activities and business well. In addition, numerous guidelines have been published as teaching materials for self-learning based on actual studies. Such guidelines was presented on the website of the Ministry of Economy Trade and Industry, Japan, and as of July 1,2019 [5] the number of published guidelines has reached over 150. However, it is difficult to grasp which parts of a security countermeasure activity the guidelines correspond to, and learners may encounter difficulty in finding and selecting appropriate guidelines to create a learning strategy.

2 Research Objective

We herein present a procedure for designing improved learning-materials that allow learners to fully understand the subject of security. We suggest that the framework core of the Cybersecurity Framework [6] will function as a correct mental model for learners and will lead them to organize their knowledge and improve their learning strategy. In addition, because many guidelines on information security have been previously published, we decided to revise a published document based on these guidelines.

In particular, we analyzed the published documents using a text-mining approach for each section and chapter to allow the mapping to the framework core of Cybersecurity Framework 1.1 and consider the accuracy of the text-mining results compared with template coding results.

We intend to examine how much this improves the learners understanding and how much it changes their mental model, although these are treated as a future study.

3 Related Studies

Security education/training studies have been conducted from various perspectives in Japan. A study on career development in the domain of cybersecurity was conducted by Kyoko Honda [7], which focused only on the dedicated (full-time) security members and the role and occupational relationship in security-related activities. The study did not consider the effect of education or training. Another study focused on the training contents; we also found other studies that used a capture the flag (CTF) approach for practice-based training [8] and some that focused on the practice environment, because an attackable environment is required for the exercise of cybersecurity practices [9]. Son et al. studied the curriculum of universities and graduate schools for the education of experts [10]. They analyzed the curriculum based on the Cybersecurity Workforce Framework [11] defined by The National Initiative for Cybersecurity Education (NICE) under the National Institute of Standards and Technology (NIST).

As mentioned by a workshop conducted by the Cyber Security Center, experts are required to learn not only the techniques but also various learning categories (in most cases, during their non-full-time security career) to connect the security fields with management along with the knowledge of the business and security activities of both these fields. To achieve this, we need to support students in good self-directed learning.

4 Analysis for Creating Training Material

To create an improved document, we decided to revise a published document based on the core of Cybersecurity Framework 1.1. We need to clarify the correspondence between the published document and the framework core. We took a similar approach as the “Proposal to visualize the contents of information security guidelines based on the Cybersecurity Framework” [12], which is a procedure used to visualize the content of a document based on the framework core; however, in this paper, we apply it to each chapter and section of the document to try and evaluate the content and clarify the correspondence.

4.1 Existing Materials Used for Creating New Improved Materials

We created improved learning materials based on two documents: Cybersecurity Framework and Guideline of Information Security for SMEs.

The Cybersecurity Framework is a document published by the National Institute of Standards and Technology and was originally created for summarizing the security measures in a critical infrastructure. It is based on aggregating the standards, guidelines, and best practices in the cybersecurity domain, and provides a systematic and structured holistic view on various security measures. We used the version translated into Japanese published by IPA, Japan [13] for the following analysis.

“Guideline of Information Security for SMEs 3rd edition” is a document published by the IPA and was created to support SMEs’ information security staff members in Japan to create a security policy and determine possible security measures.

4.2 Framework Core

The framework core has a layered structure, where each element is called either a function, category, or sub-category. Functions are the highest-level cybersecurity activities, and include identify, protect, detect, respond, and recover. Categories are the subdivisions of a function (Table 1).

Table 1. Function and category of unique identifiers (quoted from Cybersecurity Framework 1.1 [6])
Full size table

4.3 Simple Text-Mining Analysis

We created a feature word vector using tf-idf for each category of the framework core using the descriptions in Cybersecurity Framework 1.1. We calculated the cosine similarity between the feature vectors of each category and each chapter and section of “Guideline of Information Security for SMEs 3rd edition.” The results of the chapters with a function view integrating the value of categories in that function are shown in Table 2. We believe that the result of this text-mining shows how much the chapters and sections are similar to each category of the framework core. In this case, we can consider identify and protection as the main topics for all chapters.

Table 2. Text mining analysis results for chapters with functional view
Full size table

4.4 Validation of Test-Mining Analysis Compared with Template Coding Results

We evaluated the validity of the results of the text-mining of the documents, chapters, and sections using the same approach as conducted by the authors of the previous paper “Proposal to visualize the contents of information security guidelines based on the Cybersecurity Framework.”

Template Coding Based on Cybersecurity Framework

To measure the validity of the text-mining results, we need to quantitatively express how they recognize the contents of the document. We used the qualitative coding method to conduct an analysis through a quantifiable approach, similar to the previous paper.

Template coding is a procedure for defining the word group (code group) used for coding in advance. We used the sub-category of the framework core of Cybersecurity Framework 1.1 as a code group to obtain results comparable to those of the text-mining results.

The “Guideline of Information Security for SMEs 3rd edition” is analyzed herein using template coding and the results of the chapters, as shown in Table 3.

Table 3. Template coding results for chapters with functional view
Full size table

Validate the Results in Document, Chapter, and Section

We calculated the Pearson’s correlation coefficient between the text-mining results and the template coding results for each chapter and section and took the averages (Table 4). We also recalculated the Pearson’s correlation coefficient of the “Guideline of Information Security for SMEs 3rd edition” because the results of the previous paper are based on the second edition.

Table 4. Pearson’s correlation coefficient between the text-mining and the template coding
Full size table

The document level analysis results show a strong correlation, the chapter level analysis results also show a rather strong correlation, and the section level analysis results show a weak correlation. Thus, we can infer that we can use the text mining results as a clue to determine which chapter (or section) is related to which category of the framework core.

5 Considering the Design of Training Material and Experiment

We mapped each section and chapter of “Guideline of Information Security for SMEs 3rd edition” to the categories that seem to be related based on the information of the text-mining analysis and template coding the results of the chapters, as shown in Table 5.

Table 5. Map chapters to categories
Full size table

During the experiment, we intended to investigate how improvements are achieved through a change in the interface and how the mental model of the learner is changed by the learning activity. Therefore, we did not change the content or architecture of the “Guideline of Information Security for SMEs” but simply showed the relationship to the categories of the framework core. For example, we plan to show lines to the related category in when the mouse is hovered over an item of the “Guideline of Information Security for SMEs” (Fig. 1).

Fig. 1.
figure 1

Rough design of training material console

Full size image

6 Conclusion and Future Works

We analyzed the “Guideline of Information Security for SMEs” using a text-mining approach and checked the accuracy in comparison with the results of the template coding approach. Weak correlations were found even for the average sections, and it was inferred that we can use the results of text-mining as a clue for mapping between the contents of “Guideline of Information Security for SMEs” and the categories of the core of the framework. Based on these results, we created a mapping table between the sections of the “Guideline of Information Security for SMEs” and the framework core. We also presented a rough design of the user interface for the materials.

Finally, we created self-learning materials and as a future study plan to conduct a user experiment to research the effectiveness of the materials and determine how much it changes the mental models of the learners.