Leveraging Hierarchies: HMCAT for Efficiently Mapping CTI to Attack Techniques

Abstract

With the advancement of cyber technology, proactive security methods such as adversary emulation and leveraging Cyber Threat Intelligence (CTI) have become increasingly essential. Currently, some methods have achieved automatic mapping of unstructured text Cyber Threat Intelligence to attack techniques that could facilitate proactive security. However, these methods do not consider the semantic relationships between CTI and attack techniques at different abstraction levels, which leads to poor performance in the classification. In this work, we propose a Hierarchy-aware method for Mapping of CTI to Attack Techniques (HMCAT). Specifically, HMCAT first extracts Indicators of Compromise (IOC) entities in the CTI with two steps, then projects the CTI with IOC entities and the corresponding attack technique into a joint embedding space. Finally, HMCAT captures the semantics relationship among text descriptions, coarse-grained techniques, fine-grained techniques and unrelated techniques through a hierarchy-aware mapping loss. Meanwhile, we also propose a data augmentation technique based on in-context learning to solve the problem of long-tailed distribution in the Adversarial Tactics, Techniques and Common Knowledge (ATT&CK) datasets, which could further improve the performance of mapping. Experimental results demonstrate that HMCAT significantly outperforms previous ML and DL methods, improving precision, recall and F-Measure by 6.6%, 13.9% and 9.9% respectively.

Appendices

A The Comparison of Dataset Distributions

The initial distribution, depicted in Fig. 6(a), reveals numerous categories with a scarcity of samples. In some categories, as few as 3–5 samples are available due to limited data availability. Therefore, we use a data augmentation technique to enhance scarce categories. The augmented distribution and results are presented in Fig. 6(b). After using the data augmentation technique to increase samples, the model has a stronger ability to recognize different attack techniques and the text representation of different attack techniques becomes farther and farther away to avoid some classification confusion.

B Experimental Setup

In this Appendix, we introduce the datasets used in this paper and present the experimental results that demonstrate the advantages of our methods from a comparative perspective.

Fig. 6.

The comparison of distributions between the initial dataset and the augmented dataset.

1.1 B.1 Datasets and Evaluation Metrics

Mapping Datasets: We follow Orbinato’s method [21] to build the datasets that use the public knowledge base of the MITRE ATT&CK framework. Each sample in the datasets corresponds to a specific malicious technique and has been annotated with a label, representing a technique from the MITRE ATT&CK framework taxonomy. A detailed description and category distribution of the initial datasets are shown in Table 6 and Fig. 6(a), respectively. It’s obvious that the initial datasets have a long-tailed distribution which means there are a few samples for most of the categories. To address this problem, we use a method of data Augmentation with in-context learning to build the categories balance augmented datasets. We first divide the original datasets into a training set, validation set and test set with a ratio of 6:2:2, while ensuring the balance of classification. Then we keep the same validation set and test set for all experiments which would reduce data leakage and ensure fairness. Finally, for the training datasets, we use data augmentation on categories with less than 50 samples so that the model can better learn the characteristics of these categories. Specifically, we iteratively use in-context learning for data augmentation until the final number of samples reaches 50, and filter out low-quality samples through critic model.

NER Dataset: We use Dionisio et al. [12] dataset collected from tweets through a selection of manually curated accounts and passed through a filter. The fine-grained NER targets are manually labeled. Specifically, the dataset contains 11074 tweets and 12356 entities with five different types of IOCs: ID, ORG, PRO, VER and VUL, accounting for 5770, 926, 3349, 1445 and 866 respectively. The same as the mapping dataset, we divide the NER dataset into a training set, validation set and test set with a ratio of 6:2:2, while ensuring the balance of classification to train our NER model and evaluate it.

Table 6. A detailed description of datasets

Metrics: To compare the performance of different classification models, we consider three representative metrics: Precision, Recall and F-Measure.

1.2 B.2 Implementation Details

HMACT: We employ the ChatGPT(gpt-3.5-turbo-0301) model from the OpenAI API, with 175 billion parameters, as our LLM to generate new samples in the data augmentation step and we also fix the temperature to 0 to get the most certainty answer which can greatly reduce the number of low-quality samples. For categories that have less than 50 samples, we randomly select four demonstration examples as context to elicit ChatGPT to generate new samples until the total number of samples reaches 60. To further filter generated samples, we use SBERT as the critic model, which is based on dense sentence encoding. We set the upper and lower thresholds of similarity between generated samples and original samples as 0.8 and 0.3, respectively, which can eliminate duplicate and hallucinated samples. For the text encoder, we use two models: BERT [11] and RoBERTa [18]. For the label encoder in the hierarchical module, we initialize the GCN network with prior knowledge about the relationships between labels in the predefined hierarchy and corpus to accelerate the convergence of training. For the hierarchy-aware mapping, we set $\gamma$ to 0.5 to penalize siblings with the target label and 0.7, 1 for non-sibling nodes and higher-level nodes. For the NER model, we use the BERT-base model that adopts a 12-layer structure and maps text to 768 dimensions. During the training process, a fine-tuning strategy is applied, wherein the first four layers of the model are kept fixed, and the weights of the subsequent eight layers are adjusted with a low learning rate of $2\times 10^{-5}$. This approach ensures the effectiveness of training while reducing both training time and the required training set. The hidden state dimension of the BiLSTM is set to 256, and a dropout layer with a dropout rate of 0.3 is added to prevent overfitting. Because the characteristics and functions of the LSTM, BERT and CRF models are different, it is necessary to adopt a hierarchical learning rate method to learn better model parameters. The learning rate of BERT is set to $2\times 10^{-5}$, and the learning rate of LSTM is 0.001. However, due to the large transfer matrix of CRF, a relatively large learning rate of 0.01 needs to be set. The batch size is set to 16, the training epoch is 10, Adam is used as the optimizer, CRF.loss is used as the LOSS function and CRF.accuracy is selected as the evaluation function. When the validation loss is no longer reduced, it is reduced by a factor of 0.1 overall learning rate.

Hardware Environment: CPU is Intel Xeon Gold 6248R with 14 cores and 2.00GHz, memory is 72G. GPU uses A100-PCIE with a memory of 40G.