This post is also available in: 日本語 (Japanese)

Executive Summary

In January 2021, Unit 42 researchers detected a new malware campaign targeting Kubernetes clusters. The attackers gained initial access via a misconfigured kubelet that allowed anonymous access. Once getting a foothold into a Kubernetes cluster, the malware attempted to spread over as many containers as possible and eventually launched cryptojacking operations. Based on the tactics, techniques and procedures (TTP) that the attackers used, we believe this is a new campaign from TeamTNT. We refer to this new malware as Hildegard, the username of the tmate account that the malware used.

TeamTNT is known for exploiting unsecured Docker daemons and deploying malicious container images, as documented in previous research (Cetus, Black-T and TeamTNT DDoS). However, this is the first time we found TeamTNT targeting Kubernetes environments. In addition to the same tools and domains identified in TeamTNT's previous campaigns, this new malware carries multiple new capabilities that make it more stealthy and persistent. In particular, we found that TeamTNT’s Hildegard malware:

• Uses two ways to establish command and control (C2) connections: a tmate reverse shell and an Internet Relay Chat (IRC) channel.
• Uses a known Linux process name (bioset) to disguise the malicious process.
• Uses a library injection technique based on LD_PRELOAD to hide the malicious processes.
• Encrypts the malicious payload inside a binary to make automated static analysis more difficult.

We believe that this new malware campaign is still under development due to its seemingly incomplete codebase and infrastructure. At the time of writing, most of Hildegard's infrastructure has been online for only a month. The C2 domain borg[.]wtf was registered on Dec. 24, 2020, the IRC server went online on Jan. 9, 2021, and some malicious scripts have been updated frequently. The malware campaign has ~25.05 KH/s hashing power, and there is 11 XMR (~$1,500) in the wallet.

There has not been any activity since our initial detection, which indicates the threat campaign may still be in the reconnaissance and weaponization stage. However, knowing this malware's capabilities and target environments, we have good reason to believe that the group will soon launch a larger-scale attack. The malware can leverage the abundant computing resources in Kubernetes environments for cryptojacking and potentially exfiltrate sensitive data from tens to thousands of applications running in the clusters.

Palo Alto Networks customers running Prisma Cloud are protected from this threat by the Runtime Protection feature, Cryptominer Detection feature and the Prisma Cloud Compute Kubernetes Compliance Protection, which alerts on an insufficient Kubernetes configuration and provides secure alternatives.

Tactics, Techniques and Procedures

Figure 1 illustrates how the attacker entered, moved laterally and eventually performed cryptojacking in multiple containers.

1. The attacker started by exploiting an unsecured Kubelet on the internet and searched for containers running inside the Kubernetes nodes. After finding container 1 in Node A, the attacker attempted to perform remote code execution (RCE) in container 1.
2. The attacker downloaded tmate and issued a command to run it and establish a reverse shell to tmate.io from container 1. The attacker then continued the attack with this tmate session.
3. From container 1, the attacker used masscan to scan Kubernetes's internal network and found unsecured Kubelets in Node B and Node C. The attacker then attempted to deploy a malicious crypto mining script (xmr.sh) to containers managed by these Kubelets (containers 2-7).
4. Containers that ran xmr.sh started an xmrig process and established an IRC channel back to the IRC C2.
5. The attacker could also create another tmate session from one of the containers (container 4). With the reverse shell, the attacker could perform more manual reconnaissance and operations.

The indicators of compromise (IOCs) found in each container are listed below. These files are either shell script or Executable Linkable Format (ELF). The IOC section at the end of the blog contains the hash and details of each file.

• Container 1: TDGG was dropped and executed via Kubelet. TDGG then subsequently downloaded and executed tt.sh, api.key and tmate. The attacker used the established tmate connection to drop and run sGAU.sh, kshell, install_monerod.bash, setup_moneroocean_miner.sh and xmrig (MoneroOcean).
• Container 2-7: xmr.sh was dropped and executed via Kubelet.
• Container 4: The attacker also established a tmate session in this container. The attacker then dropped and executed pei.sh, pei64/32, xmr3.assi, aws2.sh, t.sh, tmate,x86_64.so, xmrig and xmrig.so.

Figure 2 maps the malware campaign's TTP to MITRE ATT&CK tactics. The following sections will detail the techniques used in each stage.

Initial Access

kubelet is an agent running on each Kubernetes node. It takes RESTful requests from various components (mainly kube-apiserver) and performs pod-level operations. Depending on the configuration, kubelet may or may not accept unauthenticated requests. Standard Kubernetes deployments come with anonymous access to kubelet by default. However, most managed Kubernetes services such as Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE) and Kubernetes operations (Kops) all enforce proper authentication by default.

We discovered that TeamTNT gained initial access with the Hildegard malware by executing commands on kubelets that allow anonymous access. This was achieved by accessing the kubelet’s run command API and executing commands on running containers.

Execution

Hildegard uses kubelet’s API to execute commands inside containers. The initial commands create a tmate reverse shell that allows the attacker to carry out the subsequent operation. Unlike the techniques that TeamTNT used in the past, this malware campaign did not pull or run any new container image.

Privilege Escalation

Although Unit 42 researchers have not observed an attempt to perform privilege escalation, the malware dropped two adversarial tools, Peirates and BOtB, which are capable of breaking out of containers via known vulnerabilities or accessing cloud resources via exposed cloud credentials.

Container Breakout

BOtB can perform a container breakout using a known vulnerability such as CVE-2019-5736. It can also escape from privileged containers that have enabled CAPS and SYSCALLS.

Access to Cloud Resources

Peirates can gather multiple infrastructures and cloud credentials. It looks for identity and access management (IAM) credentials from cloud metadata services and service account tokens from the Kubernetes clusters. With the identified credentials, it then further attempts to move laterally or gain control of the cluster. While we observed Peirates in use, the container it was executed in had no credentials.

Defense Evasion

Library Injection

Hildegard uses LD_PRELOAD to hide the malicious process launched inside the containers. The malware modified the /etc/ld.so.preload file to intercept shared libraries’ imported functions. In particular, the malware overwrites two functions: readdir() and readdir64(), which are responsible for returning the directory entries in the file system. The overwritten functions filter out queries made to directory entries under /proc. The functions then drop queries with keywords such as tmate, xmrig and ziggy. This way, when applications try to identify the running processes (by reading files under /proc) in the containers, tmate, xmrig and ziggy will not be found. Linux tools such as ps, top and many other container monitoring tools will be blinded from these malicious processes.

Encrypted ELF Binary

Hildegard deploys an IRC agent built from the open-source project ziggystartux. To avoid being detected by automated static analysis tools, the ziggystartux ELF is encrypted and packed in another binary (ziggy). When the binary is executed, the ziggystartux ELF is decrypted by a hardcoded Advanced Encryption Standard (AES) key and executed in memory.

Disguised Process Name

The malware names the IRC process “bioset”, which is the name of a well-known Linux kernel process bioset. If one is only looking at the names of the running processes on a host, one can easily overlook this disguised process.

DNS Monitoring Bypass
The malware modifies the system DNS resolvers and uses Google’s public DNS servers to avoid being detected by DNS monitoring tools.

Delete Files and Clear Shell History

All the scripts are deleted immediately after being executed. TeamTNT also uses the “history -c” command to clear the shell log in every script.

Credential Access

Hildegard searches for credential files on the host, as well as queries metadata for cloud-specific credentials. The identified credentials are sent back to the C2.

The searched credentials include:

• Cloud access keys.
• Cloud access tokens.
• SSH keys.
• Docker credentials.
• Kubernetes service tokens.

The metadata servers searched:

• 169.254.169.254
• 169.254.170.2

Discovery

Hildegard performs several reconnaissance operations to explore the environment.

• It gathers and sends back the host’s OS, CPU and memory information.
• It uses masscan to search for kubelets in Kubernetes’ internal network.
• It uses kubelet’s API to search for running containers in a particular node.

Lateral Movement

Hildegard mainly uses the unsecured kubelet to move laterally inside a Kubernetes cluster. During the discovery stage, the malware finds the exploitable kubelets and the containers these kubelets manage. The malware then creates C2 channels (tmate or IRC) and deploys malicious crypto miners in these containers. Although not observed by Unit 42 researchers, the attacker may also move laterally with the stolen credentials.

Command and Control

Once gaining the initial foothold into a container, Hildegard establishes either a tmate session or an IRC channel back to the C2. It is unclear how TeamTNT chooses and tasks between these two C2 channels, as both can serve the same purpose. At the time of writing, tmate sessions are the only way the attacker interacts with the compromised containers. Unit 42 researchers have not observed any commands in the IRC channel. However, the IRC server's metadata indicates that the server was deployed on Jan. 9, 2021, and there are around 220 clients currently connected to the server.

Impact

The most significant impact of the malware is resource hijacking and denial of service (DoS). The cryptojacking operation can quickly drain the entire system’s resources and disrupt every application in the cluster. The xmrig mining process joins the supportxmr mining pool using the wallet address 428uyvSqdpVZL7HHgpj2T5SpasCcoHZNTTzE3Lz2H5ZkiMzqayy19sYDcBGDCjoWbTfLBnc3tc9rG4Y8gXQ8fJiP5tqeBda. At the time of writing, the malware campaign has ~25.05 KH/s hashing power and there is 11 XMR (~$1,500) in the wallet.

Conclusion

Unlike a Docker engine that runs on a single host, a Kubernetes cluster typically contains more than one host and every host can run multiple containers. Given the abundant resources in a Kubernetes infrastructure, a hijacked Kubernetes cluster can be more profitable than a hijacked Docker host. This new TeamTNT malware campaign is one of the most complicated attacks targeting Kubernetes. This is also the most feature-rich malware we have seen from TeamTNT so far. In particular, the threat actor has developed more sophisticated tactics for initial access, execution, defense evasion and C2. These efforts make the malware more stealthy and persistent. Although the malware is still under development and the campaign is not yet widely spread, we believe the attacker will soon mature the tools and start a large-scale deployment.

Palo Alto Networks customers running Prisma Cloud are protected from this threat by the Runtime Protection features, Cryptominer Detection and by the Prisma Cloud Compute Kubernetes Compliance Protection, which alerts on an insufficient Kubernetes configuration and provides secure alternatives.

Indicators of Compromise

Domains/IPs:

Files: