Objective-See's research, tools, and writing, are supported by the "Friends of Objective-See" such as:

...the world’s most-loved password manager.

...next-generation apple enterprise management.

...modern apple mdm & security for enterprise & education.

...the standard in apple enterprise management.

...an all-in-one package to awesomize your Mac.

Become a Friend!

Background

In March 2017, WikiLeaks began publishing thousands of files detailing the CIA’s spying operations and hacking tools. The leak, known as Vault 7, was the largest disclosure of classified information in the agency’s history. In April, Symantec publicly linked Vault 7 to an advanced threat actor named Longhorn. Kaspersky then announced it tracks the same actor as The Lamberts, and revealed the existence of an OS X implant called Green Lambert.

Kaspersky’s research showed that The Lamberts’ toolkit includes “network-driven backdoors, several generations of modular backdoors, harvesting tools, and wipers.” A timeline of actvitiy for tools used by The Lamberts shows that “Green Lambert is the oldest and longest-running in the family.” Green Lambert is described as an “active implant” and “the only one where non-Windows variants have been found.”

This blog post, along with the [Made in America](https://objectivebythesea.com/v4/talks.html#Made In America) talk at Objective By The Sea v.4.0, provides a comprehensive analysis of Green Lambert for OS X. I’ll share how I approached the research, the tools I used, the things I figured out, and the things I didn’t. I’ll also look at whether the developers followed the agency’s guidelines for development tradecraft. Some might ask why I’d look at an implant this old? Doing so helps us better understand the capabilities of its sophisticated creator, past and present. And, if we’re being honest: I could, so I did.

Victimology

We don’t know how this implant makes it into a target system; the type of system it’s used on; or the geographical location of a typical target. Symantec said that the actor has infiltrated governments, “in addition to targets in the financial, telecoms, energy, aerospace, information technology, education, and natural resources sectors.” QI-ANXIN said the actor has previously “targeted personnel and institutions in China.”

A version of Green Lambert for OS X was first uploaded to VirusTotal, from Russia, in September 2014. Kaspersky marked it as malicious in October 2016. AegisLab, a security firm based in Taiwan, followed a couple of weeks later. VirusTotal identified that the implant calls itself GrowlHelper, possibly referencing the popular Growl notification system for OS X from 2004.

Triage

Using static analysis methods, we can triage the implant without running it. For example, we can determine that GrowlHelper is a small, unsigned Mach-O executable.

$ file GrowlHelper
GrowlHelper: Mach-O executable i386

$ codesign -dvv GrowlHelper
GrowlHelper: code object is not signed at all

$ du -h GrowlHelper
208K

We can use otool -L to print a list of linked libraries. This can sometimes provide insight into the capabilities of the malware, but doesn’t appear to be particularly helpful here. Note the few dependencies in the list below.

$ otool -L GrowlHelper
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/Security.framework/Versions/A/Security 
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/usr/lib/libSystem.B.dylib
/usr/lib/libgcc_s.1.dylib

What’s more interesting is the output of strings -. This tool can also provide insight into the capabilities of the malware.

$ strings - GrowlHelper
LoginItem
LaunchAgent
/Library/LaunchDaemons

www.google.com
Error from libevent when adding event for DNS server
1.3a

_SecKeychainFindInternetPassword
_SecKeychainItemCopyAttributesAndData
_kSCPropNetProxiesHTTPProxy
_kSCPropNetProxiesProxyAutoConfigEnable
_kSCPropNetProxiesProxyAutoConfigURLString

The references to LoginItem, LaunchAgent and LaunchDaemons suggest this implant has different options for gaining persistence on a system. In other words: how the implant ensures it’s executed again if the system is rebooted. Check out this post by Phil Stokes at SentinelOne for an overview of malware persistence techniques seen in the wild.

The following three lines appear to be related to libevent, the same event notification library that is used by Tor. The open-source library is very popular now, but was perhaps less known back when this implant was created. The reference to 1.3a may shed some light on the development timeline for this implant: version 1.3a of libevent was released in February 2007.

The references to Keychain, Proxies and AutoConfig suggest this implant determines proxy settings on the target system. According to this post, kSCPropNetProxiesProxyAutoConfigEnable and kSCPropNetProxiesProxyAutoConfigURLString were added in Xcode version 2.2. This version was released in November 2005. Could be another clue about the development timeline.

OS X Version

The static analysis methods we used were helpful, but we’re going to want to see how the implant behaves on a system. For that, we’ll turn to dynamic analysis in a virtual machine. But which version of OS X does the implant need? We know that it’s a 32-bit executable, and the latest macOS is 64-bit only. We can narrow this down further by looking at symbols using nm.

$ nm GrowlHelper
         U _CFArrayAppendValue
         U _CFArrayCreateMutable
         U _CFArrayCreateMutableCopy
         U _CFArrayGetCount
         U _CFArrayGetValueAtIndex
         U _CFArrayRemoveValueAtIndex
         U _CFDictionaryCreate
         U _CFDictionaryGetValue
         U _CFGetTypeID
         U _CFNumberGetTypeID
         ...

The next step is a bit tedious, but does provide helpful information. To better understand what these symbols represent, we can look them up in Apple’s Developer Documentation. Not only will we be able to learn how and where a given symbol is used, but we can also see when it was deprecated. With that information, we can determine which version of OS X the implant will run on.

• FSGetCatalogInfo is available in macOS 10.0 - 10.8
• FSPathMakeRef is available in macOS 10.0 - 10.8
• FSSetCatalogInfo is available in macOS 10.0 - 10.8
• SecKeychainSearchCopyNext is available in macOS 10.0 - 10.7
• SecKeychainSearchCreateFromAttributes is available in macOS 10.0 - 10.7
• SecKeychainSetUserInteractionAllowed is available in macOS 10.2 - 12.0

This means that the implant will run on (at least) 10.7: OS X Lion.

Note: I confirmed the implant runs on 10.8. It probably runs on any OS X that supports 32-bit executables.

Development / Use Timeline

Let’s look at a potential timeline for the development and use of this implant.

Growl was released in 2004 and retired in 2020. Xcode version 2.2 was released in November 2005, while libevent 1.3a was released in February 2007. OS X 10.7 was released in 2011, and 10.8 in 2012. The implant first appeared on VirusTotal in late 2014. Court records show Vault 7 was stolen sometime in early 2016 and published by WikiLeaks a year later. Based on these datapoints, it’s likely the implant was created and used between 2007 and (at least) 2013.

Setting Up a Virtual Machine

As of June 2021, OS X 10.7 is available for free from Apple. You can also do what I did: buy an old MacBook on eBay for $95.

You may have to unpack the .dmg you get from Apple to get a file that’ll work with your virtual machine software. If so, try:

$ hdiutil attach InstallMacOSX.dmg

Click on Install Mac OS X on the Desktop and use The Unarchiver (or another tool) to extract InstallMacOSX.pkg to a temporary folder. Go into this folder, click on the new copy of InstallMacOSX.pkg and select Show Package Contents. Copy InstallESD.dmg to where you keep your virtual machine images, and use that instead.

We’re going to use lldb, the default debugger, to execute the implant, modify registers, and examine memory contents. OS X 10.7 doesn’t include Xcode by default, but a quick Google search suggests we need version 4.6.3 and can get it from Apple’s Developer Downloads page. After installing Xcode and confirming that lldb is working, we isolate the machine and create a clean snapshot.

Persistence

Phil Stokes at SentinelOne wrote that “the most common way malware persists on macOS is via a LaunchAgent. Each user on a Mac can have a LaunchAgents folder in their own Library folder to specify code that should be run every time that user logs in.” We can confirm this is the case with Green Lambert by running the implant, then checking the user’s LaunchAgents folder.

$ ls ~/Library/LaunchAgents
com.apple.GrowlHelper.plist

Once installed, it’ll delete the original GrowlHelper file from the system. This is where our VM snapshot comes in handy.

From Phil’s post, we know that “LaunchAgents take the form of property list files, which can either specify a file to execute or can contain their own commands to execute directly.” We can confirm this by looking at com.apple.GrowlHelper.plist.

$ cat ~/Library/LaunchAgents/com.apple.GrowlHelper.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>com.apple.GrowlHelper</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Users/user/Library/Caches/com.apple.Growl.GrowlHelper/5d0d/GrowlHelper</string>
		<string>-f</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>OnDemand</key>
	<false/>
</dict>
</plist>

The ProgramArguments tell us where GrowlHelper is installed and that it takes at least one command line argument (-f). The RunAtLoad key confirms the implant will run every time the user logs in. To get an overview of the installation process, we can monitor file system activity for GrowlHelper events.

$ sudo fs_usage -w -f filesys > filesys.out
$ sudo grep GrowlHelper filesys.out
execve      /Users/user/GrowlHelper     0.015273 W bash.2848
execve      /Users/user/GrowlHelper     0.000383   GrowlHelper.2851

open        /Users/user/.profile        0.000018   GrowlHelper.2851
open        /Users/user/.bash_profile   0.000015   GrowlHelper.2851
open        /Users/user/.bash_login     0.000015   GrowlHelper.2851
open        /Users/user/.bashrc         0.000014   GrowlHelper.2851
open        /Users/user/.cshrc          0.000014   GrowlHelper.2851
open        /Users/user/.login          0.000014   GrowlHelper.2851
open        /Users/user/.tcshrc         0.000014   GrowlHelper.2851
open        /Users/user/.xsession       0.000007   GrowlHelper.2851
open        /Users/user/.xinitrc        0.000006   GrowlHelper.2851

We see that GrowlHelper has a handful of options for maintaining persistence, in case the LaunchAgent is removed. In one case, the implant uses a .profile file to ensure it’s launched whenever the user opens the Terminal. (Path to GrowlHelper was lightly edited due to space constraints.)

$ cat ~/.profile
GrowlHelper=`/path/to/com.apple.Growl.GrowlHelper/5d0d/GrowlHelper 2>&1` # Automatic GrowlHelper. Do not remove

Self-Update

We can compare how GrowlHelper behaves when the system is offline v. online. Here are the files it created in an isolated VM.

$ file /Users/offline/Library/Caches/com.apple.Growl.GrowlHelper/5d0d/*
GrowlHelper:	Mach-O executable i386
db:						Berkeley DB 1.85 (Hash, version 2, native byte-order)
fifo:					socket
queue:				directory

And here are the files GrowlHelper created on that old MacBook I got from eBay.

$ file /Users/online/Library/Caches/com.apple.Growl.GrowlHelper/5d0d/*
GrowlHelper:						Mach-O executable i386 
Software Update Check: 	Mach-O executable i386
db:											Berkeley DB 1.85 (Hash, version 2, native byte-order)
fifo:										socket
queue:									directory

It looks like GrowlHelper creates an executable named Software Update Check when it thinks it’s online. I was pretty excited when I first found this, but quickly realized it just drops a copy of itself with a different name.

3fcdbd3c5fa34fb8e8d58038fa1d1f13d37e8a4b GrowlHelper
3fcdbd3c5fa34fb8e8d58038fa1d1f13d37e8a4b Software Update Check

It’s possible that Software Update Check is used to update GrowlHelper.

Command Line Arguments

We know where GrowlHelper is installed and that it takes at least one command line argument (-f). With this information, we can identify other arguments by manually looping through options a - z and A - Z on the command line. The output below is the result of doing this try/fail experiment in a VM.

Hopper Disassembler is a tool that helps you disassemble, decompile and debug malware. There’s a free version, and you can get a personal license for $99. Using Hopper, we can confirm the arguments we found by looking for argc, argv, and getopt.

By using Hopper’s pseudo-code mode, we can see the full set of possible command line arguments.

Entry Points

When you open GrowlHelper in Hopper, you’ll see that it has multiple entry points: EntryPoint_1 through EntryPoint_21. These entry points are called when GrowlHelper starts executing, before the main entry point at 0x2cd8. GrowlHelper will use these entry points to initialize certain functionality. QI-ANXIN detailed these entry points in this post / this screenshot below.

It appears GrowlHelper has a preflight checklist of sorts: it initializes functionality, figures out what it needs, deletes the rest.

$ sudo grep GrowlHelper filesys.out
mkdir       /Users/user/.DS_Info       											0.000083   GrowlHelper.2851
mkdir				/Users/user/.DS_Info/5d0d												0.000044   GrowlHelper.2851
mkdir				/Users/User/Library/Caches/com.apple.advanced		0.000066   GrowlHelper.2851

rmdir				/Users/user/.DS_Info/5d0d												0.000109 W GrowlHelper.2851
rmdir				/Users/user/.DS_Info 														0.000240 W GrowlHelper.2851
rmdir				/Users/User/Library/Caches/com.apple.advanced		0.000068   GrowlHelper.2851

Decrypting a String

Given the author, it’s no surprise that most strings in this implant are encrypted. The implant appears to handle encrypted strings in a bunch of different ways, which makes it challenging to automate decryption. Hopper has done some of the analysis work for us, allowing us to at least manually decrypt strings with lldb. Here’s one example.

In the screenshot above, we have:

• The address for the program counter / call to the decryption routine (0x1549b)
• The values for ecx (0x01), edx (0x31e80), eax (0x2d487)
• The address after the decryption routine, which we’ll use as a breakpoint for lldb (0x154a0)

We load the implant into the debugger using lldb GrowlHelper, and decrypt the string:

Decrypting More Strings

Manually decrypting strings turned into a rabbit hole for me, but that’s OK. I’m sure there are ways to do this faster, but I have to admit I really enjoyed the process of learning to do this manually. Here are the strings I’ve decrypted so far, minus duplicates.

Listening Post

One of the decrypted strings is No LP configured. LP likely stands for Listening Post, a military term used in the context of signals intelligence and reconnaissance. Where other types of malware would likely use the terms C2 or Command & Control, the CIA and the NSA use LP. One Vault 7 document is titled Listening Post (LP) Creation, and another details requirements for a Listening Post.

Configuration Files

Some of the decrypted strings refer to .html, .php, and .txt files, but I’m unable to access them. But we know that Kaspersky found “a hostname and an IP address” hardcoded in the implant. And researchers at QI-ANXIN determined the implant talks to the Listening Post through login.php and getconf.php, and downloads follow-up code through getfile.php.

Configuration? Survey?

If you dig around in Hopper and use pseudo-code mode from time to time, you’ll likely find some interesting bits of information. When I stumbled upon the string Version=1.2.0, I decided to see where else = was referenced. To do that, highlight 0x132b8 as shown below and hit x.

The list of references looks like this, with the current one selected.

We can then go through all these references, decrypt the strings, and get an output that looks like this.

uname=
Time=%Y\%m\%d %H:%M:%S Z
Uptime=
Version=1.2.0
PID=

The output lists information from the target system (e.g. uname) and information from the implant (e.g. Version). This could be a combination of a configuration file and system survey.

Network Traffic

We can monitor the network traffic on our OS X 10.7 system using tcpdump and then view the output in Wireshark.

This gives us the hardcoded hostname notify[.]growlupdate[.]com. Very clever given the name of the executable.

And the hardcoded IP address: 94[.]242[.]252[.]68.

Hostname

Google and the Wayback Machine don’t have any results for the domain name. If we look it up on VirusTotal, we see that it was first submitted in October 2016. But if we look up the domain on crt.sh, we see that an SSL certificate was created on October 29, 2013. The domain may have been purchased earlier, but this at least suggests the domain was active in late 2013. This matches the timeline we created earlier, as well as Kaspersky’s timeline of activity by The Lamberts.

Note: Kaspersky sinkholed the domain to 95[.]211[.]172[.]143 between October 1, 2016 and October 2, 2017.

Development Tradecraft DOs and DON’Ts

As part of Vault 7, WikiLeaks published 52 revisions of the CIA’s development tradecraft guidelines. I mapped the revisions in a public spreadsheet to see how the guidance changed over time. Studying the development choices made by sophisticated actors may help us track them over time. For example, when Kaspersky identified a code overlap between Sunburst and Kazuar, it was because of “unusual, shared features” such as the UID generation algorithm, the sleeping algorithm, and use of the FNV-1a hash.

As Costin Raiu of Kaspersky pointed out on Twitter, “C2 jitter, secure erase / uninstall, SSL/TLS+extra crypto, size below 150K, encrypt logs and local collection, decrypt strings on the fly in mem… simply following these guidelines immediately makes the malware (“tools”) more interesting and, recognizable by a skilled analyst.” While most of these are true here as well, there are a few things that stand out.

• File size is a bit over the “ideal binary file size” for a fully featured tool (208K v. 150K)

• The references to Listening Post / LP may be CIA and USG specific terminology

• Use of English abbreviations for days of the week (mtwhfsu / MTWHFSU)

• Use of the libevent library back when it was perhaps less well-known

Conclusion

I’ve really enjoyed working on this project and certainly learned a lot along the way. I’m confident there’s more to find here, and I’d love to collaborate with anyone interested in taking a closer look. As for The Lamberts? Malware from this actor keeps turning up, along with new insights. In fact, Kaspersky’s APT trends report for Q1 2021 mentions Purple Lambert, a malware “capable of providing an attacker with basic information about the infected system and executing a received payload.”

Indicators of Compromise

• notify[.]growlupdate[.]com
• 94[.]242[.]252[.]68
• 3fcdbd3c5fa34fb8e8d58038fa1d1f13d37e8a4b

References

Patrick’s free and open-source book on Mac malware analysis was incredibly helpful during this project. If you haven’t already done so, I highly recommend checking out The Art of Mac Malware.