Malicious Life Podcast: Moonlight Maze

Transcript

Archaeologists don’t have it easy when trying to piece together a reliable picture of the past from tiny fragments of pottery and the occasional inscription. One would assume that future Archaeologists would probably have an easier task when it comes to our present lives: after all, we’re leaving behind an enormous trail of digital information detailing everything we do on a minute-by-minute basis, including pictures, videos, texts, tweets, and more. It seems that if anything, they’ll probably have the opposite problem, trying to make sense of this deluge of information.

However, the reality is quite different. While inscriptions written on stone and clay might survive for thousands of years, digital information is far more ephemeral. CDs, for example, may become unreadable in as little as 15-20 years. And even if a storage media manages to survive the ages, reading its content often requires reconstructing a suitable hardware and software environment, especially if the recovered information is a piece of software. Some scholars believe that our present might be just as inscrutable to future historians as our past is to us. 

And when it comes to cyber security, future archaeologists who might wish to reconstruct our present might face an even bigger challenge, because on top of all the difficulties I listed, cyber security has one more problematic characteristic to consider: secrecy. Criminals, naturally, wish to avoid detection by law enforcement in the present, and so will make the lives of future researchers much more difficult. This is doubly true for nation-state threat actors, for whom staying under the radar is often an absolute necessity. 

So, how difficult will it be to reconstruct an accurate picture of a historic APT attack? With the internet already more than thirty years old at this point, we can already get a glimpse of the answer to that question, as the following story will illustrate. 

1996. Colorado. 

In September 1996, a sysadmin of a university lab in Colorado noticed some unusual activity in their network. What kind of ‘strange activity’? We don’t know: not a lot of information survives from these early days of internet prehistory. But whatever activity the admin noticed, it was unusual enough that he decided to dig deeper – which is how he found the malware that was lurking in one of the lab’s machines. The admin contacted the Navy, who was funding the lab’s activity, and notified it of the breach. But since cybersecurity was practically non-existent at the time, whoever received the report on the Navy side probably had little idea what to do with it, and so nothing happened. 

Some three months later, another malware – similar to the first or different, we have no idea – was discovered, this time inside the Navy’s own network, infecting several systems in various bases across the country. Knowing what we know now, it’s likely that the hacker who penetrated the Navy’s network did so using the internet connection between the lab and the Colorado lab – but no one in the military was able to connect these dots at the time. Had they done so, they might have been able to thwart what was later discovered to be the very first known cyber espionage campaign in history. 

2016. London.

Thomas Rid is currently the Director of the Alperovitch Institute for Cybersecurity Studies. In 2016, he was professor of Security Studies at the Department of War Studies at London’s King’s College. Cyber warfare was always one of his interests, but although he was considered “one of Britain’s leading authorities on […] cyber-warfare”, as The Economist put it – Rid was uncomfortable with that title, because he felt his research into the subject wasn’t deep enough. 

“I was always a bit annoyed because I get introduced as our cyber security or just cyber expert quite often. So I thought I probably can’t kill it, cyber, as a term, so I might as well just add some historical depth to it.”

That’s how the idea for his eight book, ‘Rise of the Machines, A Cybernetic History’ – was born. 

Being a book about the history of cyber-warfare, there was one incident Rid was certain should be included in it: Moonlight Maze. That was the name given to a massive data breach that affected literally thousands of American government agencies, military bases, universities and defense contractors, and took place between 1996 and 1999 – the same incident whose origins I described earlier. 

But apart from Moonlight Maze’s importance as the first cyber-espionage campaign, Rid had another reason to be interested in this historical episode.

In 2008, a worm called Agent.BTZ infiltrated a Dept. of Defense facility in the Middle East. It spread through the building by silently replicating itself onto USB flash drives plugged into infected machines, and from there spread to multiple networks belonging to the U.S. Strategic Command, including highly classified networks. One Pentagon official called it “the worst breach of U.S. military computers in history,” and it took the DoD 14 months to scrub the worm from all of its systems. 

The 2008 attack was the work of Turla, a Russian APT, also known as Venomous Bear and Waterbug. Turla is considered by many to be one of the top five threat actors in the world: an innovative and dangerous hacking group, known for utilizing advanced attack tools and unorthodox tactics. For example, using hijacked satellite-based internet connections for its C&C infrastructure, and taking over the infrastructure of other APT groups to intercept the data that they steal. 

When Thomas Rid was contemplating the content for his planned book, he came across claims from three different investigators from three different countries that Turla was responsible for the Moonlight Maze campaign.If these claims could be proven, that meant that the Moonlight Maze hackers were operating continuously for more than 18 years: an amazingly long time to survive in the cyber world, where hacking groups and APTs usually disband or disappear within only a few short years. In fact, the only other hacking group known to have existed for such a long time is the Equation Group, widely suspected of being part of the NSA, that’s been active since at least 2001 and perhaps even since the early 1990s. 

If Moonlight Maze’s hackers were still active that would make them a sort of ‘living fossil’ of cyber security history, and Rid saw it as a rare and unique opportunity to examine the long-term evolution of a nation-state threat actor: something no one has ever done before. 

Rid, then, decided to take on the role of a “Cyber Archaeologist”, and investigate the Moonlight Maze mystery, and its ties to Turla. 

1998. Ohio.

In January of 1998, an admin for ATI-Corp, a specialist materials company in Charleston, South Carolina, discovered that someone was connecting from their network to that of Wright Patterson Air Force Base in Ohio. That, by itself, wasn’t unusual – ATI-Corp was working closely with the Air Force – except for the fact that the connection was made at 3am on a Sunday. 

The admin reached out to the account’s owner, who confirmed that it wasn’t him. The admin then notified the Air Force computer experts, who began looking into the matter: it didn’t take them long to discover the same suspicious activity that their Navy counterparts observed two years earlier. This time, however, they came upon another unusual find: one of the connections to their network came from a computer located in Russia. 

The FBI was called in, and this time the investigators did manage to connect the dots: someone was hacking into military bases and government organizations all over the country. In addition to unearthing the attacks on Navy systems in 1996, the logs revealed no less than 324 hacking attempts against the Dept. of Energy, as well as attacks against similar targets of several US allies around the world. The investigation was dubbed “Moonlight Maze”.

The analysts who tracked the hackers’ activity inside the military’s networks discovered two interesting facts. The first was that the attackers were logging into the classified networks mainly via universities, such as the University of South Carolina and the University of Cincinnati, using credentials stolen from researchers from these institutions. The FBI interviewed several individuals belonging to these universities, but soon came to the conclusion that the hackers were only using the university networks as a stepping stone for infiltrating into the military’s networks. In hindsight, this was perfectly logical: many universities were working closely with the military on various classified projects – and thus plenty of researchers were allowed to log on to the classified networks – yet university networks were usually poorly protected, as the concept of regular updating and patching security bugs was still unfamiliar to most administrators in these early days. 

The second interesting find was that the hackers were using a compromised system belonging to a small organization called “The Institute of Personnel and Development” in London, a server called “HR Test”, as an entry-point into the universities’ networks, as well as for storing some of their software tools and the documents they exfiltrated. The investigators reached out to the server’s administrator, and with the aid of the local police obtained the server’s logs. 

This turned out to be a gold mine for the researchers, and allowed them a much wider and deeper visibility into the intruders actions than they previously had. And the picture these logs painted was very, very troubling – if not outright scary. It turned out that whoever was behind the Moonlight Maze campaign was targeting no less than 1600 US organizations and agencies, including the Pentagon, NASA, many major academic institutions and defense contractors. 

With so many targets, it was difficult for the investigators to point a finger at the attackers’ goals. What were they after? Were they trying to steal classified information, or were they planning to damage these computer systems – for example, planting ‘logic bombs’ that would go off sometime in the future and disable these systems? 

2016. London.

One of Thomas Rid’s first actions was to contact the FBI and ask for the information collected during the Moonlight Maze investigation, as part of the Freedom Of Information Act. He was disappointed to learn that almost all of the inquiry’s data was destroyed in 2008, as part of the standard procedure of removing old and unnecessary evidence. 

Rid’s only hope, then, was to get hold of the people who were part of that investigation twenty years ago and interview them. This turned out to be just as difficult, since most of the names that appeared in the few documents that did survive from the 1990s had been redacted – that is, erased – by the FBI as part of the documents’ declassification process. Lucky for Rid, the FBI didn’t do a perfect job: in one of the documents he noticed a name that wasn’t redacted: David Hedges – the administrator of the HR Test server that served as a hub for the hackers. It took Rid almost a whole year to track down Hedges, but when he did – we can only imagine his delight as the now-retired sysadmin chuckled on the phone. “I hear you’re looking for HR Test. Well, it’s sitting right here, and it’s still working.”

For the second time in its history, just as it did for the FBI almost twenty years earlier, HR Test turned out to be a gold mine for Thomas Rid. It turned out that when Scotland Yard got involved in the investigation some 18 years earlier, its people installed eavesdropping software to record everything the hackers were doing with the machine and archive each and every binary file passing through the server, for a period of almost six months.
Not only that: It turned out that the attackers themselves made a huge mistake that played into Rid’s hands. One of their standard methods of operation was to install network sniffers on victim machines, to allow them to gain visibility into the targeted networks. They then used the same machines to connect to other machines – with the sniffers still recording the activity – and by doing so, essentially created complete logs of everything they themselves were doing. The attackers then exfiltrated the sniffer files along with the data they stole, and stored them in HR Test – for Rid to find in 2016.  

The sheer magnitude of information stored in these twenty-years old logs meant that Rid couldn’t hope to analyze them all by himself. He formed a group with two analysts from Kaspersky – Juan Andres Guerro-Saade and Costin Raiu – and Daniel Moore, a colleague from King’s College, and together they spent almost nine months working on the logs. 

1999. The Pentagon

In a sense, the Americans were lucky. Up until 1997, almost no one in the US military understood the magnitude of the threat posed to US security from cyber attacks, and so little was done to counter that threat. This changed in 1997, after a now almost mythical exercise known as Eligible Receiver 97. We told the story of Eligible Receiver in depth in episode 15 of Malicious Life, but the gist of it was that within only four short days, the NSA’s red team was able to take down almost any computer network it wanted to – plus, they detected signs of foreign spies who already penetrated these sensitive networks. ER97 was proof that America was vulnerable to cyber attacks, and that something had to be done about it. 

One outcome of the exercise was the decision to install intrusion detection systems – IDSs – on many Department of Defense computers. These devices should have been capable of detecting the tell-tale signs of Moonlight Maze’s intrusions, but as often happens in large organizations, many of the people who had an IDS installed in their computer had absolutely no idea what it was or what to do with it. This became apparent in one unforgettable meeting held by Deputy Secretary of Defense John Hamre, in which an Army one-star general grumbled about “these IDS things” installed on his machine that did nothing but spew incomprehensible alerts day in and day out. The others at the table immediately realized that the general’s computer was actually being hacked for months, maybe even years. 

This realization prompted Hamre to appoint Brigadier General John “Soup” Campbell, who had been the Pentagon’s point man on Eligible Receiver, to erect a new office called the Joint Task Force for Computer Network Defense – or JTF-CND, for short. As luck would have it, the JTF was established just a few short months before Moonlight Maze was exposed, and so this investigation became the first major incident the newly established organization had to deal with. 

The JTF collaborated with the FBI, and the information collected from the IDS devices installed on many DoD machines, together with the HR Test logs, helped the investigators construct a better overall image of the attack – and finally understand what the Moonlight Maze hackers were after. A former navy intelligence officer who used to track KGB operations during the Cold War, recalled that the Soviet spy agency often sent Russian scientists to international conferences to collect papers on topics the USSR was interested in. He analyzed the logs of Moonlight Maze, and then compared them to databases of recent scientific conferences: the list of topics that interested the hacker and the conferences’ subject matters matched perfectly. 

This discovery pointed a clear finger at the Russian government as the one orchestrating the espionage campaign. There was even more evidence hidden in the logs: the hackers’ work hours aligned with Russia’s time zone, and they even took a few days off during the Russian Orthodox Christmas, which takes place roughly two weeks after the same holiday in the US. 

But as Soup Campbell was very much aware – this was a very serious allegation, since post-communist Russia under then-president Boris Yeltzin was considered an ally of the United States. Was it possible that some other country – perhaps Iran or some other Middle Eastern entity – was behind the attack, and was simply using Russian servers to mask their true identity? 

Someone in the FBI came up with an idea. It was inspired by Cliff Stoll, the Berkeley computer systems administrator who in 1986 set up a honeypot – a directory full of fake “classified” military documents – as a trap for an East German hacker who infiltrated his network, a story we covered in episodes 210 & 211 of the podcast. The Moonlight Maze investigators followed Stoll’s example, and created a fake website that supposedly held information about a secretive stealth aircraft program: once the hackers, who naturally couldn’t resist the lure of the fake site, visited it – the team tracked their IP back to the Russian Academy of Sciences in Moscow, a government-supported body that has known links with the Russian military.

The amount of information the attackers managed to siphon away during the three years when no one was looking was staggering: approximately 5.5 gigabytes of data, the equivalent of nearly three million sheets of paper. These included classified naval code, information about missile-guidance systems, military maps, U.S. troop configurations and more. Although there still wasn’t definitive proof of the Russian government’s involvement in the campaign, the circumstantial evidence was enough for Soup Campbell to decide it was time to set the alarm bells ringing: when John Hamre, the Deputy Secretary of Defense, briefed his colleagues at the Congress, he told them “We’re in the middle of a cyberwar.”

As often happens when politicians get involved, the briefing was promptly leaked to the press, and Hamre’s warning was the actual headline of a story published in Newsweek a few months later. When the Russian hackers became aware that the Americans had discovered their operation – the hacking stopped, and Moonlight Maze came to an end. 

Or so it seemed.

2016. London

Thomas Rid and his colleagues from Kaspersky and King’s College, who suspected that the 1990s Moonlight Maze attack was somehow connected to present-day Turla, analyzed the logs and binary files they dug up from the old HR Test server. The code they uncovered revealed that the hackers behind Moonlight Maze – the code included the handles Iron, Max and Rinat4 – borrowed much of their exploits from public sources such as forums and security mailing lists, mostly from proof-of-concept code developed by system administrators who wished to inform the community about such dangers. 

The investigation also uncovered another interesting find: the Moonlight Maze attack targeted Solaris and Unix systems, and not Windows-based systems as the team initially assumed. It was this fact that provided Rid with the connection he was looking for between the past – and present.

In February 2014, G-Data – a German security vendor – released a report about a spyware rootkit it named ‘Uroburos’, that was part of an large espionage campaign against government, military and research institutions in 45 countries. Uroburos was a highly sophisticated rootkit which managed to stay undetected for at least three years: so sophisticated, in fact, that G-Data’s researchers suspected it was a state sponsored effort. Further research from Kaspersky validated G-Data’s finding, and concluded that Uroburos was the work of Turla.

Uroburos was targeting Windows systems – but a few months after its initial discovery in February of 2014, Kaspersky announced that they had detected a rare variant of the spyware which targeted Linux computers, which they dubbed “Penquin Turla”. That variant, Kaspersky’s researchers noted, included code which was essentially inactive, and appeared to be leftovers from older versions of the malware. 

When Rid and his team got word of the finding, they pounced on the newly discovered samples and analyzed Penquin Turla’s code. They found that portions of the malware were compiled for Linux Kernel versions 2.2.0 and 2.2.5, which were released in 1999. The code also included an exploit known as LOKI2: a program that exploited the Internet Control Message Protocol (ICMP, for short) – a mechanism originally intended for conveying error messages and similar information – to covertly tunnel arbitrary information from compromised machines. The LOKI2 exploit was published in Issue 51 of the Phrack – a hacker online magazine – in September of 1997, and apparently became a favorite tool for Moonlight Maze’s operators. Crucially, this same ancient exploit was found in the Uroburos/Agent.BTZ code of 2008 to 2014, thus establishing the link between Moonlight Maze and Turla. 

It was a unique find that allowed Rid to trace Turla’s development from their earliest roots as a group of what were basically sophisticated script-kiddies who utilized publicly available tools and exploits – to a bona-fide APT group. Analysis of the old and new malwares showed how the hackers slowly modified the original LOKI2 code over the years, adding custom functionality and features, until it became the formidable attack tool that was Uroburos. It was a never before seen peek into the evolution of a threat actor over a period of 18 years. 

There was just one more piece missing from the puzzle. The circumstantial connection between Moonlight Maze and Turla was pretty strong: for example, Turla was the only threat actor still using the 1997 LOKI2 exploit in 2016. But there was still a 9 year gap between when the Moonlight Maze halted their operation in 1999, after being exposed, and when Turla attacked the U.S. Strategic Command in 2008. To prove that Turla was indeed operating for 18 years straight, Rid needed to know what Turla was doing during these nine years. 

It was a Wall Street Journal article from 2001 that bridged that gap. The article discussed attempts by U.S officials to attribute the Moonlight Maze and other cyber attacks to the Russian and Chinese governments, and mentioned a little known cyber espionage campaign that was discovered a few months after Moonlight Maze ended. Its name was Storm Cloud, and the malware it used was based on a modified version of… LOKI2. It seems, then, that Moonlight Maze never really went away, but just increased their operational security, making them harder to detect, and that Storm Cloud was the missing link that connected Moonlight Maze with our modern Turla. 

Epilogue

Thomas Rid’s archeological research into the origin and evolution of Turla is a great illustration of the difficulties that future historians will encounter when they’ll try to get a better understanding of how cybersecurity was practiced in the first half of the 21st century. To uncover Turla’s distant origins, Rid had to submit multiple Freedom of Information requests to the government, analyze several different malware across several different operating systems – and crucially, locate an 18-year old server who miraculously wasn’t deleted or discarded. And while Rid successfully tracked Turla’s evolution in 2016, the obstacles he encountered underscore the difficulties that historians in 2116 will have to surmount in understanding how cyber wars were fought in the early 21st century. 

And when they’ll do that, they’ll probably stumble upon funny little anecdotes, like this one. 

In 1999, some time after Moonlight Maze was discovered, someone at the FBI had a neat idea: sending a team of investigators to Moscow, to confront Russian officials about the findings. If it turns out that the Russian government is not involved in the campaign, it’s likely they would be happy to learn about the rogue hacking group – and if it turns out that the government is involved – well, that’s useful information as well, obviously. 

By pure coincidence, a few weeks earlier the Russian Ministry of the Interior requested assistance from the US in identifying the owner of a website that defamed Russian President Boris Yeltzin’s daughter. As the defamation also included a terrorism threat, the FBI was able to assist the Russians. When the FBI reached out to the Ministry of the Interior and asked for their help in investigating Moonlight Maze – which the investigators presented as just another standard criminal inquiry – the Russian officials were more than happy to reciprocate. 

And so, on April 2nd, 1999, a seven men delegation landed in Moscow. They were warmly received by their hosts, and their first day in the Russian capital involved plenty of Vodka and Caviar. The next day they met with a Russian general who was their liaison for the investigation, and showed him the logs from the hacked computer. The general produced logs of his own, which clearly matched those of the Americans and proved that the attack had indeed been launched from the Academy of Sciences. The general was visibly embarrassed: he was certain it was the work of rogue intelligence operatives who acted on their own initiative. The delegation’s members sighed a sigh of relief: maybe this wasn’t a state-sponsored attack after all, and the whole matter could be resolved through quiet diplomacy. 

But then, on the third day of their visit, things took a sharp turn. None of their counterparts in the Ministry of the Interior showed up to the scheduled meetings, and the group’s driver announced that the day would be dedicated to sightseeing. So did the fourth day, and the fifth – and the sixth and seventh. The American delegation wasn’t allowed to set foot inside the Ministry again, and the helpful general disappeared without a trace. 

The conclusion was obvious: The Russian government was behind the Moonlight Maze attack. Apparently, the general and his staff weren’t briefed on the Moonlight Maze operation, and assumed they should assist their American allies. But when word of the investigation reached the Russian intelligence, who were probably directing the campaign, all cooperation stopped. 

—

Turla is still active to this very day: in 2023, for example, it was involved in espionage attacks against diplomatic and military organizations in Ukraine, as part of the Russian invasion. In May of 2023, the FBI successfully managed to infiltrate Turla’s network of hacked machines, and sent the malware a command that forced it to delete itself. This was a blow that probably set back Turla’s espionage efforts considerably, but judging from the APT’s long history – there’s little doubt that they will re-emerge from the shadows sometime in the not-so-distant future.