A long con social engineering attack that could have resulted in a large number of Internet servers being backdoored was detected and blocked this weekend thanks to the heroic work of one hacker.
Postgres developer Andres Freund, who is currently employed by Microsoft, stumbled on the issue a few weeks ago when he noticed performance degradation — measured in milliseconds — in a core Linux tool.
Freund noticed that Secure Shell (SSH), specifically OpenSSH, a tool used by just about every server on the Internet for secure administrative access, was performing poorly in a pre-release version of Debian.
Debian is a widely used Linux distribution on which many others are based, including Ubuntu.
Ubuntu, in turn, forms the basis for many other Linux distributions.
After further digging, Freund determined that the issue was not with OpenSSH but with specific customisations Debian and other distributions patched into it.
Fortunately, not all distributions or even Debian derivatives like Ubuntu use this customisation. However, many do.
These customisations rely on a compression format called LZMA. One of this standard’s widely-used open source implementations is the XZ project.
Freund’s analysis revealed that XZ’s implementation of LZMA contained a backdoor that granted attackers some level of access to infected systems.
At the very least, the backdoor allowed the attacker to remotely execute arbitrary code on a target machine.
Although not full access or an authentication bypass, remote code execution can be as good as having total administrative control of a machine.
Subsequent analyses of the attack suggest that the backdoor enabled remote code execution.
The vulnerability is being tracked as CVE–2024–3094 with a CVSS score of 10 — indicating the highest possible severity.
Freund’s initial detective work also found that the backdoor was not in the XZ Project’s source code — it only existed in already-compiled versions of the software that distributions like Debian used to create its packages.
XZ used Github for version control, and it revealed that the malicious packages were all uploaded by the same person, “Jia Tan”, with the username JiaT75.
At this point, other hackers took the baton from Freund and investigated Jia Tan’s activity on Github, revealing what appeared to be an incredibly sophisticated supply chain attack.
Software developer Evan Boehs has published a summary of the investigation’s findings so far.
Boehs and other hackers found that Tan created his Github account in 2021 and submitted his first patch to the XZ Project in 2022.
Tan became a regular contributor to the project, building trust over several months.
XZ creator Lasse Collin, under increasing pressure to add more maintainers to the project to help speed up development, gave Tan greater privileges and responsibilities as he demonstrated commitment to the project.
Tan pushed the final parts of the backdoor to GitHub on 23 February and 9 March 2024.
Shortly after this, there was a flurry of activity by Tan and another unknown moniker, “Hans Jansen”, to get the new versions of XZ included in various Linux distributions.
According to Orca Security, Tan got his backdoored code into pre-release versions of Debian and Fedora, and into OpenSUSE Tumbleweed.
As a result of getting into Debian’s unstable branch, the compromised version of XZ also found its way into the security-focused Kali Linux distribution.
While the deception was discovered shortly afterwards, a lot of it was luck.
The Internet dodged a bullet — but not because it was watching for one
In a post on Mastodon, Freund explained that he wouldn’t have spotted the issue had it not been for several coincidences.
Firstly, he happened to be micro-benchmarking changes being implemented in Postgres, a relational database management system.
He also happened to be running a pre-release version of Debian to ensure compatibility.
He also happened to have seen a complaint about a related but separate issue a few weeks earlier, and happened to have an option enabled that caused that complaint to surface again during his benchmarks.
“Just to be clear: I didn’t mean that I didn’t do good — I did. I mean that we got unreasonably lucky here, and that we can’t just bank on that going forward,” he said.
Senior Google engineer Damien Miller agrees. Miller has over 17 years of experience at Google, chiefly in information security.
“This is the nearest of near-misses. Anyone who suggests this was any kind of success is a fool,” Miller said on Mastodon.
“No system caught this, it was luck and individual heroics. That’s not acceptable when unauthorised access to [nearly] every server on the Internet is on the table. We need to find a way to do better.”
Miller said few of the software supply-chain defences people have been proposing would have prevented this compromise.
“The attacker was a (relatively) long-term maintainer, was not averse to using sockpuppet accounts and was careful to hide their exploit from automated tools,” he explained.
“Worse, many of the solutions being offered increase the workload on maintainers. But maintainer burnout was another key factor in this incident.”
Following Freund’s disclosure of the backdoor, an email sent by XZ Project creator Lasse Collin in 2022 has reignited discussion about how critical software infrastructure is built and maintained by volunteers whose conscientiousness often leads to burnout.
In the email, Collin was responding to criticism from what is now suspected to be sockpuppet account created by the attackers.
“Progress will not happen until there is a new maintainer… Submitting patches here has no purpose these days. The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this,” a person calling themselves Jigar Kumar wrote.
“I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things,” Collin replied.
“Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.”
Part of the sophistication of this supply-chain attack was that enemy hackers did not go after OpenSSH directly but instead targeted a weaker link in the chain.
Based on the available evidence, they specifically targeted a project that had become an important dependency in critical security infrastructure where they could see a single vulnerable maintainer struggling.
They then heaped more pressure on someone already struggling with their mental health to amplify the feelings that he wasn’t living up to people’s expectations or failing people who were depending on him.
“We need to find a way to support maintainers without being proscriptive or parentalistic,” Miller stated.
Miller also predicted that this wouldn’t be the last sophisticated and methodical open source software supply-chain attack.
“The actor(s) behind XZ are probably already learning their lessons ahead of their next attempt,” he said.
Indeed, XZ might not be the only attack they had in progress, Miller warned.
“The next one is going to be more carefully operated and harder to spot. How are we going to stop it?”
Author’s Note: The term “hacker” is used here in the word’s original meaning. Freund is not an information security professional, but he is a highly skilled programmer, an enthusiastic tinkerer, and when he noticed something strange he did not rest until he understood what was happening.
Tyler Fields is your internet guru, delving into the latest trends, developments, and issues shaping the online world. With a focus on internet culture, cybersecurity, and emerging technologies, Tyler keeps readers informed about the dynamic landscape of the internet and its impact on our digital lives.
