The Internet Was Weeks Away From Disaster and No One Knew

High-level story

• In 2024 an elaborate supply-chain backdoor was discovered in the XZ compression project. XZ (LZMA) is a widely used lossless compressor and a dependency in many Linux distributions and in the OpenSSH toolchain.
• A malicious contributor (alias “Jia Tan”) attempted to insert a stealthy backdoor that would give an attacker a master key to authenticate into SSH on virtually any affected Linux server.
• The backdoor reached pre-release packages for distributions including Fedora, Debian testing and Ubuntu pre-release and might have been on a path into Red Hat Enterprise Linux (RHEL).
• Detection began with a suspicious performance slowdown investigated by Andres Freund. After coordinated disclosure, Fedora rolled back the packages and the community removed the malicious code.

Technical concepts and explanations (tutorial-style)

• Cryptography and authentication

  • Diffie–Hellman key agreement: how two parties establish a shared secret over an eavesdropped channel (illustrated as public‑color + private‑color paint mixing).
  • RSA public‑key authentication: two secret primes generate a public modulus; the published public key authenticates whoever knows the factors.
  • SSH: combines a secure channel (key agreement) with authentication (RSA) for secure remote logins; OpenSSH is critical infrastructure for remote server maintenance.

• Compression algorithms

  • Huffman coding: variable-length codes based on symbol frequency.
  • LZ77: sliding-window pointers that copy repeated substrings.
  • LZMA: Lempel–Ziv plus Markov modeling with a large search window; XZ uses LZMA to achieve better compression than typical .zip/deflate.

• Shared libraries and runtime linking

  • Global Offset Table (GOT): how applications resolve addresses of shared-library functions at load/run time.
  • IFUNC resolver: lets a library pick different implementations early at runtime (for example, optimized code for different CPU vendors).
  • Dynamic audit hooks: a low-level feature that can run code when the loader writes GOT entries.

• Memory debugging

  • Valgrind: detects memory errors (invalid writes, leaks) and can expose misbehaving or malicious components.

The attacker’s technique — three technical phases

1. Trojan horse (supply‑chain entry)

   • Hide malicious payload inside binary “test blobs” in the XZ repo (binary data files packagers commonly ignore).
   • Make superficial benign changes on GitHub to build trust and be accepted as a contributor/maintainer.

2. Goldilocks timing (GOT hijack)

   • Use an IFUNC resolver inside the XZ library to run code very early during program load.
   • From the IFUNC, locate and set a dynamic audit hook so that when the loader writes the real address of a crypto function (RSA decryption) into the GOT, the audit hook runs and swaps the GOT entry with the attacker’s function implementation — timed between the real write and the subsequent read-only lock.
   • This gives the attacker control over the authentication function used by OpenSSH without modifying OpenSSH source directly.

3. Cat burglar (stealth master key and cleanup)

   • The malicious function checks for a special master-key exchange (its own mini‑cryptographic handshake) and only then grants access; otherwise it defers to the real function.
   • The backdoor obfuscated strings, performed safety checks to avoid crashing, and attempted to wipe logs to hide traces.

Detection and exploitation demonstration

• The backdoor introduced a subtle latency (~400–500 ms), which tipped off a developer (Andres Freund). That timing anomaly led to code inspection, discovery of the backdoor, and rapid community response.
• The demonstration in the video shows the exploit live (on a cloned Veritasium site) to illustrate that a compromised server could grant root control, modify website content, exfiltrate or alter data, or install ransomware.

Security analysis, consequences and community response

• Impact potential

  • A universal SSH backdoor could enable spying, ransomware, or widespread disruption on millions of servers, including critical infrastructure (banks, governments, supercomputers).

• Root causes identified

  • Dependency concentration in the open-source ecosystem: many projects rely on small, volunteer‑maintained components (single‑maintainer risk).
  • Social engineering and long multi‑year campaigns can coerce or replace maintainers to insert malicious code.
  • Not all dependencies receive equal scrutiny, even when top-level projects (like OpenSSH) are well audited.

• Response

  • Debian, Red Hat and Fedora teams rolled back or blocked tainted packages after disclosure. The community audited related projects for similar compromises.

• Broader debate

  • The incident highlights both vulnerabilities in open-source supply chains and a key strength: public code allowed community review to find and remove the backdoor.
  • It underscores the need for better funding, multiple maintainers, and stronger review/auditing guarantees for critical projects.

• Attribution

  • Attribution remains uncertain. Evidence (timezones, naming, operation patterns) is ambiguous and could be intentionally misleading. Defenders must assume sophisticated, persistent adversaries.

Key products, tools and projects mentioned

• Compromised and involved projects

  • XZ (LZMA compressor) — compromised project
  • OpenSSH — target/application relying on XZ indirectly
  • Fedora, Debian, Ubuntu, Red Hat Enterprise Linux (RHEL) — distribution ecosystems involved

• Tools and primitives

  • Git / GitHub — source hosting and contribution workflow
  • Valgrind — memory error detector that helped expose problematic code
  • Global Offset Table (GOT), IFUNC resolver, dynamic audit hooks — runtime linking/hijack primitives
  • SSH, Diffie–Hellman, RSA — crypto/authentication building blocks

• Other mentions / context

  • Postgres — unrelated project where the discovering developer worked (Andres Freund)
  • GNU Project, Free Software Foundation, Linux kernel and GNU tools — historical context for open source

Lessons, recommendations and takeaways

• Supply-chain security

  • Account for subtle maintainer takeovers and hidden binary blobs inside repositories; code review should include test data and build artifacts.

• Project sustainability

  • Widely depended-on open-source components need sustainable funding, multiple maintainers, or stronger review/auditing guarantees to avoid single‑person failure modes.

• Detection approaches

  • Public/open code enables discovery and remediation, but attackers can design backdoors to survive heavy scrutiny. Detection may rely on side signals (performance anomalies) as much as code review.

• Operational assumption

  • Treat potential attackers as sophisticated and persistent; don’t assume obvious attribution.

Main speakers / named sources referenced

• Derek (Derek Muller) — Veritasium host / on-camera interviewer
• Narrator (labeled in transcript as Narrator / Henry in parts)
• Richard Stallman — history of Free Software / GNU
• Linus Torvalds — origin of Linux (referenced)
• Tatu Ylonen — creator of SSH (explains crypto)
• Lasse Collin — original XZ maintainer (victim / single‑maintainer example)
• Jia Tan — alias used by the attacker(s) (malicious contributor)
• Rich — Fedora packager who accepted XZ update (packager perspective)
• Andres Freund — Microsoft/Postgres developer who discovered the backdoor
• Organizations quoted: Red Hat, Debian, Ubuntu, Microsoft, NTT, and the open-source community generally

Optional additional artifacts (available)

• Compact timeline of attack → discovery → mitigation steps.
• Diagram of the attack flow: Trojan → IFUNC/audit hook → GOT swap → backdoor authentication.
• Extracted technical mitigations with implementation pointers (CI checks, binary-blob scanning, reproducible builds, mandated multiple maintainers).