Why Git Wasn’t Enough for Anthropic!

Summary

Anthropic reportedly built Scion, its own version-control system, because Git’s usual workflow created too much overhead for its large-scale automated testing.

• Anthropic was merging more than 30,000 code changes a day and running about 25 million CI checkouts daily. Each checkout prepares a machine with the code needed to compile and test a change.
• With Git, machines typically clone repository data that includes its history. The video says Anthropic’s history was nearly 100 GB, while even a reduced copy exceeded 25 GB—a large download when a test may only concern one file. Repeating this process at Anthropic’s scale was estimated to generate over 600 PB of Git traffic per day.
• Git’s local-history model is useful for developers working offline, but the video argues it is poorly suited to always-online CI machines that need only a subset of the code. Git also lacks a built-in way for those machines to query a central server for historical information.
• Scion keeps history on central servers. A CI machine can query the repository’s root tree and fetch only the folder it needs—for example, the billing folder when testing a change to tax.py.
• Scion reduces repeated downloads with a three-level cache: the CI machine’s local cache, a shared cache in the same region, and then the central server.
• File versions receive unique IDs based on their contents. Editing a file creates a new ID while leaving the old version intact, helping cached versions remain valid.
• The video says this approach brought checkouts down from minutes to seconds and reduced CI machines’ CPU usage by 25%. Scion was reportedly built by seven engineers working with Claude over five months.

Reviews, guides, or tutorials: None are presented; the video is an explanatory overview of Git’s scaling limitations and Scion’s design.

Main speaker/source: A single narrator in a KodeKloud video; no other speakers or named sources are identified.