How Anthropic made Claude 3x faster

Technological concepts & product/performance improvements discussed

• Anthropic’s claim: Claude 3 / cloud.ai (Claude AI) + Claude Desktop became ~3× faster in a 2-week sprint, attributed largely to using Claude Tag (an internal Slackbot/agent) to debug performance.
• Performance focus: The sprint targeted the core user journeys that account for 95% of activity, including:
  1. App launch
  2. Starting conversations
  3. Loading existing conversations
  4. Sending messages (web + desktop)
• Measured outcomes (quantified):
  • Fresh load time for Claude (75th percentile): 3.1s → 0.55s
  • Start new cloud code session: 0.8s → 0.3s
  • Load cloud work sessions: ~3s → <1s
  • Website launch: ~3s → 550ms
  • Desktop app launch: >6s → ~3s
  • Sending messages:
    • cloud co-work: ~1s → 48ms
    • claude code on desktop: ~250ms → 52ms
  • Aggregate savings (estimate): tens of thousands of user-hours/day spent waiting (per Anthropic’s estimate).

How the sprint worked (tooling, instrumentation, and “measurement first”)

• Claude could measure → it could optimize.
  • The team kept “finding more things to measure” so the agent had more reliable targets than vague “make it faster.”

Instrumentation & baselines

• They added/adjusted telemetry so metrics were:
  • comparable
  • properly scoped (e.g., timing from user interaction → render completion, separating client vs. server work)
• Key critique (from the speaker): teams often mis-measure what users perceive—for example, measuring component render instead of user-perceived “interaction complete.”

Lab benchmarking approach + deterministic metrics

• For benchmarks, they used lab measurements and deterministic counters, such as:
  • Valgrind / Node predictable flags (for JS hot paths where instruction counts are deterministic)
  • Deterministic counts for browser paths (Chrome/React/V8-specific), including:
    • React commits per interaction
    • V8 call counts
    • layout/style recalculation counts
    • DOM mutations
• CI guardrails (“ratchet-down”): benchmarks had a ceiling so improvements couldn’t silently regress.
• Benchmark selection discipline: flaky benchmarks or those that didn’t correlate with real user latency were discarded.

Prompting/agent workflow (the “thread loop”)

• A Slack channel with standing instructions guided Claude as the performance “owner” for the website/desktop app.

Workflow loop

1. Humans open a thread when a journey seems slow (often with screenshots/recordings).
2. Claude traces the flow and builds benchmarks demonstrating the bottleneck.
3. Claude proposes PRs (often behind feature flags).
4. After shipping, Claude monitors deploys/field data.
5. If performance improves, it “locks in the win” by tightening benchmarks/ratchets; otherwise, it disables the flag and iterates.
6. Then it hunts the next bottleneck in that journey.

High throughput / safety practices

• >3,000 changes merged with no customer-facing incident or rollback (as claimed).
• Automated reviews with at least one human approval.
• Hot-path changes shipped behind flags; flags retired quickly when safe.
• Nearly 200 flags introduced during the sprint (with more than half removed by the end).

Guardrails & correctness strategies (beyond raw speed)

Static Composer guardrails (example)

• They used a “static composer” that renders HTML early and lets React paint later.
• Guardrails included:
  • JSDOM-based rendering to generate static markup
  • tests ensuring React/static never drift
  • integration tests comparing static HTML vs. React render across viewports
  • keystroke tests ensuring key ordering and no reordering issues
  • field monitoring for nonzero layout movement

Layout shift / UI stability

• The speaker describes investigating issues where caching and delayed real data could cause layout shifts (CLS-like behavior).
• They discuss tests that ensure stability by failing on shift in named regions, but caution that mismatched cached vs. network data can still cause shifts.

Speaker’s practical counterpoints and observed issues (real-world testing)

The host/observer tries the improved app and notes remaining or newly surfaced engineering challenges:

• Thread persistence bugs after refresh: submitting a prompt could create a new thread, but persistence/navigation state wasn’t consistently correct until later.
• Aggressive caching / stale sidebar content: cached “recent threads” via local storage / IndexedDB could remain incorrect (showing threads that don’t exist after refresh) and not always revalidate.
• Suggested remedy (speaker): solutions like a proper state-sync/data engine (mentioned: “Convex”) could reduce cache invalidation and sync issues.
• CSS/UX performance regression risks: performance work can introduce UX issues, so strong tests/validation are emphasized.
• Main-thread blocking and rendering offloads:
  • The host highlights worker/off-main-thread techniques (e.g., moving code syntax highlighting off the main thread) to avoid jank.
  • Example: syntax highlighting freezes due to M-dash/non-ASCII characters causing slower UTF-16 paths; they move highlighting to WebAssembly in a worker.

Additional performance tactics described

• Navigation improvements:
  • Composer kept mounted between conversations.
  • Prefetch sessions on hover.
  • Claimed ~90% reduction in re-renders.
• App/desktop startup improvements:
  • Static composer injected in HTML for faster readiness.
  • Precompiled V8 code cache to avoid recompiling at desktop shell startup.
• Streaming / frame budget work:
  • Rendering optimized at an 8.3ms frame budget (targeting 120Hz smoothness).
  • Claude stepped through long replies frame-by-frame, improved data structures, and reduced per-chunk scaling work.
• 120Hz deterministic benchmarking:
  • Headless Chrome configured for deterministic frame stepping to validate 120 FPS behavior in the lab.

Main speakers/sources (as identifiable from the subtitles)

• Primary speaker / video host: the person discussing T3 code/optimization and reacting to Anthropic’s article (referred to as “me”).
• Anthropic team / article authors: referenced as the Anthropic blog post/article, plus internal Slack channel details.
• Other mentioned contributors (not necessarily speaking):
  • Sam
  • Maria / Raymond / Pier / Shelley / Isaac / Jamie
  • Adios Mani (Chrome team hire)
  • Boris (referenced as “guy who started Cloud Code” / encouraging ambition)
  • Anthony
  • The WorkOS sponsor content: narrator promoting Work OS (OTD portion)