Video summary

Boris Cherny: Stop Hobbling Your AI

Main summary

Key takeaways

Technology

Technological/product concepts and key claims

1) Opus 5 performance jump (Claude models)

  • The speaker (Boris Cherny, Anthropic) says the new Claude model, Opus 5, was shipped recently and shows a major performance acceleration.
  • They cite an Arc AGI 3 score reaching ~30%, whereas prior versions were reportedly in low single digits / low teens.
  • They attribute improvements to:
    • Training on many things at once, where only subsets reliably transfer
    • The model sometimes learns extra skills unexpectedly

2) “Runs for days/weeks/months” with Auto Mode

  • A standout capability of Opus 5, especially with auto mode, is that it can run extremely long-lived tasks—potentially days, weeks, even months—and “won’t stop.”
  • The model can reportedly do this without scaffolding (i.e., without needing wrapper instructions like /goal).

3) Stronger resistance to prompt injection

  • Boris claims Opus 5 is “not prompt-injectable anymore” in practice (i.e., it won’t follow malicious instructions embedded in retrieved/internet content).
  • They describe a layered defense approach:
    • Alignment research (“three years of research into alignment”)
    • A prompt injection classifier running on all traffic
    • Mechanistic interpretability to detect prompt-injection behavior by observing neurons that “light up” during injection attempts (referencing “Crystal’s mechanistic interpretability work”)
    • An additional auto mode classifier
  • This property was improving earlier too (e.g., Opus 4.7/4.8, “Sonnet 5,” “People was quite good”), but Opus 5 is described as a “new frontier.”

4) Claude Code harness: deleting most of the system prompt

  • For Claude Code (agent “harness” / product wrapper), the team reportedly:
    • Deletes and rebuilds large parts of the system prompt when new models release
    • Specifically claims that with Opus 5 they deleted 80%+ of the system prompt used in Claude Code
  • Practical experiments described:
    • You can experiment by overriding the system prompt (e.g., passing a “double dash system prompt” option)
    • An “undocumented” mode: setting an environment variable like simple=1 to remove system prompts entirely (including tool prompts)
  • Internal finding:
    • The model can be more intelligent without some prompts
    • But prompts still matter for the product experience (behavior, safety, usability)

5) “Ablation as eval”: rebuild prompts/tools each model release

  • They frame harness building as continual ablation-based evaluation:
    • Delete system prompt/tools
    • Re-add instructions line by line to see which parts are truly necessary
  • This also applies to tools:
    • Unshipping/changing tool definitions and tool code
  • They argue this deletion mindset should be standard for agentic product teams:
    • “Be comfortable and brave to press delete.”
    • For Claude Code users: periodically delete configuration (system prompts, skills, hooks) and see what the model still does.

6) How to rebuild a system prompt after deleting it (workflow)

Recommended process:

  1. Delete the prompt
  2. Use it on real product/customer agent tasks
  3. Observe failures repeatedly
  4. Only add back instructions when the model keeps failing in the same way

Why:

  • If the prompt is used every turn, unnecessary instructions can be harmful or redundant.

7) Evals: mostly constant, but also need replacement as models improve

  • They say evals should be continuously appended, but:
    • Evals may only remain valid for 1–3 model generations
    • Older evals can “saturate” as models improve
  • Eventually you may need to throw away old evals and build new ones based on what the newest model struggles with.

8) “Unhobbling Claude” vs “product overhang”

  • Conceptual framework:
    • Hobbling: the product/harness gets in the way of what the model can do
    • Product overhang: the model can do more than the product currently allows, so the product fails to elicit those capabilities
  • Example used:
    • Early Claude Code (with an older coding model like Sonnet 3.5-era) removed scaffolding and gave full terminal access, enabling writing full files/functions rather than limited IDE-like behaviors.

9) “Prompt engineering” becomes verification (not rigid instruction-following)

Boris suggests the industry shifts from “prompt engineering” to:

  • Context engineering / hard task design
  • Verification of outputs along the way

Key idea:

  • Give the model a task that’s “a little too hard,” plus a way to verify work, and avoid over-specifying step-by-step instructions.

Guides/tutorial-like operational advice mentioned

A) How to get the model to run very long tasks

  • Use auto mode with Opus 5
  • Avoid relying on heavy scaffolding
  • Provide verification so it doesn’t get stuck (e.g., continuous checking like screenshot comparisons)

B) How to reach “thousands of agents” (dynamic workflows)

They describe dynamic workflows in Claude Code:

  • Triggered by telling Claude to “use a workflow”
  • Runs within a Bun runtime sandbox, then spins up agent orchestration using virtual machines
  • Can fan out across stages (initial work, verification, summaries, more parallelization)

Alternate scaling methods:

  • Loops: cron-like local repetition for Quad
  • Routines: similar repetition but cloud-based (maintaining codebases over time)

Example of routine-based maintenance:

  • Daily cleanup of dead code
  • Shipping ready experiments
  • Adding/removing tests
  • “Abstraction police”: unifying duplicated abstractions

C) Example: multi-week desktop app refactor/verification

  • Task: rewrite an Electron desktop app in Swift, run it on a macOS VM, take screenshots, and pixel-compare against the Electron version.
  • Reported runtime: still running after ~14–15 days.
  • Presented as evidence that long verification loops are feasible with the right tooling.
  • The model may even “live blog” progress via updates (screenshots).

Concrete capability examples (use cases / experiments)

1) Codebase rewrite (Zig → Rust) running ~11 days in production

Example from the Bun team:

  • They fuzzed Bun (initially to find memory leaks) using Claude
  • Then tested whether Claude could rewrite the Zig codebase into Rust
  • With newer model capability (starting at “Fable” per transcript, and including Opus 5), this became feasible

Orchestration details:

  • Uses a test suite to validate correctness
  • Employs Claude Code’s dynamic workflows

Outcome:

  • Ran for ~11 days and rewrote the entire codebase
  • Reported as running in production under Claude Code’s runtime

2) “Accidental” capability discovery: OpenCV image drawing

  • Internal experiment: give Opus 5 OpenCV and ask it to draw images
  • Claimed result: plausible outputs (portraits, animals, landscapes) despite not being explicitly trained for “drawing” in that way
  • Framed as a “solicitation gap” (capability exists if prompted appropriately)

Overall evaluation criteria / mindset emphasized

  • Treat agentic building as an empirical / scientific iteration process, not purely theoretical engineering:
    • Try harder tasks
    • Provide tools and verification
    • Fix what fails
  • Avoid “over-specifying” like in older agent/automation patterns; modern models respond better to higher-level task descriptions and guardrails.

Prizes/announcements mentioned

  • Audience received “Max 20X” (sent by email), described as a special announcement.

Main speakers or sources

  • Boris Cherny (creator of Claude Code, Anthropic) — primary speaker
  • Anthropic (team context; mentions “Crystal’s mechanistic interpretability work”)
  • Crystal — referenced for mechanistic interpretability work used in the prompt-injection classifier

Original video