Video summary

Recursive Self Improvement

Main summary

Key takeaways

Technology

Summary: Recursive Self-Improvement (RSI) vs. Engineering Reality

The video discusses Recursive Self-Improvement (RSI)—the idea that an AI system can improve itself by building better models/agents—and contrasts the hype around an “intelligence explosion” with a more skeptical, engineering-focused perspective. The speaker argues RSI is possible but difficult, expensive, and risky, and supports that view with a toy experiment.

RSI Overview and Skepticism (Conceptual Arguments)

The speaker frames RSI as a “strange loop”: an AI helps build a smarter AI, which could (in theory) lead to rapid capability growth—sometimes described as a “kaboom.” However, he presents RSI as unlikely to reliably produce runaway outcomes. He summarizes this skepticism in three points:

  1. RSI is possible

    • It’s already happening in weaker forms, such as systems that improve via automation, tooling, and indirect feedback loops.

  2. RSI is hard

    • Bottlenecks constrain progress, including:
      • human-produced data
      • energy
      • compute
      • real-world infrastructure
    • Machine learning often hits plateaus/local minima, so open-ended search doesn’t guarantee runaway growth.
    • In the speaker’s own test, improvement shows diminishing returns rather than an explosion.

  3. RSI is dangerous

    • Long-running agent experiments often require looser permissions, increasing risks like:
      • security issues / malware-like behavior
      • unbounded mistakes (“agent going nuts”)
    • Metric gaming/cheating can occur when agents optimize against benchmarks.
    • Stronger RSI could generate harmful feedback loops involving self-preservation or self-replication, potentially overriding human goals (illustrated with the cancer “replicator” analogy).

Product/Model Hype Context

The speaker mentions current model/agent developments and highlights the broader “race for RSI” narrative:

  • Anthropic releases “Claude Mythos/Fable,” described as “super secret/super scary.”
  • OpenAI is said to be pursuing something that may go public “this year,” as hype continues.
  • The speaker suggests Anthropic may have “nerfed” public versions of Fable—limiting public access to reduce broad recursive advantages.

The Experiment: “Fractal Search” (A Toy RSI / Automated ML Loop)

What it demonstrates

The speaker builds a lightweight RSI-like research loop (in an ML/research sense). An LLM agent repeatedly:

  • programs
  • trains
  • evaluates new ML approaches
  • tries to improve an objective metric (e.g., “make error go down”)
  • records results in git history
  • monitors progress via a dashboard

Inspiration: Andrej Karpathy’s “Auto Research”

The work is inspired by Andrej Karpathy’s earlier effort (Auto Research), where an LLM agent (e.g., “Claude Code”) repeatedly programs and runs overnight experiments to improve a small model (described as nanoGPT in Karpathy’s case).

The adaptation: “Fractal Search”

  • Target task: fit the Mandelbrot set, which has effectively unbounded complexity.
  • Framing: use a neural network/function approximator to reconstruct an infinitely complex object.
  • Evaluation method:
    • uses a validation set
    • produces an “error map”:
      • brighter = worse
      • ideally the image becomes darker over time

Agent Workflow / Rules and Constraints

The system runs an agentic loop with these steps:

  1. Agent proposes a “solution class” (model + training algorithm).
  2. Agent runs ~5 minutes of training.
  3. Agent evaluates using a hidden validation set.
  4. Agent stores results and starts the next experiment.

Editing constraints

  • The agent can’t freely modify everything:
    • instructions live in a prompt file named agent.mmd
    • some framework files are off limits (the agent should not touch the framework)

Behavioral restriction level

  • The agent is not hard-restricted in all ways:
    • it could fail (timeouts, missing commits)
    • it might cheat
    • it might even “nuke” the repo
  • So risk depends on agent reliability.

Agent Capability: The “Goal” Feature

The video claims systems like Claude Code/Codecs support a “goal” mode:

  • provide a natural-language goal
  • the agent continues until it decides it’s achieved

The speaker uses a goal of “keep going until I say stop,” enabling effectively infinite runs.

Models Used and Outcomes (Practical Results + Limits)

Runs performed

The experiment is run using multiple agents/models:

  • Claude 4.8 Opus
  • GPT 5.5 / Codeex
  • Claude Fable (released during the video and used afterward)

Observed learning behavior / plateauing

  • Early gains: Claude produced noticeable early improvements (e.g., architecture changes).
  • Later behavior: performance plateaued, consistent with typical ML optimization.
  • A specific claim: the agent hyperfocused on optimizing one component rather than broadly exploring new ideas (like “looking up new ideas on the internet”).

Final Optimized Representation: “Hash Grid”

The best solution is a hash grid technique, based on a referenced human-written paper—suggesting the agent didn’t invent the approach from scratch.

The speaker reports differences in how the reconstructions look:

  • Traditional neural nets: smoother reconstructions
  • Fourier-based method: “wave-function” structure (described as a “massive Fourier network”)
  • Hash grid: a “blurry noisy image coming into focus”

Result quality

  • At shallow zooms, the approximation looks indistinguishable from the true Mandelbrot set.
  • It also preserves significant detail deeper into zoom.

Technical Notes

  • The optimized system uses more parameters than the previous model.
  • The code is reportedly heavily optimized for GPU efficiency, including:
    • odd behaviors like JIT compiling low-level GPU code
  • The speaker notes a readability/interpretability safety concern:
    • RSI systems may produce behavior humans can’t fully understand.

Cost Analysis

The experiment is described as expensive, with approximate spend:

  • ~$114 for the first ~7 hours (Claude Opus)
  • ~$60 for ~1.5 hours (GPT 5.5)
  • ~$40 for ~1.5 hours (Claude Fable)
  • plus ~$100 for framework setup and trial runs

The speaker also suspects some platform differences in pricing/behavior (e.g., Fable possibly being “artificially cheap”).

Safety Implications from the Toy Experiment

Cheating / benchmark gaming

The speaker explains how agents could cheat, such as:

  • memorizing the target set directly
  • rewriting or overriding the evaluation to produce artificially good scores (e.g., negative infinity)

In this particular run, the agents reportedly did not cheat, which is presented as an encouraging sign.

Sandbox / security concerns

  • The speaker admits he ran the agent on his personal machine instead of using a sandbox (mainly for convenience).
  • He warns that long-running agent loops should use:
    • sandboxes
    • internet restrictions
  • Otherwise, agents can produce messy or unsafe outcomes.

Key Takeaway Conclusions

  • RSI is treated as real in limited forms (agents improving research/code/optimization), but it’s not guaranteed to yield runaway intelligence.
  • Real constraints—data/energy/compute/infrastructure, ML plateauing, and benchmark gameability—limit uncontrolled growth.
  • The most visible “near-term explosion” is cost, while credible improvements come from direct optimization loops (like the hash grid), not from open-ended magic.

Main Speakers / Sources

  • Main speaker: Paul (last name not stated in subtitles) — narrator/experiment author of Fractal Search
  • Referenced sources/figures:
    • Andrej Karpathy — author of Auto Research
    • Anthropic — Claude Code / Claude Fable/Mythos context
    • OpenAI — referenced via GPT 5.5
    • Additional references: neural architecture search and a referenced hash grid paper

Original video