Video summary

Wir haben Claude 5 getestet

Main summary

Key takeaways

Technology

Summary of the Video (Auto-Generated Subtitles)

The speaker tests Anthropic’s newly released AI model family: Mythos and Fable. Anthropic previously said Mythos was too dangerous due to potential misuse (e.g., helping find security vulnerabilities). The video explains how Anthropic mitigated this risk using policy/classification gating rather than fully refusing requests.

1) What Mythos / Fable Are (and Why They Differ)

  • Mythos (previously described as very dangerous) and Fable are claimed by Anthropic to be fundamentally identical.
  • The key difference is a detection mechanism in Fable:
    • It uses “Constitutional Classifiers” to detect whether a request is about a “forbidden topic.”
  • If the classifier detects harmful requests, Fable does not always stop. Instead, it routes/downgrades the request to a “next best” model—currently Opus.
    • Claude indicates when it downgrades, but the reviewer finds it annoying, especially when users want maximum capability.

Restricted / Forbidden Domains Mentioned

  • Cybersecurity
  • Biology and chemistry (framed as preventing biological weapons)
  • Knowledge distillation (using the model’s outputs to train other models)

2) Reviewer’s Security / Safety Critique

  • The reviewer finds the initial cybersecurity blocking understandable, since vulnerability discovery can enable exploitation.
  • They’re more skeptical about biology/chemistry restrictions, suggesting the risk framing may be contentious.
  • For knowledge distillation, the reviewer interprets Anthropic’s goal as protecting itself—e.g., preventing others (such as “Chinese providers”) from using Fable outputs to train competing models.

3) Major Concern: “Silent Downgrading” / Opacity

A major complaint is that Fable may switch to a less capable mode when certain categories are detected.

  • The reviewer claims this behavior is described in a long system card (noted as 319 pages).
  • They argue downgrades can happen frequently and may be hard to notice, especially because performance drops while the user is paying for Fable.

4) Pricing and Availability

Subscription (Cheapest Access)

  • Pro subscription: $20/month (about €21.50 including taxes in Germany)

Temporary Access to Fable via Subscription

  • Access ends around June 22 (roughly 10 days remaining at recording time).

Afterward: Per-Token Billing (Expected)

The reviewer claims this will be extremely expensive, citing:

  • $10 per 1M tokens input
  • $50 per 1M tokens output

They estimate costs could reach hundreds to ~$1000 in a busy day, depending on workload.

5) Tests / Demonstrations (Capabilities Compared to Other Models)

Example 1: “Cap Recognition” (Japanese logo/pun)

The reviewer tests whether the model interprets a Japanese baseball cap design written as a pun in Japanese.

They compare:

  • Google Gemini Flash Thinking: hallucinates / gives nonsensical interpretation
  • ChatGPT: also hallucinates
  • Claude Fable: better performance—identifies the pun (“Ureshii” = “happy”) and uses online search to find the exact cap listing and its available colors

The reviewer claims this combo (pun understanding + accurate online verification) worked better than other models in their tests.

Example 2: Coding (Claude Code harness + interactive web app)

Using the Claude Code coding harness, the speaker asks Fable to convert a “3003” logo PNG into an interactive web app in multiple styles.

After ~15 minutes, the reviewer considers the results good:

  • Some effects (e.g., “particle storm”) are less exciting
  • Others (oil/liquid effect, 3D hologram) are strong

Comparisons

  • Opus can produce similar results but looked “less cool” to the reviewer
  • OpenAI Codex is also compared
  • They also mention running local coding with Qwen 3.6 27B (8-bit quantized) via Hermes as a harness

Example 3: Website Generation

The reviewer says Fable is not very impressive for generating a publish-ready website (mainly for testing purposes).

Criticisms include:

  • Some content is nonsense or inconsistent:
    • Not using the official c’t 4004 logo
    • Incorrect podcaster names
  • Media embedding behavior:
    • Play buttons don’t embed/play media; they redirect to platforms (YouTube/Podigee)

They say an Opus-generated site looks similar quality, also lacking embedding/actual thumbnails.

Example 4: SVG Animation Generation

Using a known SVG test (credited to Simon Williams) about drawing an animal riding a bicycle:

  • Fable’s octopus bicycle SVG is judged better than Opus visually.
  • The reviewer also notes seeing stylistic/wording oddities typical of LLMs.

6) Tests of Safety Classifier Frequency (“Downgrade Trigger Rate”)

The reviewer explicitly checks how often Fable gets downgraded.

They report downgrades happen often, including for requests they consider “fairly harmless,” such as:

  • “Can you check my code for security vulnerabilities?” (expected to be restricted)
  • “How does the mRNA vaccine work?” (biology/chemistry; still triggers)

Result: sometimes they only get Opus-level quality while paying for Fable.

7) Final Evaluation (Overall Opinion)

Despite frequent downgrades, the reviewer concludes:

  • As of the test, Anthropic Fable is the most powerful all-around model they’ve seen.
  • It’s also the most expensive, especially after the subscription ends on/around June 22.
  • Whether it’s “worth it” remains unresolved due to the downgrade behavior.

Main Speakers / Sources

Primary Speaker / Reviewer

  • “Nerd Keno” (the channel host / tester; referenced directly during the video)

System / Model Sources Referenced

  • Anthropic:
    • Mythos, Fable
    • Constitutional Classifiers
    • system card
    • benchmarking table
    • Opus as the downgrade target

External Tools / References

  • NordVPN (sponsor)
  • Simon Williams (SVG test concept)
  • Other referenced model providers/tools:
    • Google (Gemini)
    • OpenAI (Codex)
  • Local model mentioned:
    • Qwen 3.6 27B (8-bit quantized) via Hermes

Original video