Claude Opus 5 is a freak

Overview / what the video tests

• The video reviews Anthropic’s Claude Opus 5, focusing on what it can and can’t do—especially agentic tool use, front-end generation, 3D creation, and autonomous workflows.
• The reviewer argues Opus 5 is best tested via an agentic coding harness: Claude Code (instead of only using the chat interface at claude.ai), because it can manage multiple files/folders and perform long-horizon, multi-step tasks.

Key demos / technical capabilities

1) “One prompt” Windows 11-like web replica using agentic harness

• Goal: Prompt Opus 5 to build a browser-friendly Windows 11 clone containing many real apps/programs (e.g., Office, File Explorer, Media Player, Discord, Slack, Spotify, etc.).
• Claude Code behavior:
  • Creates a full architecture plan first (e.g., drag/resize/snap/focus mechanisms).
  • Builds major UI components such as taskbar, start menu, and desktop.
  • Generates Office-like functionality, including:
    • Word editing: formatting (italics, font color, alignment) and Ctrl+S saving with persistence.
    • Excel formula engine: examples include AVERAGE(A1:A5) and SUM(A1:A5), plus save/load behavior.
  • Adds utility apps and simple games (e.g., Minesweeper, Snake).
• Verification / self-debugging:
  • The model opens the page in a browser, detects runtime errors, and autofixes issues until “everything works.”
• Reported resource use:
  • About 366,000 tokens, running over an hour.

2) 3D scene generation from a reference image (HTML output)

• Task: Upload a tricky geometric “office scene” image and generate a faithful 3D animated scene as a single HTML file.
• Claim: Other frontier models (e.g., “Fable” / “GPT 5.6”) couldn’t match it accurately, while Opus 5 produced the best result.
• Iteration:
  • The first attempt wasn’t faithful (missing arrangements, too much glow).
  • A follow-up prompt instructed Opus 5 to correct mismatches (“exactly like the attached image”), improving desk/chair layout and overall plausibility.
• Remaining issues:
  • Inconsistencies persist (e.g., extra chair, people in wrong places, wall/panel alignment issues, book placement inaccuracies).
• Reported resource use:
  • About 343,000 tokens, near an hour.

3) Autonomous “find reports → analyze → produce slide-motion video with TTS”

• Prompt: Using financial Q4 reports (Nvidia, Google, Meta, Amazon) with no direct attachments, Opus 5 should:
  1. Locate reports online
  2. Analyze financials + future outlook
  3. Generate a ~1 minute, 16:9 presentation video
  4. Use HyperFrames (referenced via a GitHub tool) for animation/motion graphics
  5. Produce voiceover via Gemini TTS using provided documentation
  6. Use provided audio background/instructions
• Output behavior:
  • It autonomously searches, installs/uses HyperFrames, generates voice via TTS, and renders the final video successfully.
  • The video is described as “cleaner/smoother” than comparable outputs.
• Reported resource use:
  • About 316,000 tokens.

4) Blender integration via MCP for 3D model + animation (X-Wing)

• The reviewer uses Blender MCP to let Claude Opus 5 operate Blender tools locally.
• Prompt: Generate an X-Wing fighter with realistic textures and motion.
• Observed behavior:
  • Plans step-by-step components (body, wings) and hinge mechanics for opening wings.
  • Produces an animation and validates via screenshots; if validation fails, it fixes issues.
• Detail:
  • Includes modular parts and positioning detail reminiscent of R2-D2 placement.
• Reported resource use:
  • About 173K tokens, but still takes close to an hour.

5) Music creation in a DAW (Waveform) with downloaded VSTs

• Prompt: Compose an “award-winning” song by:
  • Choosing instruments
  • Downloading free VST plugins
  • Loading them into Waveform DAW
  • Setting effects/automation (e.g., reverb/delay, panning automation)
  • Rendering a complete track
• Observed behavior:
  • Searches for Waveform, selects instruments, downloads plugins.
  • Handles constraints—for example, rejects a VST plugin that requires registration (can’t create accounts).
  • Creates a multi-track arrangement (kick/clap/hats, bass layers, pads/strings/plucks, risers, etc.) and renders a full song.
• Reported resource use:
  • About 350,000 tokens, over an hour; reviewer calls Opus 5 incredibly slow.

Visual understanding / biomedical evaluation results

“Find the frog” image test

• Uses a 3×3 grid visual inspection approach.
• The model attempts detection (edge enhancement, tracing “snake-like” candidates) but ultimately concludes it can’t find anything.
• Reviewer notes a positive behavior: if it can’t detect an object, it doesn’t hallucinate.

Tumor identification on 6 scan images

• Prompt: “Identify types of tumors in each of the six images, if any.”
• Result: Reviewer reports 0/6 correct—mostly incorrect identifications or mistaken “no tumor.”
• Comparison:
  • Another model (“Chimera 3”) gets 1 correct.
  • Another (“Fable 5”) refuses the biomedical request entirely.
• Takeaway: Opus 5 is more open to biomedical prompts, but appears weak/unsafe-uncertain for accurate diagnosis-like classification.

Deep biomedical research prompt (atherosclerosis)

• Prompt: Pathophysiology analysis with tables/visualizations.
• Reviewer claims Opus 5 produced flowcharts/diagrams/tables and answered successfully.
• Preference note: Reviewer subjectively prefers other models’ structure/verbosity (e.g., Gemini being too verbose; GPT/Kimika being more concise and information-dense).

Performance, specs, benchmarks, and cost analysis (as presented)

• Model role: Frontier model designed for agentic workflows and long-horizon tool use.
• Context window: ~1 million tokens (described as enough for ~700k words or a medium-sized codebase).
• Speed: Reviewer emphasizes Opus 5 is painfully slow, worse than GPT 5.6 and Fable 5, measured via output tokens/sec.
• Cost: Reviewer says Opus 5 is much more expensive, nearly 2× Gemini 5.6 Solo, and not cost-effective.
• Hallucination: Noted as “not bad,” roughly similar to Gemini K 3, but worse than a leading open-source model (JLM 5.2).

Leaderboard results (mixed interpretation)

• Artificial Analysis leaderboard: ranked #1 or near #1, but reviewer criticizes missing confidence intervals.
• Arc AGI-3: Opus 5 scored >30% (vs GPT 5.6 <10%); described as evidence of emergent learning—with a caveat that benchmark design/similarity may inflate performance (good on familiar games, weaker on novel rules).
• Other cited results:
  • LiveBench: Opus 5 #3, Gemini #1
  • Debates: Opus 5 #1
  • NYT Connections: Opus 5 around #2, behind Gemini 3.1 Pro
  • Valves index: behind Fable 5 and only slightly above Kimika 3, but at much higher cost
  • Deep Sweeps: appears #1, but confidence intervals may suggest no meaningful difference

Guardrails / “nerfing”

• Like “Fable 5,” Opus 5 may reject cybersecurity/biology-related prompts and fall back to Opus 4.8.
• Reviewer claims it’s less restrictive than Fable 5 for certain vulnerability-related tasks.
• However, it still blocks long-running autonomous biomedical research when risk is considered substantial.

Overall verdict (as stated by the reviewer)

• Opus 5 is characterized as:
  • Excellent for vibe coding, front-end UI, and 3D design
  • Extremely expensive and very slow
  • Often not clearly better than other frontier models (e.g., GPT 5.6, Gemini 3) for many tasks
• The reviewer recommends paying for Opus 5 mainly when you specifically need:
  • High-quality front-end/3D output, or
  • A model that can eventually solve a stubborn coding problem where others fail.

Main speakers / sources

• Main speaker: The video reviewer/host (name not provided in subtitles).
• Primary external sources referenced:
  • Anthropic (Claude Opus 5 release, benchmarks/model claims)
  • Claude Code / Claude.ai
  • HyperFrames (motion graphics tool repository)
  • Gemini TTS (voiceover)
  • Luma AI (sponsor: Luma Agents / Luma Skills)
  • Blender MCP (Blender integration)
  • Waveform DAW
  • Benchmark communities: Artificial Analysis, LiveBench, plus mentions of Arc AGI-3, Deep Sweeps, Valves index, and others