Video summary

Please stop using stupid models

Main summary

Key takeaways

Technology

Main topic

The speaker argues that coding/engineering AI agents and “frontier” code models are improving most in reliability over longer autonomous runs—i.e., a higher “floor” with fewer dumb failures.

They also claim many engineers misunderstand model tradeoffs and prompting practices. In particular, they criticize an idea attributed to David Kramer (“Zeg,” Sentry) that switching (“downgrading”) to lower reasoning / older models (e.g., Opus/Soul vs newer Fable/Astra) would yield roughly the same practical results.

Disputed claim: “downgrade and tasks won’t change”

A respected engineer (David Kramer / Zeg, founder of Sentry) claims that users on cheaper tiers should switch back to older “high reasoning” models (Opus and Soul) and expect no meaningful performance difference.

The speaker strongly disagrees:

  • People who report “no difference” likely have:
    • Poor prompting habits, or
    • Tasks that are too narrow/simple.
  • The meaningful differences emerge when tasks are broader/longer, where lower-floor models fail more often.

Key performance framework: “ceiling vs floor” and prompt width

The speaker uses an implied chart/diagram framework:

  • Ceiling = peak quality / best-case performance
  • Floor = worst-case reliability / how often the model plausibly goes wrong or derails

They emphasize that:

  • Raising the floor matters more than raising the ceiling, especially for agents.

They also highlight a prompting/task dimension:

  • The key factor is prompt width (how many things are involved, how long the run is, and how far the task extends), not just difficulty or “depth.”
  • Wider tasks require higher floors—otherwise failures become more likely as time increases.

Astra vs Soul / Fable vs Opus (model-specific sentiment)

Astra

  • Can sometimes outperform others in ways that are described as “incredible”
  • But may also fail in “dumb” or annoying ways
  • The speaker characterizes it as unpredictable/spiky, making it harder to know how much to trust it.

Soul

  • Presented as less spiky / more consistent for the kinds of scenarios under discussion (especially “weird” failure modes attributed to Astra).

Fable vs Opus

  • For narrow prompts (e.g., “find a file, make this change, tell me when done”), differences may be minor because both clear the bar.
  • For broader/longer end-to-end tasks, the speaker claims the difference becomes pronounced.

Core agent review/tutorial lesson: test by running, not just reading

The speaker promotes an AI code review/testing approach using an agentic tool (sponsor: Grapile’s “T-Rex”).

Common mistake:

  • AI code review bots assume correctness from code reading alone.

Better approach:

  • Use sandboxes to actually run changes and verify behavior.

What T-Rex does (as described)

  • Indexes/understands the whole codebase for context
  • Spawns review orchestrators that can create sub-agents
  • Runs multiple sandbox tests depending on suspected failure points
  • Returns evidence such as images/videos showing that tests passed

The speaker stresses:

  • “Thumbs up” approvals aren’t enough.
  • Video proof is more valuable (e.g., UI dropdown disappears correctly, signup flow works end-to-end).

Practical workflow emphasis: longer autonomous runs + verification artifacts

The speaker describes their development workflow (example: “T3 Code”) as giving an agent a low-intervention loop:

  1. Provide a screenshot/DM bug report
  2. Instruct the agent to fix, test, record a video proving it works, and link the PR
  3. Let it continue until issues are resolved in review

Claimed advantage:

  • Agents can maintain context over time and produce an end-to-end PR with verification, reducing constant supervision.

Mental model for developers

They suggest thinking in terms of two metrics:

  1. How long the model can run without input
  2. How likely it is to work before humans step in

Goal:

  • Make the model less likely to screw up within any given window.

Evidence claim: model runs are getting longer (logs + statistics)

The speaker provides log/usage comparisons across time periods:

  • Median prompt duration: ~53 seconds → 2 minutes 20 seconds
  • P95 prompt duration: ~<7 minutes → >16 minutes 20 seconds
  • They observe gradual increases over months, then note that longest requests nearly doubled after new models (mentions Fable and Soul around May–June).

Why longer success improves “exponentially” (probability math)

They present a probability argument about failure compounding:

  • If a model fails 5% per 10-minute window, extending runtime increases the chance of at least one failure nonlinearly.
  • Even improving the per-window failure rate (e.g., 5% → 3%) can dramatically increase expected maximum run length due to compounding.

They argue frontier models reduce failure rates over time windows, enabling more reliable multi-hour autonomy.

Strategic recommendation: adjust prompts/call strategy (don’t “Februaring”)

The speaker claims some respected leaders prompt as if models were earlier-generation capable (“prompting like we’re in February”).

Instead, they recommend:

  • Use more of the model’s available context window
  • Prefer “horizontal” instruction:
    • Tell the agent the overall problem and let it find files and make changes itself (instead of listing exact files)
    • Have the agent trigger build/simulation and verify, then deliver artifacts back to the user (e.g., a build pushed to phone, videos in the PR)
    • Move verification earlier/later depending on reliability

They argue this is how teams unlock the real benefit: fewer dumb failures during longer runs.

Community polling mentioned (agents watched or not)

The speaker references polling about whether people watch an agent while it runs:

  • They express disappointment that many watch only until the first output is badly wrong.
  • They suggest that “watch only if the first output is horribly wrong” can still be educational, because it helps infer failure causes (tools/files/signal tracing).

Main speakers/sources (as mentioned)

  • David Kramer (“Zeg”), founder of Sentry (the claim being criticized)
  • The video’s primary speaker/narrator (referred to as “Zeke” / “Zeie” at the end; exact real name not provided in subtitles)
  • Jamon (mentioned as framing preference for overnight autonomous work)
  • Maria (mentioned as contributing PR work and refining comments/traces)
  • AD guy (mentioned in chat)
  • Grapile / T-Rex (sponsor/tool described for sandboxed agentic code review)
  • Cursor (referenced in comparison; “Cursor 2/3” vs improved stability)

Original video