AI Models Are Eating Their Harnesses. Now What?

Summary

The video examines whether AI agents still need elaborate harnesses—the software around a model that lets it use tools and operate in real environments. Its conclusion is that better models may make some scaffolding unnecessary, but systems still need infrastructure for execution, context, state, permissions, recovery, and evaluation.

Core Concepts

• Models, agents, and harnesses are different things: A model is the neural network; an agent uses a model to carry out a task across multiple steps; and a harness runs and supports that process. Agent-building platforms provide reusable components, but even a small custom program can serve as a harness.
• A minimal coding agent: A roughly 25-line example from Focus AI demonstrates the basic tool-use loop. The program tells the model about a bash tool, executes commands the model requests, returns the results to the conversation, and repeats until the model finishes. The model decides what to try; the harness executes it.
• Runtime versus context orchestration: The runtime launches the agent, executes tools, and manages its environment and state. Context orchestration decides what information, tools, and prior work the model sees. For example, it might provide relevant log errors rather than an entire log file.
• More components are not automatically better: Multiple agents can work on separate investigations in parallel, but they may duplicate work and incur extra costs. Persistent memory can preserve useful lessons, but systems must distinguish verified findings from hypotheses. Summaries can also distort earlier work—for example, turning “the migration probably caused the slowdown” into a definitive conclusion.
• Long-running tasks need durable state, not continuous thinking: An agent can save what it has done, what remains, and what event it is waiting for, then resume when an approval or external event arrives. This persistent task state is distinct from the context sent to a model during a particular call and from memory intended to store reusable information across tasks.
• Recovery matters: The video describes systems that preserve useful artifacts outside a temporary execution sandbox, allowing work to resume in a fresh environment after a failure.
• Capability and authorization are separate: A model may identify a suitable action without being permitted to take it. Examples include limiting an agent’s access by identity and requiring human approval before a sensitive financial change. Another approach keeps real API credentials in a proxy, gives the agent dummy values, and applies controls to requests. These safeguards reduce risk but do not eliminate every security vulnerability.
• Agents can create tools and environments: Dynamic systems can let an agent build, register, use, and remove a task-specific tool, or provide a suitable runtime for its work. This raises questions about validation, permissions, resource limits, and whether created capabilities should persist.
• Evaluate the harness rather than assuming it helps: Compare configurations on representative tasks, measuring correctness, time, token use, cost, and constraint violations. Research discussed in the video found that more complex harnesses did not necessarily improve results on the tested tasks, while costs could differ substantially. The “harness tax” includes overhead from extra instructions, model calls, tool use, and orchestration.
• Improving models can make scaffolding obsolete: The video cites Cursor reducing parts of its system prompt—including instructions to use subagents—after the model became better at deciding when to delegate. It describes a “data flywheel” in which harnesses enable complex work, experience contributes to training, and improved models eventually need less support. However, broader autonomy and longer tasks can demand more infrastructure, not less.

Practical Guidance

Begin with a model, the necessary tools, and a simple execution loop. Add context management, persistent state, event handling, permissions, or delegation only to address observed needs. Compare additions against a single-agent baseline, and re-evaluate prompts and orchestration whenever the model changes.

Preserve important task status, procedures, and evaluation data so they remain usable if the model or harness is replaced. Track temporary fixes so outdated “crutches” can be removed.

Main Speakers and Sources

• Ksenia, host of Attention Span from Turing Post.
• Jediah Katz, associated with Cursor and Grok Bot, on runtime and context orchestration, harness overhead, and models absorbing scaffolding.
• Aman Singh of Arca, on event-driven, long-running financial workflows.
• Aparna Dhinakaran of Arize, on Signal, persistent artifacts, and resuming work across environments.
• Belinda Neal and Keon Shahab of Anthropic, on skills, MCP connections, agent identity, access restrictions, and human approval.
• Philipp Schmid of Google DeepMind, on protecting credentials with proxies and runtime controls.
• Shigma, associated with DeepSeek’s dynamic tool lifecycle, on creating and managing tools during execution.
• Akshat Bugna of Modal, on dynamic runtimes and task-specific environments.
• Delon Ronick of News Research, on retaining control of agent workflows and accumulated knowledge.
• The video also references research comparing harnesses from UC Berkeley, Apple, and EPFL.