Is the Agent Harness Dead? — Jediah Katz, SpaceXAI

Summary

Jediah Katz argues that the agent harness—the software layer that connects and guides an AI model—is not dead, though its role is changing. Harnesses may need to compensate less for models’ coding weaknesses, but they remain essential for enabling new kinds of agent work and providing reliable infrastructure, security, and efficiency.

What an Agent Harness Does

Katz divides a harness into two layers:

• Execution: Runs the agent loop and manages state so work can continue.
• Context and orchestration: Selects models, tools, prompts, and context, and coordinates how agents work.

He describes the harness as the model’s “eyes and hands,” connecting it to computers, applications, and services.

Why Coding Harnesses Have Become Simpler

Katz cites studies comparing different harnesses. A UC Berkeley study tested seven leading models across several coding setups, including Pi, Codex, and Claude Code. An Apple–EPFL study examined custom harnesses on MLEBench. In both cases, harness choice had little effect on benchmark scores with newer models, though the studies found meaningful differences in task-completion cost.

Katz explains this shift through a data flywheel: harnesses first make more complex tasks practical, increasing the amount of useful work and feedback generated with agents. Models then learn skills that harnesses once had to supply—such as exploring a codebase, persisting through long tasks, and checking work—allowing some prompts and workarounds to be removed.

Why Harnesses Still Matter

Katz gives two main reasons:

1. They enable new tasks. Grokbot is presented as an autonomous colleague for varied intellectual work, not just programming. It uses long-running coordinator agents that delegate to other agents. Design challenges include creating presentations, booking travel, communicating appropriately, keeping users informed without overwhelming them, acting only as intended, and learning users’ work styles over time.
2. They provide capabilities models cannot absorb on their own.
   • Infrastructure: Long-running work needs cloud execution rather than relying on a laptop that may sleep or disconnect.
   • Trust and security: Access should be limited and actions safeguarded to prevent problems such as unwanted emails, exposed credentials, or excessive API bills.
   • Efficiency: Harness choices can affect cost and speed even when task quality is similar. Katz says tuning caching, token use, and model selection improved Grokbot’s cost and speed without sacrificing quality.

Katz also describes a “software factory” vision: systems that coordinate agents across the software-development lifecycle. One example starts with an incoming bug report, classifies it, launches a cloud agent to fix it, produces before-and-after videos as evidence, and uses another agent to monitor the deployment.

Product and Engineering Directions

Katz expects harnesses to change rapidly, with product development sometimes moving faster than model development. He highlights several approaches:

• Keep harness features easy to add and roll back, and limit users’ exposure to internal implementation details.
• Build reusable agent primitives rather than one-off integrations. For example, a subscription system can wake agents in response to events such as Slack messages, CI errors, or a scheduled time.
• Track temporary behavioral workarounds, called shims, centrally. A monitoring bot can propose removing a shim when it is no longer needed.
• Build at a higher level of abstraction, using agents—not just individual LLM calls—as the basic components of larger systems.

Prediction and Conclusion

Katz predicts that within six months, 90% of agents’ work will happen off-laptop.

His conclusion is that harness features that once patched gaps in coding intelligence may fade as models improve. Harnesses will nevertheless remain essential for expanding what agents can do and for providing infrastructure, safety, and efficiency.

Main Speaker and Sources

• Jediah Katz, formerly a long-tenured agent-harness engineer at Cursor and now at SpaceXAI, is the presenter.
• Research cited includes a UC Berkeley harness comparison and a study by Apple and EPFL. The transcript does not identify the studies by title.