Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp

Summary

Ben Holmes, a DevEx engineering lead at Warp, describes a feedback loop for helping software-development agents improve their skills and configuration. In a “software factory,” agents handle development stages such as investigating tasks, implementing changes, and preparing pull requests. Factory engineering focuses on improving the system that runs those agents.

The Improvement Cycle

1. Collect agent work centrally. Gather conversations from tools such as Claude Code or Codex, either from local histories or through a shared cloud runner or sandbox.
2. Capture complete traces. Include the conversation transcript, tool and MCP calls, and the resulting code or pull-request diff. Tool activity matters because agents may work around bad instructions without making wasted effort or failed attempts obvious.
3. Score conversations against explicit rubrics. Evaluator agents can assess criteria such as redundant tests, verbosity, efficiency, code quality, and test coverage. The rubric should define what counts as a pass or failure and can be customized.
4. Look for recurring problems. Reports explain low scores. A scheduled agent can review failed evaluations, identify patterns, and suggest changes to skills or configuration.
5. Measure whether changes help. Tracking scores over time can show whether a configuration or skill change improved performance.

Holmes notes that agents can appear successful while making unnecessary tool calls or struggling with outdated instructions. For example, an agent might repeatedly try different ports or spend time figuring out how to create subagents without revealing the underlying configuration problem.

Tools and Examples

• Skill Doctor is a skill that users can run locally to review past conversations from agents such as Claude Code and Codex. It produces scores and reports, then suggests skill improvements. Examples included clearer instructions for finding test ports and more explicit guidance for writing changelogs.
• Warp Factories can run the process continuously: evaluate conversations, periodically analyze failures, and propose pull requests to improve agent settings. One example identified missing configuration for creating subagents, which had led to wasted tool calls.
• Assessment dashboards help teams relate configuration changes to performance over time.

Guide and Tutorial Points

• Centralize agent histories so team-wide patterns can be analyzed.
• Include transcripts, tool activity, and outcomes—not just the final answer.
• Start with a concise, clearly defined evaluation rubric.
• Evaluate every conversation or use sampling to control costs.
• Use recurring failures to propose targeted skill or configuration changes.
• Review suggested changes and track scores afterward to see whether they worked.

Source

Ben Holmes, DevEx engineering lead at Warp, presented and demonstrated Warp Factories and Skill Doctor.