I Stopped Coding and Started Architecting Agents (And You Should Too)

Summary

• Core idea: “Agentic harness engineering” as a way to improve code quality with AI agents

  • The speaker argues that modern generative/agentic coding tools shouldn’t be used like simple autocomplete.
  • Instead, you should constrain the agent so it reliably produces high-quality code and tests.
  • The goal is to create a feedback + guidance “flywheel” where:
    • early improvements come from prompting/iteration,
    • those improvements then update the harness, and
    • over time the codebase (and thus the outcomes) get better repeatedly.

• “Harness” = guides + sensors (constrain behavior before and after code generation)

  • Drawing on Birgitta Buckler’s terminology:
    • Guides (feedforward): advice the LLM uses before writing code (e.g., unit test design rules like Arrange-Act-Assert).
    • Sensors (feedback): deterministic checks/scripts that examine after code is written (e.g., file length limits, code smell detection, lint-like checks).
  • The harness helps the agent:
    • produce code that matches team preferences, and
    • generate tests that are better designed (not just “passing”).

• Why “make your own harness” instead of copying one

  • The speaker recommends teams learn/build their own harness because they:
    • can tailor it to their codebase and standards,
    • can adapt it as requirements evolve, and
    • avoid uncertainty/lock-in from an unknown “downloaded” harness.

• Problem addressed: legacy/unsafe-to-change code

  • The approach is presented as especially useful for teams with:
    • legacy code
    • business-critical logic
    • limited ability to refactor safely
    • weak or missing unit tests
  • Existing coaching on unit tests and incremental design improvements is extended by adding an AI harness so improvements compound rather than repeat old mistakes.

• Key mechanism: don’t let the agent copy bad patterns

  • If the codebase starts with poor design and weak tests, the agent may learn those patterns.
  • To counter this, the harness should “steer” the agent toward good design styles, starting with a strong example (e.g., one well-designed unit test).

Implementation details

• Agent capabilities

  • Some agents have a “skills” mechanism that can automatically inject guides into the LLM context for relevant tasks.
  • If the agent doesn’t support skills, the guide can be manually included in prompts.
  • Sensors may be integrated via agent hooks, but if unavailable, the same effect can be achieved by running sensor scripts in the build step or linter.

• Practical workflow: update harness continuously

  • When a team improves a test or design task successfully:
    • update guides and sensors to reinforce the preferred patterns.
  • Over time:
    • fewer guide additions are needed because the codebase provides more “examples to copy,”
    • and the models may require less constraint.

• Harness maintenance: balance adding vs removing

  • The speaker warns harnesses can grow into large, expensive, possibly unhelpful token-heavy documents.
  • An Anthropic evaluation approach for checking what should be in agents.md is described:
    • re-run previously completed tasks with vs without harness content,
    • compare outcomes to accepted results,
    • though it’s difficult due to needing a large statistically relevant sample and significant time/resources.
  • More practical advice:
    • treat harness updates as part of normal tasks,
    • encourage removing outdated or ineffective parts,
    • optionally do spot checks (run prompts with/without harness updates), though this may be too costly regularly.
  • The team should “own” the harness like other critical tooling (build scripts, shared manual tests).

Tutorial/support mentioned

• Upcoming webinar: Test-Driven Development with an “Agilitic/agentic” harness (signup link in show notes).
• Patreon is mentioned as an additional way to follow along.

Main speakers / sources

• Speaker: Emily Bache (creator of “SAM method” / “Sam An Coaching”; “technical coach”)
• Referenced sources:
  • Birgitta Buckler (terminology and explanation of harness engineering: guides and sensors)
  • Anthropic article about evaluating agents.md contents (with/without harness comparison)
• Sponsors acknowledged: Equal Experts, Transfix, Octopus Deploy