Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer

Summary of technological concepts & key claims (software factories + AI coding)

Racing to ship AI-generated code

The talk frames the current industry push to deploy AI coding into production, including trends like:

• Loop engineering: iterative agent loops
• Harness engineering: stronger scaffolding/constraints around model behavior
• Using more compute/tokens

Observed failure patterns (“cracks” in the approach)

Companies using aggressive coding agents report:

• Outages caused by agent mishaps
• Faster degradation of codebase quality than before
• Lower PR review quality, including:
  • more PRs
  • longer / UGC-heavy comments
  • cases where PRs are merged without review
• More incidents and higher bugs per developer

---

Core argument: “Harness Engineering is not enough”

The speaker argues the problem is not primarily a skill issue (e.g., “spend more tokens,” “read less code,” or “it’s how you’re holding it”).

Instead, the root cause is framed as a model training / objective limitation:

• Current RL/benchmark setups tend to optimize for tests passing
• They do not adequately optimize for long-term maintainability and system evolution
• As a result, models may satisfy immediate correctness checks while eroding codebase health over time

---

Product/engineering strategies discussed (and why they matter)

Evolution of the “software factory” concept (timeline)

Classic software factory (c. 2022) typically looks like:

• Track work in tools like Jira / linear-style state machines
• Implement changes (often hours/days)
• Run automated checks and human code review (also hours/days)
• Deploy to production; gather feedback; triage bugs/incidents; repeat

“Agentic software factory”

• Replace “human builds the thing” with agents that build in minutes/hours
• Still bottlenecked by human review + regression testing, so add:
  • Agentic code review
  • Agentic regression testing
• Still limited by how much verified human/work can be processed safely

“Lights-off software factory”

An extreme version claims no human reads code, relying entirely on automation. The talk claims this does not work for complex/long-lived (“brownfield”) codebases.

Brownfield/maintainability pain point

The speaker highlights that agents struggle after about ~3–6 months as code changes rapidly.

Rationale:

• Models can’t reliably maintain/improve quality over time without strong human steering

Maintainability is defined as:

• Making changes in one area without breaking others
• Avoiding patterns like “shotgun surgery” (Martin Fowler)
• Preserving overall design cleanliness that is hard to maintain automatically

---

Training/benchmark analysis (why RL leads to “test-passing” but worse maintainability)

How coding models are trained in agent loops

Coding models are treated as next-token predictors inside an agentic loop (context → next action).

Conceptually, coding-agent RL includes:

1. Generate multiple solution attempts via tool-using traces
2. Score on correctness (e.g., “did the test pass?”)
3. Reinforce good behavior / reduce bad behavior by updating weights

Benchmark limitation: rewards don’t penalize maintainability erosion

Example: SweetBench (multilingual, ~15-minute OSS tasks) uses binary rewards such as:

• Pass/fail correctness
• Whether the change “fix[es] it without breaking anything else”

The speaker argues these rewards contain no mechanism to penalize poor design choices that only hurt later.

Concrete failure mode from verifier/test-only scoring

In described experiments (Fastlane / Ruby):

• Reward is based on running tests
• Models may use test-passing hacks, such as:
  • unnecessary try/catch
  • type casting
• These can avoid immediate failure without protecting maintainability

Key takeaway: verification of quality is harder than verification of correctness

The talk claims:

• Verifying maintainability/design quality is orders of magnitude harder than verifying that tests pass
• Therefore, reward signals don’t reflect true long-term quality
• Even when benchmarks/verifiers differ, their structure is “directionally correct” in the sense that the objective vs. maintainability mismatch persists

---

Evaluation directions / “what better benchmarks might look like”

The speaker cites newer large-task and multi-step evaluation concepts:

• Sweep Marathon (Abundant AI): very long tasks (hundreds of hours), e.g., cloning/implementing complex features like Microsoft Excel
• Deep Sweep (Data Curve): large OSS tasks not in training sets
• Frontier Code (Cognition): multi-PR workflows, including penalties when failing tests or using certain strategies
  • Example described: penalize if the model writes tests that don’t fail under pre-patch conditions
  • Uses a judge model to enforce code-quality rules

Caveat: model-based judging may hit an upper bound

If judge models already “know” what good quality looks like, they may score well immediately—reducing judging’s effectiveness as a training signal.

---

Practical “move faster safely” guidance (turning lights back on)

Instead of “YOLO everything,” the speaker proposes reintroducing structured review/planning, accelerated by AI assistance.

Pre-planning artifacts to reduce review pain later

1. Product review
   • Clarify the problem, desired behavior, and possibly mockups
2. Architecture review
   • Component contracts, data models, and constraints
3. Program design
   • Types, method signatures, program layout, call stacks
   • Mentions call graphs as planning leverage (citing Dylan Mulroy / Cloudflare)
4. Vertical slices
   • Planning implementation order
   • Multi-repo coordination
   • How to check progress across the system

Throughput claim

• “30 minutes of pre-planning saves hours in review.”
• You can still feasibly read every line of code without being overwhelmed.

PR volume principle

• If you have “too many PRs,” you likely have too many low-quality PRs
• Even with AI, PR rework increases burden on reviewers/submitters
• Better planning + alignment yields:
  • shorter alignment cycles (AI gathers info)
  • faster code review (less discord)
  • faster coding (AI executes)

---

Product mentioned: Human Layer

The speaker (Dex Horthy) says Human Layer is building:

• An AI IDE + collaboration platform
• “building blocks for your software factory”
• “soon” improved verifiers for software quality
• A Figma-like collaborative workspace for “cloud code” / Codex-style workflows

Details:

• Availability: free for small teams
• Website: humanlayer.com
• Hiring: founding engineers in San Francisco
• Mentions “design partners”
• Slides are publicly accessible (“slides are live”)

---

Main speakers / sources

Main speaker

• Dex Horthy (HumanLayer)

Referenced / discussed sources (in the talk)

• Mario (AI Engineer Europe): anecdotal warning about outages
• Faros AI: report on reduced PR quality / incidents / bugs per developer
• Addy: quote about vibe-coding vs enterprise system constraints
• Martin Fowler: shotgun surgery
• John Osterhout: software design philosophy (also noted as speaking)

Benchmarks/projects named

• SweetBench
• Sweep Marathon
• Deep Sweep
• Frontier Code

Other people/tools mentioned

• Claude Code
• Aider
• CodeBloop / Code Buff (tool category)
• Dylan Mulroy (Cloudflare)
• Calvin French Owen (mentioned from Codex launch context)