The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI

Technological concepts & core argument

• AI coding agents massively increase code generation speed, but they do not proportionally increase review/verification speed, creating a new bottleneck at the “trust” stage (human approval still gates production).
• The central question is whether human code review can be removed or replaced by automated verification—and what “verification” should mean beyond merely passing tests.

Key data points and findings

Bottleneck shifts from writing to verification

A cited GitHub-based study (via economists) tracking 100,000+ developers found:

• Teams using autonomous agents wrote ~741% more code
• But they released only ~30% more software

Conclusion: verification became the bottleneck, because production still routes through people.

Human review limits (Cisco study)

Human code reviewers become less effective after:

• Reading more than ~400 lines at once
• Trying to “test” more than ~450 lines per hour (efficiency drops)

An example calculation suggests agent-generated PRs may require multiple business days of human review at that rate.

Testing ≠ acceptance (“merge eligibility”)

Multiple findings emphasize that passing tests is not the same as being acceptable for merge:

• Meter study: maintainers reviewed PRs that already passed SWE-bench

  • PRs were “merge-ready” only about half the time
  • Failures were often due to quality issues, hidden regressions, and scope/discipline—not necessarily correctness in the test suite

• Frontier Code / FrontierBench (Cognition):

  • Models scored ~88% on Swebench Pro
  • But only ~29% on the hardest “Would you accept this?” evaluation
  • Example mentioned: GPT-5.5 < 6% on acceptance-style criteria

Takeaway: the industry still lacks a universal metric for “would a human accept this PR?”

Product features & approaches used in industry (automated code review)

The talk frames a “status quo”:

• You can generate code automatically and run tests
• But humans still check merges
• Many companies aim to replace human reviewers with AI reviewer systems, often measuring success via human acceptance

GitHub Copilot / automated review at scale

• GitHub’s Copilot reviewer:
  • Performed ~60 million checks
  • Drives > 1 in 5 code reviews across GitHub
• Presented as broad industrial adoption rather than a pilot.

Cursor code reviewer architecture (multi-pass + remediation)

Cursor is described as using details consistent with:

• Multiple passes over each diff (e.g., 8 passes)
• Reviewer order and review passes can change outcomes, so systems iterate
• False positives are filtered out (otherwise trust degrades)
• The reviewer:
  • Analyzes diffs
  • Calls tools
  • Determines where deeper investigation is needed
• Behavioral tweak: the model is trained to be more suspicious by default (not “everything looks good, ship”)
• Cursor can generate a remediation patch, turning “review” into rewrite + diff for approval
• Cursor can also run code to prove whether findings are real, making review and rewrite increasingly overlap

Research corroboration: multi-stage verification increases quality

• Peking University research explored staged verification where the system retains what it can agree on across stages
• Reported up to ~44% improvement in review quality

The multi-pass idea is said to be rediscovered largely because false positives undermine automated review usefulness.

Other vendors and metrics

Systems mentioned include:

• CodeRabbit: reviewed >13 million PRs
• Graphite: builds a repository graph to infer how changes affect remote dependencies; evaluation based on proposals developers accept/reject

A common success metric across vendors is human acceptance rate:

• Cursor calls it a “resolution factor”
• Claimed improvement: ~52% → >70%

Attempts to remove humans entirely (and why they struggle)

“Loop design replaces verification” (agent-only pipelines)

• OpenAI is cited as demonstrating an internal product with no hand-written code, relying on agent loops.
• The speaker argues gaps likely remain because “how to do it” wasn’t fully open-sourced as a tutorial.

Experiments & caveats

• Anthropic (C compiler from scratch):

  • 16 agents built a compiler and compiled Linux without direct human intervention mid-process
  • However, the speaker stresses humans still authored the testing/verification infrastructure and controlled the system externally

• Bun port: Zig → Rust:

  • Agents migrated ~1M lines in ~6 days
  • Tests passed ~99.8%, but:
    • Another robot flagged a PR as “AI garbage”
    • Manual analysis found ~13,044 unsafe blocks vs ~74 in comparable human-written Rust
  • Conclusion: tests can miss “internal” risk assumptions; tests validate interfaces, not hidden safety claims

OpenAI’s “Codeex” framing: keep verification, shift it

Instead of removing verification, OpenAI:

• Uses an agent reviewer that reviews its own changes
• Re-invokes additional agents in loops until reviewers are satisfied
• Provides agents access to the logging stack

Lesson: verification doesn’t disappear; it becomes an agent-built verification system maintained by people.

Real-world reversal (Horty anecdote)

• Dexter Horty tried six months of releasing without people reading code
• Later retracted publicly: “I was wrong… It ended badly… had to rip out and replace significant parts.”

Security and “meta-review” concerns (automated reviewers are also attackable)

Automated review can be fooled:

• Anthropic’s security checker includes a disclaimer it is not immune to prompt injection
• A reported claim (March of that year):
  • Vulnerable code disguised in a commit message fooled an autonomous agent in ~88% of attempts vs ~35% for humans

If machines operate in a closed loop, bad code that “looks ready” can pass confidently—so humans may remain an inherently hard-to-bypass verifier.

Practical takeaway / suggested action (what to do now)

The speaker’s recommendation:

• Stop checking PRs with humans (framed as “wrong abstraction for 2026”)
• Instead, invest in building reliable verification tooling, including:
  • Codifying quality definitions, company/domain context, and rules
  • Running agents under those codified constraints

Human effort should shift upstream:

• Humans define the rubric/standards
• Humans audit the verification system itself (meta-review), rather than performing line-by-line review.

Main speakers / sources

• Main speaker: Laurie (“Lori” / “Lori Voss” as referenced) — Head of Developer Relations at Arize AI (also mentioned as co-founder associated with npm)

• Referenced/credited sources and entities:

  • Economists’ GitHub study on agent usage and verification bottlenecks
  • Cisco research on human code review limits
  • Stripe, Anthropic (Fable / Ruby migration claim)
  • Bun (Zig → Rust migration claim)
  • Peter Steinberger (OpenClaw), Andrej Karpathy (OpenAI) arguments
  • OpenAI (agent-only coding, Codeex)
  • Meter (SWE-bench merge-readiness review finding)
  • Cognition (Devon, Frontier Code / FrontierBench)
  • GitHub (Copilot reviewer)
  • Cursor (architecture details)
  • Peking University (multi-stage verification research)
  • CodeRabbit, Graphite (vendor implementations)
  • Dexter Horty (human-avoidance attempt/retraction)
  • Sarah Goh and Sarah Kuo (investor + report commentary cited)
  • Nicholas Carlini (Anthropic) (compiler-from-scratch experiment)
  • Arize AI (company mentioned as context; speaker says not to promote it)