Video summary

What The Best Engineers Solve After The Code Review Bottleneck

Main summary

Key takeaways

Business

Business problem & context (what’s bottlenecking delivery)

  • The video centers on moving beyond PR-code-review as the dominant software-delivery bottleneck.
  • Example scale of the organization/engineering operation:
    • ~3,500 engineers (growing) and ~4,000+ total engineers contributing daily
    • ~8,000 repos (spanning monorepos to microservices; from legacy monoliths to very new services)
    • Lots of automation/harness tooling already exists, but it’s insufficiently standardized across heterogeneous workflows.
  • Core thesis: delivery systems are “only as fast as the slowest link.”
    • Bottlenecks will move (e.g., from code review to other steps like security checks, release readiness, or pipeline determinism).

Strategy: “Verification as a stack” (agents + harnesses)

What the team is building

  • A generalized verification system that starts with code review but extends into a multi-layer stack of checks (often agentic).
  • Goal: predict/detect the next bottleneck before engineers hit it, so throughput doesn’t collapse elsewhere after code review is improved.

Verification stack (playbook-style layers)

  • Code review verification
  • Security verification layer
    • Agents detect issues earlier than traditional scanners
  • Compliance/regulatory verification agents
    • App store / platform compliance checks
    • Financial/regulatory compliance checks
  • Agent outputs feed downstream agents
    • This is a pipeline of capabilities, not a single PR gate.
  • Different owners per layer
    • Not everyone can “own everything”; interfaces and boundaries are required so each domain can control its workflow without breaking the overall system.

Framework/pattern referenced

  • High Output Management (Andy Grove / Intel metaphor)
    • “Fast bread tossing doesn’t help if eggs take 10 minutes”—i.e., optimize the system throughput, not isolated steps.

Operational approach: adapt verification to workflow + business constraints

Concrete example: mobile vs web release verification

  • Mobile engineers must pass app store verification and release constraints (centralized release + platform gates).
  • This differs from pushing changes to web servers (less platform gating).
  • Implication: “one harness/one workflow” won’t work; you need multiple harness flavors per discipline.

Customer/business alignment (uptime + pragmatism)

  • Leaders tailor engineering practices to the business’s reliability needs.
  • Example uptime target discussed: “9s” (99.999% type reliability).
  • Message: don’t copy Google/OpenAI/others blindly; your verification strictness must match your business risk tolerance (e-commerce trust model and uptime).

Managing cost as a first-class constraint (agents/compute)

  • Agents add latency and cost; they can’t run on every change.
  • Cost becomes a “bottleneck of its own,” similar to how cloud economics changed after early optimism.
  • The team remains open to multiple technical futures:
    • Faster inference via cloud providers
    • Local models improving enough for laptop-scale
    • Better routing / model selection strategies (“model routing” mentioned)
  • High-level investment/market note:
    • Concern that economics (margins) will reshape what’s viable; cost structures may shift dramatically.

Measurement system: “debugging the SDLC” (from targets to investigation)

The playbook: build an event-driven measurement foundation first

  • Four years prior, they built an event system to understand bottlenecks across the SDLC (software development lifecycle).
  • Purpose: investigation tool, not vanity targets.
  • Called “debugging the SDLC.”

What they measure (signals and events)

They collect events across:

  • Code artifacts
    • Commits and document metadata (with context)
  • Repo/component context
    • Monorepo component correlation downstream
  • CI/CD pipeline events
    • Build events, deployment events
    • Release granularity (e.g., which components ship together)
    • Release coordination/waiting when multiple components are needed
  • Ticket/issue linkage
    • Tickets connected to changes; labels and workflow metadata
  • Review processes
    • Review gates and related workflow info
  • Agent observability
    • Mainly cost/token usage (e.g., number of API calls)
    • Emerging need to track effectiveness vs cost
      • e.g., context length (“more context” can reduce effectiveness while increasing cost)

KPI / outcome framing (examples explicitly mentioned)

  • Separation of:
    • CTL vs Blue vs Feature deployments
      • Keeping the lights on / business-as-usual
      • Foundational work
      • Customer-visible feature output
  • They answer the AI adoption question:
    • “Do people that use AI deliver more features?”
    • Their answer: “yes or no—depends”, because measurement separates types of work and constraints.

Anti-gaming principle (Goodhart’s Law / cobra-farm story)

  • Avoid static dashboards becoming targets.
  • Principle: if a metric becomes a target, teams can game it and lose signal usefulness.
  • Emphasis:
    • Investigative metrics (leaders query/diagnose)
    • Evolving “next-generation” insights to surface bottlenecks organically

How they translate engineering metrics to executive outcomes (bridging tech ↔ business)

“Co-create queries with business leaders”

  • Practical process:
    • Engineers/business leaders run SQL together in workshops to build intuition.
  • Example query-style question:
    • “How many changes were deployed together in the past month?”
  • Tools:
    • Previously Tableau/Looker
    • Now AI-assisted querying so executives can ask questions in-room and get results faster.

ROI approach: give leaders autonomy rather than prescribing a single outcome

  • They don’t claim one universal ROI formula.
  • Different leaders prioritize different outcomes:
    • Quality
    • Velocity
    • Fewer incidents / stability
  • The system’s job is to provide leaders with the ability to see what improves and where bottlenecks are.

Concrete operational recommendations and starting points (actionable takeaways)

  • If a company doesn’t yet have an event store:
    • Start with workflow reality (understand how engineering work truly flows)
    • Connect tools into an event store when possible
    • Don’t assume the “right” metrics—learn and pivot
  • If small enough:
    • Begin measurement work with 2–4 people over ~2.5 years of iterative learning (internal reference example)
  • If choosing what to measure first:
    • Look for: “where am I slow?”
    • Identify bottlenecks using real event traces
    • Simple startup diagnostics (even without full instrumentation):
      • Check review cycle times
      • Check CI pipeline durations (can be days)

Adoption model: engineers pull tools in instead of pushing migrations

  • With AI tools, adoption is easier than traditional tooling migrations:
    • Engineers ask for interfaces/skills early (e.g., “Do you have an MCP for this?”)
  • However, governance still matters:
    • They centralize parts that require safety/auditability (“enterprise harness” direction)
    • Concern: teams building outside guardrails could create compliance risks.

Agents + human workflow: measured via context, not direct “correlation”

  • They don’t correlate “human vs agent” by who parallelized what.
  • They track:
    • Context: how many agents vs humans at scale
    • Impact on developer experience and outcomes

Developer Experience (qual + quant)

Surveys as a complementary measurement layer

  • They use a survey tool called GET.
  • Cadence: every ~6 months
  • Response rate: ~92–93% (highlighted as a standout strength)
  • They manage survey effectiveness by:
    • Explaining what they’ll do with results
    • Showing prior actions based on previous surveys
    • Running constructive interpretation cycles with leaders and teams

Examples of survey-driven improvements

  • Review/release automation issues:
    • Engineers guided tooling improvements
  • Deployment context help:
    • Engineers asked for an AI capability to locate where deployment is in the pipeline
      • e.g., “ask Claude to figure out where my deployment is in the pipeline”

OpenAI/Frontier Labs relationship (enterprise partnership, shared learning)

  • They described working with major frontier labs (example name: Frontier Labs).
  • Relationship model:
    • Labs also exploring what works at enterprise vs smaller scale
    • Labs provide cost-saving suggestions, but aren’t always sure how to achieve them at their scale
  • They aim to collaborate with multiple labs because it’s unclear who wins (and models become commodity-like).

Key “playbooks / frameworks” explicitly or implicitly used

  • High Output Management (Andy Grove)
    • Optimize system throughput, not local performance.
  • Debugging the SDLC
    • Event system for investigating bottlenecks across CI/CD + workflow + review.
  • Stacked verification (capability layers)
    • Code review → security → compliance → app/platform constraints, with agent chaining.
  • Goodhart’s Law mitigation
    • Avoid turning metrics into fixed targets; prefer investigative/iterative querying.
  • Co-creation with business
    • SQL workshops + AI-assisted executive querying.

Key presenters/sources

  • Presenter (guest): Amos (software/engineering platform leader; full last name not provided in subtitles)
  • Host/Interviewer: Patrick (also unnamed last name in subtitles)
  • Referenced source/work: Andy Grove, High Output Management (Intel metaphor)

Original video