Video summary

Designing & Building PR Review Multi Agent System (3 Hours Build)

Main summary

Key takeaways

Technology

Overview

This video is a long, “build-from-first-principles” tutorial for designing and implementing an AI-assisted PR (pull request) review multi-agent system. The focus is production-ready behavior rather than a simplistic “send diff to an LLM and comment” approach.

Core Product/Tech Idea

The system’s goal is selectivity: only surface high-value findings worth a senior engineer’s attention—rather than trying to maximize coverage.

How the senior reviewer is modeled

  • A senior reviewer performs multiple concern-driven reviews (e.g., security, correctness/quality, testing, documentation).
  • The review is done with:
    • skepticism
    • evidence
    • confidence

Architectural principles

  • Precise triggers
    • Example: a GitHub PR opened webhook.
  • Structured outputs
    • Findings include location, severity, confidence, and rationale/evidence.
  • Human-in-the-loop (HITL) gates
    • Routing based on confidence/severity.
  • Grounding + retrieval (RAG)
    • Prevent hallucinations from prompt-only context.
  • Reliability / fault tolerance
    • Retries, circuit breakers, deduplication, timeouts.
  • Observability / auditability
    • An event/trace “event spine.”
  • Cost/economics dashboards
    • Track token spend and operational efficiency.

Criticism of Existing “Diff-to-LLM” Approaches

The tutorial critiques approaches that only send code diffs to a single LLM:

  • Lack auditing/proof
  • Tend to hallucinate with confidence
  • Can’t reliably apply multiple specialized mindsets
  • Struggle when repository context is too large (context window limits)

Multi-Agent Design (Fan-Out / Fan-In)

The system uses a parallel multi-agent setup:

  • Security specialist
  • Quality/Correctness specialist
  • Testing specialist
  • Documentation/Readability specialist

An aggregator/merger combines their outputs.

Structured “findings” per specialist

Each specialist returns findings that include:

  • concern / agent type
  • severity/category (including critical vs informational)
  • exact file + line location
  • confidence score
  • rationale/evidence (auditable reasoning)

Retrieval & Grounding (“Context Engineering”)

The grounding problem is addressed by retrieving only what’s relevant:

  • Retrieve relevant code slices tied to the PR diff (not the entire repo)
  • Add PR-relevant repository context, past review context, and conventions

Memory types described

  • Semantic memory
    • Vector/embedding similarity over code
  • Episodic memory
    • Time-ordered past findings/reviews
  • Procedural memory
    • Team conventions and decision records

Human Gates and Routing Logic

Confidence-driven routing

  • If confidence is high and findings are non-critical → auto-post review
  • If confidence is low → escalate to a human approval queue
  • If findings are critical (e.g., security) → always escalate

Disputes are treated as training/learning inputs, but only after checks.

Failure Mode Engineering (Reliability Layer)

The tutorial emphasizes explicit handling of failure modes, including:

  • Hallucinations
    • Require citations/proof + confidence threshold + grounding
  • Model drift
    • Monitoring + alerting + periodic retraining/prompt updates
  • API/tool timeouts
    • Retry/backoff + graceful degradation + circuit breakers
  • Tool/API failures
    • Fallback paths
  • Orchestration deadlocks
    • Timeouts per node so the aggregator doesn’t wait forever
  • Feedback poisoning / bad feedback
    • Minimum evidence thresholds + feedback decay
  • “Almost right” misattribution
    • 90% correct but subtly wrong → confidence + random audits
  • Escalation overload
    • Human bottleneck → capacity planning + prioritization
  • Duplicate processing / event retries
    • Deduplication (e.g., idempotency keys)

Output Contract (Structured “Finding” Object)

The system defines a strict data shape for findings flowing through components:

  • agent type / concern raised
  • severity + category
  • confidence score
  • rationale/evidence text
  • file/line (precise location)

This structured contract supports:

  • auditing
  • dispute resolution
  • routing to humans

Ingress + Workflow Execution Architecture

Ingress handler (FastAPI-style)

  • Validates GitHub webhook signatures
  • Checks idempotency / unaltered mechanism
  • Quickly acknowledges GitHub to avoid webhook timeouts
  • Then queues the PR review job

Queue / job system

  • Uses Redis + ARQ
  • Discusses Rabbit-like concerns conceptually
  • Notes that durable dedup/retry state should move fully to Redis-backed durable storage

Orchestrator

  • Uses LangGraph for MVP graph workflow:
    • fan-out to 4 specialists
    • fan-in to merge results
    • checkpointing/recovery
  • Mentions an abstraction so orchestration could be swapped later (e.g., Temporal)

Technology Stack Emphasis (Tiger/Data + Postgres Unification)

A major practical focus is storage and observability using TigerData/TigerCloud (managed Postgres-compatible), along with extensions:

  • Use one durable Postgres-compatible system for multiple “data shapes” to avoid multiple DB operational overheads:
    1. Memory / vector search over code chunks (e.g., PGVector / PGVectorScale)
    2. Time series / event spine for observability (hypertables)
    3. Continuous aggregates for dashboards (cost per minute, P95 latency, tokens, etc.)

Performance features highlighted

  • PGVectorScale and DiskANN-style indexing for large-scale vector retrieval
  • HyperTable storage for events/agent traces
  • Continuous aggregates to avoid scanning massive raw event tables on each dashboard refresh

“Genesis Kit” and Harness-Based Coding (Tutorial Framework)

The video introduces Genesis kit (an AI coding harness/loop framework).

Key principle

Don’t just “generate code”; enforce milestones with:

  • demos
  • invariants
  • independent verification

Gate/checkpoint approach

  • Define a “done” spec that agents cannot modify
  • Verification steps using an independent verifier agent
  • Debug/retry loops when the verifier finds issues

Emphasis: without harnesses, AI coding becomes unreliable.

Demonstrated Implementation Milestones (High-Level)

The walkthrough includes early milestones:

  • M1: webhook ingress contract

    • signature validation
    • JSON parsing behavior
    • HTTP status decisions (e.g., returning 400 vs 500)
    • idempotency/dedup checks
    • initial job enqueueing

  • M2: tiger data provisioning

    • creating necessary tables/extensions in TigerCloud
    • enforcing DB invariants (append-only/immutability via hypertable setup)

Additional milestones are described conceptually (M3+), such as:

  • event spine append-only logging for every action
  • orchestration fan-out/fan-in
  • retrieval + RAG chunking
  • HITL gates and posting back structured PR review comments
  • dashboards for trace viewing and cost economics

Guides / Tutorial Takeaways

The tutorial emphasizes:

  • Start with first principles and map the “mess” of how humans review today.
  • Design around:
    • trigger + output contract + selectivity
    • grounding via retrieval
    • multi-agent separation of concerns
    • proof/audit trail via event logging
    • reliability engineering for production readiness
  • Use harnesses and independent verification per milestone rather than trusting the coding agent blindly.

Main Speakers / Sources (As Stated/Implied)

Primary speaker

  • The course author/instructor (speaks throughout; references “Genesis kit” and prior system-building experience).

External tech sources referenced

  • GitHub webhooks / GitHub PRs
  • OpenAI (and mentions “Entropic”/Anthropic API keys)
  • LangGraph
  • Redis + ARQ
  • TigerCloud / TigerData
  • FastAPI (webhook ingress)
  • Upstash (Redis hosting mentioned)

Original video