Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification

Main ideas (verification for self-improving AI agents)

• The lecture focuses on verification: how an AI system can automatically choose correct answers during/after generation, addressing the generation–verification gap (models can generate plausible answers, but reliably selecting the correct one is hard).
• It presents a progression across ~4 papers, showing how verification improved over time:
  1. Train verifiers for correctness on math (GSM8K), using labeled generated solutions.
  2. Improve verification by supervising reasoning processes step-by-step (PRM vs. outcome reward).
  3. Reduce human labeling by automatically generating step labels and using PRM within RL.
  4. Reduce the generation–verification gap using ensembles of weak verifiers and weak-to-strong aggregation (Beaver).

---

Paper 1 (OpenAI, 2021): Training Verifiers to Solve Math Problems (GSM8K) + “verifier as rubric”

Core concepts

• LLMs can produce wrong-but-confident solutions (“hallucinations”), so the system needs a verifier that predicts whether a candidate solution is correct.
• A verification model outputs the probability (score) that a solution is correct.
• Key benchmark: GSM8K
  • ~8,500 grade-school math problems
  • Emphasis on multi-step reasoning
  • Solutions are collected in natural language (not just symbolic math)

Training methodology (detailed)

• Given:

  • A question (Q_i)
  • A generated solution (S_i) (from a generator/LLM)
  • A binary label: correct vs incorrect (based on ground truth; problems are human-supervised so final answers are known)

• Training data creation:

  • For each problem, use repeated sampling to generate 100 solutions per question.
  • For each generated solution, label it as correct/incorrect by comparing with ground-truth final answers.

• Verifier training:

  • The verifier is a language model with a small scalar head that predicts correctness.
  • Uses two losses:
    • Binary correctness loss (correct vs incorrect)
    • Language modeling / token prediction loss (next-token objective) to keep the verifier’s representations grounded
  • Token-level masking: tokens corresponding to the question are masked; optimization focuses on the solution portion.
  • Training schedule described:
    • Fine-tune the generator for 2 epochs
    • Sample 100 completions per question, label them
    • Train the verifier for 1 epoch on that labeled set

Test-time procedure (detailed)

• For a new question:
  • Generate many candidate answers (again via repeated sampling).
  • Score each candidate with the verifier (using log-probability / correctness score).
  • Select the highest-scoring candidate as the final output.
• Token-level detail mentioned:
  • The verifier can produce per-token correctness signals, but the final decision for the whole solution is made by:
    • Taking the last token’s correctness label/score as the overall correctness.

Ablations / comparisons (high-level outcomes)

• Verifier vs baseline “direct fine-tuning”:
  • Verification helps more as verifier training set size increases.
  • With small datasets (e.g., < ~1000 labels), verification helps less.
• Generator/Verifier size tradeoff:
  • Better results when using a larger generator and smaller verifier than the reverse (noted as intuitive).
• Test-time scaling limit:
  • Increasing the number of completions helps up to about ~400.
  • Beyond that, verifier ranking becomes less reliable, and accuracy may drop (interpreted as reduced ability to distinguish closely competing candidates).

Key lesson

• A verifier can function like a rubric, enabling systems to pick correct outputs from many samples—especially when verifier training data is sufficiently large.

---

Paper 2 (OpenAI): “Let’s Verify Step by Step” (PRM vs ORM, process supervision)

Core concepts

• Problem: If a model makes an early wrong step, the final answer can fail—even if the overall output looks plausible.
• Two reward modeling approaches:
  • Outcome reward model (ORM): scores the final answer correctness.
  • Process reward model (PRM): assigns rewards to each step.
• PRM is trained with process supervision from humans:
  • Humans annotate each intermediate step as correct/incorrect relative to a correct reasoning path.

Training methodology (detailed)

• Setup:

  • Generator produces a solution as steps: step 1, step 2, …, final answer.
  • Ground truth is available to label whether the final answer is correct.

• ORM training:

  • Label correctness based on whether final answer matches ground truth.
  • (Token-level decision mentioned: final token score used for ORM scoring.)

• PRM training:

  • Humans label each step as correct/incorrect/neutral.
  • Final step reward for a completion is computed (described) as the product of stepwise rewards/probabilities.

• “Process supervision” yields a stepwise dataset:

  • Mentioned dataset size: PRM 800K (step-level labels, open-source)

Test-time procedure (high-level)

• The system scores step-by-step completions using PRM and selects/uses candidates with higher process scores.
• Emphasis: PRM can better avoid false positives:
  • A solution might end correct while steps are wrong; PRM supervision reduces that risk.

Key results (as presented)

• PRM outperforms:
  • ORM
  • majority voting
• PRM shows better performance in rarer correct-solution regimes (cases where < ~5% of samples are correct).
• PRM is discussed as more data/sample efficient than ORM:
  • ORM might need many full-solution labels, while PRM leverages stepwise structure with fewer trajectories per correct outcome.

Key lesson

• Supervising the reasoning path (not only the outcome) makes verification more reliable and improves robustness to distribution shift.

---

Paper 3 (Meta/“Matt Shepard” style): Verify and Reinforce LLM Step by Step without Human Annotation

Core concepts

• Human process labels are expensive; this paper automates step labeling for PRM.
• It introduces:
  1. Automatic annotation via rollout-based step scoring
  2. Using PRM as a reward model in an RL loop to improve the generator

Automatic step annotation (detailed)

• At any reasoning step (t), define “quality” of that step by its potential to reach a correct final answer later.

• Two estimation approaches:

  • Hard estimate
    • A step is successful if any of (n) continuations eventually lead to a final correct answer.
    • Example intuition: if 1 out of 3 rollouts succeeds → success=1.
  • Soft estimate
    • Uses the frequency of successful outcomes among (n) rollouts.
    • Example: if 2 out of 3 rollouts succeed → soft score = (2/3).

• Practical issue noted:

  • Hard/soft estimates may struggle when:
    • the correct trajectory is deep/unpredictable, requiring more samples than (n)
    • wrong intermediate steps sometimes still lead to a correct final answer (risk of crediting wrong steps)

Test-time / selection procedure (detailed)

• Generate candidates via multiple sampled trajectories.
• Score using PRM (from the learned step reward model).
• Select the candidate with the highest PRM score.

RL training / “reinforce” (high-level)

• PRM acts as the reward model.
• The generator is fine-tuned using RL to increase the probability of producing trajectories that score higher under PRM.
• Mentioned: RL+PO yields further gains, though improvements plateau.

Key results (as presented)

• Better than self-consistency / majority voting, without human annotation.
• Competitive with or better than the earlier PRM 800K model in the harder math setup.
• General theme of self-improvement:
  • generate data (via rollouts)
  • train PRM
  • use PRM to improve the generator

Key lesson

• PRM-style step verification can be created without humans, using rollout-based automatic labeling and reinforcement learning.

---

Paper 4 (Stanford, 2025): Shrinking the Generation–Verification Gap with Weak Verifiers (Beaver)

Core concepts

• Instead of training a single new strong verifier, use ensembles of weak verifiers:
  • “Weak” means each verifier correlates with correctness but is imperfect.
• Verifiers can include:
  • PRMs
  • outcome reward models (ORMs)
  • LLM judges (prompt the model to score correctness)
  • reward models using tools/rubrics

Ensembling methodology (detailed)

• Combine multiple verifiers’ scores to form a stronger aggregate verifier.

• Methods mentioned:

  1. Naive baseline / simple averaging (with filtering)
  2. Logistic regression / weak-to-strong supervised weighting when some labels are available
  3. Score-weight and select:
     • Normalize verifier scores to a common scale
     • Filter out low-quality verifiers below a threshold
     • Combine remaining verifiers using weak signals

• Important note:

  • Performance improved by ensembling, but not always monotonically with more verifiers; weighting/selection matters.

Weak-to-strong aggregation (Beaver, detailed)

• Setup described:
  • (n) queries
  • For each query: (k) solutions
  • (m) verifiers
  • Total labels: (n \times k \times m)
• Goal:
  • Estimate (P(\text{correct})) for each query-solution given verifier outputs.
• Key assumption:
  • Verifiers provide signals capturing independent aspects of correctness (independence assumption).
• Uses derived equations/optimization to compute weights and aggregate outputs.
• Compared against:
  • naive ensemble
  • majority voting / multi-agent verification (MAV)
• Outcome:
  • Large improvements on hard benchmarks (e.g., GPQA/Math/MLU-MMLU).
  • Narrows performance gaps between smaller generators and stronger models by using verification compute at inference time.

Distillation to reduce verification compute (detailed)

• Challenge:
  • Running a large ensemble of verifiers is expensive.
• Solution:
  • Train an aggregate verifier (“Beaver”) and distill it into a much smaller model.
• Result described:
  • Distilled model can capture ~97% of pool accuracy with far less compute.
  • Checkpoints are open-sourced; test-time use is cheaper.
• Compute efficiency:
  • Substantially better FLOP efficiency than running all verifiers.

Key lesson

• Reduce the generation–verification gap not only by better single verifiers, but by:
  • combining many imperfect judges
  • using weak-to-strong aggregation
  • distilling the result for efficiency

---

Overall progression / lessons across all papers

• Verification can be framed as a rubric or scoring function that selects correct outputs from candidate generations.
• Moving from:
  1. Outcome-only correctness (GSM8K verifier / ORM)
  2. to process-based correctness (PRM with step supervision)
  3. to automatic step labeling + RL (PRM without human annotations)
  4. to ensembles of weak verifiers + weak-to-strong aggregation + distillation
• Test-time scaling helps but has diminishing returns depending on how well verifiers rank candidates.
• Robustness and accuracy improve when verification is more informative (process supervision, ensembles, better aggregation) rather than relying only on sampling/majority vote.

---

Speakers / sources featured

Speakers / on-record sources

• Lecture speaker (Stanford CS329A instructor / presenter) — the only human speaker in the provided subtitles (no name given in the transcript).

Paper authors / institutions (as referenced)

• OpenAI (Paper 1 and Paper 2; also described as “another paper by OpenAI” for PRM step verification)
• Stanford (Paper 4: Beaver / weak verifiers; and “Matt Shepard” / verify-and-reinforce step-by-step paper referenced by name)
• “SNORKEL / Alex Ratner et al.” (weak-to-strong inspiration; referenced indirectly)

Specific named datasets / systems

• GSM8K (grade school math benchmark)
• PRM 800K (step-level process supervision dataset)
• Beaver (weak-to-strong verifier ensemble framework)

Specific model names mentioned

• GPT-3, GPT-4, LLaMA 3 (instruction), Mistral 7B, DeepSeek 67B, O3 mini (as a reference benchmark model), plus multiple “7B/8B/34B/70B”-style model-size mentions.