Training Agents 3: Reinforcement Learning

Main Ideas and Lessons (Reinforcement Learning for Training Agents, Session 3 / GRPO)

Series context

• This is the third live session in a reinforcement-learning post-training series.
• Prior sessions covered:
  • SFT (Supervised Fine-Tuning) on “traces”
  • Distillation
• This session focuses on GRPO (Group Relative Policy Optimization), a reinforcement-learning method.
• A fourth session will apply the same algorithm in reinforcement learning environments (i.e., the “world” the agent acts in).

What changes from SFT/distillation to RL (the learning signal)

• SFT: uses a dense target (emulate target tokens), learning is limited by what the dataset contains.
• Distillation: uses a teacher that provides more informative signals via ranked rollouts / teacher-influenced judgments (denser learning than vanilla RL).
• RL: relies on a typically sparse reward signal (often at trajectory end), though rewards can be reshaped into denser forms.
  • In GRPO specifically, the learning signal is mostly group-relative rather than fully dense token-by-token rewards.

Why RL is usually used after SFT

A common pipeline pattern:

1. SFT first to “bootstrap” behavior up to a performance ceiling
2. Then RL to push beyond that ceiling using interactive rollouts and reward feedback

Core RL loop (applies broadly, including GRPO)

• Prompt given → model generates text
• Multiple rollouts/trajectories are sampled
• Each rollout is scored with a reward/judging signal
• The model is updated
• Repeat over more prompts

Reward/scoring can be:

• a Python function
• a separate reward model
• a judging model

Analogy for learning modes (offline vs online)

• RL (“no coach”): learn from playing and getting win/loss feedback; sparse and slow.
• Off-policy distillation (like SFT on traces): learn by reading grandmaster games; dense imitation but may miss real-scenario strategies.
• On-policy distillation: learn from your own games with feedback; denser signals but still limited by what the teacher can judge.

---

GRPO: Methodology and Step-by-Step Process

GRPO training step (high-level)

• Input

  • Take a prompt from the dataset.
  • Use the current policy model to generate multiple completions for the same prompt (group).

• Group rollouts

  • Generate N completions (e.g., 3).
  • Each completion corresponds to a trajectory/sequence of actions.

• Score each completion

  • Compute rewards using a reward function (often literally a Python function).
  • Rewards can combine multiple components (e.g., format + accuracy/success).

• Compute a relative signal

  • Compute a group-relative advantage for each completion:
    • Compare each completion’s reward to the group average
    • Normalize by group standard deviation
  • Intuition:
    • If all are equally good/bad, there’s little relative learning signal.
    • If variation exists, the model learns which outcomes are better than the group.

• Update policy

  • Update model parameters using GRPO’s loss formulation with guard rails (below).

• Repeat

  • Iterate across dataset prompts, repeatedly sampling groups, scoring them, and updating.

Group-relative advantage formula (as described)

• Advantage ≈ (reward − group_mean) / group_std

“Guard rails” to prevent undesirable policy drift

GRPO uses two main constraints to keep updates safe and stable:

1. KL divergence limit (longer timescale constraint)

   • Prevents the policy from drifting too far from its base/reference behavior.
   • Helps avoid undesired behavior changes (e.g., chess turning into something else like checker-playing).

2. Batch limit / clipping (shorter timescale constraint)

   • Prevents any completion in a batch from moving too far from the rest of the batch’s behavior.
   • Uses parameters:
     • epsilon: bounds per-batch/reuse behavior
     • beta: bounds across the full run / overall deviation
   • Mentioned that in TRL, these may have defaults (described as 0 by default, but adjustable).

Simplified “code-level” computations (conceptual)

For each generated response:

• Compute token-level log probabilities for the completion (sum over tokens using a completion mask).
• Compute loss using:
  • advantages
  • ratios of new vs old per-token log-probabilities
• Apply clipping so the update coefficient stays within a range defined by epsilon.

Net effect: weight increases for completions with positive relative advantage and decreases for negative ones, while clipping/KL constrain how far the policy moves.

---

Reward Functions (“Reward Contract”) and Reward Hacking Prevention

Reward contract (how to define success)

Reward functions act like a contract between:

• what you want the model to learn
• what it is actually trained to achieve

Typical structure:

• Format reward: checks output structure
  • e.g., legal chess move formatting, correct tags, tool-call format
• Accuracy/success reward: checks correctness
  • e.g., correct answer, checkmate/win, tool execution success

Rewards can be combined, for example:

• output may be correct but improperly formatted
  • format reward could be 1
  • accuracy reward could be 0 (or vice versa), depending on the design

Why reward design needs care

If rewards are easy to game, the model learns to exploit them (reward hacking / shortcuts).

Example risk:

• Format validation too permissive (e.g., overly flexible/open regex)
  • model satisfies the letter but not the intent
• In tasks, the model may find unintended loopholes (with chess-like “silly hacks” as an analogy).

How to reduce reward hacking (practical guidance)

• Strongest approach: design reward functions that disallow loopholes
• Also use monitoring and probing:
  • track metrics like:
    • entropy (can indicate repetitiveness/low diversity)
    • situations where reward rises while real behavior quality does not
• Use human- or engineer-in-the-loop checks:
  • inspect sampled trajectories/rollouts
• Note: KL penalties help limit deviations, but are not a complete solution
  • reward hacks can occur within regions not well covered by SFT distribution

---

Interpreting GRPO Training Curves (What to Look For)

Normal / healthy patterns

• Easy reward (e.g., format) rises early.
• Harder reward (e.g., accuracy/success) rises more slowly, but still improves.
• Entropy: not expected to abruptly collapse; may be flat or slightly decline.
• Completion length: task-dependent; suspicious spikes/large shifts may indicate hacking or wandering.
• Reward spread/variation within groups:
  • should be “healthy” (not all 0, not all 1)

Stalled run

• No improvement in rewards
• Minimal/no reward spread (e.g., all rollouts fail or all succeed immediately)
• Diagnose by probing:
  • tasks may be too hard
  • rewards may be uninformative

Collapsing run

• Entropy drops significantly (model locks into a narrow strategy)
• One reward rises while test/true objective drops
  • suggests reward hacking
• KL penalties may rise a lot (policy deviating from baseline)

Workflow

• Use track.io to inspect run curves.
• Do dry runs / smoke tests:
  • validate the reward function produces variation (e.g., on a few steps)
  • only then run full training

---

Experiments Performed / Applied Using TRL (Practical Takeaways)

Tooling described

• TRL (Transformer Reinforcement Learning / post-training library)
  • includes trainers like GRPOTrainer (and others such as SFT)
• Experiments use:
  • GRPO training scripts
  • evaluation via unit tests / callable rewards
  • tracking via track.io
• Running on HF Jobs:
  • send scripts for remote execution without local GPUs
  • save outputs and tracked artifacts for later inspection

---

Experiment 1 (Dummy reward / Reward shaping demonstration)

Goal

• Train with GRPO using a dummy objective:
  • reward outputs that are “close to 20 ch ar” (character length target)

Dataset

• Simple conversation dataset from Reddit

Reward hacking lesson

• Training reward increases, but behavior may not be meaningful beyond the dummy target.
• Takeaway: the model optimizes what you tell it to optimize, even if it’s not your real goal.

---

Experiment 2 (Coding / Unit tests as reward signal)

Goal

• Train a model to solve Python coding problems by rewarding based on unit tests.

Dataset

• Python problems, each with unit tests.

Reward design

• Execute generated code in isolation (sandbox/process) with a timeout.
• Reward scheme:
  • If tests pass → reward = positive score (often described as 1)
  • If tests fail → reward = 0 (or no reward)
• Mentions improvements like better sandboxing (as an example).

Key training insight: group variation / sweet spot

The model needs a “mix” of outcomes in the group:

• not all rewards are 0 (task too hard)
• not all rewards are 1 (task too easy)
• there’s a sweet spot enabling meaningful relative advantage

Hyperparameter explored:

• number of generations per prompt (group size): e.g., 2 vs 8 vs 16

Observed behavior:

• small groups (e.g., 2) can yield noisy reward signals
• larger groups can increase useful variation

---

Experiment 3 (Explicit reward hacking demonstration)

Goal

• Demonstrate that optimizing a reward can produce behavior that satisfies the reward without solving the task.

Method

• Use the same coding setup but define a bad reward:
  • reward depends on generating a certain number of code “blocks” (e.g., 4 blocks)
  • ignores whether code passes tests

Expected outcome

• Training reward increases (appears successful)
• Test reward decreases (true task performance collapses)

Lesson

• If train rewards go up but evaluation fails:
  • likely reward hacking or reward function bugs
• Then revisit:
  • reward function correctness/loopholes
  • model choice
  • hyperparameters (e.g., group size, KL/clipping settings)

---

Pipeline / Operational Guidance (How Adjustments Happen)

• Hyperparameters and reward functions are typically changed between runs, not during training.
• Suggested iterative workflow:
  1. Start with dry runs of the reward function on sample outputs
  2. Run a small smoke test (few steps) and verify reward variation
  3. Launch full training, then iterate based on curve diagnostics

---

Speakers / Sources Featured

Speakers

• Ben (host/lecturer; referred to as “my colleague” with Sergio and leading the session)
• Sergio Paniego (co-presenter; leads experiments and answers questions)

Sources / tools / referenced libraries

• TRL (including GRPOTrainer)
• track.io
• Hugging Face (HF)
  • HF Jobs for remote execution
  • Hugging Face YouTube channel
• DeepSeek papers (referenced concept: “aha moment”)
• Algorithms mentioned:
  • SFT
  • Distillation (including on-policy distillation)
  • GPO / GRPO
  • DPO (Direct Preference Optimization) (offline; chosen/rejected pairs)
  • PPO (Proximal Policy Optimization) (reward-model grading; more complex)
  • Online DBO/DPO (briefly referenced as related online variants)