Video summary

The prompting playbook

Main summary

Key takeaways

Technology

Overview

This video is a practical session on prompting best practices for LLM-based systems, framed as two scenarios you’ll encounter at work:

  1. Maintaining and migrating an existing prompt in production (the prompt stops working well after model/architecture changes)
  2. Building a new agentic use case from scratch (prompt + system design from “zero to one”)

Scenario 1: Debugging an existing production prompt via eval-driven iteration (Claude-based)

Key idea: use evaluations to detect regressions

When migrating to a new model, failures can come from either:

  1. The new model behaving differently (the prompt can be adjusted), or
  2. The new model being less capable (prompt changes won’t fully fix it)

An eval suite is presented as the “rigor” tool to measure whether changes actually improve performance.

Representative eval suite structure (5 test cases)

The talk emphasizes three coverage types:

  • Control case: always pass; unambiguous
  • Edge cases: known prior failures; instructions should prevent regressions
  • Capability boundaries / handoff: when to refuse or escalate to a human

Example system: Meridian Mobile customer support bot (Telco)

Five eval cases include:

  • Control: “data limit in the basic plan”
  • Edge case: proration/calculation when switching plans mid-month
  • Policy/accuracy check: answering key policy-covered questions correctly
  • Escalation: escalate to a human when there’s a billing error
  • No withholding: don’t hide information the system is supposed to provide

Workflow used

  • Start with a “first pass” prompt and run it on the v0 prompt version
  • Identify failure modes and fix them one at a time
  • Before targeted fixes, apply general hygiene to clean up the prompt

“General hygiene” improvements

  1. Remove copied/irrelevant content and incorrect role assumptions

    • Example issues: prompt says the bot is human; contains web-page artifacts like “hero image”/“cookies”.

  2. Add structure with XML tags

    • Separates role, guidelines, policy, tone, etc.
    • Result: improved performance notably (a prepaid scenario improved after restructuring).

  3. Add an output contract for consistency

    • Uses XML tags for response formatting.
    • Also mentions harness-level support like stop sequences to stop at a closing tag.

  4. Note: if guidelines/policy/data can’t be distinguished by a human reader, the model likely can’t either.

Failure modes and specific prompt fixes

  1. Hotspot data / legacy plan mismatch (withholding vs giving wrong data)

    • Failure: the model deflects to a URL instead of using the customer’s provided “source of truth” (even though it has the hotspot amount).
    • Cause: an old “patch” instruction for earlier models (“never give wrong plan details—point to URL”) became overfitted/redundant.
    • Fix: replace with a “balanced view” aligned to provided account data, including the nuance of grandfathered plans.
    • Lesson: older defensive instructions can create new wrong behavior as models improve.

  2. Proration calculation returns vague/incorrect math

    • Failure: model “reasons” but outputs a non-concrete/vague bill amount; relies on mental math.
    • Fix: add a calculation tool:
      • prompt instructs to use a calculate proration tool
      • define tool schema and implement the tool logic
    • Lesson: instructions don’t add capability—tools do. Don’t ask for mental math if you can supply an executable tool.

  3. Billing error escalation fails due to asymmetric trade-off instructions

    • Failure: instead of escalating, the model diagnoses/explains.
    • Cause: instruction includes only one side of cost/benefit (escalation “costs $8”), overfitting to not escalating; also conflicts with escalation requirements in eval setup.
    • Fix: provide both sides of trade-offs (escalation costs $8, but failing costs refund + trust).
    • Lesson: as models get better at reasoning trade-offs, prompts should state trade-offs more completely.

System-wide takeaways from Scenario 1

  • Use evals to prove correlation between prompt changes and performance changes.
  • Fix in layers:
    • do hygiene/structure first
    • then target failure modes one by one
  • Avoid long “do not” bans; be precise about goals and capabilities.
  • Manage and revise defensive patches over time (ideally with version control).

Scenario 2: Building a new agentic schedule generator (generate/evaluate/repair)

Task example: week-long retail staff schedule

  • Inputs:
    • 8 employees
    • headcount targets
    • constraints (hard rules must be satisfied)
  • Output:
    • schedules that meet constraints
  • Evaluation method:
    • because constraints are hard/quantifiable, they use a Python function to compute violations rather than an LLM judge

Comparing approaches (model + prompt + harness factors)

Baseline: smaller model + basic prompt

  • Sonnet 4.6 with an XML prompt and JSON output contract
  • all test cases fail
  • model attempts reasoning but burns tokens and still produces invalid schedules

Stronger reasoning model

  • Opus 4.7 with the same prompt
  • still fails, but fewer violations
  • suggests reasoning helps, but isn’t sufficient alone

Adaptive thinking

  • Opus 4.7 with “adaptive thinking”
  • passes reliably, but increases:
    • tokens (~3x)
    • latency (~tripled)

Smaller model + improved prompt

  • Sonnet 4.6 + better prompt with explicit “reason” and “check its work”
  • passes 2/5
  • remaining failures are due to:
    • output/token limit
    • incomplete completion
  • increasing max tokens worsens cost/latency

Agentic strategy: Generate → Evaluate → Repair (LLM loop)

This introduces a multi-step architecture:

  1. Generator drafts a schedule
  2. Evaluator prompt reports violations with evidence (checked by an LLM, not Python)
  3. Repair prompt applies targeted fixes based on violations

Outcome: passes all test cases, with lower tokens and lower latency than heavy “adaptive thinking” runs.

Additional advantage: soft constraints at runtime

Because repair is modular, it supports changing preferences/soft constraints per run without rewriting the Python evaluation logic each time.

Example soft constraints:

  • “Harry doesn’t like working with Sally”
  • “need a third shift on Wednesday”

Overall conclusions

For prompt maintenance/migration

  • eval suites are essential to diagnose regressions
  • apply hygiene (structure, remove redundant/incorrect instructions, output contracts)
  • then address failure modes iteratively
  • replace vague guidance like “tell the model to be careful” with:
    • tools for computation
    • revised balanced trade-offs for escalation decisions
    • removal/updates of old patches that no longer match the model’s capabilities

For new agentic builds

  • prompt/model changes alone may be expensive
  • use a multi-prompt agentic loop (generate/evaluate/repair) to improve compliance efficiently

Main speakers / sources

  • Margo van Laar — Applied AI Engineer at Anthropic (London) (main speaker)
  • Video references Anthropic models: Claude (Sonnet 4.6, Opus 4.7) and concepts like adaptive thinking
  • Example products/domains used:
    • Meridian Mobile (Telco customer support bot)
    • retail scheduling use case (8 employees)

Original video