Building a Debugger • Sy Brand & Tim Misiak • GOTO 2025

Main ideas, concepts, and lessons

• Debuggers teach core system truths by forcing implementation

  • The speaker argues that learning debuggers by building them is more effective than studying theory alone.
  • Building a debugger reveals how hardware and operating systems actually behave.

• The book is useful beyond “people who want to build a debugger”

  • Understanding debuggers improves systems programming and helps developers:
    • Grasp hardware/OS interaction and low-level “under the hood” behavior
    • Diagnose why debugging workflows fail (e.g., stepping/variables not behaving as expected)
    • Use debugger features more effectively (once you understand what they’re doing)

• Debugging concepts highlighted

  • Memory access breakpoints (“memory breakpoints”)

    • Breakpoints can be set not only on code locations but also on reads/writes to a specific memory address.
    • Purpose: stop execution at the moment memory is corrupted/modified and identify the cause.

  • Stack unwinding (and stack walking) is far more complex than “frame pointers” suggest

    • A simplified approach: when compiled with frame pointers, you can walk the call stack by reading known memory relative to the current stack pointer.
    • The reality: without frame pointers, debuggers must interpret stack unwinding tables, which include complicated, versioned, and edge-case-heavy logic.
    • Both Linux and Windows have “horrific but fascinating” complexity:
      • Inline frames, interrupt handlers, CPU-specific edge cases
      • “Unwinding languages” / encodings embedded in metadata
    • Documentation is not enough: specs are inconsistent, and source code (e.g., GCC behavior) often becomes the true reference.

  • Windows vs. Linux differences are philosophical more than fundamental

    • Major debugging concepts transfer across platforms, but interfaces differ:
      • Linux system call interface vs. Windows’s more “ergonomic” approach to low-level debug features
    • Threading model differs:
      • Linux process/thread distinction is thin; both are kernel “tasks,” enabling debuggers to sometimes treat threading as secondary early on.
      • Windows more strongly centers register context around threads, not processes.

  • Old interfaces create ongoing complexity

    • Linux ptrace is described as a single, overloaded interface whose arguments and error handling vary by request type.
    • This “one function to do many things” approach accumulates complexity over time.

  • DWARF is compact but hard to consume

    • DWARF debug info uses multiple bytecode interpreters and prioritizes small memory footprint by encoding information as programs that must be interpreted.
    • Backward/forward compatibility is limited:
      • Parsers must handle multiple DWARF versions, and versions may be incompatible.
    • Modern needs (bigger binaries, new language features) motivate reconsideration of compact-but-complex formats.

• Language evolution forces debugging toolchain evolution

  • New language features (C++20 concepts, coroutines/“async”-like behavior, anonymous functions, closures, etc.) require:
    • New kinds of debug information
    • Updates across compiler, linker, debugger, and runtime/library boundaries
  • Debugging becomes a cross-layer problem: symbol formats, unwind data, expression evaluation, and execution models all interact.

• Stepping through code is surprisingly complex

  • Even though stepping seems like it should be easy (“go to the next line”), the “next line” is hard to define:
    • Line mappings depend on compiler-generated line tables
    • Stepping semantics differ for:
      • Step into vs. step over
      • Inline functions (hard because there may be no program-counter change)
      • Breakpoints encountered mid-step
      • Branches that jump into/out of functions
      • Conditions spanning multiple source lines
  • Stepping complexity grows as you encounter more real-world compilation behavior and language constructs.

• A practical methodology for stepping: compose “thread plans”

  • The debugger design approach discussed (inspired by LLDB/LLVM):
    • Treat stepping as a set of goals/actions, represented by a stack of “thread plans.”
    • Plans can handle “what to do when a breakpoint occurs,” without mixing that logic into every stepping action.
  • Benefit: separation of concerns
    • Each plan encodes what the debugger is trying to accomplish.
    • When execution stops, the debugger asks the plans which one explains the stop and what to report.

• Teaching/debugger implementation realism

  • While writing the book, some chapters were expected to be hard/easy, but the actual experience differed:
    • Stack unwinding ended up harder than expected.
    • Expression evaluation was also a major challenge.
  • The author mapped prerequisites early (compilation, hardware, OS basics), then re-ordered chapters as dependencies became clearer.

• Future directions in debugging

  • Time travel debugging

    • Lets developers go backward to understand why a bug happened, especially useful for:
      • Multi-threaded programs
      • Non-deterministic behavior
    • Enables reversing past corrupted state (e.g., stack/memory overwrites).

  • Debugging optimized code

    • Still under-served due to:
      • Variables optimized out
      • Stepping jumping unexpectedly
      • Inlining and other transformations
    • Presents a major tooling opportunity.

---

Methodology / instruction-style details (as presented)

Memory breakpoints (conceptual “how it works” steps)

• Set a breakpoint on a memory address for:
  • Reads or writes
• When the program accesses that address (e.g., corrupt write occurs):
  • The debugger halts execution immediately
• Inspect state at the stop point to identify:
  • What code path/cause is responsible for the corruption

Stack unwinding (practical reality compared to simplified method)

• If compiled with frame pointers:
  • Unwinding can be done by walking up the stack via known offsets from the current stack pointer.
• If frame pointers are not available:
  • The debugger must interpret unwinding tables/encodings:
    • Potentially multiple metadata formats
    • Multiple versions
    • Complex edge cases (inline frames, interrupts, CPU differences)

Stepping design approach (thread plans / separation of concerns)

• Model stepping as:
  • A stack of “thread plans” representing goals like:
    • step over a code range
    • step into a function
    • step out of a function
• As execution progresses:
  • If a breakpoint occurs during a planned step:
    • Stop execution
    • Ask each thread plan in the stack whether it can explain/handle the stop
• Report to the user based on the plan that matches the stop reason
• Keep stepping logic modular:
  • Plans encode intent and stopping behavior
  • Other concerns (like breakpoint handling relevance) are delegated to the plan framework

---

Speakers / sources featured (and roles)

• Tim Mishack — interviewer; engineer at Datadog (formerly at Microsoft working on debuggers)
• Sai Brand (spelled “Saibrand” in subtitles) — author of Building a Debugger; works on C++ standards and toolchains; background in GPU debuggers and compilers; Microsoft C++ developer advocate
• LLDB / LLVM debugger — referenced as an example stepping implementation style (thread plans)
• GCC — referenced regarding how its source behavior can contradict unwind specifications (Linux context)
• DWARF — referenced as a Linux debug information format (and its versioning/bytecode interpreter complexity)
• ptrace — referenced as the Linux low-level debugger interface that accumulates complexity over time