I Asked Claude Fable 5 to Improve llama.cpp.. and It Did

Summary of the video’s technological content (llama.cpp + “Fable” optimization)

Goal

The video’s aim is to use Claude Fable to optimize llama.cpp for local mid-range hardware. The creator tested on a 12 GB RTX 3060 with 32 GB DDR4.

Rather than focusing on flashy demos, the emphasis is on improving the existing local inference engine.

---

How Fable was used (workflow)

The creator forked llama.cpp because the upstream repository rejects PRs that are fully or predominantly AI-generated. Fable was used to:

• Read the codebase
• Analyze where optimizations might apply
• Propose changes
• Benchmark before/after
• Compare outputs to ensure no regression (described as token-identical outputs)

---

Key performance target: MoE inference

A core target is Mixture-of-Experts (MoE) inference, where only a small subset of experts is active per token.

• Example mentioned: 35B MoE
• Approx. active per token: ~3B active
• Practical implication: with offloading, expert weights may reside in system RAM instead of fully fitting in VRAM (especially relevant for the tested 12 GB GPU).

---

Main technical findings: 4 optimizations (2 succeed, 2 fail)

Baseline performance

For the tested MoE workload at UBatch 2048, the baseline prefill speed is about:

• ~1143 tokens/sec (prompt processing / prefill)

---

✅ Optimization 1 (Succeeded): “Pinning” paged memory

Issue

llama.cpp includes memory-mapping modes:

• mmap: paged/evictable by the OS (feels fast to load)
• no mmap: better transfer/compute characteristics, but slower model load/swap

What Fable found

Fable identified dead code: a feature that can pin mmapped paged memory, effectively achieving no mmap-like transfer performance while preserving the instant load/swap benefits of mmap. The original call site that should enable it wasn’t being used.

Effect

• ~1143 → ~1385 tokens/sec on prefill (~21% faster)
• The gain is similar to using no mmap, but avoids the longer load/swap penalty.

---

✅ Optimization 2 (Succeeded): Overlap attention with expert weight uploads

Core bottleneck (prefill)

During prefill on offloaded MoE (processing hundreds/thousands of tokens):

1. GPU computes attention + router
2. After routing, the system knows which experts are needed
3. Expert weights must be copied from RAM → VRAM
4. During that copy, the GPU is idle

Profiling showed GPU idle time ~41.7% in the relevant stage.

Optimization idea

Use two parallel “lanes”:

• Lane A: compute attention
• Lane B: start copying expert weights

The key trick is that in this prefill regime, most experts are selected anyway, so waiting for router IDs isn’t beneficial. It therefore pulls the whole expert layer back to VRAM early, before router completion.

Effect

• GPU idle time: ~41.7% → ~2.8%
• Combined with Optimization 1, total prefill speedup becomes ~64.5%
• Reported as: ~1143 → ~1880 tokens/sec
• Quality maintained: outputs described as token-identical to mainline

---

❌ Optimization 3 (Failed): Adaptive speculative decoding controller

Context

During decode (token generation), speculative decoding can draft tokens ahead and verify them in batches.

Corrected assumption

Fable adjusted the understanding that decode on offloaded MoE is not purely CPU-bound; it’s roughly 50/50 between GPU and CPU due to frequent handoffs.

Why longer drafts can hurt

Verification cost depends on the union of experts across drafted tokens. Drafting more tokens increases that union set, potentially toward nearly all experts, raising CPU workload and slowing things down.

• Example: drafting 8 tokens ahead was slower than no speculation.

Adaptive controller attempt

Fable tried to implement an adaptive controller to pick draft length dynamically:

• It was iterated/rebuilt five times
• Still lost to a simple fixed cap
• Overhead from learning/decision-making outweighed savings

Practical takeaway

For offloaded MoE, a fixed speculative cap of ~2–3 tokens is best:

• About ~23% faster
• With no slowdown
• The adaptive approach didn’t beat that fixed baseline

---

❌ Optimization 4 (Failed): Split expert work between CPU and GPU concurrently

Big idea

Run parts of an expert layer on CPU and GPU at the same time to parallelize expert execution.

Result

• About 14× slower in the tested setup

Why it failed

The limiting factor was hardware/transport bandwidth:

• PCIe Gen 4 was too slow compared to reading experts directly from system RAM
• Moving work across PCIe cost more than it gained

When it might work

This could become beneficial on PCIe 5 or NVLink, where transport costs would be lower.

---

Overall results

• 4 optimizations total
  • 2 worked
  • 2 did not

Total successful prefill speedup

• ~64.5% improvement (code-only): 1143 → 1880 tokens/sec
• No regression: outputs described as token-identical

Additional value

Even failed ideas helped surface:

• Latent bugs in llama.cpp mainline
• Issues in Fable’s patch logic

Availability

The creator states the changes are available on their GitHub fork, so viewers can test.

---

Main speakers/sources (at the end)

• Main speaker: the video author/experimenter (narrator; describes benchmarking, profiling, and the GitHub fork)
• AI tool referenced: Claude Fable
• Software being modified: llama.cpp (maintainers referenced regarding AI-generated PR policy)