llama.cpp Just Made 125B MoE Models 4× Faster!

Summary

The video explains how llama.cpp’s new Mixture-of-Experts (MoE) expert cache can speed up large models, why the headline “4× faster” depends on the comparison, and which hardware is most likely to benefit.

How the expert cache works

The example model, Qwen3.8-Flash-Next, has about 125 billion parameters arranged as an MoE model. It contains 48 layers, each with 512 experts, but only 10 experts per layer are selected for each generated token. Most expert weights are therefore idle at any given moment.

Previously, llama.cpp’s whole-layer placement could put complete expert layers on the GPU. This was useful, but potentially wasteful because it stored all 512 experts even though only a few were used per token. The new cache instead keeps individual, recently used experts in GPU VRAM. When a requested expert is missing, it must be copied from system RAM over PCIe.

The cache works best when expert requests show locality—that is, when the same experts are reused—and when there is enough cache space to achieve a high hit rate. The key measure is the cache hit rate, not just GPU compute power.

What the speed-up figures mean

• The often-quoted 3.8× improvement—from 9.7 to 37.2 tokens per second on a 24 GB Radeon RX 7900 XTX—compares the cache against a setup that kept essentially no experts in GPU memory. It is a real result, but uses a weak baseline.
• A more direct comparison against llama.cpp’s default whole-layer GPU placement showed about 2.2× improvement in one reported test.
• An experimental implementation was reported to be 1.55× faster than the merged implementation on the same setup. It avoids CPU routing handoffs at each layer, and pinned memory also helps transfers. The video says the experimental branch had not yet been merged and had platform-stability limitations.
• A Windows fix for pinned-memory behavior reportedly raised a test result from 37.76 to 62.73 tokens per second on an RTX 5090. The video notes that the full comparison had not been rerun after this fix.

Who benefits—and who may lose speed

The cache is most promising for large-VRAM GPUs, fast PCIe links, and generation-heavy workloads. The video estimates that the example model may need roughly a 16 GB expert cache to reach a useful hit rate, in addition to memory for attention weights and context.

The video describes a hit rate of around 85% as a break-even region: below that, cache-miss overhead can outweigh the benefit; above 90% is more promising. Reported results illustrate the risk for smaller cards:

• An 8 GB RTX 4060 laptop slowed from 23.4 to 15 tokens per second with a 1.5 GB cache; a larger cache was slower still.
• A 12 GB RTX 3060 slowed from 32 to 12 tokens per second with an 8 GB cache.
• In a 12 GB RTX 4070 SUPER test, plain llama.cpp beat the tested cache sizes.

PCIe bandwidth is another constraint. Cache misses require transfers over the bus, so a card in a slower or narrower slot may not benefit even if it has sufficient VRAM. The video gives examples where whole-layer placement beat caching on PCIe 3.0 hardware.

Other trade-offs and considerations

• Prompt processing: The cache primarily benefits decode, not long-prompt processing. Reserving VRAM for it can reduce prompt speed; one cited test improved decode but reduced prompt-reading speed to 0.73× its previous rate. Context-heavy workflows may therefore see little overall improvement.
• Speculative decoding and multi-token prediction: Verifying several proposed tokens can request more experts at once, potentially overwhelming a cache sized for ordinary one-token decoding. Results vary with implementation and cache size.
• Possible looping behavior: A user reported repetitive output with caching enabled, but the video says a sampling issue may be responsible and that a causal link was not established. Disabling the cache is suggested as a diagnostic check.
• Alternative runtime: Strata is presented as a faster specialist for this model family in the cited tests, while llama.cpp is described as a more general-purpose runtime with broader model support.

Suggested 15-minute benchmark guide

1. Update to a recent llama.cpp build that includes the relevant release and Windows fix.
2. Benchmark the model without the expert cache, using automatic GPU placement or a tuned whole-layer split. Record both decode speed and prompt-processing speed.
3. Enable the cache using most—but not all—of the spare VRAM, leaving room for context and overhead.
4. Run a few warm-up prompts before measuring, then inspect the logged hit rate.
5. Compare against the best whole-layer setup, not against a baseline that leaves GPU memory unused. Treat a hit rate below about 85% as a warning; the video describes 85–90% as a gray area and above 90% as more favorable.

Main speaker and sources: The video is presented by Kai. It draws on llama.cpp documentation and pull requests, an experimental branch attributed in the subtitles to “Lich 77” (the name transcription may be inaccurate), and user and developer benchmark reports.