llama.cpp Just Made Huge MoE Models 4× Faster

Summary

The video explains an experimental llama.cpp change that speeds up Mixture-of-Experts (MoE) models by moving expert-cache management onto the GPU. The key change is not simply caching more expert weights, but reducing the CPU–GPU coordination needed to decide which experts to keep and load for each token.

How the GPU-graph cache works

MoE models route each token to a small subset of experts. If a selected expert is not in VRAM, its weights must be fetched from system RAM over PCIe.

The proposed cache reserves a fixed number of GPU slots for each layer whose expert weights are stored on the host. It uses a least-recently-used (LRU) replacement policy and adds two operations to the GPU compute graph:

• LRU_DECIDE checks routed expert IDs, updates cache state, and identifies missing experts.
• LRU_FILL loads missing weights into their reserved slots.

The expert computation then uses those slots. This avoids a per-token CPU round trip for cache decisions. It does not increase PCIe bandwidth; it aims to reduce management overhead so the GPU can spend more time doing useful work.

Reported performance

• On a 24 GB Radeon RX 7900 XTX, running Qwen 3.8 Flash Next in IQ3 quantization with all 48 MoE layers using host-side expert pools:

  • No new cache pool: 9.7 tokens/s
  • GPU-graph LRU with 160 slots per layer: 37.2 tokens/s, with an 89.4% cache hit rate—about 3.8× the baseline
  • Adding speculative MTP decoding brought the reported result to 44.8 tokens/s, with some code-generation workloads reaching 50 tokens/s. These gains include speculative decoding, not just the cache.

• In a separate IQ4 test, with eight expert layers on the GPU and 40 using host-side caching, a roughly 6.4 GB cache budget delivered 31.2 tokens/s at a 77.9% hit rate. On the same machine and with a similar cache budget, the existing cache reached 20.1 tokens/s—about 55% slower in that comparison.

• On two RTX 3090s, reported decoding speeds were 47.6 tokens/s with layer separation and 59.6 tokens/s with MTP. Tensor-split reached 65.3 tokens/s with MTP. The video notes that much of the extra gain came from speculative decoding giving the GPU more parallel work.

• An independent test on a different system, using PCIe 3.0 and a smaller Q1 model, found the GPU-graph cache at about 56–57 tokens/s, similar to the existing cache at about 57 tokens/s. Another experimental cache approach was faster on that system.

Caveats and status

• The implementation was described as an experimental branch, not a feature in regular llama.cpp builds as of October 21. The video says it supports CUDA and HIP.
• Results depend on the model, cache size, workload, host-memory setup, and PCIe bandwidth. Expert weights need to be in pinned host memory; Windows pinning limits or other settings may prevent the cache from activating. A narrow PCIe connection or too few cache slots can also reduce the benefit.
• It is unlikely to help much when a model already fits entirely in VRAM. Faster token generation also does not automatically mean faster overall request processing or better performance with many simultaneous users.
• The developer reported nearly unchanged perplexity and identical outputs across 12 repeated greedy-decoding runs. The video cautions that this is useful evidence, but not a comprehensive quality evaluation.

Speakers and sources

The video’s narrator is from Code Unpacked. The GPU-graph LRU proposal is attributed to developer leech77, in llama.cpp issue/PR reference #29949. The video also cites an independent benchmark on a separate system.