Video summary

Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)

Main summary

Key takeaways

Technology

Technological concepts / product features emphasized

Running a very large MoE model on small VRAM

  • Uses sparse Mixture of Experts (MoE) routing so only a small subset of experts run per token.
  • Model described as 22GB QN3.6 35B A3B:
    • About 35B total parameters
    • 3B active per token (≈ “3B parameters used per token”)
  • Video claim:
    • ~17 tokens/sec on a 6GB GTX 1060 (2016) using llama.cpp

Keeping mostly-idle expert weights off the GPU

  • Each generated token “wakes” only a subset of experts (described as 8 experts + 1 shared).
  • Most expert weights remain idle and can live in system RAM instead of VRAM.
  • Net effect: reduced VRAM pressure and enabling smaller-GPU deployment.

How speed is controlled in llama.cpp (via 5 flags)

1) CPU-MOA and N-CPU-MOA

  • Moves expert weights of the first N layers into CPU/system RAM.
  • Example effect (6GB GTX 1060):
    • ~3 tok/s → ~10 tok/s without additional quantization
  • Tuning guidance:
    • N is manual
    • Start with many layers on CPU and reduce until you approach VRAM limits, then step back one layer count.

2) “Load mode” options (replacing older flags)

  • Older flags like No-MMAP were deprecated.
  • llama.cpp consolidated behavior into a single load mode with multiple settings (e.g., None, MMAP, MLOCK, DEO).
  • Important warnings/catches:
    • If your model is larger than RAM, using No-MMAP is incorrect.
    • For MoE models, leaving memory mapping (page cache) enabled can be faster because the kernel/page cache may help.

3) n_gpu_layers (push layers back onto GPU)

  • After moving experts to CPU, n_gpu_layers keeps enough layers on GPU to utilize VRAM efficiently.
  • Example effect:
    • ~10 tok/s → ~17 tok/s on the same 6GB GTX 1060 scenario

4) mLOCK / anti-paging behavior

  • If the server runs for hours, the kernel may page out cold expert pages.
  • Using mlock can prevent paging:
    • Reported benefit: improved first token latency after long idle periods.
  • Main limitation:
    • Once experts are in system RAM, throughput becomes memory-bandwidth bound.

5) op_offload_min_batch_size (prompt processing vs decoding)

  • Decoding (writing tokens):
    • Batch size is ~1
    • Often handled on CPU if below threshold
  • Prompt processing:
    • Processes many tokens at once (batch > threshold)
    • Can require copying expert weights to GPU via PCIe, so performance depends on batching/offload behavior
  • Benchmark cited:
    • Increasing micro-batch 128 → 2048
    • Prompt processing: ~22 tok/s → ~345 tok/s
  • Patch cited:
    • “Copy only the used experts when offloading prompt processing”
    • Goal: avoid transferring all experts (example mentioned: 256).

Tooling / workflow notes about where the approach works (and doesn’t)

Ollama limitations

  • Ollama doesn’t expose fine-grained control for MOE expert placement (CPU vs GPU partitioning).
  • Feature requests for this remain open.
  • Claimed behavior:
    • Ollama may split something like 17% CPU / 83% GPU without allowing exact allocation control.

LM Studio limitations

  • Reportedly provides only a coarse checkbox:
    • all experts CPU or none

Context-length and memory tradeoffs (separate from decoding speed)

Large context claimed for QN 3.6

  • Claimed context length: 262k tokens
  • KV-cache breakdown described in the video:
    • Per token per sequence KV cache: ~20 KB
    • MoE architecture provides structural memory savings
  • Full context estimate:
    • Around ~5GB KV cache
  • Architectural point:
    • Among 40 layers, only about 10 use real attention
    • The rest use gated DeltaNet (linear attention with fixed-size state)
    • This reduces KV-cache growth.

“Turbo Quant” for KV-cache to get more context on tiny VRAM

Why it’s needed

  • Even with MoE KV savings, a ~5GB context KV cache still competes with hot weights on 6GB VRAM.

What Turbo Quant is (from cited paper)

  • Based on:
    • A rotation/coordinate transform
    • Quantization against a fixed codebook
  • Claims:
    • ~6× reduction in key/value memory while maintaining benchmark scores
  • Example result (6GB hardware):
    • Context increases from ~64k → 256k with Turbo quantification

Status and tooling limitation

  • Turbo Quant is not integrated into main llama.cpp.
  • The video claims it appears in many PRs/issues but they’re closed/unmerged, including some marked as policy violations.
  • Requires building a fork ~300 commits ahead of upstream.
  • Additional performance note:
    • Turbo speed advantages reportedly diminished after upstream rewrote MoE attention kernels.
    • Example claim:
      • Turbo 2 decodes at ~45% of F16 on an MoE model
    • Conclusion implied in the video:
      • The speed win may have disappeared, but the memory win may remain.

Decoding optimizations discussed (but flagged as problematic for MoE)

Speculative decoding

  • General technique:
    • A smaller model proposes tokens; the larger model verifies.
  • Why MoE sparsity can make it worse:
    • More predicted tokens can trigger more experts, reducing the same sparsity advantage.
  • Video cites a Mixtral-related paper analysis:
    • Speculation slowed verification and increased overall time
    • Reported: ~1.0–1.5× slower than no speculation
  • Workaround:
    • Multi-token prediction / draft head trained into the model to share cache
  • Reported improvements:
    • ~1.4–2× on dense models
    • ~1.15–1.25× on MoE

Accuracy/quality considerations (not just speed)

Check whether aggressive quantization still matches the original model

  • Video discusses drift metrics:
    • 4-bit KL divergence reportedly worse than 8-bit
    • Example drift: 0.0137 vs 0.0026 (described as ~5× drift)
  • Notes:
    • Perplexity changes appear small (<1%), but:
    • KL divergence and perplexity don’t reliably predict task success
  • Recommended approach:
    • Run your own task/benchmark for a day.

Concrete guidance / tutorial-style “rules”

  • Use MoE sparsity + llama.cpp tuning to fit large models on small GPUs:
    • Tune CPU-MOA (N-CPU-MOA) first for major gains
    • Then adjust n_gpu_layers to fill VRAM effectively
    • Use the correct load mode (accounting for deprecations and RAM vs VM assumptions)
    • Consider mlock for long-running stability
    • Consider op_offload_min_batch_size to improve prompt processing speed (micro-batching)
  • Don’t rely on Ollama for fine-grained MOE expert placement control.
  • Context length:
    • MoE reduces KV cache; Turbo Quant can extend context further but may require an unsupported fork.

Main speakers / sources (as mentioned)

  • llama.cpp maintainer
    • Merged flags/patches and provided benchmark context
  • Unsloth
    • Provided 4-bit build details and prior quantization quality numbers
  • Alibaba
    • Shipped the model; license: Apache 2.0; release date cited
  • Google + NYU
    • Authors of the Turbo Quant-related arXiv paper (referenced paper dated April 2025)
  • Threads / “the Threads own verdict”
    • Internal/other commentary voice attributed in the subtitles
  • A 2025 paper on Mixtral
    • Used for speculative decoding performance analysis

Original video