I Tested Every Qwen3.8-27B Quant: Here’s the Best One For Your GPU

Tech/Product Summary (Qwen3.8-27B quantized model testing)

Core problem with local quantized LLMs

• The speaker reports that a locally running quantized Qwen 3.8-27B could be “quietly broken.”
  • GPU utilization/temps and dashboards look fine.
  • Tokens generate normally.
  • There may be no error logs.
• Behavior drifted from the full-precision BF16 version, especially on longer / multi-step context, suggesting quantization can damage context tracking without obvious warnings.

What quantization means here (and why simple “bigger Q is better” fails)

• Full precision reference: Qwen 3.8 27B is BF16 ~55GB.
• Quantizers reduce weight precision to fit in GPU VRAM (the speaker uses a suitcase analogy).
• Key architecture detail (Qwen 3.8): 64 layers total
  • 16 attention layers
  • 48 gated DeltaNet layers (a linear-attention variant with internal running state that effectively “remembers” earlier prompt information)
• Aggressive quantization can compress DeltaNet state values, causing:
  • long answers to drift
  • earlier conversation context to stop sticking
  • outputs that look “plausible” but are fundamentally off

What matters more than the “Q4/Q8” number: what components are protected

The speaker’s argument: the decisive factor is what the quantization packer chose to protect, not average bits-per-weight.

• Successful quants aim to keep critical components at ~5 bits or better, especially:
  • output layer
  • attention value projections
  • down projections
• They offload more compression onto embeddings and DeltaNet / linear-attention state in a controlled way.

Examples of “smart” quant approaches mentioned:

• Ridge
  • ~11.7GB
  • avoids compressing DeltaNet state
  • compresses elsewhere
• Unsloth
  • described as a dynamic approach (allocating precision where it matters)
• Bartowski
  • uses an importance matrix calibration pass
  • the calibration corpus was skewed toward tool-calling traffic (63%) instead of general prose

Plain/uniform quant is risky

• A “plain Q4 applied uniformly” can lock into the DeltaNet-state failure mode:
  • no visible breakage
  • but compromised behavior (e.g., quiet forgetting)

---

Evidence: benchmarking what bit-width actually preserves

Quant-maker benchmark snippet (Atomic Chat / KL divergence style)

• A seller/internal benchmark reports KL divergence across quant levels:
  • 4-bit: top token match ~96%
  • 8-bit: top token match ~98.9%
  • Higher bit widths increase file size; step above 8-bit is framed as diminishing returns
• Caveat: the speaker notes this comes from the seller, so treat it directionally.

Independent benchmark (Quesma)

• Quesma allegedly spent ~$3,000 on rented GPU time to test four quants on:
  • GPQA, Diamond, IFBench, TerminalBench
• TerminalBench: 89 real terminal tasks in containers (not “vibes”).
• Findings emphasized:
  • 4-bit matched BF16 outright on TerminalBench
  • instruction following degraded only down to 2 bits (surprising to the writer)
  • 1-bit dropped sharply: reasoning benchmarks neared random guessing
• Unsloth marketing claim mentioned:
  • “~72% top-1 accuracy while being 89% smaller”
  • the speaker argues the missing portion is substantial; effectively “don’t run that one.”

Practical takeaway from benchmarks

• 4-bit is the floor
• 8-bit is where things get “hair-splidy”
• 1-bit is effectively a science experiment

---

KV cache quantization mistake (major “quiet failure” warning)

• The speaker warns against copying a speculative decoding recipe that sets KV cache to Q4 compression.
• Benchmark vs a different ~7B coder model showed:
  • Q8 cache: output similarity ~81.6% vs full precision
  • Q4 cache: similarity drops to ~8.3%
• Crucially:
  • throughput/dashboards stayed “green”
  • generation speed stayed fast
  • answers became substantially different without obvious failure signals
• Result for the speaker:
  • Don’t use Q4 KV cache compression; keep KV cache at Q8 or full precision
• Note: they couldn’t provide the exact Qwen3.8 27B numbers because the benchmark was run on a different model, but they caution that the same problem likely applies.

---

File selection guide (recommended downloads by GPU VRAM)

The speaker provides concrete model/file recommendations and warns about version drift.

~24GB GPU (RTX 3090/4090 class)

• Download: Unsloth UDQ4KXL
• Size: ~17.92GB
• Leaves ~7GB breathing room
• Supports ~32–64K context with 8-bit cache
• Important: pin the revision
  • Unsloth replaced original builds on Aug 19
  • newer benchmark references may point to files that no longer exist

~32GB GPU

• Use: Q6 or Q8 variant
• Rationale: closer to near full-model behavior / less aggressive tradeoffs

~16GB GPU

• Use: Bartowski’s IQ4XS
• Rationale:
  • smaller
  • uses importance-matrix calibration
  • tuned for likely usage patterns (tool-calling)

≤12GB VRAM

• Recommendation: run a smaller model instead
• Reason: 2-bit variants diverge badly from full precision (speaker claims worse than “1 in 8 tokens”).

---

Multimodal (images/screenshot) gotcha

• Vision/projector is separate from the main weights.
• Many “vision” users fail to include the projector file, then assume the model can’t see images.
• Guidance: download the vision projector separately for screenshot/image capability.

---

Settings that affect real-world quality/performance more than Q level

The speaker says day-to-day results depend heavily on settings beyond “Q4 vs Q6”.

1. Reasoning effort

   • Default is highest thinking mode (X high).
   • Turning reasoning off reduced runtime dramatically (e.g., minutes → seconds).
   • For summarization/chat/quick coding: turn it down/off.

2. Speculative decoding

   • The model includes a built-in draft head; set spec type to draft MTP to gain speed.
   • Speaker notes self-reported improvements (e.g., ~31–41 tokens/sec on a 3090).
   • Mac users: this flag reportedly does nothing (no speedup on M4).

3. Chat template / “ginger flag”

   • Correct front-end handling is required so the chat template nests thinking blocks properly across multi-turn history.
   • If mishandled: history may truncate in weird ways and quality slowly degrades over long sessions.

---

Meta-lesson: benchmarks and files can become outdated

• Even the “best independent benchmark evidence” described may be invalid now, because tested artifacts were later replaced and may be unavailable.
• Key points:
  • benchmarks are snapshots tied to specific model files + cache settings at the time
  • the “file name number” is only the 4th most important factor
  • more important: what’s protected in quantization, KV cache setting, and reasoning effort

---

Main speakers / sources

• Primary speaker: an individual reviewer/testing creator (described as “I tested…”, “I posted…”) discussing Qwen3.8-27B quant variants.
• Mentioned sources/teams:
  • Alibaba (Qwen3.8 release context)
  • Atomic Chat (internal KL-divergence benchmark referenced)
  • Quesma (~$3,000 independent benchmarking)
  • Unsloth, Bartowski, Ridge (quantization approaches and specific files)
  • Speculative decoding recipe community (as the origin of KV-cache Q4 guidance)