Video summary

Qwen3.8-27B on 6GB VRAM: Bonsai 27B is HERE!

Main summary

Key takeaways

Technology

Technological concept / product claim (and what the video says to inspect)

  • Turner Bonsai 227B (a 27B-class model) is presented as fitting in about 6 GB while retaining ~98% of a full-precision score.
  • The video argues this sounds like marketing until you inspect:
    • the weight-matrix quantization method, and
    • the runtime implications (compute + KV cache).

Model architecture & parameter breakdown (Bonsai 227B)

  • Base architecture: matches the “Qwen3.8-27B” family (described as aligning with a Qwen 3.8-27B variant).
  • Total parameters: 27.36B
    • Language backbone: 24.35B across 64 blocks
    • Embedding + output head: 2.54B
    • Vision tower: ~0.47B across 27 blocks

Hybrid attention

  • ~75% linear attention layers
  • ~25% full attention layers

Other components mentioned

  • SwiGLU activations
  • RMSNorm

Core feature: ternary (“turnary”) weight quantization + activation rotation

1) Ternary weights

  • Key idea: store most weights as ternary values: -1 / 0 / +1 instead of FP16.
  • Information-theory estimate: ternary carries ~log2(3) ≈ 1.585 bits/weight.

2) Bit-rate landing point (with grouping/scaling)

  • Grouping/scaling: every 128 weights share an FP16 scale factor
  • This adds ~0.125 bits/weight, yielding roughly:
    • ~1.71 bits/weight (theoretical)

3) Low fraction kept at higher precision

  • 26.2 million parameters remain in BF16
  • Claimed as <0.1% of the model to protect sensitive components (e.g., parts connected to normalization and linear-attention recurrent states)
  • Reported averages:
    • theoretical ~1.72 bits/weight
    • packaged-file ~1.76 bits/weight

4) Walsh–Hadamard rotation (blockwise)

  • Uses a Walsh/Hadamard rotation with block size 1,024
  • Purpose: reduce outlier activation spikes so low-bit quantization doesn’t fail

5) Runtime tradeoff

  • Inference must apply a sign flip + fast Hadamard transform to input activations before matmul
  • Adds real compute overhead, especially noticeable at:
    • batch size 1 decoding

Implementation detail: custom kernels and why vanilla loaders break it

  • Weights are still packed (ternary in memory).
  • Custom kernels “unpack” ternary values on the fly during matrix multiplication.

Compatibility trap (critical)

  • You cannot run the weights on stock llama.cpp.
  • Vanilla loading skips required activation transforms, leading to garbage output.
  • To run today, you need:
    • Prism ML’s custom fork of llama.cpp, or
    • their MLX runtime

Download formats / files (GGUF / Apple MLX) and performance tradeoffs

Hugging Face repo (GGUF)

Primary GGUF weight files

  • PTQ10: ~5.93 GB (claimed ~2.10 bits/weight)
  • PQ20: ~7.2 GB (claimed ~2.16 bits/weight)

Vision projector variants

  • Q80: ~0.63 GB
  • Reference BF16: ~0.93 GB

Apple Silicon MLX package

  • ~8.66 GB

PTQ10 vs PQ20 decision (hardware-dependent)

PTQ10

  • ~18% less memory bus traffic
  • Faster token generation on memory-bandwidth constrained consumer GPUs (examples mentioned: RTX 4060 / 4090)
  • But requires more arithmetic cycles to unpack

PQ20

  • Less unpack compute
  • Faster prompt processing/prefill on high-compute hardware (example mentioned: RTX 5090)

Benchmark / evaluation results (what holds up vs what breaks)

General reasoning-suite aggregate

  • Evaluated on a 20-benchmark suite in thinking mode with high reasoning effort
  • Scores:
    • Qwen full precision (Qwen 3.827B BF16): 85.4
    • Bonsai 227B: 83.9
  • Reported retention: ~98.2% (about 1.5-point absolute gap)
  • Compared with conventional extreme quantization:
    • Qwen IQ2XXS (7.3 GB): 75.2

“Synthetic academic” benchmarks

  • Live CodeBench v6: ~90.07 vs ~90.05 (tie)
  • AIME 2026: 95.83 vs 94.58 (Bonsai higher)
  • GSM-AK: 96.66 vs 97.19 (close)

Real software engineering / agent workflows show larger drops

  • SWE-bench verified across 500 instances:
    • Full precision: 80.6%
    • Bonsai: 60.8%
  • TerminalBench 2.1:
    • Full precision: 69.7%
    • Bonsai: 52.8%

Claimed impact

  • Roughly ~25% drop in resolved tasks
  • Reason given: in multi-step agent loops, small precision errors compound over many commands/edits

Reasoning effort matters

  • The headline 98.2% retention applies to X-high reasoning effort
  • At medium effort:
    • retention falls to about 96%
    • AIME 2025 drops from 86.25 to 74.58 (as stated)

Practical guidance / “local coding agents” decision framing

  • The video frames a community question:
    • Would you accept roughly ~25% SWE-bench penalty to run a 27B model on a single consumer GPU / ~6 GB footprint?
    • Or would you prefer:
      • a smaller full-precision model (e.g., 14B), or
      • remote APIs?
  • It encourages viewers to share hardware specs and preferences (comment prompt).

Context window & VRAM reality check (KV cache dominates)

Advertised context limit

  • 262,144 tokens
  • Weight file size: ~6 GB

KV cache cost (FP16 stated)

  • ~64 KB/token

KV cache sizing examples:

  • 4K context: ~0.25 GB
  • 10K context: ~0.63 GB
  • 100K context: ~6.1 GB
  • 262K context: ~~16 GB KV cache

Total VRAM estimate (stated arithmetic)

  • 5.9 GB weights + 16 GB KV + 1.2 GB overhead = >23 GB VRAM
  • Therefore:
    • 16 GB GPUs (e.g., RTX 4070 Ti Super 16GB or 16GB laptops) cannot run full context in FP16

Experimental mitigation: 4-bit KV cache

  • ~18 KB/token
  • ~4.5 GB KV at full context
  • Downsides mentioned:
    • increased decoding latency
    • possible accuracy degradation
    • need for calibration

Speed / throughput + energy

Batch-one decode tokens/sec (PQ2020)

  • RTX 5090: 142.5 tok/s
  • RTX 4090: 90.9 tok/s
  • L4: 29.7 tok/s
  • Apple Silicon
    • M5 Max: 46.8 tok/s
    • M5 Pro: 27.7 tok/s

Energy (example)

  • RTX 4090: 0.714 mWh per token (as stated)

Main takeaway / analysis conclusion

  • Bonsai 227B is not a universal drop-in replacement for full-precision software engineering agents.
  • It does show that ternary weights + activation rotations can preserve enough reasoning quality to make a 27B-class model workable in ~6 GB.
  • However, as weight size shrinks, the limiting factor shifts toward:
    • KV cache capacity (especially for long context)
    • stability/accuracy over long agent loops, where errors compound

Main speakers / sources (as mentioned in the subtitles)

  • Prism ML (developer/vendor of Turner Bonsai 227B and associated runtimes/benchmarks)
  • Repo Chad (channel referenced for “Repo Chad channel memberships” / deeper technical scripts)

Original video