Video summary

Is Bonsai 2 27B REALLY 98% of Qwen 3.8 27B fp16?

Main summary

Key takeaways

Technology

Summary of the Video’s Technical Content (Bonsai 2 / Qwen Comparison)

Goal

  • The video tests whether Turnary Bonsai 2 27B (claims: “98% of Qwen 3.8 27B fp16”) can run well on consumer GPUs with limited VRAM.
  • Focused hardware:
    • Single RTX 3090 (24 GB)
    • Two RTX 3060 (12 GB each) (speed comparison mentioned)
  • Main theme: quality vs speed/VRAM feasibility for running a ~27B model locally.

Key Model / Quantization Concept: Turnary vs Traditional Binary States

  • Traditional binary-like representation uses -1 and +1 states.
  • Turnary representation uses -1, 0, +1 (three states).
  • Provider claim:
    • Turnary should deliver better quality at smaller sizes.
    • The full 27B model can fit in 24 GB VRAM with minimal quality loss (as asserted by the creator/provider).

Deployment Approach: Proxmox Container + llama.cpp Server

  • The video is tutorial-oriented and assumes the viewer already followed a guide:
    • Digital Spaceport Proxmox 9 LXC guide
  • Test setup details:
    • Runs inside a container (noted as container 118)
    • The model is downloaded automatically during the run
    • Server execution uses a command from a Git repository via scripts
  • Hugging Face access:
    • The run may prompt for a Hugging Face key
    • The creator skips it for the demo since the model downloads quickly enough
  • Environment/config variables and networking:
    • bonsai_host is used to override localhost
    • The server binds to 0.0.0.0 so it is reachable across the internal LAN
    • Expected access format: something like 192.168.1.61:8080
  • Mentioned tuning knobs (not enabled for this test):
    • Speculative decoder
    • KV cache set to 4
    • Context size left at auto
    • For this specific evaluation, speculative / KV4 were not enabled.

Hardware / Performance Results (RTX 3090 Single-Card Test)

  • GPU memory footprint
    • The running model uses about 16.77 GB of GPU memory (within 24 GB available), leaving headroom.

  • Throughput / token speed

    • Initial generation: roughly ~69 tokens/second (stays in the high-60s)
    • For longer/agentic work: prompt processing noted around ~1,250 tokens/second (prompt processing rate differs from decode speed)

  • Context length

    • Context set around 131
    • Creator notes this may be too low for some workflows and suggests increasing it for “agent” style usage.

Implementation Detail: llama.cpp Fork Requirement

  • The creator notes that upstream llama.cpp support isn’t integrated yet for this setup.
  • As a result, they use a specific fork to get correct outputs.

Output Quality Evaluation

Quick Test: SVG “Cat Walking”

  • Image quality described as:
    • Creatively strong (cat and some elements look good)
    • But not equal to FP16 quality (creator notes it’s “not as good as FP16”), despite the impressive speed.

Arcade Benchmark-Style Suite (Compared to FP16 Qwen Variant)

  • The video runs a “full suite” of arcade-console-style tasks and compares results to a previous FP16 run of Quinn 3.8 27B.
  • Observed issues include:
    • Font/UI problems
    • Gameplay/control problems
    • Incorrect or broken behaviors, e.g.:
      • boss shooting in the wrong direction
      • movement/control feeling broken
      • escape not working
  • Creator’s example ratings:
    • Neon Laner: “C minus / D plus”
    • Space Racer: “impossible to play”
    • Pixel Breaker: “two Fs and a D+” (overall very poor)
  • Conclusion regarding the “98%” claim:
    • Creator believes it is not really 98% of the FP16 experience.
    • Quality differences are “definitely different” and “not as good.”

Agentic Workflow / Tool Calling Behavior

  • Positive:
    • Chat speed is very good
    • Tool calling accuracy appears fine (creator did not notice missed tool calls)
    • Chat + “cat” type responses are smooth
  • Negative:
    • Quality degradation impacts agentic coding and complex tasks
    • For “agentic code development,” the creator suggests it is probably not a good time

Speed vs Quality Tradeoff Conclusion

  • The model appears:
    • Substantially faster than FP16 on comparable setups
    • Usable for a chat + single stream agentic workflow
  • But:
    • Not equivalent to FP16 in quality
    • Especially problematic for arcade/action-heavy tasks.

Pricing / Hardware Context (Brief)

  • Mentions GPU used prices tracked on DigitalSpaceport:
    • RTX 3090: lowest used around $1,400 (noted recent increase)
    • RTX 3060: lowest used around $300
  • Notes that RTX 5090 price increases also pulled 4090s / 3090s upward.

Main Speakers / Sources

  • Main speaker: The video author/host (unnamed in subtitles), running the experiments and comparing results.
  • Referenced sources and components:
    • Turnary Bonsai 2 27B (Turnary method and model claims)
    • Digital Spaceport (Proxmox 9 LXC guide, GPU pricing pages, and arcade site)
    • Llama.cpp (including a fork for correct support)
    • Git repository for the model/server scripts
    • Hugging Face (optional access key)

Original video