Video summary

GLM 4.5-Air-106B and Qwen3-235B on AMD "Strix Halo" AI Ryzen MAX+ 395 (HP Z2 G1a Mini Workstation)

Main summary

Key takeaways

Technology

Technological focus (models + hardware)

  • Runs extremely large quantized LLMs on an HP Z2 Mini G1a workstation using AMD’s new “Strix Halo” / Ryzen AI Max, with unified memory up to 128 GB.
  • Highlights two model types:
    • Qwen3-235B (“~235B parameters”)
      • Uses Q3 Excel quantization
      • Consumes >100 GB RAM
      • Runs smoothly
    • GLM 4.5-Air MoE (Mixture-of-Experts)
      • ~106B parameters
      • 12 active experts
      • Uses Q6 Excel quantization
      • Achieves >16 tokens/sec
  • Key performance claim:
    • Strix Halo works best with MoE models, while dense models can perform worse due to memory bandwidth limitations.

Product/features and why this setup exists

  • Motivating problem:
    • Local LLMs on typical consumer GPUs (especially laptops with ~8 GB VRAM) require heavy quantization and still can’t reach “serious work” sizes compared with GPT via API.
  • Unified memory as the enabler:
    • Like Apple M-series, the integrated GPU can access system RAM directly (up to 128 GB).
  • Hardware sourcing narrative:
    • Early Strix Halo devices (tablets/laptops) had power/thermal constraints.
    • Chinese mini PCs raised cooling/quality concerns.
    • HP Z2 Mini G1a was chosen for workstation build quality, compactness, and cooling design.

Reviews / third-party references mentioned

  • Notebookcheck review of the HP Z2 Mini (focused mainly on gaming benchmarks):
    • Reports the integrated Radeon 8060S performs around an RTX 4070 laptop-class GPU.
    • Can handle high CPU load without thermal throttling.
    • Louder under load, but manageable by using another room.
  • Community sources for Strix Halo configurations:
    • Strix Halo Homelab D7 WTF website + active Discord community.
    • LHL’s GitHub repo: Strix Halo testing/benchmarks and configuration quirks.

Guide/tutorial: “best performance” setup steps

1) BIOS

  • Allocate the minimum GPU memory: 512 MB (counterintuitive but recommended).

2) OS / Kernel

  • Use a modern Linux distribution.
  • Tested on Fedora 42 with kernel 6.15.6 and 6.16.

3) Kernel / GRUB parameters

  • Add kernel parameters to unlock unified memory performance:
    • Mentioned: set GDT size for unified memory access
    • Mentioned: disable IO MMU for lower latency

4) Containers / GPU driver stack

  • Use toolbox (Fedora-based containers).
  • Prefer Vulkan RADV Toolbox container for stability/compatibility.
    • AMD VLK may be faster, but can fail on some models due to buffer allocation limits.

5) Toolbox validation

  • Inside the container:
    • Run: Llama CLI list devices
    • Confirm CPU/GPU driver visibility (via Vulcan RADV driver).

6) Model selection + download

  • Suggest using Hugging Face “ANS/ExLlama quantized models” in a quantization size that fits memory.
  • Use Hugging Face CLI to download GGUF weights.

7) Inference commands

  • Run with llama.cpp / Llama CLI using:
    • --ngl 99 (offload layers to GPU)
    • --no-mmap (mentioned to prevent issues during slow/failed loading, especially with ROCm)
  • For a web UI:
    • llama serve

Quantization concept explained

  • Quantization compresses model weights (e.g., 32-bit → 8-bit/4-bit/1-bit) to fit memory, trading off quality for capacity.
  • Practical takeaway:
    • Local laptop-sized setups usually only fit ~1B–8B models, and even with quantization they remain limited versus API/GPT-class models.

Backend comparison (key analysis + “which to use”)

Main inference backends discussed

  1. Vulkan RADV (open-source Mesa/community)

    • Stable and “just works.”
    • Successfully loaded and ran every tested model.

  2. Vulkan AMD VLK (AMD’s driver)

    • Can be faster (sometimes >50% in prompt processing).
    • But fails for some models due to a 2 GB Vulkan buffer allocation limitation, causing load failure on certain models.

  3. ROCm / Rock M

    • Potentially best for certain formats (notably BF16).
    • Problems:
      • Without correct flags, model loading becomes extremely slow (hours for large models).
      • Requires --no-mmap to avoid pathological loading times for >64GB scenarios.
      • Stability issues: intermittent GPU crashes/hangs.

Benchmark methodology + key results (Llama Bench)

  • Bench tests two metrics:
    • Prompt processing speed (time until the model starts responding)
    • Text generation speed (tokens/sec while streaming output)

Observed patterns

  • AMD VLK (when it loads) is fastest for prompt processing:
    • Example (described as “Gemma 12B”):
      • VLK: ~686 tok/s prompt
      • RADV: ~509 tok/s prompt
  • For some models, AMD VLK can’t even load (e.g., BF16 or large sizes).
  • RADV
    • Slightly slower than VLK
    • But reliable across models.
  • ROCm (Rockam)
    • Strong for Qwen models in BF16:
      • Example claim: prompt processing >150 tok/s
      • generation ~23 tok/s
      • described as ~3× faster than Vulkan for generation
    • But frequent instability + load-time problems without --no-mmap.

Practical TL;DR from the video

  • Try AMD VLK first if the model loads successfully.
  • Otherwise use RADV (recommended default for compatibility).
  • Keep ROCm/ROCKm as an option mainly for BF16-format models, but expect stability/workflow quirks.

Memory + context window analysis (important technical point)

  • Not only weights must fit RAM; context window also consumes memory.
  • Context memory scaling:
    • Normally scales quadratically with token count,
    • but Flash Attention makes it more memory efficient (and is supported by llama.cpp).
  • They created a “GGUF VRAM estimator” Python script:
    • Predicts RAM needs for different context sizes and quantizations.

Example calculations (128 GB RAM)

  • Llama 4 Scout with 1M token context at Q4:
    • weights ~58 GB
    • context ~49 GB
    • total ~109 GB (fits “just about”)
  • Qwen maximum context estimate for that model:
    • ~131k tokens
    • requiring ~122 GB (suggest leaving headroom)

Latency impact

  • Longer context increases latency:
    • Example: a question over >100,000 tokens leads to very slow processing.

Speakers / sources (as end-identification)

  • Main speaker: the video creator/host (not named in the subtitles).
  • Primary external sources referenced:
    • Julia Turk (linked video on quantization)
    • Strix Halo Homelab D7 WTF (website + Discord)
    • LHL (maintains the Strix Halo testing GitHub repository)
    • Notebookcheck (review of HP Z2 Mini / gaming-focused benchmarking)

Original video