Video summary

Colibri – Run frontier MoE models on hardware you already own – pure C.

Main summary

Key takeaways

Technology

Colibri (v1.11.0) — key tech/product ideas in the subtitles

Purpose / promise

  • Runs frontier Mixture-of-Experts (MoE) models on existing consumer hardware by using local storage/RAM/VRAM rather than renting hyper-scaler GPUs.
  • Speed is not guaranteed (explicitly: “no SLA on speed”).
  • Semantics are protected: the engine should not silently change model precision or router behavior when resources are limited.

Engine design

  • Engine is written in pure C with zero engine dependencies.
  • One C file per model family, aiming for readable end-to-end code (not a plugin/dependency tree).
  • Shared frontends:
    • colibri chat (terminal)
    • colibri serve (model as a service)
    • colibri web (dashboard)

Supported models / scale

Examples mentioned in the roster include:

  • GLM 5.2 / 5.3 (744B)
  • GLM 5.3 Flash (321B, vision)
  • Inkling (975B)
  • Kimi / “Kimmy Tay 3” (2.8T)
  • DeepSeek V4 Flash (284B) / V4.1 Flash (552B, vision)
  • Qwen variants, including:
    • “Qwen 38 Flash next” (125B + 51B N-gram component)
    • “Qwen 3.6” setup
  • OLMoE (7B) for quick build testing

The write-up suggests support for INT4 quantization (at least for the 744B demo).

Mixture-of-Experts core concept

  • MoE routes each token through a small subset of experts; most experts stay idle per token.
  • The router selects experts per layer/token, so while total parameters can be huge, active parameters per token are smaller.
  • This enables the system question: if experts are mostly idle, you may not need to keep all weights in the fastest memory.

AI Memory multi-tiering (weight placement across VRAM/RAM/NVMe)

  • Treats VRAM, system RAM, and NVMe as one inference memory hierarchy for model weights.
  • Pulls experts from NVMe only when needed, rather than loading everything up front.
  • Uses a “pyramid” model:
    • Top: VRAM (fast, small, expensive)
    • Middle: RAM (slower, larger, cheaper)
    • Bottom: NVMe (slowest, but very large capacity)

Runtime strategy to reduce storage bottlenecks

Framed as “JIT for weights” (just-in-time weight loading). Main mechanisms:

  1. Per-layer LRU cache: keeps recently used experts in fast memory; evicts least recently used.
  2. Learned “pinned” hot store: uses routing frequency (“routing heat”) to keep consistently used experts resident.
  3. One-layer-ahead prefetch: while computing layer N, prefetches experts needed for layer N+1.

Conceptual loop:

route → ensure chosen experts are in memory (load from NVMe if needed) → compute → repeat

Additional IO/storage techniques

  • Batched expert unions: if tokens need overlapping expert sets, load the union once.
  • Overlapped reads and compute: overlap SSD reads with ongoing compute.
  • O_DIRECT: bypasses OS page cache to avoid duplicate caching/collisions with the engine’s own caches.
  • Weighted dual SSD striping: splits reads across two NVMe drives (RAID 0–like) and biases share by drive speed.

Important caveat:

O_DIRECT and related IO behaviors are drive/host dependent; community AB tests are needed.

Heterogeneous execution

  • A single runtime supports multiple hardware types (mentions CPU, CUDA, Metal, NUMA).
  • Mixed expert residency is supported (some experts on GPU, some on RAM, some on disk, etc.).
  • Goal: avoid separate “bolt-on” code paths per device type; instead configure the same runtime for the machine.

Demo results highlighted

  • Streaming CPU demo

    • Example: GLM 5.2 (744B) at INT4
    • Ready in ~32 seconds
    • Resident memory: ~9.9 GB
    • Routing loads experts from disk when needed

  • 6x RTX 5090 full GPU demo

    • ~4 tokens/sec
    • Time to first token: ~1.6 seconds
    • Disk counter reads: zero (full expert residency on GPU)

Correctness/identity vs performance cheating

Strong emphasis that the default policy should not:

  • silently change quantization/precision,
  • reduce the number of experts per token than the model expects,
  • skip experts purely to “cheat” benchmarks for speed.

If performance drops, it should be treated as a speed impact (e.g., insufficient fast memory), not a silent change in “what the model is.”

Visualizations

  • “Brain” page

    • Live visualization of all experts as a “cortex”
    • Color indicates current storage tier (VRAM/RAM/disk)
    • Brightness indicates experts routed during the turn
    • Hover shows topic affinity for experts

  • “Atlas” page

    • 3D expert clustering (“galaxy”)
    • Experts characterized into topic clusters, including replicated specialists
    • Not based on learned embeddings; described as based on measured routing affinity (observed router behavior)

What the project asks for / gaps

The video notes missing public benchmarks, such as:

  • No tokens/sec figures for the streaming CPU mode
  • Unclear exact host/drive configurations for specific tests
  • More measurements needed for other architectures (e.g., dual SSD striping effectiveness)

Project direction:

  • Calls for reproducible end-to-end measurements, not just microbenchmarks.

Main speakers / sources

  • Primary source: The narrator/author of the YouTube video reviewing and analyzing Colibri (GitHub project, v1.11.0) using the repository README and demos.
  • Project being discussed: Colibri / Colibri GitHub repository, referenced via its README and dashboards/demos.

Original video