Video summary

Gemma 4 Runs at 255 Tokens/sec in Your Browser Locally, No Server, No Install

Main summary

Key takeaways

Technology

Local on-device Gemma 4 WebGPU demo (no server, no API, no install)

A Hugging Face demo purportedly runs Gemma 4 E2B at ~255 tokens/sec inside a normal browser tab (example shown on a MacBook), using WebGPU and publicly available kernels. Users can:

  • Download weights locally
  • Verify speed with an on-page tokens/sec counter

Controversy: “agent optimization ceiling” and transparency

  • Kernel authorship (alleged): The kernel code was allegedly authored by an AI model agent (Fable 5), which was globally suspended ~24 hours after completing the task.
  • Optimization progression (alleged):
    • During that window, the agent reportedly optimized Gemma 4 to ~84 tokens/sec, then stopped after claiming it hit an apparent ceiling.
    • After Anthropic allegedly rolled back “invisible LLM development safeguards,” the same hardware + same task supposedly enabled further optimization to ~255 tokens/sec (about 3×).
  • Suspension effects (alleged):
    • After Fable 5 was suspended, benchmark scores were reportedly zeroed out, leading to skepticism in comments.
  • Resolution approach:
    • Joshua “Xenova” Lockner released the entire artifact (demo + WebGPU kernels) so others could re-run and verify.

Who’s behind the tooling

  • Joshua Lockner (Xenova): Known for transformers.js, a library for running ML models in JavaScript/browser.
  • The kernel + demo work is associated with Xenova hosting the Gemma 4 WebGPU kernels on Hugging Face.

Gemma 4 context (why architecture matters for performance)

Gemma 4 was released by Google (early April 2026) in multiple sizes, including:

  • E2B / E4B for phones/edge
  • 26B MoE (~4B parameters active per token)
  • 31B dense workstation model

Additional notes:

  • Models are under Apache 2.0 (noted as a first for the Gemma family).
  • The demo specifically uses Gemma 4 E2B, stated as:
    • 5.1B total parameters
    • 2.3B active per token
  • It uses PLE (Per Layer Embeddings): each decoder layer uses its own small embedding lookup per token. The video claims this can make lookups “cheap” while enabling representational depth benefits.

Claimed optimization techniques (and order)

The video presents an “optimization log” showing staged kernel improvements:

  1. Dedicated dense GEMV path for PLE projections with FP16-safe intermediates → ~76 tok/s
  2. Fused kernel combining PLE gate + GEU activation + multiply into one GPU pass → ~83 tok/s
  3. After safeguards change: retuned row counts and reduced staging → ~121 tok/s
  4. Rewrote prefill kernels with unrolled tile branches and vec4 loads (4 values per memory transaction) → ~173 tok/s
  5. Final: retuned attention geometry, increased workgroup size to 256 threads → ~253 tok/s
  6. Validation → ~254.8, rounded to 255 tok/s

The video emphasizes these are standard GPU optimization methods (fusing, vectorized loads, workgroup tuning), but stresses that an agent performed the end-to-end search/ordering, not a human hand-tuning pass for a specific model/chip.

How to run the demo (tutorial/guide-style steps)

  1. Go to Hugging Face WebML community / “gemma 4 webgpu kernels” space.
  2. Click “load model” (downloads Gemma 4 E2B weights locally).
  3. Use a browser with WebGPU enabled:
    • Chrome/Edge: should work out of the box
    • Safari: works on updated macOS
    • Firefox: may require enabling a flag
  4. After weights are cached, everything runs locally (the video claims you can even disable Wi‑Fi and continue).
  5. The chat UI streams output with a live tokens/sec counter; speed depends on GPU (example shown from an M4 Mac).

Benchmark comparison and why it mattered

  • In an Agent Swarm challenge on Hugging Face, the cited record is 508 tok/s on an A10G (data center GPU).
  • The video argues that ~255 tok/s on a consumer laptop chip via browser WebGPU is closer to real-world practicality and was underestimated by skeptics.

Follow-up claim: generalization to other models

The same approach is alleged to have continued with other agents/models after Fable 5’s suspension:

  • Opus 4.8 allegedly improved Liquid AI’s Tiny LFM 2.5 to ~1,400 tok/s in-browser.

Open questions / missing explanations

Anthropic reportedly hasn’t detailed:

  • what the “development safeguards” were,
  • why removing them tripled the optimization ceiling,
  • or when access will return.

Main speakers / sources

  • Joshua Lockner (“Xenova”) — creator of transformers.js; released the Gemma 4 WebGPU kernels and demo.
  • Anthropic’s “Fable 5” — referenced as the agent model that performed/custom-wrote the WebGPU kernels.
  • Google — referenced as the issuer of Gemma 4 (under Apache 2.0).

Original video