Video summary

SGLang + Qwen: Fastest Way to Serve Local Models Right Now

Main summary

Key takeaways

Technology

Core comparison (same GPU, same Qwen model weights; only the serving engine changes)

The video benchmarks three local serving options for Qwen (“QN”) models:

  1. llama.cpp (single-stream oriented)
  2. vLLM (general-purpose, paged attention + batching)
  3. SGLang (focused on agent/overlapping-prefix workloads with Radix attention)

Headline claim: “SG Lang is the fastest way to serve a local model right now”—but only when your traffic has shared text/prefix overlap (typical for agents/chats/tool calls).

Key technical concepts & product features

1) llama.cpp: best for single-user / single-stream

  • Designed to efficiently handle one request stream at a time.
  • Uses a KV cache strategy that reserves a flat block in VRAM.
  • Under concurrency, throughput flatlines because the reserved KV cache fills.

Example results mentioned

  • ~35 tokens/sec on a 3090 for a single user.
  • With multiple users, it does not scale throughput; stalls/queues once KV cache fills (example given: ~26 concurrent requests on an A10G).
    • With “page rewrite,” throughput can improve to ~247, but the conceptual single-stream ceiling still applies.

Overall positioning

  • Great “one-seat” server, not meant as a fleet/multi-user engine.

2) vLLM: generalist throughput via paged attention + continuous batching

vLLM addresses throughput and KV cache efficiency using:

Paged attention (KV cache efficiency)

  • KV cache is split into small pages (like virtual memory).
  • Pages are allocated only when needed, reducing waste.

Continuous batching (GPU utilization)

  • As soon as one request produces a token, another request can fill the gap.
  • Helps keep the GPU busier, especially under load.

Performance notes

  • Under heavy load: GPU utilization cited around 85–92% for vLLM.
  • Example under unique prompts:
    • Qwen 5.35B, 32 concurrent requests:
      • vLLM: ~1,743 output tokens/sec
      • SGLang: ~1,661

Compatibility / adoption

  • “Runs almost anywhere” (NVIDIA, AMD, Intel, TPUs, Trainium, Gaudi) and has strong community uptake.
  • Example: vLLM 0.20 added DeepSeek-V4 support quickly after weights release.

3) SGLang: fastest when prompts share long prefixes (agents/RAG/structured outputs)

SGLang’s advantage is driven by Radix attention:

Radix attention (prefix reuse)

  • SGLang stores seen prompts in a radix tree / prefix tree of KV cache.
  • For a new request, it reuses cached computation for the longest shared prefix.
  • Only computes the new tail, avoiding repeated “prefill” work.

Why this matches real agent traffic

  • Agents often repeatedly send:
    • the same system prompt
    • the same tool list
    • the same chat history
  • With only a small new suffix each turn.

Benchmarks mentioned

  • Advantage widens as shared prefix grows.

  • Data-center test on H100, 8B model under concurrent load:

    • SGLang: ~16,200 tokens/sec
    • vLLM: ~12,500
    • (~29% edge), attributed to prefix reuse.

  • Hobby hardware example:

    • QN3 54B on 4090:
      • SGLang: ~180 tokens/sec
      • stock vLLM: ~40 tokens/sec
    • Same card/weights; advantage from reuse.

Structured output optimization

  • SGLang compiles required JSON/fixed-schema constraints into a finite state machine.
  • Performs constrained decoding more efficiently (multiple constraint tokens can be advanced in one step).
  • Reported scaling:
    • ~3× throughput on shared chat
    • ~5× throughput on agent/tool workloads
    • Paper cited: up to 6.4× throughput and ~1/3 latency vs contemporaries on heavier structured tasks.

Additional performance engineering (mentioned)

  • Splits prefill and decode across separate workers to prevent one long prompt from stalling others.
  • Enables speculative decoding (Spec v2) and a “Nudlash” kernel by default.
  • Supports lower precision quantization (FP8, Int4, FP4) to fit models on limited VRAM.

“Round” benchmark framing (what traffic pattern decides the winner)

  1. Round 1 (unique prompts / no overlap)

    • vLLM leads; SGLang slightly behind.
    • llama.cpp scales poorly as concurrency increases.

  2. Round 2 (shared long system prefix / agent-like prompts)

    • SGLang pulls ahead increasingly as shared prefix length grows.

  3. Round 3 (high concurrency, e.g., ~100 users)

    • vLLM and SGLang maintain throughput better; llama.cpp eventually flatlines and queues.

Final takeaway

The “fastest” engine depends on workload:

  • Solo / single stream → llama.cpp
  • Mostly unique prompts or non-standard hardware → vLLM
  • Agent/chat/RAG with shared history/prefix and/or structured outputs with fixed schema → SGLang

Practical deployment notes (local model serving)

Setup simplicity (emphasis on SGLang)

  • pip install SGLang
  • Launch a server command pointing at a Qwen model from Hugging Face
  • Provides an OpenAI-compatible endpoint on localhost
  • Often requires no client changes beyond pointing to localhost

Fairness note

  • All tests used the same quantized weights; only the serving engine differed.
  • llama.cpp uses GGUF quantization (about 2-bit to 8-bit).
  • vLLM/SGLang use FP8/Int4 and other quant schemes (e.g., AWQ/GPTQ).

Model-to-hardware guidance

  • Around 24GB VRAM: a smaller Qwen (e.g., “QN4B”) fits well.
  • Mixture-of-experts (MoE) can stay fast because only a subset of parameters fire per token (example: ~3B parameters active per token for a 30B MoE).
  • Example coding-agent target:
    • QN3 Coda 30B near 130 tokens/sec on a 3090 (keeping context reasonable).

Main speaker / sources (as mentioned)

  • Original SGLang paper authors: Zheng, Sheng, and the LMSYS group (the same group behind the Chatbot Arena)
  • Engines discussed: llama.cpp, vLLM, SGLang
  • Industry mention: “Red Hat” (used in the narrative to place/benchmark engines on an H200)

Original video