Video summary

Is Local AI Coding Actually Good?

Main summary

Key takeaways

Technology

Summary of technological concepts, features, and findings

Core thesis

Local AI coding can be good, but it’s only clearly practical with high-end hardware. Even then, it often hits speed/compatibility limits compared to cloud coding models. A hybrid approach is recommended for most users.

Hardware constraints & model sizing

Systems used for testing

  • Windows workstation: RTX 4090 (24 GB VRAM) + 64 GB RAM (DDR5 implied).
  • Mac reference: M5 Max with 64 GB unified memory, but similar practical limitations were observed for “reasonable speed.”

Key bottleneck: memory capacity

  • VRAM/unified memory largely determines the maximum model size that runs without severe slowdown.
  • With 24 GB VRAM, they estimate models up to roughly ~30B parameters can be feasible depending on quantization.

Performance cliff: spilling into system RAM

  • If the model overflows into system RAM, token generation speed can drop from about 70–80 tokens/sec to roughly ~2 tokens/sec.

System-wide lag from other GPU/RAM apps

  • Even if the model “fits,” other applications using resources (e.g., Chrome, Slack/Discord) can still cause noticeable lag.

Evaluation criteria (how models were compared)

They compared coding models—especially local ones—using:

  1. Speed: tokens/sec and perceived “snappiness”
  2. Quality: frequency of mistakes and suitability for advanced tasks
  3. Compatibility: whether coding tools/agent harnesses work smoothly with local backends (e.g., LM Studio/Ollama)
  4. Cost: whether local setup truly saves money vs paid API/cloud usage
  5. Control: a major reason to go local—choosing/owning the model and avoiding provider changes after download

Compatibility & tooling friction (important limitation)

  • Local backends such as LM Studio and Ollama may work, but agent harness integration can be incomplete or error-prone.
  • Cursor is described as their preferred agent harness, but:
    • LM Studio/Ollama models can be tricky to configure in Cursor
    • local backends may lack “native” features like tool calling / agent capabilities
  • As a result:
    • they often fall back to VS Code for more reliable local integration,
    • though provider-native setups are still stronger.

Local model setup & inference optimizations

Serving approach

  • They used LM Studio, which can expose a local server and host multiple models for different devices/apps.

VS Code integration

  • Used VS Code’s “manage language models” feature:
    • added an LM Studio instance via URL/JSON config
    • set context window size
    • selected model IDs

Quantization used

  • Q4KM was used to fit models on GPU.

Recommended inference optimizations for dense models

For dense models, they recommended enabling:

  • KV cache / K cache quantization
  • Flash attention

Effect: reduced GPU memory usage (example cited: about ~24 GB down to ~20 GB, saving ~3 GB), improving memory headroom and stability for larger contexts.

Model lineup and observed behaviors

Local models tested

  • Mistral “Devster small” (~24B, 15 GB)
  • Qwen 3 Coder 30B
    • ~30B parameters, 18.6 GB
    • described as MoE-style with ~3B active parameters
    • reported as the best performer for speed
  • Qwen 3.6 27B (dense model)
    • full dense: all parameters active
    • slower inference than the MoE model
  • Gemma 4 (smaller; less explored)

Coding demos & measured outcomes

Tetris demo (VS Code + local model)

  • The MoE model generated a fully functioning Tetris game quickly.
  • Speed was tracked via token streaming (and a prior run reportedly completed before results were checked).
  • Output quality was described as “usable,” with relatively little manual intervention.

Mario platformer fix (different local model)

  • The dense model was slower:
    • observed roughly ~30–33 tokens/sec between messages
    • claimed about 2–2.3x slower than the faster MoE model
  • Quality/correctness sometimes improved with dense models.
  • They also hit an interactive testing error in-browser, implying a payload/tool execution mismatch between harness expectations and local tool-calling behavior.

Cost comparison with cloud (API billing)

Compared against

  • Anthropic Sonnet 5 via API billing.

Observations

  • Cloud runs produced higher quality results and avoided some local harness failures.
  • Example cost snapshot: ~$2.42 recorded on the Anthropic dashboard for their run.
  • They note real costs can be lower due to subscriptions/subsidies.

Claimed tradeoff

  • Cloud can be 5–10x more expensive than some “cheapest” local setups (depending on provider choice),
  • but is often not dramatically faster—it’s mainly higher quality and more reliable.

Final verdict / recommendations

When local is worth it

  • Local models are “pretty good” and can be adequate if you have high-end hardware.

When to avoid local

  • Don’t buy a ~$5,000 GPU/PC solely for local coding.
  • If hardware is low-end, local coding will likely be too slow and lower quality.

Best practical approach (hybrid)

  • Use cloud as the primary option:
    • faster, higher quality, better agent/tooling support
  • Use local as a secondary/private/backup option:
    • when you need privacy
    • when you’re quota-limited
    • for cost stacking / always-on use cases outside coding

Privacy and always-on automation scenarios were highlighted as the main cases where local makes the most sense.

Main speakers / sources

Speaker/source

  • The video narrator/reviewer (the main person performing hardware testing and comparisons).

Tools/products mentioned

  • LM Studio, Ollama (local model serving)
  • VS Code (integration/testing environment)
  • Cursor (preferred agent harness; local integration issues discussed)
  • Anthropic (Claude models; “Sonnet 5” used in cost/quality comparison)
  • Mind Sub Co-worker (sponsor; an open-source model router / “Anton” agent harness enabling switching across multiple providers/models within one workspace)

Original video