Local AI: What To Run On 8GB VRAM

Summary

The video argues that local “coding assistant” success on an ~8GB GPU is less about squeezing in a large model and more about sustaining a practical coding loop—ask → read code → patch → run → debug → repeat—without the model bogging down.

Core technological / practical points

The “it loads” problem

Many models start responding, but coding workloads quickly degrade performance. As you iterate through stack traces, multiple files, test failures, and longer context, you see:

• Token speed collapse
• Heavy KV-cache growth (memory used to “remember” context during generation)

Two main resource costs beyond weights

• Context length (and KV cache): More code + more reasoning + longer agent/tool loops ⇒ more cached state ⇒ slower generation.
• Latency from reasoning: Models that “think hard” can be accurate, but may stall the editor workflow.

What matters for coding (not generic chat)

Coding differs from short Q&A. A coding model needs:

• the edited file + surrounding code
• stack traces / test output
• dependency/package versions
• optionally tool results (from agent loops)

The “4 jobs” for local coding AI

1. Code explanation (often easy; many models do it confidently)
2. Patch writing (frequently fails: over-rewrites, misses the real bug, chooses wrong helpers, touches too many files)
3. Debugging / reasoning (must connect failing tests + stack trace + code path + root cause)
4. Repo awareness / multi-file understanding (context becomes the bottleneck)

Model comparison outcomes (high level)

• Kimi / very large agentic models: Dismissed for 8GB—too big for this hardware class, even if exciting via workstation/API/GPU rentals.
• DeepSeek: Strong for hard debugging (good reasoning), but may be too slow / too “careful” for everyday editor rhythm on 8GB.
• Gemma (multimodal): Emphasized for vision-based debugging (screenshots, UI/layout issues, error images, PDFs). For pure text coding, it’s treated more as a specialist.
• Llama (compatibility): Positioned as a safe fallback because it’s widely supported across local runners/quant guides; not the sharpest code specialist, but useful for sanity-checking setup.
• Mistral: Promising generally, but current “interesting” models can push past comfortable 8GB operation.

Scoring method introduced

Instead of “smartest,” the video proposes scoring by least waste, such as:

• wasted memory (barely fits before prompts)
• wasted context (marketing context that’s not practically usable)
• wasted reasoning (deep thinking on shallow tasks)
• wasted specialization (great at one narrow thing, awkward in daily use)

Qwen vs DeepSeek (the practical win)

• Qwen 3 8B is framed as a flexible general assistant.
• But for the narrow goal of coding on 8GB while keeping the loop fast, the video concludes:

Best pick: Qwen 2.5 Coder 7B Instruct Small enough for responsiveness, better suited to quant/local support, Apache-2.0 licensing, and designed to preserve coding “rhythm.”

Why not the larger/more famous ones? They risk speed/context collapse or mismatch with the editing workflow.

Recommended “local setup” strategy (practical workflow)

Keep the setup disciplined: one main coding model, plus optional models (reasoning, vision), and a fallback.

• Main: Qwen 2.5 Coder 7B Instruct

  • Use a sensible 4-bit quant
  • Avoid massive files that force awkward context tradeoffs
  • Start around ~8k tokens, then raise to 16k if things stay smooth

• Second (only for harder bugs): DeepSeek R1 distill

  • Use when subtle logic errors make tests misleading and slower reasoning helps

• Optional third (visual debugging): Gemma (or a Qwen vision route)

• Fallback / sanity check: Llama 3.1 8B

Key “review/test” angle of the video

The “review” is essentially a loop-based benchmark: can the model stay responsive while handling real coding artifacts (stack traces, multi-file context, tests), rather than just producing convincing answers to benchmark-style prompts?

Main speakers / sources

• Single primary speaker/host: the narrator “I” throughout the transcript.
• Models discussed (sources referenced): DeepSeek, Qwen, Gemma, Llama, Mistral, Kimi (including variants like DeepSeek R1 distill, Qwen 2.5 Coder 7B Instruct, Qwen 3 8B, Llama 3.1 8B, Gemma 3, Kimi K2 Thinking, DeepSeek V3-class, etc.).