Gemma4 12B Coder - Composer 2.5 × Fable 5 v2 vs base - 16GB Local LLM setup

Overview (models + setup)

• Speaker: Luke (“Luke’s Dev Lab”)
• Goal: Compare Gemma 4 12B fine-tune vs base, using 4-bit quantization (Q4_K_M, GGUF) for both to keep conditions consistent.
• Base model: Gemma 4 12B GGUF (sourced from “Unsllo” per subtitles)
• Fine-tune:
  • Created from training data based on reasoning/outputs from Fable 5 and Composer 2.5
  • Fine-tuned by the Cursor team on “K2.5” (as named in subtitles)
• References: Prompt/configs are linked in GitHub; model pages are also linked in the video description (per subtitles).
• Hardware mention: A separate local inference system with 16GB VRAM, which can “comfortably fit” the model. 32GB DDR4 is described as not relevant here.

Test suite performed

1) Performance benchmarks

What was measured

• Prefill speed: input/token ingest processing
• Decode speed: token generation

Result

• Base and fine-tune are essentially the same (≈ 23–24 across the board, within margin of error).

---

2) Memory / long-context retention test

Setup

• Uses 128K context.
• Places “key” data at 0%, 25%, 50%, 75%, 100% depth of context.
• Retrieves the key data and repeats tests 3 times each.

Result

• Fine-tune: 100% pass rate
• Base: 87% pass rate

Base failure modes

• At 25% depth: failed 1 out of 3 runs
• At 100% depth: failed 1 out of 3 runs (“lost in the context”)

Attribution

• The failure is attributed to a large Linux kernel PDF used as the document.

---

3) Agency benchmark (tool-using “fake company” sandbox)

What the models were asked to do

• Use tools to complete tasks involving:
  • Reasoning + tool chaining
  • Scenarios like booking rooms/appointments, department/people lookup, and currency conversion via tool chains.

Result

• Base: 91%
• Fine-tune: 52%
  • Described as the worst agency result the speaker has seen.

Observed fine-tune failure patterns

• Overuse/misuse of tools (e.g., repeatedly creating support tickets when not appropriate)
• Getting trapped in loops (finishing tasks then redoing them)
• Hitting “trap” tools (wiki mentioned as should “never be called”)
• Wrong/incomplete tool usage in currency conversion
• Incorrect outputs for headcount (returned 2 instead of 3)
• Room-booking conflict handling failure (fine-tune created tickets / did employee lookup instead of choosing alternatives)

---

4) OpenAI HumanEval (164 Python challenges)

What was tested

• The model generates Python solutions.
• Correctness is verified by running Python.

Runtime

• ~7 hours per model on the speaker’s machine.

Result

• Fine-tune is only ~10% worse than base (not catastrophic given earlier agency failures).
• All 164 answered for both.

Pass counts

• Base: 156 passed
• Fine-tune: 140 passed

Additional note

• The speaker observes occasional looping/strange “thinking” behavior even in the base.

---

5) Sand physics simulator (coding/workflow via VS Code / LlamaCPP tool harness)

Attempt

• Run an interactive programming/code-generation prompt using:
  • a VS Code client extension
  • models served via Llama.cpp.

Base model behavior

• Mostly works but shows:
  • one crash when sand hits bottom initially
  • becomes stuck in a thinking loop
  • after prompting to exit the loop, it outputs a “working-ish” simulation

Fine-tune behavior

• Much more problematic:
  • repeatedly tries unnecessary tool calls (browser/screenshot behavior implied)
  • appears stuck in loops again
  • workflow later shifts to heavy manual debugging/tweaks due to harness/model interaction issues

---

6) Coding tests ultimately fail (tool-calling/harness issues)

Additional attempts

• The speaker tries multiple coding environments/harnesses (VS Code, then other editors described as OpenWebUI / “Open code” / “Claude code” / etc. per subtitles).

Core issue

• Tool calls don’t properly execute / file writes fail / garbled tool formatting leaks upstream.
• Suspected cause: Llama.cpp / Ginger (tool format) compatibility or a client bug.

Mitigations attempted

• Toggle “native tool calling” on/off
• Try different editors/harnesses
• Copy code manually when possible
• Consult a pinned discussion suggesting:
  • need for Gemma 4 native tool format
  • using Ginger in Llama CPP
  • “garbled output leaking … client bug”

Outcome

• Even after following the recommendations (speaker says they are using Ginger/LlamaCPP and keeping Gemma tool format), required coding tests couldn’t reliably complete.

Conclusion

• Coding tests are abandoned for this video because a stable harness run couldn’t be achieved.

Key conclusions (speaker’s analysis)

• Performance: fine-tune provides no speed advantage over base (same prefill/decode).
• Memory/long-context: fine-tune does better than base (higher pass rate), though the speaker doesn’t treat this as the whole story.
• Agent/tool behavior: fine-tune is significantly worse in the agency benchmark—often calls tools when it shouldn’t and loops.
• Python capability (HumanEval): fine-tune is slightly worse (~10% drop) but still strong for a 12B model.
• Coding comparison blocked: harness/tool-calling reliability issues prevented a deeper coding comparison, and observed tool-calling problems strongly influenced the agency benchmark result.

Main speakers / sources

• Speaker: Luke (“Luke’s Dev Lab”)
• Model sources referenced:
  • Gemma 4 12B base GGUF (from “Unsllo” per subtitles)
  • Cursor team fine-tune (based on Fable 5, Composer 2.5, trained on K2.5)
• Benchmarks referenced:
  • OpenAI HumanEval (164 Python challenges)