Laguna XS 2.1 33B A3B tested vs Qwen 35B A3B - 16GB Local LLM setup

Product reviewed

• Laguna XS (Poolside): 33B A3B mixture-of-experts model (active ~3B), targeted at agentic coding and long-horizon work on local hardware.
• Compared against: Qwen 35B A3B, run in a similar “non-MTP” setup on the same local target (16GB VRAM).

---

Test setup / tasks covered

1. Performance
   • Prefill (prompt ingest) + decode (token generation speed)
2. Memory/context benchmark
   • 256K context
   • Checks whether specific data is found at depths: 0/25/50/75/100%
3. Agency benchmark
   • “Fake company sandbox” with tools
   • Measures scenario success and tool selection correctness
4. OpenAI human eval
   • 164 handwritten Python challenges
   • Emphasizes pass rate, plus how many questions were answered vs left blank
5. Coding challenges (web-based)
   • Sand physics simulator
   • Kanban
   • Dungeon crawler
6. MCP coding tasks (tool integration)
   • Blender: create + export assets, then render images
   • Godot: generate a simple 3D environment and verify gameplay

---

Key findings (features, strengths, weaknesses)

1) Speed / performance

Laguna vs Qwen

• Prefill (ingest speed)
  • Laguna slower
  • Starts around ~52 tokens, then drops into the low 20s
  • Qwen starts around ~180 and flattens as context grows
• Decode (generation speed)
  • Laguna slightly faster
  • Qwen decode: ~38
  • Laguna decode: ~45–46

Implication: Laguna “generates” faster once engaged, but is significantly slower at initial prompt ingestion.

---

2) Long-context / memory benchmark (256K)

Pass rates

• Qwen: 93%
• Laguna: 60% (described as “bit worrying”)

Notable failure pattern (Laguna)

Laguna finds data near the start and end, but dips in the middle:

• 0% depth: found
• 25%: never found
• 50%: found
• 75%: never found
• 100%: found again

Pros: correct retrieval at some depths Cons: inconsistent retrieval across the window (potentially harmful for long-horizon coding)

---

3) Agency benchmark (tool-using sandbox)

Pass rates

• Qwen: 98% (fails only 1 scenario)
• Laguna: 92% (fails 4 scenarios)

Main stated issue for Laguna

• Tool selection failure: calls tools it shouldn’t
• Example (scenario 24): it didn’t follow instructions to use a specific currency conversion tool (though the final answer was “close”).

---

4) OpenAI human eval (164 Python challenges)

Results

• Laguna: very poor
  • Qwen pass rate: 80%
  • Laguna pass rate: 13%
• Answered vs not answered
  • Qwen: failed to answer 29 questions
  • Laguna: failed to answer 137 questions
  • Laguna “only answered ~16% of questions”

Behavioral reason described

• Uncertainty / overthinking (“wait, wait, wait” pattern)
• Token burn despite prompt injection attempts (e.g., at 4K telling it to “get going”)
• Often doesn’t reach a final answer in time

---

5) Coding challenges (web-based)

Sand Physics

• Laguna
  • Could produce a working solution, but:
    • Burned lots of context and needed compaction
    • Initially behaved incorrectly (water treated like sand) until corrected
  • Performance: laggy / not very performant
• Qwen
  • Generally fixed faster
  • Hit a 32K output token limit repeatedly; after being told it was reaching the limit, it continued
  • Some functional issues (eraser/certain tools) but became workable after fixes
• Verdict from the video: both can succeed; Qwen performed better on performance, Laguna was workable but token-hungry and hesitant.

Kanban (front-end app)

• Laguna
  • Generated files + a UI, but issues included:
    • Code leaked into HTML
    • Light/dark mode broken
    • Day/night toggle didn’t work
    • Card/state handling bugs (duplicates on refresh; messy archive/restore behavior)
  • Needed multiple iterations; burned many tokens
• Qwen
  • Similar 32K output limit issues, but:
    • UI looked nicer
    • Dark/light mode worked
    • Columns draggable worked
  • Still struggled with “add card” reliably; became unsure and didn’t complete consistently in-run
• Bottom line: Laguna was messy; Qwen was more polished but still unreliable at completing the full task.

Dungeon Crawler

• Laguna
  • First run: mostly working (fog of war missing)
  • Follow-ups:
    • A later run burned ~90% of tokens, triggered compactions, then “game broke” and required refresh/fixes
  • Eventually: fog of war worked and previously explored area stayed revealed
• Qwen
  • Produced a working dungeon quickly with fog of war working
  • Also mentioned “god mode” (map reveal) that can’t be turned off—handled as not framed as a failure
• Verdict: Laguna’s style was praised, but Qwen was faster and more efficient, with fewer context explosions.

---

6) MCP tasks (integration) — Blender and Godot

Blender (create assets + render + export)

• Laguna
  • Created assets (marble and finish gate); assets were “okay,” then later got them into the correct folder
  • Render/export attempts sometimes saved screenshots of the program instead of rendered images; retried
• Qwen
  • Created assets too (marble more spherical; gate had nicer details/panels)
  • Export folder and exports worked; still had token-limit/verbosity issues but got further
• Verdict: Qwen generally edged out Laguna; both had token-limit/verbosity problems.

Godot (3D environment + gameplay)

• Laguna
  • Integration failed: attempted MCP connection but couldn’t find/attach correctly
  • Killed the running app; then couldn’t connect
  • Restart cycles caused port conflicts and repeated app termination
  • Test was ultimately abandoned/left unfinished
• Qwen
  • Integration succeeded; gameplay tested:
    • Movement keys reversed (W/S swapped)
    • Jumping worked
    • Some collision worked
    • Collecting orb not functioning / not updating
  • Still described as not terrible, especially relative to Laguna’s failure
• Verdict: Laguna very weak on Godot/MCP; Qwen managed a mostly working basic world with smaller gameplay issues.

---

Pros & cons (as evidenced in the video)

Pros (Laguna XS)

• Slightly faster decode speed than Qwen (in the reported test)
• Can complete some coding tasks after iteration (Sand Physics, Dungeon Crawler, Blender asset creation)

Cons (Laguna XS)

• Much slower prefill (prompt ingestion)
• Weak long-context retrieval
  • 60% vs Qwen 93%
  • inconsistent mid-context performance
• Agentic reliability behind Qwen
  • agency: 92% vs 98%, including tool-selection mistakes
• Major failure on OpenAI Python human eval
  • 13% pass, only ~16% of questions answered
• Token/context burn and “uncertainty loops” (“wait… wait…”) leading to non-results or compactions
• MCP struggles badly in Godot (integration/session failure)

---

Comparisons / overall ranking from the video

• Qwen 35B A3B is described as the best overall performer.
• Laguna is usable but unreliable, especially in:
  • long-context memory accuracy
  • Python human-eval answering
  • MCP-based Godot execution

---

Numerical results explicitly mentioned

• Prefill speed
  • Qwen ~180
  • Laguna ~52 → low 20s
• Decode speed
  • Qwen ~38
  • Laguna ~45–46
• Memory benchmark (256K) pass rate
  • Qwen 93%
  • Laguna 60%
• Agency benchmark pass rate
  • Qwen 98%
  • Laguna 92%
• OpenAI human eval (164 Python)
  • Qwen pass rate 80%
  • Laguna pass rate 13%
  • Laguna answered ~16% (failed to answer 137 questions)

---

Overall verdict / recommendation

Prefer Qwen over Laguna XS for practical coding/agent tasks. Laguna XS shows some strengths (decode speed; can sometimes complete coding after retries), but the tests show serious unreliability—especially Python human eval, long-context consistency, and Godot/MCP integration. If trying Laguna XS, expect token burn, uncertainty loops, and manual prompting/fixing.

---

Unique points mentioned about the product (consolidated)

1. Laguna XS is a 33B A3B mixture-of-experts with active ~3B.
2. Presented as upgraded from Laguna XS2, aimed at agentic coding and long-horizon local work.
3. Runs locally on hardware with 16GB VRAM (with 32GB system RAM mentioned for overflow).
4. Prefill slower than Qwen; decode faster.
5. 256K context retrieval inconsistent (60% pass; mid-depth failures).
6. Agency/tools: decent but behind Qwen; tool selection mistakes observed.
7. Python human eval: extremely low answer rate and pass rate (13%).
8. Token/context burn and “wait” uncertainty loops contribute to failures.
9. Web coding performance highlights:
   • Sand Physics: works after iteration; token-hungry/hesitant
   • Kanban: starts but state/alignment bugs and duplicates
   • Dungeon crawler: can work but may “break” and require refresh/extra prompting
10. MCP highlights:
    • Blender: can generate/export assets, but render steps may be off
    • Godot: failed to complete due to MCP/session handling issues (killing/connection problems)

---

Speakers/views

• Single main speaker: Luke (Luke’s Dev Lab). No other speakers contributed distinct perspectives beyond incidental reactions while he runs tests.