Video summary

Laguna XS 2.1 33B A3B tested vs Qwen 35B A3B - 16GB Local LLM setup

Main summary

Key takeaways

Product Review

Product reviewed

  • Laguna XS (Poolside): 33B A3B mixture-of-experts model (active ~3B), targeted at agentic coding and long-horizon work on local hardware.
  • Compared against: Qwen 35B A3B, run in a similar “non-MTP” setup on the same local target (16GB VRAM).

Test setup / tasks covered

  1. Performance
    • Prefill (prompt ingest) + decode (token generation speed)
  2. Memory/context benchmark
    • 256K context
    • Checks whether specific data is found at depths: 0/25/50/75/100%
  3. Agency benchmark
    • “Fake company sandbox” with tools
    • Measures scenario success and tool selection correctness
  4. OpenAI human eval
    • 164 handwritten Python challenges
    • Emphasizes pass rate, plus how many questions were answered vs left blank
  5. Coding challenges (web-based)
    • Sand physics simulator
    • Kanban
    • Dungeon crawler
  6. MCP coding tasks (tool integration)
    • Blender: create + export assets, then render images
    • Godot: generate a simple 3D environment and verify gameplay

Key findings (features, strengths, weaknesses)

1) Speed / performance

Laguna vs Qwen

  • Prefill (ingest speed)
    • Laguna slower
    • Starts around ~52 tokens, then drops into the low 20s
    • Qwen starts around ~180 and flattens as context grows
  • Decode (generation speed)
    • Laguna slightly faster
    • Qwen decode: ~38
    • Laguna decode: ~45–46

Implication: Laguna “generates” faster once engaged, but is significantly slower at initial prompt ingestion.

2) Long-context / memory benchmark (256K)

Pass rates

  • Qwen: 93%
  • Laguna: 60% (described as “bit worrying”)

Notable failure pattern (Laguna)

Laguna finds data near the start and end, but dips in the middle:

  • 0% depth: found
  • 25%: never found
  • 50%: found
  • 75%: never found
  • 100%: found again

Pros: correct retrieval at some depths Cons: inconsistent retrieval across the window (potentially harmful for long-horizon coding)

3) Agency benchmark (tool-using sandbox)

Pass rates

  • Qwen: 98% (fails only 1 scenario)
  • Laguna: 92% (fails 4 scenarios)

Main stated issue for Laguna

  • Tool selection failure: calls tools it shouldn’t
  • Example (scenario 24): it didn’t follow instructions to use a specific currency conversion tool (though the final answer was “close”).

4) OpenAI human eval (164 Python challenges)

Results

  • Laguna: very poor
    • Qwen pass rate: 80%
    • Laguna pass rate: 13%
  • Answered vs not answered
    • Qwen: failed to answer 29 questions
    • Laguna: failed to answer 137 questions
    • Laguna “only answered ~16% of questions”

Behavioral reason described

  • Uncertainty / overthinking (“wait, wait, wait” pattern)
  • Token burn despite prompt injection attempts (e.g., at 4K telling it to “get going”)
  • Often doesn’t reach a final answer in time

5) Coding challenges (web-based)

Sand Physics

  • Laguna
    • Could produce a working solution, but:
      • Burned lots of context and needed compaction
      • Initially behaved incorrectly (water treated like sand) until corrected
    • Performance: laggy / not very performant
  • Qwen
    • Generally fixed faster
    • Hit a 32K output token limit repeatedly; after being told it was reaching the limit, it continued
    • Some functional issues (eraser/certain tools) but became workable after fixes
  • Verdict from the video: both can succeed; Qwen performed better on performance, Laguna was workable but token-hungry and hesitant.

Kanban (front-end app)

  • Laguna
    • Generated files + a UI, but issues included:
      • Code leaked into HTML
      • Light/dark mode broken
      • Day/night toggle didn’t work
      • Card/state handling bugs (duplicates on refresh; messy archive/restore behavior)
    • Needed multiple iterations; burned many tokens
  • Qwen
    • Similar 32K output limit issues, but:
      • UI looked nicer
      • Dark/light mode worked
      • Columns draggable worked
    • Still struggled with “add card” reliably; became unsure and didn’t complete consistently in-run
  • Bottom line: Laguna was messy; Qwen was more polished but still unreliable at completing the full task.

Dungeon Crawler

  • Laguna
    • First run: mostly working (fog of war missing)
    • Follow-ups:
      • A later run burned ~90% of tokens, triggered compactions, then “game broke” and required refresh/fixes
    • Eventually: fog of war worked and previously explored area stayed revealed
  • Qwen
    • Produced a working dungeon quickly with fog of war working
    • Also mentioned “god mode” (map reveal) that can’t be turned off—handled as not framed as a failure
  • Verdict: Laguna’s style was praised, but Qwen was faster and more efficient, with fewer context explosions.

6) MCP tasks (integration) — Blender and Godot

Blender (create assets + render + export)

  • Laguna
    • Created assets (marble and finish gate); assets were “okay,” then later got them into the correct folder
    • Render/export attempts sometimes saved screenshots of the program instead of rendered images; retried
  • Qwen
    • Created assets too (marble more spherical; gate had nicer details/panels)
    • Export folder and exports worked; still had token-limit/verbosity issues but got further
  • Verdict: Qwen generally edged out Laguna; both had token-limit/verbosity problems.

Godot (3D environment + gameplay)

  • Laguna
    • Integration failed: attempted MCP connection but couldn’t find/attach correctly
    • Killed the running app; then couldn’t connect
    • Restart cycles caused port conflicts and repeated app termination
    • Test was ultimately abandoned/left unfinished
  • Qwen
    • Integration succeeded; gameplay tested:
      • Movement keys reversed (W/S swapped)
      • Jumping worked
      • Some collision worked
      • Collecting orb not functioning / not updating
    • Still described as not terrible, especially relative to Laguna’s failure
  • Verdict: Laguna very weak on Godot/MCP; Qwen managed a mostly working basic world with smaller gameplay issues.

Pros & cons (as evidenced in the video)

Pros (Laguna XS)

  • Slightly faster decode speed than Qwen (in the reported test)
  • Can complete some coding tasks after iteration (Sand Physics, Dungeon Crawler, Blender asset creation)

Cons (Laguna XS)

  • Much slower prefill (prompt ingestion)
  • Weak long-context retrieval
    • 60% vs Qwen 93%
    • inconsistent mid-context performance
  • Agentic reliability behind Qwen
    • agency: 92% vs 98%, including tool-selection mistakes
  • Major failure on OpenAI Python human eval
    • 13% pass, only ~16% of questions answered
  • Token/context burn and “uncertainty loops” (“wait… wait…”) leading to non-results or compactions
  • MCP struggles badly in Godot (integration/session failure)

Comparisons / overall ranking from the video

  • Qwen 35B A3B is described as the best overall performer.
  • Laguna is usable but unreliable, especially in:
    • long-context memory accuracy
    • Python human-eval answering
    • MCP-based Godot execution

Numerical results explicitly mentioned

  • Prefill speed
    • Qwen ~180
    • Laguna ~52 → low 20s
  • Decode speed
    • Qwen ~38
    • Laguna ~45–46
  • Memory benchmark (256K) pass rate
    • Qwen 93%
    • Laguna 60%
  • Agency benchmark pass rate
    • Qwen 98%
    • Laguna 92%
  • OpenAI human eval (164 Python)
    • Qwen pass rate 80%
    • Laguna pass rate 13%
    • Laguna answered ~16% (failed to answer 137 questions)

Overall verdict / recommendation

Prefer Qwen over Laguna XS for practical coding/agent tasks. Laguna XS shows some strengths (decode speed; can sometimes complete coding after retries), but the tests show serious unreliability—especially Python human eval, long-context consistency, and Godot/MCP integration. If trying Laguna XS, expect token burn, uncertainty loops, and manual prompting/fixing.

Unique points mentioned about the product (consolidated)

  1. Laguna XS is a 33B A3B mixture-of-experts with active ~3B.
  2. Presented as upgraded from Laguna XS2, aimed at agentic coding and long-horizon local work.
  3. Runs locally on hardware with 16GB VRAM (with 32GB system RAM mentioned for overflow).
  4. Prefill slower than Qwen; decode faster.
  5. 256K context retrieval inconsistent (60% pass; mid-depth failures).
  6. Agency/tools: decent but behind Qwen; tool selection mistakes observed.
  7. Python human eval: extremely low answer rate and pass rate (13%).
  8. Token/context burn and “wait” uncertainty loops contribute to failures.
  9. Web coding performance highlights:
    • Sand Physics: works after iteration; token-hungry/hesitant
    • Kanban: starts but state/alignment bugs and duplicates
    • Dungeon crawler: can work but may “break” and require refresh/extra prompting
  10. MCP highlights:
    • Blender: can generate/export assets, but render steps may be off
    • Godot: failed to complete due to MCP/session handling issues (killing/connection problems)

Speakers/views

  • Single main speaker: Luke (Luke’s Dev Lab). No other speakers contributed distinct perspectives beyond incidental reactions while he runs tests.

Original video