Video summary

how did we make deepseek outperform opus 4.7?

Main summary

Key takeaways

Technology

Core claim / context

  • After DeepSeek launched the V4 family (V4 Pro and V4 Flash), the community split:
    • Some praised their intelligence/cognition and stability.
    • Others reported tool-calling failures (slow or non-functional tool calls).
  • The speaker (from Command Code) investigates why tool calling fails for open models like DeepSeek and concludes it’s primarily a harness/validation/repair problem, not the model’s “raw ability.”

Key technological idea: “Tool repairs” in the harness

The speaker identifies a deterministic failure pattern caused by:

  • Tool input schema validation (they mention Zod as the type/argument validator).
  • Open models producing non-conformant tool call arguments, which triggers validation errors.

Instead of repeatedly sending validation errors back to the model (wasting tokens/time and degrading output over long sessions), the harness:

  1. Detects a specific validation/type/schema failure.
  2. Applies a deterministic fix (“repair migration”) to transform the tool call into a schema-valid form.
  3. Executes the repaired tool call.
  4. Sends the tool result back to the LLM along with a small “repair note” explaining what was wrong and what to do next.

Evidence / observed failure rates

Using Command Code’s bundled review skill (referred to as r/review skill), the speaker reports:

  • DeepSeek V4 Flash had ~56 failures per tool call for cases like:
    • shell
    • read file
    • the r/review scenario

They initially considered removing DeepSeek from the product because of tool failures.

“Taste” model / continuous learning layer (to understand model quirks)

Before fully fixing tool calling, they introduce “taste”:

  • A continuous learning layer that captures preferences/repair-like behavioral invariants for building CLIs/software.
  • Stored as something like taste.md, learned through feedback and/or experiment.

They also describe an experiment to learn an LLM’s “taste”:

  • Run ~1 billion tokens of multi-language programming tasks (e.g., TypeScript/JavaScript/Go/Ruby/Python/HTML-CSS).
  • Produce a taste file representing DeepSeek’s behavior at scale.
  • They correlate this “taste” with recurring tool-call errors after Command Code had enough telemetry (they mention crossing 10B tokens consumed).

The four common tool-call error types they target (examples)

They list the most frequent Open-model tool-call bugs and describe repairs:

  1. Optional fields set to null instead of omission/expected type
    • Example: models send null for optional args, but the validator is strict and rejects it.
  2. JSON type mismatches
    • Example: returns “a JSON array as a string” when an actual array is expected.
  3. Wrapping arguments in unexpected objects
    • Example: sends an empty object instead of a required scalar/value, causing validator failure.
  4. Other recurring schema/shape inconsistencies
    • Later generalized into a growing set of “repair invariants.”

Why “repair notes” matter (not just repairing once)

  • Early repair attempts improved only a small portion:
    • The speaker claims about ~1% of bad tool calls were fixable.
  • The remaining issue:
    • After repairs, open models repeat the same bad tool calls on subsequent attempts.

Their solution:

  • Alongside the corrected tool result, include a repair note that tells the LLM what went wrong (based on the validator error).

Claimed effect:

  • In the next tool call, the model stops repeating the mistake, restoring flow and quality.

How the repair harness is structured

The repair mechanism is modeled like a migration folder:

  • A deterministic “repair harness” contains many small tool repair migrations (per tool type, language, scenario, and model quirks).

They claim:

  • Repairs themselves are small compute (roughly 30–100 lines of code per repair/repair logic).
  • Repair matching is quick; tool results proceed normally after repairs.

Reported scale and impact

  • They claim a large and growing library of repairs:
    • Mention of ~56,000 repair invariants (with versions varying by language/model/scenario).

Product outcome:

  • Tool reliability improved, and DeepSeek Flash became a favorite, with claims like:
    • “better than Claude Haiku right now”
    • “cheaper … 10–20x”
  • Command Code adoption increased:
    • described as the second/third biggest coding agent by usage on leaderboards
    • mentions Vercel AI gateway and hosting GPUs themselves

Performance/operational theory:

  • They suggest DeepSeek tool issues appear more during periods of lower inference capacity (when many users are active), increasing the mistake rate.

Notable “tutorial/guide-like” elements

Rather than UI step-by-step guidance, the speaker offers a conceptual recipe:

  • Monitor tool-call validation errors (Zod).
  • Identify finite, compositional classes of schema/shape mistakes.
  • Build deterministic repairs + “repair notes.”
  • Re-run long sessions to confirm improvements and prevent repeated failure loops.

They also note the work is shareable, and others created variants (e.g., different coding agents/harnesses).

Main speakers / sources (end)

  • Main speaker: Creator/engineer from Command Code (credited via references to “my team,” “Command Code,” and their GitHub/taste work).
  • Referenced external source: DeepSeek (V4 Pro and V4 Flash models).
  • Referenced third-party entities:
    • Zod (type validator library)
    • Paul (mentioned in community commentary)

Original video