Xiaomi & DeepSeek Just Solved the Biggest Bottleneck in Your Local AI

Summary of technological concepts & key product/research claims

Observed problem: agent “amnesia” after ~45 minutes

An AI coding agent can often:

• Write code
• Run tests
• Fix bugs

…but after a long session it may begin to:

• Repeat ideas it already tried
• Forget the error it was actively fixing
• Contradict earlier outputs

The user reported trying workarounds like restarting or refactoring prompts, but the issue persisted. It was framed as hitting a performance/compute cliff rather than a “broken” model.

Core bottleneck (beyond “how many tokens fit”)

The video argues the expensive part of long-running agents is not just context size—it’s reading new input tokens, often described as KV cache misses, rather than keeping context around (KV cache hits).

It cites large cost differences:

• KV cache hits (re-reading existing context like prior conversation/system prompt/files) are cheap, ~0.003 per million tokens (as explained via DeepSeek).
• KV cache misses (new content like fresh logs/errors) are much more expensive, ~15 cents per million tokens—about 50× more expensive for the same token count.

A Hacker News anecdote is included: crossing about 60% of the context window caused a sudden cost jump, interpreted as reaching a point where “cheap rereading” no longer applies.

---

What DeepSeek and Xiaomi solved (same architecture, independently)

Shared motivation: “write little, read lots”

Both papers (released ~12 days apart) target efficiency for agent workflows that generate short actions but process long observations:

• DeepSeek: “AI agents are making workloads increasingly input heavy.”
• Xiaomi: “Agents generate short actions and process long observations.”

Common solution concept: “read once / only cache once”

Both are based on an earlier idea (Microsoft Research / “Yoko” concept): You Only Cache Once.

High-level mechanism:

• Split the transformer into two halves.
• The bottom half reads the prompt and produces shared “notes.”
• The top half does not re-read the prompt; it reuses the shared notes while generating output.

Result:

• The prompt traverses only part of the network, roughly cutting prompt reading cost by about half (for prompt encoding/processing).

DeepSeek implementation (v4.1 Flash)

• Uses a causal encoder-decoder style approach.
• Example split: 40 layers total → bottom 20 read, top 20 write.
• Only 4 layers keep growing notes; others borrow shared nodes.

Claim:

• Nearly halves the cost of reading new input on tool-call-heavy agent workloads.

Xiaomi implementation (Highsparse 2)

• 49 layers with an “early exit” idea (prompt exits around layer 25, skipping part of the top half).
• Adds a deployment optimization: if prompt reading uses only bottom layers, the top half may not need to be loaded in memory.

Claim:

• “5× fewer mathematical operations” for processing a 1 million token prompt (the video emphasizes this as the real innovation).

---

Correcting the “4× vs 4.5× memory race” framing

The video argues the hype is misleading because:

• The reported memory savings aren’t comparable:
  • DeepSeek’s “4×” is measured against its own earlier model setup (including different precision/versions).
  • Xiaomi’s “4.5×” is measured against a different prior setup (described as an 80B research model that isn’t directly runnable).
• More importantly, both papers emphasize that memory capacity wasn’t the main issue.
  • The key win is reducing the expensive cost of reading new observations.

---

Practical implications (API vs local use)

API / agent-at-scale impact

This approach reduces the expensive part of the input bill—specifically the KV cache miss cost (read-new-input).

Example workload described:

• DeepSeek v4.1 Flash in beta for 11.59M tokens total
• 99.5% were KV cache hits (cheap rereads)
• Only 0.5% were “new tokens” (expensive reads), but they still account for a noticeable share of cost (described as ~1/5 of the input bill in the video).

As rereading becomes cheaper, new reading becomes the dominant cost component, making reduction of KV-miss/read-new cost especially important.

Local/home model impact

Memory reduction helps with storage, but the video argues the bigger remaining bottleneck is reading speed (context processing throughput).

Example local measurement:

• On a 512GB M3 Ultra, reading speed is reported around ~450 tokens/sec, roughly flat across context sizes.
• Timing implications:
  • 100k tokens: ~4 minutes before the first output token
  • 1M tokens: ~39 minutes

The video also highlights an open issue: whether the split architecture retains recall as well as full-context models:

• DeepSeek: no standard recall test in the report; long-context retrieval “not fully characterized.”
• Xiaomi: recall improved vs the old design but is still under 60% on their benchmark.
• User reports suggest forgetting after ~400k tokens with v4.1 Flash; recommendation is to keep context under ~400k for now.

---

Takeaway message

The video’s main conclusion is that the next-gen improvement isn’t primarily “bigger context window.” Instead, it’s about:

1. How fast the model can read/process input
2. How well it retains/recalls what it read

It claims memory capacity issues are largely “solved enough,” while read speed and recall quality are the remaining differentiators.

---

Guides / tutorials / review elements

• This is presented as a technical breakdown correcting hype and interpreting the two research papers as solving KV cache miss cost more than raw context length alone.
• Practical guidance included:
  • If a model markets context length, a better metric is reading speed / time-to-first-token for long prompts.
  • For reliability today, user reports suggest keeping context below ~400k tokens (for v4.1 Flash).

---

Main speakers / sources

Main speaker

• The video narrator/host (not explicitly named in the subtitles).

Research/lab sources mentioned

• DeepSeek (v4.1 Flash; referenced paper/report)
• Xiaomi (Highsparse 2 paper; “Myo V3” mentioned)
• Microsoft Research and Tsinghua University (referenced “Yoko” / “You Only Cache Once” idea)

Named individuals

• Lu (Lu Fooly) — described as a researcher leading Xiaomi’s model team; previously helped build DeepSeek V2.