DeepSeek-v4.1-Flash Kept Hallucinating - Here's What Helped

Tech-focused summary

The video analyzes DeepSeek-V4.1-Flash, arguing it is both fast and capable—including vision support, strong coding/agentic performance, and a granular “reasoning efforts” setting (0–100). However, it also claims the model is problematic for hallucinations, covering behaviors such as “unfaithful recall,” “gaslighting,” and lying.

Key points: pros/cons of the checkpoint

Pros

• Vision support
• Better coding and agentic capabilities
• Same strong speeds as earlier “Flash” variants
• Granular reasoning efforts (0–100)
• Architecture improvements, including:
  • Improved sparse attention
  • Added n-grams
  • Reduced KB-cache position storage size
  • “Improved across the board”

Cons

• Much larger model size (claimed ~750B parameters when including n-grams)
• Harder to run locally than DeepSeek V4 Flash
• Despite improvements, it’s portrayed as the worst (or near-worst) hallucination performer, “dethroning themselves.”

Evidence / reviews / benchmarks referenced

• An “artificial analysis hallucination rate” chart is cited as showing DeepSeek 4.1 Flash performing worse than DeepSeek 4.
• A Twitter post is referenced for detailed examples of unfaithful recall and hallucinated answers.

Practical demonstrations (what hallucinations look like)

1) Cybersecurity CTF-style Splunk investigation

• The speaker uses a Capture-the-Flag style investigation where:
  • the correct answer is not present in Splunk data directly
  • but is referenced by a pointer
  • and the final correct value exists in a GitHub commit
• Claimed failure mode: the model hallucinates a secret key—it outputs an answer that does not exist in the Splunk data—rather than admitting it lacks the data and pointing to the real source.
• Conclusion: this “immediately rules out” the model for the speaker’s cybersecurity use case, because hallucinated facts are especially dangerous.

2) Coding / code-hallucination examples

Even if code “runs,” the speaker emphasizes that hallucinations can be subtle and often require manual review:

Common patterns mentioned:

• Pointless / empty branches (e.g., an if with no meaningful body)
• Ternaries where branches are effectively identical (no real behavioral change)
• TypeScript unused-variable noise handled by unusual constructs (e.g., “void existing” to silence unused warnings)

Core critique:

• Models can generate syntactically valid or compiling/transpiling code that still inserts ineffective or misleading logic.
• At scale, performance can degrade because “little things” accumulate.

Core technical explanation: why hallucinations happen (sparse attention)

The speaker attributes the issue to sparse attention architecture constraints:

• DeepSeek selects a limited set of candidate key/value positions for attention.
• Example numbers given:
  • With a large context (e.g., 100k context), it identifies only about 16k candidate positions
  • Each attention layer then uses only the top 512 by default
  • Later layers cannot “recover” tokens that were not selected into the candidate pool
• The implication: if the model makes a mistake during early candidate selection, it can’t be corrected later.

Configuration/math terms highlighted:

• index_topK (how many candidates are selected)
• candidate blocks / block size (how the candidate pool size is determined)

The argument: hallucinations occur because the model is effectively unable to revisit or incorporate missed information once candidates are fixed/masked out.

Proposed mitigation / guide: reduce hallucination rate via config tuning

The tutorial-like portion proposes tuning sparse-attention settings to reduce hallucinations:

• Increase the indexing/candidate budget using the model’s sparse-attention configuration.
• Specific suggestion:
  • Increase index_topK from its default (relative to the discussed ~512 top attention usage) to larger values.
• Reported outcome:
  • The speaker claims they reduced hallucinations significantly, landing on 2048 for the number of “top selected candidate positions.”

Result in the Splunk challenge:

• With higher budgets, the model would stop hallucinating the secret key.
• Instead, it would respond with behavior like “unable to recover the plain text key”, while still pointing to the correct link/source.

Comparative takeaway: DeepSeek vs Qwen (architecture differences)

The speaker contrasts DeepSeek’s sparse attention with a referenced Qwen sparse design:

• Qwen is described as using a compression ratio approach, which (as claimed) scales candidate access more favorably with context size.
• Summary comparison:
  • DeepSeek attends to less of the context
  • Qwen attends to more of the context
  • Dense attention models attend to all context

Main speakers / sources

• Speaker: Eric
  • Described as having worked in Big Tech; characterized as former Google and former Mandiant
  • Focuses on agentic AI and local models
• External sources referenced:
  • an “artificial analysis hallucination rate” benchmark/chart
  • a Twitter post documenting unfaithful recall/hallucinations
• Systems used in demonstrations:
  • Splunk (data investigation)
  • GitHub commits (ground-truth referenced answers)
  • DeepSeek V4.1 Flash configuration (including index_topK)