DeepSeek Just Shook the Entire AI Industry Again

Summary

The video analyzes DeepSeek’s open-sourced inference stack, DeepInfer, and its companion DeepSeek Recipe. It explains how the company says it serves AI requests cheaply at scale.

The video reports that DeepSeek’s system processes 155 trillion tokens per day and that a GPU can generate about $38 in revenue per hour during discounted periods. Based on those figures and estimated chip costs, the narrator says DeepSeek could be operating at roughly an 80% gross margin.

Technology and Features

• Prompt processing: The narrator describes a causal encoder-decoder design that separates the model’s “reader” and “writer” layers. The first half processes the prompt, while the upper half generates responses, reducing the computation needed to read prompts.
• Mixture of experts: The model is said to have 748 billion total parameters but activates only a smaller subset for a given task: around 8 billion for reading and 16 billion for writing.
• Wide expert parallelism: DeepInfer distributes expert components across large GPU groups—up to 64–128 GPUs for generation—instead of placing the entire workload on a smaller server. The video says this reduces memory traffic and improves throughput.
• Expert load balancing (EPLB): The system monitors which experts are busiest and can duplicate popular experts onto less-loaded GPUs to prevent bottlenecks and idle hardware.
• DSpark speculative decoding: A smaller model drafts several words for the larger model to check. A confidence mechanism can stop verification early when later draft words seem unlikely to be accepted. The narrator says this improves speed without reducing accuracy.
• Bounded replay: Conversation state is kept in fast local memory for short periods. If a user returns after that memory has cleared, the system can resume by rereading a short recent segment rather than the whole conversation, according to the video.
• DeepSeek Recipe: This Rust-based program tokenizes prompt chunks in parallel across CPU cores. The narrator also says it helps defend against prompt injection by encoding user input as plain text.
• Hardware support: DeepInfer is described as supporting Nvidia Blackwell and Huawei Ascend 950 hardware, but not Nvidia Hopper chips such as the H100.

Limitations and Implications

The video emphasizes that the release does not include DeepSeek’s entire infrastructure. The company has kept its large storage cluster, specialized network-routing system, and global internet load balancer private.

The narrator also notes that reproducing the described performance requires substantial enterprise hardware—at least 128 GPUs—and the specified flagship model.

Overall, the video argues that systems engineering and efficient inference can lower the cost of serving AI, putting pressure on cloud providers and competitors to improve efficiency and pricing.

Content Type

This is an explanatory technology analysis, not a hands-on guide or product review.

Main Speaker and Source

A single narrator from the WhatsWithAI channel.