Video summary

TPUs Are BETTER, But Why No One Uses Them?

Main summary

Key takeaways

Technology

Technological concepts & claims

TPUs vs GPUs for AI training/inference

  • The video argues that TPUs are architecturally more efficient than Nvidia GPUs for transformer-style AI workloads.
  • It contrasts hardware designs:
    • GPUs use the Von Neumann architecture, splitting work into compute units and memory, which causes heavy data movement—the “Von Neumann bottleneck”—as data repeatedly shuttles between compute and memory.
    • TPUs use a systolic array design that lets data flow through thousands of math units in pulses, enabling massive on-chip data reuse before off-chip memory is needed.
  • Claimed benefits include:
    • Up to 30× faster performance and
    • Up to 80× more efficient operation (numbers are described as comparisons against contemporary CPUs/GPUs).

ASIC rigidity

  • TPUs are described as ASICs (application-specific integrated circuits), optimized especially for matrix multiplication, which aligns well with Transformer math.
  • The historical concern: if AI model architectures change, ASICs could become obsolete because they’re hard to repurpose—unlike GPUs, which are treated as general-purpose.

Product/features and “why adoption stalled”

Core contradiction: better chip, worse product

Even if TPUs were superior, the video claims they weren’t practical for most companies to adopt due to two “locks”:

  1. Access lock (distribution / deployment)

    • For much of TPU history, the video describes TPUs as effectively available only via Google Cloud rental, not as hardware customers could buy and install.
    • As a result, building on TPUs required moving operations into Google’s environment, including following Google’s pricing and rules.

  2. Software lock

    • Nvidia’s CUDA is highlighted as a major ecosystem advantage.
    • Since 2007, CUDA is described as giving developers a long-standing and widely used programming layer.
    • The video frames this as: startups weren’t only buying hardware; they were buying years of ecosystem tooling, libraries, and engineer familiarity.

“Training without GPUs” example

  • The video claims Google trained Gemini “from start to finish without touching a single GPU”, as evidence that TPUs can fully replace GPUs for training.

What changed recently (adoption acceleration)

The TPU’s original “risk” didn’t materialize

  • The video argues the feared ASIC obsolescence didn’t occur because Transformers dominated, so TPU-optimized math remained central for about a decade.

Access becoming more real / compute deals

The video claims major companies signed large TPU-related agreements:

  • Anthropic: agreement for up to 1 million TPUs (described as tens of billions).
  • Meta: a cloud deal >$10B, reportedly negotiating for millions of TPUs in its own data centers.
  • Apple: training models using 8,000+ Google TPUs and allegedly avoiding Nvidia.

It also claims Google is increasingly moving to sell TPUs as a product, targeting a portion of Nvidia’s data center revenue (e.g., “up to 10%”).

Demand shock inside Google

  • The video claims Google DeepMind and internal researchers are queuing behind paying customers, suggesting TPU capacity is heavily sold externally.

Review/guide/tutorial elements

Hefty Search (narrator’s product, not a TPU tutorial)

  • The only tutorial/how-to style segment is not about TPUs; it’s about a tool the narrator built:
    • Hefty Search: a local file-search tool that lets users search by describing visual content or actions (e.g., “dog jumping into a lake”), returning matching photos/videos/timestamps.
  • Claims about the tool:
    • Uses the machine’s GPU when available
    • Works on weak laptops, including integrated graphics
    • Runs locally (no upload)
    • Costs money after a free trial
  • It’s not framed as a general “AI assistant,” but as a large codebase with “handcrafted systems.”

Main speakers/sources (as presented)

  • Jeff Dean (mentioned as a Google engineer) — credited with a “napkin calculation.”
  • Google (including Google DeepMind / DeepMind researchers) — source of TPUs and related training claims.
  • Nvidia — source of the CUDA ecosystem advantage described as creating the software lock.
  • Companies referenced: OpenAI, Anthropic, Meta, Apple
  • Narrator/creator (implied) — introduces and promotes Hefty Search (speaker is not named in the subtitles).

Original video