Mojo + Vulkan is INSANE: Run Local AI on ANY GPU (Goodbye CUDA)

Key technological shift: “CUDA moat” is weakening (local AI on non‑NVIDIA GPUs)

• For ~15 years, GPU AI has been effectively tied to NVIDIA CUDA (software stack + tooling + tight coupling to NVIDIA hardware).
• The video argues this “CUDA moat” is now breaking: local AI can run on many GPUs—AMD, Intel, Apple (including integrated graphics), and even cheap laptop GPUs—without CUDA.

---

Two main enabling approaches

1) Vulkan (vendor-neutral GPU compute for running today’s models)

• Vulkan is described as a low-level open GPU API, originally popular for graphics/games, but usable for AI because the GPU operations overlap (e.g., matrix math).
• Core claim: you can build/run once and target multiple GPU vendors.

Practical workflow (as described)

• Use LM Studio or llama.cpp
• Select a Vulkan runtime inside the app
• Load a single model file (GGUF), typically a quantized model designed to fit GPU memory
• Avoid heavy vendor tooling:
  • No CUDA install
  • Less driver/toolchain pain
  • “one file, any GPU, token streaming”

Performance / benchmark claims (“receipts”)

Community benchmarks are referenced using a 7B model, focusing on token throughput.

• AMD Radeon RX 7900 XTX
  • ~191 tokens/sec on Vulkan
  • Reportedly, Vulkan beats AMD’s own official stack (Rockm) on the same AMD card
• AMD newest laptop chip
  • Vulkan reportedly ~21% faster than Rockm, with steadier results
• Intel ARC (example B580)
  • ~70 tokens/sec
• Apple M3 Ultra (via Molten VK translation)
  • ~115 tokens/sec
• Integrated GPU on Ryzen
  • Claimed to run a 7B model at readable speeds

Additional bonus claim:

• Vulkan sometimes avoids the behavior where CUDA keeps a CPU core pegged, which can mean systems are quieter/cooler and leave CPU headroom for other work.

Tradeoff admitted

• At the very top end on NVIDIA hardware, CUDA may still be faster
  • Example given: RTX 5090 on CUDA ~290 tokens/sec vs ~263 tokens/sec on Vulkan (same card)

---

2) Mojo (build GPU kernels without CUDA; “write once, compile per GPU”)

• Mojo is presented as a language/toolchain that generates optimized GPU code for multiple vendors from a single source.

Claimed developer experience

• Mojo code is described as Python-like, but compiles to C++-like performance for GPU execution.
• Key idea:
  • one Mojo kernel compiles down to the correct low-level GPU code for the target vendor
  • no CUDA-specific rewrite per platform

Product / deployment stack mentioned

• Mojo ships within Max, a serving stack.
• Portability claim: one container/image can deploy on both AMD and NVIDIA data center GPUs with zero changes (no CUDA-porting and no separate Rockm-vs-CUDA logic).

Server performance claims

Examples mentioned:

• On AMD MI300X
  • Max purportedly delivers up to 53% more throughput than vLLM (described as a “gold-standard” for real-model serving)
• On another AMD chip
  • Max purportedly matches NVIDIA H200 token-for-token on the same workload

Framing:

• This is positioned as a data-center competitiveness issue—more than one viable supplier reduces single-vendor pricing power.

GPU reach expanded to consumer hardware + beyond LLMs

• Mojo targets consumer GPUs too:
  • AMD RDNA
  • NVIDIA RTX from 20-series through 50-series
• Not limited to text:
  • Mentions an image/video pipeline using Black Forest Labs Flux, claiming the Mojo engine scales from desktop to data center.
• The video also claims open-sourcing of large portions of GPU kernel code (hundreds of thousands of lines).

---

“Receipt” moment against NVIDIA’s own tuned kernels

• NVIDIA Cutlass is described as a premium, hand-tuned CUDA library for new chips (example: Blackwell).
• The video claims Modular rewrote a Cutlass-like elite kernel in Mojo and matched CUDA performance on NVIDIA:
  • Example number: 130.7 teraflops (paired with CUDA’s figure)
  • Reported code size: Mojo ~770 lines vs CUDA ~3,000 lines (suggesting easier readability/maintainability)
• Additional claims:
  • The port history was published
  • An AI agent automated much of the code-editing

---

The Qualcomm acquisition as “proof this matters”

• The video emphasizes timing and investment:
  • Qualcomm paid nearly $4B to acquire Modular (the Mojo/Max company), reportedly buying the whole company.
• Framing:
  • Qualcomm wants AI that runs across hardware without being locked to CUDA—presented as “smart money” betting on portability.

---

Tutorial-style “what to do now”

Recommended immediate steps:

• Download LM Studio
• Switch runtime to Vulkan
• Load a small model GGUF
• Expect local inference with no account and zero dollars

---

Main speakers / sources

• Speaker: the video’s narrator/host (no specific individual named in the subtitles)
• Implied sources/teams mentioned:
  • Modular (Mojo + Max serving stack)
  • Qualcomm (acquirer of Modular)
  • Community benchmarks (authors not named)
  • NVIDIA (CUDA, Cutlass, vLLM mentioned)
  • AMD (Rockm referenced)
  • Intel (Intel Arc GPUs referenced)
  • Apple (Molten VK referenced)
  • Llama.cpp and LM Studio (workflow tools)
  • vLLM (serving baseline mentioned)