NVIDIA’s New AI Just Changed Everything

Overview

• Video: Two Minute Papers review of NVIDIA’s new open AI assistant (subtitle names it “Nemotron 3 Super”).
• Notable because NVIDIA published a full 51‑page research paper and dataset details, making the model and training recipe unusually transparent and freely available for public use.

Model & training facts

• Size: ~120 billion parameters.
• Training data: ~25 trillion tokens.
• Claimed capability: roughly matches the best closed/proprietary frontier models from about 1.5 years ago and performs on par with the best open models in many tests, though it still lags on some tasks.

The release couples a full research paper and dataset description with an openly available model and recipe.

Key technical innovations (the four “secrets”)

1. NVFP4 numerical format

   • A reduced‑precision number format that compresses computations by rounding off less‑important digits.
   • Engineers selectively keep the most sensitive calculations in higher precision to avoid catastrophic accuracy loss.
   • Result: NVFP4 is reported to be ~3.5× faster than their BF16 variant and up to ~7× faster than similarly capable open models, with no meaningful accuracy drop in most tests.

2. Multi‑token prediction

   • The model predicts multiple future tokens in one batch instead of generating one token at a time (demonstrated with 7-token prediction and joint verification).
   • This approach yields a large speedup in generation throughput.

3. Mamba layers (memory compression)

   • A specialized layer design that compresses context into compact “notes,” keeping important information and discarding filler.
   • Enables efficient handling of much larger contexts without the full re‑reading cost of standard transformer attention.

4. Stochastic rounding

   • To avoid accumulation of rounding errors across many sequential steps (an issue with low‑precision arithmetic), they add carefully crafted zero‑mean random noise during rounding.
   • Over many steps the errors average out, preventing systematic bias and preserving long‑run accuracy.

Performance & practical notes

• The combination of NVFP4, multi‑token prediction, mamba layers, and stochastic rounding yields large speedups (reported up to ~7× versus comparable open models) while maintaining competitive accuracy.
• Some complex, math‑heavy reasoning tasks can still be very slow (one example reportedly took ~1 hour).
• For heavy workloads, using faster hardware instances (e.g., Lambda instances) is recommended.
• Business implication: the speaker suggests this release could shift the landscape away from closed proprietary models if NVIDIA invests heavily in open systems.

Limitations called out

• Still somewhat behind on certain benchmarks and areas compared to the latest closed models.
• Some long reasoning tasks remain slow in practice.
• NVFP4 and the other efficiency tricks require careful engineering (selective precision, stochastic rounding) to avoid failure modes.

Type of content in the release

• Fully open 51‑page paper with detailed methods and dataset description — unusually transparent compared to most proprietary systems.
• The model and techniques are presented as freely available to consumers and researchers.

Main speakers / sources

• Video speaker: Dr. Károly Zsolnai‑Fehér (Two Minute Papers).
• Primary source: NVIDIA research (Nemotron / NemoTRON 3 Super paper and model release).
• Jensen Huang (NVIDIA CEO) is mentioned in the context of NVIDIA’s public/open AI efforts.