When Will AI Make Me Scrambled Eggs? I Went To NVIDIA To Find Out.

Overview

This video is an on-site interview at NVIDIA about “world models”—models that predict and generate actions/trajectories in the physical world, analogous to how LLMs generate text. The discussion contrasts world models with LLMs and explores NVIDIA’s Cosmos initiative and the Cosmos 3 model suite for physical AI.

---

Core idea: verified tasks over perfectly creative tasks

The interview argues it’s more likely to build robots that reliably perform structured, testable tasks than tasks requiring high subjective quality, such as:

• More feasible (verifiable tasks): laundry folding
• Less feasible (subjective quality): “perfect scrambled eggs”

Reason: physical AI needs verifiable rewards and reliable evaluation. Subjective notions of “quality” are harder to measure and automate.

---

What “world models” are

A world model generates actions (e.g., commanding a robot manipulator to pick up a bottle), rather than only producing text like “pick up the bottle.”

It’s described as a set of models for different purposes—unified by the idea that they represent the same world for physical AI developers. Examples of purposes include:

• prediction
• explanation
• navigation
• interaction

---

NVIDIA Cosmos: purpose and product direction

The interviewee, Ming Yu Liu (VP at NVIDIA / Cosmos Labs lead), describes Cosmos as a platform for building physical AI across areas such as:

• driverless cars
• robots
• factory machine control

Cosmos aims to help developers with:

• world understanding
• simulation
• interaction

so that policy models can be evaluated before deployment.

---

Cosmos suite and the “one model” approach

The discussion presents Cosmos as combining three capabilities:

1. Understanding the world (perception / explanation)
2. Simulating the world (scenario testing)
3. Interacting with the world (planning/action within the model)

NVIDIA released a Cosmos suite (mentioned as in June, during Jensen’s keynote in Taipei) that “combines” these into a more unified system. The unification relies on shared representations of the same world across tasks.

---

Real-time deployment architecture (engineering constraint)

For safe robot operation, the model must be real-time and responsive. Cosmos CV uses multiple model sizes:

• Super (largest): maximum performance, suited for data centers/powerful computers
• Nano (mid-size; referenced indirectly)
• Edge (smallest): designed to run in real time on embedded/peripheral hardware
  • examples mentioned include “Media Server,” “Thor,” “DJI Spark”

A described strategy is to mix models:

• run small models on-device for immediate navigation
• use larger models for higher-level logic where latency is more tolerable

---

Scaling laws for world models

The interview extends scaling concepts beyond LLM token scaling into world modeling, including:

• Data scaling: more data improves coverage/quality
• Model size scaling: larger models yield better performance
• “Text scaling” for visual/world generation: longer/more detailed reasoning and more precise spatial/temporal instructions can improve generated video/action quality

It also adds physical-world-specific scaling dimensions:

• test/task scaling
• agent scaling

Training includes an “analyzer” that expands simulation contents iteratively, suggesting an agentic loop that refines simulation outcomes.

---

Learning physics: partial understanding + physics engines

The interview notes uncertainty: it’s difficult to know whether a model “truly knows physics” purely from observational data. As models scale and data coverage improves, physical process modeling improves.

To accelerate physics competence, NVIDIA suggests integrating physics simulators, explicitly naming:

• NVIDIA Omniverse
• Isaac

The framing is that models can approximate trajectories from video/images, and can improve further by generating synthetic interaction data using a physical engine (e.g., Newtonian motion and collisions).

---

Cosmos 3 architecture (transformer-based: listener + generator)

Cosmos 3 is described as a hybrid transformer architecture:

• A “listener/tower” that interprets what it sees and thinks in language
• A generator that produces simulation video based on that reasoning

A major emphasis is on actions: since robotics observations are consequences of prior actions, robotics needs control signals (e.g., servos/robot motion) rather than token text.

---

Key applications highlighted

1. Policy review / evaluation

   • Use Cosmos to evaluate driving/robot policy performance and safety in simulation
   • Narrow deployment to promising policy checkpoints (reducing fleet rollout)

2. Policy-as-model / action model

   • Cosmos can follow instructions and apply actions because it’s trained across text, video, and action modalities
   • Example task family: “capture and move” tasks

3. Strategy models

   • Strategy is framed as producing trajectories/checkpoints that update when new obstacles appear
     • example: cars appearing mid-turn in self-driving
   • Treated as action prediction in space-time (directing vectors)

---

Bindings and tooling for physical AI

NVIDIA discusses “harness/strapping/bindings” (or similar orchestration concepts):

• For LLMs: bindings orchestrate model calls + tools
• For physical AI: the orchestration is similar, but the tools differ, such as:
  • camera calibration
  • visual inspection
  • real robot interfaces

A visual inspector concept is mentioned to assess whether the robot successfully completed the task.

---

Agent-based AI and decomposition

A major theme is that agent systems handle complex tasks by breaking them into smaller subproblems via multiple model calls.

The interview contrasts:

• LLM approach: next-token prediction
• diffusion approach: noise removal

It argues agent-based AI is especially attractive for physical AI because it naturally decomposes work and supports graph-like execution.

---

Roadmap theme: verified rewards speed physical progress

A recurring bottleneck is not only data volume but achieving verifiable rewards—objective measurable outcomes. Tasks that can be automatically checked in simulation are expected to advance faster than tasks requiring human subjective judgment (e.g., “delicious scrambled eggs”).

---

NVIDIA broader “model family” and openness stance

The interview describes NVIDIA using multiple specialized model families, including:

• NeMo Tron (language/programming)
• Cosmos (world modeling)
• Cruise (human understanding)
• Alpha Mario (autopiloting)
• bio-animal (healthcare)

NVIDIA emphasizes openness, aiming to enable developers via open models/open source rather than limiting access to black-box-only APIs.

---

Main speakers / sources

• Ming Yu Liu — Vice President, Cosmos Labs lead (NVIDIA); main interview subject
• Interviewer — narrator/host asking questions at NVIDIA (name not provided in subtitles)