Video summary

Yann LeCun: World Models: Enabling the next AI revolution

Main summary

Key takeaways

Science and Nature

Scientific concepts / discoveries / nature phenomena mentioned

Key framing: why “world models” could enable more general intelligence

  • Limits of current machine learning vs humans/animals

    • Humans/animals learn new tasks quickly with few samples and can do zero-shot generalization.
    • Real-world environments are described as messy: continuous, high-dimensional, and noisy data.
    • Text/language is described as easier by comparison.

  • Moravec’s paradox

    • “Simple for humans, hard for computers” (e.g., tasks requiring real-world grounding).
    • “Complicated for humans, easy for computers” (e.g., chess, symbolic integral computation, theorem proving).

Grounding as a requirement for intelligence

  • Intelligence is argued to require grounding in the real world (interaction with physical reality).
  • A philosophical/linguistic debate about whether language is innate vs learned is referenced to motivate perspectives on learning capacity.

What “intelligence” is (as defined in the talk)

  • Not viewed as:
    • Accumulation of declarative knowledge (LMs are framed as primarily knowledge accumulation).
    • Accumulation of skills.
  • Instead, intelligence is defined as:
    • Ability to accomplish new tasks / solve new problems with minimal or no prior task training.
  • Therefore:
    • No single simple test metric fully captures intelligence, since any specific task can be “cracked” with enough effort.

How infants learn physical concepts (developmental science)

  • Observation-based learning early in life

    • Babies learn a lot before they can act on the world.

  • Learned physical/world models

    • 3D world perception inferred from changes in view when moved (even passive movement).
    • Object permanence and other common-sense concepts learned quickly.
    • Intuitive physics takes longer (examples: inertia, gravity).
      • Gravity-related expectation violations: age-dependent attention/surprise.
      • Violation-of-expectation experiments used to infer whether infants learned concepts.
    • Inference-by-surprise idea:
      • If an outcome violates learned physical expectations, infants show increased attention/“surprise.”

Data/scale argument against text-only scaling to human-level AI

  • A calculation compares:
    • Training data for an LLM (tens of trillions of tokens/words) vs.
    • A child’s lifetime input, especially vision (vision channels estimated from optic nerve fibers and time awake).
  • Conclusion: training on text alone is argued to be insufficient for human-like intelligence.

World models: proposed computational architecture and inference principle

Energy-based reasoning / planning vs autoregressive token prediction

  • Two inference styles are contrasted:
    1. Forward propagation through a fixed neural stack (LLM-like autoregressive next-token prediction).
    2. Search / optimization at inference time to find an output/action that best satisfies an objective.
  • Planning is framed as minimizing an energy / cost function:
    • Perception → internal representation of current world state
    • Propose an action sequence
    • World model predicts outcome
    • Objective measures task success (interpreted as cost/energy)
    • Optimization searches action sequences that minimize energy
  • This is likened to Model Predictive Control (MPC) / classical optimal control.

Safety “guard rails” via constrained optimization

  • Guard rails are modeled as additional objectives/costs:
    • Penalize action sequences that cause harmful or dangerous outcomes.
  • Claim:
    • Unlike LLM safety via fine-tuning/jailbreak vulnerabilities, an optimization-based world model with explicit constraints is harder to bypass.

Hierarchical planning (unsolved)

  • Humans/animals plan at multiple abstraction levels.
    • Example: planning “New York → Paris tomorrow”:
      • High level: get to airport within rough time
      • Mid level: get to airport via taxi
      • Low level: walk to elevator, press button, exit building
  • Core challenge:
    • How to learn hierarchical planning is described as open/unresolved.

Training world models: representation learning and why video “prediction in pixel space” fails

Why “generative” video prediction is not the proposed path

  • Pure next-frame pixel prediction is described as ineffective due to:
    • Huge combinatorial uncertainty: many plausible futures.
    • This causes averaging/blurry predictions.
  • Video generation can still produce “cute videos,” but the talk distinguishes:
    • video generation systems vs world models for control/planning.

Proposed learning approach: Joint Embedding Predictive Architecture (JEPA / “JPA”)

  • Core idea:
    • Instead of reconstructing/predicting in raw pixel space,
    • encode current and future observations into a representation space,
    • then train prediction in that representation space.
  • Claimed benefit:
    • Representation can discard information that is not predictable, enabling better predictive abstraction for control.

Collapse problem in self-supervised joint-embedding

  • Risk:
    • Networks can minimize prediction error by collapsing to constant/ignoring representations.
  • Proposed solution: information maximization (a favorite concept in the talk)
    • Encourage representations to carry maximal information and become disentangled/non-redundant.

Mutual information / information maximization objectives

  • Practical difficulties are noted:
    • True information measures require knowing underlying distributions (usually unknown).
    • Differentiable, estimable bounds are challenging:
      • empirical measures and bounds can behave differently than exact information.

Energy-based models and contrastive/regularized training to avoid collapse

Energy-based model perspective

  • Learn a contrast/energy function defining which (X, Y) pairs are compatible.
  • Inference then becomes:
    • search among possible outputs to minimize energy.

Two broad anti-collapse strategies

  • Contrasting methods

    • Push energy low for data pairs, high for non-data (“negative”) pairs.

  • Regularization methods

    • Shape representation distributions so that only small “low-energy” regions exist (reduce volume of low-energy space).

Information maximization via dimension- and sample-contrast

  • Two ways to make representations informative:
    • Dimension contrastive: ensure representation dimensions are distinct (less redundancy across variables).
    • Sample contrastive: ensure different samples yield distinct embeddings.
  • Related to known contrastive methods in vision (e.g., image-text alignment).

Specific methods / named techniques described

S.I.G.G. (sketch isotropic Gaussian regularization)

  • Goal:
    • Make encoder outputs follow an isotropic Gaussian in representation space.
    • Isotropic Gaussian implies independence across dimensions (maximally informative individually, per the talk).
  • Practical training trick:
    • Perform many 1D projections along different directions.
    • Match the projected empirical cumulative distribution to a Gaussian using gradients from a “staircase vs ideal Gaussian” comparison.
    • Repeat across many directions to recover near-isotropic behavior in higher dimensions (claimed theorem/intuition).

Distillation-style joint embedding with EMA targets (bootstrapped latent)

  • Use two encoders with shared architecture:
    • Online encoder updated by gradients.
    • Target encoder updated more slowly via exponential moving average (EMA) of online weights.
  • Cited family/idea:
    • “Bootstrap your own latent” (BYOL-like) and other EMA-based self-supervised methods.
  • Examples referenced:
    • JEPA and comparisons vs MAE-style autoencoding (max autoencoder), where JEPA is claimed to be better and faster.

DINO

  • Described as:
    • A strong self-supervised joint-embedding method developed at Meta.
    • Producing high-quality generic image representations.
  • Used as an encoder option in the demonstrated world-model/planning pipeline.

Empirical/behavioral demonstrations claimed

Planning in simulated physics/control tasks

  • A trained world model + planner:
    • produces action sequences that reach goals in < 25 steps in simulated environments.
  • Named examples:
    • double pendulum
    • push-T (push manipulation type task)
    • other simulated scenarios

Learning common sense from self-supervised video prediction (violation → error spikes)

  • A JEPA-like model trained to predict future representations in video:
    • measures internal prediction error over time steps.
  • Claim:
    • When shown an impossible physical event (unphysical disappearance/violation),
    • prediction error increases sharply—interpreted as emergent common sense about physical plausibility.

Depth from single image using learned representations

  • A model variant trained for video representation learning:
    • uses representation features with a head to predict depth from a single image.
  • Claim:
    • Performs well (even better than DINOv3 in the talk),
    • suggesting the representations encode 3D structure and object-like notions.

Methodology / “how to” lists (as described)

Architecture for planning with a world model (high-level loop)

  1. Observe environment via a perception module → obtain representation/state.
  2. Optionally incorporate memory for partial observability.
  3. Propose a candidate sequence of actions.
  4. Use a world model to predict the outcome of that sequence.
  5. Compute an objective/energy function measuring task completion:
    • low/zero when accomplished, higher when not.
  6. Add guard-rail objectives to penalize harmful/dangerous trajectories.
  7. Perform optimization/search over action sequences at inference time (energy minimization).

JEPA/JPA training in representation space

  • Encode observed frames (current context) into latent representations.
  • Predict future/masked parts in latent/representation space (not pixel reconstruction).
  • Train with self-supervised predictive objectives while preventing collapse via:
    • information maximization (e.g., SI(G)G) and/or
    • joint-embedding + anti-collapse mechanisms (contrastive or regularized).

Researchers / sources featured (explicitly named)

  • Yann LeCun (speaker)
  • Vladimir (referred to by first name only: “Vladlin/Vladlen” in subtitles; no full name given)
  • Marvin Minsky / Moravec (via Moravec’s paradox)
  • Noam Chomsky
  • Seymour Papert
  • Jean Piaget (subtitles reference “Jean P / Piaget”)
  • Emanuel Du (co-author referenced for JEPA/common-sense baby/world model related paper; “mostly written by Emanuel Du”)
  • Berkeley (institution referenced)
  • Meta (institution referenced)
  • DeepMind (institution referenced)
  • DINOv3 / DINOv3 (method name; DINO credited to “former colleagues at Meta in Paris” without names)
  • NYU (institution referenced for speaker affiliation and example context)
  • “Meta in Paris” colleagues (unnamed individuals referenced collectively)

Original video