Do we need diffusion in robotics?

Main ideas, concepts, and lessons

1) Why diffusion entered robotics—and what the talk questions

• Diffusion / generative control policies became widely used in robotics because they can represent complex action distributions conditioned on observations.
• The speaker argues that the dominant explanation—“multimodality / distribution matching”—is likely not the main reason diffusion-based policies outperform simpler regression policies.
• The talk focuses on empirical phenomena that lack strong theoretical backing, and frames hypotheses as conjectures (not fully proven).
• Framing concept: a “robotics Moravec’s paradox” / “algorithmic Moravec’s paradox”—robotics became scalable after key algorithmic breakthroughs enabled effective learning from collected data, not merely from “more data.”

2) Background: imitation learning setup and objectives

The problem is cast as a MDP / decision-process view:

• States/observations and actions
• A transition kernel describing the next state given current state and action
• A reward structure, often sparse (e.g., success/failure)
• A policy mapping (state/observation) → distribution over actions

Learning from demonstration (LfD):

• An expert generates multiple trajectories of states/actions (often human collected).
• A learned imitator policy is trained to act successfully in the environment.
• Goal: achieve near-expert performance in expectation, with error shrinking as more data is collected.

3) Baseline: behavior cloning (regression-style imitation)

• Behavior cloning trains a policy via supervised learning: minimize loss between predicted and expert actions.
• Often implemented as regression (e.g., L2 / squared error), though the effective loss depends on model parameterization.

4) What conditional diffusion policies are used for (motivation)

• Demonstration data can be multi-modal: there may be multiple valid solutions to the same situation.
  • Example: when encountering an obstacle, people might go left or right.
• Belief: diffusion models were needed because they can learn complex distributions over actions.
• Historical note:
  • Early diffusion work used diffusion for long planning trajectories.
  • Later work used diffusion for shorter action sequences, making learning easier (less multimodality / lower dimensionality).

5) “Mode learning hypothesis” vs. the speaker’s counter-argument

Mode learning hypothesis (dominant view):

• Generative control policies help by learning and reproducing action modes (i.e., choosing among multiple plausible action branches), avoiding mean-action failure.

Speaker’s counter-argument: diffusion is helpful for other reasons The speaker proposes that observed benefits arise from components intertwined in diffusion training:

1. Distributional learning (fitting a distribution)
2. Stochasticity in training
3. Iterative multi-step computation during sampling

They claim experiments suggest that distributional learning / multimodality is not the main mechanism.

---

Proposed mechanism and evidence (with detailed structure)

A) Key empirical results: diffusion helps only in certain “precise” regions

In an experiment comparing:

• Flow-based policy trained with flow matching
• Regression policy

Results:

• In many settings, regression and flow are similar.
• Diffusion/flow helps significantly only for a small subset of task conditions—especially those requiring high precision (e.g., tool insertion, where placement must be exact).

B) Evidence against “multimodality is the cause”

The speaker reports interventions intended to test whether diffusion truly learns multimodal structures.

1. Visualize action distributions

   • Classic multimodality-eliciting tasks (e.g., Push-T) were considered.
   • They found no clear multimodal structure in action trajectories (while noting visualization limitations in high dimensions).

2. Noise-seed manipulation to “destroy” multimodality

   • In a flow model, multiple noise seeds are sampled and mapped through the iterative process.
   • The resulting actions are then averaged.
   • If diffusion truly learned multimodal modes, averaging should reduce performance.
   • Observed: at most ~2% degradation, suggesting multimodality learning is not the primary factor.

3. Re-labeling with deterministic generation

   • Use the flow policy with deterministic sampling (noise = zero) to generate actions.
   • Build a dataset of these deterministically relabeled trajectories.
   • Refit flow and regression on this data.
   • The gap narrows, but diffusion still outperforms regression.

Conclusion from these tests: distribution matching / multimodality alone does not explain the gains.

C) “Minimal iterative policy” (MIP) intervention: iterative computation + stochasticity, but limited distribution expressivity

What it is:

• Train:
  1. A regression policy to predict a base action.
  2. A second policy that performs a single denoising / one-step correction:
     • Input: a noised action
     • Output: a corrected action (one denoising step)
• At inference:
  • Generate an initial action guess.
  • Condition and apply one denoising step to refine it.

Purpose:

• Test whether you need the full diffusion process and multimodal distribution learning.

Result:

• MIP performs as well as flow.
• Baselines removing only one component (either stochasticity or iterative computation alone) are said to perform worse.

D) What the speaker argues actually matters: on-manifold vs off-manifold error

Core geometry idea:

• Define a local action manifold: the set of actions reachable when you perturb (and sometimes “rewind”) a state trajectory using action noise.
• Decompose action error into:
  • In-manifold (on-manifold) error: errors aligned with manifold directions
  • Orthogonal/off-manifold error: errors pushing actions away from feasible/likely directions

Main empirical finding:

• Flow and MIP induce much lower off-manifold error than regression, even when their L2 regression error is similar.

Why this helps in robotics:

• In control tasks, off-manifold errors can be catastrophic (e.g., insertion wrong spatially).
• On-manifold errors may be less harmful (e.g., timing variation or redundant variations that still work).

Conjectured mechanism:

• Multiple denoising steps / iterative computation repeatedly provide opportunities to project back toward the learned manifold.
• Stochasticity provides coverage for which corrections are needed at different steps (framed as “data augmentation along the generation path”).
• Diffusion-like methods are argued to simulate a data-collection-with-noise effect, selectively in the directions that matter.

E) Additional conjecture about the learning bias in flow matching

• The flow matching loss includes both:
  • Action conditioning term
  • Observation conditioning term
• Hypothesis:
  • Neural function approximation may introduce smoothing in both:
    • the action variable
    • the observation/conditioning variable
  • This joint smoothing could create an inductive bias that encourages manifold adherence.
• The speaker acknowledges the formal justification remains open.

---

Methodology / intervention list (as presented)

Experiments & interventions to test the diffusion mechanism

• Regression vs flow comparison
  • Hold architecture and action chunking constant (diffusion transformer used as parameterization for regression in ablations).
  • Compare task success curves and identify where diffusion helps.
• Multimodality visualizations
  • Visualize trajectories directly or via embeddings (e.g., UMAP).
  • Color by performance (e.g., MC returns).
• Mode-destruction via noise averaging
  • Sample 64 noise seeds → run through flow → average final actions.
  • If multimodality is essential, averaging should degrade performance substantially.
• Deterministic re-labeling
  • Generate deterministic trajectories using the flow policy with noise = 0.
  • Refit both flow and regression on that deterministically labeled dataset.
  • Check whether diffusion maintains an advantage.
• Minimal Iterative Policy (MIP)
  • Start with regression for initial action.
  • Train a shared-network one-step denoiser to refine from a noised action guess.
  • Evaluate whether restricted iterative/stochastic setup matches full diffusion/flow.

---

Later discussion: implications for robot reinforcement learning (RL)

1) Why generative policies can help more in RL than in imitation

• The speaker suggests diffusion-based policies train better for RL than standard regression policies.
• Viewpoint: expand the MDP into a bi-level / two-layer process:
  • First: multi-step action generation (denoising process)
  • Then: environment transition and reward
• Idea: port RL algorithms (e.g., PPO) from standard MDPs to this expanded abstraction.

2) Evidence described: structured exploration and robustness

• Franka Kitchen task:
  • Diversity of “subtask completion in different orders” is reported as similar between regression, MIP, and flow.
  • Conclusion: improvements are not simply because diffusion encodes more overall task diversity.
• On-manifold exploration (named idea):
  • Add noise to different policy types:
    • Gaussian policy → erratic trajectory distributions
    • Gaussian mixture → collapse issues
    • Diffusion → noise projects back onto feasible action manifold, producing “fatter but semantically sensible” exploration coverage
• Sim-to-real robustness
  • Diffusion policies are reported to be smoother and more robust under real-world perturbations.

3) More FLOPs per interaction (compute asymmetry)

• Real robots are slow/expensive to interact with; test-time compute is cheaper.
• Diffusion allows more computation (sampling steps) per action selection.
• Concept: leverage critics/value estimates more by doing conditional diffusion through the policy generation process.
• Mentioned concept/name: OGPO (off-policy generative policy optimization):
  • Train using an RL method but optimize through the generative policy generation steps.

---

Conclusion / stance

• Speaker’s answer to “Do we need diffusion in robotics?”:
  • Maybe not, if “generative” means faithfully reproducing the full conditional action distribution.
  • But diffusion-like methods may provide crucial principles byproducts, such as:
    • iterative computation
    • self-correction
    • structured (manifold-adhering) action diversity
• Open problem posed:
  • Formally explain why diffusion/flow yields lower off-manifold error and stronger manifold adherence given specific losses and architectures.

---

Speakers / sources mentioned

Speakers / participants

• Host (Siddharth) — intro and moderation
• Max Simchowitz (CMU) — main speaker
• Tonghe — Max’s master’s student (example: shirt folding)
• Steven — audience question (about injecting noise at inference)
• Elon — audience question(s)
• Unnamed audience member — questions about training (one-step vs multi-token / one-step prediction)
• Yilun — audience question on iterative reasoning / transfer of insights

Sources / referenced works

• Moravec’s paradox
• Zhang et al. (2026) (mentioned for action chunking explanation)
• Berkeley work (mentioned for action chunking explanation)
• Chao Yu (mentioned as having tried hypotheses on hardware)
• DART (noise-augmented data collection / manifold fattening idea)
• Adam Block and Dylan Foster (theoretical literature about compounding error in discrete settings)
• Sergey’s group (recent paper debunking some aspect related to action chunking vs compounding error)
• Mara Daniels and Philippe Rigollet (log-concave distribution results; contraction of flow fields)
• Holden (mentioned in connection with diffusion resembling nearest-neighbor selection in unconditional settings)
• Abhishek Shetty, Navadita Sharma, Pulkit (with the speaker, referenced for conditional diffusion work)
• Jensen (mentioned as a “shill” for compute/FLOPs argument)
• PPO (Proximal Policy Optimization) — used as an RL algorithm example