Diffusion policy: Visuomotor policy learning via action diffusion
1. Paper overview
In one sentence: Denoising entire action sequences captures coherent alternatives, while executing only a short segment trades temporal consistency against feedback delay. e-identitye-formulatione-multimodale-backbonese-protocole-real-pushe-latencye-hyperparameters
| At a glance | What to know |
|---|---|
| Research problem | Source description Human demonstrations can offer several valid actions for the same observation, yet a robot must commit to a coherent trajectory and make precise corrections. Single-step regression can average incompatible choices; independently sampled multimodal actions can alternate between them. The paper investigates whether diffusion over action sequences can represent this diversity while remaining trainable and responsive. e-probleme-multimodale-horizon |
| Core mechanism | Source description The method combines conditional action diffusion, receding-horizon execution and visual features computed once per policy inference. It provides both a temporal CNN and a time-series diffusion transformer. e-formulatione-backbones |
| A key reported result | Real-world Push-T: CNN with end-to-end vision: 0.80 IoU, 95% success, 22.9 seconds. End-state IoU; thresholded success; mean episode duration. 136 demonstrations; 12-hour training budget; 20 matched initial conditions. Last checkpoint except IBC, selected by minimum training-action MSE. Human: 0.84, 100%, 20.3 seconds; best LSTM-GMM success 20%; IBC 0%. Frozen R3M diffusion reaches 80%. Executed two-stage manipulation: align the block, then clear the view. Success thresholds use minimum demonstration IoU; perturbation examples were a separate episode. e-real-pushe-push-protocole-perturbation |
| Reading caution | Source description The authors identify dependence on adequate demonstration data and higher inference cost than simpler policies. Action chunks mitigate latency but may be insufficient for high-rate control. e-limitations |
Core contributions
- Source description
The method combines conditional action diffusion, receding-horizon execution and visual features computed once per policy inference. It provides both a temporal CNN and a time-series diffusion transformer. e-formulatione-backbones
- Author claim
The authors report a 46.9% average relative improvement. Appendix B.2 selects the best diffusion variant and baseline per column of Tables 1, 2 and 4, excluding MH columns. This is a selected relative-improvement average, not a pooled success rate or a 46.9-percentage-point gain. e-aggregate
Figure 2. Observation-conditioned denoising sits inside a slower cycle of execution and new feedback. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the observation images and robot pose, then follow the timeline: recent observations condition a predicted action sequence, and a later observation starts another prediction. Section 2.3 defines the three horizons: T_o observations, T_p predicted actions and T_a executed actions. In panel b, observation features modulate temporal convolutions through FiLM's channel-wise scale and bias. In panel c, observation embeddings enter cross-attention; the caption specifies that action attention sees only the current and preceding action embeddings. The blue return arrows repeat the denoising network K times. Equation (4) fixes the update direction: subtract predicted noise from the candidate action sequence. e-formulatione-backbonese-vision
What it supports. The policy repeatedly refines actions while reusing a fixed observation representation. Its output can contain a temporally coordinated plan without paying to generate future images. Actual feedback arrives when the robot executes a segment and new observations condition the next sequence, rather than at every denoising iteration.
Where the evidence stops. The caption describes T_a output steps, while Section 2.3 distinguishes T_p predicted from T_a executed; this guide follows that explicit definition. Warm-starting is described as an option, without a separate quantified ablation.
2. Motivation
2.1 The problem and the proposed response
Human demonstrations can offer several valid actions for the same observation, yet a robot must commit to a coherent trajectory and make precise corrections. Single-step regression can average incompatible choices; independently sampled multimodal actions can alternate between them. The paper investigates whether diffusion over action sequences can represent this diversity while remaining trainable and responsive. e-probleme-multimodale-horizon
2.2 What this reading follows
A robot pushing around an obstacle can take either side, but it must maintain that choice long enough to make progress. Diffusion Policy addresses this problem by learning a distribution over action sequences from demonstrations. At inference, it turns noise into a candidate sequence, executes part of it, then conditions the next prediction on new observations. The same framework supports a temporal CNN or a transformer denoiser. This reading follows the supplied journal extension through its mechanism, simulated benchmarks, encoder ablations and real-robot evaluation. The evidence supports capable imitation under the tested conditions, while action-space choices, checkpoint selection and conflicting implementation details complicate universal performance claims. e-identitye-formulatione-multimodale-backbonese-protocole-real-pushe-latencye-hyperparameters
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational action-policy/diffusion classification is supported. World-action architecture, prediction paradigm and quadrant remain not applicable: the learned distribution is over actions conditioned on observations, without an explicit future-world prediction branch. Section 4.5's linear example motivates implicitly learning task-relevant dynamics, but does not establish a joint world/action model or an inverse-dynamics pipeline. e-formulatione-control
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Train a conditional action distribution, then sample from it
A training example supplies recent observations and a demonstrated action sequence. The learner corrupts that sequence at a sampled diffusion iteration and asks the network to recover the injected noise. Equation (5) is thus a supervised noise-prediction objective, even though the final product is a robot policy. At inference, no demonstration sequence is available: the process begins with Gaussian noise, and Equation (4) repeatedly subtracts the learned correction while conditioning on the actual observations. The visual encoder is trained jointly in the default configuration, but its features are computed once for that observation window. The state of the diffusion process is the candidate action sequence. It is not a simulated world state, and the loop does not obtain new camera feedback until execution advances. e-formulatione-ddpme-vision
Figure 3. Diversity between rollouts can coexist with commitment within a rollout. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel as multiple action trajectories from the same test state, not as a single robot taking every colored path. The blue dot marks the end-effector and the gray T is the object. In the diffusion panel, routes pass around either side before approaching the object. LSTM-GMM and IBC favor one route in the illustrated sample, while BET's paths cross back and forth. The caption specifies 40-step rollouts from the best-performing checkpoint. Section 4.3 connects this picture to sequence prediction: independently drawing successive actions from a multimodal distribution can change the selected route before the robot reaches useful contact. e-multimodal
What it supports. The example supports the authors' argument that representing multiple valid choices is only part of the problem. Actions must also remain mutually consistent over time. Diffusion's trajectory representation can retain alternatives across samples while producing a coherent route within a particular rollout, as illustrated here.
Where the evidence stops. This is a qualitative diagnostic at one illustrated state with selected checkpoints. It does not quantify mode probabilities, prove unbiased sampling over all states, or isolate sequence prediction from the other differences between methods.
5.2 Separate prediction length from time without feedback
The paper's three horizons answer different questions. T_o determines how much recent observation history the policy sees; T_p determines how far its candidate action sequence extends; T_a determines how many commands it executes before replanning. Figure 5 illustrates why the last quantity is consequential: short commitments may lose useful temporal consistency, whereas long commitments postpone correction. As a concrete configuration, Table 7 gives visual Square horizons of two observations, sixteen predicted actions and eight executed actions. Appendix B.1 also finds that adding more visual history does not monotonically improve performance. The robot's lower-level controller then converts desired commands into execution. For real Push-T, commands at 10 Hz are interpolated to 125 Hz; that controller frequency is not the rate of new diffusion-policy decisions. e-formulatione-horizone-hyperparameterse-preprocessinge-hardware
5.3 Interpret broader behavior without promoting it to a world model
The journal extension tests bimanual tasks by expanding observations and actions to both arms and grippers. It reports success rates of 55% for egg beating and 75% for both mat unrolling and shirt folding, with twenty trials per task, while describing failures around grasping and repeated corrections. These are useful demonstrations of coordinated execution, but no learned-policy baseline is supplied for those tasks. Reader interpretation: such behavior can require predictive internal features without establishing an explicit world-model architecture. Section 4.5 supports that distinction through a linear-control example in which predicting future actions depends on task-relevant closed-loop dynamics. Section 2.3 still defines the learned output distribution over actions alone. The foundational-action-policy taxonomy therefore fits the demonstrated mechanism better than a joint future-world/action quadrant. e-identitye-bimanuale-controle-formulation
5.4 Training and inference
During training
Sample a demonstration, diffusion iteration and Gaussian corruption; regress the injected noise with mean squared error. Visual encoders train jointly with the denoiser. The square-cosine noise schedule is used; the training objective does not require the negative samples used to estimate an energy-based policy's normalization constant. e-formulatione-visione-stability
Appendix A specifies per-dimension action scaling to [-1,1], except rotations and nearly constant dimensions; position control uses a 6D rotation representation. Training uses random image crops and inference uses center crops. State/image batch sizes are 256/64, with cosine learning-rate scheduling and CNN/transformer warmups of 500/1000 steps. e-preprocessing
During inference
Initialize the candidate sequence with Gaussian noise and repeatedly subtract predicted noise according to the schedule. Observations stay fixed through this loop. The resulting commands are executed by robot controllers; no explicit dynamics rollout, reward optimization or inverse-dynamics action extractor is used. e-formulatione-hardware
Section 3.4 reports 100 training diffusion steps and 10 DDIM inference steps, yielding 0.1-second latency on an Nvidia 3080. Appendix A.4 and Tables 7–8 instead specify 16 real-world inference steps; simulation uses 100. These settings are not silently reconciled. e-latencye-preprocessinge-hyperparameters
5.5 Implementation flow
- Encode observations
Each camera has a separate ResNet-18 encoder; frames are encoded independently and concatenated. The default encoder is trained from scratch with the policy, uses spatial softmax pooling, and replaces BatchNorm with GroupNorm to accommodate exponential moving averages. e-vision
- Refine a sequence
The CNN uses temporal convolutions and FiLM conditioning on observations and diffusion iteration. The transformer takes noisy action tokens, incorporates observation cross-attention, and uses causal action attention as defined in Figure 2's caption. The authors recommend starting with the CNN; the transformer can handle sharper action changes but needs more tuning. e-formulatione-backbones
- Execute and observe again
At time t, predict T_p actions using T_o observations, execute T_a without replanning, then update observations. Section 2.3 also describes warm-starting from unexecuted actions when prediction extends beyond execution. This is an available procedure, not a separately demonstrated gain. e-formulation
6. Experiments & results
Diffusion Policy learns a conditional distribution over demonstrated action sequences, then repeatedly denoises a candidate sequence and executes a short segment before observing again. Visual features condition action generation rather than being predicted as future images. The journal extension adds encoder studies and bimanual manipulation. Its strongest evidence combines simulation comparisons with executed real-robot tasks, while heterogeneous protocols and conflicting configuration statements limit simple headline comparisons.
6.1 Read the original evidence
Table 2. The visual benchmarks favor diffusion, but backbone and checkpoint choice both matter. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a task and demonstration source before comparing rows: ph denotes proficient-human data and mh denotes mixed-human data. Each cell contains best-checkpoint performance followed by the average of the last ten checkpoints, as defined in Table 1. C and T identify CNN and transformer denoisers. Transport PH is instructive: the CNN's 1.00/0.93 exceeds LSTM-GMM's 0.88/0.62, and its last-ten average exceeds the transformer's 0.81. Most columns measure success; Push-T measures target coverage according to Section 5.2. Keep those metrics separate when reading across tasks. Table 3 supplies demonstration counts and task characteristics. e-visual-resultse-protocole-datae-aggregatee-backbones
What it supports. The CNN's strong Transport result survives switching from best-checkpoint selection to the last-ten average, making the finding less dependent on one favorable checkpoint. The table also shows that selecting the transformer is not automatically an improvement: the CNN is stronger on several visual tasks, including this Transport comparison.
Where the evidence stops. The p. 8 footnote corrects Robomimic evaluation from 50 to 22 initializations. Methods use selected action spaces, and uncertainty intervals are absent. Appendix B.2's 46.9% aggregate uses selected columns and variants, not this table alone.
Table 6, setup panels and quantitative table. Real Push-T evaluates the final placement and the transition out of pushing. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The hardware panel separates front/wrist policy inputs from the top-down camera used only for evaluation. Panel b shows two stages: align the T block, then move the end-effector to its end-zone. The table measures final-state IoU rather than maximum overlap during the episode. Its Succ% row displays fractions, so 0.95 means 95% success; Dur. is mean seconds. Success is thresholded using the minimum demonstration IoU. T-E2E denotes the transformer with end-to-end vision; ImgNet and R3M use frozen pretrained features, and E2E is the end-to-end CNN variant. Appendix C.1 establishes the shared initial conditions and checkpoint rules. e-real-pushe-push-protocole-hardwaree-perturbation
What it supports. The end-to-end CNN achieves 0.80 IoU and 95% success, near the human reference of 0.84 and 100% in this experiment. The best LSTM-GMM success is 20%, while frozen R3M diffusion reaches 80%. This is evidence from executed physical behavior with a required stage transition and explicit termination condition.
Where the evidence stops. Each method has 20 matched trials after a 12-hour training budget; IBC uses a different checkpoint-selection rule. No uncertainty interval is given. Figure 8's perturbation demonstrations were a separate episode and are not included in the 95% result.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Robomimic visual Transport (PH) 200 proficient-human demonstrations; 3000 training epochs. Three training seeds; the p. 8 footnote corrects Robomimic evaluation to 22 initializations, despite the table's stated 50. | CNN: 1.00 / 0.93; transformer: 0.98 / 0.81. Success rate: best checkpoint / average of last 10 checkpoints | LSTM-GMM: 0.88 / 0.62; IBC: 0.00 / 0.00. The CNN improves both selections; action spaces differ across methods, so this is a system comparison rather than an isolated denoiser ablation. e-visual-resultse-protocole-data |
| Franka Kitchen four-subtask completion State observations; 656 demonstrations, each performing four interactions; 4500 training epochs and Section 5.2 checkpoint protocol. | CNN 0.99; transformer 0.96. p4: frequency of completing at least four interactions | BET 0.44; LSTM-GMM 0.34; IBC 0.24. Supports long-horizon execution under the benchmark. The prose's 213% improvement does not follow from the printed 0.99 versus 0.44 pair; retain the table values. e-multistagee-protocole-data |
| Vision-encoder ablation on visual Square (PH) CNN policy; Table 5 states 500 epochs, evaluation every 50 epochs and 50 initializations. | CLIP ViT-B/16: scratch 0.22, frozen 0.70, fine-tuned 0.98. Reported success rate | ResNet-18: 0.94, 0.58, 0.92 respectively; ResNet-34: 0.92, 0.40, 0.94. Adaptation matters, but pretraining does not improve every architecture. The 50-epoch claim in adjacent prose and the global 22-initialization footnote leave aggregation details unresolved. e-encoder-ablatione-protocol |
| Real-world Push-T 136 demonstrations; 12-hour training budget; 20 matched initial conditions. Last checkpoint except IBC, selected by minimum training-action MSE. | CNN with end-to-end vision: 0.80 IoU, 95% success, 22.9 seconds. End-state IoU; thresholded success; mean episode duration | Human: 0.84, 100%, 20.3 seconds; best LSTM-GMM success 20%; IBC 0%. Frozen R3M diffusion reaches 80%. Executed two-stage manipulation: align the block, then clear the view. Success thresholds use minimum demonstration IoU; perturbation examples were a separate episode. e-real-pushe-push-protocole-perturbation |
| Real-world mug flipping 20 trials; varied mug poses. Table 3 lists 250 demonstrations; LSTM-GMM uses an unspecified subset of the same data. | Diffusion Policy 90%. Task success | LSTM-GMM 0%. Demonstrates 3D reorientation, but the unequal, incompletely specified training subsets prevent a clean data-matched attribution. e-muge-data |
| Real-world sauce pouring and spreading Franka tasks; last checkpoint after 1000 epochs. Appendix C.2 says 50 demonstrations with 90% training per task, conflicting with Table 3's 90. | Pouring 0.74 / 0.79; spreading 0.77 / 1.00. Pouring IoU and success; spreading coverage and success | Human 0.79 / 1.00 for both; LSTM-GMM pouring 0.06 / 0.00 and spreading 0.27 / 0.00. Different geometric metrics must remain separate. The reported pouring success fraction's trial denominator is not explicitly resolved by the protocol. e-saucee-sauce-protocole-data |
| Bimanual egg beating, mat unrolling and shirt folding 20 trials per task; respectively 210, 162 and 284 demonstrations. | 55%, 75% and 75%, respectively. Task success | No learned-policy baseline reported for these three tasks. Extends executed behavior to coordinated tool use and deformable objects. Failures include missed grasps, unfamiliar egg-beater positions and repeated unsuccessful corrections; no cross-task transfer test is established. e-bimanual |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Longer action commitments help initially, then make feedback too slow. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue Push-T and gray Square curves separately. The vertical axis is a difference from each task's maximum performance, so zero means that task's best setting rather than zero success. The left plot varies action execution horizon across the labeled step counts; both illustrated curves peak at eight. Moving farther right eventually reduces performance, especially for Square. The right plot varies latency, defined by the caption as the gap between the last observed frame and the first executable action. Its horizontal units are control steps, not seconds. These plots concern the effect of acting on an existing prediction before refreshed information can change behavior. e-horizone-formulatione-protocol
What it supports. The curves support a compromise between commitment and responsiveness. An intermediate execution horizon improves on single-step execution in these examples, whereas long open-loop segments lose accuracy. Small observation-to-action delays are comparatively tolerable under the tested position-control setup, with degradation becoming more visible as delays grow.
Where the evidence stops. Section 5.3 says peak performance persists through four delay steps, but the plotted Square curve already declines slightly. Also, Section 5.2 defines Push-T's metric as coverage despite the caption's success-rate wording. Neither plot provides uncertainty bands.
Table 5. Fine-tuning rescues the ViT result, but pretraining gains depend on the encoder. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read within a row to compare training strategies for one visual architecture. The ResNet rows use ImageNet-21k pretraining in their pretrained conditions; ViT-B/16 uses CLIP. The CNN action denoiser and Square PH task are shared. Fine-tuning uses an encoder learning rate ten times smaller than the policy network's. ViT changes from 0.22 when trained from scratch to 0.70 when frozen and 0.98 when fine-tuned. ResNet-18 instead has 0.94 from scratch and 0.92 with fine-tuning. Cross-row comparisons therefore change both the visual architecture and the pretraining source; they do not isolate model capacity alone. e-encoder-ablatione-protocol
What it supports. The strongest supported lesson is that visual adaptation can be decisive for a particular pretrained encoder. The ViT's frozen features leave a substantial gap to fine-tuning, while a small ResNet trained from scratch remains competitive. The table gives no basis for treating all frozen encoders or all pretrained encoders as equivalent.
Where the evidence stops. Table 5 states 500 training epochs and evaluations every 50 epochs, while adjacent prose says ViT reached 98% after 50. Checkpoint aggregation is unspecified, and the general Robomimic 22-initialization footnote complicates the caption's stated 50.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify dependence on adequate demonstration data and higher inference cost than simpler policies. Action chunks mitigate latency but may be insufficient for high-rate control. e-limitations
Main comparison tables omit confidence intervals. Best-variant selection, action-space differences and the Robomimic initialization bug qualify broad superiority claims. Figure 5 shows near-peak rather than literally identical performance across small delays; the prose's four-step robustness statement should not become a general latency guarantee. e-protocole-aggregatee-horizon
Internal inconsistencies affect reproduction: 10 versus 16 real-world denoising steps, conflicting sauce dataset counts, and ambiguous encoder-ablation checkpoint reporting. Figure 2's caption describes T_a outputs, whereas Section 2.3 explicitly predicts T_p and executes T_a. e-latencye-hyperparameterse-datae-sauce-protocole-encoder-ablatione-formulation
7.2 Questions for discussion
- How much of the gain remains when every method uses matched action spaces and checkpoint rules?
- Can action-sequence prediction's implicit dynamics representation generalize beyond the demonstrated control task?
8. Reproducibility audit
8.1 Requirements and known gaps
For visual Square, Table 7 specifies position control, T_o=2, T_a=8, T_p=16, two 84×84 images cropped to 76×76, 256M denoiser parameters, 22M vision parameters, learning rate 10^-4 and weight decay 10^-6. Reproduction needs the demonstrations, exact preprocessing, sampler and checkpoint-selection rules. e-hyperparameterse-preprocessinge-protocol
Real Push-T needs synchronized front/wrist observations and UR5 execution: 10-Hz policy commands are interpolated to 125 Hz. Bimanual inference uses the non-haptic controller with explicit kinematic constraints; haptic demonstration collection is a separate control mode. The paper links resources but does not establish their inspected availability here. e-hardwaree-bimanuale-identity
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does action commitment help when feedback delay is controlled?
Reader-proposed check, not an experiment performed here: train a CNN policy on simulated visual Push-T and Square with the same demonstrations, preprocessing and sixteen-step prediction horizon. Compare execution horizons 1, 2, 4, 8 and 16 while holding observation horizon at two. Evaluate each under added delays of 0, 2 and 4 control steps, using identical initializations and three training seeds. Verify that all requested initializations actually run, addressing the disclosed Robomimic bug. Report Push-T coverage and Square success separately, together with trajectory reversals and idle stalls. An intermediate-horizon advantage at matched delay would support temporal consistency; disappearance of that advantage would weaken the proposed explanation. Fix the action space and checkpoint-selection rule throughout. e-horizone-multimodale-protocole-hyperparameterse-preprocessing
Check 2: Is the ViT gain caused by task adaptation rather than checkpoint selection?
Reader-proposed check, not a reproduced result: repeat the Square PH encoder comparison using the same CNN policy, augmentations, data split and evaluation initializations. For ViT-B/16, compare random initialization, frozen CLIP weights and fine-tuned copies of those same CLIP weights; use the paper's tenfold smaller encoder learning rate for fine-tuning. Train each for 500 epochs across matched seeds and record every 50-epoch evaluation, reporting the final checkpoint and the complete curve rather than selecting a winner retrospectively. Include scratch ResNet-18 as a practical reference. A stable fine-tuning advantage over the frozen encoder would support task-specific representation adaptation; a gain appearing only at one selected checkpoint would not. Explicitly resolve the paper's 50-versus-500-epoch reporting ambiguity. e-encoder-ablatione-protocole-preprocessinge-hyperparameters
8.3 Reading coverage
Visual audit: All 19 original PDF pages were rendered and visually inspected, including the title block, all numbered figures/tables, numerical protocols and footnotes, references, and Appendices A–D. Six final crops were inspected individually; the Table 2 crop was tightened and viewed again to remove clipped caption prose. Figure 2's loop direction and causal-attention interpretation were checked against its caption and Equation (4); prediction/execution-horizon wording is disclosed in its caution. Figure 5's near-peak latency curves and Push-T metric wording were cross-checked against Section 5.2–5.3. Pages 4, 16 and 17 support the conflicting denoising settings; pages 7, 17 and 18 support the sauce-data/protocol caveats. The bimanual montage pages were inspected as qualitative endpoint evidence, not independent measurements of unseen trajectories. Supplemental videos and external code/data remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (p. 1)
- 1 Introduction and journal-extension statement (pp. 1–2)
- 2 Diffusion Policy Formulation, including 2.1–2.3 (pp. 2–3)
- 3 Key Design Decisions, including 3.1–3.4 (pp. 3–4)
- 4 Intriguing Properties, including 4.1–4.5 (pp. 4–6)
- 5 Evaluation, including 5.1–5.4 (pp. 6–9)
- 6 Realworld Evaluation, including 6.1–6.3 (pp. 9–11)
- 7 Realworld Bimanual Tasks, including 7.1–7.5 (pp. 11–12)
- 8 Related Work; 9 Limitations and Future Work; 10 Conclusion; 11 Acknowledgement; Funding (pp. 12–13)
- References (pp. 14–16)
- Appendix A, implementation, normalization, rotations, augmentation, hyperparameters and data efficiency (pp. 16–17)
- Appendix B, observation horizon and improvement calculation (pp. 16–17)
- Appendix C, real-world demonstrations and evaluation protocols (pp. 17–18)
- Appendix D, robot stations and all bimanual rollout montages (pp. 18–19)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- All nine supplied text chunks and all 19 PDF pages were read; no supplied body or appendix section was omitted.
- Identity/version: the observed title and eight authors match the catalog apart from title capitalization. Page 2 identifies this as an extension of Chi et al. (2023), adding control-theory discussion, encoder ablations and three bimanual tasks. The supplied 19-page author-hosted IJRR artifact has no explicit revision number/date or publisher pagination establishing equivalence to the catalog's 2025 journal pages 1684–1704; the publisher edition and earlier conference PDF were not supplied.
- The title block has affiliation superscripts but no institutional key; affiliations and equal-contribution semantics cannot be verified from those marks alone.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This supplied limitation was addressed by inspecting all PDF pages and six final original crops.
- Separate supplemental material availability has not been fully verified.
- No supplemental videos, external website, linked code or dataset files were inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block; p. 2, Introduction extension paragraph; p. 14, Chi et al. (2023) reference
The title is Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Authors are Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake and Shuran Song. The paper explicitly extends the 2023 RSS work with control-theory discussion, encoder ablations and three bimanual tasks; it links project resources.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract, Figure 1 and Introduction
Motivates multimodal distributions, temporal correlation, precision, action-sequence diffusion, visual conditioning and stable training.
Go to primary source ↓e-ddpmPDF p. 2, Section 2.1, Equations (1)–(2)
Defines Gaussian-initialized iterative denoising, noise predictor εθ and schedule factors α, γ and σ.
Go to primary source ↓e-formulationPDF p. 3, Figure 2 and caption; Sections 2.2–2.3, Equations (3)–(5)
Conditions action diffusion on observations; predicts T_p actions from T_o observations and executes T_a. Describes warm-starting unexecuted actions and noise-MSE training. Caption defines causal action attention and denoising subtraction, but describes outputs as T_a steps.
Go to primary source ↓e-backbonesPDF pp. 3–4, Section 3.1; Figure 2(b–c)
CNN temporal convolution uses FiLM; transformer action tokens use observation embeddings and diffusion iteration conditioning. CNN is recommended first; transformer is more sensitive to tuning.
Go to primary source ↓e-visionPDF p. 4, Sections 3.2–3.3
Default separate camera encoders are ResNet-18 without pretraining, with spatial softmax and GroupNorm, trained end-to-end; square-cosine noise schedule is used.
Go to primary source ↓e-latencyPDF p. 4, Section 3.4
Reports 100 training and 10 DDIM inference iterations with 0.1-second latency on an Nvidia 3080 GPU.
Go to primary source ↓e-multimodalPDF p. 5, Figure 3 and caption; Section 4.3; p. 4, Section 4.1
Push-T qualitative trajectories use 40-step rollouts from best checkpoints. Diffusion shows two routes with within-rollout commitment; other methods show bias or inconsistent paths. The text motivates temporal consistency and idle-action handling.
Go to primary source ↓e-horizonPDF p. 5, Figure 5 and caption; p. 8, Section 5.3 action-horizon and latency paragraphs
Plots performance differences from task maxima for Push-T and Square. Both peak at action horizon 8; long horizons degrade. Small delays have modest effects, although the plotted Square curve slightly declines before the prose's claimed four-step peak-preservation limit. Latency is measured from last observation to first executable action.
Go to primary source ↓e-stabilityPDF p. 6, Section 4.4, Equations (6)–(8), Figure 6 and caption
The score removes the action-independent normalization term; diffusion avoids estimating it with negative samples. Figure 6 compares training-action MSE on real Push-T with rollout performance on simulated Push-T.
Go to primary source ↓e-controlPDF p. 6, Section 4.5, Equation (9) and following derivation
A linear-system/linear-feedback example recovers the demonstration policy and expresses future actions through closed-loop dynamics, motivating task-relevant implicit dynamics representation rather than explicit state prediction.
Go to primary source ↓e-visual-resultsPDF p. 7, Table 2, Transport PH column and all model rows; Table 1 caption
Transport PH visual results (best/last-ten average): CNN 1.00/0.93, transformer 0.98/0.81, LSTM-GMM 0.88/0.62, IBC 0.00/0.00. The paired format is defined in Table 1's caption.
Go to primary source ↓e-dataPDF pp. 6–7, Section 5.1; p. 7, Table 3
Robomimic PH datasets contain 200 demonstrations per task; MH variants contain 300. Kitchen has 656 demonstrations of four interactions. Table 3 lists 136 real Push-T, 250 mug-flip and 90 demonstrations for each sauce task.
Go to primary source ↓e-protocolPDF p. 8, Section 5.2 and bottom footnote; p. 7, Tables 1–2 captions
Three seeds, nominally 50 initializations, last ten checkpoints saved every 50 epochs; best checkpoints also reported for Robomimic/Push-T. Footnote corrects Robomimic to 22 initializations for all methods. State/image training lasts 4500/3000 epochs; methods use their selected action spaces. Push-T uses target coverage, unlike binary-success tasks.
Go to primary source ↓e-multistagePDF p. 8, Table 4, Kitchen p4 column and caption; Section 5.3 long-horizon paragraph
Kitchen p4 is 0.99 CNN, 0.96 transformer, 0.44 BET, 0.34 LSTM-GMM and 0.24 IBC. It measures at least four interactions. The adjacent prose claims 213% improvement, which is inconsistent with the displayed best-versus-best pair.
Go to primary source ↓e-aggregatePDF p. 17, Appendix B.2; pp. 7–8, Tables 1, 2 and 4
B.2 excludes MH columns, selects maximum baseline and diffusion-variant performance in each remaining column, averages relative improvements and reports 0.46858 ≈46.9%.
Go to primary source ↓e-encoder-ablationPDF p. 8, Section 5.4; p. 9, Table 5 and continuation of Section 5.4
ResNet-18/34 use ImageNet-21k pretraining, ViT-B/16 uses CLIP. Scratch/frozen/fine-tuned success is 0.94/0.58/0.92, 0.92/0.40/0.94 and 0.22/0.70/0.98. Fine-tuning uses tenfold lower encoder learning rate. Caption states 500 epochs and evaluation every 50 epochs under 50 initializations; prose says ViT reaches 98% after 50 epochs. Exact table checkpoint aggregation is not specified.
Go to primary source ↓e-real-pushPDF p. 9, Table 6, all metric rows and caption; Section 6.1
CNN end-to-end vision yields 0.80 end-state IoU, 0.95 success, 22.9-second duration; human 0.84/1.00/20.3. Best LSTM-GMM success is 0.20, IBC 0.00, frozen-R3M diffusion 0.80. Success uses minimum demonstration IoU; the robot must enter an end-zone. Figure labels separate policy cameras from evaluation-only top-down camera.
Go to primary source ↓e-push-protocolPDF p. 17, Appendix C.1.1–C.1.2
136 demonstrations; 12 hours training; last checkpoint except IBC's minimum training-action-MSE checkpoint; 20 matched initial conditions. Stop after 0.5 seconds in end-zone or 60 seconds. IoU is measured in top-down pixel space.
Go to primary source ↓e-perturbationPDF p. 9, Section 6.1 robustness paragraph; p. 10, Figures 7–8 and captions
Illustrates recovery after camera occlusion and block displacement. These interventions were applied in a separate episode from Table 6's quantitative comparison.
Go to primary source ↓e-mugPDF p. 10, Section 6.2, Figure 9 and caption
Mug goal is lip down and handle left. Diffusion success is 90% over 20 trials; LSTM-GMM is 0% with a subset of the same training data, whose size is not stated.
Go to primary source ↓e-saucePDF pp. 10–11, Section 6.3; p. 11, Figure 10 table and caption
Pouring uses IoU to a centered circle; spreading uses sauce coverage. Diffusion reports 0.74 IoU/0.79 success and 0.77 coverage/1.00 success; human 0.79/1.00 for both; LSTM-GMM 0.06/0.00 and 0.27/0.00. Success thresholds use minimum human performance.
Go to primary source ↓e-sauce-protocolPDF pp. 17–18, Appendix C.2.1–C.2.2
States 50 demonstrations with 90% training per task, 1000 epochs and last checkpoints. Initial positions are matched but sauce shape is not; quantity is fixed. Episodes end after upward spoon motion for 0.5 seconds or an unsafe behavior judgment. Coverage uses homography and the larger of the two camera estimates.
Go to primary source ↓e-bimanualPDF pp. 11–12, Sections 7.1–7.5 and Figures 11–13; pp. 18–19, Appendix D.2, Figures 16–21
Bimanual observations include actual/desired poses and gripper widths plus two scene/two wrist cameras; actions cover both arms and grippers. Egg beating uses haptic demonstrations, mat and shirt use VR. Success over 20 trials is 55%/75%/75% after 210/162/284 demonstrations. Reports task-specific grasp/initial-position/termination failures and shows rollout endpoints; no learned-policy baseline appears.
Go to primary source ↓e-limitationsPDF p. 13, Section 9
Identifies behavior-cloning dependence on adequate data and increased inference computation/latency, which action-sequence execution only partly mitigates.
Go to primary source ↓e-preprocessingPDF p. 16, Appendix A.1–A.5 and B.1; Figures 14–15
Specifies action scaling exceptions, rotation representation, image crops, batch sizes, warmups, simulation's 100-step iDDPM and real-world 16-step DDIM. State Push-T CNN Table 1 uses inpainting rather than FiLM. Observation horizon two is a practical compromise; data-size plots compare diffusion and LSTM-GMM.
Go to primary source ↓e-hyperparametersPDF p. 17, Tables 7–8, headers, captions and Square/Real Push-T/Block Push rows
CNN Square uses position control, horizons 2/8/16, two 84×84 images cropped to 76×76, 256M diffusion/22M vision parameters, learning rate 1e-4, weight decay 1e-6 and 100/100 diffusion steps. Real Push-T uses horizons 2/6/16 and 100/16 steps. Table 8 specifies velocity control for transformer Block Push, an exception to the general position-control description.
Go to primary source ↓e-hardwarePDF p. 18, Appendix D.0.1 and D.1; p. 9, Section 6.1; p. 11, Section 7.1
UR5 interpolates 10-Hz commands to 125 Hz, with two policy cameras. Franka non-haptic inference uses constrained differential kinematics and joint control; haptic data collection uses a distinct torque controller. The bimanual section adds wrist cameras beyond the generic station's two-camera description.
Go to primary source ↓8.5 Primary sources
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion ↗
PDF · 14,593 extracted words
Source fingerprint
efc284c0e414f90c6ef69e1a4eb43423618971be68f9401801f34b0ddff3c073