DreamGen: Unlocking Generalization in Robot Learning through Video World Models
1. Paper overview
In one sentence: DreamGen converts adapted video priors into offline robot demonstrations, trading manual action collection for imperfect pseudo-labels and costly synthetic generation. e01e02e03e04e06e12e13e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Collecting demonstrations for every behavior and environment is expensive, while contact-rich simulation requires engineering and can transfer poorly. DreamGen asks whether pretrained video priors can supply new robot demonstrations after limited embodiment adaptation. e02 |
| Core mechanism | |
| A key reported result | GR1 novel behaviors in the seen environment: 43.2 Mean partial-credit task score, reported as success rate (%). 14 tasks; GR00T N1 trained on 50 neural trajectories per task versus pick-and-place-only adaptation; ten evaluation rollouts per checkpoint with position randomization. 11.2 in Table 1; nearby prose also prints 11.8. Improvement is 32.0 percentage points using the table. It does not mean 43.2% of trials fully completed the named task. e12e13 |
| Reading caution | Source description Authors acknowledge simple tasks, limited kinematic coverage, no direct comparison with other video-learning methods, manual initial frames and evaluator hallucination. Generating 240K RoboCasa samples reportedly required 54 hours on 1,500 NVIDIA L40 GPUs. e16 |
Core contributions
- Source description
DreamGen Bench measures generated-video instruction following and physics alignment and tests their relationship to downstream policy learning. This is a diagnostic benchmark alongside a data-generation method. e14
Figure 2. Generated motion becomes reusable policy-training data through a separate labeling stage. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the numbered panels from the upper left to the upper right, then across the lower row. Step 1 adapts the video model using teleoperated robot footage. Step 2 changes the initial image and instruction to produce a new sequence, illustrated by watering flowers and picking up a tangerine. Step 3 groups frames into transitions and attaches inferred action chunks. Step 4 stores these labeled sequences as neural trajectories and uses them to train a visuomotor policy. The lower-right action arrow belongs to that policy. Sections 2.3–2.4 specify the action-label alternatives and training conditions that the overview compresses into icons. e02e03e04e06
What it supports. The main architectural contribution is the separation of generation, labeling and policy learning. Once a generated video has been converted into a neural trajectory, it can supervise different policy architectures. This places the expensive video model in the data-production stage and makes downstream execution depend on the trained policy.
Where the evidence stops. These images illustrate data flow, not independently verified physical rollouts. A convincing generated movement can still carry inaccurate action labels. The overview also omits the zero-filled state inputs and separate GR00T N1 action encoders/decoders described in Section 2.4.
2. Motivation
2.1 The problem and the proposed response
Collecting demonstrations for every behavior and environment is expensive, while contact-rich simulation requires engineering and can transfer poorly. DreamGen asks whether pretrained video priors can supply new robot demonstrations after limited embodiment adaptation. e02
2.2 What this reading follows
Imagine teaching a humanoid to water flowers when its embodiment-specific demonstrations mostly show pick-and-place. DreamGen first adapts a video model to that robot, then supplies a flower scene and a new instruction. The generated motion becomes training data only after another model infers actions between frames. A separate policy learns from these neural trajectories and later controls the robot. The evidence is strongest when read at three levels: the data pipeline, measured policy gains, and the scoring rules behind those gains. This edition also preserves the supplied revision's conflicting counts and aggregates, because they affect what a reproduction could honestly claim. e01e02e03e04e06e12e13e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The dataset-oriented catalog placement fits synthetic data generation, pseudo-action annotation and demonstrations. The resource itself is a method paper with a benchmark. Its deliberately separate generator, labeler and policy do not establish a One Model architecture or joint future/action prediction; leaving control quadrants not applicable is reasonable. e02e06e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Adapt an embodiment, then ask for a new behavior
The first learning problem is making a pretrained video model depict the target robot consistently. DreamGen fine-tunes on teleoperated footage and uses LoRA by default; the authors' rationale is to learn embodiment motion while reducing loss of prior internet-video knowledge. Generation is then conditioned on an initial frame and a language instruction. This matters for the generalization claim: the target scene is supplied explicitly, and the new verb is requested in the prompt. The GR1 generalization setup starts from pick-and-place demonstrations, whereas the existing-task augmentation experiments additionally adapt the video model on small target-task datasets. Keeping those protocols separate prevents the new-behavior result from being confused with ordinary augmentation. Initial-frame capture and prompt writing still involve human work even when no new target-behavior teleoperation is collected. e03e10e12
Figure 3. Two labeling routes connect generated visual transitions to robot-policy supervision. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two panels by their training targets. On the left, the IDM receives the current and future images through the vision encoder and learns intervening action chunks with a diffusion transformer and flow-matching objective. Language and proprioception are excluded from this labeler. On the right, both frames enter the ViT encoder, while the current frame also follows a bypass toward the decoder; reconstruction targets the future frame through a quantized representation. Section 2.3 specifies that DreamGen actually extracts continuous embeddings before quantization as latent labels, using a frame one second ahead. Thus the drawn VQ codebook belongs to representation learning and should not be read as the deployed motor-action interface. e04e05e06e07e21
What it supports. IDM and LAPA obtain supervision differently: IDM needs embodiment action demonstrations, whereas the latent model can learn visual changes from videos without target-robot action labels. DreamGen uses IDM by default in later experiments because it supports training a policy solely on neural trajectories when suitable teleoperation data exist.
Where the evidence stops. Action endpoints conflict: the IDM output ends at t+H−1, while its input and prose extend to t+H despite calling this H actions. The loop label k is undefined in the supplied account. Preserve the diagram; exact indexing and overlap reconciliation require clarification.
5.2 Turn visual change into supervision
The second learning problem is attaching a usable training target to generated motion. An IDM learns from real image/action pairs to infer what action chunk could connect two frames, then applies that mapping to synthetic video. LAPA instead compresses frame-to-frame visual change using a VQ-VAE trained on diverse robot and human footage; DreamGen takes its continuous pre-quantization representation as the label. These routes differ in what real action information they require, and the reported neural-only policies use IDM supervision. At policy-training time, images and instructions are retained but synthetic state is filled with zeros. GR00T N1 separates real and neural action encoders/decoders during equal-ratio co-training. The world model, labeler and control policy consequently have distinct jobs, rather than forming a single jointly predicting controller. e02e04e05e06e07
5.3 Trace success through the entire chain
Reader analysis: evaluate generated video, recovered motion and executed policy behavior as separate links. DreamGen Bench checks instruction following and physics alignment; its latter score averages VideoCon-Physics and Qwen judgments. Figure 6 relates video scores to RoboCasa policies trained with 7K generated trajectories per model, but this association does not isolate which link causes improvement. The replay diagnostic in Appendix A can expose implausible inferred motion before expensive policy training, though it is not a quantitative decomposition of errors. Finally, real-robot tables use task-specific progress scores, so a higher number need not mean more fully completed tasks. A persuasive replication would preserve those scores while separately reporting complete execution and uncertainty. It would also resolve the inconsistent aggregates and human-evaluation entries before treating benchmark agreement as calibrated reliability. e12e13e14e15e19
5.4 Training and inference
During training
In RoboCasa, the video model sees 1,200 human demonstrations. IDM/policy regimes use 720, 2,400 or 7,200 ground-truth trajectories across 24 tasks; these budgets must not be confused with the video-model budget. e07
For GR1 augmentation, Appendix F adapts the video model on 2,884 pick-and-place trajectories, then 10 examples per target task except 25 for folding. Its IDM stays trained on pick-and-place. Franka starts from 49,895 DROID examples and further adapts the video model on 11/10/8 target examples, without task-specific IDM post-training. e10
WAN2.1 uses learning rate 10⁻⁴ and LoRA rank/alpha 4/4; epochs/batches are RoboCasa 100/32, GR1 75/64, DROID 5/64 and SO-100 200/8. LAPA uses codebook size 8, sequence length 16, 100K steps and batch 1024 on the diverse Table 3 mixture. e17e05
During inference
Video rollout and action labeling happen during dataset construction. Deployment runs the learned visuomotor policy on current observations and instructions; DreamGen does not describe an online video planner or joint future-video/action controller. e02e06
IDM labels can directly supervise an embodiment's action space; latent labels provide a separate co-training representation. The paper's demonstration of neural-only control uses IDM actions, not uncalibrated latent codes as motor commands. e05e06e07
5.5 Implementation flow
- Adapt the video prior
Fine-tune mainly WAN2.1 with LoRA to retain pretrained knowledge while adapting embodiment motion. For RoboCasa and DROID, left/right/wrist images occupy three cells of a 2×2 grid; the fourth is black. e03
- Generate demonstrations
Condition on an initial image and task instruction. Real-world initial images are manually captured with object randomization; novel scenes and verbs are introduced through these images and prompts. e03
- Infer actions between frames
The IDM combines SigLIP-2 with a diffusion transformer trained by flow matching. It receives two frames without language or proprioception and predicts intervening action chunks using a sliding window. Alternatively, LAPA learns visual changes with a VQ-VAE; labeling uses continuous pre-quantization embeddings with a future frame one second ahead. e04e05
- Learn the control policy
Train on images, instructions and pseudo-action targets, zero-filling unavailable synthetic state. Co-training samples real and neural trajectories equally; GR00T N1 uses separate action encoders/decoders. Behavior/environment generalization uses neural trajectories alone for the downstream policy. e06
6. Experiments & results
DreamGen turns embodiment-adapted video generation into offline robot-training data: generate a trajectory, infer its actions, then train a separate visuomotor policy. It improves simulated and real-robot performance, but depends on an action labeler, manual initial frames and substantial generation compute. Reported real-world scores include partial credit.
6.1 Read the original evidence
Table 1. Novel-task scores improve, but the metric awards progress and the lower aggregate is inconsistent. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each pair of model rows vertically before looking at the Average column. The upper block holds the environment familiar and changes behavior across 14 tasks. The lower block changes the environment and separates six familiar-behavior tasks from seven new-behavior tasks. The example images show evaluation scenes; the red rectangles identify object randomization regions. Section 3.2 specifies 50 neural trajectories per upper-block task. Appendix I.1 reports ten evaluation rollouts per checkpoint, and Table 8 defines partial-credit criteria. For example, picking up a bottle earns half credit on Pour Water before any pouring occurs. These scores measure graded task progress rather than a uniform binary completion event. e03e12e13e20
What it supports. The upper block reports 43.2 versus 11.2, a calculated gain of 32.0 percentage points. Every printed lower-block baseline is zero, while the DreamGen task scores are positive. This supports transfer beyond the demonstrated pick-and-place setting under the paper's scoring protocol, with initial images from the target environments still supplied.
Where the evidence stops. The lower Average prints 28.5, but an unweighted mean of its 13 printed task scores is about 31.9; the weighting is unresolved. Table 8 also scores Open Microwave using closing criteria. Neither discrepancy is silently repaired here.
Table 4. Aggregate gains coexist with task regressions and weaknesses in neural-only learning. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first three numeric columns contain real-data-only policies with 30, 100 and 300 trajectories per task. Match each of them to its corresponding '+ NT' column; the original caption defines NT as 240K neural trajectories in total. The final column removes real trajectories from downstream policy training. Start with the Average row to see the overall effect, then inspect task rows to understand its distribution. For instance, OpenSingleDoor loses performance in the high-data augmentation comparison, while several pick-and-place tasks remain at zero in the neural-only column. The video model and IDM still depend on real data even when the downstream policy column says ONLY NT. e07e09
What it supports. At the largest real-data budget, augmentation changes the average from 49.59 to 57.61, a calculated 8.02-point gain. Neural-only training reaches 20.55, showing usable supervision but leaving a large gap to the best co-trained policy. The result supports complementing real demonstrations rather than assuming that all real-data requirements disappear.
Where the evidence stops. Average improvements do not imply per-task improvement: OpenSingleDoor falls from 58.82 to 47.12 at the 300-trajectory budget. The table provides no uncertainty estimates, and Section 3.1 refers readers to the original RoboCasa evaluation protocol rather than fully restating it.
Table 5. Real-robot gains depend on the policy and task, with larger real-data baselines providing context. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each embodiment, compare a policy's first-block row with the matching '+ Neural Traj.' row. The last block uses the larger real dataset and is a separate supervision-budget comparison. Dashes indicate missing combinations, not zero success. GR1 covers hammering, wiping, folding and stacking; Franka covers pick-and-place, cube stacking and tool use; SO-100 covers pick-and-place and tic-tac-toe. Only GR1 has an explicit Average column. Appendix I explains the small rollout counts and graded scores, and Appendix F gives task-specific adaptation data. Therefore, a single pooled average across embodiments would obscure both different tasks and different scoring rules. e10e11e13e18
What it supports. GR00T N1 improves on the listed tasks: for example, Franka pick-and-place rises from 40 to 60, and SO-100 tic-tac-toe rises from 25 to 65. Yet neural augmentation is not uniformly beneficial across policies: DP hammering falls from 35 to 15. The high-data block also shows that extra real demonstrations remain valuable.
Where the evidence stops. Preserve the source versions: this table prints a GR1 GR00T N1 augmented average of 46.0, whereas Figure 5 prints 46.4 and slightly different folding scores. Counts also vary between the main text and appendices. No confidence intervals accompany these real-robot entries.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboCasa data augmentation across 24 tasks GR00T N1; 30/100/300 ground-truth trajectories per task, with or without 240K IDM-labeled neural trajectories; original RoboCasa evaluation protocol. | 23.32 / 39.94 / 57.61 with neural trajectories; neural-only 20.55. Average task success (%) | Ground-truth-only: 17.44 / 32.07 / 49.59. Higher averages support augmentation. Neural-only training remains below the medium/high real-data baselines, and individual tasks can regress. e07e09 |
| GR1 novel behaviors in the seen environment 14 tasks; GR00T N1 trained on 50 neural trajectories per task versus pick-and-place-only adaptation; ten evaluation rollouts per checkpoint with position randomization. | 43.2 Mean partial-credit task score, reported as success rate (%) | 11.2 in Table 1; nearby prose also prints 11.8. Improvement is 32.0 percentage points using the table. It does not mean 43.2% of trials fully completed the named task. e12e13 |
| GR1 generalization to novel environments Table 1's 13 tasks: six seen-behavior and seven novel-behavior tasks; neural-only downstream training and manually captured initial frames. | 28.5, as printed; unresolved aggregate discrepancy. Mean partial-credit task score, reported as success rate (%) | Baseline: 0.0. Reader arithmetic gives approximately 31.9 from the 13 printed task scores, not 28.5. Preserve the reported aggregate without treating its averaging rule as verified. e03e12e13 |
| Real-world low-data augmentation Table 5; GR00T N1 on four GR1, three Franka and two SO-100 tasks. GR1/Franka evaluate ten rollouts per checkpoint; SO-100 uses ten one-minute trials. | GR1 average 46.0; Franka tasks 60/20/30; SO-100 tasks 26/65. Reported task success scores (%) with partial credit in GR1 and SO-100 | Baselines: GR1 37.0; Franka 40/10/20; SO-100 17/25. Table 5 shows gains for GR00T N1; Figure 5 prints GR1 average 46.4. Benefits are not universal: DP hammering drops from 35 to 15. e10e11e13e18 |
| Synthetic-data scaling and label-type diagnostic Figure 4; LAPA versus IDM with three ground-truth budgets and 0–240K neural trajectories. | Low-data LAPA peaks at 28.53 with 24K, then reaches 22.07 at 240K. Mean RoboCasa success (%) | Its no-neural baseline is 17.40. The broad upward trend does not establish monotonic or universal log-linear scaling. Band definitions and a fitted scaling law are absent. e07e08 |
| DreamGen Bench as a downstream proxy Four video models, zero-shot and fine-tuned; 7K generated trajectories per model for neural-only RoboCasa policy training. | Positive association shown in Figure 6; no numerical correlation coefficient supplied there. Mean IF(GPT-4o) and PA versus downstream RoboCasa score | Zero-shot and adapted variants are plotted together. Video scoring is a useful screening signal, not proof of physical execution or independent causal effects of physics alignment. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. More synthetic data improves endpoint averages, with visible departures from monotonic scaling. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at zero neural trajectories: each colored curve begins with a policy trained on ground-truth data alone. Follow that color across increasing synthetic-data budgets, then compare the same color between LAPA and IDM. The low, medium and high regimes correspond to 720, 2,400 and 7,200 real trajectories across 24 RoboCasa tasks. These counts govern IDM and policy training; the video model separately uses 1,200 original demonstrations. The vertical axis averages success over tasks. Read the numeric labels as well as the connecting lines: LAPA's low-data curve rises sharply at 24K but then falls, while IDM's medium-data curve declines between 72K and 240K. e07e08
What it supports. The plots support an overall augmentation benefit for both labeling routes. They also constrain the authors' log-linear interpretation: low-data LAPA moves from 17.40 without neural data to 28.53 at 24K, then 22.07 at 240K. More generated trajectories therefore do not reliably improve every displayed setting.
Where the evidence stops. The shaded regions have no supplied statistical definition, and the paper gives no fitted scaling law here. Changing the ground-truth budget also changes IDM quality and policy supervision, so cross-color comparisons cannot isolate synthetic-data quantity alone.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors acknowledge simple tasks, limited kinematic coverage, no direct comparison with other video-learning methods, manual initial frames and evaluator hallucination. Generating 240K RoboCasa samples reportedly required 54 hours on 1,500 NVIDIA L40 GPUs. e16
Generalization retains embodiment teleoperation and task-specific initial images. Partial-credit criteria weaken literal task-name interpretations: Light Candle awards full credit for approaching the candle; Open Microwave is inconsistently scored for closing it. Confidence intervals for these real-robot tables are absent. e07e12e13e20
Source discrepancies also affect benchmark validation: H.3 mentions AUC-ROC and three models while Tables 6–7 show Pearson statistics across four. Human environment scores differ from Table 2; WAN behavior scores differ between Tables 6–7. Model-level correlation cannot establish per-video evaluator accuracy. e14e15
7.2 Questions for discussion
- Would the gains survive scoring only complete task execution, with more repeated trials? [e12, e13]
- Does better generated motion or more accurate inverse dynamics account for the improvement when data budgets are held fixed? [e08, e19]
8. Reproducibility audit
8.1 Requirements and known gaps
Recover the exact checkpoint, camera grid, data partitions and module schedules. Appendix D supplies WAN settings but leaves a full IDM/policy recipe and overlapping-action reconciliation unspecified. Replay inferred actions in simulation before policy training to distinguish video errors from action-label errors. e03e17e19e21
Resolve data accounting first: Section 3.2 alternates 2,884/2,885 GR1 demonstrations. Franka's generic ten-example claim differs from Appendix F's 11/10/8. SO-100's 10/13 selected videos become 68/44 trimmed trajectories, and reported neural counts differ between Section 3.1 (40/50) and I.3 (30/40). e12e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate generated-motion quality from IDM quality
Reader-proposed, not performed: in a RoboCasa subset with known actions, hold initial scenes, prompts, policy architecture, update count and neural-trajectory count fixed. Cross two video generators with IDMs trained on two real-data budgets; also relabel held-out real videos and compare inferred actions against their ground truth. Measure replay feasibility, action error and downstream success separately. If improving IDM quality rescues weak videos as much as changing generators, the claim that neural-trajectory quality is the dominant bottleneck would need qualification. If generator gains persist with a well-validated fixed IDM, that explanation gains support. e04e07e08e19
Check 2: Rescore transfer using completed tasks and matched starts
Reader-proposed, not performed: compare the pick-and-place-only GR00T N1 baseline with a neural-only policy on Pour Water and Water Flowers, using the same randomized starting scenes and more repeated trials. Preserve the published partial-credit score, but also report actual completed pouring, blinded video judgments and uncertainty across trials. Resolve the task rubric before testing and count the target images supplied. If gains mostly reflect grasping while completed pouring remains rare, the broader new-behavior interpretation is weakened. If full execution improves as well, the transfer conclusion becomes more specific and stronger. e03e12e13
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page; all twelve original figures; all eight tables; method, evaluation, limitations and appendix configuration pages supporting the report. Each of the six final original crops was separately inspected for legibility and clipping. Figure 3's action-index mismatch and undefined k, Figure 4's nonmonotonic curves, Table 1's aggregate mismatch, Figure 5/Table 5 differences, and Table 8's task/criterion mismatch are retained and discussed. Reference-only pages 11–14 were read in the complete text chunks but not rendered. Linked videos and external supplements were not inspected; static sequences cannot independently establish full motion quality.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract; Sections 1–7
- References
- Appendices A–J, including H.1–H.4 and I.1–I.3
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/edition: reviewed arXiv:2505.12705v2, dated 17 June 2025, whose title matches the catalog. All 28 authors substantially agree; the PDF prints Lin Yen-Chen and Loïc Magne. The catalog's CoRL 2025 venue edition was not supplied or compared; equivalence to it is not established.
- All seven supplied text chunks were read. Original PDF images were inspected on pages 1–10 and 15–23; references-only pages 11–14 were read as text.
- Separate supplemental material availability has not been fully verified. Linked videos, project resources and code were not supplied for inspection; experiments were not reproduced.
- Several counts, aggregates, evaluation descriptions and action indices conflict within the supplied PDF; the relevant report entries preserve these differences.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin
The title matches the supplied identity. The artifact is arXiv:2505.12705v2, 17 June 2025. Its 28-author block prints Lin Yen-Chen and Loïc Magne; the catalog reverses the former name and encodes the latter differently. Nine affiliations are printed.
Go to primary source ↓e02PDF p. 2, Figure 2 and Section 1; p. 9, Section 5, Video World Modeling for Robotics
DreamGen separates video-model adaptation, video generation, pseudo-action labeling and policy training. The video model is an offline synthetic-data generator, and the components are deliberately separate.
Go to primary source ↓e03PDF p. 3, Sections 2.1–2.2; p. 16, Appendix C and Figure 10
WAN2.1 is the main video model; LoRA is the default adaptation. Initial frames and language condition generation. Real initial frames and novel prompts are manually supplied. Multiview data place left/right/wrist views in a 2×2 grid with a black bottom-right cell.
Go to primary source ↓e04PDF p. 4, Figure 3(a) and Section 2.3, IDM Actions
The IDM uses a SigLIP-2 vision encoder and diffusion transformer with flow matching, conditions on two frames, excludes language/proprioception, and labels overlapping action chunks. The diagram's output ends at t+H−1, while its input and the prose use t+H; the loop symbol k is not defined here.
Go to primary source ↓e05PDF p. 4, Figure 3(b) and Latent Actions; p. 15, Table 3 and Appendix A
LAPA uses a transformer VQ-VAE trained on robot, simulation and human videos. Labels are continuous pre-quantization embeddings between current and one-second-future frames. Appendix A specifies codebook size 8, sequence length 16, 100K training steps and batch size 1024; Table 3 reports 438.1M frames and 5,721.3 hours.
Go to primary source ↓e06PDF p. 4, Section 2.4 and footnote 4
Policies take image observations and language instructions and predict pseudo-action chunks. Synthetic trajectories have zero-filled state inputs. Real and neural data are sampled 1:1 for co-training; GR00T N1 uses separate action encoders/decoders for the two embodiments. Generalization policies use neural trajectories only.
Go to primary source ↓e07PDF p. 5, Section 3.1, Simulation experiments and footnotes 5–6
RoboCasa evaluation covers 24 tasks with GR00T N1 and left/right/wrist views. The video model uses 1,200 human demonstrations; IDM and policy data regimes are 720, 2.4K and 7.2K trajectories. The paper adopts the original benchmark protocol and leaves zero-ground-truth-embodiment generalization open.
Go to primary source ↓e08PDF p. 5, Figure 4 and caption
Both LAPA and IDM improve endpoint averages as neural data increase to 240K. LAPA low-data scores are 17.40, 19.40, 28.53, 21.60 and 22.07 at 0, 7.2K, 24K, 72K and 240K. IDM medium-data scores decline from 40.90 at 72K to 39.94 at 240K. Shaded bands are not defined in the caption.
Go to primary source ↓e09PDF p. 17, Table 4, header, Average row, OpenSingleDoor row and ONLY NT column
NT denotes 240K neural trajectories. Average scores for 30/100/300 real trajectories per task are 17.44/32.07/49.59, becoming 23.32/39.94/57.61 with NT; ONLY NT scores 20.55. OpenSingleDoor falls from 58.82 to 47.12 in the 300-trajectory comparison, and several ONLY NT tasks score zero.
Go to primary source ↓e10PDF p. 6, Section 3.1 and Figure 5 caption; p. 17, Appendix F
Real augmentation uses GR1, Franka and SO-100. Section 3.1 gives 300 generated trajectories per GR1 task and 100 per Franka task. Appendix F specifies GR1 adaptation on 2,884 pick-and-place trajectories then 10 target examples for hammering/wiping/stacking and 25 for folding; the IDM stays trained on pick-and-place. Franka uses 49,895 DROID examples then 11/10/8 target examples for its video model, without task-specific IDM post-training.
Go to primary source ↓e11PDF p. 18, Table 5, low-data, +Neural Traj. and High Data rows; p. 6, Figure 5
Table 5 prints GR00T N1 GR1 averages 37.0→46.0, versus 46.4 in Figure 5. Franka GR00T N1 pick/place, cube stacking and tool scores change 40/10/20→60/20/30. SO-100 scores change 17/25→26/65. DP hammering declines 35→15. High-data GR00T N1 scores include GR1 average 69.0 and SO-100 tic-tac-toe 40.0.
Go to primary source ↓e12PDF p. 7, Table 1 and Section 3.2
Seen-environment novel-behavior averages are 11.2→43.2 over 14 tasks; 50 generated trajectories per task train the new policies. The 13 novel-environment task entries are 30,10,20,45,35,45,15,55,15,25,55,30,35, with printed average 28.5 and baseline zero. Prose alternates 2,884/2,885 training trajectories and 11.8/11.2 baseline scores. The table's lower block separates six seen-behavior and seven novel-behavior tasks.
Go to primary source ↓e13PDF p. 20, Appendix I.1 and Figure 11; p. 21, Table 8
GR1 evaluations use ten rollouts per checkpoint with target-position randomization. Scores include intermediate progress: Pour Water gives 0.5 for picking up and 1 for pouring. Light Candle's full-credit criterion is approaching the candle. Table 8 lists the specific criteria for 14 seen-environment and 13 novel-environment tasks.
Go to primary source ↓e14PDF p. 8, Table 2, Section 4 and Figure 6; p. 18, Appendices H.1–H.2
DreamGen Bench compares zero-shot/fine-tuned Hunyuan, CogVideoX, WAN2.1 and Cosmos. Training/evaluation counts are RoboCasa 1,200 trajectories/48 frames and GR1 100 trajectories with object/behavior/environment evaluations of 50/47/30 frames. IF has GPT-4o, Qwen and human columns; PA averages VideoCon-Physics and Qwen judgments. Figure 6 uses mean IF(GPT) and PA versus RoboCasa policies trained on 7K generated trajectories per model. Zero-shot IF uses a different, stricter prompt in H.1.
Go to primary source ↓e15PDF p. 19, Appendix H.3, Tables 6–7; p. 8, Table 2
Tables 6–7 report model-level Pearson correlations with human IF; GPT-4o correlations are 0.94/0.93/0.96/1.00. H.3 mentions AUC-ROC and three models although its tables list four. Human GR1 environment values in H.3 differ from Table 2; WAN behavior human IF is 74.5 in Table 6 and 70.2 in Table 7. These are unresolved source inconsistencies.
Go to primary source ↓e16PDF pp. 9–10, Section 7, Limitation
Authors report limited task/kinematic coverage, no direct benchmark against other video-learning methods, costly generation, manually supplied initial frames and hallucinating automatic evaluators. Generating 240K RoboCasa samples reportedly took 54 hours on 1,500 NVIDIA L40 GPUs.
Go to primary source ↓e17PDF p. 16, Appendix D
WAN2.1 fine-tuning uses learning rate 1e−4, LoRA rank 4 and alpha 4. Epochs/batch sizes are RoboCasa 100/32, GR1 75/64, DROID 5/64 and SO-100 200/8. These settings do not specify the full downstream policy or IDM optimization recipe.
Go to primary source ↓e18PDF p. 17, Appendix F; p. 22, Appendices I.2–I.3; p. 6, Section 3.1
SO-100 selects 10/13 videos yielding 68/44 trajectories after trimming. Section 3.1 reports 40/50 neural trajectories, whereas I.3 says 30/40. Franka's generic ten-example description and irrelevant folding exception in I.2 do not resolve Appendix F's 11/10/8 counts. Franka uses ten rollouts per checkpoint; SO-100 uses ten one-minute trials with partial-credit scoring.
Go to primary source ↓e19PDF p. 15, Appendix A and Figure 7; p. 19, Appendix H.4
Replaying inferred actions in a GR1 simulation is proposed as an intermediate diagnostic of video versus IDM quality. Figure 7 shows WAN and CogVideoX generated/replay pairs. The authors attribute most bottlenecks to neural-trajectory quality; no quantitative decomposition is given.
Go to primary source ↓e20PDF p. 21, Table 8, Open Microwave row
The task named Open Microwave is scored using closing motion and closing the microwave. The supplied source does not reconcile this mismatch.
Go to primary source ↓e21PDF p. 4, Section 2.3; p. 15, Appendix A; p. 16, Appendix D
The supplied method describes a sliding window and names training objectives, but does not provide an explicit rule for reconciling overlapping IDM predictions or a complete inference schedule, action horizon and optimizer configuration for all modules.
Go to primary source ↓8.5 Primary sources
DreamGen: Unlocking Generalization in Robot Learning through Video World Models ↗
PDF · 11,323 extracted words
Source fingerprint
e4e2c5c9e664d9331adb49c230053a76cf5bb4c3501341f4d9b372ecc585684c