DreamDojo: A Real-Time Robot World Model from Large-Scale Human Videos
1. Paper overview
In one sentence: Continuous latent actions help transfer human-video experience into a robot simulator whose distilled student streams faster, while separate policy and value models are required to turn predictions into executed plans. e-identitye-latente-posttraine-speede-policye-planninge-limit
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations cover limited objects, environments and intentions. The paper asks whether diverse, mostly action-unlabeled human videos can improve robot future prediction, including unfamiliar interactions and counterfactual motions. e-problem |
| Core mechanism | Source description A 44,711-hour human-video mixture combines 55 hours of In-lab, 829 of EgoDex and 43,827 of crowdsourced DreamDojo-HV. Its scale supports broader interaction exposure; diversity counts include automated estimates. e-data |
| A key reported result | Real-time long-horizon GR-1 simulation: Student: 10.81 FPS; 13.146 / 0.379 / 0.485. FPS ↑; PSNR ↑; SSIM ↑; LPIPS ↓. GR-1 Long Eval, 600 generated frames (one minute); GR-1-adapted 2B teacher/student; single NVIDIA H100. Teacher: 2.72 FPS; 14.086 / 0.442 / 0.412. Approximately fourfold throughput comes with worse aggregate image metrics. Qualitative context recovery is a separate benefit. e-speede-context |
| Reading caution | Source description The authors report failures on slapping/fast waving, optimistic simulated success, no natural multiview support and an unresolved finetuning/knowledge-retention tradeoff. e-limit |
Core contributions
- Source description
A 44,711-hour human-video mixture combines 55 hours of In-lab, 829 of EgoDex and 43,827 of crowdsourced DreamDojo-HV. Its scale supports broader interaction exposure; diversity counts include automated estimates. e-data
Figure 3. The proxy-action bottleneck learns from observed transitions before robot adaptation. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two images at the upper left, labeled with consecutive times. Both feed the latent-action encoder; its output is the compact action code between them. Follow the separate gray route carrying the earlier image directly to the decoder. Together, that image and the code reconstruct the later image below. This matches Section 3.3.2 and Equation (3): the decoder can obtain appearance from the earlier image while the code carries transition information. The right-hand groups are retrieved frame pairs with similar latent actions across different embodiments and contexts. They illustrate the representation's intended transfer behavior, rather than the full diffusion simulator. e-latente-traininge-posttrain
What it supports. The architecture provides proxy action labels from videos without instrumented motion capture. The implemented code has 32 dimensions. Those labels condition human-video pretraining; robot post-training subsequently learns the target action interface. This explains how the paper separates broad interaction exposure from embodiment-specific control semantics.
Where the evidence stops. The encoder observes the later frame during representation learning. This is not evidence of predicting a robot command from the current image alone. Similar retrieved examples also do not prove complete causal disentanglement from scene or camera motion.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations cover limited objects, environments and intentions. The paper asks whether diverse, mostly action-unlabeled human videos can improve robot future prediction, including unfamiliar interactions and counterfactual motions. e-problem
2.2 What this reading follows
DreamDojo asks how a robot simulator can learn from human experience when videos lack robot action labels. Its answer is to infer compact transition codes, pretrain an action-conditioned video model, and then adapt that model to robot controls. This edition reads the supplied Generalist-titled arXiv v1, without asserting a verified revision relationship to the catalog's Real-Time title. The figures below connect the proxy-action mechanism to controlled comparisons, then follow the tradeoff introduced by distillation. The final chart moves beyond video quality to policy ranking and executed planning, where strong ordering coexists with optimistic success predictions and a numerical discrepancy in the paper's prose. e-identitye-latente-posttraine-speede-policye-planninge-limit
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work, neural-world-simulator and latent-action-pretraining labels fit the architecture. Its learned transition predicts video conditioned on supplied actions; planning adds separate policy proposals and a value model. This supports the recorded Not applicable architecture/paradigm/quadrant fields for a joint world-action-policy taxonomy. The proxy encoder has an inverse-dynamics role during training, but the deployed system is not a joint future/action generator. e-backbonee-latente-posttraine-planninge-value
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a shared transition interface before learning robot controls
The information flow begins with recorded human transitions, not with a desired robot action. The latent-action encoder sees both endpoints and compresses their change; its decoder receives the earlier image separately. Reader interpretation: this arrangement reduces the need to encode unchanged appearance in the small action code, but does not guarantee that every nuisance variable disappears. The code is then used as a condition for the much larger video world model. During robot adaptation, measured robot actions replace those proxy labels and the action MLP's first layer is reset. All world-model weights are finetuned. Thus cross-embodiment transfer happens through pretrained video dynamics and a learned conditioning interface, rather than through a demonstrated universal mapping from human codes directly to executable joint commands. e-latente-posttraine-training
5.2 Align actions and temporal supervision with compressed video
The world model operates on temporally compressed video, so action timing must match its prediction units. DreamDojo first expresses robot joint poses relative to the start of each latent frame, then concatenates four successive actions for that latent's condition. This differs from sending an entire trajectory as one global condition. The projected action condition is added to the diffusion-timestep embedding before adaptive normalization in each DiT block. Training also compares changes between adjacent predicted velocities with changes between target velocities, adding that temporal term to flow matching with weight 0.1. The temporal loss concerns the coherence of generated transitions; it is not a reward for task completion. Table 5 connects these choices to a cumulative ablation, with the largest incremental improvement appearing when chunked injection is added. e-backbonee-relativee-latente-losse-design
5.3 Separate a faster simulator from a successful controller
Distillation changes both computation and temporal context. A causal student first imitates teacher ODE trajectories and then trains using its own previous outputs, reducing the mismatch between training context and deployment context. Longer student rollouts expose it to accumulated errors. Reader interpretation: this helps explain why context recovery can improve even while aggregate long-horizon image scores worsen. The reported student runs faster, but deciding what to execute still requires another mechanism. In fruit packing, policy checkpoints supply five proposals, DreamDojo supplies predicted futures, and an external DINOv2-based value model estimates proximity to a subtask boundary. The lowest-value proposal is executed by the robot. The resulting planning gains evaluate that combined system; the correlation experiment separately evaluates checkpoint ordering, and neither result makes simulated success perfectly calibrated. e-distille-distill-confige-speede-contexte-policye-planninge-valuee-limit
5.4 Training and inference
During training
The 700M proxy-action model learns 32-dimensional embeddings for 400k steps, batch 256, with KL weight 10^-6. Human and in-house robot videos are included; its image resolution is 320×240. World-model pretraining uses an empty text prompt, 640×480 clips of 13 frames and In-lab:EgoDex:DreamDojo-HV sampling of 1:2:10. e-training
Both 2B and 14B variants pretrain for 140k steps, batch 1024, on 256 H100 GPUs. Default robot adaptation uses 50k steps, batch 512, on 128 H100s. These defaults differ from the shorter ablation schedules. Distillation uses 64 H100s, 10k warmup iterations and 3k subsequent iterations. e-traininge-distill-confige-actione-mixture
During inference
The teacher uses 35 denoising steps; the student uses four without classifier-free guidance. The student retains a 12-frame context and emits four-frame prediction units. This simulator consumes actions; it does not independently select a robot command. e-distill-confige-speede-backbone
For planning, five policy checkpoints propose actions; distilled DreamDojo predicts their outcomes. A separate frozen-DINOv2 backbone and trained attention head score four-frame clips by normalized time to subtask completion. The lowest-value proposal is executed. PICO control of a virtual G1 on an RTX 5090 is a separate demonstration. e-planninge-valuee-teleop
5.5 Implementation flow
- Extract proxy actions
A VAE encoder compresses two consecutive frames into a continuous action embedding. Its decoder reconstructs the later frame from that embedding and the earlier frame. Reconstruction and KL regularization encourage motion information to pass through the bottleneck; this is a learned proxy, not measured robot control. e-latent
- Condition the video model
Cosmos-Predict2.5 predicts WAN2.2 video latents. Four consecutive actions form each condition chunk. Robot poses are relative to the start of the corresponding latent frame. An MLP adds action embeddings to diffusion-timestep embeddings before adaptive normalization in every DiT block. e-backbonee-relativee-latent
- Adapt and distill
Robot adaptation reinitializes the action MLP's first layer and finetunes all world-model weights. Distillation replaces bidirectional attention with causal attention, first matching teacher trajectories and then training on the student's own generated context. e-posttraine-distill
6. Experiments & results
DreamDojo transfers interaction knowledge from human videos into an action-conditioned robot video simulator. Continuous latent actions provide pretraining labels; robot finetuning supplies executable action semantics. A causal distilled student streams faster, with reduced long-horizon image fidelity. Separate policy and value models turn its predicted futures into planning decisions.
6.1 Read the original evidence
Table 2. Latent actions improve transfer over passive video pretraining under the action-conditioning ablation. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each benchmark independently. Higher PSNR and SSIM are better; lower LPIPS is better. The no-pretraining row starts robot adaptation from Cosmos-Predict2.5. The action-free and latent-action rows differ in whether human pretraining receives transition labels. The pretrained variants use 50k human-video steps followed by 25k GR-1 steps, rather than the final model's longer schedule. Gray rows are special references with labels from capture devices: retargeted GR-1 actions for In-lab and Apple Vision Pro-derived MANO for EgoDex. Their gray styling does not indicate weak scores or a missing experiment. Compare latent actions first with action-free training, then with those instrumented references. e-actione-evale-training
What it supports. On EgoDex, latent actions yield 20.344 PSNR, 0.790 SSIM and 0.214 LPIPS, improving all three scores over action-free pretraining. The 20.474/0.795/0.211 MANO reference remains better. The supported finding is useful label-free transfer approaching the instrumented setting, with a remaining gap.
Where the evidence stops. These are future-video similarity metrics, not executed manipulation success. The MANO reference uses captured poses and should not be interpreted as performance of an off-the-shelf hand estimator on arbitrary videos. No uncertainty intervals accompany the table.
Table 6. Streaming speed and usable context improve while long-horizon image fidelity declines. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the table in two blocks. GR-1 Long Eval compares generated videos over 600 frames, representing one minute. Its three image metrics all favor the teacher. Interactivity measures a different property: the student reaches 10.81 FPS on a single H100, produces shorter prediction units and carries a longer visual context. The prediction-length column refers to generated frames per interaction unit, not diffusion denoising iterations. Section 4.1 separately states that teacher/student denoising uses 35/four steps. The 12-frame student context helps explain the selected occlusion-recovery examples in Appendix D.3, but those qualitative cases should not replace the aggregate quality measurements shown here. e-speede-distill-confige-contexte-teleope-planning
What it supports. The student is about four times faster than the 2.72-FPS teacher, while PSNR drops from 14.086 to 13.146, SSIM from 0.442 to 0.379, and LPIPS worsens from 0.412 to 0.485. Real-time throughput is achieved with a measurable quality tradeoff under this long-horizon protocol.
Where the evidence stops. The paper characterizes the degradation as minor; the table supplies no task-level threshold establishing that judgment. Its single-H100 speed result should not be transferred to the separate RTX 5090 teleoperation demonstration or to complete planning-loop latency.
Figure 5. Policy ordering is strong, and planning improves the shown groups, but the simulator is optimistic. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) places simulated success on the horizontal axis and real success on the vertical axis. Its six points represent checkpoint comparisons over 20 fruit-packing scenes, not six independent task families. Success is graded by the number of fruits bagged, with five defining full completion. Panel (b) is a separate planning experiment over ten scenes. For each group, compare the green planning bar with both the highest gray checkpoint and the red uniform-sampling line. The planner simulates five proposals and uses an external value model to choose an action for real execution. Keep checkpoint ranking, absolute success calibration and planning improvement as separate conclusions. e-policye-planninge-valuee-limit
What it supports. The reported ranking statistics are strong: r=0.995 and MMRV=0.003. Planning reaches 70% versus a 53% best checkpoint in the first group, and 68% versus 63% in the second. These results concern the complete proposal–simulator–value pipeline; the world model alone does not select actions.
Where the evidence stops. The prose calls the first best-checkpoint gain “17%”; the bars imply 17 percentage points. Its second-group “nearly 2×” claim conflicts with 68%/55%, about 1.24×. Use the plotted values. Neither panel reports uncertainty, and ranking agreement alone does not demonstrate calibrated failure prediction.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Latent-action transfer on EgoDex Eval GR-1 with unseen objects/actions; pretrained variants use 50k human-video steps and 25k robot steps. Most undistilled evaluations use 100 generated 49-frame videos. | 20.344 / 0.790 / 0.214 PSNR ↑ (dB), SSIM ↑, LPIPS ↓ | Action-free: 19.924 / 0.783 / 0.222; instrumented MANO: 20.474 / 0.795 / 0.211. Proxy labels improve all three metrics over passive pretraining here, while remaining below the instrumented reference. e-actione-eval |
| Data-mixture scaling on Counterfactual Eval Uniform dataset sampling; 50k pretraining plus 25k GR-1 post-training steps. | In-lab: 20.755; plus EgoDex: 20.797; plus DreamDojo-HV: 20.852. PSNR ↑ (dB) | Cosmos-Predict2.5 without human pretraining: 20.472. PSNR rises across these mixtures; SSIM and LPIPS do not improve universally. Final-model rows use a different training recipe. e-mixturee-training |
| Human preference in edited novel environments 50 background-edited samples, 12 volunteers, 30k robot post-training steps; matching edited ground-truth videos unavailable. | DreamDojo-14B over Cosmos-Predict2.5: physics 73.50%; action following 72.55%. Pairwise preference rate ↑ | DreamDojo-2B over Cosmos: 62.50% and 63.45%. These are subjective video comparisons, not physical task-success rates or fully unseen real-environment trials. e-evale-human |
| Action-conditioning design ablation Cosmos-Predict2.5 finetuned only on GR-1 for 30k steps; sequential additions evaluated on held-out GR-1 and Counterfactual Eval. | Baseline 19.448 → relative 19.482 → chunked 20.783 → temporal 20.980. Counterfactual PSNR ↑ (dB) | GR-1 Val baseline/full: 16.199/17.630. Chunking gives the largest incremental gain in this ordering; the cumulative design does not measure every factor independently. e-design |
| Real-time long-horizon GR-1 simulation GR-1 Long Eval, 600 generated frames (one minute); GR-1-adapted 2B teacher/student; single NVIDIA H100. | Student: 10.81 FPS; 13.146 / 0.379 / 0.485. FPS ↑; PSNR ↑; SSIM ↑; LPIPS ↓ | Teacher: 2.72 FPS; 14.086 / 0.442 / 0.412. Approximately fourfold throughput comes with worse aggregate image metrics. Qualitative context recovery is a separate benefit. e-speede-context |
| AgiBot fruit-packing policy evaluation Six checkpoint points; 20 scenes; about 80 seconds per rollout. Five bagged fruits define 100%; scores average scene-level fruit completion. | r = 0.995; MMRV = 0.003. Pearson correlation ↑; MMRV ↓ | Human-scored DreamDojo rollouts versus corresponding real-world performance. Strong ordering does not establish calibration: simulated success is often too high. e-policye-limit |
| AgiBot fruit-packing model-based planning Ten scenes; two groups of five checkpoint proposals; distilled simulator plus external value selection and robot execution. | Planning: 70% and 68% for the two groups. Reported success rate ↑ | Best checkpoints: 53% and 63%; uniform sampling: 37% and 55%. Figure-derived gains over the best are 17 and 5 percentage points. The prose's nearly-twofold description fits 70/37 approximately, but not 68/55. e-planninge-value |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Broader human-video mixtures raise PSNR consistently in this ablation, while other metrics are mixed. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the horizontal rules to distinguish three comparisons. The top row is the Cosmos-Predict2.5 initialization without the additional human-video pretraining. The middle block adds In-lab, EgoDex and DreamDojo-HV successively, with uniform dataset sampling and the same 50k/25k pretraining/post-training schedule. The final two rows are the final 2B and 14B variants, which use a different pretraining recipe. Across each benchmark, inspect PSNR, SSIM and LPIPS together. The middle rows offer the cleaner data-mixture comparison, although changing a mixture changes its content and diversity as well as the amount of available video. e-mixturee-traininge-eval
What it supports. Counterfactual PSNR rises from 20.755 to 20.797 to 20.852 as datasets are added. The table supports useful transfer from broader experience. However, adding DreamDojo-HV to In-lab plus EgoDex worsens EgoDex SSIM from 0.791 to 0.790 and LPIPS from 0.215 to 0.216.
Where the evidence stops. The caption's claim of consistent improvement should not be read as improvement in every cell. The final-model rows also change training recipe and model capacity, so their gains cannot isolate dataset diversity under the middle block's controls.
Table 5. Aligning action chunks with video latents gives the largest incremental gain in the tested ordering. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the three checkmark columns. Each successive row keeps previous modifications and adds one more: relative joint poses, action chunks aligned to video latents, then temporal consistency loss. The unmarked row is the starting Cosmos-Predict2.5 architecture. All variants are finetuned only on GR-1 for 30k steps, which makes this a separate experiment from human-video transfer. GR-1 Val measures held-out expert demonstrations; Counterfactual Eval tests atypical actions. Trace the Counterfactual PSNR column downward and then check whether SSIM and LPIPS agree. Finally connect the additions to Sections 3.3.1–3.3.2: coordinate normalization, temporal alignment and transition supervision play different roles. e-designe-relativee-loss
What it supports. Counterfactual PSNR changes only slightly with relative actions alone, from 19.448 to 19.482, then reaches 20.783 after chunking and 20.980 after temporal loss. The final row also improves the other Counterfactual metrics. This supports the combined design and a particularly strong incremental contribution from chunking.
Where the evidence stops. This is a cumulative ablation, not a full factorial study. The chunking gain is conditional on relative actions already being enabled; the table cannot determine its standalone effect or exclude interactions among the components.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report failures on slapping/fast waving, optimistic simulated success, no natural multiview support and an unresolved finetuning/knowledge-retention tradeoff. e-limit
Video metrics and preference votes offer indirect evidence of dynamics. Edited backgrounds lack matching ground-truth videos, and reported comparisons provide no confidence intervals. Table 7 also contradicts an all-metric reading of improved generalization: Counterfactual SSIM and LPIPS worsen after human pretraining. e-evale-actione-humane-student-transfer
7.2 Questions for discussion
- Would proxy actions retain their transfer advantage under equal data, compute and action-shuffled controls?
- Can policy ranking remain accurate when evaluated on failure-rich scenes and policies outside the checkpoint family?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the pretrained backbone/tokenizer, human mixture, target robot trajectories, action alignment and evaluation splits. In-house data are essential to the reported recipe; the supplied paper does not establish complete dataset access or a runnable release. e-backbonee-datae-traininge-eval
Resolve supervision-window ambiguity before distilling: Section 3.3.4 describes a random teacher-length window, whereas Section 4.1 specifies the last 13 frames of a 13–49-frame rollout. Planning also needs subtask annotations and the Appendix D.6 value-aggregation rule; optimizer/schedule details for that value head are not given there. e-distille-distill-confige-value
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the proxy label convey action information beyond extra video exposure?
Reader-proposed check, not an experiment performed here: start from the same Cosmos checkpoint and compare no human pretraining, action-free pretraining, latent-action pretraining and latent-action pretraining with codes shuffled between clips. Match the human-video sampling, training updates and GR-1 adaptation budget for the three pretrained arms; retain the unshuffled encoder and code distribution in the shuffled control. Evaluate fixed held-out initial frames and counterfactual action sequences, reporting PSNR/SSIM/LPIPS alongside blinded action-following judgments. Repeat training seeds and report paired uncertainty. If correctly aligned codes do not outperform shuffled codes on action following, the claim that the proxy transfers actionable transition information is weakened even if both improve visual appearance. e-actione-mixturee-evale-latent
Check 2: Can checkpoint ranking survive deliberate failure-rich evaluation?
Reader-proposed check, not a reproduced result: hold a fruit-packing policy family and initial scenes fixed, then add predefined missed grasps, near-contact motions and awkward object placements to the evaluation set. Run the same action sequences in the robot and simulator, scoring fruit completion under the paper's five-fruit rule. Report both Pearson/MMRV and signed simulated-minus-real success error, with scene-level uncertainty. For planning, hold the five proposals and value model fixed and compare value-selected proposals against uniform selection and each checkpoint. Strong rank correlation combined with a large positive calibration error would reproduce the authors' optimism limitation; a loss of ranking on failure-rich cases would challenge the simulator's usefulness for selecting policies beyond the original scenes. e-policye-planninge-valuee-limite-eval
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all main-text scientific pages, all seven tables, Figures 1–14, and Appendix D's value-model and context details on PDF pp. 1–24. Every final crop was separately opened and checked for readable headers, labels and values. Figure 3's encoder/decoder arrows and earlier-frame bypass agree with Section 3.3.2; Table 5's checkmarks were checked against Section 4.5. Figure 5's plotted planning values are retained with an explicit prose discrepancy. The random-window versus last-window distillation descriptions are disclosed in the base report. Reference-only pages 25–33 were read as text. External videos, code, datasets and separate supplements were outside the supplied visual pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2
- Sections 3.1–3.2
- Sections 3.3.1–3.3.4: architecture, pretraining, robot adaptation and distillation
- Sections 4.1–4.7: setup, action/data/design ablations, generalization, distillation and applications
- Section 5, including Limitations
- Appendices A–C
- Appendices D.1–D.6
- References, PDF pp. 25–33
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity notes: this reading verifies the explicitly supplied primary artifact, DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos, arXiv:2602.06949v1 (margin: 6 February 2026; title-page header: 2026-2-9). The catalog title is DreamDojo: A Real-Time Robot World Model from Large-Scale Human Videos. No primary revision chain connecting these titles was supplied; this report does not assert their edition equivalence or verify the catalog's ICML 2026 venue claim.
- The title-page author list agrees substantially with the catalog author string, but the catalog's shorter BibTeX list omits several observed credits. Metadata preserves all 30 title-page authors; no added-author revision history is inferred.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition omission was addressed by visually inspecting PDF pp. 1–24 and every final crop.
- Separate supplemental material availability has not been fully verified.
- No external project pages, source code, datasets or video supplements were inspected; no experiments were reproduced. References were read as part of the supplied text, not independently verified.
- Visual review covers all scientific pages (1–24); reference-only pages 25–33 were read as text.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author/affiliation block and arXiv margin
The title is DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos. Thirty authors and nine institutional labels are printed. The margin identifies arXiv:2602.06949v1, 6 February 2026; the page header separately reads 2026-2-9.
Go to primary source ↓e-problemPDF p. 2, Section 1, first three paragraphs
The motivating gaps are limited robot-data coverage, predominantly expert demonstrations, and scarce action annotations in broader human video.
Go to primary source ↓e-backbonePDF p. 3, Figure 1 and Section 2, Eqs. (1)–(2); p. 4, Section 3.1
DreamDojo predicts action-conditioned video transitions using Cosmos-Predict2.5 and the WAN2.2 video tokenizer, followed by robot post-training and optional autoregressive distillation.
Go to primary source ↓e-dataPDF pp. 4–5, Section 3.2; p. 5, Table 1, human dataset rows and dagger note
In-lab, EgoDex and DreamDojo-HV contain 55, 829 and 43,827 hours respectively; the mixture totals 44,711 hours. DreamDojo-HV is crowdsourced in-house. The 6,015 skill count is GPT-estimated. Table 1's scene column and the prose's unique-scene description use different reported counts without reconciling their definitions.
Go to primary source ↓e-relativePDF p. 5, Section 3.3.1, final paragraph; p. 6, continuation before Section 3.3.2
Joint poses are made relative to the start of each latent frame. Four consecutive actions are concatenated and injected into their corresponding video latent, matching the tokenizer's temporal compression factor.
Go to primary source ↓e-latentPDF p. 6, Figure 3 and caption; p. 7, Section 3.3.2, Eq. (3) and following paragraph
An encoder sees consecutive frames and emits a compact continuous action embedding. A decoder receives that embedding and the earlier frame to reconstruct the later frame. Reconstruction and KL regularization train the bottleneck. An action MLP with a zero-initialized final layer adds projected actions to timestep embeddings before adaptive layer normalization.
Go to primary source ↓e-lossPDF p. 3, Eq. (2); p. 7, Eqs. (4)–(5) and definitions
The world-model objective combines flow matching with squared differences between predicted and target velocity changes across adjacent video latents; the temporal coefficient is 0.1.
Go to primary source ↓e-posttrainPDF p. 7, Section 3.3.3; p. 9, Section 4.1, final paragraph
Robot adaptation replaces latent proxy labels with robot actions, reinitializes the action MLP's first layer, and updates all pretrained world-model weights. The setup uses 13-frame clips and 12 relative actions.
Go to primary source ↓e-distillPDF p. 8, Section 3.3.4, Eqs. (6)–(8) and final paragraph
A causal student warms up on teacher ODE trajectories, then conditions on its own generated latents under distribution-matching supervision. Longer student rollouts are supervised through a teacher-length window; this description says the window is randomly selected.
Go to primary source ↓e-trainingPDF p. 9, Section 4.1, Training and inference
The 700M latent-action model uses 32-dimensional actions, 24 encoder and 24 decoder blocks, 400k steps, batch 256, beta 10^-6 and AdamW. World models use 640×480, 13-frame clips, an empty prompt, a 1:2:10 dataset ratio, 140k pretraining steps, batch 1024 and 256 H100 GPUs. Default robot post-training uses 50k steps, batch 512 and 128 H100 GPUs.
Go to primary source ↓e-distill-configPDF p. 10, Section 4.1, first two paragraphs below Table 2
The student uses a 12-frame context. Warmup uses 10k ODE trajectories and 10k iterations; subsequent distillation uses 3k iterations, 13–49 generated frames and loss on the last 13. Both stages use 64 H100 GPUs, with batches 256 and 64. Teacher/student inference uses 35/4 denoising steps, without classifier-free guidance.
Go to primary source ↓e-evalPDF p. 10, Section 4.1, Benchmark construction and Evaluation protocol; p. 11, opening continuation; p. 17, Appendix C and Figure 7
Robot evaluation recreates interactions from human datasets and includes counterfactual actions. Most undistilled evaluations use 100 videos of 49 frames. Two novel-background sets have 25 samples each, edited with Gemini 2.5 Flash Image; matching ground-truth videos are unavailable. Twelve volunteers judge physics and action following, with randomized left/right order and action-following ties permitted.
Go to primary source ↓e-actionPDF p. 10, Table 2, both subtables; p. 11, Section 4.2
Compared pretrained variants use 50k human-video pretraining steps and 25k GR-1 post-training steps. EgoDex latent-action scores are 20.344/0.790/0.214 for PSNR/SSIM/LPIPS; action-free scores are 19.924/0.783/0.222 and device-derived MANO scores are 20.474/0.795/0.211.
Go to primary source ↓e-mixturePDF p. 11, Table 3, Data Mixture block and final-model rows; Section 4.3
The mixture ablation uses uniform dataset sampling with 50k pretraining and 25k post-training steps. Counterfactual PSNR increases from 20.755 to 20.797 to 20.852 as datasets are added. Other metrics are not uniformly monotonic. Final 2B/14B models have a different recipe.
Go to primary source ↓e-humanPDF p. 12, Table 4 and Section 4.4
With 30k post-training steps, 12 volunteers assess 50 edited-background samples. DreamDojo-14B wins over Cosmos-Predict2.5 at 73.50% for physics correctness and 72.55% for action following.
Go to primary source ↓e-designPDF p. 12, Table 5 and Section 4.5; p. 19, Figure 9
Sequentially adding relative actions, chunked injection and temporal loss to Cosmos-Predict2.5 trained only on GR-1 for 30k steps improves the final Counterfactual PSNR to 20.980 from 19.448. Relative-only, relative-plus-chunked and all-three scores are 19.482, 20.783 and 20.980. GR-1 Val rises from 16.199 to 17.630.
Go to primary source ↓e-speedPDF p. 12, Section 4.6; p. 13, Table 6; p. 21, Figure 10
GR-1 Long Eval generates 600 frames, representing one minute. On one H100, the teacher/student run at 2.72/10.81 FPS; PSNR is 14.086/13.146, SSIM 0.442/0.379 and LPIPS 0.412/0.485. Prediction/context lengths are 12/1 for the teacher and 4/12 for the student.
Go to primary source ↓e-student-transferPDF p. 13, Table 7, Counterfactual Eval columns
Human pretraining before distillation raises Counterfactual PSNR from 19.782 to 19.891 but lowers SSIM from 0.758 to 0.746 and raises LPIPS from 0.232 to 0.234.
Go to primary source ↓e-policyPDF p. 13, Section 4.7, Policy evaluation; p. 14, Figure 5a
A single-view, state-free GR00T N1.5 variant and DreamDojo-2B are trained on AgiBot fruit packing. Twenty scenes have about 80-second rollouts; successfully bagging five fruits defines 100%. Figure 5a plots six checkpoint points and reports Pearson r=0.995 and MMRV=0.003.
Go to primary source ↓e-planningPDF pp. 13–14, Section 4.7, Model-based planning; p. 14, Figure 5b
Planning evaluates ten AgiBot scenes with five checkpoint proposals, distilled DreamDojo-2B futures and an external value model. The first group has 70% planning, 53% best checkpoint and 37% uniform sampling. The second has 68%, 63% and 55%. The prose describes the first best-checkpoint gain as 17% and calls the second uniform-baseline comparison nearly 2×.
Go to primary source ↓e-teleopPDF p. 14, Figure 6 and Section 4.7, Live teleoperation
The authors demonstrate PICO VR control of a virtual G1 using DreamDojo-2B on a desktop RTX 5090. This demonstration is distinct from the single-H100 FPS benchmark.
Go to primary source ↓e-limitPDF p. 15, Section 5, Limitations
The authors acknowledge uncommon-action failures, overestimated simulated success, missing nuanced failures, no natural multiview support, and insufficient study of knowledge retention during finetuning.
Go to primary source ↓e-valuePDF p. 20, Appendix D.6; p. 24, Figure 14
The external value model freezes DINOv2, processes four-frame clips with a trainable global-attention head, and predicts normalized time to a subtask boundary. Stride-one scores are averaged until a dip preceding an increase; the lowest-value proposal is executed. Figure 14 shows two illustrative prediction traces.
Go to primary source ↓e-contextPDF p. 18, Appendix D.3; p. 22, Figure 11
Selected teacher/student examples illustrate better object persistence after occlusion and camera changes with the student's temporal context; these are qualitative comparisons.
Go to primary source ↓e-curvesPDF p. 20, Appendix D.5; p. 24, Figure 13
Post-training PSNR curves compare no pretraining, action-free pretraining, latent actions and instrumented action labels on In-lab and EgoDex. The displayed interval ends at 25k steps.
Go to primary source ↓8.5 Primary sources
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos ↗
PDF · 13,466 extracted words
Source fingerprint
c8f02e22c8fbe98e0b50215e10e1f6baf2f589b1987e5c6dc07175bfe6fe8587