DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
1. Paper overview
In one sentence: Frozen spatial features let DINO-WM learn action-conditioned dynamics and plan toward goal images, but coverage of offline actions and the cost of test-time search remain limiting factors. e-identitye-probleme-encodere-dynamicse-planninge-datae-planning-results
| At a glance | What to know |
|---|---|
| Research problem | Source description Can an offline visual dynamics model solve new image-specified goals without training a separate policy or reward model for each goal? The paper replaces task-specific representation learning with pretrained spatial features, while still requiring environment trajectories containing actions and adequate state-action coverage. e-probleme-limitations |
| Core mechanism | Source description A frozen perceptual encoder supplies spatial patch targets for an action-conditioned transformer, decoupling dynamics learning from pixel reconstruction. e-encodere-dynamicse-loss |
| A key reported result | PushT visual goal reaching: DINO-WM 0.90 Success rate ↑. 50 sampled initial/goal pairs; feasible target within 25 steps; both pusher and block must match the target; offline-trained models with MPC. IRIS 0.32; DreamerV3 0.30; TD-MPC2 0.00. A 58-percentage-point advantage over the strongest Table 1 comparator. These baselines were retrained without rewards, so this is not their standard online-RL evaluation. e-protocole-planning-resultse-data |
| Reading caution | Source description The authors require sufficient offline state-action coverage and ground-truth action labels; actionless internet video does not directly satisfy these assumptions. Hierarchical planning is proposed future work. e-limitations |
Core contributions
- Source description
A frozen perceptual encoder supplies spatial patch targets for an action-conditioned transformer, decoupling dynamics learning from pixel reconstruction. e-encodere-dynamicse-loss
- Source description
The same latent distance drives test-time trajectory optimization; an optional decoder makes predictions visible without participating in control. e-decodere-planning
- Source description
Six control suites, three configuration-generalization suites and targeted ablations test where this representation supports planning. e-protocole-generalizatione-mask
Figure 2. Prediction runs through latent states; test-time search changes the candidate actions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the black arrows from left to right. The observation history is encoded into patch grids, which enter the first dynamics block, pθ. Each candidate action enters its corresponding transition, producing a predicted grid for the next step. On the right, the goal image goes through DINOv2 separately, supplying zg for comparison with the terminal prediction. The orange arrows mark which actions are optimized at test time; they do not establish that the optimizer uses gradients. Section 3.2 and Appendix A.5.1 specify CEM for the principal MPC procedure. The blue region contains the latent rollout, with no pixel decoder in the planning path. e-encodere-dynamicse-planninge-decoder
What it supports. The architecture supports a forward-model interpretation: candidate actions cause predicted changes, and goal matching selects among them. It does not show a network directly outputting an action from a goal. The decoder-free planning route also explains why later image reconstructions are interpretability aids rather than necessary control signals.
Where the evidence stops. The arrows agree with the caption and planning objective. The diagram abbreviates recurrent history handling and does not draw the causal attention mask. Its orange callout should not be read as evidence of gradient-descent planning.
2. Motivation
2.1 The problem and the proposed response
Can an offline visual dynamics model solve new image-specified goals without training a separate policy or reward model for each goal? The paper replaces task-specific representation learning with pretrained spatial features, while still requiring environment trajectories containing actions and adequate state-action coverage. e-probleme-limitations
2.2 What this reading follows
Imagine giving a controller a picture of where a block should end up, instead of a reward function or a demonstration of how to move it. DINO-WM encodes that picture and the current scene with frozen DINOv2 patch features. A learned transition model predicts what candidate actions would do to those features; a planner searches for a sequence whose predicted endpoint resembles the goal. The useful distinction is between learning consequences during training and choosing behavior during deployment. This edition follows that separation through the architecture, executed simulation results, representation and attention ablations, and decoded rollout examples, while preserving the reviewed arXiv revision's data and reporting caveats. e-identitye-probleme-encodere-dynamicse-planninge-datae-planning-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-world-model and visual-representation categories fit. A frozen encoder plus learned forward dynamics and an external action optimizer supports the catalog's Not applicable quadrant: this is neither joint future/action generation nor inverse dynamics. The architecture evidence does not justify a One Model world-action label. e-encodere-dynamicse-planning
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the perceptual target fixed while learning consequences
Training begins with observation–action trajectories rather than desired policies. The frozen DINOv2 encoder turns every frame into a spatial grid of features, so the transition learner has a stable target for the next frame. Its action MLP attaches the same encoded action to every patch, letting the transformer relate a control input to scene-wide consequences. Causal attention supplies temporal context without exposing future observations. Teacher forcing makes training efficient by conditioning on observed history, while the loss compares predicted features with the encoded next observation. An optional decoder learns to turn features back into pixels through its own reconstruction objective. This separation is operational: the planning objective never needs those reconstructed pixels, and the decoder's training does not define the primary dynamics target. e-encodere-dynamicse-losse-decoder
5.2 Convert a goal picture into an optimization problem
At deployment, encode the current scene and the requested goal using the same frozen observation model. Sample possible action sequences and roll each through the transition model. CEM scores the predicted terminal feature grid against the goal grid, selects low-cost sequences, and updates its sampling distribution. MPC then executes only a prefix before receiving a fresh observation and searching again. The goal changes the behavior selected at test time; it does not require fitting a new reward model or inverse model. Appendix Table 8 separates the benefit of feedback from open-loop action optimization: its CEM and GD rows execute whole plans, while MPC replans with CEM. Calling all three simply different optimizers would conceal that change in feedback. The resulting action execution is evaluated in the supplied simulation suites. e-probleme-planninge-optimizere-data
5.3 Read zero-shot as new goals under learned dynamics
Reader interpretation: the strongest supported meaning of zero-shot is planning for a supplied goal without learning a new goal-specific policy. It does not mean learning dynamics without environment data. The appendix's PushT data construction even replays expert trajectories with noise, qualifying the abstract's demonstration-free language. The generalization suites strengthen the claim by withholding wall/door positions or object shapes and changing particle count, yet these remain designed shifts within known task families. The data-scaling table also shows planning success rising as more PushT trajectories become available. Together, these observations favor a conditional conclusion: frozen spatial features can make offline dynamics useful for new goals when the trajectories cover relevant interactions. Neither good decoded pictures nor a single high success rate removes the coverage requirement or establishes physical deployment. e-identitye-probleme-datae-generalizatione-scalinge-limitationse-image-metrics
5.4 Training and inference
During training
Use teacher forcing on trajectory segments of length H + 1, supervising H predicted frames against frozen observation features. This trains the transition and action encoders with latent consistency, without rewards, termination labels or a required decoder. e-dynamicse-loss
Reported settings are AdamW, 100 epochs, batch size 32, predictor learning rate 5 × 10⁻⁵, action-encoder rate 5 × 10⁻⁴ and action embedding dimension 10. The predictor has six layers, 16 heads, MLP dimension 2048 and approximately 19M parameters. History and frameskip vary by environment. e-implementation
Training is environment-specific. Appendix A.1 describes random data for several suites, but PushT uses 18,500 samples obtained by replaying released expert trajectories with noise. Thus the abstract's blanket demonstration-free wording does not fully describe data provenance. e-datae-identity
During inference
CEM samples Gaussian action sequences, rolls them forward, retains low-cost elites and updates the distribution. MPC executes a prefix, observes the resulting environment state and replans. A separate optional decoder does not influence this loop. e-planninge-decoder
Appendix Table 8 distinguishes open-loop CEM and gradient descent from MPC with CEM. On PushT their reported success rates are 0.86, 0.28 and 0.90, respectively; do not attribute the entire MPC advantage to optimizer choice. e-optimizer
5.5 Implementation flow
- Encode a spatial state
Keep DINOv2 frozen during training and inference. Each image becomes N patch vectors of dimension E, rather than one global CLS vector. Appendix A.9 specifies a 14 × 14 grid of 384-dimensional features from images resized to 196 × 196. e-encodere-implementation
- Predict action-conditioned changes
A ViT without its image-tokenization layer becomes a decoder-only transition transformer. An MLP embeds each action and concatenates it to every patch; available proprioception is concatenated similarly. Frame-level causal attention uses earlier frames to predict the next complete patch grid, without autoregressing over target-frame patches. e-dynamics
- Score candidate futures
Roll out the learned dynamics under candidate actions and compare the terminal predicted embedding with the encoded goal. The world model predicts consequences; CEM selects actions. There is no learned inverse-dynamics action extractor. e-planninge-dynamics
6. Experiments & results
DINO-WM learns how actions change frozen DINOv2 patch features, then searches for actions that bring predicted features close to a goal image. Its strongest evidence concerns simulated manipulation under an offline, reward-free planning protocol. Spatial features and causal prediction matter, but data provenance, appendix discrepancies and planning latency constrain the zero-shot claim.
6.1 Read the original evidence
Table 1. Manipulation improves under the shared offline planning protocol, while navigation remains competitive. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the metric row: the first four columns contain success-rate fractions, where larger is better; the final two contain Chamfer distance, where smaller is better. Compare methods down one column before looking across tasks. Section 4.3 uses 50 initial/goal cases for the success-rate tasks and 10 cases each for Rope and Granular. PushT requires both the pusher and the block to match a feasible target configuration. The comparison is specifically offline world-model training followed by MPC: Section 4.2 removes reward and task information from the IRIS, DreamerV3 and TD-MPC2 training setups. e-protocole-planning-resultse-datae-mask
What it supports. DINO-WM reaches 0.90 PushT success versus 0.32 for IRIS and 0.30 for DreamerV3. It also improves Reach to 0.92 and achieves the lowest Rope/Granular distances, 0.41/0.26. Conversely, DreamerV3 scores 1.00 on both Maze and Wall, above DINO-WM's 0.98 and 0.96; the improvement is task-dependent.
Where the evidence stops. These are executed simulated-control outcomes under modified baseline protocols, without reported uncertainty intervals. Reward-free TD-MPC2 performance does not measure its usual reward-supervised capability. Appendix PushT ablations report 0.92, which should remain separate from this table's 0.90.
Table 3. Transfer to held-out configurations is measurable, but new object shapes remain difficult. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column with its separate distribution shift in mind. WallRandom excludes test wall and door positions from training. PushObj trains on four shapes and tests on two unseen shapes, with both object and pusher required to reach their target locations. GranularRandom changes the particle count while reusing the model trained with a fixed amount of material; the main text says testing uses fewer than half the training particles. Success rate applies to the first two columns, whereas the third reports Chamfer distance. Figure 5 and Appendix A.2 define these splits, so the table should be interpreted as transfer within specified environment families. e-generalizatione-data
What it supports. DINO-WM has the best reported entry in each column: WallRandom 0.82 versus DreamerV3's 0.76, PushObj 0.34 versus the best alternative 0.18, and GranularRandom distance 0.63 versus IRIS's 0.86. The absolute PushObj score is still modest, showing that relative gains coexist with frequent failures on novel shapes.
Where the evidence stops. The table provides neither per-suite trial counts nor uncertainty intervals, and these structured shifts do not establish broad real-world transfer. Section 4.5 refers to Table 5 in prose, but the displayed generalization results are Table 3.
Figure 4. Decoded rollouts expose prediction drift before asking whether a plan succeeds. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each horizontal row as time advancing from left to right. In the upper PushT group and lower Granular group, compare a model row with the GT row at the bottom of that group. Each model receives an initial frame and an action sequence, then predicts subsequent observations through its learned dynamics and decoder. This is an open-loop prediction comparison, not a sequence of newly observed states fed back into the model. In PushT, track the gray block and blue pusher; Appendix A.1 specifies that the green T is a static reference anchor rather than the task's goal marker. e-rolloutse-image-metricse-planning-resultse-data
What it supports. In these displayed examples, DINO-WM's block orientation, pusher position and particle distribution track the corresponding ground-truth sequence more closely than the other shown predictions. That visual reading supports the paper's case for informative latent dynamics. The separate LPIPS/SSIM tables quantify decoded-image similarity; Table 1 supplies the distinct evidence about executed planning.
Where the evidence stops. Selected rollouts cannot establish physical correctness across a dataset, and decoder quality influences their appearance. The caption's claim of visual indistinguishability is stronger than this qualitative inspection alone warrants. A matching green anchor does not establish PushT goal completion.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PushT visual goal reaching 50 sampled initial/goal pairs; feasible target within 25 steps; both pusher and block must match the target; offline-trained models with MPC. | DINO-WM 0.90 Success rate ↑ | IRIS 0.32; DreamerV3 0.30; TD-MPC2 0.00. A 58-percentage-point advantage over the strongest Table 1 comparator. These baselines were retrained without rewards, so this is not their standard online-RL evaluation. e-protocole-planning-resultse-data |
| Navigation, reaching and deformable control Table 1: 50 cases for Maze, Wall and Reach; 10 cases each for Rope and Granular. | Maze 0.98; Wall 0.96; Reach 0.92; Rope CD 0.41; Granular CD 0.26. Success rate ↑; Chamfer distance ↓ | DreamerV3 reaches 1.00, 1.00 and 0.64 on the first three; IRIS has CD 1.11 and 0.37. Manipulation improves, but navigation does not exceed DreamerV3. Chamfer distance is not a success percentage; its units and uncertainty are not reported. e-protocole-planning-results |
| Generalization to unseen configurations Wall/door positions excluded from training; two held-out object shapes after four training shapes; reduced particle count with the fixed-count Granular model. | WallRandom 0.82; PushObj 0.34; GranularRandom CD 0.63. Success rate ↑; Chamfer distance ↓ | Best listed alternatives: DreamerV3 0.76; DreamerV3/DINO CLS 0.18; IRIS CD 0.86. Restricted within-family transfer is supported. The low absolute PushObj success leaves substantial difficulty; these are not unseen real-world environments. e-generalization |
| Spatial representation and causal-attention ablations Table 2 encoder substitution; separate Table 6 PushT history/mask experiment. | PushT patch features 0.90 versus DINO CLS 0.44. At history h = 3, causal mask 0.92 versus no mask 0.08. Success rate ↑ | At h = 1, masked and unmasked models both score 0.76. Patch structure helps this planning setup, and causality prevents access to future inputs during teacher forcing. Table 6's 0.92 is retained separately from Table 1's 0.90. e-encoder-resultse-maske-planning-results |
| Decoded future prediction quality Predicted future states decoded to images; Tables 4 and 9. | PushT LPIPS 0.007 and SSIM 0.985; Granular LPIPS 0.035 and SSIM 0.940. LPIPS ↓; SSIM ↑ | DINO CLS: 0.039/0.973 on PushT and 0.086/0.912 on Granular. These are perceptual reconstruction metrics, not action-execution success or a complete test of physical fidelity. e-image-metrics |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Keeping spatial patches matters most when the controller must resolve detailed interactions. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row changes the observation representation supplied to a learned world model. R3M and ResNet are pretrained ResNet-18 encoders; DINO CLS uses a global vector from the DINOv2 model, while DINOPatch keeps its spatial patch embeddings. Read Maze first as a relatively easy control comparison: every row scores at least 0.94. Then move to Wall, Reach and PushT, where global-vector models separate sharply from the patch representation. Finally, inspect Rope and Granular using the reversed metric direction, since lower Chamfer distance means a closer final configuration. The paper interprets this pattern as evidence that spatial information matters for precise planning. e-encoder-resultse-encodere-dynamicse-protocol
What it supports. On PushT, DINO patch features score 0.90 against 0.44 for DINO CLS; on Wall, the pair scores 0.96 against 0.58. The corresponding Granular distances are 0.26 and 0.79. These comparisons make a more specific case for retaining spatial structure than a claim that any pretrained encoder is sufficient.
Where the evidence stops. The comparison changes the representation and its token structure; it is not a fully matched test of spatial information alone. R3M and ResNet also differ in pretraining. The DINO CLS comparison narrows, but does not remove, those interpretation limits.
Table 6. Longer history helps only when training respects the information available at prediction time. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Columns increase the number of input observations, h. Rows switch the frame-level causal mask off or on; the displayed numbers are PushT planning success rates. Appendix A.4.2 defines the training input as a sequence ending at the current observation and the output as that sequence shifted one step forward. Without masking, some output targets can attend to future frames already present elsewhere in the input. This shortcut is unavailable during an autoregressive test rollout. At h = 1 there is no such cross-frame opportunity, so the two versions are equivalent. Read the divergence at larger h in that context. e-maske-dynamicse-planning-results
What it supports. Both variants start at 0.76 for one observation. At three observations, the masked model reaches 0.92 while the unmasked model falls to 0.08. The pattern supports the importance of preventing temporal leakage when training with shifted sequences; simply providing more observations does not ensure a better usable dynamics model.
Where the evidence stops. This tests masked versus unmasked training, not frame-level versus token-level causal autoregression. No repeated-seed uncertainty is shown. The 0.92 endpoint belongs to this appendix experiment and differs from the main PushT planning table.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors require sufficient offline state-action coverage and ground-truth action labels; actionless internet video does not directly satisfy these assumptions. Hierarchical planning is proposed future work. e-limitations
All evaluated environments are simulated. Tables lack confidence intervals and repeated-training-seed summaries. Reward removal particularly disadvantages TD-MPC2's representation objective, narrowing the comparison's interpretation. e-datae-protocole-planning-results
Several source discrepancies remain unresolved: PushT success is 0.90 in Table 1 but 0.92 in appendix ablations; Table 5 reports LPIPS 0.005 versus 0.007 in Table 4; Rope/Granular trajectory length is 20 in A.1 but 5 in Table 11. Figure 1 labels DreamerV3's Granular CD 1.04, versus 1.05 in Table 1. e-planning-resultse-scalinge-maske-image-metricse-datae-implementatione-overview
7.2 Questions for discussion
- How much of the PushT advantage survives matched state-action coverage from fully random trajectories, given its expert-replay training provenance?
- Does lower terminal feature error rank action sequences reliably when object appearance changes but dynamics remain fixed?
8. Reproducibility audit
8.1 Requirements and known gaps
A PushT reproduction needs the noisy expert-replay data construction, H = 3 and frameskip 5, the frozen DINOv2 features, and the training settings above. Table 12 lists image size 224 while A.9 specifies resizing to 196 for encoding; the exact preprocessing stages require clarification. e-datae-implementation
Resolve success thresholds, dataset split construction, noise levels, planning horizon, elite count, executed-prefix length and random seeds before comparing scores. The supplied description does not fully specify these. Training hardware count/time and software versions are also absent from the implementation section. e-datae-protocole-planninge-implementation
Table 10 reports A6000 timing: 0.014 s for batch-32 inference, 3.0 s for batch-1 simulation, and 53.0 s for CEM planning with 100 samples over 10 iterations. Different batch sizes preclude a direct per-case speedup claim, and fast prediction alone does not establish real-time control. e-timing
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test temporal leakage independently of apparent prediction quality
Reader-proposed check, not run: reproduce the PushT mask-by-history comparison with histories of one, two and three observations. Hold the data split, frozen encoder, predictor size, training updates and CEM/MPC evaluation budget fixed; repeat across documented seeds. Measure teacher-forced error, open-loop feature error at matched horizons, and executed success on identical initial/goal pairs. At history one, masked and unmasked models are the equivalence control. At longer histories, the leakage explanation predicts that the unmasked model can look good under teacher forcing while degrading in rollout and planning. If that divergence disappears under controlled training or the history-one control fails systematically, inspect sequence alignment and masking before attributing the result to temporal reasoning. e-losse-maske-implementatione-protocole-planning
Check 2: Separate the value of patch features from expert-replay coverage
Reader-proposed check, not run: compare DINO patch and DINO CLS world models on equal-size PushT datasets generated by noisy expert replay and by fully random actions. Keep the same held-out feasible goals, action/observation preprocessing, optimization budget and predictor training schedule; document unavoidable token-count or capacity differences. Measure interaction coverage, open-loop error and MPC success with repeated seeds. Include a coverage-matched subset comparison where feasible, rather than equating trajectory count with contact diversity. If the patch advantage persists across both data sources at comparable coverage, it strengthens the spatial-representation explanation. If success collapses mainly when expert replay is removed, the result would qualify the method's practical data requirements. Report the replay noise and split construction explicitly because the source leaves them incomplete. e-encoder-resultse-datae-scalinge-implementatione-protocole-limitations
8.3 Reading coverage
Visual audit: Original PDF pages 1–8 and 13–21 were rendered and visually inspected, including the title/authors/version, all Figures 1–11, all Tables 1–12, equations, training and planning details, simulation protocols and appendix implementation/timing evidence. Six final original crops were individually viewed; the causal-mask crop was corrected to exclude a caption fragment and viewed again. Figure 2's action arrows and terminal comparison were checked against Sections 3.1–3.2 and Appendix A.5; no mechanism-changing discrepancy was found there. Source differences in numerical entries, trajectory lengths, preprocessing descriptions and the generalization table cross-reference are disclosed in the report or local cautions. Pages 9–12 contain references and were read as text. No separate supplements, code, external videos or experiments were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16, 17, 18, 19, 20, 21. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2: Introduction and Related Work
- Sections 3, 3.1.1–3.1.3 and 3.2: formulation, observation model, transition model, decoder and visual planning
- Sections 4.1–4.7: environments, baselines, planning, encoders, generalization, generative models and decoded predictions
- Section 5: Conclusion, Limitations and Future Work; Impact Statement; Acknowledgements; References
- Appendix A.1–A.3: datasets, generalization families and pretrained features
- Appendix A.4.1–A.4.3: data scaling, causal attention and reconstruction loss
- Appendix A.5.1–A.5.3: CEM, gradient descent and planning results
- Appendix A.6–A.10: action-conditioned diffusion comparison, image metrics, timing, implementation and additional planning visualizations
Outside the original text pass
- Identity/version note: the title and four authors match the catalog. The inspected title page identifies arXiv:2411.04983v2 [cs.RO], 1 February 2025. The catalog labels ICML 2025, but that proceedings edition and earlier revisions were not supplied for comparison; no equivalence between editions is asserted.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction's visual limitation was addressed by inspecting original PDF pages 1–8 and 13–21, including every figure and table. Reference-only pages 9–12 were read as text.
- Separate supplemental material availability has not been fully verified.
- Only the supplied 21-page primary PDF and its complete text were reviewed. Linked code, models, project videos and external references were not opened, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author line, affiliation footnote and arXiv margin stamp; Abstract
Observed title matches DINO-WM. Authors are Gaoyue Zhou, Hengkai Pan, Yann LeCun and Lerrel Pinto. Affiliations are Courant Institute, New York University and Meta AI. Stamp identifies arXiv:2411.04983v2, 1 Feb 2025. The abstract claims operation without expert demonstrations.
Go to primary source ↓e-problemPDF pp. 1–3, Introduction and Section 3 problem formulation
Offline action-labeled trajectories support a task-agnostic dynamics model; evaluation asks for action sequences reaching an RGB goal from a current observation, rather than reward-maximizing task-specific policies.
Go to primary source ↓e-encoderPDF p. 4, Section 3.1.1 and Figure 2
DINOv2 is frozen during training and testing and returns an N × E matrix of spatial patch embeddings. Current/history images and a separate goal image enter the same observation representation.
Go to primary source ↓e-dynamicsPDF pp. 4–5, Section 3.1.2
The transition ViT removes tokenization, predicts complete frames with causal attention, and concatenates MLP-encoded actions and available proprioception to patch vectors. Training does not require reward, discount or termination targets.
Go to primary source ↓e-lossPDF p. 5, Section 3.1.2, Equation (1)
Teacher forcing uses segments of length H + 1 and latent consistency on H predicted frames. Equation (1) compares the action-conditioned next feature prediction to the encoded next observation.
Go to primary source ↓e-decoderPDF p. 5, Section 3.1.3, Equation (2); pp. 14–15, A.4.3 and Table 7
An independently trained transposed-convolution decoder reconstructs pixels for interpretation and is unnecessary for planning. Feeding decoder loss into the predictor yields PushT SR 0.80 versus 0.92 without it.
Go to primary source ↓e-planningPDF p. 4, Figure 2; p. 5, Section 3.2; p. 15, A.5.1–A.5.2, Equations (3)–(8)
CEM samples Gaussian action sequences, evaluates terminal latent squared distance, selects K elites and updates mean/covariance. MPC executes an action prefix and replans from feedback. Gradient descent is another optimizer. Numeric T, K and executed-prefix settings are not supplied here.
Go to primary source ↓e-protocolPDF pp. 5–6, Sections 4.1–4.3
Observations are 224 × 224 RGB. Baselines IRIS, DreamerV3 and TD-MPC2 are trained on the offline data without rewards/task information and evaluated with MPC. Maze/Reach/PushT/Wall use 50 initial/goal cases; Rope/Granular use Chamfer distance on 10 cases.
Go to primary source ↓e-planning-resultsPDF p. 6, Table 1, all method rows and six environment columns
DINO-WM: SR 0.98/0.96/0.92/0.90 on Maze/Wall/Reach/PushT; CD 0.41/0.26 on Rope/Granular. DreamerV3: 1.00/1.00/0.64/0.30 and 2.49/1.05. IRIS: 0.74/0.04/0.18/0.32 and 1.11/0.37. TD-MPC2: four zero SRs and CD 2.52/1.21. No uncertainty intervals appear.
Go to primary source ↓e-encoder-resultsPDF p. 6, Section 4.4 and Table 2; p. 7, continuation; p. 14, A.3
Encoder comparison includes R3M, ImageNet ResNet-18, DINOv2 CLS and DINO patch embeddings. PushT SR is 0.42/0.20/0.44/0.90, respectively. DINO CLS is a global image vector. The authors attribute patch-feature improvements to spatial information.
Go to primary source ↓e-generalizationPDF pp. 7–8, Section 4.5, Figure 5 and Table 3; p. 13, A.2
WallRandom changes wall/door positions with nonoverlapping train/test positions. PushObj uses four training and two unseen shapes. GranularRandom reuses fixed-count models with fewer than half as many particles. Ours scores 0.82/0.34/0.63; best listed alternatives are 0.76/0.18/0.86. Section 4.5 mistakenly points to Table 5; these results are Table 3.
Go to primary source ↓e-rolloutsPDF p. 7, Figure 4 and caption; p. 5, Section 3.1.3; p. 13, A.1(d)
Figure 4 compares action-conditioned open-loop decoded trajectories for PushT and Granular, with ground truth at the bottom of each group. The green T in PushT is a fixed visual anchor, not the goal marker in this setup. Reconstruction is for interpretation.
Go to primary source ↓e-image-metricsPDF p. 8, Section 4.7 and Table 4; p. 17, A.7 and Table 9
Decoded predicted futures are evaluated by LPIPS (lower better) and SSIM (higher better). Ours has PushT 0.007/0.985 and Granular 0.035/0.940; DINO CLS has 0.039/0.973 and 0.086/0.912. Ours LPIPS is 0.0016 on Wall and 0.009 on Rope.
Go to primary source ↓e-scalingPDF p. 14, A.4.1 and Table 5
PushT dataset sizes 200/1000/5000/10000/18500 give SR 0.08/0.48/0.72/0.88/0.92, SSIM 0.949/0.973/0.981/0.984/0.987 and LPIPS 0.056/0.013/0.007/0.006/0.005. This appendix endpoint differs from main-table values.
Go to primary source ↓e-maskPDF p. 14, A.4.2 and Table 6; p. 4, Section 3.1.2
Input observations o_(t-h+1) through o_t predict o_(t-h+2) through o_(t+1). Unmasked training permits future-input leakage. For h = 1/2/3, unmasked SR is 0.76/0.36/0.08 and masked SR is 0.76/0.88/0.92.
Go to primary source ↓e-optimizerPDF p. 16, A.5.3 and Table 8
CEM and GD rows execute complete plans without replanning; MPC uses receding-horizon CEM. PushT is 0.86/0.28/0.90. PointMaze is 0.80/0.22/0.98. Several GD/CEM task entries are NA, and Reach is not included.
Go to primary source ↓e-diffusionPDF p. 8, Section 4.6 and Figure 6; p. 16, A.6 and Figure 7
The paper distinguishes goal-conditioned AVDC plans from an action-conditioned AVDC variant that predicts the next image. Figure 7 shows two validation-trajectory examples with ground truth, DINO-WM and AVDC-AC rows. No quantitative AVDC-AC planning table appears in A.6.
Go to primary source ↓e-dataPDF p. 13, A.1(a)–(f), especially Push-T and Rope/Granular descriptions; p. 18, Table 11
The six suites are simulated. PushT requires both agent and block target poses within a feasible 25-step task and uses noisy replay of expert trajectories. A.1 says Rope/Granular each have 1000 trajectories of 20 steps; Table 11 lists trajectory length 5. Exact replay noise levels and complete success thresholds are not provided.
Go to primary source ↓e-implementationPDF pp. 17–18, A.9 and Tables 11–12
A.9 specifies resized 196 × 196 inputs, features (14 × 14,384), six predictor layers, 16 heads, MLP width 2048 and about 19M parameters. Table 12 gives image size 224, AdamW, decoder/predictor/action-encoder rates 3e-4/5e-5/5e-4, action dimension 10, 100 epochs and batch 32. Table 11 gives PushT H=3, frameskip=5 and environment-dependent settings; training GPU count/time and software versions are not listed.
Go to primary source ↓e-limitationsPDF p. 8, Section 5, Limitations and Future Work
The authors acknowledge dependence on adequate offline state-action coverage and ground-truth actions, and propose exploration/model updating and hierarchical action planning as future directions.
Go to primary source ↓e-timingPDF p. 17, A.8 and Table 10
On an NVIDIA A6000, Table 10 gives batch-32 inference 0.014 s, batch-1 simulation rollout 3.0 s, and CEM planning 53.0 s using 100 samples per iteration and 10 optimization steps. No matched-batch per-environment timing breakdown is given.
Go to primary source ↓e-overviewPDF p. 3, Figure 1(c); p. 6, Table 1, Granular column
The overview chart labels DreamerV3 Granular Chamfer distance 1.04, whereas Table 1 gives 1.05; IRIS 0.37, TD-MPC2 1.21 and DINO-WM 0.26 agree.
Go to primary source ↓e-additional-plansPDF pp. 18–21, A.10 and Figures 8–11
Additional planning visualizations distinguish observed trajectories after executing actions from imagined trajectories. Figures 10–11 show the same starting configurations paired with different goals; they are qualitative examples rather than additional success-rate measurements.
Go to primary source ↓8.5 Primary sources
DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning ↗
PDF · 10,507 extracted words
Source fingerprint
7683c366109c091a47b7ae8a3d6520131edd3db1a5aa594d74c7636d2f9745d6