Navigation World Models
1. Paper overview
In one sentence: A diffusion model can help select navigation actions by predicting their visual consequences, but accurate short-horizon ranking does not make unfamiliar-scene imagination a reliable map. overviewarchitectureplanning-energyplanning-protocolnavigationvideo-generationtransferlimitations
| At a glance | What to know |
|---|---|
| Research problem | Source description Supervised navigation policies make it difficult to introduce new constraints or spend additional computation on difficult goals. NWM instead learns visual consequences of candidate motion, allowing a planner to compare possible futures. The question is whether this perceptual simulator improves goal-directed trajectory selection across environments and embodiments. introductionplanning-energy |
| Core mechanism | Source description The Conditional Diffusion Transformer (CDiT) separates target-frame self-attention from cross-attention to past frames. Its dominant attention cost grows linearly with context length, supporting larger predictors without full attention over the video context. architecture |
| A key reported result | Goal-conditioned navigation: NWM planning: 1.13 ± 0.02 / 0.35 ± 0.01. ATE / RPE, lower is better; units are not labeled in Table 2.. RECON; predicted two-second trajectories, Table 2. The main paper does not specify evaluation sample selection. NoMaD: 1.93 ± 0.04 / 0.52 ± 0.00; NWM + NoMaD with 32 candidates: 1.78 ± 0.03 / 0.48 ± 0.01. This supports lower short-horizon trajectory error. It is not a goal-completion rate or collision-avoidance result. navigationplanning-protocol |
| Reading caution | Author claim The authors report context loss and drift toward training scenes in unfamiliar environments, difficulty simulating pedestrian motion, and restriction to three-degree-of-freedom navigation. Longer context and more data are suggested remedies, not demonstrated solutions. limitations |
Core contributions
- Source description
The Conditional Diffusion Transformer (CDiT) separates target-frame self-attention from cross-attention to past frames. Its dominant attention cost grows linearly with context length, supporting larger predictors without full attention over the video context. architecture
- Source description
One trained forward model supports standalone search and ranking of NoMaD proposals. A separate experiment tests whether unlabeled human video improves prediction in an unseen environment. planning-protocoltransfer
Figure 1. The same forward predictor supplies imagined views for trajectory scoring and exploratory visualization. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at panel (a), where the observation stack and navigation action enter the Conditional Diffusion Transformer. The action label includes translation, yaw change and time shift. The output is an image, so action choice must happen elsewhere. In panel (b), follow each indoor row from the shared input through generated views to a score against the goal image on the right. The branches visualize alternative consequences of candidate motion. Panel (c) changes the setting to unfamiliar environments: a single starting image supports an imagined traversal. These panels illustrate uses of the predictor, rather than a single measured navigation protocol. overviewformulationplanning-energyplanning-protocollimitations
What it supports. NWM makes candidate actions comparable through their predicted visual outcomes. The planner can therefore reuse a learned forward model while changing proposals or constraints at inference. The arrows terminate in generated images and goal scores; they do not depict an action decoder or an inverse-dynamics network.
Where the evidence stops. The unknown-environment panels are qualitative imagination examples. They do not verify the layout beyond the input image, collision-free traversability, or successful robot execution; the paper later documents hallucinated paths and context drift.
2. Motivation
2.1 The problem and the proposed response
Supervised navigation policies make it difficult to introduce new constraints or spend additional computation on difficult goals. NWM instead learns visual consequences of candidate motion, allowing a planner to compare possible futures. The question is whether this perceptual simulator improves goal-directed trajectory selection across environments and embodiments. introductionplanning-energy
2.2 What this reading follows
Imagine a robot choosing between two routes toward a photographed destination. NWM predicts the views each route would produce, then uses the final predicted image to judge which route better matches the goal. This separates learning visual dynamics from choosing actions: a derivative-free optimizer can search directly, or an existing NoMaD policy can supply candidates. The paper’s efficient CDiT block makes history-conditioned prediction more tractable, and its experiments connect prediction quality to short navigation trajectories. Read the results in three layers: generated-video fidelity, trajectory error in known environments, and visual generalization to unseen scenes. They support different conclusions, especially when additional unlabeled video helps one domain while hurting another. overviewarchitectureplanning-energyplanning-protocolnavigationvideo-generationtransferlimitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Other mechanisms / Outside quadrants is supported: actions condition forward observation prediction, and CEM or a separate NoMaD policy supplies action choices. There is neither joint future/action generation nor inverse dynamics. Dual-system describes predictor/planner separation, especially the NoMaD pipeline; standalone CEM does not imply two jointly trained neural models. formulationarchitectureplanning-energyplanning-protocol
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the future predictor separate from the goal selector
Start with the variables the learned model actually receives. A VAE converts recent RGB observations into compact latents. A proposed action describes planar translation and yaw, and temporal shift requests the interval to the next observation. CDiT then predicts a visual latent under that supplied motion. The goal image enters a different stage: the planner compares it with the decoded terminal prediction and prefers low LPIPS. In standalone planning, CEM searches over candidate actions; in the ranking pipeline, NoMaD supplies the candidates. Both reuse forward simulation, but neither makes the world model jointly output actions and future frames. This distinction explains the taxonomy and the evidence boundary: producing a plausible goal-like image is useful for choosing a trajectory, while actual execution and feedback require additional evaluation. formulationarchitectureplanning-energyplanning-protocolnavigation
Figure 2. CDiT spends self-attention on the target and accesses history through cross-attention. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the Future State box. Section 3.2 specifies that training corrupts this target latent before denoising. The first attention layer operates within the target frame; the middle layer uses current-target queries and the context branch’s keys and values, labeled K,V. The top feedforward block completes the repeated CDiT unit. On the right, action, time shift and diffusion time enter an embedding and AdaLN branch that supplies modulation coefficients. Plus signs indicate residual additions. This layout avoids expensive self-attention among all context tokens. Keep navigation time shift separate from diffusion time: one changes the requested observation interval, the other the corruption level. architecturetraining-objectiveformulationscaling
What it supports. The architectural saving comes from how history is accessed. With fixed tokens per frame, cross-attention grows linearly with the number of context frames, while full joint attention has a quadratic context term. Figure 5 separately tests the resulting prediction-versus-FLOPs tradeoff; this block diagram alone is not a runtime benchmark.
Where the evidence stops. The dashed connector from the lower target residual junction points toward context Scale, Shift, leaving its meaning ambiguous. The prose explicitly assigns past-frame keys/values and target queries; this explanation follows that definition. The Future State label also omits the noisy-target superscript used in training.
5.2 Understand why CDiT can afford more visual context
A full video-token attention layer pays for interactions among every context token as well as the target. CDiT focuses self-attention on the frame being denoised and retrieves history through cross-attention. The paper gives quadratic versus linear dependence on frame count for these dominant attention terms. My interpretation is that this design spends model capacity on predicting the target while avoiding repeated all-to-all processing of history. Table 1 then asks whether retaining more history is useful: four context frames outperform one in its prediction metrics. Figure 5 addresses a different question, comparing accuracy against FLOPs as model size changes. Neither experiment alone demonstrates real-time control, and the ambiguous dashed connector in Figure 2 should not override the explicit attention roles in Section 3.2. architectureablationscalingtraining-objective
5.3 Trace the chain from good video to useful navigation
The experiments test different links in a causal chain. Figure 6 asks whether the simulator produces better video distributions under known actions. Table 2 asks whether predicted terminal views help select short trajectories toward a goal. Table 4 asks whether additional action-free video transfers visual prediction to a new environment. A gain in one link need not guarantee a gain in the others. The transfer experiment makes this concrete: Go Stanford improves while RECON becomes worse. My reading is that a useful world model must be judged both by visual prediction and by how its errors affect action selection. The authors’ context-drift and pedestrian-motion limitations make that distinction operationally relevant. An imagined path through a new scene remains a prediction, even when the sequence looks coherent. video-generationnavigationplanning-protocoltransferlimitations
5.4 Training and inference
During training
Robot training uses SCAND, TartanDrive, RECON and HuRoN. Pose changes approximate relative actions; displacement is normalized by each agent’s average step size and backward movements are filtered. Optional Ego4D training omits explicit navigation-action conditioning while retaining time shift. data-trainingarchitecture
The objective reconstructs clean target latents from noise, with an additional variational lower-bound loss for predicted noise covariance. The paper uses a pretrained VAE and follows DiT noise-schedule settings; the supplied main text does not explicitly specify whether the VAE is frozen. training-objectivedata-training
Default CDiT-XL has 1B parameters, four context frames and four targets per context: batch size 1,024 yields 4,096 target examples. AdamW uses learning rate 8e-5. XL training uses eight machines with eight H100 GPUs each. Means and standard deviations use five sampling repetitions, not explicitly five independent training runs. data-training
During inference
Standalone planning uses derivative-free Cross-Entropy Method (CEM) optimization of expected energy. The experiment averages terminal LPIPS over three stochastic rollouts per action sequence and plans eight steps with temporal shift k = 0.25, giving a two-second horizon. Detailed optimizer settings are deferred to the unsupplied appendix. planning-energyplanning-protocol
Alternatively, sample 16 or 32 eight-step trajectories from NoMaD and choose the lowest terminal LPIPS. Demonstrated constraints zero selected action components, such as forward motion during the initial turning phase. The formal MPC formulation does not itself document a physical closed-loop deployment protocol. planning-protocolconstraints
5.5 Implementation flow
- Represent motion and time
Planar displacement and yaw describe navigation; temporal shift specifies how far forward or backward to predict. Training samples targets within ±16 seconds. Multiple targets per context are intended to expose natural counterfactuals and discourage relying on time while ignoring action. formulationablation
- Denoise using history
Target tokens self-attend, then query past-frame keys and values through cross-attention. Sine/cosine features and two-layer MLPs embed action scalars; action, time-shift and diffusion-time embeddings are summed to condition adaptive layer normalization. The figure’s dashed connector near the context branch is ambiguous; the prose explicitly defines target queries and past-frame keys/values. architecture
- Roll out and evaluate
Predicted latents become context for subsequent predictions under candidate actions. Decoded terminal images are compared with the goal. Action selection occurs in the planner; CDiT predicts observations rather than extracting actions through inverse dynamics. planning-energyplanning-protocolvideo-generation
6. Experiments & results
Navigation World Models turns action-conditioned video prediction into a navigation evaluator: simulate candidate trajectories, compare their final views with a goal image, and select actions through search or policy reranking. The strongest main-paper result concerns short RECON trajectory errors; unfamiliar-scene imagination remains vulnerable to drift.
6.1 Read the original evidence
Table 2. Standalone planning reports the lowest errors for the table’s two-second RECON trajectories. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Both columns point downward: ATE measures absolute trajectory error and RPE measures relative pose error. Compare the unassisted policy rows with NWM + NoMaD to isolate the reported pipeline change from candidate generation to simulation-based selection. The ×16 and ×32 labels count sampled NoMaD trajectories, not training epochs or video frames. The final row instead searches using CEM, with terminal LPIPS averaged over three stochastic predictions per sequence. Section 4.4 describes eight steps at a temporal shift of 0.25. Read these numbers under the caption’s RECON scope; nearby prose mentions all in-domain datasets, but this table only supplies RECON. navigationplanning-protocoldata-trainingplanning-energy
What it supports. Standalone NWM reports ATE 1.13 ± 0.02 and RPE 0.35 ± 0.01. NoMaD reports 1.93 ± 0.04 and 0.52 ± 0.00. Reranking 32 proposals reaches 1.78 ± 0.03 and 0.48 ± 0.01, supporting a smaller improvement when the action candidates come from an existing policy.
Where the evidence stops. The table labels no units and supplies neither success percentages nor collision statistics. Evaluation example selection and complete CEM settings are deferred to an unsupplied appendix; the larger standalone gain should not be interpreted as a compute-matched comparison.
Figure 6 (FVD table). Generated-video quality is evaluated with supplied ground-truth actions. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This compact visual is labeled Figure 6 in the paper but contains a quantitative table. Its single row measures FVD, with the downward arrow indicating that smaller is better. The caption and Section 4.3 specify sixteen-second RECON videos generated autoregressively at four frames per second. The model receives ground-truth actions, feeds its own predicted visual states back into the rollout, and is evaluated on the resulting video distribution. Keep this protocol distinct from Table 2, where the system must select an action trajectory. The reported deviations follow the paper’s sampling-repeat convention, without establishing variability across independently trained models. video-generationdata-training
What it supports. NWM’s FVD is 200.969 ± 5.629, compared with DIAMOND’s 762.734 ± 3.361. This supports improved distributional video quality under the stated action-conditioned rollout protocol. It complements the trajectory results by testing the simulator’s generated outputs, rather than measuring whether a planner chooses the right route.
Where the evidence stops. FVD is a distribution-level synthesis metric, not a certificate that every predicted turn or obstacle is correct. The baselines differ architecturally, and this table supplies no matched-compute or physical-control latency comparison.
Table 4. Unlabeled video produces a transfer benefit alongside an in-domain cost. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each dataset group independently, first comparing the two training rows within Go Stanford, then doing the same within RECON. The additional Ego4D videos lack explicit navigation-action labels, so their training uses temporal shift without the motion embedding. LPIPS and DreamSim favor lower values; PSNR favors higher values. The left group evaluates an unknown environment, while the right group evaluates the known environment. All measurements are four seconds into the future. Bold values switch rows across the two dataset groups, revealing a tradeoff that is easy to miss if one reads only the caption’s emphasis on unseen-environment improvement. transferarchitecturedata-traininglimitations
What it supports. Go Stanford LPIPS improves from 0.658 ± 0.002 to 0.652 ± 0.003 with Ego4D. RECON LPIPS worsens from 0.295 ± 0.002 to 0.368 ± 0.003. DreamSim and PSNR agree directionally, so the evidence supports transfer with a cost to familiar-domain prediction rather than a uniform improvement.
Where the evidence stops. The table provides reported means and deviations, not a significance test or unseen-environment navigation success rates. Section 4.5 and the failure discussion acknowledge faster breakdown and hallucinated paths when unfamiliar scenes are rolled out.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Goal-conditioned navigation RECON; predicted two-second trajectories, Table 2. The main paper does not specify evaluation sample selection. | NWM planning: 1.13 ± 0.02 / 0.35 ± 0.01. ATE / RPE, lower is better; units are not labeled in Table 2. | NoMaD: 1.93 ± 0.04 / 0.52 ± 0.00; NWM + NoMaD with 32 candidates: 1.78 ± 0.03 / 0.48 ± 0.01. This supports lower short-horizon trajectory error. It is not a goal-completion rate or collision-avoidance result. navigationplanning-protocol |
| Video synthesis quality RECON; 16-second autoregressive videos at 4 FPS, conditioned on ground-truth actions. | NWM: 200.969 ± 5.629. FVD, lower is better. | DIAMOND: 762.734 ± 3.361. This evaluates generated-video distributions with supplied actions, independently of autonomous action choice. video-generation |
| Action and time conditioning Single-step four-second future prediction on RECON validation trajectories. | Action + time: 0.295 ± 0.002. LPIPS, lower is better. | Time only: 0.760 ± 0.001; action only: 0.318 ± 0.002. Action carries substantial predictive information, while time adds a smaller benefit. This does not prove complete disentanglement of motion and time. ablationdata-training |
| Unlabeled-video transfer Four-second prediction on held-out Go Stanford and known RECON, Table 4. | Adding Ego4D changes Go Stanford from 0.658 ± 0.002 to 0.652 ± 0.003. LPIPS, lower is better. | RECON worsens from 0.295 ± 0.002 to 0.368 ± 0.003. DreamSim and PSNR show the same direction of tradeoff. Improved unseen-domain means accompany degraded known-domain accuracy. The table provides no significance test. transfer |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Four-second prediction benefits from more history and targets; action conditioning carries most of the measured signal. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three row blocks separately. The upper block varies how many target goals are sampled for a fixed context; these are training targets, not a planner’s candidate count. The middle block varies the number of conditioning frames. The final block compares time-only, action-only and combined conditioning. LPIPS and DreamSim decrease with better predictions, whereas PSNR increases. All rows concern four-second future prediction on RECON, rather than autonomous navigation. Notice that the four-goal and four-context rows report LPIPS 0.296, while action plus time reports 0.295: preserve the separate blocks instead of forcing their rounded results into one shared baseline. ablationdata-trainingformulation
What it supports. Combined conditioning obtains LPIPS 0.295 ± 0.002, compared with 0.760 ± 0.001 for time alone and 0.318 ± 0.002 for action alone. Four goals improve the upper block from 0.312 to 0.296. These results support useful action information and benefits from additional training targets and context.
Where the evidence stops. More goals also increase target examples per context in the reported training setup, so this is not a fixed-target-budget counterfactual test. The main text also leaves the conditioning rows’ exact retraining-versus-input-masking protocol insufficiently detailed.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report context loss and drift toward training scenes in unfamiliar environments, difficulty simulating pedestrian motion, and restriction to three-degree-of-freedom navigation. Longer context and more data are suggested remedies, not demonstrated solutions. limitations
Table 2 is explicitly RECON, although nearby ranking prose says all in-domain datasets. This report follows the table’s scope. Perceptual goal matching and action-zeroing constraints do not establish safe-state enforcement, long-horizon completion or robustness to execution errors. navigationplanning-protocolconstraintsplanning-energy
CDiT scaling is plotted against FLOPs, not measured control latency. Baselines also use different training collections and architectures. The comparisons establish reported system performance rather than isolating every architectural or data contribution. scalingdata-training
7.2 Questions for discussion
- Would terminal LPIPS still identify better trajectories under perceptual aliasing?
- Does multi-target sampling help beyond its increased number of training targets?
- Can unlabeled-data mixing improve Go Stanford without degrading RECON?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful reproduction needs robot pose streams, average-step normalization, backward-motion filtering, the VAE/CDiT configuration and matched splits. The main paper defers dataset and CEM details to Appendices 7–8; training duration, complete optimizer configuration and evaluation sample counts are not supplied here. data-trainingplanning-energyplanning-protocol
Proposed checks should hold data and prediction budgets fixed while varying target sampling, then test whether terminal LPIPS selects lower-error NoMaD trajectories from identical candidate pools. Preserve rollout stochasticity and distinguish sampling uncertainty from retraining variability. ablationdata-trainingplanning-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate multi-target diversity from target-example budget
Reader-proposed check: repeat the one-goal versus four-goal RECON training comparison with the same architecture, four-frame context, data split, optimizer and total number of target predictions. Use gradient accumulation or resampling so target-example budgets match, and also report a run matching unique-context exposures to expose that tradeoff. Preserve the ±16-second target window. Evaluate the same four-second validation pairs using LPIPS, DreamSim and PSNR, with independent training seeds and matched sampling repeats. If the four-goal advantage disappears once target counts are controlled, the original table would support extra prediction supervision more strongly than the proposed counterfactual-diversity mechanism. formulationablationdata-training
Check 2: Test whether terminal image distance selects better actions
Reader-proposed check: use fixed RECON observation/goal pairs and identical pools of 32 eight-step NoMaD trajectories. Compare random selection, NWM selection by terminal LPIPS averaged over three stochastic rollouts, and an oracle selecting the lowest ground-truth trajectory error from the same pool. Report ATE/RPE, the relationship between LPIPS rank and trajectory error, and complete selection latency. Repeat with fixed 16-candidate subsets to separate candidate availability from scoring quality. If LPIPS has weak ranking agreement or fails to beat random selection on held-out pairs, plausible terminal images would not suffice to explain reliable action selection. Resolve the missing evaluation-split details before claiming an exact reproduction. navigationplanning-protocolplanning-energydata-training
8.3 Reading coverage
Visual audit: Visually inspected the title/author/affiliation block and accepted-version watermark on p. 1; formulation on p. 3; attention arrows, conditioning equation and diffusion objective on p. 4; planning equations, datasets, metrics, optimizer and hardware on p. 5; all main-paper Figures 1–10 and Tables 1–4 across pp. 1, 4, 6–8. All six final original crops were inspected, including headers, metric directions and bounds. Figure 2’s ambiguous dashed context connector is disclosed in its caution. Every page supporting retained method, numerical, training and proposed-check details is included. References on pp. 9–11 were read as text. The separate supplement, linked animations, code and experiments remain outside this review.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not read.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, abstract and Figure 1 (PDF p. 1)
- 1. Introduction and 2. Related Work (PDF pp. 2–3)
- 3. Navigation World Models: Sections 3.1–3.3 (PDF pp. 3–5)
- 4. Experiments and Results: Sections 4.1–4.5 (PDF pp. 5–8)
- 5. Limitations, 6. Discussion and acknowledgments (PDF p. 8)
- Complete references (PDF pp. 9–11; proceedings pp. 15799–15801)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental materials were discovered but are not part of the primary text; inspect supplements individually.
- The entire 11-page main paper was read. The separate supplement listed in the acquisition record was outside the supplied reading material; referenced Appendices 7, 8.1 and 8.2 were not read. Supplement-only claims in the preliminary report are not retained.
- Text extraction does not reconstruct figure images. This edition adds visual inspection of the original main-paper figures and tables; linked animations were not watched.
- Code, checkpoints and datasets were not inspected, and experiments were not reproduced.
- Identity/version: the title and five author credits match the catalog author field. The title-page watermark identifies the CVF open-access version as identical to the accepted paper except for the watermark. The final IEEE proceedings edition was not supplied; no numbered revision is established.
- The catalog BibTeX spells the third author Dung Tran, whereas the supplied title page and catalog author field say Danny Tran. This report preserves the observed credit without asserting that the names are equivalent.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1 / proceedings p. 15791, title, author/affiliation block and CVF watermark
Observed title: Navigation World Models. Authors: Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell and Yann LeCun. Affiliations: FAIR at Meta, New York University and Berkeley AI Research. The watermark describes the accepted-version relationship.
Go to primary source ↓overviewPDF p. 1, Figure 1 panels (a)–(c) and caption
The model predicts views from observations and navigation actions; terminal images score trajectories, while unknown-environment examples illustrate imagined traversals.
Go to primary source ↓introductionPDF p. 2, Section 1, paragraphs on policy constraints and planning
The motivation is test-time constraints and variable planning computation, with separate standalone MPC and external-policy ranking setups.
Go to primary source ↓formulationPDF p. 3, Section 3.1, Equations (1)–(2) and time-shift paragraphs
Pretrained VAE latents condition stochastic forward prediction. Actions represent planar translation and yaw, extended with temporal shifts up to ±16 seconds. Multiple goals encourage natural counterfactuals.
Go to primary source ↓architecturePDF p. 4, Section 3.2, Figure 2 and Equation (3)
Target self-attention precedes cross-attention using past-frame keys/values. Action, time shift and diffusion embeddings drive AdaLN. Dominant costs are O(m²n²d) for full attention and O(mn²d) for CDiT, with m frames, n tokens/frame and token dimension d. Figure 2 has an ambiguous dashed connector from the lower target residual junction toward context Scale, Shift.
Go to primary source ↓training-objectivePDF p. 4, Section 3.2, Diffusion Training and Training Objective
Gaussian corruption at sampled diffusion timesteps trains clean-latent reconstruction; predicted noise covariance receives a variational lower-bound loss.
Go to primary source ↓planning-energyPDF pp. 4–5, Section 3.3, Equations (4)–(5)
Expected energy combines terminal perceptual similarity with invalid-action and unsafe-state penalties. CEM optimizes action sequences; detailed optimization is deferred to Appendix 7.
Go to primary source ↓data-trainingPDF p. 5, Section 4.1, Datasets, Evaluation Metrics, Baselines and Implementation Details; opening of Section 4.2
Robot datasets, normalization/filtering, unlabeled Ego4D, baseline settings, CDiT-XL size, context/target batch, AdamW learning rate, H100 hardware and five sampling repetitions are specified. Ablations use four-second RECON validation prediction.
Go to primary source ↓ablationPDF p. 6, Table 1, all rows and metric headers; Section 4.2 continuation
Four goals and four context frames each obtain LPIPS 0.296 ± 0.002 in their blocks. Conditioning-block LPIPS is 0.760 ± 0.001 for time only, 0.318 ± 0.002 for action only and 0.295 ± 0.002 for action + time.
Go to primary source ↓scalingPDF p. 6, Figure 5 axes, point labels, bubble-size legend and caption; Section 4.2 continuation
CDiT and DiT are compared using LPIPS versus TFLOPs for four-second RECON prediction, with bubble diameter encoding parameter count. This is a compute plot, not a latency measurement.
Go to primary source ↓video-generationPDF p. 6, Figures 4 and 6 and Section 4.3; PDF p. 7, Section 4.3 continuation and Generation Quality
Autoregressive evaluation uses ground-truth actions. Figure 6 reports 16-second, 4-FPS RECON FVD: DIAMOND 762.734 ± 3.361 and NWM 200.969 ± 5.629. Figure 4 compares 1- and 4-FPS rollouts.
Go to primary source ↓navigationPDF p. 7, Table 2, all rows, ATE/RPE headers and caption
The caption specifies RECON and two-second predicted trajectories. NWM planning reports ATE/RPE 1.13 ± 0.02 / 0.35 ± 0.01, versus NoMaD 1.93 ± 0.04 / 0.52 ± 0.00; 32-candidate reranking reports 1.78 ± 0.03 / 0.48 ± 0.01.
Go to primary source ↓planning-protocolPDF p. 7, Section 4.4, Standalone Planning and Using a Navigation World Model for Ranking; Figure 7
CEM uses terminal LPIPS averaged over three simulations, eight steps and k = 0.25. Ranking samples 16 or 32 eight-step NoMaD trajectories. Details are deferred to Appendix 8.2; the prose’s all-dataset wording exceeds Table 2’s RECON scope.
Go to primary source ↓constraintsPDF p. 7, Table 3 and Section 4.4 Planning with Constraints; PDF p. 8, Figure 9 and caption
Demonstrated constraints zero selected action components. Table 3 reports terminal position/yaw changes relative to unconstrained planning; Figure 9 illustrates a turn-first comparison.
Go to primary source ↓transferPDF p. 8, Table 4, both dataset groups and training rows; Section 4.5 continuation
Ego4D changes Go Stanford LPIPS/DreamSim/PSNR from 0.658/0.478/11.031 to 0.652/0.464/11.083. RECON changes from 0.295/0.091/15.343 to 0.368/0.138/14.072; uncertainty values are shown in the table.
Go to primary source ↓limitationsPDF p. 8, Sections 5–6, Figures 8 and 10 and captions
Unfamiliar-environment rollouts hallucinate paths and can drift toward training scenes. Pedestrian dynamics and three-degree-of-freedom actions limit current scope; the discussion leaves the learned spatial representation unresolved.
Go to primary source ↓8.5 Primary sources
Navigation World Models ↗
PDF · 8,340 extracted words
Source fingerprint
2cfe50f6350a9bd68f8bfa8fbc61d38bf2c186106ecb28fb6e37ab716bf34d26