Pathdreamer: A World Model for Indoor Navigation
1. Paper overview
In one sentence: Pathdreamer turns remembered geometry into stochastic future panoramas that help a separate navigation planner, but plausible room completions can still misrepresent the real route. e02e03e04e06e12e13
| At a glance | What to know |
|---|---|
| Research problem | Source description An agent entering an unfamiliar room must infer objects and geometry that no previous camera view reveals. Pathdreamer addresses this room-reveal problem while trying to keep revisited regions consistent, even when only one prior panoramic observation is available. e02 |
| Core mechanism | |
| A key reported result | Vision-and-language navigation with three-step planning: Pathdreamer 50.4%. Success rate (SR, higher is better). R2R Val-Unseen; identical ground-truth navigation-graph access; compatibility-based path ranking and first-action execution. Repeated panorama 40.6%; blank panorama 41.9%; ground-truth future observations 59.3%. Reader calculation: +9.8 percentage points over repeated panoramas, closing about 52.4% of that baseline's 18.7-point gap to privileged look-ahead. This is graph-based simulated navigation, not physical robot deployment. e12e13 |
| Reading caution |
Core contributions
Figure 2. Geometry supplies remembered context; learned structure supplies what the next view has not yet revealed. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the semantic/depth history on the left and follow it through the 3D point cloud to projection at the requested pose. The encoder receives sparse guidance, while the noise branch contributes latent information before decoding. The upper encoder sees target structure during training; the dashed connection means shared weights, and the red slash marks no gradients on the target-encoder/posterior connection. Section 3.1 and Equation (1) specify posterior noise during training and prior noise during inference. Follow the structural output into the right-hand Multi-SPADE stack, alongside previous RGB and reprojected RGB guidance. These are distinct sources of layout and appearance context. e03e04e05e06e07e14
What it supports. Pathdreamer's persistent state includes explicit geometry, while its stochastic component completes unseen structure. RGB synthesis then conditions on that structure. The diagram explains why a new room can vary across latent samples while already visible appearance still contributes to the generated view.
Where the evidence stops. The cross-stage arrow depicts inference. The stages train separately, and RGB training receives ground-truth structure. Figure 2 lists feature-matching loss, but Appendix C sets its weight to zero; it is not an active term in the reported configuration.
2. Motivation
2.1 The problem and the proposed response
An agent entering an unfamiliar room must infer objects and geometry that no previous camera view reveals. Pathdreamer addresses this room-reveal problem while trying to keep revisited regions consistent, even when only one prior panoramic observation is available. e02
2.2 What this reading follows
Imagine an agent following an instruction through a building it has never seen. A doorway reveals only part of the next room, yet choosing a route requires reasoning about what lies beyond it. Pathdreamer addresses this uncertainty by completing semantic structure and depth before generating RGB. Its point clouds preserve observation history, and a latent variable permits alternative unseen layouts. This reading follows the connection from that architecture to image-quality ablations and instruction-following navigation. The distinction to keep in view is between predicting useful observations and knowing the environment: the reported planner benefits from generated images while still receiving feasible path geometry from the true navigation graph. e02e03e04e06e12e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system classification is supported by the separation between visual prediction and the VLN proposal/ranking/action loop, not merely by the two image-generation stages. The mechanism is pose-conditioned future-observation synthesis with point-cloud memory and external planning, rather than joint future/action prediction or inverse dynamics. This supports Other mechanisms and Outside quadrants, while Navigation and Memory & long-horizon modeling describe the task and persistent context. e03e04e06e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Start with what projection can preserve
A future pose changes which surfaces should be visible. Pathdreamer handles this known geometric part before asking a neural network to invent anything: previous observations are back-projected into point clouds and rendered at the new pose. The resulting guidance can preserve already observed surfaces, but it cannot contain a room that no observation has exposed. The Structure Generator therefore completes the semantic and depth image, and its output supplies context for later predictions. A separate RGB point cloud helps preserve appearance. Reader interpretation: this architecture divides prediction into reuse of observed geometry and learned completion of missing content. It also creates a route for earlier prediction errors to influence later views; Figure 8(c) illustrates divergence, although it does not experimentally isolate the point cloud as the cause. e03e04e06e17
Figure 3. The same partial observation can support several different room completions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each example from left to right. The first column is projected guidance, with black regions indicating missing observations. The middle columns use three noise vectors; each pairs an RGB panorama with a semantic prediction. The last column is the actual scene. Compare the labeled bed, chair and table alternatives in the upper example, then the shelving, drawers and door alternatives below. These semantic changes appear in the associated RGB views, which is the visible consequence of putting stochasticity in the Structure Generator. The columns are alternative futures for a fixed input, rather than successive frames of one navigation rollout. e04e08e16
What it supports. The model can express ambiguity about a room's contents rather than committing to one deterministic completion. Figure 3 provides qualitative evidence that latent structural variation reaches the rendered image: changing the predicted object layout changes corresponding appearance, while the observed context continues to constrain the scene.
Where the evidence stops. Diversity does not establish calibrated uncertainty or recovery of the actual room. The lower example misses the matching lamp identified in the caption. Quantitative experiments use the prior mean, so their navigation gains do not demonstrate benefits from sampling several futures.
5.2 Learn a prior that can operate without the answer
The training target contains information about unseen rooms that the guidance image lacks. Pathdreamer's posterior network can inspect that target and provide a useful latent code to the decoder during training. A second network learns a prior from guidance alone, with Equation (1) penalizing divergence between the two distributions while also supervising semantics and depth. At inference, only the prior is available. The Image Generator translates the resulting layout into RGB, so alternatives in Figure 3 affect object structure as well as appearance. Training remains modular: the RGB generator sees true structure during training, then predicted structure at deployment. Appendix C further specifies zero weight for feature matching. Finally, the numerical evaluations replace prior sampling with its mean, distinguishing demonstrated prediction quality from the model's qualitative ability to generate alternatives. e04e05e06e07e08e14e16
5.3 Trace the prediction into an executed navigation decision
Pathdreamer does not read the navigation instruction and directly output an action. A baseline VLN agent proposes candidate trajectories with beam search; a separate dual encoder assesses instruction compatibility from path geometry and visual observations. Future RGB can come from Pathdreamer, actual look-ahead, blank images or repeated current views. The agent executes the first action of the highest-ranked trajectory, then repeats the process. Table 3 therefore tests whether imagined visual content helps that decision pipeline. The three-step success gain is meaningful because the comparison holds access to ground-truth graph geometry constant. Reader interpretation: this supports useful visual foresight in a modular planner, while leaving open how much of the benefit survives when the agent must also estimate the graph or detect an incorrect room prediction. e12e13e17
5.4 Training and inference
During training
Matterport3D supplies 90 indoor environments and 40 semantic classes. Training samples 400k R2R-graph trajectories with 5–8 panoramas, approximately 2m apart. Habitat renders structural supervision. Structure uses 512×256 panoramas with Gaussian viewpoint perturbations; RGB uses 1024×512 panoramas with random crops and horizontal rolls. Neither generator is pretrained, although the perceptual VGG-19 network is. e06e07
Appendix C specifies structure training for 50 epochs, batch 64: 30 epochs of teacher forcing with context count reduced from path length minus one to one, then recurrent feedback. Adam uses beta values 0.9/0.999 and a learning rate warming from 0.0001 to 0.0002 over 10 epochs. e14
RGB training uses 500 epochs, batch 128, Adam beta values 0.5/0.999 and learning rate 0.0002 for generator and discriminator, with two discriminator steps per generator step. Inference uses generator EMA decay 0.999. Despite Figure 2 and Equation (2) listing feature matching, the actual weights are GAN 1, VGG 0.07 and feature matching 0. e06e14
During inference
Both stages use horizontal circular padding. Structural predictions are enlarged to RGB resolution using nearest-neighbor interpolation. Although the model can sample its learned prior, all quantitative experiments use its mean; those results do not test the benefit of multiple stochastic rollouts. e08
For VLN, a separate agent proposes beam-search paths. A dual encoder scores instruction compatibility using visual observations and path geometry; it was trained with true observations, including truncated paths. The agent executes the best path's first action and replans over horizons of 1–3 steps. e12
5.5 Implementation flow
- Project persistent context
Back-project previous semantic/depth observations into a 3D point cloud and reproject it at the requested pose. The resulting sparse guidance preserves visible structure and leaves holes where the new view reveals unknown regions. A parallel RGB point cloud supplies appearance context. e03e04e06
- Complete structure stochastically
A RedNet-based encoder-decoder receives one-hot semantic guidance plus depth. Skip connections retain spatial detail; a spatial noise tensor is concatenated at the bottleneck. Softmax produces semantic classes and sigmoid produces normalized depth. Predictions are back-projected to support subsequent steps. e04
- Learn uncertainty without future inputs at deployment
Training samples noise from a posterior conditioned on target semantics/depth and matches a guidance-conditioned prior through KL divergence. At inference the posterior is unused. Thus alternative room layouts arise from latent structural variation, rather than merely recoloring a fixed scene. e04e05e16
- Render appearance
The Image Generator starts from previous RGB and uses seven Multi-SPADE blocks conditioned on structure and projected RGB guidance. Partial convolutions handle sparse guidance. A PatchGAN discriminator and VGG-19 perceptual features supervise realistic synthesis; the two generator stages are trained separately. e06e07
6. Experiments & results
Pathdreamer imagines what an indoor agent might see along a supplied future trajectory. It completes semantic segmentation and depth first, then renders RGB, using accumulated 3D point clouds to preserve visual context. Its predictions improve a separate instruction-following navigation system on unseen Matterport3D buildings. The central tradeoff is useful visual foresight without reliable recovery of the actual unseen layout; navigation still assumes a ground-truth graph of feasible movements.
6.1 Read the original evidence
Table 1. Recurrent feedback helps seen-building rollouts much more than unseen-building rollouts. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First fix the Context row group: one, two or three previous observations. Then compare training variants within one split and one prediction column. The table reports mIOU on a 0–100 scale, where higher is better. Its '1–6 Steps' column summarizes the rollout range and is not the sixth-step endpoint. With one context observation, recurrent training raises the seen-building aggregate from 59.2 to 65.9 relative to teacher forcing; the corresponding unseen values are 50.8 and 50.9. Move down to three contexts only after making this matched comparison. Figure 5a on the same source page separately shows the per-step deterioration. e09e10e14e15
What it supports. Learned semantic completion clearly outperforms nearest-neighbor filling on unseen buildings: the one-context aggregate is 50.9 for recurrent training versus 30.6 for the baseline. Yet recurrent feedback itself adds little over teacher forcing there. With three context observations, recurrent unseen mIOU rises to 60.8.
Where the evidence stops. The comparison tests the reported training variants, not universal benefits of recurrence. Appendix C describes a teacher-forcing-to-recurrent curriculum, and evaluation selects a checkpoint for each validation split. No seed-level uncertainty is supplied; Figure 5 bands vary context count.
Table 3. Generated look-ahead improves path selection when feasible route geometry is already supplied. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each horizontal group fixes the planning horizon while changing future RGB observations. The ranker uses the instruction, images and path geometry; execution takes the best path's first action before replanning. Read the three-step Pathdreamer row across: navigation error (NE) 5.32, success rate (SR) 50.4, SPL 47.3, nDTW 61.8 and sDTW 44.4. This is the penultimate row, immediately above Ground truth. Now compare SR within that group: Pathdreamer's 50.4 versus repeated panoramas at 40.6, blank panoramas at 41.9 and actual future observations at 59.3. Higher SR is better; the NE arrow points downward. The ground-truth row is privileged visual look-ahead. All rows receive future trajectory geometry from the true R2R navigation graph, so this comparison isolates the observation source given that geometry. e12e13
What it supports. At three planning steps, generated images add 9.8 percentage points of success over repeating the current view. Dividing that gain by the 18.7-point repeated-to-ground-truth gap gives approximately 52.4%, a reader calculation supporting the paper's 'about half' claim. Pathdreamer also improves SPL from 37.7 to 47.3 over repeated panoramas.
Where the evidence stops. This is a downstream navigation result within graph-based simulation. It does not establish free-space discovery, physical robot execution or performance without privileged graph geometry. The ranker is trained on true observations, and the table reports no repeated-run uncertainty.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Vision-and-language navigation with three-step planning R2R Val-Unseen; identical ground-truth navigation-graph access; compatibility-based path ranking and first-action execution. | Pathdreamer 50.4%. Success rate (SR, higher is better) | Repeated panorama 40.6%; blank panorama 41.9%; ground-truth future observations 59.3%. Reader calculation: +9.8 percentage points over repeated panoramas, closing about 52.4% of that baseline's 18.7-point gap to privileged look-ahead. This is graph-based simulated navigation, not physical robot deployment. e12e13 |
| Semantic prediction and recurrent-training ablation R2R Val-Seen/Val-Unseen, one context observation, aggregate 1–6-step prediction column. | Recurrent: 65.9 seen and 50.9 unseen. mIOU (higher is better; table's 0–100 scale) | Teacher forcing: 59.2 seen and 50.8 unseen; nearest-neighbor baseline: 30.6 unseen. Recurrent training helps seen-building rollouts substantially but shows almost no unseen-building improvement; the geometry-aware learned predictor nevertheless exceeds hole filling. e09e10 |
| RGB prediction and context ablation R2R Val-Unseen, one context observation, aggregate 1–6-step prediction column. | Full model with predicted structure: 70.4. FID (lower is better) | No Semantics: 90.8; Ours with ground-truth structure: 39.9; SPADE with ground-truth structure: 50.3. Structural prediction is a bottleneck. RGB context helps under matched ground-truth structural inputs. No Semantics removes both semantics and depth, so this comparison does not isolate semantics alone. e09e11 |
| Additional observation context R2R Val-Unseen, aggregate 1–6-step prediction column; three versus one context observations. | Three-context recurrent mIOU 60.8; full-model FID 52.9. mIOU upward; FID downward | One-context recurrent mIOU 50.9; full-model FID 70.4. More observed context improves both reported metrics. These aggregate columns should not be read as measurements at the sixth step alone. e10e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. RGB context helps unfamiliar scenes, while imperfect structural predictions limit longer rollouts. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep context count and split fixed before comparing rows. Lower FID is better. 'GT' denotes ground-truth semantic inputs; 'SG' denotes Structure Generator predictions, so the GT rows are privileged component evaluations. The check marks distinguish semantic and RGB context. For one context on Val-Unseen, compare Ours (GT), 39.9, with SPADE (GT), 50.3, in the aggregate column to assess RGB context under matched structural supervision. Next compare Ours (SG), 70.4, with Ours (GT) to see the gap associated with imperfect structure. The No Semantics variant actually removes both semantic and depth inputs, as defined in the surrounding text. e09e11e15
What it supports. The full predicted-structure model improves aggregate unseen FID over No Semantics, from 90.8 to 70.4, and reaches 52.9 with three contexts. Ground-truth structure yields substantially better scores. Together, the comparisons support structural guidance and retained RGB context, while identifying structural prediction as a major remaining bottleneck.
Where the evidence stops. Do not present the 39.9 GT score as end-to-end prediction. Removing semantics also removes depth, and the SPADE ablation changes multiple RGB pathways. FID uses augmented samples and measures image-distribution similarity; it does not verify the exact future scene or navigation success.
Figure 8(c). An early missed entrance produces a plausible but increasingly incorrect imagined environment. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read left to right along the distance labels, keeping the four modality rows aligned. The input column gives the single true context observation. Each later column shows predicted depth and semantics above generated RGB, with actual RGB in the bottom row. At 2.1m, compare the center of the generated panorama with the real room entrance below it: the caption identifies this as the missed structural feature. Continue through later columns and inspect how the predicted semantic layout and rendered room evolve together while separating from the true route. The figure is one of Appendix A's randomly selected Val-Unseen examples, not one of its selected success cases. e03e04e17e18
What it supports. A convincing-looking indoor panorama can encode the wrong spatial continuation. This example makes the limitation concrete: structural error appears early, and subsequent generated observations follow an increasingly different scene. It motivates evaluating the effect of prediction errors on route decisions, rather than judging RGB plausibility alone.
Where the evidence stops. This is a qualitative diagnostic, not a controlled ablation or a measured failure rate. It does not isolate which memory or generation component caused the error. The depth row has no numerical error scale and cannot establish metric depth accuracy.
7. Analysis & limitations
7.1 What the evidence leaves open
Future path geometry comes from the true R2R graph for every navigation condition. The experiment therefore establishes value from predicted visual observations given feasible paths, not autonomous discovery of free space or robustness to odometry and perception errors. e03e12
Plausible completions can be wrong: Figure 8(c) misses a room entrance at 2.1m and diverges later. Figure 3 demonstrates diversity but also a missed matching lamp; it provides no calibrated probability of the actual room layout. e16e17
Reported generation results use the best checkpoint separately on each validation split, not an untouched test set. Figure 5's bands represent different context counts, not repeated-training uncertainty. The tables provide no seed-level uncertainty, and depth has no separate quantitative result table. e09e10e11e15
7.2 Questions for discussion
- Would the visual-planning gain persist when candidate paths come from estimated free space instead of the true navigation graph?
- Can sampling several structural futures improve path ranking beyond the prior-mean predictions used in the reported experiments?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs Matterport3D RGB/semantic meshes, R2R graph paths and splits, Habitat rendering, the two training schedules and the separate VLN ranker. FID uses 10,000 samples per step, obtained with horizontal rolls and flips; checkpoint evaluation occurs every 2,000 steps. Hardware, runtime and pinned software versions are not specified. e07e12e14e15e19
Reader-proposed checks: first repeat the teacher-forcing/recurrent comparison with matched training budgets and repeated seeds; then freeze candidate paths and the ranker while swapping generated, blank, repeated and true future images. Keep split, context and horizon fixed to isolate each mechanism. e10e12e13e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does recurrent training improve held-out rollouts?
Reader-proposed experiment, not performed: compare teacher-forcing-only structure training with the Appendix C curriculum, matching initialization, total epochs, data order, context schedule and compute budget across repeated seeds. Fix one context observation and use the prior mean at evaluation. Measure per-step mIOU and the 1–6-step aggregate on seen and unseen buildings, with a checkpoint-selection split distinct from the final evaluation set. The reported pattern predicts a stronger recurrent advantage on seen buildings than on unseen ones. A consistent unseen gain across seeds would challenge that observed asymmetry; no reproducible seen gain would weaken the claimed curriculum benefit. e08e09e10e14e15
Check 2: Does predicted visual content change path ranking usefully?
Reader-proposed experiment, not performed: freeze the VLN proposal model, compatibility ranker, three-step candidate sets and graph geometry. At matched decision states, swap only future RGB among Pathdreamer, repeated, blank and ground-truth panoramas. Record ranking changes and the outcomes of executing the selected first actions; then compare complete episodes under the same replanning protocol. Add shuffled Pathdreamer images from other candidate paths as a control for scene-specific information versus merely realistic imagery. Useful predictive content should outperform both noninformative baselines and the shuffled-image control. If shuffling preserves the gain, the proposed explanation through route-specific visual foresight would be weakened. e12e13e17
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, all Figures 1–12, Tables 1–3, Equations (1)–(3), method and evaluation pages, and the complete included implementation/qualitative appendix. The six final crops were each opened and checked for readable labels, complete tables and faithful bounds. Figure 2's shared-weight and no-gradient markers were checked against Section 3.1; its cross-stage flow is qualified by the separate-training description. Its feature-matching label is qualified by Appendix C's zero weight. Table 2's GT and SG conditions and Figure 5's context-dependent bands were checked explicitly. Page 8 was inspected at both 200 and 450 DPI, and Table 3 was cropped from the latter render. The page and enlarged crop agree: the three-step Pathdreamer row has NE 5.32, SR 50.4, SPL 47.3, nDTW 61.8 and sDTW 44.4. Bibliography pages 9–11 were read as text but were not rendered for visual review. External videos, code and separate supplements remain outside the inspection.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Pathdreamer
- 3.1. Structure Generator: Segmentation & Depth
- 3.2. Image Generator: RGB
- 3.3. Training and Inference
- 4. Experiments
- 4.1. Pathdreamer Results
- 4.2. VLN Results
- 5. Conclusion
- References
- Appendix A. Qualitative Results, including A.1–A.3
- Appendix B. Generated Videos
- Appendix C. Implementation Details
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read individually, covering the complete 18-page PDF. Original figures, tables and equations were also visually inspected; the extraction-only visual omission is addressed by this edition.
- Identity/edition note: the inspected source is arXiv:2105.08756v2, 16 August 2021, with the exact catalog title and all five authors. The catalog names ICCV 2021; the venue edition and other arXiv revisions were not supplied for comparison, so equivalence or revision differences beyond the observed version cannot be established.
- The externally linked videos and code were not inspected; no experiments were reproduced. No separate supplement beyond the PDF's included appendices was supplied.
- Figures 11–12 retain 'Prescient Predictions' labels despite Pathdreamer captions and appendix text. The supplied source does not explain this alternate label.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and arXiv margin
The title and five authors match the catalog. Affiliations are Google Research and University of Michigan. The supplied edition is arXiv:2105.08756v2, 16 August 2021.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1
The task is to predict indoor panoramic observations in unseen buildings, including room reveals, while preserving consistency with previous observations.
Go to primary source ↓e03PDF p. 3, Section 3
Inputs are RGB, segmentation, depth, previous camera poses and camera intrinsics; future poses condition predictions. Point clouds provide context to both stages.
Go to primary source ↓e04PDF pp. 3–4, Section 3.1 and Figure 2
A RedNet-based encoder-decoder uses projected semantic/depth guidance, skip connections and spatial latent noise. A posterior supplies training noise; the learned prior supplies inference noise. Predicted semantics are back-projected for future context.
Go to primary source ↓e05PDF p. 4, Equation (1) and following paragraph
Structure training combines semantic cross-entropy, depth absolute error and posterior-to-prior KL divergence, weighted 1, 100 and 0.5.
Go to primary source ↓e06PDF pp. 4–5, Section 3.2, Figure 2 and Equations (2)–(3)
The Image Generator uses seven Multi-SPADE blocks, previous RGB and projected RGB guidance with partial convolutions; training uses ground-truth structure, a PatchGAN discriminator and a pretrained VGG-19 perceptual network.
Go to primary source ↓e07PDF p. 5, Section 3.3, Dataset, Trajectories and Training
Training uses Matterport3D and 400k graph-sampled trajectories of 5–8 panoramas. Stages train separately with 512×256 structural and 1024×512 RGB panoramas, using Habitat renders and stage-specific augmentation.
Go to primary source ↓e08PDF pp. 5–6, Section 3.3, Inference
Both stages use horizontal circular padding. Structural outputs are enlarged by nearest-neighbor interpolation for RGB generation. Quantitative experiments set the noise tensor to the prior mean.
Go to primary source ↓e09PDF p. 6, Section 4 and Section 4.1
Evaluation uses 340 Val-Seen and 783 Val-Unseen R2R paths; predictions span 1–6 steps, approximately 2–13m. Semantic quality uses mIOU; RGB quality uses FID.
Go to primary source ↓e10PDF pp. 6–7, Section 4.1, Table 1 and Figure 5a
At one context observation, 1–6-step mIOU is 65.9 recurrent versus 59.2 teacher forcing on Val-Seen, but 50.9 versus 50.8 on Val-Unseen; unseen nearest-neighbor performance is 30.6. Three-context unseen recurrent mIOU is 60.8.
Go to primary source ↓e11PDF p. 7, Table 2, Section 4.1 ablation definitions and Figure 5b; p. 8, Figure 6
With one context, Val-Unseen 1–6-step FID is 70.4 for Ours (SG), 90.8 without semantics/depth, 50.3 for SPADE (GT), and 39.9 for Ours (GT). Three-context Ours (SG) reaches 52.9. Figure 6 uses ground-truth structure.
Go to primary source ↓e12PDF pp. 7–8, Section 4.2
A separate VLN agent proposes beam-search paths; a dual-encoder instruction-trajectory model ranks them. Training uses true observations and truncated paths; inference executes the first action then replans. All conditions use ground-truth R2R graph geometry.
Go to primary source ↓e13PDF p. 8, Table 3, Plan Steps = 3 rows and accompanying discussion
Val-Unseen success rates are 50.4% for Pathdreamer, 40.6% for repeated panoramas, 41.9% for blank panoramas, and 59.3% for ground-truth look-ahead. Corresponding SPL scores are 47.3, 37.7, 38.8 and 55.8. The complete three-step Pathdreamer row, immediately above Ground truth, reads NE 5.32, SR 50.4, SPL 47.3, nDTW 61.8 and sDTW 44.4.
Go to primary source ↓e14PDF p. 12, Appendix C, Structure Generator and Image Generator
Structure training uses batch 64 for 50 epochs, initially 30 teacher-forcing epochs, then recurrent training. RGB training uses batch 128 for 500 epochs, two discriminator updates per generator update, EMA decay 0.999, and loss weights GAN=1, VGG=0.07, feature matching=0.
Go to primary source ↓e15PDF p. 12, Appendix C, Evaluation Details; p. 7, Figure 5 caption
FID uses 10,000 augmented samples per prediction step. Evaluation occurs every 2,000 training steps with best checkpoints selected separately on each reported validation split. Figure 5 bands describe variation across 1–3 context observations.
Go to primary source ↓e16PDF p. 5, Figure 3 and caption; p. 4, Section 3.1
Different latent values produce alternative semantic layouts and corresponding RGB room reveals; the lower example misses the ground-truth matching lamp.
Go to primary source ↓e17PDF p. 14, Figure 8(c) and caption
A randomly selected Val-Unseen rollout misses a room entrance at 2.1m and progressively diverges from the true scene while retaining plausible-looking outputs.
Go to primary source ↓e18PDF p. 12, Appendix A; pp. 13–18, Figures 7–12 and captions
Appendix examples distinguish selected from random sequences, seen from unseen buildings, and varied noise. Figures 11–12 contain the internal label 'Prescient Predictions' although captions and surrounding text identify Pathdreamer.
Go to primary source ↓e19PDF p. 8, Section 5; p. 12, Appendices B–C
Other navigation tasks are proposed as future work; video results are linked externally. The implementation appendix provides optimization and evaluation settings but no hardware model/count, wall-clock budget or pinned software versions.
Go to primary source ↓8.5 Primary sources
Pathdreamer: A World Model for Indoor Navigation ↗
PDF · 10,555 extracted words
Source fingerprint
5b66b257dcca796e646e4c80fc283d50321b1e244074edac6284f01baab52acb