DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation
1. Paper overview
In one sentence: A learned panoramic world model makes short-horizon waypoint search useful for continuous navigation, while prediction errors and goal-distance estimation limit the benefit of imagining further. e02e03e04e05e06e07e11e12e15e18
| At a glance | What to know |
|---|---|
| Research problem | Source description Continuous vision-language navigation requires an agent to follow instructions through partially observed environments without a supplied navigation graph. Actions are 15° left/right turns, 0.25 m forward moves, and stop. Success requires stopping within 3 m of the goal within 500 steps. DREAMWALKER addresses the risk of selecting locally attractive waypoints without anticipating what comes next. e02e03 |
| Core mechanism | |
| A key reported result | VLN-CE test navigation: 49%; 44; 5.48 m; 11.8 m SR; SPL; NE; TL. 3,408 trajectory-instruction pairs in 18 held-out scenes; simulated low-level execution. Sim2Sim: 44%, 37, 6.17 m, 11.4 m. BridgingGap: 42%, 36, 5.89 m, 13.3 m. The SR improvement is five percentage points over Sim2Sim and seven over BridgingGap. It is navigation performance in simulation, not physical deployment. e02e10e11 |
| Reading caution | Author claim Occlusion, distant viewpoints, and waypoints inside walls produce sparse guidance and blurred or incorrect synthesis. Only 61 training scenes support the world model. Collision can separate predicted and reached coordinates; larger search spaces increase computation. e16 |
Core contributions
- Reader analysis
The authors present imagined panoramas and search trees as explanations of navigation decisions. These are qualitative demonstrations of inspectability, without a measured human-interpretability study. e19
Figure I. Geometry supplies a partial view; learned synthesis fills the unseen content used for planning. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: panoramic RGBD subviews enter the waypoint network, which produces a heatmap and candidate positions. Filled blue circles denote visited waypoints; yellow rings denote unvisited ones. Follow the candidate translation downward into the scene synthesizer. The lower branch converts a panorama and its RedNet semantic estimate into a point cloud, reprojects it from the candidate position, and feeds guidance images into a generator. The synthesized panorama then moves upward into the graph in 'Mind'. The bidirectional equirectangular arrow reconciles the subview and panoramic representations; it does not signify executed movement. Actual navigation happens after the separate planner selects an action. e03e04e05e09e16
What it supports. The world model has explicit geometry and memory as well as learned image completion. Its output is a hypothetical observation associated with a candidate waypoint, enabling another round of imagined expansion. The source describes the synthesizer as independently trained, so the diagram supports a modular model-and-planner interpretation rather than one joint action decoder.
Where the evidence stops. Figure I labels guidance as RGB/semantic, while §I also specifies depth guidance and the original RGB input to the generator. Those textual inputs are not fully drawn. Sparse projections and mesh-intersecting waypoints remain documented failure cases.
2. Motivation
2.1 The problem and the proposed response
Continuous vision-language navigation requires an agent to follow instructions through partially observed environments without a supplied navigation graph. Actions are 15° left/right turns, 0.25 m forward moves, and stop. Success requires stopping within 3 m of the goal within 500 steps. DREAMWALKER addresses the risk of selecting locally attractive waypoints without anticipating what comes next. e02e03
2.2 What this reading follows
Following an instruction through an unfamiliar building requires choosing paths before seeing what lies around the next corner. DREAMWALKER makes those choices by maintaining a waypoint graph, generating possible future panoramas, and scoring imagined continuations with an instruction-conditioned distance estimator. Its reported gains concern low-level navigation in simulated continuous environments. The central reading question is how useful search survives imperfect visual predictions. Read the architecture and MCTS diagrams first, then separate ordinary benchmark performance from privileged-information ablations. Finally, compare planning depth with synthesis drift. Several inconsistencies in the published equations and implementation descriptions matter for anyone trying to reproduce the reported behavior. e02e03e04e05e06e07e11e12e15e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
A separately trained observation-generating world model feeds an explicit MCTS planner and learned distance estimator, supporting Dual-system and Other mechanisms. Actions are selected by search, with no joint future/action decoder or learned inverse-dynamics module specified. Thus Outside quadrants is reasonable. Navigation and the visual-planning part of 'Visual planning & IDM' fit; the IDM wording should not imply an implemented inverse model. e03e04e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a continuous building into a graph that can be extended in imagination
The agent begins with an instruction and panoramic RGBD observations, not a preloaded navigation graph. A waypoint predictor gives nearby candidate positions; the environment graph records observations at locations actually reached and their geometric relationships. To reason about a candidate before moving, the scene synthesizer first uses depth to reproject known content into that viewpoint. Learned generation completes the missing regions, producing a panorama that can itself support another predicted waypoint. This separates the current episode's memory from general scene knowledge learned during training. In the supplement, semantic information comes from RedNet and accompanies the geometric synthesis pipeline. The distinction between actual and imagined observations is essential: an appealing generated doorway is a hypothesis, and collision can make the reached location differ from the intended waypoint. e02e03e04e09e16
Figure 3. Search evaluates several imagined continuations before committing to one root action. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right. Each small graph is a whole possible environment state, not a single physical location. Selection follows maximum UCT, balancing estimated action value against visit-count exploration. 'Extension' in the graphic corresponds to expansion in the text: a new leaf receives an imagined state. Rollout samples additional waypoint transitions using the distance function, with K marking the maximum depth. The right-hand upward arrow backs the resulting reward through the tree. The outer loop repeats this process. After search, Eq. (7) selects the root edge with greatest Q; the exploration bonus is used for search selection, not the final action. e05e06e07e18
What it supports. Imagined futures directly participate in inference. The system executes the selected waypoint through low-level actions, refreshes its real graph after a first visit, and plans again. Selecting an already visited waypoint instead triggers movement there followed by stop, a specific termination mechanism that a reproduction must preserve.
Where the evidence stops. The arrows agree with the four-phase description, but reward/rollout signs in Eqs. (11)–(12) conflict with distance-reduction intuition. Algorithm I also differs from Eqs. (4),(6) in exploration scaling and backup. The graphic does not resolve those numerical ambiguities.
5.2 Teach a distance estimator to score graphs containing synthetic observations
A graph attention network combines visual waypoint features, geometric edges and the instruction; a small MLP estimates remaining goal distance for each node. Training progressively builds graphs along target-directed trajectories and adds random accessible alternatives, so the estimator sees more than one narrow demonstration path. It first learns from real observations, then receives graphs with some observations replaced by the synthesizer's outputs. At inference, this estimator helps score and sample imagined continuations inside MCTS. The search expands candidate states, estimates rollout value and propagates it toward the root before selecting a waypoint. The source's intended role is to prefer progress toward the goal. Its printed positive distance softmax and distance-increase reward remain inconsistent with that role, so a faithful implementation requires resolving the signs rather than assuming a correction. e05e06e07e08e18
5.3 Separate the value of prediction from the value of a good goal score
The ablations probe two different bottlenecks. Perfect Imagination changes the observations available to search, while ground-truth distance replacement changes how imagined outcomes are scored. Neither intervention is available to the ordinary deployed policy. Reader analysis: their benefits suggest that plausible visual futures are useful only when the scorer ranks their relevance to the instruction correctly. The greedy comparison provides evidence for lookahead within this architecture, but does not establish that every world model needs MCTS. Likewise, the horizon sweep and rising FID support a short planning horizon without proving that image error alone causes the six-step decline. A stronger causal test would independently vary observation accuracy and distance accuracy while holding waypoint candidates, search budget and evaluation episodes fixed, then check which intervention restores the benefit of deeper planning. e12e13e14e15
5.4 Training and inference
During training
The waypoint predictor and synthesizer follow referenced training schedules rather than a reproduced joint objective. Distance regression uses progressively enlarged training graphs, adding the best next waypoint and up to five random accessible waypoints. Training uses real observations for 10 epochs, then mixes synthesized observations for 10 more; AdamW uses learning rate 2.5×10⁻⁵ and batch size 16. e08
The supplement freezes RGB/depth ResNet-50 encoders, whereas §4.2 names ResNet-18 for RGB and ResNet-50 for depth. Language uses GLoVE and a bidirectional LSTM; the distance MLP has 1,024 hidden neurons. The RGB discrepancy remains unresolved. e08e09
During inference
Default planning uses 50 search iterations, horizon four, discount 0.98, and exploration constant 1.0. The observation interface comprises 12 RGBD views separated by 30°, with 224×224 RGB, 256×256 depth, and 90° vertical field of view. Evaluation enables sliding. e08e09
5.5 Implementation flow
- Build episodic memory
The environment graph starts at the initial location and adds reached waypoints. Node embeddings encode panoramic observations; edges encode angle and relative position, with connectivity established by mutual waypoint detection. A 120-angle by 12-distance heatmap proposes nearby waypoints; supplementary NMS limits candidates to five. e02e03e09
- Synthesize an unvisited view
The scene synthesizer unprojects RGBD and semantic information into a point cloud, translates to a candidate waypoint, and reprojects sparse guidance images. A generator fills missing content. RedNet supplies initial semantics; equirectangular projection connects separate subviews to a panorama. Synthetic observations permit further hypothetical waypoint expansion. e04e09
- Evaluate imagined progress
A graph attention network fuses waypoint observations, edges, and instruction embeddings. An MLP predicts each node's distance to the goal; the state distance is the minimum prediction over its graph. These predictions supply rollout guidance and rewards rather than actual goal access at deployment. e06e07
- Search, act, and observe again
MCTS selects branches by UCT, expands a leaf, rolls out imagined transitions, and backs up values. The root action with greatest Q is executed. Reaching an unvisited waypoint updates memory and triggers replanning; selecting a previously visited waypoint means navigating there and stopping. e05e07
6. Experiments & results
DREAMWALKER builds an episodic waypoint graph, synthesizes future panoramic observations, and searches those imagined states before executing navigation actions. It reports 49% test success on VLN-CE, with a short planning horizon limiting both computation and prediction drift. The evidence supports simulated navigation benefits; conflicting implementation descriptions leave exact reproduction unresolved.
6.1 Read the original evidence
Table 1. DREAMWALKER improves reported success across the three splits, with distinct baseline gaps. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose a split, then compare matching metric columns within that block. SR is success rate, OR is oracle success rate, SPL weights success by path length, NE is navigation error, and TL is traveled length. The source prioritizes SR; lower NE and higher SR, OR and SPL are preferred. For held-out test scenes, compare DREAMWALKER's last row with Sim2Sim and BridgingGap. Keep success improvements separate from path cost: DREAMWALKER's test TL is 11.8 m, between those baselines' 11.4 and 13.3 m. These are continuous-environment simulator episodes, with the success criterion defined by stopping near the target. e02e10e11
What it supports. On test, DREAMWALKER reports 49% SR and 44 SPL, compared with Sim2Sim's 44% and 37 and BridgingGap's 42% and 36. These are gains of five and seven SR percentage points respectively. On unseen validation, its 49% SR exceeds BridgingGap's 44%.
Where the evidence stops. Table 1's original caption mistakenly describes component impacts. Its seen SR values imply a nine-point gain over BridgingGap, although §4.3 says seven. No seeds or uncertainty are reported; identical baseline compute and evaluator configurations are not established.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| VLN-CE test navigation 3,408 trajectory-instruction pairs in 18 held-out scenes; simulated low-level execution. | 49%; 44; 5.48 m; 11.8 m SR; SPL; NE; TL | Sim2Sim: 44%, 37, 6.17 m, 11.4 m. BridgingGap: 42%, 36, 5.89 m, 13.3 m. The SR improvement is five percentage points over Sim2Sim and seven over BridgingGap. It is navigation performance in simulation, not physical deployment. e02e10e11 |
| VLN-CE validation navigation Val seen: 778 pairs/53 scenes; val unseen: 1,839 pairs/11 scenes. | Seen: 59%, 48. Unseen: 49%, 44. SR; SPL | BridgingGap: seen 50%, 44; unseen 44%, 39. Table 1 implies gains of nine and five SR percentage points. The prose's seven-point seen gain disagrees with the table. No uncertainty is supplied. e10e11 |
| World-model and planning ablation VLN-CE val unseen; Table 2 variants. | DREAMWALKER: 49%, 44. SR; SPL | Perfect Imagination: 54%, 49; Copy Memory: 27%, 24; Greedy Selection: 42%, 36. Looking ahead helps relative to this greedy control. Perfect Imagination substitutes actual future observations and is an oracle diagnostic, not deployable performance; Copy Memory changes prediction quality, not only search. e12 |
| Planning horizon and computation VLN-CE val unseen; 50 search iterations. | Horizon 0: 42%, 0.09; 2: 47%, 1.15; 4: 49%, 1.43; 6: 48%, 2.05. SR; runtime (s/step) | Separately, 70 search iterations give 50% SR and 1.74 s/step versus 49% and 1.43 at 50. Deeper search has diminishing returns. Figure 5 shows increasing synthesis FID with prediction step; it supports, but does not causally prove, the authors' error-accumulation explanation. e14e15 |
| Oracle distance replacement diagnostic Random replacement of distance estimates by ground truth; §4.6 does not explicitly name the split. | Reported 70% at 20% replacement; 100% with perfect distance estimates. SR | Figure 4 starts near the ordinary agent's roughly 49% SR with no replacement. Privileged goal information exposes a scoring bottleneck. The unspecified split and oracle access prevent treating this as an additional ordinary benchmark result. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Oracle views, memory copying, and greedy decisions expose different limits of imagined planning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row as the ordinary learned system. 'Perfect Imagination' replaces predicted future scenes with the actual observations at those locations, granting information the ordinary agent cannot access. 'Copy Memory' uses the nearest stored observation as its future prediction. 'Greedy Selection' chooses a currently visible waypoint from estimated goal distance without explicit lookahead planning. Read the unseen block for generalization: its SR column is 49, 54, 27 and 42 from top to bottom. Then inspect SPL in the same block to check whether better success comes with useful path efficiency. Keep these interventions conceptually separate: they change different parts of the decision process. e12
What it supports. DREAMWALKER exceeds the greedy control by seven unseen SR percentage points and eight SPL points. Copying memory is substantially worse. Replacing synthesis with actual future views adds five SR points, suggesting remaining headroom in the world model under this particular planner and distance estimator.
Where the evidence stops. Perfect Imagination is an oracle diagnostic, not an attainable deployment result or universal upper bound. Copy Memory changes the quality and content of predictions, so its gap does not isolate the value of MCTS alone. The table supplies no uncertainty.
Table 4. Four imagined steps give the strongest tested horizon tradeoff on unseen validation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first substantive column sets the maximum imagined trajectory length. Table 4's caption fixes the maximum search rounds at 50, so the primary intervention is planning horizon. Horizon zero matches the greedy setting: no future rollout is used. Compare the unseen SR and SPL columns before looking at runtime on the far right. Moving from zero to four steps increases SR from 42 to 49 and SPL from 36 to 44. Going further to six steps reduces these to 48 and 41 while runtime rises. Runtime is reported per step; the source does not fully specify how that timing was measured. e08e12e14e15
What it supports. In this sweep, more imagination helps up to four steps, then costs more without improving navigation. The six-step configuration takes 2.05 s/step against 1.43 for four steps. This supports a bounded-horizon design choice, while Figure 5 motivates the possibility that poorer distant predictions help explain the reversal.
Where the evidence stops. Horizon changes both prediction depth and computation. No confidence intervals establish whether a one-point SR difference is reliable. The training GPU is specified elsewhere, but the runtime table does not explicitly identify its measurement hardware.
Figure 5. Longer imagined trajectories drift further from the distribution of observed panoramas. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is prediction step, and the vertical axis is Fréchet Inception Distance between synthesized views and corresponding ground-truth scenes as described in §4.7. The yellow dashed line is the seen setting and the green solid line is unseen. Follow each curve along the horizontal axis: both generally rise as predictions extend further, with brief flat stretches. The unseen curve remains above the seen curve. Compare this tendency with Table 4 rather than interpreting a plotted FID value as an episode success probability. The graph is a diagnostic of synthesis fidelity; it does not show the selected action or the correctness of a particular route. e15e16
What it supports. The observed trend is consistent with recursive prediction accumulating error, and unseen scenes are harder for synthesis throughout the displayed sweep. Together with the horizon table, it motivates limiting imagined depth. The paper attributes the weakening of long-horizon planning to overloaded world-model capacity, but the plot alone does not isolate that cause.
Where the evidence stops. FID is not a direct test of obstacle geometry, goal-distance calibration, or executable paths. This figure provides neither error bars nor a complete FID sampling and feature-extraction protocol. Exact point values are not tabulated, so this reading retains the qualitative trend.
7. Analysis & limitations
7.1 What the evidence leaves open
Occlusion, distant viewpoints, and waypoints inside walls produce sparse guidance and blurred or incorrect synthesis. Only 61 training scenes support the world model. Collision can separate predicted and reached coordinates; larger search spaces increase computation. e16
All navigation is in static virtual environments. Moving objects, lighting changes, crowding, physical transfer, and inclusive language/accessibility support remain future concerns. e17
Tables 1–4 omit seeds and uncertainty, and the cross-method comparison does not establish equal training compute or identical evaluator configurations. Visual plausibility and selected successful trajectories alone cannot establish reliable dynamics or calibrated explanations. e11e12e14e15e19
7.2 Questions for discussion
- Would improved synthesis preserve its benefit when distance-estimator accuracy and search cost are controlled?
- Which reward signs and backup rules produced the published results?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires Matterport3D/VLN-CE data, pretrained perception and synthesis components, and graph-distance training. The source names PyTorch, Habitat, and one NVIDIA RTX 3090 for training, but gives no software versions, total training time, or explicit hardware attribution for the runtime table. e08e09e14
Resolve RGB backbone choice, reward/rollout signs, and backup implementation first. Algorithm I uses an extra factor two inside the UCT square root and a depth-discounted total-reward backup, unlike Eqs. (4),(6). The stated L2 objective is printed as an unsquared norm; replacement probability, gradient-clipping threshold, and detailed inherited synthesizer/predictor losses are unspecified here. e05e07e08e09e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Resolve reward signs and MCTS backup semantics before benchmark training
Reader-proposed check, not executed: construct a small deterministic waypoint graph with known goal distances, one progress branch, one regress branch, and stopping states on either side of the 3 m threshold. Compare the printed Eqs. (11)–(12) with explicitly sign-reversed progress variants, holding candidate order, visits, random seeds and rollout depth fixed. Record rollout probabilities, backed-up Q values and selected root actions. Separately compare Eq. (6)'s recursive backup with Algorithm I's depth-discounted accumulation; keep the UCT coefficient matched when testing its factor-two difference. A progress-consistent variant should favor the nearer branch and valid stop. Failure to do so would falsify the proposed correction; success would still not identify which implementation generated the paper's results. e05e06e07e18
Check 2: Factor observation accuracy and distance accuracy across planning horizons
Reader-proposed check, not executed: on the same VLN-CE val-unseen episodes, compare learned versus actual future observations crossed with learned versus ground-truth distances, at horizons two, four and six with 50 search rounds. Freeze candidate waypoint sets for each matched search state to prevent improved synthesis from silently changing the action space; use the same initial graphs and random seeds. Report SR, SPL, distance-estimation error, decision latency and paired episode uncertainty. The actual-observation and ground-truth-distance conditions are diagnostics only. If prediction drift limits depth, actual observations should reduce the four-to-six-step loss with the scorer held fixed. If oracle distances remove that loss while actual views do not, scoring rather than image fidelity is the stronger bottleneck. e10e12e13e14e15
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author page; all main and supplementary method, training, evaluation and discussion pages; Figures 1–6 and I–V; Tables 1–4; and Algorithm I. All six final crops were viewed with readable labels, axes and legends. Figure I's abbreviated input labels were checked against supplementary §I; Figure 3's arrows and selection rule were checked against Eqs. (4)–(13) and Algorithm I. Equation/pseudocode, backbone and results-prose discrepancies are preserved in the reading. References on pp. 17–19 were read as text; cited external works and the linked code were not inspected. No separately hosted supplement or physical experiment was reviewed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (PDF p. 1)
- 1. Introduction (pp. 1–2)
- 2. Related Work (pp. 2–3)
- 3. Methodology, including §§3.1–3.2 and Eqs. (1)–(13) (pp. 3–6)
- 4. Experiments, §§4.1–4.8 (pp. 6–8)
- 5. Conclusion and Discussion (p. 8)
- Embedded Supplementary Material: I. Implementation Details and Algorithm I (pp. 9–11)
- Embedded Supplementary Material: II. Qualitative Results and Figures II–V (pp. 9, 11–15)
- Embedded Supplementary Material: III. Discussions, Limitations and Social Impact (pp. 11, 16)
- References (pp. 16–19)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The reviewed artifact is arXiv:2308.07498v1 [cs.CV], dated 14 August 2023. Its title and all four authors match the catalog apart from typographic title spacing/capitalization. The catalog identifies ICCV 2023; equivalence to a separate proceedings edition or later revision was not checked.
- Text extraction does not reconstruct figure images; the retained PDF was rendered and all Figures 1–6 and I–V and Tables 1–4 were visually inspected.
- Separate supplemental material availability has not been fully verified. The supplement embedded in this 19-page PDF was read in full.
- No external references, repository, checkpoints, or separately hosted material were inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin stamp
DREAMWALKER title; authors Hanqing Wang, Wei Liang, Luc Van Gool, Wenguan Wang; four affiliations; arXiv:2308.07498v1, 14 August 2023.
Go to primary source ↓e02PDF p. 3, §3 task setup and Waypoint Action Space
Actions use 15° turns, 0.25 m forward movement and stop; success within 3 m and 500 steps. Waypoint heatmap is 120 angles by 12 distances, 3° bins and 0.25–3 m range.
Go to primary source ↓e03PDF pp. 3–4, §3.1 Environment Graph, Eqs. (1)–(2), Figure 2(a–g)
The episodic graph stores reached waypoint observations and geometric relations; mutual waypoint validity defines connections. Synthesizer and graph support discrete planning in continuous navigation.
Go to primary source ↓e04PDF p. 4, §3.1 Scene Synthesizer, Eq. (3), Figure 2(h–j); p. 11, Figure I
A separately trained synthesizer uses observations and geometry-aligned reprojections to imagine unvisited panoramas. Figure I traces waypoint prediction, translation, point-cloud projection, generation, and hypothetical graph updates.
Go to primary source ↓e05PDF p. 5, §3.2, Figure 3 and Eqs. (4)–(7)
MCTS selection, expansion, rollout and backup; UCT exploration during search; highest root Q for execution; first visits update the graph, while selection of a visited waypoint terminates navigation after reaching it.
Go to primary source ↓e06PDF p. 5, §3.2 Reward, Eqs. (8)–(10)
GAT and MLP estimate instruction-conditioned waypoint-to-goal distances. D is their minimum across the graph. Stop reward is +5 within predicted distance 3 and −5 otherwise.
Go to primary source ↓e07PDF p. 6, §3.2, Eqs. (11)–(13) and surrounding reward/rollout prose
Printed reward is D(s′)−D(s), although prose describes getting closer; rollout applies softmax to positive F_d distances. Rollout ends at a visited waypoint or depth K; values sum discounted rewards.
Go to primary source ↓e08PDF p. 6, §4.2 Network Architecture, Network Training and Reproducibility; p. 9, §I Learning of Distance Function and Mental Planning
Distance training expands graphs and mixes synthesized observations; 10+10 epochs, AdamW 2.5e−5, batch 16. Main text supplies language encoder, MLP size, horizon 4, 50 iterations, γ=0.98, PyTorch/Habitat and one RTX 3090; supplement sets C=1.
Go to primary source ↓e09PDF p. 9, §I Experimental Configurations and Details of World Model; p. 6, §4.2; p. 11, Figure I
Twelve RGBD subviews, resolutions, FOV, frozen ResNet-50 backbones, sliding, five NMS waypoints, equirectangular conversion and RedNet semantics. Main text instead specifies RGB ResNet-18. Supplement text lists RGB/depth/semantic guidance; Figure I labels guidance RGB/semantic.
Go to primary source ↓e10PDF p. 6, §4.1 Dataset and Evaluation Metric; p. 9, §I Experimental Configurations
VLN-CE has 16,844 pairs over 90 scenes; train 10,819/61, seen validation 778/53, unseen validation 1,839/11, test 3,408/18. Metrics include NE, TL, SR, OR and SPL; sliding is enabled.
Go to primary source ↓e11PDF p. 7, Table 1, DREAMWALKER/BridgingGap/Sim2Sim rows, all splits; p. 6, §4.3
DREAMWALKER SR is 59/49/49 and SPL 48/44/44. BridgingGap SR is 50/44/42; Sim2Sim test SR/SPL is 44/37. Table values conflict with the prose's seven-point seen gain over BridgingGap. Table 1's caption incorrectly describes component impacts.
Go to primary source ↓e12PDF p. 7, Table 2 and §§4.4–4.5
Unseen SR/SPL: DREAMWALKER 49/44, actual-future-view oracle 54/49, nearest-memory copying 27/24, greedy distance-based selection 42/36. Seen oracle SR/SPL is 64/60.
Go to primary source ↓e13PDF p. 7, §4.6 and Figure 4
Ground-truth distance replacement improves the plotted SR; prose reports 70 at 20% replacement and 100 with perfect distances. This subsection and caption do not explicitly identify the evaluation split.
Go to primary source ↓e14PDF p. 8, Table 3; p. 7, §4.7 Search Iterations
Iterations 10/30/50/70 yield unseen SR 44/46/49/50 and runtime 0.43/1.08/1.43/1.74 s/step. Runtime attribution lacks an explicit hardware/methodology specification.
Go to primary source ↓e15PDF p. 8, Table 4, Figure 5 and §4.7 Horizons of Mental Planning
At 50 search rounds, horizons 0/2/4/6 yield unseen SR 42/47/49/48 and runtime 0.09/1.15/1.43/2.05. Seen and unseen FID generally increase with prediction step; the authors suggest accumulated synthesis error limits useful depth.
Go to primary source ↓e16PDF p. 11, §II Qualitative Results and §III Limitations; pp. 13–15, Figures III–V
The source identifies occluded, distant and mesh-intersecting waypoints as synthesis failures; collision alters reached coordinates. Limited 61-scene training and increased planning cost constrain the method. Collision checks and alternative synthesis are proposed future work.
Go to primary source ↓e17PDF p. 16, §III Limitations and Social Impact
Static virtual evaluation leaves dynamic-world robustness and physical transfer unresolved. The discussion also identifies language and accessibility needs as future considerations.
Go to primary source ↓e18PDF p. 10, Algorithm I, _backup and _getBestChild; p. 5, Eqs. (4),(6)
Pseudocode accumulates rollout reward multiplied by gamma to node depth and uses sqrt(2 log parent-visits / child-visits); displayed equations use a different recursive backup and omit the factor two.
Go to primary source ↓e19PDF p. 8, Figure 6 and §4.8; p. 12, Figure II; p. 11, §II
Selected navigation examples pair actual/imagined panoramas with routes and search values. The authors interpret these as transparent planning; no quantitative human interpretability evaluation is presented.
Go to primary source ↓8.5 Primary sources
DREAMWALKER : Mental Planning for Continuous Vision-Language Navigation ↗
PDF · 11,523 extracted words
Source fingerprint
927fc75286d9f782803bb1508407e3351956ff93a38dbd7aef0e3b976faf7500