Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
1. Paper overview
In one sentence: Removing the navigation graph exposes long-horizon control and observation failures that instruction-following scores can otherwise hide. task-motivationtask-interfacetrajectory-conversionattention-policytraining-resultscomparison-caveats
| At a glance | What to know |
|---|---|
| Research problem | Source description A panorama graph silently supplies traversability, reliable movement between nodes, and precise localization. Consequently, strong instruction-following scores can conceal unresolved obstacle avoidance and observation-selection problems. VLN-CE makes these responsibilities part of the task while retaining R2R instructions, allowing researchers to examine the interface between language understanding and control. task-motivationtask-interface |
| Core mechanism | Source description The benchmark transfers R2R trajectories into Habitat/Matterport3D and provides 4,475 trajectories from train and validation splits. The average reference path becomes 55.88 low-level actions, substantially extending the decision horizon. dataset-horizon |
| A key reported result | Instruction following in unseen continuous environments: 0.32 (32%); 0.30; 7.37 m; 0.51 SR; SPL; NE; nDTW. VLN-CE val-unseen; cross-modal PM + DAgger fine-tuning + augmentation, Table 3 row 10. Cross-modal teacher forcing: SR 0.23, SPL 0.22; Seq2Seq teacher forcing: SR 0.20, SPL 0.18. The full recipe improves unseen performance, but combines architecture and training choices. These are reported point estimates, without uncertainty intervals. training-resultstraining-protocol |
| Reading caution | Reader analysis Only navigable converted trajectories enter the continuous dataset. Mesh holes and moved objects cause exclusions, so the retained task cannot be assumed representative of every original R2R route. trajectory-conversion |
Core contributions
- Source description
The benchmark transfers R2R trajectories into Habitat/Matterport3D and provides 4,475 trajectories from train and validation splits. The average reference path becomes 55.88 low-level actions, substantially extending the decision horizon. dataset-horizon
- Source description
Sequence-to-sequence and cross-modal attention baselines, modality removals, and training-regime comparisons establish which existing navigation techniques carry into the continuous setting. policy-encodersmodality-resultstraining-results
Figure 2. Transferring a panorama route requires both reachable waypoints and connected free space. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right. Panel (a) measures horizontal displacement for nodes that needed adjustment; the dashed marker is 0.19 m, not an error averaged over every original node. The caption says only 3% needed this adjustment. Panel (b) shows why a plausible panorama connection can fail in the reconstructed mesh: missing surfaces and furniture can disconnect the route. Panel (c) shifts attention from geometry to decision horizon, marking 55.88 low-level actions per reference path. Section 3.1 supplies the missing procedural link: project candidate waypoints, then verify navigation between successive waypoints with an A*-based search. trajectory-conversiondataset-horizon
What it supports. The conversion succeeds for 98.3% of nodes but only 77% of trajectories. A collection of individually reachable points does not guarantee a complete route. The resulting dataset contains 4,475 train/validation trajectories, and their long action sequences expose control decisions that graph hops abstract away.
Where the evidence stops. The 0.19 m statistic is conditional on adjusted nodes. The histogram vertical axes are not explicitly named, so their heights should not be compared as common units. Removed routes also change which instructions are represented in the continuous benchmark.
2. Motivation
2.1 The problem and the proposed response
A panorama graph silently supplies traversability, reliable movement between nodes, and precise localization. Consequently, strong instruction-following scores can conceal unresolved obstacle avoidance and observation-selection problems. VLN-CE makes these responsibilities part of the task while retaining R2R instructions, allowing researchers to examine the interface between language understanding and control. task-motivationtask-interface
2.2 What this reading follows
A route instruction is only useful if an agent can execute it through the space between landmarks. VLN-CE makes that space part of evaluation: an agent receives egocentric RGBD observations and chooses small movements inside reconstructed indoor scenes. This reading follows three connected questions: which original routes survive conversion, how a recurrent policy grounds language while moving, and which experimental gains survive the harder interface. The strongest model reaches 32% success in unseen continuous environments. The figures and tables also show why that result cannot be directly equated with either graph-navigation performance or physical robot reliability. task-motivationtask-interfacetrajectory-conversionattention-policytraining-resultscomparison-caveats
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The benchmark/simulator classification is supported: the principal contribution is a task and evaluation setting. Habitat supplies environment dynamics; the baselines predict actions from recurrent perceptual state, without a learned future-state rollout or inverse-dynamics action extraction. Their shared training objectives therefore do not establish a One Model world-action architecture. The architecture, prediction-paradigm, and quadrant fields appropriately remain not applicable. The physics-related subcategory is supported at the level of movement and collision simulation, without evidence here of validated contact dynamics. task-motivationtask-interfaceattention-policytraining-objectives
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First make the route physically navigable in the simulator
A panorama node records where a camera stood, which may be above furniture or at a height unsuitable for a ground robot. Section 3.1 therefore searches downward for nearby navigable mesh points and permits manual corrections. The collision body is a 1.5 m tall cylinder with 0.2 m diameter; horizontal displacement must remain below 0.5 m. After node conversion, an A*-based search checks whether each successive waypoint can be reached within 0.5 m. This second stage matters because valid endpoints can lie in disconnected mesh regions. My interpretation is that route feasibility is part of the benchmark definition, rather than a neutral preprocessing detail: excluding failed conversions changes the available instruction distribution before any policy is trained. trajectory-conversiondataset-horizon
Figure 3. The attention policy selects relevant words before selecting relevant image regions. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with panel (a): depth is flattened, RGB is mean-pooled, and the instruction LSTM supplies one summary to the policy GRU. In panel (b), the left GRU updates the visual-history state using current observations and the previous action. Its output queries the BiLSTM instruction states. Follow the attended instruction into the RGB and depth attention blocks, then into the right GRU together with the previous action and visual-history state. The downward dashed arrows carry recurrent state through time. Equations (3)–(7) on page 9 confirm this query direction and distinguish the visual-history state from the final action-policy state. policy-encodersseq2seq-policyattention-policytraining-results
What it supports. The structural change gives the policy a way to revisit particular words and spatial features as its observations change. Under teacher forcing, cross-modal attention reaches unseen SPL 0.22 versus Seq2Seq’s 0.18. The figure explains the information flow; Table 3 supplies that measured comparison.
Where the evidence stops. These are recurrent action policies, without a learned future-environment rollout in the displayed architecture. The figure names pretrained encoders but does not establish a frozen-parameter policy or the independent effect of each attention block.
5.2 Use history to choose words, then words to choose spatial evidence
The cross-modal policy addresses changing observations by retaining a visual-history state and all BiLSTM instruction states. Its first GRU consumes the current RGB/depth summaries and the previous action embedding. That state queries instruction attention; the selected instruction content then queries the spatial RGB and depth features. A second GRU combines the attended features, previous action, and visual-history state to predict the next action. The order is important: Eq. (5) makes the visual attention conditional on the attended instruction, rather than simply pooling the entire image again. At inference, the classifier’s argmax command is executed in Habitat and the observation changes. The representation is recurrent perceptual memory; the paper does not use it to simulate candidate future worlds before acting. policy-encodersattention-policytask-interface
5.3 Separate recovery experience from progress supervision
Teacher forcing teaches the policy what to do on reference paths, but does not expose it to the consequences of its own mistakes. DAgger addresses that distribution shift by collecting mixed oracle/policy rollouts and retaining data from every collection. Progress monitoring solves a different learning problem: it supervises an estimate of the fraction of the route completed. The authors find that this auxiliary signal alone hurts unseen SPL, yet helps within augmented pretraining followed by DAgger fine-tuning. They also report progress-monitor validation loss of 0.67 without augmentation versus 0.47 with it. This supports their overfitting interpretation, but my reading is that staged training and changed data volume remain coupled. The strongest row should be understood as a successful recipe, not proof that any one component is universally beneficial. training-objectivestraining-protocoltraining-resultsprogress-diagnostic
5.4 Training and inference
During training
Teacher forcing maximizes reference-action likelihood, using inflection weighting to emphasize changes rather than long runs of repeated forward actions. All experiments use coefficient 3.2. Adam uses learning rate 2.5 × 10⁻⁴, batches of five full trajectories, and at most 30 teacher-forcing epochs. training-objectivestraining-protocol
DAgger collects 5,000 trajectories per stage, mixes oracle and policy actions, and trains four epochs over all accumulated data. Training converges after 6–10 collections depending on the model. Early stopping uses val-unseen, so this split also selects checkpoints. training-protocol
The strongest cross-modal recipe pretrains with approximately 150,000 converted synthetic trajectories and a progress-monitor mean squared error loss, then fine-tunes with DAgger on original data. Progress predicts fraction through the trajectory; it is auxiliary supervision, not a separately specified test-time stopping controller. training-objectivestraining-resultsattention-policy
During inference
At each step, encode the current observation, update recurrent states, and choose the argmax action from the softmax classifier. Habitat executes the action and supplies the next observation. Position and heading are not policy inputs. The learned stop action terminates navigation; these experiments do not establish physical robot deployment. task-motivationtask-interfaceattention-policycomparison-caveats
5.5 Implementation flow
- Construct executable reference trajectories
Project panorama nodes toward navigable mesh locations using a downward ray and manual corrections. Accept nodes within the displacement threshold, then use A*-based searches between consecutive waypoints, requiring arrival within 0.5 m. Successful node transfer is 98.3%, but only 77% of trajectories are navigable; geometry and connectivity both matter. trajectory-conversion
- Encode observations and language
GLoVE tokens feed an LSTM or BiLSTM. RGB uses ImageNet-pretrained ResNet50 features; depth uses a modified ResNet50 pretrained for point-goal navigation. Pretraining is specified, but the supplied text does not establish which encoder parameters remain frozen. policy-encoders
- Form the policy state
Seq2Seq concatenates pooled RGB, flattened depth, and one final instruction state into a GRU. Cross-modal attention instead maintains two GRU states: visual history selects instruction content, which selects spatial RGB/depth content, and the second GRU integrates these features to select an action. seq2seq-policyattention-policy
- Evaluate navigation and path following
Evaluation reports trajectory length (TL) and terminal navigation error (NE) in meters, oracle success (OS), success rate (SR), success weighted by inverse path length (SPL), and normalized dynamic-time warping (nDTW). Higher SR/SPL indicate better success and efficiency; nDTW assesses trajectory agreement. The supplied paper delegates full metric definitions to citations, so it does not specify a success-distance threshold here. training-protocol
6. Experiments & results
VLN-CE turns language-guided graph traversal into low-level navigation through reconstructed indoor spaces. Its recurrent policies choose movements from egocentric RGBD observations without supplied location or topology. The strongest recipe attains 32% success and 0.30 SPL on unseen continuous environments; this is simulated navigation performance, with graph-based comparisons subject to conversion caveats.
6.1 Read the original evidence
Table 3. The best recipe combines techniques whose individual effects are mixed. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left switches mean progress monitor (PM), DAgger (DA), and synthetic data augmentation (Aug.). The source caption defines the asterisk as fine-tuning: starred DA follows pretraining rather than being another simultaneous loss. First compare rows 1 and 6 to hold teacher forcing constant across architectures. Then use rows 6–9 to see each training change separately for the cross-modal model. Finally compare row 10 with rows 11 and 12 to examine the staged combination and its omissions. In the rightmost block, SR measures success and SPL additionally accounts for path length; NE is terminal distance from the goal in meters. training-resultstraining-protocoltraining-objectivesprogress-diagnostic
What it supports. Cross-modal DAgger alone increases unseen SPL from 0.22 to 0.25, whereas PM or augmentation alone gives 0.19. Combining PM and augmentation pretraining with DAgger fine-tuning reaches 0.30 SPL and 32% success. Removing PM from the augmented fine-tuning recipe lowers SPL to 0.26.
Where the evidence stops. The all-techniques row changes both data and training schedule, so it cannot isolate a single cause. Checkpoints use val-unseen performance, and the table reports no uncertainty intervals. Page 13’s progress-loss diagnosis is supporting evidence rather than a causal decomposition.
Table 4. A cross-setting performance gap must be read together with the trajectory-conversion caveats. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. All three column blocks are labeled VLN: these are graph-evaluation scores, unlike the continuous-environment validation scores in Table 3. The upper rows use existing graph-based approaches; the bottom row takes the paper’s continuous policy and converts its rollouts for the VLN leaderboard. Keep the Test block separate from both validation blocks. A dash means a value was not reported in this table, not zero. In Test, the bottom row has 0.24 SR and 0.21 SPL; RCM has 0.38 SPL and Back-Translation has 0.47. Read the adjacent Caveats paragraph before treating those gaps as the effect of topology alone. graph-comparisoncomparison-caveatstraining-results
What it supports. The paper’s comparison makes the graph prior an important research question: strong scores in a graph interface need not carry over to low-level continuous navigation. However, the reported test gap also includes differences in trajectory feasibility, representation, and training. It supports concern about transfer without measuring the isolated value of the graph.
Where the evidence stops. The authors estimate about 20% of VLN routes are non-navigable in VLN-CE and attribute roughly 0.10 validation SPL loss to feasibility/conversion effects together. The exact conversion procedure is in an unsupplied supplement; this is not a matched causal ablation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Instruction following in unseen continuous environments VLN-CE val-unseen; cross-modal PM + DAgger fine-tuning + augmentation, Table 3 row 10. | 0.32 (32%); 0.30; 7.37 m; 0.51 SR; SPL; NE; nDTW | Cross-modal teacher forcing: SR 0.23, SPL 0.22; Seq2Seq teacher forcing: SR 0.20, SPL 0.18. The full recipe improves unseen performance, but combines architecture and training choices. These are reported point estimates, without uncertainty intervals. training-resultstraining-protocol |
| Input-modality dependence VLN-CE val-unseen; Seq2Seq imitation learning without augmentation or auxiliary losses, Table 2. | RGBD plus instruction: 20%; no image: 17%; no depth: 4%; no vision: 0%; no instruction: 17%. Success rate | Random and hand-crafted agents each achieve 3%. Depth removal has a large effect in this setup. The small instruction-removal gap suggests exploitable trajectory regularities; it does not prove absence of language use. The table’s 4% RGB-only result contradicts the adjacent ≤1% prose. modality-results |
| Training mechanisms in the cross-modal model VLN-CE val-unseen, Table 3 rows 6–12. | Baseline 0.22; DAgger 0.25; PM alone 0.19; augmentation alone 0.19; all three 0.30. SPL | Augmentation plus DAgger fine-tuning without PM achieves 0.26. Benefits depend on the combination and staging. The authors associate PM’s improvement with reduced overfitting under augmentation; this diagnosis is not an isolated causal proof. training-resultsprogress-diagnostic |
| Converted trajectories on the VLN test leaderboard Continuous rollouts translated to nav-graph paths; Table 4 Test (VLN), distinct from VLN-CE validation. | 0.24; 0.21 SR; SPL | RCM test SPL 0.38; Back-Translation 0.47. The gap is consistent with a useful graph prior, but differs in training/interface and is confounded by non-navigable trajectories and imperfect conversion. It is not a controlled graph-removal ablation. graph-comparisoncomparison-caveats |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Depth matters strongly in these baselines, while instruction removal leaves substantial residual success. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the right-hand Val-Unseen block for generalization to new environments. The Vision column distinguishes RGBD, depth alone, RGB alone, and no vision; the instruction and history checkmarks show retained inputs. The learned rows use imitation learning without augmentation or auxiliary losses. Read success rate (SR) alongside success weighted by inverse path length (SPL), rather than interpreting short trajectories as automatically good. Random and hand-crafted rows provide context at 0.03 SR. The complete model reaches 0.20 SR; removing the instruction still leaves 0.17, while removing depth leaves 0.04. All-vision removal reaches zero. modality-resultspolicy-encoderstraining-protocol
What it supports. The full-to-no-depth comparison is a 16-percentage-point drop in unseen success. The instruction-removal drop is only 3 points, consistent with useful trajectory regularities. These ablations establish dependence within this architecture and training setup, while leaving room for stronger language grounding and alternative RGB representations.
Where the evidence stops. The prose on this page says models without depth achieve at most 1% success, but the No Depth row reports 4% unseen and 3% seen. This edition preserves the table values and flags the unresolved conflict. Depth also uses navigation-specific pretraining.
Figure 4. A plausible nearby couch does not satisfy an instruction’s ordered route. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the instruction first, especially the requirement to pass the kitchen before turning and stopping. The map above distinguishes the blue agent path from the green ground-truth path, with separate markers for the start, intermediate waypoints, and goal. Now follow the sampled views from step 1 to step 37; the ellipses indicate omitted intermediate observations. The agent approaches a couch and windows, but the authors explain that it selected the wrong windows and never first passed the kitchen. The final blue cross and Failure label therefore concern the full route instruction, not simply the presence of a couch in view. failure-exampletask-interfacetask-motivation
What it supports. The example shows how grounding a few visible nouns can still produce the wrong trajectory. The authors link such failures to instruction referents that the agent never observes under its limited egocentric field of view. This motivates studying observation selection alongside low-level movement and stopping.
Where the evidence stops. This is one selected failed rollout, not an ablation experiment or a failure-frequency estimate. The map and displayed reference path are explanatory annotations; they are not evidence that the policy receives location or a navigation graph as input.
7. Analysis & limitations
7.1 What the evidence leaves open
Only navigable converted trajectories enter the continuous dataset. Mesh holes and moved objects cause exclusions, so the retained task cannot be assumed representative of every original R2R route. trajectory-conversion
The Figure 4 agent stops at the wrong couch without first passing the kitchen. Limited egocentric viewing can hide relevant landmarks; the example illustrates a failure mechanism, not its population frequency. failure-example
Unseen validation selects checkpoints, tables omit seed-level uncertainty, and the leaderboard comparison changes the evaluation representation. Physical robustness to actuation error or dynamic objects remains future work. training-protocoltraining-resultscomparison-caveats
7.2 Questions for discussion
- How much of the depth ablation reflects geometric information versus the depth encoder’s navigation pretraining?
- Would training benefits persist on unseen scenes reserved entirely from checkpoint selection?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the original R2R instructions and scene splits, Matterport3D meshes, Habitat collision geometry, converted waypoints and shortest paths, both pretrained visual encoders, and synthetic augmentation data. Match camera and action settings before comparing policy scores. task-interfacetrajectory-conversiondataset-horizonpolicy-encoderstraining-objectives
The supplied paper names PyTorch and Habitat but omits software versions, hardware budget, random seeds, full encoder-freezing details, progress-loss coefficient, and the exact ray-sampling interval. Full metric definitions are delegated to citations, and graph-conversion details to the absent supplement; these gaps prevent an exact implementation from this PDF alone. trajectory-conversionpolicy-encoderstraining-objectivestraining-protocolgraph-comparison
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does DAgger help especially when long rollouts drift off the reference path?
Reader-proposed, not run: compare cross-modal teacher forcing with DAgger using the same converted split, initial encoders, inflection weighting, action interface, and matched gradient-update budget. Use multiple paired seeds and keep a scene subset out of checkpoint selection. Stratify held-out routes by reference action count; record SR, SPL, collision/stall counts, and recovery after the same small heading perturbation. Report both overall differences and how the DAgger advantage changes with horizon. A larger benefit on long or perturbed routes would support the exposure-bias account. No such interaction, or a gain that disappears after matching updates, would weaken that specific explanation while allowing other DAgger benefits. dataset-horizontraining-objectivestraining-protocoltraining-resultstask-interface
Check 2: Does synthetic augmentation change the effect of progress supervision?
Reader-proposed, not run: run a two-by-two comparison of PM enabled/disabled and synthetic augmentation present/absent with identical cross-modal architecture and matched pretraining updates. Apply the same DAgger fine-tuning schedule to all four groups, rather than comparing starred and unstarred recipes. Log unseen SR/SPL and, for PM-enabled runs, training and validation progress MSE before and after fine-tuning; repeat across seeds on scenes withheld from checkpoint selection. The paper’s interpretation predicts a positive augmentation-by-PM interaction and a smaller progress-loss generalization gap with augmentation. If the MSE gap shrinks without improved navigation, progress estimation is insufficient evidence for better control. training-objectivestraining-protocoltraining-resultsprogress-diagnostic
8.3 Reading coverage
Visual audit: The title/author page and all main-text pages were rendered and visually inspected. Figures 1–4 and Tables 1–4 were read in their original layouts. Six final crops were individually viewed; the architecture crop was expanded and viewed again to retain the top recurrent-state labels. Figure 3 arrow directions agree with Equations (3)–(7). Table 2’s no-depth row conflicts with the adjacent ≤1% prose; the report explicitly preserves the table values. Table 3’s caption defines the asterisk as fine-tuning, retained in the reading guide. Figure 2’s unlabeled vertical units are not inferred. Pages 15–16 contain references and were read as text. No appendix is present in the supplied PDF, and the referenced separate supplement was unavailable to this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (p. 1)
- 1 Introduction (pp. 1–4)
- 2 Related Work (pp. 4–5)
- 3 VLN in Continuous Environments and 3.1 trajectory transfer (pp. 5–7)
- 3.2 VLN-CE Dataset (p. 8)
- 4.1 Sequence-to-Sequence and 4.2 Cross-Modal Attention (pp. 8–9)
- 4.3 Auxiliary Losses and Training Regimes (pp. 9–10)
- 5 Experiments, including 5.1–5.3 (pp. 10–14)
- 6 Discussion and acknowledgements (p. 14)
- References (pp. 15–16)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read completely. The title and all five authors match the catalog. The supplied ECVA PDF has internal pages 1–16 and no explicit revision label; no alternate edition or revision chain was supplied or compared.
- Acquisition omission: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This was addressed by visually inspecting PDF pages 1–14 and every final crop.
- Separate supplemental material availability has not been fully verified. No supplement was supplied or read; Section 5.3 places path-to-graph conversion details there. There is no appendix in the supplied PDF.
- Code, linked resources, and external metric definitions were not inspected; no experiments were reproduced. References were read as text, without opening the cited works.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title and author block
The title matches the catalog. Authors, in order, are Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Affiliations are Oregon State University, Georgia Institute of Technology, and Facebook AI Research.
Go to primary source ↓task-motivationPDF pp. 2–4, Figure 1 and Section 1 contributions
VLN-CE removes supplied navigation topology, oracle movement between panorama nodes, and precise localization; it introduces a continuous simulation task, policy baselines, and diagnostic evaluations.
Go to primary source ↓task-interfacePDF pp. 5–6, Section 3, observations and actions
Habitat supports movement and collision checking in reconstructed Matterport3D meshes. The agent receives 256 by 256 RGBD images with 90-degree horizontal field of view and chooses forward 0.25 m, left/right 15 degrees, or stop.
Go to primary source ↓trajectory-conversionPDF pp. 6–7, Section 3.1 and Figure 2(a–b)
The ground agent is a 1.5 m tall, 0.2 m diameter cylinder. Downward ray projection searches up to 2 m; mapped nodes require less than 0.5 m horizontal displacement, with manual corrections. 98.3% of nodes and 77% of trajectories transfer. A*-based pairwise waypoint checks require arrival within 0.5 m. Adjusted nodes average 0.19 m displacement; mesh discontinuities and moved objects cause failures.
Go to primary source ↓dataset-horizonPDF pp. 7–8, Figure 2(c) and Section 3.2
VLN-CE contains 4,475 trajectories converted from R2R train and validation splits, with original instructions and precomputed low-level paths. Paths average 55.88 actions versus approximately 4–6 graph steps in R2R.
Go to primary source ↓policy-encodersPDF p. 8, Figure 3 and Section 4 representations
Both baselines use GLoVE instruction embeddings, ImageNet-pretrained ResNet50 RGB features, and modified ResNet50 depth features pretrained for point-goal navigation. Figure 3 contrasts a single policy GRU with two recurrent states and cross-modal attention.
Go to primary source ↓seq2seq-policyPDF p. 9, Section 4.1, Equations (1)–(2)
The baseline concatenates mean-pooled RGB features, flattened depth features, and the final LSTM instruction state; a recurrent GRU state feeds the action classifier.
Go to primary source ↓attention-policyPDF p. 9, Section 4.2, Equations (3)–(7)
A first GRU processes observations and the 32-dimensional previous-action embedding. Its state attends to BiLSTM instruction states, which query RGB and depth spatial features. A second GRU consumes attended features, the previous action, and the first state, and predicts the argmax softmax action.
Go to primary source ↓training-objectivesPDF p. 10, Section 4.3
Teacher forcing uses inflection weighting at action changes. DAgger aggregates experiences across iterations. Approximately 150,000 synthetic trajectories are converted from prior speaker-generated data. The progress monitor predicts fraction through a trajectory with a training-time mean squared error loss.
Go to primary source ↓training-protocolPDF pp. 10–11, Section 5, setting, metrics, and implementation details
Adam uses learning rate 2.5e-4, batches of five complete trajectories, and inflection coefficient 3.2. Teacher forcing runs at most 30 epochs. DAgger uses oracle probability 0.75^n, 5,000 trajectories per collection, four epochs over the aggregate, and 6–10 collections. Early stopping uses val-unseen. Implementation uses PyTorch and Habitat; full metric definitions are referred to prior papers.
Go to primary source ↓modality-resultsPDF p. 11, Table 2, Val-Unseen columns and Section 5.1
Unseen SR/SPL: Seq2Seq 0.20/0.18; no image 0.17/0.15; no depth 0.04/0.04; no vision 0.00/0.00; no instruction 0.17/0.16; random and hand-crafted 0.03/0.02. Table 2 contradicts the nearby prose asserting at most 1% success for models without depth.
Go to primary source ↓training-resultsPDF p. 12, Table 3, rows 1–12, Val-Unseen columns and Section 5.2
Cross-modal baseline row 6 has SR 0.23 and SPL 0.22. DAgger row 8 reaches 0.27/0.25; PM row 7 and augmentation row 9 each have SPL 0.19. Full combination row 10 reaches SR 0.32, SPL 0.30, NE 7.37 m, and nDTW 0.51. Row 12 without PM reaches 0.29/0.26. Asterisk denotes fine-tuning; the full recipe pretrains with PM and augmentation, then uses DAgger on original data with oracle probability 0.75^(n+1).
Go to primary source ↓progress-diagnosticPDF p. 13, Section 5.2, first paragraph; PDF p. 12, Table 3 rows 7, 9, 11
Progress-monitor validation loss is 0.67 without augmentation versus 0.47 with augmentation. The authors interpret this as overfitting on the smaller dataset. PM plus augmentation has unseen SPL 0.22, versus 0.19 for either alone.
Go to primary source ↓failure-examplePDF p. 13, Figure 4 and Section 5.2 Example
The illustrated unseen rollout ends in failure after 37 actions: the agent goes toward the wrong windows, does not first pass the kitchen, and stops at the nearest couch. The map distinguishes blue agent and green ground-truth paths; the authors discuss referents missed under egocentric viewing.
Go to primary source ↓graph-comparisonPDF pp. 13–14, Section 5.3 and Table 4, Test (VLN) columns
Continuous rollouts are converted to graph trajectories and evaluated through the VLN leaderboard. The proposed cross-modal model has test SR 0.24 and SPL 0.21, versus RCM SPL 0.38 and Back-Translation SPL 0.47. Conversion details are assigned to separate supplemental material.
Go to primary source ↓comparison-caveatsPDF p. 14, Section 5.3 Caveats and Section 6
The authors note about 20% of VLN trajectories are non-navigable in VLN-CE and sparse graph coverage can distort converted rollouts, with an approximately 0.10 SPL validation drop. Imperfect actuation, dynamic objects, and modular controllers are discussed as future research directions.
Go to primary source ↓8.5 Primary sources
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments ↗
PDF · 7,262 extracted words
Source fingerprint
ec4cd997441c061221e5fa76b3da1205a019d25214d6f1c8557438b4627ddcc6