Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments
1. Paper overview
In one sentence: R2R couples human route instructions to a deterministic simulator built from real panoramas, revealing that learning to act in familiar buildings transfers poorly to unseen ones. e02e03e06e09e15e16
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | Source description The Matterport3D Simulator reuses 10,800 panoramic views from 90 building-scale scenes. These derive from 194,400 RGB-D images, but this simulator version outputs RGB; depth and semantic observations are future extensions. e03 |
| A key reported result | R2R navigation in unseen test buildings: 20.4%; 26.6%; 7.85 m; 8.13 m. Success; oracle success; navigation error; trajectory length. 4,173 test instructions in 18 held-out scenes; student-forcing submission trained on train and validation data; greedy single-shot evaluation. RANDOM: 13.2% success, 18.3% oracle success and 9.77 m error. Humans: 86.4% success and 1.61 m error on a separate 1,390-instruction test subset. The learned agent exceeds the exploitative random baseline, but passing near a goal and stopping there remain distinct. The human row is contextual, not a matched full-test comparison. e09e10e12e15e19 |
| Reading caution |
Core contributions
- Source description
The Matterport3D Simulator reuses 10,800 panoramic views from 90 building-scale scenes. These derive from 194,400 RGB-D images, but this simulator version outputs RGB; depth and semantic observations are future extensions. e03
Figure 2. Navigation turns sequence prediction into a feedback process through the camera. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper row from the blue instruction-encoding blocks toward the green decoding blocks. Each green VLN block receives a room image from below and emits an action above; horizontal arrows carry sequence state forward. The lower VQA row instead emits the words of an answer. The caption establishes the crucial connection: navigation actions manipulate the viewpoint that supplies later observations. Section 5.1 makes the policy concrete with an LSTM decoder receiving image and previous-action features and attending to language states. The diagram displays readable word order; the implementation reverses instruction words before encoding. Attention connections and the full simulator loop are not drawn. e02e03e11e17e19
What it supports. The task couples perception to behavior: an action changes which photograph the agent sees next, so errors can change the inputs for subsequent decisions. The recurrent baseline uses observation history and language attention to choose actions. It does not predict future images or learn the simulator's transition graph.
Where the evidence stops. This is a conceptual sequence diagram, not a complete network schematic. It cannot establish encoder/decoder dimensions, attention equations or a learned world model; those architectural details must be checked in Sections 5.1–5.2.
2. Motivation
2.1 The problem and the proposed response
An agent must interpret an unfamiliar instruction while moving through an unfamiliar building, using changing first-person RGB observations and deciding when to stop. R2R makes this problem measurable without assuming enumerated object labels or continuous robot motion. e02e03e09
2.2 What this reading follows
A navigation instruction describes places that may be invisible from the starting camera view. R2R therefore asks an agent to gather new observations through its own actions and decide when it has arrived. The paper contributes a photographic simulator, a dataset of 21,567 instructions, and baseline policies that make this challenge reproducible. Read the visuals as a chain: language produces camera-changing actions, a graph constrains movement, route sampling shapes the dataset, and held-out buildings expose the limits of learned behavior. The headline baseline reaches 20.4% test success. These results concern simulated navigation through captured real scenes; they do not demonstrate physical robot deployment. e02e03e06e09e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dataset, language-annotation and benchmark categories fit the primary contribution. Architecture, prediction paradigm and quadrant are appropriately not applicable to the dataset. Its baseline predicts actions from observed images and language; neither simulator transitions nor recurrent memory constitute a learned future-observation world model or a joint world/action predictor. e02e03e06e09e11e17
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the simulator's graph from the policy's memory
The word 'environment' hides two different representations here. The simulator knows a navigation graph over captured camera positions and uses it to decide which transitions are legal. The policy, by contrast, updates an LSTM state from its current image and previous action while consulting the instruction. To follow a route, it rotates until an appropriate destination is available, moves forward, and processes the new observation. Equation (1) explains the simulator's legal moves, not a learned prediction of where an action will lead. Reader interpretation: this separation is why R2R can test language-conditioned sequential decision making without establishing world-model learning. Photographic inputs increase visual diversity, while precomputed connectivity removes much of the low-level navigation problem that a physical robot would face. e03e04e11e17
Figure 3. Photographic realism is paired with a discrete, manually checked movement graph. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The blue network overlays a partial floor rather than a complete building. Nodes correspond to panoramic viewpoints and edges indicate navigable transitions, not trajectories generated by the baseline. Section 3.2.2 explains how mesh ray tracing and manual verification construct this graph, with edges longer than 5 m removed. Equation (1) then narrows the available destinations to adjacent viewpoints within the camera's horizontal view region, while always allowing the current viewpoint. Thus the drawing shows potential connectivity; the agent's immediate options also depend on its orientation. The simplified baseline's forward action chooses the available destination nearest the center of the visual field. e03e04e11
What it supports. The environment supplies navigation structure before policy learning. This makes viewpoint transitions reproducible while retaining photographic observations of real interiors. The graph therefore explains both a strength of the benchmark and a major abstraction: the policy need not learn collision-free continuous locomotion from pixels.
Where the evidence stops. The cutaway is an explanatory map, not evidence that the policy receives a global map. The simulator constrains movement using its graph; the baseline described in Section 5.1 consumes local images and action history.
5.2 Understand why sampled actions still receive supervised targets
Suppose the agent turns incorrectly while following an instruction. Teacher-forcing avoids this situation during training by executing the shortest-path target action at every step. Student-forcing instead samples the current policy, allowing the learner to visit states caused by its own mistakes. Crucially, the graph still supplies a shortest path from the new pose to the known training goal, so a target action can be recomputed there. Cross entropy trains the policy against that target; exploration through sampling does not turn this objective into reward-only reinforcement learning. At evaluation, the agent switches to greedy decoding and must select stop itself. The distinction matters when reproducing the comparison: change which actions generate training states while retaining the same target rule, architecture and metric definitions. e04e09e11e18e19
Figure 5. The benchmark's language and route lengths reflect its collection procedure. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left horizontal axis in words and the right horizontal axis in meters. Both panels have a concentrated central region and a thinner right tail; their vertical scales differ. Use the adjacent source text for exact averages: 29 words per instruction and 10 m per sampled route. The route distribution must be interpreted alongside the sampling rule: shortest paths shorter than 5 m are rejected, and accepted paths have four to six graph edges. These constraints limit geometric route complexity without fixing the wording. The three instructions collected for a route can describe the same destination at different levels of detail, as Figure 4 illustrates. e06e07
What it supports. R2R contains 7,189 sampled paths with three instructions each, giving 21,567 instructions. Its collection protocol combines bounded route structure with variable natural language. The two distributions characterize the annotation benchmark; they are not measurements of the distances or instruction lengths a deployed robot would encounter.
Where the evidence stops. The panels do not label the vertical-axis quantity or formally define their dashed lines in the caption. Avoid reading precise bin probabilities from this crop, and do not mistake instruction count for the number of independent routes.
5.3 Read generalization and stopping as separate failures
A new instruction in a familiar building tests a different ability from an instruction in a building excluded from training. The split design makes that distinction visible: student-forcing achieves 38.6% success on seen validation and 21.8% on unseen validation. Oracle success asks a second question: did the route ever get sufficiently close, regardless of the agent's chosen stopping point? On unseen validation, student-forcing has higher final success than teacher-forcing but slightly lower oracle success. Reader interpretation: a single success number therefore cannot tell whether an improvement comes from visiting better locations or choosing a better stopping point. Figure 7 adds temporal evidence that more training alone does not close the environmental gap. These observations motivate controlled tests; they do not identify a unique cause. e09e10e13e14e16
5.4 Training and inference
During training
Both regimes minimize per-step cross entropy against the next shortest-path action from the current pose to the goal. Teacher-forcing executes that target during training. Student-forcing samples the policy's own action, then recomputes the target from the resulting state, exposing the learner to its own deviations. e18
The implementation uses 512 hidden units per LSTM, 256-dimensional word embeddings, 32-dimensional action embeddings and dropout 0.5. Lowercased whitespace tokens occurring fewer than five times are filtered. Images are 640 × 480 with 60-degree vertical field of view. CNN features are pre-cached, so recurrent training does not jointly fine-tune the CNN. Adam with weight decay and batch size 100 trains the policy in PyTorch. e19
During inference
Evaluation is single-shot with greedy decoding, rather than the sampling used in student-forcing training. The test submission is trained on training plus validation data. Stop ends the episode; camera heading and elevation do not enter navigation error. These are executed simulator actions, not physical robot trials. e08e09e19
5.5 Implementation flow
- Construct navigable transitions
A weighted undirected graph connects panoramic viewpoints. Mesh ray tracing detects obstacles, edges longer than 5 m are removed, and manual verification corrects missing obstacles. Edge weights are straight-line distances; actions deterministically change viewpoint and camera orientation. e03e04
- Collect routes and language
Sample start/goal pairs predominantly in different rooms, keep shortest paths at least 5 m long with four to six edges, and show annotators interactive marked routes. Instructions need only reach the goal, not reproduce the displayed path. Screened US-based workers supply varied levels of linguistic detail. e06e07
- Encode instruction and observation history
An LSTM encodes reversed instruction tokens. Mean-pooled ImageNet-pretrained ResNet-152 features and the previous action embedding form the decoder input. The decoder retains recurrent history and uses global general attention over language encoder states to predict the next action. Visual attention is left to future work. e11e17
- Map predictions to simulator actions
The baseline reduces the richer simulator interface to left, right, up, down, forward and stop. Rotations change heading or elevation by 30 degrees. Forward selects the reachable viewpoint nearest the center of the current visual field; the simulator then supplies the resulting observation. e03e11
- Separate environmental generalization from stopping
Training has 14,025 instructions and seen validation 1,020 within 61 scenes; unseen validation has 2,349 instructions in 11 other scenes; test has 4,173 in 18 scenes. Navigation error is final-to-goal shortest-path distance. Success requires error strictly below 3 m; oracle success instead uses the closest visited point. e09e10
6. Experiments & results
R2R turns natural-language route following into a reproducible benchmark over photographs of real buildings. Its simulator executes viewpoint transitions; an attention-based recurrent policy learns to choose navigation actions. Student-forcing improves baseline success, but the large seen/unseen gap exposes limited environmental generalization (e02, e13–e16).
6.1 Read the original evidence
Table 1. Student-forcing helps the baseline, but unseen-building success remains low. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the three split labels, then compare methods within a block. Trajectory length and navigation error are in meters; the two success columns are percentages. Success requires the agent to stop less than 3 m from the goal, whereas oracle success asks whether any visited point met that distance rule. SHORTEST has privileged goal information and serves as a reference. RANDOM exploits typical route structure by taking five successful forward steps. The test student-forcing model uses training plus validation data, unlike the validation experiments. The human test row covers 1,390 instructions, so its denominator differs from the automated test evaluation's 4,173 instructions. e09e10e12e13e14e15e18e19e16
What it supports. Student-forcing reaches 38.6% success on seen validation, 21.8% on unseen validation and 20.4% on test. Teacher-forcing reaches 27.1% and 19.6% on the two validation splits. The unseen comparison is nuanced: student-forcing improves final success while its oracle success, 28.4%, is slightly below teacher-forcing's 29.1%.
Where the evidence stops. The human row is a smaller-subset benchmark, not a paired full-test comparison. No uncertainty intervals or seed counts accompany the table, and the different training regimes are not established as matched in compute.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| R2R navigation in unseen test buildings 4,173 test instructions in 18 held-out scenes; student-forcing submission trained on train and validation data; greedy single-shot evaluation. | 20.4%; 26.6%; 7.85 m; 8.13 m. Success; oracle success; navigation error; trajectory length | RANDOM: 13.2% success, 18.3% oracle success and 9.77 m error. Humans: 86.4% success and 1.61 m error on a separate 1,390-instruction test subset. The learned agent exceeds the exploitative random baseline, but passing near a goal and stopping there remain distinct. The human row is contextual, not a matched full-test comparison. e09e10e12e15e19 |
| R2R navigation in seen validation buildings 1,020 held-out instructions from the 61 pooled training/seen-validation scenes. | Student-forcing: 38.6%; 52.9%; 6.01 m. Success; oracle success; navigation error | Teacher-forcing: 27.1%; 36.7%; 8.01 m. RANDOM success: 15.9%. Student-forcing improves all three reported quantities in familiar environments. This split tests new instructions without requiring new buildings. e10e13e18 |
| R2R navigation in unseen validation buildings 2,349 instructions in 11 scenes excluded from baseline training. | Student-forcing: 21.8%; 28.4%; 7.81 m. Success; oracle success; navigation error | Teacher-forcing: 19.6%; 29.1%; 8.61 m. RANDOM success: 16.3%. The student-forcing advantage is smaller here, and oracle success is slightly lower than teacher-forcing. Learning from sampled actions does not establish robust transfer to new buildings. e10e14e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. More training improves familiar-building navigation while unseen performance plateaus. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: blue and green are student-forcing on seen and unseen validation; orange and red are the corresponding teacher-forcing curves. Purple and brown appear in the loss panel for training. Next compare panels rather than treating cross entropy as navigation success. Student-forcing's seen navigation error continues falling, but unseen error levels off; the success panel shows the corresponding separation. The caption and Section 6 interpret this as overfitting to training environments. Teacher-forcing curves cover a shorter training span, so the plot is also a reminder to control the training budget when comparing regimes. Exact endpoint percentages are tabulated separately in Table 1. e13e14e16e18e19
What it supports. The learning curves show that better performance on familiar buildings can coexist with limited progress on new buildings. The authors suggest that visual grounding may be environment-specific. As reader analysis, the plot motivates testing that explanation, but it does not distinguish visual memorization from language, graph or route correlations.
Where the evidence stops. The success axis is labeled '%' but marked with decimal ticks; Table 1 supplies explicit percentages. Preserve that source ambiguity. These curves lack uncertainty bands and do not establish a matched-compute ablation or isolate which model component causes the gap.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify clean, often luxurious interiors, very few people or animals, and viewpoints chosen for commanding views rather than typical robot positions. Discrete panoramic transitions preserve image realism while limiting continuous-motion realism. e03e05
Figure 7 shows unseen validation performance plateauing while seen performance continues improving. The authors infer environment-specific visual grounding; the curves support overfitting concerns but do not isolate its visual cause from language, layout or route correlations. e16
Endpoint success does not certify instruction-faithful routes or efficient navigation. Workers were allowed alternate paths; heading and elevation are ignored by the metric. Human instructions can confuse left and right, and this setup offers neither gesture nor clarification dialog. e06e09e20
Table 1 provides averages without seed counts or uncertainty intervals. Figure 7 uses decimal success ticks despite a percent axis label; exact percentages should come from Table 1. e13e14e15e16
7.2 Questions for discussion
- How much of the seen/unseen gap comes from visual grounding, language reuse or graph-layout regularities? (e16)
- Would route-sensitive scoring change conclusions when several paths reach the same instructed goal? (e06, e09)
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the matching panoramas, manually checked connectivity graphs, annotations and scene splits, plus the six-action adapter and shortest-path teacher. The paper describes C++/OpenGL simulation with Python bindings and advertises simulator, dataset and baseline access through its project site; it does not state a release license. e02e04e06e10e11e21
Section 5.2 specifies major dimensions and optimization choices but not the learning rate, weight-decay coefficient, numerical iteration budget, hardware, runtime or software versions. Figure 7 displays training spans without resolving all checkpoint-selection details. The attention description also leaves final action-projection dimensions unspecified. e17e19e16
A minimal proposed check is to hold data, initialization, architecture and evaluation fixed while comparing teacher- and student-forcing across seen and unseen buildings. Measure both final and oracle success, since their rankings differ on unseen validation; this proposal has not been executed. e13e14e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Control the training budget when comparing forcing regimes
Reader-proposed check; not executed. Train teacher- and student-forcing policies with the same splits, feature cache, initialization scheme, optimizer settings and shortest-path target rule. Compare both at matched update counts and at matched environment-interaction budgets, since the source shows unequal training spans. Record seeds and checkpoint rules explicitly. On seen and unseen validation, measure final success, oracle success and navigation error with the same greedy evaluator. If the student-forcing gain disappears under these controls, the published comparison would not by itself establish a benefit from visiting self-induced states. Persistent gains would strengthen that mechanism claim. e10e11e13e14e16e18e19
Check 2: Test whether successful routes depend on the supplied instruction
Reader-proposed check; not executed. Freeze a trained policy and pair each validation episode with a second run using a length-matched instruction shuffled from another route in the same scene. Keep its start pose, images, action budget, decoding rule and original scoring goal unchanged. Compare the change in success, oracle success and navigation error separately for seen and unseen buildings. Little degradation would support an instruction-insensitive shortcut explanation; a clear drop would refute that simple explanation, while leaving compositional understanding unproven. This control tests the authors' generalization concern without treating the learning curves as a causal ablation. e06e07e09e10e11e16
8.3 Reading coverage
Visual audit: Visually inspected the title/author/affiliation block and open-access notice, all eight body pages, Figures 1–8, Equation (1), model/training/evaluation text, Table 1 and the training diagnostics. Inspected every final original crop at its returned dimensions. Figure 2's arrows were checked against its caption and Section 5.1; it is a conceptual diagram without full attention wiring. Figure 7's percent-label/decimal-tick discrepancy is retained and disclosed. All source pages supporting numerical, method, training and proposed-check claims are included here. References on PDF pages 9–10 were read in text only. No appendix occurs in the supplied PDF; the separately mentioned collection-interface supplement was not supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and 1. Introduction (PDF pp. 1–2)
- 2. Related Work (PDF pp. 2–3)
- 3. Matterport3D Simulator, including 3.1 and 3.2.1–3.2.4 (PDF pp. 3–4)
- 4. Room-to-Room Navigation, including task, collection, analysis and evaluation (PDF pp. 4–6)
- 5. Vision-and-Language Navigation Agents, including model, training and baselines (PDF pp. 6–7)
- 6. Results and 7. Conclusion and Future Work; acknowledgements (PDF pp. 7–8)
- References (PDF pp. 9–10)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/edition note: the supplied CVPR 2018 open-access paper spans printed pages 3674–3683. The title wording matches the catalog; the printed subtitle uses sentence case. All nine authors match, with Ian Reid printed where the catalog gives Ian D. Reid. The title-page notice says this version is identical to IEEE Xplore except for its watermark; no separate revision identifier or revision date is printed. No other edition was compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by visually inspecting PDF pages 1–8 and all five final crops.
- Separate supplemental material availability has not been fully verified. The collection interface described as supplementary in Section 4.2 was not supplied, and no separate supplement was read.
- Code, external project resources and referenced papers were not inspected; experiments were not reproduced. Reference pages were read as text, but their page images were not inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1 (printed p. 3674), title, author/affiliation block and open-access notice
The inspected title matches the supplied observedTitle in wording. Credits are Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould and Anton van den Hengel. Institutions are Australian National University, University of Adelaide, Queensland University of Technology and Macquarie University. The notice states this open-access CVPR version differs from IEEE Xplore only by the watermark.
Go to primary source ↓e02PDF pp. 1–2 (3674–3675), Abstract, Section 1 contribution list and Figure 2/caption
The paper introduces the simulator, R2R benchmark and sequence-to-sequence baselines. Figure 2 contrasts action outputs with VQA word outputs and associates successive VLN actions with changing camera observations. The paper advertises access through its project website.
Go to primary source ↓e03PDF p. 3 (3676), navigation-based simulators, Sections 3.1, 3.2.1 and 3.2.2
Matterport3D contains 10,800 panoramas from 194,400 RGB-D images across 90 scenes. Rendering from panoramas trades continuous motion for image realism. The simulator returns RGB perspective views, proposes depth/segmentation as future extensions, and uses deterministic viewpoint/orientation actions.
Go to primary source ↓e04PDF pp. 3–4 (3676–3677), Section 3.2.2, Equation (1), footnote 3 and Figure 3/caption
Graph edges encode navigable transitions weighted by straight-line distance. Construction uses mesh ray tracing, removes edges longer than 5 m and includes manual correction. Equation (1) allows staying at the current viewpoint or moving along an edge within the camera's horizontal field; vertical glancing is allowed. Figure 3 shows a partial-floor graph.
Go to primary source ↓e05PDF p. 4 (3677), Section 3.2.4 Biases
The authors describe tidy and often luxurious spaces, sparse people/animals, and camera viewpoints with commanding views that may differ from robot locations.
Go to primary source ↓e06PDF p. 5 (3678), Section 4.2 Data Collection and Figure 5
Start/goal pairs predominantly cross rooms. Shortest paths shorter than 5 m or outside four to six edges are discarded. There are 7,189 paths averaging 10 m, three instructions per path, and workers may describe alternate routes to the goal.
Go to primary source ↓e07PDF p. 5 (3678), Sections 4.2–4.3, Figures 4–5 and captions
Screened US-based AMT workers produced 21,567 instructions averaging 29 words. Vocabulary is about 3.1k words, with about 1.2k occurring at least five times. Instructions vary in abstraction. Figure 5 plots word and metric route lengths; the full annotation interface is described as supplementary.
Go to primary source ↓e08PDF pp. 4–5 (3677–3678), Section 4.1 Task
The agent receives language and an initial pose/image, takes actions yielding new poses and observations, and ends an episode with stop.
Go to primary source ↓e09PDF pp. 5–6 (3678–3679), Section 4.4 Evaluation Protocol
Navigation error is shortest-path graph distance from final viewpoint to goal, ignoring orientation. Success is strictly less than 3 m. Entire trajectories are not scored for route adherence. Oracle stopping scores the closest visited point to the goal.
Go to primary source ↓e10PDF p. 6 (3679), Section 4.4 Dataset Splits
Train/seen-validation share 61 scenes with 14,025/1,020 instructions. Unseen validation has 11 scenes and 2,349 instructions; test has 18 scenes and 4,173 instructions. Test goals are withheld for server evaluation.
Go to primary source ↓e11PDF p. 6 (3679), Section 5.1, language encoding, model action space and image/action embedding
Reversed word embeddings enter an encoder LSTM. A decoder consumes mean-pooled ImageNet-pretrained ResNet-152 image features plus a previous-action embedding, preserving recurrent history and attending to encoder states. The six actions include 30-degree rotations and forward to the reachable viewpoint nearest the visual center.
Go to primary source ↓e12PDF p. 7 (3680), Section 5.3 Additional Baselines
RANDOM selects a heading and executes five successful forward moves, turning right when blocked; SHORTEST uses the goal's shortest path. Human evaluation uses 1,390 instructions from one third of test, a freely navigable first-person interface, and bonuses for stopping near the goal.
Go to primary source ↓e13PDF p. 7 (3680), Table 1, Val Seen block, all metric columns
Student-forcing reports trajectory 11.33 m, error 6.01 m, success 38.6%, oracle success 52.9%. Teacher-forcing reports 10.95 m, 8.01 m, 27.1%, 36.7%; RANDOM success is 15.9%. No uncertainty intervals are tabulated.
Go to primary source ↓e14PDF p. 7 (3680), Table 1, Val Unseen block, all metric columns
Student-forcing reports trajectory 8.39 m, error 7.81 m, success 21.8%, oracle success 28.4%. Teacher-forcing reports 10.67 m, 8.61 m, 19.6%, 29.1%; RANDOM success is 16.3%. No uncertainty intervals are tabulated.
Go to primary source ↓e15PDF p. 7 (3680), Table 1, Test (unseen) block, all metric columns
Student-forcing reports 8.13 m trajectory, 7.85 m error, 20.4% success and 26.6% oracle success. RANDOM gives 9.93 m, 9.77 m, 13.2%, 18.3%. Humans give 11.90 m, 1.61 m, 86.4%, 90.2%; SHORTEST gives 9.93 m, 0 m, 100%, 100%. No uncertainty intervals are tabulated.
Go to primary source ↓e16PDF p. 8 (3681), Figure 7, its caption and Section 6 discussion; PDF p. 7 Table 1 for percentage scale
Training plots compare teacher/student regimes and seen/unseen validation, with training loss curves. Unseen performance plateaus while seen performance improves; the authors suggest environment-specific visual grounding. The success axis has a percent label but decimal ticks; Table 1 supplies explicit percentages. Plotted teacher/student training spans differ.
Go to primary source ↓e17PDF pp. 6–7 (3679–3680), Section 5.1 Action prediction with attention mechanism
The decoder uses global general language alignment to obtain context, concatenates it with decoder state, applies W_c and tanh, then writes the action distribution as a softmax of the attentional hidden state. Visual attention is deferred. The text does not specify final action-projection dimensions.
Go to primary source ↓e18PDF p. 7 (3680), Section 5.2 Training and footnote 4
Cross entropy targets the next shortest-path action from the current pose. Teacher-forcing executes targets; student-forcing samples predicted actions. The authors relate always sampling to online DAGGER and report that scheduled sampling performed worse in initial experiments without numerical results.
Go to primary source ↓e19PDF p. 7 (3680), Section 5.2 Implementation Details
The paper specifies lowercase whitespace tokenization, a five-occurrence filter, 640 × 480 images, 60-degree vertical field, 512-unit LSTMs, 256/32-dimensional word/action embeddings, dropout 0.5, pre-cached CNN features, PyTorch, Adam with weight decay and batch size 100. Training uses an unspecified fixed iteration count; testing uses greedy decoding and train-plus-validation data for the submission. Learning rate, weight-decay value, hardware and software versions are not given.
Go to primary source ↓e20PDF pp. 7–8 (3680–3681), Section 6 human-performance discussion and Figure 8/caption
The authors mention occasional left/right confusion and the absence of gesture/dialog for disambiguation. Figure 8 plots seen-validation error distributions and its caption reports 38.6% success for student-forcing.
Go to primary source ↓e21PDF p. 4 (3677), Section 3.2.3 Implementation Details; PDF p. 2 project-availability paragraph
The simulator uses C++ and OpenGL with Python bindings, configurable image resolution/field of view, and a separate WebGL annotation interface. The paper advertises project access but gives no release-license terms.
Go to primary source ↓8.5 Primary sources
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments ↗
PDF · 7,764 extracted words
Source fingerprint
4ffdbfd13363a2f07301d9b173245a700bce5834dcc5fbe6b930df06a0105b30