Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
1. Paper overview
In one sentence: RxR makes multilingual route following a richer grounding problem by pairing less predictable paths with synchronized human demonstrations, while its baseline agents remain far below human performance. e-motivatione-samplinge-annotatione-agente-gfe-multilinguale-grounding-resultse-test
| At a glance | What to know |
|---|---|
| Research problem | Source description Shortest, narrowly distributed navigation paths can reward destination seeking and route priors while weakening the need to understand instructions. RxR seeks more varied, indirectly approaching routes, denser references to visible entities, and multilingual grounding in reconstructed indoor environments. e-motivatione-sampling |
| Core mechanism | Source description The dataset contains 126,069 instructions over 16,522 sampled paths, with English, Hindi and Telugu collected independently from speakers rather than translated. Table 1 reports 9.8 million words; Table 2 reports 78 whitespace-tokenized words per instruction on average. e-datasete-motivation |
| A key reported result | RxR heldout navigation: Monolingual: 25.9 / 21.0 / 41.5. SR (%) / SDTW / NDTW; higher is better. Table 8, test-standard; Guide+Follower training; averages across English, Hindi and Telugu. Multilingual: 21.3 / 17.1 / 37.2; random: 7.5 / 3.1 / 15.4; human followers: 93.9 / 76.9 / 79.5. A large simulated-navigation gap remains. Human collection uses audio and quality filtering, so the human row is a task reference, not a matched-input ablation. e-teste-annotation |
| Reading caution | Reader analysis Camera poses measure what was visible, not eye fixation or exact object–word correspondence. The linguistic analysis covers only 25 sampled paths; the authors cannot separate language, dialect and annotator-pool effects. e-alignmente-language-analysis |
Core contributions
- Source description
The dataset contains 126,069 instructions over 16,522 sampled paths, with English, Hindi and Telugu collected independently from speakers rather than translated. Table 1 reports 9.8 million words; Table 2 reports 78 whitespace-tokenized words per instruction on average. e-datasete-motivation
- Source description
Each instruction has Guide and Follower pose traces aligned to words. Most sampled paths receive instructions in all three languages, while language-specific paths add coverage. Follower trajectories provide alternative human interpretations and a quality-control signal. e-annotatione-dataset
- Source description
The evaluation includes monolingual and multilingual agents, Guide/Follower training comparisons, attention supervision, transfer, multitask learning and modality ablations. e-gfe-multilinguale-grounding-resultse-transfere-ablation
Figure 2. Room constraints permit an indirect route while retaining a shortest path inside the permitted sequence. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the left panel: small circles labeled p are panorama viewpoints, and the surrounding regions labeled r are rooms. In the right panel, the selected room order is r0, r2, r3; room r1 and its connections fade into the background. Follow the arrowheads from p8 through p13, p10, p7 and p11 to p6. This is a shortest panorama route subject to the selected room order. Section 3 formalizes the restriction using a directed graph, so the visible arrows should be read in that order rather than as freely reversible room transitions. Subsequent greedy selection across candidate paths adds the coverage objective. e-samplinge-motivation
What it supports. The diagram explains how shortest-path computation can coexist with indirect instructions. Choosing the room sequence first changes the permitted navigation graph: a shorter route available in the full building need not remain valid. This supplies a concrete mechanism for weakening the simple destination-seeking prior described in the motivation.
Where the evidence stops. The original caption calls its endpoints r8 and r6, but the diagram and §3 define panorama endpoints p8 and p6. The crop is preserved; the explanation follows those verified panorama labels. Coverage optimization is specified in prose, not depicted here.
2. Motivation
2.1 The problem and the proposed response
Shortest, narrowly distributed navigation paths can reward destination seeking and route priors while weakening the need to understand instructions. RxR seeks more varied, indirectly approaching routes, denser references to visible entities, and multilingual grounding in reconstructed indoor environments. e-motivatione-sampling
2.2 What this reading follows
A navigation dataset can accidentally reward guessing where a route ends instead of understanding how to follow it. RxR changes both the routes and the evidence available for learning: speakers describe paths in English, Hindi and Telugu, while the collection tool records what Guides and Followers could see as the instruction unfolds. This reading follows that chain from room-level path sampling to aligned camera traces and recurrent navigation policies. The experiments separate several questions: whether alternative human paths help, whether multilingual training transfers, whether pose supervision improves attention, and how far agents remain from the filtered human reference. e-motivatione-samplinge-annotatione-agente-gfe-multilinguale-grounding-resultse-test
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dataset category and multilingual/spatiotemporal/benchmark subcategories fit the contribution. Architecture, prediction paradigm and quadrant are not applicable to this dataset. Its baseline is a recurrent navigation policy with optional attention supervision; neither joint future/action prediction nor inverse dynamics is established, and pose traces are not a learned world model. e-datasete-agente-grounding
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Choose the route distribution before asking what an agent learns
RxR starts by changing the training problem itself. A shortest path between arbitrary endpoints can encourage an agent to infer a destination and take a shortcut. Here, a simple room sequence is chosen first, and panorama-level shortest-path search is constrained by that order. The path can therefore remain easy to describe at the room level while approaching its destination indirectly. Greedy selection then considers two properties: the ratio of unrestricted shortest distance to actual path length, and how often the candidate's panoramas have already appeared. Minimizing the first favors detours; minimizing the second spreads coverage. Reader interpretation: this makes path-adherence evaluation essential, because a model might reach the endpoint while violating the requested route. Figure 3 checks the resulting distribution, and Table 4 tests one family of straight-motion shortcuts. e-samplinge-path-statse-metrics
Figure 4. The same instruction supports related but nonidentical human paths and viewing histories. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down the Guide column and then the Follower column, comparing the instruction fragments associated with each panorama. The sequence moves past a wall painting into a gym and ends near a gray ball. Faded pixels indicate panorama regions the annotator did not observe; the red marker identifies the next selected action, as defined by the caption. It is not an eye-tracking fixation. The Follower takes additional moves but produces similar visual–textual alignments. Section 4 explains how speech timestamps, manual transcripts and recorded camera poses create these correspondences. Section 5 then distinguishes using the Follower path as another trajectory example from using visible-region masks to supervise attention. e-annotatione-alignmente-groundinge-gf
What it supports. The annotation contains more than a destination label: it records a human's changing visual access while producing or interpreting the instruction. The example makes alternative trajectories concrete, motivating both Guide-plus-Follower training and pose-derived attention supervision. Those are separate uses of the same collection process, with separate empirical comparisons.
Where the evidence stops. These are human demonstrations, not predictions from the navigation agent. Visibility is a coarse grounding cue and does not identify exactly which object a word denotes. Similar-looking alignments in one example do not establish alignment accuracy across the dataset.
5.2 Separate trajectory supervision from attention supervision
A Follower trace supplies an alternative instruction–path example even if the model never uses its camera orientations. RxR's baseline treats Guide and Follower pairs alike during behavioral cloning and policy-gradient training. Pose-based attention supervision is an additional mechanism: observed pixels are projected onto the agent's panoramic feature bins, while word timestamps identify text available at each step. On gold-path states, the auxiliary loss rewards attention mass inside the observed-view mask. Reader interpretation: the loss constrains where attention may concentrate without specifying a unique object-level interpretation among the allowed views. This also explains the boundary at inference: the learned recurrent policy still chooses actions from current visual and textual features, rather than consulting a live human trace. Table 5 tests combined trajectories and the auxiliary loss in different comparisons. e-annotatione-gfe-traininge-groundinge-agente-grounding-results
5.3 Keep data scaling, grounding and navigation success as separate conclusions
Three comparisons prevent a single optimistic story from obscuring the results. First, combining Guide and Follower paths improves success, but also increases the reported number of instruction–path pairs; the table alone cannot isolate interpretive diversity from exposure. Second, multilingual pooling and automatic cross-translations add examples while lowering performance for this architecture. The authors suggest that already-shared routes may limit translation value, but do not establish a general law about multilingual learning. Third, visual attention supervision improves Hindi and Telugu NDTW while leaving English unchanged, and its success changes are mixed. Reader interpretation: better route alignment need not imply better stopping or task completion. Finally, the test-standard table should anchor expectations: the strongest reported monolingual baseline remains far below the human reference, with different human and agent input protocols. e-gfe-multilinguale-grounding-resultse-teste-annotation
5.4 Training and inference
During training
VALAN minibatches mix 50% behavioral-cloning gold-path rollouts with 50% sampled policy-gradient rollouts. Rewards use incremental NDTW plus a linear navigation-error term after stopping. Adam runs for 100,000 iterations, batch size 32, initial learning rate 0.0001. Guide and Follower instruction–path pairs are treated as independent examples. e-traininge-gf
Optional grounding supervision converts observed pixels into a projected, max-pooled mask over visual features. On gold-path training steps, an auxiliary loss discourages attention outside this mask. An analogous text mask marks words spoken/heard at or before the step; textual attention supervision did not improve reported performance. e-groundinge-grounding-results
During inference
The agent repeatedly observes its current panorama and available graph moves, updates its recurrent state and predicts the next action. Pose-derived masks supervise training; the described inference policy does not require a human trace or predict future images. Action execution occurs in the graph simulator. e-agente-grounding
Evaluation reports navigation error, success, SPL, NDTW and SDTW. NDTW measures path adherence; SDTW additionally weights by success. These are prioritized because an indirect instruction route can differ from a shortest route to the same goal. e-metrics
5.5 Implementation flow
- Sample rooms, then panoramas
Form a room graph from connected components within room annotations. Generate simple room paths through at most five rooms and two building levels. Sample endpoints in the first and last rooms, then find the shortest panorama path constrained to that room sequence. Greedy selection favors detours and underrepresented viewpoints, capped at 40 m and 500 paths per building. e-samplinge-path-stats
- Collect and align human descriptions
Guides speak while navigating, then transcribe their recordings. Timestamped ASR output is aligned to the manual transcript using dynamic time warping, linking words to recorded camera poses. Followers navigate while listening and can pause, rewind or skip audio. Failure to finish within 3 m triggers another follower and, if necessary, reannotation; the most successful resulting pair is retained. e-annotation
- Preserve environment splits
The sampled paths comprise 11,089 train, 1,232 val-seen, 1,517 val-unseen and 2,684 test paths, following Matterport3D/R2R building splits. Val-seen uses training environments; unseen validation and test assess different environments. e-path-statse-datasheet
- Encode instructions and score actions
Pretrained multilingual BERT embeddings enter residual one-dimensional convolutions with ReLU activations. An LSTM decoder attends to encoded text and 36 panoramic views, conditions on the previous action, and ranks navigable directions by hidden-state/feature dot products. EfficientNet-B4 visual features are pretrained in an image–text dual encoder on Conceptual Captions and concatenated with angle encodings. e-agent
6. Experiments & results
Room-Across-Room (RxR) pairs multilingual navigation instructions with routes and synchronized human camera poses. Its sampling and annotation procedures make route adherence central to evaluation. The recurrent navigation baselines benefit from complementary demonstrations but remain far below human followers: test-standard success averages 25.9% for monolingual agents versus 93.9% for humans.
6.1 Read the original evidence
Figure 3. The sampled paths vary more in length and distribute visits more evenly across panoramas. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first two panels separately: the left uses meters and the middle uses navigation-graph edges. Their scales differ, so visual height is not directly comparable across panels. In the right panel, panoramas are sorted by visitation; the steep dashed R2R curve concentrates visits near the beginning, whereas the solid RxR curve is less concentrated. The dotted horizontal line is the uniform reference, not another measured dataset. The caption restricts these comparisons to train and validation because the R2R test set is sequestered. Read this as a diagnostic of the resulting path distribution, alongside the selection rule on the preceding page. e-path-statse-sampling
What it supports. Both length plots show a broader RxR distribution, and the visitation curve indicates more even coverage than R2R. The displayed meter labels are 14.9 for RxR and 9.4 for R2R, matching the prose's reported averages. This supports the sampling design's intended change in dataset structure, rather than directly measuring learned navigation quality.
Where the evidence stops. The RxR edge label is 7.0, while §3 reports an average of 8 edges. The box-plot summary statistic is not explicitly defined, and the plotted scope is train/validation; the paper does not reconcile these values. They should remain distinct.
Table 5. Complementary demonstrations help success, while multilingual expansion and attention supervision require more qualified conclusions. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First inspect the setting columns: G and F mark Guide and Follower path pairing, while X adds automatically cross-translated instructions. Mono denotes separate language-specific agents; Multi uses one agent across languages; Multi* adds visual attention supervision. Within every metric group, columns run English, Hindi and Telugu. Navigation error decreases with improvement, while success, SDTW and NDTW increase. Compare rows 1–3 for path supervision, rows 3–4 for multilingual training, rows 4–5 for translations and rows 4–6 for the attention loss. The reported pair counts change between several settings, so these comparisons answer different questions and do not all hold data exposure fixed. e-gfe-multilinguale-grounding-resultse-agent
What it supports. Guide-plus-Follower training attains English success of 26.1%, versus 25.6% with Guides and 23.9% with Followers alone. Its success advantage holds across the three language columns. Multilingual and cross-translated settings score lower than the corresponding simpler settings; visual attention supervision produces mixed changes rather than a uniform gain.
Where the evidence stops. The prose's every-metric improvement claim has an exception: Hindi NDTW is 45.5 with both path types versus 45.9 with Followers only. Pair counts also increase with combined supervision, and no seed variation is reported, limiting causal and significance claims.
Table 8. Heldout results expose substantial remaining headroom under the paper's benchmark protocol. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Locate the Test-Std block before comparing the agent and human rows; the upper blocks are seen and unseen validation, not additional test runs. Mono and Multi refer to the Guide-plus-Follower agents from Table 5, without the Multi* attention-loss variant. Each metric group has English, Hindi, Telugu and an average column. Read success alongside NDTW: reaching the goal and following the described route are related but distinct requirements. The paper also defines a separate sequestered test-challenge split, whose results are not presented here. The human row comes from Follower annotations collected with audio playback and the quality-control process described in Section 4. e-teste-gfe-annotation
What it supports. On test-standard, average success is 25.9% for monolingual agents and 21.3% for the multilingual agent, compared with 7.5% random and 93.9% human performance. Average NDTW is 41.5 for monolingual agents and 79.5 for humans. The gap concerns both goal completion and route adherence in the simulator.
Where the evidence stops. Human followers heard audio, could control playback and were subject to pair selection/reannotation; agents consume text. The human row is therefore a benchmark reference rather than a matched-input experiment. These results establish neither physical deployment nor statistical significance across training seeds.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RxR heldout navigation Table 8, test-standard; Guide+Follower training; averages across English, Hindi and Telugu. | Monolingual: 25.9 / 21.0 / 41.5. SR (%) / SDTW / NDTW; higher is better | Multilingual: 21.3 / 17.1 / 37.2; random: 7.5 / 3.1 / 15.4; human followers: 93.9 / 76.9 / 79.5. A large simulated-navigation gap remains. Human collection uses audio and quality filtering, so the human row is a task reference, not a matched-input ablation. e-teste-annotation |
| Complementary Guide and Follower supervision Table 5, val-unseen; monolingual experiments 1–3, language order en/hi/te. | Guide+Follower: 26.1 / 29.6 / 29.8. SR (%) | Guide only: 25.6 / 24.8 / 28.0; Follower only: 23.9 / 28.0 / 27.0. Success improves in every language, but the prose's every-metric claim is too broad: Hindi NDTW is 45.5 combined versus 45.9 Follower-only. More training pairs also confound attribution. e-gf |
| Multilingual training and automatic translation Table 5, val-unseen; experiments 3, 4 and 5; en/hi/te. | Multilingual Guide+Follower: 22.2 / 23.0 / 23.1. SR (%) | Monolingual: 26.1 / 29.6 / 29.8; multilingual with cross-translations: 20.0 / 18.7 / 20.3. More multilingual or translated examples do not improve this baseline; this does not establish an inherent disadvantage of multilingual navigation. e-multilingual |
| Visual attention supervision Table 5, val-unseen; Multi* experiment 6 versus Multi experiment 4; en/hi/te. | Multi*: NDTW 38.6 / 39.9 / 39.7; SR 21.9 / 22.6 / 23.2. NDTW / SR (%) | Multi: NDTW 38.6 / 39.2 / 38.8; SR 22.2 / 23.0 / 23.1. Hindi/Telugu path adherence improves while English ties; success changes are mixed. No uncertainty estimates establish significance. e-grounding-results |
| R2R–RxR multitask learning Table 6, val-unseen; R2R baseline omits generated-instruction augmentation and uses RxR-tuned hyperparameters. | Joint training: R2R 52.3; RxR en/hi/te 38.8 / 39.4 / 38.4. NDTW | R2R-only on R2R: 48.1; RxR-only on RxR: 38.6 / 39.2 / 38.8. R2R benefits, but the claim of uniformly best multitask performance has exceptions, including Telugu NDTW. Cross-dataset transfer is weaker than in-domain training. e-transfer |
| Dependence on vision and language Table 7, English val-unseen; modality removed during both training and evaluation. | Both modalities: 22.2 / 38.6. SR (%) / NDTW | Language-only: 16.0 / 30.9; vision-only: 7.8 / 16.5. Both modalities help. Language-only still uses navigable graph directions, so its performance does not demonstrate image grounding. e-ablatione-agent |
| Resistance to straight-line route priors Table 4, val-unseen; oracle first step followed by straight motion, stopping after each dataset's mean training-path step count. | RxR: 13.7 / 25.9. SR (%) / NDTW | R2R: 27.2 / 52.6; stopping budgets are 8 and 5 steps respectively. The heuristic is less effective on RxR. This cross-dataset diagnostic changes routes and horizon together and partly uses the gold path. e-metrics |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 7. Removing either input weakens the baseline, but meaningful language-only performance remains. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the checkmarks to identify the retained modality before reading the numbers: row 4 retains both, row 9 retains language, and row 10 retains vision. The experiment removes the missing input during both training and evaluation, so it tests a separately trained restricted agent rather than an abrupt test-time failure. Compare a fixed language column within each metric group; lower navigation error and higher success or path-adherence scores indicate improvement. Section 5 explains why language can still help without images: commands about direction remain meaningful relative to the available navigation graph. Vision alone cannot identify which instruction-dependent route was requested. e-ablatione-agent
What it supports. For English, the full model reaches 22.2% success and 38.6 NDTW; language-only gives 16.0% and 30.9, and vision-only gives 7.8% and 16.5. The ordering supports a contribution from both inputs, while showing that useful task information remains available without visual recognition.
Where the evidence stops. Language-only is not environment-free: navigable directions and graph structure remain available. This ablation therefore does not prove that every successful full-model episode required visual grounding, nor that all route priors have been eliminated.
7. Analysis & limitations
7.1 What the evidence leaves open
Camera poses measure what was visible, not eye fixation or exact object–word correspondence. The linguistic analysis covers only 25 sampled paths; the authors cannot separate language, dialect and annotator-pool effects. e-alignmente-language-analysis
Human success is conditioned on retraining/reannotation and pair selection. It should not be read as an unfiltered first-attempt success rate. The graph-based results do not establish physical robot or continuous-control performance. e-annotatione-annotatorse-test
Figure 3 labels the RxR edge statistic 7.0, whereas the path-statistics prose reports an average of 8. The figure is restricted to train/validation and does not explicitly define its box-plot summary; the relationship is unresolved. Figure 2's caption also calls the endpoints r8/r6 although the diagram and section define them as panoramas p8/p6. e-path-statse-sampling
Guide audio is withheld pending privacy review of home recordings during COVID-19. The publication documents text and pose data, not an available spoken-input benchmark release. e-annotatorse-datasheet
7.2 Questions for discussion
- Does Follower supervision help because it supplies alternative interpretations, or because it increases training exposure? [e-gf]
- Would pose supervision help more with perspective-camera agents than with panoramic agents? [e-alignment, e-grounding, e-test]
8. Reproducibility audit
8.1 Requirements and known gaps
Required resources include Matterport3D reconstructions, R2R navigation graphs, RxR transcripts/paths/poses, multilingual BERT, Conceptual Captions visual pretraining and VALAN. The datasheet specifies CC-BY for RxR and separate Matterport3D terms; it reports collection during March–September 2020. e-agente-traininge-datasheete-access
The publication plans code, annotation-tool and test-server releases. It does not provide complete CNN configuration, loss/reward coefficients, random seeds, hardware or software versions; the full freezing policy and evaluation action-selection rule are not explicit. e-agente-traininge-access
Proposed checks: compare Guide+Follower training against an exposure-matched duplicated-Guide control, and compare genuine pose masks against shuffled masks and no auxiliary loss. Match optimization budgets and evaluate per-language val-unseen NDTW, SDTW and success over multiple seeds. e-gfe-groundinge-grounding-results
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does an alternative human path help beyond additional exposure?
Reader-proposed, not run: train three monolingual conditions for each language—Guide-only, Guide-plus-Follower, and Guide-only with duplicated pairs. Match total sampled examples, optimizer updates, features and model capacity between the combined and duplicated conditions; keep the original building splits. Evaluate per-language val-unseen SR, SDTW and NDTW over multiple seeds, grouping uncertainty estimates by building. A reproducible combined-path advantage over duplication would support value from alternative human trajectories. If duplication matches the gain, the original table would be consistent with an exposure explanation. Track Hindi NDTW separately because the published combined setting trails Follower-only there. e-gfe-traininge-path-stats
Check 2: Do pose masks teach useful grounding or only suppress attention?
Reader-proposed, not run: keep multilingual Guide-plus-Follower training fixed and compare no auxiliary loss, genuine pose masks, and masks with view indices permuted while preserving the number of active bins at every step. Apply losses on the same gold-path states with matched weights and training budgets. Measure heldout gold-path attention outside genuine observed regions and free-running val-unseen SR, SDTW and NDTW, reporting every language over multiple seeds. If genuine masks improve both attention alignment and navigation beyond permuted masks, the grounding interpretation gains support. If attention alignment improves without navigation gains, or permuted masks perform similarly, the published mixed result may reflect regularization or an insufficient connection between attention and action. e-groundinge-grounding-resultse-traininge-agent
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, all figures (1–11), all tables (1–8), the §5 model/training/attention-loss page, Appendix A and the full embedded datasheet. The neural baseline is specified in prose; Figure 2 is the original dataset-method diagram. Six final original crops were individually inspected, including labels, arrow directions, metric headers, legends and Table 5's setting key. Figure 2 endpoint naming, Figure 3 edge summaries and Table 5/6 prose-versus-cell differences are disclosed. References on PDF pages 10–11 were read as text but not rendered; no retained scientific detail depends on them. External code, audio, dataset files, linked supplements and later revisions were outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, abstract and 1 Introduction (PDF p. 1–2; printed p. 4392–4393)
- 2 Motivation (PDF p. 2–3)
- 3 Two-Level Path Sampling (PDF p. 3–4)
- 4 Data Collection and Metrics (PDF p. 4–6)
- 5 Experiments (PDF p. 7–9)
- 6 Conclusion (PDF p. 9)
- Acknowledgments and References (PDF p. 10–11)
- A Supplementary Material: annotators, collection interface, alignment and instruction examples (PDF p. 12–18)
- B Datasheet: motivation, composition, collection, preprocessing, uses, distribution and maintenance (PDF p. 19–21)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity matches the catalog title and all five authors. The inspected artifact is the EMNLP 2020 proceedings paper, printed pages 4392–4412, dated November 16–20, 2020. No explicit revision number or revision chain is supplied; no later edition or updated online datasheet was compared (e-identity).
- The acquisition notes that text extraction does not reconstruct figure images. This limitation was addressed by inspecting the original PDF figures and tables, including all final crops.
- Separate supplemental material availability has not been fully verified. Appendices A and B embedded in this PDF were read completely.
- Code, dataset files, audio, linked websites and external cited works were not inspected; no experiments were reproduced. Hindi and Telugu example panels were inspected as supplied, without an independent linguistic annotation or translation audit.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1 (printed p. 4392), title, author/affiliation block, equal-contribution footnote and proceedings footer
Exact title and authors: Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie and Jason Baldridge. Affiliations are Google Research and Brown University; the first two authors contributed equally. Footer identifies EMNLP 2020, November 16–20, pages 4392–4412.
Go to primary source ↓e-motivationPDF p. 2–3 (printed p. 4393–4394), §2 Multilinguality, Scale, Fine-Grained Grounding, Follower Demonstrations and Path Desiderata; Table 1
Instructions originate independently in English, Hindi and Telugu. The design addresses limited scale, coarse grounding and shortest-path/length priors; Table 1 reports 126K instructions and 9.8M words.
Go to primary source ↓e-samplingPDF p. 3 (printed p. 4394), §3 Preliminaries, Path Generation, Greedy Selection for Coverage; Figure 2 and caption
Room vertices derive from connected components within annotated rooms. Simple room paths visit at most five rooms and two levels; shortest panorama routes respect that order. Greedy scoring combines endpoint-distance/path-length ratio with mean previous panorama visitation. Figure 2 arrows run p8→p13→p10→p7→p11→p6 through r0,r2,r3; caption's r8/r6 endpoint names differ from panel labels.
Go to primary source ↓e-path-statsPDF p. 4 (printed p. 4395), §3 continuation and Path Statistics; Figure 3 and caption
Selection limits are 40 m and 500 paths/building. Splits contain 11,089/1,232/1,517/2,684 paths. Prose reports RxR averages of 14.9 m and 8 edges, versus R2R 9.4 m and 5 edges; 44.5% of RxR paths are not shortest and mean excess length is 27.4%. Figure 3's RxR edge label is 7.0; caption restricts plotted comparison to train/val.
Go to primary source ↓e-datasetPDF p. 5 (printed p. 4396), Table 2 Counts/Averages and §4 Guide Task/Dataset Analysis; PDF p. 2, Table 1
Table 2 totals are 126,069 instructions and 16,522 paths; English/Hindi/Telugu instruction counts are 42,002/42,068/41,999. Mean instruction length is 78 whitespace words. Approximately 12.8K paths are shared across all three languages and 1.2K per language are unique; English paths receive two Indian and one US instruction.
Go to primary source ↓e-annotationPDF p. 4–5 (printed p. 4395–4396), §4 Guide Task and Follower Task; Figure 4 and caption
Guides use a 640×480 canvas and 75° vertical field of view, speak and manually transcribe, and receive word timing through ASR/manual-transcript DTW alignment. Timestamped 6-DOF poses are recorded. Followers can control playback. Success is within 3 m; additional followers and reannotation select the most successful pair. Figure 4 shows distinct Guide/Follower routes with similar alignments.
Go to primary source ↓e-alignmentPDF p. 4, Figure 4 and caption; PDF p. 6 (printed p. 4397), Figure 5 and accompanying §4 paragraphs
Figure 4 marks the next action in red and fades unobserved panorama pixels. Figure 5 relates word progress to path progress and shows wider viewing at the first viewpoint. Reported visible fractions are Guide/Follower 43%/44% initially and 27%/28% subsequently; these are virtual-camera observations.
Go to primary source ↓e-language-analysisPDF p. 5–6 (printed p. 4396–4397), Dataset Analysis, Table 3 and following paragraph
Manual linguistic analysis uses instructions for 25 sampled paths; Table 3 reports sentence prevalence and frequency. The authors explicitly leave language/dialect/annotator-pool explanations unresolved.
Go to primary source ↓e-metricsPDF p. 6 (printed p. 4397), §4 Evaluation, Table 4 rows 1–3 and caption
The metrics are PL, NE, SR, SPL, SDTW and NDTW, with emphasis on path adherence. Simple baselines stop after the mean training-path step count, R2R 5/RxR 8. Oracle-first-step straight motion yields R2R SR 27.2/NDTW 52.6 and RxR SR 13.7/NDTW 25.9 on val-unseen.
Go to primary source ↓e-agentPDF p. 7 (printed p. 4398), §5 Agent
Residual ReLU CNN instruction encoding uses pretrained multilingual BERT features; the LSTM attends to text and 36 panorama features and ranks navigable-direction encodings against its hidden state. Visual features use EfficientNet-B4 pretrained with Conceptual Captions in an image–text dual encoder, plus angles.
Go to primary source ↓e-groundingPDF p. 7 (printed p. 4398), §5 Grounding Supervision, mask definitions and unnumbered attention-loss equation
Text masks indicate words spoken/heard at or before each step. Binary observed-pixel masks are projected/max-pooled onto the 36 views. On gold-path training states, the log-partition difference loss penalizes attention outside observed inputs; textual supervision is analogous.
Go to primary source ↓e-trainingPDF p. 7 (printed p. 4398), §5 Implementation Details
VALAN training mixes equal proportions of behavioral cloning and sampled policy-gradient rollouts. Reward uses incremental NDTW and a linear navigation-error term after stopping. Adam uses 100K iterations, batch size 32 and initial learning rate 1e-4; detailed coefficients, seeds, hardware and software versions are not given.
Go to primary source ↓e-gfPDF p. 7–8 (printed p. 4398–4399), §5 Monolingual Results; Table 5 rows (1)–(3), SR and NDTW columns
Guide/Follower pairs are independent training examples. Table 5 lists 42K pairs for each single-path setting and 84K combined. Combined SR exceeds both alternatives in all languages, but Hindi NDTW is 45.5 combined versus 45.9 Follower-only despite the prose saying every metric benefits.
Go to primary source ↓e-multilingualPDF p. 8 (printed p. 4399), Table 5 rows (3)–(5); §5 Multilinguality
Monolingual SR en/hi/te is 26.1/29.6/29.8; multilingual is 22.2/23.0/23.1; adding automatic cross-translations gives 20.0/18.7/20.3. Most Guide paths are shared across languages; the authors suggest this may limit translation value.
Go to primary source ↓e-grounding-resultsPDF p. 8 (printed p. 4399), Table 5 rows (4), (6), NDTW/SR/NE/SDTW columns; §5 Spatiotemporal Grounding Supervision
Visual supervision changes NDTW from 38.6/39.2/38.8 to 38.6/39.9/39.7 and SR from 22.2/23.0/23.1 to 21.9/22.6/23.2. The authors call results mixed and textual supervision unsuccessful. No seed variation or confidence intervals are reported.
Go to primary source ↓e-transferPDF p. 8–9 (printed p. 4399–4400), Table 6 rows (7), (4), (8); §5 Multitask and Transfer Learning
Joint R2R/RxR training gives R2R NDTW 52.3 versus 48.1 R2R-only, and RxR 38.8/39.4/38.4 versus 38.6/39.2/38.8 RxR-only. Cross-transfer is weaker. R2R-only omits generated-instruction augmentation and uses RxR-tuned hyperparameters; Telugu columns contain exceptions to the broad best-performance claim.
Go to primary source ↓e-ablationPDF p. 9 (printed p. 4400), Table 7 rows (4), (9), (10), English columns; §5 Unimodal Ablations
Vision or language is removed during training and evaluation. English SR/NDTW is 22.2/38.6 with both, 16.0/30.9 language-only and 7.8/16.5 vision-only. The authors explain that graph directions preserve some meaning of language without images.
Go to primary source ↓e-testPDF p. 9 (printed p. 4400), Table 8 Test-Std rows and average columns; §5 Test Set; §6 Conclusion
Test-standard average SR/SDTW/NDTW is monolingual 25.9/21.0/41.5, multilingual 21.3/17.1/37.2, random 7.5/3.1/15.4 and human 93.9/76.9/79.5. Test-standard and test-challenge are held out. Perspective-camera and continuous-simulator extensions are future opportunities.
Go to primary source ↓e-annotatorsPDF p. 12 (printed p. 4403), Appendix A Annotators
247 annotators contributed, 97 in the USA and the remainder in India. Guides do not repeat paths or follow their own instructions. Performance below the 80% target prompts training or removal. Audio is withheld pending privacy/safety review of home recordings during COVID-19.
Go to primary source ↓e-datasheetPDF p. 19–20 (printed p. 4410–4411), Appendix B Composition, recommended splits, external dependencies and Collection
Instances contain transcripts, viewpoint paths and timestamped camera traces for Guides/Followers; audio is absent. Building IDs and Matterport3D/R2R splits are retained. Dependencies are Matterport3D reconstructions and R2R graphs. Collection ran March–September 2020.
Go to primary source ↓e-accessPDF p. 2 (printed p. 4393), opening release statement; PDF p. 21 (printed p. 4412), Appendix B Preprocessing, Uses, Distribution and Maintenance
The paper states the dataset is distributed, while code, annotation tool and test server are planned. The datasheet specifies GitHub/Google Cloud Storage distribution, CC-BY licensing, separate Matterport3D terms, and no planned dataset updates. These are publication-time statements.
Go to primary source ↓8.5 Primary sources
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding ↗
PDF · 12,552 extracted words
Source fingerprint
c9e0c0bfeeb2726fcbb8d318b66b01863e01a808de502bbefae3ebadcadbbe0b