On Evaluation of Embodied Navigation Agents
1. Paper overview
In one sentence: SPL rewards explicitly completed navigation and efficient paths, but meaningful comparison also requires matching goals, sensors, exploration access and success rules. e-identitye-purposee-goalse-exposuree-stope-distancee-sple-source-scope
| At a glance | What to know |
|---|---|
| Research problem | Source description Navigation research had accumulated incompatible goals, simulator assumptions and evaluation protocols. A high score could therefore reflect different sensory privileges or prior exposure to the test environment. The authors seek a common task framework that makes such assumptions explicit while preserving freedom to investigate internal representations and control methods. e-purposee-goalse-exposuree-architectures |
| Core mechanism | Source description The report separates PointGoal, ObjectGoal and AreaGoal from the modality used to communicate each goal, and distinguishes unseen-environment evaluation from recorded or agent-controlled prior exploration. e-goalse-exposure |
| Reading caution | Source description There is no measured agent comparison, ablation or human calibration experiment. The authors expect SPL 0.5 to represent good performance in complex unseen environments and suggest dataset-specific human calibration; this expectation is not an empirical performance threshold. e-sple-source-scope |
Core contributions
- Source description
The report separates PointGoal, ObjectGoal and AreaGoal from the modality used to communicate each goal, and distinguishes unseen-environment evaluation from recorded or agent-controlled prior exploration. e-goalse-exposure
- Source description
It recommends explicit completion, geodesic proximity and Success weighted by normalized inverse Path Length (SPL), combining binary success with efficient travel in one summary score. e-stope-distancee-spl
- Source description
It specifies scenario splits and task settings for SUNCG, Matterport3D, AI2-THOR and Gibson, alongside recommendations for continuous simulator state, SI units and software supporting physical deployment. e-testbedse-scenariose-suncg-matterporte-thore-gibson
Equation (1), original mathematical display. The original SPL equation combines a completion gate with a normalized path-efficiency factor. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the product inside the summation first. The binary success indicator S_i decides whether episode i contributes at all; the paper defines success at the agent's DONE action. For a successful episode, ℓ_i is the shortest-path distance from its start to its goal, and p_i is the path actually traveled. The denominator is max(p_i, ℓ_i), as confirmed in the surrounding symbol definitions; it prevents a contribution larger than one. Finally, sum across all N test episodes and divide by N, retaining failures in that average. These are trajectory measurements. The equation does not prescribe an agent architecture or action-selection procedure. e-sple-stope-distancee-exposuree-architecturese-source-scope
What it supports. The source's worked examples yield SPL 0.5 both when half the episodes succeed along optimal paths and when every episode succeeds along paths twice optimal. Half success with twice-optimal successful paths yields 0.25. The equality illustrates how reliability and travel efficiency can trade off inside one score; these values are explanatory calculations.
Where the evidence stops. The crop is an equation, not an experimental chart or a numbered figure. Its interpretation depends on the success rule, geodesic calculation and prior-exploration regime. SPL alone cannot reveal whether equal scores arise from frequent failures or inefficient successful routes.
2. Motivation
2.1 The problem and the proposed response
Navigation research had accumulated incompatible goals, simulator assumptions and evaluation protocols. A high score could therefore reflect different sensory privileges or prior exposure to the test environment. The authors seek a common task framework that makes such assumptions explicit while preserving freedom to investigate internal representations and control methods. e-purposee-goalse-exposuree-architectures
2.2 What this reading follows
A navigation score depends on what an agent was asked to find, what it could sense, and how much it already knew about the scene. This working-group report turns those assumptions into an evaluation protocol. Its central equation weights each successful episode by travel efficiency, while its surrounding recommendations define completion, traversable distance and prior exploration. Read the equation together with those conditions: the same SPL can describe very different behaviors. The supplied 2018 v1 contains protocol recommendations and scenario specifications, with illustrative arithmetic rather than measured agent results. The edition therefore centers on its original mathematical display. e-identitye-purposee-goalse-exposuree-stope-distancee-sple-source-scope
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded evaluation-metrics classification is supported by SPL, its success protocol and scenario recommendations. Architecture, prediction paradigm and quadrant are correctly not applicable: discussing environmental representations does not instantiate a world-action model, joint future/action predictor or inverse-dynamics controller. e-purposee-sple-architectures
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Establish what information the agent may use
Begin with the goal, then separate its content from how it is communicated. PointGoal selects a location; ObjectGoal and AreaGoal select object and area categories. An image or a language description can communicate a goal and may select a particular instance. Next declare the sensors and localization privileges: an agent receiving perfect odometry has different information from one estimating motion from images. Finally account for prior exposure. The paper distinguishes no exploration, a recording collected by a third party, and exploration chosen by the agent within a trajectory-length budget. In the last case, evaluate navigation over a range of budgets instead of reporting one unexplained amount of preparation. This protocol places each SPL measurement within a specified information and experience regime. e-goalse-exposure
5.2 2. Make completion meaningful before averaging efficiency
The source first requires an agent to declare DONE, so evaluation uses the configuration it chooses as its endpoint. A trajectory that merely passes the goal does not qualify. Next use geodesic distance, because a wall can separate locations that are close in Euclidean space. The general proximity threshold is twice the agent's body width; AreaGoal instead tests whether the center of mass lies inside the requested area. Only after deciding success does Equation (1) apply the path-efficiency factor. Its examples deliberately expose an ambiguity: reliable but indirect navigation and less reliable optimal navigation can both score 0.5. Reader interpretation: retain success and efficiency diagnostics together when diagnosing a score, because the average alone does not identify the underlying behavior. e-stope-distancee-sple-auxiliary
5.3 3. Carry dataset-specific rules into the evaluator
The standard scenarios do not erase differences among environments. SUNCG and Matterport3D specify a 0.2 m body width and 0.4 m success threshold. AI2-THOR v1.0 builds 125 PointGoal scenarios per test scene from five targets, five starts and five scene configurations. Its ObjectGoal setting uses visibility within 1 m and the camera field of view, and distinguishes navigation-only targets from targets requiring opening or closing objects. Gibson allows different agent action spaces and notes that geodesic distance can change with embodiment; its default calculation uses a fixed height over the floor and same-floor endpoints. Reader interpretation: a shared SPL implementation is only part of a comparable experiment. Preserve these task-specific success rules, reachable spaces and exposure settings when interpreting differences between scores. e-suncg-matterporte-thore-gibsone-exposuree-distance
5.4 Training and inference
During training
This is a protocol and metric contribution: no new network, optimization loss, training stages, frozen modules or compute budget is specified. Section 7 discusses reactive, recurrent-memory and richer-memory agents without selecting or training one. e-purposee-architecturese-source-scope
The training/validation/test environment splits are SUNCG 300/100/100 and Matterport3D 61/11/18. AI2-THOR v1.0 assigns the first 20, next five and last five scenes of each of four categories to those splits, yielding 80/20/20 scenes. Gibson defines tiny 25/5/5, medium 100/20/20 and full 402/85/85 partitions. e-suncg-matterporte-thore-gibson
During inference
SPL evaluates an executed navigation trajectory; it does not generate actions, predict future observations or provide inference-time planning. The agent remains responsible for movement and completion. The architecture discussion spans sensory-input-to-action reactive networks, recurrent vector memory and richer environmental representations. e-stope-sple-architectures
AI2-THOR PointGoal offers forward/backward movement, left/right rotation and Terminate. ObjectGoal adds looking up/down; its interaction-based setting additionally permits Open X and Close X. Gibson leaves the agent and action space configurable, including discrete or continuous navigation commands. e-thore-gibson
5.5 Implementation flow
- Define the navigation problem
PointGoal identifies a location, ObjectGoal an object category, and AreaGoal an area category. Coordinates or category labels are basic specifications; images and language can also communicate goals, sometimes selecting a particular instance. Declare RGB, depth, haptics, map access and any perfect odometry or GPS assumptions. e-goals
- Separate exploration from evaluation
Report whether the environment is unseen, supplied through a third-party exploration recording, or explored by the agent within a trajectory-length budget. The last regime varies exposure to obtain a navigation-versus-exploration profile. The source also suggests measuring exploration itself by environment coverage. e-exposure
- Evaluate the agent's decision to finish
A dedicated DONE action triggers evaluation of the current configuration. Passing near the goal earlier is insufficient, and absence of DONE means failure. Geodesic distance respects intervening walls. The general PointGoal/ObjectGoal rule uses distance below a threshold of twice body width; AreaGoal uses the center of mass lying inside the target area. e-stope-distance
- Aggregate success and travel efficiency
For every test episode, multiply binary success by shortest-path distance divided by the larger of actual and shortest-path length, then average over all episodes. Failed episodes contribute zero. The maximum in the denominator caps successful contributions at one; this is a consequence of the displayed formula. e-spl
- Retain diagnostic information
Alongside SPL, the authors suggest terminal distance, success-versus-efficiency curves, threshold sweeps, distributions of normalized inverse path length, obstacle contacts, and actuation time or energy. These characterize aspects of performance compressed by the primary score. e-auxiliary
6. Experiments & results
This working-group report defines how to compare embodied navigation agents: specify goals, sensors and prior exploration, evaluate an explicit completion action using traversable distance, and summarize successful navigation with SPL. Its contribution is an evaluation contract and standard scenarios across four environment datasets. It proposes no trained agent and reports no measured agent comparison; the numerical SPL examples explain the metric.
The complete supplied v1 is a consensus metric and protocol report. It contains no original figures, tables, empirical agent comparisons or ablation studies; Section 7 describes architecture families only in prose. Equation (1) is its sole displayed mathematical visual and is faithfully cropped here as a figure-kind asset, explicitly labeled as an equation. One crop is therefore appropriate: adding architecture, quantitative-result or ablation visuals would require recreating material or treating body prose as a graphic. The edition makes no empirical featured-result selection. e-source-scopee-purposee-sple-architectures
6.1 Read the original evidence
No results visual is included in this edition. The source-specific visual limitations are recorded above.
6.2 Results and evaluation conditions
No quantitative results are included in this reading.
6.3 Ablations and diagnostic examples
No ablation or diagnostic visual is included in this edition. The source-specific visual limitations are recorded above.
7. Analysis & limitations
7.1 What the evidence leaves open
There is no measured agent comparison, ablation or human calibration experiment. The authors expect SPL 0.5 to represent good performance in complex unseen environments and suggest dataset-specific human calibration; this expectation is not an empirical performance threshold. e-sple-source-scope
Reader analysis: equal SPL can conceal different reliability and efficiency, as the paper's two 0.5 examples demonstrate. Binary failure also gives no partial credit for progress. Consequently, a single SPL value cannot explain how an agent fails; the proposed auxiliary measures remain important. e-sple-auxiliary
The dataset rules are not interchangeable. AI2-THOR ObjectGoal defines success through target visibility within 1 m and the camera field of view, differing from the general geodesic proximity formulation. Its single-room scenes exclude AreaGoal. Gibson's agent-dependent reachability and same-floor sampling also constrain comparisons. e-distancee-thore-gibson
Section 6 first states more than 100 configurations per scene, but the Gibson subsection specifies 100 point pairs per space. Preserve this difference when implementing scenario counts. Simulator reproducibility and recommended deployment software also do not establish transfer success on physical robots. e-scenariose-gibsone-testbeds
7.2 Questions for discussion
- When two agents share SPL but differ in success rate and path efficiency, which auxiliary measures best reflect the intended deployment?
- How should prior exploration budgets and agent-specific geodesics be standardized before comparing navigation scores?
- Should interaction-based ObjectGoal evaluation report interaction costs separately from traveled path length?
8. Reproducibility audit
8.1 Requirements and known gaps
A protocol implementation requires the original scene partitions and scenario specifications, a traversability-aware shortest-path routine, trajectory logging and explicit terminal success evaluation. Start/goal pairs are sampled in free space and checked by a tile-based shortest-path computation, with minimum distance 1 m. The paper links scenario files but does not enumerate scene identifiers, random seeds or tile resolution in its text. e-scenariose-suncg-matterporte-gibson
Keep geometry and thresholds consistent: SUNCG and Matterport3D use body width 0.2 m and threshold 0.4 m. Gibson's default distances are computed at fixed height over the floor and may need recalculation for the chosen agent; its default threshold is 0.4 m, with twice body width suggested generally. e-suncg-matterporte-gibson
Reconstruct AI2-THOR settings separately: five targets, five starts and five scene configurations yield 125 PointGoal scenarios per test scene; ObjectGoal similarly uses five categories, starts and configurations. Document sensors, control space, noise and odometry privileges. A universal episode step limit and a concrete learned-agent training recipe are not supplied. e-thore-testbedse-architecturese-source-scope
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check the completion gate and SPL arithmetic
Reader-proposed check, not performed: construct controlled episode logs with the same positive shortest-path distances. First verify the paper's three combinations of success rate and successful path length, expecting SPL 0.5, 0.5 and 0.25. Then hold a successful trajectory fixed and remove its DONE signal: its contribution must become zero. Add a trajectory that passes within the success threshold but issues DONE after leaving it, and one ending across a wall with small Euclidean but large geodesic distance. Under the general proximity protocol, neither should be credited as successful. A mismatch would falsify conformance of the evaluator to the stated completion, distance or aggregation rules. e-stope-distancee-spl
Check 2: Measure the exploration–navigation trade-off under matched conditions
Reader-proposed comparison, not performed: evaluate the same memory-capable agent on the same held-out start/goal episodes after no prior exploration and after controlled trajectory-length exploration budgets, for example the paper's 500 m and 1,000 m budgets where feasible. Hold weights, sensors, body geometry, success threshold, episode limit and geodesic implementation fixed; reset memory between scenes and preserve only the permitted exploration memory within a condition. Report SPL alongside success rate and terminal distance. Improvement would support the usefulness of that agent's acquired representation under these conditions; a flat or decreasing profile would refute an assumption that more exploration necessarily helps this agent. This tests a protocol-motivated question, not a result claimed by the paper. e-exposuree-architecturese-auxiliarye-distancee-gibson
8.3 Reading coverage
Visual audit: All seven PDF pages were rendered and visually inspected. Page 1 establishes the exact title, 11 authors, affiliations and v1 date; page 2 supports goals, sensors and exploration; page 3 supports DONE, geodesic proximity, Equation (1), its symbols and numerical examples; page 4 supports auxiliary metrics, simulator recommendations and scenario construction; page 5 supports dataset splits, counts, thresholds and AI2-THOR action/success rules; page 6 supports Gibson geometry, configurable controls and the architecture discussion; pages 6–7 contain the references. The final 600 DPI equation crop was separately viewed and retains the complete expression and equation number without surrounding prose. No empirical tables, architecture diagrams, ablation pages or appendices exist in this supplied PDF; separate supplements and linked artifacts were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, all 11 authors and affiliations; arXiv:1807.06757v1 [cs.AI], 18 Jul 2018; Abstract; Section 1, Introduction
- PDF p. 2: Section 2, Goal Specification and Sensory Input
- PDF pp. 2–3: Section 3, Generalization and Exploration
- PDF pp. 3–4: Section 4, Evaluation Measures; Recommendations 1–3; Equation (1)
- PDF p. 4: Section 5, Experimental Testbeds; Recommendations 4–6
- PDF pp. 4–6: Section 6, Standard Scenarios; SUNCG, Matterport3D, AI2-THOR and Gibson
- PDF p. 6: Section 7, Agent Architectures; Recommendation 7; Section 8, Acknowledgments
- PDF pp. 6–7: References [1]–[32]
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The supplied text extraction does not reconstruct figure images. Its visual gap was addressed by inspecting all seven PDF pages and the final original Equation (1) crop.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- Linked scenario files, code and cited works were not inspected. No experiments were reproduced.
- Edition scope is the supplied arXiv v1 dated 18 July 2018. Its title and 11-author list match the catalog author string; the supplied BibTeX uses Angel X. Chang where the title page reads Angel Chang. No later revision was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, author affiliations and arXiv margin stamp
The title is On Evaluation of Embodied Navigation Agents. Eleven named authors and their institutional affiliations are printed; the stamp identifies arXiv:1807.06757v1 [cs.AI], 18 Jul 2018.
Go to primary source ↓e-purposePDF p. 1, Abstract and Section 1, final three paragraphs
A working group presents consensus recommendations on navigation problem statements, generalization, evaluation measures and standard scenarios to address incompatible experimental methodologies.
Go to primary source ↓e-goalsPDF p. 2, Section 2, three goal bullets and subsequent specification/sensor paragraphs
PointGoal targets a location, ObjectGoal an object category and AreaGoal an area category. Goal modalities include coordinates, labels, images and language. Sensor access and idealized localization assumptions should be explicit.
Go to primary source ↓e-exposurePDF p. 2, Section 3, three exploration-regime bullets; PDF p. 3, opening paragraph
No exploration, third-party recorded exploration and agent-controlled exploration with a trajectory-length budget are distinguished. Performance versus exposure and coverage within a path-length budget are suggested evaluations.
Go to primary source ↓e-stopPDF p. 3, Section 4, completion discussion, Recommendation 1 and binary-success paragraph
Evaluation occurs when the agent issues DONE, using that configuration rather than an earlier favorable one. An episode without the signal is unsuccessful.
Go to primary source ↓e-distancePDF p. 3, Section 4, Recommendation 2 and success definition immediately above Equation (1)
Geodesic distance accounts for environmental reachability. The general PointGoal/ObjectGoal success condition is distance below τ, with twice body width recommended; AreaGoal requires the center of mass inside the specified area.
Go to primary source ↓e-splPDF p. 3, Section 4, Equation (1), symbol definitions, three worked examples and final paragraph
SPL averages S_i ℓ_i/max(p_i,ℓ_i) over N test episodes. Examples produce 0.5, 0.5 and 0.25 under distinct success/path-length conditions. SPL 0.5 as good unseen-environment performance is an expectation; human-subject calibration is proposed.
Go to primary source ↓e-auxiliaryPDF p. 4, Recommendation 3 and Section 4 auxiliary-measures paragraph, items (a)–(f)
SPL is recommended as the primary score, complemented by success/efficiency profiles, terminal distance, threshold sweeps, normalized path-length distributions, infractions, actuation time and energy.
Go to primary source ↓e-testbedsPDF p. 4, Section 5, Recommendations 4–6 and customization/deployment paragraphs
The report advocates simulation for reproducibility without preferring one platform, continuous state spaces, SI units, documented simulator modifications and agent assumptions, and open-source software for deployment to physical robots.
Go to primary source ↓e-scenariosPDF p. 4, Section 6 opening and sampling paragraphs; PDF p. 5, opening paragraph
Scenes are partitioned into training, validation and test sets. Random free-space start/goal samples undergo tile-based shortest-path checking. The general description states more than 100 configurations per scene and minimum geodesic distance 1 m.
Go to primary source ↓e-suncg-matterportPDF p. 5, Section 6, SUNCG and Matterport3D paragraphs
SUNCG uses 500 houses split 300/100/100; Matterport3D uses 61/11/18 houses. Both specify body width 0.2 m, success threshold 0.4 m and linked scenario files.
Go to primary source ↓e-thorPDF p. 5, Section 6, AI2-THOR introductory paragraph, PointGoal and ObjectGoal subsections including both action lists
AI2-THOR v1.0 contains 120 scenes in four categories, each split 20/5/5. PointGoal combines five targets, starts and configurations for 125 scenarios. ObjectGoal uses visibility within 1 m and field of view, distinguishes navigation-only from interaction-based actions, and likewise specifies 125 scenarios. Single-room scenes do not support AreaGoal.
Go to primary source ↓e-gibsonPDF p. 5, Section 6, Gibson paragraphs; PDF p. 6, opening continuation before Section 7
Gibson gives tiny/medium/full splits of 25/5/5, 100/20/20 and 402/85/85; 100 random reachable point pairs per space; minimum separation 1 m. Action spaces and geodesics depend on the agent. Default geodesics use fixed height over the floor, endpoints are on the same floor, and the recommended threshold is twice body width, default 0.4 m.
Go to primary source ↓e-architecturesPDF p. 6, Section 7 and Recommendation 7
The report discusses reactive deep-network agents, recurrent vector memory and richer internal representations. It explicitly refrains from recommending a representation and calls for broad study; it supplies no new trained architecture.
Go to primary source ↓e-source-scopePDF pp. 1–7, complete document: Abstract, Sections 1–8 and References; sole displayed Equation (1) on p. 3
The complete document is a consensus metric/protocol report with scenario specifications and a conceptual architecture discussion. It contains no numbered figures, tables, empirical agent results, ablation study or appendix. Equation (1) provides the original mathematical visual used in this edition.
Go to primary source ↓8.5 Primary sources
On Evaluation of Embodied Navigation Agents ↗
PDF · 5,394 extracted words
Source fingerprint
340604b15502d321510d74c2e506963eca31520afbc6d46ff2c00c63702dad79