NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
1. Paper overview
In one sentence: NAVSIM trades reactive feedback for inexpensive evaluation on real sensor observations, using short simulated trajectories and curated scenes to make planning scores more informative. e02e03e04e05e08e13
| At a glance | What to know |
|---|---|
| Research problem | Source description Displacement from one human trajectory confounds acceptable alternatives with errors: Figure 1 gives a collision-free, on-road plan higher displacement error than a colliding plan. Interactive sensor simulation is costly and introduces domain gaps. NAVSIM asks how much useful planning evaluation real observations can support without synthesizing new sensor streams. e02e03 |
| Core mechanism | |
| A key reported result | Sensor-based planning on navtest: TransFuser 84.0; PARA-Drive 84.0; LTF 83.8; UniAD 83.4. PDMS (%) ↑. Table 1, 12k curated navtest samples; four-second non-reactive evaluation; methods use their stated sensor configurations. Ego Status MLP 65.6; Constant Velocity 20.6; human trajectories 94.8. Sensor information matters. The leading baselines remain 10.8 percentage points below humans, computed from Table 1. Small model-to-model differences lack uncertainty estimates here. e05e08 |
| Reading caution | Source description Non-reactivity omits traffic responses and compounding feedback errors. The authors warn that high PDMS need not imply high CLS and recommend complementary closed-loop benchmarks. e13 |
Core contributions
Figure 1. Different geometric errors can conceal very different driving outcomes. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the photograph: three colored paths correspond to the three candidate BEV panels on the right. The human log has zero average displacement error (ADE) by definition. Orange has 2.3 m ADE and leaves the road; green has 1.1 m ADE and remains on-road without collision; red has only 1.0 m ADE but collides. Read the road and collision checkmarks separately from the ADE labels. Their ordering explains why NAVSIM evaluates simulated consequences. Section 3 implements that idea by tracking a fixed four-second plan with a controller and applying collision, compliance, progress and comfort subscores. e02e03e04
What it supports. The green and red examples reverse the preference suggested by smaller displacement alone: the lower-ADE red path collides, while the green alternative is collision-free and on-road. This is a concrete motivation for checking multiple driving properties. It does not require recovering the exact human path to recognize a plausible alternative.
Where the evidence stops. This is a conceptual illustration, not a statistical evaluation or a complete controller diagram. Its checkmarks cover only the depicted road and collision conditions; they do not certify every PDMS component or interactive safety.
2. Motivation
2.1 The problem and the proposed response
Displacement from one human trajectory confounds acceptable alternatives with errors: Figure 1 gives a collision-free, on-road plan higher displacement error than a colliding plan. Interactive sensor simulation is costly and introduces domain gaps. NAVSIM asks how much useful planning evaluation real observations can support without synthesizing new sensor streams. e02e03
2.2 What this reading follows
A driving policy can disagree with a recorded driver and still choose a reasonable path. NAVSIM therefore asks what a proposed trajectory would do in a simplified scene, rather than judging only its distance from the demonstration. The policy observes real sensors once, commits to four seconds of motion, and receives a score combining collision avoidance, road compliance, progress and comfort. This edition follows the link from that evaluation choice to data filtering and model comparisons. Its central boundary is equally concrete: traffic does not react, the policy cannot replan, and a favorable score is not evidence that the same system can recover from closed-loop mistakes. e02e03e04e05e08e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The catalog correctly identifies a driving benchmark/simulator and evaluation diagnostic. Fixed-plan simulation and metric aggregation are its central mechanisms. Comparing neural architectures does not make NAVSIM a One Model system, joint future/action predictor or inverse-dynamics policy; architecture, paradigm and quadrant are appropriately not applicable. e03e04e09
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 One sensor observation, two different kinds of execution
The policy's interface ends at a sequence of future poses. NAVSIM then turns those poses into motion using an LQR controller and a kinematic bicycle model. Although the controller runs at 10 Hz, the policy is queried only once and receives no newly simulated camera or LiDAR input. This distinction is the computational bargain: real observations remain usable because the evaluator never needs to render what the policy would see after deviating from the recorded route. Other traffic also remains non-reactive. By contrast, the nuPlan experiment used to validate the metric repeatedly queries an ego planner throughout its closed-loop rollout. Calling both procedures simulation is correct, but they test different abilities: committing to a locally viable plan versus updating that plan as execution unfolds. e03e06e13
Figure 2. Scenario selection makes constant-motion shortcuts less effective. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the shared red/blue legend to compare the same quantities before and after filtering. Panel (a) contrasts human and constant-velocity PDMS. Panel (b) shows longitudinal endpoint frequencies, while panel (c) shows lateral endpoints with a logarithmic vertical axis; equal vertical distances there represent multiplicative frequency changes. The filter removes a scene if constant velocity and heading score above 0.8, or if the human trajectory scores below 0.8. The first rule removes easy examples; the second keeps a demonstrated acceptable solution under the same metric and helps exclude annotation failures. The endpoint plots show the resulting distribution change. e04e05e08
What it supports. The accompanying text reports constant-velocity PDMS falling from 79% to 22% and human PDMS rising from 91% to 95%. The filtered data contain fewer stationary or fast straight-driving shortcuts and relatively more lateral variation. These are aggregate filtering-analysis values, distinct from the exact navtest results in Table 1.
Where the evidence stops. Difficulty is defined using PDMS itself, and the human-score threshold removes some failure cases. This supports a deliberately curated benchmark, not an estimate of ordinary traffic frequencies or a guarantee that all annotation errors are removed.
5.2 Read PDMS as a hierarchy of constraints and preferences
Equation (1) makes collision and drivable-area scores multiplicative gates, then averages progress, TTC and comfort with weights 5, 5 and 2. A zero gate eliminates the score even when the plan is smooth or makes rapid progress. Within admissible behavior, progress is measured relative to a safe estimate from PDM-Closed, not relative to how far the human happened to drive. This explains why human trajectories can score below 100 on EP. Reader inference: the aggregate is a useful ranking convention, but inspecting its components is essential to understand what improved. The filter reinforces the same convention by selecting scenes where a constant-motion policy struggles and the human scores adequately. Consequently, the benchmark's difficulty, solvability criterion and final ranking share one definition of acceptable driving. e04e05e08
5.3 Separate model simplicity from what the benchmark rewards
Table 1 makes a strong case that sensor information matters on navtest: the Ego Status MLP trails every tested sensor model. It makes a weaker case about which sensor architecture is best, because those scores cluster closely and the table lacks uncertainty estimates. Table 2 adds mechanism clues: segmentation supervision helps in the reported runs, and shrinking LiDAR coverage hurts despite retaining the sensor. The authors hypothesize that range changes also alter auxiliary-task difficulty. Reader inference: apparent sensor value depends on representation, supervision and the evaluator's penalties together. In particular, ignored rear-end collisions can reduce the payoff from surround perception. Table 3 should then be read as a separate leaderboard snapshot with multi-seed baselines and a single Hydra-MDP entry, rather than as a controlled architectural ablation. e08e09e10e11e13
5.4 Training and inference
During training
NAVSIM is an evaluator, not a learned world model. Sensor agents learn human-trajectory imitation. TransFuser also uses BEV segmentation and 3D detection supervision; LTF replaces LiDAR with a learned embedding. UniAD sequences task decoders; PARA-Drive trains auxiliary heads in parallel on a shared encoder. e09e10
Alignment experiments include 37 rule-based and 114 learned planners. Learned variants use 25%, 50% or 100% of navtrain and equally sized uniformly sampled OpenScene subsets. Full optimization settings are deferred to the unsupplied supplement. e06e15
During inference
Trajectory prediction and simulated action execution are separate stages. Auxiliary perception training does not make NAVSIM a joint future/action generator. Its four-second commitment differs from the repeatedly queried ego planner used to measure closed-loop alignment. e03e06e09
PDMS is calculated per frame and averaged across frames. Tables display percentages, although the underlying score ranges from zero to one. Reader inference: a high average cannot guarantee safe behavior in every scene. e04e08
5.5 Implementation flow
- Construct the scenario
OpenScene redistributes nuPlan at 2 Hz. Available observations comprise eight 1920 × 1080 cameras and merged point clouds from five LiDAR sensors. Lane-graph navigation avoids the human-trajectory-endpoint shortcut discussed in related work. e03e05
- Filter easy and suspect examples
Remove scenes where constant velocity and heading exceed 0.8 PDMS, or the human trajectory scores below 0.8. In the filtering analysis, constant-velocity performance falls from 79% to 22%, while human performance rises from 91% to 95%. e05
- Execute a committed plan
Query the policy only initially. Keep its planned poses fixed; an LQR controller calculates steering and acceleration, and a kinematic bicycle model propagates the ego vehicle at 10 Hz for four seconds. Other agents do not respond; the policy receives no new environmental feedback. e03
- Aggregate consequences
At-fault road-user collisions or leaving the drivable area zero the scene score; static-object collisions halve the collision factor. Progress is normalized against a safe PDM-Closed estimate and clipped to [0,1], with special handling below a five-meter reference. TTC uses constant-velocity/heading projections; comfort thresholds acceleration and jerk. e04
6. Experiments & results
NAVSIM benchmarks sensor-based driving by simulating a fixed four-second plan against non-reactive traffic, then scoring safety, progress and comfort. Its curated scenes expose the weakness of blind ego-state policies, while compact sensor models match more elaborate baselines. The evidence supports an evaluation proxy, not a guarantee of reactive driving competence.
This benchmark paper supplies a conceptual trajectory-evaluation illustration and a scoring equation, rather than a neural network architecture diagram. Figure 1 therefore serves as the method visual, with Section 3 explaining execution. The main PDF refers detailed implementations and metric definitions to a separate supplement that was not supplied; those details cannot be reconstructed from the displayed figures and tables. e02e03e04e15
6.1 Read the original evidence
Figure 3. Validate the proxy by asking whether it tracks repeatedly replanned driving. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In panel (a), each point is a planner: circles denote learned models and triangles rule-based models; red is PDMS and blue is OLS. The horizontal axis is closed-loop score (CLS), with default planning frequency f = 10 Hz and duration d = 15 s. Panels (b–e) distinguish three easily confused quantities: d is closed-loop rollout duration, f is how often its ego planner replans, and h is the horizon scored by PDMS. Read the legend's rank and linear bars as Spearman and Pearson correlations. The comparison uses privileged planners on 396 navmini scenes, not the sensor architectures listed in Table 1. e04e06e07e13
What it supports. PDMS tracks CLS more closely than OLS across the reported settings. Shorter closed-loop duration, lower replanning frequency and longer PDMS horizon improve agreement in these experiments; changing replayed traffic to reactive IDM traffic has relatively little effect. Figure 4 separately checks planner families and retains the PDMS advantage, reducing concern about the aggregate planner mix.
Where the evidence stops. The correlations are imperfect and lack uncertainty intervals in this figure. PDMS and CLS also share scoring ingredients. Evidence from privileged planners and limited IDM reactions does not establish that a high-scoring sensor policy will remain safe under sustained interaction.
Table 1. Sensor models separate from blind baselines, while the leading architectures cluster closely. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the modality columns before ranking models. Ego Stat. supplies kinematics and navigation; Image and LiDAR indicate sensor inputs, while Video marks the temporal camera methods. UniAD and PARA-Drive still use images despite their checks appearing under Video. The rightmost PDMS column is the aggregate; NC, DAC, TTC, Comf. and EP expose its ingredients. All displayed scores are percentages and higher is better. Human trajectories provide a reference, but their EP is not 100 because progress is normalized against privileged PDM-Closed planning. Compare the perception gap first, then the much smaller differences among sensor architectures. e04e08e09e13
What it supports. TransFuser and PARA-Drive both score 84.0 PDMS, versus LTF at 83.8, UniAD at 83.4 and the Ego Status MLP at 65.6. The human score is 94.8, leaving a calculated 10.8-point gap from the leading baselines. Near-perfect comfort contrasts with weaker drivable-area compliance and progress, directing attention to the remaining errors.
Where the evidence stops. The table supplies no error bars and does not equalize modalities or training compute. These close sensor-model scores do not prove architectural equivalence. NC is an at-fault collision score, and ignored rear-end events can reduce the value of rear sensing.
Table 3. Keep the multi-seed leaderboard distinct from the main comparison. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this as the paper's NAVSIM 1.1 leaderboard snapshot. Its surrounding paragraph says the reopened public server uses navtest and includes three training seeds for the learned baselines. TransFuser, LTF and the Ego Status MLP therefore have displayed plus/minus values. Hydra-MDP is included with one seed, and Constant Velocity has no training-seed distribution. Compare this reporting setup with Table 1 before reconciling numbers: TransFuser's 83.9 here is a leaderboard aggregate, while Table 1 reports 84.0. The table does not define the precise statistic represented by the plus/minus notation, so it should not be relabeled as a confidence interval. e08e11
What it supports. TransFuser is reported at 83.9 ± 0.4, LTF at 83.5 ± 0.6 and the Ego Status MLP at 66.4 ± 0.9. Hydra-MDP's single entry reaches 91.3. The snapshot shows progress beyond the original sensor baselines while exposing unequal uncertainty reporting between the baseline aggregates and the winning method.
Where the evidence stops. This table is part of the supplied proceedings artifact, not a live leaderboard query. It cannot establish current rankings, explain all differences from Table 1, or demonstrate Hydra-MDP's training-seed robustness; its single score has no displayed dispersion.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Alignment with nuPlan closed-loop evaluation 396 independent, similarly filtered navmini scenarios; 37 rule-based and 114 privileged learned planners; default CLS duration 15 s and planning frequency 10 Hz; replayed traffic unless varied. | Figures 3–4 show stronger correlation for PDMS than the displacement/heading-based open-loop score (OLS), including within every planner family. Exact coefficients are not tabulated. Spearman rank and Pearson linear correlation with closed-loop score (CLS) | OLS combines displacement and heading errors with a miss-rate penalty; duration, frequency, horizon and background reactivity are varied. Supports a proxy in this simulator/planner population, not direct closed-loop validation of sensor policies. e06e07 |
| Sensor-based planning on navtest Table 1, 12k curated navtest samples; four-second non-reactive evaluation; methods use their stated sensor configurations. | TransFuser 84.0; PARA-Drive 84.0; LTF 83.8; UniAD 83.4. PDMS (%) ↑ | Ego Status MLP 65.6; Constant Velocity 20.6; human trajectories 94.8. Sensor information matters. The leading baselines remain 10.8 percentage points below humans, computed from Table 1. Small model-to-model differences lack uncertainty estimates here. e05e08 |
| TransFuser input and supervision ablations Table 2, navtest; default uses goal, velocity, acceleration, approximately 140° camera FOV, 32 m LiDAR range in each direction and both auxiliary tasks. | Default seeds 83.3, 84.0, 84.4; goal only 81.8; 60° camera 80.3; 16 m LiDAR 79.1; no BEV segmentation 81.6; no 3D detection 84.0. PDMS (%) ↑ | The paper reports default-seed standard deviation ±0.56; altered configurations lack matching multi-seed distributions. Segmentation and adequate spatial coverage help in these runs. LiDAR range also changes auxiliary-task geometry, preventing isolation of coverage alone. e10 |
| NAVSIM 1.1 public leaderboard on navtest Table 3, reopened public evaluation; three training seeds for learned baselines and one seed for Hydra-MDP. | TransFuser 83.9 ± 0.4; LTF 83.5 ± 0.6; Ego Status MLP 66.4 ± 0.9; Hydra-MDP 91.3. PDMS (%) ↑ | Constant Velocity 20.6. These leaderboard entries do not replace Table 1 values. The passage does not define the displayed ± statistic. Hydra-MDP has no reported multi-seed uncertainty here. e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Use the seed range as context before attributing a score change to one component. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read group A first: the three default training seeds score 83.3, 84.0 and 84.4. The caption defines that default as goal, velocity and acceleration inputs, about 140° camera FOV, 32 m LiDAR in every direction, and both auxiliary tasks. Groups B–E vary ego status, camera coverage, LiDAR range and supervision respectively. F, B, L and R mean front, back, left and right, with range measured in meters. In particular, the 160° three-camera row is a changed configuration, not the default 140° crop. Use the subscore columns to see whether a change affects compliance, progress or several properties together. e08e09e10
What it supports. Removing BEV segmentation gives 81.6 PDMS, whereas removing 3D detection gives 84.0, within the default seed range. Restricting LiDAR to 16 m in every direction gives 79.1, below camera-only LTF's 83.8 in Table 1. Thus merely supplying LiDAR does not guarantee improvement under the reported representation and training setup.
Where the evidence stops. Only the default configuration has three seeds here. LiDAR-range changes also change the coordinate support of auxiliary tasks, so they do not isolate coverage alone. The authors offer this training explanation as a hypothesis, not a demonstrated causal mechanism.
7. Analysis & limitations
7.1 What the evidence leaves open
Non-reactivity omits traffic responses and compounding feedback errors. The authors warn that high PDMS need not imply high CLS and recommend complementary closed-loop benchmarks. e13
Rear-end collisions into ego are ignored as non-at-fault. PDMS/CLS omit stop-sign/light compliance and some efficiency objectives. Dataset label, calibration and pose noise persist, and road elevation is absent. e13e14
Reader inference: filtering conditions difficulty and human solvability on PDMS itself. The sensor-versus-blind gap is compelling for this curated distribution, but cannot establish performance on unfiltered traffic or removed failures. e05e08
7.2 Questions for discussion
- How much alignment survives a different planner mix or unfiltered scenario distribution? [e05, e06, e07]
- Would rear-end and traffic-light penalties change the apparent sufficiency of forward cameras? [e09, e13, e14]
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs exact splits, maps, boxes, sensor preprocessing, LQR/bicycle execution and subscore definitions. The curated release is described as 450 GB. Detailed TTC/comfort thresholds, losses, optimizers and implementation settings are not recoverable from this PDF alone. e03e04e05e15
The authors report one GPU for one day for TransFuser versus 80 GPUs for three days for PARA-Drive on navtrain. Reader caution: this is not a hardware-normalized efficiency experiment; GPU models and full compute settings are deferred to the supplement. e09e15
Proposed checks: repeat family-level correlations with scene-level uncertainty and matched reactive/replayed traffic; then repeat segmentation and LiDAR-range ablations across paired seeds while controlling auxiliary-task support. These test proxy robustness and the proposed representation confound. e06e07e10
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does alignment survive planner-family and traffic controls?
Reader-proposed, not performed: use the same navmini scenarios and planner predictions to recompute PDMS, OLS and CLS at the reported four-second PDMS horizon and 15-second, 10 Hz closed-loop setting. Pair replayed-background and reactive-IDM runs, holding the ego planner and starting scenes fixed. Report Spearman and Pearson correlations within each planner family and with equal family weighting; bootstrap scenes to estimate uncertainty. The proxy claim is strengthened if PDMS retains its advantage over OLS across families and both traffic conditions. A reversal after balancing families, or unstable differences under paired traffic changes, would expose limits hidden by pooled correlations. e03e06e07
Check 2: Is the LiDAR-range drop a coverage effect or a supervision effect?
Reader-proposed, not performed: repeat the default 32 m and restricted 16 m LiDAR settings across paired training seeds on fixed navtrain/navtest splits. Hold camera inputs, optimizer, training budget and evaluation constant. First reproduce the paper's range-coupled auxiliary-task setup; then use a common 16 m support for auxiliary targets while still varying the LiDAR input range. Cross this comparison with segmentation enabled or disabled, retaining the detection setting. Report paired changes in PDMS, DAC and EP. A persistent range advantage under common target support favors information coverage; a substantially smaller gap supports the authors' auxiliary-task-geometry hypothesis. Exact implementation settings require the referenced supplement. e05e08e09e10e15
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/affiliations on p. 1; Figure 1 and its road/collision markers on p. 2; navigation and protocol text on pp. 3–4, including Eq. (1); Figure 2 and filtering/data details on p. 5; Figure 3 and alignment protocol on p. 6; Figure 4, Table 1 and model descriptions on p. 7; Table 2, model/training details and ablation discussion on p. 8; Figure 5, Table 3, ablation continuation and discussion on p. 9; limitations on p. 10; and the supplement-dependent checklist on p. 14. All six final crops were viewed at their original rendered resolution and retain required labels, legends and table rows. All five text chunks, including reference-only pp. 11–13, were read. The separate supplement and external resources were not supplied or inspected. No claim depends on an uninspected appendix page.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction and Contributions (pp. 1–3)
- 2 Related Work (p. 3)
- 3 NAVSIM: Non-Reactive Autonomous Vehicle Simulation (pp. 3–5)
- 3.1 Generating Standardized and Challenging Train and Test Splits (p. 5)
- 4 Experiments (p. 5)
- 4.1 Alignment Between Open-Loop and Closed-Loop Evaluation (pp. 6–7)
- 4.2 Analysis of the State of the Art in End-to-End Autonomous Driving (pp. 7–9)
- 5 Discussion (pp. 9–10)
- Acknowledgments (p. 10)
- References (pp. 10–14)
- Checklist (p. 14)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This supplied acquisition caveat was addressed by inspecting the original PDF pages and final crops.
- Separate supplemental material availability has not been fully verified.
- No separate supplement was supplied. The complete 14-page proceedings PDF was read, including its checklist; no technical appendix is present in this artifact. The referenced supplement covering implementation, compute, licenses, consent and societal impacts remains outside this review.
- Code, model weights, datasets and external leaderboard pages were not inspected; no experiments were reproduced.
- Identity and edition: the printed NAVSIM title, all 12 authors and the NeurIPS 2024 Datasets and Benchmarks venue match the catalog. The supplied observedTitle spells the acronym as NA VSIM; that extraction spacing is preserved in provenance and metadata, not interpreted as a changed title. No numbered manuscript revision or revision date is printed. The artifact includes a NAVSIM 1.1 leaderboard section. No earlier manuscript or later release was supplied for comparison.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block and venue footer
The printed title is NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking. All 12 authors match the catalog. Nine affiliations and the NeurIPS 2024 Datasets and Benchmarks venue are printed. No manuscript revision number/date is shown.
Go to primary source ↓e02PDF p. 2, Figure 1 and caption; pp. 1–2, Introduction
Figure 1 contrasts an off-road 2.3 m ADE plan, an on-road collision-free 1.1 m plan and an on-road colliding 1.0 m plan with the human log. The introduction motivates evaluation beyond displacement from one demonstration.
Go to primary source ↓e03PDF p. 3, Contributions and final Related Work paragraph; p. 4, Task description and Non-Reactive Simulation
Navigation comes from the lane graph. Policies predict poses from sensors and ego status. NAVSIM queries once, fixes the trajectory for h=4 s, and uses LQR with a kinematic bicycle model at 10 Hz without environmental feedback or responsive traffic.
Go to primary source ↓e04PDF p. 4, PDM Score, Eq. (1), Penalties and Weighted Average; p. 5, paragraph above Section 3.1
PDMS multiplies NC/DAC penalties by a weighted EP/TTC/comfort average, with weights 5/5/2. At-fault road-user collisions and off-road violations zero the score; static-object collisions use NC=0.5. EP uses clipped relative progress to PDM-Closed with special treatment below 5 m. Detailed threshold definitions are deferred.
Go to primary source ↓e05PDF p. 5, Section 3.1, Dataset and Filtering for challenging scenes; Figure 2
OpenScene supplies 120 h at 2 Hz, eight 1920×1080 cameras, merged LiDAR from five sensors and optional three-frame history. Filtering removes constant-velocity PDMS>0.8 or human PDMS<0.8, yielding 103k/12k train/test samples and 450 GB. The filtering analysis reports constant-velocity 79%→22% and human 91%→95%; Figure 2 shows changed endpoint distributions.
Go to primary source ↓e06PDF p. 6, Section 4.1, Benchmark and Motion Planners
Alignment uses 396 navmini scenarios independent of navtrain/navtest and privileged planners with ground-truth perception/maps. Default CLS is 15 s at 10 Hz with replayed traffic. There are 37 rule-based and 114 learned variants; learned training varies fractions and filtered versus uniform sampling.
Go to primary source ↓e07PDF p. 6, Figure 3(a–e) and caption; p. 7, Figure 4(a–c), caption and Section 4.1 results
PDMS is more correlated with CLS than OLS overall and within every planner family. The comparisons vary CLS duration, planning frequency, PDMS horizon and traffic behavior. Shorter duration, lower frequency and longer horizon improve alignment in these experiments; IDM reactive traffic changes correlation little.
Go to primary source ↓e08PDF p. 7, Table 1, all rows and PDMS/subscore columns; p. 8, Results
Table 1 percentages: TransFuser/PARA-Drive 84.0, LTF 83.8, UniAD 83.4, Ego Status MLP 65.6, Constant Velocity 20.6, human 94.8 PDMS. Comfort is near saturation; DAC and EP are lower. Human EP is 87.5 because the reference is privileged planning rather than the demonstration.
Go to primary source ↓e09PDF p. 7, Section 4.2 Methods; p. 8, Methods continuation and Results
TransFuser uses three forward cameras combined to 1024×256, rasterized LiDAR and two auxiliary tasks; LTF replaces LiDAR with a learned embedding. UniAD has semi-sequential decoders and PARA-Drive parallel auxiliary heads. Both use BEVFormer with eight cameras over four frames. Reported training is one GPU for one day versus 80 GPUs for three days for TransFuser/PARA-Drive.
Go to primary source ↓e10PDF p. 8, Table 2, caption and Analyzing TransFuser; p. 9, continuation before CVPR challenge
Default PDMS seeds are 83.3/84.0/84.4, with reported SD ±0.56. Goal-only scores 81.8; 60° FOV 80.3; 16 m LiDAR 79.1; no segmentation 81.6; no detection 84.0. Default is 140° FOV and 32 m LiDAR in each direction. Authors hypothesize range changes affect auxiliary-task difficulty; deterministic agents repeat identical evaluations.
Go to primary source ↓e11PDF p. 9, NAVSIM 1.1 Leaderboard paragraph and Table 3
Reopened evaluation uses navtest and three training seeds for learned baselines: TransFuser 83.9±0.4, LTF 83.5±0.6, Ego Status MLP 66.4±0.9. One Hydra-MDP seed scores 91.3; Constant Velocity scores 20.6. The displayed ± statistic is not defined in this passage.
Go to primary source ↓e12PDF p. 9, CVPR 2024 NAVSIM Challenge and Figure 5
The March–May 2024 challenge withheld private-data annotations and scored submitted trajectories. It received 463 submissions from 143 teams. The winner extended TransFuser to predict proxy subscores for sampled trajectories and combine them with imitation scores; the runner-up used a VLM.
Go to primary source ↓e13PDF pp. 9–10, Section 5, Need for Reactive Simulation
High PDMS does not always imply high CLS: reactivity and accumulated errors are untested. Rear-end collisions into ego are ignored as non-at-fault. The authors recommend complementary closed-loop simulation.
Go to primary source ↓e14PDF p. 10, Section 5, Simplicity of Metrics and Call for Datasets
PDMS/CLS omit stop-sign/light compliance, transit and fuel-efficiency objectives. Data issues include missing classes, camera-parameter errors, pose/3D-annotation noise and absent road elevation.
Go to primary source ↓e15PDF p. 4, PDM Score; p. 8, implementation-details sentence; p. 14, Checklist items 1(c), 3(a–d), 4(b,d,e)
The main paper defers detailed metrics and model implementations to supplementary material. The checklist points there for compute/resource types, training details, licenses, consent, privacy and societal impacts. The supplied artifact contains no such supplement.
Go to primary source ↓8.5 Primary sources
NA VSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking ↗
PDF · 8,746 extracted words
Source fingerprint
2c47052cd1761dc4b40a02778aa3db9ff4e1818779002791d2d5af963d72b37e