Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
1. Paper overview
In one sentence: Short, scenario-focused CARLA routes make driving failures easier to diagnose, while shared expert data improve comparability within the limits of simulated behavior and uneven coverage. designteacherevaluationoverall-resultsdistillationdistributionpreprocessingpenaltiescompute
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Open-loop trajectory error omits the consequences of a policy’s actions. Long simulated routes compound penalties and obscure which skill failed, while privately collected training sets confound algorithm comparisons. Bench2Drive uses common demonstrations and short scenario tests; reduced evaluation variance is a design rationale rather than a quantified repeated-run result. designevaluationuncertainty |
| Core mechanism | |
| A key reported result | Closed-loop driving on Bench2Drive base: DriveAdapter*: 64.22 / 33.08%. Driving score / success rate; higher is better. 950 training clips; 220 CARLA routes; stars indicate expert distillation. ThinkTwice*: 62.44 / 31.23%; UniAD-Base: 45.81 / 16.36%. DriveAdapter leads these baselines on both metrics but succeeds on about one third of routes. Simulator point estimates have no reported error bars. protocoloverall-resultsuncertainty |
| Reading caution | Author claim The authors identify simulation-to-reality gaps and scarce interactive frames within demonstrations. Their explanation involving limitations of imitation learning is a hypothesis, not an isolated ablation. limits |
Core contributions
- Source description
The 220-route suite reports five abilities alongside driving score, success, efficiency and comfortness. Baseline adaptations and expert features support comparisons using shared data. evaluationmetricscomfortimplementation
- Source description
The paper states Apache License 2.0 distribution for data, code and checkpoints. This is a release statement, not an independently tested installation. release
Figure 1. Expert demonstrations, reactive route tests and an ability profile form the benchmark’s three components. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right as the organization of the benchmark. The left panel identifies Think2Drive as the RL expert supplying demonstrations for a unified training set. The center shows several traffic situations used to motivate closed-loop evaluation. The right panel represents multidimensional assessment. These are conceptual components, not a layer-by-layer neural architecture. Section 3.1 explains the teacher’s privileged information, while Section 3.3 defines the route tests and skill groups. The red radar shapes have no labeled quantitative axes here, so they should be read as illustrations of an ability profile rather than measured baseline results. designteacherevaluationprotocolimplementation
What it supports. The scientific contribution lies in coordinating training data and evaluation. Sharing demonstrations reduces one source of comparison variability, and testing individual scenarios makes failure categories visible. The figure also locates the world model’s role: Think2Drive supplies expertise during data collection; it does not define a universal inference mechanism for the students.
Where the evidence stops. This overview does not specify model layers, loss weights or inference-time access to teacher features. The right-hand plots are schematic. Quantitative claims require Tables 3–4 and their evaluation protocol.
2. Motivation
2.1 The problem and the proposed response
Open-loop trajectory error omits the consequences of a policy’s actions. Long simulated routes compound penalties and obscure which skill failed, while privately collected training sets confound algorithm comparisons. Bench2Drive uses common demonstrations and short scenario tests; reduced evaluation variance is a design rationale rather than a quantified repeated-run result. designevaluationuncertainty
2.2 What this reading follows
A planner can predict a recorded trajectory accurately and still drive poorly once its own decisions change what it sees next. Bench2Drive makes that distinction testable through a common expert dataset and a suite of short, interactive simulation routes. Read the paper as the design of an experiment: what observations are available, how demonstrations are collected, what constitutes a successful route, and how failures are grouped. Its baseline tables show both the value and the limits of expert-feature distillation. The appendices are essential: they qualify claims about balanced coverage, describe preprocessing, and expose ambiguities in metric and compute specifications. designteacherevaluationoverall-resultsdistillationdistributionpreprocessingpenaltiescompute
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The benchmark/simulator classification is supported. Bench2Drive standardizes data, scenario execution and diagnostics. Think2Drive supplies demonstrations and features; this does not make Bench2Drive a One Model architecture or joint future/action predictor. Architecture, prediction paradigm and quadrant are appropriately not applicable. teacherevaluationimplementation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 From a privileged teacher to a sensor-driven student
The training pipeline begins with Think2Drive navigating CARLA using privileged information about the simulated scene. Bench2Drive keeps successful demonstrations and packages raw observations, annotations, teacher values and features. Those are different kinds of information: camera images can be student inputs, boxes and segmentation can supervise perception, and teacher features can guide representation learning. The benchmark’s baseline adaptations matter at this boundary. ThinkTwice and DriveAdapter regenerate expert BEV features and use six cameras without LiDAR, while AD-MLP uses only ego history. Training uses 950 clips from the base subset; 50 clips support open-loop validation. At evaluation, the selected baseline actually drives the simulator. Think2Drive’s role as a data source therefore does not establish that every student runs a world model or receives privileged teacher information at inference. teachersensorsprotocolimplementation
Figure 2. Short routes change the unit of diagnosis from a long journey to a specific scenario. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Both panels depict Town12. Colors distinguish routes, so a color change on the right indicates another short test rather than a change in danger or success. Compare the extended paths in the Leaderboard V2 panel with the fragmented route coverage in Bench2Drive. The design is explained in Sections 1 and 3.3: routes are approximately 150 meters and contain a particular scenario, with five configurations per scenario across towns and weather. The map illustrates route segmentation. The metric argument comes separately from Equation (1), where each route’s contribution includes cumulative multiplication of infraction penalties. designevaluationmetricsuncertaintyrelease
What it supports. Shorter tests give a failure a more local interpretation: it can be associated with the tested scenario instead of buried inside a long trip with multiple penalties. This is the rationale for ability-level reporting. The visual supports the route-design contrast, while the claimed variance advantage still needs repeated measurements.
Where the evidence stops. Map coverage does not establish generalization to unseen geography or lower empirical variance. The figure gives no outcome measurements, and the paper supplies no error bars for the reported baseline comparisons.
5.2 Why a short route changes what a score can tell you
Consider the two parts of driving score: distance completed and the product of infraction penalties. Equation (1) makes repeated infractions progressively shrink the contribution of a route. Bench2Drive limits each test to a short route with a specific scenario, then repeats that scenario across five configurations. The intended benefit is that a poor score can be associated with a particular driving demand. Reader interpretation: this improves diagnostic locality, but does not prove that composing successful short maneuvers yields reliable long journeys. Success rate adds a stricter requirement of timely completion without infractions. The five ability groups then summarize related scenarios, with overlapping membership. Their mean is consequently a separate summary from overall route success. Without repeated-run uncertainty, the claim of reduced variance remains a rationale to test. evaluationmetricsdesignabilitiesuncertainty
Figure 3. The dataset separates observable sensor streams from rich supervision supplied by simulation. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the six camera views around the ego vehicle. The coverage diagram labels five sectors as 70 degrees and the rear sector as 110 degrees. Then read the annotation montage: it includes bounding boxes, LiDAR BEV, point clouds, segmentation, depth, remote sensing and lane information. Section 3.2 additionally lists radar, IMU/GNSS, HD-map and teacher values/features. These are available dataset channels, not a claim that all are inputs to every evaluated policy. Appendix C explicitly changes ThinkTwice and DriveAdapter to six cameras without LiDAR, which is essential context when interpreting their results. sensorsimplementationpreprocessing
What it supports. Simulation supplies supervision that would be costly or unavailable in ordinary driving logs. That makes auxiliary perception training and expert-feature guidance practical. A fair comparison still needs to record which channels each baseline consumes, since a shared dataset does not automatically equalize sensing or supervision.
Where the evidence stops. The figure depicts sample observations and annotations, not a prediction pipeline. Appendix A describes annotation repairs and JPEG compression; retaining those details matters for reproducing the actual baseline data distribution.
5.3 Read improvements through three different lenses
First inspect execution: TCP-traj success rises from 20.45% without distillation to 30.00% with it. Next inspect the mechanism-oriented contrast: these rows vary expert-feature distillation within the named TCP-traj baseline, making them more informative than comparing unrelated architectures. Finally inspect the ability profile. Overtaking rises from 6.67% to 24.29%, but merging falls from 17.14% to 8.89%; an aggregate gain is not a universal skill gain. Reader interpretation: teacher guidance may redistribute which behaviors the student learns, and seed variability must be measured before attributing individual regressions to the mechanism. Comfortness also falls in Table 3. Because the metric rewards kinematic smoothness even for stopped failures, it should be interpreted jointly with progress and success rather than used as a substitute for them. distillationabilitiescomfortuncertainty
5.4 Training and inference
During training
The base protocol uses 950 training and 50 validation clips covering every scenario. UniAD, VAD, ThinkTwice and DriveAdapter use one tenth of original epochs because denser data supply similar training-step counts. Detailed optimizer, loss-weight and frozen-module schedules are delegated to baseline configurations. protocolimplementation
ThinkTwice/DriveAdapter regenerate Think2Drive BEV features; TCP-traj is compared with and without distillation. There is no single new student objective. Shared data still leave architecture, sensor and training differences in cross-model comparisons. implementationdistillationprotocol
During inference
Each baseline drives the 220 CARLA routes; simulator-crash routes receive zero. TCP predicts trajectories and controls, while the benchmark does not define a universal controller. JPEG quality-20 compression/decompression at inference matches stored training images. protocolpreprocessing
Open-loop testing compares predictions against recorded trajectories over two seconds at 2 Hz. It does not execute those predictions and must be distinguished from closed-loop success. protocoldesign
5.5 Implementation flow
- Collect successful demonstrations
Think2Drive uses privileged simulator information to drive predefined routes. Collection keeps infraction-free runs, filters rendering/behavior bugs and repairs annotations. Teacher features and values accompany sensor data as supervision resources; they do not imply every student performs world-model planning. teacherpreprocessingsensors
- Expose multimodal observations
Data sampled at 10 Hz include six cameras, LiDAR, five radars, IMU/GNSS, a BEV camera and HD-map, with boxes, depth and segmentation. Modalities vary by baseline: AD-MLP uses ego history, while ThinkTwice/DriveAdapter are adapted to six cameras without LiDAR. sensorsprotocolimplementation
- Isolate scenarios with feedback
Five approximately 150-meter routes per scenario yield 220 tests across town/weather configurations. The model drives toward target waypoints; its actions change subsequent observations. NPC vehicles react through rule-based behavior, so interaction is bounded by the simulator’s behavior model. evaluationdesignnpc
- Separate progress, speed and smoothness
Success requires timely, infraction-free completion. Driving score multiplies completion by penalties. Efficiency averages speed relative to traffic at 20 checkpoints, excluding routes before the first 5% and values above 1000%. Comfortness counts smooth 20-frame segments; stopped failures can still be smooth. evaluationmetricscomfort
6. Experiments & results
Bench2Drive standardizes demonstrations and tests end-to-end driving through short, interactive CARLA routes. Its contribution is an evaluation system: a common expert dataset, scenario-level success measurements and five ability groups. Baselines reveal gaps between fitting recorded trajectories and successful closed-loop driving. Expert-feature distillation improves aggregate TCP-traj performance, but skill-specific regressions and simulator limitations constrain the interpretation.
6.1 Read the original evidence
Table 3. Shared base-data results show that trajectory error and closed-loop success can rank models differently. An asterisk denotes expert feature distillation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the column arrows first: lower average L2 is better, whereas the closed-loop columns point upward. Average L2 covers predictions over two seconds at 2 Hz. Training uses 950 base clips; open-loop validation uses 50 clips, and closed-loop evaluation uses 220 routes. Asterisks denote expert feature distillation. Compare UniAD-Base with DriveAdapter: the former has lower L2, but the latter has higher driving score and success. Then compare the two TCP-traj rows to examine the distillation contrast. Efficiency values can exceed 100 because they compare ego speed with nearby traffic; they are not route-success percentages. protocoloverall-resultsdistillationmetricsresult-conflictuncertainty
What it supports. DriveAdapter reports 64.22 driving score and 33.08% success versus UniAD-Base’s 45.81 and 16.36%, despite worse L2, 1.01 versus 0.73. Within TCP-traj, distillation raises success from 20.45% to 30.00%, a calculated 9.55-percentage-point improvement. These findings concern executed simulation performance under the base protocol.
Where the evidence stops. The prose below Table 4 calls UniAD-Base worse than VAD in closed loop, although this table gives it higher driving score/success and lower efficiency/comfortness. Preserve that metric-dependent discrepancy. No error bars are reported.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Closed-loop driving on Bench2Drive base 950 training clips; 220 CARLA routes; stars indicate expert distillation. | DriveAdapter*: 64.22 / 33.08%. Driving score / success rate; higher is better | ThinkTwice*: 62.44 / 31.23%; UniAD-Base: 45.81 / 16.36%. DriveAdapter leads these baselines on both metrics but succeeds on about one third of routes. Simulator point estimates have no reported error bars. protocoloverall-resultsuncertainty |
| Open-loop versus closed-loop ranking Base protocol; L2 on 50 validation clips over two seconds at 2 Hz. | UniAD-Base: L2 0.73, success 16.36%. Average L2, lower better; success rate, higher better | DriveAdapter*: L2 1.01, success 33.08%. Better trajectory fitting does not imply better executed success. Architecture and distillation differ, so this illustrates ranking disagreement rather than isolating its cause. protocoloverall-results |
| TCP-traj expert-feature distillation Base subset and 220-route evaluation. | With distillation: 30.00% / 59.90. Success rate / driving score | Without: 20.45% / 49.30; calculated differences: +9.55 percentage points / +10.60 score. Aggregate improvement coexists with lower comfortness, 18.08 versus 22.96. Missing seed variability limits confidence. protocoldistillationuncertainty |
| Skill-specific effects of TCP-traj distillation Table 4 groups defined by Table 2; base training. | With distillation: mean 34.22%; overtaking 24.29%; merging 8.89%. Ability percentage and five-group mean | Without: mean 28.51%; overtaking 6.67%; merging 17.14%. Benefits vary by skill. Groups overlap, so their mean is distinct from overall 220-route success. abilitiesevaluation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. The TCP-traj distillation contrast improves the mean while exposing skill-specific regressions. An asterisk denotes expert feature distillation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The five columns use scenario groups defined in Table 2, and the final column averages their reported percentages. Some scenarios belong to more than one group, so this mean is not the overall success rate from Table 3. Asterisks again denote expert feature distillation. For the mechanism-oriented comparison, move between TCP-traj* and TCP-traj without distillation, keeping each ability column aligned. Overtaking and emergency braking improve, but merging and giving way decrease. Finally inspect DriveAdapter’s row: its strongest numbers occur in different columns from its weakest ones, which demonstrates why a single overall score can hide an uneven skill profile. abilitiesevaluationdistillationuncertaintyimplementation
What it supports. TCP-traj’s mean rises from 28.51% to 34.22% with distillation, yet merging falls from 17.14% to 8.89% and giving way from 50.00% to 40.00%. DriveAdapter’s 42.08% mean also coexists with only 28.82% merging and 26.38% overtaking. The table supports a differentiated assessment of driving abilities.
Where the evidence stops. These point estimates do not establish that distillation reliably harms a specific skill. Matched repeated seeds and route-level outcomes are absent. Comparisons between unrelated baseline rows additionally change architecture and supervision.
Figure 8. The distribution diagnostic qualifies broad claims of uniformly distributed data. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legends to identify categories, especially where small town-slice labels overlap in the original chart. The left pie is heavily concentrated in Town12 and Town13; the right pie distributes mass more evenly across weather identifiers, but still contains unequal shares. Appendix B explains the geographic concentration: newer towns are larger and support layouts required by recently introduced scenarios. Read this figure alongside the abstract’s uniformity wording and Figure 6’s unequal scenario counts. The defensible description is diverse coverage with deliberate balancing efforts, rather than exact equality across towns, weather and behavior. distributionreleaseprotocol
What it supports. Town12 contributes 64.4% and Town13 22.1% in the displayed distribution, together a calculated 86.5%. A benchmark can cover many conditions while remaining concentrated in a few environments. The result motivates reporting geographic coverage explicitly and checking whether evaluation locations overlap with training neighborhoods.
Where the evidence stops. This is a dataset diagnostic, not an intervention showing how imbalance changes performance. The chart does not identify train/test overlap, and overlapping tiny slice labels should be read through the intact legends rather than guessed.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify simulation-to-reality gaps and scarce interactive frames within demonstrations. Their explanation involving limitations of imitation learning is a hypothesis, not an isolated ablation. limits
Diversity is not uniformity: Figure 8 assigns 64.4% to Town12 and 22.1% to Town13; Figure 6 shows unequal scenario counts. Nearby locations may recur across clips, and the PDF does not establish geographic disjointness of the reported splits. distributionreleaseprotocol
The prose calls UniAD-Base worse than VAD in closed loop, but Table 3 gives it higher driving score/success and lower efficiency/comfortness. The unqualified comparison is ambiguous. Error bars and repeated-seed uncertainty are absent. result-conflictuncertainty
Table 7 retains a Too Slow penalty although Section 3.3 removes speed checks from driving score. The comfort discussion’s 60-second blocking description differs from Table 7’s 180-second shutdown threshold. These should not be silently conflated. penaltiescomfort
7.2 Questions for discussion
- Would TCP-traj distillation still help merging under matched repeated seeds?
- How would strict geographic holdouts change the ability profile?
- Do short-route rankings remain stable when scenarios are composed into long journeys?
8. Reproducibility audit
8.1 Requirements and known gaps
Recover exact route/split files, CARLA build, baseline configurations and checkpoints before reproducing numbers. The PDF delegates split/configuration details externally. Preserve JPEG quality 20, coordinate conventions, annotation repairs and the zero-score crash rule. releaseimplementationpreprocessingprotocol
Compute descriptions conflict: Section 4.1 lists A6000/A100 training, while Table 5 prints A600/H800. Table 5 reports eight H800s for five days for VAD and nine days for UniAD-base; BEVFormer/stage1/stage2 take three days each. Both evaluations use eight H800s for two days. These requirements remain unreconciled. compute
Proposed checks: repeat TCP-traj distillation with matched configurations and seeds, reporting every ability; independently rescore route traces to test speed-penalty removal and exclusions. Neither check has been run. distillationabilitiesmetricspenaltiesuncertainty
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the TCP-traj distillation contrast with matched runs
Reader-proposed, not executed: train TCP-traj with and without expert-feature distillation on the same 950 clips using three paired initialization seeds. Keep architecture, augmentations, optimizer, training-step budget, image compression and control interface fixed; change only the distillation term and required teacher features. Evaluate matched simulator seeds on the same 220 routes. Report per-route success, driving score, each ability, and variability across runs. The reported aggregate gain should persist if teacher features provide robust guidance; the merging regression may disappear if it reflects run variation. Failure to retain a positive paired success difference would weaken the paper’s mechanism interpretation. Exact implementation settings must first be recovered from the referenced configurations. protocolimplementationpreprocessingdistillationabilitiesuncertainty
Check 2: Reconcile scoring with an independent trace rescore
Reader-proposed, not executed: freeze several baseline checkpoints and record route completion, every infraction, checkpoint speeds and frame-level smoothness variables. Implement a separate scorer from Section 3.3 and compare its route scores with the evaluation output. Use matched traces that differ only in slow-speed events: the specified Bench2Drive driving score should remain unchanged while efficiency changes. Separately test traces ending before the first 5% checkpoint, traces shorter than 20 frames and stopped segments. Report included-route counts for each metric and distinguish the comfort paragraph’s blocking condition from Table 7’s shutdown threshold. A driving-score reduction solely from a Too Slow event would reveal the unresolved mismatch between the main metric definition and the appendix. metricscomfortpenaltiesprotocol
8.3 Reading coverage
Visual audit: The title/author/version page, Figures 1–8, Tables 1–7, the unnumbered real-world/simulation comparison, metric equations and NPC Algorithm 1 were visually inspected on these pages. Appendix pages covering preprocessing, training modifications, hardware, behavior, penalties, licensing and the datasheet were included. All six final original crops were viewed and checked for legibility and bounds. Table stars mean expert feature distillation, as stated in the source captions and explained locally. No conflicting arrow or branch marking was found in the selected diagrams. Table 3 versus its results prose, the distribution figures versus broad uniformity wording, and metric/compute inconsistencies are disclosed. All seven text chunks, including all scenario descriptions, were read; scenario images on pages 20–24 and reference pages 11–13 were not visually reviewed. External code, case-analysis pages and supplements were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 19, 25, 26. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/version block (p. 1)
- 1 Introduction (pp. 1–3)
- 2 Related Work, 2.1–2.2 (pp. 3–4)
- 3 Bench2Drive, 3.1–3.3 and all metrics (pp. 4–8)
- 4 Experiments, 4.1–4.3 (pp. 8–10)
- 5 Conclusion, Limitations and Social Impact (p. 10)
- References (pp. 11–14)
- Checklist (pp. 14–15)
- A Details of Data Collecting (pp. 15–16)
- B Distribution of Scenarios, Towns and Weathers (pp. 16–17)
- C Implementation Details of Baselines (p. 17)
- D Training and Evaluation Resource Requirements (pp. 17–18)
- E Behavior Model of NPC Agents (pp. 18–19)
- F Details about Infraction Score (pp. 18–19)
- G Description of Scenarios, entries 1–44 (pp. 19–25)
- H Author Statement and I License (p. 25)
- J Datasheet, J.1–J.6 (pp. 25–26)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the retained PDF pages and final crops.
- Separate supplemental material availability has not been fully verified.
- All seven supplied chunks were read. Visual inspection covered pages 1–10, 14–19 and 25–26; scenario illustrations on pages 20–24 and reference pages 11–13 were not visually inspected.
- Linked repositories, datasets, checkpoints and external case-analysis materials were not inspected; no experiments were executed.
- The reviewed artifact is arXiv:2406.03877v3, dated 27 November 2024. Title and authors match the catalog. Earlier revisions and a separate venue PDF were not supplied, so edition-to-edition content differences were not assessed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author block, arXiv margin and venue footer
Title and five authors match the catalog. The artifact is arXiv:2406.03877v3 [cs.RO], 27 November 2024, bearing the NeurIPS 2024 Datasets and Benchmarks track footer. The first four authors have equal-contribution marks; Junchi Yan is corresponding author. The affiliation is Dept. of CSE & School of AI & MoE Key Lab of AI, Shanghai Jiao Tong University.
Go to primary source ↓designPDF pp. 1–3, Abstract, Section 1, Figures 1–2 and captions
Bench2Drive combines expert training data, closed-loop scenario evaluation and ability assessment. The release is described as 2 million frames from 13,638 clips, 44 scenarios, 23 weathers and 12 towns. Figure 2 contrasts long Leaderboard V2 routes with short Bench2Drive routes in Town12; colors identify routes, not performance.
Go to primary source ↓teacherPDF p. 4, Sections 3 and 3.1
The benchmark contains a dataset, evaluation toolkit and baseline implementations. Think2Drive is a world-model-based RL teacher using privileged simulator information to collect demonstrations. The authors say it solved all 44 scenarios during construction; they acknowledge the subsequent PDM-Lite release.
Go to primary source ↓sensorsPDF p. 5, Section 3.2, Figure 3 and caption
Data are sampled at 10 Hz. The configuration includes six 900×1600 cameras with JPEG quality 20, one 64-channel LiDAR with 85-meter range and 600,000 points/second, five radars, IMU/GNSS, a BEV camera and HD-map. Annotations include boxes, depth, segmentation and expert values/features. Figure 3 labels five camera sectors 70 degrees and the rear sector 110 degrees. Mini/base/full subsets contain 10/1,000/10,000 clips.
Go to primary source ↓evaluationPDF p. 6, Section 3.3 and Table 2
Five approximately 150-meter routes for each of 44 scenarios produce 220 evaluation routes with different towns and weathers. Inputs are raw sensors and target waypoints. Success requires timely arrival without infractions. Table 2 maps scenarios to merging, overtaking, emergency brake, give way and traffic sign; some scenarios occur in multiple groups.
Go to primary source ↓metricsPDF p. 7, Section 3.3, Equations (1)–(3) and efficiency paragraphs
Success rate is successful routes divided by total routes. Driving score averages route completion multiplied by each infraction penalty. Efficiency averages ego speed relative to nearby traffic at 20 checkpoints, every 5% of route distance, and is excluded from driving score. Routes failing to reach the first checkpoint and speed percentages above 1000% are excluded from that metric.
Go to primary source ↓comfortPDF pp. 7–8, Section 3.3, Equation (4), segment and smoothness definitions
Frame smoothness uses nuPlan-derived bounds on acceleration, yaw rate, yaw acceleration and jerk. Bench2Drive evaluates 20-frame segments and reports their smooth fraction. A blocked vehicle can still have smooth segments; routes shorter than 20 frames are excluded. Blocking here is described as speed below 0.1 for over 60 seconds.
Go to primary source ↓protocolPDF p. 8, Section 4.1; p. 9, Table 3 caption
Baselines use 950 training and 50 open-loop validation clips from the base subset; each scenario appears in validation and weather is balanced. Closed-loop evaluation uses 220 routes, with simulator-crash routes assigned zero. Average L2 covers predictions over two seconds at 2 Hz. Models include AD-MLP, UniAD, VAD, TCP, ThinkTwice and DriveAdapter.
Go to primary source ↓overall-resultsPDF p. 9, Table 3, DriveAdapter*, ThinkTwice*, UniAD-Base, VAD and AD-MLP rows
DriveAdapter*: L2 1.01, driving score 64.22, success 33.08%, efficiency 70.22, comfortness 16.01. ThinkTwice*: 0.95, 62.44, 31.23%, 69.33, 16.22. UniAD-Base: 0.73, 45.81, 16.36%, 129.21, 43.58. VAD: 0.91, 42.35, 15.00%, 157.94, 46.01. AD-MLP: 3.64, 18.05, 0.00%, 48.45, 22.63. Stars denote expert feature distillation.
Go to primary source ↓distillationPDF p. 9, Table 3, TCP-traj* and TCP-traj w/o distillation rows; Section 4.2, expert feature distillation paragraph
TCP-traj with distillation has L2 1.70, driving score 59.90 and success 30.00%; without distillation these are 1.96, 49.30 and 20.45%. Comfortness is 18.08 versus 22.96. The authors attribute guidance to expert features and note their difficulty of acquisition in the real world.
Go to primary source ↓abilitiesPDF p. 9, Table 4, DriveAdapter*, TCP-traj* and TCP-traj w/o distillation rows; p. 6, Table 2
DriveAdapter has merging 28.82%, overtaking 26.38%, emergency brake 48.76%, give way 50.00%, traffic sign 56.43%, mean 42.08%. TCP-traj with/without distillation has mean 34.22/28.51, merging 8.89/17.14, overtaking 24.29/6.67, emergency brake 51.67/40.00, give way 40.00/50.00 and traffic sign 46.28/28.72. Grouped ability percentages are distinct from the all-route success rate.
Go to primary source ↓result-conflictPDF p. 9, Section 4.2, first paragraph beneath Table 4, compared with Table 3
The prose calls UniAD-Base worse than VAD in closed loop despite lower L2. Table 3 gives UniAD-Base higher driving score and success rate, while VAD has higher efficiency and comfortness. The direction depends on the metric; the prose cannot support an unqualified success-ranking statement.
Go to primary source ↓limitsPDF pp. 9–10, Section 4.2 interactive-behavior discussion; Section 4.3; Section 5 Limitations and Social Impact
The authors hypothesize that scarce interactive frames and limitations of imitation learning explain weak interactive skills. They acknowledge rendering and behavior gaps relative to reality and position simulation as complementary to on-road testing. Detailed case analysis is linked externally rather than included in the PDF.
Go to primary source ↓uncertaintyPDF p. 14, Checklist item 3(c); p. 9, Tables 3–4
The checklist explicitly answers No to reporting error bars. Tables 3–4 show point estimates without seed-level variability or confidence intervals.
Go to primary source ↓preprocessingPDF pp. 15–16, Appendix A, Route, Annotations, Coordinate System and Data Compression
Collection retains expert routes without infractions and manually removes buggy clips. Corrections include pedestrian speeds by differentiation, zero-padding missing speedometer/IMU values, stop-sign trigger rectangles and static-vehicle boxes. CARLA uses Z-up left-handed coordinates. JPEG quality 20 is used for stored images and in-memory compression/decompression during inference.
Go to primary source ↓distributionPDF p. 16, Appendix B and Figure 6; p. 17, Figure 8 and caption; p. 1, Abstract
Appendix B states Town12 and Town13 dominate because larger newer maps support more scenario layouts. Figure 8 reports Town12 64.4% and Town13 22.1%; weather shares are closer but unequal, including 0.6% and 7.4%. Figure 6 shows unequal scenario counts. These qualify the abstract's wording about uniform distribution.
Go to primary source ↓implementationPDF p. 17, Appendix C
Baseline code/configurations are followed with CARLA detection classes. UniAD, VAD, ThinkTwice and DriveAdapter use one tenth of original epochs, justified by denser frames and similar training-step counts. ThinkTwice and DriveAdapter use regenerated Think2Drive expert BEV features and are modified to six cameras without LiDAR.
Go to primary source ↓computePDF p. 8, Section 4.1; p. 18, Table 5 and caption
Section 4.1 specifies one A6000 for AD-MLP/TCP and eight A100s for the larger models. Table 5 instead prints one A600 for one day for AD-MLP/TCP, eight H800s for five days for VAD and eight H800s for nine days for UniAD-base. UniAD training is BEVFormer, stage1 and stage2, three days each. Evaluation is four A6000s for six hours for AD-MLP/TCP or eight H800s for two days for VAD/UniAD. Hardware descriptions remain unresolved.
Go to primary source ↓npcPDF pp. 18–19, Appendix E, Table 6 and Algorithm 1
NPC vehicles have cautious, normal and aggressive presets and a rule-based decision procedure with stopping, car following and PID control. They respond to the ego vehicle. Pedestrians follow their route at constant speed and cannot walk backward.
Go to primary source ↓penaltiesPDF p. 19, Appendix F, Table 7; pp. 7–8, Section 3.3
Table 7 lists pedestrian/vehicle/other collision factors 0.50/0.60/0.65 and red-light factor 0.70. It retains a Too Slow factor of 0.70, whereas Section 3.3 removes speed checking from driving score. Table 7 stops blocked agents after 180 seconds, distinct from the comfort paragraph's 60-second blocking description.
Go to primary source ↓releasePDF pp. 25–26, Appendices H–J, especially I, J.2, J.4 and J.6
The authors state data, code and checkpoints are distributed through GitHub and Hugging Face under Apache License 2.0. The datasheet repeats 13,638 clips and 10-Hz sampling, acknowledges nearby locations/shared scenario types, and points to the repository for split definitions. Other possible tasks include detection, segmentation, depth estimation, tracking and motion prediction.
Go to primary source ↓8.5 Primary sources
Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving ↗
PDF · 10,401 extracted words
Source fingerprint
c4b3514da643b7e564c3bb1efbcdaa17eaf6a61231da22125aeb5af708540c93