nuScenes: A Multimodal Dataset for Autonomous Driving
1. Paper overview
In one sentence: nuScenes makes sensor coverage, temporal context and evaluation rules part of the perception benchmark, so a detector’s ranking must be read together with its inputs and error profile. e01e02e04e08e09e10e12e14e18
| At a glance | What to know |
|---|---|
| Research problem | Source description Autonomous-driving perception needs benchmarks that expose complementary sensor strengths and failures. Cameras provide appearance but difficult 3D localization; lidar provides accurate geometry with sparse returns; radar provides velocity information with coarser localization. The authors motivate multimodal data, varied conditions and contextual maps as a basis for studying detection and tracking beyond front-facing image benchmarks. e01e02 |
| Core mechanism | |
| A key reported result | Temporal accumulation for 3D detection: 1 sweep: 31.8 / 21.9 / 1.21; 5 sweeps: 42.9 / 27.7 / 0.34; 10 sweeps: 44.8 / 28.8 / 0.30. NDS (%), mAP (%), mAVE (m/s; lower is better). nuScenes validation set; PointPillars; KITTI pretraining; 1, 5 or 10 lidar sweeps. One to ten sweeps improves NDS by 13.0 percentage points and mAP by 6.9 points; mAVE decreases by 0.91 m/s. Measured validation results support temporal accumulation, with smaller gains after five sweeps. They do not isolate time-offset information from denser geometry. No uncertainty is reported. e12 |
| Reading caution | Reader analysis Manual scene selection and two-city coverage constrain representativeness. The paper reports 19.4% rainy and 11.6% nighttime keyframes, plus a 1:10,000 rarest-to-most-common class imbalance. Diverse examples do not by themselves establish geographic or adverse-weather generalization. e06e07 |
Core contributions
Figure 4. The acquisition architecture places complementary sensors around the vehicle. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the front of the car, on the left of the drawing, and follow the labeled camera and radar placements around its perimeter. The roof lidar and IMU complete the illustrated setup. Red, green and blue arrows label local X, Y and Z axes; the circled markers indicate directions into or out of the image plane. These are coordinate conventions, not a neural-network information flow. Read the diagram with Table 2 and the synchronization text: cameras, radar and lidar operate at different rates, and camera exposure is triggered as the lidar crosses the camera’s field of view. e04e11e18
What it supports. The dataset’s surround coverage is produced by a physical sensor arrangement whose observations must be related geometrically and temporally. Six cameras, five radars and a lidar provide the raw modalities; the separate detection experiments later select camera or lidar inputs rather than demonstrating that every sensor is fused.
Where the evidence stops. Figure 4 supplies placements and axis conventions, not calibration matrices or a complete fusion algorithm. The paper explicitly notes that 12 Hz camera exposures do not pair one-to-one with all 20 Hz lidar scans.
2. Motivation
2.1 The problem and the proposed response
Autonomous-driving perception needs benchmarks that expose complementary sensor strengths and failures. Cameras provide appearance but difficult 3D localization; lidar provides accurate geometry with sparse returns; radar provides velocity information with coarser localization. The authors motivate multimodal data, varied conditions and contextual maps as a basis for studying detection and tracking beyond front-facing image benchmarks. e01e02
2.2 What this reading follows
A driving dataset is also a set of decisions about what a perception system can observe and what counts as success. nuScenes combines cameras, lidar, radar, object annotations and maps, then uses detection and tracking baselines to examine those decisions. This reading follows the supplied CVPR 2020 paper: the sensor platform explains how observations are gathered, the temporal ablation tests richer inputs, and the metric comparison shows why rankings depend on the evaluation rule. The experiments concern estimated objects and tracks. They offer evidence about perception quality, while leaving executed driving performance and the benefit of full sensor fusion unestablished. e01e02e04e08e09e10e12e14e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dataset, multisensor annotation and benchmark categories are supported. Architecture, prediction paradigm and quadrant are not applicable because nuScenes is a data and evaluation contribution. Its perception baselines do not instantiate a unified future-and-action model, inverse-dynamics action extractor or inference-time world-model controller. e02e08e10e11e18
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a driving log into a bounded perception example
The dataset is built by selecting interesting scenes, synchronizing sensors and annotating keyframes, rather than labeling every captured sensor frame. Its 2 Hz object annotations coexist with camera, radar and lidar streams at 12, 13 and 20 Hz. The intermediate measurements matter because a detector may use up to half a second of history. To understand an input example, first identify the target keyframe, then the permitted earlier observations, and finally the transformations relating them to that keyframe. Scene descriptions and semantic maps add context, but they are different forms of information from measured returns and object cuboids. Tracking has a longer permissible history, from the scene start to the current time. Neither task permits future observations for the current prediction, and neither evaluates an executed driving action. e04e05e06e08e10e18
Figure 3. The vector map supplies context around recorded ego poses. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to distinguish lanes, road segments, walkways, crossings and other semantic layers. Then follow the black dots along the road: the caption identifies them as keyframe ego poses from scene-0121. They are the recorded vehicle’s positions, not predicted trajectories for surrounding agents. The illustrated map belongs to the vector expansion described in Section 2; the original raster map contained roads and sidewalks. The neighboring text also introduces idealized baseline routes under an assumption of no obstacles. Keep these routes conceptually separate from the observed poses overlaid in this figure. e05e11e16e18
What it supports. nuScenes offers a structured spatial prior in addition to sensor measurements and object labels. The map helps explain how a perception or prediction system could constrain plausible locations and routes. This illustration establishes the available context, while the reported detector tables do not measure an isolated gain from using it.
Where the evidence stops. The overlay is neither a planning result nor a demonstration of obstacle avoidance. The baseline-route assumption excludes obstacles, and no map-use ablation in the supplied paper establishes transfer to unseen city layouts.
5.2 Ask what multiple lidar sweeps actually contribute
The PointPillars adaptation combines two changes in its input: additional points and explicit time offsets. The source describes moving each sweep into the keyframe coordinate system, appending a scalar time difference and learning box velocity as an additional regression target. Table 3 then shows better detection and substantially smaller velocity error as more sweeps are used. Reader interpretation: those observations are compatible with both improved geometric coverage and a useful motion signal. The table cannot allocate the gain between them because it never removes or corrupts time offsets while holding the accumulated points fixed. Pretraining is a separate comparison: at ten sweeps, final scores are similar across KITTI, ImageNet and random initialization. That result helps focus a reproduction on the temporal input mechanism, while the absence of uncertainty prevents strong claims about small initialization differences. e11e12e15
5.3 Read the benchmark score as a bundle of error types
nuScenes first decides whether a detection is close enough to an object using ground-plane center distance. Its mAP averages four distance thresholds and the evaluated classes. It then measures translation, scale, orientation, velocity and attribute errors on matches at two meters. NDS assigns half its weight to mAP and distributes the other half across clipped complements of these five errors. This explains why the nearly equal PointPillars and MonoDIS test mAP values coexist with different NDS values. Figure 7 supplies another perspective: changing the matching criterion alters the bicycle ranking before these quality terms are combined. Tracking adds a continuity question through recall-averaged sAMOTA, initialization delay and longest detection gaps. Read those tracking metrics separately from detection: the reported tracking comparison uses validation data, while Table 4 uses test data. e08e09e10e14e16e17
5.4 Training and inference
During training
PointPillars uses one network across classes and adds box-velocity regression. Earlier sweeps are transformed into the keyframe coordinate system, and each point receives its time offset. OFT is reimplemented with an SSD head. Both baselines assign the most common training-set attribute for each class. e11e12
The data-volume experiment gives PointPillars six times fewer epochs and a one-cycle optimizer schedule. This reduced-training experiment must be distinguished from the main baseline tables. The supplied body does not specify complete loss weights, optimizer configurations or frozen-module policies. e13e11
During inference
PointPillars predicts boxes and velocities from accumulated lidar input. OFT processes single images and combines predictions from all six cameras using non-maximum suppression; its box velocities are set to zero. These outputs describe perceived objects, not action commands. e11e12
Tracking baselines pass Megvii, PointPillars or MonoDIS detections to the same cited tracking approach. The main paper does not provide its complete association or state-update implementation. No closed-loop vehicle-control evaluation is presented. e10e17e18
5.5 Implementation flow
- Collect and align
Two identically equipped cars collect Boston and Singapore data. Six cameras capture at 12 Hz, five radars at 13 Hz and a 32-beam lidar at 20 Hz. Camera exposure is triggered as lidar crosses its field of view; not every lidar scan has an image. Lidar-map localization supplies motion compensation. e04
- Select and annotate
Interesting twenty-second scenes are manually selected from 84 logs totaling 15 hours. At 2 Hz, annotators label class, attributes and cuboid position, dimensions and yaw. Objects are annotated continuously when covered by at least one lidar or radar point; intermediate sensor frames remain available. e03e06
- Provide spatial context
The original raster map covers roads and sidewalks; the vector expansion has eleven semantic classes. Baseline routes describe an ideal obstacle-free path. These are contextual priors, not evidence of obstacle-aware planning or executed control. e05
- Enforce the task information window
Detection at time t may use sensor data in [t − 0.5, t]; tracking may use [0, t]. Detection uses ten classes, while tracking excludes barrier, construction and trafficcone, using the paper’s category wording. e08e10
- Evaluate detection quality and track continuity
True-positive errors measure translation in meters, aligned-box scale as 1 − IoU, yaw in radians, velocity in m/s and attribute error as 1 − accuracy. If a class fails to reach 10% recall, its TP errors become one. Tracking uses sAMOTA, averaging recall-adjusted MOTA over 40 recall levels from 0.1 to 1. Track initialization duration measures delay to first detection; longest gap duration measures the longest missing interval. e08e09e10
6. Experiments & results
nuScenes pairs surround-view camera, lidar and radar recordings with 3D object annotations and maps, then defines detection and tracking evaluations. Its central lesson is that data scale, temporal input and the matching rule all affect perceived progress. The baselines estimate objects and tracks; they do not demonstrate a learned driving controller.
This dataset paper supplies an acquisition-platform diagram and semantic-map example, rather than a neural-network architecture figure. Those original method visuals are used here. The main PDF contains quantitative detector tables and ablations, but no quantitative radar-fusion comparison or executed-control evaluation. Referenced supplementary class statistics and tracking-challenge tables are outside the supplied source. e04e05e11e18e19
6.1 Read the original evidence
Table 4. Similar mAP can hide different localization and motion accuracy. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the retained modality note: dagger-marked OFT, SSD+3D and MDIS use monocular camera images; PP and Megvii use lidar. MDIS abbreviates MonoDIS and PP abbreviates PointPillars. NDS and mAP are percentages where higher is better. The five remaining columns are errors where lower is better: translation in meters, aligned-box scale as one minus IoU, yaw in radians, velocity in meters per second and attribute error as one minus accuracy. Compare PointPillars and MonoDIS across the row, rather than stopping at their similar mAP. This table evaluates the test set, unlike the validation ablations in Table 3. e08e09e16e12
What it supports. PointPillars and MonoDIS report 30.5% and 30.4% mAP, yet their NDS values are 45.3% and 38.4%. PointPillars has smaller translation and velocity errors, explaining why mAP alone is incomplete. Megvii leads the table with 63.3% NDS and 52.8% mAP, within this mixture of baselines and challenge submissions.
Where the evidence stops. The adjacent prose says MonoDIS has larger scale error, but the table gives mASE 0.26 versus PointPillars’ 0.29; the prose conflicts with the table. Different architectures and training procedures also prevent treating these rows as an isolated causal comparison of sensor modalities.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Temporal accumulation for 3D detection nuScenes validation set; PointPillars; KITTI pretraining; 1, 5 or 10 lidar sweeps. | 1 sweep: 31.8 / 21.9 / 1.21; 5 sweeps: 42.9 / 27.7 / 0.34; 10 sweeps: 44.8 / 28.8 / 0.30. NDS (%), mAP (%), mAVE (m/s; lower is better) | One to ten sweeps improves NDS by 13.0 percentage points and mAP by 6.9 points; mAVE decreases by 0.91 m/s. Measured validation results support temporal accumulation, with smaller gains after five sweeps. They do not isolate time-offset information from denser geometry. No uncertainty is reported. e12 |
| Pretraining sensitivity for 3D detection nuScenes validation set; PointPillars; ten lidar sweeps. | KITTI: 44.8 and 28.8; ImageNet: 44.9 and 28.9; none: 44.2 and 27.6. NDS (%) and mAP (%) | Final scores differ little across these initializations; the authors report faster convergence with KITTI pretraining. This is a single reported comparison without seed variation or significance estimates; it does not establish general equivalence of pretraining strategies. e15 |
| Lidar and camera 3D detection nuScenes test set; Table 4 baselines and 2019 detection-challenge submissions. | Megvii: 63.3 and 52.8; PointPillars: 45.3 and 30.5; MonoDIS: 38.4 and 30.4. NDS (%) and mAP (%) | PointPillars and MonoDIS nearly tie in mAP, but translation error is 0.52 versus 0.74 m and velocity error 0.32 versus 1.55 m/s. NDS exposes quality differences hidden by mAP. Architectures and training differ, so this is not a controlled experiment proving one sensor intrinsically superior. e09e16 |
| Tracking with alternative detector inputs nuScenes validation set; the same cited tracker driven by three detectors. | Megvii: 17.9 and 1.50; PointPillars: 3.5 and 1.69; MonoDIS: 4.5 and 1.79. sAMOTA (%; higher is better) and AMOTP (m; lower is better) | Megvii leads both metrics, while MonoDIS exceeds PointPillars in sAMOTA but has worse AMOTP. Detector quality matters, but ranking is metric-dependent. The detection comparison above uses test data; it is not a matched-split correlation analysis. e10e17e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. Detector ordering changes as the training dataset grows. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as the percentage of nuScenes training data used and the vertical axis as validation mAP. The blue curve is SSD+3D, orange is PointPillars and green is OFT. The dashed black line marks a data volume comparable to KITTI, as specified in the caption; it is not a test-set boundary. Around that marker, the blue and orange curves are close. Farther right, PointPillars separates from SSD+3D. Section 4.2 says this experiment shortens PointPillars training by a factor of six and uses a one-cycle optimizer schedule, so its curve is distinct from the main baseline tables. e13e12e16
What it supports. The figure supports the paper’s argument that a small training set can obscure differences between detector designs. PointPillars benefits more from added data over the illustrated range, changing its position relative to SSD+3D. This is evidence for a benchmark-size interaction, not a universal scaling law.
Where the evidence stops. The plotted curves have no uncertainty bands, and the reduced training schedule limits comparison with Tables 3 and 4. The figure does not isolate sample diversity, class composition and optimization effects from the amount of data.
Figure 7. The matching rule can reverse the ranking for bicycles. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare CD and IOU within each class panel before comparing detectors. Colors identify Megvii, PointPillars, MonoDIS and OFT. The caption and surrounding text specify a 2 m center-distance threshold, versus IoU thresholds of 0.7 for cars and 0.5 for pedestrians and bicycles. The vertical axis is AP in percent. This is a diagnostic at particular matching thresholds, not the four-threshold mean AP reported elsewhere. Watch the green MonoDIS bar in the bicycle panel: it leads with center-distance matching, while the blue Megvii bar leads under IoU. The change concerns what qualifies as a correct detection. e14e08e09
What it supports. The figure illustrates why localization tolerance and object footprint affect comparisons between camera and lidar methods. For bicycles, MonoDIS is favored by center-distance matching relative to the two lidar methods. The authors use this result to motivate reporting detection separately from box-quality errors rather than embedding both in IoU matching.
Where the evidence stops. The prose’s broad zero-AP wording overstates the plot: lidar methods retain nonzero small-object IoU AP. Near-zero camera bars do not imply universal failure. The figure caption does not independently name a split, so no additional split assignment is inferred here.
Table 3. Temporal accumulation improves detection and especially velocity estimation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three rows as the sweep-count comparison: KITTI pretraining is fixed while input changes from one to five to ten lidar sweeps. Higher NDS and mAP are better; lower mAVE, measured in meters per second, is better. Then compare the final three rows as a different experiment: the sweep count stays at ten while pretraining changes among KITTI, ImageNet and none. Section 4.2 explains that accumulated points are moved into the keyframe coordinate system and carry time offsets. The table therefore evaluates both richer geometric coverage and temporal information in the model’s input. e12e15
What it supports. With KITTI pretraining, moving from one to ten sweeps raises NDS from 31.8% to 44.8% and mAP from 21.9% to 28.8%, while mAVE falls from 1.21 to 0.30 m/s. Most of the improvement appears by five sweeps. The pretraining comparison shows much smaller differences in final reported scores.
Where the evidence stops. All rows use the validation set. No repeated-run uncertainty is shown, so tiny pretraining differences are not evidence of superiority. The sweep experiment lacks a control separating point density from explicit time-offset information.
7. Analysis & limitations
7.1 What the evidence leaves open
Manual scene selection and two-city coverage constrain representativeness. The paper reports 19.4% rainy and 11.6% nighttime keyframes, plus a 1:10,000 rarest-to-most-common class imbalance. Diverse examples do not by themselves establish geographic or adverse-weather generalization. e06e07
The authors report no promising radar-only PointPillars result and do not give a quantitative radar-fusion baseline here. Providing multiple modalities therefore does not establish a sensor-fusion advantage. e18
Section 4.2 says MonoDIS has larger scale errors, but Table 4 lists mASE 0.26 for MonoDIS and 0.29 for PointPillars; lower is better. The table contradicts that sentence. Figure 7 also limits the broad zero-AP discussion: lidar methods retain nonzero small-object IoU AP. e16e14
7.2 Questions for discussion
- How much of the multi-sweep improvement comes from time information rather than additional geometric samples?
- How should center-distance matching and the separate box-quality errors be weighted for a particular driving application?
- Would controlled geographic and weather holdouts preserve the reported detector ordering?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful attempt needs the described dataset edition, official split assignments, sensor transforms, time offsets, class mapping and evaluator. The supplied body names train/validation/test but omits split counts and construction rules, complete training recipes, hardware and runtimes. Referenced supplementary material is needed to resolve details rather than guessing them. e02e04e08e11e13e19
Reader-proposed checks should separate temporal information from point density, and rescore identical detections with center-distance and IoU matching. Hold splits, training budgets and class definitions fixed. Report repeated-run uncertainty and native-unit errors alongside aggregate scores. These are proposed controls, not completed reproductions. e12e13e14e09
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate temporal information from additional lidar points
Reader-proposed experiment, not performed: train the same PointPillars adaptation with one sweep, ten sweeps with correct time offsets, ten with all offsets zeroed, and ten with shuffled offsets. Fix the split, pretraining, class sampling, training budget and evaluation code, and repeat with multiple seeds. Add point-count-matched versions of the one- and ten-sweep inputs to probe geometric-density effects. Report validation mAP, NDS and mAVE, including moving-object breakdowns where the supplied annotations support them. A repeatable velocity advantage for correct offsets over zeroed or shuffled offsets would support a temporal-information mechanism. If the advantage disappears after these controls, the original sweep-count result cannot establish that mechanism independently of geometry. e04e06e08e11e12e15
Check 2: Measure ranking sensitivity without changing predictions
Reader-proposed experiment, not performed: freeze one set of predictions from each compared detector on the same split, with identical class filters and confidence handling. Rescore car, pedestrian and bicycle detections using the Figure 7 rules: center distance at 2 m, or IoU at 0.7 for cars and 0.5 for pedestrians and bicycles. Also retain official four-threshold mAP and native-unit TP errors as separate outputs. Check whether the bicycle ordering reverses and whether camera small-object IoU AP collapses while center-distance AP remains useful. Failure to recover those patterns would challenge reproducibility of this diagnostic. Explicitly report any missing checkpoint or preprocessing detail; do not silently tune predictions separately for each matching rule. e08e09e14e16
8.3 Reading coverage
Visual audit: All eleven supplied PDF pages were visually inspected, including title/authors and the CVF version notice, Figures 1–7, Tables 1–4, the metric equations, training/inference descriptions, results and references. All six final original crops were inspected. Figure 4 axis markings were checked against its legend and the collection description; Figure 7 was checked against its caption and metric definitions. Table 4’s modality and abbreviation note is retained with its table. The scale-error prose/table conflict and the overbroad zero-AP wording are disclosed. No separate supplementary pages were supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1. Introduction; 1.1. Contributions (PDF pp. 1–2)
- 1.2. Related datasets (PDF p. 3)
- 2. The nuScenes dataset (PDF pp. 3–5)
- 3. Tasks & Metrics; 3.1. Detection (PDF pp. 5–6)
- 3.2. Tracking (PDF p. 6)
- 4. Experiments; 4.1. Baselines (PDF pp. 6–7)
- 4.2. Analysis; 5. Conclusion; Acknowledgements (PDF pp. 7–8)
- References, entries 1–91 (PDF pp. 9–11)
Outside the original text pass
- Identity/version: the title and all ten authors match the catalog, apart from sentence case in the printed title. The title-page watermark identifies the CVPR 2020 Open Access version and states that it is identical to the accepted version except for the watermark. No numbered revision is printed. The supplied PDF is identified by its immutable hash; no other edition was inspected.
- Pagination discrepancy: the supplied PDF prints 11621–11631, whereas the catalog BibTeX records 11618–11628. Locations here use the supplied PDF page indices. The catalog was not changed.
- This 2020 paper discusses the March 2019 full dataset release, subsequent map expansion and 2019 challenge results. It is not a reading of an original 2019 preprint or a current dataset release. Section 3.2 notes that preprints called sMOTAr MOTAR.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all eleven supplied PDF pages and all six final crops. No separate supplement was supplied. Referenced supplementary figures, class mappings, per-class tables and tracking-challenge results were not inspected; no appendix is present in this PDF.
- The dataset, devkit, evaluation implementation, cited papers and external links were not opened. No code was inspected and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, CVF watermark, Abstract, Introduction and footnote 2; PDF pp. 1 and 11, printed page numbers
The title page credits Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan and Oscar Beijbom to nuTonomy: an APTIV company. The watermark identifies the CVPR 2020 Open Access accepted version. Printed pagination runs 11621–11631. The abstract gives sensor counts, scene count, classes and attributes; footnote 2 dates the full release to March 2019.
Go to primary source ↓e02PDF p. 2, Section 1.1 Contributions and Figure 2
The contribution is a multimodal surround-view dataset, task metrics and baselines. The paper reports publishing the devkit, evaluation code, taxonomy, annotator instructions and database schema, and specifies CC BY-NC-SA 4.0 for the data. Figure 2 shows clear, night, rain and construction examples.
Go to primary source ↓e03PDF p. 3, Table 1, nuScenes row; Section 2, Drive planning; PDF p. 4, opening continuation
The nuScenes row lists 1,000 scenes, 5.5 hours, 1.4M RGB images, 400k lidar pointclouds, 1.3M radar pointclouds, 40k annotated frames and 1.4M boxes. Drive planning describes 84 selected logs with 15 hours of Boston/Singapore driving.
Go to primary source ↓e04PDF p. 4, Table 2, Figure 4, Car setup, Sensor synchronization, Localization and footnote 5
The platform has six 12 Hz cameras, one 32-beam 20 Hz lidar, five 13 Hz radars and GPS/IMU. Figure 4 labels sensor coordinate axes. Two cars share a layout. Camera triggering is tied to lidar azimuth, with no image for every scan. Lidar-map Monte Carlo localization with odometry supports motion compensation.
Go to primary source ↓e05PDF p. 4, Maps, Figure 3 and paragraph immediately below Figure 3
The original raster map has roads and sidewalks; the vector map expansion provides eleven semantic classes. Figure 3 overlays black keyframe ego poses for scene-0121. Baseline routes are ideal paths assuming no obstacles; they are suggested priors for trajectory prediction.
Go to primary source ↓e06PDF p. 4, Scene selection and Data annotation
The authors manually select 1,000 interesting twenty-second scenes, provide scene descriptions, and annotate 2 Hz keyframes with category, attributes and 3D cuboids. Continuous object annotation requires at least one lidar or radar point; intermediate frames are released.
Go to primary source ↓e07PDF p. 5, Annotation statistics and Figure 5
The text reports 40k keyframes, rain in 19.4% and night in 11.6%, and a 1:10k imbalance between rarest and most common annotated classes. The location breakdown and spatial coverage show concentrated, nonuniform coverage; most data comes from intersections.
Go to primary source ↓e08PDF p. 5, Section 3 Tasks & Metrics; Section 3.1 Detection, Eqs. (1)–(2) and TP metric definitions
Detection is restricted to [t−0.5,t], tracking to [0,t]. Ten detection classes use center-distance AP at 0.5, 1, 2 and 4 m, excluding precision/recall below 10%. TP errors use 2 m matches. Translation, scale, yaw, velocity and attribute errors have explicit definitions and class exclusions.
Go to primary source ↓e09PDF p. 6, Section 3.1, nuScenes detection score, Eq. (3)
NDS combines five times fractional mAP with the sum of five clipped mean-TP-error complements, divided by ten. Translation, orientation and velocity errors can exceed one, motivating clipping. Half the score is detection AP and half is box, motion and attribute quality.
Go to primary source ↓e10PDF p. 6, Section 3.2 Tracking, sMOTAr/sAMOTA definitions, TID and LGD; PDF p. 7, Tracking baselines
Tracking excludes barrier, construction and trafficcone. sAMOTA averages adjusted MOTA over 40 recall levels from 0.1 to 1; preprints used the name MOTAR for sMOTAr. TID measures time to first detection and LGD the longest missing interval. Three detector inputs share the tracking approach cited as reference 77.
Go to primary source ↓e11PDF p. 6, Section 4.1, Lidar detection baseline and Image detection baseline
PointPillars is trained jointly across classes and extended with velocity regression. OFT is reimplemented with an SSD head; separate camera predictions are merged with NMS. Attributes use the most common class attribute; OFT velocity is zero. Complete training details are not given in these baseline descriptions.
Go to primary source ↓e12PDF p. 7, Section 4.2, Multiple lidar sweeps improve performance; PDF p. 8, Table 3, KITTI rows
Sweeps are moved to keyframe coordinates and decorated with time offsets. For 1/5/10 sweeps with KITTI pretraining, validation NDS is 31.8/42.9/44.8%, mAP 21.9/27.7/28.8%, and mAVE 1.21/0.34/0.30 m/s. The experiment varies temporal input without a separate time-offset ablation.
Go to primary source ↓e13PDF p. 7, Section 4.2, The case for a large benchmark dataset, Figure 6 and caption
Figure 6 compares validation mAP against training-data fraction for PointPillars, SSD+3D and OFT. The dashed line marks KITTI-scale training volume. PointPillars separates from SSD+3D as more data is used; this ablation uses six times fewer PointPillars epochs and a one-cycle optimizer schedule.
Go to primary source ↓e14PDF p. 7, Figure 7, caption and The importance of the matching function
Figure 7 compares 2 m center-distance matching with IoU thresholds 0.7 for cars and 0.5 for pedestrians/bicycles. MonoDIS leads bicycle AP under center distance; Megvii leads under IoU. The prose broadly describes zero small-object IoU AP, but plotted lidar bars include clearly nonzero results. Camera small-object IoU bars are near zero.
Go to primary source ↓e15PDF p. 8, Table 3, ten-sweep rows; The importance of pre-training
Ten-sweep validation NDS/mAP is 44.8/28.8 with KITTI, 44.9/28.9 with ImageNet and 44.2/27.6 with no pretraining. The authors report faster convergence with KITTI. The table gives no repeated-run uncertainty.
Go to primary source ↓e16PDF p. 8, Table 4, all rows and modality footnote; Which sensor is most important?
Test NDS/mAP is 63.3/52.8 for Megvii, 45.3/30.5 for PointPillars, and 38.4/30.4 for MonoDIS. PointPillars/MonoDIS translation errors are 0.52/0.74 m and velocity errors 0.32/1.55 m/s. Table mASE is 0.29/0.26, contradicting the prose claim of larger MonoDIS scale error. Dagger rows use camera images; other rows use lidar.
Go to primary source ↓e17PDF p. 8, Better detection gives better tracking
On validation, Megvii, PointPillars and MonoDIS obtain sAMOTA 17.9%, 3.5%, 4.5% and AMOTP 1.50, 1.69, 1.79 m. The text reports shortest LGD and most fragmentations for MonoDIS. Its cross-task comparison references test-set detection Table 4.
Go to primary source ↓e18PDF p. 8, Which sensor is most important? and Section 5 Conclusion
The authors focus on lidar and image detectors because no competitive radar-only method was available to them and their preliminary radar PointPillars study was not promising. The conclusion describes detection/tracking contributions and lists semantic labels and a trajectory-prediction benchmark as future work; no executed-control experiment is reported.
Go to primary source ↓e19PDF p. 5, references to Figures 8-SM, 9-SM, 10-SM, 14-SM and Table 5-SM; PDF p. 7, Tracking baselines; PDF p. 8, reference to Table 7-SM
The main paper directs readers to supplementary annotation statistics, class mappings, per-class detection results and tracking-challenge results. Those items are not contained in the supplied eleven-page main-paper PDF.
Go to primary source ↓8.5 Primary sources
nuScenes: A Multimodal Dataset for Autonomous Driving ↗
PDF · 9,303 extracted words
Source fingerprint
4a2822800db7e5709262e79f56fe7814350f3d7e67a16e5a072a04f7082b86fd