Are we ready for autonomous driving? The KITTI vision benchmark suite
1. Paper overview
In one sentence: KITTI exposes road-scene perception failures through calibrated references and diagnostic evaluation, while its selected scenes and partial labels limit claims about autonomous driving readiness. e02e03e05e07e11e12e13e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Small or controlled vision benchmarks can reward assumptions that fail on roads. KITTI asks whether existing perception methods transfer to urban, rural and highway imagery, using sensor-derived references and task-specific evaluation rather than a vehicle-level success score. e02 |
| Core mechanism | |
| A key reported result | Optical flow estimation: TGV2CENSUS: 11.14% / 18.42%. Percentage with endpoint error >3 px, non-occluded / all labeled pixels.. Same 195-pair test benchmark and ranking date; static scenes with camera motion. HS: 19.92% / 28.86%; both report 100% estimated density. Substantial errors remain even without independently moving objects; Figure 4 illustrates large-motion failure. e07e08e12 |
| Reading caution |
Core contributions
Figure 1. One recording platform supplies several distinct forms of perception ground truth. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the roof-mounted stereo rig, laser scanner and GPS labels at upper left. The neighboring trajectory represents the odometry task; the stacked image and colored maps at upper right introduce stereo disparity and optical flow. The lower image shows projected object boxes. Read these as different products of synchronized acquisition. Sections 2.2–2.3 explain the missing processing steps: calibrate the coordinate systems, accumulate laser measurements for image-space references, transform localization output into the camera frame, and annotate object tracklets. The montage is a guide to the benchmark interfaces; it does not specify a neural architecture or connect the outputs through prediction arrows. e02e03e04e05e06
What it supports. The suite can compare visual estimators against complementary sensor-derived references using a common acquisition platform. That makes the calibration and annotation pipeline scientifically central: the apparent simplicity of a disparity map or bounding box hides registration, filtering and human labeling decisions that determine which predictions can be scored.
Where the evidence stops. The colored maps have no numerical color key here, and the trajectory has no coordinate scale. They establish task modalities, not accuracy, planned motion or executed autonomous control. Use Sections 2.2–2.3 for how references are constructed.
2. Motivation
2.1 The problem and the proposed response
Small or controlled vision benchmarks can reward assumptions that fail on roads. KITTI asks whether existing perception methods transfer to urban, rural and highway imagery, using sensor-derived references and task-specific evaluation rather than a vehicle-level success score. e02
2.2 What this reading follows
The useful starting point for KITTI is the measurement pipeline: what can a moving car observe, and which parts can another sensor verify? The original 2012 paper answers with synchronized cameras, a laser scanner and GPS/IMU localization, then builds separate stereo, flow, odometry and object-perception tasks. Its results show that realistic imagery can change familiar rankings and reveal severe failures. This reading follows that chain from sensors to labels, sampling, metrics and comparisons. It also keeps the original evaluation boundaries visible: static scenes for stereo and flow, partial ground truth, odometry without loop closure, and a restricted object subset. These are perception benchmarks, not a driving-policy evaluation. e02e03e05e07e11e12e13e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The dataset and perception/odometry benchmark classification is supported. Architecture, prediction paradigm and quadrant are not applicable: this work defines observations, labels and evaluation protocols, not a world/action model. Separate baseline algorithms do not establish either a One Model or Multiple Models WAM architecture. e02e05e09e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the measurement pipeline from the tested estimator
A laser reference is only useful to a camera algorithm after coordinate systems and time have been aligned. KITTI first calibrates and rectifies cameras, registers the laser and localization sensors, and compensates laser egomotion. For stereo and flow, it then aligns neighboring scans with ICP, projects accumulated points, and removes ambiguous regions by hand. The remaining points define approximately half-image ground truth; they are deliberately not interpolated. Predictions, however, are interpolated for the ranking comparison. Reader interpretation: these two choices answer different questions. Leaving references sparse avoids creating unmeasured truth, whereas filling predictions makes incomplete methods comparable under a chosen policy. A faithful evaluation must therefore preserve both the reference mask and prediction-filling rule. A dense-looking result image alone cannot establish dense ground-truth accuracy. e03e04e05e08
Figure 2. Recorded road scenes supply unequal class frequencies and strongly structured object geometry. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top row as a description of what was labeled: class counts, visibility restrictions, and the number of pedestrians or cars in an image. The lower-left histograms describe orientation, while the stacked lower-right plots describe object height, width and length in meters. Axes change between counts, percentages, angles and dimensions, so bar heights cannot be compared across panels. Cars and pedestrians dominate, but their orientation distributions differ. Section 2.4 then describes a separate selection process that rewards visible-object counts and class-specific orientation entropy. The histograms characterize the labeling effort; they are not proof that the selected training and test subsets are balanced. e06e07e14e17
What it supports. KITTI makes dataset composition inspectable. Unequal class frequencies, visibility and orientations help explain why object selection matters in addition to raw annotation volume. The greedy entropy term is motivated by diversity, but this figure does not demonstrate how much that term changes the selected benchmark or improves a detector.
Where the evidence stops. These statistics must not be substituted for the filtered object-evaluation population. Section 3.4 evaluates a smaller subset and excludes truncated, small and more heavily occluded objects; no before/after ablation of the selection rule is shown.
5.2 Read the benchmark population before interpreting the winner
The collected data and each evaluated task are different populations. Stereo and flow select static scenes using clustering over coarse disparity and motion descriptors, then remove poor illumination. Object selection instead rewards non-occluded counts and orientation entropy, while sequence separation protects its training/test division. Detection subsequently applies additional size, occlusion and truncation restrictions. This distinction explains why the paper can describe a large, cluttered recording collection yet evaluate a simpler object subset. Reader interpretation: a benchmark result is conditional on the selection and scoring rules, so a strong row in Table 2 or a high precision curve does not establish coverage of moving objects, tunnels or severe occlusion. Figure 2 helps inspect the recorded distribution, but a selection ablation would be needed to quantify the entropy rule’s effect. e06e07e08e14e17
5.3 Keep localization, detection and orientation scores separate
Odometry is scored through relative camera-pose discrepancies, split into translation and rotation and examined by distance and speed. This avoids making the final trajectory endpoint the sole summary. Object orientation uses a different construction: detections are matched to boxes, each accepted orientation contributes a normalized cosine similarity, and unmatched detections receive zero. Equation (5) produces a similarity value at a recall level; Equation (4) aggregates its interpolated envelope over eleven recall levels. Figure 6 shows curves, while Table 3 tests classification or regression on cropped car boxes. Reader interpretation: the 0.93 classification similarity in Table 3 cannot be read as a detector AOS score or localization success. Reproduction must also resolve the paper’s inconsistent overlap wording and the right-hand Figure 6 variant description. e09e10e14e15
5.4 Training and inference
During training
There is no shared learned KITTI model. The detector comparison uses 100 non-occluded examples per each of 16 orientation classes: variable latent components, orientation-initialized components allowed to vary, or components fixed to orientation classes. In the odometry experiment, ground truth was not used to fit parameters. e14e13
The separate car-orientation experiment uses 100 instances per orientation bin and HOG descriptors from resized boxes: 19 by 13 blocks, 8 by 8 pixel cells, 12 HOG orientation bins. It compares SVM/nearest-neighbor classification and GP/SVM/nearest-neighbor regression. e15
During inference
Each baseline independently estimates disparity, image motion, camera poses, boxes or orientations. Evaluation occurs against recorded references; no common latent rollout, action extraction, feedback controller or executed driving policy is specified. Odometry comparisons exclude loop closure. e02e08e13e14
5.5 Implementation flow
- Acquire and register
Four cameras and a Velodyne scanner record at 10 Hz; same-type cameras have roughly 54 cm baseline. Calibrate cameras, register laser to camera, then GPS/IMU to laser. Laser-camera refinement combines manual reprojection residuals with disparity outliers relative to top benchmark methods. e03e04
- Construct task references
ICP registers five preceding and five following frames around a target. Project accumulated points, remove ambiguous regions, and derive disparity and next-frame flow without interpolating ground truth. GPS/IMU supplies camera-frame trajectories; annotators supply 3D box tracklets. e05e06
- Select examples
For static-scene stereo/flow, cluster 144-dimensional motion/disparity descriptors from a 12 by 4 grid with k=400, select cluster representatives, then remove poor illumination. Object selection greedily balances visible-object counts and orientation entropy; images from one sequence cannot enter both object training and test sets. e07
- Score compatible outputs
Count matching errors above three pixels separately for non-occluded and all labeled pixels, averaging per-image percentages. Interpolate missing predictions for comparison. For odometry, compare relative pose errors over subsequences by length and speed. Object evaluation combines 2D box matching with orientation similarity. e08e09e10
6. Experiments & results
KITTI turns synchronized driving recordings into separately scored stereo, optical-flow, odometry and object-perception benchmarks. Its contribution is calibrated reference data, selection rules and diagnostic metrics. The 2012 results expose failures under large motion and difficult surfaces; they do not measure autonomous driving success (e02, e05, e12, e13).
This dataset paper has no learned-model architecture diagram: Figure 1 supplies the original acquisition/task overview, with the construction mechanism explained from Sections 2.1–2.4. It contains detector-variant comparisons and failure diagnostics, but no controlled ablation of sensor calibration or benchmark selection. Figures 4–6 are therefore presented as diagnostics and a restricted detector comparison, not as proof that a new model component causes the overall benchmark results. e02e03e04e05e07e12e13e14
6.1 Read the original evidence
Table 2. The April 2, 2012 ranking separates prediction density from error on labeled pixels. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a task half before comparing rows. Non-Occluded and All are error percentages, where lower is better; Density records the fraction of estimated pixels and is not an accuracy score. The retained note defines the default threshold as error greater than three pixels and says that percentages are averaged over test images. It also explains the interpolation of invalid disparities and flow vectors. Read PCBP against ITGV within stereo, or TGV2CENSUS against HS within flow. Do not treat the two halves as the same prediction task. The word All refers to pixels with ground truth, not to every image pixel. e07e08e11e12e05
What it supports. PCBP reports 4.72% non-occluded error against ITGV’s 6.31%; their all-labeled errors are 6.16% and 7.40%. For flow, TGV2CENSUS reports 11.14% and 18.42%, versus HS at 19.92% and 28.86%. These measured errors demonstrate substantial remaining difficulty even for the leading tested methods under the paper’s restricted benchmark.
Where the evidence stops. This is a dated comparison on static scenes with semi-dense references and interpolated predictions. Neither uncertainty intervals nor vehicle-level consequences are reported. “Non-occluded” also has the source’s narrower projection-based meaning, rather than a complete visibility mask.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Stereo matching 2012 benchmark, 195 test pairs; April 2 ranking; invalid predictions interpolated. | PCBP: 4.72% / 6.16%. Percentage of pixels with disparity error >3 px, non-occluded / all labeled pixels; lower is better. | ITGV: 6.31% / 7.40%; both have 100% estimated density. PCBP leads these tested methods. All-pixel coverage still means only pixels with ground truth. e07e08e11 |
| Optical flow estimation Same 195-pair test benchmark and ranking date; static scenes with camera motion. | TGV2CENSUS: 11.14% / 18.42%. Percentage with endpoint error >3 px, non-occluded / all labeled pixels. | HS: 19.92% / 28.86%; both report 100% estimated density. Substantial errors remain even without independently moving objects; Figure 4 illustrates large-motion failure. e07e08e12 |
| Visual odometry without loop closure 39.2 km benchmark; errors analyzed by subsequence length and speed. Train/test sequence allocation is not stated in this PDF. | VISO2-S: 2.2%; 0.016 degrees/m. Average translation error (%) and rotation error (degrees/m), lower is better. | Authors identify VISO2-S as closest overall; Figure 5 compares six labeled variants, including monocular VISO2-M. Aggregate performance does not imply leading every rotation bin or successful loop-closing SLAM. e02e09e13 |
| Car orientation from cropped boxes 100 instances per each of 16 orientation bins; HOG features; exact train/test allocation unstated. | Classification SVM: 0.93; regression GP: 0.92. Average orientation similarity, higher is better. | Classification NN: 0.85; regression SVM: 0.91, NN: 0.86. These are separate classification/regression comparisons on object crops, not end-to-end detection AOS. e15e10 |
| Detector orientation-component variants 12,000 images/40,000 objects overall; train/test counts unstated. Evaluate non-/weakly-occluded (<20%), untruncated objects at least 40 px high, with box overlap at least 50%. | Authors report no substantial difference; no scalar AP/AOS values supplied. Precision/recall and orientation-similarity curves. | Variable, fixed-init and fixed for detection; fixed-init versus fixed in the orientation plot. Evidence is restricted to filtered objects; figure legends conflict with the prose label for the orientation variants. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Large image displacement exposes a severe optical-flow failure case. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each column, follow the input road image down to estimated flow and then endpoint error. The paper’s caption maps zero error to black and errors of at least five pixels to white; interpret that scale only where a reference exists. The left and right labels identify selected best and worst cases. Section 3.2 connects them to flow bounds of at most 55 and 176 pixels, respectively, and discusses the difficulty of close-range road pixels and missing texture in pyramidal estimation. The flow colors themselves have no legend in this figure, so use the error image and accompanying text rather than assigning numerical motion from hue. e12e05e08
What it supports. The best example is labeled below 1% errors and the worst 59%, with a broad bright road region in the latter error map. This supports the paper’s warning that a good average ranking can conceal extreme failures. The examples are consistent with its large-displacement diagnosis, but they do not isolate speed from scene structure.
Where the evidence stops. These are selected extremes, not an error distribution or a controlled speed ablation. The figure does not explicitly identify which evaluation mask its percentage labels use. Unlabeled or filtered regions must not be interpreted as demonstrated zero error.
Figure 5. Subsequence diagnostics distinguish translation drift from rotation error and expose speed dependence. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read columns as error type and rows as conditioning variable. The left column uses translation error in percent; the right uses rotation error in degrees per meter. The top row varies path length in meters, and the bottom row varies speed in kilometers per hour. Follow one legend entry across panels before comparing methods. VISO2-M is the monocular entry; the others use stereo. All evaluated methods omit loop closure. The source’s introductory sentence says five approaches, but the actual legends identify six variants. The plotted curves represent averages over subsequences, not separate training schedules or individual end-point trajectories. e09e13
What it supports. VISO2-S has low translation-error curves, while the monocular VISO2-M is substantially worse across these plots. Rotation rankings differ from translation rankings, illustrating why a single endpoint score would hide useful distinctions. The low-speed and high-speed regions expose different failure regimes discussed in the accompanying text: accumulated drift and large inter-frame motion.
Where the evidence stops. These comparisons do not isolate sensor modality because the algorithms also differ. Speed covaries with road structure, and no confidence bands are plotted. The source does not provide the complete implementation of distance normalization, binning and aggregation in this PDF.
Figure 6. Orientation supervision changes component constraints but yields similar curves in the filtered detector experiment. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First compare the three precision-recall traces on the left. Their legends distinguish variable components, orientation-initialized components that can change, and components fixed to orientation classes. Each setting uses 100 non-occluded training examples per orientation class. The right-hand legend contains only fixed init and fixed. Its vertical label says AOS, but Eqs. (4)–(5) distinguish the similarity curve at a recall level from the final eleven-point average. Section 3.4’s phrase “two unsupervised variants” conflicts with these right-hand legends and the training descriptions. This edition follows the visible names while retaining the discrepancy; it does not relabel either curve. e14e10
What it supports. The authors find no substantial difference among the investigated detector variants, and the plotted traces are close. That finding concerns the retained objects and this training configuration. Precision remains an upper bound on orientation similarity because an assigned detection contributes at most one to the cosine-based score, while unmatched detections contribute zero.
Where the evidence stops. Objects are untruncated, at least 40 pixels high and below 20% occlusion. The curves contain no uncertainty or printed scalar AP/AOS values. The inconsistent prose description leaves the exact intended characterization of the right-hand variants unresolved.
7. Analysis & limitations
7.1 What the evidence leaves open
Stereo/flow excludes dynamic scenes and poor illumination. Roughly half the image has ground truth; ambiguous regions are removed, and the non-occluded mask does not reliably resolve within-image occlusions. These choices narrow the measured difficulty. e05e07
Laser-camera calibration partly minimizes errors relative to benchmark methods. This motivates a possible evaluator-dependence concern, but the paper does not quantify its effect. e04
Keep the over-200k recorded annotations, Table 1 two-category average of 80,000 labels, and evaluated subset of 12,000 images/40,000 objects distinct; the source does not fully reconcile the annotation totals. No uncertainty estimates accompany the retained ranking and orientation values. e17e11e12e15
Figure 6(b) names fixed-init/fixed, whereas Section 3.4 says two unsupervised variants. Section 2.5 also alternates between >50% and at-least-50% overlap; Section 3.4 uses the latter. Neither inconsistency is silently resolved. e10e14
7.2 Questions for discussion
- How sensitive are rankings to calibration choices and the evaluated-pixel mask (e04, e05, e08)?
- Would the detector-variant conclusion survive the harder occlusion/truncation cases proposed as future work (e14, e16)?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful evaluation needs this historical release, calibration, ground-truth masks, sequence splits, interpolation policy and per-image aggregation. The paper defers stereo parameters to a website and provides no complete software versions, compute recipe or license; linked kits alone do not establish reproducibility. e04e05e07e08e16
Selection weight alpha, calibration objective weights and the car-orientation train/test allocation remain unspecified. Odometry equations average pose discrepancies, while plotted units normalize by distance; exact binning and weighting require implementation confirmation. e04e07e09e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Measure whether calibration refinement favors its reference methods
Reader-proposed, not run: on a fixed calibration/evaluation split, compare laser-camera transforms refined using manual correspondences alone against transforms refined with the additional disparity-outlier term described in Section 2.2. Hold initial registration, evaluation frames, masks, interpolation and tested predictors fixed. Reserve manually labeled correspondences for independent reprojection checks, and distinguish predictors used in calibration from held-out predictors. Report reprojection residuals and paired changes in three-pixel error rates. A gain concentrated in calibration-reference methods without improved held-out geometric residuals would support evaluator dependence; broadly improved residuals and rankings would weaken that concern. The original objective weights must first be recovered or explicitly varied. e04e05e08
Check 2: Test whether fixed orientation components remain equivalent on harder objects
Reader-proposed, not run: first reproduce variable, fixed-init and fixed detector training on the same sequence-separated data and equal orientation-bin sample counts. Lock features, optimization budget and matching rules. Compare precision-recall, similarity curves and scalar AOS on the original filtered objects, then separately on additional occlusion/truncation strata with verified labels. Use repeated training runs and paired resampling by sequence to assess differences. A consistent fixed-versus-fixed-init gap confined to harder strata would show that the paper’s near-equivalence is population-dependent. Predeclare the 50% overlap boundary and retain the Figure 6 legend names until the prose discrepancy is resolved. e07e10e14e16
8.3 Reading coverage
Visual audit: All eight pages were rendered and visually inspected after reading all four text chunks. Page 1 verifies title, authors, affiliations and Figure 1; pages 2–3 verify sensor/calibration details, Table 1 and reference construction; page 4 verifies Figure 2, sampling and Eq. (1); page 5 verifies Eqs. (2)–(5), evaluation rules and Figures 3–4; page 6 verifies Table 2, Figure 5 and matching/odometry results; page 7 verifies Figure 6, Table 3, training details, restrictions and future work; page 8 contains the remaining references. Every final asset was viewed, including the corrected wider Table 2 crop. Its evaluation note is retained because it defines the threshold, masks, density and interpolation. Figure 6 legend/prose and curve/scalar distinctions are disclosed. The supplied PDF has no appendix; separate supplements and external code/configuration remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and author block; Abstract
- 1. Introduction
- 2. Challenges and Methodology: 2.1 Sensors and Data Acquisition; 2.2 Sensor Calibration; 2.3 Ground Truth; 2.4 Benchmark Selection; 2.5 Evaluation Metrics
- 3. Experimental Evaluation: 3.1 Stereo Matching; 3.2 Optical Flow Estimation; 3.3 Visual Odometry/SLAM; 3.4 3D Object Detection / Orientation Estimation
- 4. Conclusion and Future Work
- References, pp. 7–8
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The preceding acquisition limitation was addressed by visually inspecting all eight PDF pages and every final crop.
- Separate supplemental material availability has not been fully verified.
- No separate supplement was supplied. No code, development kit, website configuration or evaluation server was inspected; no experiments were reproduced.
- Identity/version: the supplied author-hosted CVPR 2012 eight-page paper matches all three catalog authors. The observed title differs from the catalog only in capitalization. No revision identifier or date is printed on the title page; no arXiv version or later KITTI edition is asserted. Current dataset sizes, rules and availability were not verified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title and author block
The observed title is Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. Andreas Geiger and Philip Lenz are affiliated with Karlsruhe Institute of Technology; Raquel Urtasun with Toyota Technological Institute at Chicago.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Introduction and Figure 1
The paper introduces stereo, optical flow, visual odometry/SLAM and object detection/orientation benchmarks from a driving platform. It reports 389 stereo/flow pairs, 22 odometry sequences totaling 39.2 km, and over 200k object annotations in the recorded data.
Go to primary source ↓e03PDF p. 2, Section 2.1; p. 1, Introduction and Figure 1
Acquisition uses two color and two grayscale PointGrey Flea2 cameras and a Velodyne HDL-64E at 10 Hz, with GPS/IMU localization. Same-type camera baseline is roughly 54 cm. GPS/IMU positions compensate laser egomotion; p. 1 identifies the localization system as OXTS RT 3003.
Go to primary source ↓e04PDF pp. 2–3, Section 2.2, all three calibration paragraphs
Camera calibration minimizes reprojection error. Laser-to-camera refinement combines disparity outliers relative to top benchmark methods and manual correspondences using Metropolis-Hastings sampling. GPS/IMU-to-laser registration uses ICP and hand-eye calibration, sampling 1000 pose pairs from a parking sequence.
Go to primary source ↓e05PDF pp. 3–4, Section 2.3, stereo/flow and odometry ground-truth paragraphs
ICP aligns five frames before and five after a target. Accumulated laser points are projected into images; ambiguous windows/fences are manually removed. Ground truth is not interpolated and averages about 50% density. Non-occluded excludes out-of-image projections but cannot reliably resolve within-image occlusions. Odometry truth is GPS/IMU output in the rectified left-camera frame.
Go to primary source ↓e06PDF p. 4, Figure 2 and Section 2.3, object ground truth
Annotators create 3D box tracklets using laser points and camera images and label visibility, occlusion and truncation. Figure 2 shows unequal class counts, object multiplicity, orientation and size distributions; cars and pedestrians predominate.
Go to primary source ↓e07PDF p. 4, Section 2.4 and Eq. (1); p. 1, final paragraph
Approximately 3 TB is reduced to task-specific benchmarks. Stereo/flow uses static scenes, 144-dimensional descriptors over 12 by 4 blocks, k-means with k=400, and removes bad illumination, yielding 194 training and 195 test pairs at rectified 1240 by 376 resolution. Odometry has 41,000 frames at 10 fps over 39.2 km. Object selection favors non-occluded counts and class orientation entropy, with 8/16 pedestrian/car bins and sequence-separated training/test sets; alpha is not specified numerically.
Go to primary source ↓e08PDF p. 5, Section 2.5, first paragraph and Section 3.1; p. 6, Table 2 caption
Matching errors use disparity or flow endpoint thresholds of 2–5 pixels, default 3, without image downsampling. Table 2 averages percentages exceeding 3 pixels over test images and distinguishes non-occluded, all labeled pixels, and estimated density. Invalid predictions are interpolated; stereo uses background interpolation.
Go to primary source ↓e09PDF p. 5, Section 2.5, Eqs. (2)–(3); p. 6, Figure 5 and caption
Odometry separately averages rotation and translation discrepancies of estimated versus true relative poses over frame-pair sets. Figure 5 reports translation error in percent and rotation error in degrees per meter, grouped by path length and speed.
Go to primary source ↓e10PDF p. 5, Section 2.5, Eqs. (4)–(5) and detection-matching paragraphs
AOS averages the interpolated orientation-similarity envelope at eleven recall levels. Similarity is cosine-based, normalized by all detections at a recall level; unmatched or duplicate detections receive zero via the assignment indicator. The text says more than 50% box overlap once and at least 50% subsequently.
Go to primary source ↓e11PDF p. 6, Table 2, stereo PCBP/ITGV/CostFilter/GC+occ rows; pp. 5–6, Figure 3 and Section 3.1
In the April 2, 2012 ranking, PCBP has 4.72% non-occluded and 6.16% all-pixel error; ITGV has 6.31% and 7.40%. Both report 100% density. The authors associate difficult examples with saturation, disparity shadows and non-Lambertian surfaces and discuss poor transfer of some Middlebury-leading methods.
Go to primary source ↓e12PDF p. 6, Table 2, TGV2CENSUS/HS rows and Section 3.2; p. 5, Figure 4 and caption
TGV2CENSUS has 11.14% non-occluded and 18.42% all-pixel error; HS has 19.92% and 28.86%, both at 100% density. Figure 4 labels best/worst examples below 1% and 59%. Section 3.2 reports flow bounds of at most 55 and 176 pixels in those examples and discusses large displacements and missing texture. Error images run from black at zero to white at at least five pixels.
Go to primary source ↓e13PDF pp. 6–7, Section 3.3 and Figure 5
VISO2-S averages 2.2% translation error and 0.016 degrees/m rotation error. Compared methods do not use loop closure; all except VISO2-M are stereo. Odometry parameters were not trained using ground truth. Large motions, limited highway structure and slow-speed drift are discussed. The plot has six labeled variants although the paragraph calls them five approaches.
Go to primary source ↓e14PDF p. 7, Section 3.4, first four paragraphs and Figure 6 legends; p. 5, Eqs. (4)–(5)
The object benchmark subset contains 12,000 images and 40,000 objects. Detector training uses 100 non-occluded examples per each of 16 orientation classes and variable, fixed-init and fixed latent-component settings. Evaluation retains non-/weakly-occluded objects below 20% occlusion, untruncated and at least 40 pixels high. Figure 6(a) compares all three variants, and 6(b) labels fixed init and fixed, contradicting the prose description of two unsupervised variants. The authors report no substantial performance difference; no scalar AP/AOS values are printed.
Go to primary source ↓e15PDF p. 7, Table 3 and Section 3.4, final paragraph
Car orientation evaluation extracts 100 instances per 16 bins, uses HOG on cropped/resized boxes with 19 by 13 blocks, 8 by 8 pixel cells and 12 orientation bins. Classification similarity is SVM 0.93 and NN 0.85; regression is GP 0.92, SVM 0.91 and NN 0.86. Table 3 calls these errors but specifies higher is better. The train/test allocation for this experiment is not specified.
Go to primary source ↓e16PDF p. 7, Section 3.4, evaluation restrictions and Section 4; p. 2, development-kit paragraph; p. 5, Section 3.1
More object classes, difficult occlusions/truncation, loop-closing SLAM and further tasks are future extensions. MATLAB/C++ kits and an online evaluation server are described; stereo parameters are deferred to the website. This PDF supplies no software release versions, complete runtime/compute configuration or dataset license.
Go to primary source ↓e17PDF p. 3, Table 1, Proposed Dataset rows; p. 1, Abstract; p. 7, Section 3.4
Table 1 lists two object categories with 80,000 labels per category on average; the abstract describes over 200k annotations across the recorded data, while the evaluated object subset has 40,000 objects in 12,000 images. The evaluated subset is explicitly smaller than the full collection; the source does not fully reconcile all reported annotation totals.
Go to primary source ↓8.5 Primary sources
Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite ↗
PDF · 6,436 extracted words
Source fingerprint
e52e29804edbca87f3cb2577c56466898f4d90f71bbda56ce9e11def02129692