TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models
1. Paper overview
In one sentence: TartanDrive shows how multimodal driving interactions improve short-horizon off-road pose prediction, while leaving the conversion of those gains into autonomous navigation unresolved. e-probleme-collectione-modele-difficultye-conclusion
| At a glance | What to know |
|---|---|
| Research problem | Source description A terrain label alone cannot determine traversability: the same mud or vegetation can affect vehicles differently depending on speed and approach. The authors therefore collect synchronized observations, actions and resulting motion to support dynamics learning beyond scene segmentation. e-problem |
| Core mechanism | Source description The release is described as roughly 200,000 interactions; Table I specifies 184,000 samples. It contains 630 human-operated trajectories, approximately five hours at 10 Hz, from a Pittsburgh test site. Seven sensing modalities are accompanied by action and intervention records. e-collectione-featurese-site |
| A key reported result | Exteroception on more uneven terrain: Original: 0.2176→0.1896, reported 13% improvement. Difficult: 0.7313→0.5394, reported 26% improvement. Reported prediction error and relative improvement from adding exteroception.. Original evaluation data versus a separately collected difficult set; median height-change difficulty 0.0866 versus 0.2253 m/s. Proprioception-only versus proprioception plus visual inputs; 87% of difficult-set trajectories exceed the original median difficulty. The relative benefit increases on the difficult collection, while absolute error rises for both models. Table IV does not label its loss; its original-set values match Table III's State column, an inference rather than an explicit protocol statement. e-difficultye-main-results |
| Reading caution | Reader analysis Human operation, intervention-based trajectory termination and one test site restrict coverage of risky behaviors and geographic variation. Trajectory splitting alone does not establish spatially independent evaluation; repeated routes may share visual context. e-collectione-sitee-protocol |
Core contributions
- Source description
The release is described as roughly 200,000 interactions; Table I specifies 184,000 samples. It contains 630 human-operated trajectories, approximately five hours at 10 Hz, from a Pittsburgh test site. Seven sensing modalities are accompanied by action and intervention records. e-collectione-featurese-site
- Source description
The benchmark crosses four input configurations with three objectives and compares against a kinematic bicycle model. A separate, more uneven evaluation collection tests the contribution of exteroception under harder dynamics. e-protocole-main-resultse-difficulty
Figure 2. One driving interaction connects the vehicle's surroundings with its measured response. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at a colored trajectory on the central satellite image, then follow its connector to the corresponding observation bundle. Each bundle shows a forward RGB view alongside height and color maps, plus IMU and suspension-shock traces. These are complementary descriptions of the same driving situation: maps summarize visible terrain geometry, while inertial and suspension measurements describe how the vehicle is responding. The caption states that trajectory colors come from clustering ResNet image features. They should therefore be read as visual groupings, rather than as a manually verified terrain-label system or a train/test partition. The full collection also includes pose, wheel RPM and control records. e-sitee-collectione-features
What it supports. The dataset's distinctive unit is an interaction with synchronized sensing, commands and measured motion. This figure makes the relationship concrete, but it is an illustrative sample of one site. The collection scale comes from the dataset text and Table I, rather than from counting colored paths in the map.
Where the evidence stops. The caption calls blue open road and yellow puddles, yet the blue-framed RGB example visibly contains a puddle. Preserve this mismatch. The colors are image-feature clusters; neither their terrain names nor geographic generalization are established by this montage.
2. Motivation
2.1 The problem and the proposed response
A terrain label alone cannot determine traversability: the same mud or vegetation can affect vehicles differently depending on speed and approach. The authors therefore collect synchronized observations, actions and resulting motion to support dynamics learning beyond scene segmentation. e-problem
2.2 What this reading follows
Driving through a puddle is an interaction between terrain, vehicle state and control. TartanDrive records those ingredients together: human steering and throttle, measured pose, camera-derived maps, and fast proprioceptive histories. Its baseline compresses observations into a shared latent state, advances that state under a supplied action sequence, and predicts future pose. The most useful comparison asks what vision adds once wheel motion and inertial sensing are already available. On a separately collected uneven-terrain set, the reported relative error reduction is larger than on the original evaluation data. Read the results as evidence about prediction, with navigation costs, planning and closed-loop performance still open. e-probleme-collectione-modele-difficultye-conclusion
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dataset classification fits the primary contribution: driving demonstrations with multisensor observations and motion targets. The architecture, prediction-paradigm and quadrant fields appropriately remain Not applicable to the dataset. Its benchmark is an action-conditioned forward model; joint multimodal training does not establish a unified future-and-action generator or inverse-dynamics controller. e-probleme-collectione-modele-conclusion
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Treat intervention boundaries as part of the data-generating process
Begin with how an example is produced. The driver sends throttle and steering commands through the vehicle's servo system, while the logger records pose, images and fast proprioceptive measurements. Braking is a separate human intervention that terminates a trajectory. This matters because the dataset contains the motion resulting from selected human actions, with dangerous continuations curtailed by intervention. Its observations combine immediate vehicle response with local terrain information: wheel RPM and IMU histories describe motion, while stereo-derived maps describe visible structure ahead. Training then pairs initial observations and a recorded action sequence with future measured pose. The paper states a trajectory-level train/evaluation split, but does not identify the trajectories or establish a spatially disjoint split; any reproduction must preserve that distinction. e-collectione-featurese-mape-protocol
Figure 5. Stereo geometry turns a forward view into local terrain inputs for dynamics learning. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two branches from left to right. Stereo matching supplies disparity, while stereo visual odometry supplies camera motion. Both enter point-cloud registration, together with the RGB image and camera intrinsics. The resulting local colored cloud is projected into a top-down plane, producing a height map and an RGB map. Section III.B identifies TartanVO for motion estimation and describes binning at 0.02 metres per pixel over a ten-metre square. These are observation-construction steps. They precede the modality encoders and latent recurrent predictor; the arrows do not depict future-state prediction or an action-selection policy. e-mape-featurese-main-resultse-architecture
What it supports. The Image + Maps baseline tests whether this representation helps beyond a forward image. Its state-loss RMSE is 0.3521 versus 0.5263 for Image alone. This supports the value of the combined mapping inputs under that benchmark, while the diagram explains where those inputs originate.
Where the evidence stops. Map construction and vision are changed together in this comparison. The source does not isolate height from RGB maps. It also describes two height channels, whereas the appendix map encoder lists {1,3} input channels; the conversion is not explained.
5.2 Follow the shared latent state from sensing to pose
The model compresses different sensor types using different encoders: dense networks for vectors, CNNs for images, and WaveNet-style temporal processing for histories. The main text combines their Gaussian latent predictions through a product of experts. Its stated motivation is that experts can constrain smaller regions of latent space and yield sharper aggregate predictions. The recurrent dynamics then receives an action embedding and advances the latent representation. State decoding recovers position and quaternion orientation, while separate observation decoders are available when an auxiliary loss needs them. At inference, the initial observations need be encoded only once; future pose can be decoded without repeatedly reconstructing images. The conceptual flow is clear, but the appendix leaves the exact initial-state injection and GRU input construction unresolved. e-modele-forwarde-architecturee-features
Appendix Table VIII. The benchmark predicts pose after conditioning recurrent dynamics on supplied actions. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward through the layer column. The two-dimensional throttle/steering input passes through two tanh layers, producing a sixteen-dimensional action embedding. The GRU row lists recurrent quantities of dimension 128, and the state decoder ends in a distribution over seven pose coordinates. Read this alongside Section IV.A: modality-specific encoders first form a shared latent representation, after which the recurrent model advances it under actions. Appendix Algorithm 1 explicitly separates action embedding, latent transition and state decoding. Its optional observation-decoding loop supports auxiliary reconstruction; producing an image is not required to obtain the state predictions used in the benchmark. e-modele-forwarde-architecturee-featurese-conclusion
What it supports. The architecture supports action-conditioned forward dynamics. Actions are inputs to the predictor, while pose is its output. That distinction explains why the dataset benchmark can inform later planning research without demonstrating a learned policy, inverse-dynamics controller or a joint future-and-action generator.
Where the evidence stops. The GRU input is printed as (128,23), although the action embedding has size 16. The source does not explain that construction. Algorithm 1 also leaves initial-state use and several indices unclear; do not silently supply missing connections.
5.3 Separate the value of a sensor from the value of predicting it
Reader analysis: a useful input need not be an equally useful reconstruction target. IMU and wheel measurements describe current motion, so their strong baseline is consistent with the dynamics task. Vision and maps add information about terrain that the vehicle may encounter, which is consistent with the larger relative gain on the uneven collection. Yet a model can use the visible scene without being able to predict everything it will see after rounding a corner. The authors use this partial-observability argument to explain why auxiliary sensory losses yield limited gains. Table III supports a narrower conclusion: the preferred objective changes with the input configuration, and the contrastive Time Series and All scores are nearly equal. A convincing reproduction should therefore cross sensor choice with objective choice and quantify uncertainty. e-main-resultse-difficultye-losse-conclusion
5.4 Training and inference
During training
State-only training maximizes the likelihood of GNSS position and orientation and does not train observation decoders. Reconstruction adds modality-weighted future-observation likelihoods. Contrastive training instead favors future latent codes under matching observations over other observations in the batch. e-loss
Appendix Table IX specifies Adam, learning rate 0.001, 5,000 epochs, 10 gradient steps per epoch, batch size 64, gradient-norm clipping at 100 and 20 training timesteps. Reconstruction scales are 100 for RGB image/map, 1 for heightmap, and 0.1 for IMU/wheel RPM; contrastive scale is 10 and EMA τ is 0.05. e-training
Map generation is described separately from dynamics training. The supplied text does not establish end-to-end joint training or a frozen-module schedule for the stereo and visual-odometry networks. e-mape-modele-training
During inference
The reported evaluation forward-simulates 20 steps from initial observations and given actions. It can decode only states, avoiding repeated high-dimensional observation decoding. There is no learned action-selection head, inverse-dynamics action extraction or evaluated replanning loop. e-modele-forwarde-protocole-conclusion
5.5 Implementation flow
- Record actions and feedback
A human commands throttle and steering through a joystick, ROS and position-controlled servos. Throttle ranges from 0 to 1; steering ranges from −1 to 1, with −1 hard left. Braking remains under direct human control and ends a trajectory. GNSS supplies pose targets; stereo vision, IMU and wheel sensors provide observations. e-collectione-features
- Construct observation inputs
Stereo matching and TartanVO motion estimates register colored point clouds, then project them into local height and RGB maps. Maps cover 10 by 10 metres at 0.02 m/pixel. Training uses 128×128 RGB images and 64×64 maps; wheel histories are upsampled from 50 to 200 Hz to align with IMU sampling. e-mape-features
- Fuse modalities into a latent state
Dense, convolutional and WaveNet-style encoders handle vectors, images and time series respectively. Each provides a diagonal Gaussian latent distribution; the main model combines experts multiplicatively. A GRU predicts latent transitions conditioned on actions, and decoders recover pose or observations. e-modele-architecture
- Roll out compact predictions
Encode initial observations once, embed the supplied actions, advance the latent dynamics and decode future states. Appendix Table VIII specifies two action layers, 2→16→16, and 128-dimensional recurrent/state-decoder features, ending in a distribution over seven pose coordinates. Its GRU input entry needs clarification before implementation. e-modele-forwarde-architecture
6. Experiments & results
TartanDrive supplies human-driven ATV interactions for learning how terrain, vehicle motion and commands jointly determine future pose. Its benchmark fuses visual and proprioceptive observations into an action-conditioned latent dynamics model. Additional modalities improve reported prediction error, particularly on uneven terrain; whether those improvements enable autonomous navigation remains untested.
6.1 Read the original evidence
Table IV. Visual sensing contributes more on the separately collected uneven-terrain evaluation set. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the first two error columns within each row before comparing the rows themselves. Prop. uses proprioception; Prop. + Ext. adds exteroception from images and maps. The Improvement column reports relative error reduction, not a navigation success rate. The difficult collection was gathered separately in more uneven terrain. Its median difficulty is 0.2253 metres of height change per second, compared with 0.0866 for the original data; this proxy captures steepness and unevenness. The original row matches Table III's State-loss values, but Table IV does not explicitly name its objective. That correspondence is a reader inference, not an additional documented training condition. e-difficultye-main-resultse-conclusion
What it supports. The reported relative gain grows from 13% on the original data to 26% on the difficult collection. At the same time, the full-model error rises from 0.1896 to 0.5394. Thus the finding is a larger contribution from visual inputs under harder conditions, alongside a substantial remaining prediction challenge.
Where the evidence stops. The difficult-set size and repeated-run uncertainty are not supplied. Height-change difficulty does not isolate slip, speed or visibility, and this table does not restate the rollout horizon. It establishes neither route-completion performance nor safe autonomous driving.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| 20-step off-road pose prediction Original dataset split by trajectories into training and evaluation sets; identical reported horizon across modality/loss comparisons. | All modalities: 0.1896 (State), 0.1674 (Reconstruction), 0.1958 (Contrastive). RMSE of mean state prediction; lower is better. Units and coordinate aggregation are unspecified. | Image: 0.5263/0.4740/0.4952; Image + Maps: 0.3521/0.3386/0.3741; Time Series: 0.2176/0.2285/0.1966, in the same loss order. KBM: 1.1638. All has the lowest reported error in each column. Reconstruction gives the lowest overall entry, but Time Series favors contrastive training; the latter nearly eliminates the All advantage. No repeated-run uncertainty is reported. e-protocole-main-results |
| Exteroception on more uneven terrain Original evaluation data versus a separately collected difficult set; median height-change difficulty 0.0866 versus 0.2253 m/s. | Original: 0.2176→0.1896, reported 13% improvement. Difficult: 0.7313→0.5394, reported 26% improvement. Reported prediction error and relative improvement from adding exteroception. | Proprioception-only versus proprioception plus visual inputs; 87% of difficult-set trajectories exceed the original median difficulty. The relative benefit increases on the difficult collection, while absolute error rises for both models. Table IV does not label its loss; its original-set values match Table III's State column, an inference rather than an explicit protocol statement. e-difficultye-main-results |
| Action-to-displacement variation diagnostic 10,000 one-second subsequences; 10 action clusters using Euclidean distance; five initial-velocity bins. | Action colors overlap, and similar action sequences can map to separated displacement regions. Qualitative t-SNE displacement structure colored by action cluster. | Figure 4 illustrates both different actions with nearby outcomes and similar actions with separated outcomes; appendix plots show all bins and centroids. Motivates richer observations but does not causally isolate terrain effects or supply a numerical prediction score. e-diagnostic |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table III. Sensor choice changes the reported errors more consistently than auxiliary-loss choice. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down a column to compare input configurations under one objective; read across a row to compare objectives for one configuration. State is pose-likelihood training, Reconstruction adds future sensory reconstruction, and Contrastive adds a latent matching objective. Image + Maps adds height and RGB maps; Time Series uses IMU, shocks and wheel RPM; All combines these visual and proprioceptive inputs. Bold entries identify the best modality set within each loss, and red entries identify the best loss within each modality set. KBM is a separate wheel-RPM-based kinematic baseline: its identical entries repeat one evaluation, not three independently optimized models. e-losse-protocole-main-results
What it supports. All has the lowest error in every column, with 0.1674 under Reconstruction the overall minimum. However, Time Series improves from 0.2285 under Reconstruction to 0.1966 under Contrastive, and then nearly matches All at 0.1958. The evidence supports a modality-dependent interaction with the training objective rather than a universal auxiliary-loss winner.
Where the evidence stops. These are errors after 20 forward-simulation steps on trajectory-held-out data. No confidence intervals or repeated-run counts appear, and the aggregation of position/quaternion errors is unspecified. The small contrastive gap cannot establish a statistically reliable advantage.
Figure 4. Similar commands do not uniquely determine the vehicle's observed displacement. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two sides as complementary examples. On the left, three differently colored action clusters point into a nearby region of the displacement embedding. On the right, three sequences from the same cluster point into separated regions. The central colors identify clusters of action sequences, not terrain categories. The procedure samples one-second subsequences, uses Euclidean action distance to preserve command timing, and bins by initial speed before embedding state displacement. Appendix Figure 2 shows all five velocity bins, while Figure 3 supplies the ten action-centroid plots with throttle and steering legends. Those appendix views complete the context missing from this compact illustration. e-diagnostice-main-resultse-difficulty
What it supports. The visual motivates conditioning dynamics on observations beyond the command sequence: action similarity and displacement similarity do not coincide cleanly. It provides a qualitative reason to test richer sensing. The quantitative support for improved prediction comes separately from the modality ablation and the difficult-terrain comparison.
Where the evidence stops. The embedding is descriptive, not a controlled terrain intervention. Initial-speed binning reduces one confound but cannot remove all state differences. This main figure lacks numerical axes and a trace legend; do not infer physical distances or exact command magnitudes from its layout.
7. Analysis & limitations
7.1 What the evidence leaves open
Human operation, intervention-based trajectory termination and one test site restrict coverage of risky behaviors and geographic variation. Trajectory splitting alone does not establish spatially independent evaluation; repeated routes may share visual context. e-collectione-sitee-protocol
The authors attribute limited auxiliary-loss gains to partial observability: future views around corners cannot be inferred reliably without prior mapping. Short training sequences also limit long-term mapping behavior. They explicitly leave navigation integration, costs and planning for future work. e-losse-conclusion
Split counts/identifiers, difficult-set size, seed variation and confidence intervals are absent. The aggregate state RMSE is not defined finely enough to interpret it as metres or compare it to another benchmark's pose metric. e-protocole-main-resultse-difficulty
Source inconsistencies remain unresolved: the dataset describes two height channels and six IMU channels, whereas appendix encoders list map inputs {1,3} and temporal inputs {4,9}. Algorithm 1's indices and use of the initial state are underspecified; Eq. (1)'s Gaussian dispersion convention is unclear. e-featurese-mape-modele-forwarde-architecture
7.2 Questions for discussion
- Would the visual advantage persist after matching initial speed and holding out complete spatial routes?
- Which pose-error components explain the difficult-terrain benefit, and would a planner benefit from those components?
8. Reproducibility audit
8.1 Requirements and known gaps
Start from synchronized trajectories, sensor frames, action conventions and map preprocessing. Appendix A reports a 3 m wheelbase and 1.57 m GPS height. The brake-intervention threshold, exact split, map/IMU channel reconciliation and state-error normalization must be resolved or declared as assumptions. e-collectione-featurese-mape-framese-protocole-architecture
The PDF links a dataset repository but does not state a data license, training hardware/runtime or software versions. Table IX omits a shock reconstruction weight and does not explain where its contrastive EMA is applied. These are specification gaps, not evidence that the linked implementation lacks them. e-probleme-traininge-loss
Proposed checks: compare All against Time Series with identical speed/terrain strata, and cross the three losses under a fixed update budget. Use trajectory-level uncertainty and explicit pose metrics; the illustrated edition provides controls and falsifiable outcomes. e-main-resultse-difficultye-training
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does visual context help beyond matching current motion?
Proposed, not performed: train Time Series and All with the same state objective, trajectory split, initialization schedule and update budget. Evaluate paired 20-step predictions on original and difficult terrain, stratifying by initial speed and the paper's height-change difficulty. Add an All-model control with visual observations shuffled within those strata, leaving proprioception and actions untouched. Report the original aggregate error only after clarifying its definition, plus separate translation and orientation errors and trajectory-bootstrap intervals. A visual advantage that persists within strata and disappears under shuffling would support useful scene information beyond current motion. An unchanged advantage under shuffling would challenge that explanation. Record any assumptions needed because the paper omits split IDs and difficult-set size. e-protocole-main-resultse-difficultye-traininge-diagnostic
Check 2: Does the objective ranking survive a controlled rerun?
Proposed, not performed: cross State, Reconstruction and Contrastive objectives with Time Series and All inputs on one documented trajectory split. Use the appendix's 20-step horizon and equal optimizer-update budget, with multiple seeds. Keep the shared encoder/GRU capacity fixed; document the additional decoders needed by reconstruction and resolve the missing shock-loss weight and unspecified EMA application before claiming an exact reproduction. Test whether reconstruction remains best for All while contrastive remains best for Time Series, and whether the 0.1966 versus 0.1958 contrastive gap exceeds run-to-run variation. If these rankings change within uncertainty, the evidence would favor a conditional, unstable loss preference. None of these checks would establish navigation success. e-losse-main-resultse-traininge-architecturee-conclusion
8.3 Reading coverage
Visual audit: All 12 pages were rendered and visually inspected after the complete four-chunk reading. This covers the title/byline and affiliations; main Figures 1–5 and Tables I–IV; the appendix vehicle frames, Algorithms 1–6, Tables I–X, all five t-SNE velocity bins and all ten action centroids. Every final crop was viewed; Appendix Table VIII was recropped and viewed again to retain its complete label. Figure 5's branch directions were checked against III.B.4; Figure 4's callouts were checked against its caption and appendix procedure. The Figure 2 color/caption mismatch and appendix input/index ambiguities are disclosed in the corresponding explanations. All pages supporting retained method, result and reproduction details are included. No external supplement, dataset files, linked code or separate edition was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, abstract and I. Introduction
- PDF pp. 2–3: contributions and II. Related Work
- PDF pp. 3–4: III. The Dataset, A–C
- PDF pp. 4–5: IV. Multi-modal Modeling, A–B, Eqs. (1)–(4)
- PDF pp. 5–6: V. Experiments and Analysis, A–B; VI. Conclusions and Future Work
- PDF p. 7: References
- PDF pp. 8–12: Appendix A–D, references, Algorithms 1–6, Tables I–X and Figures 1–3
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Pages with little extracted text require visual inspection: 9, 12
- The extraction-related visual cautions above were addressed by inspecting all 12 PDF pages, including pages 9 and 12, and all six final crops.
- Identity scope: arXiv:2205.01791v1, 3 May 2022. The exact title and six authors match the catalog apart from its expanded name Sebastian A. Scherer; the PDF byline reads Sebastian Scherer. No separate venue edition or later revision was inspected.
- The embedded appendix was fully read. No external supplemental material, linked repository, dataset files or prior HTML artifact was inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, affiliation footnotes and arXiv margin stamp
The observed title is TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models. The six authors are Samuel Triest, Matthew Sivaprakasam, Sean J. Wang, Wenshan Wang, Aaron M. Johnson and Sebastian Scherer. The stamp identifies 2205.01791v1, 3 May 2022; affiliations are at Carnegie Mellon University and the University of Pittsburgh.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract, I. Introduction and numbered contributions
The paper motivates interaction data through vehicle- and speed-dependent traversability, describes roughly 200,000 interactions with seven sensing modalities, provides a dataset repository link and proposes dynamics-learning benchmarks.
Go to primary source ↓e-sitePDF p. 2, Figure 2 and caption
A Pittsburgh test-site map overlays trajectories colored by clusters of ResNet RGB features; four examples pair RGB, heightmap, color map, IMU and shock observations. The caption calls blue open road and yellow puddles, while the blue-framed RGB example visibly contains a puddle; the color-to-terrain wording should not be treated as verified annotation.
Go to primary source ↓e-collectionPDF p. 3, III.A–B and Figure 3
The Yamaha Viking ATV uses joystick-controlled throttle/steering and direct human braking. ROS records sensors and commands. Collection yields 630 trajectories, approximately five hours at 10 Hz; braking terminates trajectories. Action ranges and sensor models are described.
Go to primary source ↓e-featuresPDF pp. 3–4, III.B.1–6; main Tables I–II
Table I lists 184,000 TartanDrive samples. Pose has position plus quaternion; IMU data are 20×6, shock/RPM data 5×4, upsampled to 20×4 for training. RGB images and maps are resized to 128×128 and 64×64. A brake threshold defines interventions, without a numerical threshold value.
Go to primary source ↓e-mapPDF p. 3, III.B.4; PDF p. 5, Figure 5 and caption
Stereo matching and TartanVO supply disparity and camera motion; colored point clouds are registered and projected into height/RGB maps. The heightmap is described with minimum and maximum height channels. The 500×500 maps cover forward 0–10 m and lateral −5–5 m at 0.02 m/pixel, updated at 10 Hz.
Go to primary source ↓e-diagnosticPDF p. 4, III.C and Figure 4; PDF p. 9, appendix Figure 2; PDF p. 11, Algorithm 6 and Table X; PDF p. 12, appendix Figure 3
The diagnostic clusters 10,000 length-10 action subsequences into 10 Euclidean clusters, bins initial velocity into five ranges, and embeds resulting state displacements with t-SNE. Figure 4 contrasts different-cluster nearby outcomes with same-cluster separated outcomes. Appendix plots show every velocity bin and action centroid; they are qualitative evidence.
Go to primary source ↓e-modelPDF pp. 4–5, IV and IV.A, including Eq. (1)
The task predicts future states from initial state/observations and actions. Modality encoders produce Gaussian latent distributions combined using a product of experts, a GRU advances latent dynamics, and decoders recover states or observations. Initial encoding suffices for rollout. The printed Gaussian aggregation does not clearly establish its dispersion convention.
Go to primary source ↓e-lossPDF p. 5, IV.B, Eqs. (2)–(4), and V.A; PDF p. 6, continuation of V.A
State loss fits future pose likelihoods. Reconstruction adds weighted modality likelihoods; contrastive loss compares codes with matching and other batch observations. The source attributes limited gains to real-world partial observability.
Go to primary source ↓e-protocolPDF p. 6, V.B, first four paragraphs and Table III caption
Data are split by trajectories into training and evaluation sets, without counts or identifiers. Four observation configurations are trained with three losses. The KBM uses average wheel RPM; the reported metric is RMSE of mean state prediction after 20 forward-simulation steps.
Go to primary source ↓e-main-resultsPDF p. 6, Table III, every modality row and State/Reconstruction/Contrastive columns; V.B
KBM is 1.1638 in each displayed column, copied rather than separately trained. Image scores are 0.5263/0.4740/0.4952; Image + Maps 0.3521/0.3386/0.3741; Time Series 0.2176/0.2285/0.1966; All 0.1896/0.1674/0.1958. Bold marks column minima, red row minima; no uncertainty is tabulated.
Go to primary source ↓e-difficultyPDF p. 6, Table IV and V.B difficult-dataset paragraphs
Original proprioceptive/full errors are 0.2176/0.1896 with 13% improvement; difficult-set errors are 0.7313/0.5394 with 26%. Median difficulty, average height change per second, is 0.0866 versus 0.2253 m/s; 87% of new trajectories exceed the original median. The table does not label a loss or report the new set's size.
Go to primary source ↓e-conclusionPDF p. 6, VI. Conclusions and Future Work
Navigation-stack integration remains future work; prediction improvements have not been shown sufficient for intelligent navigation. The authors call for cost/planning development and mapping-based memory, noting partial observability and training sequences too short for learning SLAM.
Go to primary source ↓e-framesPDF p. 8, appendix A, Figure 1 and Table I
NovAtel, Multisense and map frames are identified and illustrated. Table I gives wheelbase 3 m and GPS height 1.57 m. Exact relative quantities are said to be supplied in the dataset.
Go to primary source ↓e-forwardPDF p. 10, appendix Algorithms 1–5
Algorithm 1 encodes modalities, aggregates them, embeds actions and decodes states and selected observations. It mentions DeepSets or product-of-experts aggregation, although the main method specifies the latter. It lists initial state as input but does not explicitly use it, mixes action/observation time indices and leaves the recurrent z unindexed. Algorithm 5 begins with x despite listing latent input z.
Go to primary source ↓e-architecturePDF p. 8, appendix B; PDF p. 10, Tables II–III; PDF p. 11, Tables IV–VIII
The appendix gives image/map CNNs, WaveNet-style temporal encoding, interpolation/convolution decoding and a GRU model. Table VIII lists action layers 2→16→16, GRU input (128,23), output 128,128 and a seven-dimensional Gaussian state output. The source does not explain the 23-dimensional input construction. Map input {1,3} and temporal input {4,9} do not directly match main-text height/IMU dimensions.
Go to primary source ↓e-trainingPDF p. 11, appendix Table IX; PDF p. 8, appendix B
Adam uses learning rate 1e−3, 5,000 epochs, batch 64, 10 gradient steps per epoch, norm clip 100 and 20 train timesteps. Reconstruction scales: RGB image 100, RGB map 100, heightmap 1, IMU 0.1, wheel RPM 0.1; no shock scale is listed. Contrastive scale is 10 and EMA τ 0.05; application of that EMA, compute hardware/runtime and software versions are not specified.
Go to primary source ↓8.5 Primary sources
TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models ↗
PDF · 7,019 extracted words
Source fingerprint
36d96acc3b6ca56110c7d02d3c607a86d0778e8cf49e7f052f8c1ce004606835