DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers
1. Paper overview
In one sentence: A shared causal transformer predicts interleaved visual and relative-motion tokens, improving the reported video and non-reactive planning benchmarks while leaving the causal value of joint training unresolved. e-identitye-tokenizatione-sequencee-planninge-copye-curatione-position
| At a glance | What to know |
|---|---|
| Research problem | Source description The paper seeks a shared sequence model for an observation-space driving transition model and a driving policy. The authors argue that predominantly visual diffusion world models make this integration difficult. Their solution learns image and motion continuations together; differentiable model-based planning is a motivation and future direction, rather than a demonstrated reward-search procedure. e-probleme-sequence |
| Core mechanism | Source description A driving vocabulary combines compressed images with three separately quantized relative-motion components. Frame-wise interleaving allows one next-token objective to cover both world modeling and planning. e-tokenizatione-sequence |
| A key reported result | NAVSIM end-to-end planning: PDMS 82.4; NC 98.9, DAC 90.7, TTC 94.9, comfort 95.6, ego progress 79.7. PDMS and subscores, percentages; higher is better. navmini; two-second history, four-second non-reactive simulation; front camera, no additional ego-status input. ResNet-50 + MLP: PDMS 77.8, comfort 96.3. Constant velocity: PDMS 24.2. The PDMS gain is 4.6 percentage points over the visual baseline, with slightly worse comfort. This is a non-reactive benchmark result, not physical deployment. e-evaluatione-planning |
| Reading caution | Reader analysis Front-only sensing omits surrounding cameras, LiDAR and IMUs. Non-reactive evaluation and qualitative examples leave reactive interaction, physical execution and safety under distribution shift unestablished. e-tokenizatione-evaluatione-qualitative |
Core contributions
- Source description
A driving vocabulary combines compressed images with three separately quantized relative-motion components. Frame-wise interleaving allows one next-token objective to cover both world modeling and planning. e-tokenizatione-sequence
- Source description
The study reports video-generation comparisons, non-reactive planning scores, and diagnostics of motion copying, data curation, tokenizers and action position embeddings. e-videoe-planninge-tokenizerse-copye-curatione-position
Figure 2. One predictor connects two tokenization routes and two output decoding routes. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: each image becomes a grid of blue tokens through the VQ-VAE encoder. Below, a frame-to-frame pose increment becomes an orange triplet through component-wise binning. Follow the arrows into the shared input column, where image blocks and action triplets alternate. The central DrivingGPT block predicts the next token in that ordering. On the right, image-token groups go to the VQ-VAE decoder and motion-token groups to unbinning. Section 3.2 supplies the causal mask and conditional distributions that the schematic does not draw: an action block sees its current image, whereas an image block sees preceding actions. e-tokenizatione-sequencee-integratione-refinement
What it supports. The architectural evidence for joint prediction is the shared autoregressive stream. The visual and action branches are alternative token types handled by the same predictor, with different output decoding operations. Decoded pose increments are subsequently composed into a trajectory using the transformation matrices in Equation (2).
Where the evidence stops. The steering/pedal icons do not redefine actions: the equations specify translations and yaw. This schematic omits the supplementary SVD refinement decoder and shows no reward-guided search, executed control loop or measured driving latency.
2. Motivation
2.1 The problem and the proposed response
The paper seeks a shared sequence model for an observation-space driving transition model and a driving policy. The authors argue that predominantly visual diffusion world models make this integration difficult. Their solution learns image and motion continuations together; differentiable model-based planning is a motivation and future direction, rather than a demonstrated reward-search procedure. e-probleme-sequence
2.2 What this reading follows
DrivingGPT asks whether predicting a driving scene and predicting the ego vehicle's motion can use the same sequence model. Front-camera images become visual tokens, and local translations and yaw become three action tokens per frame. A single transformer continues this interleaved stream, after which separate decoding operations recover images and trajectories. This reading follows the verified December 2024 arXiv v1, including its embedded supplement; the catalog's ICCV 2025 edition was not compared. The central distinction is between jointly predicting useful futures and demonstrating that imagined futures improve driving. The paper establishes the former architecture and reports promising benchmark comparisons, while leaving the latter causal question open. e-identitye-tokenizatione-sequencee-planninge-copye-curatione-position
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
Figure 2 and the two conditional distributions support One Model × Joint prediction: one causal transformer generates future visual and action tokens in one stream. This is not an inverse-dynamics-only action extractor. Tokenizers and supplementary SVD refinement remain separate components; the classification describes the shared prediction core, not an entirely single-network rendering pipeline. e-sequencee-refinement
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the action vocabulary causal before learning it
Begin with the labels, because an autoregressive model can exploit information accidentally placed in its prefix. The paper argues that attaching a whole future trajectory to every observation creates overlap: an earlier action label can already contain later targets. DrivingGPT instead assigns a single frame-to-frame relative pose increment to each image. Longitudinal translation, lateral translation and yaw are clamped separately to their first and 99th percentiles and quantized into separate 128-entry vocabularies. The resulting triplet follows that frame's visual tokens. At inference, unbinning recovers local motion and Equation (2) composes the increments into a trajectory. Reader interpretation: the label design removes the particular overlapping-horizon shortcut described by the authors, but does not establish that every remaining historical-motion cue is harmless or that quantization preserves rare maneuvers. e-tokenizatione-sequencee-traininge-integration
5.2 Separate the shared predictor from the image decoder
Follow one cycle through the token stream. Given earlier images and actions, the transformer predicts the next image block. Once that block is present, it can predict the corresponding action triplet, which becomes context for the following image. Equation (1) trains this shared sequence with ordinary next-token cross entropy, and Figure 2 routes outputs to modality-specific decoders. This architecture supports the catalog's One Model and Joint prediction labels for the prediction core. The supplement adds an important qualification: independently decoded frames can be temporally inconsistent, so a separately trained SVD decoder refines predicted visual tokens into higher-resolution video. Reader interpretation: a better-looking refinement is not automatically a better action plan. The paper supplies qualitative refined videos but no numerical comparison showing that refinement improves planning, nor a reward-search algorithm using the refined imagery. e-sequencee-refinemente-problem
5.3 Ask each experiment a narrower question
The evaluation has three distinct roles. Table 1 tests generated-video distributions on 512 navtest videos; Table 2 uses a dedicated long-clip selection to test degradation with horizon. Table 3 evaluates planned trajectories on navmini through four-second non-reactive simulation. Their scores therefore answer different questions and should remain separate. Within planning, Table 5 rejects specific history-copying substitutes, while Table 7 shows a large association between action position embeddings and PDMS. Table 6 favors curated NAVSIM data, but also changes sequence length and sampling rate relative to uniform nuPlan training. Reader interpretation: this collection supports the feasibility of the unified representation. Establishing that joint visual prediction itself causes the planning gain would require a matched training comparison beyond the copying and position-embedding interventions that the paper reports. e-evaluatione-videoe-long-videoe-planninge-copye-positione-curatione-training
5.4 Training and inference
During training
Train next-token cross entropy for 100k iterations, batch 16, with AdamW: learning rate 10^-4, weight decay 0.05, momenta 0.9/0.95, gradient norm clipping 1.0 and token dropout 0.1. Horizontal image flips also transform waypoints/yaws. nuPlan clips contain 16 frames at 10 Hz; NAVSIM clips contain four history and eight future frames at 2 Hz. e-training
Supplement A separately fine-tunes an SVD video decoder on tokens from input images and conditions it on predicted tokens at inference. VQ-VAE features are upsampled, reduced to four channels and concatenated with Gaussian noise for the denoising UNet, producing 576 × 1024 videos. e-refinement
During inference
Image sampling uses temperature 1.0 and top-k 2000; modality-incompatible logits are masked. NAVSIM planning conditions on two seconds of observations/actions and generates a four-second trajectory. The paper excludes an additional ego-status input but retains historical relative actions. e-inferencee-planning
Section 4.1 says long videos produce 16 frames conditioned on eight previous frames. Supplement A instead specifies eight new frames from eight preceding frames, then SVD decoding in four 16-frame blocks. These descriptions are preserved without assuming identical schedules. e-inferencee-refinement
5.5 Implementation flow
- Compress observations
A pretrained VQ-VAE maps each image to a spatial token grid. Images are resized to 288 × 512; downsampling by 8 or 16 gives 2,304 or 576 tokens. Only the front camera is used. e-tokenizatione-training
- Represent local motion
Each action is the next frame's longitudinal translation, lateral translation and yaw relative to the current frame. Overlapping long-horizon action labels would expose future targets in the prefix. Components are clamped to their first/99th percentiles and uniformly quantized using separate vocabularies. e-tokenization
- Predict the interleaved stream
Image blocks precede their action triplets in z1, q1, z2, q2, … . A Llama-like causal transformer uses frame-wise 1D rotary embeddings for both modalities. Its shared vocabulary has 16,384 image entries plus 128 entries per action component, totaling 16,768. e-sequencee-training
- Decode a plan
Predicted image tokens enter the VQ-VAE decoder. Action tokens are unbinned into relative poses, converted to planar transformation matrices, and composed into an absolute trajectory. Figure 2's steering/pedal icons are illustrative: the defined outputs are pose increments, not actuator commands. e-tokenizatione-sequencee-integration
6. Experiments & results
DrivingGPT turns front-camera frames and relative ego motion into interleaved discrete tokens. A single causal transformer predicts both future imagery and motion, then integrates motion into trajectories. Its strongest planning comparison is on NAVSIM navmini, with separate navtest video-quality evaluations. The results support a unified prediction architecture, while leaving reactive driving, causal benefits of joint training and deployment efficiency unresolved.
6.1 Read the original evidence
Table 1. The navtrain-adapted comparison supports better generated-video distribution scores. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Fine-tune Dataset column before comparing scores. The first two rows use released SVD and CogvideoX models without the listed driving-data adaptation; the next SVD row is fine-tuned on navtrain. DrivingGPT is trained on navtrain and described in the accompanying text as trained from scratch. Both right-hand arrows point downward: lower FVD and FID are preferred. The most informative local comparison is therefore the bottom two rows, which both use the driving training split. Section 4.2 locates evaluation on 512 navtest videos, separate from the navmini planning evaluation and from the dedicated long-clip experiment in Table 2. e-evaluatione-videoe-long-videoe-refinement
What it supports. DrivingGPT reports FVD 142.61 and FID 12.78, compared with 227.54 and 24.03 for navtrain-fine-tuned SVD. Both distributional measures favor DrivingGPT in this evaluation. The off-the-shelf rows also show why training-data adaptation matters when interpreting the table's much larger gaps to released general video models.
Where the evidence stops. Training provenance and model families differ, so this is not a compute-matched architecture isolation. The table supplies no uncertainty or refinement-specific comparison, and FID/FVD cannot establish collision avoidance or faithful response to a particular action.
Table 2. Longer rollouts degrade, but DrivingGPT retains a substantial advantage over SVD. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First compare each method horizontally across 16, 32 and 64 frames. Both FID and FVD rise as the requested sequence grows. Then compare corresponding metric rows vertically; do not compare an FID row to an FVD row. The experiment selects 512 navtest source clips containing more than 64 frames. Its scores are consequently a separate result series from Table 1, even in the 16-frame column. Figure 3 on PDF page 5 provides the accompanying 32-second qualitative example, with SVD in panel (a) and DrivingGPT in panel (b). That montage illustrates the authors' repetition argument but does not replace this aggregate comparison. e-long-videoe-inferencee-refinement
What it supports. At 64 frames, DrivingGPT reaches FID 20.45 and FVD 506.95, compared with SVD's 46.45 and 1079.28. The supported finding is a relative advantage over longer horizons. DrivingGPT's own 16-frame scores, 15.04 and 278.11, show that extending generation still incurs deterioration.
Where the evidence stops. The main text describes 16-frame generation conditioned on eight frames; Supplement A describes eight new token frames from eight previous frames. The source does not reconcile those schedules or clearly assign SVD refinement to these table entries.
Table 3. Planning improves overall while retaining a visible comfort tradeoff. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the rightmost PDMS column, then examine the subscores that qualify the aggregate. NC means no at-fault collision, DAC drivable area compliance, TTC time-to-collision, Comf. comfort and EP ego progress. The caption reports these values as percentages, with upward arrows indicating the preferred direction. The relevant visual planner baseline is ResNet-50 with an MLP trajectory decoder. Section 4.4 says both it and DrivingGPT use front-camera imagery without an additional ego-status input. DrivingGPT still conditions on past actions. Section 4.2 specifies navmini evaluation after a four-second non-reactive simulation of the predicted trajectory, which bounds the driving claim. e-evaluatione-planninge-sequence
What it supports. DrivingGPT's 82.4 PDMS exceeds the visual baseline's 77.8 by 4.6 percentage points. NC increases from 92.6 to 98.9 and TTC from 86.2 to 94.9, while comfort falls from 96.3 to 95.6. The table therefore supports an overall gain without uniform improvement across every driving criterion.
Where the evidence stops. These are non-reactive benchmark percentages, not road-test success rates. The comparison changes the architecture and training formulation together; it cannot by itself attribute the gain specifically to generated future images or establish closed-loop physical driving capability.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM end-to-end planning navmini; two-second history, four-second non-reactive simulation; front camera, no additional ego-status input. | PDMS 82.4; NC 98.9, DAC 90.7, TTC 94.9, comfort 95.6, ego progress 79.7. PDMS and subscores, percentages; higher is better | ResNet-50 + MLP: PDMS 77.8, comfort 96.3. Constant velocity: PDMS 24.2. The PDMS gain is 4.6 percentage points over the visual baseline, with slightly worse comfort. This is a non-reactive benchmark result, not physical deployment. e-evaluatione-planning |
| Video-generation quality 512 navtest videos; Table 1; DrivingGPT trained on navtrain, SVD also compared after navtrain fine-tuning. | 142.61 / 12.78. FVD / FID; lower is better | Fine-tuned SVD: 227.54 / 24.03; off-the-shelf SVD: 483.76 / 27.80; CogvideoX: 848.87 / 31.78. Distributional visual quality improves over the listed models. Pretraining differs, and these metrics do not measure action execution. e-evaluatione-video |
| Long-video generation Separate 512-clip navtest selection with more than 64 frames per source clip; Table 2 evaluates 16/32/64-frame outputs. | At 16/32/64 frames: FID 15.04/16.30/20.45; FVD 278.11/454.28/506.95. FID / FVD; lower is better | SVD: FID 30.48/35.57/46.45; FVD 418.93/786.68/1079.28. DrivingGPT deteriorates with horizon but remains better in this comparison. These values belong to Table 2's long-clip protocol, not Table 1. e-long-video |
| Replacing predicted motion with history copying Planning diagnostic; replace one predicted component with its last historical value. | All predicted: 82.4; copy longitudinal x: 53.5; copy lateral y: 79.4; copy yaw: 73.1. PDMS, percentage; higher is better | Copying x produces the largest degradation, despite comfort increasing to 100.0. Predicted motion outperforms these copying controls. The intervention does not isolate visual conditioning or the world-model loss. e-copy |
| Training-data and position-embedding diagnostics Table 6 compares nuPlan uniform (651k sequences, 16 frames) with NAVSIM train (104k, 12 frames); Table 7 changes action position embeddings. | Data: 74.6 versus 82.4. Position embeddings: images only 65.3; images and actions 82.4. PDMS, percentage; higher is better | Smaller curated training data and action position embeddings are associated with better scores. Data curation is confounded with sampling rate, sequence length and distribution; Table 6 does not establish a universal quality-over-quantity law. e-traininge-curatione-position |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Predicting relative motion beats copying the last historical action component. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three left columns as intervention switches. The paper defines x as longitudinal translation, y as lateral translation and theta as yaw. A pred entry uses DrivingGPT's output; copy substitutes the value from the last historical frame. The first row retains all predictions, while each subsequent row replaces one component. The right-hand columns use the same percentage subscores as Table 3. Compare each replacement to the first row, especially PDMS, TTC and ego progress. The longitudinal-copy row is revealing: comfort reaches 100.0, yet the trajectory makes much less progress and has a substantially worse TTC score. e-copye-planning
What it supports. Copying longitudinal motion reduces PDMS from 82.4 to 53.5; copying lateral motion gives 79.4, and copying yaw gives 73.1. The model therefore provides value beyond these constant-component continuations. The authors explain the particularly large longitudinal penalty by NAVSIM scenes requiring acceleration from a stop.
Where the evidence stops. This intervention tests copying at the output level. It does not remove observations, isolate the visual-prediction loss, or prove general scene understanding. The stop-and-go explanation is the authors' interpretation rather than a separately reported stratified evaluation.
Figure 7. Sampling breadth changes generated texture and object shape in a qualitative example. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from T+4 through T+6 to T+8 to follow a generated continuation, then compare the same column vertically across K=1, 5, 25, 125, 625 and 2000. The scene offers two useful reference regions: cracks and surface texture on the road, and the nearby car's body shape. Supplement B describes lower k as producing smoother images and higher k as preserving more detail while sometimes distorting objects. Section 4.1 uses top-k 2000 and temperature 1.0 for image tokens. Keep the displayed T offsets as labeled; the figure does not explicitly attach time units to them. e-samplinge-inference
What it supports. The example makes the sampling choice visible as part of the generated-video result. The authors identify a smoothness-versus-detail tradeoff rather than reporting one setting that dominates every criterion. This motivates evaluating sampling alongside tokenizer and decoder choices when interpreting apparently sharper or more textured driving videos.
Where the evidence stops. This is a qualitative diagnostic from one displayed scene, with no aggregate quality curve or planning-score comparison. Visual texture and plausible car shape alone cannot certify correct dynamics, and the figure does not establish an optimal top-k value.
7. Analysis & limitations
7.1 What the evidence leaves open
Front-only sensing omits surrounding cameras, LiDAR and IMUs. Non-reactive evaluation and qualitative examples leave reactive interaction, physical execution and safety under distribution shift unestablished. e-tokenizatione-evaluatione-qualitative
The reported ablations do not compare matched joint versus action-only training. No repeated-run uncertainty is supplied. LlamaGen wins reconstruction FVD/FID, whereas Cosmos wins PSNR/SSIM, so tokenizer selection depends on the criterion. e-tokenizerse-copye-curatione-position
Supplement A acknowledges temporal inconsistency from independent frame decoding. Its SVD refinement adds another model, while Supplement B shows a qualitative smoothness/detail tradeoff with top-k; neither supplies a quantified planning benefit. e-refinemente-sampling
7.2 Questions for discussion
- Would matched action-only training preserve the planning gain attributed to the unified formulation?
- How much does visual-token sampling alter later predicted actions and accumulated pose error?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the matching nuPlan/NAVSIM splits, pretrained tokenizer and its downsampling choice, component percentile thresholds, token ordering, rotary indexing, modality masks and pose integration. Preserve navmini planning versus navtest visual evaluation and the separate long-clip sample. e-tokenizatione-sequencee-integratione-traininge-evaluatione-long-video
Missing specifications include numerical action thresholds, exact transformer scale, full frozen-module inventory, action sampling/unbinning convention, hardware, runtime, software versions and SVD training hyperparameters. Resolve the rollout wording and whether reported video tables use refinement before claiming exact replication. e-traininge-inferencee-videoe-long-videoe-refinement
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the visual prediction objective improves planning
Reader-proposed experiment, not performed: train matched models on the same navtrain sequences with identical tokenizer, relative-action bins, transformer size, rotary indexing, optimizer and training budget. Compare the full next-token loss with a variant masking loss on visual target tokens while retaining the same training stream. Evaluate both with the same causal rollout and modality masks on navmini, reporting PDMS and every subscore over multiple seeds. Also evaluate a history-only action predictor as a separately labeled baseline. If the full objective fails to improve planning over the matched loss-masked variant, the claim that visual prediction supervision causes the planning advantage would weaken; any difference must be interpreted alongside the variants' different visual-rollout quality. e-sequencee-traininge-inferencee-evaluatione-planninge-copy
Check 2: Separate sampling, rollout schedule and SVD refinement effects
Reader-proposed experiment, not performed: fix a DrivingGPT checkpoint, initial histories, action conditions, random seeds and a held-out 512-clip navtest sample longer than 64 frames. Generate with each Figure 7 top-k setting at temperature 1.0. For each resulting token sequence, compare VQ-VAE decoding with supplementary SVD refinement and report FID/FVD at 16, 32 and 64 frames, plus a predefined object-persistence diagnostic. Resolve the main-text versus supplementary rollout wording first, or report both schedules as separate configurations. If refinement improves texture metrics without improving object persistence, visual polish should not be interpreted as more faithful dynamics. If sampling rankings reverse across metrics, there is no single empirically dominant setting under this check. e-inferencee-long-videoe-qualitativee-refinemente-sampling
8.3 Reading coverage
Visual audit: Actually inspected the rendered title/author/version page; the problem and method pages, Figure 2 arrows and token branches, Equations (1)–(2), training and sampling details, all Figures 1–7 and Tables 1–7, the conclusion, and embedded supplementary A/B. PDF pages 13–15 correspond to supplementary printed pages 1–3. Pages 10–12 contain references and were read in the supplied text, not visually inspected. Every final crop was viewed, including the revised sampling crop. The cropped tables retain their complete headers, rows and numerical cells; definitions and evaluation qualifications from their captions are explained in the accompanying guides. The architecture's causal ordering was checked against Section 3.2; its control icons are explicitly distinguished from defined pose increments. The source's main-text/supplement rollout wording difference is disclosed. Supporting pages for retained method, numerical, evaluation and proposed-check claims are all included above. No external video, code or separate supplement was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Works, including 2.1–2.3
- 3. Driving as Next Token Prediction, including 3.1–3.2
- 4. Experiments, including 4.1–4.5
- 5. Conclusion
- References
- Supplementary A. Refining Video Generation with SVD Decoder
- Supplementary B. The Effect of Sampling Parameters on Video Generation
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/edition note: the inspected title and all three authors match the catalog. PDF p. 1 identifies arXiv:2412.18607v1, 24 December 2024. The catalog records ICCV 2025; this report reviews the supplied v1 artifact, not a verified ICCV camera-ready edition. Differences from that later edition were not supplied or compared (e-identity).
- All five supplied text chunks were read individually, covering all 15 PDF pages. References were read as supplied bibliography; the cited works were not independently inspected.
- Acquisition omission preserved: text extraction does not reconstruct figure images. The retained PDF was subsequently visually inspected for Figures 1–7, Tables 1–7, equations and supporting method/evaluation pages.
- Separate supplemental material availability has not been fully verified. The supplementary pages embedded in this PDF were read and inspected; no separate supplement was supplied.
- Code, project links and external resources were not inspected. No experiments were run or reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin stamp
Exact observed title matches the catalog. Authors are Yuntao Chen, Yuqi Wang and Zhaoxiang Zhang. The stamp is arXiv:2412.18607v1 [cs.CV], 24 Dec 2024. Three institutional affiliations appear below the author names.
Go to primary source ↓e-problemPDF p. 2, Section 1; p. 3, Section 3.1; pp. 8–9, Section 5
The authors motivate joint world modeling and planning, define observation-space dynamics and a policy, and describe differentiable model-based planning as future exploration in the conclusion.
Go to primary source ↓e-tokenizationPDF pp. 3–4, Section 3.2, Observation Tokenization and Action Tokenization
Front-camera images become VQ-VAE tokens. Relative translations/yaw avoid overlapping long-horizon labels leaking future actions. Each component is clamped to first/99th percentiles and uniformly binned with its own vocabulary.
Go to primary source ↓e-sequencePDF p. 4, Figure 2 and caption; Section 3.2, Unified Visual Action Sequence Modeling, Eq. (1) and following conditionals
A single causal transformer predicts interleaved image/action tokens with frame-wise rotary embeddings and cross entropy. Image tokens condition on earlier image/action blocks; current actions also condition on current image tokens. The figure routes images through VQ-VAE and motion through binning/unbinning.
Go to primary source ↓e-integrationPDF p. 4, Section 3.2, Integrating Action into Trajectory, Eq. (2) and subsequent product
Decoded longitudinal/lateral translations and yaw define a 3×3 planar transformation; consecutive relative transformations are multiplied into an absolute future pose.
Go to primary source ↓e-trainingPDF pp. 4–5, Section 4.1, Training
Input resolution is 288×512; spatial rates 8/16 yield 2304/576 image tokens. The vocabulary is 16384+3×128=16768. AdamW uses learning rate 10^-4, weight decay 0.05, momenta 0.9/0.95, clip norm 1.0, dropout 0.1, batch 16 and 100k iterations. nuPlan/NAVSIM clips are 16 frames at 10 Hz and 12 frames at 2 Hz respectively. Horizontal flips transform actions too. Exact hardware and transformer scale are not stated here.
Go to primary source ↓e-inferencePDF p. 5, Section 4.1, Inference
Image sampling uses temperature 1.0, top-k 2000 and masking of foreign-modality logits. The main-text long-generation description says 16 frames at a time conditioned on the previous eight.
Go to primary source ↓e-evaluationPDF pp. 5–6, Section 4.2, NAVSIM, Planning Metrics and World Modeling Metrics
The paper reports navtrain with 1192 scenarios, navtest with 136 and an independent navmini with 396. Planning is measured on navmini after four-second non-reactive simulation using NC, DAC, TTC, comfort and ego progress combined as PDMS. Visual evaluation selects 512 navtest videos for FVD/FID.
Go to primary source ↓e-videoPDF p. 6, Table 1, FVD/FID columns, and Section 4.3, Comparison of Generated Videos
DrivingGPT: 142.61/12.78; navtrain-fine-tuned SVD: 227.54/24.03; off-the-shelf SVD: 483.76/27.80; CogvideoX: 848.87/31.78. The text says DrivingGPT is trained from scratch and diffusion comparisons use released models, with SVD additionally fine-tuned on navtrain. No refinement-specific rows or uncertainty are given.
Go to primary source ↓e-long-videoPDF p. 5, Figure 3/caption; p. 6, Section 4.3, Long Video Generation and Table 2
The long-video evaluation selects 512 navtest clips longer than 64 frames. At 16/32/64 frames, DrivingGPT FID is 15.04/16.30/20.45 and FVD 278.11/454.28/506.95; SVD FID is 30.48/35.57/46.45 and FVD 418.93/786.68/1079.28. Figure 3 shows a 64-frame, 32-second example and its caption labels SVD as panel (a), DrivingGPT as (b).
Go to primary source ↓e-planningPDF p. 7, Table 3, all rows/columns, and Section 4.4
DrivingGPT NC/DAC/TTC/comfort/EP/PDMS: 98.9/90.7/94.9/95.6/79.7/82.4 percent. ResNet-50+MLP: 92.6/89.9/86.2/96.3/73.7/77.8. Constant velocity PDMS is 24.2. Planning conditions on two seconds of observations/actions for four seconds ahead; DrivingGPT and the visual baseline use front images without additional ego status.
Go to primary source ↓e-tokenizersPDF p. 7, Table 4 and Section 4.5, Comparison of Different Visual Tokenizers
Evaluation uses navtest, 512×288 images and spatial downsampling 16; the text states 12146 video samples. LlamaGen rFVD/rFID/PSNR/SSIM: 68.40/5.67/23.09/0.652; OpenMAGVIT2: 96.32/6.70/15.57/0.410; Cosmos: 178.50/27.12/25.05/0.695. The authors select LlamaGen.
Go to primary source ↓e-copyPDF pp. 7–8, Section 4.5, Does DrivingGPT Learn or Copy Planning Solutions?; p. 8, Table 5 and caption
Replacing one component with the last historical action gives PDMS 53.5 for x, 79.4 for y and 73.1 for yaw versus 82.4 for all predicted. Copy-x gives NC 75.9, TTC 63.1, comfort 100.0 and EP 48.2. The authors attribute its large drop to stop-and-go acceleration scenes. These are copying interventions, not action-only training comparisons.
Go to primary source ↓e-curationPDF p. 8, Table 6 and Section 4.5, Training Data Curation for Planning
Uniform nuPlan: 651k sequences, 16 frames/sequence, PDMS 74.6. NAVSIM train: 104k sequences, 12 frames/sequence, PDMS 82.4. The authors hypothesize that many ordinary driving trajectories contain little action change.
Go to primary source ↓e-positionPDF p. 8, Table 7 and Section 4.5, Position Embedding for Actions
Rotary position embeddings for image tokens only give PDMS 65.3; applying them to both image and action tokens gives 82.4.
Go to primary source ↓e-qualitativePDF p. 6, Mitigating Object Hallucination; p. 7, Figure 4/caption; p. 8, Figure 5/caption
Figure 4 contrasts object appearance/disappearance in SVD with a DrivingGPT example. Figure 5 displays unprotected left turn, curved turn, merging and an author-labeled better-path example; blue denotes generated and brown human trajectories. These are qualitative selections without execution evidence.
Go to primary source ↓e-refinementPDF p. 13 (supplement p. 1), Appendix A; p. 14 (supplement p. 2), Figure 6/caption
A separately fine-tuned SVD decoder addresses temporally inconsistent independent decoding. Training conditions on input-image tokens; inference uses predicted tokens. Eight new token frames are predicted from eight previous frames until 64 frames/32 seconds. Upsampled VQ-VAE features are reduced to four channels and concatenated with noise; SVD decodes four 16-frame blocks at 576×1024. Figure 6 provides four qualitative nuPlan examples without a numerical refinement comparison.
Go to primary source ↓e-samplingPDF p. 13 (supplement p. 1), Appendix B; p. 15 (supplement p. 3), Figure 7/caption
Figure 7 has columns T+4, T+6, T+8 and rows K=1,5,25,125,625,2000. The authors describe smaller k as smoother and larger k as more detailed but sometimes distorted. No quantitative sampling-quality or planning comparison is supplied.
Go to primary source ↓8.5 Primary sources
DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers ↗
PDF · 8,111 extracted words
Source fingerprint
d0d9b8e4d5d847c3f63680fe04afc8f18f84bff2398fa3bc18c972bd4530c027