DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
1. Paper overview
In one sentence: DriveDreamer forecasts structural scene features from actions, renders them with a shared diffusion backbone, and decodes future actions, trading strong conditional generation for dependence on structured inputs and incompletely specified evaluation details. e03e06e07e10e12e14e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Real driving videos have a large modeling space and strong geometric constraints. The paper asks whether a diffusion model can first learn those constraints from maps and boxes, then forecast future scenes and actions. The authors frame real-world training as a step beyond predominantly simulated world-model settings; their historical priority claim is not independently verified. e02 |
| Core mechanism | |
| A key reported result | Open-loop driving trajectory prediction: 0.29 m / 0.15%. Average L2 error (m) / average collision rate (%). nuScenes validation; future three seconds; ST-P3 evaluation settings. AD-MLP: 0.29 m / 0.19%; VAD: 0.37 m / 0.14%. Matches AD-MLP's error with 0.04 percentage points lower collision rate, approximately 21% relative. VAD has lower collision rate. Inputs differ across baselines; this is not a closed-loop safety result. e08e14 |
| Reading caution |
Core contributions
- Source description
ActionFormer predicts future structural features from an initial observation and supplied actions, enabling video forecasting without future annotated layouts. e06
Figure 2. One shared visual representation supports video rendering and a separate action decoder. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the inputs on the left before following the two output branches. Text takes the green CLIP route, and the reference image supplies a style embedding. Initial maps and boxes enter ActionFormer, whose predicted structural features condition Auto-DM. The lower action route splits: it conditions structural prediction and also reaches the action decoder. At the right, video decoding and action decoding are separate operations; the latter receives Auto-DM features as well as action history. The fire and snowflake markings agree with the training description: attention additions are trained while the original Stable Diffusion parameters remain frozen. The denoising arrow runs from noisy latent ZT toward Z₀. e03e04e05e06e07e08e09
What it supports. The diagram supports a shared-model interpretation because the action head uses the same Auto-DM representation that supports video generation. It also identifies an important distinction: actions are decoded from internal features and history. The architecture does not depict an inverse-dynamics model that converts completed generated frames into controls.
Where the evidence stops. The steering and pedal icons are schematic; Section 4.1 specifies yaw angle and velocity as inputs. Figure 2's horizon labels differ from Section 3.2's conventions, so they should not be treated as an exact temporal implementation contract.
2. Motivation
2.1 The problem and the proposed response
Real driving videos have a large modeling space and strong geometric constraints. The paper asks whether a diffusion model can first learn those constraints from maps and boxes, then forecast future scenes and actions. The authors frame real-world training as a step beyond predominantly simulated world-model settings; their historical priority claim is not independently verified. e02
2.2 What this reading follows
A driving world model must do more than produce convincing road images: it must connect what the vehicle does to what the scene becomes. DriveDreamer splits that problem into learning how traffic structure maps to appearance and learning how actions change the structural representation. Its diffusion backbone also supplies features to an action decoder. Read the architecture and recurrent update first, then distinguish three experimental questions: whether samples match the video distribution, whether synthetic images improve perception training, and whether predicted trajectories match recorded driving. These questions have separate metrics, and the paper's open-loop planning results leave actual driving performance untested. e03e06e07e10e12e14e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Q1 classification is supported at the integrated-system level: Auto-DM's shared UNet representation feeds video and action decoders, with ActionFormer providing recurrent structural conditioning. Joint prediction is architectural and appears in the factorized output objective. However, this is a modular network with separate heads, not a single diffusion process over video and action tokens; it is not inverse dynamics or evidence of closed-loop control. e03e06e07e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn the structural renderer before asking it to forecast
Start with the distinction between a road layout and its visual appearance. Auto-DM receives projected maps, projected boxes with category information, and scene text, but those conditions enter different parts of the model. Maps align spatially with image latents, whereas boxes become position tokens used by gated self-attention. Text influences appearance through cross-attention. The first training step uses a single image and omits temporal attention, concentrating supervision on the structural relationship. The next step adds temporal attention and sequential conditions so the same generator learns coherent video. Eq. (4) remains a Gaussian-noise prediction objective. Original Stable Diffusion v1.4 weights are frozen while the paper's named attention parameters are trained. This first stage teaches conditional rendering; it still receives the structural sequence rather than predicting all of it. e04e05e09
Figure 6. ActionFormer supplies future structural features when future annotated layouts are unavailable. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin in the blue Observing region: H₀ and B₀ become tokens, then self-attention and an MLP produce h₀. The reference image and text have separate upward paths to Auto-DM. Move right along the purple hidden-state arrows. At each step, encoded action Aₜ enters cross-attention; Eq. (5) specifies a Gaussian latent sₜ from this interaction. The upward sₜ arrow and horizontal hₜ arrow meet at the GRU, consistent with hₜ₊₁ = FGRU(hₜ, sₜ). Each new hidden state is concatenated with action features and decoded into the colored structural tokens that travel upward to Auto-DM. The repeated blocks show recurrent forecasting, not repeated access to future ground truth. e05e06e07
What it supports. The forecast lives at the structural-feature level. Auto-DM can therefore reuse its learned relationship between structure and appearance while ActionFormer handles action-conditioned evolution. This is the mechanism linking the first-stage conditional generator to second-stage prediction, rather than simply asking an image generator to extrapolate from pixels alone.
Where the evidence stops. The arrows agree with Eqs. (5)–(6), but the diagram does not establish calibrated uncertainty or long-horizon accuracy. The objective explicitly omits posterior–prior matching, and the supplied PDF does not quantify uncertainty calibration.
5.2 Replace unavailable future conditions with recurrent predictions
Video forecasting changes the information available at test time. An initial map and set of boxes can be supplied, but future layouts are the unknown quantities. ActionFormer compresses the initial structure into h₀ and then repeatedly combines a hidden state with the supplied action. Eq. (5) describes the resulting stochastic latent; Eq. (6) advances the hidden state through a GRU. The decoder converts each action-conditioned state into structural features for Auto-DM. Figure 6's arrows therefore implement an explicit bridge from actions to future rendering conditions. Second-stage training also predicts future actions through pooled Auto-DM features and action history. The paper models video and action likelihoods separately given shared conditions, and uses MSE and L1 objectives while omitting posterior–prior matching. Joint prediction here does not mean that actions and pixels share one diffusion variable. e03e06e07
5.3 Separate generation evidence from evidence about decisions
Reader interpretation: assess the result tables as different tests of the representation. Table 3 measures image and video distribution quality; its capability columns prevent treating every row as an identical-input baseline. Table 1 asks whether generated images improve a detector trained on real data, offering a downstream test of visual usefulness. Table 4 associates staged learning and recurrent structural prediction with better generation metrics, but does not match compute or isolate each recurrent component. Finally, Table 5 tests three-second trajectories against logged nuScenes data. DriveDreamer matches the action-only baseline's L2 error and reduces its collision metric, while VAD still has the lower collision rate. These findings motivate testing the shared features more carefully; they do not show that generated futures were searched to select actions or that a vehicle executed the predictions safely. e07e08e10e12e13e14
5.4 Training and inference
During training
First-stage step 1 uses single-frame supervision without temporal attention; step 2 adds temporal attention and sequential conditions. Both optimize denoising. Second-stage training replaces unavailable future conditions with ActionFormer predictions and combines video MSE with action L1 loss. e05e06e07
Training uses nuScenes: 700 training/150 validation videos, six cameras at 12 Hz, approximately one million training frames. The devkit supplies maps; supplemental 12 Hz boxes complement nuScenes' 2 Hz annotations. Scene descriptions supply text; ego yaw and velocity supply actions. e08
Stable Diffusion v1.4's original weights are frozen. The two first-stage steps run 40 epochs/batch 16 and 10 epochs/batch 1; step 2 uses 32 frames at 448 × 256. Second-stage training runs 10 epochs/batch 1, predicting 16 frames and 16 actions. AdamW uses 5 × 10⁻⁵ on A800 GPUs; GPU count is unstated. e09
During inference
For forecasting, start with the current image, structure, text, and an action sequence. Roll ActionFormer forward, condition Auto-DM's denoising on the resulting features, and decode video. Pool Auto-DM features with action history for the action output. Fully supplied sequential layouts instead support conditional video generation. e03e05e06e07
The demonstrated action output is evaluated against logged trajectories. No executed feedback loop or rollout-based action-search procedure is reported. Sampling-step count, guidance settings, action normalization, and conversion to the three-second trajectory protocol remain unresolved in the supplied body. e08e14e16
5.5 Implementation flow
- Give geometry explicit entry points
Project lane boundaries, lane dividers, pedestrian crossings, and eight-corner boxes into the image plane. Concatenate convolutionally encoded maps with noisy latents. Combine Fourier box coordinates with CLIP category features for gated self-attention; text enters cross-attention. Temporal attention connects frames, with neighboring-view attention in the multi-view extension. e04e05
- Forecast structure before rendering
Encode initial maps and boxes, aggregate them into hidden state h₀, and use action-conditioned cross-attention to parameterize a Gaussian latent. A GRU advances the hidden state. Decoding it together with action features supplies future structural features to Auto-DM; these are latent conditions rather than newly observed maps. e06
- Decode two kinds of prediction
Reference-image and text features accompany the predicted structure during video generation. The action branch pools multi-scale UNet features, concatenates historical action features, and uses MLP layers to predict future actions. It does not recover actions by applying inverse dynamics to generated video frames. e03e07
6. Experiments & results
DriveDreamer learns a driving-scene generator from structured real-world observations, then predicts the missing future structure from actions. Auto-DM renders that structure into video, while an action decoder combines its features with action history. The experiments support generation quality, synthetic-data usefulness, and open-loop trajectory prediction; they do not establish closed-loop driving safety.
6.1 Read the original evidence
Table 1. Generated training images improve both reported 3D detection baselines. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each detector as a separate pair of columns, with higher values preferred. The upper row is real-data training without synthetic additions; the lower row adds 4K images generated from nuScenes training conditions. For FCOS3D, mAP moves from 30.2 to 30.9 and NDS from 38.1 to 38.3. For BEVFusion, the corresponding changes are 32.8 to 35.8 and 37.6 to 39.5. Parentheses are absolute metric-point gains, not relative percentages. The accompanying text specifies 12 training epochs for the detection baselines. Table 2 on the same inspected page separately tests BDD100K lane detection; its data and metrics should not be pooled into this comparison. e10e11e16
What it supports. The clearest improvement here is BEVFusion's 3.0-point mAP gain, accompanied by a 1.9-point NDS gain. This provides downstream evidence that generated images contain useful training information beyond looking plausible. It tests augmentation with synthetic images, not the quality of action execution or a learned closed-loop policy.
Where the evidence stops. The table and surrounding paragraph do not explicitly name the detector evaluation split or report uncertainty. Supplementary training details were not supplied, and the experiment does not isolate whether extra training exposure alone explains part of the gain.
Table 3. Generation metrics improve, with different conditioning and view coverage across methods. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the capability columns: they determine what each row generates and what conditions it receives. The first three methods produce structured multi-view images; their FVD cells are dashes. DriveGAN has video and action checkmarks. Both DriveDreamer rows also have structure conditioning, while only the last has multi-view checked. The surrounding text identifies the preceding DriveDreamer row as front-view. Now read FID and FVD, both lower-is-better: the front-view model records 14.71 and 110.23; the multi-view model records 15.22 and 129.89. Evaluation uses nuScenes validation and images resized to 448 × 256. The adjacent ablation table has a different printed final FVD. e08e12e13e16
What it supports. Both DriveDreamer configurations report substantially lower FVD than DriveGAN's 390.82, and their FID values are below the image baselines listed here. This supports improved distributional generation quality under the paper's comparison. It does not by itself quantify traffic-law compliance, response accuracy under interventions, or driving-policy performance.
Where the evidence stops. Capability differences prevent an identical-input comparison. Preserve the original discrepancy: Table 3 gives front-view FVD 110.23, while Table 4 gives 110.32 for its final configuration. Neither value has a reported error bar.
Table 5. DriveDreamer matches the action-only baseline's trajectory error while reducing its reported collision rate. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the input columns before comparing scores. ST-P3 and UniAD use visual information, AD-MLP uses action information, and VAD and DriveDreamer use both. The last columns measure average trajectory L2 error in meters and average collision rate in percent, under the stated ST-P3 protocol on nuScenes validation over three future seconds. DriveDreamer's 0.29 m equals AD-MLP's value; its 0.15% collision rate is below AD-MLP's 0.19%. Relative to VAD, the tradeoff reverses between metrics: DriveDreamer has lower error than 0.37 m but higher collision rate than 0.14%. The bold values therefore do not identify one method as best on every metric. e07e08e14
What it supports. The authors' approximately 21% relative collision reduction is against AD-MLP: the absolute decrease is 0.04 percentage points. The result is compatible with visual features helping an action-history predictor avoid some logged collisions, while the unchanged L2 score shows that average trajectory error alone misses this difference.
Where the evidence stops. This is open-loop prediction against recorded data. It neither demonstrates physical collision reduction nor isolates visual-feature causality through a matched ablation of the same model. VAD has the lowest collision rate among these reported rows.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Driving video generation nuScenes validation; frame-wise FID and FVD, evaluated images resized to 448 × 256. | Front-view: 14.71 / 110.23; multi-view: 15.22 / 129.89. FID / FVD; lower is better | DriveGAN: 27.83 / 390.82; MagicDrive: FID 16.20, no FVD reported. Generation improves in the reported comparison, but methods differ in conditioning and view coverage. Table 4 separately prints FVD 110.32 for the full configuration. e08e12e13 |
| Training-pipeline ablation Table 4, generation evaluation accompanying the nuScenes comparison; row labels identify first-stage training and ActionFormer. | Both marked: 14.71 / 110.32. FID / FVD; lower is better | Neither marked: 15.77 / 142.07; first stage only: 15.10 / 120.92. The sequence supports the combined design; compute-matched controls and uncertainty are absent, limiting attribution to recurrence alone. e13 |
| Synthetic augmentation for 3D detection 4K generated images from nuScenes training conditions mixed with real data; detectors trained 12 epochs. Evaluation split not explicitly named. | FCOS3D: 30.9 / 38.3; BEVFusion: 35.8 / 39.5. mAP / NDS; higher is better | Without synthetic data: 30.2 / 38.1 and 32.8 / 37.6, respectively. BEVFusion gains 3.0 mAP points. This tests downstream training usefulness, not action execution. e10 |
| Synthetic augmentation for lane detection BDD100K pretraining; 10K generated training-condition images; 30-epoch detector training. Evaluation split not explicitly named. | TwinLiteNetV2-Small: 77.1 / 29.6; Large: 82.5 / 34.9. Lane IoU / drivable-area mIoU; higher is better | Without synthetic data: 75.9 / 28.7 and 81.7 / 34.2. Both sizes improve on both reported metrics; no uncertainty accompanies these point estimates. e11e16 |
| Open-loop driving trajectory prediction nuScenes validation; future three seconds; ST-P3 evaluation settings. | 0.29 m / 0.15%. Average L2 error (m) / average collision rate (%) | AD-MLP: 0.29 m / 0.19%; VAD: 0.37 m / 0.14%. Matches AD-MLP's error with 0.04 percentage points lower collision rate, approximately 21% relative. VAD has lower collision rate. Inputs differ across baselines; this is not a closed-loop safety result. e08e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Staged training and ActionFormer coincide with incremental generation gains. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The checkmarks identify the experimental configurations; an empty cell should not be read as an entirely untrained network. From top to bottom, neither named component is marked, only first-stage Auto-DM training is marked, and both first-stage training and second-stage ActionFormer are marked. Follow the metrics across these rows: FID decreases from 15.77 to 15.10 to 14.71, while FVD decreases from 142.07 to 120.92 to 110.32. The nearby paragraph interprets the last step as replacing a concatenation baseline with iterative structural prediction. Figure 3, inspected on page 6, explains why first-stage training itself contains an image step and a video step. e05e06e12e13
What it supports. The ordering is consistent with the proposed mechanism: learning structural conditioning first helps, and adding action-conditioned recurrent prediction helps further. The table supports the complete training design. It provides less specific evidence about which part of ActionFormer, such as stochastic sampling or the GRU, causes the incremental improvement.
Where the evidence stops. Compute, parameter count, and optimization exposure are not shown as matched controls, and no repeated-run uncertainty is given. The final FVD is faithfully retained as 110.32 despite Table 3's 110.23.
7. Analysis & limitations
7.1 What the evidence leaves open
The paper's claim of enhanced safety exceeds what logged, open-loop trajectory metrics establish. Selected weather, action, and multi-view examples also do not measure long-tail robustness or physical consistency under sustained intervention. e14e15
Eqs. (7)–(10) factor video and action likelihoods through shared conditioning; posterior–prior matching is explicitly omitted. The authors report similar plausible results, but provide no matching-term ablation or uncertainty-calibration test in this PDF. e07e13
Preserve source inconsistencies: full-model FVD is 110.23 in Table 3 and 110.32 in Table 4. Figure 2 and Section 3.2 use different horizon/index conventions; Section 4.1 prints the future action sequence using image symbol I₁₇:₃₂. Exact alignment needs clarification. e03e06e09e12e13
7.2 Questions for discussion
- Does action-conditioned recurrence improve measured future structure beyond a compute-matched concatenation baseline?
- Do visual features improve trajectory prediction when action history, capacity, and training data are controlled?
- How sensitive are generation and planning results to errors in the initial map and box conditions?
8. Reproducibility audit
8.1 Requirements and known gaps
Required inputs include Stable Diffusion v1.4, nuScenes imagery/calibration/maps, dense 12 Hz boxes, ego yaw/velocity, and scene text. Reproducing augmentation also requires the specified detectors and BDD100K. The supplied body gives hardware type and schedules but defers several evaluation and implementation details to a separate supplement. e08e09e10e11e16
Reader-proposed priorities are a compute-matched ActionFormer-versus-concatenation experiment and a controlled action-history-only versus shared-visual-feature action decoder comparison. Hold splits and evaluation code fixed, repeat seeds, and report uncertainty before attributing gains to future dynamics. e06e13e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does recurrence learn action-dependent structural evolution?
Reader-proposed check, not an executed experiment: initialize both variants from the same first-stage Auto-DM checkpoint and compare ActionFormer with the paper's concatenation baseline using identical nuScenes splits, training updates, resolution, and forecast horizon. Match parameter count where feasible and report any remaining compute difference. Fix the sampling settings and seeds, then evaluate FID/FVD and a predeclared distance between predicted structural features and features encoded from withheld future annotations. As a diagnostic, permute action sequences across otherwise matched clips and measure the change. A recurrent advantage on appearance metrics without better structural prediction, or insensitivity to action permutation, would weaken the proposed action-to-structure explanation. Repeat seeds and resolve which printed FVD value is reproducible. e05e06e08e09e12e13
Check 2: Do shared visual features add information beyond action history?
Reader-proposed check, not an executed experiment: train matched action decoders with the same histories, targets, split, optimizer budget, and three-second evaluation code. Compare history alone, history plus pooled Auto-DM features, and history plus visual features shuffled across scenes. Hold decoder capacity fixed with equal-sized input slots, and use matched random seeds. Report average L2 and collision rate with scene-level uncertainty intervals. If correctly aligned visual features consistently reduce collisions beyond both controls while shuffled features lose the benefit, the result would support a contribution from scene information. If the history-only or shuffled control matches the full model, the causal interpretation of Table 5's cross-model comparison would be weakened. All variants remain open-loop tests. e07e08e09e14
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, all body pages, Figures 1–6, Tables 1–5, and equation/training/evaluation pages. Cross-checked Figure 2's conditioning/output arrows and frozen/trainable markings against Sections 3.1 and 4.1, and Figure 6's action, latent, and GRU arrows against Eqs. (5)–(6). Inspected all six final 300-DPI crops, retaining complete legends and table headers while excluding captions and body prose. All supporting body pages for tutorial, numerical, training, and proposed-check claims are included. References on pages 15–18 were read as text. No appendix appears in the supplied PDF; separate supplementary figures, videos, and implementation details were not supplied. The FVD and temporal-index discrepancies remain explicit.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and Abstract (p. 1)
- 1 Introduction (pp. 1–3)
- 2 Related Work: 2.1–2.3 (pp. 3–4)
- 3 DriveDreamer: 3.1 First-stage Training and 3.2 Second-stage Training, Eqs. (1)–(10) (pp. 5–11)
- 4 Experiment: 4.1–4.3 (pp. 11–14)
- 5 Conclusion and Acknowledgements (p. 14)
- References (pp. 15–18)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read. The extraction's image limitation was addressed by inspecting PDF pages 1–14 and every final crop; references on pages 15–18 were read as text. No appendix is present in this PDF; the separate supplement repeatedly cited by the body was not supplied.
- The supplied ECVA/ECCV 2024 artifact is the reviewed edition. Its title and all six authors match the catalog; the catalog BibTeX has the wording 'real-world-drive' instead of the observed 'Real-world-driven'. No explicit revision identifier or revision chain was supplied; no other edition was compared.
- Code and external links were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title and author block
The observed title matches the catalog title. Credits are Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu; affiliations are GigaAI and Tsinghua University.
Go to primary source ↓e02PDF pp. 1–3, Abstract and Section 1
The authors motivate real-driving world modeling through structured traffic conditioning and a two-stage diffusion-based pipeline, with video generation, perception augmentation, and action prediction as applications.
Go to primary source ↓e03PDF p. 5, Figure 2 and Section 3
Initial image, HDMap, boxes, text, and actions enter an integrated architecture. ActionFormer supplies predicted structural features to Auto-DM; separate video and action decoders use its representation, with historical actions also entering the action branch.
Go to primary source ↓e04PDF pp. 6–7, Section 3.1, Figure 4 and Eqs. (1)–(3)
Map features are concatenated with noisy image latents. Fourier-embedded box coordinates and CLIP category features form position tokens for gated self-attention. Text uses cross-attention; temporal attention connects frames. The multi-view extension attends to neighboring views.
Go to primary source ↓e05PDF pp. 6–8, Figure 3, Section 3.1 training steps and Eq. (4)
First-stage step 1 learns from single-frame structural conditions without temporal attention. Step 2 adds temporal attention and video supervision. The denoising objective predicts injected Gaussian noise; the named trainable attention parameters are gated self-attention, temporal attention, and cross-attention.
Go to primary source ↓e06PDF pp. 9–10, Figure 6, Section 3.2 and Eqs. (5)–(6)
ActionFormer encodes initial maps and boxes into a hidden state, conditions a Gaussian latent on the hidden state and action through cross-attention, updates the hidden state with a GRU, and decodes action-conditioned future structural features.
Go to primary source ↓e07PDF pp. 10–11, Eqs. (7)–(10) and action-decoder paragraph
Video and action likelihoods factorize given shared conditioning. The authors omit posterior–prior matching from their simplified variational objective and use MSE and L1 losses. Pooled multi-scale Auto-DM UNet features are concatenated with historical action features and decoded by MLP layers.
Go to primary source ↓e08PDF p. 11, Section 4.1, Dataset and Evaluation
nuScenes supplies 700 training and 150 validation videos, approximately 20 seconds each, six cameras at 12 Hz, and approximately one million training frames. Maps come from the devkit; 2 Hz boxes are supplemented with 12 Hz annotations from reference [70]. Action inputs are ego yaw angle and velocity. Generation images are resized to 448 × 256; trajectory evaluation covers three future seconds.
Go to primary source ↓e09PDF p. 11, Section 4.1, Training
Stable Diffusion v1.4's original parameters are frozen. First-stage steps use 40 epochs/batch 16 and 10 epochs/batch 1, respectively, with 32-frame 448 × 256 videos in step 2. Second-stage training uses 10 epochs/batch 1 and predicts 16 frames and 16 future actions. A800 GPUs and AdamW at 5 × 10^-5 are reported; GPU count is absent. The action sequence is printed as I17:32.
Go to primary source ↓e10PDF p. 12, Table 1 and Section 4.2, 3D detection paragraph
With 4K synthetic images from nuScenes training conditions, FCOS3D mAP/NDS changes from 30.2/38.1 to 30.9/38.3; BEVFusion changes from 32.8/37.6 to 35.8/39.5. Detectors are trained for 12 epochs. The table and paragraph do not explicitly name the evaluation split.
Go to primary source ↓e11PDF p. 12, Table 2 and Section 4.2, lane detection paragraph
After DriveDreamer pretraining on BDD100K, 10K synthetic images from training conditions augment real images, with 30-epoch baseline training. TwinLiteNetV2-Small lane IoU/drivable mIoU changes from 75.9/28.7 to 77.1/29.6; Large changes from 81.7/34.2 to 82.5/34.9. The evaluation split is not explicitly named here.
Go to primary source ↓e12PDF p. 13, Table 3 and generation-comparison paragraph
On nuScenes validation, front-view DriveDreamer reports FID/FVD 14.71/110.23, multi-view 15.22/129.89, DriveGAN 27.83/390.82, and MagicDrive FID 16.20. Capability columns distinguish video, multi-view, structure, and action conditioning; image baselines have no FVD values.
Go to primary source ↓e13PDF p. 13, Table 4 and final ablation paragraph
Rows with neither stage marked, first-stage training marked, and both stages marked report FID/FVD 15.77/142.07, 15.10/120.92, and 14.71/110.32. The text compares ActionFormer with concatenation. The final FVD differs from Table 3's 110.23; uncertainty and matched compute are not reported.
Go to primary source ↓e14PDF p. 14, Table 5 and Section 4.3; p. 11, Evaluation
Three-second open-loop nuScenes validation planning under ST-P3 settings yields DriveDreamer average L2 0.29 m and collision 0.15%, versus AD-MLP 0.29 m/0.19% and VAD 0.37 m/0.14%. Table 5 explicitly distinguishes visual and action inputs. The text claims a 21% relative collision reduction against AD-MLP.
Go to primary source ↓e15PDF p. 2, Figure 1; p. 8, Figure 5 and captions
Qualitative examples illustrate text-controlled weather/time, action-conditioned future videos, predicted actions, and multi-view alignment against nuScenes validation conditions. These are selected visual examples rather than executed driving trials.
Go to primary source ↓e16PDF p. 7, multi-view extension; pp. 11–14, Sections 4.1–4.3
The body delegates multi-view video, metric, perception-training, and action-prediction implementation details to supplemental material. The supplied paper provides neither a closed-loop evaluation nor error bars for Tables 1–5.
Go to primary source ↓8.5 Primary sources
DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving ↗
PDF · 7,010 extracted words
Source fingerprint
a731c6a6805c90fa64970a550fadd44fe3dd79baaf5330082df3e82a6ea5b21a