DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
1. Paper overview
In one sentence: Language specifies traffic, trajectory-conditioned diffusion supplies roads, and concatenated camera views support coherent video synthesis, with a tradeoff between visual conditioning and appearance diversity. e2e3e4e5e7e8e9e10e11
| At a glance | What to know |
|---|---|
| Research problem | Source description Customized driving-video generation usually depends on existing maps, boxes or real images, restricting scenario variation and user interaction. The paper seeks a language interface that creates moving agents and compatible roads, then renders temporally coherent surround-view video. e2 |
| Core mechanism | |
| A key reported result | Multi-view video generation from initial images: 11.2; 55.7 FID ↓; FVD ↓. nuScenes validation; first-frame multi-view image condition. DriveDreamer: 14.9/340.8 with initial images; Panacea: 16.9/139.0 with generated initial images. Best tabulated scores in this group. Generated-condition daggers prevent treating all baselines as identical-input comparisons. e6e7 |
| Reading caution |
Core contributions
Figure 2. Motion and road structure are built before the video is rendered. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the black arrows through the left panel first. The user description and function library enter the LLM; the diagram then places a Python script before the BEV trajectory. That trajectory branches toward the HDMap generator and camera projection, so the road generator receives foreground motion rather than choosing a road independently. The road map also reaches camera projection. In the right panel, projected multi-view structure passes through an encoder into a condition embedding. A separate appearance prompt passes through CLIP. Both branches feed the video generator. Pages 3–5 supply the library, map-training and latent-conditioning details behind these compact blocks. e2e3e4e5e12
What it supports. The method exposes two distinct controls: traffic structure determines where agents and roads appear, while language also supplies visual context such as rain. Reader interpretation: this decomposition can vary road and appearance around specified motion, but it also makes compatibility between separately produced intermediate representations an important failure point.
Where the evidence stops. Figure 2 shows a script intermediate, whereas Figure 3 and page 4 describe direct trajectory-array output at inference. The exact runtime bridge is unspecified. The camera symbol represents projection of structure; it is not evidence of an executed vehicle-control loop.
2. Motivation
2.1 The problem and the proposed response
Customized driving-video generation usually depends on existing maps, boxes or real images, restricting scenario variation and user interaction. The paper seeks a language interface that creates moving agents and compatible roads, then renders temporally coherent surround-view video. e2
2.2 What this reading follows
A request for a car cutting in on a rainy day leaves two problems: deciding how traffic moves and deciding what the cameras should see. DriveDreamer-2 separates them. A language model and trajectory library specify motion; a diffusion model proposes compatible roads; a video model renders the resulting structure across six cameras. Its central architectural choice is to concatenate those camera views before video generation. This reading follows that information flow, then separates three kinds of evidence: video-distribution quality, the contribution of UniMVM, and the usefulness of synthetic videos for training perception systems. e2e3e4e5e7e8e9e10e11
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
Foundational-work categories fit a neural video simulator and synthetic-data-generation method. Architecture, prediction paradigm and quadrant are appropriately not applicable to a world/action-policy taxonomy: UniMVM unifies camera views, not future prediction with executed-action prediction. This multi-stage pipeline does not establish a One Model controller. e2e5e8e9e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build traffic before rendering its appearance
Begin with the user request as a specification of motion and scene appearance. The trajectory library supplies vehicle, pedestrian and utility operations; manually written language/script pairs adapt GPT-3.5 to that vocabulary. In the cut-in example, another vehicle’s maneuver is constructed before the ego trajectory that responds to it. The resulting BEV trajectories then condition a separate HDMap diffusion model. Its training pairs teach associations between foreground motion and lane boundaries, dividers and crossings. Only after this stage are map elements and category-sized boxes projected into camera views for rendering. Reader interpretation: this ordering reduces the burden on the video model to invent traffic geometry from language alone. It does not guarantee that every requested maneuver has a physically or legally valid matching road; Figure 4 provides examples rather than constraint-satisfaction statistics. e2e3e4e5
Figure 5. Concatenating views moves cross-camera reasoning into a shared spatial representation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each vertical column from input to generated output. The two left columns retain separate camera images and explicitly insert a cross-view module before the cross-frame module. The three right columns place camera views side by side and omit that separate cross-view block. Their inputs distinguish a supplied initial frame, a supplied front-view video and no supplied images. Noise-textured regions illustrate portions to generate. The caption explicitly suppresses structural conditions and text prompts, so the last column is not unconditional synthesis. Page 5 gives the actual concatenation order, FL, F, FR, BR, B, BL, and the mask-based factorization in Eq. (3). e5e10
What it supports. UniMVM provides one representation for future prediction, view outpainting and generation without image inputs. It retains temporal modeling while removing dedicated cross-view parameters. The drawing establishes the architectural distinction; the within-SVD rows in Table 4, rather than the diagram alone, provide evidence that this choice improves the reported generation scores.
Where the evidence stops. The drawing and prose agree on which visual inputs are supplied. Eq. (3) conditions on x′·m, while the prose describes regions as masked without defining bit polarity. An implementation must resolve that convention; the figure supplies neither a mask legend nor tensor-level code.
5.2 Use one wide video representation for three conditioning tasks
The key UniMVM operation is spatial concatenation. A sequence with T frames, K camera views and per-view height H and width W becomes a video tensor of shape T × 3 × H × KW. The source fixes the camera order as FL, F, FR, BR, B, BL. Encoded structural conditions align with this wide image layout, and the SVD backbone is fine-tuned for video denoising. Supplying the first multi-view frame supports future prediction; supplying front-view video supports outpainting into other views; supplying no images still leaves text and structure as conditions. Reader interpretation: cross-camera relationships can now be processed inside the shared spatial representation rather than by an additional cross-view module. Eq. (3) expresses the conditional partition, but its mask-bit convention needs clarification before implementing these tasks. e5e6
5.3 Ask what each experiment actually tests
The experiments answer related but different questions. Table 1 tests similarity of generated image/video distributions to nuScenes validation data, with different amounts and origins of visual conditioning. Table 4 asks whether UniMVM improves generation over cross-view attention while retaining SVD, which is more informative about its mechanism than comparing only against the older backbone. Tables 2 and 3 then ask whether adding generated videos helps a perception model trained with real data. A useful tension appears here: removing initial-image conditioning worsens generation-distribution scores, yet its augmentation row has higher mAP, NDS and AMOTA. Reader interpretation: fidelity under stronger visual constraints and usefulness as additional training data need not rank configurations identically. Missing data-volume controls and uncertainty estimates prevent attributing that difference conclusively to diversity. e7e8e9e10e11e12
5.4 Training and inference
During training
Training uses nuScenes: 700 training and 150 validation videos, approximately 20 seconds each, six cameras at 12 Hz and roughly one million training frames. Annotations are prepared at 12 Hz, in BEV for map training and image coordinates for video training. e6
The SD2.1 map generator has trainable ControlNet parameters: 55K iterations, batch 24, 512×512. SVD fine-tunes all parameters by denoising score matching: 200K iterations, batch 1, eight frames, six views, 256×448. Hardware is A800 80GB GPUs; optimizer is AdamW at 5×10^-5. A complete frozen-module inventory is absent. e5e6
During inference
A template expands user prompts for trajectory generation; trajectories condition road generation, and projected structure conditions video denoising. Mask selection enables future prediction from initial multi-view images, outpainting from front-view video, or generation without images. Structural and text conditions remain present in the last regime. e2e3e5
This is an offline scene-generation pipeline. No feedback loop observing executed vehicle actions, inverse-dynamics action extraction or control-policy deployment is demonstrated. e2e5e12
5.5 Implementation flow
- Specify foreground motion
GPT-3.5 is fine-tuned on manual text-to-script pairs using 18 vehicle, pedestrian and utility functions. A cut-in example creates another vehicle trajectory, an ego forward trajectory and saved arrays. Figure 2 shows an intermediate script; Figure 3 and inference prose describe direct trajectory-array output, leaving the runtime bridge unclear. e2e3
- Construct a compatible road
A diffusion HDMap generator conditions on a colored BEV trajectory map. Convolutional features inject motion information; output channels represent lane boundaries, dividers and pedestrian crossings. Multiple maps can accompany one trajectory, but illustrated compatibility is not a measured guarantee. e4
- Project and concatenate
Trajectories determine category-sized 3D boxes, projected into camera images. Views follow FL, F, FR, BR, B, BL order. Three encoders produce map, box and image latents; aligned map/box features concatenate with noisy image latents. Text passes through CLIP in Figure 2. e2e5
6. Experiments & results
DriveDreamer-2 turns language requests into traffic trajectories, trajectory-conditioned road maps and six-view driving videos. UniMVM concatenates camera views so a video diffusion backbone processes them together. Experiments support improved video-distribution scores and synthetic training data for perception; they do not establish closed-loop driving performance.
6.1 Read the original evidence
Table 1. The strongest reported video scores use initial multi-view images. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Conditions column, not the bold numbers. Horizontal rules divide no-image, video-conditioned and initial-image groups. The dagger footnote is essential: several baselines first generate their visual conditions. In the video group, Drive-WM uses three generated views whereas DriveDreamer-2 uses one view, so the inputs are not matched. Both FID and FVD point downward, but they assess image and video distributions respectively. DriveDreamer-2 has lower FVD than Drive-WM here while its FID is higher. Finally, compare the two DriveDreamer-2 endpoints to see how supplying initial images changes the scores. Training and evaluation details span pages 5–6. e5e6e7e11e12
What it supports. With initial multi-view images, DriveDreamer-2 reports FID 11.2 and FVD 55.7. Without image conditioning it reports 25.0 and 105.1, versus DriveDreamer’s 26.8 and 353.2. These results support stronger distributional generation quality, while showing that the headline numbers describe a particular conditioning regime rather than every language-driven generation setting.
Where the evidence stops. Tables 1 and 5 give front-view scores of 18.4/74.9; page 7 prose gives 17.2/94.6. This edition retains the table values and discloses the conflict. Neither metric establishes prompt compliance, physical validity or closed-loop driving success.
Table 2. Synthetic training videos improve detection, with different conditioning choices favoring different metrics. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three columns as properties of the training-data comparison. Every row retains real data; the second and third rows add generated videos. The Initial frame check marks how those videos were generated, not whether the detector receives only an initial frame during evaluation. Page 6 identifies the downstream model as ResNet-50 StreamPETR trained at 256×448. Follow the metric arrows: mAP and NDS increase, while mAOE and mAVE decrease. The final row wins three columns, but the middle row has the lowest mAVE. Keep the printed metric scales because this table does not supply unit-conversion instructions. e6e8e12
What it supports. Adding synthetic videos generated without images raises mAP from 31.7 to 32.9 and NDS from 43.5 to 45.4. Reader arithmetic gives increases of 1.2 and 1.9 reported points, or approximately 3.8% and 4.4% relative. Initial-frame-conditioned augmentation instead has the lowest mAVE, 29.7 versus 30.4 in the final row.
Where the evidence stops. The paper does not report synthetic-data counts, mixing ratios or a matched extra-real-data training control. It also does not explicitly restate the downstream evaluation split. The table therefore supports augmentation utility but does not isolate diversity as its cause.
Table 3. Tracking improves with augmentation, but the preferred conditioning depends on the metric. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the Real and Generated columns to identify the baseline and augmented training sets. Both augmented rows include real data; their difference is whether synthesis receives an initial frame. AMOTA is read upward, while AMOTP and IDS are read downward. The lower row is best for the first two metrics, but the middle row is best for identity switches. This prevents compressing the table into a claim that removing image conditions wins everywhere. The evaluation paragraph links these comparisons to StreamPETR, and the augmentation paragraph says generated videos come from the nuScenes training set. e6e9e12
What it supports. AMOTA rises from 28.9 to 31.3 with synthetic videos generated without images: a 2.4-point increase, approximately 8.3% relative by reader arithmetic. AMOTP improves from 1.419 to 1.387. Identity switches fall from 687 to 593, although initial-frame-conditioned synthesis lowers them further to 542. The tradeoff remains visible across metrics.
Where the evidence stops. The small AMOTA difference between the two augmented rows has no accompanying uncertainty estimate. These aggregate tracking metrics cannot establish that rare cut-ins are handled better specifically, and they do not measure the safety of a vehicle acting on those tracks.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multi-view video generation from initial images nuScenes validation; first-frame multi-view image condition. | 11.2; 55.7 FID ↓; FVD ↓ | DriveDreamer: 14.9/340.8 with initial images; Panacea: 16.9/139.0 with generated initial images. Best tabulated scores in this group. Generated-condition daggers prevent treating all baselines as identical-input comparisons. e6e7 |
| Video generation without image conditioning nuScenes validation; structural/text conditions remain. | 25.0; 105.1 FID ↓; FVD ↓ | DriveDreamer: 26.8; 353.2. Both improve in the no-image group; headline 11.2/55.7 does not apply here. e5e7 |
| Front-view video outpainting nuScenes validation; one-view video versus three generated views for Drive-WM. | 18.4; 74.9 (Tables 1 and 5) FID ↓; FVD ↓ | Drive-WM: 15.8; 122.7. Lower FVD but higher FID with different inputs. Page 7 prose instead states 17.2/94.6; this conflict remains unresolved. e7e11 |
| Synthetic-data augmentation for 3D detection ResNet-50 StreamPETR at 256×448; real plus synthetic nuScenes-training videos. Downstream evaluation split is not explicitly restated. | 32.9; 45.4 using synthetic videos generated without images. mAP ↑; NDS ↑ | Real only: 31.7/43.5; initial-frame-conditioned augmentation: 32.6/45.2. Reader arithmetic: +1.2 mAP and +1.9 NDS on the reported scale, approximately 3.8% and 4.4% relative. These are perception gains. e6e8 |
| Synthetic-data augmentation for multi-object tracking Same StreamPETR augmentation study; downstream evaluation split not explicitly restated. | 31.3; 1.387; 593 without image-conditioned synthesis. AMOTA ↑; AMOTP ↓; IDS ↓ | Real only: 28.9/1.419/687; initial-frame-conditioned augmentation: 31.2/1.396/542. AMOTA improves by 2.4 reported points, approximately 8.3% relative. Initial-frame conditioning gives fewer identity switches. e6e9 |
| Backbone and UniMVM ablation Table 4; scores align with initial-image results, although the table has no condition column. | SVD plus UniMVM: 11.2; 55.7. FID ↓; FVD ↓ | SVD plus cross-view attention: 17.2/94.6; original DriveDreamer: 14.9/340.8. Within-SVD results support the representation change. Endpoint comparisons also include a backbone change; the first replacement worsens FID despite improving FVD. e7e10 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Separate the video-backbone gain from the contribution of concatenating camera views. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first transition and the second transition separately. The prose identifies the original DriveDreamer backbone as SD1.4 and the DriveDreamer-2 backbone as SVD. Moving between those method families while retaining cross-view attention changes both the backbone and the overall method. The cleaner comparison is between the two DriveDreamer-2 rows: cross-view attention is replaced by UniMVM while SVD remains the backbone. The check marks show that these are alternative treatments, not two modules enabled together. Lower FID and FVD favor the UniMVM row. Figure 7, also inspected on page 6, adds qualitative foreground/background examples, not an independent numerical consistency metric. e5e7e10e12
What it supports. SVD with cross-view attention reports FID/FVD 17.2/94.6; SVD with UniMVM reports 11.2/55.7. This within-backbone comparison supports the proposed representation. The original DriveDreamer scores are 14.9/340.8, so the preceding backbone change greatly lowers FVD while worsening FID. Attributing the full endpoint improvement solely to UniMVM would combine these effects.
Where the evidence stops. Table 4 omits an explicit condition column; its scores match the initial-image setting elsewhere. The prose calls the first FID change a decrease, although 14.9 to 17.2 is an increase. No uncertainty or matched resource accounting is supplied for these comparisons.
7. Analysis & limitations
7.1 What the evidence leaves open
FID/FVD and selected examples do not measure physical validity, language compliance or closed-loop safety. Figure 8 illustrates appearance diversity without quantifying its distribution. No uncertainty intervals or dedicated failure analysis accompany the results. e11e12
Front-view scores conflict between prose and tables; mask polarity and the script-to-array interface need clarification. Preserve these ambiguities rather than silently filling gaps from implementation conventions. e3e5e11
7.2 Questions for discussion
- Would the within-SVD benefit persist under matched compute and explicit cross-camera consistency measurements?
- How much downstream improvement comes from additional data quantity versus novel traffic content?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs nuScenes with 12 Hz annotation preparation, curated language/script pairs and function library, SD2.1/ControlNet, SVD, camera projections and StreamPETR. Missing details include corpus size, synthetic-data quantity/mixing, diffusion sampler settings, GPU count and wall time. e3e5e6e12
Reader-proposed checks: fix SVD and training budget when comparing cross-view attention against concatenation; compare synthetic augmentation against equal-budget real-data repetition. Measure generation and downstream metrics separately. Neither check was run. e8e9e10e12
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate concatenation with a fixed SVD backbone
Reader-proposed experiment, not performed: initialize both arms from the same SVD checkpoint and train on the same nuScenes clips, structural conditions and initial-image inputs. Compare separate views with cross-view attention against the reported FL/F/FR/BR/B/BL concatenation. Match updates, effective batch, eight-frame clips and 256×448 per-view resolution; record parameter count, memory and elapsed time rather than assuming equal compute. Evaluate FID/FVD on identical validation samples and seeds. Add a blinded audit of object-appearance disagreements across overlapping camera views, anchored by the projected boxes. If UniMVM fails to improve the distribution scores or disagreement rate across repeated runs, the claimed mechanism needs qualification even if the original endpoint comparison is reproduced. Resolve mask polarity before either arm is run. e5e6e7e10e12
Check 2: Separate useful synthetic variation from extra training exposure
Reader-proposed experiment, not performed: train the same ResNet-50 StreamPETR under four arms: the real-only baseline, repeated real data for an equal added training budget, real plus initial-frame-conditioned synthetic clips, and real plus synthetic clips generated without images. Match added clip count and optimization steps between the latter three arms, and keep synthetic source scenes disjoint from a fixed nuScenes validation set. Report the chosen mixing ratio, all Table 2/3 metrics and variation across seeds. A synthetic advantage over both real-only controls would support added content value; disappearance of the gain under matched exposure would weaken a diversity explanation. Preserve the mAVE and identity-switch tradeoffs rather than selecting only the best aggregate metric. e6e8e9e11e12
8.3 Reading coverage
Visual audit: Visually inspected the title, all seven authors and four affiliations on p. 1; Figure 2 pipeline on p. 2; trajectory-library and HDMap examples on p. 3; Figure 5 and Eq. (1)–(2) on p. 4; Eq. (3), video-conditioning details, dataset/training text, Figure 6 and Tables 1–2 on p. 5; hardware/training/evaluation details, Figure 7 and Tables 3–4 on p. 6; Figure 8, Table 5 and the conflicting front-view prose on p. 7. All six final crops were individually viewed and are readable. Table 1 retains the generated-condition footnote. Narrow Tables 2–4 were cropped from original 400-DPI renders; no crops were upscaled or recreated. Reference-only pp. 8–9 were read in the complete text pass, not visually inspected. No appendix or separate supplement was supplied. No code or experiments were inspected or run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title/authors, Abstract and Introduction
- PDF pp. 2–3: Introduction and DriveDreamer-2 overview
- PDF pp. 3–4: Customized Traffic Simulation, LLM fine-tuning and HDMap Generation
- PDF pp. 4–5: UniMVM formulation and Video Generation, Eqs. (1)–(3)
- PDF pp. 5–6: Experiment Details, training, evaluation and generation results
- PDF pp. 6–7: downstream results, Ablation Study, Discussion and Conclusion
- PDF pp. 8–9: References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting PDF pages 1–7, including all figures and tables, and all six final crops. Reference-only pages 8–9 were read as text.
- Identity/version scope: title and seven-author order match the catalog. This is the AAAI-25 proceedings article, printed pages 10412–10420, copyright 2025. No revision identifier/date is stated; no other edition was supplied for comparison.
- No appendix is present in this nine-page PDF. No separate supplement was supplied.
- Code and project-site contents were not inspected; no training, inference or reproduction experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e1PDF p. 1 (10412), title, author block, conference header and copyright
The proceedings article bears the catalog title and seven authors in the same order, identifies AAAI-25 and copyright 2025. No revision identifier/date is stated.
Go to primary source ↓e2PDF p. 2 (10413), Figure 2 and Introduction; p. 3 (10414), DriveDreamer-2 overview
The pipeline generates trajectories from language, HDMaps conditioned on trajectories, projected camera-view structural conditions and multi-view video. It aims to reduce reliance on pre-existing structured scenes or real imagery.
Go to primary source ↓e3PDF p. 3 (10414), Figure 3 and Finetuning LLM for Trajectory Generation; p. 4 (10415), continuation above HDMap Generation
An 18-function library covers vehicle, pedestrian and utility operations. Manual text-to-Python-script pairs fine-tune GPT-3.5. The cut-in example produces another vehicle trajectory, an ego forward trajectory and saved arrays. Inference uses a prompt template and is described as directly outputting trajectory arrays.
Go to primary source ↓e4PDF p. 3 (10414), Figure 4 and caption; p. 4 (10415), HDMap Generation and Eq. (1)
Conditional diffusion maps BEV trajectories to three-channel maps for lane boundaries, dividers and crossings. Convolutional trajectory features condition diffusion; training predicts added Gaussian noise. Figure 4 illustrates multiple maps for one trajectory without a quantitative constraint-satisfaction test.
Go to primary source ↓e5PDF p. 4 (10415), Figure 5, caption and Eq. (2); p. 5 (10416), UniMVM formulation, Eq. (3) and Video Generation
Views are concatenated in FL, F, FR, BR, B, BL order into a spatial patch without separate cross-view parameters. Masks enable initial-frame prediction, front-video outpainting or generation without images. Encoded projected maps and category-sized boxes concatenate with noisy image latents. Figure 5 omits structural/text conditions. Eq. (3) conditions on the region multiplied by m; the prose does not explicitly define mask-bit polarity.
Go to primary source ↓e6PDF p. 5 (10416), Experiment Details: Dataset and Training; p. 6 (10417), Training continuation and Evaluation
nuScenes supplies 700 training and 150 validation videos, roughly 20 seconds each, six cameras at 12 Hz and approximately one million training frames. Annotations are prepared at 12 Hz. SD2.1 with trainable ControlNet parameters uses 55K iterations, batch 24, 512×512. All SVD parameters are fine-tuned for 200K iterations, batch 1, eight frames, six views, 256×448. Hardware is A800 80GB GPUs; optimizer is AdamW at 5×10^-5. Evaluation uses FID/FVD and ResNet-50 StreamPETR at 256×448.
Go to primary source ↓e7PDF p. 5 (10416), Table 1, all rows and dagger footnote; p. 6 (10417), Quality Evaluation of Generated Videos
On nuScenes validation, DriveDreamer-2 reports FID/FVD 25.0/105.1 without images, 18.4/74.9 with one-view video and 11.2/55.7 with initial multi-view images. A dagger denotes generated conditions. DriveDreamer scores 26.8/353.2 without images and 14.9/340.8 with initial images; Drive-WM scores 15.8/122.7 with three generated views; Panacea scores 16.9/139.0 with generated initial images.
Go to primary source ↓e8PDF p. 5 (10416), Table 2, all rows; p. 6 (10417), Evaluation and downstream augmentation paragraph
Real-only StreamPETR detection scores are mAP 31.7, mAOE 67.9, mAVE 33.0, NDS 43.5. Adding initial-frame-conditioned synthetic data gives 32.6/61.7/29.7/45.2; adding synthetic data generated without images gives 32.9/61.5/30.4/45.4. Synthetic videos derive from nuScenes training. The downstream evaluation split is not explicitly restated.
Go to primary source ↓e9PDF p. 6 (10417), Table 3, all rows; Evaluation and downstream augmentation paragraph
Tracking AMOTA/AMOTP/IDS are 28.9/1.419/687 for real-only training, 31.2/1.396/542 with initial-frame-conditioned synthetic data and 31.3/1.387/593 with synthetic data generated without images. The last row has the best AMOTA/AMOTP but not the fewest identity switches.
Go to primary source ↓e10PDF p. 6 (10417), Table 4, Figure 7 and Ablation Study; p. 7 (10418), Ablation Study continuation
DriveDreamer gives 14.9 FID/340.8 FVD. DriveDreamer-2 with SVD and cross-view attention gives 17.2/94.6; SVD with UniMVM gives 11.2/55.7. Figure 7 qualitatively compares cross-view consistency. The prose calls the initial FID change a decrease although its tabulated score increases.
Go to primary source ↓e11PDF p. 7 (10418), Table 5, Figure 8 and Ablation Study, right-column continuation; compare p. 5 Table 1
Tables 1 and 5 agree on 18.4 FID/74.9 FVD for front-view conditioning. Page 7 prose instead gives 17.2/94.6, an unresolved inconsistency. Figure 8 illustrates increased appearance variation as image conditions are removed; no separate diversity score is reported.
Go to primary source ↓e12PDF pp. 5–7 (10416–10418), Experiment Details, Tables 1–5, User-Customized Driving Video Generation and Discussion and Conclusion; p. 3, LLM fine-tuning
Experiments concern generated videos and perception augmentation, without a closed-loop control benchmark, numerical physical-validity or prompt-compliance tests, confidence intervals or dedicated failure analysis. Details omit GPU count, wall time, diffusion sampling settings, synthetic-data quantity/mixing ratio and LLM fine-tuning corpus size.
Go to primary source ↓8.5 Primary sources
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation ↗
PDF · 6,546 extracted words
Source fingerprint
849ae8e110aa3fe1b23757a2a0ff726fbe0e3440e9ef524d70eca1e32573d8cf