DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning
1. Paper overview
In one sentence: Depth-supervised queries enrich a shared driving representation for video and trajectory prediction, while separate generators let planning proceed without rendering an imagined world. e02e04e08e14e15e18
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that appearance-based world prediction can miss geometry relevant to occlusion, free space and collision avoidance. Their goal is to make geometric supervision useful to both future-video generation and driving trajectories through a compact shared representation. e02 |
| Core mechanism | |
| A key reported result | Navsim v2 trajectory planning: 88.7 EPDMS ↑. Navsim v2 EPDMS evaluated on navtest, following the comparison protocol stated in Section 4.1. DriveVLA-W0 86.1; DiffusionDrive 84.5. A 2.6-score-point aggregate gain over DriveVLA-W0. Table 2 lists no competing world-model-based row and does not show dominance on every component. e11e13 |
| Reading caution |
Core contributions
Figure 2. Three generative experts share context through query embeddings. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the instruction, cameras and current action into the LLM. The green and orange world queries become conditioning embeddings for depth and video; the blue action queries condition the trajectory expert. At upper left, the dashed training-only box corrupts the depth target, which is concatenated with RGB. The middle branch combines VAE image latents with noisy future latents and uses CLIP features alongside world embeddings. The right branch starts from trajectory noise. The depth-to-video-to-action order is specified in the accompanying text as attention between query groups; the figure does not draw that internal mask or a generated-video-to-action arrow. e04e05e06e07e08e09
What it supports. The useful connection is through learned embeddings. Section 3.2.2 explicitly allows the action generator to run without producing depth or video, so planning need not wait for those renderings. Depth predicts the current scene, while video and trajectories describe the future; these outputs have different temporal roles.
Where the evidence stops. The snowflake/flame markings distinguish frozen and trainable parts, qualifying Section 3.3's broad wording about training all components. Exact freeze boundaries remain underspecified. The diagram also supplies no measured latency or controller interface.
2. Motivation
2.1 The problem and the proposed response
The authors argue that appearance-based world prediction can miss geometry relevant to occlusion, free space and collision avoidance. Their goal is to make geometric supervision useful to both future-video generation and driving trajectories through a compact shared representation. e02
2.2 What this reading follows
A driving model can predict plausible images without representing distances well enough to plan safely. DriveDreamer-Policy addresses that gap by learning current depth, future video and future trajectories together. Its key interface is a small collection of learned queries inside a multimodal language backbone. Depth context reaches video queries, and both world groups inform action queries. The diagrams and tables below explain how this information flow differs from explicitly feeding generated frames into a planner. The evidence favors joint supervision on Navsim, but the depth targets come from another model, the video metrics disagree in one comparison, and no measured latency frontier is supplied. e02e04e08e14e15e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
Every supplied catalog category is Not assigned, so there is no positive classification to confirm. Architecture evidence supports an integrated world–action system with a shared LLM and three separate generative experts, joint world/action learning and ordered query conditioning. Action prediction does not require inverse dynamics on a rendered future. A One Model judgment needs an explicit rule for this modular architecture; joint training alone cannot settle it. e04e05e08e09
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Let geometry shape the shared representation before generating pixels
At a decision step, the instruction, synchronized images and current action provide the observed context. DriveDreamer-Policy appends learned query groups that will become depth, video and action embeddings. The depth queries are trained to support reconstruction of current geometry through the depth expert. Video queries may attend to that depth context, and action queries may attend to both world groups. Thus the directed pathway is defined inside the LLM's representation computation. The three generators then read their own embeddings through cross-attention. This explains why a depth map can be useful as a training target even when it is not rendered for a planning request. The source explicitly makes the action generator independent of explicit depth/video generation, while retaining their learned contextual influence through the queries. e04e06e08
5.2 Separate shared flow training from each expert's output space
The flow formulation starts with a data sample at time zero and noise at time one. Training interpolates between them and regresses the velocity x₁−x₀; generation integrates in the reverse direction. The experts apply this continuous-generation principle to different representations. Depth is generated directly in pixels after log/percentile normalization, video is generated in VAE latent space, and trajectories encode position and heading with cosine and sine. DA3 supplies the depth supervision. The joint objective weights depth loss by 0.1 and the other losses by 1.0. Initializing the LLM, depth expert and video expert from existing backbones supplies substantial prior learning even though task training uses only Navsim. Figure 2's frozen-module markings also mean the one-stage description should not be interpreted as an explicit guarantee that every parameter is updated. e03e05e08e09e10
5.3 Read the results as evidence for supervision, then test the pathway
Reader analysis: the strongest mechanism evidence is the pair of internal comparisons. Table 4 improves planning when either world task is learned and improves it further when both are learned. Table 5 improves video generation when depth learning and depth-query conditioning are added. Together they make a plausible case that geometry is useful shared supervision. They do not yet isolate the directed attention pathway, because the depth-video comparison changes two ingredients at once. Table 6 introduces a second ambiguity by enlarging all query groups together. Cross-method results add useful context but answer different questions: the Navsim v2 aggregate improves, video PSNR falls relative to PWM, and depth is evaluated against the training teacher's outputs. A convincing reproduction should preserve these separate evaluation targets rather than compress them into one claim of better world understanding. e13e14e15e16e17e18
5.4 Training and inference
During training
Initialization uses Qwen3-VL-2B, PPD for depth, and Wan-2.1-T2V-1.3B adapted to image-to-video generation. Depth/video training uses 144×256 resolution and a nine-frame video horizon. DA3 supplies depth labels; log-depth and per-map percentiles normalize them to [−0.5, 0.5]. e09
One-stage optimization uses 100k steps, batch size 32, eight NVIDIA H20 GPUs and AdamW with learning rate 10⁻⁵. Defaults are 64 depth, 64 video and 8 action queries; the action encoder is a two-layer MLP with layer normalization. No additional training dataset beyond Navsim is claimed, apart from initialized backbones. e10
Figure 2 marks the vision encoder, tokenizer, CLIP and VAE with snowflakes, and the LLM, queries, action encoder and generators with flames. This qualifies Section 3.3's broad statement that all components are trained: a complete parameter-level freeze specification is absent. e05e09
During inference
Sample noise and integrate the learned flow backward from time 1 to 0. Planning activates the action expert independently; depth/video experts can also run for imagination or generation. World-query context remains available to action queries without explicitly generating those world outputs. e03e04e08
The source states that normalized depth is inverted to metric or relative depth as needed, but does not specify percentile values or how inference-time scale parameters are obtained. e09
5.5 Implementation flow
- Encode context and read out queries
Text, visual patches and encoded current action enter the LLM alongside depth, video and action queries. Video queries attend to depth context; action queries attend to both world groups. This ordering operates within the backbone in one pass. e04
- Generate current depth
A pixel-space diffusion transformer concatenates noisy depth with the corresponding RGB image. Depth-query embeddings provide cross-attention keys/values. The output represents current geometry; downstream queries use its embeddings, not a mandatory rendered depth map. e05e06
- Imagine future video
A VAE encodes the current image; noisy latents represent the future horizon. Video-query embeddings condition the video transformer through cross-attention, supplemented by concatenated CLIP image features. The VAE decodes predicted latents into frames. e05e07
- Predict motion
A separate diffusion transformer converts a noise trajectory into states \((x,y,\cos\theta,\sin\theta)\), where position and heading describe each trajectory state. Action embeddings supply context. The source describes trajectory prediction, without a detailed actuator controller or candidate-rollout selection loop. e08
6. Experiments & results
DriveDreamer-Policy trains a shared multimodal backbone with depth, video and trajectory generators. Ordered query embeddings transfer geometry and future-scene context into planning without requiring rendered depth or video at inference. Navsim scores and controlled modality ablations support useful joint supervision, while depth evaluation against a learned teacher and incomplete runtime details limit the conclusions.
6.1 Read the original evidence
Table 2. Navsim v2 improves the aggregate score while retaining component tradeoffs. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the rightmost EPDMS column, where higher is better. The highlighted row reaches 88.7, compared with 86.1 for DriveVLA-W0 and 84.5 for DiffusionDrive. Then read leftward to check whether the aggregate gain is uniform. DriveDreamer-Policy leads the displayed DDC, TLC and LK columns, but DriveVLA-W0 has higher NC, DAC and TTC. ARTEMIS has a higher EC value. Section 4.1 identifies this as Navsim v2 evaluation on navtest. The family headings organize the listed comparisons; the world-model-based block contains just the proposed model, rather than a separate head-to-head comparison within that family. e10e11e13
What it supports. The reported improvement over DriveVLA-W0 is 2.6 EPDMS score points. It is a benchmark aggregate gain with a mixed component profile. For example, the proposed model's EC is 79.4 versus ARTEMIS's 89.1, so the highest aggregate should not be read as winning every criterion.
Where the evidence stops. The comparison uses officially reported baseline results. The table lacks uncertainty estimates and does not establish physical deployment or uniform experimental control across methods; its empty competitor set within the world-model block limits within-family claims.
Table 3(a–b). Video quality has a metric tradeoff; depth quality measures agreement with DA3. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the two panels as different evaluations. Panel (a) compares front-camera videos against recorded future RGB frames, using lower-is-better LPIPS and FVD and higher-is-better PSNR. The front-only restriction is stated on page 9 because PWM supports single-view generation. Panel (b) compares depth against dense DA3 targets: the first PPD row is zero-shot, the next is fine-tuned on Navsim, and the highlighted row uses LLM-conditioned depth generation. Lower AbsRel and higher threshold accuracies indicate better agreement with those targets. The printed depth values are retained in the source's scale; neither the table header nor the metric paragraph specifies a numerical scaling factor. e09e14e15
What it supports. Against PWM, FVD falls from 85.95 to 53.59 and LPIPS from 0.23 to 0.20, while PSNR falls from 21.57 to 21.05. For depth, AbsRel is 8.1 versus 9.3 for fine-tuned PPD. The evidence therefore supports perceptual/video-distribution gains and closer teacher-depth agreement, rather than universal improvement.
Where the evidence stops. DA3 supplies both training labels and evaluation targets, so this panel is not an independent metric-depth calibration test. Exact depth threshold settings and uncertainty are unreported; generated-video quality also cannot stand in for executed planning safety.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Navsim v1 trajectory planning navtrain/navtest: reported 100k/12k samples at 2 Hz; v1 navtest, three-camera model. Baselines are their officially reported results. | 89.2 PDMS ↑ | AutoVLA 89.1; WoTE 88.3; PWM 88.1; human reference 94.8. Highest listed learned-model aggregate, only 0.1 score point above AutoVLA. Sensor configurations differ; the human reference remains higher. e11e12e10 |
| Navsim v2 trajectory planning Navsim v2 EPDMS evaluated on navtest, following the comparison protocol stated in Section 4.1. | 88.7 EPDMS ↑ | DriveVLA-W0 86.1; DiffusionDrive 84.5. A 2.6-score-point aggregate gain over DriveVLA-W0. Table 2 lists no competing world-model-based row and does not show dominance on every component. e11e13 |
| Future front-camera video generation Navsim recorded future RGB targets; only the front view is compared because PWM supports single-view generation. | 0.20 / 21.05 / 53.59 LPIPS ↓ / PSNR ↑ / FVD ↓ | PWM: 0.23 / 21.57 / 85.95. FVD improves by 32.36 and LPIPS improves, but PSNR decreases by 0.52. These generation metrics do not measure executed driving safety. e14 |
| Current-depth generation Navsim dense DA3 targets; comparison with zero-shot and Navsim-fine-tuned PPD. | 8.1 / 92.8 / 98.6 / 99.5 (as printed; scaling not specified) AbsRel ↓ / δ₁ ↑ / δ₂ ↑ / δ₃ ↑ | Fine-tuned PPD: 9.3 / 91.4 / 98.3 / 99.5; zero-shot PPD: 18.5 / 80.4 / 94.0 / 97.2. Better agreement with the teacher targets, including lower AbsRel than fine-tuned PPD. This does not independently validate metric depth against physical measurements. e15 |
| World supervision for planning Four modality variants under identical training budgets, Table 4. | Action-only 88.0; depth+action 88.5; video+action 88.9; full 89.2. PDMS ↑ | Full model gains 1.2 score points over action-only and 0.3 over video+action. Both supervision sources help in this comparison; no uncertainty estimates establish the reliability of the smaller increment. e16 |
| Depth supervision for video Matched data and compute; compare video-only against depth joint learning plus depth-to-video query conditioning. | With depth: 0.20 / 21.05 / 53.59 LPIPS ↓ / PSNR ↑ / FVD ↓ | Without depth: 0.22 / 19.89 / 65.82. All three metrics improve within this ablation, but the experiment changes both supervision and conditioning. e17 |
| Query-budget sensitivity Compare 32/32/4 depth/video/action queries with 64/64/8. | 64/64/8: 89.2 / 53.59 / 8.1 PDMS ↑ / FVD ↓ / depth AbsRel ↓ | 32/32/4: 88.9 / 57.97 / 9.7. Increasing all query groups together helps these metrics; per-group effects and measured latency costs remain unresolved. e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Joint world learning yields the strongest planning score among four variants. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Depth and Video checkmarks before comparing the PDMS column. The top row trains the action-only variant without either world objective. The next two rows add one modality, and the last adds both. Pages 9–10 state that these variants use identical training budgets. Depth adds 0.5 PDMS score points over 88.0, video adds 0.9, and the full model adds 1.2. The component columns let you inspect the nature of that improvement: all rows report comfort C of 100.0, while the full row has higher NC, DAC, TTC and EP. Consequently, the aggregate gain cannot be attributed to a changed reported comfort score. e08e16
What it supports. Both world-learning variants improve planning within this experiment, and combining them improves PDMS by another 0.3 over video alone. This is evidence of complementary supervision under the reported setup. It does not demonstrate a super-additive interaction, nor does it require rendered depth or video to be consumed during planning.
Where the evidence stops. The rows change which world tasks are learned, not just one attention edge. No seed variation or confidence intervals are given, so the reliability of the 0.3-point increment remains unresolved.
Table 5. Depth learning helps all three video metrics in the internal comparison. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows column by column. The checkmark switches on depth learning; the metric arrows show that LPIPS and FVD should decrease and PSNR should increase. All three move favorably in this internal ablation. Page 10 explains the intervention more precisely than the table alone: the depth-enabled model is jointly trained on depth, and its video queries receive causal conditioning from depth queries. The authors state that training data and compute are matched. Keep this comparison separate from Table 3(a): its baseline is this architecture without depth learning, whereas the other table compares against PWM, whose PSNR is higher than the full model's. e14e17
What it supports. FVD decreases by 12.23, from 65.82 to 53.59, accompanied by improved LPIPS and PSNR. The result supports the combined depth-learning design as useful for future-video prediction. It provides a stronger internal all-metric result than the cross-method video comparison, which contains a PSNR tradeoff.
Where the evidence stops. Depth supervision and depth-to-video conditioning change together. The table cannot determine whether the gain comes from the auxiliary learning signal, the directed connection, or their interaction; separating them requires an additional controlled experiment.
Figure 4. Selected scenes show what the authors mean by useful world context. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare horizontally within each row, keeping the displayed instruction and road layout fixed. Columns add depth, video or both to action training. Green marks the human trajectory and red the model prediction; each panel retains that legend. In the top keep-straight scene, compare how far the red trajectory extends toward the vehicle ahead. In the middle turn-left scene, examine the bend relative to the green reference. The bottom row emphasizes alignment during another left turn. The caption interprets these cases as slower collision avoidance, maneuver correction and improved expert alignment. Read these as selected spatial examples of the quantitative ablation, without treating every panel as a measured safety result. e16e20
What it supports. The full variant visibly changes the predicted trajectory relative to action-only, including a shorter forward extent in the top scene and closer expert-path alignment in the bottom scene. These examples make the authors' proposed geometric and future-context benefits concrete, while Table 4 supplies the aggregate planning comparison.
Where the evidence stops. Static overlays do not show executed collisions, speed traces or a failure distribution. The figure does not establish that the full model is closest to the expert in every panel; the safety interpretation belongs to the authors.
7. Analysis & limitations
7.1 What the evidence leaves open
Navsim evaluation uses real-log-derived benchmark data; the authors call it closed-loop planning. The supplied experiments do not establish physical vehicle deployment, reactive-traffic performance, or rare-event robustness beyond the reported benchmark and selected examples. e11e20
The same DA3 teacher provides training and evaluation depth targets. Improved agreement can reflect teacher imitation; Table 3 does not print depth scaling or exact δ thresholds. No independent geometric calibration is reported. e09e15
Tables provide point estimates without seed variation or confidence intervals. Table 5 does not isolate the causal mask, and Table 6 changes all query groups simultaneously. The qualitative scenes illustrate the authors' interpretation without quantifying failure frequency. e16e17e18e20
7.2 Questions for discussion
- How much of the planning gain comes from depth supervision versus access to depth-query embeddings?
- Does lower DA3-relative error correspond to better measured distance estimates in safety-critical scenes?
- What accuracy/latency frontier emerges when query groups and sampling steps are varied independently?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs Navsim split definitions, initialized backbones and DA3 targets plus the published resolution, loss weights and optimizer budget. Exact depth percentiles, flow solver and sampling steps, action horizon/context encoding, and a complete freeze list need clarification; hardware count alone gives no measured runtime. e03e05e08e09e10e11
Reader-proposed checks should separate depth supervision from the directed query connections, and compare planning-only with full generation using identical action noise. Neither experiment is reported as completed here. e04e08e16e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Factor depth supervision apart from depth-query access
Reader-proposed experiment, not run: train a 2×2 comparison with depth loss enabled/disabled and depth-query access by video/action queries enabled/blocked. Keep query counts, backbones, action/video losses, Navsim splits, training steps and batch size fixed; when enabled use the published depth weight 0.1. Repeat with matched seeds and evaluate PDMS plus front-video LPIPS, PSNR and FVD. Compare gains from depth loss with and without query access. A gain that persists when access is blocked would support shared-backbone auxiliary supervision; an additional reproducible gain when access is enabled would support the directed pathway beyond that regularization effect. e04e09e10e11e14e16e17
Check 2: Verify that rendering can be skipped without changing planning
Reader-proposed experiment, not run: use one trained checkpoint with identical observations, instructions, current-action context, queries, action noise and action solver settings. Compare action-only expert execution with execution of all three experts, resetting randomness so extra generators cannot change action noise. Measure per-state trajectory differences and PDMS, plus synchronized wall-clock latency and peak memory on the same hardware. The source's stated independence predicts matching trajectories within numerical tolerance while omitted generators reduce computation. Systematic action changes would expose an undocumented dependency or stochastic coupling. Report sampling steps and both timing distributions, since the source gives neither a latency measurement nor a complete solver recipe. e03e04e05e08e10
8.3 Reading coverage
Visual audit: Actually inspected the title/author/version page and every main-body page, including Figures 1–4, Tables 1–6 and Equations (1)–(3). All six final original crops were viewed after extraction; architecture and ablation bounds were corrected and the revised images viewed again. Checked the query-flow description against Figure 2, its caption and Sections 3.2–3.3, preserving the frozen-module wording ambiguity. Pages 4–6 supply method, loss and inference evidence; pages 6–8 supply training and evaluation details; pages 9–11 supply comparison conditions and qualitative/ablation interpretation. Bibliography pages 12–17 were read in the complete supplied text but not visually inspected. No appendix is present; separate supplements, project media and code remain unverified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1. Introduction (pp. 1–3)
- 2. Related Works, including 2.1 Driving World-Action Models and 2.2 Driving Vision-Language-Action Models (pp. 3–4)
- 3. Methodology: 3.1 Preliminaries; 3.2 DriveDreamer-Policy; 3.2.1 World Understanding; 3.2.2 World and Action Prediction (pp. 4–6)
- 3.3 Training Details (p. 6)
- 4.1 Experimental Setup (pp. 6–8)
- 4.2 Quantitative Results and world/depth ablation discussion (pp. 8–10)
- 4.3 Ablation Studies; 4.4 Qualitative Results (p. 11)
- 5. Conclusion (p. 11)
- References (pp. 12–17)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The preceding acquisition omission was addressed by visually inspecting PDF pages 1–11, including all four figures, all six tables and all three equations. References on pages 12–17 were read as supplied text.
- Separate supplemental material availability has not been fully verified.
- Identity/version note: the title and all eleven authors match the catalog, allowing typographic World–Action/World-Action variation. The inspected title page identifies arXiv:2604.01765v1 [cs.CV], 2 Apr 2026, while its internal date is 2026-4-3. This report covers that supplied v1 artifact; no other revision was supplied or compared.
- No appendix or dedicated limitations section appears in the supplied PDF.
- The linked project website and code were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, author/affiliation lines, internal date and arXiv margin stamp
The title and eleven named authors match the catalog. Affiliations are GigaAI, University of Toronto and CUHK MMLab. The artifact is stamped arXiv:2604.01765v1, 2 Apr 2026, and internally dated 2026-4-3.
Go to primary source ↓e02PDF pp. 1–3, Abstract and Section 1; p. 2, Figure 1 and caption
The authors motivate explicit depth as geometric grounding for appearance prediction and planning, and introduce a shared LLM with depth, video and action experts.
Go to primary source ↓e03PDF p. 4, Section 3.1, Equations (1)–(2) and inference paragraph
Linear data-to-noise interpolation defines target velocity x₁−x₀ and squared velocity regression; inference integrates backward from t=1 to t=0. No numerical solver or step count is specified here.
Go to primary source ↓e04PDF pp. 4–5, Sections 3.2 and 3.2.1, Input Processing and Embeddings Generation
Instructions, image patches and current-action tokens enter the LLM with learned depth/video/action queries. Video queries attend to depth context and action queries to both world groups in a single pass; modular inference modes are described.
Go to primary source ↓e05PDF p. 5, Figure 2, caption and surrounding method text; p. 6, Section 3.3
Figure 2 shows three separately conditioned diffusion transformers, training-only noisy depth supervision, VAE/CLIP video conditioning and frozen/trainable pictograms. Snowflakes appear on tokenizer, vision encoder, CLIP and VAEs; the prose broadly describes training all components.
Go to primary source ↓e06PDF p. 5, Section 3.2.2, Depth Generator
The pixel-space depth model concatenates noisy depth and RGB, uses flow matching, and cross-attends to depth-query embeddings. Those embeddings provide upstream context without requiring generated depth as an input to later experts.
Go to primary source ↓e07PDF pp. 5–6, Section 3.2.2, Video Generator; p. 5, Figure 2
Current RGB is VAE encoded; future noisy latents are conditioned on world-video queries through cross-attention, with CLIP image features concatenated to the conditioning representation.
Go to primary source ↓e08PDF p. 6, Section 3.2.2, Action Generator
A standalone diffusion transformer conditions noisy trajectories on action embeddings and predicts position/heading states (x,y,cosθ,sinθ). The action expert can run independently of explicit depth/video generation.
Go to primary source ↓e09PDF p. 6, Section 3.3, Depth Normalization, Model Initialization and Adaptation, Equation (3)
Log-depth per-map percentile normalization targets [−0.5,0.5]. Backbones are Qwen3-VL-2B, PPD and Wan-2.1-T2V-1.3B; depth/video resolution is 144×256 and video horizon nine frames. DA3 provides depth labels; the depth loss weight is 0.1 and others default to 1.0.
Go to primary source ↓e10PDF p. 8, Section 4.1 continuation, Baselines and Implementation Details
Baselines use their official reported Navsim performance. Training uses a two-layer normalized action MLP, 100k steps, batch 32, eight NVIDIA H20 GPUs, AdamW at 10⁻⁵ and 64/64/8 queries, with Navsim data and initialized backbones.
Go to primary source ↓e11PDF pp. 6–7, Section 4.1, Datasets and Planning Metrics
The paper reports 100k navtrain and 12k navtest samples at 2 Hz, PDMS for v1 and EPDMS for v2, evaluating v2 on navtest. It describes Navsim as real-log-derived and calls its planning evaluation closed-loop.
Go to primary source ↓e12PDF p. 7, Table 1, Sensors and PDMS columns, Human/AutoVLA/WoTE/PWM/Ours rows
PDMS is 89.2 for the three-camera model, 89.1 for AutoVLA, 88.3 for WoTE, 88.1 for PWM and 94.8 for human. Sensors vary across methods; the imitation-learning asterisk applies to Recogdrive.
Go to primary source ↓e13PDF p. 7, Table 2, EPDMS and component columns
EPDMS is 88.7 for Ours, 86.1 for DriveVLA-W0 and 84.5 for DiffusionDrive. Ours has DDC 99.5, TLC 99.9, LK 97.6 and EC 79.4; ARTEMIS has EC 89.1. The world-model-based block contains only Ours.
Go to primary source ↓e14PDF p. 7, Table 3(a) and World Generation Metrics; p. 9, World Performance Comparison
Front-view comparison gives LPIPS/PSNR/FVD 0.20/21.05/53.59 versus PWM 0.23/21.57/85.95. Recorded RGB frames are targets; front-only evaluation accommodates PWM's single-view generation.
Go to primary source ↓e15PDF p. 7, Table 3(b) and World Generation Metrics; p. 9, depth comparison paragraph
Depth targets are DA3 outputs. AbsRel/δ₁/δ₂/δ₃ are 8.1/92.8/98.6/99.5 for Ours, 9.3/91.4/98.3/99.5 for fine-tuned PPD and 18.5/80.4/94.0/97.2 for PPD. The table does not state scaling or exact δ thresholds.
Go to primary source ↓e16PDF p. 8, Table 4; pp. 9–10, Ablations on World Learning for Planning
Under stated identical budgets, action-only, depth+action, video+action and full training yield PDMS 88.0, 88.5, 88.9 and 89.2. Full NC/DAC/TTC/C/EP are 98.4/97.1/95.1/100.0/83.5. No uncertainty is tabulated.
Go to primary source ↓e17PDF p. 8, Table 5; p. 10, Ablations on Depth Learning for Video Generation
Without/with depth learning, LPIPS is 0.22/0.20, PSNR 19.89/21.05 and FVD 65.82/53.59. The description states matched data/compute and adds both joint depth learning and causal depth-to-video query conditioning.
Go to primary source ↓e18PDF p. 8, Table 6; p. 11, Section 4.3
Query counts increase together from 32/32/4 to 64/64/8. PDMS improves 88.9 to 89.2, FVD 57.97 to 53.59 and depth AbsRel 9.7 to 8.1; LPIPS remains 0.20. No runtime measurement accompanies the table.
Go to primary source ↓e19PDF p. 9, Figure 3 and caption; p. 11, Section 4.4, first paragraph
Two selected examples show left/front/right generated depth, first/last video frames and human/predicted trajectories. Depth visualization is truncated below 80 m; the authors interpret the scenes as spatially stable and useful for planning.
Go to primary source ↓e20PDF p. 10, Figure 4 and caption; p. 11, Section 4.4, second paragraph
Three scenarios compare action-only, depth-action, video-action and full variants. Green is human and red predicted trajectory. The caption interprets rows as slower collision avoidance, maneuver correction and closer expert alignment; no failure-rate statistic accompanies these scenes.
Go to primary source ↓8.5 Primary sources
DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning ↗
PDF · 8,245 extracted words
Source fingerprint
3ff851e73e9a647a8cb8ffe764b6a58d8c9a2d845e54497e0f371a492cd63fd4