DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving
1. Paper overview
In one sentence: DriveWAM predicts ego motion through a generated future in one video-action transformer, gaining benchmark accuracy while retaining dependence on semantic guidance and the quality of its imagined future. ev-problemev-flowev-guidanceev-memoryev-physicalaiev-routeev-curationev-memory-results
| At a glance | What to know |
|---|---|
| Research problem | Author claim Driving needs temporal dynamics as well as scene semantics. The authors argue that video pretraining supplies motion priors that image-text-centric policies must otherwise learn downstream. Their challenge is to connect those priors to continuous ego actions, update intent as scenes change, and retain useful history without unbounded KV storage. ev-problem |
| Core mechanism | Source description A shared transformer preserves video prediction during policy adaptation and implements future-conditioned inverse-dynamics action generation. ev-flowev-objective |
| A key reported result | PhysicalAI trajectory prediction on the curated test subset: 3 s: 0.47 / 1.35; 4 s: 0.83 / 2.47. ADE / FDE (m), lower is better. 100k training clips; 1,000 curated test clips from the official test split; front camera and one output trajectory per method. Alpamayo-1.5: 0.80 / 2.31 at 3 s and 1.44 / 4.18 at 4 s. VaVAM: 2.31 / 4.32 at 3 s; its released checkpoint has no 4 s result. Strong performance on this curated protocol; baseline pretraining data and model sizes differ. This is trajectory prediction from driving logs. ev-physicalaiev-datasetsev-implementationev-curation |
| Reading caution | Reader analysis The visual input and motion history are causal, but Appendix B derives route labels from the upcoming chunk's ground-truth yaw: left above 15 degrees, right below −15 degrees, otherwise straight. This coarse future-derived intent qualifies the unconditional no-future-information claim; performance with independently supplied navigation remains untested. ev-guidanceev-route |
Core contributions
- Source description
A shared transformer preserves video prediction during policy adaptation and implements future-conditioned inverse-dynamics action generation. ev-flowev-objective
- Source description
Chunk-specific VLM guidance is localized by an attention mask; inference-only memory selection separately protects video and action history. ev-guidanceev-maskev-memory
Figure 1. One predictive backbone connects observed history, semantic guidance, imagined video, and ego actions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the real-observation block on the left: the VAE produces video tokens and the action encoder produces motion tokens. Their blue and green colors persist in the history cache above. Follow the gold path from the frozen VLM into the text encoder, then into the transformer alongside ego-state conditioning. Snowflakes mark frozen modules; flames mark trainable ones. The two central prediction streams share the transformer. Their displayed upward arrows do not specify the sampling order: Equation (3) and the rollout text establish that future video is generated first and conditions action generation. The right-hand real-observation block shows how subsequent evidence refreshes history. ev-architectureev-tokenizationev-flowev-objectiveev-guidanceev-route
What it supports. The architectural support for a unified world-action model is the shared video/action transformer and the explicit future-latent input to action prediction. The VLM contributes semantic context, while ego motion is predicted through the video backbone. This is stronger classification evidence than a joint training loss alone.
Where the evidence stops. The guidance arrow assumes a supplied route command. Appendix B derives that label from upcoming ground-truth yaw, qualifying the main text's causal-input description. The diagram also represents predicted motion, without documenting physical vehicle execution.
2. Motivation
2.1 The problem and the proposed response
Driving needs temporal dynamics as well as scene semantics. The authors argue that video pretraining supplies motion priors that image-text-centric policies must otherwise learn downstream. Their challenge is to connect those priors to continuous ego actions, update intent as scenes change, and retain useful history without unbounded KV storage. ev-problem
2.2 What this reading follows
A driving policy must connect what it sees now to how the scene could change and what the ego vehicle should do next. DriveWAM makes that connection inside a pretrained video diffusion transformer: the future video latent becomes a condition for action generation. A frozen VLM adds changing semantic intent, and separate video/action caches keep historical attention bounded. This reading follows those information paths before examining planning comparisons and component ablations. The experiments show useful trajectory-prediction gains, but their meaning depends on the curated test distribution, future-derived route labels, and the distinction between short-clip accuracy and long-rollout memory profiling. ev-problemev-flowev-guidanceev-memoryev-physicalaiev-routeev-curationev-memory-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | IDM |
| Quadrant | Q2 · One Model × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The shared T_omega performs video and action flow prediction, supporting One Model for the predictive core. Equation (3) conditions actions on the future latent, supporting IDM and Q2. A separate frozen semantic VLM does not become a separate action planner. Driving and memory tags are directly supported by the task and bounded modality caches. ev-flowev-objectiveev-guidanceev-memory
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the predicted future an input to the action model
DriveWAM begins with synchronized video and ego-motion history. Video passes through the pretrained VAE, while an MLP embeds normalized motion increments. The shared transformer learns video flow prediction in its inherited latent space and action flow prediction through an action decoder. Equation (3) supplies the crucial dependency: the action branch receives the next video latent as a condition. In teacher-forced training that latent is clean ground truth; at inference it comes from video generation. Both branches retain supervision in Equation (4), and all chunks are processed with a causal mask during training. Deployment-style rollout instead samples one future chunk and its actions sequentially, then updates history when a real observation arrives. The implementation uses only partial video denoising before action generation. ev-tokenizationev-flowev-objectiveev-maskev-implementation
Figure 2. Teacher forcing preserves the future-to-action dependency while localizing each chunk's text condition. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read rows as queries and columns as keys. Colored cells permit attention; white cells block it. The subscript tau identifies noised tokens. In the first predicted chunk, the noisy action row u_{1,tau} can read clean future video z_1, together with the initial observation z_0, guidance g_0, and ego state e_0. The noisy video row z_{1,tau} has no corresponding access to future action. Move downward to the second chunk: clean video/action history becomes available, and its yellow guidance cells move from g_0 to g_1. This visually matches the inverse-dynamics factorization and the text's localized guidance rule. ev-maskev-flowev-objectiveev-guidanceev-implementationev-route
What it supports. The mask permits efficient parallel training without treating every token as simultaneously available. It encodes two useful restrictions: future actions depend on future visual content, and each target chunk receives its own semantic intent. The frozen VLM's later guidance cannot enter an earlier chunk through these displayed text-attention cells.
Where the evidence stops. This is a training mask. Clean future video is a teacher-forcing condition; inference uses a generated latent, with video denoising stopping at tau 0.6. The mask does not remove the future-derived route-label condition described in Appendix B.
5.2 Separate current scene reasoning from the origin of route intent
A plausible visual continuation does not uniquely specify the desired maneuver. The VLM supplies a short description and behavioral intent from the latest camera frame, recent ego trajectory, and supplied route direction. Localized cross-attention lets each future chunk receive updated guidance without reading text generated for later chunks. Figure 7 illustrates this distinction: a straight route can remain unchanged while guidance responds to pedestrians and traffic lights. However, Appendix B constructs route direction from upcoming ground-truth yaw because explicit annotations are unavailable. Reader analysis: the visual and motion inputs are causal, but the complete conditioning package includes future-derived coarse intent. This does not reveal a full future trajectory; it does mean that the guidance ablation cannot by itself establish the same gain with an independent navigation source. ev-guidanceev-maskev-routeev-promptev-guidance-ablation
5.3 Read bounded memory as a retention tradeoff
Once inference repeatedly consumes real observations, historical video tokens can greatly outnumber action tokens. DriveWAM therefore assigns separate budgets instead of letting both compete in one pool. Current-query attention supplies relevance, while similarity to other cached keys supplies redundancy. The score rewards useful evidence and penalizes repetition; low-scoring historical tokens leave when new KVs need space. Figure 3 illustrates the resulting retention patterns. Reader analysis: this is a computational selection rule, not a guarantee that every safety-critical object survives compression. Table 5 gives the appropriate empirical test: selective retention improves on FIFO at equal reported memory and approaches full-cache short-clip accuracy. Its separate long-rollout profile establishes reduced storage and attention cost; longer-horizon decision quality remains a distinct question. ev-memoryev-memory-results
5.4 Training and inference
During training
Fine-tuning uses Wan2.2-TI2V-5B initialized from the checkpoint of cited work [19], the full diffusion transformer, and new action/ego modules. Figure 1 marks the VAE, text encoder, and VLM frozen. Action encoder/decoder MLPs have hidden dimension 3072. ev-architectureev-implementation
Training uses 256×448 images, 48 NVIDIA H20 GPUs, AdamW with learning rate 0.00001, betas (0.9, 0.95), weight decay 0.1, per-device batch size 1, and action-loss weight 1. NAVSIM uses 100k iterations with learning-rate halving at 50k/70k/90k. PhysicalAI uses 50k iterations on random 12-second segments from 20-second clips; video is 1 Hz and actions 10 Hz. ev-implementation
All chunks are processed in parallel with causal teacher forcing. NAVSIM supplies only one four-second prediction horizon per sample, so its evaluation does not exercise extended autoregressive memory. ev-objectiveev-implementation
During inference
Generate video first using three Euler steps from flow time 1 to 0.6, then actions using ten steps from 1 to 0. Thus the operational future latent is only partially denoised. Encode the next real observation into history when it arrives. Selective memory uses 448 video tokens, 160 action tokens, and lambda 0.07. ev-objectiveev-implementationev-memory
5.5 Implementation flow
- Encode synchronized history
A pretrained VAE maps video chunks to latents z; an MLP action encoder maps normalized translation/yaw increments to tokens u. The inherited video input embedding maps latents into the shared transformer dimension. History H preserves chunk order; a separate MLP and cross-attention branch supply ego state e. ev-tokenizationev-flow
- Predict a future, then infer motion
The same transformer predicts video flow velocity and, through an MLP decoder, action flow velocity. Action prediction conditions on clean future video during teacher-forced training and generated future latents at inference. Noisy-history augmentation addresses this mismatch. Joint prediction therefore has an explicit video-to-action dependency. ev-flowev-objective
- Refresh semantic intent
Frozen Qwen3-VL-8B receives the latest frame, previous-chunk trajectory visualization, and route command. It produces two qualitative sentences under 50 words. Guidance is cached before training and queried once per decision step at inference. Each target chunk attends only to its own guidance. ev-guidanceev-maskev-routeev-prompt
- Retain useful history
Separate modality pools score cached keys by current-query attention relevance minus key-similarity redundancy. Low-scoring historical tokens are evicted to make room for new KVs. This selection changes inference memory, without retraining or modifying the loss. ev-memory
6. Experiments & results
DriveWAM turns a pretrained video diffusion transformer into a driving policy: it predicts a future video latent, then generates ego motion conditioned on that future. A frozen VLM supplies changing semantic guidance, while selective video/action caches bound historical memory. The evidence supports benchmark trajectory prediction and a useful cache tradeoff, with route-label, data-curation, and deployment limits.
6.1 Read the original evidence
Table 2. DriveWAM improves trajectory prediction on the authors' curated 1,000-clip PhysicalAI test subset. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four error columns in two horizon pairs: average displacement error and final displacement error at three seconds, then the same metrics at four seconds. Lower values are better. Every row uses a single camera, and Section 4.3 states that each method outputs one trajectory. The asterisk matters: VaVAM's released checkpoint only supports three seconds, so its dashes are missing evaluations rather than failed four-second predictions. The parameter column also matters: DriveWAM lists a 5B video model plus an 8B semantic guide, whereas Alpamayo-1.5 lists 10B. Keep those system differences separate from the error comparison. ev-physicalaiev-datasetsev-curationev-qualitative
What it supports. At four seconds, DriveWAM reports ADE/FDE of 0.83/2.47 m against Alpamayo-1.5's 1.44/4.18 m. At three seconds it reports 0.47/1.35 m, lower than both listed baselines. These are concrete improvements under the shared front-camera, single-trajectory evaluation protocol on this selected test subset.
Where the evidence stops. The models have different pretraining histories and parameter counts. Appendix A enriches the test subset for rare/high-interest scenes, so this table does not estimate the full test distribution. It supplies no uncertainty intervals or physical-execution evidence.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PhysicalAI trajectory prediction on the curated test subset 100k training clips; 1,000 curated test clips from the official test split; front camera and one output trajectory per method. | 3 s: 0.47 / 1.35; 4 s: 0.83 / 2.47. ADE / FDE (m), lower is better | Alpamayo-1.5: 0.80 / 2.31 at 3 s and 1.44 / 4.18 at 4 s. VaVAM: 2.31 / 4.32 at 3 s; its released checkpoint has no 4 s result. Strong performance on this curated protocol; baseline pretraining data and model sizes differ. This is trajectory prediction from driving logs. ev-physicalaiev-datasetsev-implementationev-curation |
| NAVSIM v1 planning Standard benchmark with 103k trainval and 12k test samples; single front-view camera; four-second prediction. | 90.1. PDMS, higher is better | WorldDrive 89.0; Epona 86.2; DriveVLA-W0 88.4. The separately marked multi-anchor DriveVLA-W0 variant reports 90.2. Competitive single-view planning, not the highest number in every table row; sensors and training variants differ. ev-navsimev-datasetsev-implementation |
| Scene-evolving guidance and data scaling PhysicalAI; fixed 50k iterations at 4k, 20k, and 100k training clips. | With guidance: 1.01 / 2.95, 0.94 / 2.65, 0.83 / 2.47. ADE@4s / FDE@4s (m) | Fixed global prompt: 1.21 / 3.65, 0.95 / 2.94, 0.92 / 2.75, respectively. Both metrics improve with guidance at each scale. Three data sizes do not establish an asymptotic scaling law. ev-guidance-ablation |
| Video-prior adaptation ablation PhysicalAI; 100k clips and 50k iterations. | Pretrained initialization plus video supervision: 0.83 / 2.47. ADE@4s / FDE@4s (m) | Scratch initialization with video supervision: 1.10 / 3.26. Pretrained initialization without video supervision: 1.23 / 3.79. Supports preserving video supervision during adaptation; the table omits the scratch/action-only combination. ev-adaptation |
| Selective KV memory tradeoff Accuracy on 20-second PhysicalAI clips; memory/attention costs separately profiled at 300 seconds. | Selective: 0.89 / 2.52; 0.25 GB; 1.44 GFLOPs. ADE@4s / FDE@4s (m); KV GB; attention GFLOPs | Full: 0.83 / 2.47; 3.07 GB; 17.37 GFLOPs. FIFO: 1.40 / 3.47; 0.25 GB; 1.05 GFLOPs. Selective improves accuracy over equal-memory FIFO but costs more attention computation. GFLOPs cover one self-attention layer, and KV GB sum all DiT layers; neither measures complete-system cost. ev-memory-results |
| Per-chunk inference efficiency One NVIDIA H20; a four-second prediction chunk. | Default: guidance 125, video 372, action 765. Five action steps: 374 ms for action, approximately 871 ms total, with 0.84 / 2.45. Stage latency (ms); ADE@4s / FDE@4s (m) | Ten steps give 0.83 / 2.47. Alpamayo-1.5 totals 900 ms with 1.44 / 4.18. The five-step variant is a measured speed/accuracy option, not evidence of safety under execution delays. ev-efficiency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Changing semantic guidance helps at every tested data scale, with uneven gains across metrics. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare adjacent rows before comparing different dataset sizes. Within each pair, the clip count and iteration count stay fixed; the cross uses a fixed global prompt and the checkmark uses scene-evolving guidance. Then follow only the checked rows down the table to see the scaling trend under the changing-guidance configuration. This two-pass reading separates a text-conditioning comparison from a data-volume comparison. At 20k clips, ADE changes only from 0.95 to 0.94, while FDE moves from 2.94 to 2.65, so the two metrics tell different parts of the story. Figure 5 plots these same trends. ev-guidance-ablationev-guidanceev-route
What it supports. Guided ADE/FDE improves from 1.01/2.95 m at 4k clips to 0.83/2.47 m at 100k. Guidance also improves both errors within each data-size pair. The evidence supports complementary semantic conditioning across these tested scales; it does not show that the gain has a fixed magnitude.
Where the evidence stops. The ablation changes the semantic-conditioning package, without isolating scene description from route intent. All runs use 50k iterations, not equal numbers of dataset passes. Three scales and no seed intervals are insufficient to infer an asymptotic scaling law.
Table 4. Video supervision remains useful after adapting the pretrained backbone to action prediction. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two left columns are interventions, and the two right columns measure trajectory error. The first row removes pretrained initialization but keeps video supervision. The second keeps initialization but removes video supervision. The last keeps both. The caption specifies a common setting of 100k clips and 50k iterations, and Section 4.4 places the comparison on PhysicalAI. Read the first-to-last comparison as evidence about initialization under joint supervision, then the second-to-last comparison as evidence about video supervision with pretrained initialization. There is no row removing both, so this is not a complete two-by-two factorial experiment. ev-adaptationev-objectiveev-flow
What it supports. The full setup reaches 0.83/2.47 m ADE/FDE, compared with 1.10/3.26 m from scratch and 1.23/3.79 m without video supervision. The latter result motivates preserving video prediction during adaptation, even though the downstream score evaluates the ego trajectory rather than video quality.
Where the evidence stops. The authors interpret the action-only degradation as failure to preserve video priors. That explanation is plausible, but this table measures trajectory errors rather than representation preservation. It does not isolate whether generated future content is causally used at inference.
Table 5. Selective retention recovers much of full-cache accuracy within FIFO's memory budget. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the protocol note above the columns: trajectory errors come from 20-second clips, whereas memory and GFLOPs come from a 300-second profile. Next compare FIFO and Selective. Their 0.25 GB cache sizes match, so the error difference tests retention policy at the reported equal-memory setting. Finally compare Selective with Full to see the remaining accuracy gap. Section 4.4 defines the profiling scope: KV memory sums all diffusion-transformer layers, while GFLOPs cover one causal self-attention layer per step. The attention cost is consequently neither total model computation nor a direct wall-clock latency measurement. ev-memoryev-memory-resultsev-physicalaiev-implementation
What it supports. Selective caching achieves 0.89/2.52 m ADE/FDE versus FIFO's 1.40/3.47 m, while Full reaches 0.83/2.47 m. It cuts the profiled memory from 3.07 to 0.25 GB and attention cost from 17.37 to 1.44 GFLOPs. Selection improves the accuracy–memory tradeoff but costs more attention computation than FIFO's 1.05 GFLOPs.
Where the evidence stops. The 300-second profile establishes bounded cache cost, not 300-second prediction accuracy. Main Table 2's 0.83/2.47 matches the Full row; the explicit Selective result here is 0.89/2.52. Do not combine full-cache accuracy and selective-cache cost into one measured operating point.
7. Analysis & limitations
7.1 What the evidence leaves open
The visual input and motion history are causal, but Appendix B derives route labels from the upcoming chunk's ground-truth yaw: left above 15 degrees, right below −15 degrees, otherwise straight. This coarse future-derived intent qualifies the unconditional no-future-information claim; performance with independently supplied navigation remains untested. ev-guidanceev-route
VLM-based test curation enriches rare and high-interest scenes and includes 200 common-scene controls. Aggregate performance need not transfer to the full test distribution. Tables provide no seed variability or confidence intervals, and illustrative videos do not establish physical deployment or 300-second planning accuracy. ev-curationev-physicalaiev-guidance-ablationev-memory-resultsev-qualitative
7.2 Questions for discussion
- How much of the guidance gain remains when route commands come from an independent navigation source?
- Does selective retention preserve rare-event accuracy over long rollouts, beyond the reported short-clip aggregate?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the specified pretrained initialization, camera/ego synchronization, normalization, guidance cache, and exact subset membership. Curation retains scores at least 2.0 plus half the lower-score clips; rare test tags occur below 1%, with up to 30 clips per tag. The paper gives example weights but omits a complete weighting table, selection seeds, and detailed quota/deduplication rules. ev-tokenizationev-implementationev-curation
Further gaps include the precise action-normalization statistics, noisy-history augmentation schedule, and software versions. Appendix C names default vLLM compilation without a version. Proposed checks should separate route-command provenance from semantic guidance and test action sensitivity to generated versus shuffled future latents. ev-tokenizationev-objectiveev-routeev-memory-resultsev-efficiencyev-flow
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether actions depend on the generated future
Reader-proposed check, not performed: freeze one DriveWAM checkpoint and evaluate identical held-out clips with identical history, ego state, guidance, and action-noise seeds. Compare the default generated future latent, the clean ground-truth future latent as an oracle control, and generated latents shuffled between clips with similar route direction and ego speed. Measure paired changes in action outputs and ADE/FDE. A shuffled future that leaves both nearly unchanged would challenge the claim that future content drives action prediction; a large oracle advantage would quantify the teacher-forcing gap. Keep the default video solver fixed so denoising compute does not confound the intervention. ev-flowev-objectiveev-implementationev-adaptationev-physicalai
Check 2: Separate scene-guidance gains from future-derived route labels
Reader-proposed check, not performed: train matched route-only and scene-evolving-guidance variants with the same initialization, clips, iterations, and several seeds. Evaluate each using either the paper's yaw-derived route labels or an independently provided navigation command available before the target chunk. Regenerate guidance from the same current frame and recent trajectory under each command source. Compare paired ADE/FDE and report turn/intersection cases separately. If the guidance advantage disappears with independent commands, the published gain is contingent on route-label provenance; persistence would strengthen the semantic-guidance interpretation. Preserve test membership to separate this effect from curation changes. ev-guidanceev-routeev-guidance-ablationev-curationev-implementation
8.3 Reading coverage
Visual audit: The title/author/version block on p. 1, problem and method text on pp. 2–5, attention mask and retention diagnostic on p. 6, all main tables on pp. 7–8, ablation interpretation and qualitative/scaling figures on p. 9, and Appendices A–D on pp. 14–17 were rendered and visually inspected. This includes the route-label derivation and prompt, curation details, training hardware/settings, and the H20/vLLM latency evidence. Figures 1–8 and Tables 1–6 were visually read. All six final crops were individually inspected after adjustment; narrow originals were rendered at higher DPI without upscaling. Required table-caption protocol notes were retained. The diagram arrows, trainable/frozen markers, and attention cells were checked against Equations (2)–(4) and Sections 3.1–3.3. The causal-guidance qualification from Appendix B is disclosed. Pages 10–13, containing the conclusion continuation and references, were read in the supplied text; they supply no additional numerical or implementation claims in this edition. Separate supplements were not verified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/author/version block (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work; 2.1 Vision-Language-Action Models in Autonomous Driving; 2.2 World-Action Models (pp. 2–3)
- 3 Method; 3.1 Autoregressive Video-Action Generation (pp. 3–5)
- 3.2 Scene-Evolving Driving Guidance (p. 5)
- 3.3 Selective KV Memory for Long-Horizon Rollout (pp. 5–7)
- 4 Experiments; 4.1 Datasets; 4.2 Implementation Details; 4.3 Main Results (pp. 7–8)
- 4.4 Ablation Study and 5 Conclusion (pp. 9–10)
- References (pp. 10–13)
- A Dataset Curation (p. 14; Figure 6 on p. 15)
- B VLM Guidance Details (pp. 14–16)
- C Efficiency Analysis (p. 16)
- D Additional Qualitative Results (pp. 16–17)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation above was addressed by inspecting original PDF pages and all final crops; all six supplied text chunks were read without truncation.
- The supplied artifact is arXiv:2605.28544v1 [cs.CV], 27 May 2026. Its title and six authors match the catalog after name-order normalization. The catalog URL is unversioned; no other revision was supplied or compared.
- Code, external project pages, and separate supplements were not inspected; experiments were not reproduced. References were read as part of this paper, without opening cited works.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title, author/affiliation block, arXiv margin and preprint label
The title matches the supplied observed title. Authors are Chen Shi, Jinrui Xu, Shaoshuai Shi, Kehua Sheng, Bo Zhang, and Li Jiang. Affiliations are The Chinese University of Hong Kong, Shenzhen and Voyager Research, Didi Chuxing. The artifact is arXiv:2605.28544v1 [cs.CV], 27 May 2026; Chen Shi and Jinrui Xu share equal contribution.
Go to primary source ↓ev-problemPDF pp. 1–2, Section 1 Introduction, motivation and contributions
The authors motivate video-derived temporal priors, semantic guidance, and bounded historical memory for driving, and identify future-first action generation in a shared transformer.
Go to primary source ↓ev-architecturePDF p. 4, Figure 1 and caption
Video and action tokens enter one Video Diffusion Transformer. The VLM and text encoder carry snowflake markers, as does the video VAE; the transformer and action/ego encoders carry flame markers. Real observations feed history, and VLM guidance feeds text conditioning.
Go to primary source ↓ev-tokenizationPDF pp. 3–4, Section 3.1, Tokenization and Equation (1)
Clips are divided into chunks. A VAE encodes video, while an MLP encodes normalized ego-frame translation and yaw increments. Video latents are projected to transformer dimension; ordered video/action pairs form history. Numerical normalization statistics are not specified here.
Go to primary source ↓ev-flowPDF p. 4, Section 3.1, World-action flow and Equations (2)–(3); p. 5, opening paragraph
Video flow prediction uses history, ego state, and guidance. The same transformer predicts action velocity through an MLP decoder, additionally conditioned on future video. That latent is clean during training and generated at inference.
Go to primary source ↓ev-objectivePDF p. 5, Section 3.1, opening paragraph, Equation (4), and Full-clip training and autoregressive rollout
The joint loss fits video/action flow velocities with action weight beta_a. Training uses causal teacher forcing and noisy-history augmentation; inference generates video then actions and appends newly available real observations. The augmentation schedule is not provided.
Go to primary source ↓ev-guidancePDF p. 5, Section 3.2, Equation (5) and Temporally localized guidance injection
Frozen Qwen3-VL-8B uses the latest observation, recent motion, and a supplied route command. Guidance is precomputed for training and queried once per decision step at inference; a block-diagonal text mask restricts target chunks to their corresponding guidance.
Go to primary source ↓ev-maskPDF p. 6, Figure 2 and caption; p. 5, Sections 3.1–3.2
Colored cells allow attention and white cells mask it. The noisy action row u_{1,tau} can read clean future video z_1; noisy video z_{1,tau} cannot read future action. The next chunk uses g_1 rather than g_0, matching localized guidance and the stated conditional dependencies.
Go to primary source ↓ev-memoryPDF p. 6, Section 3.3, Figure 3 and Equations (6)–(7); p. 7, Equation (8)
Video and action caches have separate budgets. Relevance averages attention mass from current queries; redundancy averages cosine similarity among cached keys. The score subtracts weighted redundancy from relevance. Top-ranked historical tokens are retained while space is reserved for new KVs. This mechanism applies only at inference.
Go to primary source ↓ev-datasetsPDF p. 7, Section 4.1 Datasets; p. 8, Section 4.2 dataset-specific sampling
NAVSIM uses 103k trainval and 12k test samples and reports NC, DAC, TTC, Comfort, Ego Progress, and PDMS. PhysicalAI uses the front camera and ego-motion labels, a curated 100k training subset, and a curated 1,000-clip test subset; trajectory horizons are three and four seconds.
Go to primary source ↓ev-implementationPDF pp. 7–8, Section 4.2 Implementation Details
The source specifies Wan2.2-TI2V-5B initialization through cited work [19], full-transformer fine-tuning, 3072-dimensional action MLPs, resolution, 48 H20 GPUs, optimizer settings, dataset schedules, four-second chunks, 1 Hz video and 10 Hz PhysicalAI actions. Inference uses 3 video steps over tau 1 to 0.6 and 10 action steps over tau 1 to 0; cache capacities are 448/160 and lambda is 0.07.
Go to primary source ↓ev-navsimPDF p. 7, Table 1, DriveWAM, WorldDrive, Epona, and DriveVLA-W0 rows and caption; p. 8, Section 4.3 NAVSIM
PDMS is 90.1 for DriveWAM, 89.0 for WorldDrive, 86.2 for Epona, and 88.4 for standard DriveVLA-W0. The dagger-marked multi-anchor DriveVLA-W0 row reports 90.2; the caption distinguishes sensors and training variants.
Go to primary source ↓ev-physicalaiPDF p. 7, Table 2, all method rows and checkpoint footnote; p. 8, Section 4.3 PhysicalAI; p. 9, Figure 5 metric units
DriveWAM reports ADE/FDE 0.47/1.35 at three seconds and 0.83/2.47 at four seconds. Alpamayo-1.5 reports 0.80/2.31 and 1.44/4.18; VaVAM reports 2.31/4.32 only at three seconds. All use a front camera and one trajectory, but parameter counts and training histories differ. The paper labels displacement error axes in meters.
Go to primary source ↓ev-guidance-ablationPDF p. 8, Table 3, all six rows; p. 9, Section 4.4 Scene-evolving Driving Guidance, Data Scaling, and Figure 5
At fixed 50k iterations, guidance versus fixed prompt yields 1.01/2.95 versus 1.21/3.65 at 4k clips, 0.94/2.65 versus 0.95/2.94 at 20k, and 0.83/2.47 versus 0.92/2.75 at 100k. No uncertainty intervals appear.
Go to primary source ↓ev-adaptationPDF p. 8, Table 4 and caption; p. 9, Section 4.4 Video Foundation Model Adaptation and Figure 5 metric units
With 100k clips and 50k iterations, pretrained plus video supervision achieves 0.83/2.47 ADE/FDE. Scratch plus video supervision gives 1.10/3.26; pretrained without video supervision gives 1.23/3.79. No scratch-without-video row or repeated-seed uncertainty is supplied.
Go to primary source ↓ev-memory-resultsPDF p. 8, Table 5 and caption; p. 9, Section 4.4 Selective KV Memory
Full/FIFO/selective have ADE/FDE 0.83/2.47, 1.40/3.47, and 0.89/2.52 on 20-second clips. The separate 300-second profile reports KV GB 3.07/0.25/0.25 and attention GFLOPs 17.37/1.05/1.44. Memory sums DiT layers; FLOPs refer to one causal self-attention layer per step.
Go to primary source ↓ev-curationPDF p. 14, Appendix A Dataset Curation; p. 15, Figure 6 and caption
Qwen3-VL-8B tags 20 sampled frames per clip using four prompt categories. Weighted tags determine interest scores. Training retains scores at least 2.0 and samples half the lower-score pool. Test curation combines rare tags below 1%, up to 30 clips per rare tag, grouped high-interest clips above the 75th percentile, and 200 common controls. Only example weights and approximate quotas are given.
Go to primary source ↓ev-routePDF pp. 14–15, Appendix B pipeline and Route command paragraph; p. 15, Figure 7
The latest front-camera frame and previous four-second BEV ego trajectory accompany route guidance. Because explicit route labels are unavailable, the command is derived from ground-truth yaw change over the upcoming chunk: left above 15 degrees, right below −15 degrees, otherwise straight. It contains coarse direction rather than trajectory coordinates.
Go to primary source ↓ev-promptPDF p. 16, Appendix B, Prompt template for VLM guidance
The supplied template requests two present-tense sentences, under 50 words total, describing current road context and qualitative upcoming behavior. It excludes numerical trajectory values and treats the route command as authoritative.
Go to primary source ↓ev-efficiencyPDF p. 16, Appendix C Efficiency Analysis, all three stage paragraphs and Table 6
One H20 gives default DriveWAM stage latencies 125/372/765 ms for VLM/video/action, with ADE/FDE 0.83/2.47. Five action steps reduce the action stage to 374 ms and total to approximately 871 ms, with 0.84/2.45. Alpamayo totals 900 ms. Default vLLM compilation is named without a software version.
Go to primary source ↓ev-qualitativePDF p. 9, Figure 4 and caption; p. 17, Appendix D, Figure 8 and accompanying paragraphs
Selected examples juxtapose predicted ego trajectories and generated future frames on NAVSIM and PhysicalAI. The authors describe coherence and physical plausibility; these illustrations do not supply execution trials or quantitative long-horizon safety measurements.
Go to primary source ↓8.5 Primary sources
DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving ↗
PDF · 9,035 extracted words
Source fingerprint
2cac17398b8e3d82f4edc4b70bc7ab664cca31600c1d4e17a9dd51c49c205581