UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
1. Paper overview
In one sentence: UNIVERSE uses video supervision to train a shared trajectory denoiser, while a visibility mask removes the need for future-video generation during planning. e02e03e04e05e07
| At a glance | What to know |
|---|---|
| Research problem | Source description A planner can fit source-dataset driving habits without learning transferable dynamics. The paper asks whether dense future-video supervision transfers better when video and trajectory denoising share transformer parameters. It studies autonomous-driving prediction under cross-dataset and real-to-simulation shifts. e02 |
| Core mechanism | Source description A single DiT shares attention and feed-forward blocks across video and trajectory objectives while retaining modality-specific projections and heads. e03 |
| A key reported result | NAVSIM v1 Navtest planning: 91.0 PDMS, higher is better. Front-camera UNIVERSE with default two-step trajectory-only inference; the paper labels these closed-loop metrics. Two-DiT 89.6; no video co-training 88.2; no mask 90.9; DriveLaW 89.1; DiffusionDrive 88.1. The variant differences support shared parameters and video supervision. External methods differ in inputs; no uncertainty estimates accompany the table. e08e09e10 |
| Reading caution | Source description The authors identify missing side/rear cues with front-view input and costly video-backbone training. Multi-view input, lighter tokenizers, smaller backbones, and distillation remain future directions. e18 |
Core contributions
- Source description
A single DiT shares attention and feed-forward blocks across video and trajectory objectives while retaining modality-specific projections and heads. e03
Figure 2. Shared denoising parameters let video supervision shape a planner whose future tokens remain separate. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the conditions on the left into the shared DiT, then read the central matrices by query row and key/value column, using Appendix B.1 to establish the orientation. C denotes context, V future video, and A future trajectory. Colored cells permit attention; hatched cells block it. Both future groups read C, but neither reads the other. The C row also blocks future targets, preventing an indirect path through context. The lower matrices remove an entire future group for single-modality inference. The right-hand outputs are predicted video and trajectory representations, not an actuator interface. e03e04e05e06e20
What it supports. The architecture combines parameter sharing with restricted information flow. Video and trajectory objectives update the same DiT, yet trajectory tokens are trained without access to future video. This is the structural reason the model can omit video tokens during planning while retaining benefits learned from video supervision.
Where the evidence stops. Figure 2 groups text tokens with other conditions near C. Equation (2) and Section 3.2 instead define masked context from visual history and velocity, with text supplied separately by cross-attention. The explicit formulation governs this explanation.
2. Motivation
2.1 The problem and the proposed response
A planner can fit source-dataset driving habits without learning transferable dynamics. The paper asks whether dense future-video supervision transfers better when video and trajectory denoising share transformer parameters. It studies autonomous-driving prediction under cross-dataset and real-to-simulation shifts. e02
2.2 What this reading follows
A driving model can learn from future video without generating that video every time it plans. UNIVERSE makes this separation explicit: one diffusion transformer learns both visual evolution and ego trajectories, while a mask prevents their future tokens from reading one another. The useful question is then where the benefit travels. Here it travels through shared trained parameters and common historical context. The paper tests that choice using variants that remove video supervision, remove the mask, or split the transformer. This edition follows those comparisons from information flow to transfer performance, computational cost, and the limits of visual-motion agreement. e02e03e04e05e07
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Insufficient evidence to decide
One Model is supported by shared attention/feed-forward blocks with modality-specific heads. This is not inverse dynamics: actions cannot read future video. Both modalities are jointly predicted during training and optional joint rollout, while default deployment uses video through learned parameters. Other mechanisms/Outside quadrants therefore depends on whether the catalog classifies deployment or joint prediction; that convention is not established by the supplied labels. The snapshot is preserved. e03e04e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow the gradient when the future tokens cannot communicate
Consider a training sample with historical images, current velocity, a future trajectory, and future video. The model corrupts both targets and learns their flow velocities. Its loss adds the trajectory term to a weighted video term, so both losses update the same attention and feed-forward parameters. This provides the route by which dense visual supervision can influence later trajectory prediction. It does not require a future-video token to transmit information to a future-action token. The mask explicitly prohibits that transmission, and it also prevents context tokens from reading future targets. Appendix B explains why this second restriction matters: otherwise context could become an indirect information bridge. Full attention inside each future modality remains valid because the model denoises chunks rather than decoding successive future tokens autoregressively. e03e04e06
5.2 Remove a token group while preserving the learned dependency
At deployment, trajectory-only mode instantiates historical context and noisy trajectory tokens, with instruction tokens supplied separately through cross-attention. The video token group is absent. Appendix B argues that this preserves the trajectory branch’s learned attention dependencies, since those queries never read future video during training. Reader analysis: the strongest implementation check would hold conditions, action noise, and flow times fixed, then compare trajectory outputs with and without video tokens. The equations suggest invariance apart from numerical effects, but the supplied paper does not report that exact test. Its efficiency comparison instead contrasts the masked model with a separately trained no-mask variant that needs coupled rollout. Distinguishing structural reasoning from the measured comparison explains both the deployment rationale and the boundary of the advertised speedup. e03e04e05e07e14
5.3 Use complementary measurements to evaluate the design
The ablations ask three different questions. Setting the video weight to zero tests the contribution of video supervision. Replacing one DiT with two tests shared parameterization, although resource usage also changes. Removing the mask tests a coupled future-token design and simultaneously changes what inference must retain. The near tie between full and no-mask NAVSIM scores therefore cannot settle the deployment argument. Cross-domain collision metrics and latency provide additional evidence, while the DPVO diagnostic asks whether the generated visuals imply a trajectory consistent with the predicted one. Reader analysis: these measurements strengthen the case when interpreted together, but none substitutes for another. Similarity-aligned odometry can look good despite scale error; good video distribution metrics do not establish safe action execution; horizon-based collision rates do not report route completion. e07e10e11e12e13e14
5.4 Training and inference
During training
Flow matching trains both modalities with Gaussian corruption and a common sampled flow time. Weighted video and trajectory losses update shared parameters. The no-video variant sets the video weight to zero; only the text encoder is explicitly called frozen. e03e06e07
NAVSIM training uses Wan2.2-TI2V-5B, four history/eight future frames at 832×480, NVIDIA H20 GPUs, distributed bf16, and AdamW (learning rate 10^-4, weight decay 0.01). It runs 20k steps at batch 80, then 10k at effective batch 640 via accumulation. A 1k-step warm-up starts at 10^-3 of the base rate, followed by a constant rate. e09
During inference
Default planning uses context and trajectory tokens with two flow sampling steps. Video-only and joint modes instantiate the requested future groups with unchanged weights. Joint mode still blocks future cross-modal attention; actions do not read imagined future videos. e04e05e09
The inference controller selects token groups. The source does not specify a low-level vehicle controller, actuation interface, or feedback/replanning frequency; generated trajectories and videos must not be equated with physical execution. e03e05e08
5.5 Implementation flow
- Encode conditions
A frozen Wan2.2-TI2V-5B text encoder supplies instruction tokens through cross-attention. A 3D-causal VAE encodes historical images; an MLP embeds velocity. Equation (2) defines self-attention context as visual history plus velocity tokens. e03e09
- Share the denoiser
Noisy future video and trajectory tokens receive modality-specific projections, modality embeddings, and temporal embeddings. Shared DiT blocks process both; separate heads predict their outputs. Actions encode future poses, not low-level steering/throttle commands. e03
- Restrict future dependencies
Each future modality reads context and itself. Context cannot read future targets, blocking an indirect information bridge between modalities. Within-modality attention remains bidirectional because entire chunks are denoised. e04
6. Experiments & results
UNIVERSE trains future driving video and ego trajectories in one diffusion transformer while blocking attention between their future tokens. Video supervision updates the planner’s parameters, but trajectory-only deployment omits video generation. Reported benchmark gains support this design without establishing physical driving safety (e03, e04, e05, e10, e11).
6.1 Read the original evidence
Table 1. The shared model leads the reported composite score, with different variant gaps for supervision and masking. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the shaded PDMS column at the right, then compare the four UNIVERSE rows at the bottom. Higher scores are better. Section 4.1 defines the ingredients: No Collision, Drivable Area Compliance, Time-to-Collision, Comfort, and Ego Progress. The full row scores 99.1, 97.6, 98.5, 100, and 83.6 on those components. Read across before attributing the aggregate gain to one behavior: the no-mask variant slightly exceeds the full model on NC and TTC. The Image and Lidar columns also matter when comparing external baselines, since the input configurations differ. e07e08e10e11e14
What it supports. UNIVERSE reports 91.0 PDMS, versus 89.6 for Two-DiT and 88.2 without video co-training: differences of 1.4 and 2.8 score points. Removing the mask barely changes the in-domain aggregate, at 90.9. The mask’s deployment and transfer effects therefore need the other tables rather than this score alone.
Where the evidence stops. The paper calls these closed-loop metrics, but this table does not establish physical driving safety. External baselines differ in sensors and supervision, and no repeated-seed variation or confidence intervals are reported.
Table 2. Transfer comparisons separate positional imitation error from collision-related planning performance. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read nuScenes and Bench2Drive as separate target-domain blocks. Within each, L2 displacement is measured in meters and collision rate in percent; both decrease with better reported performance. Each metric has one-, two-, and three-second columns plus an average. Red crosses in Finetune indicate that the rows use NAVSIM training without adaptation to the target dataset. Compare the UNIVERSE variants before the external methods to examine architectural changes under the stated common protocol. Pay particular attention to collision columns: similar L2 averages can coexist with appreciably different collision rates, so trajectory closeness is not a complete safety measure. e07e08e10e11
What it supports. The full model reports 0.83 m and 0.03% on nuScenes, and 1.33 m and 1.19% on Bench2Drive. Two-DiT’s collision averages rise to 0.21% and 1.82%; no-mask reaches 0.06% and 1.79%. These results support transfer benefits that the nearly tied full/no-mask NAVSIM scores alone would obscure.
Where the evidence stops. The averages concern the displayed horizons, not route-level completion or a calibrated on-road crash probability. Split selection, collision implementation, and uncertainty are not specified sufficiently to infer deployment reliability from these percentages.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM v1 Navtest planning Front-camera UNIVERSE with default two-step trajectory-only inference; the paper labels these closed-loop metrics. | 91.0 PDMS, higher is better | Two-DiT 89.6; no video co-training 88.2; no mask 90.9; DriveLaW 89.1; DiffusionDrive 88.1. The variant differences support shared parameters and video supervision. External methods differ in inputs; no uncertainty estimates accompany the table. e08e09e10 |
| Zero-shot transfer to nuScenes and Bench2Drive NAVSIM-trained models without target fine-tuning. nuScenes has 150 validation scenes; Bench2Drive uses its stated validation split. Averages cover 1, 2, and 3 seconds. | nuScenes: 0.83 m, 0.03%; Bench2Drive: 1.33 m, 1.19%. Average L2 displacement (m); average collision rate (%) | PWM: 3.99 m/0.36% and 2.80 m/3.76%; Two-DiT: 0.92 m/0.21% and 1.43 m/1.82%. These are horizon-based planning measures; Table 2 does not supply route completion or intervention counts. e08e11 |
| Future-video generation on nuScenes validation Reported comparison of driving-video generators; baseline training/sampling settings are not fully documented as matched. | 6.7; 81.7 FID; FVD, lower is better | Vista 6.9/89.4; Epona 7.5/82.8. Best reported values among listed methods, measuring visual quality rather than executed control or causal correctness. e12e19 |
| Video-action consistency NAVSIM and nuScenes, four-second DPVO ego motion aligned to each reference by a 2D similarity transform. | 0.12 NAVSIM; 0.14 nuScenes; 0.13 average. Average aligned L2; Table 4 does not print a unit | GT reference 0.08 average; no mask 0.15; Two-DiT 0.19. Scale, rotation, and translation are fitted, so agreement does not verify absolute motion scale. DPVO is evaluation-only. e13e19 |
| Trajectory inference and memory Default model versus no-mask coupled rollout and Two-DiT; timing configuration is incompletely specified. | 376 ms; 23.3/52.1 GB. Latency (ms); inference/training peak memory (GB) | No mask: 1623 ms, 26.3/54.4 GB; Two-DiT: 551 ms, 32.8/88.3 GB. The approximately 4.3× ratio is 1623/376 against no-mask UNIVERSE, not a same-checkpoint mode comparison. e07e09e14 |
| Future horizon and sampling ablations NAVSIM sweeps; defaults are eight future frames and two sampling steps. | Frames 4/8/12: 84.9/91.0/88.1. Steps 1/2/3: 13.7/91.0/91.0. PDMS, higher is better | Eight frames beat both tested alternatives; two steps match three at reported precision. One-step sampling fails sharply here. These sweeps do not establish a universally optimal horizon or sampler. e15 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Removing the need for future-video denoising reduces reported inference cost; shared parameters also reduce memory. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a different model configuration, not simply a switch applied to one checkpoint. The first column is trajectory inference latency, followed by inference and training peak memory. The default UNIVERSE row uses trajectory-only rollout. Section 4.5 explains that the no-mask model retains future-video denoising because its modalities can depend on each other. Two-DiT separates the denoising transformers and has the largest memory figures here. Keep the training and inference columns distinct: saved test-time video computation does not remove the large backbone or the video objective from training. e05e07e09e14
What it supports. The reported default is 376 ms with 23.3 GB inference and 52.1 GB training peak memory. The 1623 ms no-mask latency yields the approximately 4.3× ratio. Two-DiT takes 551 ms and 88.3 GB training peak memory, so the 4.3× claim should not be attached to that comparison.
Where the evidence stops. The table does not time joint and trajectory-only modes of the same masked checkpoint. Although H20 training hardware is named, timing batch size, warm-up, synchronization, and the inclusion of preprocessing or decoding remain unspecified.
Table 6. Eight video frames and two sampling steps are the best tested operating point, with a sharp one-step failure. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the two panels as different experimental knobs. The left varies future video frames used for supervision; it does not specify a count of control commands. The right varies flow sampling steps at inference. Both use the same NAVSIM metric columns as Table 1, with higher scores preferred. Start with PDMS, then inspect the components to see how broadly a configuration degrades. One-step sampling has very low comfort and progress as well as reduced collision/compliance scores. The eight-frame and two-step rows reproduce the default model’s component values, linking these sweeps to the main result. e03e08e09e15
What it supports. Four, eight, and twelve future frames yield PDMS 84.9, 91.0, and 88.1. One, two, and three sampling steps yield 13.7, 91.0, and 91.0. Thus more future frames do not monotonically help, while adding a third sampling step gives no aggregate improvement at the reported precision.
Where the evidence stops. These are limited sweeps without uncertainty bars or a matched-compute horizon analysis. Changing frame count changes the supervised temporal extent and token workload together; the table does not isolate why twelve frames underperform eight.
Table 4. Aligned visual odometry tests whether generated appearance and predicted ego motion agree in shape. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use Appendix C to understand what each error compares. DPVO reconstructs ego motion from a future video. For the GT reference row, reconstruction from real video is compared with the real trajectory; for model rows, reconstruction from generated video is compared with the predicted trajectory. Each reconstruction is first fitted to its own reference using scale, rotation, and translation, and then average L2 is measured over four seconds. Read NAVSIM and nuScenes separately before the average. The GT row provides a reconstruction reference, not a guaranteed universal error floor. The original table does not print a unit. e13e19
What it supports. UNIVERSE’s average is 0.13, between the GT reference’s 0.08 and the variants’ 0.15 and 0.19. On nuScenes, full and no-mask both score 0.14, so the full model does not improve every individual cell. The aggregate is consistent with shared-denoiser alignment under this particular measurement procedure.
Where the evidence stops. Fitting scale and pose to the reference can hide absolute motion errors. Low aligned L2 does not prove metric-scale dynamics or safe control. Appendix G also states that DPVO is an evaluation tool, not a training signal.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify missing side/rear cues with front-view input and costly video-backbone training. Multi-view input, lighter tokenizers, smaller backbones, and distillation remain future directions. e18
Table 8 mixes target-trained/fine-tuned baselines with zero-shot UNIVERSE. Variant tables lack seed variation or confidence intervals. Two-DiT changes memory requirements alongside parameter sharing, leaving resource confounds. e07e10e11e14e17
Figure 2 groups text with other conditions under C, whereas Equation (2) and Section 3.2 exclude text from masked self-attention. This report follows the explicit formulation and preserves the schematic ambiguity. e20
7.2 Questions for discussion
- Does video supervision improve difficult interactive cases after matching compute and reporting seed variation? (e07, e10, e11)
- How much video-action agreement survives evaluation without fitting scale to each future reference? (e13)
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include NAVSIM/OpenScene data, the Wan backbone, the specified optimizer schedule and mask, target validation data, and DPVO for the consistency diagnostic. Appendix G describes distinct data/model terms; external licenses were not independently checked. e04e08e09e19
Missing details include numerical video-loss weight, concrete trajectory-token count/normalization, precise flow solver, GPU count/training duration, latency setup, VAE freezing policy, training sample count, and exact Bench2Drive validation selection. Exact replication requires these choices. e03e06e08e09e14
Proposed checks: compare joint and trajectory-only outputs using identical action noise; repeat video co-training versus zero video weight across seeds with matched budgets. These test dependency consistency and transfer gains, respectively; neither was run. e04e05e07e10e11
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test trajectory invariance to future-video tokens
Reader-proposed check, not executed: use the same masked checkpoint, observations, instruction, trajectory noise, and flow time grid in joint and trajectory-only modes. Compare trajectory velocity predictions at every denoising step and final poses, setting a numerical tolerance before inspection. In joint mode, change only future-video noise and token values; also check that context activations remain unchanged. As a negative control, repeat with the no-mask variant, whose outputs may depend on video. Material differences in the masked model would falsify the claimed dependency isolation or reveal an unreported implementation path. Time both modes of the same checkpoint separately to isolate the actual mode-switch speedup. e03e04e05e07e14
Check 2: Separate useful future supervision from generic regularization
Reader-proposed check, not executed: train matched shared-DiT runs with correctly paired future-video targets, zero video-loss weight, and batch-shuffled future-video targets. Preserve initialization, action targets, token groups, optimizer schedule, and training budget; publish the chosen positive video weight because the paper omits its value. Repeat across at least three seeds. Evaluate NAVSIM PDMS and unchanged zero-shot nuScenes/Bench2Drive L2 and collision protocols with uncertainty estimates. If correctly paired video consistently beats both controls, the evidence favors learning relevant scene dynamics. If shuffled targets match the benefit, generic regularization remains a competing explanation for the co-training gain. e06e07e08e09e10e11
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, Figures 1–7, Tables 1–8, method and loss equations, the appendix visibility matrix and token configurations, DPVO alignment equations, implementation details, limitations, and asset/evaluation statements. All six final original crops were separately viewed at readable resolution. Figure 2’s condition grouping is qualified against Equations (2) and (7) and Section 3.2; its mask cells agree with the explicit row-query convention. Pages 10–13 contain the conclusion/references and were read in the complete text pass but not rendered for this visual pass. No separate supplements, source code, videos, or external assets were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work, including 2.1 and 2.2
- 3 Method, Sections 3.1–3.5
- 4 Experiments, Sections 4.1–4.6
- 5 Conclusion
- References
- A Supplementary Material
- B Additional Architecture Details, B.1–B.2
- C DPVO-based Verification of video-action Consistency
- D Additional Comparison on nuScenes
- E Additional Zero-Shot Visualization
- F Limitations
- G Licenses for Existing Assets
- H Broader Impacts
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure gap was addressed by inspecting the retained PDF: all seven figures and eight tables were visually reviewed. All six supplied text chunks were read completely, including the embedded appendices.
- Identity/edition scope: arXiv:2607.05133v1 [cs.CV], 6 July 2026. The title and all ten authors match the catalog. Xiaomi EV appears on the title page but is omitted from the catalog affiliation string. No other revision was supplied or compared.
- Code, checkpoints, external cited works, and linked assets were not inspected; no experiments were reproduced. Separate supplements beyond the embedded appendix remain unverified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
The exact catalog title appears with ten authors in matching order, arXiv:2607.05133v1 [cs.CV], 6 July 2026. Affiliations are University of Twente, Xiaomi EV, University of Cambridge, and University of Bath.
Go to primary source ↓e02PDF pp. 1–3, Abstract and Sections 1–2
The motivation is dense video supervision for transferable driving dynamics; shared denoising is contrasted with cascaded or separate video/action modules.
Go to primary source ↓e03PDF p. 4, Figure 2 and Section 3.1, Equations (1)–(3)
Historical images, velocity, and optional instruction condition pose/video generation. Text uses a frozen encoder and cross-attention. Shared DiT blocks have modality-specific projections/heads; historical trajectories are excluded.
Go to primary source ↓e04PDF p. 5, Section 3.2; p. 14, Appendix B.1, Equation (7)
Query-to-key mask rows C,V,A are 100,110,101. Full intra-modality attention is retained; neither future modality reads the other directly or through context.
Go to primary source ↓e05PDF p. 5, Section 3.3; pp. 14–15, Appendix B.2 and Table 7
Requested future token groups determine inference mode with the same weights. Removing future video preserves trajectory attention dependencies.
Go to primary source ↓e06PDF p. 5, Section 3.4, Equations (4)–(6)
Uniform flow time interpolates Gaussian noise toward clean targets. Weighted video and trajectory squared velocity errors train the shared DiT. A numerical default video weight is not given.
Go to primary source ↓e07PDF pp. 5–6, Section 3.5; p. 8, Section 4.5 opening
No-mask allows future mutual attention and requires coupled inference; no-video sets lambda_vid to zero; Two-DiT retains conditions/objectives with separate transformer parameters. Other schedules/protocols are stated to match.
Go to primary source ↓e08PDF p. 6, Section 4.1, dataset paragraphs
NAVSIM v1 uses NC, DAC, TTC, comfort, progress and aggregate PDMS. Input is front-camera imagery without map/box/depth/occupancy/LiDAR supervision. nuScenes has 150 validation scenes. Targets are evaluated with horizon L2/collision metrics.
Go to primary source ↓e09PDF p. 6, Section 4.1, Implementation Details
H20, AdamW, bf16, learning rate 10^-4, weight decay 0.01, 20k steps/batch 80 plus 10k/effective batch 640, 1k warm-up from 10^-3 base rate, Wan2.2-TI2V-5B, four history/eight future frames at 832×480, and default two-step trajectory-only inference are stated.
Go to primary source ↓e10PDF p. 6, Table 1, PDMS column and UNIVERSE variant rows
PDMS is 91.0 full, 90.9 no mask, 88.2 no video co-training, 89.6 Two-DiT, 89.1 DriveLaW, 88.1 DiffusionDrive. Full-model NC/DAC/TTC/comfort/progress are 99.1/97.6/98.5/100/83.6.
Go to primary source ↓e11PDF p. 8, Table 2, both datasets, Avg. columns and variant rows
All rows are NAVSIM-trained without target fine-tuning. Full nuScenes averages are 0.83 m/0.03%, Bench2Drive 1.33 m/1.19%; no-mask collision averages 0.06%/1.79%, no-video 0.35%/1.57%, Two-DiT 0.21%/1.82%. PWM averages are 3.99 m/0.36% and 2.80 m/3.76%.
Go to primary source ↓e12PDF p. 8, Table 3 and Section 4.4
nuScenes FID/FVD are 6.7/81.7 for UNIVERSE, 6.9/89.4 for Vista, 7.5/82.8 for Epona. No uncertainty or fully matched video-generation protocol is supplied.
Go to primary source ↓e13PDF p. 9, Table 4 and Section 4.5 consistency paragraph; p. 15, Appendix C, Equations (8)–(9)
DPVO reconstructs motion from real/generated video, aligned to ground-truth/predicted references by independently optimized scale, rotation, and translation. Four-second average L2 is 0.08 GT reference, 0.13 full, 0.15 no mask, 0.19 Two-DiT. The table does not label a unit.
Go to primary source ↓e14PDF p. 9, Table 5 and Section 4.5 visibility/efficiency paragraphs
Full/no-mask/Two-DiT latency is 376/1623/551 ms, inference peak memory 23.3/26.3/32.8 GB, training peak 52.1/54.4/88.3 GB. No-mask retains future-video denoising.
Go to primary source ↓e15PDF p. 9, Table 6 and Section 4.6
Future frames 4/8/12 give PDMS 84.9/91.0/88.1; sampling steps 1/2/3 give 13.7/91.0/91.0. Defaults are eight frames and two steps.
Go to primary source ↓e16PDF p. 7, Figures 3–4; p. 16, Figure 5; p. 17, Figures 6–7
Qualitative examples show video/trajectory alignment, bypass versus stop, following/lane-change/curved-road cases, and zero-shot nighttime/foggy scenes; these are not aggregate success statistics.
Go to primary source ↓e17PDF p. 15, Appendix D; p. 16, Table 8, caption and Fine-tune/Input columns
The extended nuScenes comparison includes target-trained/fine-tuned baselines and zero-shot UNIVERSE; the asterisk denotes front-camera-only inputs.
Go to primary source ↓e18PDF pp. 16–17, Appendix F
The authors identify front-view coverage and video-backbone training cost as limitations, proposing multi-view input and smaller models/tokenizers.
Go to primary source ↓e19PDF pp. 17–18, Appendix G; p. 18, Appendix H
Asset-specific licenses are described; DPVO is evaluation-only. Literature baseline settings are retained as closely as possible. Broader impacts discuss transfer benefits and computational cost.
Go to primary source ↓e20PDF p. 4, Figure 2 condition grouping versus Equation (2); p. 5, Section 3.2 first paragraph
The schematic brackets text, image, and velocity conditions near C. The explicit formulation defines self-attention context from images/velocity and excludes cross-attention text tokens from the mask.
Go to primary source ↓8.5 Primary sources
UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation ↗
PDF · 9,099 extracted words
Source fingerprint
bc84493af7a2b719598f370d13d99a59e1d92e44bc6b27876be423305004ec5e