VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
1. Paper overview
In one sentence: Video pre-training supports trajectory imitation, but stronger synthesis and closer expert matching do not guarantee safer closed-loop driving. e-overviewe-actione-opene-diagnostice-limits
| At a glance | What to know |
|---|---|
| Research problem | Source description Can unlabeled driving-video prediction provide useful features for decisions? The paper separates perceptual plausibility, semantic transfer and driving behavior, because matching recorded expert trajectories cannot measure how errors change subsequent observations. e-overviewe-simulator |
| Core mechanism | |
| A key reported result | Frontal collision avoidance in NeuroNCAP: VaVAM-L: 2.375 NNS, 56.80% collisions, 40.2% progress. NeuroNCAP score ↑; collision rate ↓; progress ↑. Neural-rendered closed-loop frontal hazards; VaVAM-L uses a front camera without trajectory post-processing; compared UniAD is the authors' reproduced, post-processed baseline. UniAD: 1.18 NNS, 78.8% collisions, 47.1% progress. The reader-calculated collision reduction is 22.0 percentage points, about 27.9% relative, accompanied by 6.9 percentage points less progress. This is scenario-specific, not overall superiority: post-processed VAD has lower all-scenario collisions (50.70%) than VaVAM-L (57.90%). e-closede-diagnostic |
| Reading caution | Reader analysis The authors attribute part of the safety tradeoff to rigid imitation of expert-derived guidance. That explanation is a hypothesis: the presented comparisons do not isolate guidance construction, pre-training benefit or flow matching against alternatives. e-hypothesise-opene-diagnostic |
Core contributions
- Reader analysis
The experiments expose a tradeoff between tighter trajectory following and collision avoidance, motivating progress and route-deviation metrics alongside safety scores. e-diagnostice-metric-gap
Figure 2. The action expert repeatedly refines a trajectory while reading causally available video features. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) upward: the scattered green points and noisy coordinate tokens enter the Action Encoder, command embeddings enter the Action Transformer, and the Action Decoder supplies the denoising prediction. The dashed return arrow represents repeated refinement, not a new camera observation. Panel (b) makes the information boundary explicit: keys run horizontally and queries vertically. The visual blocks form a causal triangle. An action block can read visual tokens up to its observation time and all waypoints within its own block, but not another action-time block. VaViM interacts with the expert through attention across layers; the action stream keeps a smaller feed-forward width. e-actione-maske-flowe-implementatione-limits
What it supports. The mask lets one training clip supervise actions at different observation times without exposing future visual tokens. It also permits a compact trajectory expert to use representations across the video backbone's depth. This supports observed-history-conditioned action generation; no branch here evaluates imagined future driving outcomes.
Where the evidence stops. The diagram compresses the numerical update: the decoder predicts a vector field used by Eq. (9). Eq. (7) and noise initialization place noise at tau=0; the following prose reverses those endpoints. That inconsistency is preserved here.
2. Motivation
2.1 The problem and the proposed response
Can unlabeled driving-video prediction provide useful features for decisions? The paper separates perceptual plausibility, semantic transfer and driving behavior, because matching recorded expert trajectories cannot measure how errors change subsequent observations. e-overviewe-simulator
2.2 What this reading follows
Imagine a car learning visual patterns from ordinary driving videos, then learning how to act from recorded ego-motion. VaViM supplies the first component: a causal model of discrete image tokens. VaVAM adds a compact expert that turns camera history and a directional command into a three-second trajectory. The paper's most useful lesson emerges when its evaluations are read together. Increasing scale helps generated images and best-of-five trajectory matching, yet simulated hazard avoidance behaves differently. These visuals trace that gap from the attention mask to the safety tables, while keeping source inconsistencies and the absence of predictive planning explicit. e-overviewe-actione-opene-diagnostice-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The separate video backbone and action expert support Dual-system. Actions are denoised from observed-history features, rather than inferred from predicted before/after states or jointly emitted with future video; Other mechanisms and Outside quadrants fit this reactive design. The broad WAM label describes video-to-action transfer, while the authors explicitly reserve predictive planning for future work. Efficient sampling is architecturally motivated; real-time control is not established by a latency benchmark. e-actione-maske-implementatione-limits
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a visual sequence before learning a driving trajectory
VaViM's first compression step operates on individual images. The pre-trained LlamaGen tokenizer converts each frame to a grid of discrete code indices; the temporal model then learns relationships among those indices across frames. The separation matters: reconstructing a sharp image and predicting how a scene evolves are different burdens. VaViM learns next-token probabilities with teacher forcing, while spatial and temporal embeddings identify where tokens belong. The training sequence first uses OpenDV video, then adapts with a mixture that includes the downstream driving domains. No ego-trajectory supervision is needed for those video objectives. The subsequent action-learning stage does require recorded trajectories. This explains how the method can exploit unlabeled video at scale without implying that the complete driving system learns actions without demonstrations. e-videoe-datae-mixe-implementation
5.2 Use scene features to transport noise into a path
The action expert receives a noisy sequence of future ego-positions, not discrete video tokens as its prediction target. Its encoder adds information about the command, action time and flow step; joint attention supplies the video representation at multiple layers. During training, Eq. (8) fits the vector field that should transform interpolated trajectories toward demonstrations. During inference, Eq. (9) repeatedly adds a predicted increment, starting from Gaussian noise and using ten Euler steps. Eq. (7) supports noise at zero and expert actions at one, despite the reversed sentence immediately below it. Keeping that distinction explicit is necessary for reproduction. The result is six future waypoints. The source leaves the video backbone's exact freezing policy during this action-training stage unspecified, and does not use generated future videos to rank candidate actions. e-actione-maske-flowe-implementatione-limits
5.3 Evaluate the action after it changes what the model sees
Reader analysis: the open-loop and closed-loop tables answer different questions. Best-of-five minADE asks whether one sampled trajectory resembles a recorded expert future. NeuroNCAP instead executes decisions in a neural-rendered scene and returns new observations, including scripted hazards that can demand a departure from the recorded route. Even collision avoidance alone is insufficient: the authors accidentally obtained a high-scoring model that left the road and missed the hazards. Their added progress and closest-point deviation metrics expose this loophole. Table 6 then shows why a single scaling claim is misleading: more imitation data can improve route adherence while worsening collisions. Rigid command following is the authors' proposed explanation, but the comparisons do not isolate it. A useful follow-up must preserve progress while testing whether safer deviations become possible. e-opene-simulatore-metric-gape-diagnostice-hypothesis
5.4 Training and inference
During training
Video learning uses teacher-forced next-token cross-entropy. OpenDV provides over 1,700 hours before filtering: intros/outros are removed, only Full HD videos retained, and overlapping eight-frame clips sampled at 2 Hz. Target adaptation mixes OpenDV, nuPlan and nuScenes; imitation requires synchronized expert ego-trajectories from the latter two datasets. e-videoe-datae-mix
Maximal Update Parametrization transfers settings across widths, and an empirical loss law models data/model scaling. Its flagship prediction error is reported as 0.003 (0.06%); this concerns token loss, not driving safety. Flow matching then regresses a conditional transport field from interpolated noise/expert trajectories, using a beta-distributed schedule. e-scalinge-flowe-implementation
During inference
Initialize the six-waypoint trajectory with Gaussian noise and apply ten forward-Euler updates conditioned on observations and the command. NeuroNCAP executes model decisions in simulation and renders subsequent observations. The reported pipeline does not score imagined future videos, use a reward model, or demonstrate physical-road deployment. e-flowe-simulatore-limits
5.5 Implementation flow
- Compress and predict video
A pre-trained LlamaGen VQGAN tokenizer maps each 512×288 frame to 576 discrete tokens. VaViM uses a 24-layer GPT-2-style decoder, learned spatial/temporal embeddings and causal attention; eight frames occupy 4,608 tokens. Tokenizer pre-training is described separately from the imported tokenizer used here. e-videoe-implementation
- Condition a trajectory expert
An MLP encodes noisy waypoints with flow-time, action-time and command embeddings. Layer-wise joint attention supplies video features, while a linear decoder predicts a vector field. Action queries see available camera history and their own waypoint block; visual queries remain causal and do not consume actions. e-actione-mask
- Keep action sampling compact
The expert matches the backbone's attention dimensions but reduces its MLP dimensionality fourfold. This supports repeated denoising passes without requiring the entire action stream to use the video's feed-forward width. e-maske-implementation
6. Experiments & results
VaViM learns camera-history representations by predicting discrete video tokens; VaVAM adds a flow-matching expert that generates driving trajectories. Larger models improve image-distribution fidelity and best-of-five trajectory matching, yet closed-loop collision avoidance does not improve consistently. This is a study of generative representation transfer to reactive imitation, with simulated driving evaluation and an explicit gap to predictive planning.
6.1 Read the original evidence
Table 2. Video FID improves with model size, but the oracle uses real future images. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a dataset block, then follow FID@1 through FID@4 horizontally to see how distributional distance changes across generated frames. Compare S, B and L vertically at a fixed frame index to assess model scale. Section 5.1.1 specifies four context frames and four generated frames, with DINOv2 features and the context images as the reference distribution. The top LlamaGen row encodes and decodes actual future images, so it removes forecasting uncertainty. Its printed 'Oracle upper-bound' describes a quality ceiling, while the FID values themselves are better when lower. It is not an action model or a fair prediction baseline. e-fide-limits
What it supports. At the fourth frame, scaling from S to L lowers KITTI FID from 34.5 to 24.7 and nuScenes FID from 39.8 to 26.1. All four reported horizons favor the largest model. The remaining gap to the oracle shows that better capacity does not eliminate forecasting and reconstruction errors.
Where the evidence stops. This is per-frame image-distribution evaluation with an unusual context-frame reference. It does not directly measure temporal physics, trajectory feasibility or safety. The paper does not report uncertainty for these values.
Table 3. Better video synthesis does not imply better semantic probing performance. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the three dataset columns; every entry is mIoU, so higher is better. KITTI-1f uses a single-frame setup, whereas KITTI uses the downsampled video setup. The lower two groups compare VaViM before and after target-domain video fine-tuning, not two segmentation-training runs. Section 5.1.2 uses layer-12 features and ten sampled patch features per image in the Humming-bird protocol. Cityscapes and KITTI were absent from the video-training mixture. Compare B with L within each group to test whether the synthesis scaling trend carries over; then consult individual DINO rows rather than treating all discriminative baselines as interchangeable. e-semanticse-fide-implementation
What it supports. On Cityscapes, fine-tuned VaViM-B scores 20.8 mIoU and L scores 18.4 despite L's stronger synthesis results. DINOv2-B reaches 43.4. The source also contains an exception to its blanket DINO-superiority statement: DINOv2-L scores 13.0, below every VaViM row in that column.
Where the evidence stops. This probe measures accessible semantic features under one extraction protocol, not all information in the backbone. 'Zero-shot' concerns transfer beyond the video-training datasets; it should not be read as a label-free driving-safety evaluation.
Table 4. Matched-data scaling improves best-sample imitation accuracy. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The parameter column separates the video backbone from the action expert, for example 185+21 million for S. The data column varies the action-learning data amount as labeled in the table. Read the two dataset columns as minADE with five sampled trajectories: compute each candidate's average pointwise distance from the recorded expert path, then retain the closest candidate. This is not a separate nearest prediction at every waypoint. First compare the 139K rows across model sizes, which hold the printed data count fixed. Then inspect the intermediate rows within S or B; data increases do not produce uniformly monotonic improvements. e-opene-sizese-implementation
What it supports. At 139K data, nuScenes error falls from 1.00 for S to 0.85 for B and 0.80 for L; nuPlan falls from 0.68 to 0.53 and 0.52. These measurements establish better coverage of recorded expert behavior among five samples. They do not establish which sampled action will be safe when executed.
Where the evidence stops. The source gives neither an exact evaluation scene list nor uncertainty intervals here, and the table omits a distance unit. Best-of-five selection uses the recorded future; it is not an available inference-time selection rule.
Table 5. VaVAM-L leads the listed frontal comparison, while post-processed VAD performs better overall. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each row as a complete evaluation configuration. The post-processing column distinguishes planners with additional trajectory optimization; the dagger on UniAD denotes a run reproduced by the paper's authors. The left metric block is NeuroNCAP score, where larger is better; the right block is collision percentage, where smaller is better. Within each block, compare average, stationary, frontal and side columns separately. Gray cells emphasize the absence of VaVAM trajectory post-processing and its limited side view. Its front camera differs from the baselines' surrounding-camera inputs. Also, 'raw data' still includes recorded expert ego-trajectories for action learning, despite avoiding dense manual semantic annotations. e-closede-simulatore-datae-metric-gap
What it supports. VaVAM-L reports frontal NNS 2.38 and 56.80% collisions, compared with reproduced UniAD's 1.18 and 78.8%. Across all scenarios, however, post-processed VAD achieves 50.70% collisions and NNS 2.75, versus VaVAM-L's 57.90% and 2.46. Frontal success therefore supports a narrower claim than overall driving superiority.
Where the evidence stops. Cameras, additional sensor inputs and post-processing differ, so this is not a controlled architecture ablation. Section 5.2.2 calls collision rate the percentage 'without collision'; that wording conflicts with these downward arrows and the reported interpretation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Future-frame generation on KITTI and nuScenes Four observed frames followed by four generated frames; fine-tuned models; DINOv2-feature FID against context-frame features. | VaViM-L: KITTI 24.7; nuScenes 26.1. FID@4 ↓ | VaViM-S: 34.5/39.8; VaViM-B: 30.0/38.6; tokenizer oracle: 8.1/10.4. Scaling improves this image-distribution metric. The oracle has access to real future frames; neither FID nor this comparison measures collision avoidance. e-fid |
| Semantic transfer to Cityscapes Humming-bird probing of layer-12 features; ten sampled patch features per image; Cityscapes absent from video-training mix. | Fine-tuned VaViM-B 20.8; VaViM-L 18.4. mIoU ↑ | DINOv2-B 43.4; DINOv2-L 13.0. The larger video model is worse in this probe. The table also contradicts the prose's claim that DINO models always outperform VaViM. e-semantics |
| Open-loop trajectory matching nuScenes/nuPlan, best of five sampled three-second trajectories; Table 4 data count 139K; exact evaluation scene list not supplied. | VaVAM-L: 0.80/0.52. minADE5 ↓; table does not print a distance unit | VaVAM-S: 1.00/0.68; VaVAM-B: 0.85/0.53. Matched-data scaling improves best-sample accuracy; the metric does not identify which sample a deployed controller should execute. e-opene-implementation |
| Frontal collision avoidance in NeuroNCAP Neural-rendered closed-loop frontal hazards; VaVAM-L uses a front camera without trajectory post-processing; compared UniAD is the authors' reproduced, post-processed baseline. | VaVAM-L: 2.375 NNS, 56.80% collisions, 40.2% progress. NeuroNCAP score ↑; collision rate ↓; progress ↑ | UniAD: 1.18 NNS, 78.8% collisions, 47.1% progress. The reader-calculated collision reduction is 22.0 percentage points, about 27.9% relative, accompanied by 6.9 percentage points less progress. This is scenario-specific, not overall superiority: post-processed VAD has lower all-scenario collisions (50.70%) than VaVAM-L (57.90%). e-closede-diagnostic |
| Data-scaling safety diagnostic VaVAM-S on frontal NeuroNCAP scenarios, comparing Table 6 data counts 38K and 139K. | 38K → 139K: collisions 44.4% → 50.0%; deviation 2.616 → 2.022; progress 39.8% → 42.5%. Collision rate ↓; mean route deviation ↓; progress ↑ | The same model follows the reference more closely while colliding more frequently. This is a scaling diagnostic, not an isolated causal ablation. Reported rows have no confidence intervals or seed variation. e-diagnostic |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 6. The scaling diagnostic makes route adherence and safety separately visible. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read all four metric columns together. NNS rewards collision avoidance and reduced impact severity; collision rate records hazard outcomes. Mean deviation measures instantaneous distance to the closest point on the reference route, rather than matching equally timed waypoints. Progress is the clipped fractional reduction in distance to the goal, displayed as a percentage. Hold S fixed and move from 38K to 139K data to see a clear counterexample to 'more data makes driving safer.' Then compare the 139K S, B and L rows: larger models do not produce a monotonic safety progression. The baseline also makes the cost in progress visible. e-diagnostice-metric-gape-simulatore-hypothesis
What it supports. For S, collisions rise from 44.4% to 50.0% as deviation falls from 2.616 to 2.022 and progress rises from 39.8% to 42.5%. For L, the 56.80% collision rate improves on UniAD's 78.8%, but progress is 40.2% rather than 47.1%. Safety and route completion require separate judgments.
Where the evidence stops. This is a model/data scaling diagnostic, not a controlled ablation of video pre-training or command design. The authors' rigid-imitation explanation remains a hypothesis, and no seed variation or confidence intervals are supplied.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors attribute part of the safety tradeoff to rigid imitation of expert-derived guidance. That explanation is a hypothesis: the presented comparisons do not isolate guidance construction, pre-training benefit or flow matching against alternatives. e-hypothesise-opene-diagnostic
Front-only perception limits side-hazard visibility; urgent braking is weak in some scenes. Video prediction remains unused for planning, and physical understanding is not comprehensively tested. e-limits
Collision scores can reward leaving the road: a noisy model exceeded 4.0 NNS by missing hazards. Progress and closest-point route deviation reveal behavior hidden by collision rate alone. Section 5.2.2's 'without collision' wording conflicts with the downward-arrow collision tables; this report follows the tables. e-metric-gape-simulator
7.2 Questions for discussion
- Would matched-capacity action experts still benefit from video pre-training under identical data and compute?
- Would independently specified route guidance permit safer deviations without reducing useful progress?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the specified tokenizer, video clips and ego-trajectories, joint mask and ten-step sampler. Video AdamW uses base learning rate 0.0041 and batch 384; the action learning rate is 0.0194. The flagship video run reports 192 H100 GPUs for approximately 25 hours (4,800 GPU-hours), with Lightning and DeepSpeed stage 2. e-datae-maske-computee-flow
Resolve source inconsistencies before training: the printed 40%/58.72%/1.28% mixture does not match its 2,385,300/2,765,278/76,120 clip counts, and the compute-optimal frontier swaps D/N between Figure 5 and prose. Freezing policy, beta parameters, trajectory normalization, action-training duration, evaluation seeds and software versions are not explicitly specified. e-mixe-scalinge-implementatione-computee-opene-diagnostic
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate video pre-training under matched action-learning budgets
Reader-proposed check, not performed: train the same VaVAM-S action architecture with either the supplied-style pre-trained video backbone or a randomly initialized backbone. Keep the tokenizer, camera history, mask, expert-trajectory split, command construction, optimization budget and inference sampling identical. Explicitly declare and match which modules are trainable, since the PDF does not settle freezing. Repeat with multiple seeds and evaluate both minADE5 and paired frontal NeuroNCAP scenarios, reporting collisions, NNS, progress and deviation. A repeatable improvement in safety without losing progress would support transfer to useful driving behavior; an improvement only in minADE5 would support the narrower representation-to-imitation claim. e-videoe-maske-implementatione-opene-diagnostic
Check 2: Test whether independent route guidance permits safe deviations
Reader-proposed check, not performed: first specify how the paper's three-way commands are derived from a reference route, an implementation detail requiring clarification. Compare the original expert-derived guidance with independently defined nominal routes in demonstrations where experts safely deviate and rejoin. Hold model size, video checkpoint, number of action examples, hazard scenarios and sampling policy fixed. Include no-hazard paired scenes to detect gratuitous departures. The rigid-imitation hypothesis predicts fewer collisions at comparable goal progress under independent guidance, with deviation increasing mainly where avoidance is needed. A collision gain caused by stopping short or leaving the route without useful progress would fail this check. e-actione-metric-gape-diagnostice-hypothesis
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page and every body page through the conclusion (PDF pp. 1–19), including Figures 1–8, Tables 1–6, equations, data preparation and implementation/hardware details. All six final original crops were opened and checked; the semantic table's bottom edge and diagnostic table's right edge were corrected and re-inspected. Figure 2's arrows and query/key blocks were cross-checked against Sections 3.2 and 4.4. The flow-time prose/equation conflict, scaling-frontier D/N conflict, mixture-count inconsistency and collision-rate wording conflict remain explicit. Contributions/references on pp. 20–26 were read as text; no appendix or separate supplement was supplied. The sixth visual is a scaling diagnostic rather than an isolated mechanism ablation.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Section 1 Introduction (pp. 1–2)
- Sections 2.1–2.3 Related work and evaluation (pp. 2–5)
- Sections 3.1–3.2 Video model and action expert (pp. 5–8)
- Sections 4.1–4.5 Data, scaling, fine-tuning, imitation learning and implementation (pp. 8–13)
- Sections 5.1–5.2 Generation, semantics, open-loop and closed-loop experiments (pp. 13–19)
- Section 6 Conclusion and Acknowledgements (p. 19)
- Detailed Contributions and References (pp. 20–26)
Outside the original text pass
- All eight supplied text chunks were read individually, covering all 26 PDF pages. No appendix is present.
- Identity: inspected arXiv:2502.15672v1, dated 21 February 2025; all 17 authors agree with the catalog. No later revision or alternative edition was supplied or compared. The manifest's observedTitle retains extracted 'VaV AM' spacing; the rendered title uses 'VaVAM'.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The retained PDF was visually inspected on pages 1–19, including every method/result evidence page and all eight figures and six tables. Reference/contribution pages 20–26 were read as text only.
- No code, external project pages, separate supplements or model weights were inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, affiliation footnotes and arXiv margin
The title and all 17 authors match the catalog. The artifact is arXiv:2502.15672v1, 21 February 2025. The manifest's 'VaV AM' spacing is extraction typography; the rendered title reads 'VaVAM'.
Go to primary source ↓e-overviewPDF pp. 1–2, Abstract and Section 1
The work investigates transferring autoregressive video representations to an imitation-trained action expert, comparing generation, semantics and driving performance.
Go to primary source ↓e-sizesPDF p. 3, Table 1, all model groups
Video backbones have 185M, 318M and 1,200M parameters; action experts add 21M, 38M and 150M. Video pre-training, mixed-data fine-tuning and action learning are distinct stages.
Go to primary source ↓e-videoPDF pp. 6–7, Sections 3.1.1–3.1.2, Eqs. (1)–(5)
A vector-quantized image tokenizer supplies discrete codes. GPT-2-style causal decoding learns next-token cross-entropy with teacher forcing, spatial/temporal embeddings, tied weights and a sampling KV cache.
Go to primary source ↓e-actionPDF p. 5, Figure 1; pp. 7–8, Section 3.2 and Figure 2a
Observed camera history and left/right/straight commands condition an action encoder, joint-attention transformer and linear vector-field decoder. Actions are future BEV ego-positions; the figure shows iterative refinement of six waypoints.
Go to primary source ↓e-maskPDF p. 8, Figure 2b and Section 3.2 architecture bullets
Query rows and key columns show causal visual attention, no visual-to-action conditioning, and each action block attending available visual history and its own action block. Separate action-time blocks do not attend each other.
Go to primary source ↓e-dataPDF p. 9, Section 4.1, OpenDV/nuPlan/nuScenes paragraphs
Only front-camera RGB is used. OpenDV processing retains Full HD videos, trims intros/outros, extracts 512×288 frames at 10 FPS and forms eight-frame 2-Hz clips. nuPlan and nuScenes supply synchronized ego-position trajectories.
Go to primary source ↓e-scalingPDF pp. 10–11, Section 4.2, Figure 5 and Eq. (6)
Maximal Update Parametrization transfers hyperparameters across widths. The fitted validation-loss law uses clip count and non-embedding parameter count. Reported flagship loss prediction error is 0.003 (0.06%). Figure 5's frontier legend writes N=1.38 D^0.96, whereas p. 11 writes D=1.384 N^0.962.
Go to primary source ↓e-mixPDF p. 11, Section 4.3, fine-tuning bullets
The stated OpenDV/nuPlan/nuScenes mix is 40%/58.72%/1.28%, alongside 2,385,300/2,765,278/76,120 clips. These counts do not reproduce the printed percentages. Alternative mixture composition remains future work.
Go to primary source ↓e-flowPDF p. 12, Section 4.4, Eqs. (7)–(9) and adjacent prose
Eq. (7) interpolates from Gaussian noise at tau=0 to expert action at tau=1; the following sentence reverses these endpoints. Eq. (9) and the stated noise initialization support ten forward-Euler steps from tau=0. Eq. (8) regresses the conditional transport vector field.
Go to primary source ↓e-implementationPDF pp. 12–13, Section 4.5
The pre-trained 72M LlamaGen stride-16 tokenizer has 16,384 codes, producing 576 tokens per frame. The 24-layer video model uses up to 4,608 tokens. The action expert reduces MLP dimensionality fourfold; the beta schedule parameters and freezing policy are not explicitly specified.
Go to primary source ↓e-computePDF p. 12, Section 4.5 optimizer paragraph; p. 13, action settings and Training infrastructure
Video AdamW uses base learning rate 0.0041, weight decay 1e-7, betas (0.9,0.95), gradient clipping 1.0 and batch size 384. Action learning rate is 0.0194. VaViM-L training uses 192 H100 GPUs, about 25 hours/4,800 GPU-hours, Lightning and DeepSpeed stage 2; software versions are not given.
Go to primary source ↓e-fidPDF p. 13, Table 2 and Section 5.1.1
Four context frames condition four generated frames. DINOv2-feature FID uses context frames as the reference distribution. At frame four, S/B/L score 34.5/30.0/24.7 on KITTI and 39.8/38.6/26.1 on nuScenes; tokenizer-oracle scores are 8.1/10.4.
Go to primary source ↓e-semanticsPDF p. 14, Table 3; p. 15, Section 5.1.2
Humming-bird probing uses layer-12 features and ten sampled patch features per image. Fine-tuned B/L Cityscapes mIoU is 20.8/18.4; DINOv2-B/L is 43.4/13.0. The prose's blanket DINO superiority does not hold for the DINOv2-L row.
Go to primary source ↓e-openPDF p. 15, Section 5.2.1; p. 16, Table 4
minADE with k=5 selects the closest sampled trajectory by mean pointwise L2 distance. At data count 139K, S/B/L score 1.00/0.85/0.80 on nuScenes and 0.68/0.53/0.52 on nuPlan. Intermediate data-count rows are not monotonically better.
Go to primary source ↓e-simulatorPDF pp. 16–17, Section 5.2.2, NeuroNCAP, collision metrics and baselines
Neural rendering supplies feedback after simulated model decisions, with stationary, frontal and side hazards. Eq. (10) rewards avoidance and impact-speed reduction. The text calls collision rate the percentage without collision, conflicting with the tables' downward arrows and interpretation.
Go to primary source ↓e-closedPDF p. 17, Table 5, VaVAM-L, UniAD dagger and post-processed VAD rows; caption
VaVAM-L has frontal NNS 2.38 and collision rate 56.80%, versus reproduced UniAD's 1.18/78.8%. VaVAM-L's all-scenario average is 2.46/57.90%, versus post-processed VAD's 2.75/50.70%. Cameras and post-processing differ; dagger means reproduced by the paper's authors.
Go to primary source ↓e-metric-gapPDF p. 17, Limitations of the benchmark and Eq. (11)
An off-road noisy model scored above 4.0 NNS by avoiding scripted hazards. Added measures are closest-point mean route deviation and fractional reduction of goal distance, clipped below at zero.
Go to primary source ↓e-diagnosticPDF p. 18, Table 6, UniAD and VaVAM-S/B/L rows
In frontal scenarios, VaVAM-S changes from 44.4% collisions, 2.616 deviation and 39.8% progress at 38K data to 50.0%, 2.022 and 42.5% at 139K. VaVAM-L at 139K achieves 2.375 NNS, 56.80% collisions, 1.541 deviation and 40.2% progress; UniAD progress is 47.1%.
Go to primary source ↓e-hypothesisPDF pp. 18–19, Section 5.2.2 discussion and Section 6 future-work bullets
The authors hypothesize that stronger imitation encourages rigid adherence to expert-derived guidance, impairing responses to adversarial vehicles. They propose decoupling command paths from demonstrated trajectories; the study does not experimentally isolate this cause.
Go to primary source ↓e-limitsPDF p. 17, qualitative limitations; p. 19, Section 6
VaVAM rarely stops or brakes urgently in some hazardous scenes, uses only a front camera, and does not predict occupancy. The authors explicitly say video prediction is not yet used for planning/control, identify the missing reward model, and call for better physical-understanding evaluation and finer tokenization.
Go to primary source ↓8.5 Primary sources
VaViM and VaV AM: Autonomous Driving through Video Generative Modeling ↗
PDF · 13,589 extracted words
Source fingerprint
1a794642eaa61c2d67884b67a7b280ce3eedbfda435d3daefaeb88c99702d0dc