Diffusion Transformer World-Action Model for AV Scene Prediction
1. Paper overview
In one sentence: Calibrated latent diffusion improves the appearance distribution of compact driving predictions, while a separate re-anchored jump model improves coarse motion at the cost of accumulating blur. e-architecturee-realisme-calibratione-motione-jump-resulte-limits
| At a glance | What to know |
|---|---|
| Research problem | Source description Which representations and prediction objectives support compact action-conditioned driving models, and which metrics distinguish realistic images from accurate dynamics? A single distortion score can reward an averaged, blurry future while hiding the appearance benefit of generative prediction. e-architecturee-realisme-qualitative |
| Core mechanism | Source description The paper combines a six-encoder action-probe benchmark, controlled DiT diagnosis, a calibrated perception–distortion comparison, and a re-anchored transition model for coarse motion. e-encodere-diagnosise-realisme-jump-method |
| A key reported result | Decoded-frame distribution at eight seconds: Calibrated diffusion: 0.078 KID, 162.5 FID, 0.260 CosSim. KID and FID (lower); latent CosSim (higher). SD-VAE pipeline, t+16, 150-scene held-out nuScenes test split; Table 2 point values. Exact window mapping across plots is unresolved. Direct: 0.375, 370.8, 0.471; raw diffusion: 0.294, 341.9, 0.233. Halfway latent interpolation: 0.084, 166.6, 0.316. The stated 4.8-fold KID advantage trades away paired latent agreement and depends strongly on calibration. The prose separately gives three-seed KID 0.076±0.005; do not attach that uncertainty to the table point. Distribution agreement is not driving success. e-realisme-calibratione-evaluatione-data |
| Reading caution | Reader analysis The diagnosis text claims 88.5% gap recovery, while Figure 3 labels 99%, 100% and 97%. Figure 4 uses 600 windows, Figure 5 uses 150, and plotted values do not consistently match Table 2. These discrepancies remain unresolved, rather than forming a single pooled estimate. e-diagnosise-evaluation |
Core contributions
- Source description
The paper combines a six-encoder action-probe benchmark, controlled DiT diagnosis, a calibrated perception–distortion comparison, and a re-anchored transition model for coarse motion. e-encodere-diagnosise-realisme-jump-method
- Reader analysis
The authors propose spatial tokens, clean-target prediction, residual anchoring and uncertainty-matched sampling as a compact-model recipe. Calling these universally necessary and sufficient exceeds the configurations tested. e-diagnosise-scaling
Figure 1. The present latent conditions prediction and also anchors every future residual. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the camera image: the frozen VAE converts appearance into a latent grid. Follow the separate action branch into Fourier embeddings, then the conditioning box that combines actions, present context and diffusion timestep. The lower input supplies noisy future latents during diffusion training; sampling begins with Gaussian noise. The red DiT produces a residual that meets the long blue present-latent bypass at Anchor Add. Decoding happens after this addition. Direct regression reuses the architecture with replicated present input and no diffusion noise. The bypass is therefore a central modeling assumption: every horizon is expressed relative to the same observed scene. e-architecturee-objectivee-datae-motione-jump-method
What it supports. The diagram explains both training stability and the later motion concern. A small residual leaves the present scene largely intact, providing a sensible initial prediction. However, producing several residuals around one fixed anchor does not itself make later frames accumulate the displacement of earlier predictions; the jump model changes that information flow.
Where the evidence stops. The column labels the Fourier encoder as frozen, but Section 4.2 specifies learned frequencies. The tiny diffusion-input label uses the present index; the main label, caption and Eq. (4) specify noised future targets. These drawing inconsistencies remain unresolved; this guide follows the method text.
2. Motivation
2.1 The problem and the proposed response
Which representations and prediction objectives support compact action-conditioned driving models, and which metrics distinguish realistic images from accurate dynamics? A single distortion score can reward an averaged, blurry future while hiding the appearance benefit of generative prediction. e-architecturee-realisme-qualitative
2.2 What this reading follows
Imagine predicting a driving video from one front-camera image and a supplied steering-and-acceleration sequence. This paper shows why choosing the prediction with the best average pixel or latent agreement can select a blurry future. Its compact diffusion transformer produces stronger image structure and better distribution scores, especially after calibration learned from training data. Yet plausible individual frames do not guarantee that the scene moves correctly. A motion decomposition exposes that gap, and a separate chained transition model partially addresses it. Read the six visuals as distinct tests of representation, appearance and dynamics; none demonstrates a deployed driving policy. e-architecturee-realisme-calibratione-motione-jump-resulte-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The neural-world-simulator and evaluation categories fit the architecture and metric study. Keeping the joint world/action-prediction quadrant not applicable is reasonable: actions condition future observations, while the encoder probe and inverse-control diagnostic are separate experiments. No shared generative model jointly emits future states and actions, so the evidence does not justify a One Model action-policy classification. e-architecturee-encodere-control
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the action probe separate from the world model
The paper first asks whether frozen visual features retain information useful for predicting logged actions. It projects embeddings to a common dimension and trains the same small steering-and-acceleration probe, making the encoder comparison easier to interpret. The temporal V-JEPA2 variant performs best, but this does not mean that V-JEPA2 supplies the latents in the illustrated generative pipeline. That pipeline uses a frozen SD-VAE so that predicted latents can be decoded into camera frames. Here the direction of information flow is different: actions are provided as conditioning, and future observations are predicted. Keeping these experiments separate prevents an action-prediction benchmark from being mistaken for a jointly trained policy or a driving controller. e-encodere-datae-architecture
Figure 8. The jump model feeds its own predicted state into the next transition. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the black arrows through the pink latent chain. Each Jump DiT consumes an anchor and the next four actions, then predicts a latent four keyframes later. The blue re-anchor labels show that the next transition uses that prediction, not a refreshed ground-truth frame. Decoder arrows point downward to the blue image row; those decoded images are shown for inspection, while the chain itself propagates latents. Compare them with the gray ground-truth row at the corresponding times. The four transitions reach t+16. Training differs from this displayed rollout: the model learns from ground-truth anchors, then runs on its own outputs at evaluation. e-jump-methode-jump-resulte-limits
What it supports. This structure accompanies the reported 1.7M-model motion result: direction correlation 0.48 and low-frequency magnitude 1.02×GT on 30 held-out scenes, versus direction correlation 0.41 for the 5.4M single-pass baseline. The displayed predictions also make the tradeoff visible: coarse structure is present early, but blur increases along the chain.
Where the evidence stops. The result concerns an open-loop prediction chain, not vehicle execution. The comparison changes transition horizon, anchoring, training task and capacity together; it cannot uniquely identify re-anchoring as the cause. Matching motion magnitude also does not establish accurate object trajectories or high-fidelity appearance.
5.2 Read the realism gain through the calibration control
A paired latent error asks whether a prediction resembles the particular recorded future. A distribution score asks whether a collection of predictions resembles a reference collection in feature space. The table shows these preferences diverging: the deterministic mean has better cosine similarity, whereas calibrated diffusion has better KID and FID. Reader interpretation: the raw-diffusion row is an essential control, because much of the reported improvement appears after the per-channel transformation. The source fits that transformation on training data, avoiding the need for test-time ground truth, but does not fully specify its estimator. Reproduction should therefore compare raw and calibrated outputs on identical held-out windows and seeds, and explicitly retain the decoded-ground-truth reference used by the paper. e-realisme-calibratione-evaluation
5.3 Separate a moving latent chain from a working driving policy
The residual equation gives every single-pass future the same observed anchor. The authors argue that this encourages repeated renderings of the present layout, and their low-frequency diagnostic reveals limited scene change. The jump model changes the recurrence: a predicted latent becomes the anchor for the next four-step transition. Teacher-forced training and self-anchored evaluation are distinct, so accumulated blur is a meaningful failure rather than a cosmetic issue. Reader interpretation: the higher motion correlation supports this combined redesign, but does not isolate its individual causes. The steering sweep supplies another piece of evidence, showing response to action changes among detectable cases. Neither experiment includes environmental feedback or establishes that choosing actions through this model would safely control a vehicle. e-architecturee-motione-jump-methode-jump-resulte-controle-limits
5.4 Training and inference
During training
nuScenes uses scene-disjoint 630/70/150 training/validation/test splits. Actions use training-only normalization; horizons 4, 8 and 16 correspond to 2, 4 and 8 seconds. Predictions are described as aggregated within scenes before final metrics. e-data
Diffusion minimizes clean-latent squared error under a 1,000-step cosine noise schedule, with action dropout 0.1. EMA decay is 0.999. Jump training uses ground-truth anchors; its objective/optimizer schedule is not fully specified. e-objectivee-jump-methode-limitse-configuration
During inference
Diffusion starts from Gaussian noise and uses 50 deterministic DDIM steps. Direct regression uses the same architecture at zero diffusion time in one forward pass. Logged actions condition both; the model does not itself select or execute them. e-objectivee-data
The separate 1.7M jump model predicts four-step transitions with mean-pooled embeddings of the intervening actions. Four sequential applications re-anchor on predicted latents to reach t+16; this is open-loop latent feedback, without fresh camera observations. e-jump-method
5.5 Implementation flow
- Separate representation probing from scene generation
The encoder study uses frozen 384-dimensional embeddings and an action-prediction MLP. The renderable world model instead uses SD-VAE grids, scaled by 0.18215 and patchified into 64 spatial tokens. V-JEPA2 does not supply the renderable world-model latents. e-datae-encoder
- Condition residual prediction
The approximately 5.4M-parameter DiT uses four blocks, four attention heads and width 256. adaLN-Zero combines diffusion time, pooled present context and per-step Fourier action embeddings. Each future residual is added to the same present latent. The text alternates spatial-token and H-token descriptions without fully specifying their arrangement. e-architecturee-data
- Decode and calibrate
A frozen VAE decodes predicted latents. Per-channel mean/scale calibration fitted on training data is applied at test time. FID/KID compare generated frames with decoded ground-truth latents; this reference differs from untouched camera RGB. e-calibratione-evaluation
6. Experiments & results
This compact driving world model predicts future front-camera latents from a present frame and supplied ego-actions. Calibrated diffusion improves distributional appearance over deterministic regression, while coherent motion remains weak. A separate chained jump model recovers coarse motion but accumulates blur. These are offline prediction and controllability findings, with closed-loop driving left untested (e-architecture, e-realism, e-jump-result, e-limits).
6.1 Read the original evidence
Table 2. At t+16, calibrated diffusion wins distribution scores while direct regression wins cosine similarity. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Check the arrows before comparing rows: lower KID and FID indicate closer feature distributions, while higher CosSim indicates closer paired latents. The experiment is at t+16, corresponding to eight seconds at the stated keyframe rate. Compare direct regression with calibrated diffusion, then inspect raw diffusion to see how much the calibration changes the outcome. The interpolation row mixes direct and diffusion latents equally; it is a separate operating point. Finally, interpret VAE-GT as the decoded ground-truth reference. Section 5.6 clarifies that the distribution metrics use these reconstructed test frames, so the near-zero reference row is not proof of perfect reconstruction of raw camera imagery. e-realisme-datae-calibratione-evaluation
What it supports. Calibrated diffusion reports KID 0.078 and FID 162.5 versus 0.375 and 370.8 for direct regression. CosSim reverses that ranking: 0.260 versus 0.471. Raw diffusion reaches only 0.294 KID, making calibration a substantial part of the reported distribution advantage. Halfway interpolation offers an intermediate distortion score, rather than dominating both objectives.
Where the evidence stops. These are Table 2 point values. Figure 4 names 600 windows and has differing bar heights; Figure 5 names 150 windows. The prose separately reports three-seed KID 0.076±0.005. The source does not reconcile these protocols, and KID sampling deviation should not be conflated with seed uncertainty.
Figure 10. The VAE retains scene structure; the two prediction modes introduce different errors. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down a column to compare methods at the same labeled time, then across a row to inspect progression. Camera RGB provides the observed scene, and VAE-GT isolates what encoding and decoding alone preserve. The direct-prediction row loses fine structure much more severely than that reconstruction row. Diffusion restores prominent edges and recognizable scene elements, but its contrast and texture differ markedly from the camera images. Across time, watch the leading car and road geometry: the generated row remains close to a repeated layout while the ground truth progresses. The final column is t+15, not the t+16 setting of Table 2. e-qualitativee-realisme-motione-limits
What it supports. This example supports a narrow, useful claim: regression blur is not explained solely by the VAE bottleneck, because the VAE-GT row remains much clearer. Diffusion produces stronger visual structure, consistent with the distribution-score comparison. Yet the visible artifacts and weak scene progression show why a favorable appearance score cannot substitute for a motion evaluation.
Where the evidence stops. This is one selected held-out scene, without an uncertainty estimate or a representative failure frequency. The caption describes realistic appearance; visual inspection supports recognizable structure with substantial artifacts. It cannot establish accurate futures, reliable motion or performance on unseen driving conditions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Decoded-frame distribution at eight seconds SD-VAE pipeline, t+16, 150-scene held-out nuScenes test split; Table 2 point values. Exact window mapping across plots is unresolved. | Calibrated diffusion: 0.078 KID, 162.5 FID, 0.260 CosSim. KID and FID (lower); latent CosSim (higher) | Direct: 0.375, 370.8, 0.471; raw diffusion: 0.294, 341.9, 0.233. Halfway latent interpolation: 0.084, 166.6, 0.316. The stated 4.8-fold KID advantage trades away paired latent agreement and depends strongly on calibration. The prose separately gives three-seed KID 0.076±0.005; do not attach that uncertainty to the table point. Distribution agreement is not driving success. e-realisme-calibratione-evaluatione-data |
| Frozen-encoder steering prediction 150 test scenes, shared action MLP, three seeds; reported bootstrap 95% confidence intervals. | V-JEPA2 rep64: 0.058±0.012. Normalized steering RMSE (lower) | Best single-frame encoder, rep1: 0.097±0.019; approximately 40% relative reduction. Temporal context benefits this probe. Methods specify 16-frame input, although Figure 2 labels it 64-frame; exact clip configuration needs resolution. e-encoder |
| Single-pass coherent scene change Held-out scenes, 16-step decoded rollouts; Gaussian low/high-frequency decomposition. | Diffusion: low-frequency 0.44×GT; high-frequency 0.98×GT. Frame-to-frame change divided by ground-truth change | Direct low-frequency change: 0.56×GT. In the high-motion quartile, diffusion/direct low-frequency values are 0.31/0.47. Near-ground-truth texture variation coexists with weak coherent motion. The reported temporal-loss fine-tune did not fix it. e-motion-methode-motion |
| Open-loop jump-model motion 30 held-out test scenes; own predicted anchors across four transitions. | Jump: 0.48 direction correlation; 1.02×GT magnitude. Image-plane direction correlation; low-frequency magnitude ratio | Larger single-pass baseline: 0.41 direction correlation; 1.7M versus 5.4M parameters. Coarse motion improves, but blur compounds. Transition length, anchoring, training task and capacity change together, so this is not an isolated anchor-only ablation. e-jump-resulte-jump-method |
| Steering-conditioned scene displacement Steering sweep over training percentiles 5–95, fixed noise, t+15; 40 held-out windows, called scenes in Figure 9. | Diffusion: rho 0.81, 100% sign-correct among 18/40 detectable cases. Spearman correlation; sign correctness among detected shifts | Direct: rho -0.18, 35% sign-correct among 39/40. An inverse-control probe reports 0.67×chance error versus 1.24×. The response is measurable on a selected detectable subset. Unknown threshold and incomplete inverse-probe details limit interpretation; this does not establish closed-loop control. e-controle-limits |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Changing the diffusion target removes most of the plotted performance deficit. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one encoder group at a time, using the legend to distinguish red noise-prediction bars, purple direct DiT, blue clean-target DiT and orange residual MLP. Higher cosine similarity is better, and the vertical axis begins at 0.4 rather than zero. Red bars sit far below the other three methods across these pooled representations. Blue and purple bars approach the orange reference, so the key contrast is between objectives and prediction modes within a transformer architecture. This figure belongs to the compact pooled-latent diagnosis; the subsequent spatial-token comparison is described separately in Section 5.2. e-diagnosis
What it supports. The plot supports the claim that poor initial DiT performance need not indicate insufficient transformer capacity. Clean-target prediction nearly reaches the residual-MLP reference in the displayed groups. The separate spatial-token comparison reports DiT gains of 0.020±0.002 and 0.023±0.002 CosSim for ViT and DINOv2 at matched 12M parameters and three seeds.
Where the evidence stops. The plotted recovery annotations are 99%, 100% and 97%, whereas the caption and Section 5.2 claim 88.5%. Their relationship is unexplained. Preserve the qualitative objective comparison, but do not treat either annotation set as a reconciled universal recovery estimate.
Figure 6. Texture variation can approach ground truth while coherent scene change remains much smaller. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The dashed horizontal line marks ground-truth change normalized to one. For each group, compare the two solid bars first: yellow is direct regression and blue is diffusion after low-frequency filtering. Then compare the hatched bar, which measures diffusion high-frequency variation. Section 4.4 defines the filtering through Gaussian blur with sigma 8 and kernel size 31; high frequencies are the residual after blur. The left group aggregates all scenes, while the right restricts attention to the high-motion quartile. This decomposition asks whether visible frame changes occur in broad scene structure or predominantly in fine texture. e-motion-methode-motione-jump-result
What it supports. Across all scenes, diffusion reaches 0.98×GT high-frequency change but only 0.44×GT low-frequency change. Direct regression reaches 0.56×GT low-frequency change despite its blur. The high-motion subset is more demanding: diffusion coherent change falls to 0.31×GT versus 0.47×GT for direct. Appearance realism therefore leaves a substantial temporal gap.
Where the evidence stops. Filtered image changes are proxies for motion, not measured 3D trajectories or collision outcomes. The figure does not plot jump-model bars, although its caption mentions the jump result. That separate comparison uses 30 scenes and should not be silently merged with these groups.
7. Analysis & limitations
7.1 What the evidence leaves open
The diagnosis text claims 88.5% gap recovery, while Figure 3 labels 99%, 100% and 97%. Figure 4 uses 600 windows, Figure 5 uses 150, and plotted values do not consistently match Table 2. These discrepancies remain unresolved, rather than forming a single pooled estimate. e-diagnosise-evaluation
Appearance artifacts and accumulated jump blur remain. The authors limit evidence to compact, single-camera, 2 Hz prediction. A two-capacity, one-seed-per-model probe cannot establish a scaling law; predicted-action closed-loop and multi-camera evaluation remain future work. e-limitse-scaling
7.2 Questions for discussion
- Does train-only calibration preserve steering response and motion as well as marginal appearance?
- Would a capacity- and objective-matched anchor ablation retain the jump model’s motion benefit?
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuild the scene-disjoint nuScenes/CAN-bus pipeline and frozen VAE preprocessing first. The encoder probe specifies Adam, learning rate 0.001, batch 256 and 50 epochs; the main world-model optimizer, full training duration, hardware, exact checkpoint versions and detailed token layout are not comparably documented. e-datae-encodere-architecturee-limitse-configuration
Resolve calibration estimation, KID sampling/aggregation, figure inconsistencies and the displacement threshold before comparing scores. The paper claims linked code/checkpoints/artifacts; availability and reproducibility require separate verification. e-calibratione-evaluatione-controle-code
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does train-only calibration preserve action response?
Reader-proposed, not executed: fix one trained checkpoint, held-out windows, action sweeps and diffusion noise; compare raw predictions with the paper’s per-channel calibration estimated strictly from training data. Include direct regression as a reference. Report KID/FID against both decoded ground-truth frames and raw RGB as explicitly separate protocols, plus CosSim, motion ratios, steering correlation and detection coverage. Predeclare the displacement threshold and repeat over seeds. The claim that calibration improves useful predictions is weakened if the appearance gain disappears under matched evaluation or if action sensitivity and coherent motion degrade despite lower KID. e-calibratione-realisme-evaluatione-controle-motion-method
Check 2: Isolate the benefit of replacing the present anchor
Reader-proposed, not executed: train matched-capacity four-step predictors with the same data, action embeddings, objective and optimization budget. During four-transition rollouts, compare an updating predicted anchor with a controlled fixed-present-anchor variant; add ground-truth-anchor evaluation only as a diagnostic ceiling. Use identical held-out scenes and report direction correlation, low-frequency magnitude, appearance scores and degradation by horizon across seeds. A consistent updating-anchor gain at matched capacity would support the mechanism. If the gain vanishes after matching training and transition length, the original comparison would support the combined jump redesign rather than anchoring alone. e-architecturee-jump-methode-jump-resulte-motion-method
8.3 Reading coverage
Visual audit: All ten original PDF pages were rendered and visually inspected, including the title/version/affiliations on page 1, method equations on pages 3–4, Tables 1–2, Figures 1–10, evaluation and limitation text on pages 5–8, the source-code availability statement on page 9, and Appendix A on page 10. All six final original crops were viewed; the architecture crop was tightened and viewed again. The guides preserve conflicting Figure 1 labels, Figure 3 recovery percentages, and Table 2/Figure 4 protocol and value differences. Figure 2’s clip-length label and Figure 9’s windows/scenes terminology are disclosed in the base report. No supplied page remains outside this visual pass; separate supplemental material was not verified, linked code was not inspected and no experiment was run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, version and Abstract
- 1. Introduction
- 2. Related Work
- 3. Dataset and Features
- 4. Methods (4.1–4.4)
- 5. Experiments (5.1–5.6)
- 6. Discussion
- 7. Conclusion and Future work
- Contributions and Acknowledgements; Source Code; References
- Appendix A. Additional Qualitative Results
Outside the original text pass
- Identity: the observed title and all three authors match the catalog; the inspected artifact is arXiv:2606.12987v1, 11 June 2026. No other revision or edition was supplied or compared (e-identity).
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks and all ten PDF pages were read; original Figures 1–10 and Tables 1–2 were visually inspected, resolving the text-only figure omission.
- Code, checkpoints, external references and linked artifacts were not inspected. No experiments were executed or reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
The title matches the catalog. Credits are Ruslan Sharifullin, Benjamin Jiang, and Kai Xi Chew, each at Stanford University. The margin identifies arXiv:2606.12987v1 [cs.CV], 11 Jun 2026.
Go to primary source ↓e-dataPDF p. 3, Section 3, Dataset and Features
nuScenes v1.0-trainval has 850 scenes and 33,552 keyframes; scene splits are 630/70/150. CAN-bus steering and acceleration use training-only z-score statistics. CAM_FRONT frames are 256×256; SD-VAE latents are 32×32×4, scaled by 0.18215 and patchified into 64 spatial tokens. Horizons are 4, 8, and 16 at 2 Hz.
Go to primary source ↓e-encoderPDF p. 3, Table 1 and Section 4.1; p. 4, Section 5.1 and Figure 2
Frozen embeddings are projected to 384 dimensions for a shared MLP probe, trained with Adam, learning rate 0.001, batch 256, 50 epochs, three seeds. Table 1 gives rep64 steering 0.058±0.012 versus rep1 0.097±0.019; acceleration is 0.055±0.004 versus 0.059±0.004. Methods specify bootstrap 95% CIs and 16-frame clips, whereas Figure 2 labels rep64 as 64-frame.
Go to primary source ↓e-architecturePDF p. 2, Figure 1 and caption; p. 3, Section 4.2, Eqs. (1)–(3)
The anchored VAE DiT has four blocks, four heads, width 256 and about 5.4M parameters. Conditioning combines diffusion time, present latent and learned Fourier action embeddings with 64 frequencies per action dimension. Future predictions add residuals to the same present latent. Figure 1 puts the Fourier encoder under Frozen Encoders and its small diffusion-input label uses the present index, unlike the learned embedding and noised-future descriptions.
Go to primary source ↓e-objectivePDF p. 4, Section 4.2, diffusion objective, Eq. (4), sampling and baselines
Diffusion trains clean-latent prediction under a cosine noise schedule with 1,000 timesteps and action dropout 0.1; inference uses 50 deterministic DDIM steps from Gaussian noise. Direct regression uses the same architecture at zero diffusion time in one pass.
Go to primary source ↓e-calibrationPDF p. 4, Section 4.3; p. 6, Section 5.3, calibration paragraph
FID compares Inception feature statistics; KID is polynomial-kernel squared MMD. Per-channel mean/scale calibration is estimated on training data and applied to test predictions. Train-derived KID is 0.078 versus a stated post-hoc oracle 0.086; the detailed calibration estimator is not supplied.
Go to primary source ↓e-diagnosisPDF p. 5, Section 5.2 and Figure 3
The diagnostic chain rejects capacity and longer horizon as explanations in pooled latents, attributes improvement to clean-target prediction and reports action-sequence interactions. Text and caption claim 88.5% gap recovery, while Figure 3 annotates 99%, 100%, and 97%. Spatial-token comparisons at 12M parameters and three seeds report DiT gains of 0.020±0.002 on ViT and 0.023±0.002 on DINOv2.
Go to primary source ↓e-realismPDF p. 5, Section 5.3 and Table 2, all rows, horizon t+16
Direct has KID/FID/CosSim 0.375/370.8/0.471; raw diffusion 0.294/341.9/0.233; interpolation at alpha 0.5 has 0.084/166.6/0.316; calibrated diffusion 0.078/162.5/0.260. The VAE-GT row is approximately zero KID/FID and 1.000 CosSim. The prose separately reports three-seed KID 0.076±0.005.
Go to primary source ↓e-evaluationPDF p. 3, Section 3, aggregation; p. 5, Figure 4; p. 6, Figure 5; p. 8, Section 5.6, Memorization/Overfitting
Section 3 specifies scene aggregation. Figure 4 describes 600 held-out windows and seed-standard-deviation bars; Figure 5 describes 150 test windows and KID-standard-deviation bars. Figure 4 bar heights do not consistently reproduce Table 2 point values. Section 5.6 clarifies that FID/KID reference held-out frames decoded from ground-truth latents and states three-seed evaluation generally.
Go to primary source ↓e-motion-methodPDF p. 4, Section 4.4, motion metrics
Consecutive decoded-frame differences are decomposed using Gaussian blur with sigma 8 and kernel 31; the residual measures high-frequency change. Horizontal/vertical profile cross-correlation supplies displacement magnitude and direction comparisons.
Go to primary source ↓e-motionPDF p. 6, Section 5.4 and Figure 6
All-scene low-frequency change is 0.56×GT for direct and 0.44×GT for diffusion; diffusion high-frequency change is 0.98×GT. High-motion-quartile values are 0.47, 0.31 and 0.83 respectively. A 30-epoch temporal-difference-loss fine-tune did not improve the motion results.
Go to primary source ↓e-jump-methodPDF p. 4, Section 4.4, chain-anchor jump model and Eq. (6); p. 7, Figure 8
A two-block, width-192, 1.7M-parameter model predicts a four-step transition from an anchor and four Fourier-embedded, mean-pooled actions. Training uses ground-truth anchors. Testing chains four transitions using its own latent predictions, reaching t+16 without new observations.
Go to primary source ↓e-jump-resultPDF p. 6, Section 5.4, jump-model result; p. 7, Figure 8 and continuation of Section 5.4
On 30 held-out test scenes, the open-loop jump model obtains direction correlation 0.48 versus 0.41 for the 5.4M single-pass baseline and low-frequency magnitude 1.02×GT. Figure 8 shows blur compounding across the predicted chain.
Go to primary source ↓e-controlPDF p. 7, Section 5.5 and Figure 9
The steering sweep spans training percentiles 5–95 with fixed diffusion noise and evaluates horizontal displacement at t+15 on 40 held-out windows, called scenes in the caption. Diffusion rho is 0.81 with 100% sign-correct among 18/40 detectable cases; direct rho is -0.18 with 35% sign-correct among 39/40. The inverse-control probe reports 0.67×chance error versus 1.24×, without a full protocol or detection-threshold value.
Go to primary source ↓e-limitsPDF pp. 7–8, Section 5.6; p. 8, Sections 6–7
The source reports tint, over-sharpening, regression blur and increasing jump-rollout blur. It limits conclusions to compact single-front-camera models at 2 Hz; closed-loop predicted-action evaluation and multiple cameras are future work. EMA decay is 0.999.
Go to primary source ↓e-scalingPDF p. 6, Section 5.3, capacity probe
The 3.0M versus 5.4M comparison uses one seed per model: the larger has KID 0.078 versus 0.089 and FID 162.5 versus 172.8. The authors explicitly call this two-point probe preliminary.
Go to primary source ↓e-codePDF p. 9, Source Code
The paper states that source code, trained checkpoints and evaluation artifacts are available at https://github.com/dlcv-team/latent-world-models-av. This is an availability statement, not verification of the linked contents.
Go to primary source ↓e-qualitativePDF p. 10, Appendix A, Figure 10 and caption
A held-out scene is shown at t+0, t+4, t+8, t+12 and t+15, with RGB, VAE-GT, direct and calibrated diffusion rows. VAE reconstruction retains structure, direct predictions blur strongly, and diffusion preserves stronger edges while exhibiting artifacts and limited progression.
Go to primary source ↓e-configurationPDF pp. 3–4, Sections 3–4.4; p. 8, Section 5.6; p. 10, Appendix A
The paper specifies VAE preprocessing, architecture dimensions, probe optimization, diffusion schedule, sampling steps and EMA. It does not provide a complete world-model optimizer/training schedule, hardware configuration, exact pretrained checkpoint versions, explicit spatial/horizon token layout or full jump objective. Appendix A contains only an additional qualitative figure.
Go to primary source ↓8.5 Primary sources
Diffusion Transformer World-Action Model for AV Scene Prediction ↗
PDF · 5,725 extracted words
Source fingerprint
e91dae0327fa6fb1072506ffcd933c8b0a07a60041edbc72044527c141f0a092