4D-WAM: 4D Consistent World Modeling for Autonomous Driving
1. Paper overview
In one sentence: 4D-WAM improves joint video-and-trajectory prediction by supervising generated futures with a frozen geometric teacher and emphasizing early decision timesteps, while keeping that teacher outside inference. ev-problemev-architectureev-teacherev-earlyev-navtest-v2ev-navhard
| At a glance | What to know |
|---|---|
| Research problem | Author claim Plausible RGB futures can still misrepresent moving-object shape, distance and velocity, misleading a jointly trained planner. The authors seek geometric and temporal supervision beyond video appearance, while keeping geometric reconstruction outside inference. ev-problem |
| Core mechanism | Source description A Mixture-of-Transformers backbone separates video and action experts while exchanging their attention representations; an asymmetric mask protects observed history from noisy future tokens. ev-architecture |
| A key reported result | NAVSIM v2 navhard two-stage trajectory planning: 35.9 EPDMS ↑. Real-observation stage S1 followed by synthetic follow-up stage S2; EPDMS is the final two-stage score. Metis 32.2; DriveLaW 30.6; DriveFine 30.5. A 3.7-point gain over Metis. This benchmark evidence is stronger than selected video examples, but does not establish safety in road deployment or superiority on every component. ev-navhard |
| Reading caution | Reader analysis Teacher consistency is a proxy for physical correctness. Figures 4 and S1 show selected reconstruction and motion improvements, without a dataset-level geometric-error evaluation. Tables report no confidence intervals or seed dispersion. ev-teacherev-qualitativeev-eponaev-navtest-v2ev-navhardev-components |
Core contributions
- Source description
A Mixture-of-Transformers backbone separates video and action experts while exchanging their attention representations; an asymmetric mask protects observed history from noisy future tokens. ev-architecture
- Source description
Frozen VGGT-Ω supplies feature and depth consistency targets. A decision-oriented noise sampler concentrates training on the region where trajectories approach their final denoising prediction. ev-teacherev-earlyev-sampler
Figure 2. Two interacting prediction experts receive geometric supervision during training. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower left: noisy actions and history-plus-noisy-video tokens enter different experts, then meet in the central MoT attention block. Outputs return to modality-specific processing. In the lower-right mask, C, V and A denote conditions, future video and actions. Reading rows as attending queries, the C row permits only C; V and A permit all groups, matching the caption and method text. Follow predicted frames into the orange training-only branch. Equation (6) clarifies that both predicted and recorded videos produce both features and depth. The snowflake freezes teacher parameters, while the dashed red arrow carries loss gradients back toward the backbone. ev-architectureev-teacher
What it supports. The geometry model supervises the video-producing pathway during training, while mixed attention connects that pathway to trajectory prediction. This architecture explains how a geometric training objective can influence planning without adding a geometric teacher to the inference sequence. The backbone still predicts both video and actions.
Where the evidence stops. The teacher’s responses to recorded RGB frames are estimated targets, not measured physical ground truth. The schematic is consistent with the stated attention mask, but it does not specify exact checkpoint sizes or how the two branches’ sampled noise levels are coupled.
2. Motivation
2.1 The problem and the proposed response
Plausible RGB futures can still misrepresent moving-object shape, distance and velocity, misleading a jointly trained planner. The authors seek geometric and temporal supervision beyond video appearance, while keeping geometric reconstruction outside inference. ev-problem
2.2 What this reading follows
A driving model can draw a convincing future frame while placing a moving vehicle at the wrong distance. When video and trajectory predictions interact, that error can also distort the planned maneuver. 4D-WAM addresses this connection through training: a frozen VGGT-Ω compares the geometric features and depth inferred from predicted and recorded futures. A second change directs more training samples toward the high-noise region where action predictions stabilize. The figures below trace these mechanisms into NAVSIM results and ablations. Read the planning improvements alongside their boundaries: geometric targets are teacher estimates, and early agreement with a final trajectory does not itself establish that the trajectory is correct. ev-problemev-architectureev-teacherev-earlyev-navtest-v2ev-navhard
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The separate video/action DiT experts support the recorded Dual-system architecture, while mixed attention and joint future/trajectory denoising support Joint prediction rather than inverse dynamics. The frozen teacher is auxiliary training supervision, not a second inference-time planner. The driving/4D theme fits; the multiview tag needs qualification because evaluated visual input is a single front camera, without an explicit 4D output representation. ev-architectureev-teacherev-navtest-v2
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Protect observed history while letting video and actions interact
Start from the information the planner actually receives: three historical front-view frames, the current observation, ego state and a textual instruction. The VAE turns the visual history into clean conditioning latents. Future-video latents and trajectory tokens begin noisy, so permitting them to rewrite the conditioning representation would mix predictions back into the observed anchor. The paper’s mask prevents that particular attention path: condition queries see conditions, while video and action queries can see all groups. Separate DiT experts preserve modality-specific processing, but concatenated query, key and value representations allow joint reasoning at each layer. This is the architectural basis for the Dual-system × Joint prediction assessment. The term dual refers here to the video and action experts; the geometric teacher is a training component outside their inference loop. ev-architectureev-training
5.2 Make generated futures comparable to geometric targets
The geometry teacher expects clean RGB, while a training example contains noisy video latents. The model therefore first subtracts predicted velocity scaled by the noise level to estimate a clean latent, then decodes it. Both that decoded future and the recorded future pass separately through frozen VGGT-Ω. Their features supervise broad scene structure, and their depth responses supervise local geometric relationships. The supplement makes the objective concrete: feature distances cover layers 11, 17 and 23 and camera, register and pooled patch tokens; depth supervision excludes invalid pixels and each frame’s lowest-confidence portion. Both geometric terms have weight one and enter gradually during Stage II. Reader interpretation: this teaches the backbone to agree with a particular geometry estimator. It does not independently prove that the predicted scene satisfies physical dynamics. ev-teacherev-feature-implev-depth-implev-training
5.3 Connect early convergence to training allocation and inference cost
Figure 3 defines early decision through agreement with the model’s final twenty-step trajectory. The authors place the decision boundary at noise level 0.90, then adjust a rational transform of uniform samples so half the training mass falls above that boundary. Table 5 supplies the empirical reason for choosing that allocation: its 90.6 EPDMS exceeds both uniform sampling and the more concentrated 60% setting. Training allocation and inference truncation are separate choices. Table 6 tests the latter and reports 90.5 EPDMS at two steps versus 90.6 at ten, with RTX 4090 latency falling from 0.79 to 0.31 seconds. Reader interpretation: the results suggest an economical operating point, but the convergence diagnostic should also be checked against ground-truth errors and collision outcomes before early stability is treated as evidence of a correct decision. ev-earlyev-samplerev-samplingev-latency
5.4 Training and inference
During training
Under flow matching, Stage I trains the joint video-action objective on navtrain for 40 epochs; Stage II adds geometric supervision for another 40. AdamW uses cosine scheduling with learning rates 1×10⁻⁴ and 2×10⁻⁵. Training uses 16 NVIDIA H200 GPUs. Frames lose 28 pixels at both top and bottom before resizing to 352×640. ev-training
Feature supervision uses teacher layers {11,17,23}, separately comparing camera, register and patch tokens with equal weights. Patch grids are average-pooled from 22×40 to 11×20. Depth filtering requires finite values, positive depths and target confidence above 10⁻⁵, then retains confidence at or above each frame’s 30th percentile. Smooth-L1 β=0.1 and log-depth ε=10⁻⁶. Both geometric loss weights are one, introduced by a half-cosine warm-up over Stage II’s first 0.5 epoch. ev-feature-implev-depth-impl
During inference
Joint denoising predicts video and trajectory, normally in ten steps. The geometric teacher and training sampler are absent from inference. The paper evaluates planned trajectories; it does not specify a deployed low-level actuator controller or demonstrate road execution. ev-architectureev-trainingev-metrics
5.5 Implementation flow
- Encode conditions and noisy futures
A video VAE encodes observed frames as clean history latents, concatenated with noisy future-video latents. Noisy trajectories become action tokens through linear projection. A pretrained T5 encodes instructions; projected ego state is appended to those text tokens. ev-architectureev-training
- Exchange information between experts
The video expert starts from Wan2.2 video DiT; a structurally aligned action DiT handles trajectories. Each layer concatenates the experts’ queries, keys and values for mixed attention, then routes outputs back to their respective experts. Condition tokens attend only to conditions; future video and actions attend to all three groups. ev-architecture
- Compare geometry through a frozen teacher
Recover a clean latent estimate before VAE decoding. Apply the same frozen VGGT-Ω separately to predicted and recorded future RGB sequences, obtaining features and depth from each. Feature cosine distance and confidence-filtered log-depth Smooth-L1 loss train the backbone through the predicted-video path. Target depth is teacher-estimated, not measured ground-truth depth. ev-teacherev-architecture
- Concentrate supervision where decisions stabilize
A 20-step diagnostic on 1,000 NAVSIM validation scenes compares intermediate clean action estimates with the final prediction. The authors set the decision boundary to noise level 0.90 and shift training probability toward the interval [0.90, 1]. This observed convergence pattern does not guarantee correctness. ev-earlyev-sampler
6. Experiments & results
4D-WAM jointly predicts driving video and ego trajectories, using a frozen geometry teacher during training to align generated futures with geometric responses to recorded futures. It emphasizes high-noise training timesteps and improves reported NAVSIM planning scores; the teacher adds no inference module. The evidence supports benchmark gains, while physical consistency and cross-domain robustness remain less directly quantified.
6.1 Read the original evidence
Table 1. The strongest learned aggregate score comes with uneven component performance. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the rightmost EPDMS column, then trace each candidate back through its component scores and sensor entry. Bold and underline mark the best and second-best learned results; the human row is a separate reference. The retained note identifies reinforcement-learning fine-tuning through the RL superscript. NC concerns at-fault collisions; DAC, DDC and TLC concern drivable area, driving direction and traffic lights. EP and TTC describe progress and time to collision. LK is lane keeping; HC and EC concern history and extended comfort. The 4D-WAM row uses a single camera, but other rows differ in sensor configuration and training approach. ev-navtest-v2ev-metrics
What it supports. 4D-WAM reaches 90.6 EPDMS, ahead of DriveFine’s 89.7 by 0.9 points and Metis’s 89.5 by 1.1 points. Its strong collision, direction, progress and time-to-collision components support the aggregate result. The 94.5 human reference remains higher, and several learned competitors lead individual components.
Where the evidence stops. The table does not establish uniform superiority: 4D-WAM’s traffic-light compliance is 97.3 and lane keeping is 86.9. Different sensors and training regimes prevent a controlled architectural comparison; no confidence intervals accompany the reported scores.
Table 3. 4D-WAM’s largest highlighted benchmark gain appears in the two-stage navhard setting. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each method as a pair of rows. S1 evaluates the real-observation stage; S2 evaluates the synthetic follow-up stage. The far-right EPDMS entry summarizes the full two-stage protocol, so it is neither an S1-only nor an S2-only score. First compare these aggregate entries, then inspect the paired component values to see where gains or losses occur. For example, 4D-WAM’s collision score is lower than DriveFine’s in S1 but higher in S2. This structure matters for the paper’s argument: the authors associate its synthetic-follow-up performance with coherent predicted geometry and motion. The table establishes planning scores, while that causal explanation also depends on the ablations. ev-navhardev-metricsev-components
What it supports. The final score is 35.9, exceeding Metis’s 32.2 by 3.7 points. 4D-WAM also leads several S2 components, including collision, direction, progress, time-to-collision and lane keeping. These results strengthen the case for improved planning under the benchmark’s difficult follow-up observations.
Where the evidence stops. 4D-WAM’s S1 collision score is 92.8 versus DriveFine’s 97.6, so the aggregate gain should not be described as universal safety improvement. Navhard’s two-stage benchmark also does not establish performance in unrestricted physical driving.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM v2 navtest trajectory planning Official v2 navtest protocol; single front camera; default ten-step inference. | 90.6 EPDMS ↑ | DriveFine 89.7; Metis 89.5; human reference 94.5. Best learned aggregate in Table 1, by 0.9 points over DriveFine. Component tradeoffs remain: TLC is 97.3 and LK 86.9. Baselines use differing sensors and training strategies. ev-navtest-v2ev-trainingev-metrics |
| NAVSIM v1 navtest trajectory planning Official v1 navtest protocol. | 90.9 PDMS ↑ | ReCogDrive with RL 90.8; DriveVLA-W0 90.2. A 0.1-point lead over the strongest listed learned baseline; no uncertainty is reported. PDMS and EPDMS are different composites. ev-navtest-v1ev-metrics |
| NAVSIM v2 navhard two-stage trajectory planning Real-observation stage S1 followed by synthetic follow-up stage S2; EPDMS is the final two-stage score. | 35.9 EPDMS ↑ | Metis 32.2; DriveLaW 30.6; DriveFine 30.5. A 3.7-point gain over Metis. This benchmark evidence is stronger than selected video examples, but does not establish safety in road deployment or superiority on every component. ev-navhard |
| Cumulative component ablation NAVSIM v2 navtest; components added cumulatively. | Base 88.8 → history 89.6 → feature loss 90.1 → depth loss 90.4 → sampling 90.6. EPDMS ↑ | Table 5 separately reports uniform sampling 90.2, 50% decision-region allocation 90.6, and 60% allocation 90.1. Both studies favor the full configuration, but the 90.4 versus 90.2 presampling/uniform discrepancy is unexplained. Cumulative additions do not isolate all interactions. ev-componentsev-sampling |
| Denoising speed and planning tradeoff NAVSIM EPDMS ablation; Table 6 does not separately restate its split. Latency is measured on NVIDIA RTX 4090. | Two steps: 90.5, 0.31 s; ten steps: 90.6, 0.79 s. EPDMS ↑; latency in seconds ↓ | One step: 88.1, 0.22 s. Two steps retain most reported score with lower latency; timing scope, batching and repeated-run variability are unspecified. ev-latency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Most trajectory change occurs near the start of the illustrated denoising process. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the left pair of future-frame and trajectory panels with the middle pair: they show clean estimates after steps one and twenty, not observations at different driving times. Both future images concern the four-second horizon. Then read the curve from left to right: the horizontal noise coordinate decreases from one to zero. Its vertical quantity is mean squared difference from the final twenty-step action prediction, aggregated over 1,000 NAVSIM validation scenes. The marked boundary at 0.90 separates the high-noise decision region from later refinement. Equation (11) assigns the boundary itself to the decision region, consistent with the sampling formulation. ev-earlyev-sampler
What it supports. The diagnostic supports the authors’ interpretation that the model selects much of its maneuver early, with later steps refining an already similar prediction. It motivates allocating more geometric supervision to high-noise training examples. This is an empirical observation about the examined rollout and validation sample.
Where the evidence stops. Action-to-final MSE measures agreement with the model’s own final output. A wrong maneuver can converge early too. The illustrated frame pair and averaged curve do not show maneuver-specific exceptions, seed variability or ground-truth trajectory error at the decision boundary.
Table 4. Geometric supervision and timestep sampling add gains after historical conditioning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward as cumulative additions, as specified by the source caption. The feature-loss row already contains historical conditioning, and the sampling row includes both geometric losses. Follow the EPDMS column to separate the increments associated with each addition. Then inspect component columns: adding feature loss raises NC from 98.4 to 99.0, while depth loss raises EPDMS from 90.1 to 90.4. The last row is the complete reported configuration. Sampling means the decision-oriented training distribution, not a change to the camera history or the number of inference steps. The component values help reveal that a higher aggregate need not improve every individual metric. ev-componentsev-sampling
What it supports. The sequence improves EPDMS from 88.8 to 90.6. After history reaches 89.6, feature and depth consistency bring the score to 90.4; sampling then reaches 90.6. This supports contributions beyond historical conditioning, though the cumulative design alone does not identify independent effects or interactions.
Where the evidence stops. Table 5’s uniform-sampling configuration scores 90.2, whereas this table’s row before adding sampling scores 90.4. The source does not explain that difference. Preserve both values; neither table reports variability sufficient to reconcile them statistically.
Table 5. A balanced emphasis on high-noise training outperforms the most aggressive tested allocation. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each column is a training distribution. The top row gives the probability mass ρ_dec assigned to noise levels at or above the 0.90 decision boundary; the second row gives the corresponding shift parameter s from Equation (12). Uniform sampling is the first column: because the decision interval occupies the upper tenth of the noise range, its probability is 10%. Moving right reallocates training examples toward that interval. The third row reports EPDMS for each choice. These percentages are not fractions of inference steps executed, and s is not a denoising-step count. Compare the peak column with both the uniform baseline and the more concentrated 60% setting. ev-earlyev-samplerev-sampling
What it supports. The best tested allocation is 50%, with s=9 and EPDMS 90.6, compared with 90.2 under uniform sampling. Increasing the decision-region mass to 60% lowers the score to 90.1. The reported trend favors emphasizing early decision formation while retaining substantial training coverage for refinement.
Where the evidence stops. The explanation that excessive high-noise emphasis harms refinement is the authors’ interpretation of the score drop. The table does not directly measure refinement quality or establish a universal optimum; uncertainty and a separate hyperparameter-selection protocol are not reported.
7. Analysis & limitations
7.1 What the evidence leaves open
Teacher consistency is a proxy for physical correctness. Figures 4 and S1 show selected reconstruction and motion improvements, without a dataset-level geometric-error evaluation. Tables report no confidence intervals or seed dispersion. ev-teacherev-qualitativeev-eponaev-navtest-v2ev-navhardev-components
Waymo transfer uses a NAVSIM-only checkpoint and four illustrated validation scenes, including nighttime cases. Section S3 provides qualitative trajectory agreement, without aggregate planning metrics or a quantified transfer comparison. ev-waymo
The additional Epona comparison uses 100 denoising steps versus ten for WAM/4D-WAM, and stitches the first two seconds from each of two Epona predictions to extend its three-second horizon to four seconds. Its selected-case scores therefore require this protocol qualification. ev-epona
7.2 Questions for discussion
- Would geometric gains persist under an independent depth or motion evaluator rather than the supervising teacher?
- Does the decision threshold remain stable across maneuvers, seeds and domain shifts?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires NAVSIM training/evaluation data, Wan2.2, the VAE and T5, frozen VGGT-Ω, the two-stage schedule and S1’s exact loss implementation. The supplied text leaves backbone/checkpoint size, batch size, seed, full joint-loss weighting, action parameterization, instruction construction and video/action noise coupling insufficiently specified. ev-architectureev-trainingev-feature-implev-depth-impl
Reader-proposed priorities are a matched factorial test of geometric losses and timestep allocation, and an independent check that early convergence predicts correct, collision-avoiding trajectories. These are proposed experiments, not completed reproductions. ev-earlyev-componentsev-samplingev-latency
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate geometric supervision from timestep allocation
Reader-proposed experiment, not performed: start every run from the same Stage-I checkpoint and hold historical input, Stage-II duration, optimizer schedule and inference settings fixed. Cross four geometric objectives—none, feature only, depth only and both—with uniform versus 50% decision-region sampling. Use several matched seeds and the same held-out NAVSIM scenes; report EPDMS and its safety/progress components with paired uncertainty. First resolve why Tables 4 and 5 give different presampling/uniform scores. A repeatable interaction in which geometric supervision benefits more under shifted sampling would support the proposed mechanism. Similar sampling gains without geometry would weaken the claim that its benefit specifically comes from directing 4D supervision toward decisions. ev-trainingev-feature-implev-depth-implev-componentsev-sampling
Check 2: Test whether early stability identifies correct trajectories
Reader-proposed experiment, not performed: reproduce the twenty-step diagnostic on a fixed NAVSIM validation sample, stratified by maneuver and difficult interactions. At every step, record both action-to-final MSE and error to the recorded trajectory, then evaluate collision and progress outcomes using the same benchmark protocol. Compare the full model with its variant without geometric supervision, holding the sampled initial noise fixed, and separately compare two-step with ten-step inference. Stable trajectories that remain wrong or collision-prone would falsify using early convergence as a correctness indicator. Report latency with explicit batch size, included operations and repeated measurements, rather than assuming Table 6’s timing transfers to another setup. ev-earlyev-metricsev-qualitativeev-latency
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version and Figure 1 on p. 1; motivation and method opening on p. 2; architecture, attention mask and equations on p. 3; teacher losses and early-decision diagnostic on p. 4; Tables 1–2, metrics and training hardware/schedule on p. 5; Table 3 and ablation discussion on p. 6; Figure 4, Tables 4–6 and conclusion on p. 7; all loss implementation and Epona protocol details on p. 10; Figure S1 on p. 11; and Figure S2, qualitative case interpretation and Waymo scope on p. 12. All six final crops were inspected. Table 1 and Table 3 retain their short explanatory notes for ranking/RL and stage definitions. Figure 2’s arrows, frozen-teacher marker and attention mask were checked against its caption and Eqs. (1)–(9); Figure 3’s decreasing noise axis and region boundary were checked against Eqs. (10)–(13). Reference pages 8–9 were read as text only. No separate supplementary artifact or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Introduction (pp. 1–2)
- Related Work (p. 2)
- Method: architecture, 4D consistency supervision, decision-oriented timestep sampling (pp. 2–5)
- Experiments: datasets, metrics, implementation, results and ablations (pp. 5–7)
- Conclusion (p. 7)
- References (pp. 8–9)
- Supplement S1: feature and depth loss implementation (p. 10)
- Supplement S2: additional qualitative comparison (pp. 10–12)
- Supplement S3: zero-shot generalization (p. 12)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The supplied PDF, including its embedded supplement, was read completely. Original figures and tables were visually inspected, resolving the extraction-only image limitation; no separate supplement was supplied.
- Identity/version: the inspected title page states arXiv:2608.10107v1, 10 August 2026. Title and all eleven authors agree with the catalog after name-order normalization. No other revision was supplied or compared.
- Code was not inspected and experiments were not reproduced. References were read as bibliography, not independently verified sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title, author/affiliation block and arXiv margin stamp
Exact catalog title; eleven matching authors; arXiv:2608.10107v1 dated 10 August 2026. The title page lists four institutions.
Go to primary source ↓ev-problemPDF pp. 1–2, Abstract, Introduction and Figure 1
The authors connect inconsistent object geometry and lead-vehicle motion predictions with poor planning, motivating training-time geometric supervision.
Go to primary source ↓ev-architecturePDF p. 3, Figure 2/caption and 4D-WAM Architecture, Eqs. (1)–(3)
Historical front-view latents, text and ego state condition separate video/action DiTs. Mixed QKV attention protects condition queries and permits bidirectional video/action interaction. The teacher and timestep sampling are marked training only.
Go to primary source ↓ev-teacherPDF p. 3, 4D Consistency Supervision, Eq. (4); p. 4, Eqs. (5)–(9)
Clean latent estimation precedes VAE decoding. Frozen VGGT-Ω processes predicted and recorded videos separately; feature cosine and confidence-filtered log-depth losses form the geometric objective.
Go to primary source ↓ev-earlyPDF p. 4, Figure 3/caption and Decision-Oriented Timestep Sampling, Eqs. (10)–(11)
A 20-step rollout over 1,000 validation scenes measures intermediate-to-final action MSE; the decision boundary is set to 0.90. The reference is the model’s final prediction.
Go to primary source ↓ev-samplerPDF p. 4, Eqs. (11)–(13); p. 5, paragraph before Experiments
The decision region is [θ_dec,1]; rationally transformed uniform noise shifts probability toward high noise. A 50% allocation corresponds to s=9.
Go to primary source ↓ev-trainingPDF p. 5, Implementation Details
Four observed frames, 2 Hz/four-second outputs, top/bottom 28-pixel cropping, 352×640 resolution, two 40-epoch stages, AdamW/cosine learning rates, 16 H200 GPUs and ten default denoising steps are specified.
Go to primary source ↓ev-metricsPDF p. 5, Dataset and Metrics
PDMS combines collision, drivable-area, progress, time-to-collision and comfort components; EPDMS extends compliance/comfort criteria. Navhard uses two-stage closed-loop evaluation.
Go to primary source ↓ev-navtest-v2PDF p. 5, Table 1, 4D-WAM/DriveFine/Metis/Human rows, Sensors and score columns
EPDMS is 90.6/89.7/89.5/94.5 respectively. 4D-WAM uses 1C; its NC/DDC/EP/TTC are 99.1/99.7/98.3/98.6, and TLC/LK are 97.3/86.9. Caption defines bold/underline ranking and RL fine-tuning.
Go to primary source ↓ev-navtest-v1PDF p. 5, Table 2, PDMS column; p. 6, Results on navtest
4D-WAM 90.9, ReCogDrive RL 90.8 and DriveVLA-W0 90.2 under the v1 protocol.
Go to primary source ↓ev-navhardPDF p. 6, Table 3 and caption, EPDMS and S1/S2 component columns
Final two-stage EPDMS is 35.9 for 4D-WAM, 32.2 Metis, 30.6 DriveLaW and 30.5 DriveFine. S1 is real observation; S2 is synthetic follow-up. 4D-WAM S1 NC is 92.8 versus DriveFine 97.6; S2 NC is 82.6 versus 82.1.
Go to primary source ↓ev-componentsPDF p. 7, Table 4 and caption; pp. 6–7, Ablation Study of Key Components
Cumulative EPDMS values are 88.8, 89.6, 90.1, 90.4, 90.6. Feature loss raises NC from 98.4 to 99.0. Sampling changes LK from 87.0 to 86.9.
Go to primary source ↓ev-samplingPDF p. 7, Table 5 and Ablation Study of Timestep Sampling Distribution
Decision-region allocations 10/20/30/40/50/60% correspond to s=1/2.25/3.86/6/9/13.5 and EPDMS 90.2/90.3/90.3/90.5/90.6/90.1. The difference from Table 4’s 90.4 is not explained.
Go to primary source ↓ev-latencyPDF p. 7, Table 6/caption and Ablation Study of Denoising Steps
One/two/ten steps yield 88.1/90.5/90.6 EPDMS and 0.22/0.31/0.79 seconds on an RTX 4090.
Go to primary source ↓ev-qualitativePDF p. 6, Qualitative Effects of 4D Consistency Supervision; p. 7, Figure 4/caption
Two cases compare an ablated model, full model and ground truth for moving-object reconstruction and lead-vehicle motion prediction.
Go to primary source ↓ev-feature-implPDF p. 10, S1 Feature Consistency Loss, Eqs. (S1)–(S3)
Teacher layers 11/17/23 and camera/register/patch groups receive equal weights. Group distances average over frames/tokens; 2×2 pooling reduces the patch grid from 22×40 to 11×20.
Go to primary source ↓ev-depth-implPDF p. 10, S1 Depth Consistency Loss, Eqs. (S4)–(S8)
Finite positive depths and confidence above 10⁻⁵ precede per-frame 30th-percentile confidence filtering. β=0.1, ε=10⁻⁶, both loss weights one, and half-cosine warm-up lasts 0.5 epoch.
Go to primary source ↓ev-eponaPDF p. 10, S2 protocol; p. 11, Figure S1/caption; p. 12, Case 1 and Case 2
Epona uses 100 steps and a stitched four-second trajectory; WAM/4D-WAM use ten steps. Figure S1’s case scores and examples illustrate selected geometry/motion failures under these differing settings.
Go to primary source ↓ev-waymoPDF p. 12, Figure S2/caption and S3 Zero-Shot Generalization Capability
A checkpoint trained exclusively on NAVSIM is qualitatively evaluated on Waymo validation sequences, with four illustrated scenes including turning, straight and nighttime driving; no aggregate metric is supplied.
Go to primary source ↓8.5 Primary sources
4D-WAM: 4D Consistent World Modeling for Autonomous Driving ↗
PDF · 5,836 extracted words
Source fingerprint
97751f41743c415f9e13ecb1bbe30ca0a2575dd7398f75b8d5ee36ec767b19e0