Toward Physically Consistent Driving Video World Models under Challenging Trajectories
1. Paper overview
In one sentence: Learning to repair impossible trajectories and learning to render difficult physical interactions are complementary, but their measured gains depend on synthetic supervision and a stylized evaluation domain. problempipelinecounterfactual-pairsvideo-modelpipeline-resultsgenerator-resultsablationstyle-transfer
| At a glance | What to know |
|---|---|
| Research problem | Source description A trajectory can demand that vehicles pass through one another, while a feasible collision can still be unfamiliar to a generator trained on safe driving. The authors target both failures: correct the motion condition, then learn to depict its physical consequences. problempipeline |
| Core mechanism | Source description Counterfactual trajectory supervision maps impossible post-impact continuations to simulated 6-DoF outcomes; nominal examples discourage unnecessary corrections. counterfactual-pairs |
| A key reported result | Video generation from physics-violating CARLA Ego trajectories: 11.03; 72.48; 0.71; 0.71 FID ↓; FVD ↓; PHY ↑; Pref. ↑. Table 1; 150 sampled test clips; stylized CARLA initial frames and reference videos; constant-velocity post-collision corruption. DiST-4D: 19.84; 197.57; 0.39; 0.10. Large gains for the complete pipeline. PHY combines VLM judgments of physical and visual quality; preference is a separate human study. Neither is a driving success rate. pipeline-resultsevaluationhuman-studystyle-transfer |
| Reading caution | Reader analysis The experiments measure short generated clips and rectified trajectories, without closed-loop driving or real collision ground truth for the nuScenes stress test. PHY is a perceptual proxy, not a direct mechanics test; reported tables provide no confidence intervals. evaluationstress-testscope-boundarypipeline-resultsgenerator-results |
Core contributions
- Source description
Counterfactual trajectory supervision maps impossible post-impact continuations to simulated 6-DoF outcomes; nominal examples discourage unnecessary corrections. counterfactual-pairs
- Source description
A heterogeneous training set combines 9.7 hours of selected CARLA interactions with 4.6 hours of nuScenes. A time-wise head targets abrupt motion, and a separately trained video generator learns challenging appearances. datatemporal-headvideo-model
Figure 2. Correct motion first; render its consequences second. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the lower panel from left to right. Initial-image perspective-view features enter spatial cross-attention, other agents enter self-attention, and the map supplies road context. The refined token is copied across future times for the temporal output head. Its six-component motion becomes camera-view box and map layouts for PE-MVGen. The upper panel supplies nominal and challenging training examples. Separate the two learning problems: corrupted trajectories train the corrector, whereas the video model receives ground-truth physical layouts during training. The circled-plus in the rendering branch must be read with Section 3.4, which specifies channel concatenation of layout and noisy-video latents. pipelinecorrectortemporal-headcounterfactual-pairsvideo-modelrectification-resultsgenerator-resultstraining
What it supports. The architecture gives an explicit intermediate representation at which to inspect physical mistakes. A failure can originate in trajectory correction or in rendering a valid corrected layout. Figure 8 and Table 3 separately examine rectification; Table 2 evaluates the renderer with ground-truth conditions. This separation makes the proposed mechanism testable.
Where the evidence stops. Figure 2 places h labels after MLP blocks, while Equations (5)–(6) define h before the output MLP; follow the equations. N also labels block repetition although the formulation uses N for agents, leaving the implementation’s N=2 ambiguous.
2. Motivation
2.1 The problem and the proposed response
A trajectory can demand that vehicles pass through one another, while a feasible collision can still be unfamiliar to a generator trained on safe driving. The authors target both failures: correct the motion condition, then learn to depict its physical consequences. problempipeline
2.2 What this reading follows
Imagine asking a driving video model to continue a car straight through a guardrail. A convincing renderer needs a motion condition that depicts the impact, and it must have learned what that impact looks like. PhyGenesis addresses these requirements in sequence: a physical condition generator predicts corrected six-degree-of-freedom trajectories, then a Wan2.1-based model renders the projected scene layouts. CARLA supplies challenging interactions that ordinary driving logs rarely contain. The figures below explain this division of labor and compare three different questions: whether the whole pipeline works, whether rendering improves when trajectories are already valid, and which components account for the gains. problempipelinecounterfactual-pairsvideo-modelpipeline-resultsgenerator-resultsablationstyle-transfer
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational neural-simulator and training-optimization labels fit trajectory-conditioned generation and heterogeneous training. No distillation is demonstrated. Architecture, prediction-paradigm and quadrant Not applicable are appropriate for this catalog’s action-model taxonomy: the sequential rectifier/renderer does not jointly predict executable actions and futures or infer controls through inverse dynamics. pipelinevideo-modelscope-boundary
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build a correction target from the collision that actually happened
The counterfactual training pair is the key to understanding the first stage. Begin with a CARLA clip containing a collision and its logged physical motion. Retain each agent’s pre-impact path, then replace post-impact positions with a continuation at the pre-impact velocity. The input now describes motion that can penetrate another object; the target remains the simulator’s actual trajectory. Initial images, nearby agents and map context let the model infer where that continuation becomes implausible. The output includes vertical position and all three rotation angles, because a difficult interaction is not fully represented in a ground-plane path. Uncorrupted nuScenes pairs supply a complementary requirement: preserve ordinary driving. This training task learns a mapping to observed consequences; it is not an objective to avoid every crash. Figure 8’s corrected guardrail collision and stop illustrate that distinction. counterfactual-pairscorrectorrectification-results
5.2 Keep physical supervision separate from visual synthesis
The video generator is not trained by asking it to imitate frames under deliberately impossible trajectories. It sees ground-truth physical layouts from a balanced mixture of real and simulated clips. Projected boxes and map polylines constrain geometry, while the initial frame and scene caption provide appearance and semantic context. The pretrained VAE turns video and layout images into latent tensors. Concatenating views along width lets the existing self-attention connect cameras, and concatenating layout and noisy-video latents along channels introduces structural conditioning. Rectified flow learns a velocity from noise toward a clean video latent under those conditions. At inference the corrected trajectories occupy the layout interface. That interface is therefore also a possible distribution shift: imperfect predicted layouts replace the ground-truth layouts used for renderer training. The latter observation is a reader deduction from the specified training and inference paths. pipelinevideo-model
5.3 Read the evaluation as three tests, then inspect its measurement assumptions
Table 1 tests the complete system under nominal or deliberately corrupted 2D paths. Table 2 supplies valid ground-truth trajectories, asking whether difficult interactions still challenge the renderer. Table 3 measures trajectory correction itself, while Table 4 removes components. Together these are more informative than any single video-quality score, but the reader should keep their protocols separate. PHY includes frame and temporal quality alongside mass and impenetrability judgments, so an improved score need not isolate dynamics. CARLA appearance is translated by a model trained on nuScenes; its outputs become metric references as well as initial conditions. Appendix C also adapts the rectifier to stylized frames for one epoch, leaving the video generator unchanged. These details make the reported comparison specific and reproducible in principle, while leaving the independence of appearance and physical fidelity as an open experimental question. pipeline-resultsgenerator-resultsrectification-resultsablationevaluationstyle-transferbaseline-settings
5.4 Training and inference
During training
The rectifier sees unchanged pre-collision paths followed by constant-pre-impact-velocity continuations, supervised by actual simulator motion. Nominal nuScenes pairs remain uncorrupted. Event-window and agent weights emphasize impacts and participants. counterfactual-pairsphysical-objective
Physical training uses T=36 at 12 Hz, batch 256, and learning rates 9×10⁻⁵ for ResNet50 features and 9×10⁻⁴ for the main network; default event/agent weights are 10/5. Video training samples real/simulated clips 1:1 using ground-truth trajectories, not rectifier outputs. AdamW trains 2,850 steps at 224×400, learning rate 5×10⁻⁵, batch 480, then 350 steps at 448×800, learning rate 10⁻⁴, batch 240, on 48 H20 GPUs. trainingvideo-model
During inference
Rectify supplied 2D paths, construct camera layouts, and generate video conditioned on initial images, text and those layouts. The reported video is produced in one pass. No action-selection objective, vehicle command execution or feedback controller is demonstrated; correction can produce a collision and stop rather than an avoidance maneuver. pipelinevideo-modelbaseline-settingsrectification-resultsscope-boundary
5.5 Implementation flow
- Ground the trajectory tokens
Encode trajectories with sine–cosine features and an MLP. Deformable spatial cross-attention reads initial-image features; agent self-attention models interactions; map cross-attention supplies road context; an FFN refines the tokens. corrector
- Predict abrupt 6-DoF motion
Copy each refined token across future steps, concatenate learned time embeddings, project and apply temporal convolutions, then predict six state components with an MLP. This is supervised regression, without an explicit hard-constraint solver. temporal-headphysical-objective
- Render the corrected world
Project agent boxes and map polylines into calibrated cameras. Encode these layouts and videos using a pretrained 3D VAE. Concatenate views along latent width and layout/noisy-video features along channels before the Wan2.1 DiT. pipelinevideo-model
6. Experiments & results
PhyGenesis first repairs potentially impossible driving trajectories, then renders their consequences as multi-view video. Counterfactual CARLA trajectories supervise a 6-DoF rectifier, while real and simulated clips train a layout-conditioned Wan2.1 generator. Its strongest gains concern challenging simulated inputs; the evidence establishes improved trajectory and video metrics, with substantial evaluation and reproducibility qualifications.
6.1 Read the original evidence
Table 1. The largest full-pipeline improvements occur under corrupted CARLA trajectories. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the dataset groups: nuScenes uses nominal x/y trajectories, while the two CARLA groups use post-collision continuations that violate physics. Within each group, lower FID and FVD mean better visual distribution matching; higher PHY and preference are favored. Read the CARLA Ego columns vertically to compare methods under the same benchmark condition. The video evaluation samples 150 clips per test split, but human preferences use separate, smaller scene selections. Appendix B also matters: CARLA videos are translated into nuScenes style, providing both the initial frames and the reference videos used for FID and FVD. pipeline-resultsevaluationhuman-studystyle-transferbaseline-settings
What it supports. On CARLA Ego, PhyGenesis reports FVD 72.48 and PHY 0.71, compared with DiST-4D’s 197.57 and 0.39. On CARLA Adv, the corresponding values are 77.83 and 0.87 versus 128.88 and 0.56. These support the complete pipeline under this corruption and stylization protocol; they do not identify a single causal component.
Where the evidence stops. PHY averages perceptual VLM judgments, not measured mechanics. Appendix A specifies choosing among candidates with uncertain votes split equally, despite the main text’s pairwise description. Baseline resolutions, depth inputs and training histories also differ.
Table 2. Even a valid trajectory needs a renderer trained on difficult interactions. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Here every method receives ground-truth trajectories, removing the deliberately impossible motion conditions used for CARLA in Table 1. This asks whether a video generator can depict challenging dynamics when the structural input is already physically feasible. Read FID, FVD and PHY as before, then inspect CtrlErr, where lower is better. Section 4.1 defines it from rotation and translation errors, with camera poses estimated from generated video by ViPE. It is therefore an indirect condition-following measurement. For CARLA, keep the translated appearance domain in mind. Compare methods within this table before comparing their results with the distinct full-pipeline protocol. generator-resultsevaluationstyle-transferbaseline-settingsvideo-model
What it supports. On CARLA Ego, PhyGenesis achieves FVD 57.02 and CtrlErr 0.85, versus DiST-4D’s 133.10 and 1.19. On nuScenes, its FVD is 31.14 versus 45.24. The improvements under valid conditions support the need to address rendering as well as trajectory feasibility.
Where the evidence stops. This is not a matched-data renderer ablation: architectures, pretrained weights, output resolutions and conditioning differ. Estimated camera-pose error cannot establish accurate motion for every surrounding agent or prove physically correct contact.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Video generation from physics-violating CARLA Ego trajectories Table 1; 150 sampled test clips; stylized CARLA initial frames and reference videos; constant-velocity post-collision corruption. | 11.03; 72.48; 0.71; 0.71 FID ↓; FVD ↓; PHY ↑; Pref. ↑ | DiST-4D: 19.84; 197.57; 0.39; 0.10. Large gains for the complete pipeline. PHY combines VLM judgments of physical and visual quality; preference is a separate human study. Neither is a driving success rate. pipeline-resultsevaluationhuman-studystyle-transfer |
| Video generation from nominal nuScenes trajectories Table 1; ground-truth x/y inputs, 150 sampled nuScenes test clips. | 10.24; 40.41; 0.97 FID ↓; FVD ↓; PHY ↑ | DiST-4D: 10.49; 46.95; 0.86. Nominal performance also improves, with a much smaller FID difference than on CARLA Ego. pipeline-resultsevaluation |
| Video generation under ground-truth CARLA Ego trajectories Table 2; physically feasible ground-truth layouts; same stylized evaluation domain. | 10.98; 57.02; 0.69; 0.85 FID ↓; FVD ↓; PHY ↑; CtrlErr ↓ | DiST-4D: 19.94; 133.10; 0.38; 1.19. Rendering remains a distinct challenge even with valid trajectories. This protocol must not be merged with Table 1; CtrlErr uses estimated camera motion. generator-resultsevaluationstyle-transfer |
| 6-DoF trajectory rectification Table 3; physical agent and interaction partner; nominal nuScenes and corrupted CARLA conditions. | nuScenes 0.19; CARLA Ego 0.65; CARLA Adv 0.86. 6-DoF L2 distance ↓; units/coordinate normalization unspecified | Without correction: 0.21; 1.78; 1.05. The nominal improvement mainly recovers missing degrees of freedom. Mixed translation/rotation errors should not be read as meters. rectification-results |
| Contribution of trajectory correction and mixed training data Table 4; compare one removed component against the complete model. | Full model: CARLA Ego FVD 72.48, PHY 0.71; CARLA Adv FVD 77.83, Pref. 0.57. FVD ↓; PHY ↑; separate-study Pref. ↑ | Without correction: Ego 116.51/0.65. Without mixed data: Adv 89.83/0.15. Both components help selected challenging metrics. Table 4 lacks the neither-component control, and improvements are not uniform across every metric. ablationhuman-study |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. The temporal head targets the abrupt change around contact. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three panels as a local collision diagnostic, preserving the printed t and dY axes. The caption interprets the fall as a velocity decrease, but the graphic does not supply physical units for dY. Compare the timing and sharpness of the drop rather than treating the panels as identical calibrated measurements. The mechanism on page 8 first concatenates a copied agent feature with learned time embeddings, then applies a projection and temporal convolutions before the output MLP. This lets neighboring future steps interact while retaining step-specific information, which the authors motivate as useful for the rapid change at impact. temporal-headrectification-results
What it supports. The shown time-wise prediction resembles the abrupt ground-truth stop more closely than the displayed MLP prediction. This is a qualitative motivation for the head design, separate from the aggregate rectification result: Table 3 reports CARLA Ego 6-DoF distance decreasing from 1.78 without correction to 0.65 with the full corrector.
Where the evidence stops. The panels have different vertical ranges, and this example is not a head-only aggregate benchmark. The caption’s instantaneous-stop wording is stronger than a finite sampled trace can establish; no uncertainty or physical units are provided here.
Table 4. Both components help challenging cases, with metric-specific exceptions. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the two left columns as switches. The first row enables both mixed real/simulated training and the physical condition generator. Comparing it with the second row tests correction with mixed data retained; comparing it with the third tests mixed training while correction remains present. Follow a single metric and dataset across those rows. CARLA Ego FVD and CARLA Adv FVD make the two contributions particularly clear. Then inspect the exceptions: nuScenes preference is higher without mixed data, and CARLA Adv FID is slightly lower without correction. These are separate human studies from Table 1, so preference values have different comparison sets. ablationhuman-study
What it supports. Removing correction raises CARLA Ego FVD from 72.48 to 116.51 and lowers PHY from 0.71 to 0.65. Removing mixed training raises CARLA Adv FVD from 77.83 to 89.83. Table 4 reports Adv preference increasing from 0.15 without mixed data to 0.57 with it; the prose gives different values.
Where the evidence stops. Page 15 instead says preference 0.13→0.53; the discrepancy is unresolved. With no neither-component row and no uncertainty estimates, this table cannot fully estimate component interaction or establish uniform gains across all metrics.
7. Analysis & limitations
7.1 What the evidence leaves open
The experiments measure short generated clips and rectified trajectories, without closed-loop driving or real collision ground truth for the nuScenes stress test. PHY is a perceptual proxy, not a direct mechanics test; reported tables provide no confidence intervals. evaluationstress-testscope-boundarypipeline-resultsgenerator-results
Comparison conditions differ: baselines use pretrained weights, resolutions vary, UniMLVG uses two passes, and DiST-4D receives depth. CARLA FID/FVD references are themselves translated videos. These results do not isolate training data from architecture or establish unbiased physical fidelity. baseline-settingsstyle-transfer
Reporting conflicts remain: Appendix A describes multi-candidate voting despite the main text calling it pairwise; Table 4 gives CARLA Adv preference 0.15→0.57 whereas its discussion says 0.13→0.53. Use the table values without silently reconciling them. human-studyevaluationablation
Figure 2 places h labels after MLPs, contrary to Equations (5)–(6), and uses a circled-plus where Section 3.4 specifies concatenation. N denotes agents in the formulation but also labels diagram repetition; the implementation statement N=2 is ambiguous. The tutorial follows the explicit equations and text. pipelinecorrectortemporal-headvideo-modeltraining
7.2 Questions for discussion
- How much of the advantage survives matched data, resolution, initial conditions and generation budget?
- Does the temporal head generalize to held-out impact speeds and objects without erasing legitimate abrupt nominal maneuvers?
- Can independent geometric checks distinguish plausible-looking contact from mechanically accurate contact?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs nuScenes, CARLA/Bench2Drive routes, event annotations, synchronized cameras/calibration, Wan2.1 and the style-transfer pipeline. Appendix D gives perturbation ranges and rollout timing but omits exact sampling grids/interval widths. Split identities, simulator version, rectifier training duration, inference solver/step count, frozen-module policy and 36-step/33-frame alignment remain insufficiently specified. datascenario-constructionvideo-modeltrainingstyle-transferbaseline-settings
Appendix E defines decaying event windows and participant selection, but its weighting tables do not fully document fixed settings or reproduce the default agent weight 5. Table 6 CARLA Adv values range from 0.70 to 0.90; without uncertainty, the claimed limited sensitivity warrants a controlled rerun. Proposed checks should separately measure abrupt-event correction and rendering under matched layouts. physical-objectiveweighting-ablationtraining
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the temporal head recover impact timing beyond the training corruption?
Reader-proposed, not run: hold the attention backbone, training pairs, event/agent weights, optimizer budget and split fixed, and compare the time-wise head against a parameter-matched MLP. Hold out routes and impact-speed ranges from the CARLA construction, retaining a nominal nuScenes control. Measure per-coordinate trajectory error, impact-time error and post-contact penetration against simulator logs, reporting physical units separately rather than only a combined 6-DoF score. Repeat across seeds and inspect the same event-aligned traces. A reliable reduction in impact-time and penetration errors without worse nominal preservation would support the proposed mechanism; a benefit confined to the illustrated collision or seen speeds would weaken it. temporal-headcounterfactual-pairsphysical-objectivetrainingscenario-constructionrectification-results
Check 2: Can mixed training improve physical rendering under matched conditions?
Reader-proposed, not run: train the same Wan2.1-based renderer with nominal-only versus 1:1 mixed clips, matching initialization, total updates, resolution and inference settings. Evaluate a fixed held-out set with ground-truth layouts first, then corrected layouts, and include the neither-component control missing from Table 4. Score identical outputs against both raw and stylized CARLA references, while checking geometry against simulator annotations and using a declared multi-candidate human-vote protocol. Report confidence intervals over scenes and seeds. Persistent contact-quality gains across reference styles would support learned interaction rendering; gains confined to stylized FID/FVD would suggest that appearance alignment explains part of the improvement. video-modeltraininggenerator-resultsablationstyle-transferhuman-studybaseline-settings
8.3 Reading coverage
Visual audit: Visually inspected the title/byline, all Figures 1–13 and Tables 1–6, all method/evaluation/training pages and Appendices A–E, including equations, hardware, baseline settings, scenario construction and weighting details. The five final crops were individually viewed with labels, axes and table rows intact. The page-5 extraction damage was checked against the PDF; figure/notation and preference-reporting inconsistencies are disclosed. Reference-only pages 16–18 were read as text but not rendered. Separate videos, external supplements and code were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 20, 21, 22, 23, 24, 25. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract
- 1 Introduction
- 2 Related Work
- 3.1 Overview of PhyGenesis
- 3.2 Heterogeneous Multi-view Data
- 3.3 Physical Condition Generator
- 3.4 Physics-Enhanced Multi-view Video Generator
- 4.1 Experimental Setup
- 4.2 Performance of PhyGenesis
- 4.3 Performance of Physics-enhanced Multi-view Video Generator
- 4.4 Performance of Physical Condition Generator
- 4.5 Ablation Study
- 5 Conclusion
- References
- A User Study Setting
- B Style Transfer Model Used for Video Generation
- C Implementation Details of Different Baselines
- D Physical-Challenging Scenario Construction in CARLA Ego and CARLA Adv
- E Weighting Design and Ablation Study of event and agent weights
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Some PDF font mappings yielded unpaired Unicode surrogates; replacement characters are recorded by page in extractionNotes and require PDF visual verification.
- All six supplied text chunks were read, covering all 25 PDF pages. Original Figures 1–13 and Tables 1–6 were visually inspected; the page-5 symbol damage was checked against the rendered PDF.
- Version scope: the inspected title page identifies arXiv:2603.24506v2, 1 April 2026; the catalog submission date is 25 March 2026. Title and all thirteen authors agree. The manifest identifies the earlier retained HTML as the same v2 revision, but that HTML and v1 were not independently read or compared.
- Separate videos, external supplements, project resources and code were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author byline, affiliations and arXiv margin identifier
The title and all thirteen authors match the catalog. The inspected artifact is arXiv:2603.24506v2, dated 1 April 2026. PhyGenesis is the method name. The four affiliations are Zhejiang University, Xiaomi EV, The Hong Kong Polytechnic University, and Shenzhen Loop Area Institute.
Go to primary source ↓problemPDF pp. 2–3, Abstract and Section 1
The authors distinguish infeasible input trajectories from a lack of training examples of physically challenging but feasible interactions, motivating trajectory rectification and physics-rich video training.
Go to primary source ↓pipelinePDF pp. 4–5, Section 3.1 and Figure 2
Inputs include initial multi-view images, a static map and future agent x/y trajectories. A physical condition generator predicts corrected 6-DoF motion, which supplies projected layouts to PE-MVGen. Figure 2 shows spatial cross-attention, agent self-attention, map cross-attention, an FFN and a temporal output head.
Go to primary source ↓dataPDF p. 6, Section 3.2; p. 7, Figure 3
About 31 simulated hours yield 9.7 hours of selected challenging clips, combined with 4.6 hours of real nuScenes data. CARLA Adv and Ego contain 15.5 and 15.2 hours respectively. Six surround cameras record at 12 Hz. Collision sensors and map metadata identify events; Figure 3 shows more aggressive maximum-acceleration distributions.
Go to primary source ↓correctorPDF pp. 7–8, Section 3.3, Equations (1)–(4); p. 5, Figure 2
Sine–cosine trajectory encoding and an MLP produce agent tokens. Deformable spatial cross-attention uses initial-image perspective-view features, followed by agent self-attention, map cross-attention and an FFN. The predicted state contains x, y, z, pitch, yaw and roll.
Go to primary source ↓temporal-headPDF p. 7, Figure 4 and caption; p. 8, Equations (5)–(6); p. 5, Figure 2
The refined token is copied across future times and concatenated with learned temporal embeddings before projection and a TCN; an MLP maps temporal features to six outputs. Figure 4 contrasts a smoothed MLP response with a sharper time-wise response. Figure 2 places h labels after MLP blocks whereas Equations (5)–(6) define h before the final MLP.
Go to primary source ↓counterfactual-pairsPDF p. 8, Section 3.3, Training Pair Construction
Simulated pre-collision trajectories are retained; post-collision trajectories are corrupted by continuing all agents at their pre-collision velocities. Actual simulated trajectories supervise correction. Uncorrupted nominal nuScenes pairs are also included.
Go to primary source ↓physical-objectivePDF p. 9, Equation (7); pp. 23–24, Appendix E, weighting definitions
The physical generator minimizes a weighted 6-DoF L1 distance. Event windows extend from max(0, event time minus 1) to min(T minus 1, event time plus 10), with exponential decay from lambda_event to 1; overlapping windows use the maximum. Agent weighting emphasizes the perturbed agent and, for dynamic collisions, its nearest interaction partner.
Go to primary source ↓video-modelPDF pp. 9–10, Section 3.4, Equations (8)–(9)
PE-MVGen adapts Wan2.1. A pretrained 3D VAE encodes videos and camera-projected box/map layouts. View latents are concatenated along width; layout and noisy video latents are concatenated along channels. Rectified flow conditions on initial-frame features, text and layouts. Real and simulated clips are sampled 1:1; video training uses ground-truth physical trajectories, without counterfactual corruption.
Go to primary source ↓trainingPDF p. 11, Section 4.1, Implementation Details
Physical-model training uses 12 Hz trajectories, T=36, ResNet50 features, learning rates 9e-5 for the feature network and 9e-4 for the main network, batch 256, N=2, lambda_event=10 and lambda_agent=5. AdamW video training uses 2,850 steps at 224×400, learning rate 5e-5, batch 480, then 350 steps at 448×800, learning rate 1e-4, batch 240, on 48 NVIDIA H20 GPUs. Outputs have 33 frames at 12 Hz.
Go to primary source ↓evaluationPDF pp. 10–11, Section 4.1, Datasets, Evaluation Metrics and Baseline
Video evaluation samples 150 clips per test split. PHY averages WorldModelBench Mass, Impenetrability, Frame-wise Quality and Temporal Quality VLM judgments. CtrlErr is the geometric mean of rotation and translation errors with camera poses extracted using ViPE. DiST-4D additionally receives initial depth. The main text calls Pref. a pairwise preference percentage.
Go to primary source ↓pipeline-resultsPDF pp. 11–12, Section 4.2; p. 12, Table 1, all method rows and dataset columns
Table 1 uses nominal 2D nuScenes trajectories and corrupted CARLA trajectories. PhyGenesis reports FID/FVD/PHY/Pref. of 10.24/40.41/0.97/0.67, 11.03/72.48/0.71/0.71 and 9.28/77.83/0.87/0.66 for nuScenes, CARLA Ego and CARLA Adv. DiST-4D gives 10.49/46.95/0.86/0.13, 19.84/197.57/0.39/0.10 and 16.07/128.88/0.56/0.05. Baseline yaw is inferred from x/y displacement.
Go to primary source ↓generator-resultsPDF p. 13, Table 2 and Section 4.3
Under ground-truth trajectories, PhyGenesis reports FID/FVD/PHY/CtrlErr of 10.20/31.14/0.97/0.25 on nuScenes, 10.98/57.02/0.69/0.85 on CARLA Ego and 9.07/59.44/0.83/0.37 on CARLA Adv. DiST-4D gives 10.48/45.24/0.84/0.28, 19.94/133.10/0.38/1.19 and 16.12/105.70/0.50/0.57 respectively.
Go to primary source ↓rectification-resultsPDF p. 13, Table 3 and Figure 8; p. 14, Section 4.4
Reported 6-DoF L2 distances before/after rectification are 0.21/0.19 for nuScenes, 1.78/0.65 for CARLA Ego and 1.05/0.86 for CARLA Adv, evaluated on the physical agent and interaction partner. Figure 8 shows a penetrating guardrail trajectory corrected to a collision and stop. The nuScenes gain primarily recovers missing degrees of freedom.
Go to primary source ↓ablationPDF p. 14, Table 4 and Figures 9–10; p. 15, Section 4.5
With mixed data fixed, removing the physical generator changes CARLA Ego FVD/PHY from 72.48/0.71 to 116.51/0.65. With correction fixed, removing mixed data changes CARLA Adv FVD from 77.83 to 89.83 and Table 4 preference from 0.57 to 0.15. The p. 15 prose instead states preference 0.13 to 0.53. Table 4 includes three component combinations, without a neither-component row; nuScenes preference favors the nominal-only row, and CARLA Adv FID favors the row without correction.
Go to primary source ↓stress-testPDF p. 12, Section 4.2 and Figure 6; p. 13, Figure 7
The nuScenes stress test scales ego speed, retains collision cases and leaves initial frames unchanged. Qualitative comparisons and the plotted human preference and PHY scores favor PhyGenesis under these corrupted conditions.
Go to primary source ↓human-studyPDF pp. 19–20, Appendix A and Figure 11
Participants choose the highest-quality video among candidates or select uncertain. A choice earns one point; an uncertain response splits one point equally among candidates. Separate studies for Table 1, Figure 7 and Table 4 use five scenes, with each score based on 150 responses from 30 participants; 1,050 responses are reported overall.
Go to primary source ↓style-transferPDF pp. 20–21, Appendix B, Figure 12 and Equations (10)–(11); p. 22, Figure 13
The nuScenes-trained Wan2.1-Fun-V1.1-1.3B-Control translator uses Depth Anything V2 depth and Qwen2.5-VL full-video captions without initial-frame conditioning. Stylized CARLA clips supply initial frames and FID/FVD reference videos. Figure 13 specifies the caption content.
Go to primary source ↓baseline-settingsPDF pp. 21–22, Appendix C
Baselines use official pretrained weights. All outputs contain 33 frames, with UniMLVG using two passes and others one. PhyGenesis uses 448×800, MagicDriveV2 and DiST-4D 424×800, and UniMLVG 256×448. On stylized CARLA frames, the physical generator is adapted for one epoch, with PE-MVGen unchanged.
Go to primary source ↓scenario-constructionPDF pp. 22–23, Appendix D
Bench2Drive routes are perturbed in speed (predefined values between 0 and 30 m/s) and lateral offset (between −200 and 200 m), with three equally likely perturbation modes. After a 24-step warm-up, ego or nearby adversarial vehicles follow modified routes. Collection continues 48 frames after an event, or stops after 120 post-warm-up steps without an event. Exact discrete sampling values and interval widths are not enumerated.
Go to primary source ↓weighting-ablationPDF pp. 24–25, Appendix E, Tables 5–6
The authors describe limited sensitivity to weighting. Table 5 CARLA Adv entries for lambda_event=1,5,10 are 0.69,0.76,0.77; Table 6 entries for lambda_agent=1,10,20 are 0.70,0.76,0.90. Tables do not label metric units or fully specify the other fixed settings. Table 6 does not include the main-text default lambda_agent=5.
Go to primary source ↓scope-boundaryPDF pp. 10–15, Sections 4.1–5
Reported experiments assess generated videos, trajectory rectification, corrupted-condition stress tests and ablations. The conclusion proposes use in simulation-driven evaluation and safety testing; no executed driving-policy or closed-loop safety outcome is reported.
Go to primary source ↓8.5 Primary sources
Toward Physically Consistent Driving Video World Models under Challenging Trajectories ↗
PDF · 8,432 extracted words
Source fingerprint
c7ad34262d83f2b53badf19f059245d0d242cc7513267585e2163119e5f347b7