Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving
1. Paper overview
In one sentence: Uni-World VLA alternates imagined frames and waypoint predictions in a shared backbone, improving reported NAVSIM planning while leaving the causal contribution of feedback and deployment behavior unresolved. ev-identityev-inputsev-interleavingev-depthev-componentsev-schemesev-planningev-videoev-training
| At a glance | What to know |
|---|---|
| Research problem | Author claim A planner conditioned on a complete, fixed imagined rollout may use futures inconsistent with its evolving decisions. The authors propose alternating prediction and planning so each waypoint uses the latest imagined scene and later scenes condition on earlier predictions. This motivation is plausible; the experiments do not directly establish reactive behavior by surrounding traffic. ev-motivationev-interleavingev-evaluation |
| Core mechanism | Source description A shared autoregressive backbone interleaves visual prediction and action querying across a four-second horizon, with bidirectional attention inside each frame and causal attention across frame/action blocks. ev-interleavingev-mask |
| A key reported result | NAVSIM test-set trajectory planning: PDMS 89.4; NC 98.7; DAC 96.7; EP 83.2; TTC 96.1; Comfort 100.0. PDMS ↑; NC, DAC, EP, TTC, Comfort ↑. Official NAVSIM test split; single front camera; two-second history and four-second prediction. Authors describe PDMS evaluation as closed-loop. PWM: PDMS 88.1 with single camera; ResWorld: 89.0 with multi-view cameras and LiDAR. Best PDMS among listed methods, 1.3 score points above PWM. Sensor and training differences prevent a controlled architectural comparison. Page 10 prose says TTC 96.4; Table 1 and the full-model row of Table 3 say 96.1. ev-evaluationev-planningev-components |
| Reading caution | Reader analysis All reported experiments use NAVSIM; the paper provides no physical-driving study, latency measurement or uncertainty estimates. Its internal imagined feedback and benchmark scores do not demonstrate responsive surrounding agents or deployment readiness. ev-evaluationev-planningev-interleaving |
Core contributions
- Source description
A shared autoregressive backbone interleaves visual prediction and action querying across a four-second horizon, with bidirectional attention inside each frame and causal attention across frame/action blocks. ev-interleavingev-mask
- Source description
Historical visual embeddings query monocular-depth features through cross-attention. Component and scheduling ablations separately examine depth, future-frame modeling, pretraining and temporal alignment. ev-depthev-componentsev-schemes
Figure 2. Historical geometry conditions a shared sequence of imagined frames and waypoint queries. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a): orange blocks carry prompts, green and olive blocks represent dynamic and contextual imagery, and blue represents ego information. The depth-fusion arrow summarizes a more specific operation in Equations (10)–(11): visual embeddings query depth-encoder features. Then follow panel (b) from left to right. The current frame precedes the first predicted frame; an action query requests the waypoint at that predicted frame's timestamp. Dashed links carry generated visual tokens into subsequent context. Equations (2)–(3) additionally condition later generation on earlier actions. This is a recurrent use of one backbone, not evidence of multiple independent planners. ev-inputsev-depthev-interleavingev-objective
What it supports. The architectural change is when planning receives visual information: it can use each newly imagined frame before the next frame is produced. Historical depth supplies geometric conditioning, while action-query hidden states produce waypoints. The diagram supports shared computation and interleaved prediction; its practical value still needs the ablations.
Where the evidence stops. Figure 2(a) places ego before the user prompt, whereas the p. 6 input list reverses them. Preserve this packing ambiguity. The text also leaves the exact predicted-action feedback representation unclear; arrows alone do not resolve its implementation.
2. Motivation
2.1 The problem and the proposed response
A planner conditioned on a complete, fixed imagined rollout may use futures inconsistent with its evolving decisions. The authors propose alternating prediction and planning so each waypoint uses the latest imagined scene and later scenes condition on earlier predictions. This motivation is plausible; the experiments do not directly establish reactive behavior by surrounding traffic. ev-motivationev-interleavingev-evaluation
2.2 What this reading follows
A driving world model is useful to planning only if its imagined future informs decisions at the right time. Uni-World VLA makes that timing explicit: predict a frame, request a waypoint for the same timestamp, and continue the sequence using the growing context. Estimated depth enriches the historical visual embeddings. This reading follows the token flow, distinguishes joint training from inference, and uses the ablations to separate large pretraining gains from smaller scheduling and depth gains. The supplied v1 paper reports strong NAVSIM scores and competitive video FVD, but contains unresolved implementation and numerical inconsistencies. Its internal prediction loop should not be confused with a demonstrated physical driving system. ev-identityev-inputsev-interleavingev-depthev-componentsev-schemesev-planningev-videoev-training
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | IDM |
| Quadrant | Q2 · One Model × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM and One Model are supported by a shared backbone that predicts future visual tokens and regresses waypoint queries. Future-conditioned action prediction resembles the catalog's visual-planning interpretation of IDM, but the paper specifies joint interleaved generation rather than an explicit inverse-dynamics mapping from a state transition to its causing control. The precise IDM/Q2 assignment therefore remains uncertain; the recorded classification is preserved. ev-inputsev-interleavingev-objectiveev-mask
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Compress what stays stable and what changes
The information bottleneck starts before the shared backbone. A high-resolution MagVIT-v2 branch supplies 448 contextual tokens per frame, while a lower-resolution branch supplies only 28 dynamic tokens. The historical dynamic stream captures fine temporal changes, and context supplies scene structure. Depth Anything 3 estimates geometry from the input RGB images; two depth encoders produce features at the matching resolutions. In cross-attention, the visual embeddings are queries and the depth features are keys and values, so the resulting conditioning remains attached to the visual representation. Appendix A also matters: ego status is projected through an MLP at inference rather than quantized like imagery. These distinctions prevent the phrase 'unified tokens' from hiding differences in how images, ego state and action queries enter the backbone. ev-tokenizerev-inputsev-depth
Figure 3. Joint supervision and cached inference share a blockwise temporal attention structure. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. At upper left, panel (a) aligns the interleaved inputs with two supervised outputs: visual-token logits and trajectories. Beside it at upper right, panel (b) explains how information can flow. The solid green squares form fully connected within-frame blocks; the colored area expands toward earlier blocks as time advances, while gray cells exclude future information. Action rows can access the corresponding frame. This agrees with the p. 7 description of bidirectional intra-frame and causal inter-frame attention. Below both panels, read panel (c)'s four backbone calls as successive stages: generate the next frame, query its action, extend the visual context, and query again. The caption and inference text specify KV-cache reuse. ev-maskev-interleavingev-objectiveev-training
What it supports. World prediction is present during inference, rather than serving only as an auxiliary training target. A waypoint can depend on the current imagined frame. The block mask preserves temporal ordering while allowing spatial dependencies within each frame, and the cache avoids recomputing all preceding key/value representations.
Where the evidence stops. The figure does not specify how a frame is sampled under bidirectional attention, nor provide measured latency. It depicts joint supervision; the separate depth-pretraining and freezing schedule must be taken from p. 10.
5.2 2. Trace the same-timestamp frame-to-action dependency
Let the model reach the first future timestamp. Equation (2) predicts its dynamic visual tokens from the preceding visual and action context. Equation (3) then predicts the corresponding action representation with access to that newly predicted frame; an MLP converts the action hidden state to an ego position. Repeating this order gives eight frame/waypoint pairs over four seconds. Training supervises visual logits and positions together, with increased visual-loss weight on changing tokens. Inference instead extends a generated context and reuses cached keys and values. The paper calls this a closed loop, but the loop is between imagined representations. Fresh sensor feedback, vehicle dynamics during execution and the reaction of surrounding agents are separate questions not settled by the described autoregressive sequence. ev-interleavingev-objectiveev-maskev-evaluation
5.3 3. Separate benchmark gains from a causal explanation
Start with the internal component ladder rather than the leaderboard. Pretraining increases PDMS from 82.1 to 88.2, future-frame modeling raises it to 89.2, and depth raises it to 89.4. This identifies pretraining as the largest reported contributor, while depth's more pronounced benefit is in FVD. Next compare the five depth-free schedules: aligned 2-Hz interleaving leads, but timing, supervision density and action horizons vary together. Finally check the historical-input ablation: context-only nearly matches the full representation, while dynamic-only is much weaker. My interpretation is that the evidence supports a useful combined recipe, but leaves multiple explanations for its gains. A decisive follow-up would hold temporal frequency, training budget and initialization constant while changing only whether planning can access generated frames. ev-componentsev-schemesev-history
5.4 Training and inference
During training
Visual supervision uses dynamic-weighted cross-entropy, upweighting tokens that change between adjacent frames. Trajectory supervision is L1 waypoint error. Both losses train the shared predictor; no reinforcement-learning objective is described. ev-objective
Stage 1 freezes the foundation model and trains depth-related modules using action-free, one-second video prediction at 10 Hz for five epochs, learning rate 3×10⁻⁵. Stage 2 freezes the depth encoders and unfreezes fusion and foundation modules for 16 epochs, learning rate 2×10⁻⁵, using aligned Scheme E. ev-trainingev-schemes
Reported resources are 32 NVIDIA H20 GPUs, AdamW, cosine annealing and batch size 3, without clarifying global versus per-device batch size. The same page also states 30 epochs with the best PDMS checkpoint at epoch 16; its relationship to the five-plus-16-stage schedule is unresolved. ev-training
During inference
Generate a frame block, insert its action query, predict the corresponding waypoint, then continue to the next timestamp. Previous keys and values are cached. Within-frame attention is bidirectional, while later temporal blocks remain masked; the detailed within-frame sampling procedure is not supplied. ev-interleavingev-mask
The feedback described here is among generated representations. Newly imagined frames are not fresh camera measurements. Inference-time world prediction is integral to planning, but this internal loop alone does not establish real-world closed-loop control. ev-interleavingev-evaluation
5.5 Implementation flow
- Compress history
PWM-pretrained MagVIT-v2 branches encode 256×448 images into 448 contextual tokens and 128×224 images into 28 dynamic tokens; each codebook has 8,192 entries. Historical dynamic inputs capture 10-Hz motion. Appendix A clarifies that inference-time ego status is MLP-projected into embeddings rather than discretized. ev-inputsev-tokenizer
- Fuse geometric cues
Depth Anything 3 estimates depth. Context-depth and dynamic-depth encoders process the two image resolutions; visual embeddings provide cross-attention queries and depth features supply keys and values. Depth enriches historical conditioning, without explicit future-depth prediction. ev-depth
- Alternate frame and waypoint
The Phi-1.5-based Show-o backbone, initialized through PWM, generates future visual tokens, then processes a same-timestamp action query. An MLP regresses ego position from action-token hidden states. The formulation conditions later frames on preceding visual and action predictions. ev-inputsev-interleavingev-objectiveev-training
- Decode the imagined trajectory
MagVIT-v2 reconstructs RGB frames; waypoint outputs constitute the trajectory. Equation (4) also conditions decoding on per-second contextual tokens, but the future contextual-token construction is insufficiently explained. The output is a plan; the paper does not specify an actuator-level execution interface. ev-interleaving
6. Experiments & results
Uni-World VLA shares a multimodal backbone between future-frame generation and driving-trajectory prediction. It alternates imagined frames with waypoint queries and enriches historical RGB embeddings with estimated depth. NAVSIM results favor aligned 2-Hz generation, but the evidence concerns benchmark planning and generated imagery, not demonstrated physical deployment.
6.1 Read the original evidence
Table 1. The full model leads this table in aggregate planning score, with a single front-camera input. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Input column before comparing scores: SC means single-view camera, C means multi-view cameras, and C&L adds LiDAR to multi-view cameras. The five submetrics are no-at-fault collisions (NC), drivable-area compliance (DAC), ego progress (EP), time-to-collision (TTC), and comfort; larger scores are preferred throughout. The shaded rightmost column is aggregate PDMS. Compare Uni-World VLA first with the single-camera PWM row, then with multi-sensor ResWorld. Finally scan individual subscores: the best aggregate score does not imply that every safety or compliance component is maximal. The paper evaluates NAVSIM test trajectories over its four-second prediction setup. ev-evaluationev-planningev-componentsev-training
What it supports. Uni-World VLA reports PDMS 89.4 against PWM's 88.1 and ResWorld's 89.0. Its EP is 83.2, TTC 96.1 and comfort 100.0. These are the strongest listed aggregate results, but the mixed sensor inputs and inherited training pipelines make this a benchmark comparison rather than a controlled mechanism test.
Where the evidence stops. The p. 10 prose reports TTC 96.4; this table and Table 3's full-model row report 96.1. No uncertainty is supplied. The scores concern the supplied NAVSIM evaluation and do not document physical vehicle deployment.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM test-set trajectory planning Official NAVSIM test split; single front camera; two-second history and four-second prediction. Authors describe PDMS evaluation as closed-loop. | PDMS 89.4; NC 98.7; DAC 96.7; EP 83.2; TTC 96.1; Comfort 100.0. PDMS ↑; NC, DAC, EP, TTC, Comfort ↑ | PWM: PDMS 88.1 with single camera; ResWorld: 89.0 with multi-view cameras and LiDAR. Best PDMS among listed methods, 1.3 score points above PWM. Sensor and training differences prevent a controlled architectural comparison. Page 10 prose says TTC 96.4; Table 1 and the full-model row of Table 3 say 96.1. ev-evaluationev-planningev-components |
| Future-video generation NAVSIM, front view, maximum four seconds at 2 Hz. | 141.8. Fréchet Video Distance (FVD) ↓ | DrivingGPT 142.6 and SVD 227.5 under the table's matching dataset/view/horizon labels. The 0.8 FVD advantage over DrivingGPT has no reported uncertainty. Other columns use nuScenes or OpenDV and sometimes different views/horizons, so the entire table is not a matched ranking. ev-evaluationev-video |
| Pretraining, future-frame and depth ablation NAVSIM; cumulative component additions in Table 3. | PDMS 82.1 → 88.2 → 89.2 → 89.4; FVD 164.2 without depth and 141.8 with depth. PDMS ↑; FVD ↓ | Rows successively add pretraining, future-frame modeling and depth. Pretraining contributes the largest reported PDMS increment. Depth adds 0.2 PDMS and lowers FVD by 22.4, while TTC falls from 96.4 to 96.1 and NC from 98.8 to 98.7. Improvements are not uniform across safety subscores. ev-components |
| Frame-action generation schedule ablation NAVSIM; all five schemes exclude depth fusion. Some dense variants use nuPlan-supplemented 10-Hz trajectories. | A 88.3; B 86.1; C 87.8; D 85.7; E 89.2. PDMS ↑ | E aligns frame/action pairs at 2 Hz; alternatives change frequency, horizon ordering or overlapping action windows. Aligned Scheme E leads these variants. Frequency, supervision and token ordering change together; this does not isolate interleaving independently of protocol alignment. ev-evaluationev-schemes |
| Historical visual information ablation NAVSIM without depth fusion; Table 5. | Two-second context+dynamic: 89.2 / 164.2; one-second context+dynamic: 88.8 / 170.7; context only: 89.1 / 165.5; dynamic only: 81.7 / 203.6. PDMS ↑ / FVD ↓ | Full history and both branches outperform the listed reduced-input settings in aggregate. Context-only nearly matches the full representation, whereas dynamic-only loses substantial performance. Longer history does not lead every safety submetric. ev-history |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Pretraining dominates the planning gain; depth mainly improves the reported video metric. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three columns as a cumulative ladder. The first row excludes all three components; the second adds pretraining; the third also predicts future frames; and the fourth adds depth. Compare adjacent rows for the reported incremental change, while remembering that this is not a complete factorial experiment. PDMS increases to the right's preferred direction, whereas FVD decreases. The dashes in the first two FVD cells mean no video score is reported for those non-frame-generating configurations. After reading the aggregate columns, inspect NC and TTC: their maxima occur before depth is enabled, so the full configuration does not dominate every submetric. ev-componentsev-training
What it supports. The PDMS ladder is 82.1, 88.2, 89.2 and 89.4: increments of 6.1, 1.0 and 0.2 score points. Adding depth reduces FVD from 164.2 to 141.8, a 22.4-point decrease. Its planning gain is much smaller, and TTC decreases from 96.4 to 96.1.
Where the evidence stops. No repeated-run variability or compute-matched controls are shown. The depth model uses a staged training procedure, so the table alone does not isolate geometric information from all extra training and module effects.
Table 4. Alignment with the evaluation frequency matters more than simply increasing prediction frequency. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. All rows omit depth fusion. Read Scheme E as the paper's chosen order: a frame at 0.5 seconds, its action, then a frame at 1.0 seconds and its action, continuing to four seconds. The token-order formulas on p. 14 clarify the alternatives. A pairs dense early frames with more distant action timestamps. B uses dense frame/action alternation in the first second followed by dense action outputs; C changes the later action outputs to a coarser schedule. D inserts overlapping one-second action windows. Compare their PDMS and progress columns, rather than assuming more frequent queries necessarily improve planning. ev-schemesev-evaluation
What it supports. Aligned Scheme E scores 89.2 PDMS, compared with A's 88.3, B's 86.1, C's 87.8 and D's 85.7. It also leads EP at 82.9 in this table. The result favors the particular aligned schedule over these alternatives under the shared NAVSIM evaluation.
Where the evidence stops. The schemes change timing, output horizons and supervision together; some require nuPlan-derived 10-Hz targets. Frequency mismatch and conflicting-window explanations are author interpretations, not separately measured causes. The table cannot isolate a pure benefit from interleaving.
Figure 5. Selected depth-fused predictions retain sharper scene structure at the displayed horizons. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare each row horizontally: the left column is the reference image, the center is the prediction without depth fusion, and the right is the prediction with depth. The rows are labeled +2.0, +3.0 and +4.0 seconds, but depict different road scenes; do not read them as successive frames of one trip. In the middle row, examine vehicles and building edges. In the bottom row, compare the skyline, building facades and roadside barrier. The depth-fused images preserve more of this visible structure than the blurred center-column examples. The actual depth maps are absent here; the figure shows their downstream effect on RGB generation. ev-qualitativeev-componentsev-depth
What it supports. The selected examples make the reported FVD improvement visually plausible: depth-conditioned predictions look clearer in the displayed scenes, especially the later-horizon examples. They complement Table 3's FVD change from 164.2 to 141.8, but neither this crop nor distributional FVD measures the metric accuracy of inferred depth.
Where the evidence stops. These are selected qualitative examples without a stated sampling rule or failure count. Crisper pixels do not establish collision avoidance, calibrated geometry or reliable prediction of other road users. Different scenes across rows also prevent a controlled temporal-error comparison.
7. Analysis & limitations
7.1 What the evidence leaves open
All reported experiments use NAVSIM; the paper provides no physical-driving study, latency measurement or uncertainty estimates. Its internal imagined feedback and benchmark scores do not demonstrate responsive surrounding agents or deployment readiness. ev-evaluationev-planningev-interleaving
Figure 2(a) places ego information before the user prompt, while the p. 6 sequence reverses those items. Exact prompt packing, future contextual-token refresh, within-frame sampling and the representation used to feed predicted actions back remain insufficiently specified. ev-inputsev-interleavingev-mask
Figure 5 shows clearer structures with depth in selected examples. Figures 4, 7 and 8 compare predictions with reference frames/trajectories; these visualizations do not establish failure frequency or the prose's superiority over unpictured baselines. ev-qualitative
7.2 Questions for discussion
- Does access to generated frames improve decisions when temporal frequency and compute are held fixed?
- Can depth improve geometric prediction without reducing TTC or collision-related subscores?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires PWM/Show-o initialization, the dual-branch tokenizer, Depth Anything 3, NAVSIM splits and nuPlan alignment for dense ablations. Preserve the front-view restriction, two-second history, eight-step horizon and separate depth-training stages. ev-tokenizerev-depthev-trainingev-evaluationev-schemes
Before comparison, resolve the contradictory epoch accounting, batch-size scope, loss-weight values, precise NAVSIM release/split identifiers, depth preprocessing, action-feedback representation and contextual-token refresh. Software versions, seeds and FVD sample counts are not specified in the supplied method/setup. ev-trainingev-objectiveev-depthev-interleavingev-evaluationev-video
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Hold timing fixed and remove access to imagined frames
Reader-proposed check, not performed: initialize matched models from the same PWM checkpoint and train on the same NAVSIM split with two-second history, eight aligned 2-Hz outputs, no depth, identical loss weights and training budget. In the control, block action-query attention to predicted future-frame tokens while retaining historical context and the visual-generation objective; keep the original mask in the treatment. Evaluate paired scenes for PDMS and all subscores across multiple seeds, and measure latency. A reproducible planning decrease in the masked control would support an inference-time benefit from imagined visual evidence. Little or no difference would weaken that explanation even if visual prediction remains a useful shared-training task. ev-maskev-objectiveev-trainingev-componentsev-schemesev-evaluation
Check 2: Test whether depth contributes aligned geometry
Reader-proposed check, not performed: reproduce the final two Table 3 configurations with identical initialization and stage budgets, then add a parameter-matched control whose depth maps are shuffled across scenes before depth-encoder training. Preserve each map's value distribution and record the preprocessing. Use the same NAVSIM evaluation scenes and FVD sample set, report multiple seeds and paired confidence intervals, and stratify qualitative errors by horizon and turns. Compare valid-depth, shuffled-depth and no-depth runs on FVD, PDMS, NC and TTC. A reliable advantage confined to correctly aligned depth would support the geometric-conditioning account; equal gains from shuffled depth would suggest capacity or training effects. Explicitly test whether the reported TTC decline persists. ev-depthev-trainingev-componentsev-qualitativeev-videoev-evaluation
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block; all method equations and training/setup pages; Figures 1–8; and Tables 1–5. All six final original crops were separately viewed. Figure 2's prompt/ego ordering differs from the p. 6 sequence; Figure 3's mask agrees with the p. 7 blockwise attention description. The TTC discrepancy and the p. 10 training-schedule ambiguity are retained. Appendix A's token representations and Appendix B's selected examples were included in the visual pass. Reference-only pages 16–19 were read as text, not visually inspected. No external supplements or linked code were inspected, and no experiment was run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block and abstract (p. 1)
- 1 Introduction (pp. 1–4)
- 2 Related Work (p. 4)
- 3 Methods (pp. 5–9)
- 4.1 Experimental Setup (pp. 9–10)
- 4.2 Main Results (pp. 10–12)
- 4.3 Ablation Study (pp. 12–15)
- 5 Conclusion (p. 15)
- References (pp. 16–19)
- A Unified Discrete Tokenizer (p. 20)
- B More Visualization (pp. 20–22)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity: the inspected title and all eight authors match the catalog. The supplied PDF is arXiv:2603.27287v1 [cs.RO], dated 28 March 2026. No revision or title difference is evident in this artifact; no other edition was supplied for comparison.
- Acquisition omission preserved: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This reading therefore includes direct inspection of all eight figures and five tables.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read individually. PDF pages 1–15 and 20–22 were visually inspected; reference-only pages 16–19 were read as text.
- The linked code repository was not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title, author/affiliation block and arXiv margin
Exact title matches the catalog. Authors: Qiqi Liu; Huan Xu; Jingyu Li; Bin Sun; Zhihui Hao; Dangen She; Xiatian Zhu; Li Zhang. Affiliations: Fudan University; Shanghai Innovation Institute; Li Auto Inc.; University of Surrey. Margin identifies arXiv:2603.27287v1 [cs.RO], 28 Mar 2026.
Go to primary source ↓ev-motivationPDF pp. 2–3, Section 1 and Figure 1
The authors contrast predict-and-plan, predict-then-plan and interleaved generation, motivating stepwise coupling by possible inconsistency between a fixed imagined rollout and changing ego decisions.
Go to primary source ↓ev-inputsPDF pp. 5–6, Section 3 Overview/Inputs and tokenization, Figure 2(a–b)
Show-o uses MagVIT-v2 visual representations and ego velocity, acceleration, command and prompts. Historical dynamic tokens are sampled at 10 Hz. Figure 2(a) visually places ego before the user prompt; the p. 6 bracketed sequence puts user prompt before ego.
Go to primary source ↓ev-tokenizerPDF p. 10, implementation details; p. 20, Appendix A and Figure 6
Each MagVIT-v2 branch has an 8192-entry codebook. High-resolution 256×448 inputs yield 448 contextual tokens; 128×224 inputs yield 28 dynamic tokens. Appendix A states ego status is MLP-projected at inference and describes action/visual delimiters.
Go to primary source ↓ev-depthPDF p. 4, final paragraph; pp. 8–9, Depth integration, Eqs. (9)–(11)
Depth Anything 3 maps RGB to depth; CDE and DDE process 256×448 and 128×224 resized depth. Visual embeddings are queries, depth features keys/values. Fusion supplies historical conditioning without explicit future-depth modeling.
Go to primary source ↓ev-interleavingPDF pp. 6–8, Interleaved frame-action generation, Decoding and output, Inference; Eqs. (2)–(4); Figure 3(c)
Eight frame/action steps at 0.5-second intervals form a four-second horizon. Frames condition on preceding visual/actions; actions condition on the current predicted frame. An MLP produces positions; RGB decoding uses contextual tokens indexed c at t+2 floor(k/2). Inference appends generated representations and reuses a KV-cache.
Go to primary source ↓ev-objectivePDF p. 7, Training objectives, Eqs. (5)–(8)
Changed visual tokens receive weight alpha, unchanged tokens beta with alpha greater than beta. Weighted visual cross-entropy combines with L1 trajectory regression through lambda1 and lambda2. Numerical loss weights are not given.
Go to primary source ↓ev-maskPDF pp. 7–8, attention-mask description, inference text and Figure 3(a–c)
The mask permits attention within a frame and to earlier blocks while masking later blocks; the same construction is used for historical frames and inference. Training compares frame logits and regressed trajectories with ground truth. Within-frame sampling details are not specified.
Go to primary source ↓ev-trainingPDF pp. 9–10, Section 4.1 Implementation details and Stage 1/Stage 2
PWM initializes the Show-o model. Reported training uses AdamW, cosine annealing, 32 NVIDIA H20 GPUs and batch size 3. The text states 30 epochs with best checkpoint at 16, then describes five epochs at 3e-5 for frozen-foundation action-free video training and 16 epochs at 2e-5 with frozen depth encoders and trainable fusion/foundation modules.
Go to primary source ↓ev-evaluationPDF pp. 9–11, Section 4.1 Dataset/Evaluation metrics, implementation and Tables 1–2 captions
Experiments use NAVSIM official train/validation/test split, typically eight future frames over four seconds; front-camera input and two-second history are specified. Dense action variants obtain 10-Hz trajectories from nuPlan. PDMS includes NC, DAC, TTC, comfort and progress; FVD evaluates video distributions.
Go to primary source ↓ev-planningPDF p. 11, Table 1, Uni-World VLA/PWM/ResWorld rows; p. 10, Section 4.2 opening paragraph
Uni-World VLA: NC 98.7, DAC 96.7, EP 83.2, TTC 96.1, Comfort 100.0, PDMS 89.4. PWM PDMS is 88.1 and ResWorld 89.0. Their inputs are respectively single-camera, single-camera and multi-camera plus LiDAR. The p. 10 prose instead reports full-model TTC 96.4. No uncertainty is tabulated.
Go to primary source ↓ev-videoPDF p. 11, Table 2, FVD and Max Duration/Fps/Dataset/View rows
Ours FVD 141.8, DrivingGPT 142.6 and SVD 227.5 share NAVSIM/front/4s/2Hz labels. WoVoGen uses nuScenes/multiview/2.5s/2Hz, DriveDreamer nuScenes/multiview/4s/2Hz and GenAD OpenDV/front/4s/2Hz. FVD sample counts and uncertainty are not given.
Go to primary source ↓ev-componentsPDF pp. 12–13, Section 4.3; Table 3, all four rows
Cumulative addition of pretraining, future frames and depth changes PDMS from 82.1 to 88.2 to 89.2 to 89.4. Future-frame models have FVD 164.2 without depth and 141.8 with depth; corresponding TTC changes 96.4 to 96.1 and NC 98.8 to 98.7.
Go to primary source ↓ev-schemesPDF pp. 13–14, Section 4.3; Table 4 and Scheme A–E token-order formulas
Without depth, PDMS for A/B/C/D/E is 88.3/86.1/87.8/85.7/89.2. Scheme E alternates same-timestamp frames and actions at 0.5-second intervals through four seconds. Other schemes vary sampling rates, dense/coarse action horizons or sliding windows; explanations of mismatch/conflicting supervision are author interpretations.
Go to primary source ↓ev-historyPDF pp. 14–15, historical visual information paragraph and Table 5
Without depth: two-second context+dynamic gives PDMS/FVD 89.2/164.2; one-second gives 88.8/170.7; context-only 89.1/165.5; dynamic-only 81.7/203.6. One-second history has higher NC/TTC, and context-only has higher DAC.
Go to primary source ↓ev-qualitativePDF p. 12, Figure 4 and discussion; p. 13, Figure 5 and discussion; pp. 20–22, Appendix B and Figures 7–8
Figure 4 pairs original/predicted frames and BEV plans. Figure 5 shows reference, no-depth and depth-fused examples at 2, 3 and 4 seconds. Appendix B shows six selected scenarios with predicted/reference frames and BEV trajectories; Figure 8 identifies green as ground truth and red as planned.
Go to primary source ↓8.5 Primary sources
Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving ↗
PDF · 7,712 extracted words
Source fingerprint
09aa65ced8445e87e0d93dd8d9d88fdcfbe335ec30579e47579e5ddfe801d41e