UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
1. Paper overview
In one sentence: Predicting an image after a driving plan supplies useful training feedback, while discrete and continuous visual decoders trade speed against image fidelity. architectureplannerjoint-trainingclosed-loopspeed-tradeoffconfigurationorder-ablation
| At a glance | What to know |
|---|---|
| Research problem | Source description Driving requires geometric understanding, continuous plans and predictions of what follows those plans. The authors argue that text-only interfaces discard visual detail and weaken coupling between these tasks. UniDrive-WM learns them within a shared VLM, treating ego trajectories as actions and front-view images as part of the future state. motivationarchitecture |
| Core mechanism | Source description A shared VLM connects structured multi-view/history representations, a differentiable latent waypoint planner and trajectory-conditioned image prediction. architectureplannerjoint-training |
| A key reported result | Bench2Drive closed-loop driving: AR: 79.22 / 56.36%; AR+Diffusion: 79.31 / 56.42%. Driving Score (higher) / Success Rate (%) (higher). 950 training scenes; 220 routes across 44 scenarios; cameras and navigation commands. ORION: 77.74 / 54.62%. AR+Diffusion gains 1.57 score units and 1.80 percentage points (reader-calculated). Other methods retain higher efficiency or comfort scores. protocolclosed-loop |
| Reading caution | Reader analysis Continuous generation uses sampled noise and flow matching, then is described as deterministic reconstruction. Noise handling and the inference solver are insufficiently specified to reconcile these descriptions; reduced diversity is acknowledged. continuous-decoder |
Core contributions
- Source description
A shared VLM connects structured multi-view/history representations, a differentiable latent waypoint planner and trajectory-conditioned image prediction. architectureplannerjoint-training
- Source description
Two visual decoders expose a fidelity/latency tradeoff; component and prediction-order ablations test whether image supervision helps planning. ar-decodercontinuous-decoderspeed-tradeoffcomponent-ablationorder-ablation
Figure 2. A shared reasoning backbone receives structured visual history and serves three output tasks. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the multi-view input from left to right. The vision encoder extracts image features; QT-Former combines them with perception, scene and history queries before projecting visual representations into the LLM. Instructions take a separate tokenizer path into text embeddings. The LoRA marker identifies adaptation of the language backbone, while the output panel lists trajectory planning, future-image generation and VQA. The figure is an architectural overview: Equation (2) and Section 3.4 supply the missing ordering detail, namely that the predicted plan conditions the future image. History is retrieved through the memory mechanism described alongside the figure, rather than through the text prompt alone. architecturejoint-trainingperception-training
What it supports. The common LLM and structured visual inputs support a shared-backbone interpretation of the system. The output tasks are connected through representations and training, while the planner and image decoders retain different numerical interfaces. This architecture gives a concrete basis for the joint-prediction classification.
Where the evidence stops. The three output arrows do not establish simultaneous independent predictions or a replanning loop. Read their dependency from Equation (2) and Section 3.4. Appendix C also specifies a frozen detection head, despite the broad end-to-end description.
2. Motivation
2.1 The problem and the proposed response
Driving requires geometric understanding, continuous plans and predictions of what follows those plans. The authors argue that text-only interfaces discard visual detail and weaken coupling between these tasks. UniDrive-WM learns them within a shared VLM, treating ego trajectories as actions and front-view images as part of the future state. motivationarchitecture
2.2 What this reading follows
UniDrive-WM asks whether learning to picture a planned future can improve the plan itself. Multi-view images and temporal memory enter a shared vision–language backbone; a latent planner produces waypoints, and a visual decoder predicts the future front-view image conditioned on planning features. The useful distinction is between this forward prediction order and the backward training signal supplied by image losses. The paper reports gains over ORION in simulated closed-loop driving and open-loop prediction, but the architecture does not specify an iterative imagination-based controller. Read the decoder comparison alongside its resolution and latency differences, and use the ablations to judge what the coupling actually establishes. architectureplannerjoint-trainingclosed-loopspeed-tradeoffconfigurationorder-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Pending verification |
| Prediction paradigm | Joint prediction |
| Quadrant | Pending verification |
3.1 Evidence-based assessment
Supports the recorded classification
WAM/joint prediction is supported: a shared VLM predicts actions and their conditioned visual future. Figures 2–3 establish a shared backbone with distinct planner/visual decoders, stronger evidence than joint losses alone. This supports a shared-model interpretation, not an identical decoder for all modalities. No inverse-dynamics extraction or explicit inference-time image search is specified. The pending architecture/quadrant fields remain unchanged. architecturear-decodercontinuous-decoderjoint-trainingplanner
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 From camera history to a differentiable action representation
Begin with what the model knows before it imagines anything. QT-Former compresses current multi-view image features through scene and perception queries, while history queries retrieve temporally indexed memory. The current scene and updated history representations enter the same LLM space as the language instruction. The planner then reads the planning-related hidden representation and predicts the parameters of a Gaussian action latent. Reparameterization makes this stochastic interface differentiable, and a recurrent decoder turns the latent into continuous two-dimensional waypoints in ego BEV coordinates. Collision and boundary terms connect those waypoints to predicted traffic boxes and road structure; expert-waypoint error supplies imitation supervision. The paper omits explicit KL regularization and explains that choice as a training-stability decision. The resulting output is a numeric trajectory, rather than a natural-language driving instruction. architectureplannerperception-training
Figure 3. Two image decoders attach to the shared planning and reasoning backbone. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the visual paths rather than just the output photographs. On the left, the visual token sequence lies between image-start and image-end markers, and the MoVQGAN decoder converts it to pixels. On the right, learnable queries provide continuous features through an MLP and cross-attention to the diffusion model. Both panels retain a trajectory planner and a VQA route. The drawing does not show an explicit arrow from the predicted trajectory to image decoding; Sections 3.4 and 4.3 establish that planning tokens or features precede and condition the visual representation. That textual specification is essential for interpreting the apparent parallel branches. ar-decodercontinuous-decoderjoint-trainingspeed-tradeoff
What it supports. The two paths instantiate different representations of the planned visual future. AR uses discrete next-token prediction; AR+Diffusion uses continuous queries, flow matching and semantic image alignment. The figure explains where the additional decoder enters, while the measured speed and fidelity tradeoff must be read from the experiments.
Where the evidence stops. The continuous branch's equations sample noise, but the following prose calls inference deterministic reconstruction. The source leaves noise handling and solver details unresolved. This schematic should not be used to infer a standard diffusion sampling schedule.
5.2 Why a later image can help an earlier plan
Planning precedes generation in the forward sequence. In AR, visual tokens follow the planning token; in AR+Diffusion, learnable image queries follow planning features. During training, however, a loss on those visual predictions can send gradients into the representations on which they depend. This is the useful distinction between the solid information arrows and dashed backpropagation arrows in Figure 1. Reader interpretation: learning to explain future pixels may regularize the shared scene and planning representations, even when no generated image is fed back into a controller at inference. Table 6 supports a benefit from adding generation, and Table 7 supports planning-first ordering. Neither comparison isolates that gradient mechanism from extra supervision, altered conditioning or training cost. An explicit stop-gradient control would sharpen the causal claim. motivationjoint-trainingcomponent-ablationorder-ablation
5.3 Keep fidelity, planning accuracy and executed driving separate
The paper evaluates three different consequences of the architecture. FID measures future-image distributions, open-loop L2 and collision compare predicted plans under a dataset protocol, and closed-loop Driving Score and Success Rate measure simulator routes. Improvement on one does not logically guarantee improvement on another. AR+Diffusion improves FID over AR, but it also uses much larger output images and runs at 0.4 fps rather than 2 fps in the reported A100 measurement. Its closed-loop gains over AR are small. Across other baselines, the pattern is also mixed: FSDrive has lower nuScenes collision, while DriveTransformer-Large has lower Table 1 L2. Reader interpretation: the evidence favors useful multitask coupling, with a practical decoder tradeoff, rather than a universal dominance claim or demonstrated real-vehicle readiness. image-fidelityprotocolconfigurationspeed-tradeoffclosed-loopnuscenes
5.4 Training and inference
During training
Initialize from ORION perception pretraining with Hungarian assignment and focal/L1 supervision. Freeze the detection head. Stage 1 combines planning/image prediction with LoRA; Stage 2 adds VQA data. The continuous-decoder description also freezes its vision encoder. End-to-end therefore does not mean every module is trainable. perception-trainingjoint-trainingcontinuous-decoder
The setup uses EVA-02-L, Vicuna 1.5 and eight H200 GPUs. LoRA rank/alpha are 16/16. Scene/perception/history query counts are 512/600/16, with 16-frame memory. Inputs are 640×640; outputs are 192×128 for AR and 512×1024 with 64 latent queries for AR+Diffusion. configuration
During inference
Encode observations/history, predict a trajectory, then generate its conditioned image. The source describes supervisory feedback but provides no explicit iterative image-based trajectory search or replanning algorithm. The learned output is a waypoint plan; closed-loop route evaluation separately measures executed simulator behavior. architecturejoint-trainingplannerprotocol
5.5 Implementation flow
- Compress scene and history
QT-Former attends to image features with 3D positional encodings. History queries retrieve timestamp-embedded memory and current scene context; MLPs project updated history and scene features into the LLM. architecture
- Translate reasoning into waypoints
The planning representation predicts Gaussian latent parameters. Reparameterized samples feed a recurrent waypoint decoder. This provides a numeric interface instead of parsing free-form trajectory text; explicit KL regularization is omitted. planner
- Generate the planned future
AR predicts visual tokens after planning and detokenizes them with MoVQGAN. AR+Diffusion places continuous queries after planning features, maps them through an MLP, and conditions a diffusion decoder through cross-attention. ar-decodercontinuous-decoderjoint-training
6. Experiments & results
UniDrive-WM couples a shared driving VLM to a waypoint planner and a future front-view image generator. Planning tokens precede visual prediction, allowing image supervision to improve shared representations. Discrete AR is faster; continuous AR+Diffusion yields lower FID. Bench2Drive closed-loop gains over ORION are modest, and do not establish real-vehicle safety or an inference-time imagination-and-replanning loop.
6.1 Read the original evidence
Table 1. Closed-loop improvements over ORION coexist with metric-specific strengths of other baselines. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Condition and Modality columns: NC means navigation command, TP target point, C camera and L LiDAR; an asterisk marks expert feature distillation. The most relevant architectural baseline is ORION, which shares camera input and navigation-command conditioning with both proposed variants. Driving Score combines completion with infraction penalties, while Success Rate measures routes completed within the time limit. Keep these closed-loop columns separate from the final open-loop L2 column, whose caption specifies a two-second horizon sampled at 2 Hz. The official closed-loop protocol comprises 220 short routes across 44 scenarios; it is a simulator evaluation. closed-loopprotocolspeed-tradeoff
What it supports. AR+Diffusion reports Driving Score 79.31 and Success Rate 56.42%, versus ORION's 77.74 and 54.62%. Those are gains of 1.57 score units and 1.80 percentage points, calculated from the table. AR is close at 79.22 and 56.36%, so the slower decoder adds little on these two measures.
Where the evidence stops. Do not turn this into an across-the-board ranking: DriveTransformer-Large has lower open-loop L2 (0.62 versus 0.63), and MomAD has higher efficiency and comfortness. The table gives no uncertainty intervals for the small AR/AR+Diffusion differences.
Table 4. Lower FID favors both proposed decoders, with a further gain for continuous decoding. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally within each dataset row. The Type row groups methods by generative architecture, but the actual comparison is the FID value below it: lower means the generated frame distribution more closely matches the ground-truth future-frame distribution. The two rightmost columns compare the paper's own AR and AR+Diffusion variants; FSDrive provides a nearby unified-model comparison. Keep Bench2Drive and nuScenes separate because they contain different image distributions. Then return to the implementation section: AR predicts at 192×128, whereas AR+Diffusion predicts at 512×1024. The fidelity comparison therefore includes a substantial resolution and decoder-capacity change. image-fidelityconfigurationspeed-tradeoff
What it supports. On Bench2Drive, AR and AR+Diffusion report FID 7.2 and 6.6, compared with FSDrive's 9.3. On nuScenes the corresponding values are 7.8 and 7.3, versus 10.1. These reported values support a distributional image-quality gain for the proposed systems under the paper's evaluation setup.
Where the evidence stops. FID does not measure whether a particular trajectory causes the correct particular future, nor does it verify safety. The different output resolutions and reported speeds also prevent interpreting the AR/AR+Diffusion gap as a controlled representation-only ablation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Bench2Drive closed-loop driving 950 training scenes; 220 routes across 44 scenarios; cameras and navigation commands. | AR: 79.22 / 56.36%; AR+Diffusion: 79.31 / 56.42%. Driving Score (higher) / Success Rate (%) (higher) | ORION: 77.74 / 54.62%. AR+Diffusion gains 1.57 score units and 1.80 percentage points (reader-calculated). Other methods retain higher efficiency or comfort scores. protocolclosed-loop |
| Bench2Drive open-loop planning and detection 50 validation scenes; VAD/ORION cumulative-time evaluation; 3-second entries. | AR+Diffusion: 1.066 m, 0.657%; mAP 0.675, NDS 0.755. L2 (m), collision (%), mAP and NDS | ORION: 1.129 m, 0.743%; mAP 0.646, NDS 0.723. These ORION metrics improve. VAD has lower collision (0.296%); UniAD uses different temporal aggregation. protocolplanning-perception |
| nuScenes open-loop planning Official train/validation split. | AR: 0.30 / 0.31%; AR+Diffusion: 0.29 / 0.31%. Average L2 (m) / collision (%) | ORION: 0.34 / 0.37%; FSDrive: 0.60 / 0.19%. Both metrics improve over ORION; FSDrive retains lower collision. Dataset-specific evaluation does not establish zero-shot transfer. protocolnuscenes |
| Future-frame generation Bench2Drive and nuScenes future-frame evaluation. | AR: 7.2 / 7.8; AR+Diffusion: 6.6 / 7.3, in dataset order. FID (lower) | FSDrive: 9.3 / 10.1. Distributional fidelity improves. Different resolutions and decoder costs confound attributing the gap solely to representation; FID does not verify collision avoidance. image-fidelityconfiguration |
| DriveLM GVQA understanding DriveLM GVQA benchmark. | 0.59. Final Score (higher) | FSDrive: 0.57. This aggregate understanding gain is distinct from the separately scaled Chat-B2D ablation and from physical execution. vqaprotocol |
| AR component ablation Bench2Drive open-loop; detection+planning versus all three components. | With image generation: 1.079 / 0.668%. 3-second L2 (m) / collision (%) | Without image generation: 1.130 / 0.746%. Image supervision helps this system, but the comparison does not isolate gradient coupling from extra supervision or compute. component-ablation |
| AR prediction-order ablation Bench2Drive open-loop; planning/generation head order. | Planning then generation: 0.64, 0.43%, 7.2. Average L2 (m), collision (%), FID | Generation then planning: 0.67, 0.45%, 9.1. All three metrics favor the proposed order; counterfactual dynamics accuracy is not established. order-ablation |
| Bench2Drive multi-ability evaluation Closed-loop base-set ability groups. | AR: 59.00%; AR+Diffusion: 59.23%. Mean ability (%) | ORION: 54.72%. The mean improves, while Give Way remains 40.00%, below several target-point baselines at 50.00%. multi-abilityprotocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 6. Image prediction and perception supervision both help the AR planner in the reported ablation. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the checkbox columns to identify the experiment before reading the numbers. The first row keeps planning and image generation without detection; the second keeps detection and generation without planning, so its planning entries are dashes. The third row keeps detection and planning but removes image generation. The last row includes all three. Compare the last two rows to examine adding image prediction to a perception-informed planner. Then compare the first and last to see the dependence on detection supervision. The right-hand metrics assess detection features, but Appendix C says the detection head is frozen during the joint planning/generation stage. component-ablationperception-trainingjoint-training
What it supports. Adding image generation changes three-second L2 from 1.130 m to 1.079 m and collision from 0.746% to 0.668%. Without detection, the planning-plus-generation row reaches 2.146 m and 1.77%. The evidence therefore supports complementary benefits from perception supervision and image prediction within this AR setup.
Where the evidence stops. These switches alter available supervision and training objectives; they do not isolate the image-loss gradient pathway under matched compute. The detection checkbox should be interpreted with the pretraining/freeze description, not as proof that detection loss is jointly optimized throughout.
Table 7. Planning before generation improves both visual fidelity and open-loop planning metrics. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The plus signs in the row labels denote the order being compared, not merely the inclusion of two losses. Section 4.6 spells this out as Generation→Planning versus Planning→Generation. Read each column downward: every metric is lower-is-better, so the second row is favorable throughout. This experiment is specific to the AR architecture on Bench2Drive. Its most direct architectural interpretation is that visual tokens benefit from the preceding planning representation. The paper associates this sequence with the action-to-observation direction, while the joint loss also allows supervision on later image tokens to influence earlier shared representations during training. order-ablationjoint-training
What it supports. Planning→Generation yields average L2 0.64 m, collision 0.43% and FID 7.2, compared with 0.67 m, 0.45% and 9.1 for the reverse order. These values support the proposed token order in this AR comparison, with a particularly visible improvement in the image-quality metric.
Where the evidence stops. The result does not prove causal world knowledge or inference-time plan refinement. No uncertainty intervals are shown, and swapping token order changes the conditioning context. A gradient-detachment control would be needed to separate forward conditioning from backward supervisory benefits.
7. Analysis & limitations
7.1 What the evidence leaves open
Continuous generation uses sampled noise and flow matching, then is described as deterministic reconstruction. Noise handling and the inference solver are insufficiently specified to reconcile these descriptions; reduced diversity is acknowledged. continuous-decoder
A100 inference is 2 fps for AR and 0.4 fps for AR+Diffusion. These measurements alone do not establish a real-time control deadline. Longer-horizon, more interactive driving remains future work. speed-tradeofffuture-work
Principal comparisons and ablations provide no uncertainty intervals. Figure 6 also combines speed keeping with a red-light stop instruction, illustrating why fluent VQA needs consistency checks. closed-looporder-ablationqualitative
7.2 Questions for discussion
- Would planning gains survive detaching image-loss gradients under a matched training budget?
- Does changing only the trajectory produce correct ego-motion changes in generated frames?
- How should latency, collision rate and image fidelity be balanced under fixed closed-loop compute?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires ORION/QT-Former initialization, the specified visual/language backbones, driving/VQA annotations, decoder configurations and benchmark splits. Resolve optimizer/schedule, stage duration, data mixture, future-frame offset, controller/simulator settings and decoder integration details: the supplied method and appendix do not provide a complete executable configuration. configurationprotocoljoint-trainingcontinuous-decoderperception-trainingplanner
Reader-proposed checks: repeat component ablations with matched optimization budgets and detached image gradients; compare prediction orders with identical data, resolution and decoding budgets. Measure planning and image metrics across seeds before attributing gains to coupling. component-ablationorder-ablationconfiguration
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate image supervision from gradient coupling
Reader-proposed experiment, not performed: train three AR variants from the same perception/VLM initialization—planning without image generation, full joint training, and joint forward conditioning with image-loss gradients blocked from the planner and shared backbone. Keep training scenes, ordering, main-task losses, optimization steps and VQA mixture fixed; record actual compute rather than assuming equal cost. Evaluate the Table 6 planning and detection metrics plus FID across multiple seeds. A persistent planning gain for full coupling over the detached variant would support the backward-feedback explanation. If detached and coupled variants perform alike, extra supervision, initialization or another shared training effect would remain a plausible explanation. component-ablationjoint-trainingperception-trainingconfiguration
Check 2: Test prediction order under matched budgets and changed plans
Reader-proposed experiment, not performed: repeat Table 7 with the same AR resolution, token budget, data split, initialization and optimizer schedule for both orders. First test whether the reported L2, collision and FID ranking persists across seeds. Then hold the scene input fixed and replace the planning condition with feasible alternative trajectories, examining whether generated ego-motion and road geometry change consistently. Use matched simulator rollouts as references where available, recording the reference protocol explicitly. If planning-first only improves marginal FID while images scarcely respond to changed plans, the ordering advantage would not substantiate action-conditioned counterfactual prediction. This proposed intervention goes beyond the paper's reported ablation. order-ablationar-decoderarchitectureconfigurationprotocol
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, Figures 1–8, Tables 1–9, main-text method equations, training/evaluation details, and Appendix A–D evidence pages. All six final original crops were separately viewed. Figure 3 omits a direct planning-to-image arrow; its conditioning was checked against Sections 3.4 and 4.3. The flow-matching versus deterministic-reconstruction ambiguity is preserved. Bibliography pages 16–20 were read as supplied text but not rendered; no independent scientific claims are based on those references. No separate supplement or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 21, 22, 23, 24, 25, 26, 27. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract
- 1 Introduction
- 2 Related Work
- 3 Method (3.1–3.4)
- 4 Experiments (4.1–4.6)
- 5 Conclusion and Future Work
- Bibliography
- Appendix A Evaluation Metrics
- Appendix B More Results
- Appendix C Perception Module
- Appendix D Trajectory Planner
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the supplied PDF is arXiv:2601.04453v4, dated 1 July 2026. Its title and all eight authors match the catalog; the catalog submission date is 7 January 2026. Earlier revisions were not supplied, so their scientific differences are unverified.
- All six supplied text chunks were read, covering the 27-page PDF. Text extraction does not reconstruct figure images; this was addressed by inspecting the retained PDF figures and equation/table layouts.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- Code, project links and external resources were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author/affiliation block and arXiv margin
The title and eight authors match the catalog. The supplied artifact identifies itself as arXiv:2601.04453v4, 1 July 2026, and prints four institutional affiliations.
Go to primary source ↓motivationPDF pp. 2–3, Section 1; PDF p. 1, Figure 1 and legend
The authors motivate jointly learning understanding, planning and image prediction to avoid text-only bottlenecks. Figure 1 distinguishes forward information flow from dashed red backpropagation.
Go to primary source ↓architecturePDF pp. 4–6, Section 3, Equations (1)–(5), Figure 2; PDF p. 22, Appendix C.1
Multi-view images, history and perception features form the multimodal state. QT-Former processes scene, perception and history queries, retrieves time-embedded memory, and projects history/scene representations to the LLM. Equation (2) factors planning before future-state generation.
Go to primary source ↓plannerPDF p. 6, Section 3.2; PDF pp. 23–24, Appendix D, Equations (16)–(18)
MLPs predict Gaussian mean and variance from VLM planning embeddings; reparameterized samples feed a recurrent decoder of continuous 2D ego-BEV waypoints. Explicit KL regularization is omitted. Planning uses collision, boundary and expert-trajectory losses.
Go to primary source ↓ar-decoderPDF pp. 7–8, Section 3.3, Figure 3(a), Equation (7); PDF p. 9, Section 3.4
The discrete branch extends the visual codebook, brackets image tokens with special tokens, and decodes through MoVQGAN. Image prediction is conditioned on preceding planning tokens and trained with teacher-forced cross-entropy.
Go to primary source ↓continuous-decoderPDF pp. 7–9, Figure 3(b), Section 3.3, Equations (8)–(12)
Learnable queries connect VLM features to a diffusion decoder. Equations (9)–(11) interpolate between Gaussian noise and future-image embeddings and define velocity matching; Equation (12) adds CLIP alignment. The vision encoder is frozen for decoder training. The following prose describes deterministic reconstruction and reduced inference diversity without specifying noise handling.
Go to primary source ↓joint-trainingPDF p. 9, Section 3.4, Equations (13)–(15); PDF p. 10, Section 4.3
Planning precedes visual generation. AR combines cross-entropy and planning losses; AR+Diffusion adds flow matching and CLIP alignment. Stage 1 jointly trains planning/generation with LoRA; Stage 2 mixes VQA and driving data. Continuous latent queries follow planning features.
Go to primary source ↓configurationPDF p. 10, Section 4.1
The setup lists eight NVIDIA H200 GPUs, EVA-02-L, Vicuna 1.5, LoRA rank/alpha 16/16, scene/perception/history queries 512/600/16, a 16-frame memory, 640×640 inputs, 192×128 AR and 512×1024 AR+Diffusion outputs, and 64 continuous latent queries.
Go to primary source ↓protocolPDF p. 10, Section 4.2; PDF p. 11, Section 4.4 and Table 1 caption; PDF p. 21, Appendix A
Bench2Drive uses 950 training and 50 open-loop validation scenes, plus 220 closed-loop routes across 44 scenarios. nuScenes uses its official train/validation split. Table 1 L2 averages a 2-second horizon at 2 Hz. The paper distinguishes UniAD per-timestep metrics from VAD/ORION cumulative averages. Appendix A defines driving, detection, image and VQA metrics.
Go to primary source ↓closed-loopPDF p. 11, Table 1, ORION, Ours, DriveTransformer-Large and MomAD rows
ORION DS/SR/L2 are 77.74/54.62%/0.68; AR gives 79.22/56.36%/0.64; AR+Diffusion gives 79.31/56.42%/0.63. DriveTransformer-Large has L2 0.62. MomAD efficiency/comfortness are 170.21/48.63, above both proposed branches.
Go to primary source ↓planning-perceptionPDF p. 11, Table 2, VAD, Orion and Ours rows
At 3 seconds ORION L2/collision are 1.129 m/0.743%; AR gives 1.079 m/0.668%; AR+Diffusion gives 1.066 m/0.657%. ORION mAP/NDS are 0.646/0.723, versus 0.675/0.755 for AR+Diffusion. VAD collision is 0.296% at 3 seconds.
Go to primary source ↓nuscenesPDF p. 12, Table 3, ORION, FSDrive and Ours columns
Average L2/collision are 0.34 m/0.37% for ORION, 0.30 m/0.31% for AR and 0.29 m/0.31% for AR+Diffusion. FSDrive reports 0.60 m/0.19%.
Go to primary source ↓image-fidelityPDF p. 12, Table 4; PDF p. 21, Appendix A, FID definition
Bench2Drive/nuScenes FID is 7.2/7.8 for AR and 6.6/7.3 for AR+Diffusion, versus 9.3/10.1 for FSDrive. FID compares generated and ground-truth future-frame distributions.
Go to primary source ↓speed-tradeoffPDF pp. 12–14, Section 4.5, especially PDF p. 13, Speed and Resolution and Compression Trade-offs
Open-loop Bench2Drive inference on an A100 is 2 fps for AR and 0.4 fps for AR+Diffusion. The discussion links discrete resolution to token count and describes continuous decoding as the higher-resolution, heavier alternative.
Go to primary source ↓vqaPDF p. 14, Tables 5 and 8; PDF p. 15, Section 4.6
DriveLM GVQA final scores are 0.59 for UniDrive-WM and 0.57 for FSDrive. The separate Chat-B2D ablation changes CIDEr/BLEU/ROUGE-L from 65.7/52.4/77.5 without image generation to 66.7/53.5/78.6 with it.
Go to primary source ↓component-ablationPDF p. 14, Table 6, all four checkbox rows; PDF pp. 14–15, Section 4.6
AR detection+planning without image generation gives 3-second L2/collision 1.130 m/0.746%; all three give 1.079 m/0.668%. Planning+image without detection gives 2.146 m/1.77%. Detection+image without planning has no planning metrics.
Go to primary source ↓order-ablationPDF p. 14, Table 7; PDF p. 15, Section 4.6
Generation+Planning has average L2 0.67 m, collision 0.45% and FID 9.1; Planning+Generation gives 0.64 m, 0.43% and 7.2. The authors interpret this order as aligned with action-to-observation causality.
Go to primary source ↓perception-trainingPDF pp. 22–23, Appendix C.2; PDF p. 9, Section 3.4
Perception pretraining uses Hungarian matching, focal classification and L1 regression, with auxiliary traffic-state and motion supervision. The detection head is frozen during joint planning/generation training; Appendix C distinguishes this from jointly optimizing detection as an end task.
Go to primary source ↓multi-abilityPDF p. 21, Table 9; PDF p. 22, Appendix B.1
Mean ability is 54.72% for ORION, 59.00% for AR and 59.23% for AR+Diffusion. Proposed models obtain 40.00% Give Way, while several target-point baselines attain 50.00%.
Go to primary source ↓qualitativePDF p. 13, Figures 4–5; PDF pp. 25–27, Figures 6–8 and captions
The source shows selected future-frame and VQA examples. Figure 6 first-scene Q3 combines maintaining speed/lane following with an instruction to stop for a red light. These examples do not measure failure rates.
Go to primary source ↓future-workPDF p. 15, Section 5
More interactive and longer-horizon driving scenarios are identified as future work.
Go to primary source ↓8.5 Primary sources
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving ↗
PDF · 9,983 extracted words
Source fingerprint
9557a3a40bf9e405ab76b290848ca4a8996cfdae894da854b51d1ff7bacc0fce