Learning Vision-Language-Action World Models for Autonomous Driving
1. Paper overview
In one sentence: VLA-World conditions future-image tokens on a short-term motion prediction and uses them to refine a driving plan, improving offline nuScenes metrics while leaving the accuracy and causal usefulness of imagined evidence unresolved. e02e03e05e10e12e18
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that direct vision-language-action policies inadequately model scene evolution, while image-generating world models do not necessarily evaluate consequences. Their proposed bridge is action-conditioned imagination followed by reflective planning. Claims of human-like reasoning and causal understanding are motivations and author interpretations, not independently measured capabilities. e02e03 |
| Core mechanism | |
| A key reported result | Trajectory planning on nuScenes — unstarred models: ST-P3: 0.30 m / 0.10%; UniAD: 0.83 m / 0.16%. Average L2 error (m) and collision rate (%), lower is better. Table 1, Qwen2-VL-2B rows without the additional-ego-state asterisk; evaluated split not explicitly labeled. FSDrive: 0.53 m / 0.17% under ST-P3; 0.96 m / 0.40% under UniAD. Both protocol blocks improve. ST-P3 averages preceding timesteps; UniAD evaluates individual timesteps. The two protocols must remain separate. e11e12 |
| Reading caution |
Core contributions
- Source description
nuScenes-GR-20K supplies approximately 20K structured generation-and-reasoning examples for supervised fine-tuning and reinforcement learning. e09
Figure 3. Three training stages teach a single model to generate an intermediate future before completing its plan. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panels (a), (b) and (c) as training stages, then inspect the response in (d). Panel (a) predicts visual tokens from images and a view instruction; decoding makes a future image visible. Panel (b) places perception, prediction, generation, reasoning and planning around the shared VLM. Panel (c) shows a policy update from sampled outputs. In panel (d), Perception and Prediction precede the dashed Condition line, then Visual precedes Think, Action and Answer. This ordering agrees with Equations (3–4): the imagined view becomes context for trajectory refinement. The dashed line is not a safety threshold or an attention-mask specification. e03e04e05e08e18
What it supports. The method's distinctive claim is about where future generation sits in the computation. It occurs before reasoning and the final trajectory, making imagined content available at inference. Appendix A.3 explicitly describes a single autoregressive transformer, supporting One Model with architectural evidence rather than inferring it only from joint training.
Where the evidence stops. Flame icons, including the one in the inference panel, do not establish test-time parameter updates or which encoders are frozen. Appendix A.2's kinematic predictor is not separately drawn here; its exact integration with the generated Prediction block remains unspecified.
2. Motivation
2.1 The problem and the proposed response
The authors argue that direct vision-language-action policies inadequately model scene evolution, while image-generating world models do not necessarily evaluate consequences. Their proposed bridge is action-conditioned imagination followed by reflective planning. Claims of human-like reasoning and causal understanding are motivations and author interpretations, not independently measured capabilities. e02e03
2.2 What this reading follows
A driving model can describe the present yet still miss how a nearby vehicle will move. VLA-World inserts a visual prediction into the decision sequence: estimate the next ego motion, imagine a camera view, reason over that future, then output a maneuver and waypoints. The same autoregressive transformer learns these steps through visual pretraining, supervised examples and reinforcement learning. Read the figures as a claim about information flow, and the tables as tests of offline prediction. The strongest evidence is improved nuScenes planning under matched protocol blocks; the unresolved question is whether the generated scene reliably contributes the right evidence when a decision is difficult. e02e03e05e10e12e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM and One Model are supported: A.3 explicitly identifies a single autoregressive transformer for policy and world modeling, consistent with Figure 3. Visual planning is supported, but no separate inverse-dynamics model is established. The operational sequence is predict → imagine → reason → plan, while Eq. (12) models image and trajectory jointly. Thus Other mechanisms/Outside quadrants is an interpretive placement requiring review, not a consequence of joint training alone. The recorded catalog snapshot is preserved. e03e05e18
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use the first waypoint as a hypothesis about the future
Begin with the requested maneuver and current evidence. Six camera views, ego kinematics and history support structured perception; the model then predicts a near-term waypoint and direction. Appendix A.2 supplies a physical interpretation: finite differences estimate current motion, while an acceleration toward a command-derived target modifies the inertial forecast. This short-term motion conditions visual tokens for the requested camera view. Only afterward does the output move to reasoning and the longer waypoint plan. My interpretation is that the first waypoint narrows the imagination problem to one plausible ego-motion hypothesis. That can make the generated scene useful, but it also creates a dependency: a poor initial hypothesis can contaminate the evidence used to refine it. The source does not fully specify how its kinematic formula is integrated with the textual prediction block. e03e05e08e09
5.2 Separate learning the response format from rewarding its consequences
The three stages teach different aspects of the same output sequence. Visual pretraining learns to emit future-image codebook tokens under camera instructions. SFT then supplies structured examples connecting perception, a short prediction, visual tokens, reasoning, action and waypoints. GRPO samples eight responses to a prompt and favors those with higher group-relative rewards. Its rules check format, short-term consistency, valid visual tokens, action labels and trajectory quality. Appendix A.3 emphasizes that future-image tokens occur inside the optimized sequence, so generation can receive task-related reinforcement. This explains why the image is more than a training-only auxiliary target. It does not make the generated content a verified observation: token validity guarantees decodability, and reward optimization may favor useful outputs without guaranteeing faithful scene dynamics. The reward formulas and weights need clarification before reproduction. e04e06e07e09e10e18
5.3 Ask which evidence would establish useful foresight
Read the empirical argument as three linked but distinct observations. Table 2 reports a lower image FID, Table 1 reports improved planning under matched metric and ego-state settings, and Table 4 reports degradation when generation or reasoning is removed. Together they support the complete pipeline as a useful recipe. They do not prove that the image improvement causes the planning improvement. A removal ablation changes the sequence and training signals, while FID can improve without fixing the motion cue that matters for a maneuver. The appendix adds resolution, backbone and mixed-data sensitivity, but these also change model inputs or training conditions. My proposed next step is to hold the trained planner and prefix fixed and intervene on the future tokens. A plan that ignores relevant interventions would weaken the claim that reflective visual evidence drives its decisions. e11e12e13e15e16
5.4 Training and inference
During training
Visual pretraining uses next-token prediction across requested camera views; SFT teaches the structured multi-task output sequence. Figure 1 labels pretraining 480K, whereas B.1 says approximately 500K; the precise sample count is unresolved. Both later stages use the approximately 20K collection. e04e09
Starting from Qwen2-VL-2B, the stages run for 30, 12 and one epoch, at learning rates 5×10⁻⁴, 10⁻⁴ and 10⁻⁶. Pretraining and SFT use AdamW. GRPO samples eight responses per prompt with global batch size 16 and KL coefficient 0.01. e10
Rule-based rewards cover format, short-term accuracy/consistency, visual-token length and validity, action F1, and trajectory accuracy/kinematics. GRPO normalizes rewards within each response group. These are training-time verifiers, not a demonstrated inference-time safety controller. e06e07
During inference
Figure 3 orders the response as Perception → Prediction → Visual → Think → Action → Answer. The generated future is a conditioning cue for the later plan. No multi-candidate search at inference, repeated environment feedback cycle or physical vehicle execution is demonstrated. e03e05e11
5.5 Implementation flow
- Ground the current scene
Structured perception describes traffic participants, positions, motion cues and distances to road shoulders. The observation history and mission goal provide the context for planning. e02e05e09
- Construct a short-term motion condition
The next 0.5-second waypoint and direction condition imagination. Appendix A.2 describes finite-difference velocity and acceleration, combined with goal-driven acceleration; its relationship to the model-generated prediction block is not fully specified. e05e08
- Generate and reflect
The transformer autoregressively emits discrete VQGAN codebook tokens for the requested future view. Subsequent reasoning identifies notable objects and possible effects before producing the maneuver and final trajectory. Images are decodable for visualization; a separate decode-and-re-encode inference pass is not established. e03e04e05e09
6. Experiments & results
VLA-World turns a predicted near-term ego motion into a generated camera view, then uses that imagined future as context for reasoning and a three-second waypoint plan. A Qwen2-VL-2B autoregressive model learns through visual pretraining, structured imitation and GRPO. Its nuScenes results improve planning and future-image FID, but establish offline prediction performance rather than closed-loop driving safety. The key research question is whether the generated future provides accurate decision-relevant evidence, rather than merely useful intermediate tokens.
6.1 Read the original evidence
Table 1. Planning improves in matched comparisons, but the two metric protocols and ego-state variants must remain distinct. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the caption: the asterisk means additional ego-state information. Compare unstarred VLA-World with unstarred FSDrive, or the two starred rows, before comparing across model families. The left block contains ST-P3 metrics and the right contains UniAD metrics; each has L2 error in meters and collision rate in percent. Appendix B.2 explains that ST-P3 uses preceding-timestep averages whereas UniAD evaluates individual timesteps. Read each Avg. within its own block. The final LLM column helps identify the shared Qwen2-VL-2B backbone for the closest baseline. The retained caption also discloses the secondary provenance of VAD and UniAD results. e11e12
What it supports. For the unstarred comparison, ST-P3 average L2 falls from FSDrive's 0.53 m to 0.30 m, and collision from 0.17% to 0.10%. Under UniAD, the corresponding averages improve from 0.96 m and 0.40% to 0.83 m and 0.16%. These are reported offline planning results.
Where the evidence stops. Average gains do not cover every entry: starred UniAD three-second L2 is 0.80 m versus FSDrive* 0.77 m. No uncertainty intervals are given, and neither collision percentages nor the table's end-to-end label demonstrate physical or closed-loop driving safety.
Table 2. The nearest same-resolution baseline is FSDrive; the broader table includes different image resolutions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Type and Resolution rows before the final FID row. The table compares GAN, diffusion and autoregressive generators, but their output sizes differ substantially. VLA-World and FSDrive share 128×192 output and both are autoregressive, making their adjacent columns especially useful. The downward arrow means lower Fréchet Inception Distance is preferred. GEM's 576×1024 output should remain visible when discussing its nearby FID value; otherwise a comparison would hide an important protocol difference. Finally, keep the purpose of this table separate from Table 1: here the evaluated output is an image, while the driving trajectory is evaluated elsewhere. e11e13
What it supports. VLA-World reports FID 9.8 versus FSDrive's 10.1 at the same resolution. That supports an improvement in the reported image-generation metric, despite generation serving as an intermediate planning step. The table does not tell us whether the small FID difference explains the larger planning differences.
Where the evidence stops. The source does not give uncertainty intervals or the exact FID sample-selection procedure. A favorable FID cannot certify correct object motion, calibrated future risk or a safer executed trajectory; those require additional evidence.
Figure 6. Selected future images illustrate the object-appearance differences discussed in the appendix. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare vertically within each of the five columns. The top row is labeled VLA-World and the bottom FSDrive; both represent a future frame 0.5 seconds ahead. The red dashed ellipses are the paper's original highlights, directing attention to vehicle shape and local scene structure. They are not error contours, predicted uncertainty or measured collision boundaries. Inspect the highlighted regions together with the surrounding roadway: the figure is intended to show the visual character of each prediction. There is no recorded-future reference row here, so this view alone cannot reveal which generated motion matches what subsequently happened. e13e17
What it supports. The authors use these examples to argue that trajectory-conditioned generation better preserves object structure. Several highlighted vehicles make the appearance comparison easy to inspect. The examples complement the aggregate FID table by showing selected local artifacts, but neither their selection nor their visual appeal quantifies driving-relevant prediction accuracy.
Where the evidence stops. These are selected examples, with no failure frequency, scene-level uncertainty or paired ground-truth future displayed. The appendix's causal attribution to trajectory conditioning should be treated as an author interpretation; visual inspection alone cannot establish that mechanism.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Trajectory planning on nuScenes — unstarred models Table 1, Qwen2-VL-2B rows without the additional-ego-state asterisk; evaluated split not explicitly labeled. | ST-P3: 0.30 m / 0.10%; UniAD: 0.83 m / 0.16%. Average L2 error (m) and collision rate (%), lower is better | FSDrive: 0.53 m / 0.17% under ST-P3; 0.96 m / 0.40% under UniAD. Both protocol blocks improve. ST-P3 averages preceding timesteps; UniAD evaluates individual timesteps. The two protocols must remain separate. e11e12 |
| Trajectory planning on nuScenes — additional ego state Table 1 starred VLA-World* and FSDrive* rows. | ST-P3: 0.26 / 0.08; UniAD: 0.42 / 0.12. Average L2 (m) / collision (%) | FSDrive*: 0.28 / 0.10 and 0.45 / 0.16, respectively. Average improvements do not imply every horizon improves: UniAD 3-second L2 is 0.80 m versus FSDrive* 0.77 m. e12 |
| Future-frame generation on nuScenes Table 2; 128×192 output for VLA-World and FSDrive; FID sample-selection details unspecified. | 9.8 FID, lower is better | FSDrive 10.1 at the same resolution; GEM 10.5 at 576×1024. A modest same-resolution improvement; comparisons across all listed generators also differ in resolution. FID does not measure collision avoidance. e13 |
| High-level action prediction on nuScenes Table 3; comparison with the nuScenes-trained Qwen2-VL-2B† baseline. | Left 74.22; right 75.06; stop 81.24. Per-class F1 (%) | Baseline: 61.78, 66.52 and 74.85, respectively. Label prediction improves; no executed-control success rate is measured here. e14 |
| Pipeline and reward ablations Table 4, nuScenes ST-P3 evaluation. | Full 0.30; without pretraining 0.57, SFT 0.85, RL 0.71; without generation 0.68 or reasoning 0.85. Average L2 error (m) | Removing trajectory reward gives 0.72; removing action reward gives 0.62. Each deletion hurts this setup, but changes to supervision, token sequences and optimization limit causal isolation. e15 |
| Appendix sensitivity experiments Tables 5–7, nuScenes ST-P3. | Resolution labels 36,000/52,884: 0.38/0.30; 2B/3B/7B backbones: 0.30/0.29/0.18; without mixed data: 0.49. Average L2 error (m) | The standard 2B mixed-data model is 0.30. The 3B backbone also changes to Qwen2.5-VL; this is not a controlled parameter-only scaling experiment. Table 5 does not provide image height/width. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Removing generation, reasoning or any reported training stage worsens ST-P3 planning in this setup. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use block (d), the complete model, as the common reference. Block (a) removes pretraining, SFT or RL; block (b) removes perception, generation or reasoning. Block (c) removes prediction, visual-validity, action or trajectory rewards, corresponding to the terms described in Section 3.5. The three horizon columns and Avg. all report L2 error in meters under ST-P3. For the central mechanism, compare without Generation and without Reasoning against the final row. For optimization, compare without SFT and without RL: the former is worse here, consistent with the authors' argument that structured supervision provides a useful starting policy before reinforcement learning. e06e15
What it supports. Average L2 increases from 0.30 m to 0.68 m without generation and 0.85 m without reasoning. Removing SFT gives 0.85 m, whereas removing RL gives 0.71 m. The source therefore supports the usefulness of both the intermediate sequence and the staged training recipe within the evaluated configuration.
Where the evidence stops. These deletions alter supervision or computation as well as information content. They do not isolate whether accurate imagined motion is essential. The authors' explanation involving visual-token-dominated gradients is a hypothesis in the discussion, not a directly measured diagnostic.
7. Analysis & limitations
7.1 What the evidence leaves open
The experiments report nuScenes prediction metrics, with no closed-loop rollout, on-road trial, latency distribution or uncertainty intervals. FID and selected qualitative examples cannot establish calibrated dynamics or real-world safety. e11e12e13e17
No dedicated limitations section is present. Appendix A.3's ELBO argument does not establish that this learned generator actually tightens the bound; joint modeling and greater expressivity alone do not prove superior driving decisions. e18e15
The source does not specify a separate geometric multi-view consistency loss, despite claiming coherent viewpoints. Reward descriptions omit executable definitions; Eq. (8) leaves the clipping term abbreviated, and its KL notation uses the old policy while the prose describes the SFT reference. e04e06e07
7.2 Questions for discussion
- Would planning remain accurate if generated future tokens were replaced by matched but incorrect futures?
- How much of the apparent safety gain survives matched ego-state inputs and closed-loop evaluation?
8. Reproducibility audit
8.1 Requirements and known gaps
B.1 lists 28,130 training, 6,019 validation and 193,082 unlabeled nuScenes samples. Reproduction needs the exact 20K subset, annotation construction, view/time pairing and split mapping; these are not fully documented. e09e11
Reported training uses eight A100 GPUs (80 GB in §4.1), inference four A100s, PyTorch, LLaMA Factory and Easy-R1. B.2 specifies maximum pixels 524,288, accumulation 2, cosine scheduling and warm-up 0.1. Software versions, inference latency, freezing policy and the precise tokenizer checkpoint are unstated. e10e04
Before replication, resolve reward weights and verifiers, prediction-prior integration, the adaptive λ rule, decoding settings and the distinction between the maximum-pixel cap and Table 5 resolution labels. Proposed controlled token substitutions and matched training ablations are detailed in the illustrated edition. e06e07e08e10e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Intervene on the future while holding the planning prefix fixed
Reader-proposed check, not executed: on fixed held-out nuScenes scenes, keep the checkpoint, camera/ego inputs, mission command and Perception/Prediction prefix identical. Continue planning with four visual-token conditions: the model's generated future, the current image encoded with the same tokenizer, a same-view future shuffled from a speed/turn-matched scene, and the recorded next frame as a diagnostic oracle unavailable at deployment. Keep token lengths, output budget and decoding settings matched. Measure L2 and collision under each protocol separately, along with action changes in hazard-relevant scenes. If shuffled futures perform indistinguishably from genuine generated futures, the claim of decision-relevant visual reflection is weakened. Oracle improvement combined with generated-future failures would instead implicate imagination accuracy. Verify split separation before sampling. e03e04e09e11e12e15
Check 2: Test whether visual reasoning beats extra sequence and supervision
Reader-proposed check, not executed: start from the same pretrained checkpoint and identical 20K scene split. Compare the complete SFT-plus-GRPO sequence against a variant replacing each future image with a fixed valid codebook-token sequence of the same length while preserving the other target blocks. Retain the same training steps, token budget, decoding budget and nonvisual reward settings; match the validity reward rather than rewarding image realism in only one arm. Resolve and publish the missing reward weights first. Repeat across seeds, stratify by starred versus unstarred ego-state inputs, and report ST-P3/UniAD metrics separately with uncertainty. A consistent advantage for scene-specific futures would support an information benefit beyond extra tokens; similar performance would challenge that interpretation of Table 4. e04e06e07e09e10e11e12e15
8.3 Reading coverage
Visual audit: Visually inspected the title, authors, affiliations and v1 identifier on p. 1; Figures 1–7; Tables 1–7; and all method, equation, training, protocol and appendix pages supporting this edition and its proposed checks. Figure 3's conditioning boundary and output order were checked against Eqs. (3–4) and §§3.3–3.4; it supplies no hard safety gate, explicit attention mask or inference-update rule. Figure 6's row labels agree with B.4. All five final crops were individually viewed and are readable; Table 1 retains its extra-ego-state/provenance caption because it is needed to interpret comparisons. PDF pages 9–12 were read in the supplied text as references and acknowledgements, but were not visually inspected; no technical claim relies on their images. The separate video demo, external code and other editions were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and arXiv version block (PDF p. 1)
- Abstract; 1. Introduction; 2. Related Work (PDF pp. 1–3)
- 3.1–3.5: preliminaries, architecture, visual pretraining, supervised fine-tuning and reinforcement learning (PDF pp. 3–6)
- 4.1–4.3: setup, main results and ablations; 5. Conclusion (PDF pp. 6–8)
- Acknowledgements and References (PDF pp. 9–12)
- A.1–A.3: GRPO, short-term predictor and theoretical analysis (PDF pp. 13–15)
- B.1–B.4: datasets, implementation, additional experiments and visualizations (PDF pp. 15–18)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Verified artifact: arXiv:2604.09059v1 [cs.CV], 10 April 2026. Its exact title and all six authors match the supplied catalog. No different revision or edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all seven figures, all seven tables and the supporting method pages.
- Separate supplemental material availability has not been fully verified.
- The supplementary material embedded in PDF pages 13–18 was fully read. The additional video demo mentioned in B.4 was not supplied or inspected.
- Code, project websites and external references were not inspected; no training, inference or reproduction experiments were executed.
- All seven supplied text chunks were read in full. References were read as bibliography entries, not as independently verified research sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Observed title: Learning Vision-Language-Action World Models for Autonomous Driving. Authors: Guoqing Wang, Pin Tang, Xiangxuan Ren, Guodongfang Zhao, Bailan Feng and Chao Ma. arXiv:2604.09059v1 [cs.CV], 10 Apr 2026. Affiliations are the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University; and Central Research Institute, Huawei.
Go to primary source ↓e02PDF pp. 2–3, §1, Figure 2 and §3.1, Eq. (1)
Motivates combining temporal prediction and reflection; defines multi-view observations, ego status, goal and ego-centric waypoint trajectory. The introductory pipeline imagines a view after a 0.5-second prediction.
Go to primary source ↓e03PDF p. 4, Figure 3 and §3.2, Eqs. (2–4)
Training panels show visual activation, structured SFT and policy updates. Inference emits perception/prediction before visual tokens, followed by thinking, action and answer. Equations (3–4) specify conditioned imagination then reflective trajectory refinement.
Go to primary source ↓e04PDF p. 5, §3.3, Eq. (5)
Future tokens are predicted autoregressively from current multi-view/ego-state encoding and the requested view instruction. Tokens use a VQGAN codebook and can be decoded to images. No separate geometric consistency objective or precise tokenizer checkpoint is specified.
Go to primary source ↓e05PDF pp. 5–6, §3.4, Perception through Action and Trajectory Planning
Defines structured scene perception, predicted short-term waypoint/direction, conditioned future visual tokens, reflection and high-level action followed by three-second waypoints at 0.5-second intervals.
Go to primary source ↓e06PDF p. 6, §3.5, reward descriptions and Eq. (6)
Five reward components cover structured formatting, prediction accuracy/consistency, visual token validity/count, action F1 and trajectory accuracy/kinematics. Weights are symbolic; complete reward formulas and values are absent.
Go to primary source ↓e07PDF p. 13, §A.1, Eqs. (7–8)
GRPO uses group-normalized reward advantages and rule-based verifiers. Eq. (8) contains an unexpanded clip term and KL to the old policy; nearby prose describes the reference as the SFT checkpoint. Collision/generation-quality verifiers are mentioned without complete definitions.
Go to primary source ↓e08PDF pp. 13–14, §A.2, Eqs. (9–11)
Finite differences estimate velocity and historical acceleration; a goal acceleration is derived from desired displacement. An adaptive λ blends accelerations for kinematic prediction, but the λ rule and exact command-to-target mapping are not supplied.
Go to primary source ↓e09PDF p. 1, Figure 1; PDF p. 15, §B.1 and Figure 5
Figure 1 labels pretraining 480K; B.1 calls it approximately 500K. SFT/RL examples total approximately 20K. Listed nuScenes counts are 28,130 training, 6,019 validation and 193,082 unlabeled samples. Figure 5 shows tagged examples, kinematics, history, goal and camera instructions.
Go to primary source ↓e10PDF p. 6, §4.1 Implementation Details; PDF pp. 15–16, §B.2
Qwen2-VL-2B initialization; eight 80-GB training GPUs, identified as A100s in B.2, and four A100s for inference. Names PyTorch, LLaMA Factory and Easy-R1. Gives 30/12/1 epochs, learning rates, GRPO group size 8, global batch 16, KL 0.01, maximum pixels 524,288, accumulation 2, cosine scheduler and warm-up 0.1.
Go to primary source ↓e11PDF p. 6, §4.1; PDF pp. 15–16, §§B.1–B.2, evaluation protocol paragraph
Experiments evaluate nuScenes planning with L2/collision and generation with FID. UniAD uses individual timesteps; ST-P3 averages preceding timesteps. The source lists dataset splits but does not explicitly name the evaluated split in the main results tables or describe closed-loop deployment.
Go to primary source ↓e12PDF p. 7, Table 1, caption and VLA-World/FSDrive starred and unstarred rows
Asterisk denotes additional ego state. Unstarred ST-P3 average L2/collision: VLA-World 0.30/0.10, FSDrive 0.53/0.17; UniAD 0.83/0.16 versus 0.96/0.40. Starred averages: 0.26/0.08 versus 0.28/0.10 and 0.42/0.12 versus 0.45/0.16. Starred UniAD 3s L2: 0.80 versus 0.77.
Go to primary source ↓e13PDF p. 7, Table 2, Resolution and FID rows
VLA-World FID 9.8 and FSDrive 10.1 at 128×192; GEM 10.5 at 576×1024. Listed generators use different resolutions. No uncertainty intervals or exact FID sample-selection specification accompany this table.
Go to primary source ↓e14PDF p. 7, Table 3, VLA-World and Qwen2-VL-2B† rows
The dagger marks nuScenes training. Left/right/stop F1 are 74.22/75.06/81.24 for VLA-World and 61.78/66.52/74.85 for the trained Qwen baseline; the table measures action classification.
Go to primary source ↓e15PDF p. 8, Table 4, blocks (a)–(d), and §4.3
Full average ST-P3 L2 is 0.30 m. Removing pretraining/SFT/RL yields 0.57/0.85/0.71; perception/generation/reasoning gives 0.75/0.68/0.85; prediction/visual/action/trajectory rewards gives 0.41/0.42/0.62/0.72. Discussion attributes weaker generation impact to token-dominated gradients, without measuring that explanation directly.
Go to primary source ↓e16PDF p. 16, Tables 5–7 and §B.3; PDF p. 17, mixed-data discussion
Table 5 resolution labels 36,000 and 52,884 yield average L2 0.38 and 0.30, and 3s L2 0.98 and 0.52. Table 6 uses Qwen2-VL-2B, Qwen2.5-VL-3B and Qwen2-VL-7B with averages 0.30/0.29/0.18. Table 7 removes mixed data, increasing average L2 from 0.30 to 0.49.
Go to primary source ↓e17PDF p. 8, Figure 4/caption; PDF p. 16, Figure 6/caption; PDF pp. 17–18, Figure 7 and §B.4
Figure 4 defines red predicted and green ground-truth trajectories. Figure 6 places VLA-World above FSDrive across five selected future-image examples with red highlighted regions; B.4 attributes improved appearance to trajectory conditioning. Figure 7 shows selected three-second trajectory comparisons. A separate video is mentioned.
Go to primary source ↓e18PDF pp. 14–15, §A.3, Eqs. (12–19) and special cases
Explicitly describes a single autoregressive transformer for policy and world model, with full tokens including trajectory, image and reasoning optimized through reward. Presents joint factorization, an ELBO argument and special-case expressivity claims; it does not empirically measure bound tightness.
Go to primary source ↓8.5 Primary sources
Learning Vision-Language-Action World Models for Autonomous Driving ↗
PDF · 11,580 extracted words
Source fingerprint
4719e8f3bfc7c4314c06eeee71b0b1aa81341e3c3de8a61b4a07d762f163552d