Vega: Learning to Drive with Natural Language Instructions
1. Paper overview
In one sentence: Vega couples instruction-conditioned trajectory denoising to future-image prediction, gaining dense training supervision while leaving general instruction-following reliability unmeasured. e02e03e05e06e10e11e12e15
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis An averaged imitation policy or a small navigation-command vocabulary cannot express all user-requested driving behaviors. The authors argue that sparse trajectory supervision makes mapping rich visual-language inputs to actions difficult. Vega adds future visual prediction as dense supervision; the claim that this teaches causal dynamics is stronger than the behavioral evidence. e02e12e15 |
| Core mechanism | Source description InstructScene pairs NAVSIM scenes with automatically generated driving instructions: 85,109 training scenes and 12,144 test scenes. Annotation combines a VLM with rule-derived ego-motion cues. e03 |
| A key reported result | NAVSIM v2 trajectory planning: Vega: 86.9; Vega†, best-of-six: 89.4. Extended Predictive Driver Model Score (EPDMS), higher is better. Reported NAVSIM v2 benchmark, described as using reactive traffic. Separate v2 split counts are not supplied. Ordinary and best-of-six variants are reported separately. Table 1 reports DriveVLA-W0 at 86.1 and DiffusionDrive at 84.5. Vega† improves over ordinary Vega by 2.5 score points. Best aggregate among listed rows, with extra candidate selection for the dagger variant. Components have different leaders: ordinary Vega DAC is 95.3 versus DriveVLA-W0’s 99.1. No uncertainty is reported. e09e10 |
| Reading caution | Reader analysis Instruction following and image consistency use selected examples without dedicated quantitative adherence or image-quality evaluation. NAVSIM scores do not isolate compliance with novel requests, and annotations describe recorded futures. These results do not establish causal dynamics identification or real-world deployment safety. e03e09e15 |
Core contributions
- Source description
InstructScene pairs NAVSIM scenes with automatically generated driving instructions: 85,109 training scenes and 12,144 test scenes. Annotation combines a VLM with rule-derived ego-motion cues. e03
Figure 3. Separate modality transformers communicate through an ordered multimodal sequence. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: hatching distinguishes noisy prediction targets from clean conditioning tokens. Move along the bottom time arrow from the observed image and instruction to action and future-image tokens. The repeated action label is important: Section 3.3 explains that training uses a noisy copy for supervision and a clean copy for later conditioning. The Und., Act. and Gen. blocks have separate parameters, but their tokens meet in global causal attention. The upward arrows lead to planning and generation outputs. The shared attention bar does not mean unrestricted bidirectional attention: later blocks can use preceding clean context, and subsequent tokens cannot attend to noisy target copies. e04e05e06e07e08
What it supports. The design integrates understanding, action prediction and image generation within layerwise attention. It therefore supports a unified-model interpretation despite separate transformer parameters. Joint training supplies both denoising losses; when both outputs are requested at inference, the model completes the action before generating the image conditioned on it.
Where the evidence stops. The schematic labels the future image I_(t+1); Section 3.4 uses the horizon endpoint I_(t+N). It also shows a VAE snowflake without a complete freezing-policy explanation. Use the prose for the mask and horizon rather than inferring all details from the iconography.
2. Motivation
2.1 The problem and the proposed response
An averaged imitation policy or a small navigation-command vocabulary cannot express all user-requested driving behaviors. The authors argue that sparse trajectory supervision makes mapping rich visual-language inputs to actions difficult. Vega adds future visual prediction as dense supervision; the claim that this teaches causal dynamics is stronger than the behavioral evidence. e02e12e15
2.2 What this reading follows
A driving request can change the desired speed or route even when the camera view stays the same. Vega approaches this problem by annotating recorded driving with instructions and teaching one integrated system to predict both actions and their visual consequences. The crucial connection is architectural: modality-specific transformers exchange information through causal attention, with actions preceding future images. Read the architecture alongside the ablations, then compare the two NAVSIM protocols separately. The tables support a planning benefit from future prediction, while the illustrated alternatives reveal what the model can express. They do not establish how reliably arbitrary instructions will be followed. e02e03e05e06e10e11e12e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
Every recorded category is unassigned, leaving no positive catalog classification to confirm. Reader assessment: layerwise global attention supports an integrated One Model interpretation despite modality-specific parameters. The mechanism is jointly trained action prediction and action-conditioned future prediction, ordered action-to-image at inference; it is neither inverse dynamics nor demonstrated future-rollout search for control. e05e06e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a recorded future into an instruction label
Begin with how the supervision is obtained. InstructScene does not ask drivers to execute a new request during data collection. Its annotation model sees the existing scene and future behavior, describes both, and formulates an instruction that would elicit that behavior. Motion rules help compensate for the VLM’s difficulty estimating ego motion. The deployed planner receives history and the instruction, while the annotation pipeline’s future frames are withheld. Reader interpretation: this makes the target instruction closely aligned with a demonstrated trajectory, which is useful for training but leaves a transfer question. A system may learn to match language to recorded behaviors without reliably handling valid alternatives that never occurred in that scene. Figure 6 illustrates such alternatives, but does not measure their frequency of success. e03e15
5.2 Use visual denoising without reversing the causal order
Imagine training the action and image objectives in one sequence. If the future-image target could attend only to a noisy action target, its conditioning would differ from inference, where the action has already been denoised. Vega addresses that mismatch with two copies: the noisy action participates in its own prediction loss, while a clean copy conditions later image prediction. The noisy copy is hidden from subsequent tokens. Modality-specific transformer parameters allow understanding, action and image generation to specialize, while global causal attention connects their representations. The image loss can therefore contribute to learning the integrated model without requiring a future image to be generated before each planned action. At inference, joint examples proceed action first, image second; the paper primarily evaluates the action-planning capability. e05e06e07e15
5.3 Separate supervision evidence from deployment evidence
The most direct support for future prediction is the Action Only versus Next Frame comparison, not the attractive generated scenes. Its large planning-score difference suggests that the auxiliary visual task matters within the reported setup. Yet the random-frame variant is almost as strong, so the evidence does not identify a uniquely necessary future horizon. The action-expert comparison asks another question: it finds little score difference from the VLM module, despite a large difference from reusing the image diffusion module. Reader interpretation: these results motivate studying supervision and specialization separately. Before extrapolating either result to deployment, reconcile the ablation configuration with the higher headline scores and separate ordinary generation from best-of-six selection. Neither these scores nor Figure 4’s training curves quantify novel-instruction reliability. e10e11e12e13e14e15
5.4 Training and inference
During training
Single-stage joint training predicts an action sequence and its endpoint image using noise-prediction MSE losses. Text, ViT, clean VAE and clean action tokens are randomly dropped by modality for classifier-free guidance. Figure 3 marks the VAE with a snowflake; the text does not enumerate a complete freezing policy. e05e07
Training uses 200,000 steps on eight H20 GPUs, per-device batch size one, learning rate 2 × 10⁻⁵, 2,500 warmup steps, equal action/image loss weights, and EMA decay 0.9999 for saved checkpoints. Four images condition eight actions and an endpoint image. e08
During inference
Actions are denoised with image and text classifier-free guidance. For joint visualization, a future image is then denoised conditioned on the completed action plan. Inference primarily uses action planning; the source does not describe selecting trajectories by scoring generated futures or an inverse-dynamics controller. e06e07e15
Dagger-marked benchmark variants use best-of-N with N = 6. Trajectory prediction and simulated evaluation do not establish physical vehicle execution. e09e10e11
5.5 Implementation flow
- Construct instructions from demonstrations
Qwen2.5-VL-72B-Instruct receives 14 front-view frames at 2 Hz and 1920 × 1080 resolution. Four are history/current observations; ten are future annotation inputs unavailable to the deployed agent. Two stages describe the scene and subsequent behavior, then formulate an instruction. Speed, acceleration and turn-rate rules supplement ego-motion perception. e03
- Encode observations, language and motion
Qwen2.5 tokenizes instructions. Front-view images provide VAE latents plus SigLIP2 ViT features. Absolute planar trajectory poses become consecutive relative movements, A = (Δx, Δy, Δθ), normalized and linearly projected into action tokens. Normalization constants are not supplied. e04
- Connect specialized transformers
Understanding and image-generation transformers have hidden size 3584 and 28 layers, initialized from Bagel-7B; the action module uses width 256. Attention and feed-forward parameters are modality-specific. Segments pass through their respective modules and are reassembled for global attention, providing architectural integration beyond a shared loss. e05
- Preserve the conditioning order
Blocks attend causally across the sequence; image/action tokens use full within-block attention, whereas text uses causal within-block attention. Targets that also condition later predictions receive separate noisy and clean copies. Later tokens are masked from the noisy copy and use the clean condition. e06
6. Experiments & results
Vega learns instruction-conditioned driving by training trajectory denoising together with future-image denoising. Modality-specific transformers exchange information through global causal attention. InstructScene supplies automatically generated instructions describing recorded driving. The strongest reported NAVSIM v2 score uses best-of-six trajectory selection; NAVSIM v1 results are weaker than leading VLA baselines. Future-prediction ablations support visual supervision, while selected images illustrate instruction sensitivity without measuring general instruction-following reliability.
6.1 Read the original evidence
Table 1. The highest listed v2 aggregate belongs to the best-of-six variant. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rightmost EPDMS column first, then move left to see which components contribute to each method’s profile. The two shaded Vega rows represent different selection settings: the dagger means best-of-six, as preserved in the table note. Ordinary Vega reports 86.9 EPDMS and the dagger row 89.4; DriveVLA-W0 reports 86.1. Now inspect DAC, drivable-area compliance: ordinary Vega has 95.3, while DriveVLA-W0 has 99.1. This prevents treating a leading aggregate as dominance on every driving property. The caption supplies the metric abbreviations; Section 4.1 describes v2 as adding reactive traffic and extended metrics. e09e10e11e15
What it supports. Vega† has the best aggregate among the rows shown. Candidate selection improves Vega’s EPDMS by 2.5 score points, while ordinary Vega is 0.8 points above DriveVLA-W0. These are table-level comparisons under the reported configurations, and the component scores show meaningful tradeoffs.
Where the evidence stops. Do not transfer these differences to NAVSIM v1, which uses another score and traffic protocol. The dagger changes candidate selection; no seed uncertainty or fully specified best-of-six selector is supplied. EPDMS is not a direct measure of language compliance.
Table 2. The v1 comparison places Vega below the strongest listed VLA baselines. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use three columns together: method, sensors and PDMS. Vega uses one front camera and scores 87.9 without the dagger or 89.8 with best-of-six. AutoVLA’s corresponding values are 89.1 and 92.1, with three cameras. DriveVLA-W0† uses one camera and scores 93.0, so the single-view input alone does not explain all of Vega’s gap. The BEV and VLA group labels organize approaches, not controlled experiments. Read the dagger and camera abbreviations above the table before comparing rows. Section 4.1 defines this benchmark as non-reactive simulation over four seconds at 10 Hz, with PDMS aggregating its component metrics. e09e11e15
What it supports. The two benchmark tables tell different stories. Vega leads the listed v2 aggregate with best-of-six but remains below several VLA entries on v1. Even among the single-camera, dagger-marked rows, DriveVLA-W0 has a higher PDMS than Vega. The source therefore supports competitive performance rather than universal benchmark leadership.
Where the evidence stops. Sensor count is only one protocol difference; training and reasoning additions also vary. The authors’ suggestion that v1 favors risk-averse policies is an interpretation, not an isolated result. PDMS alone does not determine whether the chosen motion follows a requested instruction.
Figure 6. Changing the instruction changes both the planned path and the generated scene. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each scene as a separate group. The left column provides the current observation, expert action and recorded future at +4 seconds. Arrows from that observation lead to three instruction alternatives; each row then pairs a planned action with a predicted image. In the upper scene the alternatives include accelerating straight, turning right and remaining stationary. In the lower scene they include turning left, turning right and remaining stationary. Compare the path direction with the changed camera viewpoint in the corresponding prediction. Section 4.4 states that these examples generate actions first and images afterward, consistent with the ordering explained in Section 3.3. e06e07e15
What it supports. The examples demonstrate that Vega can express different instruction-conditioned plans and associated visual outcomes for the same starting scene. Their paired arrangement makes the intended action/image consistency visible. The evidence is qualitative: selected predictions vary meaningfully with the instructions, rather than repeating a single displayed expert trajectory.
Where the evidence stops. Only one recorded future is shown per scene; it cannot validate all alternative futures. The caption’s consistency claim is not an aggregate adherence or image-quality measurement. Generated frames are predictions, and these illustrations provide no evidence of physical vehicle execution.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM v2 trajectory planning Reported NAVSIM v2 benchmark, described as using reactive traffic. Separate v2 split counts are not supplied. Ordinary and best-of-six variants are reported separately. | Vega: 86.9; Vega†, best-of-six: 89.4. Extended Predictive Driver Model Score (EPDMS), higher is better | Table 1 reports DriveVLA-W0 at 86.1 and DiffusionDrive at 84.5. Vega† improves over ordinary Vega by 2.5 score points. Best aggregate among listed rows, with extra candidate selection for the dagger variant. Components have different leaders: ordinary Vega DAC is 95.3 versus DriveVLA-W0’s 99.1. No uncertainty is reported. e09e10 |
| NAVSIM v1 trajectory planning NAVSIM test split, non-reactive simulation at 10 Hz for four seconds; Vega uses one front camera. Dagger rows use best-of-six. | Vega: 87.9; Vega†: 89.8. Predictive Driver Model Score (PDMS), higher is better | AutoVLA: 89.1; AutoVLA†: 92.1; single-camera DriveVLA-W0†: 93.0. Vega is competitive but not the leading VLA in this table. Sensor inputs and training recipes differ. PDMS and EPDMS are different metrics. e09e11 |
| Future-image prediction ablation Section 4.4 ablation on NAVSIM v1/v2; eight H20 GPUs and otherwise Section 4.2 hyperparameters. These are ablation scores, not headline scores. | Next Frame: 77.9 / 76.0; Random Frame: 77.3 / 75.2; Action Only: 51.8 / 48.9. PDMS / EPDMS, higher is better | Next Frame exceeds Action Only by 26.1 / 27.1 score points, but Random Frame by only 0.6 / 0.8. Supports visual prediction supervision within this comparison. The source does not reconcile its baseline with higher headline scores or resolve all horizon terminology. e08e10e11e12 |
| Action-module ablation Table 4 compares the image diffusion module, understanding VLM, or dedicated action expert for planning. | Action Expert: 77.9 / 76.0; Use VLM: 77.6 / 75.7; Use Diffusion: 19.7 / 19.6. PDMS / EPDMS, higher is better | The dedicated expert improves over the VLM by 0.3 points on each score. The large gap concerns reusing the image diffusion module. The small VLM gap lacks error bars; a compute benefit is argued but not measured here. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Tables 3 and 4. The visual-supervision gap is large; the action-expert advantage over the VLM is small. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper table vertically within each benchmark column. Next Frame scores 77.9 PDMS and 76.0 EPDMS, compared with 51.8 and 48.9 for Action Only. Random Frame is close to Next Frame at 77.3 and 75.2; its index is included in the instruction prompt. Then read the lower table as a different intervention: which module predicts actions. The dedicated expert reaches 77.9 and 76.0, the VLM 77.6 and 75.7, and the image diffusion module 19.7 and 19.6. Keep these comparisons within their tables; the ordinary headline Vega scores are substantially higher. e07e08e10e11e12e13
What it supports. The upper table supports the usefulness of visual supervision within the stated ablation setup: Next Frame gains 26.1 PDMS and 27.1 EPDMS points over Action Only. The lower table suggests that a narrow action expert preserves the VLM planner’s performance much more closely than repurposing the image diffusion module does.
Where the evidence stops. The paper does not reconcile the ablation baseline with Tables 1–2. It also uses Next Frame terminology alongside endpoint-image training. These gaps prevent assigning the large ablation gain directly to the headline system; no uncertainty establishes the significance of the 0.3-point expert/VLM difference.
Figure 4. Interleaved histories improve the plotted training-loss trajectory after an initial disadvantage. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the horizontal axis from the start of fine-tuning to 80k steps and read the vertical axis as log(loss), not a planning score. The teal original curve begins below the interleaving variants. The pink 2VA, brown 4VA and yellow 6VA curves descend more rapidly and finish below it, with the longer interleaving variants ending lower in the shown run. The caption identifies these as image-action sequence variants. Read the accompanying text on both pages 7 and 8: it describes image-only history as the original design and later discusses interleaving four images with three past actions. e04e06e14
What it supports. The plotted optimization behavior is consistent with a benefit from providing image-action history during training. It is a diagnostic of model fitting, complementing the planning-score ablations. It does not directly show that the lower final loss produces better instruction adherence, better NAVSIM scores or faster wall-clock execution.
Where the evidence stops. The prose ambiguously describes both equal and different sequence lengths, so exact configurations remain unresolved. A source overlay near the final curves is retained. No exact endpoint values are inferred, and lower training loss is not treated as a measured control improvement.
7. Analysis & limitations
7.1 What the evidence leaves open
Instruction following and image consistency use selected examples without dedicated quantitative adherence or image-quality evaluation. NAVSIM scores do not isolate compliance with novel requests, and annotations describe recorded futures. These results do not establish causal dynamics identification or real-world deployment safety. e03e09e15
Section 3.3 places past actions in history, but Section 4.4 calls image-only history the original design. The interleaving discussion also describes both equal and differing sequence lengths. Figure 4 measures training log-loss, not driving performance, so convergence alone cannot resolve the planning effect. e04e06e14
The authors attribute weaker NAVSIM v1 performance partly to metrics favoring risk-averse policies and competitors’ extra sensors or training; these explanations are not isolated experimentally. No dedicated limitations section, multi-seed uncertainty, or measured inference latency is supplied. e11e13e15
7.2 Questions for discussion
- Does visual supervision improve adherence when instructions describe valid alternatives to the recorded future?
- How much of the best-of-six gain depends on selection rather than the learned planning distribution?
- Which history, frame-horizon and checkpoint configurations produced headline versus ablation scores?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the annotated NAVSIM splits, annotation pipeline, Bagel-7B initialization, SigLIP2/VAE encoders, relative-action normalization, and clean/noisy token mask. Exact prompts, motion-rule thresholds, normalization constants, diffusion schedule/step count, guidance scales, complete freezing policy, software versions and best-of-six selection implementation are unspecified here. e03e04e05e06e07e08e10
Proposed checks should fix initialization, data, optimization and candidate selection while removing future prediction, then evaluate alternative instructions on identical observations. Resolve the headline/ablation gap before claiming numerical reproduction. These are proposed experiments, not reproduced results. e10e11e12e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate the benefit of the future-image target
Reader-proposed experiment: reproduce the three Table 3 variants with identical initialization, data order, history format, action expert, action-loss normalization and optimization schedule. Keep training steps fixed and report compute separately; fix a single-candidate inference protocol, guidance and denoising settings. Explicitly define the Next Frame target index and log it. Evaluate PDMS and EPDMS separately across multiple seeds, then repeat with a shared best-of-six selector. A reproducible improvement over Action Only would support the supervision mechanism. If the advantage disappears after matching configurations, or appears only with different selection, the published ablation would not establish the same benefit for the headline system. e06e07e08e10e11e12
Check 2: Test alternative instructions without using the held-out future
Reader-proposed experiment: select held-out scenes and have annotators see only the same history available to the planner. For each scene, write two feasible requests that require distinct motion, plus meaning-preserving paraphrases. Fix model checkpoint, random seeds and candidate-selection settings; compare Vega with the action-only ablation. Define adherence in advance using requested speed changes, stopping or turn direction, and report it alongside collision and drivable-area metrics. Include a shuffled-instruction control. Consistent motion changes for semantic alternatives, stability to paraphrases, and degradation under shuffling would support language-conditioned control. Failure on these tests despite good NAVSIM scores would limit the broad instruction-following claim. e03e09e12e15
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block and Figure 1 on p. 1; motivation and formulation on pp. 2–3; Figure 2 and encoding on p. 4; Figure 3, causal-mask construction, architecture dimensions, losses and inference on p. 5; Table 1, annotation pipeline, hardware/training recipe and benchmark protocols on p. 6; Tables 2–4 and Figure 4 on p. 7; the interleaving/action-expert discussion and Figure 5 on p. 8; Figure 6 and conclusion on p. 9. Every final crop was inspected; the Figure 4 crop was widened to retain the full 80k axis label. Original legends, labels and table notes are retained, including captions where they define metrics or dagger protocols. The architecture arrows were checked against the sequential action-to-image formulation and clean/noisy masking; schematic horizon notation and missing freezing details are disclosed. Reference pages 10–13 were read as text. No appendix or separate supplement was supplied, no external media/code was inspected, and no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1. Introduction (pp. 1–2)
- 2. Related Work, including 2.1–2.3 (pp. 2–3)
- 3. Proposed Approach, including 3.1–3.4 and Equations (1)–(8) (pp. 3–5)
- 4. Experiments, including 4.1–4.4 (pp. 6–8)
- 5. Conclusion (pp. 8–9)
- References [1]–[87] (pp. 10–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The retained PDF figures and tables were subsequently inspected on pages 1–9; reference pages 10–13 were read as text. No appendix is present in this 13-page PDF.
- Identity/version: the observed title and all six authors match the catalog. The title page identifies arXiv:2603.25741v2 [cs.CV], 30 March 2026. The catalog records submission on 26 March 2026. This report reads v2; v1 and a revision history were not supplied, so changes between versions were not assessed.
- The code and project links printed in the paper were not opened. No code was inspected and no experiments were reproduced. No separate supplement was supplied.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin stamp
Title: Vega: Learning to Drive with Natural Language Instructions. Authors: Sicheng Zuo, Yuxuan Li, Wenzhao Zheng, Zheng Zhu, Jie Zhou and Jiwen Lu. Affiliations: Tsinghua University and GigaAI. Stamp: arXiv:2603.25741v2 [cs.CV], 30 Mar 2026.
Go to primary source ↓e02PDF pp. 2–3, Introduction and Sections 3.1–3.2
The paper motivates open-ended instructional driving, describes sparse action supervision as a difficulty, and proposes joint image generation and planning to learn instruction/action/outcome relationships.
Go to primary source ↓e03PDF p. 3, Section 3.1 annotation paragraph; p. 6, Section 4.2, Instruction Annotation
The two-stage Qwen2.5-VL-72B-Instruct pipeline uses 14 front-camera frames at 2 Hz and 1920 × 1080, four past/current and ten future, plus motion rules. It annotates 85,109 train and 12,144 test scenes. Future images are unavailable to the agent at inference. Exact prompts and rule thresholds are not listed.
Go to primary source ↓e04PDF p. 4, Section 3.3, Encoding Inputs and Constructing Input Sequence
Qwen2.5 tokens, VAE latents and SigLIP2 features encode inputs. Relative normalized planar movement is projected with a linear head. The described sequence includes historical images/actions and a noisy action target or clean action plus noisy image target.
Go to primary source ↓e05PDF p. 5, Figure 3 and Section 3.3, Integrated Transformer
Figure 3 shows understanding/action/generation modules, noisy and clean action tokens, causal attention and a snowflake beside VAE. MoT duplicates attention and FFN parameters. Understanding/generation use width 3584, depth 28 and Bagel-7B initialization; action width is 256. Segments are reassembled for global attention.
Go to primary source ↓e06PDF pp. 4–5, Section 3.3, sequence construction and paragraphs before Integrated Transformer
Block-causal attention combines with within-block text causality or full image/action attention. Training duplicates noisy targets and clean conditions, masking noisy copies from subsequent tokens. Inference denoises actions before images, conditioning the latter on fully denoised actions.
Go to primary source ↓e07PDF p. 5, Section 3.4, Equations (7)–(8) and training/inference paragraphs
Action noise prediction conditions on image history and instruction; image noise prediction also conditions on the action plan. Their weighted MSE sum is optimized jointly. Modality dropout enables classifier-free guidance; inference primarily performs action planning with image/text guidance.
Go to primary source ↓e08PDF p. 6, Section 4.2, Training
Training uses 200k steps, eight H20 GPUs, four history images, eight predicted actions and an endpoint image, learning rate 2e-5, 2500 warmup steps, per-device batch size one, equal loss weights 1.0 and EMA 0.9999.
Go to primary source ↓e09PDF p. 6, Section 4.1, Datasets and Benchmarks
NAVSIM v1 uses train/test splits and non-reactive 10 Hz simulation for four seconds with PDMS; v2 is described as enabling reactive traffic and using EPDMS with additional compliance and comfort metrics.
Go to primary source ↓e10PDF p. 6, Table 1, EPDMS and DAC columns, Vega/Vega† and baseline rows; dagger note
Vega scores 86.9 EPDMS and Vega† 89.4; dagger means best-of-N, N=6. DriveVLA-W0 scores 86.1 and DiffusionDrive 84.5. Vega DAC is 95.3 against DriveVLA-W0 99.1. Vega† has the highest listed aggregate, but components have different leaders.
Go to primary source ↓e11PDF p. 7, Table 2, Sensors and PDMS columns and dagger note; Section 4.3 continuation
Vega and Vega† use one front camera and score 87.9 and 89.8 PDMS. AutoVLA and AutoVLA† score 89.1 and 92.1 with three cameras; DriveVLA-W0† scores 93.0 with one. The authors discuss metrics, sensors and extra training as potential reasons for weaker v1 performance.
Go to primary source ↓e12PDF p. 7, Table 3 and Section 4.4, Future Frame Prediction
Random Frame scores 77.3/75.2, Action Only 51.8/48.9, and Next Frame 77.9/76.0 PDMS/EPDMS. The random variant selects one of eight future frames and specifies its index in text. All use eight H20 GPUs and otherwise Section 4.2 hyperparameters; the source does not reconcile these scores with Tables 1–2.
Go to primary source ↓e13PDF p. 7, Table 4; p. 8, Section 4.4, Independent Action Module
Use Diffusion, Use VLM and Action Expert score 19.7/19.6, 77.6/75.7 and 77.9/76.0. The authors argue that the narrower expert reduces action computation; the table has no runtime, compute measurements or uncertainty.
Go to primary source ↓e14PDF p. 7, Figure 4 and Interleaving Observation and Action; p. 8, continuation of that paragraph
Figure 4 plots log(loss) against fine-tuning steps for original, 2VA, 4VA and 6VA. Interleaved variants start above original and end below it. Prose identifies image-only history as original, describes four images interleaved with three actions, and gives inconsistent equal/differing sequence-length statements.
Go to primary source ↓e15PDF p. 8, Figure 5 and Section 4.4 Visualizations/VLA Baseline; p. 9, Figure 6 and Conclusion
Selected scenes show instruction-dependent speed/trajectory changes and sequentially generated future images. Figure 6 includes current images, expert actions, recorded +4s futures and three instruction-conditioned alternatives per scene. Qualitative consistency is claimed, without an aggregate instruction-adherence or image-quality metric.
Go to primary source ↓8.5 Primary sources
Vega: Learning to Drive with Natural Language Instructions ↗
PDF · 8,353 extracted words
Source fingerprint
8a7570cc1312442b78a418cb70649c1b7f64cf1315fb6f19c0ba1a1c0f7a2811