OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
1. Paper overview
In one sentence: OpenDriveVLA teaches a shared autoregressive waypoint decoder through structured visual-language alignment and auxiliary agent forecasting, yielding competitive open-loop planning while leaving closed-loop safety untested. perceptionalignmentinstructioninteractionplanningstage-ablationinput-ablationevaluation
| At a glance | What to know |
|---|---|
| Research problem | Source description The paper asks how a language model can produce spatially grounded driving actions when generic 2D VLMs lack explicit object structure and 3D interaction priors. The authors aim to combine semantic instruction following with perception-trained tokens while avoiding expensive reasoning traces during trajectory inference. probleminstructionperception |
| Core mechanism | Source description A single autoregressive action decoder consumes hierarchical scene, tracked-agent and map representations, joining driving-specific perception to a pretrained language model through separate token projectors. perceptionplanning |
| A key reported result | Open-loop trajectory planning under UniAD metrics: 0.66 m; 0.25%. Average L2 displacement (m) and collision rate (%), lower is better.. nuScenes validation; three-second horizon; OpenDriveVLA-7B. UniAD: 1.03 m; 0.31%. Best reported averages in these Table 1 columns; comparison uses the paper's compiled baseline results, without evidence of matched training compute. planning-tabledataevaluation |
| Reading caution | Reader analysis Open-loop nuScenes evaluation cannot establish closed-loop safety, recovery or long-tail deployment robustness. No measured inference latency supports the intended efficiency advantage; tables also omit repeated-run uncertainty. evaluationreporting-limits |
Core contributions
- Source description
A single autoregressive action decoder consumes hierarchical scene, tracked-agent and map representations, joining driving-specific perception to a pretrained language model through separate token projectors. perceptionplanning
- Source description
A staged curriculum separates visual-language alignment, driving instruction tuning, auxiliary agent forecasting and final ego-trajectory learning. The proposed benefit is implicit interaction knowledge in the planner. alignmentinstructioninteractionplanning
Figure 2. Perception creates structured tokens that a shared language-model decoder turns into waypoints. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read from the camera images at the lower left toward the central VLA. The backbone extracts 2D features, and the BEV branch supports spatial perception. TrackQFormer, MapQFormer and SceneSampler produce different environmental token types, which pass through separate projectors. The token strip distinguishes environmental content from the system prompt, ego state and driver command; its colors are explained in the lower-right legend. The inset prompt specifies coordinates in meters, with x pointing right and y forward, and requests six waypoints over three seconds. The two action panels illustrate command-conditioned paths. For the map module's precise input, use the written perception formulation and the caution below. perceptionalignmentplanningdata
What it supports. The architectural contribution is an interface between structured driving perception and a shared autoregressive action decoder. The scene, agent and map tokens carry different kinds of information into the same language-model input. This makes instance-level grounding an explicit design choice, but the figure itself does not measure how accurately those tokens represent the scene.
Where the evidence stops. The MapQFormer arrow branches at the 2D-feature level in this drawing, while the text defines map queries on BEV features. The crop preserves that discrepancy. Follow the explicit textual formulation; the source does not explain the arrow mismatch.
2. Motivation
2.1 The problem and the proposed response
The paper asks how a language model can produce spatially grounded driving actions when generic 2D VLMs lack explicit object structure and 3D interaction priors. The authors aim to combine semantic instruction following with perception-trained tokens while avoiding expensive reasoning traces during trajectory inference. probleminstructionperception
2.2 What this reading follows
A driving VLA must connect language such as “turn right” to the geometry and motion of a particular scene. OpenDriveVLA approaches that connection through its inputs and training sequence: perception-trained scene, agent and map tokens enter a language model, which first learns captions and driving answers, then other-agent motion, and finally ego waypoints. The central question is whether those intermediate tasks improve the path that the model generates. Read the architecture alongside the two ablations: the interaction stage lowers reported collisions without changing rounded displacement error, while removing history or ego state hurts substantially more than removing vision. These are open-loop findings, not evidence of executed driving. perceptionalignmentinstructioninteractionplanningstage-ablationinput-ablationevaluation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded VLA, Autoregressive VLA and Autonomous driving VLA labels fit direct language-model waypoint generation. One Model is supported by a shared VLA decoder producing trajectories from projected perception and text, rather than a language decision-maker feeding a separately specified planner. It still contains perception and projection modules. The world/action prediction quadrant remains Not applicable: agent forecasting is auxiliary training, and the demonstrated planner does not jointly generate future world observations and ego actions or infer actions through inverse dynamics. problemperceptioninteractionplanning
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Give the language model identifiable parts of the driving scene
OpenDriveVLA's first design choice is to let driving perception determine the visual units supplied to language learning. Multi-view features support a pooled global scene representation and BEV perception; tracked agents and map structure receive dedicated query representations. Stage 1 then teaches a separate projector for each token type. An agent query is matched to an individual caption that combines appearance and spatial information, while scene and map tokens align to broader descriptions. This hierarchy matters because the same visual-language bridge must represent an individual moving vehicle and a road layout at different granularities. The language model and visual encoding stay frozen at this point, so the trainable interface must make the existing representations useful to caption generation. The later driving QA stage trains the projectors and LLM together. perceptionalignmentinstructiondata
Figure 3. The auxiliary forecasting stage sits between language instruction tuning and ego planning. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the top row as a change in supervision: matched captions, driving answers, another agent's motion, and finally the ego trajectory. Then inspect the snowflake and flame symbols underneath. In Stage 1 only projectors train; in Stage 2 the projectors and LLM train while visual encoding stays frozen. Stage 2.5 uses the same depicted trainable modules for agent forecasting conditioned on environment and ego information. Stage 3 expands training into 3D perception. Its top panel receives environment and ego state and outputs a trajectory. These panels describe a curriculum, not a required inference sequence in which every intermediate answer must be generated. alignmentinstructioninteractionplanningimplementationevaluation
What it supports. The figure makes the interaction mechanism concrete: forecasting is an auxiliary objective used to shape the decoder before final ego planning. Instruction tuning likewise teaches driving responses before trajectory tuning. The paper evaluates language after Stage 2, so those results should be associated with that checkpoint rather than automatically attributed to the final planner.
Where the evidence stops. The Stage 3 Vision Encoder flame is broader than the written rule: pages 5–6 explicitly keep the 2D backbone frozen while tuning 3D perception. The diagram does not display this internal exception; it should not be read as permission to unfreeze the entire visual stack.
5.2 Use other-agent forecasting to change what the planner learns
Stage 2.5 introduces motion supervision before ego planning. For each detected agent, the LLM predicts future waypoints from previous waypoints, environmental context, ego state and the agent's projected embedding. This is an auxiliary learning problem; the final planning formulation instead conditions ego trajectory generation on environment, ego state and a driving command. Reader interpretation: the intended transfer is a better internal representation of interactions, not a separately simulated future that the planner searches at inference. Table 5 offers a specific test of that idea: adding forecasting reduces collision while leaving the rounded L2 averages unchanged. However, the additional stage also adds optimization. The result supports the curriculum's usefulness under the reported setup, but it does not isolate whether agent-specific motion supervision, extra training, or both produce the improvement. interactionplanningstage-ablationreporting-limits
5.3 Evaluate the action claim separately from the language claim
The paper produces several kinds of evidence from different checkpoints. Stage 2 answers driving questions and generates captions; the final model predicts ego trajectories over three seconds. The language scores therefore establish a capability of the instruction-tuned checkpoint, while the planning table measures path prediction under two open-loop conventions. Reader interpretation: combining these into a single claim of safe reasoning-driven control would require additional evidence. The input ablation is especially informative because removing history or ego state harms the planner much more than removing visual input. A researcher should ask how much of the trajectory score follows from motion continuity, and whether the remaining visual dependence responds correctly to changed hazards or commands. Closed-loop execution and retention of language abilities after planning tuning remain untested in the supplied paper. evaluationdataplanning-tablelanguage-tableinput-ablationreporting-limits
5.4 Training and inference
During training
Stage 1 freezes visual encoding and the LLM, training only projectors on hierarchical captions. Stage 2 freezes visual encoding while tuning projectors and the LLM on perception, prediction, attention, action-reasoning and decision QA. alignmentinstruction
Stage 2.5 maximizes the likelihood of each detected agent's future trajectory given its embedding, scene/map context, ego state and preceding waypoints. Figure 3 keeps visual encoding frozen and trains projectors plus LLM. interaction
Stage 3 jointly optimizes 3D perception, projectors and the LLM for ego trajectories. The 2D backbone remains frozen, an exception obscured by Figure 3's broad trainable Vision Encoder icon. planningimplementation
Training uses the nuScenes training split with TOD3Cap, nuCaption, nuScenesQA, nuX and GPT-Driver-derived material. Object captions add BEV coordinates, scene captions merge camera views, and map descriptions translate ground-truth map annotations. data
During inference
Encode observations, assemble environmental and textual inputs, autoregressively emit trajectory tokens at temperature 0, then decode them into numerical waypoints. The described inference does not require generating a chain-of-thought or an agent forecast first. instructionplanningimplementation
The output is a planned path. The paper supplies neither an evaluated vehicle controller nor a closed-loop feedback experiment; auxiliary forecasting should not be interpreted as inference-time world-model rollout or action-conditioned simulation. interactionplanningevaluation
5.5 Implementation flow
- Build structured environmental tokens
A shared 2D backbone extracts camera features. SceneSampler pools multi-view 2D context; BEV perception produces tracked-agent and map queries. The written method defines both agent and map queries on BEV features. Detection, tracking and segmentation pretraining supplies their grounding. Figure 2's map-input arrow is ambiguous relative to this formulation. perceptionimplementation
- Translate perception into language space
Separate two-layer GeLU MLPs project scene, agent and map tokens. Agent captions supervise individual matched queries; scene and map captions supervise holistic tokens. The resulting environment representation joins ego text and a question or command in the language model. alignmentinstructionimplementation
- Represent motion as text
Agent forecasting and ego planning predict entity-relative displacements. Ego waypoints are 2D coordinate pairs serialized as discrete textual tokens. Both tasks cover three seconds at 0.5-second intervals. Historical trajectory is an explicit ablated input, although its serialization and window are unspecified. planningdatainput-ablationreporting-limits
6. Experiments & results
OpenDriveVLA turns multi-view driving images, textual ego information and commands into six future ego waypoints through a Qwen-based autoregressive VLA. Its distinctive training sequence aligns scene, agent and map tokens to language, teaches driving QA, then forecasts other agents before tuning ego planning. The strongest mechanism evidence is a collision reduction from auxiliary forecasting without a rounded L2 improvement; the evaluation establishes open-loop planning performance, not executed driving safety.
6.1 Read the original evidence
Table 1. Keep the evaluation convention fixed before comparing planning numbers. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with either the ST-P3 block on the left or the UniAD block on the right; they are separate evaluation conventions, despite sharing metric names. Within that block, compare like horizon columns and keep displacement in meters separate from collision in percent. The final three rows vary OpenDriveVLA's language-model size, while the last two columns identify the LLM and input type for each method. Dashes mean a result was not reported. In the UniAD average columns, compare 7B with the UniAD baseline; in ST-P3, check the EMMA row before accepting an unrestricted best-performance claim. Baseline entries are compiled from cited result summaries. planning-tableevaluationdatareporting-limits
What it supports. Under UniAD metrics, 7B reports average L2 of 0.66 m and collision of 0.25%, versus UniAD's 1.03 m and 0.31%. Under ST-P3, 3B and 7B both reach 0.33 m, while EMMA reports 0.32 m. OpenDriveVLA therefore offers strong results without uniformly minimizing every reported measure.
Where the evidence stops. The table reports open-loop prediction, with no seed uncertainty and no matched-compute rerun of all baselines established here. EMMA's missing collision values prevent a full comparison. Do not merge ST-P3 and UniAD averages or treat logged-scene collision estimates as deployment safety.
Table 2. Language gains are measured before the final motion and planning stages. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left block as caption-generation quality: BL-1 through BL-4 are BLEU scores and BERT-S is BERT-Score. The right block evaluates question answering; its caption identifies existence, counting, object, status and comparison questions. Acc is the reported overall accuracy. H0 and H1 are retained in the original table, but the supplied paper does not define their meaning, so no interpretation of those columns is added here. Compare the bottom three rows to see that model scaling affects the two tasks differently. The evaluation section states that these answers are produced after driving instruction tuning, before the subsequent forecasting and planning stages. language-tableinstructionevaluationreporting-limits
What it supports. The 7B model leads the displayed captioning scores, including BLEU-4 of 27.6 versus LiDAR-LLM's 19.3. Overall QA accuracy peaks at 58.5 for 3B, compared with 57.0 for BEVDet+BUTD; 7B reaches 58.2. Increasing language-model size therefore does not monotonically improve the displayed QA aggregate.
Where the evidence stops. Caption and QA metrics do not demonstrate safe action execution. The paper does not establish retention of these scores after Stage 3. H0/H1 definitions and evaluation details beyond the supplied main text remain unresolved; no definitions are inferred from their labels.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Open-loop trajectory planning under UniAD metrics nuScenes validation; three-second horizon; OpenDriveVLA-7B. | 0.66 m; 0.25%. Average L2 displacement (m) and collision rate (%), lower is better. | UniAD: 1.03 m; 0.31%. Best reported averages in these Table 1 columns; comparison uses the paper's compiled baseline results, without evidence of matched training compute. planning-tabledataevaluation |
| Open-loop trajectory planning under ST-P3 metrics nuScenes validation; separate ST-P3 evaluation convention. | 3B/7B: 0.33 m and 0.10%; 0.5B: 0.35 m and 0.09%. Average L2 (m) and collision (%). | GPT-Driver: 0.44 m and 0.17%; EMMA: 0.32 m, collision unreported. Competitive planning, but the broad state-of-the-art claim does not mean lowest displacement in every column. Missing collision values prevent a complete EMMA safety comparison. planning-tableevaluation |
| Driving captioning and question answering nuScenes-derived language evaluation after Stage 2, before subsequent motion/planning tuning. | 7B captioning: 27.6/92.2; 3B QA accuracy: 58.5. nu-Caption BLEU-4/BERT-Score; nuScenes-QA overall accuracy. | LiDAR-LLM captioning: 19.3/91.3; BEVDet+BUTD QA: 57.0. Nu-X CIDEr is 32.3/25.5/26.2 for 0.5B/3B/7B versus Hint-VAD 22.4. Language gains are checkpoint-specific. Scaling is uneven: 0.5B leads Nu-X CIDEr, while its METEOR 12.5 is below Hint-VAD's 13.2. language-tablenux-tableevaluation |
| Auxiliary interaction-stage ablation OpenDriveVLA-0.5B; compare Stages 1+2+3 with 1+2+2.5+3. | UniAD collision: 0.31% → 0.26%; L2: 0.68 → 0.68 m. Average collision (%) and L2 (m), separately by evaluation convention. | ST-P3 collision: 0.11% → 0.09%; L2: 0.35 → 0.35 m. The UniAD change is 0.05 percentage points. It supports a collision-specific benefit at reported precision, but training-budget and seed controls are absent. stage-ablationreporting-limits |
| Input-modality ablation OpenDriveVLA-0.5B; nuScenes open-loop; UniAD averages. | All inputs: 0.26/0.68; no ego: 0.77/1.34; no history: 1.14/1.30. Collision (%) / L2 (m). | No vision: 0.29/0.77; no command: 0.33/0.80. History and ego-state removals hurt more than removing vision in this setup. This motivates checking benchmark dependence on motion context before attributing gains to visual reasoning. input-ablationevaluation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. The benchmark is particularly sensitive to ego state and motion history. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the checkmarks before reading the scores. Visu, Ego, Hist and Cmd mean visual perception, ego state, historical trajectory and command. The first row omits ego state, the second history, the third vision and the fourth command; the last row includes all four. This row mapping is essential because text extraction alone loses the blank cells. For an initial comparison, stay within the UniAD collision and L2 columns. Full inputs give 0.26% and 0.68 m, whereas omitting history gives 1.14% and 1.30 m. Then compare the no-vision row, which gives 0.29% and 0.77 m. input-ablationevaluationreporting-limits
What it supports. In this ablation, removing ego state or history degrades the reported open-loop metrics much more than removing vision. Visual input still helps, but this result makes motion context a major competing explanation for overall benchmark performance. Reader interpretation: claims about visual reasoning should be tested while explicitly controlling those strong nonvisual inputs.
Where the evidence stops. These are input removals within the reported setup, not causal interventions in a live driving environment. The history representation and window are unspecified in the supplied text. The table also gives no repeated-run variability, so small score differences should not be assigned statistical significance.
Table 5. The clearest auxiliary-forecasting gain appears in collision, not rounded displacement error. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the training-stage columns, not the score columns. Every row contains Stage 3, so the first row is already a trajectory-trained model. Moving downward adds alignment, then driving instruction tuning, then Stage 2.5 agent forecasting before the final planning stage. Compare the last two rows to focus on the added forecasting stage: both include Stages 1, 2 and 3. UniAD collision changes from 0.31% to 0.26%, while UniAD L2 remains 0.68 m at reported precision. The ST-P3 columns show the same qualitative pattern: collision changes from 0.11% to 0.09%, with L2 remaining 0.35 m. stage-ablationinteractionreporting-limits
What it supports. Adding Stage 2.5 yields a 0.05-percentage-point reduction in average UniAD collision. Its unchanged rounded L2 suggests that the gain is not simply a large improvement in mean path imitation. This is useful evidence for the proposed interaction curriculum, while still leaving the source of the gain incompletely isolated.
Where the evidence stops. The added stage also adds training. No equal-update control or repeated-seed uncertainty is reported, and rounded equality cannot prove identical underlying L2. A controlled continuation experiment is needed to distinguish motion supervision from the benefit of additional optimization.
7. Analysis & limitations
7.1 What the evidence leaves open
Open-loop nuScenes evaluation cannot establish closed-loop safety, recovery or long-tail deployment robustness. No measured inference latency supports the intended efficiency advantage; tables also omit repeated-run uncertainty. evaluationreporting-limits
Figure 2 branches MapQFormer from the 2D feature level, whereas the text defines map queries on BEV features. Figure 3's Stage 3 encoder icon is broader than the written frozen-backbone exception. Reproduction should follow the explicit text while flagging these visual ambiguities. perceptionplanningimplementation
Figure 4 is one qualitative intersection example assembled from different training stages. Its caption calls keep-forward the original instruction, whereas the prose calls the right turn ground truth; the example cannot resolve this intent-label ambiguity or establish generalization frequency. qualitative
7.2 Questions for discussion
- Would the Stage 2.5 collision gain survive an equal-update control and repeated seeds?
- How much visual dependence remains after controlling ego state and trajectory history?
- Do Stage 2 language abilities persist after Stage 3 planning tuning?
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include the nuScenes split and derived annotations, multitask-pretrained ResNet-101 perception with 200 × 200 BEV features, three two-layer projectors and Qwen2.5-Instruct. The paper reports four H100 GPUs, batch size 1 and roughly two days, without specifying which model size that timing covers or whether batch size is global. dataimplementation
The supplied paper defers detailed data and training configurations to a supplement. Stage lengths, optimizer, learning rates, loss weighting, trajectory precision and history construction remain unresolved. A proposed minimal check would compare Stage 2.5 against an equal-update continuation control before repeating final planning tuning. reporting-limitsinteractionstage-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Control the extra optimization in Stage 2.5
Reader-proposed check, not performed: start from one OpenDriveVLA-0.5B Stage 2 checkpoint and compare three continuations: skip Stage 2.5, use the reported agent-forecasting objective, or spend the same number of optimizer updates on continued driving QA. Keep the nuScenes split, initialization, trainable modules and final Stage 3 schedule fixed; use repeated seeds. Report UniAD and ST-P3 collision and L2 separately, with scene-level uncertainty. The mechanism receives stronger support if forecasting improves collision beyond both controls without materially degrading displacement. If equal-update QA matches the gain, additional optimization is a viable alternative explanation. instructioninteractionstage-ablationdatareporting-limits
Check 2: Test whether the planner responds to the visual scene when motion context is fixed
Reader-proposed check, not performed: reconstruct the full-input and no-vision 0.5B conditions in Table 4 using identical training schedules, then evaluate on held-out nuScenes scenes. Keep each example's ego state, history and command unchanged while comparing its correct visual tokens with tokens shuffled from a different scene. Report the existing open-loop metrics separately for all scenes and a prespecified subset with nearby agents or intersections; record invalid trajectories as failures rather than discarding them. If the full model uses scene-specific visual evidence, mismatched visuals should measurably alter plans or worsen performance, especially on that subset. Near invariance would weaken the claim that its aggregate advantage reflects substantial visual interaction reasoning. input-ablationperceptioninteractionevaluationreporting-limits
8.3 Reading coverage
Visual audit: The title, author block and AAAI-26 header were inspected on page 1; Figure 1 on page 2; architecture and perception formulation on page 3; training stages and objectives on page 4; trajectory decoding, training data and Table 1 on page 5; implementation, evaluation protocol and Tables 2–3 on page 6; and Figure 4, Tables 4–5 and their discussion on page 7. All six final crops were separately viewed. Arrow directions, train/freeze icons, table checkmarks, units and column assignments were cross-checked against the text and equations. The map-input arrow, frozen-2D-backbone exception and qualitative example's instruction/ground-truth ambiguity are disclosed. References on pages 8–9 were read as text; they supply no retained experimental claims. No appendix or separate supplement was supplied, and no code or external project material was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Introduction (PDF p. 1)
- Related Work and Figure 1 (p. 2)
- OpenDriveVLA: 3D perception and Stages 1, 2, 2.5, 3, Eqs. (1)–(6) (pp. 3–5)
- Experiments: training datasets, evaluations, implementation and main results (pp. 5–6)
- Ablation Study, command example and Conclusion (p. 7)
- References (pp. 8–9)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The complete supplied nine-page AAAI-26 proceedings edition was read, including references. Its title and six authors match the catalog; no title/author discrepancy or numbered revision was observed. No earlier edition was supplied for revision comparison.
- Text extraction does not reconstruct figure images; this limitation was addressed by inspecting PDF pages 1–7, all four figures and all five tables, and six final original crops.
- Separate supplemental material availability has not been fully verified. No supplement was supplied; the paper's deferred data details and training configurations remain outside this reading.
- No appendix is present in the supplied PDF. Code and project links were not opened, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1 (printed p. 13782), title, author block, conference header and copyright
The title and six authors match the catalog. This is the AAAI-26 proceedings paper, copyright 2026, by Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, Volker Tresp and Alois Knoll. Affiliations are Technical University of Munich, Germany, and Ludwig Maximilian University of Munich, Germany. No numbered revision is printed.
Go to primary source ↓problemPDF p. 1, Abstract and Introduction; p. 2, Figure 1 and Vision Language Models in Autonomous Driving
The authors motivate spatially grounded driving actions from the limits of static 2D, instance-agnostic VLMs. Figure 1 distinguishes direct VLA planning from language heads and high-level decision-makers feeding separate planners.
Go to primary source ↓perceptionPDF p. 3, Figure 2 and 3D Visual Environmental Perception
Multi-view 2D features feed scene sampling and BEV-based perception. The text defines scene tokens from 2D features and agent/map tokens from BEV features. Figure 2's visible MapQFormer input arrow instead branches at the 2D-feature level; the diagram and textual map-input formulation are not fully consistent.
Go to primary source ↓alignmentPDF pp. 3–4, Stage 1 – Hierarchical Vision-Language Alignment, Eqs. (1)–(2), Figure 3
Separate scene, map and agent projectors align visual tokens to captions. Agent alignment is instance-wise, with appearance and spatial descriptions; scene/map alignment is sample-wise. Only projectors train in Stage 1; visual encoding and the LLM are frozen.
Go to primary source ↓instructionPDF p. 4, Stage 2 – Driving Instruction Tuning, Eq. (3), Figure 3
Instruction-response training conditions on environmental tokens, textual ego state and a driving question. Projectors and the LLM train while visual encoding stays frozen. The authors intend learned driving knowledge to avoid inference-time chain-of-thought.
Go to primary source ↓interactionPDF p. 4, Stage 2.5 – Agent Environment Ego Interaction, Eq. (4), Figure 3
An auxiliary autoregressive task predicts each detected agent's future waypoints conditioned on previous waypoints, environment tokens, ego state and the projected agent token. Figure 3 marks the projector and LLM trainable, with visual encoding and QFormer frozen.
Go to primary source ↓planningPDF p. 4, Stage 3 – End-to-end Trajectory Planning Tuning; p. 5, Eqs. (5)–(6) and following paragraph
Ego waypoints are 2D coordinates tokenized into text and predicted autoregressively from environment, ego state and driving command, then decoded into numerical waypoints. Stage 3 jointly tunes 3D encoding, projectors and the LLM but freezes the 2D encoder. Figure 3's broad trainable Vision Encoder icon does not display that exception.
Go to primary source ↓dataPDF p. 5, Experiments / Training Datasets and all three data subsections; p. 6, opening paragraph
The paper uses the standard nuScenes train/validation split and data from TOD3Cap, nuCaption, nuScenesQA, nuX and GPT-Driver. Object captions receive BEV coordinates; multi-view captions are merged; map descriptions come from ground-truth annotations. Ego planning and agent forecasting use entity-relative displacements, six waypoints over three seconds at 0.5-second intervals.
Go to primary source ↓implementationPDF p. 6, Implementation Details
The implementation uses ResNet-101, perception pretraining on detection/tracking/map segmentation, a 200 × 200 BEV map, adaptive pooling for scene tokens, final-layer agent/map query tokens, two-layer GeLU MLP projectors and Qwen2.5-Instruct with full-parameter tuning. Reported training uses four NVIDIA H100 GPUs, batch size 1 and approximately two days. Decoding temperature is 0; detailed configurations are deferred to supplementary material.
Go to primary source ↓evaluationPDF p. 6, Evaluations and Driving Question Answering; p. 5, Training Datasets
Planning uses nuScenes validation with separate ST-P3 and UniAD metric settings, reporting displacement and collision at one, two and three seconds and their averages. Language results are measured after Stage 2. The supplied experiments are open-loop; no closed-loop or physical execution evaluation is reported.
Go to primary source ↓planning-tablePDF p. 5, Table 1, OpenDriveVLA, UniAD, GPT-Driver and EMMA rows, average columns and caption
Under UniAD metrics, OpenDriveVLA-7B reports average L2 0.66 m and collision 0.25%, versus UniAD 1.03 m and 0.31%. Under ST-P3 metrics, 3B/7B report 0.33 m and 0.10%, 0.5B reports 0.35 m and 0.09%, GPT-Driver reports 0.44 m and 0.17%, and EMMA reports 0.32 m with collision unreported. The caption says prior results are taken from cited summaries.
Go to primary source ↓language-tablePDF p. 6, Table 2, OpenDriveVLA, LiDAR-LLM and BEVDet+BUTD rows, BL-4, BERT-S and Acc columns; Table 2 caption
OpenDriveVLA-7B reports nu-Caption BLEU-4 27.6 and BERT-Score 92.2, versus LiDAR-LLM 19.3 and 91.3. nuScenes-QA accuracy is 58.4/58.5/58.2 for 0.5B/3B/7B, versus BEVDet+BUTD 57.0. The caption defines five question types but does not explain H0/H1.
Go to primary source ↓nux-tablePDF p. 6, Table 3, all OpenDriveVLA rows and Hint-VAD row
Nu-X CIDEr is 32.3/25.5/26.2 for 0.5B/3B/7B, versus Hint-VAD 22.4. The 0.5B model has METEOR 12.5 versus Hint-VAD 13.2. Larger model size does not improve every reported language metric.
Go to primary source ↓input-ablationPDF p. 7, Table 4, all checkmark patterns and UniAD columns; Effect of Input Modalities
For 0.5B, full inputs give average collision 0.26% and L2 0.68 m. Removing ego state gives 0.77%/1.34 m; history 1.14%/1.30 m; vision 0.29%/0.77 m; command 0.33%/0.80 m. Each omitted input is identified from the blank cell, not text-extraction order.
Go to primary source ↓stage-ablationPDF p. 7, Table 5, all rows and caption; Effect of Multi-Stage Training Strategy
All ablation rows include Stage 3. Stages 3 only, 1+3, 1+2+3 and 1+2+2.5+3 yield UniAD average collisions 0.37%, 0.32%, 0.31% and 0.26%, and L2 0.70, 0.69, 0.68 and 0.68 m. Adding Stage 2.5 changes ST-P3 collision from 0.11% to 0.09%, with L2 unchanged at 0.35 m.
Go to primary source ↓qualitativePDF p. 7, Figure 4, caption and Effect of Driving Command
One intersection illustrates different plans for keep-forward and turn-right prompts, with QA and agent predictions from earlier training stages. The caption calls keep-forward the original dataset instruction and turn-right modified; the prose calls the right turn ground truth. This leaves the example's original-intent/ground-truth relationship unclear.
Go to primary source ↓reporting-limitsPDF pp. 5–6, Training Datasets, Implementation Details and Tables 1–3; p. 7, Tables 4–5 and Conclusion
Data and detailed training settings are deferred to supplementary material. The supplied main paper gives no optimizer, learning rates, stage lengths, coordinate-token precision, history-window specification, latency benchmark or seed uncertainty. Its tables do not report error bars or repeated-run variation.
Go to primary source ↓8.5 Primary sources
OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model ↗
PDF · 7,047 extracted words
Source fingerprint
2801eed1a489ed1e4f2cc4a9253576c7c0fceb4632b9ff7d88b3d1a40e7117b8