AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
1. Paper overview
In one sentence: A shared language-and-motion decoder can improve driving plans and shorten reasoning, but its gains depend on data scale, decoding choices and benchmark conditions. e-frameworke-codebooke-sfte-rfte-navsime-carla
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors target two costs of language-based driving: numerical waypoint outputs can be inaccurate, while separate planners complicate the architecture; mandatory long reasoning also delays simple decisions. AutoVLA asks whether a shared token generator can connect visual understanding to motion while learning when extra textual reasoning is useful. e-frameworke-sfte-rft-results |
| Core mechanism | Source description A motion codebook turns trajectory planning into next-token prediction inside the VLM, with deterministic trajectory reconstruction rather than a separately learned trajectory head. e-frameworke-codebook |
| A key reported result | NAVSIM planning after reinforcement fine-tuning: Post-RFT: 89.11 PDMS; 1.31 s. PDMS ↑; average runtime ↓. nuPlan navtest; recorded-data/non-reactive benchmark, reported as open-loop in the paper. Before RFT: 80.54 and 3.95 s; runtime averages 500 test scenarios. An 8.57-point PDMS gain and reported 66.8% runtime reduction. Oracle best-of-six reaches 92.12, below TrajHF's 93.95 in Table 1; the prose's 'highest PDMS' is not valid across all listed methods. e-navsime-rft-resultse-data |
| Reading caution | Reader analysis Reported tables and curves do not provide repeated-seed uncertainty. Runtime remains GPU-intensive: Table 2's fast/slow means are 1.072/10.518 seconds. CARLA's 2 Hz simulation query schedule does not establish wall-clock 2 Hz deployment. e-navsime-carlae-rft-resultse-limits |
Core contributions
- Source description
A motion codebook turns trajectory planning into next-token prediction inside the VLM, with deterministic trajectory reconstruction rather than a separately learned trajectory head. e-frameworke-codebook
Figure 3. One decoder learns motion tokens and optional reasoning through two training stages. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower left: camera streams and tokenized prompts, instructions and ego states enter Qwen2.5-VL-3B. Its outgoing green and red token rows represent reasoning and actions from the same decoder. Above, the larger 72B model receives ground-truth decisions while creating reasoning data; trajectory labels supply action supervision. These arrows describe training information, not additional sensor inputs available at deployment. On the right, candidate outputs differ in both trajectory and response length. Their rewards feed the reinforcement update. The block labeled 'Reward Model' is specified in the text through benchmark-dependent driving scores and a reasoning-length penalty, rather than an independently documented learned reward network. e-frameworke-reasoninge-sfte-rfte-rewarde-codebooke-sign
What it supports. The architectural unification concerns language and motion generation. Knowledge distillation supplies an offline teacher, and GRPO supplies a training signal. Neither requires a second learned trajectory generator at ordinary inference, where the physical tokens are translated through the action codebook.
Where the evidence stops. The arrows agree with the caption and training description. The loss notation needs separate care: Algorithm 1 line 13 has a sign inconsistency with Equations (3)–(4). This illustration does not resolve the implementation used.
2. Motivation
2.1 The problem and the proposed response
The authors target two costs of language-based driving: numerical waypoint outputs can be inaccurate, while separate planners complicate the architecture; mandatory long reasoning also delays simple decisions. AutoVLA asks whether a shared token generator can connect visual understanding to motion while learning when extra textual reasoning is useful. e-frameworke-sfte-rft-results
2.2 What this reading follows
AutoVLA makes trajectory planning part of a vision-language model's token vocabulary. Its central choice is to describe a short vehicle movement with one physical token, then let the same decoder produce either a brief response or a longer explanation before the motion sequence. The paper combines supervised learning with GRPO to reward driving quality and discourage lengthy reasoning. Read the results through three distinctions: trajectory prediction versus executed control, ordinary generation versus oracle selection, and reasoning supervision versus inference-time reasoning. Those distinctions reveal both meaningful gains and limits that broad claims of autonomous-driving superiority would conceal. e-frameworke-codebooke-sfte-rfte-navsime-carla
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded VLA/One Model classification is supported by a shared autoregressive decoder that emits language and physical action tokens. The codebook is a deterministic motion representation; the offline teacher, reward computation and CARLA PID controller do not create a second learned planning generator. No explicit learned future-world prediction or inverse-dynamics action extraction is specified, supporting the recorded prediction-paradigm and quadrant 'Not applicable' values. e-frameworke-codebooke-rfte-carla
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a language vocabulary into a motion interface
The key interface is a local motion segment. A token retrieves displacement and heading change over half a second, so an autoregressive model can learn motion through the same next-token objective used for language. During training, continuous expert trajectories are mapped to nearby codebook entries. During inference, ten generated tokens are decoded sequentially to form a five-second trajectory. Composition matters: a token's displacement is local to the evolving pose rather than a waypoint in an unchanged global frame. This keeps trajectory decoding tied to an explicit motion dictionary. The paper uses a separate dictionary for CARLA and feeds the resulting trajectory to a PID controller. Thus the unified learned component predicts language and motion; execution still depends on feedback and control outside the decoder. e-frameworke-codebooke-inferencee-carla
Figure S1. Physical tokens encode short local motions, with different vocabularies for recorded driving and CARLA. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each arrow as a codebook motion's spatial displacement and heading, not a predicted route or probability. The left panel is built from WOMD trajectories and the right from CARLA-Garage. Gray arrows represent all 2,048 tokens; 300 are highlighted in blue for visibility, so blue does not indicate a special maneuver class. Compare the tick ranges before judging their shapes: the panels use different vertical scales. Appendix A explains how vehicle-contour distance spreads the selected half-second motions across the observed movement space. At inference, successive token displacements and rotations are composed from the ego pose to construct a longer trajectory. e-codebooke-inference
What it supports. The vocabulary carries a concrete motion meaning rather than an arbitrary latent action code. The paper deliberately changes that vocabulary for simulation because the underlying dynamics differ. This helps explain why CARLA results should not be described as deploying an identical real-world model configuration.
Where the evidence stops. The plot does not measure how likely a token is or certify smooth transitions between tokens. Its axes have numerical ticks but no explicit variable/unit labels; the displacement-and-heading interpretation comes from Appendix A.
5.2 Teach the model that reasoning is optional
The supervised target is not always a long explanation. Action-only examples contain a short standard template followed by motion tokens; reasoning examples add scene analysis, critical objects and intent before the same action interface. The full-sequence language loss teaches both formats, while an additional action loss emphasizes the trajectory portion. A large multiplier on CoT examples compensates for their relative scarcity. The teacher receives ground-truth driving hints while preparing annotations, whereas the driving policy must infer actions from observations and instructions. Figure 4 then asks whether this extra supervision helps. Its crossover shows that the benefit depends on training scale. Reader interpretation: adding explanations creates another learning burden, and the reported small-data degradation cautions against assuming that verbal supervision automatically improves control. e-sfte-traininge-reasoninge-scaling
5.3 Optimize a tradeoff, then separate its possible causes
GRPO samples several responses for one scenario, decodes their trajectories and compares their combined driving and reasoning-length rewards. The advantage is relative to the sampled group; KL regularization anchors learning to the supervised reference. NAVSIM supplies PDMS, while Waymo supplies normalized ADE because RFS labels are scarce. The authors report one update per step, allowing a simplified unclipped objective. Their NAVSIM experiment improves PDMS from 80.54 to 89.11 while average runtime falls from 3.95 to 1.31 seconds. Reader interpretation: this establishes a useful joint outcome, but does not isolate why it occurs. Driving-reward optimization, the length penalty and decoding choices can all affect behavior. A reproduction should also resolve Algorithm 1's printed sign mismatch before testing those causal explanations. e-rfte-rewarde-rft-resultse-inferencee-sign
5.4 Training and inference
During training
The base SFT mixture contains 166.3k nuPlan and 19.0k nuScenes samples. Waymo adds task fine-tuning; CARLA trains a separate model on 274.5k samples. Reasoning annotations cover subsets of each dataset, so action-only and reasoning-enhanced targets coexist. e-datae-inference
SFT updates the vision encoder and language model for five epochs on eight NVIDIA L40S GPUs using FSDP, BF16, learning rate 1e-5 and effective batch size 32. Action-loss weight is 1; CoT examples receive weight 40. e-traininge-sft
RFT freezes the vision encoder and uses LoRA rank/alpha 8, dropout 0.1, learning rate 3e-5 and KL weight 0.04 for 6,000 steps. Group-relative rewards compare candidates without a critic. NAVSIM uses navtrain and PDMS; Waymo uses validation and normalized ADE, not RFS. e-rfte-reward
During inference
Fast responses retain a short fixed template; slow responses generate structured reasoning before actions. Sampling also changes this behavior: Appendix E.4 contrasts temperature/top-p/top-k settings 1/0.5/20 with 0.1/0.01/1. Thus reasoning frequency must be assessed together with decoding settings. e-sfte-inference
Best-of-N is a separate evaluation setting: an oracle selects among six trajectories. Neither the teacher VLM nor the training reward is required for ordinary single-output decoding; CARLA still needs its trajectory-tracking controller. e-frameworke-navsime-carla
5.5 Implementation flow
- Encode the driving context
The vision encoder and text tokenizer feed Qwen2.5-VL-3B. Scene analysis, critical-object identification, intention reasoning and action generation share its autoregressive decoder. There is no learned future-image rollout in this formulation. e-framework
- Build a vocabulary of local motion
K-disk clustering selects 2,048 half-second displacements and heading changes using vehicle-contour distance with a 0.05 m separation threshold. WOMD supplies the real-world vocabulary; CARLA-Garage supplies a separate simulation vocabulary. Training maps each trajectory segment to its nearest token. e-codebook
- Distill reasoning with action hints
Qwen2.5-VL-72B annotates scenes with ground-truth meta-action hints; reformatted DriveLM answers add supervision. The reported human check scores 88.8% of 3,000 samples fully correct. This evaluates annotations, not the deployed policy's causal reasoning. e-reasoning
- Compose a trajectory
Generated tokens retrieve local displacements and rotations, composed from the current ego pose. In CARLA, the controller executes the plan and the model receives new observations every 0.5 simulation seconds. Locally observed motion primitives alone do not prove every composed plan safe. e-codebooke-carla
6. Experiments & results
AutoVLA extends Qwen2.5-VL-3B with discrete vehicle-motion tokens so one autoregressive decoder can produce scene reasoning and a driving trajectory. Supervised training teaches short and extended responses; GRPO then rewards planning quality while penalizing long reasoning. Gains depend on data, decoding and evaluation protocol, and physical execution is demonstrated in simulation.
6.1 Read the original evidence
Table 1. Reward-based post-training raises NAVSIM PDMS, while oracle selection remains a separate setting. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the three AutoVLA rows and keep their generation settings separate. One-shot reports 80.54 PDMS, post-RFT 89.11, and best-of-N 92.12. Section 4.2 explains that best-of-N uses an oracle to choose among six trajectories. Then scan the baseline rows: TrajHF reports 93.95 PDMS. The arrows point upward for every column; 'Collision' is a favorable no-at-fault-collision score, not a collision count or a failure percentage. The reward definition combines collision and drivable-area compliance with time-to-collision, comfort and progress. This table displays scores on a percentage-like scale, whereas the training reward is normalized to the unit interval. e-navsime-rewarde-data
What it supports. RFT improves AutoVLA's ordinary reported PDMS by 8.57 points over its one-shot baseline. Oracle best-of-six improves its result further, but does not lead the complete table. The results support substantial reward-aligned improvement without establishing across-method superiority or an executable oracle-free best-of-N system.
Where the evidence stops. The nearby claim of 'highest PDMS' must be limited to the listed AutoVLA variants: TrajHF is higher. This non-reactive benchmark and its oracle variant are not equivalent to closed-loop road testing.
Table 3. Closed-loop CARLA evidence comes from the separately trained SFT policy. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read driving score and success rate first, then check the other columns before declaring an overall winner. Driving score combines route completion with infraction penalties; success rate counts routes completed within the time limit without major infractions. AutoVLA leads these two columns, whereas VAD leads efficiency and comfort. Appendix F.4 supplies the execution mechanism behind these numbers: four front-camera frames, route instructions and sensor ego states feed the model; its five-second trajectory drives a PID controller for throttle, brake and steering. Queries occur every half second of simulation time. This is the CARLA-specific, single-camera model evaluated after SFT. e-carlae-datae-inferencee-rft-results
What it supports. AutoVLA reports 78.84 driving score and 57.73% success, versus Orion's 77.74 and 54.62%. The success advantage is 3.11 percentage points. The table demonstrates a closed-loop simulation result for the complete trajectory-and-controller setup, complementing the recorded-data planning tests.
Where the evidence stops. This table does not evaluate a CARLA RFT improvement or real-road deployment. Its simulation query frequency is not a measured wall-clock throughput claim, and its scores do not make AutoVLA best on every driving dimension.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM planning after reinforcement fine-tuning nuPlan navtest; recorded-data/non-reactive benchmark, reported as open-loop in the paper. | Post-RFT: 89.11 PDMS; 1.31 s. PDMS ↑; average runtime ↓ | Before RFT: 80.54 and 3.95 s; runtime averages 500 test scenarios. An 8.57-point PDMS gain and reported 66.8% runtime reduction. Oracle best-of-six reaches 92.12, below TrajHF's 93.95 in Table 1; the prose's 'highest PDMS' is not valid across all listed methods. e-navsime-rft-resultse-data |
| Bench2Drive closed-loop driving CARLA, separate single-camera SFT model; 2 Hz simulation replanning with PID control. | 78.84; 57.73%. Driving score ↑; success rate ↑ | Orion: 77.74; 54.62%. A 3.11-percentage-point success advantage over Orion. AutoVLA leads these two table columns, but not efficiency or comfort. This evaluates simulated execution, not real-world deployment or RFT's closed-loop benefit. e-carlae-datae-inference |
| Waymo end-to-end driving 1,505 test samples; nuPlan/nuScenes pretraining, Waymo SFT, then validation-set RFT. | Post-RFT: 7.557; 2.958 m. RFS Overall ↑; ADE at 5 s ↓ | Multi-camera, pretrained CoT SFT: 7.447; 3.115 m. ADE-based training improves the reference-matched RFS metric. The dated leaderboard reports 6.9436 Spotlight RFS, highest among its listed entries, but not the highest Overall RFS. e-waymoe-rfte-data |
| nuScenes planning Validation split; Table S2's UniAD protocol and its Avg. columns. | CoT: 0.70 m; 0.31%. Average L2 ↓; collision rate ↓ | Action-only: 0.71 m; 0.35%. OpenDriveVLA-3B: 0.67 m; 0.30%. The gain is small and protocol-dependent: ST-P3 average collision worsens from 0.18% to 0.20% with CoT. These two protocols must remain separate. e-nuscenes |
| Physical tokens versus textual waypoints Same mixed nuPlan/nuScenes training data; respective standard test sets, open-loop planning. | Physical tokens: 80.54; 0.70 m; 0.31%; 3.95 s. PDMS ↑; average L2 ↓; collision ↓; runtime ↓ | Text waypoints: 71.31; 0.89 m; 0.36%; 7.65 s. Supports this representation choice, but changes both motion representation and serialization length. It does not isolate a universal guarantee of dynamic feasibility. e-token-ablation |
| Reasoning supervision under data scaling Mixed nuPlan/nuScenes training at 10k, 50k, 100k and approximately 185k samples. | Reasoning: 44.06 at 10k; 80.54 at 185k. nuPlan PDMS ↑ | Action-only: 51.17 at 10k; 74.97 at 185k. The full-data advantage reverses at small data sizes. Figure 4 supports data-dependent benefits of reasoning supervision, not a blanket advantage at every scale. The caption describes a log-scaled axis, but the equal spacing of unequal size ratios conflicts with that description; comparisons here use the labeled points. e-scaling |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Reasoning supervision becomes beneficial as the training mixture grows. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First decode the line styles: open circles with dashed lines are action-only training; filled squares with solid lines include reasoning supervision. Each panel compares four labeled training sizes: 10k, 50k, 100k and 185k; read the values at those markers. Higher is better for the two nuPlan panels, while lower is better for the nuScenes panels. At 10k and 50k, the reasoning-trained model is worse on all four displayed metrics. By 100k, the ordering reverses, and it remains favorable at the full approximately 185k samples. These are supervised-training comparisons that precede the paper's RFT experiment; the solid lines should not be read as reinforcement-learning curves or as the effect of forcing longer reasoning at test time. e-scalinge-sfte-rft-results
What it supports. The nuPlan PDMS comparison changes from 44.06 versus 51.17 at 10k to 80.54 versus 74.97 at 185k, with reasoning listed first. The figure therefore supports a data-dependent benefit. It motivates asking how much supervision is needed before reasoning improves the driving policy.
Where the evidence stops. The caption says 'log-scaled x-axis', but these labels occupy equal horizontal intervals. Treat the points as discrete settings, not a verified logarithmic scale. Reasoning also underperforms at 50k, beyond the text's 'fewer than 50k' wording. No repeated-seed uncertainty or isolated test-time reasoning effect is shown.
Table 5. Physical motion tokens outperform the paper's textual waypoint baseline. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare horizontally within each row, following its direction arrow. Physical tokens have higher PDMS and lower L2 error, collision rate and runtime. Section 4.3 says both models use the same mixed nuPlan/nuScenes training data and the respective standard test sets. PDMS describes the NAVSIM comparison, while L2 and collision describe the nuScenes planning comparison; they are not four measurements of one common success probability. The output representation changes from written numerical waypoints to discrete motion tokens. The authors connect the runtime difference to the extra decoding needed for numerical text, so serialization cost is part of this comparison. e-token-ablatione-data
What it supports. PDMS rises from 71.31 to 80.54, while runtime falls from 7.65 to 3.95 seconds. The table supports the practical value of this physical-token representation against the chosen baseline. It is evidence about learned planning performance and output cost, beyond the separate reconstruction study in Table 4.
Where the evidence stops. This comparison changes representation and output length together. It does not show that the codebook guarantees collision-free motion, nor does it separate quantization benefits from easier sequence prediction. Table 5 provides no uncertainty estimates.
7. Analysis & limitations
7.1 What the evidence leaves open
Reported tables and curves do not provide repeated-seed uncertainty. Runtime remains GPU-intensive: Table 2's fast/slow means are 1.072/10.518 seconds. CARLA's 2 Hz simulation query schedule does not establish wall-clock 2 Hz deployment. e-navsime-carlae-rft-resultse-limits
Ground-truth hints can encourage plausible retrospective explanations; annotation accuracy does not establish faithful policy reasoning. Physical-token reconstruction and benchmark safety scores likewise do not certify arbitrary token compositions or unseen traffic conditions. e-reasoninge-codebooke-token-ablation
The authors identify unsafe language instructions and adversarial inputs as deployment concerns, and propose safeguarded communication and gated model updates. These are recommendations, not evaluated defenses. e-limits
7.2 Questions for discussion
- Would the reasoning penalty retain its benefit with identical decoding and equal inference budgets?
- How often do individually valid motion tokens produce incompatible transitions when composed?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires dataset-specific codebooks, split manifests, reasoning annotations, prompts and decoding settings. Table 4 reports 0.0182 m reconstruction ADE at K=2048 but does not identify its exact reconstruction evaluation split; that gap prevents a precisely matched tokenizer check. e-codebooke-datae-inferencee-token-ablation
Resolve Algorithm 1 line 13 before implementation: its outer and inner negative signs disagree with minimizing the negative of Equations (3)–(4). The intended main-text objective favors higher advantage and penalizes KL; the implementation actually used is unverified. e-sign
Other missing details include the numerical Waymo stationary-motion threshold, CARLA PID gains, software versions and the inference hardware/timing protocol. Reproduction should separately test reasoning-penalty effects under fixed decoding and tokenization effects under controlled compute. e-datae-carlae-traininge-inferencee-rft-resultse-token-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the reasoning penalty add value under fixed decoding?
Reader-proposed, not performed: start from one SFT checkpoint and compare no RFT, PDMS-only GRPO with lambda_r=0, and the reported lambda_r=0.3 reward. Hold training scenes, group size, update count, random seeds and inference temperature/top-p/top-k fixed. Before training, verify that the chosen loss increases a positive-advantage candidate's likelihood and discourages KL growth, resolving the Equation (3)/Algorithm 1 sign discrepancy explicitly. On untouched navtest scenes, measure PDMS components, reasoning-token count, slow-response frequency, invalid outputs and latency on declared hardware. Stratify by scene difficulty. If the penalty reduces reasoning without a practically meaningful safety/progress loss relative to PDMS-only RFT, it supports adaptive efficiency; equal results or worse difficult-scene performance would weaken that explanation. e-rfte-rewarde-rft-resultse-inferencee-signe-data
Check 2: Does the codebook improve executable motion beyond shorter output?
Reader-proposed, not performed: reproduce the text-waypoint versus K-disk comparison using identical training samples, backbone, reasoning targets, camera inputs and declared compute budgets. Report results both at matched optimization steps and matched inference budgets. First measure reconstruction ADE/FDE on an explicitly held-out trajectory split, including rare turns and stationary segments; report token usage and transition acceleration/jerk, not only nearest-segment error. Then evaluate single-output NAVSIM PDMS and its safety components with identical decoding rules and no oracle selection. A persistent planning advantage with fewer implausible transitions would support the physical representation mechanism. An advantage confined to runtime, or smooth individual segments that compose poorly, would narrow the claim to efficient serialization. e-codebooke-token-ablatione-navsime-datae-inference
8.3 Reading coverage
Visual audit: The title, author line, affiliation and v3/date markings were visually verified on p. 1. All main figures (1–6), supplementary figures (S1–S10), main tables (1–6), supplementary tables (S1–S4), method equations and Algorithm 1 were inspected on their original PDF pages. All six retained crops were individually viewed after their final extraction; Figure 4 is the scaling diagnostic crop. Supporting uncropped pages include training hardware and configuration, reward definitions, split/metric descriptions, decoding, Waymo results, qualitative cases, control details and broader impacts. Figure 3's dataflow and Figure 4's styles, labeled values and horizontal spacing were checked against their captions and text. Figure 4's caption calls its x-axis log-scaled, but the equal spacing of 10k, 50k, 100k and 185k conflicts with that claim; the evidence entry, result interpretation and visual caution explicitly preserve this discrepancy. The report preserves Algorithm 1's sign inconsistency and the Table 1/prose ranking discrepancy. References and acknowledgements on pp. 11–18 were read as text; external links, separately hosted supplements and videos were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and 1 Introduction (pp. 1–3)
- 2 Related Work (pp. 3–4)
- 3 AutoVLA, including 3.1 Framework, 3.2 Reasoning Data, 3.3 Supervised Fine-tuning and 3.4 Reinforcement Fine-tuning (pp. 4–7)
- 4 Experiments, including setup, main results and ablations (pp. 7–10)
- 5 Conclusions and limitations (p. 10)
- Acknowledgements and References (pp. 11–18)
- Appendix A Action Tokenization (p. 19)
- Appendix B Reasoning Data Collection (pp. 19–22)
- Appendix C Details of Supervised Fine-tuning (pp. 21–22)
- Appendix D Details of Reinforcement Fine-tuning, including Algorithm 1 and D.1–D.3 (pp. 23–24)
- Appendix E Experiment Details, E.1–E.4 (pp. 24–26)
- Appendix F Additional Results, F.1–F.4 (pp. 27–30)
- Appendix G Broader Impacts (p. 30)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction-only image gap was resolved by inspecting the original PDF pages and all final crops in this reading.
- Separate supplemental material availability has not been fully verified.
- All 12 supplied text chunks were read individually, including the supplementary material embedded in this 30-page PDF. External project pages, code, videos and separately hosted supplements were not inspected; experiments were not reproduced.
- Identity/version note: the reviewed artifact is arXiv:2506.13757v3, dated 5 November 2025, with a NeurIPS 2025 footer. Its title matches the catalog; the title page gives Seth Z. Zhao where the catalog abbreviates Seth Zhao. The catalog BibTeX year 2026 differs from the PDF's 2025 revision/venue labels. Earlier revisions and a separate proceedings edition were not supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, author line, arXiv margin and venue footer
The title matches the supplied observed title. Authors are Zewei Zhou, Tianhui Cai, Seth Z. Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou and Jiaqi Ma, affiliated with University of California, Los Angeles. The PDF identifies arXiv:2506.13757v3, 5 November 2025, and NeurIPS 2025.
Go to primary source ↓e-frameworkPDF pp. 2–3, Introduction and Figure 2; pp. 4–5, Figure 3 and Sections 3–3.1
Qwen2.5-VL-3B uses one autoregressive decoder for reasoning and physical action tokens. Real-world inputs use front, front-left and front-right camera streams with four frames per camera at 2 Hz, navigation instructions and ego states. The action vocabulary has 2,048 entries encoding displacement and heading change. The Introduction identifies textual-action feasibility, separate-planner complexity and fixed long reasoning as problems motivating the architecture.
Go to primary source ↓e-codebookPDF p. 19, Appendix A.1–A.2 and Figure S1
K-disk clustering selects 0.5-second motion segments using average vehicle-contour distance and a 0.05 m threshold. WOMD supplies the real-world codebook; CARLA-Garage supplies a separate 2,048-token simulation codebook. Decoding composes local displacements and rotations from the ego pose. Figure S1 highlights 300 of each codebook's 2,048 tokens.
Go to primary source ↓e-reasoningPDF p. 5, Section 3.2; p. 20, Appendix B.1 and Figure S2
Qwen2.5-VL-72B generates structured scene, critical-object, intention and decision annotations with ground-truth meta-actions as hints. DriveLM QA data is reformatted. The generation cap is 700 textual tokens; a binary human check of 3,000 randomly chosen annotations reports 88.8% accuracy, with erroneous checked samples corrected or discarded.
Go to primary source ↓e-sftPDF p. 6, Section 3.3, Equations (1)–(2)
SFT combines mean causal-language-model cross-entropy over the full output with an extra action-token cross-entropy. CoT examples receive a per-example multiplier. Fast-mode responses retain a fixed short reasoning template; slow-mode responses add structured reasoning before the action sequence.
Go to primary source ↓e-trainingPDF p. 2, Figure 1 training panels; p. 7, Section 4.1 Implementation Details; pp. 21–22, Appendix C
SFT trains the vision encoder and language model. The reported setup uses five epochs, eight NVIDIA L40S GPUs, FSDP, learning rate 1e-5, effective batch size 32, action-loss weight 1 and CoT-example weight 40. Appendix C specifies BF16, gradient checkpointing, 500 warm-up steps, 2% learning-rate decay every 2,000 steps and gradient clipping at 1.0.
Go to primary source ↓e-rftPDF p. 6, Section 3.4, Equations (3)–(4); p. 7, Implementation Details; pp. 23–24, Appendix D.1–D.3
GRPO compares sampled candidate outputs through group-standardized rewards and penalizes KL divergence from the SFT reference. One policy update per step permits the reported unclipped simplification. RFT runs 6,000 steps at learning rate 3e-5 with KL weight 0.04; the best-performing checkpoint is selected. The vision encoder is frozen; LoRA rank and alpha are 8, dropout 0.1. RFT uses nuPlan navtrain or Waymo validation.
Go to primary source ↓e-rewardPDF pp. 23–24, Appendix D.2–D.3, Equations (S2)–(S5)
Reward is driving reward minus 0.3 times a sigmoid CoT-length penalty, with tolerance 400 and slope 0.002. NAVSIM driving reward is NC times DAC times the weighted TTC, comfort and ego-progress mean; it is normalized to [0,1], and output failures receive zero. Waymo uses (2−ADE)/10. GRPO sampling uses temperature 1, top-p 1 and top-k 0.
Go to primary source ↓e-dataPDF pp. 24–25, Appendix E.1 and Table S1
Table S1 lists 166.3k nuPlan, 19.0k nuScenes, 23.8k Waymo and 274.5k CARLA training samples, with 45.6k, 2.9k, 7.2k and 53.2k reasoning samples respectively. NAVSIM evaluates navtest, nuScenes evaluates validation and Waymo evaluates 1,505 test samples. CARLA uses front-camera inputs only. The Waymo stationary-heading motion threshold is described but its value is not supplied.
Go to primary source ↓e-inferencePDF p. 7, Implementation Details; pp. 25–26, Appendix E.3–E.4
Ten action tokens define a five-second plan. The base model uses mixed nuPlan/nuScenes data; Waymo further fine-tunes it and uses current acceleration plus four seconds of position/velocity history. CARLA uses a separately trained, higher-image-resolution, single-camera model. Example inference settings are temperature/top-p/top-k 1/0.5/20 for diverse slow reasoning and 0.1/0.01/1 for deterministic fast responses.
Go to primary source ↓e-navsimPDF p. 7, Table 1, PDMS and component columns; p. 9, Section 4.2 nuPlan Benchmark Results
Reported NAVSIM PDMS is 80.54 one-shot, 89.11 post-RFT and 92.12 best-of-N. The latter uses an oracle scorer over six candidates. Table 1 lists TrajHF at 93.95 and Centaur at 92.10. AutoVLA post-RFT has collision score 98.41, area 95.64, direction 95.40, progress 81.87, TTC 98.04 and comfort 99.94.
Go to primary source ↓e-scalingPDF p. 8, Figure 4, four panels, legends and caption; Section 4.2 Data Scaling Results
Dashed circles denote action-only training; solid squares denote reasoning training. At 10k examples, nuPlan PDMS is 51.17 action-only versus 44.06 reasoning; at 185k it is 74.97 versus 80.54. At 50k, reasoning also underperforms action-only on all four displayed metrics. At 100k and 185k reasoning improves all four displayed metrics. The caption calls the x-axis log-scaled, but the four labels (10k, 50k, 100k and 185k) occupy equal horizontal intervals in the graphic, inconsistent with logarithmic spacing. The labeled values support comparisons at the four discrete training sizes; a logarithmic scale is not visually established.
Go to primary source ↓e-rft-resultsPDF p. 8, Figure 5(a)–(c), Table 2 and RFT Performance; p. 9, opening continuation
Figure 5(a) reports PDMS 80.54 to 89.11 and average runtime 3.95 to 1.31 seconds, with runtime averaged over 500 test scenarios. Authors report 10.6% relative PDMS improvement and 66.8% runtime reduction. Table 2 separately gives average fast/slow runtimes 1.072/10.518 seconds. Figure 5(b) compares training rewards for group sizes 2, 4 and 8, with the group-8 curve highest late in training.
Go to primary source ↓e-carlaPDF p. 9, Table 3 and CARLA Closed-loop Performance; p. 25, Appendix E.2; p. 30, Appendix F.4 and Figure S10
Bench2Drive evaluates the SFT model in CARLA. AutoVLA reports driving score 78.84, success rate 57.73%, efficiency 146.93 and comfort 39.33; Orion reports 77.74, 54.62%, 151.48 and 17.38. VAD reports efficiency 157.94 and comfort 46.01. Four front-camera frames, route instructions and sensor ego states feed a five-second trajectory prediction; a PID controller computes throttle, brake and steering, with queries every 0.5 simulation seconds.
Go to primary source ↓e-token-ablationPDF pp. 9–10, Section 4.3; p. 10, Tables 4–6
With the same mixed nuPlan/nuScenes training set, Table 5 compares text waypoints against physical tokens: PDMS 71.31/80.54, average L2 0.89/0.70 m, collision 0.36/0.31%, runtime 7.65/3.95 s. Table 4 reports K-disk K=2048 reconstruction ADE/FDE 0.0182/0.0203 m, movement coverage 99.42% and codebook usage 100%; K=4096 gives 0.0141/0.0155 m and 91.46% usage. Table 6 reports FAST/K-disk NAVSIM PDMS 67.63/80.54.
Go to primary source ↓e-nuscenesPDF p. 25, Appendix E.2 nuScenes; p. 26, Table S2; p. 27, Appendix F.1
Table S2 separates ST-P3 and UniAD evaluation protocols. Under UniAD, action-only/CoT average L2 is 0.71/0.70 m and collision 0.35/0.31%; OpenDriveVLA-3B reports 0.67 m and 0.30%. Under ST-P3, AutoVLA action-only/CoT average collision is 0.18/0.20%. Appendix F.1 suggests many nuScenes scenes do not need complex reasoning.
Go to primary source ↓e-waymoPDF p. 25, Appendix E.2 Waymo; p. 28, Table S3 and Appendix F.3; p. 29, leaderboard date; p. 30, Table S4
RFS matches a prediction to rated reference trajectories and penalizes deviations outside a trust region. Table S4 reports multi-camera, nuX-pretrained CoT SFT at RFS 7.447 and ADE@5s 3.115 m, versus post-RFT 7.557 and 2.958 m. Table S3 gives AutoVLA overall RFS 7.5566 and Spotlight RFS 6.9436, versus Poutine 7.9860 and 6.8929. The leaderboard snapshot is dated May 22, 2025; it is not a current ranking.
Go to primary source ↓e-limitsPDF p. 10, Limitation and Future Work; p. 30, Appendix G Broader Impacts
Authors acknowledge substantial GPU, memory and runtime requirements and propose quantization and runtime optimization. Broader impacts call for safeguards against adversarial inputs and unsafe instructions, secure communication and gated model updates rather than online continual RFT on vehicles.
Go to primary source ↓e-signPDF p. 6, Equations (3)–(4); p. 23, Algorithm 1 line 13 and Appendix D.1
The main text maximizes a positive policy-surrogate term minus KL regularization. Algorithm 1 line 13 prints an outer negative sum around an inner negative policy-surrogate term plus positive KL. Interpreted as a minimized loss, that sign arrangement differs from the main objective; the supplied PDF does not resolve the discrepancy.
Go to primary source ↓8.5 Primary sources
AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning ↗
PDF · 18,867 extracted words
Source fingerprint
ab726d03d44020c67792cf4783a0e05a5615f5f6ee5f2b760f4d69e1502e18d3