Unifying Language-Action Understanding and Generation for Autonomous Driving
1. Paper overview
In one sentence: LinkVLA couples language and driving trajectories through one token decoder, then accelerates action generation with endpoint-conditioned refinement whose reported latency excludes textual reasoning. e02e03e04e06e07e09e11e13
| At a glance | What to know |
|---|---|
| Research problem | Author claim A driving model can describe a lane change yet predict a lane-keeping trajectory. The authors attribute this mismatch partly to separate language/action representations and seek stronger semantic coupling without expensive sequential waypoint decoding. Shared representation is their proposed mechanism, not a guarantee of obedience. e02e04 |
| Core mechanism | |
| A key reported result | Closed-loop driving on Bench2Drive: LinkVLA: 91.01 DS; 74.55% SR; 73.40% multi-ability mean. Driving Score (DS); Success Rate (SR); multi-ability mean, higher is better. CARLA v2 protocol: 44 scenarios, five routes each; 220 official routes. SimLingo: 85.07 DS; 67.27% SR; 67.28% mean. Gains are 5.94 DS points and 7.28 percentage points SR. Efficiency is slightly lower (255.84 versus 259.23), so LinkVLA does not dominate every metric. e09e10 |
| Reading caution |
Core contributions
- Source description
Train endpoint prediction and coarse-path refinement inside that same transformer, replacing sequential waypoint decoding with two action-generation passes. e07
Figure 2. One multimodal decoder links language, spatial tokens and two-pass action generation. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left side as input preparation: InternViT turns the front view into visual embeddings, the tokenizer handles language, and the action codebook represents navigation and trajectory points. The legend distinguishes those token types from the red endpoint star. Follow arrows (a), (b), and (c) in sequence: predict the endpoint, interpolate coarse positions, then refine them. The right-hand stack expands that sequence and shows the two model forward passes. Figure 2 calls the cue an End Token; implementation text specifies separate path-goal and waypoint-goal tokens. The shared vocabulary is defined explicitly in Section 3.1, while the interpolation formula and refinement procedure are in Section 3.3. e03e04e05e07e08
What it supports. The mechanism reuses one transformer for goal selection and waypoint refinement. Interpolation inserts a cheap structural prior between learned passes. Thus the architecture supports a One Model VLA reading: it predicts ego trajectories and language, with no separate future-scene generator shown or specified.
Where the evidence stops. The drawing is schematic and does not specify attention masks or low-level vehicle control. Its two forward passes cover action decoding; the separately generated commentary precedes them. A smooth drawn path is not a formal feasibility guarantee.
2. Motivation
2.1 The problem and the proposed response
A driving model can describe a lane change yet predict a lane-keeping trajectory. The authors attribute this mismatch partly to separate language/action representations and seek stronger semantic coupling without expensive sequential waypoint decoding. Shared representation is their proposed mechanism, not a guarantee of obedience. e02e04
2.2 What this reading follows
A command such as changing lanes is useful only if the planned motion expresses it. LinkVLA tackles that connection at two levels: spatial actions become vocabulary entries in the language model, and the model learns to describe trajectories as well as generate them. Its second problem is computational: sequentially emitting every waypoint is slow. Endpoint prediction supplies a scaffold that the same transformer refines in parallel. The visuals below connect those choices to the reported driving, language-action and timing evidence. Keep the evaluation boundaries in view: closed-loop CARLA driving, offline instruction tests and trajectory-only latency answer different questions. e02e03e04e06e07e09e11e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
VLA and One Model are supported by architecture: one decoder handles language/action targets, endpoints and refinement. The vision encoder is a component of that system. No future-scene predictor or inverse-dynamics action extraction is described, supporting Not applicable for the world/action prediction quadrant. Autoregressive VLA describes the foundation and AR variant; deployed C2F uses parallel action refinement. e03e04e06e07e11
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn geometry into a vocabulary with a spatial prior
Begin with a continuous waypoint in the ego-centered BEV frame. LinkVLA applies a signed logarithm independently to its coordinates, bins the transformed position, and uses the corresponding action token alongside the language vocabulary. The implementation reports 5,656 action tokens from a 56-by-101 grid. This is a vocabulary of spatial cells whose embeddings are learned, not a learned dynamics state representing the future environment. Prediction returns a token distribution, and decoding returns the selected cell's center. Spatial soft-labeling modifies the training target: a Gaussian assigns some mass to neighboring tokens instead of treating every wrong location identically. These two choices serve different roles. The logarithmic transform allocates physical resolution; soft labels encode adjacency in the learning objective. Their effects should therefore be tested separately. e04e05e19
Figure S1. Logarithmic coordinates allocate more physical precision close to the ego origin. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two left panels before examining the heatmaps. Uniform physical grid lines keep equal spacing; the logarithmic grid concentrates lines around the origin. The right panels show waypoint distributions after assignment to token coordinates, so their axes are token indices rather than driving distance. Read each color bar separately: color conveys the plotted distribution, not a performance score. Equation (1) explains the remapping, and Section 4.2 supplies the actual working domain, scaling factor and transformed-grid step. The illustrative grid axes here should not be substituted for that full domain when implementing tokenization. The lower histogram spreads the represented waypoints over a different set of token indices. e04e05e19
What it supports. This visual explains why a vocabulary of action locations can encode a spatial prior before learning. Nearby physical positions receive finer distinctions where the log grid is dense. Gaussian soft targets add a second, separate prior by assigning probability to neighboring action tokens during training.
Where the evidence stops. Figure S1 is a representation diagnostic, not an isolated accuracy ablation. Table S1 changes both logarithmic scaling and vocabulary size, so its performance difference cannot establish a pure token-count effect.
5.2 Use the trajectory as evidence for language during training
Ordinary action generation asks the decoder to predict a trajectory from vision and an instruction. LinkVLA also supplies vision plus a trajectory and asks for the language describing that behavior. Training alternates which modality comes last and receives supervision, while sharing the decoder and action embeddings. Equation (4) scores the language tokens with preceding ground-truth text, and lambda weights this term alongside action generation. The reader's interpretation is that useful action embeddings must now support both motion prediction and linguistic discrimination. This is a plausible coupling mechanism, but the paper does not impose an exact cycle-consistency test on generated actions at inference. Because both tasks retain visual context and use structured labels, an important alternative explanation is that scene cues or templates carry part of the language signal. The proposed alignment check below tests that possibility. e06e12e15
5.3 Predict a goal, then refine a scaffold
Sequential action decoding creates a dependency chain through waypoint tokens. LinkVLA trains special goal cues with endpoint-first target order, allowing a first action pass to predict the endpoint. Linear interpolation from the ego origin constructs coarse waypoint tokens; a second pass maps their positions to refined trajectory points in parallel. The geometric path and temporal waypoint streams use this procedure symmetrically. This explains the computational change without assuming that the interpolated line is safe or executable. Refinement receives vision-language context to adjust the plan. A reader-identified concern is the training/inference difference: refinement is trained on scaffolds from ground-truth endpoints, but deployed scaffolds come from predicted endpoints. The paper's strong aggregate results do not isolate robustness to endpoint errors. Also remember that commentary is generated before these action passes and is excluded from the 48 ms measurement. e07e08e11
5.4 Training and inference
During training
Use AdamW with cosine scheduling: learning rate 1e-4, weight decay 0.1, betas 0.9/0.999 and dropout 0.1. Training lasts 30 epochs on 32 H20 GPUs, with reported batch size 48 and LoRA rank 32, alpha 64. Frozen modules are not enumerated. e08
Driving demonstrations use PDM-lite in CARLA. Adopted language data include 28M QA pairs over 1M Town 12 frames, heuristic labels and paraphrase augmentation. Commentary templates describe route actions, speed actions and IDM-derived reasons. e09e15
During inference
Generate commentary first, then condition action generation on it. The two C2F passes describe trajectory decoding, not the entire pipeline. Plans are evaluated in CARLA; the paper does not specify a complete deployment controller or demonstrate physical-vehicle execution. e07e08e09e11
5.5 Implementation flow
- Encode the scene and instruction
InternVL2-1B combines InternViT-300M-448px with Qwen2-0.5B-Instruct. Visual embeddings and language/navigation tokens enter the multimodal decoder; geometric and temporal trajectories use the action vocabulary. e03
- Discretize with spatial structure
Transform each coordinate logarithmically before uniform binning. The stated domain is x in [0,50] m and y in [-30,30] m; k=5 and transformed step 0.1 yield 5,656 tokens. Gaussian soft labels use radius 10 cells and sigma=1.2. Physical bins are finer near the ego vehicle. e04e05
- Make actions linguistically interpretable
Randomly arrange vision/language/action tuples as [V,A,L] or [V,L,A], supervising the last modality with the same decoder. Action understanding supplies auxiliary training supervision; no separate inference-time verifier is described. e06
- Teach a two-pass action decoder
Reorder targets endpoint-first, then train refinement on tokens interpolated from ground-truth endpoints. At inference, predict the endpoint, interpolate from the ego origin, and refine coarse waypoint positions in parallel using vision-language context. e07
6. Experiments & results
LinkVLA combines a shared language/action vocabulary, trajectory-to-language supervision and endpoint-conditioned parallel refinement. It improves reported CARLA driving and instruction-following scores, while its 48 ms trajectory-generation timing excludes textual reasoning. It predicts ego plans without learning future scene generation (e03, e06, e07, e10–e13).
6.1 Read the original evidence
Table 1. LinkVLA improves driving score and route success, while the full table preserves tradeoffs. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the highlighted LinkVLA row and compare it with SimLingo directly above. Driving Score and Success Rate are the principal closed-loop columns; the right-hand group breaks performance into merging, overtaking, braking, giving way and traffic signs, followed by their mean. Success Rate and the multi-ability group explicitly use percent units. Section 4.1 defines the evaluation as 220 CARLA routes across 44 interactive scenarios. Keep the asterisks attached to earlier methods: the retained note identifies expert feature distillation. Dashes in AutoVLA's multi-ability columns indicate unreported entries, not zero scores. The upward arrows specify the preferred direction of the displayed metrics. e09e10
What it supports. LinkVLA reaches 91.01 Driving Score and 74.55% Success Rate, versus SimLingo's 85.07 and 67.27%. Its multi-ability mean is 73.40% versus 67.28%. The efficiency column favors SimLingo, 259.23 versus 255.84, illustrating why the headline gain should not be generalized to every driving criterion.
Where the evidence stops. These are reported simulator outcomes. The table neither establishes equal training data and compute across architectures nor reports repeated-run uncertainty. Higher scores here do not by themselves demonstrate physical-road deployment reliability.
Table 2. Two-pass decoding reduces LinkVLA's trajectory-generation time from 361 to 48 ms. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare rows 3 and 4 to examine the within-LinkVLA decoding change. The autoregressive variant takes 361 ms and scores 90.66; C2F takes 48 ms and scores 91.01. Then use the external baselines to locate the tradeoff: SimLingo's MLP is faster at 34 ms, while Orion's VAE takes 65 ms. The latency arrow points downward and the Driving Score arrow upward. Table 2's caption identifies H20 hardware, and the adjacent discussion on page 7 defines the timing boundary more narrowly than the table header: it measures trajectory generation per frame and deliberately omits chain-of-thought computation. e07e08e10e11
What it supports. C2F retains strong driving performance while removing most of the cost of sequential waypoint emission. Against the fastest listed method, however, LinkVLA spends an additional 14 ms and gains 5.94 Driving Score points. The useful finding is a measured latency–performance tradeoff under the paper's timing convention.
Where the evidence stops. The 48 ms figure excludes commentary generation and cannot be interpreted as total perception-to-control latency. Table 2 does not specify latency dispersion or enough timing settings to reproduce the measurement exactly.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Closed-loop driving on Bench2Drive CARLA v2 protocol: 44 scenarios, five routes each; 220 official routes. | LinkVLA: 91.01 DS; 74.55% SR; 73.40% multi-ability mean. Driving Score (DS); Success Rate (SR); multi-ability mean, higher is better | SimLingo: 85.07 DS; 67.27% SR; 67.28% mean. Gains are 5.94 DS points and 7.28 percentage points SR. Efficiency is slightly lower (255.84 versus 259.23), so LinkVLA does not dominate every metric. e09e10 |
| Trajectory-generation latency Average per-frame trajectory inference on H20; CoT cost excluded. | C2F: 48 ms and 91.01 DS. Milliseconds, lower is better; DS, higher is better | LinkVLA AR: 361 ms and 90.66 DS; SimLingo: 34 ms and 85.07 DS. C2F removes most sequential action-decoding cost, but remains 14 ms slower than SimLingo. This is not end-to-end system latency. e11 |
| Instruction following on Action Dreaming Town 13 validation; offline, nonreactive tests with six category-specific success definitions. | Full LinkVLA: 87.16%. Mean success rate (%) | Baseline 70.11%; tokenization 81.63%; tokenization+C2F 81.87%. Alignment adds 5.29 percentage points over the C2F configuration. Final Lane Change reaches 97.42%, while Slower remains 65.57%; category differences matter. e09e12e13 |
| Language ability on DriveLM-hard Town 13 validation: ten examples per answer type; 330 VQA and 190 commentary types. | Full model: VQA 73.0/74.7/77.0; commentary 57.4/65.7/70.8. SPICE / BLEU / ROUGE-L, higher is better | Baseline: VQA 66.7/68.9/71.5; commentary 49.2/60.3/64.3. These gains concern generated text under the supplied label protocol; they do not independently establish faithful causal reasoning. e14e15 |
| Component ablation in closed-loop driving Bench2Drive; successive tokenization, C2F and alignment configurations in Table 5. | Full model: 91.01; 74.55%. DS; SR (%) | Tokenization: 89.57; 73.18%. Tokenization+C2F: 89.85; 72.27%. C2F alone slightly improves DS while lowering SR; adding alignment improves both. This is a sequential, not full factorial ablation; uncertainty is unreported. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Reciprocal action understanding raises mean instruction success, with uneven category gains. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the checkmarks as a sequence of configurations: baseline, tokenization, tokenization plus C2F, then the full model with Align., which means the unified action-understanding objective. The last column summarizes the six task columns but does not erase their differences. For the effect of alignment within the C2F setup, compare rows 3 and 4. For the cost of introducing C2F before alignment, compare rows 2 and 3, especially Slower and Stop. Appendix B.1 is essential context: these are offline, nonreactive trajectory tests with different success rules for speed, endpoint choice and stopping, rather than six versions of route-completion success. e06e09e12e13
What it supports. Mean success increases from 81.87% to 87.16% when alignment is added to tokenization+C2F, a 5.29-percentage-point gain. Yet final Slower success is only 65.57%, and the best Stop score belongs to tokenization alone at 99.88%. The final configuration improves the overall mean without winning every cell.
Where the evidence stops. The Objects task can include instructions to collide with objects; its success metric measures command alignment, not safe behavior. The main-text category list omits Stop despite saying six classes; Table 3 and Appendix B.1 resolve that omission.
Table 5. Closed-loop ablations support the complete combination, with a mixed intermediate C2F result. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the same Token, C2F and Align. switches as in Table 3, but change the question being asked: these columns measure closed-loop driving, not offline instruction success. Tokenization moves Driving Score from 85.07 to 89.57. Adding C2F brings it to 89.85, while Success Rate falls from 73.18% to 72.27%. Alignment then raises both columns to 91.01 and 74.55%. Read the printed units carefully: Table 5 marks only Success Rate as a percentage. The discussion on page 7 attaches percent signs to Driving Score, but this edition follows the table's distinction between score points and percentages. e12e16
What it supports. The complete combination has the highest displayed scores. The C2F-only step gives a small Driving Score increase and a Success Rate decrease, so the latency benefit should not be presented as an unqualified accuracy improvement in every configuration. Alignment contributes a further 1.16 DS points and 2.28 percentage points SR here.
Where the evidence stops. This is a sequential ablation, not a complete factorial design. It does not report tokenization plus alignment without C2F, and it provides no uncertainty estimates. The same component's effect may depend on which other components are enabled.
7. Analysis & limitations
7.1 What the evidence leaves open
Evidence covers simulator driving and offline language/action tests. The Objects category can reward collision-directed instructions, so instruction success is not a safety score. Selected qualitative examples do not estimate failure rates. e09e13e20
Reported tables lack seed dispersion or confidence intervals. Shared vocabulary and auxiliary captioning support an alignment hypothesis, but do not prove the authors' stronger claim that the modality gap is eliminated. e02e10e12e16
The token-count ablation changes k and vocabulary size together: 5,656 to 7,245 tokens accompanies DS 91.01 to 89.85. Spatial compression and capacity are confounded. Larger soft-label spread also lowers performance. e17e19
7.2 Questions for discussion
- Would action understanding retain its benefit under held-out instruction paraphrases and matched training compute?
- How much does C2F depend on endpoint quality when refinement was trained with ground-truth endpoints?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the backbone, expert-generated driving data, adopted language labels, token grid, both sequence orders and endpoint/refinement supervision. Verify the 56-by-101 vocabulary and cell-center decoding before evaluating the 220 routes. e03e04e05e06e07e09e15
Unspecified details include lambda, task-sampling probabilities, exact attention masks, LoRA target modules/freeze policy, batch-size scope, driving-data quantity, software versions, waypoint timing and deployment control settings. Radius R is stated, but Eq. (2) does not explicitly encode truncation. These gaps prevent an exact implementation from the paper alone. e04e05e06e07e08e09
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does action understanding use the action sequence?
Reader-proposed experiment, not performed: keep the backbone, token grid, C2F decoder, data split and training budget fixed. Compare correct trajectory-to-language supervision with within-batch shuffled action inputs for that auxiliary task, plus a generation-only control matched for training-token budget. Use held-out instruction paraphrases, preserving the source's six separate Action Dreaming success rules, and report per-category differences over multiple seeds alongside the mean. Evaluate trajectory-description accuracy as a diagnostic. A repeatable advantage for correctly paired trajectories over both controls would support action-dependent semantic grounding. Similar gains with shuffled actions would weaken that interpretation and suggest visual/template shortcuts or a generic extra-supervision effect. e04e06e12e13e15
Check 2: Measure refinement sensitivity and complete latency
Reader-proposed experiment, not performed: evaluate a fixed C2F model with ground-truth endpoints, its predicted endpoints, and controlled endpoint offsets measured in action-grid cells. Keep visual input, instruction, starting state and refinement weights unchanged; stratify results by endpoint error and route type. Compare against the AR variant using the same input frames. Measure trajectory error and closed-loop DS/SR, then time commentary and action decoding separately and together on the same H20 setup with identical batching and synchronized measurements. A sharp loss outside oracle endpoints would expose sensitivity to the training/inference scaffold mismatch. Report whether C2F's action-only speedup remains material after adding commentary cost. e04e05e07e08e09e11e16
8.3 Reading coverage
Visual audit: All 14 original PDF pages were rendered and visually inspected after all six text chunks were read. The pass includes the title/authors/version on p. 1; Figures 1–4 and S1–S2; Tables 1–7 and S1–S2; method equations and training/evaluation details on pp. 3–7; appendix tokenization and dataset protocols on pp. 12–13; and references on pp. 9–11. All six final crops were viewed at their final bounds. Figure 2's arrows were checked against its caption and Section 3.3; Figure 3's reciprocal directions were checked against Eq. (4). The schematic goal-token label is explained in the architecture guide. Table 5's units, the omitted Stop category in the main list, and the Table S2/Table 6 cross-reference mismatch are preserved in the evidence record. No supplied PDF page remains outside this visual pass. Separate supplemental availability remains unverified; external code/data and physical deployment were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work (2.1–2.3)
- 3. Method (3.1–3.3)
- 4. Experiments (4.1–4.4)
- 5. Conclusion
- References
- Appendix A. Action Tokenization (A.1–A.3)
- Appendix B. Dataset (B.1–B.3)
- Appendix C. Qualitative Results
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only image limitation above was addressed by visually inspecting all 14 supplied PDF pages, including the embedded supplementary material on PDF pp. 12–14.
- Only the supplied arXiv:2603.01441v1 (2 March 2026) was reviewed; no later revision was compared. The observed title and all ten authors match the catalog.
- Code and external datasets were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block, affiliations and arXiv margin
The title and all ten authors match the catalog. The artifact identifies arXiv:2603.01441v1, 2 March 2026. Affiliations are State Key Lab of CAD&CG, Zhejiang University; Li Auto. No revision difference is established.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1
LinkVLA addresses instruction–trajectory mismatch and sequential decoding latency through unified tokenization, action understanding and coarse-to-fine generation. The authors claim structural elimination of the modality gap.
Go to primary source ↓e03PDF p. 3, Figure 2 and caption; p. 6, Section 4.2, Training Details
InternViT embeddings and language/navigation tokens feed one multimodal model. InternVL2-1B combines InternViT-300M-448px and Qwen2-0.5B-Instruct. Figure 2 shows endpoint prediction, interpolation and refinement.
Go to primary source ↓e04PDF p. 4, Section 3.1, Unified Token Space and Eqs. (1)–(3)
Action grid tokens augment the text vocabulary. Signed logarithmic coordinates prioritize near-origin precision; Gaussian soft targets supervise action probabilities. Predicted tokens decode to cell centers. CoT adds language cross-entropy.
Go to primary source ↓e05PDF p. 6, Section 4.2, Action Tokenize
The domain is x in [0,50] m and y in [-30,30] m. With k=5 and transformed step 0.1, the grid is 56 by 101, or 5,656 tokens. Soft labels use R=10 cells and sigma=1.2; separate path-goal and waypoint-goal tokens support C2F.
Go to primary source ↓e06PDF pp. 4–5, Figure 3 and Section 3.2, Eq. (4); p. 7, Unified understanding and generation
One decoder predicts language from vision/actions or actions from vision/language. Training swaps [V,A,L] and [V,L,A] sequences. The total objective weights action understanding by lambda.
Go to primary source ↓e07PDF p. 5, Section 3.3, training, initialization, refinement and Eq. (5)
Endpoint-first targets and ground-truth-endpoint interpolation train C2F. Inference predicts endpoints, linearly interpolates from the ego origin, and refines points in parallel. Geometric and temporal waypoint streams are processed symmetrically.
Go to primary source ↓e08PDF pp. 6–7, Section 4.2, Training Details and inference continuation
AdamW uses cosine scheduling, learning rate 1e-4, weight decay 0.1, betas 0.9/0.999 and dropout 0.1. Training uses 30 epochs, 32 H20 GPUs, batch size 48 and LoRA rank 32/alpha 64. Commentary precedes 20 geometric and 10 temporal tokens per frame.
Go to primary source ↓e09PDF pp. 5–6, Section 4.1, Bench2Drive and validation settings
Driving data use PDM-lite in CARLA. Bench2Drive follows CARLA v2 protocol with 44 scenarios and five routes each, totaling 220 routes. Action Dreaming and DriveLM-hard validation use Town 13.
Go to primary source ↓e10PDF p. 6, Table 1, LinkVLA, SimLingo, TCP-ctrl and AutoVLA rows
LinkVLA reports DS 91.01, SR 74.55%, efficiency 255.84, comfort 34.62 and multi-ability mean 73.40%; SimLingo reports 85.07, 67.27%, 259.23, 33.67 and 67.28%. Asterisks denote expert feature distillation; AutoVLA multi-ability values are absent.
Go to primary source ↓e11PDF p. 6, Table 2 and caption; p. 7, Section 4.3, Inference Latency
On H20, AR takes 361 ms with DS 90.66; C2F takes 48 ms with DS 91.01; SimLingo takes 34 ms with DS 85.07; Orion takes 65 ms with DS 77.74. Timing excludes chain-of-thought computation.
Go to primary source ↓e12PDF p. 8, Table 3, all four rows and six task columns
Mean success progresses 70.11%, 81.63%, 81.87%, 87.16%. Final Faster/Slower/Target Speed/Lane Change/Object/Stop values are 96.48/65.57/74.73/97.42/91.41/97.34%. Tokenization-only Stop is 99.88%; C2F without alignment reduces Slower to 55.86%.
Go to primary source ↓e13PDF p. 13, Appendix B.1, simulator and success definitions; p. 6, Section 4.1 category list
Action Dreaming is offline and nonreactive, using bicycle-model/PID trajectories. Speed tasks use slope thresholds, lane changes compare endpoints, Stop requires minimum predicted speed below 0.1 m/s, and Objects can request collision. The main list says six classes but omits Stop; Appendix B.1 supplies all six.
Go to primary source ↓e14PDF p. 6, Section 4.1, DriveLM-hard; p. 8, Table 4, rows 1 and 4
Validation samples ten examples per answer type, with 330 VQA and 190 commentary types. Final VQA SPICE/BLEU/ROUGE-L is 73.0/74.7/77.0 versus 66.7/68.9/71.5; commentary is 57.4/65.7/70.8 versus 49.2/60.3/64.3.
Go to primary source ↓e15PDF p. 13, Appendix B.2–B.3
Adopted VQA training has 28M QA pairs over 1M Town 12 frames, heuristic labels and up to twenty GPT-4 paraphrases per QA. Commentary templates derive route actions, speed actions and IDM-based reasons from simulator state.
Go to primary source ↓e16PDF p. 8, Table 5, rows 1–4; p. 7, Section 4.4, Ablation Experiment
DS/SR progresses 85.07/67.27%, 89.57/73.18%, 89.85/72.27%, 91.01/74.55%. Only SR is labeled percent in the table; this report follows the table despite percent signs attached to DS in the prose.
Go to primary source ↓e17PDF p. 8, Table 6; p. 12, Appendix A.3 and Table S2; p. 7, Effect of Soft-labeling
Without/with soft labels, DS/SR is 90.85/72.73% and 91.01/74.55%. Increasing sigma from 1.2 to 3.0 yields 89.73/69.55%. The on/off prose cites Table S2, but its values match Table 6; actual Table S2 varies sigma.
Go to primary source ↓e18PDF p. 8, Table 7 and Navigation Modalities continuation
GPS target points yield DS 91.01/SR 74.55%; navigation commands yield DS 91.25/SR 73.18%. Neither modality wins both metrics.
Go to primary source ↓e19PDF p. 12, Appendix A.1–A.2, Figure S1, Eq. (S1), Table S1
Figure S1 contrasts uniform/log grids and waypoint distributions in token-index space. Changing k from 5 to 10 changes compression and token count from 5,656 to 7,245; DS/SR changes 91.01/74.55% to 89.85/70.45%.
Go to primary source ↓e20PDF p. 8, Figure 4; pp. 13–14, Appendix C and Figure S2
Selected CARLA sequences illustrate intersections, lights, parking exits and obstacle avoidance with generated language and plotted trajectories. They are illustrative cases rather than failure-frequency estimates.
Go to primary source ↓8.5 Primary sources
Unifying Language-Action Understanding and Generation for Autonomous Driving ↗
PDF · 9,857 extracted words
Source fingerprint
874bd354d18beb421be42cce4562b64704c452debb47b050a1d876463a84f018