AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Wenhui Huang; Songyan Zhang; Qihang Huang; Zhidong Wang; Zhiqi Mao; Collister Chua; Zhan Chen; Long Chen; Chen Lv

Affiliations: Nanyang Technological University, Singapore; Harvard University, US; Xiaomi EV, China

Source: ICML 2026 · ref-6c70da879cfe4f99ac28 ↗ · Project page ↗ · Catalog record

Reading: 288 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: AutoMoT uses shared attention and a reusable scene cache to accelerate action prediction, trading fresh semantic context for faster replanning while keeping current action-side vision. e01 e03 e04 e06 e09 e11 e12 e14 e16 e17

At a glance	What to know
Research problem	Author claim A driving policy needs current geometric observations and rapid replanning, while a large VLM supplies slower semantic reasoning. The authors seek to avoid the latency of synchronized language/action generation and the alignment problems of a VLM that merely conditions an independent planner. e02
Core mechanism	Source description A mixture-of-transformers (MoT) shares attention layer by layer between understanding and action experts, with different update rates and direct numerical trajectory supervision. e03 e04 e07 e09
A key reported result	Closed-loop driving on CARLA Bench2Drive: AutoMoT: 87.34 DS / 70.00% SR; AutoMoT+: 89.42 / 74.09%. Driving Score (DS) and Success Rate (SR, %); higher is better.. Official Bench2Drive evaluation; AutoMoT trained with PDM-Lite expert data. SimLingo: 85.07 / 67.27%. Refinement adds 2.08 DS and 4.09 percentage points SR to AutoMoT. Best values in Table 1. AutoMoT+ is reported as a multiple-run average, with no run count or dispersion; baselines differ in expert data and augmentation. e11
Reading caution	Reader analysis The authors acknowledge resource-intensive onboard deployment. Desktop-GPU timings and simulated closed-loop scores do not establish physical-vehicle readiness or rare-event safety. e22 e21 e11

Core contributions

Source description
A mixture-of-transformers (MoT) shares attention layer by layer between understanding and action experts, with different update rates and direct numerical trajectory supervision. e03 e04 e07 e09
Source description
A proposal-initialized diffusion refiner and backbone-adaptation experiments address trajectory quality and preservation of general VQA ability, respectively. e06 e08 e14

AutoMoT diagram with frozen 4B understanding expert, trainable 1.6B action expert, layer-wise joint attention, language/decision/planning heads, and a timeline reusing KV states between understanding updates. — **Figure 2.** Current action inference reuses a slower expert’s scene representations through shared attention. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Author claim

A driving policy needs current geometric observations and rapid replanning, while a large VLM supplies slower semantic reasoning. The authors seek to avoid the latency of synchronized language/action generation and the alignment problems of a VLM that merely conditions an independent planner. e02

2.2 What this reading follows

A large vision-language model can describe a driving scene, but a vehicle’s trajectory predictor needs to react before that model finishes every new reasoning pass. AutoMoT separates those update rates while connecting the models inside their attention layers. Its frozen understanding expert supplies scene keys and values; a smaller, trained action expert combines them with current observations to predict decisions and waypoints. An optional diffusion transformer refines those proposals. This reading follows that information flow, then separates three kinds of evidence: simulated closed-loop driving, predictions on recorded real-world data, and controlled ablations of caching and adaptation. The supplied artifact is the May 2026 arXiv v3 preprint. e01 e03 e04 e06 e09 e11 e12 e14 e16 e17

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	VLA
Architecture	Dual-system
Prediction paradigm	Not applicable
Quadrant	Not applicable

3.1 Evidence-based assessment

Insufficient evidence to decide

Reader analysis

VLA and the fast/slow functional split are supported. The categorical Dual-system architecture label is ambiguous: UE and AE have separate transformer blocks and update rates, but layer-wise shared attention integrates them into what the authors call one MoT VLA, explicitly distinguished from an auxiliary VLM plus independent planner. The source predicts actions and routes, not future world observations or inverse-dynamics actions, supporting the recorded Not applicable prediction paradigm/quadrant. e02 e03 e04 e09

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
Historical multi-view RGB images and system/user or navigation prompts for the understanding expert [e03, e04]. Current RGB, LiDAR BEV features, ego information, action queries and target points on the action side [e03, e04, e06].	Semantic language responses; three future meta-actions at one-second intervals [e03, e04]. Six temporal waypoints at 0.5-second intervals over three seconds and 20 spatial route points; optionally refined trajectories [e04, e07, e08].

4.2 Equations and their role

\begin{aligned}\widetilde K^l(t)&=[K^l_{\mathrm{scene}}(\tau(t))\Vert K^l_{\mathrm{act}}(t)],\\\widetilde V^l(t)&=[V^l_{\mathrm{scene}}(\tau(t))\Vert V^l_{\mathrm{act}}(t)],\\\operatorname{Attn}^l(t)&=\operatorname{softmax}\!\left(\frac{Q^l_{\mathrm{act}}(t)\widetilde K^l(t)^\top}{\sqrt d}\right)\widetilde V^l(t).\end{aligned}

Equations (4)–(5): l indexes the attention layer; τ(t) is the latest UE update; Q, K and V are queries, keys and values. Concatenation is along the sequence dimension, and d is their shared embedding dimensionality. Current actions can attend to cached scene context. e09

\tau'=(1+\epsilon_{\mathrm{mul}})\odot\tau

The refiner perturbs proposal trajectory τ by multiplicative Gaussian noise ε_mul; ⊙ denotes elementwise multiplication. Here τ is a trajectory, distinct from the cache-update time τ(t). The paper does not specify the noise variance or denoising-step count. e06 e08

5. Method in detail

5.1 Transfer representations without retraining the understanding expert

Reader analysis

The key distinction is between changing a model’s weights and updating its representation of a scene. AutoMoT freezes the Qwen3-VL-4B understanding expert during default action training, yet each UE forward can still produce new scene states. A separate action transformer learns to use those states through layer-wise shared attention. Figure 5 makes the structure explicit: the experts retain distinct normalization, QKV and feed-forward blocks around the shared attention space. Figure 3 supplies a task hierarchy: decisions condition on understanding, and planning conditions on both. The decision head is supervised with meta-action token labels, while numerical spatial and temporal trajectories receive L1 losses. Reader interpretation: this design places driving adaptation in a learned consumer of semantic features, rather than assuming that good scene descriptions automatically constitute an accurate control policy. e03 e04 e05 e07 e09

Action-refiner diagram showing noisy spatial and temporal trajectories entering MoA blocks, with BEV features, latent decision states, conditioning modulation, and parallel plus sequential cross-attention labels. — **Figure 4.** Diffusion refinement begins from the action expert’s trajectory proposals. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

5.2 Understand exactly what a cache-hit action step saves

Reader analysis

At action time t, the current AE observations produce queries, keys and values. The UE contribution comes from its most recent completed update, indexed by τ(t), which may precede t. Equations (4)–(5) concatenate those two sets of keys and values before the attention calculation. A stale semantic cache therefore does not imply stale current RGB/BEV input to the action branch. The controlled experiment tests a one-second mismatch and reports nearly unchanged average open-loop displacement. Table 6 then measures the computation avoided when the cached UE states are already available. Reader interpretation: this is evidence for a useful reuse mechanism under the tested delay, but a deployment scheduler must also pay for cache refreshes. Evaluating the mechanism requires logging both the age of each scene representation and the work needed to replace it. e04 e09 e16 e17

5.3 Distinguish the benefit of a module from the cause of that benefit

Reader analysis

AutoMoT’s optional refiner starts with an AE trajectory proposal, injects multiplicative noise, and denoises toward expert spatial and temporal trajectories. The proposed attention fusion gives the refiner access to both BEV geometry and decision latents. Table 1 shows that adding the complete refiner improves simulated driving scores, but it cannot attribute the improvement specifically to the fusion rule. Likewise, Table 9 shows that removing decision supervision harms average planning accuracy, whereas replacing the pretrained UE also changes how that branch is trained. These are useful interventions with different causal scope. Reader interpretation: a reproduction should preserve these distinctions. It should compare the full refiner against a matched alternative and test decision supervision separately from latent transfer, rather than treating every improvement as direct evidence that explicit language reasoning caused better driving. e06 e08 e11 e20 e04

5.4 Training and inference

During training

Source description

NuSync contains 80.1K curated samples with four historical RGB frames and an additional RGB–BEV pair; asynchronous pairs use 0.5- or 1-second offsets. Labels span five lateral and four longitudinal actions over three future seconds. CARLA’s PDM-Meta includes only longitudinal labels because lateral boundaries are ambiguous. e07 e18

Source description

AE jointly minimizes decision-token negative log-likelihood and spatial/temporal L1 trajectory losses. Refinement also uses L1 reconstruction; its PDM-Lite data contain over 700,000 samples across over 5,000 scenarios, with four historical frames at 2 Hz. Training uses FSDP, eight NVIDIA A100 GPUs and learning rates from 10^-4 to 2×10^-5. e07 e08 e10

During inference

Source description

At action time t, use current AE queries and concatenate its current keys/values with UE keys/values from the latest available update τ(t)≤t. Refresh the cache when new UE representations become available. This retains fresh action-side perception even when semantic context is stale. Optional refinement follows the AE proposal; caching does not eliminate periodic UE computation. e09 e06 e17

5.5 Implementation flow

Preserve the semantic backbone
The understanding expert (UE) uses Qwen3-VL-4B and remains frozen during default action-policy training. Historical images and prompts generate semantic responses and layer-wise latent representations. Freezing is a training choice; its scene cache still changes with new observations. e03 e04 e09
Predict actions through shared attention
The approximately 1.6B-parameter action expert (AE) is trained from scratch. Current RGB/BEV observations and queries enter separate transformer blocks; joint attention connects their representations to UE states. Cross-task ordering lets decision tokens attend to understanding and planning tokens attend to both. Separate heads decode meta-actions and spatial/temporal trajectories. e03 e04 e05
Refine a trajectory proposal
AutoMoT+ perturbs AE trajectories with multiplicative Gaussian noise, then performs truncated reverse denoising with a diffusion transformer. Adaptive layer normalization conditions on diffusion time, ego state and state history. Mixture-of-Attention combines trajectory self-attention, BEV attention and decision-latent attention, with learned gates and pooled residual bypasses. e06 e08
Separate prediction from execution
The learned action outputs are decisions and waypoint proposals. Bench2Drive tests their consequences in a closed-loop simulator; nuScenes tests predictions on recorded driving data. The supplied method does not spell out the complete waypoint-to-actuator controller, so a physical-vehicle execution pipeline cannot be reconstructed from these outputs alone. e04 e10 e11 e12

6. Experiments & results

AutoMoT transfers a frozen VLM’s scene representations into a separately trained action expert through layer-wise shared attention. Reusing the understanding expert’s key–value cache lets trajectory prediction run faster than semantic reasoning; an optional diffusion refiner improves simulated driving results. The strongest evidence combines CARLA closed-loop scores with a controlled cache-delay comparison, while real-world data are evaluated open-loop.

6.1 Read the original evidence

Bench2Drive comparison table with expert source, VLM and generative-planner indicators, Driving Score, and Success Rate; final AutoMoT and AutoMoT+ rows report 87.34/70.00 and 89.42/74.09. — **Table 1.** The refiner improves both reported Bench2Drive metrics. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
Closed-loop driving on CARLA Bench2Drive Official Bench2Drive evaluation; AutoMoT trained with PDM-Lite expert data.	AutoMoT: 87.34 DS / 70.00% SR; AutoMoT+: 89.42 / 74.09%. Driving Score (DS) and Success Rate (SR, %); higher is better.	SimLingo: 85.07 / 67.27%. Refinement adds 2.08 DS and 4.09 percentage points SR to AutoMoT. Best values in Table 1. AutoMoT+ is reported as a multiple-run average, with no run count or dispersion; baselines differ in expert data and augmentation. e11
Open-loop trajectory planning on nuScenes ST-P3 evaluation protocol, vector ego status; benchmark training/evaluation protocol followed.	0.32 m L2; 0.07% collision. Average L2 displacement (m) and collision rate (%); lower is better.	DriveTransformer-Large: 0.33 m / 0.07%; Drive-R1: 0.31 m / 0.09%. Competitive displacement and a tied best average collision rate among listed methods; these are recorded-data predictions, not physical driving success. e12 e10
Cache reuse versus synchronized planning Dedicated nuScenes validation with UE context offset by two steps (1.0 s); AutoMoT-S recomputes both experts.	Asynchronous: 0.324 m, 37.0 ms, 27.0 Hz without refinement; 63.0 ms, 16.0 Hz with refinement. Average L2 (m); action-step latency (ms) and frequency (Hz).	Synchronous: 0.322 m, 117.3 ms, 8.5 Hz; with refinement 143.3 ms, 7.0 Hz. Reported average error increases 0.62% while unrefined action-step latency falls 68.5%. The timing omits UE work on cache-hit steps, not from the entire system. e16 e17
Backbone adaptation and general knowledge retention Table 4 prompted frozen backbone versus AD-fine-tuned backbone; Section 4.3 describes held-out test splits.	LingoQA: 67.00→67.20; OmniDrive counterfactual GPT-Score: 18.20→67.80; TallyQA token accuracy: 81.40→52.40. Lingo-Judge, GPT-Score and token accuracy, as individually labeled.	InfographicVQA GPT-Score also falls from 89.30 to 50.20. The adaptation tradeoff is task-dependent. Counterfactual VQA improvement is not itself a measured improvement in executed planning; dataset descriptions conflict. e14 e15
Understanding and decision-objective ablations nuScenes planning; random-backbone variant and planning-only AE compared with default AutoMoT.	Default 0.32; randomly initialized UE 0.36; without decision objective 0.34. Average L2 displacement (m).	At 3 s: 0.54, 0.60 and 0.58 m, respectively. Decision supervision helps reported planning. The random-UE comparison changes both initialization and training treatment, so it does not isolate reasoning alone. e20
Discrete decision prediction NuSync synchronous/asynchronous comparison and separate Senna nuScenes benchmark.	NuSync: asynchronous 53.10%, synchronous 53.49%; Senna: AutoMoT 90.92%, Senna baseline 88.47%. Reported joint average accuracy (NuSync) and accuracy (Senna).	NuSync decreases 0.39 percentage points; Senna increases 2.45 percentage points. Protocols are separate. Table 7 omits an explicit 3-second column despite three-second labels; its reported averages cannot be reconstructed from the displayed 1/2-second columns alone. e18 e19

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Backbone adaptation table comparing prompted AutoMoT with a fine-tuned variant on LingoQA, OmniDrive and five general-knowledge datasets, with metric and variant definitions retained. — **Table 4.** Driving-specific adaptation helps counterfactual VQA while hurting several general benchmarks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Latency table comparing synchronous and asynchronous AutoMoT, with and without generative refinement, showing UE, AE, AR, total milliseconds and frequency. — **Table 6.** Cached action steps avoid recomputing the understanding expert. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Component ablation table reporting L2 at one, two and three seconds and average L2 for AutoMoT, a random-UE variant, and a planning-only variant. — **Table 9.** Both pretrained understanding and decision supervision contribute to the reported planning result. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

The authors acknowledge resource-intensive onboard deployment. Desktop-GPU timings and simulated closed-loop scores do not establish physical-vehicle readiness or rare-event safety. e22 e21 e11

Reader analysis

Section 4.1 names LingoQA and CODA-LM for UE fine-tuning, whereas Section 4.3 names LingoQA and OmniDrive counterfactual reasoning. Figure 4 additionally labels a sequential cross-attention path that is not mapped clearly to the prose’s parallel attention and pooled bypass formulation. e15 e06

Reader analysis

The 1-second cache-offset result is an aggregate open-loop check. It does not establish robustness to longer delays, abrupt scene changes or scheduling contention. Table 9’s planning-only variant equals default L2 at 1 second after rounding, despite prose claiming degradation at every horizon. e16 e17 e20

7.2 Questions for discussion

Would cache reuse remain effective in scenes with sudden hazards rather than average validation traffic?
How much of the planning benefit comes from pretrained visual representations versus language reasoning and decision supervision?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

A reproduction requires the Qwen3-VL-4B backbone, NuSync/PDM-Meta label construction, nuScenes/PDM-Lite data, benchmark evaluators and shared-attention/cache implementation. Learning rates and GPU count are given, but batch size, training duration, optimizer details, loss weights, exact prompts, noise schedule and denoising-step count remain unspecified in the supplied method/setup. e03 e06 e07 e08 e10

Reader analysis

Proposed checks: sweep cache age with weights and action observations fixed, logging cache-refresh cost and planning errors; separately ablate latent transfer and decision supervision under matched training. Table 10 places AutoMoT inference on RTX 5090; its unknown-GPU/L40S-training footnote concerns AutoVLA, not AutoMoT. e09 e16 e17 e20 e21

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Measure the accuracy and full cost of increasing cache age

Reader-proposed, not performed: use one trained AutoMoT checkpoint and identical nuScenes action-side observations, holding the refiner setting fixed. Compare synchronous recomputation with cached UE states aged 0.5, 1.0 and 2.0 seconds; keep the extra 2.0-second condition explicitly beyond the paper’s reported validation. Log L2 at each horizon, collision rate, joint decision accuracy, cache age, refresh latency and action-step latency on the same hardware. Report both cache-hit and amortized costs, with uncertainty across scenes. Stratify a predefined subset with abrupt changes such as braking or cut-ins. A sharp error increase at one second in those scenes, or disappearance of the timing advantage when refresh work is counted, would limit the paper’s aggregate cache-reuse conclusion. e09 e10 e16 e17 e18 e21

Check 2: Separate decision supervision from access to decision latents

Reader-proposed, not performed: first train matched default and planning-only AE variants with the same pretrained frozen UE, data, initialization scheme, optimization budget and seeds, following the distinction in Table 9. Independently compare normal versus blocked decision-to-planning attention during training and evaluation, preserving understanding-to-planning access, inputs and model size. Leave the optional refiner off for this primary comparison so its decision-latent pathway does not confound the intervention. Compare per-horizon L2, collision rate and decision accuracy across seeds. If the auxiliary decision loss helps equally when its latent pathway is blocked, shared training regularization becomes a plausible explanation; an additional benefit from the intact pathway would support actual use of decision representations by the planner. e03 e04 e05 e06 e07 e12 e20

8.3 Reading coverage

Visual audit: The title/author/affiliation page, Figures 1–5, Tables 1–10, and all pages supporting retained method, training, numerical, evaluation and reproduction details were rendered and visually inspected. Every final PNG crop was separately viewed. Figure 2’s KV directions and frozen/trainable markers were cross-checked against Sections 3.1/3.3 and Figure 5; Figure 3’s task ordering was checked against its prose. Figure 4’s unresolved parallel/sequential labeling and the fine-tuning dataset inconsistency are disclosed. Table 4 retains its caption because it contains the required variant and metric key. Reference-only pages 11–12 were read in the complete text chunks, but were not part of the visual pass. Separate supplements, code and external resources were not inspected.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Abstract
1. Introduction
2. Related Work (2.1–2.2)
3. AutoMoT (3.1 Network Architecture; 3.2 Training Strategy; 3.3 Asynchronous Inference with KV Caching)
4. Experiments (4.1–4.4)
5. Conclusion
6. Impact Statement
References
A.1. Data Flow of AutoMoT
A.2. Decision-Making Benchmark Results
A.3. Impact of Individual Component
A.4. Inference Latency Comparisons

Outside the original text pass

Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
The retained PDF was inspected separately: all five figures and ten tables, plus every page supporting retained technical claims, were visually reviewed.
Separate supplemental material availability has not been fully verified.
All six supplied text chunks were read individually, covering the complete 15-page PDF including references and Appendix A. No code, external project pages, or separate supplements were inspected; no experiments were reproduced.
Identity/version: the title and all nine authors match the supplied catalog. The inspected artifact is arXiv:2603.14851v3 [cs.CV], dated 14 May 2026 in the margin, with a title-page preprint date of May 15, 2026. The catalog lists ICML 2026; this reading verifies the supplied preprint, not a separate proceedings edition. Earlier revisions and any differences from them were not supplied.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e01PDF p. 1, title, author/affiliation block, arXiv margin and preprint footer.Inspect

Exact catalog title and nine authors are present. Margin identifies arXiv:2603.14851v3, 14 May 2026; footer says Preprint, May 15, 2026. Affiliations are Nanyang Technological University, Singapore; Harvard University, US; Xiaomi EV, China.

1. Paper overview

Core contributions

2. Motivation

2.1 The problem and the proposed response

2.2 What this reading follows

3. Research context

3.1 Evidence-based assessment

4. Problem formulation

4.1 Inputs and outputs

4.2 Equations and their role

5. Method in detail

5.1 Transfer representations without retraining the understanding expert

5.2 Understand exactly what a cache-hit action step saves

5.3 Distinguish the benefit of a module from the cause of that benefit

5.4 Training and inference

During training

During inference

5.5 Implementation flow

6. Experiments & results

6.1 Read the original evidence

6.2 Results and evaluation conditions

6.3 Ablations and diagnostic examples

7. Analysis & limitations

7.1 What the evidence leaves open

7.2 Questions for discussion

8. Reproducibility audit

8.1 Requirements and known gaps

8.2 Proposed reproduction checks

Check 1: Measure the accuracy and full cost of increasing cache age

Check 2: Separate decision supervision from access to decision latents

8.3 Reading coverage

Outside the original text pass

8.4 Traceable evidence

8.5 Primary sources

AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving ↗