BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving
1. Paper overview
In one sentence: BrainWAM coordinates separate semantic and predictive action streams to improve simulated driving plans, while retaining an inference cost that still limits vehicle deployment. e02e03e04e05e07e08e09e18
| At a glance | What to know |
|---|---|
| Research problem | Source description A driving planner must combine route and traffic-rule semantics with anticipated scene dynamics. The authors find that Tri-MoT, which puts vision-language, video and action tokens into shared attention, favors clean semantic tokens over denoising video tokens. Their proposed remedy is to exchange compact action representations after each branch has processed its own modality. e02e03e05 |
| Core mechanism | |
| A key reported result | NAVSIM v2 trajectory planning: 89.6 EPDMS ↑. The v2 non-reactive planning benchmark adds direction, traffic-light, lane and comfort criteria; exact split/counts are not specified. DriveDreamer-Policy: 88.7; DriveVLA-W0: 86.1. The 0.9-point advantage over DriveDreamer-Policy accompanies EP 88.2 versus 87.9 and EC 85.8 versus 79.4. BrainWAM's NC is lower than that baseline's; aggregate leadership is not universal safety-metric leadership. e07e09 |
| Reading caution | Source description Keeping both branches and the video backbone increases compute and memory relative to a single branch. The authors explicitly state that 475–644 ms inference does not meet strict real-time vehicle deployment requirements. e18 |
Core contributions
Figure 3. Two specialized branches exchange action representations before decoding a shared trajectory. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the inputs. The blue side receives history and noisy future video alongside a noisy action; the pink side receives scene images and an instruction alongside its own action representation. Each Dual-model MoT block couples an action expert to its respective backbone. The central purple CAB exchanges messages between action streams. The green CIF sits above them and feeds the action decoder. The right-hand insets separate CAB's bidirectional cross-attention and gating from CIF's self-attention. The diagram also separates video and action timestep conditioning, which enables their different inference schedules. e03e04e05e06e13e14e18
What it supports. BrainWAM combines semantics and prediction after both have been expressed as action tokens. CAB operates during representation refinement, whereas CIF integrates the final intents. This architectural separation supports the dual-system classification and identifies the learned trajectory decoder as the action output mechanism.
Where the evidence stops. The clean future video shown here describes the generative branch. Efficient inference can stop that branch early and reuse its features. The diagram does not demonstrate physical execution, nor does it imply that both large backbones are cheap to run.
2. Motivation
2.1 The problem and the proposed response
A driving planner must combine route and traffic-rule semantics with anticipated scene dynamics. The authors find that Tri-MoT, which puts vision-language, video and action tokens into shared attention, favors clean semantic tokens over denoising video tokens. Their proposed remedy is to exchange compact action representations after each branch has processed its own modality. e02e03e05
2.2 What this reading follows
A useful driving plan must respect instructions and traffic rules while anticipating how a scene will evolve. BrainWAM gives these demands separate pathways: a vision-language action expert and a video-based world-action expert. Instead of mixing their raw modality tokens, it lets each pathway form action representations and then coordinates those representations through CAB and CIF. This reading follows that information flow, examines the attention diagnostic motivating it, and separates aggregate NAVSIM improvements from individual safety metrics. The final latency table explains both the value of early predictive context and why the authors still identify deployment efficiency as unfinished work. e02e03e04e05e07e08e09e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The dual-system/joint-prediction classification is supported by separate pretrained VLA and WAM branches, with video/action prediction inside WAM and action-space coordination across branches. Frozen branches remain distinct during joint training. This is not inverse dynamics, and video computation is used at inference rather than only as auxiliary supervision. The efficiency subcategory describes an optimization goal; real-time deployment is explicitly unmet. e03e05e06e18
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Train the two experts to predict compatible action velocities
The two pathways do not begin as an undifferentiated jointly trained network. Stage 1 trains the WAM branch to denoise future video and action trajectories; Stage 2 trains the VLA branch to turn scene semantics, instructions and ego history into action predictions. Both use the paper's linear interpolation between clean trajectories and Gaussian noise, whose target action velocity is noise minus clean action. The WAM additionally learns a video velocity field with its own timestep. In Stage 3, both experts receive the same noisy trajectory and action timestep, making their representations comparable at the same refinement state. Their pretrained parameters are frozen, while CAB, CIF and the final decoder learn the fused velocity prediction. This training sequence preserves specialization while providing a shared action-level interface. e03e04e05e06
5.2 Understand what CAB communicates and what CIF combines
CAB connects the intermediate representations, rather than choosing between two completed trajectories. Each action expert supplies eight tokens with 1024 features per token. At layers 9 and 18, one stream queries the other through cross-attention and receives a gated residual message; the reverse direction is computed in parallel. The tanh gates start at zero, so the bridge initially leaves the pretrained representations unchanged. CIF then handles a different problem: combining the refined intents at the output. It separately projects the streams, adds source embeddings, and processes their concatenation with a two-layer Transformer conditioned on action timestep. The resulting streams are averaged and decoded into fused action velocity. Table 4 supports retaining both mechanisms: CAB alone scores 88.7 PDMS, CIF alone 88.5, and both 89.5. e05e11e13e14
5.3 Separate useful predictive context from complete video generation
Independent video and action timesteps matter at inference because the planner need not repeatedly run the video backbone after it has supplied useful context. The source uses three action sampling steps and caches video features after early termination. Table 5 tests how many video updates are retained. The zero-step row performs poorly, whereas one step recovers most of the score; further steps change the score only slightly while increasing latency. A reader interpretation is that planning can benefit from partially refined predictive features before investing in all available video updates. This is not evidence that early features are fully accurate future scenes, nor that later video refinement is universally unnecessary. The appendix's ten-step joint-denoising ablations test architectural capacity under a different protocol, and the deployment limitation remains explicit. e03e06e12e15e18
5.4 Training and inference
During training
Stage 1 optimizes video and action flow losses in the WAM branch. Stage 2 independently trains the VLA branch with action flow loss. Stage 3 freezes both pretrained branches and trains only CAB, CIF and the final decoder against fused action velocity. Shared action noise aligns the two branches; video retains its own timestep. e03e04e05e06
Each stage runs 100K steps on eight NVIDIA H20 GPUs, batch six per GPU. The optimizer is AdamW with peak learning rate 5×10⁻⁵, weight decay 0.01, cosine decay and 200 warmup steps. Training uses bf16 and DeepSpeed ZeRO-2, saving checkpoints every 3K steps. e06
During inference
Both action experts start from identical trajectory noise and use three action denoising steps. CAB and CIF coordinate each prediction. The video branch stops early and caches intermediate features for subsequent action updates, avoiding repeated video-backbone evaluation after termination. e05e06e12
The planner is evaluated by submitting predicted trajectories to short-horizon, non-reactive simulation of logged driving scenes. The paper does not demonstrate a physical controller executing them or an online feedback policy adapting to reactive road users. e07
5.5 Implementation flow
- Build predictive action tokens
Wan2.2-TI2V-5B denoises future visual latents while a lightweight action expert denoises the ego trajectory. Dual-MoT shares self-attention between visual and action tokens while retaining modality-specific feed-forward networks. The WAM predicts both video and action vector fields. e03
- Build semantic action tokens
Qwen3-VL-4B encodes images and instructions into semantic tokens U and ego history into state tokens E. A separate rectified-flow action expert creates semantic-grounded action tokens; Dual-MoT couples these with semantic and state information. e04
- Bridge matching action representations
Both experts receive the same noisy trajectory and action timestep. CAB uses each stream as queries and the other as keys/values, then injects messages through zero-initialized tanh gates. Default bridges sit at action-expert layers 9 and 18; each stream has eight 1024-dimensional tokens. e05e13
- Fuse and decode
CIF separately projects the streams, adds source embeddings, concatenates them and applies a two-layer, eight-head Transformer with action-timestep-conditioned AdaLN. It averages the resulting paired streams and decodes the fused action velocity. The trajectory is learned directly, without a separate inverse-dynamics action extractor. e05e14
6. Experiments & results
BrainWAM turns a vision-language planner and a video/action world model into two specialized action streams, then coordinates them through gated cross-attention and learned fusion. It improves reported NAVSIM planning scores while retaining costly video computation at inference; the evidence concerns non-reactive simulation, not physical vehicle deployment.
6.1 Read the original evidence
Table 1. BrainWAM leads the listed non-human planners in aggregate NAVSIM v1 score. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the shaded PDMS column, then inspect its contributing metrics rather than treating the aggregate as a success percentage. NC denotes no at-fault collision, DAC drivable-area compliance, TTC time-to-collision, C comfort and EP ego progress; higher values are preferred. Image and lidar checkmarks reveal differing sensor inputs among external methods. Bold and underlining mark the paper's best and second-best results. Compare BrainWAM with AutoVLA and DriveLaW, both at 89.1 PDMS, then inspect DAC and EP to see where BrainWAM's row is strongest. The Human row is a separate reference. e07e08e19
What it supports. BrainWAM reaches 89.5 PDMS, 0.4 score points above AutoVLA and DriveLaW. Its DAC is 97.5 and EP is 83.8. Those aggregate gains coexist with NC 98.1 and TTC 94.9, which are below several competing rows. The result therefore supports improved overall benchmark planning rather than superiority on every component.
Where the evidence stops. The source does not provide repeated-run uncertainty, exact split/sample counts, or matched external training budgets. Listed inputs also differ. Human PDMS is 94.8, and the evaluation remains short-horizon, non-reactive simulation of logged scenes.
Table 2. The aggregate advantage extends to NAVSIM v2's broader rule and comfort criteria. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the rightmost EPDMS column, then move left to distinguish its components. NC, DAC, DDC and TLC are the collision, drivable-area, driving-direction and traffic-light penalty terms. TTC, EP, HC, LK and EC contribute the weighted time-to-collision, progress, history-comfort, lane-keeping and extended-comfort terms. Compare BrainWAM first with DriveDreamer-Policy, the strongest listed baseline by EPDMS. Then check which component columns favor each row. Several rule scores are close to their ceiling, so the progress and comfort columns help explain the aggregate comparison without implying uniform improvement. e07e09
What it supports. BrainWAM reports 89.6 EPDMS versus DriveDreamer-Policy's 88.7. Its EP is 88.2 versus 87.9 and EC is 85.8 versus 79.4. However, its NC is 98.1 versus 98.4. This is a favorable overall tradeoff on the benchmark's composite metric, not a clean sweep of the component scores.
Where the evidence stops. EPDMS and PDMS use different components and cannot be compared as interchangeable measurements. These are source-reported benchmark results without uncertainty estimates; they do not measure the safety of a deployed vehicle interacting with reactive traffic.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM v1 trajectory planning Logged OpenScene/nuPlan scenes; four-second, eight-waypoint predictions in non-reactive simulation. Exact split identifier and sample counts are not stated. | 89.5 PDMS ↑ | AutoVLA and DriveLaW: 89.1 each; WAM-only: 88.1. Reader-calculated gains are 0.4 score points over the two strongest listed external planners and 1.4 over WAM-only. BrainWAM reports DAC 97.5 and EP 83.8, but does not lead NC or TTC. e07e08e10 |
| NAVSIM v2 trajectory planning The v2 non-reactive planning benchmark adds direction, traffic-light, lane and comfort criteria; exact split/counts are not specified. | 89.6 EPDMS ↑ | DriveDreamer-Policy: 88.7; DriveVLA-W0: 86.1. The 0.9-point advantage over DriveDreamer-Policy accompanies EP 88.2 versus 87.9 and EC 85.8 versus 79.4. BrainWAM's NC is lower than that baseline's; aggregate leadership is not universal safety-metric leadership. e07e09 |
| Branch and coordination ablations NAVSIM v1; Table 3 branch comparison and Table 4 component removals. | BrainWAM 89.5; VLA-only 86.1; WAM-only 88.1; Tri-MoT 87.8. PDMS ↑ | CAB alone: 88.7; CIF alone: 88.5; both: 89.5. Action-level coordination outperforms raw-token fusion in this setup. The source states identical backbones and comparable parameter counts for BrainWAM/Tri-MoT, but gives no repeated-run uncertainty. e10e11 |
| Video-denoising latency tradeoff NAVSIM v1/v2 scores; latency measured on one NVIDIA H20; three action sampling steps. | 0/1/2/3 video steps: 382/475/565/644 ms; PDMS 79.3/89.3/89.5/89.4; EPDMS 75.8/89.4/89.6/89.6. Latency ↓; PDMS ↑; EPDMS ↑ | One video step is 169 ms faster than three, with 0.1 lower PDMS and 0.2 lower EPDMS. The large zero-to-one-step recovery supports predictive-context use. Extra video steps yield small, nonmonotonic score changes; the table reports no uncertainty. e06e12 |
| Coordination capacity and Stage-3 updates NAVSIM v1. Tables 6–8 use ten joint video/action denoising steps; Table 9 is a separate update-strategy comparison. | Two CAB blocks: 89.3; Transformer CIF: 89.3; two CIF layers: 89.3; selective Stage-3 updates: 89.5. PDMS ↑ | One CAB: 88.9; MLP/gated CIF: 88.8/89.1; one CIF layer: 89.0; full-model fine-tuning: 88.8. These ablations favor a compact coordinator and freezing. Their architectural scores must not be directly equated with the asynchronous headline configuration. e15e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 2. Tri-MoT action tokens attend more to semantic tokens across most layers. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis indexes Transformer block layers; the vertical axis is the ratio of attention to VLM tokens versus attention to VGM tokens. This is one ratio curve, not two independently plotted attention traces. The dashed line at one marks equal allocation. Points above it indicate more attention to the semantic stream; points below it favor the video stream. Follow the early peaks, then the later values closer to parity. Read this diagnostic alongside Table 3, where adding the semantic stream through raw-token fusion fails to beat WAM-only planning. e02e10
What it supports. The plotted imbalance is consistent with the authors' explanation that clean, compact semantic features compete advantageously against still-denoising video features. It motivates preserving branch specialization before communication. The branch ablation supplies a separate performance observation: merely adding a modality does not guarantee a planning gain.
Where the evidence stops. Attention allocation is an observed association. The plot supplies no uncertainty bands or attention-balancing intervention, and the reviewed text does not fully specify how these ratios were aggregated. It cannot establish that attention imbalance alone causes the score gap.
Table 3. Action-space coordination improves on both single branches and raw-token fusion. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first two rows as the specialized starting points: VLA-only supplies semantic planning, and WAM-only supplies prediction-grounded planning. Tri-MoT then tests combining modalities in shared raw-token attention. BrainWAM changes where the two branches communicate. Follow PDMS across these rows, then compare DAC and EP to locate the largest visible gains over the single branches. Keep NC separate: Tri-MoT's NC is higher than BrainWAM's despite its lower aggregate score. Section 4.4 states that Tri-MoT and BrainWAM use identical backbones and comparable parameter counts. e03e04e10e15
What it supports. WAM-only scores 88.1, while raw-token Tri-MoT drops to 87.8. BrainWAM reaches 89.5: a reader-calculated gain of 1.4 points over WAM-only and 1.7 over Tri-MoT. This supports the usefulness of the coordination design under the reported configuration and is consistent with complementary branch information.
Where the evidence stops. The comparison does not isolate attention competition from every training or architectural change. No seed variation is reported. Separate ten-step appendix capacity ablations use another inference protocol and should not be folded into this table as equivalent runs.
Table 5. One video step recovers most planning performance; two produce the best reported PDMS. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row changes the number of video denoising steps while action generation uses three sampling steps. Read latency downward to see the extra cost, then compare PDMS and EPDMS in the same row. The caption specifies a single NVIDIA H20 GPU. Zero video steps are an ablation, not the optimized recommended planner: both scores drop sharply. One step restores most performance, two yield the best PDMS, and three tie the best EPDMS with greater latency. Appendix E explains that terminated video features are cached for the remaining action updates. e06e12e18
What it supports. One video step gives 89.3 PDMS and 89.4 EPDMS at 475 ms. Two give 89.5 and 89.6 at 565 ms. Three take 644 ms for 89.4 and 89.6. The large improvement over zero steps supports using predictive features, while small later differences motivate truncation and reuse.
Where the evidence stops. The table reports neither latency variance nor uncertainty on the small score differences. Its caption names hardware but does not fully specify timing boundaries. Appendix G explicitly says these runtimes still miss strict real-time vehicle deployment requirements.
7. Analysis & limitations
7.1 What the evidence leaves open
Keeping both branches and the video backbone increases compute and memory relative to a single branch. The authors explicitly state that 475–644 ms inference does not meet strict real-time vehicle deployment requirements. e18
The attention plot shows association, not a causal intervention. Tables report neither error bars nor multiple-seed variation. Selected qualitative successes in Figures 5–6 illustrate complementary failures but do not measure their prevalence. e02e08e09e10e12e17
The benchmark's four-second non-reactive setting does not establish long-horizon closed-loop interaction, deployment robustness, or physical vehicle safety. e07e18
7.2 Questions for discussion
- Would a parameter-matched, attention-balanced Tri-MoT close the gap, weakening the proposed modality-competition explanation?
- Does cached predictive context remain useful when road users react to the ego vehicle beyond the benchmark's short horizon?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the named Wan and Qwen backbones, driving observations/history/instructions, target future video and trajectories, the three-stage recipe and the correct NAVSIM evaluator. CAB uses eight heads of dimension 128 and bias-free projections; its two blocks total about 16.8M parameters. CIF totals about 49.3M. e03e04e06e07e13e14
Implementation gaps include exact data splits/counts, image/video preprocessing, future-video horizon, trajectory parameterization, numerical WAM action-loss weight, precise flow timestep grids, random seeds, checkpoint-selection rule and software versions. The supplied optimization and module details do not resolve these choices. e19
Proposed checks should hold pretrained checkpoints, data and noise fixed while testing coordination or video-feature reuse, and report per-scene differences with uncertainty. The illustrated edition specifies two such experiments; neither has been run. e10e12e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether balancing raw-token attention closes the planning gap
Reader-proposed, not run: compare original Tri-MoT, a Tri-MoT variant with attention normalized separately over VLM and VGM tokens before equal-weight combination, and BrainWAM. Hold backbone initialization, scenes, training budget and action noise fixed; use three action steps and two video steps for evaluation. Repeat across seeds and report paired per-scene PDMS changes, DAC/EP, attention-ratio distributions and uncertainty. If balanced Tri-MoT approaches BrainWAM, modality competition remains plausible but the necessity of action-space bridging weakens. If attention becomes balanced without a planning gain, the ratio diagnostic alone is an incomplete explanation. e02e06e10e12
Check 2: Measure the value and cost of cached predictive features
Reader-proposed, not run: freeze one trained BrainWAM checkpoint and evaluate zero, one, two and three video steps with identical scenes, action noise and three action steps. Keep the source's feature cache behavior, and add a control that substitutes cached video features from another scene while preserving tensor shape and computation. Measure paired PDMS/EPDMS differences, component scores and explicitly bounded single-H20 latency after warmup. A one-step recovery that disappears with mismatched features would support scene-specific predictive context. Similar scores after substitution would challenge that interpretation; substitution also introduces a distribution shift, so report it as a diagnostic control. e05e06e12e18
8.3 Reading coverage
Visual audit: All 13 supplied PDF pages were rendered and visually inspected after all five text chunks were read. Page 1 verifies the title, authors, affiliations and v2 marker; pages 2–5 cover the attention plot, architecture, training pipeline, equations and benchmark definitions; pages 6–7 contain Tables 1–5 and Figure 5. Pages 8–9 contain the qualitative continuation, conclusion and references. Pages 10–11 cover CAB/CIF implementation, ten-step ablations, freezing and hardware/optimizer/cache details; page 12 contains Figure 6; page 13 states deployment limitations. All six final original crops were separately viewed with complete axes, diagram inputs/outputs or table headers and rows. No separate supplement or external code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and Abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work (pp. 2–3)
- 3 Method, including 3.1 WAM Branch, 3.2 VLA Branch and 3.3 Joint Training with CAB and CIF (pp. 3–5)
- 4 Experiments, Sections 4.1–4.5 (pp. 5–8)
- 5 Conclusion (p. 8)
- References (pp. 8–9)
- Appendix A: Modality Imbalance in Tri-MoT (p. 10)
- Appendix B: Implementation and Analysis of CAB, B.1–B.2 (p. 10)
- Appendix C: Implementation and Analysis of CIF, C.1–C.3 (pp. 10–11)
- Appendix D: Freezing the Pretrained Branches Stabilizes Stage-3 Optimization (p. 11)
- Appendix E: Additional Implementation Details (p. 11)
- Appendix F: Additional Qualitative Comparisons (pp. 11–12)
- Appendix G: Limitations (p. 13)
Outside the original text pass
- Identity/version: PDF p. 1 displays arXiv:2608.12854v2 [cs.RO], 19 Aug 2026. Its exact title and all nine authors match the catalog after name-order normalization. The catalog URL is unversioned; v1 and a revision history were not supplied, so differences from v1 cannot be established.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting all 13 PDF pages, including every figure and table, and all six final crops. All five supplied text chunks were read individually and completely.
- Separate supplemental material availability has not been fully verified.
- No source code, model checkpoints or external linked resources were inspected; no experiments were reproduced. Reading is limited to the supplied PDF.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, arXiv margin and author footnotes
Exact title matches the catalog. Authors in order: Bing Zhan, Shuyao Shang, Shuo Lu, Yuan Xu, Zhao Wang, Yida Wang, Xueyang Zhang, Kun Zhan, Jiahao Gu. Margin identifies arXiv:2608.12854v2 [cs.RO], 19 Aug 2026. First four authors list NLPR, Institute of Automation, Chinese Academy of Sciences (CASIA); remaining five list Li Auto Inc.; all list Beijing, China.
Go to primary source ↓e02PDF p. 2, Section 1 and Figure 2; p. 10, Appendix A
Figure 2 plots VLM/VGM attention ratio against Transformer layer, with most points above the equal-attention line. The authors explain this through competition between stable semantic tokens and denoising video tokens.
Go to primary source ↓e03PDF p. 3, Figure 3 and Sections 3–3.1, Eq. (1)
WAM uses Wan2.2-TI2V-5B and an action expert, coupled by Dual-MoT shared self-attention with modality-specific feed-forward layers. Separate video/action timesteps support predictive context and asynchronous inference.
Go to primary source ↓e04PDF p. 4, Sections 3.1–3.2, Eqs. (2)–(8)
Linear clean-to-noise interpolation gives velocity targets ε−x₀ and squared-error objectives. Qwen3-VL-4B encodes multi-view images/instructions and ego history, coupled with semantic action tokens. WAM's action loss has symbolic weight λᵃ_pred.
Go to primary source ↓e05PDF pp. 4–5, Section 3.3, Eqs. (9)–(13) and Inference
Both frozen branches receive the same action noise/timestep. CAB performs bidirectional cross-attention with zero-initialized gated residuals. CIF concatenates, transforms and averages action streams; its decoder predicts the fused action velocity supervised in Stage 3.
Go to primary source ↓e06PDF p. 4, Figure 4; p. 5, Section 4.2; p. 11, Appendix E
Three independent/joint stages use 100K steps each, eight H20 GPUs, batch six per GPU, AdamW, peak 5×10⁻⁵ learning rate, cosine decay, 200 warmup steps, bf16 and checkpoints every 3K steps. Appendix E adds weight decay 0.01, ZeRO-2, three-step action sampling and cached video features.
Go to primary source ↓e07PDF p. 5, Section 4.1 and PDMS/EPDMS definitions
NAVSIM uses OpenScene reprocessed from nuPlan logs, four-second predictions at 2 Hz and non-reactive simulation. PDMS multiplies NC/DAC penalties by weighted EP/TTC/comfort; EPDMS adds DDC/TLC penalties and HC/LK/EC terms. EP and TTC weights are five; comfort-related and lane terms have weight two.
Go to primary source ↓e08PDF p. 6, Table 1, BrainWAM, AutoVLA, DriveLaW and Human rows
BrainWAM: NC 98.1, DAC 97.5, TTC 94.9, C 100.0, EP 83.8, PDMS 89.5. AutoVLA/DriveLaW each score 89.1 PDMS; Human scores 94.8. Image/lidar columns distinguish sensor settings; no uncertainty is reported.
Go to primary source ↓e09PDF p. 6, Table 2, BrainWAM, DriveDreamer-Policy and DriveVLA-W0 rows
BrainWAM: NC 98.1, DAC 97.5, DDC 99.6, TLC 99.9, EP 88.2, TTC 97.4, LK 97.6, HC 98.4, EC 85.8, EPDMS 89.6. DriveDreamer-Policy: EPDMS 88.7, EP 87.9, EC 79.4, NC 98.4, DAC 97.1. DriveVLA-W0 scores 86.1 EPDMS.
Go to primary source ↓e10PDF p. 5, Section 4.4, branch/token-fusion discussion; p. 6, Table 3
VLA-only/WAM-only/Tri-MoT/BrainWAM PDMS are 86.1/88.1/87.8/89.5. BrainWAM and Tri-MoT use identical backbones and comparable parameter counts according to Section 4.4. BrainWAM improves DAC/EP over both single branches; Tri-MoT has higher NC than BrainWAM.
Go to primary source ↓e11PDF p. 6, Table 4; p. 5, Section 4.4, Effectiveness of CAB and CIF
CAB alone scores 88.7 PDMS, CIF alone 88.5, both 89.5. Joint use improves DAC and EP; NC stays at 98.1 across all three rows.
Go to primary source ↓e12PDF p. 7, Table 5 including caption; p. 6, asynchronous video-denoising discussion
With 0/1/2/3 video denoising steps, single-H20 latency is 382/475/565/644 ms, PDMS 79.3/89.3/89.5/89.4 and EPDMS 75.8/89.4/89.6/89.6. The video branch caches features for subsequent action denoising.
Go to primary source ↓e13PDF p. 10, Appendix B.1–B.2
Two CAB blocks at layers 9 and 18 connect eight-token, 1024-dimensional streams. Cross-attention has eight 128-dimensional heads, separate normalizations and bias-free projections. Gates are zero-initialized 1024-dimensional vectors; two bridges total approximately 16.8M parameters.
Go to primary source ↓e14PDF pp. 10–11, Appendix C.1
CIF separately projects both eight-token streams into 1024 dimensions, adds learned source embeddings, and uses two Transformer layers with eight heads and timestep-conditioned AdaLN. It contains approximately 49.3M parameters.
Go to primary source ↓e15PDF p. 10, Appendix B and Table 6; p. 11, Appendix C.2–C.3 and Tables 7–8
Tables 6–8 use ten-step joint video/action denoising. One/two/three/five/28 CAB blocks score 88.9/89.3/89.2/89.3/89.3. MLP/gated/Transformer CIF score 88.8/89.1/89.3; one/two/three CIF layers score 89.0/89.3/89.3.
Go to primary source ↓e16PDF p. 11, Appendix D and Table 9
Selective CAB/CIF/decoder updating scores 89.5 PDMS versus full-model fine-tuning at 88.8. VLA and WAM reach their single-branch scores after 54K and 81K steps respectively; the authors attribute freezing's benefit to more stable coordination inputs.
Go to primary source ↓e17PDF pp. 7–8, Section 4.5 and Figure 5; pp. 11–12, Appendix F and Figure 6
Selected examples contrast navigation/red-light semantics with pedestrian/curve dynamics. Figure 6 includes cases favoring either branch and one final example where both branches fail but BrainWAM produces a reasonable trajectory.
Go to primary source ↓e18PDF p. 13, Appendix G
Both branches and a generative video backbone remain active at inference, increasing compute and memory. The authors explicitly say 475–644 ms is insufficient for strict real-time in-vehicle deployment and propose compression/distillation, less redundant computation and feature reuse.
Go to primary source ↓e19PDF pp. 3–5, Sections 3–4.2; pp. 10–11, Appendices B–E
These sections specify backbones, objectives, module sizes and optimizer settings but do not state exact data split IDs/counts, preprocessing, video horizon, trajectory coordinates, numerical action-loss weight, timestep grids, seeds, checkpoint-selection criterion or software version numbers.
Go to primary source ↓8.5 Primary sources
BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving ↗
PDF · 8,119 extracted words
Source fingerprint
345f26d920810a66610774bdb300f33fd3a3bc5dfc5f258ca9a7683b7bb8328d