Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation
1. Paper overview
In one sentence: Metis uses action-conditioned video prediction to train a trajectory policy, then removes future-video synthesis from inference to reduce latency. e-architecturee-maske-gradientse-navteste-citye-latency
| At a glance | What to know |
|---|---|
| Research problem | Author claim Driving and urban navigation need timely trajectories informed by environmental dynamics. The authors argue that explicit future synthesis adds inference cost and that tightly mixing video and action representations can destabilize planning. Metis asks whether future prediction can teach a policy during training without remaining on its inference path. e-problem |
| Core mechanism | Source description A Mixture-of-Transformers separates video generation and trajectory prediction into specialized experts with shared attention interactions and expert-specific projections, feed-forward layers and heads. e-architecture |
| A key reported result | NAVSIM-v2 navtest trajectory planning: 89.5; best-of-six variant 90.3. EPDMS ↑. 12,146 scenarios; one front camera; ordinary Metis row separated from best-of-six selection. DriveFine: 87.1; Epona: 85.1; human-agent score: 90.3. Ordinary Metis leads the listed learned planners. Matching the human aggregate requires best-of-six; sensor, reinforcement-learning and training-data protocols vary across rows. e-datae-navtest |
| Reading caution | Source description The authors acknowledge unvalidated extreme cases, expensive video-expert training and a VAE downsampling tradeoff. Monocular long-horizon planning can deviate; generated distant vehicles can lose detail. Qualitative panels do not quantify these failure frequencies. e-limitse-qualitative |
Core contributions
- Source description
A Mixture-of-Transformers separates video generation and trajectory prediction into specialized experts with shared attention interactions and expert-specific projections, feed-forward layers and heads. e-architecture
- Author claim
An asymmetric visibility rule lets future video tokens read actions, while actions cannot read future video. The authors attribute better planning to video-loss gradients entering the action expert during joint training. e-maske-gradients
Figure 2. Two expert branches share controlled attention while retaining their own transformations. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom: the current frame and future training frames enter the VAE encoder, while the action chunk enters a separate encoder. Noise is added on the future/action paths. At left, the text encoder and ego-state encoder provide conditioning through the connection marked C and cross-attention. Inside the dashed boundary, each expert has its own QKV and feed-forward blocks; the horizontal attention block regulates exchange. Finally compare the right-hand insets and their legend: future-frame processing appears during training, whereas the inference inset retains current-frame context and the action path. This visual does not say that the entire video backbone disappears. e-architecturee-inferencee-confige-backbone
What it supports. The architectural distinction is between specialized representations and controlled communication. Metis still extracts visual context at inference, but future-video generation is removed from the action-producing path. The method text describes current-context extraction in one backbone pass, followed by action denoising; a single context pass does not mean a single-step policy.
Where the evidence stops. The overview supports two interacting experts, not two independently executing controllers. Implementation text names a 5B video expert, but Table 6 names 14B variants; the exact backbone configuration needs clarification before reproduction.
2. Motivation
2.1 The problem and the proposed response
Driving and urban navigation need timely trajectories informed by environmental dynamics. The authors argue that explicit future synthesis adds inference cost and that tightly mixing video and action representations can destabilize planning. Metis asks whether future prediction can teach a policy during training without remaining on its inference path. e-problem
2.2 What this reading follows
Metis makes an unusual division of labor: video prediction helps train the policy, but the deployed planner does not have to render an imagined future. Its two transformer experts exchange information through an asymmetric attention rule, preserving a route for video supervision to shape actions. Reading the paper therefore requires separating forward information flow, backward training gradients and runtime computation. The six visuals below follow that chain into driving scores, urban trajectory errors and latency measurements. They also expose important evidence boundaries: a contradictory mask drawing, differently selected result rows and a speedup comparison that changes the denoising budget. e-architecturee-maske-gradientse-navteste-citye-latency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded WAM/Dual-system/Other mechanisms/Outside quadrants classification is supported with a qualification: two specialized transformer experts interact within one trained MoT framework, rather than two independent runtime controllers. Architecture evidence, not joint training alone, supports the dual-expert reading. Inference neither jointly samples future video and actions nor infers actions from generated frames; world prediction supplies training supervision. Driving, navigation and efficient inference are directly evaluated. e-architecturee-inferencee-maske-gradientse-latency
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a one-way forward connection can still teach the action expert
The main puzzle is how video prediction can improve a policy that never reads predicted video. Separate forward information flow from backward optimization. During training, the video expert predicts future visual latents conditioned on action tokens, so its prediction error can differentiate through the action-side representations it uses. Appendix A.1 explicitly identifies this backward route. Section 3.3 blocks the opposite forward dependency: actions must not need future-video tokens. Reader interpretation: the arrangement lets video supervision shape useful action representations without creating a future-generation requirement at deployment. Table 11 is consistent with this explanation, showing higher scores with co-training, but it does not uniquely identify gradient transfer as the cause. A gradient-stop control would distinguish that explanation from benefits of shared training or other implementation differences. e-maske-losse-gradientse-cotrain
Figure 3. The stated asymmetric rule and the printed matrix disagree on current-image access. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a row as the querying token and a column as the token it can access, consistent with the caption's video-to-action visibility statement. Blue future-video rows can access action columns; action rows have empty future-video columns. Orange marks computation retained for training and inference, while blue denotes training only. Crucially, the action rows also show empty cells in the current-observation column z_t. This conflicts with Section 3.3 and the caption, which explicitly give actions access to the current observation. Equation (4) and Figure 2 likewise condition action prediction on current visual context. The faithful crop preserves that unresolved graphical discrepancy. e-maske-inferencee-architecturee-gradients
What it supports. The source consistently states that actions must not depend on generated future video. Appendix A.1 explains how video supervision can nevertheless help: gradients from generation backpropagate into the action expert during training. That backward learning signal is different from permitting actions to read future-video tokens in the forward pass.
Where the evidence stops. The intended current-context dependency is supported by the equations and prose, but the exact mask cannot be reconstructed unambiguously from this matrix alone. No stop-gradient symbol appears here; gradient transfer is specified in Appendix A.1.
5.2 What remains after future-video synthesis is removed
Action-only inference still begins with perception and conditioning. The current image is encoded through the video-model components, language and ego state provide context, and the action expert denoises a low-dimensional trajectory chunk. Equation (4) expresses this as actions conditioned on the backbone representation of the current observation and instruction. The paper's single-forward-pass statement applies to extracting that context; the action sampler still has a configurable number of steps. For driving, the output consists of eight waypoints across four seconds, whereas CityWalker uses five planar waypoints. Physical Go2 deployment adds an adapted PD controller that produces velocity commands. Thus the learned prediction, action sampling and controller are separate parts of the runtime account. Neither the architecture drawing nor the qualitative deployment panels specify a complete feedback schedule or end-to-end control latency. e-inferencee-architecturee-confige-reale-latency
5.3 Follow each result back to its evaluation contract
The driving leaderboards, CityWalker errors and robot panels establish different things. NAVSIM-v2 navtest summarizes rule-based planning over a broad scenario set, while navhard adds staged synthetic evaluation and different aggregation details. Their EPDMS values are not interchangeable success rates. The ordinary navtest Metis row should be compared before considering its best-of-six variant; the latter changes the selection protocol. CityWalker instead measures geometric and angular agreement with reference trajectories, with scenario-balanced Mean and sample-level All summaries. The Go2 sequences show physical obstacle-avoidance examples but supply no trial denominator. Reader interpretation: taken together these results motivate the method across driving and navigation, while leaving deployment reliability and extreme-case behavior unresolved. The strongest mechanism evidence comes from controlled ablation rows, not from turning qualitative plausibility into a numerical safety claim. e-datae-navteste-navharde-citye-metrice-reale-ablationse-limits
5.4 Training and inference
During training
Both experts use flow matching with equally weighted action and video losses. Appendix A.1 explicitly describes generation gradients backpropagating into the action expert. The text describes joint optimization, without a complete frozen-module schedule. e-losse-gradientse-config
NAVSIM-v2 uses 1,192 navtrain scenarios, 640×768 images and 60 epochs; CityWalker uses six hours for fine-tuning, 384×384 images and 30 epochs. Batch size is 64. Appendix F specifies AdamW, learning rate 10⁻⁴, weight decay 0.01, cosine annealing, mixed precision, gradient clipping 1.0, and eight H200 GPUs with 140 GB each. e-datae-config
During inference
Obtain current-context backbone latents in one forward pass, then denoise the action chunk. Future-video synthesis is bypassed; current-image processing remains. The default is ten denoising steps and CFG 1.0; the fast latency configuration uses two steps. e-inferencee-confige-latency
For physical deployment, an adapted PD controller converts planning output into velocity commands for a Unitree Go2. This controller is distinct from learned trajectory prediction; deployment is illustrated qualitatively rather than summarized by success rates. e-real
5.5 Implementation flow
- Encode context
A pretrained video VAE encodes observations; a T5-based encoder encodes language. Ego-state features join the conditioning. Current visual tokens, noisy future-video tokens and noisy action tokens enter expert-specific projections after language cross-attention. e-architecturee-config
- Keep specialized computation
The text specifies Wan2.2-5B for the video generation expert and a smaller diffusion-transformer action expert of matching depth, hidden dimension 1024 and approximately 1B parameters. The stated combined size is 6B; Table 6 labels a different backbone. e-architecturee-confige-backbone
- Restrict forward visibility
According to Section 3.3, action tokens access current visual context but no future video; future video accesses current observations and future actions. Figure 3 agrees about future-video/action asymmetry but leaves action-to-current cells blank, contrary to its caption and prose. The intended rule is clear; the exact depicted mask is inconsistent. e-maske-inference
6. Experiments & results
Metis trains separate video and action transformer experts together, then predicts trajectories without synthesizing future video. Its asymmetric attention supplies a training-time path from video supervision into the action expert. Driving and navigation results support this design, but latency comparisons and several implementation details require careful qualification.
6.1 Read the original evidence
Table 2. Separate the 89.5 ordinary score from the 90.3 best-of-six result. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the rightmost EPDMS column, then inspect the two Metis rows separately. The unmarked row reports 89.5; the double-dagger row reports 90.3 and uses best-of-six selection. The caption retained above the table also marks reinforcement learning and full-navtrain training, while the Sensors column shows different observation budgets. Then look across individual submetrics: the best aggregate does not imply leading every component. In particular, ordinary Metis has DAC 97.2 and LK 97.8. Appendix F defines EPDMS through penalty terms and a weighted average of other submetrics, so its aggregate cannot be read as a direct collision-free success percentage. e-navteste-datae-metrice-ablations
What it supports. Ordinary Metis scores above the listed learned planners, including DriveFine at 87.1 and Epona at 85.1. Its best-of-six variant reaches the human-agent aggregate of 90.3. This is a benchmark-score comparison under the table's specified protocols, not evidence that ordinary inference achieves human driving performance.
Where the evidence stops. Baseline training and sensor regimes differ. The paper reports no confidence intervals here, and best-of-six selection adds an evaluation condition. Table 4 also mixes the selected row's DAC/LK values with the ordinary aggregate.
Table 3. Read scenario balance before comparing navigation errors. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each method has an L2 row in meters and an MAOE row in degrees; lower is better for both. The source caption defines Mean as the average across the six scenario categories, whereas All averages over all samples. The percentages show why these answer different questions: Other contributes 55% of samples, while individual difficult categories are smaller. Read the upper pretrained ABot-N0 block separately from the fine-tuned methods; its asterisk denotes large-scale navigation pretraining, and dashes mean L2 was not reported. Metis improves several challenging scenario errors, but the upper block prevents a blanket claim of superiority on aggregate orientation. e-citye-datae-real
What it supports. Metis reports 0.64 m L2 and 9.8° MAOE over all samples, versus 1.07 m and 11.5° for the CityWalker baseline. ABot-N0 remains better on All MAOE at 7.6°. The table supports strong trajectory imitation among fine-tuned comparisons, with a narrower claim against the pretrained navigation model.
Where the evidence stops. L2 and orientation alignment are offline imitation metrics. They neither count successful obstacle avoidance nor measure robot interventions. Missing ABot-N0 L2 values must remain missing; they cannot be inferred from its orientation results.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM-v2 navtest trajectory planning 12,146 scenarios; one front camera; ordinary Metis row separated from best-of-six selection. | 89.5; best-of-six variant 90.3. EPDMS ↑ | DriveFine: 87.1; Epona: 85.1; human-agent score: 90.3. Ordinary Metis leads the listed learned planners. Matching the human aggregate requires best-of-six; sensor, reinforcement-learning and training-data protocols vary across rows. e-datae-navtest |
| NAVSIM-v2 navhard planning 244 real Stage-1 scenarios and 4,164 synthetic Stage-2 counterparts; two-stage evaluation. | 32.2 overall; stage scores 75.8 and 41.7. EPDMS ↑ | DiffusionDrive: 27.5; SGDrive: 25.5. Symbolic-input PDM-Closed: 51.3. Best listed learned-planner aggregate, not best among all input regimes. Navhard aggregation differs from navtest. e-datae-navharde-metric |
| CityWalker trajectory imitation Nine-hour test split; six-hour fine-tuning split. Reported All column averages all samples. | All: 0.64 m and 9.8°; scenario Mean: 0.71 m and 11.8°. L2 (m) ↓; MAOE (degrees) ↓ | CityWalker baseline All: 1.07 m and 11.5°. Pretrained ABot-N0: 7.6° All MAOE; L2 unreported. Strong fine-tuned comparison, but ABot-N0 has lower aggregate orientation error. This measures imitation alignment, not collision-free execution. e-citye-data |
| Attention-mask and co-training ablations Mask comparison at fixed 320×384 resolution; separate co-training comparison in Table 11. | Mask navtest/navhard: 88.8/31.6; co-training: 89.5. EPDMS ↑ | Joint mask: 87.4/28.0; isolated: 88.3/29.4. Without co-training: 87.9. Supports asymmetric interaction and video supervision, while not isolating the asserted gradient mechanism by itself. e-ablationse-cotrain |
| Action-only inference efficiency Single RTX 4090; Table 7 compares two action-only steps with ten video-generation steps. | 0.17 s and 89.2. Latency (seconds) ↓; navtest EPDMS ↑ | Metis with video: 1.38 s and 89.5. The roughly eightfold speedup changes two factors. Table 12 separately reports 147 ms for two steps and 480 ms for ten on RTX 4090; its two-step timing differs from Table 7. e-latencye-sweep |
| NAVSIM-v1 navtest planning Table 9 distinguishes ordinary inference from best-of-six; several baselines use full navtrain. | 89.1 ordinary; 89.7 best-of-six. PDMS ↑ | UniWorldVLA, with full navtrain: 89.4. The appendix prose's 89.7 refers to selected sampling, not the ordinary row; PDMS must remain separate from v2 EPDMS. e-v1e-metric |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Tables 4–5. Use matched-resolution rows to assess masking, then examine the sampling budget separately. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. On the left, compare the first three rows, all at 320×384, before looking at the higher-resolution row. Joint attention couples future video and actions; isolated attention makes those future token groups mutually invisible; Ours uses the asymmetric rule. This is the appropriate block for comparing mask designs without changing image resolution. On the right, inspect the denoising-step sweep independently: navtest and overall navhard scores rise as steps increase, but the Stage-1 column is not monotonic. These are distinct experiments, so a mask improvement should not be explained by borrowing gains from higher resolution or a larger sampling budget. e-ablationse-mask-detaile-navteste-gradients
What it supports. At matched resolution, asymmetric attention reaches navhard EPDMS 31.6 versus 28.0 for joint and 29.4 for isolated attention; navtest also improves. Separately, two action-denoising steps retain navtest EPDMS 89.2 versus 89.5 at ten, while navhard changes from 31.2 to 32.2. Harder scenarios retain more sensitivity to the sampling budget.
Where the evidence stops. Table 4's high-resolution DAC/LK pair is 97.5/98.0, matching Table 2's selected-sampling row despite ordinary EPDMS 89.5. Table 8 also disagrees on isolated-mask DAC. These discrepancies do not resolve the causal gradient-transfer question.
Table 12. Latency and planning quality change together as action sampling becomes shorter. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down one hardware column to compare step counts without mixing devices, then move left to the associated planning scores. On RTX 4090, the two-step row reports 147 ms, navtest EPDMS 89.2 and navhard EPDMS 31.2. Ten steps take 480 ms with scores 89.5 and 32.2. The H200 column provides a separate hardware measurement, not an additional training configuration. Keep PDMS and EPDMS distinct: Appendix F defines different NAVSIM-v1 and v2 scoring rules. Finally cross-check Table 7 on page 9, which uses a different two-step timing and compares action-only inference with a ten-step video-generation configuration. e-sweepe-latencye-metric
What it supports. The sweep makes the accuracy/latency tradeoff concrete. Much of the navtest score is retained with two steps, while the navhard aggregate benefits more from ten. Table 12 supports choosing a sampling budget; it does not by itself measure the speedup caused specifically by removing future-video generation.
Where the evidence stops. Table 7 reports 170 ms rather than 147 ms for two-step action-only inference. Its roughly eightfold speedup also changes denoising steps. Timing boundaries, repetitions and uncertainty are insufficiently specified to reconcile these values.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge unvalidated extreme cases, expensive video-expert training and a VAE downsampling tradeoff. Monocular long-horizon planning can deviate; generated distant vehicles can lose detail. Qualitative panels do not quantify these failure frequencies. e-limitse-qualitative
Reported tables provide no seed variation or confidence intervals. Figure 7 shows indoor scenes despite an outdoor caption; Appendix B's time-of-day description does not cleanly match the visible deployment panels. Demonstrations lack trial counts and intervention statistics. e-navteste-ablationse-real
Source inconsistencies remain unresolved: Figure 3 omits the stated action/current link; Table 6 says Wan2.2-14B where implementation text says 5B; Table 4's high-resolution DAC/LK values resemble the best-of-six row while its EPDMS matches ordinary inference. Table 8 also gives a different isolated-mask DAC value. e-maske-backbonee-ablationse-navteste-mask-detail
7.2 Questions for discussion
- Does stopping only the video-loss gradient into the action expert remove the co-training benefit? [e-gradients, e-cotrain]
- How much latency is saved by bypassing video when action denoising steps are held constant? [e-latency, e-sweep]
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the specified pretrained components, synchronized front images/actions, ego-state and instruction construction, the exact NAVSIM splits and scoring, and CityWalker preprocessing. The source omits a complete freeze schedule, action normalization, instruction-generation procedure, sampling-solver configuration, software versions, random seeds and best-of-six selection implementation. e-architecturee-losse-confige-datae-navtest
Proposed checks: compare video-to-action gradients enabled versus stopped under matched training; separately compare video enabled versus bypassed at identical denoising-step counts and hardware. Measure planning scores and latency with uncertainty before attributing gains to either mechanism. e-gradientse-cotraine-latencye-sweep
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Stop only the video-loss gradient into actions
Reader-proposed check, not performed: at the 320×384 mask-ablation setting, train otherwise identical asymmetric models with the video-loss gradient into action-side representations enabled or stopped. Keep the video objective, action objective, current-image access, initialization, data order and training budget fixed. Add an action-only training control corresponding to Table 11. First verify that changing future-video noise cannot affect the action output under the intended mask. Then compare navtest/navhard EPDMS across multiple seeds, particularly the hard split. If the gradient-stopped model retains the full gain over action-only training, the claim that transfer through this gradient path explains the improvement would be weakened. Resolve Figure 3's missing current-context cells before treating either implementation as a faithful reproduction. e-maske-gradientse-ablationse-cotraine-mask-detail
Check 2: Factor video bypass apart from denoising-step reduction
Reader-proposed check, not performed: on the same RTX 4090 and checkpoint, evaluate a factorial comparison of future-video synthesis enabled/bypassed and two/ten action-denoising steps. Fix inputs, precision, batch size, conditioning, action noise and selection protocol; explicitly record any separate video-step count. Time both model computation and the full input-to-trajectory pipeline after warmup, reporting median and tail latency over repeated scenes. Score every configuration on the same navtest and navhard splits. A substantial bypass benefit at matched action steps would support the architectural efficiency claim. If most of the reported reduction appears only when action steps also fall, the headline eightfold comparison would primarily reflect a combined configuration change. This also creates a documented basis for investigating 170 versus 147 ms. e-latencye-sweepe-confige-data
8.3 Reading coverage
Visual audit: The title/author page, Figures 1–10, Tables 1–12, method equations, training configuration, metric definitions and limitation pages were visually inspected. All six final original crops were inspected, including the mask legend and table headers. Table 2 retains its essential protocol notes; Table 3's aggregation and pretraining definitions are explained in its reading guide. The mask discrepancy, backbone-label conflict, inconsistent submetrics, deployment-caption mismatch and differing latency reports are disclosed without changing the source graphics. Complete text reading also covered the conclusion and references. Separate supplements, deployment videos and code remain outside this review; no reproduction was performed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related work (pp. 2–3)
- 3.1 Problem formulation and notation (pp. 3–4)
- 3.2 Network architecture (p. 4)
- 3.3 Structured attention mechanisms (pp. 4–5)
- 3.4 Training Objective (p. 5)
- 4 Experiments; 4.1 Main results; 4.2 Ablation studies; 4.3 Qualitative Analysis (pp. 5–9)
- 5 Conclusion (p. 10)
- References (pp. 11–16)
- Appendix A Discussion, including A.1–A.2 (pp. 17–18)
- Appendix B Real world experiments (p. 18; figures pp. 19–20)
- Appendix C Additional experiments results (p. 18; Table 9 p. 21)
- Appendix D Ablation studies, including D.1–D.2 (pp. 18–20)
- Appendix E Qualitative results, including E.1 and E.1.1 (p. 19; figures pp. 22–23)
- Appendix F Experiment details and evaluation metric (pp. 20–22)
- Appendix G Limitation and H Broad Impact (pp. 22–23)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting Figures 1–10, Tables 1–12 and all pages supporting retained scientific claims; no separate supplement or deployment video was supplied or inspected.
- Only the supplied arXiv:2606.15869v1, dated 14 June 2026, was reviewed. Its exact title and all 13 authors match the catalog; no different revision or edition was supplied or compared.
- Code and external links were not inspected; no experiment, installation or robot deployment was reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
Exact title matches the supplied primary metadata. The page lists Jingyu Li, Zhe Liu, Dongnan Hu, Junjie Wu, Zipei Ma, Wenxiao Wu, Chao Han, Zhihui Hao, Zhikang Liu, Kun Zhan, Jiankang Deng, Xiatian Zhu and Li Zhang; eight affiliations; arXiv:2606.15869v1, 14 Jun 2026.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1; Figure 1
The authors motivate action/video decoupling through inference latency and representational interference, contrasting VLA-based and video-generation-based WAMs.
Go to primary source ↓e-architecturePDF p. 4, Section 3.2 and Figure 2
VGE and AE use separate QKV projections, feed-forward layers and output heads with structured shared-latent attention. Wan2.2-5B, a video VAE, T5 text encoder and ego-state encoder are described; Figure 2 distinguishes training and inference.
Go to primary source ↓e-inferencePDF pp. 3–4, Section 3.1, Eqs. (1)–(4); Figure 2(b)
The paper contrasts joint future/action generation and inverse dynamics with actions conditioned on current-observation backbone latents. Equation (4) and its following paragraph specify a single backbone context pass without future generation.
Go to primary source ↓e-maskPDF p. 5, Section 3.3, Figure 3 matrix and caption; Eqs. (5)–(6)
Text and caption say actions attend current observations but not future video, while video attends actions. The matrix visibly shows empty action-row/current-observation-column cells, contradicting that portion of the stated rule; its future-video/action asymmetry agrees.
Go to primary source ↓e-lossPDF p. 5, Section 3.4, Eqs. (5)–(6) and total objective
Both branches predict flow velocities from noise-interpolated tokens; video is conditioned on predicted actions. The action target is data minus Gaussian noise. The two losses have equal weight λ=1.
Go to primary source ↓e-gradientsPDF p. 17, Appendix A.1 Motivation
The authors explicitly describe gradients from the video generation expert backpropagating into the action expert during joint training; inference bypasses generation.
Go to primary source ↓e-configPDF pp. 5–6, implementation details; p. 20, Appendix F, first two paragraphs
States 5B VGE plus approximately 1B AE, hidden dimension 1024, horizons eight/five, front camera, video/action temporal ratio 1:1, resolutions 640×768/384×384, 60/30 epochs, batch 64, ten inference steps, CFG 1.0, eight H200 GPUs (140 GB each), AdamW 10⁻⁴, weight decay 0.01, cosine schedule, mixed precision and clipping 1.0.
Go to primary source ↓e-dataPDF p. 6, NAVSIM-v2 and CityWalker dataset paragraphs; p. 20, Appendix F
NAVSIM training uses 1,192 scenarios; navhard has 244 real plus 4,164 synthetic cases; navtest has 12,146 cases. CityWalker uses six of fifteen teleoperation hours for fine-tuning and nine for testing, with L2 and MAOE evaluation.
Go to primary source ↓e-navhardPDF p. 6, Table 1, Metis/DiffusionDrive/SGDrive/PDM-Closed rows and caption
Metis has stage scores 75.8/41.7 and overall EPDMS 32.2; DiffusionDrive 27.5 and SGDrive 25.5. PDM-Closed has 51.3 using ground-truth symbolic inputs. Caption distinguishes copied and reproduced baseline values.
Go to primary source ↓e-navtestPDF p. 7, Table 2, caption, Sensors and EPDMS columns, Metis/DriveFine/Epona/Human rows
Ordinary Metis EPDMS is 89.5; best-of-six is 90.3, equal to the human aggregate. DriveFine is 87.1 and Epona 85.1. Caption marks RL, full-navtrain and best-of-six differences. Ordinary Metis DAC/LK are 97.2/97.8; selected sampling gives 97.5/98.0.
Go to primary source ↓e-cityPDF p. 7, Table 3, caption and Mean/All/scenario columns
Metis All L2/MAOE are 0.64 m/9.8°; Mean is 0.71 m/11.8°. CityWalker All is 1.07 m/11.5°. ABot-N0 has All MAOE 7.6° and Mean 11.2°, with L2 omitted. Caption distinguishes scenario mean from all-sample aggregation and flags large-scale pretraining.
Go to primary source ↓e-ablationsPDF p. 8, Tables 4–5 and Section 4.2
At 320×384, joint/isolated/asymmetric masks score navtest 87.4/88.3/88.8 and navhard 28.0/29.4/31.6. At 640×768, asymmetric scores 89.5/32.2 with DAC/LK 97.5/98.0. One/two/five/ten steps score navtest 87.2/89.2/89.4/89.5 and navhard 30.4/31.2/31.4/32.2; individual S1 scores are not monotonic.
Go to primary source ↓e-mask-detailPDF p. 18, Appendix A.2 and Table 8; p. 8, Table 4
Table 8 agrees with Table 4 on EPDMS aggregates but lists isolated navtest EPDMS-block DAC as 96.9, whereas Table 4 lists 96.5. Appendix prose interprets the mask comparisons in terms of noise and gradient transfer; it does not directly measure either mechanism.
Go to primary source ↓e-cotrainPDF p. 19, Appendix D.2; p. 20, Tables 10–11
Without/with video co-training yields PDMS 87.4/89.1 and EPDMS 87.9/89.5. AE capacity table reports approximately 0.21/0.45/1.04B parameters and navhard EPDMS 31.2/31.4/32.2; S1 does not improve monotonically.
Go to primary source ↓e-backbonePDF p. 9, Table 6 and accompanying paragraph; pp. 4–5 and 20, implementation descriptions
Table 6 labels its larger VGE Wan2.2-14B, with approximately 0.21B or 1.04B AE, while the main and appendix implementation text names Wan2.2-5B and a 6B combined model. The source does not resolve this discrepancy.
Go to primary source ↓e-latencyPDF p. 9, Table 7 and 'Inference latency of different methods' paragraph
On one RTX 4090, Metis with video reports 1.38 s/89.5 EPDMS; action-only reports 0.17 s/89.2. The text specifies ten steps for video generation versus two for action-only inference and claims up to eightfold speedup.
Go to primary source ↓e-sweepPDF p. 21, Table 12, all rows and hardware columns
For one/two/five/ten steps, RTX 4090 latency is 110/147/280/480 ms and H200 latency is 100/140/240/430 ms. Navtest EPDMS is 87.2/89.2/89.4/89.5 and navhard is 30.4/31.2/31.4/32.2. The 147 ms two-step value differs from Table 7's 0.17 s.
Go to primary source ↓e-v1PDF p. 18, Appendix C.1; p. 21, Table 9, caption and Metis/UniWorldVLA rows
Appendix prose highlights 89.7 PDMS. Table 9 assigns ordinary Metis 89.1 and best-of-six 89.7, while full-navtrain UniWorldVLA scores 89.4.
Go to primary source ↓e-metricPDF pp. 21–22, Appendix F, Eqs. (7)–(8) and adjacent definitions
EPDMS multiplies penalties for NC/DAC/DDC/TLC by a weighted average of TTC/EP/LK/HC/EC. Navhard adds two-stage aggregation, reactive traffic and human-failure exceptions. V1 PDMS uses NC/DAC penalties and EP/TTC/comfort averaging.
Go to primary source ↓e-realPDF p. 6, real-world experiments; p. 18, Appendix B; pp. 19–20, Figures 6–7 and captions
Metis and baselines are deployed on Unitree Go2 using an adapted PD velocity controller. Four qualitative sequences illustrate obstacle avoidance without task-specific training. Appendix B describes indoor daytime/outdoor nighttime; Figure 6 appears outdoors in daylight and Figure 7 visibly indoors despite an outdoor caption. No quantitative trial denominator or intervention rate is reported.
Go to primary source ↓e-qualitativePDF p. 19, Appendix E.1/E.1.1; p. 22, Figure 8; p. 23, Figures 9–10
Authors describe distant-vehicle detail loss in complex generated scenes and monocular long-horizon planning deviations. Figure 8 pairs generation with ground truth; Figure 10 depicts a roundabout planning failure.
Go to primary source ↓e-limitsPDF p. 22, Appendix G; pp. 22–23, Appendix H
Authors acknowledge unvalidated extreme cases, dependence on expensive pretrained-video training, VAE downsampling tradeoffs and possible unpredictable OOD behavior; broader safety and social benefits are discussed as impacts rather than measured outcomes.
Go to primary source ↓8.5 Primary sources
Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation ↗
PDF · 10,747 extracted words
Source fingerprint
34870523248f06c752b4639be40d1397ab63ec04f8487c6569315a6ce1508c6c