PerceptDrive: Perception Prior World-Action Modeling with Adaptive Expert Routing for End-to-End Autonomous Driving
1. Paper overview
In one sentence: PerceptDrive preserves distinct frozen perception priors and routes their influence before future-conditioned trajectory generation, trading evaluator-specific training for inference without candidate selection. e-probleme-interfacee-supervisione-protocole-limits
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors identify a prior-to-plan transfer problem: imitation may discard source-specific knowledge when queries compress rich perception features, while static fusion cannot change each prior's importance with the scene. They target the provider-to-planner interface. e-problem |
| Core mechanism | Source description Retention probes anchor each compressed expert branch to a distinct frozen prior; a dense router combines expert conditions before action generation. e-interface |
| A key reported result | NAVSIM v2 navtest planning: 90.2 ± 0.11 across training seeds. EPDMS, higher is better. Official EPDMS evaluation; three training seeds, each evaluated with three fixed base-noise seeds. DriveFuture 89.9; IDOL 89.6; Latent-WAM 89.3. A 0.3-point aggregate lead does not imply superiority on every sub-metric. Reported bootstrap separation concerns internal ablations, not external baselines. e-v2e-stabilitye-protocol |
| Reading caution |
Core contributions
- Source description
Retention probes anchor each compressed expert branch to a distinct frozen prior; a dense router combines expert conditions before action generation. e-interface
- Source description
Privileged trajectory sub-scores supervise the representation and router during training, allowing single-trajectory deployment without evaluator-based selection. e-supervisione-gradiente-inference
Figure 2. Build the priors first, then learn how the planner reads and combines them. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left column as provider construction: driving QA adapts InternVL3, then GEO, SEM, and DYN slots align to VGGT, JEPA, and Wan teachers. The right side is WAM training and the planning information flow. Frozen perception and video encoding feed query banks; ego state and command enter through separate encoders. The shared Transformer produces expert conditions and a scene representation. The dashed decoder inset expands the expert router, future head, and action head. The future head's output feeds the actor. Equations (6)–(7) add details omitted from this overview: retention anchors branch readouts, and soft gates mix conditions before generation. e-architecturee-providere-interfacee-futuree-gradiente-inferencee-protocol
What it supports. The two visual streams serve different roles: compact teacher-aligned priors and dense observation latents. Their interaction happens inside a trainable shared planning representation. The expert branches influence one actor through a condition mixture, while the predicted action-free future supplies temporal context for generating a single trajectory.
Where the evidence stops. The diagram is an overview, not a complete gradient graph. Appendix C specifies detached future conditioning and training-only metric/routing supervision. The displayed trajectory is a planning output, and the picture does not establish executed driving performance.
2. Motivation
2.1 The problem and the proposed response
The authors identify a prior-to-plan transfer problem: imitation may discard source-specific knowledge when queries compress rich perception features, while static fusion cannot change each prior's importance with the scene. They target the provider-to-planner interface. e-problem
2.2 What this reading follows
A driving planner can receive strong perception features and still lose useful information when compressing them into a few tokens. PerceptDrive tackles this interface with two linked choices: each expert branch must retain its designated prior, and a scene-dependent router decides how much each branch should influence the action generator. A predicted latent future supplies additional temporal context. The six visuals below connect that information flow to planning scores, controlled design changes, and representation diagnostics. The key boundary is evaluation: privileged NAVSIM sub-scores shape training, while reported planning performance comes from non-reactive benchmark protocols. e-probleme-interfacee-supervisione-protocole-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The frozen provider and separate trainable WAM support a functional Dual-system reading. Predicted action-free futures condition action generation, supporting an IDM-style future-to-action interpretation of Q4. This is not explicit inverse-dynamics regression on observed state pairs: future and actor heads share a backbone, and action-conditioned future prediction serves training only. The classification describes information flow rather than wholly independent world/action networks. e-architecturee-interfacee-futuree-inference
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Preserve a prior where compression can erase it
Start at the query bottleneck rather than the final waypoint loss. The provider has already acquired teacher-aligned knowledge, but the planner operates on compressed query readouts. A trajectory imitation objective can be satisfied without requiring three branches to retain three different sources of information. PerceptDrive therefore gives each branch a designated reconstruction target: its frozen, pooled expert-slot representation. The retention probe sees the branch readout before the shared Transformer, so its gradients shape how queries read the perception pool. All branches can still access the common pool, including dense video latents. Specialization is encouraged by the target assignment rather than enforced by hard separation of inputs. The lower cross-branch similarity and higher matched-target selectivity in Table A7 provide direct diagnostics of this intended effect. e-interfacee-capacitye-gradiente-retention-diagnostic
5.2 2. Convert training-time trajectory quality into a gate
The router predicts weights from a pooled scene representation before trajectory generation. To teach those weights, training temporarily asks what each expert condition would produce on its own. The actor forms one-step endpoint estimates using the same noisy interpolation as its flow loss, with each expert replacing the mixed condition. The drafts are detached and assigned interpolated privileged scores from the navtrain-only pool. Averaging eight sub-scores gives a quality value per branch; temperature-softmax converts those values to a target distribution. Cross-entropy trains the router toward that distribution. This is an approximation to branch quality, not direct optimization of the official aggregate score. The auxiliary regressor is a separate route for teaching quality-aware scene features. Both supervision paths disappear at inference, leaving the learned gate to combine conditions. e-interfacee-supervisione-gradiente-protocole-surrogates
5.3 3. Generate one trajectory while keeping the evidence boundary visible
At deployment, the future head predicts an action-free latent once, and the actor uses that prediction together with the gated expert condition throughout Euler integration. The action loss sees the same kind of predicted future during training; it does not receive a ground-truth future that will be unavailable later. Detachment also keeps action gradients from changing the future head. My interpretation is that this separation makes future prediction a learned conditioning signal rather than a test-time simulator used to search over actions. The distinction matters when reading the results: no candidate scoring is needed, yet the default actor still takes 25 integration steps. The non-reactive NAVSIM scores support this planning procedure. They leave unresolved how prediction errors, changed objectives, or interactive responses from other road users affect deployment. e-futuree-supervisione-gradiente-inferencee-protocole-limits
5.4 Training and inference
During training
WAM training uses navtrain only: 30,000 AdamW steps, effective batch 128, bf16, peak learning rate 0.0001, and 1,000 learning-rate warmup steps. A separate 3,000-step objective warmup disables action and routing losses and keeps gates uniform. Joint training weights future, retention, and routing losses by 0.2, 0.1, and 0.1; action and auxiliary losses have unit weight. e-capacitye-supervisione-optimizatione-protocol
Each navtrain scene supplies a demonstration, 16 smooth perturbations, and a constant-velocity trajectory with eight privileged sub-scores. History comfort is excluded because it depends on executed history. Five-neighbor interpolation scores training predictions. Three detached one-step branch drafts yield mean-quality scores and temperature-0.5 soft gate targets. Auxiliary and routing losses reach the backbone through scene state; detached futures prevent the action loss from updating the future head. e-supervisione-gradient
During inference
Encode the observation once, predict gates and the action-free future once, then integrate the flow actor for 25 Euler steps from Gaussian noise to one trajectory. Retention probes, auxiliary regression, branch drafts, and action-conditioned prediction are absent. Single-trajectory generation still requires repeated integration; these evaluations do not establish physical action execution. e-inferencee-capacitye-protocol
5.5 Implementation flow
- Acquire stationary priors
InternVL3-2B is fully fine-tuned on 1,398,858 cleaned driving-QA samples from DriveLM, LingoQA, DriveQA, NuScenes-QA, and Reason2Drive. Registered GEO, SEM, and DYN slots then receive VGGT geometry alignment, V-JEPA 2 ViT-g masked-feature prediction, and Wan 2.1 velocity-field supervision. Teachers stay frozen; slots, projection heads, a dynamic bridge, and LoRA adapters learn, with QA replay. Merged LoRA weights are frozen for planning. e-provider
- Compress complementary streams
The provider supplies high-level features; a separate frozen V-JEPA 2-L supplies dense observation latents and future targets. Width-1024 projections form a shared pool. Context, action, video, and three expert query banks read this pool; their outputs join ego and command embeddings in a bidirectional Transformer. The backbone cannot revisit the original pool after compression. e-interfacee-inputse-capacity
- Retain, route, and predict
Train-only probes reconstruct detached mean expert-slot targets from mean branch readouts. A scene-vector MLP predicts three soft gates; every expert remains active. Their weighted conditions accompany the action condition. A future head predicts an action-free latent for the actor and an action-conditioned latent for auxiliary training, both supervised against frozen-encoder future targets. e-interfacee-future
6. Experiments & results
PerceptDrive converts frozen geometric, semantic, and dynamic perception priors into driving trajectories. Branch-specific reconstruction preserves these priors through query compression; scene-dependent gates combine their conditions before a predicted latent future guides a flow actor. Its strongest evidence concerns NAVSIM planning scores and internal controls; interactive driving and transfer beyond the training evaluator remain open.
6.1 Read the original evidence
Table 2. The aggregate lead is small, and individual sub-metrics tell a less uniform story. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the shaded EPDMS column, then compare the same rows across the sub-metrics. PerceptDrive reaches 90.2, while DriveFuture reaches 89.9. Appendix A explains that NC, DAC, DDC, and TLC act as multipliers for collision, drivable-area, direction, and traffic-light compliance; EP, TTC, LK, HC, and EC contribute weighted progress, time-to-collision, lane-keeping, history-comfort, and extended-comfort terms. All arrows indicate higher is better. Bold and underlined values denote the best and second reported entries. An aggregate cannot be reconstructed by simply averaging these displayed dataset-level columns, and a stronger final score does not require winning each column. e-v2e-protocole-stability
What it supports. PerceptDrive has the highest listed EPDMS, but Latent-WAM's extended comfort is 87.3 versus 86.3, while PerceptDrive's history comfort is 98.5. This supports a claim about the reported aggregate objective, with a 0.3-point lead over DriveFuture, rather than universal superiority across driving qualities.
Where the evidence stops. These are published-system comparisons with differing training and inference resources. The paper's three-seed variability and paired bootstrap support its internal comparisons; they do not supply a paired significance test against DriveFuture or other external baselines.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM v1 navtest planning Official non-reactive evaluation; one front camera; default 25-step actor. | 90.4; training-seed mean ± standard deviation 90.4 ± 0.05. PDMS, higher is better | DriveVLA-W0 90.2; DriveWAM 90.1; human reference 94.8. Highest listed non-human aggregate, with a 0.2-point margin over DriveVLA-W0. External systems differ in sensors, pretraining, and inference budget; their uncertainty is not reported here. e-v1e-stabilitye-protocole-inference |
| NAVSIM v2 navtest planning Official EPDMS evaluation; three training seeds, each evaluated with three fixed base-noise seeds. | 90.2 ± 0.11 across training seeds. EPDMS, higher is better | DriveFuture 89.9; IDOL 89.6; Latent-WAM 89.3. A 0.3-point aggregate lead does not imply superiority on every sub-metric. Reported bootstrap separation concerns internal ablations, not external baselines. e-v2e-stabilitye-protocol |
| NAVSIM v2 navhard two-stage planning Public two-stage pseudo-simulation with pre-generated follow-up observations. | 34.5. Combined EPDMS, higher is better | DriveLaW 30.6; MindDrive 30.5; DiffusionDrive 27.5. The 3.9-point lead over DriveLaW belongs to a distinct protocol. Stage-2 lane keeping falls to 48.7 from 96.4; this is not an interactive closed loop. e-navharde-protocol |
| Nested WAM mechanism ablation NAVSIM v2 navtest; components added in construction order; uniform gates until routing is enabled. | 84.6 → 85.4 → 87.1 → 88.7 → 90.2. EPDMS, higher is better | Imitation → future conditioning → metric supervision → prior retention → metric-distilled routing. Gains depend on this ordering. Separately, removing retention from the full system yields 88.8, and disabling routing distillation yields 89.5, supporting both mechanisms outside the nested progression. e-nestede-controlse-ablation-scope |
| Inference solver trade-off Same trained model; end-to-end latency on one AMD MI308X; no retraining. | 25 steps: 90.2 at 68 ms; 10 steps: 90.1 at 53 ms. EPDMS and latency | One step: 87.5 EPDMS at 44 ms. Fewer integration steps offer a measured accuracy/latency trade-off on the stated hardware, without establishing deployment-hardware latency. e-cost |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The construction sequence improves both planning aggregates, with different components affecting comfort. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward as a sequence of increasingly complete systems. FC means future conditioning, MS metric-aware supervision, PR per-branch prior retention, and MR metric-distilled routing. Missing MR means that all three branches use uniform gating. PDMS and EPDMS belong to NAVSIM v1 and v2 navtest respectively; EC is the v2 extended-comfort sub-metric. The small subscripts are standard deviations over three training seeds, not confidence intervals. Appendix F reports greater precision than this rounded table. Notice that EC jumps from 78.9 to 84.0 when future conditioning is added, while later components continue improving the aggregate scores. e-nestede-ablation-scopee-stabilitye-controls
What it supports. EPDMS progresses from 84.6 through 85.4, 87.1, and 88.7 to 90.2. Retention and routing contribute in addition to future conditioning and metric supervision in this construction order. The comfort column suggests that much of the extended-comfort improvement appears when the actor first receives future information.
Where the evidence stops. These increments depend on the preceding configuration. They are not order-independent component effects. Appendix D also specifies different active warmup objectives, with imitation-only skipping objective warmup entirely; Table A2 provides separate full-system controls.
Figure 4. Routing varies with context, and masking exposes different patterns of prior reliance. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In the left panel, compare each colored bar with the dashed uniform-weight line. Blue denotes geometry, gold semantics, and purple dynamics; the whiskers are within-command standard deviations, not uncertainty in the mean. Command counts appear under the labels. In the right panel, the black outline marks the full model's score on each axis. Every axis covers a seven-point interval below that score, with rings at decreases of two, four, and six points and shaded 95% bootstrap bands. Follow each masked-prior line within its metric sector rather than treating the polygon area as a planning score. e-routing-visuale-prior-ablatione-ablation-scopee-controls
What it supports. The geometric branch has the largest mean weight during turns, and the dynamic branch during straight travel. Masking geometry mainly harms road compliance and lane keeping; masking semantics affects collision/TTC measures; masking dynamics affects progress and extended comfort. These patterns link branch identity with distinct aspects of the benchmark.
Where the evidence stops. Command-grouped means alone do not prove scene sensitivity; the command-only control and within-command dispersion provide additional evidence. Masking occurs without retraining, so the radar measures reliance under changed inputs, not isolated causal effects of learning each prior.
Table A2. Full-model controls distinguish scene-conditioned routing and prior-specific targets from simpler alternatives. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the top row as the reference for every subsequent row. The first block tests future-projection initialization, objective warmup, and metric gradients through the scene state. The middle block changes fusion or router inputs: static weights ignore the scene, trajectory averaging combines completed plans, and the end-to-end gate lacks metric distillation. The last block changes retention or access to the perception pool. Removing retention preserves routing, unlike the nested table's earlier stages. Shuffled targets mismatch branch identity, shared targets remove specificity, and source-masked reads restrict prior-slot access while retaining observation latents. Appendix D defines these retrained controls. e-controlse-ablation-scopee-gradient
What it supports. Static weighting scores 88.8 EPDMS, an end-to-end gate 89.5, and full metric-distilled routing 90.2. Retention removal scores 88.8; shuffled branch targets score 88.6. Together, the results favor scene-dependent gate supervision and correct branch-to-prior assignment over merely adding learnable weights or a reconstruction objective.
Where the evidence stops. Trajectory averaging generates three plans per scene and therefore has a different inference budget. Small gaps among retention variants have no uncertainty shown in this table. Shuffled targets test the specified assignment design, not a general impossibility of learning from remapped experts.
Table A7. Retention changes what branches represent, beyond changing the final trajectory score. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows column by column. CKA is the reported similarity measure between branch readouts: lower values here indicate less redundant branches. Diag. and Off-diag. are held-out reconstruction cosines for matched and mismatched branch–prior pairs. Select. is their difference, so a larger value means that a branch better preserves its own target than other targets. Own attn. is the fraction of cross-attention directed to the branch's designated slot group. It is a proportion, not a percentage already scaled by one hundred. These measurements are on navtrain-dev and address representation specialization rather than driving execution. e-retention-diagnostice-controlse-interface
What it supports. With retention, matched reconstruction improves to 0.83 while mismatched reconstruction drops to 0.41, producing selectivity 0.42 instead of 0.08. Designated-slot attention rises from 9% to 31%, alongside lower cross-branch similarity. These diagnostics support the proposed mechanism that a distinct target helps each compressed branch preserve distinct prior information.
Where the evidence stops. Representation separation is supporting evidence, not sufficient proof of better control or generalization. This table gives no uncertainty estimates. The planning controls in Table A2 are needed to connect target specificity to the reported NAVSIM gains.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors leave interactive closed-loop evaluation and transfer to differently weighted evaluators untested. The future predictor is deterministic; predictions under unobserved actions extrapolate beyond demonstrated-action supervision. e-limitse-future
Token masking measures reliance and changes inputs; only dynamic-teacher removal is verified by provider and WAM retraining. That retrained variant reaches 88.6 EPDMS versus 90.2 full. e-prior-ablatione-ablation-scope
Router supervision is approximate: interpolated sub-scores reach overall Spearman correlation 0.91, and one-step versus full-rollout expert choices agree in 86.2% of tested dev scenes. These diagnostics support useful approximations, not exact evaluator equivalence. e-surrogates
7.2 Questions for discussion
- Would routing gains persist under different evaluator weights?
- Do retention-induced representation differences predict robustness beyond NAVSIM?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the driving-QA mixture, frozen teacher checkpoints, NAVSIM splits/evaluators, and navtrain-only privileged pools. Appendix C gives spline perturbation scales and the five-neighbor distance/fallback rule. Reported training costs 409.6 MI308X GPU-hours, using eight GPUs per provider stage and two for WAM training; the WAM has 421M trainable parameters. e-providere-supervisione-protocole-capacitye-cost
The supplied description does not fix LoRA rank, numeric provider-loss weights, the complete teacher-distillation schedule, exact checkpoint revisions, or software versions. These omissions prevent an exact build from the paper alone despite detailed WAM settings. e-providere-inputse-capacitye-cost
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether correct retention targets improve planning beyond representation separation
Reader-proposed check, not performed: freeze the same distilled provider and retrain the WAM with correct per-branch targets, shuffled targets, a shared concatenated target, and no retention. Hold query counts, routing supervision, the 30k-step budget, and training/base-noise seeds fixed. Measure navtrain-dev reconstruction selectivity and cross-branch similarity, then evaluate official navtest PDMS/EPDMS with paired scene comparisons. Correct targets should improve both selectivity and planning relative to shuffled or absent targets if prior-specific preservation explains the gain. If specialization improves without a corresponding planning benefit, the representation diagnostic alone would not validate the proposed control mechanism. e-controlse-ablation-scopee-retention-diagnostice-capacitye-stability
Check 2: Separate routing-surrogate error from evaluator-specific alignment
Reader-proposed check, not performed: on a fixed navtrain-dev scene set, compare expert rankings and soft targets from one-step drafts with five-neighbor interpolation against full 25-step per-expert rollouts scored directly. Stratify disagreements by collision/TTC boundaries rather than reporting only a global correlation. Then, using navtrain-only supervision and identical initialization/budgets, compare routers trained with the original sub-score mean and a predeclared alternative weighting; retain an end-to-end gate without distillation as a control. Evaluate held-out scenes under both objective definitions. Concentrated boundary errors would limit the surrogate, while a routing gain that vanishes after objective reweighting would indicate narrow evaluator alignment. e-supervisione-surrogatese-controlse-protocole-limits
8.3 Reading coverage
Visual audit: Visually inspected the title, all 14 authors, affiliations and version on p. 1; motivation and Figures 1–2 on pp. 2–4; provider objectives, retention, gating and optimization on pp. 5–7; Tables 1–3, Figure 4 and limitations on pp. 8–10; and every appendix page, pp. 15–24. The supporting appendix pages include evaluation and inputs (p. 15), capacity and inference (p. 16), branch drafts and supervision pools (p. 17), gradient routes and control protocols (pp. 18–19), stability and surrogate validation (pp. 19–21), retention diagnostics (pp. 20–21), hardware and solver latency (pp. 21–22), and qualitative Figures A1–A2 (pp. 23–24). Figure 2 was checked against Equations (6)–(8) and Appendix C: the overview omits detailed retention and gradient wiring, which the explanation takes from the formulation. Figure 4 was checked against its caption and Table A1, including whiskers and the relative radar scale. All six final crops were inspected with their complete headers, legends and labels. References on pp. 11–14 were read as text; separate supplements and code remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 Method, including 3.1–3.4
- 4 Experiments, including 4.1–4.3 and Discussion
- 5 Conclusion
- References
- Appendix A Evaluation Protocols
- Appendix B Implementation Details
- Appendix C Staged Optimization and Gradient Routing
- Appendix D Ablation Protocol and Interpretation
- Appendix E Results on the navhard Split
- Appendix F Statistical Stability and Extended Diagnostics
- Appendix G Computational Cost and Parameters
- Appendix H Additional Visualization
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected title and all 14 authors match the catalog after name-order normalization. The title page identifies arXiv:2607.20175v1, 22 July 2026, while the manuscript's internal date is July 20, 2026. No alternative revision was supplied or compared.
- The acquisition notes that text extraction does not reconstruct figure images. This limitation was addressed by visually inspecting the retained PDF's figures, tables, and all pages supporting retained scientific claims.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- All seven supplied text chunks were read individually, covering all 24 pages. Reference pages 11–14 were read as text; visual inspection covered pages 1–10 and 15–24.
- Code, external sources, and model artifacts were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block, date and arXiv margin
Exact title and 14-author sequence match the catalog. Affiliations are Tsinghua University and AMap, Alibaba Group. The margin identifies v1 dated 22 July 2026; the manuscript date is July 20, 2026.
Go to primary source ↓e-problemPDF pp. 2–3, Section 1 and Figure 1 caption
The authors separate query-compression losses of prior information from scene-invariant allocation, motivating retention and scene-conditioned fusion.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and Section 3
Stages 1a/1b construct a frozen provider; Stage 2 joins its features and frozen video latents in a shared Transformer with expert routing, a future head, and an action head.
Go to primary source ↓e-providerPDF p. 5, Section 3.2, Equations (1)–(5)
QA adaptation uses 1,398,858 samples. Frozen VGGT, V-JEPA 2 ViT-g, and Wan 2.1 supervise geometric, semantic, and dynamic slots. Only slots, lightweight heads/bridge, and LoRA learn during distillation, with replay; numeric provider-loss coefficients are not supplied.
Go to primary source ↓e-interfacePDF p. 6, Section 3.3, Equations (6)–(7)
All query banks read a shared perception pool. Retention reconstructs detached pooled prior targets. Scene-state softmax gates densely mix three expert conditions before generation; the predicted action-free future conditions the actor.
Go to primary source ↓e-inputsPDF p. 4, Section 3.1; p. 15, Appendix B, Inputs and frozen provider
Four frames at 2 Hz and ego/command inputs predict eight 4-second waypoints, normalized by (50,50,pi). Provider features have 1,048 tokens of width 1,536; dense video features have shape 256 by 2 by 1,024.
Go to primary source ↓e-capacityPDF p. 16, Appendix B, WAM capacity and optimization
The 421,481,483-parameter WAM uses width 1,024; context/action/video query counts are 64/8/32, with eight queries per expert. It trains 30k AdamW steps at batch 128, bf16, peak 1e-4, 1k learning-rate warmup and cosine decay. The flow loss samples uniform time and Gaussian base noise.
Go to primary source ↓e-futurePDF p. 16, Appendix C, Equations (A1)–(A4); p. 17, extrapolation paragraph
Action-free and demonstrated-action-conditioned futures receive L1 supervision against the same target. Auxiliary regression consumes scene, trajectory, and pooled imagined outcome; futures for unobserved actions are extrapolations.
Go to primary source ↓e-optimizationPDF p. 7, Section 3.4, Equation (8); p. 16, Appendix C, Two-phase WAM training
Objective warmup disables action/routing losses and keeps gates uniform; joint optimization activates all five losses. Future-conditioning output projection starts at zero.
Go to primary source ↓e-supervisionPDF p. 17, Appendix C, Equations (A5)–(A7), Offline supervision pool; p. 18, opening paragraph
Flow matching uses noise-to-demonstration interpolation. Detached one-step branch drafts are scored with five-neighbor interpolation over demonstration, 16 perturbations, and constant-velocity pools. Eight score components exclude HC; routing uses mean score and temperature 0.5. Objective warmup is 3,000 steps; future/retention/routing weights are 0.2/0.1/0.1. Spline noise scales and distance/fallback rules are specified.
Go to primary source ↓e-gradientPDF p. 18, Appendix C, Gradient routes
Detached future input blocks action-loss gradients into the future head; auxiliary/routing gradients reach the backbone through scene state only. Retention updates probes/readouts; detached draft targets do not update action/future heads.
Go to primary source ↓e-inferencePDF p. 16, Appendix B, Inference
Provider, router and action-free future run once, followed by 25 Euler integration steps to one trajectory. Auxiliary metric branch, retention probes, drafts, and action-conditioned future prediction are removed.
Go to primary source ↓e-protocolPDF p. 10, Discussion; p. 15, Appendix A, all evaluation protocols
All evaluations are non-reactive. Privileged pools contain navtrain only. Cross-paper configurations differ. EPDMS combines four multiplier metrics and five weighted metrics with false-positive filtering; navhard uses pre-generated second-stage observations rather than interactive feedback.
Go to primary source ↓e-v1PDF p. 8, Table 1, PDMS column and sensor column
PerceptDrive uses one camera and scores 90.4; DriveVLA-W0 90.2, DriveWAM 90.1, and human 94.8. Comfort is saturated among listed systems.
Go to primary source ↓e-v2PDF p. 8, Table 2, EPDMS and sub-metric columns
PerceptDrive scores 90.2 EPDMS versus DriveFuture 89.9, IDOL 89.6, and Latent-WAM 89.3; its EC is 86.3 versus Latent-WAM 87.3 and its HC is 98.5.
Go to primary source ↓e-nestedPDF p. 10, Table 3, all rows
EPDMS increases 84.6, 85.4, 87.1, 88.7, 90.2 through imitation, future conditioning, metric supervision, retention, and routing. EC changes 78.9 to 84.0 with future conditioning and reaches 86.3 full.
Go to primary source ↓e-controlsPDF p. 19, Table A2, all rows
EPDMS is 88.8 for static weights, 88.3 for trajectory averaging, 89.5 without routing distillation, 89.0 command-only, 89.9 without command, 88.8 without retention, 88.6 shuffled targets, 89.5 shared targets, 90.1 token-level targets, and 89.7 source-masked reads, versus 90.2 full.
Go to primary source ↓e-ablation-scopePDF pp. 18–19, Appendix D
Provider-composition variants are retrained; token removal is inference-only. Nested effects are conditional on ordering. Design controls alter one choice, while trajectory averaging uniquely generates three trajectories. Source-masked reads retain observation latents but restrict prior-slot access.
Go to primary source ↓e-prior-ablationPDF p. 17, Table A1, provider and token-removal blocks
Full EPDMS is 90.2; removing dynamic distillation with retraining yields 88.6, no prior distillation 85.5, no VLM 81.9. Inference GEO/SEM/DYN removal yields 88.1/88.5/88.3 with different sub-metric deficits.
Go to primary source ↓e-routing-visualPDF p. 9, Figure 4 and caption; p. 21, Routing dispersion within commands
Geometry has highest mean gate weight during turns, dynamics during straight travel. Whiskers show within-command standard deviations. Radar axes span seven points below each full score, with 95% bootstrap shading. Mean within-command gate standard deviation is 0.11; command-only prediction explains 38% of gate variance.
Go to primary source ↓e-navhardPDF p. 19, Table A3 and Appendix E
Combined navhard EPDMS is 34.5 versus DriveLaW 30.6, MindDrive 30.5, DiffusionDrive 27.5. PerceptDrive LK changes from 96.4 in stage 1 to 48.7 in stage 2.
Go to primary source ↓e-stabilityPDF pp. 19–20, Appendix F, Table A4 and Multi-seed stability
Three training seeds, each with three fixed base-noise seeds, yield full-model 90.4 ± 0.05 PDMS and 90.2 ± 0.11 EPDMS. Evaluation-seed standard deviations are at most 0.05. A seed-0 paired bootstrap with 10k resamples reports separation from internal ablations at 95%.
Go to primary source ↓e-provider-generalityPDF p. 20, Table A5
Retention plus routing improve QA-only EPDMS from 83.7 to 85.5 and distilled-provider EPDMS from 87.1 to 90.2.
Go to primary source ↓e-surrogatesPDF p. 20, Validity of routing supervision; p. 21, Table A6
Four held-out perturbations per 2,048 navtrain-dev scenes yield overall sub-score Spearman 0.91. On 500 dev scenes, one-step drafts and 25-step rollouts agree on the best expert 86.2% of the time, with mean target-gate L1 gap 0.06.
Go to primary source ↓e-retention-diagnosticPDF p. 20, Retention diagnostics; p. 21, Table A7 and caption
Retention changes mean pairwise CKA from 0.76 to 0.52, matched/mismatched reconstruction cosines from 0.69/0.61 to 0.83/0.41, selectivity from 0.08 to 0.42, and designated-slot attention share from 0.09 to 0.31.
Go to primary source ↓e-costPDF p. 21, Table A9; p. 22, Appendix G and Table A10
Provider stages use eight MI308X GPUs for 30.4 and 152.0 GPU-hours; WAM uses two for 227.2, totaling 409.6. Inference retains 2.82B active parameters. One/10/25 Euler steps yield EPDMS 87.5/90.1/90.2 at 44/53/68 ms on one MI308X.
Go to primary source ↓e-limitsPDF p. 10, Discussion and Section 5
The authors acknowledge evaluator alignment, untested transfer to other objectives, non-reactive evaluation, deterministic futures, extrapolated action-conditioned predictions, and masking's limited causal interpretation.
Go to primary source ↓8.5 Primary sources
PerceptDrive: Perception Prior World-Action Modeling with Adaptive Expert Routing for End-to-End Autonomous Driving ↗
PDF · 10,629 extracted words
Source fingerprint
ad2e5d93829e49e7d6d985742329be3c927f2d75b1c72aa45fc370ee6e18339e