WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving
1. Paper overview
In one sentence: WA-JEPA couples latent-future flow generation with ego-trajectory prediction, improving simulated planning while leaving inference fidelity, comfort and deployment speed incompletely established. rationalemaskingflowjointgradientnavsim2hugsimseedstemporalmetrics
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that random-mask completion does not adequately train past-to-future prediction, while deterministic regression can smooth uncertain futures. Their goal is to retain V-JEPA's semantic features and make them useful for directly predicting ego motion. The broader claim that existing representations are fundamentally inadequate remains the authors' motivation, rather than a universal experimental conclusion. rationale |
| Core mechanism | Source description Hybrid future masking combines strictly causal prediction with easier partial-future completion during video-only adaptation. masking |
| A key reported result | NAVSIM-v2 planning: 91.7; unrounded mean 91.7014, SD 0.0531, 95% t-CI [91.6634, 91.7393]. EPDMS, higher is better. Held-out navtest, corrected evaluator; ten sampling-noise seeds with fixed model weights. SparseDriveV2 90.1; Discrete-WAM 90.4: gains of 1.6 and 1.3 score points. These are pseudo-simulation planning scores. The separate pre-fix EPDMS* value is 88.0 and must not be mixed with corrected results. The interval quantifies sampling variability, not retraining uncertainty. navsim2metricsseeds |
| Reading caution | Reader analysis Overall closed-loop gains conceal weaker comfort: 0.6620 versus VAD's 0.9534. On extreme scenarios WA-JEPA scores 0.1362, below DrivoR's 0.1407. The aggregate lead therefore does not establish uniformly better driving behavior. hugsim |
Core contributions
- Source description
Hybrid future masking combines strictly causal prediction with easier partial-future completion during video-only adaptation. masking
Figure 2. Future representation learning becomes trajectory planning through a shared flow predictor. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start on the left with history frames and the two future-condition choices. The dashed Patch-mask box is available only in Stage 1; full masking remains in Stage 2. Follow the EMA arrow to the target encoder, which sees actual future frames to construct training targets. On the right, the green predictor joins future, context and action streams. The action branch is added in Stage 2 and predicts the trajectory shown above it. Pages 3–5 distinguish the noisy training inputs from deployment: at inference the future scene and action streams begin from noise. Appendix C on page 11 explains the gradient boundary that the overview drawing does not show. maskingflowjointgradientinference
What it supports. The architecture supports joint world/action prediction within one shared predictor. Future targets provide dense training supervision, and action supervision can shape scene representations. The target encoder is part of the learning procedure; its presence is not evidence of a separate inference-time world model and planner.
Where the evidence stops. The diagram mixes training and prediction paths. Do not infer that future images are available during driving, or that every cross-stream connection carries gradients in both directions. The text explicitly blocks scene-loss gradients at the action-token interface.
2. Motivation
2.1 The problem and the proposed response
The authors argue that random-mask completion does not adequately train past-to-future prediction, while deterministic regression can smooth uncertain futures. Their goal is to retain V-JEPA's semantic features and make them useful for directly predicting ego motion. The broader claim that existing representations are fundamentally inadequate remains the authors' motivation, rather than a universal experimental conclusion. rationale
2.2 What this reading follows
A useful driving representation must do more than recognize a road scene: it must support decisions about what happens next. WA-JEPA adapts a video JEPA encoder using both fully hidden futures and partially visible future patches, then trains one predictor to generate future scene embeddings and ego trajectories together. The dense scene objective and sparse action objective interact through a carefully restricted gradient path. This reading follows those information paths before examining the benchmark tables. The results support stronger planning under specified simulators, but corrected evaluator versions, fixed-model sampling uncertainty and training-only latent diagnostics sharply define what the evidence establishes. rationalemaskingflowjointgradientnavsim2hugsimseedstemporalmetrics
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Conflicts with the recorded classification
Joint prediction and latent driving WAM labels are supported. The recorded Dual-system/Q3 architecture conflicts with Eq. (13), which explicitly identifies two output streams of the same MMDiT-style predictor, and with the shared inference process. A One Model × Joint prediction assessment is better supported; the EMA target encoder is a supervision mechanism. This is an editorial assessment, and the supplied classification snapshot is preserved. maskingjointgradientinference
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use partial futures to learn, then remove them for planning
Stage 1 has two jobs that are easy to conflate. Full-mask forces prediction from historical observations because every future condition token is a learned placeholder. Patch-mask supplies some future image patches and asks the model to infer the rest, preserving an easier completion task alongside causal prediction. In both cases, a separate EMA target encoder sees unmasked future images to define latent supervision. The online encoder processes only the patches permitted by the branch. Stage 2 then removes the partial-future condition entirely and adds trajectory supervision. This distinction explains why the training diagram contains future images without making them deployment inputs. Table 4(b) tests the proposed complementarity: the hybrid obtains 91.7 EPDMS, compared with 91.0 for Patch-mask and 91.3 for Full-mask alone. maskingjointmask-ablation
5.2 Separate the generative objective from the gradient routing
Flow matching here predicts a clean endpoint, so its loss still looks like ordinary mean-squared error. The difference is that the predictor receives a time-dependent mixture of noise and the true future latent, together with history and future-condition tokens. During planning, the model instead starts from Gaussian noise and repeatedly converts estimated endpoints into velocities. The action stream follows the same normalized-noise construction and shares the joint predictor with the scene stream. Forward information exchange does not imply symmetric learning: the scene output receives stop-gradient action tokens, while action supervision can propagate into differentiable scene representations. Appendix C describes this as a way to retain scene prediction while making representations relevant to ego planning. Table 4(c) tests joint modeling and future objectives, but does not separately ablate this gradient rule. flowjointgradientinferencejoint-ablation
5.3 Read three kinds of evidence at their own scope
The planning tables, simulated rollouts and latent diagnostics answer different questions. NAVSIM's corrected EPDMS assesses planned trajectories through a specified pseudo-simulation and aggregation procedure; the pre-fix column is a separate protocol. HUGSIM adds executed plans through a shared simulator controller and reactive scenarios, extending the evidence to simulated closed-loop transfer. It also exposes comfort and extreme-scenario weaknesses that an overall score hides. The feature diagnostics examine temporal variation relative to learned EMA targets, and the PCA maps offer selected qualitative illustrations. Reader interpretation: their agreement with the ablation makes a coherent mechanism hypothesis, but does not prove that preserving these latent statistics causes better driving. In particular, Appendix E evaluates a one-step training prediction, while deployment uses twelve integration steps from noise. metricsnavsim2hugsimclosed-protocolpcatemporalinference
5.4 Training and inference
During training
Stage 2 uses NAVSIM navtrain; held-out navtest supplies v1/v2 evaluation. Inputs are four historical frames from left, front, right and rear cameras at 256×512. The online encoder is optimized, while its target follows EMA. setupmasking
Training uses AdamW, bfloat16 and DeepSpeed ZeRO-2, with 64 A800 GPUs for Stage 1 and 32 for Stage 2, batch size 4 per GPU. Encoder, scene-projector and joint-predictor learning rates are respectively 10⁻⁵, 10⁻⁴ and 1.5×10⁻⁴; weight decay is 0.04. setup
During inference
History and ego state condition two Gaussian-initialized future streams. Twelve iterations convert predicted clean endpoints into flow velocities and integrate them; denormalization yields the trajectory. Future images and ground-truth future actions are absent. In HUGSIM a common simulator controller executes the plans. inferenceclosed-protocol
5.5 Implementation flow
- Encode history and construct targets
A V-JEPA 2 ViT-L processes each camera video independently with shared weights. Historical tokens remain visible. An EMA target encoder processes unmasked future frames and supplies stop-gradient supervision; it is a training target network, not a separate planner. masking
- Adapt using two future masks
Full-mask replaces every future condition token with a learned mask token. Patch-mask encodes only visible future patches, then scatters them into a full token sequence. The two branches adapt the representation on nuPlan without action labels. maskingflow
- Generate latent futures
The predictor receives historical context, mask-aware future conditions, noisy future tokens and flow time. MMDiT-style joint attention estimates the clean latent endpoint. The objective is still MSE; the generative change is the noise/time-conditioned prediction problem. flow
- Add coupled action generation
Stage 2 initializes from Stage 1 and uses only Full-mask. Linear and MLP encoders form action tokens from noisy normalized future actions, past actions and ego state. A shared predictor emits both futures. Scene loss cannot backpropagate through the action-token interface, while action loss can shape scene features. jointgradient
6. Experiments & results
WA-JEPA turns video JEPA features into a driving planner through future-masked adaptation and joint flow generation of scene latents and ego trajectories. It reports strong NAVSIM and simulated closed-loop results, with evaluator-version and sampling-uncertainty qualifications detailed below.
6.1 Read the original evidence
Table 1. The headline 91.7 belongs to the corrected EPDMS column. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two rightmost columns separately. EPDMS* is the older aggregation, while EPDMS applies the corrected human-reference filtering procedure. The retained table note also explains the dagger on methods using auxiliary simulator-derived supervision. For the headline comparison, stay in the final column: WA-JEPA reaches 91.7, SparseDriveV2 90.1 and Discrete-WAM 90.4. A dash is missing reporting, not zero performance. Pages 12–13 explain that EPDMS combines metric-specific filtering with two stages of pseudo-simulation and scene-weighted aggregation. Table 7 and Appendix C on page 11 establish that WA-JEPA's reported score averages sampling noise with the model held fixed. navsim2metricsseedssetup
What it supports. The supported margins are 1.6 EPDMS points over SparseDriveV2 and 1.3 over Discrete-WAM under the corrected protocol. They are benchmark-score differences, not percentage changes in real-world driving success. Comparing 91.7 against an EPDMS* baseline would mix incompatible evaluator settings.
Where the evidence stops. Backbones and training resources differ across rows, and the table is not a compute-matched experiment. The ten-seed interval in Appendix C measures inference-noise variability; it does not establish stability across retraining or independent data samples.
Table 2. Closed-loop transfer improves the aggregate score while leaving visible weaknesses. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The upper block compares metrics across the same 436 HUGSIM scenarios, all on a zero-to-one scale with higher values preferred. Find the HD-Score row for the aggregate result, then inspect comfort rather than assuming the bold overall score implies every behavior improves. The lower block separates easy, medium, hard and extreme scenarios and prints each group's size. Appendix A on page 11 pins the corrected controller/evaluator and explains that scenario counts weight the difficulty groups after within-dataset averaging. It also states that neither WA-JEPA adaptation stage uses any of the four HUGSIM source datasets. The rollout examples on page 12 show selected behaviors, while this table measures the full evaluation. hugsimclosed-protocolqualitative
What it supports. WA-JEPA's overall HD-Score is 0.4462, compared with 0.3252 for DrivoR. Its comfort score is 0.6620, below VAD's 0.9534, and its extreme-difficulty HD-Score is 0.1362, below DrivoR's 0.1407. The result supports aggregate simulated transfer with clear remaining weaknesses.
Where the evidence stops. Methods retain native camera configurations, and DrivoR uses additional NAVSIM validation data. These are common-evaluator comparisons, not identical-input or identical-training experiments. Simulated closed-loop driving does not establish physical-road deployment performance.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NAVSIM-v2 planning Held-out navtest, corrected evaluator; ten sampling-noise seeds with fixed model weights. | 91.7; unrounded mean 91.7014, SD 0.0531, 95% t-CI [91.6634, 91.7393]. EPDMS, higher is better | SparseDriveV2 90.1; Discrete-WAM 90.4: gains of 1.6 and 1.3 score points. These are pseudo-simulation planning scores. The separate pre-fix EPDMS* value is 88.0 and must not be mixed with corrected results. The interval quantifies sampling variability, not retraining uncertainty. navsim2metricsseeds |
| NAVSIM-v1 planning Held-out navtest under the v1 protocol. | 91.8 PDMS, higher is better | DriveWorld-VLA 91.3. A 0.5-point improvement within v1; PDMS and v2 EPDMS use different aggregations. navsim1metrics |
| Zero-shot HUGSIM closed-loop driving 436 HUGSIM scenarios; no HUGSIM-specific fine-tuning; neither WA-JEPA adaptation stage uses HUGSIM renderings or its four source datasets; common corrected controller/evaluator. | 0.4462 HD-Score on [0,1], higher is better | DrivoR 0.3252; UniAD 0.3124. The lead persists under dataset-uniform and scenario-uniform aggregation. Native sensors differ; DrivoR additionally uses NAVSIM validation data in training. This is simulated execution. hugsimclosed-protocol |
| Masking and scene/action ablations NAVSIM-v2 navtest; Table 4 separates initialization, Stage 1 masking and Stage 2 components. | Patch-only 91.0; Full-only 91.3; hybrid 91.7. EPDMS | No Stage 1: 89.5. Stage 2 historical-feature cascade: 89.9; cascade with separate flow predictor: 90.8; joint without future supervision: 91.1; joint regression: 90.7; joint flow: 91.7. Hybrid exceeds Patch-only by 0.7 and Full-only by 0.4 points. Joint flow exceeds joint regression by 1.0 and joint without future supervision by 0.6. These controlled comparisons support the combined design; skipping Stage 1 also changes exposure to nuPlan. mask-ablationjoint-ablation |
| Temporal representation preservation Projected scene tokens; target-selected dynamic locations; raw metrics averaged over 0–36k training. | Flow: 0.10 and 0.80. Directional similarity gap ↓; temporal change ratio closer to 1 | Direct regression: 0.30 and 0.45. Flow better preserves target-relative variation. These are one-step training diagnostics, not measurements of complete inference rollouts or calibrated physical dynamics. temporal |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4(a–c). Masking, representation initialization and scene/action coupling are tested in separate comparisons. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the three panels as separate experiments. Panel (a) skips Stage 1 adaptation for every encoder and compares initialization under a common Stage 2 setup. Panel (b) starts at 89.5 without Stage 1, then enables Patch-mask, Full-mask or both. Panel (c) uses Stage 1 initialization: its first row is an action-only cascade, and the next adds a separate flow future predictor. The last three rows use joint modeling, respectively without explicit future supervision, with regression and with flow matching. The prose on pages 7–8 resolves these row meanings; an empty checkbox cell is an absent component, not an unavailable measurement. encoder-ablationmask-ablationjoint-ablationseeds
What it supports. Hybrid masking improves over Full-mask alone by 0.4 EPDMS points. Within joint prediction, direct regression gives 90.7, below 91.1 without future supervision, while flow reaches 91.7. Thus simply adding future prediction is not sufficient in this setup; the objective and coupling matter.
Where the evidence stops. Panel (a) does not isolate pretraining objective from every difference in pretrained weights or data. Do not compare its baseline directly with panel (c)'s action-only baseline as though only one component changed. The table provides no separate uncertainty interval for each row.
Figure 4. Flow matching retains more target-relative temporal variation during training. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the blue and orange legends before comparing the curves. On the left, smaller directional-similarity gap means predicted future steps are less excessively alike relative to the EMA targets. On the right, the change-magnitude quantity is a ratio: one means matching the target's mean adjacent-step variation, so proximity to one is desirable. Appendix E on pages 13–14 selects 64 dynamic camera/spatial locations using target changes and evaluates four future token steps. Its protocol uses flow's one-step clean-endpoint prediction at sampled training times. The headline values on page 8 average raw logs over the common training interval; they are not simply the final plotted markers. temporalflowinferencejoint-ablation
What it supports. The paper reports directional-gap means of 0.10 for flow and 0.30 for regression, with change ratios of 0.80 and 0.45. These target-referenced diagnostics support less temporal collapse for flow under the reported training protocol and complement the planning ablation.
Where the evidence stops. Flow's diagnostic input includes a noisy interpolation toward the true target, whereas deployment starts from pure noise. These curves therefore cannot directly establish the fidelity of twelve-step inference or prove that improved latent variation causes better planning.
Figure 3. Compare each prediction with its own target before interpreting the colored latent maps. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row of latent maps represents two consecutive future frames, whose originals appear at the far right. First compare the flow prediction with the flow target in the adjacent column; then repeat that comparison for regression. The caption on page 6 specifies an essential restriction: each method has its own PCA basis fitted to its EMA targets. That same basis is applied to the corresponding predictions, enabling within-method comparison. It does not align colors across methods. Follow the rows downward to examine how spatial structure changes over the future horizon, rather than treating the colors as semantic classes or recovered RGB images. pcatemporaljoint-ablation
What it supports. In this example, the flow predictions retain more of their targets' spatial structure, while regression predictions are visibly smoother. This is qualitative support for the representation-collapse analysis. The quantitative temporal curves and planning ablations supply separate tests of the proposed benefit.
Where the evidence stops. These are projected feature maps from a selected example, not generated camera frames or calibrated measurements of physical state. Separate PCA bases prevent direct cross-method color comparison, and this visual cannot establish the frequency of failure across the test set.
7. Analysis & limitations
7.1 What the evidence leaves open
Overall closed-loop gains conceal weaker comfort: 0.6620 versus VAD's 0.9534. On extreme scenarios WA-JEPA scores 0.1362, below DrivoR's 0.1407. The aggregate lead therefore does not establish uniformly better driving behavior. hugsim
The encoder comparison changes pretrained representations as well as objectives; it cannot isolate video pretraining alone. PCA panels use separate target-fitted bases, so cross-method colors are not directly comparable. Neither diagnostic establishes action-conditional causal fidelity. encoder-ablationpcatemporal
Reported tests cover pseudo-simulation and HUGSIM, without physical deployment or measured inference latency. Twelve sampling steps alone do not establish interactive deployment speed; the reported confidence interval does not include independent training runs. inferencesetuphugsimseeds
HUGSIM source exclusion is documented for WA-JEPA's two adaptation stages. The paper does not audit possible overlap with the upstream V-JEPA 2 pretraining corpus; source-disjoint adaptation therefore does not establish disjointness of every pretraining source. maskingclosed-protocol
7.2 Questions for discussion
- Does better target-relative temporal variation predict better planning across independently trained models?
- How much of the closed-loop comfort deficit can be changed without losing progress or collision performance?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs V-JEPA 2 initialization, nuPlan video preparation, NAVSIM splits and the specified distributed training setup. The text leaves stage-specific training duration, mask proportions and scheduling, predictor dimensions, EMA momentum, flow-time distribution, normalization constants, ego-state fields, loss weights and numerical integration solver insufficiently specified. The diagnostic's 0–36k averaging window does not supply a complete two-stage training schedule. maskingsetupimplementation-gapstemporal
Pin NAVSIM devkit 359c7f7 and HUGSIM ead17f2. Preserve metric-specific human-reference filtering, two-stage EPDMS aggregation and the stated HUGSIM averaging rule. Keep sampling seeds separate from retraining seeds and compare baselines within the same evaluator version. metricsclosed-protocolseeds
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does asymmetric gradient routing improve planning?
Reader-proposed experiment, not performed: initialize three Stage 2 runs from the same Stage 1 checkpoint and compare the published stop-gradient rule, unrestricted scene-loss gradients into action tokens, and blocked action-loss gradients into scene features. Resolve the missing numerical loss weights first, then hold them, data order, optimization budget and sampling protocol constant. Repeat independent training seeds and evaluate corrected NAVSIM-v2 EPDMS, comfort-related sub-metrics and future-latent error. The specific hypothesis is that allowing action supervision to shape scene features while protecting actions from scene loss improves planning. If the published rule fails to outperform both controls across training seeds, its claimed optimization role would need qualification. gradientjointsetupmetricsseedsimplementation-gaps
Check 2: Do training-time temporal gains survive the full sampler?
Reader-proposed experiment, not performed: retain the published joint-flow and joint-regression variants and a fixed held-out sample set. First reproduce Appendix E's target-selected locations and one-step training-time diagnostics. Then evaluate flow futures generated from pure Gaussian noise using the published twelve-step sampler, with no true-future interpolation supplied as input; compare them against EMA targets using the same metric definitions. Across checkpoints and noise seeds, also measure corrected EPDMS on the same evaluation set. A falsifiable expectation is that flow's lower directional gap and change ratio nearer one persist at inference and track planning gains. If the advantage disappears under pure-noise sampling, the training plots alone do not validate deployment-time future fidelity. flowinferencetemporaljoint-ablationmetricsseeds
8.3 Reading coverage
Visual audit: Visually inspected the title, fourteen authors, affiliations and v1 date on page 1; conceptual Figure 1 on page 2; all method, training, inference and hardware evidence on pages 3–5; Tables 1–4 and Figures 3–4 on pages 6–7; interpretation and conclusion on page 8; Appendix A–C protocols, evaluator commit, seed statistics and Tables 5–7 on page 11; qualitative Figures 5–6 and Appendix D metrics on pages 12–13; and the full Appendix E diagnostic protocol on pages 13–14. All six final original crops were viewed. The Table 1 crop retains its protocol and supervision note. Reference-only pages 9–10 were read as text but not rendered. Separate supplements, code, linked resources and other revisions were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Introduction (pp. 1–2)
- Related Work (pp. 2–3)
- Method: Preliminaries, Overview, Stage 1, Stage 2 and Planning inference (pp. 3–5)
- Experiments: setup, comparisons and ablations (pp. 5–8)
- Conclusion (p. 8)
- References (pp. 8–10)
- Appendix A: HUGSIM Closed-Loop Evaluation (p. 11)
- Appendix B: Qualitative Results (pp. 11–13)
- Appendix C: Additional Experimental Details (p. 11)
- Appendix D: NAVSIM Evaluation Metrics (pp. 11–13)
- Appendix E: Temporal Representation Metrics (pp. 13–14)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Only the supplied arXiv:2608.20974v1 dated 21 August 2026 was reviewed; no other revision or edition was supplied. Its title and fourteen-author list match the catalog; no identity discrepancy was observed.
- Acquisition note: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This gap was addressed by inspecting all six figures, all seven tables and relevant equation pages, including the preliminary extraction ambiguities in Tables 1 and 4.
- Separate supplemental material availability has not been fully verified.
- Code and linked resources were not inspected; no experiments were reproduced. Reference-list text was read, but the cited works were not independently reviewed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin
Title and all fourteen authors match the catalog after name-order normalization; the artifact is arXiv:2608.20974v1, 21 August 2026. Five affiliations are printed.
Go to primary source ↓rationalePDF pp. 1–3, Introduction, Figure 1 and Related Work
The authors motivate future masking, generative latent prediction and joint trajectory prediction as extensions of video JEPA representation learning.
Go to primary source ↓maskingPDF pp. 3–4, Stage 1; Eqs. (4)–(7); Figure 2
A shared ViT-L independently encodes camera videos. Full-mask hides all future patches; Patch-mask exposes some. An EMA target encoder supplies unmasked future targets.
Go to primary source ↓flowPDF p. 4, flow-based latent future prediction and training objective; Eqs. (7)–(10)
Gaussian noise and target latents are linearly interpolated at flow time t; an MMDiT-style predictor estimates clean future tokens using an MSE objective.
Go to primary source ↓jointPDF pp. 4–5, Stage 2; Eqs. (11)–(15); Figure 2
Normalized noisy future actions, historical actions and ego state form action tokens. Scene and action outputs are streams of the same joint predictor; Stage 2 uses Full-mask only.
Go to primary source ↓gradientPDF p. 5, Eq. (13); p. 11, Appendix C, stop-gradient design
Scene-loss gradients are blocked at the action-token interface. Action supervision can update differentiable scene representations through joint interactions.
Go to primary source ↓inferencePDF p. 5, Planning inference and Eq. (16); p. 11, Appendix C
Both future streams start from Gaussian noise. Clean endpoints are converted to flow velocities and integrated in twelve sampling steps; actions are denormalized.
Go to primary source ↓setupPDF p. 5, Experimental Setup, datasets and implementation details
Stage 1 uses nuPlan and Stage 2 NAVSIM navtrain. Four historical frames from four cameras at 256×512 yield eight actions at 2 Hz; optimizer, GPU counts and learning rates are specified.
Go to primary source ↓navsim2PDF p. 6, Table 1, WA-JEPA, SparseDriveV2 and Discrete-WAM rows; caption
Corrected EPDMS is 91.7, 90.1 and 90.4 respectively; WA-JEPA's pre-fix EPDMS* is 88.0. The dagger denotes auxiliary simulator-derived supervision.
Go to primary source ↓metricsPDF pp. 12–13, Appendix D, Eqs. (17)–(21) and EPDMS* versus EPDMS
PDMS combines penalties with progress, TTC and comfort. Corrected EPDMS applies a metric-specific human-reference filter and two-stage aggregation using NAVSIM devkit commit 359c7f7.
Go to primary source ↓navsim1PDF p. 7, Table 3, WA-JEPA and DriveWorld-VLA rows
NAVSIM-v1 navtest PDMS is 91.8 for WA-JEPA and 91.3 for DriveWorld-VLA.
Go to primary source ↓hugsimPDF p. 7, Table 2, overall, comfort and difficulty rows
On 436 scenarios WA-JEPA scores 0.4462 HD-Score against DrivoR's 0.3252. Comfort is 0.6620 versus VAD's 0.9534; extreme HD-Score is 0.1362 versus DrivoR's 0.1407.
Go to primary source ↓closed-protocolPDF p. 11, Appendix A, Protocol, Aggregation robustness and Tables 5–6
HUGSIM commit ead17f2 fixes trajectory-to-heading coordinate order. Methods share controller and scenarios but retain native sensors. WA-JEPA uses none of the four source datasets in training and leads under all three aggregations.
Go to primary source ↓seedsPDF p. 11, Table 7 and Appendix C, inference details
Ten noise seeds with fixed model parameters give EPDMS mean 91.7014, SD 0.0531 and 95% t-confidence interval [91.6634, 91.7393]; deterministic models are evaluated once.
Go to primary source ↓encoder-ablationPDF p. 7, Table 4(a) and Vision encoder initialization
Without Stage 1 adaptation, V-JEPA 2 scores 89.5 EPDMS, MAE and DINOv3 83.8, and SigLIP2 83.1 under a common Stage 2 protocol.
Go to primary source ↓mask-ablationPDF p. 7, Table 4(b); p. 8, Stage 1 masking strategies
No Stage 1 scores 89.5; Patch-mask alone 91.0; Full-mask alone 91.3; combined masks 91.7 EPDMS.
Go to primary source ↓joint-ablationPDF p. 7, Table 4(c) and caption; p. 8, Stage 2 scene–action coupling
Cascaded action-only, cascaded with flow futures, joint without explicit future supervision, joint regression and joint flow score 89.9, 90.8, 91.1, 90.7 and 91.7.
Go to primary source ↓pcaPDF p. 6, Figure 3 and caption; p. 8, Future representation analysis
Each method's PCA basis is fitted to its own EMA targets and applied to target and predicted features; each map covers two future frames. Regression predictions appear smoother.
Go to primary source ↓temporalPDF p. 7, Figure 4; p. 8, Future representation analysis; pp. 13–14, Appendix E, Eqs. (22)–(25)
Training-interval means are 0.10 versus 0.30 for directional gap and 0.80 versus 0.45 for change ratio. Diagnostics use 64 target-selected locations, four token steps and one-step flow x-predictions over 0–36k training.
Go to primary source ↓qualitativePDF p. 11, Appendix B; p. 12, Figure 5; p. 13, Figure 6
Selected HUGSIM rollouts illustrate turning, oncoming encounters and overtaking; NAVSIM examples compare predicted and reference trajectories.
Go to primary source ↓implementation-gapsPDF pp. 3–5, Method and Implementation details; p. 11, Appendix C
The specification names mask branches, action normalization, ego-state conditioning and loss weights without numeric mask schedules, normalization constants, ego-state fields or loss weights. It does not fully specify stage-wise training duration, predictor dimensions, EMA momentum, flow-time distribution or the integration solver.
Go to primary source ↓8.5 Primary sources
WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving ↗
PDF · 6,867 extracted words
Source fingerprint
3d239a0bbbfb5cf0307771c02b825bcee89a67b52c8600df072210256aaf23fa