ReWorld: Representation Learning for World Action Models
1. Paper overview
In one sentence: ReWorld explicitly trains a video-to-action representation pathway, improving reported video and simulated planning scores while requiring careful separation of curriculum effects from video self-guidance. e02e03e04e05e06e07e09e10e14
| At a glance | What to know |
|---|---|
| Research problem | Author claim Exposing video features to a planner does not ensure they predict dynamics or remain useful after cross-attention. Expert imitation also weakly distinguishes geometrically similar trajectories with different safety outcomes. The authors call this a representation bottleneck and target the intermediate world-to-action pathway. e02e03 |
| Core mechanism | |
| A key reported result | Closed-loop planning on NAVSIM Navtest: PDMS 90.4; components 99.1, 98.2, 97.7, 99.8, 82.0. PDMS ↑; NC, DAC, TTC, comfort, ego progress ↑. NAVSIM v1, approximately 12k Navtest scenes; camera-based methods on the same chained architecture; non-reactive simulation. DriveLaW: PDMS 89.1; components 99.0, 97.1, 96.7, 100, 81.3. A 1.3-point aggregate gain with slightly lower comfort; evidence concerns benchmark planning, not physical deployment. e08e10 |
| Reading caution |
Core contributions
Figure 1. Three training objectives shape the existing chained video-to-action interface. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the green video branch: the VAE supplies visual latents and the text encoder supplies motion-conditioned context. The upper auxiliary output head introduces intermediate supervision in addition to the final generation loss. Follow the vertical arrows into the blue Action DiT; these carry the intermediate features called F in the method text. The action branch also receives noisy trajectories, command and ego status. The yellow alignment block and SG marker indicate a stop-gradient grounding target. On the right, the candidate pool is scored and filtered to obtain a hard negative. Read these additions as a curriculum spread across three stages. e03e04e05e06e07
What it supports. The method changes how internal states are supervised while retaining separate video and action generators. Stage 1 shapes video features; Stage 2 freezes the video branch and grounds action states; Stage 3 lets behavior-oriented action gradients reach both networks. Access to a representation and supervision of that representation are distinct design choices.
Where the evidence stops. All losses appear together in the overview, but alignment and repulsion are not active together. The candidate assessment is offline training machinery. Planning uses first-step video features without decoding a future video or scoring candidate trajectories at test time.
2. Motivation
2.1 The problem and the proposed response
Exposing video features to a planner does not ensure they predict dynamics or remain useful after cross-attention. Expert imitation also weakly distinguishes geometrically similar trajectories with different safety outcomes. The authors call this a representation bottleneck and target the intermediate world-to-action pathway. e02e03
2.2 What this reading follows
A video generator can describe a plausible future without giving a planner the features it needs. ReWorld starts from a chained Video DiT and Action DiT and asks what should be learned along their connection. Its answer has three parts: make intermediate video states predict the future flow target, teach action states to retain attended video information, and use nearby low-scoring trajectories to refine behavior. Read the architecture alongside the ablations: the largest video gain includes a sampling change, while the planning gain comes from sequential training. The evidence supports improved benchmark performance, with unresolved questions about checkpoint continuity, causal attribution and reactive driving. e02e03e04e05e06e07e09e10e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
Separate Video and Action DiTs support Dual-system despite joint Stage 3 gradients. IDM is plausible only as a broad latent-world-to-action label: the source specifies conditional trajectory flow from first-step video features, not an explicit inverse map from observed transitions or decoded futures. The architecture is supported, while the Q4 prediction paradigm remains unclear. Offline simulator-score mining is not reinforcement learning. e01e03e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow the features before following a generated future
The central interface is F, a collection of Video DiT block activations passed to a separate Action DiT. The paper's phrase mid-denoising states can be misleading unless read alongside Section 3.1: the planner extracts them at the first discrete video denoising step, at maximum noise, rather than waiting for a completed future video. During planning, one Video DiT pass produces these activations and the Action DiT reuses them across five flow-matching steps. Ego kinematics and navigation command also enter the action model as structured context. The returned object is a waypoint trajectory. This information flow explains why improving a video-trained representation could help planning without paying for full video decoding. It also distinguishes two connected networks from a single shared-backbone model. e03e07
5.2 Separate future-flow supervision from the sampling correction
Stage 1 requires an intermediate Video DiT head to solve the same velocity regression problem as the final head. The target comes from the generator's own clean latent and sampled noise, so no external representation teacher supplies semantics. The resulting intermediate and final predictions also provide two estimates for self-guidance. When γ is one, the sampler uses the final estimate; at the default 1.4, it extrapolates along the difference from the intermediate estimate to the final one. A reader's causal interpretation must separate these two interventions. The supervised model reaches FVD 78.9 under standard sampling, compared with the baseline's 81.3; the full 61.9 requires self-guidance. The convergence curve additionally concerns optimization steps to comparable validation quality, not a measured twofold wall-clock speedup. e04e09e14e19
5.3 Ground action states, then reshape them around poor neighbors
Stage 2 aligns an action token's post-attention state with the video information it just retrieved. Stop-gradient makes that readout a fixed target within the optimization step, even though future readouts can change with the action model's attention. Stage 3 deliberately changes the optimization setting: remove alignment, unfreeze video parameters and use offline hard negatives. A negative must score below 0.6 and be the nearest such candidate to the expert after trajectory normalization. The planner's instantaneous clean estimate is repelled from that neighbor in delta space while expert flow matching remains active. Gradients can now reach both branches. Table 8 illustrates why this is a weak regularizer: excessive repulsion damages PDMS. Reader analysis: the progressive description leaves Stage 2's stated DriveLaW initialization insufficiently reconciled with Stage 1. e05e06e07e16
5.4 Training and inference
During training
Video data use nuPlan/nuScenes at 8 Hz; NAVSIM supervision uses 2 Hz observations. Baseline pretraining progresses from 740×352×121 to 1280×704×25 clips. Stage 1 initializes from LTX-Video: 20k steps, batch 64, AdamW learning rate 10⁻⁵, weight decay 0.05. e07e08
Stage 2 freezes video parameters for 6k steps, batch 128, alignment weight 0.05. Stage 3 updates both branches for 10k steps, batch 160, repulsion weight 0.04; alignment is removed and online video features remain attached to action-loss gradients. e05e06e07
Checkpoint continuity is ambiguous: the method describes a progressive curriculum, but Section 4.1 says Stage 2 initializes from DriveLaW rather than explicitly from Stage 1. No undocumented checkpoint merge is assumed. e06e07
During inference
Planning samples a Gaussian video latent, runs Video DiT once for F, then generates a trajectory in five action flow-matching steps without decoding future frames. Video generation separately uses 30 sampling steps and guidance scale 1.4. The reported method uses neither reinforcement learning nor test-time trajectory scoring. e01e03e07
The output is a plan evaluated in simulation. No physical actuator controller or on-road execution study is established; self-guidance refines video sampling without changing the first-step planner interface. e03e04e08
5.5 Implementation flow
- Connect separate video and action generators
The backbone combines a 2B Video DiT and 133M Action DiT. Features F come from the first discrete video denoising step, at maximum noise. Cached means reused activations within the current graph across action-flow steps, not an offline feature database. e03
- Supervise the video interior
An auxiliary head at block 8 predicts the same noise-minus-clean-latent velocity target as the final head, directly constraining intermediate features. Training supervision and inference self-guidance are distinct mechanisms. e04
- Retain the attended world readout
At Action DiT cross-attention layer 12, cosine alignment compares post-attention states with stop-gradient attended video readouts. Video parameters are frozen, but readouts are recomputed and depend on current action-side attention weights. e05
- Repel nearby poor trajectories
Score 64 candidates per training scene offline with NAVSIM PDM. Among scores below 0.6, select the nearest to the expert in normalized waypoint space. Repel the clean trajectory estimate using negative mean absolute distance over normalized position increments and sine/cosine yaw. Expert flow matching anchors the prediction; invalid-negative scenes contribute no repulsion. e06e07
6. Experiments & results
ReWorld trains the internal connection between a video generator and a trajectory planner. It supervises intermediate video states, aligns action states with attended video information, then fine-tunes both networks using nearby low-scoring trajectories. Reported gains cover generated video, simulated driving and frozen action-recognition features. The strongest video result combines representation training with changed sampling; the driving result concerns non-reactive NAVSIM simulation.
6.1 Read the original evidence
Table 1. ReWorld improves both video metrics; its strongest FVD includes self-guided sampling. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read FID and FVD as different evaluations of generated imagery: the paper associates FID with frame fidelity and FVD with video quality and temporal coherence. Begin with the last two rows because DriveLaW supplies the same chained architecture before ReWorld's representation curriculum. The ReWorld row uses intermediate supervision and guidance scale 1.4 with 30 sampling steps. Section 4.2 gives the missing bridge between those rows: using intermediate supervision with standard sampling, guidance scale 1, produces FVD 78.9. Keep this validation table separate from the 120k-step, no-text experiment in Table 4, whose protocol and FVD scale differ. e07e08e09e12e14
What it supports. The full method lowers FVD from 81.3 to 61.9, a reported 23.9% relative reduction, while FID shifts from 4.6 to 4.4. The intermediate-only result of 78.9 shows that the headline FVD gain cannot be assigned entirely to training: self-guided sampling contributes substantially.
Where the evidence stops. The table shows point estimates without statistical uncertainty. These distributional image/video metrics do not certify driving safety or physically correct dynamics. The strongest same-architecture comparison still changes both representation learning and sampling.
Table 2. NAVSIM planning gains improve the aggregate score, with a small comfort tradeoff. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First check the Image and Lidar columns before comparing methods: this table mixes sensor configurations. Then compare the adjacent DriveLaW and ReWorld rows, which use the same chained architecture. NC is no-at-fault collision, DAC is drivable-area compliance, TTC is time-to-collision, Comf. is comfort, and EP is ego progress. All columns point upward. PDMS aggregates these components; the metric definition on page 6 gates its weighted terms by collision and drivable-area compliance. The retained dagger note identifies a baseline trained with the same flow-matching objective. NAVSIM Navtest is a non-reactive simulation benchmark, not an on-road vehicle trial. e08e10
What it supports. ReWorld reaches 90.4 PDMS versus 89.1 for DriveLaW. DAC rises from 97.1 to 98.2 and TTC from 96.7 to 97.7, while NC changes from 99.0 to 99.1. The gain is not uniform: comfort falls from 100 to 99.8, even as ego progress improves from 81.3 to 82.0.
Where the evidence stops. An aggregate score can conceal component tradeoffs, and this table supplies no uncertainty estimates. Cross-method rankings also mix modalities and training histories. Non-reactive simulation limits conclusions about interactions with traffic that responds to the ego vehicle.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Closed-loop planning on NAVSIM Navtest NAVSIM v1, approximately 12k Navtest scenes; camera-based methods on the same chained architecture; non-reactive simulation. | PDMS 90.4; components 99.1, 98.2, 97.7, 99.8, 82.0. PDMS ↑; NC, DAC, TTC, comfort, ego progress ↑ | DriveLaW: PDMS 89.1; components 99.0, 97.1, 96.7, 100, 81.3. A 1.3-point aggregate gain with slightly lower comfort; evidence concerns benchmark planning, not physical deployment. e08e10 |
| Future video generation on nuScenes validation Table 1 validation protocol; intermediate supervision plus 30 sampling steps with γ=1.4. | FID 4.4; FVD 61.9. FID ↓; FVD ↓ | DriveLaW 4.6/81.3; supervised ReWorld with γ=1 has FVD 78.9. The reported 23.9% relative FVD reduction combines training and inference changes. Video distribution scores do not measure executed driving. e07e09e14 |
| Frozen action-recognition transfer on UCF-101 split 1 33-frame 224×224 clips; frozen final block of the 28-layer Video DiT, token mean pooling, common linear-classifier protocol. | 80.2%. Top-1 accuracy (%) ↑ | LTX-Video 66.8%; DriveLaW 68.3%; ReWorld Stage 1 71.7%. An 11.9 percentage-point gain over DriveLaW supports feature transfer, without isolating the cause of the later-stage gain. e07e11 |
| Controlled driving-video representation learning Separate 120k-step protocol: nuPlan/nuScenes, 224×224×25 clips, batch 32, no text encoder; nuScenes test FVD. | FVD 270.4; per-step video cost 1.003× at batch 1. FVD ↓; normalized video training cost | Vanilla Flow FVD 304.1 and cost 1.0×; Self-Flow FVD 283.3. Figure 2 reports roughly half the steps to comparable validation quality. This FVD protocol differs from Table 1. The 0.3% increment is video-side per-step cost, not total curriculum or mining cost. e07e12e13e19 |
| Planning curriculum and regularizer ablations NAVSIM Navtest, same architecture; Stage 3 weight sweep starts from the best Stage 2 checkpoint. | Align only 89.5; RDE only 89.8; sequential ReWorld 90.4. PDMS ↑ | Baseline 89.1. RDE weight 0.04 gives 90.4; 0.10 gives 85.5. The stages appear complementary; excessive repulsion competes with imitation. Table 7 checkmarks do not mean simultaneous losses. e06e15e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 6(a)–(b). Layer choice and sampling strength both matter, and more guidance is not consistently better. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top panel as a comparison of where the auxiliary velocity head supervises the Video DiT. Block 8 gives the lowest listed FVD; moving the head earlier or deeper changes the representation exposed to later blocks. The lower panel fixes block 8 and changes the self-guidance scale. The sampling rule combines intermediate and final velocities as v_w = v_i + γ(v_f − v_i). At γ=1 it reduces to the final prediction. Follow the lower panel from 1.0 through 1.8 rather than assuming a monotonic benefit from stronger correction. This panel concerns video generation, not an action-guidance controller. e03e04e14
What it supports. Block 8 achieves FVD 61.9, compared with 65.5 at block 2 and 64.3 at block 20. Guidance 1.4 improves on 1.0's 78.9, but 1.6 and 1.8 worsen to 69.7 and 68.2. The evidence favors a moderate sampling correction within the tested range.
Where the evidence stops. The two panels are separate sweeps, not a full layer-by-guidance factorial experiment. They establish neither a universally optimal block nor statistical significance between nearby values. Planning continues to use the first discrete video-step features.
Table 7. Alignment and later behavior-aware shaping improve the same planning baseline. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use DriveLaW's first row as the reference point. The next two rows add the alignment or repulsive-distance objective separately; the final row records the progressive ReWorld configuration. The checkmarks summarize which mechanisms were used over training, not which losses coexist in a single optimization step. Section 3.4 freezes Video DiT during alignment, creating a stable representation space for the action model. Section 3.5 then removes alignment, unfreezes video parameters and applies RDE alongside expert flow matching. Consequently, the final row should be understood as sequential grounding followed by behavior-oriented adaptation. e05e06e07e15
What it supports. Alignment alone raises PDMS from 89.1 to 89.5; RDE alone reaches 89.8. Their sequential use reaches 90.4, higher than either isolated addition. This is evidence that the two stages can work together on this backbone, consistent with their different targets along the world-to-action pathway.
Where the evidence stops. The table does not by itself separate objective choice from extra optimization steps or changed freezing schedules. It also lacks nearest-versus-random negative selection controls and run-to-run variability, so the mechanistic interpretation remains broader than the measured comparison.
Table 8(a)–(b). The auxiliary objectives help only within a balanced training regime. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The upper panel is Stage 2 only, with no Stage 3: vary the alignment weight and look for the largest PDMS. Its best setting, 0.05 with PDMS 89.5, supplies the starting checkpoint for the lower panel. The lower panel therefore asks a different question: how much repulsion should be added after grounding has already been learned? Compare weights within each panel rather than treating their rows as one common initialization experiment. In Stage 3, expert flow matching attracts predictions toward the demonstration while RDE pushes them away from a nearby low-scoring trajectory. e06e15e16
What it supports. Alignment peaks at 0.05 in the reported sweep; larger values fall below the baseline score. Stage 3 peaks at RDE weight 0.04 with 90.4 PDMS, then drops to 85.5 at 0.10. This matches the paper's warning that repulsion can compete with imitation when weighted too strongly.
Where the evidence stops. These are finite sweeps without uncertainty estimates. RDE has no lower bound in isolation, and simulator scores select training negatives rather than impose a hard safety constraint. The best tested weights do not establish robustness across datasets or normalization choices.
7. Analysis & limitations
7.1 What the evidence leaves open
NAVSIM is non-reactive and illustrated futures cover three seconds. Selected qualitative successes do not establish reactive traffic handling, calibrated physical dynamics or long-horizon reliability. The conclusion proposes longer horizons and multimodal extensions. e08e17e18
Reported tables omit seed variability and confidence intervals. Cross-paper comparisons differ in sensor inputs, and the controlled video protocol differs from the main one. Additional fine-tuning and freezing changes complicate causal attribution even within the same architecture. e07e09e10e12e15e16
RDE is unbounded below alone and is intended as a weak regularizer beside quadratic imitation. Table 8 shows weight sensitivity. Nearest-negative selection is motivated but not compared experimentally against random low-scoring negatives. e06e15e16
7.2 Questions for discussion
- Would planning gains survive equal training budgets and identical checkpoint provenance?
- Does nearest-negative mining beat random low-scoring negatives with matched score distributions?
- Do feature gains predict performance in reactive traffic or longer planning horizons?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the specified backbones/data, train-only PDM candidate mining, identical normalization for mining and training, and the frozen probe protocol. A proposed first check holds checkpoints, validation clips, noises and sampling steps fixed while sweeping self-guidance. e03e07e14
Missing details include λMid, numerical trajectory length L and normalization constants, candidate guidance/noise scales, exact probe timestep/classifier schedule, and hardware/runtime. Resolve Stage 2 checkpoint provenance. A proposed second check compares nearest and random low-scoring negatives under equal fine-tuning budgets. e03e04e06e07e11
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate supervision gains from self-guided sampling
Reader-proposed check, not executed: use the same data split, model size, optimizer budget, validation clips and initial noise seeds for a baseline and an intermediate-supervised model. Resolve the missing λMid before training. Evaluate both with the final-head sampler; then evaluate the supervised checkpoint at γ=1.0, 1.2, 1.4, 1.6 and 1.8 using 30 steps. Report FID/FVD with variation across seeds and convergence versus both steps and wall time. If the full improvement appears only after changing sampling, attribution to representation training alone is falsified; a consistent standard-sampling gain would support its independent contribution. e04e07e09e14e19
Check 2: Test whether geometric proximity makes the negative useful
Reader-proposed check, not executed: start every arm from the same verified Stage 2 checkpoint and share the same train-only 64-candidate pools, normalization, PDM threshold, unfreezing policy and fine-tuning budget. Compare no RDE, nearest low-scoring negatives, and randomly selected low-scoring negatives; add a score-matched random control to separate geometric proximity from severity. Track valid-negative coverage, expert trajectory error, Navtest PDMS and its safety/comfort components across seeds. A stable nearest-negative advantage would support the local ambiguity argument. Equal or worse performance than score-matched random negatives would weaken that explanation even if RDE still beats no RDE. e06e07e10e15e16
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, all method equations on pp. 4–6, setup on p. 7, Figures 1–5 and Tables 1–8. Six original crops were inspected individually: Figure 1 and Tables 1, 2, 6, 7 and 8. Narrow tables were rendered at 400 DPI rather than upscaled. Table 2 retains its caption because it contains the dagger footnote needed to interpret a baseline. All method/protocol pages supporting the visual explanations are declared. References and biographies on pp. 12–15 were read in the complete text chunks; their page images were not inspected. No separate supplement, code or experiment execution was included.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/author block (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work, 2.1–2.3 (pp. 2–3)
- 3 Method, 3.1–3.6, Eqs. (1)–(21) (pp. 3–6)
- 4 Experiments, 4.1–4.5, Eqs. (22)–(23) (pp. 6–11)
- 5 Conclusion (p. 11)
- Acknowledgments and all references (pp. 12–14)
- Author biographies (pp. 14–15)
Outside the original text pass
- Version scope: the title page identifies arXiv:2606.27504v2 [cs.CV], 24 August 2026, submitted to IEEE TPAMI. Title and all eleven authors match the catalog. No earlier edition was supplied, so differences from v1 cannot be established; submission does not establish acceptance.
- The catalog affiliation string includes an IEEE Publication Technology Department entry unsupported by the inspected author block; verified metadata includes only HUST and Xiaomi EV.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting PDF pp. 1–11, all five figures, all eight tables and every final crop.
- Separate supplemental material availability has not been fully verified.
- No separate supplement was supplied. Code was not inspected and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block, arXiv margin and abstract
Title and eleven authors match; arXiv:2606.27504v2 is dated 24 Aug 2026, submitted to IEEE TPAMI. Affiliations are HUST and Xiaomi EV. Abstract states no RL or test-time scoring.
Go to primary source ↓e02PDF pp. 1–2, Section 1, bottleneck and contributions
Authors motivate future-predictive video features, retained attended information and distinction of nearby unsafe trajectories.
Go to primary source ↓e03PDF pp. 3–4, Section 3.1, Eqs. (1)–(7); p. 2, Figure 1
Separate 2B Video/133M Action DiTs use VAE/T5 and driving context. Action flow predicts normalized L×3 waypoints from first-step video features; planning uses one video pass without decoding future frames.
Go to primary source ↓e04PDF p. 5, Section 3.3, Eqs. (8)–(11)
Block 8 uses the final flow target; self-guidance combines intermediate and final velocities only at sampling. FVD 61.9 includes both mechanisms.
Go to primary source ↓e05PDF p. 5, Section 3.4, Eqs. (12)–(14)
Cosine alignment targets post-attention states at layer 12 with stop-gradient video readouts. Video parameters are frozen; attention-dependent targets are recomputed.
Go to primary source ↓e06PDF p. 6, Sections 3.5–3.6, Eqs. (15)–(21)
Hard negatives are nearest normalized candidates scoring below 0.6. RDE uses negative mean absolute delta-space distance on a clean estimate; empty valid sets give zero. Stage 3 jointly updates branches without alignment.
Go to primary source ↓e07PDF p. 7, Section 4.1, backbone/curriculum/mining/sampling/probing paragraphs
Specifies clip shapes, LTX initialization, 20k/6k/10k steps, 64/128/160 batches, optimizer, alignment 0.05, RDE 0.04, 64 candidates, 30 video/five action steps. Stage 2 explicitly starts from DriveLaW; several implementation constants are absent.
Go to primary source ↓e08PDF p. 6, Section 4.1, tasks/datasets/metrics and Eq. (22)
NAVSIM v1 is non-reactive, approximately 103k/12k train/test scenes. Video uses 8 Hz; trajectory observations 2 Hz. PDMS combines collision, drivable-area, progress, time-to-collision and comfort.
Go to primary source ↓e09PDF p. 7, Table 1, DriveLaW/ReWorld rows; Section 4.2 video paragraph
nuScenes validation FID/FVD: DriveLaW 4.6/81.3, ReWorld 4.4/61.9. Supervised ReWorld at γ=1 has FVD 78.9; reported full relative reduction is 23.9%.
Go to primary source ↓e10PDF p. 8, Table 2, headers and DriveLaW/ReWorld rows
Navtest PDMS 89.1 versus 90.4; NC/DAC/TTC/comfort/EP 99.0/97.1/96.7/100/81.3 versus 99.1/98.2/97.7/99.8/82.0. Sensor columns differ; dagger marks the same flow objective.
Go to primary source ↓e11PDF p. 8, Table 3 and Section 4.3; p. 7, probing and Eq. (23)
UCF-101 split 1 uses frozen final-block pooled features from 33-frame 224×224 clips: LTX 66.8%, DriveLaW 68.3%, Stage 1 71.7%, full ReWorld 80.2%.
Go to primary source ↓e12PDF p. 9, Table 4; p. 7, unified protocol
Separate 120k-step, batch-32, 224×224×25 no-text experiment reports nuScenes test FVD: Vanilla 304.1, Self-Flow 283.3, ReWorld 270.4.
Go to primary source ↓e13PDF p. 9, Table 5, caption and ReWorld row
Video-side per-step cost at 224×224×25 and batch 1: ReWorld 1.003× versus vanilla 1.0×.
Go to primary source ↓e14PDF p. 10, Table 6(a)–(b); p. 9, Section 4.4
Blocks 2/8/12/16/20 yield FVD 65.5/61.9/62.7/63.0/64.3. At block 8, γ=1.0/1.2/1.4/1.6/1.8 gives 78.9/72.0/61.9/69.7/68.2.
Go to primary source ↓e15PDF p. 11, Table 7; pp. 9–10, Section 4.4
Baseline/alignment-only/RDE-only/ReWorld yield Navtest PDMS 89.1/89.5/89.8/90.4. Alignment-only freezes video; the combined method is progressive.
Go to primary source ↓e16PDF p. 11, Table 8(a)–(b) and caption; p. 10, Section 4.4
Stage 2 alignment weights 0.01/0.03/0.05/0.07/0.10 give 88.8/89.2/89.5/88.2/87.7. From its best checkpoint, RDE 0.02/0.03/0.04/0.05/0.10 gives 89.4/89.6/90.4/89.7/85.5.
Go to primary source ↓e17PDF p. 8, qualitative protocol; pp. 9–11, Figures 3–5 and captions
Video examples show endpoints of one-second history and selected three-second future frames. Planning examples show red predictions and green experts for straight, left, right and intersection scenes.
Go to primary source ↓e18PDF p. 11, Section 5, final paragraph
Longer temporal horizons and multimodal scenarios are proposed as future work.
Go to primary source ↓e19PDF p. 3, Figure 2(a)–(b) and caption; p. 9, Section 4.4
The figure describes self-guidance and roughly twofold convergence acceleration at comparable validation FVD. The horizontal axis is optimization steps, not wall-clock time.
Go to primary source ↓8.5 Primary sources
ReWorld: Representation Learning for World Action Models ↗
PDF · 10,872 extracted words
Source fingerprint
ac7653eb88f7c681815b5397af28a84066cb0d713cf86fec8a88e3e31a643f96