WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory
1. Paper overview
In one sentence: WorldScape Policy 2.0 turns event captions and retrieved task history into conditions for a shared video-action model, improving measured control while leaving clean-only transfer and implementation completeness as important limits. e02e03e04e05e06e14e16e18
| At a glance | What to know |
|---|---|
| Research problem | Source description A visually similar scene can require different actions depending on completed subtasks. The authors argue that short observation windows miss this progress, episode-level captions weakly identify atomic actions, and text-only interfaces exclude useful goal or demonstration evidence. Their target is a controllable policy that retains local dynamics while retrieving semantic task history. e02 |
| Core mechanism | |
| A key reported result | RoboTwin 2.0: standard 50-task manipulation: Clean 94.3%; randomized 94.2%; reported average 94.3%. Task success rate. Clean-plus-randomized training; 50K fine-tuning steps; 100 trials per task under each evaluation setting. Average: Abot-M0.5 94.1%, Fast-WAM 91.9%, π0.5 79.8%. Highest tabulated average, but only 0.2 percentage points above the nearest baseline; uncertainty is absent. e13e14 |
| Reading caution | Reader analysis No confidence intervals or seed variability accompany the result tables. Real-world evidence spans five tasks on one dual-arm platform, and semantic forcing is tested through a small aggregate increment. These results do not establish open-ended planning or explain every gain causally. e13e14e16e17e18e19 |
Core contributions
Figure 3. Event memory supplies a latent subgoal; recent visual memory supplies interaction context. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left: fine-grained text goes through T5, whereas the head-view image and high-level task enter the VLM. In the center, current reasoning tokens supply the query and historical views supply keys and values. The retrieved signal is gated, multiplied and added back to the current tokens, matching Equation (14). The bottom switch selects T5 embeddings or memory-enhanced tokens for DiT conditioning. Semantic forcing is a training alignment arrow, not an additional autonomous inference input; Equation (16) stops gradients through the fixed T5 target. On the right, noisy video and action tokens share DiT blocks. Follow the green feedback arrow from action execution to real observations, which refresh the visual context. e03e04e05e06e07e13
What it supports. The mechanism separates deciding the next subgoal from preserving recent dynamics while coupling both to one video-action predictor. Equations (12)–(14) additionally retrieve from the compact full-history bank, beyond the three named views emphasized in this drawing. The predicted actions are executed before real observations update the next control cycle.
Where the evidence stops. Figure 3 labels the VLM Qwen3.5; Section 4.2 specifies Qwen3-VL-4B. That discrepancy is unresolved. The faithful graphic is retained, while the report uses the implementation text for the stated model configuration.
2. Motivation
2.1 The problem and the proposed response
A visually similar scene can require different actions depending on completed subtasks. The authors argue that short observation windows miss this progress, episode-level captions weakly identify atomic actions, and text-only interfaces exclude useful goal or demonstration evidence. Their target is a controllable policy that retains local dynamics while retrieving semantic task history. e02
2.2 What this reading follows
A robot may see almost the same tabletop before and after completing a subtask, yet need to act differently. WorldScape Policy 2.0 addresses that ambiguity through two kinds of history: recent visual latents preserve motion, while VLM event representations carry evidence about task progress. Its other central idea is to teach autonomous planning with the same event semantics that directly control fine-grained actions. The six visuals below trace this information flow and examine its measured effects. Keep the evaluation protocols separate: the headline RoboTwin table includes randomized training, the ablations use clean-only training, and the PiPER tables measure executed real-robot behavior under different prompt interfaces. e02e03e04e05e06e14e16e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
One Model × Joint prediction is supported at the video-action backbone: modalities share DiT blocks, interact through attention and yield velocity fields at inference. Separate conditioning encoders and embodiment adapters remain. Event retrieval and latent subgoals support the memory categories; this is not inverse dynamics from a separately generated video. e03e05e06e07e13
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make an event the unit of supervision
The training unit is an aligned event rather than a long episode carrying one repeated caption. Robot motion and gripper signals first propose temporal changes; the annotation VLM then describes what happens inside those boundaries. Episode descriptions summarize behavior actually completed, and event captions identify an atomic action with its end effector, object and state change. The resulting sample can also carry an event-terminal goal image or a task-matched demonstration. This gives the same transition several ways to express intent. Stage 1 uses those conditions to train joint video-action flow prediction while recent visual history is already available. Table 1's 4.89 million segments describe the reported decomposed portion of a heterogeneous mixture; datasets without further decomposition retain single-segment episodes. The annotation procedure should therefore be reproduced alongside the model, not treated as interchangeable preprocessing. e08e09e10e11e12
Figure 4. Joint video-action attention remains compatible with causal observation history. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the matrix by choosing a query row and scanning across its permitted key/value columns. The p blocks are persistent visual context; z blocks are clean video; v and a blocks are noisy video and action. The clean-video region forms a causal staircase. In each illustrated noisy pair, video can attend to the corresponding action block and action to the corresponding video block, matching Section 4.2's bidirectional video-action interaction. Their access to clean observations respects the depicted temporal prefix. At the right, the heading says Auto OR Fine: the gradient-colored reasoning columns and gray T5 columns represent alternative cross-attention conditions. They should not be read as simultaneous language inputs. e04e05e06e13
What it supports. The model can jointly denoise video and actions without allowing future clean observations to answer the prediction problem. Persistent goal or demonstration tokens remain available throughout the sequence, while the selected semantic condition enters through cross-attention. This is an architectural specification; task success is established separately by the experiment tables.
Where the evidence stops. The two drawn reasoning columns are schematic: Section 4.2 specifies four generated planning tokens, alongside perception tokens. The diagram alone cannot verify an implementation's leakage behavior or establish runtime latency.
5.2 Turn explicit captions into an autonomous latent condition
At autonomous inference the user supplies a global task, not the next event caption. One VLM context encodes the head-view image and task prompt; greedy continuation produces planning tokens whose hidden states are concatenated with perception features. Current tokens then retrieve from compressed history and detailed recent/boundary anchors. A learned gate adds the retrieved information to those tokens. During Stage 2, a separate training target bridges this pathway to the controllable interface learned earlier: the event caption is encoded by fixed T5, pooled and normalized, while the reasoning summary is projected into the same space. Equation (16) aligns the two with stop-gradient on the T5 side. Thus semantic forcing supervises the latent condition without giving the autonomous policy an event caption at deployment. The source describes latent subgoal conditioning, not an externally executed textual plan. e04e05e06e12
5.3 Separate prediction, feedback and experimental attribution
The policy jointly predicts video and actions, but only execution followed by new observations closes its control loop. The action adapters supply embodiment-specific interfaces; integration still takes place in raw action coordinates, while video integration takes place in VAE space. Reader analysis: this distinction matters because good-looking predicted futures would not alone establish useful control. Here, the simulation and PiPER tables do report task success. They nevertheless answer different questions. Table 2 measures performance with randomized training, Tables 5–6 test additions under clean-only training, and Table 3 changes prompt interfaces across physical tasks. The progressive gains support the proposed ingredients in those settings, but do not allocate every improvement uniquely to memory semantics or planning. A reproduction should therefore match the protocol before testing an architectural explanation. e03e07e11e13e14e16e18e19
5.4 Training and inference
During training
ManipEvent-5M mixes real robots, simulation, UMI and egocentric human data. Motion/gripper signals propose event boundaries; Qwen3-VL supplies hierarchical semantics. Episode captions describe completed behavior in hindsight. Human hands are retargeted to gripper controls; demonstration pairs align task, ordered events and outcome. Reader analysis: constructed annotations are not independently verified semantic ground truth. e08e09e10
Stage 1 trains multimodal event control and short-term visual memory. Stage 2 adds VLM/event memory and semantic forcing on extended trajectories; captions supply a training target rather than the autonomous control input. Stage 3 adapts the task's interaction mode. Figures mark T5, VLM and VAE components frozen; the memory/projector and WAM pathways are trainable. e03e06e12
The DiT initializes from Wan2.2-5B. Section 4.2 specifies Qwen3-VL-32B annotation and Qwen3-VL-4B reasoning, a 320×160 head view, four recent visual chunks and eight history chunks by default. Pretraining uses batch 768 and learning rate 5×10⁻⁴; post-training uses batch 128, 6×10⁻⁵ and 50K steps. e13
During inference
Integrate video velocity in VAE space and action velocity in raw control space from flow time 1 (noise) to 0 (data), re-encoding the action estimate each step. Execute the chunk and update memory using real observations. Autonomous inference lacks the fine-grained caption/T5 target. Reader analysis: this is joint prediction with feedback, not a reported search over alternative imagined plans. e06e07e11
5.5 Implementation flow
- Choose the control interface
Fine-grained captions pass through T5. Autonomous mode uses a head-view image and task prompt in one VLM prefill, then greedily generates planning tokens and retains their final-layer hidden states. These join perception tokens; the implementation uses four planning tokens. The language conditions are selected by mode, not concatenated. e03e04e06e13
- Retrieve progress evidence
Historical perception tokens are attention-pooled into gist tokens while planning tokens remain. Retrieval combines the compact history bank with global-history slots, recent local-active chunks and sparse event-boundary chunks. Boundaries use changes in mean VLM representations with temporal separation; local and boundary anchors retain full tokens. Current tokens query this bank and receive a gated residual update. e04e05
- Preserve visual dynamics
Clean VAE latents of recent observations enter DiT self-attention. Goal/video prompt latents persist while recent history slides. The attention mask permits paired video-action interaction and access to available clean context without future-chunk leakage; semantic conditioning enters cross-attention. e05e13
- Predict executable chunks
The shared DiT produces video and action velocity fields. Embodiment-specific adapters map raw actions into tokens and back. Each arm uses chunk-relative 3D translation, continuous 6D relative rotation and an absolute gripper command. Flow matching and integration operate in raw action space, not on action embeddings. e03e07e11
6. Experiments & results
WorldScape Policy 2.0 combines a shared video-action diffusion transformer with recent visual history and retrieved event memory. Event captions directly steer fine-grained actions or supervise an autonomous latent subgoal pathway. The paper reports strong RoboTwin and PiPER execution results, but its headline simulation score uses randomized training; clean-only generalization is substantially weaker. The evidence below separates these protocols and the two instruction modes.
6.1 Read the original evidence
Table 2. The headline result measures performance after clean-plus-randomized training. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First establish the protocol from Section 4.1 and the table caption: methods are fine-tuned for 50K steps on clean-plus-randomized data and evaluated over 50 tasks, with 100 trials per task in each condition. Then compare rows within a column. WorldScape Policy 2.0's clean and randomized columns are nearly equal, but randomized training examples were available. Inspect the nearest average, Abot-M0.5 at 94.1%, as well as broader reference baselines such as Fast-WAM at 91.9%. The source's displayed averages are already rounded and are preserved as reported. The gray separators organize model families; they do not indicate different evaluation protocols. e13e14e15
What it supports. WorldScape Policy 2.0 reports the highest average, 94.3%, with a 2.4-percentage-point advantage over Fast-WAM and a 0.2-point advantage over Abot-M0.5. It ties Abot-M0.5 at 94.2% under randomization. This supports strong performance under the shared benchmark training protocol, with a small margin over the nearest competitor.
Where the evidence stops. No intervals or seed variability are reported, so the 0.2-point lead is not a demonstrated statistical separation. Figure 10's clean-only C2R average is 47.9%; it answers a different generalization question.
Table 3. Physical execution is evaluated through several distinct control interfaces. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Prompt / Context column before comparing success rates. Folding uses a global instruction and tests autonomous sequencing. The shell game supplies video context for tracking hidden state. Block stacking has separate goal-image and video-demonstration rows, which should not be merged. The last row evaluates the complete cleaning sequence driven by successive subtask captions. Section 4.1 specifies 20 trials per real-world task. Section 4.4 also says that π0.5 receives extended observation history for the shell game and demonstration-conditioned stacking, rather than being restricted to a single frame. Figure 11 on the same inspected page shows the corresponding PiPER setups and context/execution distinction. e13e16e17
What it supports. The proposed policy achieves 75% on both folding tasks and the shell game, 60% with goal-image stacking, 70% with video stacking, and 80% on full-sequence cleaning. All displayed comparisons favor it over the two baselines. These are reported robot episode outcomes, not video-quality scores or demonstrations of simulated success.
Where the evidence stops. The evidence covers five tasks on one dual-arm platform, with no uncertainty estimates. Table 4 separately resets scenes for atomic instructions; its 73.3% overall instruction success cannot replace the 80% full-sequence result here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0: standard 50-task manipulation Clean-plus-randomized training; 50K fine-tuning steps; 100 trials per task under each evaluation setting. | Clean 94.3%; randomized 94.2%; reported average 94.3%. Task success rate | Average: Abot-M0.5 94.1%, Fast-WAM 91.9%, π0.5 79.8%. Highest tabulated average, but only 0.2 percentage points above the nearest baseline; uncertainty is absent. e13e14 |
| RoboTwin 2.0: clean-to-randomized transfer Clean-only training; Figure 10 averages clean and randomized evaluation. | 47.9%. Average task success rate | Fast-WAM 39.1%; π0.5 37.5%; π0 31.4%. An 8.8-point advantage over Fast-WAM; this is not randomized-only success. e15 |
| PiPER: folding and sequential cleaning Dual-arm real robot; 20 trials per task; global instructions for folding, sequential captions for cleaning. | Clothes folding 75%; box folding 75%; full cleaning sequence 80%. Episode success rate | π0.5: 60%, 65%, 70%; DreamZero: 45%, 55%, 60%, respectively. Evidence for executed manipulation under the specified interfaces. e13e16 |
| PiPER: visual reasoning and block-stacking transfer 20 trials per task; video-conditioned shell game and goal/video-conditioned stacking. | Shell game 75%; stacking with goal image 60%, with video 70%. Episode success rate | π0.5: 30%, 10%, 20%; DreamZero: 50%, 20%, 20%, respectively. π0.5 receives extended observation history for temporal tasks; broad cross-embodiment generality remains untested. e13e16 |
| PiPER: reset-scene atomic instruction following Six independent cleaning instructions, 20 trials each; held-out categories are black pen, green tape and beige shoe. | In-domain average 86.7%; held-out average 60.0%; overall 73.3%. Instruction success rate | Overall: π0.5 63.3%, DreamZero 51.7%; held-out: 43.3% and 30.0%. Reset-scene atomic success cannot substitute for full-sequence success. e17 |
| RoboTwin 2.0: memory-component ablation Clean-only training; progressively add short-term visual memory, long-term event memory and latent subgoal reasoning. | 40.91% → 44.67% → 46.25% → 47.89%; full model: clean 69.74%, randomized 26.03%. Average task success rate | No-memory baseline average 40.91%. The ladder supports complementary gains, but is not a factorial test of all interactions. e18 |
| RoboTwin 2.0: curriculum and semantic-forcing ablation Clean-only training; Stage-3 post-training held fixed. | No Stages 1–2: 43.19%; Stage 1: 46.63%; Stages 1–2 without forcing: 47.30%; full: 47.89%. Average task success rate | Semantic forcing adds 0.59 percentage points to the preceding row. Stage 1 supplies the largest observed gain; small later increments lack uncertainty estimates. e18e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Recent visual memory, event memory and latent reasoning each improve the reported ladder. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Translate the headings before following the checkmarks: STM is short-term visual memory, LTM is long-term event memory, and LSR is latent subgoal reasoning. Each row adds one component to the previous row; it is not a table of every possible combination. Section 4.5 explicitly uses clean-only training, unlike Table 2. In the average column, adding STM changes 40.91% to 44.67%; adding LTM changes 44.67% to 46.25%; adding LSR reaches 47.89%. Inspect the randomized column separately: its final value is 26.03%, so the average should not be described as randomized-domain success. The clean score contributes substantially to that aggregate. e04e05e13e14e18
What it supports. The complete configuration gains 6.98 percentage points in average success over the no-memory row, calculated from the displayed values. The incremental gains are 3.76, 1.58 and 1.64 points. The table supports a benefit for each addition in this order, while preserving a substantial clean-to-randomized performance gap.
Where the evidence stops. The ladder does not isolate every interaction, memory-view choice or token-budget effect. Its 47.89% average must not be compared directly with Table 2's 94.3%, which uses additional randomized training data.
Table 6. Semantic forcing adds a small final increment after the two earlier training stages. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the caption's control: every row uses the same Stage-3 downstream post-training protocol. The first row omits Stages 1 and 2. The second introduces event-grounded Stage 1, the third adds memory-aware Stage 2 without semantic forcing, and the fourth enables SF. Section 4.5's clean-only setting applies here. The average column progresses from 43.19% to 46.63%, 47.30% and 47.89%. This ordering places the largest observed increment at Stage 1. For the final comparison, Stage 2 is present in both rows; the difference is the caption-alignment objective described in Equation (16), rather than access to fine-grained captions at autonomous inference. e06e11e12e18e19
What it supports. Semantic forcing raises the average by 0.59 percentage points, with clean success increasing from 68.95% to 69.74% and randomized success from 25.64% to 26.03%. The table is consistent with a benefit from aligning planning latents to event semantics, but the magnitude is modest relative to the Stage-1 gain.
Where the evidence stops. Fixed Stage-3 training does not equalize the total earlier training budget. No seed variability or shuffled-caption control is reported, leaving the statistical stability and semantic specificity of the SF increment unresolved.
7. Analysis & limitations
7.1 What the evidence leaves open
No confidence intervals or seed variability accompany the result tables. Real-world evidence spans five tasks on one dual-arm platform, and semantic forcing is tested through a small aggregate increment. These results do not establish open-ended planning or explain every gain causally. e13e14e16e17e18e19
Source discrepancies remain unresolved: Figure 3 labels the VLM Qwen3.5, whereas Section 4.2 specifies Qwen3-VL-4B. Figure 2 includes CLIP embeddings, while Equations (4)–(5) define VAE visual prompting. This report follows the explicit formulation and implementation text without correcting the graphics. e03e13
The full-history formulation and default eight-history-chunk implementation leave retention beyond that budget unclear. The reviewed paper has no dedicated limitations section and does not report control latency or compute cost. e04e13e19
7.2 Questions for discussion
- Does event memory help when current observations and recent visual history match across different completed-subgoal histories?
- Would aligned captions outperform shuffled semantic-forcing targets under equal training budgets?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the data mixture and sampling ratios, event segmentation/captioning, human retargeting, modality pairing, embodiment adapters and mode-specific masks. Segmentation is described, but numerical motion/gripper thresholds and merging settings are not supplied. e03e05e08e09e10
Missing details include Stage-1/2 step counts and Stage-2 hyperparameters, hardware/software versions, flow solver/step count, action horizon, world-loss weight, gist/anchor budgets and history eviction. Resolve the VLM-name discrepancy before choosing a checkpoint. Proposed memory/alignment checks should hold data and compute fixed. e04e05e07e11e12e13e18e19
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether event memory distinguishes matched observations
Reader-proposed check; not run. Construct paired manipulation episodes that reach matching current observations and matching recent visual windows but differ in completed subtasks and required next actions. Hold the instruction, visual buffer, backbone, action horizon and memory-token budget fixed. Compare intact event memory with episode-shuffled event memory and a zeroed retrieval residual. Measure next-subgoal selection, repeated/skipped subtasks and executed episode success across multiple seeds. Also retain a subset where history is unnecessary. The progress-memory explanation predicts a selective advantage for intact history on the ambiguous pairs. Similar performance with shuffled histories would weaken that explanation even if the unmodified model retains its aggregate benchmark score. e02e04e05e18
Check 2: Distinguish semantic alignment from an extra training signal
Reader-proposed check; not run. Start from identical Stage-1 weights and train Stage-2 variants with correctly aligned event-caption targets, within-task shuffled caption targets, or no semantic-forcing loss. Keep examples, optimizer steps, Stage-3 adaptation, frozen T5 encoder and all other losses identical; use the paper's 0.001 weight for both nonzero variants. Evaluate autonomous control without event captions on the same clean-only RoboTwin splits, reporting clean and randomized success separately across multiple seeds. Measure agreement with held-out event labels as a secondary diagnostic. A stable advantage for correctly aligned targets over both controls would support the intended semantics-transfer mechanism; an equal gain from shuffled targets would point toward a nonspecific training effect. e06e11e12e18e19
8.3 Reading coverage
Visual audit: The title/author page and every page supporting retained method, training, numerical, evaluation and reproduction claims were rendered and visually inspected. Figures 1–11 and Tables 1–6 were read in their original layouts. Six final original crops were separately viewed; the attention-mask crop was corrected to retain its complete query label and then viewed again. Query/key directions, residual gating, the Auto/Fine switch, stop-gradient in Equation (16), and Figure 4's mask were cross-checked against text and equations. Figure 3's Qwen3.5 label conflicts with Section 4.2's Qwen3-VL-4B; Figure 2 also depicts CLIP whereas Equations (4)–(5) specify VAE prompting. Both discrepancies are disclosed. All 24 pages were read as text; page 3 and reference pages 20–24 were not visually inspected. No appendix or separate supplement was supplied.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract (p. 1)
- 1 Introduction
- 2 Related Work
- 2.1 World Action Models for Robotic Manipulation
- 2.2 Long-Term Memory and Long-Horizon Planning
- 2.3 Fine-Grained Prompts for Steerable Policies
- 3 Method
- 3.1 Overview and Problem Formulation
- 3.2 Event-Grounded World Action Modeling
- 3.3 Reasoning-Augmented Long Short-Term Memory
- 3.4 Implicit Subgoal Latent Planning
- 3.5 ManipEvent-5M: Event-Based Multimodal Dataset
- 3.6 Training Objectives and Three-Stage Curriculum
- 4 Experiments
- 4.1 Benchmark Setup and Evaluation Protocol
- 4.2 Implementation Details
- 4.3 Simulation Benchmark Results
- 4.4 Real-World Evaluation Results
- 4.5 Ablation Study
- 5 Conclusion
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All eight supplied text chunks were read individually, covering the complete 24-page PDF, including references. The extraction-only figure omission was resolved by inspecting all original figures and tables in the PDF. No appendix is present.
- No separate supplement was supplied. Code and external links were not inspected; no experiments were reproduced.
- Identity/version note: the title and all 16 authors match the catalog. The inspected artifact is arXiv:2607.18840v1, stamped 21 July 2026; its title block is dated July 20, 2026. This one-day document/submission-date difference is preserved. No other revision or edition was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and arXiv margin stamp
Exact catalog title and 16-author continuity are verified. The document is dated July 20, 2026; the margin identifies arXiv:2607.18840v1 [cs.RO], 21 Jul 2026. Affiliations are Manifold AI, Tsinghua University and Shanghai Jiao Tong University.
Go to primary source ↓e02PDF pp. 1–2, Section 1 and Figure 1
The problem statement distinguishes limited progress memory, coarse language grounding and restricted text interfaces, motivating separate semantic and visual memory.
Go to primary source ↓e03PDF pp. 4–6, Figures 2–3; Section 3.1, Equations (1)–(3); Section 3.2, Equations (4)–(5)
The shared DiT uses mode-dependent language conditions and VAE visual prompts. Actions are 10D per arm, 20D dual-arm, with embodiment-specific interfaces. Figures mark pretrained encoders frozen. Figure 2 additionally depicts CLIP; Figure 3 labels the VLM Qwen3.5 and shows real-observation feedback.
Go to primary source ↓e04PDF pp. 6–7, Section 3.3, Equations (6)–(11)
One VLM prefill supplies perception hidden states and cached greedy planning-token generation. History is compressed into gist/planning tokens plus global, local and boundary views. Boundary selection uses cosine change and minimum temporal separation; local/boundary anchors keep full tokens.
Go to primary source ↓e05PDF p. 8, Section 3.3, Equations (12)–(15) and Figure 4
Current reasoning tokens query the concatenated memory views and compact history bank, then receive a token-wise gated residual. Clean visual history and persistent prompts enter causal self-attention. Figure 4 shows query rows, key/value columns, paired video-action attention and alternative Auto/Fine cross-attention conditions.
Go to primary source ↓e06PDF p. 9, Section 3.4, Equations (16)–(18)
Semantic forcing aligns normalized projected reasoning summaries to pooled frozen T5 targets with stop-gradient. Fine-grained captions are absent at autonomous inference. Language cross-attention selects T5 or memory-enhanced VLM tokens; shared DiT hidden states yield separate velocity outputs.
Go to primary source ↓e07PDF p. 10, continuation of Section 3.4 immediately below Table 1
Inference integrates video in VAE-latent space and actions in raw space from noise time 1 to data time 0, re-encoding raw actions at each step and updating memory with new observations.
Go to primary source ↓e08PDF p. 10, Table 1, Total and dataset rows; Section 3.5, Equation (19) and action canonicalization
Table 1 reports 512.14M frames, 4982.81 hours, 744.43K episodes and 4.89M decomposed segments. RoboMIND, RoboCOIN and DROID have dashes for segment counts and 100% single-segment ratios. Data mix human, UMI, simulation and robot sources; human hands are retargeted to gripper control.
Go to primary source ↓e09PDF pp. 9–12, Figures 5–6 and Section 3.5, text-prompt construction steps (1)–(4)
Motion/gripper signals establish candidate boundaries before VLM relabeling. Episode-level hindsight descriptions and event captions use hierarchical frame sampling, embodied prompts and gripper-view priority; invalid outputs have retry/fallback handling. Numerical segmentation thresholds are not given.
Go to primary source ↓e10PDF pp. 12–14, Section 3.5, goal-image/video-prompt construction and Figures 8–9
First-view goals come from event terminal observations; third-view goals come from task-matched human demonstrations. Cross-embodiment video pairs align semantic tasks, ordered events and outcomes. Cleaning and coffee examples illustrate aligned modalities.
Go to primary source ↓e11PDF p. 13, Section 3.6, Equations (20)–(21)
Joint loss includes action/world flow matching and semantic forcing weighted by 0.001 when active. Gaussian noise interpolates with clean action/video targets; action loss remains in raw space. A value for the world-loss weight is not stated.
Go to primary source ↓e12PDF pp. 14–15, Section 3.6, Stages 1–3 and Equations (22)–(23)
Stage 1 establishes event-conditioned control with recent visual history; Stage 2 introduces event memory and semantic forcing; Stage 3 adapts task-dependent interfaces. Stage-1/2 training durations and detailed Stage-2 configuration are not specified.
Go to primary source ↓e13PDF p. 15, Sections 4.1–4.2
The standard benchmark uses 50K fine-tuning steps and 100 trials per task over 50 tasks per condition; real tasks use 20 trials. Text specifies Qwen3-VL-32B annotation, Qwen3-VL-4B reasoning, four planning tokens, 320×160 head view, Wan2.2-5B, shared video-action attention, four recent chunks and eight history chunks. Batch sizes and learning rates are stated; hardware, software, solver, horizon and latency are not.
Go to primary source ↓e14PDF p. 16, Table 2, WorldScape Policy 2.0 / Abot-M0.5 / Fast-WAM / π0.5 rows, all columns
Clean-plus-randomized training yields WorldScape scores 94.3%, 94.2%, 94.3%. Average comparators are 94.1%, 91.9% and 79.8%, respectively. Reported rounded values are retained; no uncertainty is tabulated.
Go to primary source ↓e15PDF p. 16, Figure 10, C2R panel and caption; Section 4.3 final paragraph
C2R trains only on clean demonstrations and averages clean/randomized success: WorldScape 47.9%, Fast-WAM 39.1%, π0.5 37.5%, π0 31.4%. The standard panel uses clean-plus-randomized training.
Go to primary source ↓e16PDF p. 17, Figure 11, Table 3 all rows, and Section 4.4
PiPER episode success is 75% on each folding task, 75% shell game, 60%/70% goal/video stacking and 80% sequential cleaning. Table 3 provides π0.5 and DreamZero comparisons. π0.5 input history is extended for shell game and demonstration-conditioned stacking.
Go to primary source ↓e17PDF p. 18, Table 4, caption, domain rows and average rows
Each cleaning instruction is independently tested under a reset scene for 20 trials. Held-out objects are pen, tape and shoe. WorldScape averages are 86.7% in-domain, 60.0% held-out, 73.3% overall; π0.5 gives 83.3%, 43.3%, 63.3%; DreamZero gives 73.3%, 30.0%, 51.7%.
Go to primary source ↓e18PDF p. 18, Table 5 all rows and Section 4.5 protocol/component analysis
Ablations use clean-only training. Adding STM, LTM and LSR gives averages 40.91%, 44.67%, 46.25%, 47.89%; final clean/randomized scores are 69.74%/26.03%. The design is a progressive component ladder without reported uncertainty.
Go to primary source ↓e19PDF p. 19, Table 6 all rows and caption; Section 4.5 training analysis and Section 5
With Stage 3 fixed, no Stages 1–2, Stage 1, Stages 1–2 without SF and full training give averages 43.19%, 46.63%, 47.30%, 47.89%. The final SF increment is 0.59 percentage points by subtraction. The body ends with conclusions rather than a dedicated limitations section.
Go to primary source ↓8.5 Primary sources
WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory ↗
PDF · 11,828 extracted words
Source fingerprint
6336af1b71fdac06ddcb2afb6c24a5e8320c376612b4f406e992609c2ccce6fe