WMPO: World Model-based Policy Optimization for Vision-Language-Action Models
1. Paper overview
In one sentence: WMPO turns action-conditioned video generation into an on-policy training environment for a VLA, trading expensive environment rollouts for dependence on learned dynamics and reward accuracy. e03e04e05e06e08e12e15
| At a glance | What to know |
|---|---|
| Research problem | Source description Imitation-trained VLAs can compound errors outside expert trajectories, while physical on-policy RL consumes many interactions. WMPO makes repeatable grouped rollouts available through a learned visual environment. Its interaction-free claim concerns the inner optimization loop: expert data, initial observations and policy behavior trajectories are still required. e02e03e07 |
| Core mechanism | |
| A key reported result | MimicGen four-task manipulation, P=1280: Mean 57.6; Coffee 75.0, StackThree 64.1, ThreePieceAssembly 46.1, Square 45.3. Mean and per-task success (%). Same four tasks and evaluation protocol; 1280 environment trajectories. DPO mean 42.4; GRPO 37.1; base 33.6. A 15.2-point mean advantage over DPO supports rollout-budget efficiency. No multiple-seed uncertainty is reported. e07e08e18 |
| Reading caution |
Core contributions
Figure 2. Generated observations close the training loop between action selection and policy optimization. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the instruction and initial image in the upper left. The old policy sends an action chunk downward to the world model, which generates future frames. Follow the arrows marked (d) and (e): they recycle the generated state into subsequent prediction, rather than depicting a gradient update to the video model. On the right, several complete trajectories pass through the reward model and group computation before entering the buffer. The upper-right policy uses that buffer to maximize the clipped objective. The arrow labelled θ refreshes the rollout policy; Algorithm 1 confirms that the inner loop updates policy parameters. e03e04e05e06
What it supports. The architecture separates three jobs: predicting actions, predicting their visual consequences, and judging outcomes. Repeated groups from a shared initial state make relative-reward optimization possible. Its benefit is training experience generated on demand, while the action policy continues to consume ordinary images and language.
Where the evidence stops. Figure 2 summarizes policy training, not a test-time planning controller. Its state-feedback arrows should not be read as joint policy/world-model optimization. Separate world-model alignment precedes this loop; the learned transition and reward functions can both be wrong.
2. Motivation
2.1 The problem and the proposed response
Imitation-trained VLAs can compound errors outside expert trajectories, while physical on-policy RL consumes many interactions. WMPO makes repeatable grouped rollouts available through a learned visual environment. Its interaction-free claim concerns the inner optimization loop: expert data, initial observations and policy behavior trajectories are still required. e02e03e07
2.2 What this reading follows
Imagine letting a robot policy retry the same initial scene many times, learning which action sequences recover from mistakes. WMPO supplies these retries with a video world model: the policy proposes actions, generated frames supply its next observations, and a learned classifier judges whether the full trial succeeds. This makes grouped policy optimization practical with limited collected behavior data. The strongest evidence is improved success under matched environment-trajectory budgets, including one physical manipulation task. Read the architecture together with the failure example: plausible video is useful training experience only when it preserves the contact events that decide success. e03e04e05e06e08e12e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system is supported by the separate policy and generative world-model modules. The policy predicts actions and the world model predicts action-conditioned observations; they do not jointly emit future/action tokens in one model, nor infer actions from desired futures. Other mechanisms and Outside quadrants fit world-model-assisted policy post-training, with imagination used during optimization. e03e04e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the VLA's interface visual, even when diffusion is latent
WMPO's pixel-space description refers to the observations consumed by the action policy. The video generator still performs diffusion in the latent space of a VAE. Section 3.2 replaces OpenSora's temporally compressing 3D VAE with SDXL's 2D VAE, then decodes predicted frames before giving them to the VLA. The policy receives recent images and the instruction and predicts an eight-action chunk; the world model uses four conditioning frames plus that chunk to generate eight future frames. These outputs become conditioning for another cycle. This distinction matters when interpreting the taxonomy: the architecture couples separate action and dynamics models through images. It neither requires a policy trained directly on the video model's latent representation nor describes a single network jointly predicting future observations and actions. e03e04e07
5.2 Learn failure dynamics before asking a classifier for success
A simulator trained mainly on expert successes has limited evidence about what happens after bad actions. WMPO therefore aligns its OXE-pretrained generator using trajectories collected by the base policy, including failures. Noisy conditioning addresses imperfect generated context, while frame-level action modulation addresses action-to-frame correspondence. These mechanisms have different intended roles, although this version does not isolate their effects. The reward classifier is a separate component: it learns from terminal success clips and negative clips, then marks a full trajectory successful if any sliding window crosses its threshold. Reader inference: trustworthy optimization requires both faithful generated events and a calibrated classifier. A classifier can correctly interpret a fabricated successful ending and still provide the wrong training signal. Figure 9 motivates testing this possibility but does not itself measure reward exploitation. e04e05e15e17e18
5.3 Turn whole-trial outcomes into an on-policy update
For a shared initial state, the rollout policy and world model generate a group of complete trials. Table 4 sets the group size to eight. The binary outcomes are centered and divided by their group standard deviation, giving each trajectory an advantage relative to its peers. Groups with identical outcomes are discarded, because they provide no within-group reward contrast. Action-chunk log probabilities sum over action positions and control dimensions. The current-to-old policy ratio then weights the trajectory advantage inside a clipped objective. Removing the KL term removes a separate reference-policy penalty; it does not remove the old-policy probabilities needed for that ratio. Algorithm 1 updates the policy and refreshes the rollout snapshot. Fresh sampling is therefore on-policy with respect to the learned environment, whose dynamics may still differ from the physical task. e06e14
5.4 Training and inference
During training
First imitation-finetune OpenVLA-OFT on 300 expert trajectories per simulated task. Pretrain the world model on OXE, then align it using 128 or 1280 collected base-policy trajectories. The reward classifier learns balanced success-terminal positives and earlier/failed-trajectory negatives with binary cross-entropy. e04e05e07
Algorithm 1 updates policy parameters with world and reward models supplied as inputs. It removes KL regularization but retains an old-policy snapshot for importance ratios, refreshing that snapshot after updates. Specific frozen/trainable VLA layers are not identified. e06e13
During inference
At deployment, the optimized VLA produces executable actions from actual observations and the instruction. The described method uses the world model to supply training experience, without specifying test-time imagined search or inverse-dynamics action extraction. Proprioception and wrist-camera inputs are omitted in the experiments. e03e07e09e12
5.5 Implementation flow
- Construct the visual state
The MDP assumes image observations define robot state, with language specifying the task. Each action dimension uses 256 bins. This simplifies sensing but leaves partial observability unresolved. e03e16
- Alternate action and video prediction
The VLA predicts an action chunk; the world model predicts the next eight frames from that chunk and four preceding frames. Generated observations feed the next policy call until a complete trial is produced. e04e07
- Preserve contact detail
OpenSora’s 3D VAE is replaced by SDXL’s 2D VAE. Latent diffusion outputs are decoded to pixels. Training perturbs conditioning frames at diffusion step 50/1000; frame-level AdaLN injects action and timestep information into transformer blocks. e04
- Score and optimize grouped trials
VideoMAE plus a linear head scores sliding clips; any score above the selected threshold yields trajectory reward one. GRPO retains mixed-outcome groups, computes relative advantages and updates the policy using clipped probability ratios. e05e06
6. Experiments & results
WMPO post-trains a vision-language-action policy using complete trajectories generated by a separate action-conditioned video model and judged by a learned success classifier. Its interface to the policy is decoded imagery, although diffusion itself uses VAE latents. It improves manipulation success under matched environment-rollout budgets; model fidelity, reward calibration and substantial training compute remain central qualifications.
The supplied v1 has no controlled component-removal ablation of policy behavior alignment, noisy-frame conditioning or frame-level action control. Figures 3 and 9 serve as behavior and model-failure diagnostics in the ablation section; they do not establish each component's causal effect. Physical success rates are reported in Section 4.6 prose rather than an original quantitative table, and the real-versus-imagined sequences are qualitative examples. e09e12e15e18
6.1 Read the original evidence
Table 1. WMPO improves the mean success rate at both reported environment-rollout budgets. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each budget block horizontally before comparing budgets vertically. P counts full trajectories obtained from the environment for optimization; in this experiment that environment is MimicGen, despite the caption's use of “real.” Each task uses 128 evaluation initial states, and the rightmost column averages the four task rates. The base policy is shared context, trained using 300 expert demonstrations per task. At P=1280, WMPO's four values are 75.0, 64.1, 46.1 and 45.3 percent. Compare its mean against DPO within the same block to assess the reported data-budget advantage, rather than confusing a larger budget with a different algorithm. e07e08e13e14e18
What it supports. At P=128, mean success is 47.1% for WMPO versus 37.3% for DPO. At P=1280, the means are 57.6% and 42.4%. The gaps are 9.8 and 15.2 percentage points. WMPO also leads each task column within both budget blocks.
Where the evidence stops. The comparison matches collected environment trajectories, not compute or effective update counts. World-model pretraining is additional. Appendix B explains the online GRPO baseline's update constraints; the table supplies no seed variation or confidence intervals.
Table 2. WMPO leads under three specified shifts, with substantial failures still remaining. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Expand the abbreviated columns using Figure 4 on page 8. Position disruption randomizes the stick location in Square; background disruption changes the StackThree tabletop to gray; texture disruption changes the ThreePieceAssembly base from red to dark wood. Thus, the columns are different task-and-disruption pairs, not three perturbations of one universal task. The mean summarizes those three rates. Compare each optimized policy with the base row to see whether in-distribution improvements transfer. DPO improves the position column but falls below the base policy on the two appearance changes, while WMPO improves all three listed comparisons. e10
What it supports. WMPO reports 22.3% position, 50.0% background and 16.4% texture success, averaging 29.6%. The means for GRPO, the base policy and DPO are 24.7%, 23.7% and 19.5%. This supports a relative advantage on these shifts while showing that absolute generalization is still weak.
Where the evidence stops. Section 4.4 calls these novel scenarios; they do not establish generalization to new language goals or manipulation families. The table does not restate the training rollout budget, so it should not be assigned to either Table 1 budget.
Figure 6. Two rounds of newly collected behavior support further policy improvement. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as additional trajectories beyond the initial training setup. WMPO collects 128 trajectories using its policy, performs another optimization phase, and repeats with the updated policy. DPO receives corresponding rounds of policy-collected data. The red “Base Policy” line has a different data source: Section 4.5 describes imitation training with 300, 428 and 556 expert demonstrations. Follow the direction of each segment rather than treating the chart as a table of exact percentages. The WMPO line increases in both rounds, whereas DPO's initial improvement reverses after the second collection. e11e18
What it supports. The graph supports continued improvement over the two reported StackThree rounds. WMPO finishes above both comparison curves, while static imitation benefits from additional expert data. This is useful evidence that repeated behavior collection can extend the initial post-training gain, rather than merely reusing one fixed batch forever.
Where the evidence stops. The comparison mixes expert-collected and policy-collected additions, which entail different costs. Only one task and two rounds are shown, without uncertainty bars or tabulated point values. Long-term stability, forgetting and cross-task retention remain untested here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| MimicGen four-task manipulation, P=128 Coffee_D0, StackThree_D0, ThreePieceAssembly_D0 and Square_D0; 128 evaluation initial states per task; 128 environment trajectories for optimization. | 47.1 Mean task success (%) | DPO 37.3; online GRPO 33.2; base policy 33.6. WMPO exceeds DPO by 9.8 percentage points. The budget excludes expert initialization and world-model pretraining; this is not a compute-matched comparison. e07e08e13 |
| MimicGen four-task manipulation, P=1280 Same four tasks and evaluation protocol; 1280 environment trajectories. | Mean 57.6; Coffee 75.0, StackThree 64.1, ThreePieceAssembly 46.1, Square 45.3. Mean and per-task success (%) | DPO mean 42.4; GRPO 37.1; base 33.6. A 15.2-point mean advantage over DPO supports rollout-budget efficiency. No multiple-seed uncertainty is reported. e07e08e18 |
| Generalization under three disruptions Randomized Square stick position, changed StackThree background and ThreePieceAssembly base texture; Table 2 does not restate the training budget. | Position 22.3; background 50.0; texture 16.4; mean 29.6. Success (%) | Base mean 23.7; GRPO 24.7; DPO 19.5. Best listed mean, but these are shifts of existing tasks, not new instruction families; absolute success remains limited. e10 |
| Physical square-on-stick insertion Cobot Mobile ALOHA; 5 mm clearance; 200 expert demonstrations and 128 policy trajectories; 30 trials per model. | 70 Success (%) | DPO 60; base policy 53. Executed-robot evidence, distinct from video plausibility. One task and 30 trials limit precision and breadth; percentages are retained as reported. e12 |
| StackThree iterative policy improvement Two successive collections of 128 policy trajectories; imitation baseline uses 300, 428 and 556 expert demonstrations. | WMPO increases in both rounds and finishes above both baselines; exact point values are not tabulated. Success-rate trend | DPO improves then declines; imitation learning improves with added expert data. Figure 6 supports short iterative improvement, not indefinite lifelong learning or retention across multiple tasks. e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. One collision example illustrates recovery behavior after imagined policy training. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the top and bottom rows from left to right. The dashed black region identifies the collision stage, with small red boxes drawing attention to the obstructed contact. In the upper row, the red downward arrow and failure symbol mark continued pushing by the base policy. In the lower row, the green upward arrow begins the recovery sequence; Section 4.3 describes lifting, realigning and then inserting the square. These are test-time policy behaviors in the simulated Square task. The frames explain what the authors mean by self-correction, rather than showing the world model's video-prediction accuracy. e09e18
What it supports. The displayed WMPO policy escapes a collision state that traps the imitation baseline. This is qualitative evidence for the type of behavior that can accompany better success rates. It makes the claimed benefit more concrete than a table alone, but does not quantify how often recovery occurs.
Where the evidence stops. This is a selected diagnostic example, not a controlled ablation or a recovery-rate benchmark. The source does not isolate which model modification causes the behavior, and the separate trajectory-length analysis includes only successful trials.
Figure 9. A late contact error exposes the boundary of visually plausible imagination. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare matching columns between the green-labelled real trajectory and the peach-labelled imagined trajectory. Early frames show similar approach and manipulation, so the important discrepancy is near the final contact. Appendix C and the caption identify the missed event: subtle perturbations leave the real square stuck on the stick, which the model does not faithfully reproduce. The images alone do not provide force, depth or pose measurements; use the authors' contact interpretation with that boundary in mind. Figure 8 on the same inspected page provides a contrasting example where a failed real trajectory is predicted correctly. e15e05e06
What it supports. This example demonstrates that a world model can preserve much of a trial's appearance yet miss the event that decides its outcome. Reader implication: when such video is used as an RL environment, outcome fidelity matters beyond perceptual similarity, because optimistic generated endings could steer policy updates.
Where the evidence stops. The source calls these failures relatively rare but gives no validation count or rate. It does not show that this particular example fooled the reward classifier. A possible optimization failure should therefore remain a proposed test, not a measured conclusion.
7. Analysis & limitations
7.1 What the evidence leaves open
No removal ablation isolates behavior alignment, noisy conditioning or action modulation. The self-correction example is qualitative, and shorter trajectories are measured only among successful trials. e09e18
Reward F1 above 0.95 is an author-reported summary with insufficient split/calibration detail to establish resistance to reward hacking. Figure 9 shows the world model missing a subtle sticking event; its frequency is not quantified. e15e17
Image-only state and discretized actions constrain the demonstrated scope; partially observable formulations and flow-based policies remain future work. e16
7.2 Questions for discussion
- How far can an improving policy move beyond the behavior-alignment data before imagined successes become unreliable?
- Would WMPO retain its advantage under matched training compute and stronger baseline update budgets?
8. Reproducibility audit
8.1 Requirements and known gaps
Reported hardware is 8 H100 GPUs for supervised policy finetuning and 32 H100 GPUs for world-model training/policy optimization. Table 3 lists 12,000,000 pretraining and 3,000,000 finetuning steps, AdamW at 0.0001 and batch 128. These are printed settings, not independently verified compute costs. e13
GRPO uses group size 8, learning rate 5×10^-6, clipping 0.20/0.28 and temperature 1.6. Table 4’s training batch 64/minibatch 128 conflicts with Algorithm 1’s trajectory-count convention; Appendix B’s 64×8 accounting suggests different units. Resolve this before implementation rather than silently choosing one. e14
A faithful reproduction needs the OXE mixture, exact checkpoints, rollout horizon, policy-layer freezing, reward validation split/threshold and inherited software settings. The supplied PDF does not fully specify these. Test alignment’s effect on held-out failure prediction and audit reward decisions against environment outcomes before scaling imagined RL. e04e05e06e13e15e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does behavior alignment improve failure fidelity and downstream control?
Reader-proposed check, not performed: start from the same pretrained world-model checkpoint and compare downstream finetuning on expert-only data with finetuning on base-policy behavior, matching trajectory count and optimization steps. Keep noisy conditioning, action modulation, reward model, base policy and GRPO update budget fixed. On held-out Square starts, execute identical action sequences in MimicGen and both world models, measuring success/failure agreement and missed-collision frequency. Then evaluate the two optimized policies on the same 128 environment starts. Better imagined outcome agreement together with better executed success would support the alignment mechanism; prettier videos without outcome or control gains would not. Keep validation trajectories out of finetuning. e04e07e09e15e18
Check 2: Separate reward-classifier errors from optimistic video endings
Reader-proposed check, not performed: freeze a WMPO policy and evaluate matched action sequences from held-out MimicGen initial states in both the simulator and the learned world model. Apply the same reward classifier to actual simulator videos and imagined videos, comparing each prediction with the simulator's task-success signal. Report false-positive success rates, stratified by contact failures and rollout length, alongside overall F1. Select thresholds using a separate validation split, then lock them for testing. If the classifier is accurate on actual frames but optimistic on imagined frames, dynamics error is implicated; errors on both streams would also question reward calibration. This tests the specific gap between reported F1 and reliable imagined returns. e04e05e06e15e17
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author block on page 1; Figure 1 on page 2; related-work and method context on page 3; Figure 2 and formulation on page 4; world-model and reward details on pages 5–6; Figures 3–7 and Tables 1–2 with evaluation prose on pages 7–10; Algorithm 1, hardware, baseline accounting, Tables 3–4, limitations and Figures 8–9 on pages 14–16. Figure 2 arrows were cross-checked against its caption, Eqs. (2)–(5) and Algorithm 1: observation feedback is distinct from policy-parameter updates. All six final original crops were individually viewed with labels, legends and table boundaries intact. Narrow Table 2 and Figure 6 crops were rendered at higher native PDF resolution. Reference pages 11–13 were read in the complete text chunks but not visually inspected. No separate supplement or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Section 1 Introduction (pp. 1–3)
- Section 2 Related Work (p. 3)
- Sections 3.1–3.4: formulation, world model, reward model and GRPO (pp. 4–6)
- Sections 4.1–4.6: settings, comparisons, behavior, disruptions, lifelong learning and real robot (pp. 7–10)
- Section 5 Conclusion (p. 10)
- References (pp. 11–13)
- Appendices A–E, Algorithm 1 and Tables 3–4 (pp. 14–16)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: title and all six authors match. This report reviews arXiv:2511.09515v1, whose margin is dated 12 November 2025 and manuscript box November 13, 2025. The catalog labels ICLR 2026; no separate venue edition or revision comparison was supplied, so equivalence to that edition is unverified (e01).
- The supplied extraction omits figure reconstruction; the retained PDF was visually inspected to cover all figures and tables. Reference pages 11–13 were read as text only.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- Code and external links were not inspected; no training or evaluation was reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block, arXiv margin and date box
The title and all six authors match the supplied catalog. The margin identifies arXiv:2511.09515v1, 12 November 2025; the manuscript date box says November 13, 2025. Affiliations are Hong Kong University of Science and Technology and ByteDance Seed.
Go to primary source ↓e02PDF pp. 1–3, Abstract and Section 1; p. 2, Figure 1
The authors motivate pixel-observation world-model rollouts by the cost of physical RL and the mismatch between abstract latent states and pretrained VLA visual features. Policy behavior data is collected before imagined optimization.
Go to primary source ↓e03PDF p. 4, Figure 2 and Section 3.1, Eq. (1)
Separate policy, world and reward models generate trajectories, assign binary outcomes and update policy parameters. States combine image observations and language; action chunks contain D-dimensional controls discretized into 256 bins per dimension. Image observations are assumed sufficient for state.
Go to primary source ↓e04PDF p. 5, Section 3.2, Eq. (2), Model Architecture and Policy Behavior Alignment
The world model uses an OpenSora backbone with SDXL’s 2D VAE; diffusion operates in latent space and generated images are decoded for VLA input. Noisy conditioning uses diffusion step 50/1000; frame-level AdaLN incorporates action and timestep signals. OXE pretraining is followed by alignment on policy rollouts including failures.
Go to primary source ↓e05PDF pp. 5–6, Section 3.3, Reward Model
A VideoMAE encoder and linear head use binary cross-entropy with balanced positive and negative clips. Success-terminal clips are positive; earlier clips and failed-trajectory clips are negative. Any sliding-window clip exceeding a validation-selected threshold makes a trajectory successful.
Go to primary source ↓e06PDF p. 6, Section 3.4, Eqs. (3)–(5); p. 14, Algorithm 1
GRPO samples groups from shared initial states, discards uniform-outcome groups, sums action-token log probabilities, and uses normalized trajectory rewards with a clipped new/old policy ratio. KL regularization is removed. Algorithm 1 takes world and reward models as inputs and updates policy parameters, then refreshes the old policy.
Go to primary source ↓e07PDF p. 7, Section 4.1, Implementation Details and Simulation Environment Details
OpenVLA-OFT is imitation-finetuned with 300 expert trajectories per MimicGen task. Proprioception and wrist camera are omitted. K=8 actions/frames, c=4 conditioning frames, reward clip L=8, stride 1, and behavior budgets P=128 or 1280 are used. Each task is evaluated on 128 initial states.
Go to primary source ↓e08PDF p. 7, Table 1, all task and Mean columns; p. 8, Section 4.2
Mean success rates are base 33.6%; at P=128, GRPO 33.2%, DPO 37.3%, WMPO 47.1%; at P=1280, GRPO 37.1%, DPO 42.4%, WMPO 57.6%. WMPO’s P=1280 task rates are 75.0, 64.1, 46.1 and 45.3%. Budgets count environment trajectories, and baselines differ in online/offline data reuse.
Go to primary source ↓e09PDF p. 7, Figure 3; p. 8, Section 4.3; p. 9, Figure 5
The illustrated Square execution contrasts continued downward pushing with lifting, realignment and insertion after collision. Figure 5 reports relative average lengths only for successful trials, normalized to the base policy; WMPO bars are shorter on all four tasks.
Go to primary source ↓e10PDF p. 8, Figure 4 and caption; p. 9, Section 4.4 and Table 2
Disruptions change Square stick position, StackThree tabletop background, and ThreePieceAssembly base texture. WMPO success rates are 22.3%, 50.0%, 16.4%, mean 29.6%; base/GRPO/DPO means are 23.7/24.7/19.5%. Table 2 does not identify its training rollout budget.
Go to primary source ↓e11PDF p. 9, Figure 6 and Section 4.5; p. 10, Section 4.5 continuation
The StackThree lifelong experiment adds two rounds of 128 policy-collected trajectories. The imitation comparison uses 300, 428 and 556 expert demonstrations. The graph shows WMPO increasing each round and DPO declining after its first improvement; exact success values are not tabulated.
Go to primary source ↓e12PDF p. 9, Figure 7 and caption; p. 10, Section 4.6
Cobot Mobile ALOHA performs square-on-stick insertion with 5 mm clearance. Training uses 200 expert demonstrations and 128 subsequent base-policy trajectories. Over 30 evaluation trials per model, reported base/DPO/WMPO success is 53/60/70%. Figure 7 compares real and imagined base-policy trajectories, not two trained-policy success rates.
Go to primary source ↓e13PDF p. 14, Appendix A, first two paragraphs; p. 15, Table 3
Supervised policy finetuning uses 8 H100 GPUs; world-model training and policy optimization use 32 H100 GPUs. The policy head predicts discrete tokens in parallel. Table 3 reports AdamW, learning rate 0.0001, batch 128, gradient clip 0.1, 12,000,000 pretraining steps, 3,000,000 finetuning steps, EMA 0.9999, zero weight decay and epsilon prediction; other settings refer to OpenSora.
Go to primary source ↓e14PDF p. 14, Algorithm 1 and Appendix B; p. 15, Appendix B continuation; p. 16, Table 4
Algorithm 1 defines B in trajectories and M dividing B. Table 4 lists training batch 64, group 8 and minibatch 128, learning rate 5×10^-6, clipping 0.20/0.28 and temperature 1.6. Appendix B instead counts a batch-64 update as 64×8=512 real trajectories; GRPO also tries batch 8 with learning rate reduced eightfold and reports the better configuration.
Go to primary source ↓e15PDF p. 15, Appendix C; p. 16, Figures 8–9 and captions
Figure 8 illustrates correctly predicted failure; Figure 9 misses a late contact event where the square gets stuck. The authors describe such failures as relatively rare on validation data without reporting a count or rate.
Go to primary source ↓e16PDF p. 4, Section 3.1, State space; p. 15, Appendix D
The formulation assumes image-defined state and leaves POMDPs to future work. Experiments use discretized actions; extension to flow-matching policies is proposed rather than evaluated.
Go to primary source ↓e17PDF p. 8, Section 4.2, final sentences; pp. 5–6, Section 3.3
The authors report reward-model F1 above 0.95 across tasks, without per-task scores, validation-set sizes or uncertainty. The PDF does not supply the selected reward threshold or a generated-rollout calibration study.
Go to primary source ↓e18PDF pp. 7–10, Sections 4.1–4.6, Tables 1–2 and Figures 3–7; pp. 14–16, Appendices A–D
The supplied version presents budget comparisons, disruption tests, behavior/length diagnostics, iterative learning and one physical task, plus model prediction examples. It does not report a controlled removal ablation of behavior alignment, noisy-frame conditioning or frame-level action control. Main success tables do not give uncertainty or multiple-seed statistics.
Go to primary source ↓8.5 Primary sources
WMPO: World Model-based Policy Optimization for Vision-Language-Action Models ↗
PDF · 7,540 extracted words
Source fingerprint
1f3e88874bcb716ad0417347f9c6a73f1dd70c873e7872d3ceed6c57af2b7598