World2Act: Latent Action Post-Training from World Model Dynamics

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: An Dinh Vuong; Tuan Van Vo; Abdullah Sohail; Haoran Ding; Liang Ma; Xiaodan Liang; Anqing Duan; Ivan Laptev; Ian Reid

Affiliations: MBZUAI

Source: 2603.10422 ↗ · Project page ↗ · Catalog record

Reading: 250 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: A frozen video world model can teach a small residual controller through aligned action latents, improving manipulation without pixel-derived action labels while retaining a difficult timing and contact gap. e-identity e-wm e-bridge e-residual e-robocasa e-ablation e-real

At a glance	What to know
Research problem	Reader analysis Pixel-based world-model post-training can convert hallucinated video details into incorrect action supervision. The authors propose using pre-decoding dynamics latents as weak targets instead. Their controlled frozen-IDM diagnostic supports one route by which generated videos corrupt labels; it does not establish that latent dynamics are universally accurate. e-problem e-idm e-failure
Core mechanism	Source description A two-stage bridge connects a separately trained video world model to an existing VLA through reconstructable, temporally aligned action embeddings. e-bridge e-stage1
A key reported result	RoboCasa manipulation: 0.726 (72.6%) Mean success rate. GR00T-N1.6-ft post-training; 50 trials per task for each of five seeds; evaluation scenes disjoint from post-training. GR00T-N1.6-ft 0.701; DreamGen 0.705; BC 0.704; VLA-RFT 0.710; Ctrl-World 0.698. A 2.5 percentage-point gain over the same backbone and 2.1 points over DreamGen. No uncertainty estimates accompany these tables. e-evaluation e-robocasa
Reading caution	Source description The authors identify contact-rich non-prehensile dynamics, modest physical success and rigid temporal correspondence as limitations. Figure 16 shows plausible stove-knob imagination followed by failed physical grounding during policy execution. e-failure

Core contributions

Source description
A two-stage bridge connects a separately trained video world model to an existing VLA through reconstructable, temporally aligned action embeddings. e-bridge e-stage1
Source description
Residual-only post-training preserves the VLA and bridge parameters while transferring imagined dynamics through a contrastive trajectory objective. e-residual
Reader analysis
Same-backbone comparisons, temporal-alignment ablations and an IDM diagnostic support the proposed mechanism, with limited physical-task success. e-robocasa e-ablation e-idm e-real

Two training stages connect frozen world-model latents and VLA actions through adapters. A right-hand inset shows the residual Transformer and the addition of its output to the base action latent. — **Figure 2.** The learned interface changes between stages: first align the bridge, then freeze it and learn a residual controller. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Reader analysis

Pixel-based world-model post-training can convert hallucinated video details into incorrect action supervision. The authors propose using pre-decoding dynamics latents as weak targets instead. Their controlled frozen-IDM diagnostic supports one route by which generated videos corrupt labels; it does not establish that latent dynamics are universally accurate. e-problem e-idm e-failure

2.2 What this reading follows

A video world model can imagine a robot finishing a task, but an imagined image does not specify a reliable motor command. World2Act learns an interface between those two descriptions. Expert demonstrations establish a shared video–action space and an action decoder; imagined trajectories subsequently guide corrections to a frozen VLA's proposed controls. The six source excerpts below follow that interface from architecture to execution, then examine whether the experiments support it. Read the simulation gains alongside the temporal-alignment ablation and low physical success rates. This edition reviews the supplied arXiv v2 of 29 May 2026. e-identity e-wm e-bridge e-residual e-robocasa e-ablation e-real

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	WAMs
Architecture	Dual-system
Prediction paradigm	Other mechanisms
Quadrant	Outside quadrants

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

Separate WM and VLA modules connected by adapters support Dual-system and Other mechanisms / Outside quadrants. This is trajectory-guided residual post-training, without a jointly predictive video–action backbone or inference-time IDM. The policy-post-training subcategory fits; the WM-RL wording should not imply reinforcement learning here, because Stage 2 uses contrastive supervision without rewards. e-residual e-architecture e-taxonomy

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
Stage 1: synchronized expert videos and robot action sequences Stage 2: initial observations, language instructions, imagined video latents and current-policy rollouts Execution: current image/proprioceptive state and instruction	A shared video–action latent space with an action decoder A residual controller producing executable corrected action chunks

4.2 Equations and their role

\mathcal{L}_{\mathrm{recon}}=\|a_{\mathrm{gt}}-\hat a\|^2,\qquad \mathcal{L}=\mathcal{L}_{\mathrm{recon}}+\mathcal{L}_{\mathrm{contrastive}}

The ground-truth action sequence is a_gt; hat a is reconstructed by decoding action embeddings. This Stage-1 objective combines control reconstruction with cross-modal alignment. e-bridge

\operatorname{sim}(z_i^v,z_j^a)=\frac{1}{T}\sum_{t=1}^{T}\cos(z_{i,t}^v,z_{j,t}^a)

Video and action trajectories z^v and z^a contain T corresponding chunks. Similarity averages cosine agreement at matching chunk indices rather than collapsing time into a single trajectory embedding. e-bridge

\mathcal{L}_{\mathrm{contrastive}}=-\log\frac{\exp(\operatorname{sim}(z_i^v,z_i^a)/\tau)}{\sum_{j=1}^{B}\exp(\operatorname{sim}(z_i^v,z_j^a)/\tau)}-\log\frac{\exp(\operatorname{sim}(z_i^a,z_i^v)/\tau)}{\sum_{j=1}^{B}\exp(\operatorname{sim}(z_i^a,z_j^v)/\tau)}

Equation 2 contrasts the positive pair i against B batch samples in both directions, with temperature tau = 0.1. Stage 2 substitutes corrected action trajectories for the action side and retains only this alignment loss. e-bridge e-residual

z_{\mathrm{base},t}^{a}=B_a(\bar a_{\mathrm{base},t}),\quad \Delta z_t^a=f^{\theta}(s_t,z_{\mathrm{base},t}^a),\quad \bar a_{\mathrm{final},t}=D_a(z_{\mathrm{base},t}^a+\Delta z_t^a)

The action adapter B_a embeds the base action chunk, the residual policy uses current state s_t, and the action decoder D_a decodes the corrected latent. Only the residual policy changes in Stage 2; addition occurs before action decoding. e-residual

5. Method in detail

5.1 First make video similarity mean something about controls

Reader analysis

Stage 1 starts with paired demonstrations, not generated pseudo-actions. The frozen VAE encodes each expert video, and matching action chunks are embedded separately. Reconstruction teaches the action embedding to retain the controls needed by the decoder. Contrastive learning then asks video and action trajectories from the same demonstration to agree more than mismatched trajectories. Same-task negatives matter because merely recognizing “open drawer” should not be sufficient to distinguish two executions. Averaging similarities at corresponding chunk indices further rewards matching progression through the task. Reader interpretation: the bridge attempts to turn a visual notion of progress into a decodable control representation. It does not establish that all nearby points are executable; reconstruction was learned on demonstrations, while later residuals can move away from those examples. e-stage1 e-bridge e-residual e-failure

An instruction to set up coffee passes through an LLM into ordered pick and place prompts, then through a world model into sequentially merged video segments. — **Figure 9.** Long imagined behavior is assembled from short skill-conditioned segments. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

5.2 Then teach a correction while keeping the execution path small

Reader analysis

Stage 2 creates two trajectories from matching initial conditions. The skill-conditioned WM imagines how the task could unfold, while the current augmented policy acts in the simulator. Only the residual controller receives gradients from their contrastive alignment. Within the policy branch, the frozen VLA proposes controls, the action adapter embeds them, and the residual combines that embedding with current visual and proprioceptive observations. The sum is decoded and executed for four steps before observations are refreshed. No simulator reward or differentiable transition is required. Reader interpretation: supervision transfers the teacher's preferred temporal evolution into a reusable controller, rather than planning with the WM at every deployed action. This also exposes the timing assumption: corresponding chunk indices can cease to describe the same contact event when execution falls behind imagination. e-residual e-architecture e-synthesis e-failure

5.3 Judge transfer at three different levels of evidence

Reader analysis

The benchmark tables ask whether actual policy success improves, and their same-backbone comparisons are the clearest test of the post-training package. Table 5 then asks which design choices matter: global similarity underperforms the temporally aligned objective, but this remains an ablation rather than proof of a uniquely correct representation. Table 15 addresses a different question, showing that one frozen IDM produces worse labels from generated videos. It supports the motivation for avoiding those labels without certifying WM latent physics. Finally, Figure 6 measures physical execution and shows both an improvement and a low ceiling. Reader interpretation: these levels form a useful argument for the method, but none cancels the source's transfer failure, where plausible imagined knob turning fails to produce a successful grip. e-evaluation e-robocasa e-ablation e-idm e-real e-failure

5.4 Training and inference

During training

Source description

Stage 1 runs 30K steps with batch size 16, hard-negative ratio 0.25 and temperature 0.1. Table 12 specifies four-step chunks and 32-dimensional embeddings; the residual attends to visual, proprioceptive and base-action tokens using two layers and four heads. e-stage1 e-bridge e-architecture

Reader analysis

The supervised GR00T reference uses 1,000 expert trajectories per simulator. Post-training uses approximately 1,000 imagined trajectories per simulator with evaluation-disjoint initial scenes. DeepSeek decomposes instructions using manual schemas; sequential WM segments condition on preceding final frames. Avoiding pixel supervision therefore does not establish a wholly decoder-free generation pipeline. e-evaluation e-synthesis

During inference

Reader analysis

Encode the VLA's proposed action chunk, add the observation-conditioned latent residual, and decode through the frozen action decoder. Execute four controls open-loop, then query again with fresh observations. The specified deployed controller requires no WM rollout as an input. Table 11 reports 251.9 Hz / 4.0 ms per predicted action on an RTX 4090, versus 274.1 Hz / 3.6 ms for GR00T; this is the paper's inference measurement, not a measured physical feedback-loop rate. e-residual e-architecture e-speed

5.5 Implementation flow

Prepare the dynamics teacher
Cosmos-Predict2 generates spatial video-VAE latents from initial observation, instruction and noise. It is frozen during bridge and residual learning. Atomic skill prompts organize longer imagined trajectories. e-wm e-synthesis
Align demonstrations
Appendix B.1 specifies VAE-encoded expert videos as Stage-1 anchors. A CNN video adapter and MLP action adapter map corresponding chunks into shared embeddings. An action decoder reconstructs the expert controls; bidirectional InfoNCE uses chunk-averaged cosine similarity, with same-task demonstrations supplying hard negatives. e-bridge e-stage1
Learn a residual from imagination
The frozen VLA proposes a base action chunk. Its embedding and current observations condition a Transformer residual. Parallel simulator rollouts supply executed state sequences, while the WM supplies imagined targets from matching initial conditions. Contrastive gradients update only the residual; rewards, success labels, expert action labels and differentiable simulator transitions are unnecessary in Stage 2. e-residual e-synthesis e-architecture

6. Experiments & results

World2Act transfers an instruction-conditioned video world model's dynamics into a frozen GR00T-N1.6 policy through a learned video–action bridge and residual action-latent controller. Expert demonstrations first teach the bridge to align video and action chunks while reconstructing actions. Imagined latents then supervise residual corrections without IDM pseudo-actions or simulator rewards. Reported success improves on three simulation benchmarks and three physical tasks, but temporal correspondence remains necessary and plausible imagination can still fail at contact. The method and results below are from the supplied v2 artifact.

6.1 Read the original evidence

Three original result tables for RoboCasa, LIBERO and Bridge-SIMPLER. Highlighted rows compare GR00T-N1.6-ft, DreamGen post-training and World2Act post-training. — **Tables 1–3.** The same-backbone rows show consistent average gains, with a particularly large difference from DreamGen on LIBERO. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Grouped success-rate bars for cup-and-plate pick and place, bowl pickup, microwave closure and their average, comparing GR00T-ft, DreamGen and World2Act. — **Figure 6.** Physical gains accompany a low absolute success rate across three Franka tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
RoboCasa manipulation GR00T-N1.6-ft post-training; 50 trials per task for each of five seeds; evaluation scenes disjoint from post-training.	0.726 (72.6%) Mean success rate	GR00T-N1.6-ft 0.701; DreamGen 0.705; BC 0.704; VLA-RFT 0.710; Ctrl-World 0.698. A 2.5 percentage-point gain over the same backbone and 2.1 points over DreamGen. No uncertainty estimates accompany these tables. e-evaluation e-robocasa
LIBERO four-suite evaluation Spatial, Object, Goal and Long; main five-seed, 50-trial-per-task protocol.	0.981 (98.1%) Average success rate	GR00T-N1.6-ft 0.970; DreamGen 0.921; Cosmos Policy 0.985. Gains are 1.1 points over GR00T and 6.0 over DreamGen, not a new overall best. Long decreases from 0.943 to 0.940 despite the average gain. e-evaluation e-libero
Bridge-SIMPLER seven-task evaluation AutoEval's WidowX-based seven-task suite; main five-seed protocol.	0.590 (59.0%) Average success rate	GR00T-N1.6-ft 0.576; DreamGen 0.583. A 1.4-point average gain over GR00T; individual tasks need not improve, including stacking green on yellow (0.044 to 0.042). e-evaluation e-bridge-results
RoboCasa alignment and adapter ablation Variants of latent objectives and policy adaptation in Table 5.	Chunk-wise bidirectional InfoNCE with residual: 0.726; 6.8 h Success rate; reported training time	Global similarity 0.693; single-direction InfoNCE 0.720; marginal loss 0.707; rank-32 LoRA 0.721 in 15.3 h. Temporal matching has the largest listed objective effect. LoRA changes the adaptation mechanism; timing hardware for this table is not specified. e-ablation
Franka Research 3 physical manipulation Cup-to-plate, bowl pickup and microwave closure; 20 demonstrations and 20 evaluation trials per task; 100 generated training trajectories, with per-task allocation unstated.	0.10, 0.25, 0.25; average 0.20 Task success rate and task average	GR00T: 0.05, 0.20, 0.15; DreamGen: 0.05, 0.15, 0.15. The reported average gains are 6.7 and 8.3 percentage points. Figure 6 rounds baseline averages to 0.13 and 0.12. Small trial counts and no error bars limit precision. e-real
Frozen-IDM sensitivity on LIBERO-Object Same frozen IDM; 50 held-out scenes; ground-truth versus decoded-WM input videos; temporally aligned expert action targets.	Decoded WM: 0.097 / 0.199 Action MSE / MAE, lower is better	Ground-truth video: 0.082 / 0.167; reported ratios 1.18× / 1.19×. Input-video replacement increases pseudo-label error. This is not an error measurement of World2Act's residual controller. e-idm
RoboCasa held-out task transfer Twelve seen and twelve unseen tasks; base policies without task-specific finetuning; Table 10 defines the split.	+2.4 percentage points for GR00T-N1.6-base; +1.1 for Cosmos Policy Reported success-rate gain at twelve seen tasks	Corresponding vanilla policies on the unseen split. The held-out tasks reuse related interaction skills. This tests new compositions within RoboCasa rather than arbitrary novel physics. e-transfer

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Ablation table comparing single and bidirectional contrastive objectives, chunk and global similarity, LoRA adaptation ranks, and the final residual design with success rates and training times. — **Table 5.** Preserving the temporal correspondence of video and action chunks matters more than the small gain from bidirectional contrast alone. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Frozen inverse-dynamics model errors with ground-truth versus decoded world-model videos: MSE 0.082 versus 0.097 and MAE 0.167 versus 0.199, followed by error ratios. — **Table 15.** Replacing real videos with generated rollouts makes one fixed IDM's pseudo-actions less accurate. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Source description

The authors identify contact-rich non-prehensile dynamics, modest physical success and rigid temporal correspondence as limitations. Figure 16 shows plausible stove-knob imagination followed by failed physical grounding during policy execution. e-failure

Reader analysis

Latent targets are weak supervision, not verified executable plans. The IDM experiment changes the entire video source, so it cannot isolate localized rendering artifacts from motion or timing differences. Latent alignment and t-SNE overlap alone do not certify physical validity. e-residual e-idm e-action-validity e-artifacts

7.2 Questions for discussion

How much of the benefit survives a controlled temporal offset between imagined and executed chunks?
Can latent alignment improve while decoded controls move farther from valid contact trajectories?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Reconstruct synchronized demonstrations and manual skill schemas. Appendix D specifies a 0.005 m gripper-closure threshold, chronological LLM alignment and 2×2 camera stitching; incomplete alignments are discarded. Both Figure 11 and D.1 include equality in the closing/contact branch. e-data

Source description

The WM preparation uses full finetuning of Cosmos-Predict2 2B for 10K steps; Appendix E reports eight AMD MI210 GPUs with 64 GB each. This is WM-training hardware, not an established hardware attribution for Table 5. Skill-WM versus Base-WM yields downstream GR00T success 0.726 versus 0.715. e-wm-training e-wm-evaluation e-ablation

Reader analysis

The PDF promises a future code/data release. It does not give a complete optimizer, learning-rate, software-version or Stage-2 schedule specification. Reproduction also needs action normalization, exact scene lists and handling of unequal imagined/executed trajectory lengths; these cannot be reconstructed confidently from the reported architecture alone. e-release e-stage1 e-residual e-architecture e-synthesis

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Distinguish temporal guidance from task-identity matching

Reader-proposed, not run: use the same frozen WM, bridge, initial scenes, base VLA, residual initialization and training budget for three Stage-2 arms. Align chunks correctly in the first, shift WM targets by one four-control-step chunk in the second using a common valid overlap, and use global trajectory similarity in the third. Keep task labels and negative sampling fixed. Evaluate held-out RoboCasa success, chunk similarity and action error under the reported five-seed protocol. Genuine temporally specific guidance predicts the aligned arm will outperform both controls. If task-level retrieval stays high while success falls under the shift, that would distinguish task recognition from useful dynamics matching; if success is unchanged, the claimed temporal mechanism needs narrowing. e-bridge e-residual e-architecture e-ablation e-evaluation e-action-validity

Check 2: Separate the residual's benefit from the frozen decoder's effect

Reader-proposed, not run: compare the raw frozen GR00T policy, its actions passed through the frozen action adapter and decoder with zero residual, and the learned residual policy. Use identical held-out scenes, action normalization, four-step execution cadence and random seeds, with no further finetuning of any frozen component. Measure task success and decoded-action error against held-out expert trajectories, including OpenDrawer, PnPCounterToCab and TurnOnStove. A successful reproduction should show whether the learned residual adds value beyond the adapter–decoder round trip. If the zero-residual controller explains most of the gain, or learned corrections increase contact failures despite better latent similarity, attribute the effect accordingly rather than crediting latent dynamics transfer alone. e-stage1 e-residual e-architecture e-action-validity e-failure

8.3 Reading coverage

Visual audit: Visually inspected the title/author/version page, all Figures 1–16 and Tables 1–15, the method equations and the appendix pages supporting training, hardware, synthesis, evaluation and proposed checks. All six final original crops were viewed and retain relevant labels, legends, rows and columns. Figure 2 was checked against Eqs. (3), (5)–(9) and Appendix B.1; Figure 9 against its caption and Appendix B.2. Figure 11's >= closing condition agrees with Appendix D.1. Page 2 and reference-only pages 10–12 were text-read but not visually inspected. Separate supplemental videos and external implementations remain outside this reading.

PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Title, authors, abstract and version stamp (p. 1)
1 Introduction; 2 Related Work (pp. 1–3)
3 Method, including 3.1–3.3 (pp. 3–5)
4 Experiments, including 4.1–4.3 (pp. 5–8)
5 Discussion and limitations (pp. 8–9)
References (pp. 9–12)
A Supplementary Experiment Results (pp. 13–14)
B Experimental Details, B.1–B.6 (pp. 14–18)
C Network Specifications (pp. 18–19)
D Data for World Model, D.1–D.4, including the quoted alignment prompt (pp. 19–22)
E World Model Training and Evaluation, E.1–E.5 (pp. 22–23)
F Failure Cases Analysis, F.1–F.3 (pp. 24–25)
G Broader Impact; H Remarks on Related Work (pp. 25–26)

Outside the original text pass

Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
The preceding acquisition limitation was addressed by visually inspecting every numbered figure and table in the retained PDF, alongside the method and appendix evidence pages.
Separate supplemental material availability has not been fully verified.
The supplied artifact is arXiv:2603.10422v2, dated 29 May 2026. Its title and nine-author list match the catalog; the catalog submission date is 11 March 2026. The earlier edition was not supplied, so changes from v1 cannot be established.
All eight supplied text chunks were individually read, covering all 26 pages. External references, project pages, separate videos, code and processed datasets were not inspected; no experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title block, author affiliations and arXiv margin stampInspect

World2Act: Latent Action Post-Training from World Model Dynamics; arXiv:2603.10422v2, 29 May 2026. Authors: An Dinh Vuong; Tuan Van Vo; Abdullah Sohail; Haoran Ding; Liang Ma; Xiaodan Liang; Anqing Duan; Ivan Laptev; Ian Reid. All list MBZUAI; the first two share equal contribution.

1. Paper overview

Core contributions

2. Motivation

2.1 The problem and the proposed response

2.2 What this reading follows

3. Research context

3.1 Evidence-based assessment

4. Problem formulation

4.1 Inputs and outputs

4.2 Equations and their role

5. Method in detail

5.1 First make video similarity mean something about controls

5.2 Then teach a correction while keeping the execution path small

5.3 Judge transfer at three different levels of evidence

5.4 Training and inference

During training

During inference

5.5 Implementation flow

6. Experiments & results

6.1 Read the original evidence

6.2 Results and evaluation conditions

6.3 Ablations and diagnostic examples

7. Analysis & limitations

7.1 What the evidence leaves open

7.2 Questions for discussion

8. Reproducibility audit

8.1 Requirements and known gaps

8.2 Proposed reproduction checks

Check 1: Distinguish temporal guidance from task-identity matching

Check 2: Separate the residual's benefit from the frozen decoder's effect

8.3 Reading coverage

Outside the original text pass

8.4 Traceable evidence

8.5 Primary sources

World2Act: Latent Action Post-Training from World Model Dynamics ↗