Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising
1. Paper overview
In one sentence: X-WAM couples geometric supervision with early action decoding, improving the reported quality/latency tradeoff while retaining the delays and context limits of a large video-based controller. e03e05e06e12e14e15e17e18e20
| At a glance | What to know |
|---|---|
| Research problem | Author claim A unified robot model must predict executable movement and coherent visual and geometric futures despite their different computational demands. The authors argue that pixel-only world-action models lack spatial grounding, while synchronized video/action diffusion makes control wait for expensive video synthesis. X-WAM addresses both problems within a shared predictor. e02 |
| Core mechanism | Source description A replicated, interleaved depth branch learns inverse depth from shared features without adding depth tokens to the main denoising sequence. e05 |
| A key reported result | RoboCasa closed-loop manipulation: 79.2 Average success rate (%). 24 simulated tasks; pretrained then benchmark-fine-tuned model; 100 evaluation episodes per task. Cosmos Policy 67.1; a 12.1 percentage-point lead. Strong benchmark performance under differing baseline pretraining. Per-task outcomes include TurnOffStove 35.0% and CloseDrawer 100.0%. e10e11e12e21 |
| Reading caution | Author claim The authors identify fixed context without history or autoregressive rollout, and inference delay that leaves the robot acting on older observations. RTC hides pauses but does not eliminate stale predictions. e20 |
Core contributions
- Source description
A replicated, interleaved depth branch learns inverse depth from shared features without adding depth tokens to the main denoising sequence. e05
- Source description
Asynchronous Noise Sampling couples training noise levels and explicitly includes clean-action conditioning, supporting early action release and subsequent video-only denoising. e06
Figure 2. Shared prediction, one-way geometric readout and early action release are distinct parts of the same architecture. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the left panel downward: camera images enter the causal VAE, while state and action inputs use projectors. Shared blocks feed both branches. The blue dashed arrow permits main-to-depth information; the crossed reverse arrow forbids depth-to-main attention. Equation (3) and Algorithm 1 clarify that depth reads the main branch’s input at each interleaved layer. On the right, the horizontal axis is action noise and the vertical axis is video noise. The proposed sampler occupies t_O≥t_a and explicitly includes t_a=0. In the lower plot, the orange action curve reaches zero at T_a while the blue video curve continues to T_O. e03e05e06e22
What it supports. The model can learn a geometric readout and still release executable actions before completing video synthesis. The figure’s arrows and inequalities agree with the equations and algorithms. Its noise-zero line is consequential: the later video phase conditions on already clean actions, so training explicitly allocates probability to that regime.
Where the evidence stops. The crossed arrow denotes an attention restriction, not a documented stop-gradient or frozen backbone. The schematic aligns training support with inference phases; it does not prove exact equality between the training distribution and discrete inference trajectories.
2. Motivation
2.1 The problem and the proposed response
A unified robot model must predict executable movement and coherent visual and geometric futures despite their different computational demands. The authors argue that pixel-only world-action models lack spatial grounding, while synchronized video/action diffusion makes control wait for expensive video synthesis. X-WAM addresses both problems within a shared predictor. e02
2.2 What this reading follows
A robot can begin moving before its model finishes imagining the scene. X-WAM builds this separation into a unified predictor: RGB, robot states and actions share a video diffusion transformer, while a one-way depth branch learns the geometry needed for multi-view reconstruction. Its training noise distribution includes the clean-action, noisy-video situation that arises after actions are ready. Read the architecture first, then distinguish the pretrained benchmark results from the smaller ablations, and finally examine physical deployment. The paper’s evidence spans executed simulation policies, geometric prediction and real-robot stage progress; these measure different capabilities and should not be collapsed into one success claim. e03e05e06e12e14e15e17e18e20
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is unassigned, so no recorded quadrant can be confirmed or contradicted. The shared bidirectional video/state/action DiT supports a One Model, joint future/action prediction assessment. Depth is auxiliary; action extraction uses no separate inverse-dynamics model and need not wait for completed video. e03e05e06e22
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Put observations and actions on a shared temporal axis
Begin with one multi-view observation and the current proprioceptive state, both held clean throughout denoising. The model adds noisy future RGB, state and action latents to the same sequence, with language as a condition. RGB uses the pretrained causal VAE; state and action use learned MLP interfaces. Their common transformer applies bidirectional attention, so action prediction can use partially denoised visual features. The temporal horizons differ: eight future frames and states align with thirty-two actions over the same interval. View embeddings identify cameras, while temporal RoPE helps associate the faster action sequence with the slower visual sequence. This is joint future/action prediction: an independent inverse-dynamics network does not wait to translate a completed future video into controls. e03e06
5.2 Learn geometry without requiring depth decoding for every action
The geometry branch begins from shared hidden states rather than a separate depth-token sequence. At each of the replicated final blocks, it reads the main branch’s input through one-way attention and predicts inverse depth. The source allows this branch to be disabled during action decoding. Reader interpretation: its policy benefit can therefore arise through training supervision of shared representations, rather than through an explicit depth-to-action connection during each inference step. This distinction matters when reading the no-depth ablation: equal decoding latency does not mean identical training. For reconstruction, the branch supplies depth while predicted end-effector poses and hand-eye calibration place wrist views into a shared coordinate frame. A good depth map alone is insufficient if the moving-camera pose is wrong. e04e05e14e15
5.3 Train the noise regimes that early action release will encounter
The two denoising schedules start from noise but finish at different times. State and action reach clean outputs first; if video generation continues, their latents stay fixed while video noise decreases. ANS trains both phases. With probability p, actions are already clean and video noise is sampled uniformly. Otherwise, action noise is sampled first and a Beta variable places video noise between that level and one. This keeps video at least as noisy as actions while emphasizing noisier video conditions. Algorithm 2 implements the handoff explicitly. Benchmark evaluation uses ten action and fifty video steps; the ablation instead studies five and twenty-five. The relevant causal test is the quality difference between training samplers at the same asynchronous inference budget, not merely the difference in step counts. e06e09e10e15
5.4 Training and inference
During training
Pretraining uses 1,492,026 episodes totaling 5,873.9 hours from AgibotWorld-Beta, DROID, InternA1 variants, RoboCasa MimicGen and RoboTwin 2.0. Videos become 320×256 at 3.75 FPS; Video Depth Anything supplies depth labels. Filtering excludes locomotion, dexterous manipulation, failed episodes and stationary DROID frames. e08
Pretraining runs 40,000 AdamW steps on 256 H20 GPUs, batch 2,048, peak learning rate 10^-4, 1,000 warmup steps and cosine decay. Loss weights equal one; ANS uses p=0.5. Benchmark fine-tuning uses 32 H20 GPUs, batch 128, 20,000 steps and learning rate 3×10^-5. e09e10
Benchmark depth comes from replaying official demonstrations without changing data volume or initial configurations; replay seeds avoid test overlap. RoboCasa uses raw dataset actions. RoboTwin relative actions become absolute end-effector poses using the action chunk’s initial state. e10
During inference
Separate UniPC schedules use 10 action steps and 50 video steps for benchmark evaluation, with guidance scale 1.0. Actions can be dispatched immediately when clean; remaining video steps condition on fixed clean states/actions. Depth can be disabled during action decoding. e05e06e10
Physical deployment uses eight steps on an RTX 5090 D, approximately 300 ms per action chunk, and Real-Time Chunking (RTC) to overlap computation with execution. Control runs at 15 Hz, executing 15 actions per chunk with a six-action inference delay. e16
5.5 Implementation flow
- Encode a shared sequence
Wan2.2-TI2V-5B supplies the video DiT and causal VAE. MLPs encode/decode states and actions. Initial observation/state tokens remain clean; future tokens use bidirectional full attention. View embeddings distinguish cameras, and temporal RoPE aligns eight future video/state samples with thirty-two actions. e03
- Read geometry through an auxiliary branch
After the shared trunk, ten replicated final blocks form a depth branch. Each reads the main branch’s input at the corresponding layer; the main branch cannot attend back to depth. Depth maps use the RGB VAE representation after three-channel replication, with inverse-depth MSE supervision. e05e09
- Recover a common spatial frame
Predicted end-effector poses and fixed hand-eye calibration yield wrist-camera poses. These and static-camera poses permit multi-view RGB-D fusion. Thus reconstruction depends on state prediction and calibration as well as depth. e04e14
- Represent heterogeneous robots
States are 16-dimensional absolute dual-arm position/quaternion/gripper vectors; actions are 14-dimensional relative position/axis-angle/gripper vectors. Single-arm data supervise only the first arm. Dataset-specific quantiles normalize values; action scaling omits bias to preserve zero movement. e09
6. Experiments & results
X-WAM adapts a pretrained video diffusion transformer to predict robot actions, future states and multi-view RGB-D observations together. A one-way depth branch adds geometric supervision, while asynchronous denoising releases actions before completing video generation. The paper reports strong simulated manipulation and reconstruction results plus a small physical earphone-packing evaluation. Its central tradeoff is useful spatial supervision without depth decoding during every action step; limited temporal context and delayed control remain unresolved.
6.1 Read the original evidence
Tables 1 and 2. X-WAM leads the listed simulation averages under each benchmark’s own protocol. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two tables independently. The left averages success over twenty-four RoboCasa tasks; the right averages fifty RoboTwin tasks and separates Clean from Randomized evaluation. All cells are percentages. Appendix B specifies one hundred evaluation episodes per task and benchmark inference with ten action steps and fifty video steps. Compare X-WAM with Cosmos Policy in the left table and Motus in the right, rather than comparing percentages across environments. These are the main pretrained-policy results. The authors import most baseline values from earlier papers and implement selected Wan-based baselines themselves, so the rows do not represent one uniformly retrained experiment. e10e11e12e13e21
What it supports. X-WAM reports 79.2% on RoboCasa versus Cosmos Policy’s 67.1%, a 12.1 percentage-point difference. RoboTwin results are 89.8% Clean and 90.7% Randomized, versus Motus at 88.7% and 87.0%. These support strong performance on the specified simulation benchmarks, while the appendix’s per-task outcomes show that high averages still contain substantial task failures.
Where the evidence stops. Pretraining datasets and implementations differ across baselines, and the tables give no uncertainty intervals. The Randomized column uses the benchmark’s defined variation; it cannot establish arbitrary out-of-distribution physical robustness.
Table 3. Integrated depth prediction improves the reported geometry relative to post-hoc depth estimation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Move across the grouped headers: RGB quality uses PSNR, SSIM and LPIPS; depth uses AbsRel and delta_1; reconstructed point clouds use Chamfer Distance. Follow each arrow, since high PSNR but low LPIPS is desirable. The third row is particularly informative: it keeps X-WAM’s RGB generator without its depth branch and applies Depth Anything 3 afterward. Compare that row with the last to examine integrated versus post-hoc geometry. Section 4.2 restricts pixel metrics to the two static cameras because predicted wrist poses cause pixel misalignment. The fused point cloud, by contrast, incorporates multiple viewpoints and therefore also tests pose consistency. e04e14
What it supports. Relative to X-WAM without depth plus DA3, the integrated model reduces AbsRel from 0.1045 to 0.0349 and Chamfer Distance from 0.0401 to 0.0049, while PSNR changes from 23.09 to 23.46 dB. The largest contrast is geometric: similar-looking RGB predictions can produce substantially different reconstructed spatial structure.
Where the evidence stops. The pixel scores exclude wrist views. Chamfer Distance mixes depth, viewpoint consistency and pose errors, so this table does not isolate each source. Reconstruction sample counts and CD units are unspecified, limiting exact reproduction.
Table 8. Physical progress improves in several settings, with time reported only for completed episodes. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the original caption’s averaging rule in view: progress includes all six episodes, but completion time includes only episodes reaching full progress. The four stages are opening the case, inserting one earbud, inserting the other, and closing/returning the case. The table’s ‘Pack 1/2/3 earphones’ labels denote the source’s repeated packing settings; their stage denominators are twenty-four, forty-eight and seventy-two, not numbers of trials. The lower block tests novel placements, tablecloth colors and distractors. Compare progress before time, since a method can have a different set of successful episodes contributing to its time average. e16e17e18
What it supports. X-WAM’s pack-two progress is 93.8% (45/48 stages) versus XR-0’s 79.1% (38/48). Novel-placement progress is 70.8% versus 58.3%. Both methods reach 100% in the single-packing setting and 66.7% on unseen tablecloths. Thus the table supports several improvements and two progress ties, with shorter reported completion times throughout.
Where the evidence stops. Six trials per setting provide limited uncertainty information. Stage progress is not episode success, and success-conditioned times do not directly measure all-attempt throughput. The physical comparison does not independently isolate depth supervision, ANS or RTC.
Figure 4. Original physical keyframes illustrate the workspace and bimanual manipulation, while aggregate outcomes remain in Table 8. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the numbered sequence left to right across the upper row, then continue through the lower row. The photos show the AC One arms changing configuration around a small tabletop object; they are execution keyframes rather than generated RGB-D predictions. Read them alongside Appendix D’s four-stage task description, without assigning an exact stage or insertion success to every partially occluded image. The platform has one main camera and two wrist cameras, although these frames alone do not show all input streams. This figure gives visual context for why grasping, holding a case and inserting small parts require coordinated viewpoints and movements. e16e17e19
What it supports. The paper supplies a representative physical execution, complementing its simulation measurements. Appendix D reports roughly 300 ms action-chunk inference with eight denoising steps and overlaps computation with execution using RTC. The photographs ground the deployment setting; the latency and control-frequency claims come from the setup text, not from spacing between these images.
Where the evidence stops. These selected stills have no timestamps and cannot establish continuous motion, per-stage success rates or how often corrections occur. No project video was inspected, and the source-selected example is not evidence of typical reliability.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboCasa closed-loop manipulation 24 simulated tasks; pretrained then benchmark-fine-tuned model; 100 evaluation episodes per task. | 79.2 Average success rate (%) | Cosmos Policy 67.1; a 12.1 percentage-point lead. Strong benchmark performance under differing baseline pretraining. Per-task outcomes include TurnOffStove 35.0% and CloseDrawer 100.0%. e10e11e12e21 |
| RoboTwin 2.0 closed-loop manipulation 50 AgileX tasks; Clean/Randomized protocols; 100 evaluation episodes per task. Training includes 50 clean and 500 randomized trajectories per task. | Clean 89.8; Randomized 90.7 Average success rate (%) | Motus: 88.7 and 87.0, respectively. The randomized score concerns specified benchmark variations, not unrestricted real-world generalization. e10e11e13 |
| RoboCasa 4D reconstruction and generation Predictions versus simulator renders during policy execution; pixel metrics use only two static cameras; fused point clouds assess multi-view geometry. | 23.46 dB; 0.8942; 0.0513; 0.0349; 0.9738; 0.0049 PSNR↑, SSIM↑, LPIPS↓, AbsRel↓, delta_1↑, Chamfer Distance↓ | Without depth plus DA3: AbsRel 0.1045, CD 0.0401. Robot4DGen: LPIPS 0.1026, CD 0.0134. Supports integrated geometry; wrist-camera pixel accuracy is excluded, and CD units are unspecified. e14 |
| RoboCasa architecture and scheduling ablations Direct Wan2.2 fine-tuning without robotic pretraining; one RTX 3090; five asynchronous action steps versus 25 synchronous steps. | Interleaved depth/ANS: 67.8% SR, 1,033 ms; no depth: 63.0%, 1,033 ms. Success rate, action latency and reconstruction quality | Sequence depth: 68.7%, 1,888 ms. Synchronous: 66.4%, 4,665 ms. Decoupled asynchronous: 67.2%, 1,033 ms, PSNR 22.60 versus ANS 23.46. Depth helps at unchanged decoding latency; sequence depth has better quality but costs more. ANS improves fast-decoding generation quality. This differs from the 79.2% pretrained setting. e15 |
| Physical earphone packing AC One dual-arm robot; approximately 20 demonstration hours; six trials per setting. | Pack 2: 93.8% (45/48), 113.25 s; novel placements: 70.8% (17/24), 46.68 s. Average stage progress (%); time over fully completed episodes only (s) | XR-0: 79.1% (38/48), 115.44 s; novel placements: 58.3% (14/24), 89.63 s. Counts represent stages, not successful episodes. Small samples and success-conditioned timing limit reliability and speed conclusions. e16e17e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. The preferred branch trades a small quality loss against sequence depth for lower action latency; ANS improves asynchronous generation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read block (a) as an architectural cost comparison and block (b) as a scheduling comparison. The source uses bold for best and underline for second best. These variants are fine-tuned directly from Wan2.2 without the large robotic pretraining stage, which explains why the preferred row has 67.8% success rather than the main table’s 79.2%. Latency measures action generation on one RTX 3090. Synchronous variants use twenty-five joint denoising steps; asynchronous variants release actions after five. First compare no depth against the interleaved branch, then compare the two asynchronous rows to assess ANS at the same measured latency. e15
What it supports. The interleaved branch improves success from 63.0% to 67.8% while retaining 1,033 ms action latency. Sequence depth reaches 68.7% with better reconstruction but costs 1,888 ms. Under asynchronous inference, ANS improves PSNR from 22.60 to 23.46 compared with independent sampling. Early release reduces latency from the synchronous 4,665 ms to 1,033 ms.
Where the evidence stops. ANS does not win every reconstruction column: synchronous training/inference has better RGB scores and CD. No uncertainty intervals establish significance of the 0.6-point asynchronous success difference. These reduced-training results cannot isolate the full pretrained model’s gains.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify fixed context without history or autoregressive rollout, and inference delay that leaves the robot acting on older observations. RTC hides pauses but does not eliminate stale predictions. e20
Comparisons mix published baselines and authors’ implementations with different pretraining; averages lack uncertainty intervals. Physical results chiefly compare XR-0. The authors omitted pi_0.5 from Table 8 after no completed episode and only 25–50% single-earphone progress. e11e12e13e17e18
ANS covers inference’s noise ordering and clean-action phase; its two-dimensional training mixture does not establish exact equality with a discrete inference trajectory. Ablations support a quality/latency tradeoff, not universal necessity of explicit depth. e06e15
7.2 Questions for discussion
- Does depth supervision help after matching large-scale pretraining and multiple seeds?
- How does success change with progressively older observations?
- How much fused-geometry error comes from wrist-pose prediction rather than depth?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the Wan checkpoint, listed data, depth labels, simulator replay, normalization and camera calibration. Physical fine-tuning uses 64 H20 GPUs, batch four per GPU, for 40,000 steps. Software versions, calibration values, full filtering thresholds and a complete frozen-module policy are unspecified. e03e04e05e08e09e10e16
A smaller proposed check can repeat the no-pretraining RoboCasa ablation with matched data/seeds: compare ANS with independent sampling at the same asynchronous schedule, and depth supervision on/off with depth decoding disabled. Measure policy success separately from reconstruction and latency. e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does spatial supervision improve actions when geometry decoding is absent?
Reader-proposed check: repeat the direct-from-Wan RoboCasa setting with identical data, training budgets, seeds and interleaved architecture. Compare correct inverse-depth targets, zero depth-loss weight, and shuffled depth targets; keep the depth branch disabled during action decoding in every condition. Evaluate matched episodes for success, action latency and held-out geometry. A repeatable benefit only from correct targets, at equal decoding latency, would support spatial supervision as the mechanism. Equal gains from shuffled targets would weaken that interpretation and motivate testing generic auxiliary regularization. Report seed variation rather than treating a single percentage difference as decisive. e05e07e15
Check 2: Separate the coupled-noise benefit from clean-action training
Reader-proposed check: hold the five-action/twenty-five-video inference schedule, initial checkpoint, data and optimization budget fixed. Compare independent sampling, coupled sampling with p=0, and the reported ANS mixture with p=0.5. Across matched tasks and seeds, measure policy success, video quality after action release, depth/point-cloud metrics and action latency. If the clean-action branch explains the late-video benefit, p=0.5 should improve that phase beyond coupling alone without requiring more action steps. A result confined to differing compute or dataset exposure would fail this mechanism test. This is a proposed experiment, not a reproduction result. e06e09e14e15
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version, Figures 1–4, Tables 1–8, Eqs. (1–6), Algorithms 1–2 and supporting method, training, protocol and limitation text on the declared pages. Figure 2 arrows, forbidden reverse connection, t_O≥t_a region, clean-action line and early-release trajectory were cross-checked against Eqs. (3–4) and Algorithms 1–2; no claim-relevant discrepancy was found. All six final original crops were separately viewed and retain relevant labels, headers and legends; Table 8 retains its averaging-rule caption. All seven source chunks were read, including references on pages 10–15; those reference-only pages were not visually inspected. External media, code and separate supplements were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 17, 18, 19, 20, 21. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and Abstract
- 1 Introduction
- 2.1 Unified World Action Modeling
- 2.2 3D Modeling in Embodied Models
- 3.1 Model Architecture
- 3.2 Lightweight Depth Adaptation
- 3.3 Asynchronous Noise Sampling
- 3.4 Training Details
- 4.1 Policy Evaluation
- 4.2 4D Reconstruction and Generation
- 4.3 Ablation Studies
- 5 Conclusion
- References
- A Detailed Algorithms (Algorithms 1–2)
- B.1 Pretraining Data
- B.2 Implementation Details
- B.3 Baseline Details
- C.1 Per-Task Results on RoboCasa
- C.2 Per-Task Results on RoboTwin 2.0
- D Real Robot Experiments
- E Limitations and Future Work
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only visual gap was resolved by inspecting the retained PDF figures, tables, equations and algorithms. All seven supplied text chunks were read individually, including references and Appendices A–E. Reference pages 10–15 were read as text, not rendered.
- Identity/edition note: the inspected title page identifies arXiv:2604.26694v2 [cs.RO], 7 May 2026, with the catalog title and all ten authors. The catalog submission date is 29 April 2026; this report concerns the supplied v2 artifact. No earlier version or revision comparison was supplied, so changes from v1 remain unverified.
- No external project media, code or separate supplement was inspected, and no experiment was reproduced. The project link does not establish code availability.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, byline, affiliations and arXiv margin
Exact title and ten authors match the catalog. The artifact is arXiv:2604.26694v2, dated 7 May 2026; affiliations are Tsinghua University, Xiaomi Robotics, Peking University and CASIA.
Go to primary source ↓e02PDF pp. 2–3, Section 1; Figure 1
Motivation combines spatial grounding, unified policy/world prediction and lower action latency; depth adaptation and ANS are the proposed mechanisms.
Go to primary source ↓e03PDF p. 5, Section 3.1, Eq. (1)
Wan2.2-TI2V-5B, causal VAE, state/action MLPs, clean initial conditions, full attention, view embeddings and temporal RoPE; H=8 and K=32.
Go to primary source ↓e04PDF p. 5, Section 3.1, Eq. (2)
Fixed static-camera poses and predicted end-effector poses with hand-eye calibration provide extrinsics for multi-view fusion.
Go to primary source ↓e05PDF pp. 5–6, Section 3.2, Eq. (3); p. 16, Algorithm 1, lines 7–13
Depth shares the VAE representation, reads main-branch inputs unilaterally and regresses inverse depth. Its branch can be toggled at inference; the text does not state that the shared backbone is frozen.
Go to primary source ↓e06PDF p. 6, Section 3.3, Eq. (4); p. 17, Algorithm 2
ANS combines clean-action and Beta-coupled branches satisfying t_O≥t_a. States/actions finish first, after which only video updates; actions are available after T_a.
Go to primary source ↓e07PDF pp. 6–7, Section 3.4, Eqs. (5–6)
Flow matching predicts noise minus clean latent for video/state/action; inverse-depth MSE supplies geometry supervision. States and actions share a noise level.
Go to primary source ↓e08PDF p. 16, Appendix B.1 and Table 5
Seven dataset rows total 1,492,026 episodes and 5,873.9 hours. Processing uses 320×256 at 3.75 FPS, Video Depth Anything labels and the stated episode/frame exclusions.
Go to primary source ↓e09PDF p. 17, Appendix B.2, representation and pretraining paragraphs
States/actions have 16/14 dimensions, single-arm masking and quantile normalization. Pretraining: 256 H20 GPUs, batch 2,048, 40,000 steps, AdamW, peak learning rate 10^-4, 1,000-step warmup, cosine decay, M=10, p=0.5, unit loss weights.
Go to primary source ↓e10PDF pp. 17–18, Appendix B.2, benchmark fine-tuning and inference
Fine-tuning: 32 H20 GPUs, batch 128, 20,000 steps, learning rate 3×10^-5. Replay seeds avoid test overlap. Benchmark UniPC uses 10/50 action/video steps, guidance 1.0 and 100 episodes per task.
Go to primary source ↓e11PDF p. 18, Appendix B.3
Most baseline values are imported. DreamZero on RoboCasa and UWM on RoboTwin use authors’ Wan2.2-5B implementations. DreamZero lacks robotic pretraining; other baselines have differing robotic pretraining.
Go to primary source ↓e12PDF p. 7, Section 4.1; p. 8, Table 1, X-WAM and Cosmos Policy rows
RoboCasa averages 24 tasks: X-WAM 79.2%, Cosmos Policy 67.1%; no uncertainty intervals appear.
Go to primary source ↓e13PDF p. 7, Section 4.1; p. 8, Table 2, X-WAM and Motus rows
RoboTwin covers 50 tasks and clean/randomized training trajectories. X-WAM scores 89.8/90.7%; Motus 88.7/87.0%.
Go to primary source ↓e14PDF p. 8, Table 3, all rows/metric columns; Section 4.2
X-WAM records 23.46, 0.8942, 0.0513, 0.0349, 0.9738 and 0.0049 for PSNR, SSIM, LPIPS, AbsRel, delta_1 and CD. Pixel metrics exclude the wrist camera; point clouds combine viewpoints. CD units and reconstruction sample counts are not given.
Go to primary source ↓e15PDF p. 9, Table 4(a–b), all rows; Section 4.3
Ablations omit robotic pretraining. RTX 3090 latency: no/interleaved depth 1,033 ms at 63.0/67.8% SR; sequence depth 1,888 ms at 68.7%. Synchronous latency is 4,665 ms; decoupled/ANS asynchronous PSNR is 22.60/23.46 with five action steps.
Go to primary source ↓e16PDF pp. 18–19, Appendix D, Setup; Figure 3
AC One: one main/two wrist cameras at 320×256; about 20 demonstration hours; 64 H20 GPUs, batch four per GPU, 40,000 fine-tuning steps. Eight-step RTX 5090 D inference takes about 300 ms/chunk; RTC uses 15-Hz control, 15 executed actions and a six-action delay.
Go to primary source ↓e17PDF pp. 19–20, Appendix D, task design, settings and quantitative-results paragraphs
Four stages contribute 25% each; six trials per setting test length, placements, tablecloths and distractors. Time averages only completed episodes. The authors omit pi_0.5 after no complete episodes and 25–50% single-earphone progress.
Go to primary source ↓e18PDF p. 21, Table 8, caption and all rows
XR-0/X-WAM progress: pack one 100/100, two 79.1/93.8, three 63.9/68.0, placements 58.3/70.8, tablecloth 66.7/66.7, distractors 66.7/75.0. Times: 54.66/41.63, 115.44/113.25, 195.66/160.72, 89.63/46.68, 65.73/62.01, 76.32/51.53 seconds.
Go to primary source ↓e19PDF p. 21, Figure 4 and caption
Ten numbered keyframes depict an author-selected representative physical AC One execution; they supply neither continuous timing nor aggregate success.
Go to primary source ↓e20PDF p. 21, Appendix E
Fixed-context/no-history and latency limitations remain; RTC does not remove delayed-observation effects. History, KV caching, autoregression and acceleration are future directions.
Go to primary source ↓e21PDF p. 19, Table 6, CloseDrawer and TurnOffStove rows; p. 20, Table 7, task rows and Average
RoboCasa success includes CloseDrawer 100.0% and TurnOffStove 35.0%; Table 7 supplies separate clean/randomized per-task RoboTwin outcomes.
Go to primary source ↓e22PDF p. 4, Figure 2(a–b) and caption; p. 6, Eqs. (3–4); pp. 16–17, Algorithms 1–2
The allowed arrow goes from main to depth; the reverse path is crossed out. Joint-noise plots label t_O≥t_a and t_a=0; inference releases actions at T_a while video continues to T_O. Markers agree with the formulation.
Go to primary source ↓8.5 Primary sources
Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising ↗
PDF · 10,609 extracted words
Source fingerprint
2de8a99be736c0a4715238aa61b9610802f7e4c961e1e23ede6bebb426a20332