SelfWAM: A Self-Grounded Unified World Action Model for Fast Robot Control
1. Paper overview
In one sentence: SelfWAM uses selectively action-conditioned RGB and robot-mask prediction to improve a direct policy while keeping future generation out of the deployed control loop. problemattentionmaskssim-resultsphysical-resultsfidelityobjective
| At a glance | What to know |
|---|---|
| Research problem | Source description Observation-conditioned future prediction can learn likely task progression without distinguishing which action caused it. SelfWAM asks whether conditioning visual learning on demonstrated actions, and emphasizing the visible robot body, improves the policy while preserving action-only execution. problem |
| Core mechanism | Source description A directed clean-action route grounds future visual prediction while preventing clean targets and future observations from entering action-prediction queries. attention |
| A key reported result | RoboTwin 2.0 multitask policy success: SelfWAM: 92.16 clean, 93.08 randomized, 92.62 average. Unweighted mean task success (%). 50 tasks; 100 rollouts per task in each clean/randomized setting; same 27,500-demonstration training dataset. FastWAM: 91.82, 91.86, 91.84; Motus average 87.84; GigaWorld-Policy 85.70; pi0.5 79.75; pi0 62.16. Calculated gains over FastWAM are 0.34 and 1.22 percentage points, averaging 0.78. Task gains vary: randomized Open Microwave improves 41→67%, while Dump Bin Bigbin falls 98→91%. sim-protocolsim-resultstask-breakdown |
| Reading caution | Source description The authors explicitly exclude contact, force and occluded interactions from the modeled targets. limitations |
Core contributions
- Source description
A directed clean-action route grounds future visual prediction while preventing clean targets and future observations from entering action-prediction queries. attention
- Source description
Prompt-selected robot self-mask prediction adds body-motion supervision through the existing video backbone; it does not introduce another expert or a deployment segmentation requirement. masksphysical-data
Figure 2. Clean actions condition future video through a one-way information boundary. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the middle matrix: rows are queries, columns are keys, colored cells permit attention and white cells block it. The future-video row can read the first frame, future video and clean action. The future-action row can read only the first frame and its own noisy action tokens. These entries agree with Equation 6; text and proprioceptive conditioning are described separately in the method. The left panel retains distinct video/action projections and feed-forward blocks around mixed attention. On the right, action prediction removes both clean actions and future video, while optional video rollout keeps a supplied clean action. architectureattentionmasksconfigurationpath-consistency
What it supports. The architecture can train an action-conditioned consequence model without giving the policy access to its demonstrated answer. The same video backbone serves RGB and self-mask prompts, so body supervision adds an output domain rather than another expert. Deployment keeps current-context encoding and iterative action denoising.
Where the evidence stops. This matrix establishes permitted information flow, not empirical proof that every implementation avoids leakage. The source does not fully specify action-expert dimensions or frozen-module choices; the zero-noise appendix diagnostic addresses representation consistency only.
2. Motivation
2.1 The problem and the proposed response
Observation-conditioned future prediction can learn likely task progression without distinguishing which action caused it. SelfWAM asks whether conditioning visual learning on demonstrated actions, and emphasizing the visible robot body, improves the policy while preserving action-only execution. problem
2.2 What this reading follows
A robot can predict a plausible next scene without knowing which action produced it. SelfWAM addresses that ambiguity during policy training: its future-video branch sees the demonstrated action, while its action-prediction branch is blocked from that answer. A second visual target, the robot’s own silhouette, emphasizes body motion using the same video backbone. This edition follows that information boundary into the experiments. Simulation success improves modestly overall, the physical comparison contains only ten trials per task, and the strongest video-fidelity numbers concern recorded demonstration trajectories. The central distinction is between learning from action-conditioned futures and requiring those futures to execute a policy. problemattentionmaskssim-resultsphysical-resultsfidelityobjective
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system label is supported at the modality-expert level: distinct video/action projections and feed-forward networks interact through directed mixed attention. The authors’ word unified does not imply one shared Transformer. Other mechanisms and Outside quadrants fit direct action denoising aided by training-time action-conditioned world prediction, without mandatory joint future/action generation or inverse-dynamics extraction at deployment. architectureattentionobjective
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why the world model can see the answer while the policy cannot
Begin with two different prediction problems. The policy predicts an action chunk from current RGB, language and proprioception; the world model predicts a future given that context plus an action. During training, the same demonstrated chunk supplies a noisy action target and a clean conditioning copy. Sharing action blocks and temporal positions aligns their representation, while timestep zero identifies the clean copy. Equation 6 keeps future-video and clean-action information out of noisy-action queries. The visual branch can therefore learn the consequence of the demonstrated action without turning the policy problem into copying its answer. Appendix D checks aligned representations when noise is removed and reports very small discrepancies. That check supports consistency of the two paths at zero noise; it does not establish correct dynamics for arbitrary actions. problemattentionpath-consistency
5.2 Why silhouettes change the learning target without changing policy inputs
RGB prediction must reconstruct the robot, objects and appearance variation together. SelfWAM adds a prompt-selected target containing the robot’s binary silhouette, using the same video backbone and RGB context. Appendix A specifies common temporal offsets and the same three-view layout for RGB and masks. The mask prompt changes what is predicted, not what the policy sees. Only RGB instances include noisy-action tokens and action loss; mask instances contribute visual loss alone, with a stated 9:1 sampling ratio. Simulation supplies robot masks directly, whereas physical targets are generated offline with RobotSeg and require no segmentation during deployment. The main text describes two available instance types per demonstration; the appendix describes sampling either type. A reproduction must implement both the target alignment and the supervision accounting, rather than simply doubling every policy example. problemmasksconfigurationsim-protocolphysical-data
5.3 How to connect predictive learning to the deployed controller
The deployment loop encodes the latest context, predicts 32 actions with ten denoising steps, executes 24 and observes again. Clean-action slots, future RGB and masks are absent. The H200 experiment measures 323.9 ms for that action-only computation, close to FastWAM’s 320.7 ms; the robot’s 30 Hz low-level command rate is a separate quantity. Reader interpretation: the experiments test whether richer training objectives help an inexpensive policy path, not whether online imagination improves action selection. Three evidence layers should remain distinct. Policy tables concern executed task success, Table 3 concerns matching demonstration videos, and Figure 4 plus Appendix C concern sensitivity to changed actions. The observed gains across these layers are compatible with the authors’ motivation, but they do not by themselves prove that more sensitive predictions cause better control. objectiveconfigurationlatencyphysical-datasim-resultsphysical-resultsfidelityperturbationsensitivity
5.4 Training and inference
During training
Flow matching combines action loss with a 9:1 RGB/self-mask mixture, using unit action and visual weights. Appendix A describes sampling either instance type; action supervision occurs only on RGB instances. Training runs five epochs with AdamW at 1e-4, global batch 1024 across 64 A800 GPUs, taking approximately 30 hours. masksobjectiveconfiguration
Simulation uses 27,500 demonstrations. Physical training uses 17,133 episodes and 9,461,919 transitions; an episode-level seed-42 split reserves 174 episodes and 95,500 transitions for validation. Compared physical methods use the same training split. sim-protocolphysical-data
During inference
Encode current context once, denoise 32 actions in ten steps, execute the first 24 and replan. Clean-action and future-visual tokens are absent. Physical actions contain six joint targets and one gripper command per arm; low-level execution is 30 Hz. objectiveconfigurationphysical-data
Optional rollout inserts a policy sample or candidate action into clean slots and generates RGB or masks. The reported controller does not rank candidates with imagined futures or recover actions through inverse dynamics. attentionobjective
5.5 Implementation flow
- Encode context and align targets
Three camera views form a 384×320 T-shaped composite. A 32-action window is aligned with 32 future observations, temporally downsampled to eight visual targets; RGB and masks use matching offsets. WAN2.2-5B supplies the video backbone. problemconfiguration
- Separate modality computation
Video and action experts have their own projections and feed-forward layers around mixed attention. Appendix A qualifies the main text’s broad initialization statement: matching weights are copied, mismatched axes interpolated, and action-specific layers without video counterparts initialized separately. architectureconfiguration
- Expose the demonstrated action selectively
A second, uncorrupted action copy uses the same tokenizer and action blocks, denoising timestep zero and aligned positions. It conditions future visual queries but cannot be read by noisy action queries. It is a condition, not a second action target. attention
- Select the visual supervision domain
RGB and binary robot-mask instances retain RGB context and the same action condition. Changing the prompt selects the future target. Mask instances omit noisy-action tokens and action loss; simulation provides exact masks, while physical masks come from offline RobotSeg. maskssim-protocolphysical-data
6. Experiments & results
SelfWAM co-trains a direct robot policy with action-conditioned RGB and robot-mask prediction. Its two-expert architecture isolates action denoising from clean demonstration targets, while future-video queries can access those targets. At deployment it generates actions without future video. The evidence supports stronger action sensitivity and modest aggregate policy gains, with important limits on held-out video generalization and physical-trial scale.
6.1 Read the original evidence
Table 1. RoboTwin success improves by 0.78 percentage points on average over FastWAM. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read Clean and Random separately before looking at Average. Appendix B specifies 50 tasks, 100 rollouts per task per setting, and an unweighted task mean; all methods use the same 27,500-demonstration training dataset. The bottom two rows are especially informative because SelfWAM extends FastWAM’s efficient architecture. SelfWAM changes clean success from 91.82 to 92.16 and randomized success from 91.86 to 93.08. The average column summarizes those two settings rather than introducing another evaluation split. The upper rows provide broader policy comparisons, but they do not isolate the contribution of clean-action conditioning or mask prediction. sim-protocolsim-resultstask-breakdownablation
What it supports. The calculated improvements are 0.34 points in clean scenes and 1.22 under randomization, giving a 0.78-point mean gain. This supports a modest aggregate policy improvement with stronger gains under the tested randomization protocol. The component table is needed to identify how the two proposed changes contribute.
Where the evidence stops. No confidence intervals or training-seed variation accompany these means. Appendix Table S4 shows both gains and regressions across tasks, so the aggregate result does not establish uniform robustness or performance on unseen task families.
Table 2. Physical gains come from cup and mouse placement in a small trial set. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task row represents ten physical trials per method on the AgileX ALOHA platform. Read the individual rows before the bold average: SelfWAM and FastWAM tie on pen insertion and ball disposal, while cup and mouse placement each increase from 80 to 100 percent. Success requires grasping and placing the object at its specified destination without human intervention. Appendix E allows regrasping and correction within an episode, so this is episode success rather than first-attempt success. The compared methods share the training split and initial-state protocol; object and target positions vary within predefined workspace regions. physical-resultsphysical-dataphysical-protocol
What it supports. The average rises from 85 to 95 percent against FastWAM and from 90 to 95 percent against pi0.5. With 40 trials per method, these rates correspond to 38 successful episodes for SelfWAM, 34 for FastWAM and 36 for pi0.5, calculated from the task rates. This is evidence from executed robot control.
Where the evidence stops. Ten trials per task yield coarse estimates, and the source supplies no confidence intervals. The four tasks do not establish broad physical generalization. Exact placement-region bounds and a fixed maximum episode duration are not specified.
Table 3. Action-conditioned predictions better match recorded futures on the sampled demonstrations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The arrows in the header indicate the favorable direction: lower LPIPS and FVD-I3D, higher PSNR. Read each column within its own metric; their scales cannot be averaged. Here, unconditioned means the baseline video stream does not receive the demonstrated action, although it still receives observation context. The evaluation uses matched context windows over 275 trajectories sampled from the offline demonstration dataset. SelfWAM additionally receives the action aligned with each recorded future. Appendix A explicitly states that this analysis measures fidelity to demonstrations and should not be interpreted as held-out video generalization. fidelitymasksperturbationsensitivity
What it supports. SelfWAM improves all three reported fidelity measures: LPIPS falls from 0.0636 to 0.0429, PSNR rises from 29.24 to 32.79, and FVD-I3D falls from 45.92 to 34.55. Conditioning on the demonstrated action provides information relevant to matching its recorded consequence.
Where the evidence stops. These reference-based scores do not measure whether an arbitrary candidate action produces a physically correct future. They also do not isolate the mask contribution. Controlled action changes and held-out trajectories answer different questions from this table.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0 multitask policy success 50 tasks; 100 rollouts per task in each clean/randomized setting; same 27,500-demonstration training dataset. | SelfWAM: 92.16 clean, 93.08 randomized, 92.62 average. Unweighted mean task success (%) | FastWAM: 91.82, 91.86, 91.84; Motus average 87.84; GigaWorld-Policy 85.70; pi0.5 79.75; pi0 62.16. Calculated gains over FastWAM are 0.34 and 1.22 percentage points, averaging 0.78. Task gains vary: randomized Open Microwave improves 41→67%, while Dump Bin Bigbin falls 98→91%. sim-protocolsim-resultstask-breakdown |
| Four physical pick-and-place tasks AgileX ALOHA; ten trials per task per method; same training split and initial-state protocol; autonomous placement success. | SelfWAM cup/pen/mouse/ball: 100/90/100/90; mean 95. Task and mean success (%) | FastWAM: 80/90/80/90, mean 85; pi0.5: 90 on every task. This is executed control, with corrective attempts allowed within an episode. The ten-point mean gain over FastWAM comes from cup and mouse tasks; only 40 trials support each method’s mean. physical-dataphysical-resultsphysical-protocol |
| Future-video reconstruction on demonstration trajectories 275 sampled offline trajectories; matched windows across their full duration; supplied demonstrated actions for SelfWAM. | 0.0429; 32.79; 34.55. LPIPS↓; PSNR↑; FVD-I3D↓ | Unconditioned FastWAM: 0.0636; 29.24; 45.92. All three fidelity metrics improve, but this is not a held-out video-generalization test or proof of correct counterfactual physics. fidelity |
| Clean-action and self-mask component ablation RoboTwin clean/randomized protocol; neither component, action conditioning only, then both. | Full SelfWAM: 92.62. Average policy success (%) | FastWAM: 91.84; action conditioning without masks: 90.80. Conditioning alone lowers the mean by 1.04 points; masks recover 1.82 points. The authors describe the first change as largely preserving performance. The table lacks mask-only control and uncertainty. ablationsim-protocol |
| Action sensitivity under changed conditioning actions Fixed context and sampling noise; WorldArena feature metrics, directional masks and hold-to-demonstration interpolation. | SelfWAM: 0.0184. Action Following score↑ | FastWAM: 0.0000; without masks: 0.0181. Conditioning explains most feature sensitivity. Masks slightly improve subject/background consistency but lower raw photometric consistency from 1.5168 to 1.4361. Distinct futures do not establish physical correctness. sensitivityperturbationinterpolation |
| Inference latency and memory Single H200; fixed episode, 32 actions, ten denoising steps; four runs after warm-up; joint mode includes RGB but no masks. | SelfWAM: 323.9 action-only; 677.5 action+video; 13.959 GiB. Latency (ms); peak memory (GiB) | FastWAM: 320.7; 668.4; 13.959. Reported unrounded relative increases: 0.97% and 1.36%. Comparable action-only cost supports the deployment claim under this setup. It does not mean the neural policy runs at the physical controller’s 30 Hz command rate. latencyphysical-data |
| Clean/noisy action-path consistency at zero noise Aligned action representations across Transformer layers and sampled validation windows. | Approximately 1e-5; 6e-11. Maximum NRMSE; maximum cosine distance | Two paths receive the same unperturbed demonstrated action. The diagnostic supports representation agreement at the zero-noise endpoint, not equivalent behavior for noisy actions or proof of correct robot dynamics. path-consistency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Conditioning alone lowers policy success; self-mask supervision raises the combined model above baseline. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the two component columns as the experiment key. FastWAM has neither clean-action conditioning nor future self-mask supervision. The middle row introduces action conditioning for future RGB prediction; the bottom row also adds masks. Then compare the success columns horizontally within each row. The conditioning-only model scores 90.80 percent in both clean and randomized environments, below FastWAM. The full model reaches 92.16 and 93.08 percent. Read the authors’ phrase largely preserves performance alongside these actual values: it is their interpretation of a measured decrease, not a numerical tie or an isolated improvement from action conditioning. ablationsim-protocolmasks
What it supports. Adding masks to the conditioned model raises average success by 1.82 percentage points, from 90.80 to 92.62, and leaves the full system 0.78 points above FastWAM. The ablation therefore supports the combined design and a useful mask addition within that design; conditioning by itself is insufficient to improve this policy metric.
Where the evidence stops. The table has no mask-only row or uncertainty estimates. Mask instances carry no action loss and are sampled at a stated 9:1 RGB/mask ratio, so effective action-supervision exposure must be controlled before assigning the entire gain to target semantics.
Figure 4. Changing the supplied action changes the predicted robot silhouette in the expected direction. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two panels arrange their comparisons differently. In the vertical panel, columns are upward, expert and downward action variants; rows are ground truth, unconditioned prediction and conditioned prediction. In the horizontal panel, rows are leftward, expert and rightward variants, while columns compare ground truth and the two prediction modes. Follow the white robot silhouette relative to the orange dashed reference lines. The detailed protocol holds observation, instruction and sampling noise fixed while altering the action. That control makes changes between variants interpretable as responses to the conditioning action, rather than differences caused by resampling the generative model. perturbationfidelitysensitivity
What it supports. The conditioned predictions move qualitatively with the simulator’s directional trend, while the unconditioned predictions remain almost unchanged across variants. This is more direct evidence of action dependence than reconstruction scores alone. The panels show directional response in selected examples; they do not supply a calibrated numerical error in robot position or contact outcome.
Where the evidence stops. The caption and Appendix C use translation-axis language, but the main protocol specifies joint-space offsets ramping to ±0.10 rad over eight steps, with directions assigned in the camera plane. Exact joint selection and mapping remain unresolved.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors explicitly exclude contact, force and occluded interactions from the modeled targets. limitations
Policy tables provide no confidence intervals or training-seed variation. Forty physical trials per method and four placement tasks cannot establish broad deployment robustness. sim-resultsablationphysical-results
Directional captions use translation-axis wording, while the detailed main protocol specifies joint-space offsets and image-plane direction labels. Exact joint selection remains unclear. The qualitative panels support directional response, not a quantified dynamics-error bound. perturbation
7.2 Questions for discussion
- Would mask supervision still improve success after matching the number of action-supervised updates across variants?
- Does action sensitivity predict accurate held-out contact outcomes, rather than merely different-looking futures?
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include WAN2.2-5B, paired trajectories and masks, the directed attention mask, interpolation initialization, and synchronized multi-view sampling. Table S2 specifies Ubuntu 22.04, Python 3.10.12, PyTorch 2.7.1, CUDA 12.8 and 1 TB system memory; these are reported settings, not independently tested requirements. configurationattentionmasks
The source leaves exact eight-frame indices, full action-expert dimensions, frozen-module choices and training seeds unspecified. Physical placement-region bounds and an explicit trial time cap are absent. These gaps require resolution before a faithful replication. configurationphysical-protocol
Proposed checks should test action-query independence from clean targets and separate mask supervision from changes in effective action-loss exposure. Match initialization, sampling budget, seeds and evaluation contexts. attentionmasksablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Audit target isolation while testing controlled action response
Reader-proposed check, not an executed experiment: in a fixed checkpoint, freeze current context, noisy-action state and all sampling noise, then replace clean-action tokens and future-video inputs independently. Action denoising outputs should remain unchanged within declared numerical tolerance if Equation 6 is implemented correctly. Future-video outputs should respond to changes in clean actions. Test the source’s hold-to-demonstration coefficients 0, 0.25, 0.75 and 1 on held-out simulator contexts, and compare predicted robot masks with actual rollouts, adding mask overlap and motion-direction error as proposed metrics. Unchanged action outputs but incorrect simulated consequences would separate successful leakage prevention from inadequate world modeling; changed action outputs would falsify the intended information boundary. attentioninterpolationperturbationpath-consistency
Check 2: Separate self-mask semantics from the training-sample budget
Reader-proposed check, not an executed experiment: compare conditioning-only, full SelfWAM and a control replacing each mask instance with an RGB-only visual instance that also has no action loss. Keep demonstration windows, initialization, optimization steps, visual-target count and action-supervised updates matched across variants; record the actual 9:1 sampler behavior. Use multiple training seeds and the same clean/randomized evaluation starts, reporting task success with uncertainty alongside Action Following. If true masks consistently outperform the exposure-matched RGB-only control, that supports a benefit from body-focused targets. If the gap disappears after matching action-supervision exposure, the published ablation alone cannot attribute its improvement specifically to mask semantics. masksconfigurationablationsim-protocolsensitivity
8.3 Reading coverage
Visual audit: All 14 pages of the supplied v1 PDF were rendered and visually inspected after all five text chunks were read. The pass includes the title/version/authors, Figures 1–4 and S1–S3, Tables 1–5 and S1–S4, equations, hardware/software configuration, dataset splits, physical protocol and limitations. Figure 2 query/key directions and colored cells were checked against Equation 6 and the caption; no conflicting access rule was found. Figure 4’s translation-axis wording is disclosed alongside the joint-space protocol. All six final original crops were viewed and retain their headers, labels and relevant markers, without surrounding body prose or long captions. Numerical and method evidence on uncropped appendix pages is included in the declared pages. Separate supplements, linked resources, code and experiments remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 SelfWAM: formulation, architecture, information flow, masks, training and inference
- 4 Experiments: implementation, benchmarks, policy, future prediction, ablation and efficiency
- 5 Conclusion and Acknowledgements
- Appendix A: Implementation and Reproducibility Details
- Appendix B: Extended RoboTwin Evaluation
- Appendix C: Action Sensitivity and Controlled Rollouts
- Appendix D: Consistency of the Clean and Noisy Action Paths
- Appendix E: Real Robot Experimental Details
- Appendix F: Limitations and Future Work
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks and all 14 PDF pages were read; the extraction-only figure omission was resolved by inspecting the original PDF pages and six final crops.
- Identity/version note: the title and five authors match the catalog; the inspected source is arXiv:2608.00725v1, dated 1 August 2026. No different revision was supplied or substituted.
- No external project, code, checkpoints or separate supplements were inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin stamp
The title matches the supplied identity. Authors are Bikang Pan, Fan Liu, Haotao Lu, Jingya Wang and Ye Shi; Pan and Liu contributed equally. Affiliations are ShanghaiTech University and InstAdapt. The artifact identifies arXiv:2608.00725v1, 1 August 2026.
Go to primary source ↓problemPDF pp. 1–2, Introduction and Section 3, Problem Formulation, Eqs. (1)–(5)
The policy conditions on multi-view RGB, proprioception and instruction; the world model additionally conditions on the demonstrated action. The motivation is to distinguish action-specific consequences from generic task progression.
Go to primary source ↓architecturePDF p. 3, Figure 1 and Architecture; p. 4, Figure 2, left panel
A pretrained video backbone and lightweight action expert retain separate projections and feed-forward layers, exchanging permitted information through mixed attention. Video generation is auxiliary during policy training and optional during rollout.
Go to primary source ↓attentionPDF p. 4, Figure 2 attention matrices, Information Flow and Eq. (6)
Rows are query groups and columns are keys. Noisy action queries access current context and noisy actions only. Future-video queries access context, future video and clean actions. Clean actions access context and clean actions; clean tokens use denoising timestep zero and matching temporal positions.
Go to primary source ↓masksPDF p. 4, Robot Mask Prediction; p. 8, Appendix A, Training Instance Construction and Table S1
RGB and self-mask targets share the video backbone, context, clean-action condition, noise schedule and positions. The output prompt changes the target domain. Only RGB instances include noisy actions and action loss; mask instances have only visual loss. Appendix A describes sampling either instance type at 9:1.
Go to primary source ↓objectivePDF p. 5, Training and Inference, Eq. (7)
The loss combines action flow matching and a mixture expectation of RGB/mask video flow matching. Action and visual weights are both one. Deployment omits clean-action and future-video tokens.
Go to primary source ↓configurationPDF p. 5, Implementation Details; p. 8, Appendix A, Temporal Sampling and Configuration; p. 9, Figure S1 and Table S2
WAN2.2-5B initializes the video backbone; corresponding action weights are copied or linearly interpolated with input-width scaling, while unmatched action-specific layers are separately initialized. Three views form a 384×320 T-shaped composite. Training uses 32 actions, eight sampled future frames, five epochs, AdamW at 1e-4, batch 1024, 64 A800 GPUs and approximately 30 hours. Table S2 gives software versions and deployment settings.
Go to primary source ↓sim-protocolPDF p. 5, RoboTwin 2.0; pp. 8–9, Appendix A, Datasets and Splits; p. 9, Appendix B, Evaluation Protocol
All methods use 2,500 clean and 25,000 randomized demonstrations. Evaluation covers 50 tasks with 100 rollouts per task per setting, using simulator success and unweighted task means. Simulator masks provide exact per-camera robot segmentation.
Go to primary source ↓sim-resultsPDF p. 5, Table 1, all rows and Clean/Random/Average columns
SelfWAM scores 92.16/93.08/92.62%; FastWAM 91.82/91.86/91.84%; Motus 88.66/87.02/87.84%; GigaWorld-Policy 86.36/85.04/85.70%; pi0.5 82.74/76.76/79.75%; pi0 65.92/58.40/62.16%. No uncertainty intervals are shown.
Go to primary source ↓physical-dataPDF p. 9, Appendix A, Datasets and Splits; p. 12, Appendix E, Platform and Data and RobotSeg Mask Generation
The physical dataset contains 17,307 episodes and 9,557,419 transitions. Seed-42 episode splitting gives 17,133 training episodes/9,461,919 transitions and 174 validation episodes/95,500 transitions, excluded from gradients. ALOHA uses 14-dimensional joint/gripper actions and matching proprioception at 30 Hz. RobotSeg generates offline whole-robot targets independently per camera; it is absent at deployment.
Go to primary source ↓physical-resultsPDF p. 6, Figure 3, Table 2 and Real-robot evaluation
For cup/pen/mouse/ball tasks, SelfWAM scores 100/90/100/90%, FastWAM 80/90/80/90%, and pi0.5 90/90/90/90%. Averages are 95/85/90%. Each method has ten trials per task; success requires grasping and placement without human intervention.
Go to primary source ↓physical-protocolPDF p. 13, Appendix E, Real-Robot Evaluation Details
Each task uses a fixed instruction. Objects and targets are randomly positioned within predefined regions. Regrasping and corrective attempts are allowed within an episode; completion or inability to continue without human intervention ends the episode.
Go to primary source ↓fidelityPDF p. 6, Table 3 and Future prediction evaluation; p. 9, Appendices A–B, video evaluation scope and protocol
On 275 trajectories sampled from the 27,500-demonstration offline dataset, SelfWAM/FastWAM obtain LPIPS 0.0429/0.0636, PSNR 32.79/29.24 and FVD-I3D 34.55/45.92. Matched windows span each trajectory. Appendix A explicitly excludes interpreting this as held-out video generalization.
Go to primary source ↓ablationPDF p. 7, Table 4, component flags and success columns
FastWAM has neither component and averages 91.84%; clean-action conditioning without masks scores 90.80% in both settings; the full model scores 92.16% clean, 93.08% randomized and 92.62% average. A mask-only variant is absent.
Go to primary source ↓latencyPDF p. 7, Inference Efficiency and Table 5
Single-H200 timing uses a fixed episode, 32-action horizon and ten denoising steps, averaged over four warmed-up runs. Action-only latency is 320.7/323.9 ms for FastWAM/SelfWAM; action plus RGB is 668.4/677.5 ms. Peak memory is 13.959 GiB for both. Relative changes of 0.97% and 1.36% use unrounded timings; joint timing excludes masks.
Go to primary source ↓perturbationPDF p. 6, Directional action perturbation; p. 7, Figure 4 and caption; p. 11, Appendix C, Directional Action Perturbation
The main protocol specifies joint-space offsets ramping from zero to ±0.10 rad over eight steps and then held. Directional names follow high-camera image-plane displacement. Figure 4 and Appendix C use translation-axis wording. Context, instruction and sampling noise are fixed; conditioned masks qualitatively follow simulator motion trends.
Go to primary source ↓sensitivityPDF p. 9, Table S3; p. 11, Appendix C, WorldArena Metrics, Eqs. (S1)–(S4)
Action Following is pairwise dissimilarity of global CLIP features. FastWAM/no-mask/full scores are 0.0000/0.0181/0.0184. Subject consistency is 0.9138/0.9321/0.9326, background 0.9309/0.9380/0.9385 and photometric 0.8126/1.5168/1.4361. Sensitivity is not physical correctness; raw photometric scores need not lie in [0,1], and static videos can appear consistent.
Go to primary source ↓interpolationPDF p. 11, Appendix C, Continuous Action Interpolation, Eq. (S5) and Figure S2
The action is interpolated between hold and demonstration at coefficients 0, 0.25, 0.75 and 1 with fixed context and noise. Conditioned RGB outcomes progress toward the demonstration, whereas the unconditioned baseline is nearly invariant.
Go to primary source ↓path-consistencyPDF pp. 11–12, Appendix D, Figure S3 and accompanying definitions/results
At zero perturbation, layerwise clean/noisy representations are compared by NRMSE and cosine distance. Text reports approximate maxima 1e-5 and 6e-11 over sampled validation windows and layers; curves show means and one standard deviation. This checks unperturbed representation consistency.
Go to primary source ↓task-breakdownPDF p. 10, Table S4, Open Microwave, Dump Bin Bigbin and Average rows
Randomized Open Microwave success is 67% for SelfWAM versus 41% for FastWAM; randomized Dump Bin Bigbin is 91% versus 98%. Task-level gains are not uniform despite the improved mean.
Go to primary source ↓limitationsPDF p. 13, Appendix F, Limitations and Future Work
The model does not explicitly represent contact, force or interactions hidden by occlusion; the authors propose adding such supervision.
Go to primary source ↓8.5 Primary sources
SelfWAM: A Self-Grounded Unified World Action Model for Fast Robot Control ↗
PDF · 7,242 extracted words
Source fingerprint
33f6920f515fafac2cb471dd0593d0483cb0e76eb433cd8f9c927906db98a263