MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models
1. Paper overview
In one sentence: Predicting a target's future mask makes first-frame visual prompting useful for joint action generation, at the cost of mask annotation and segmentation dependence. problemjointencodingablationambiguousinference
| At a glance | What to know |
|---|---|
| Research problem | Author claim RGB prediction can spend representational capacity on irrelevant backgrounds, while language may not distinguish similar objects in clutter. The authors therefore seek both object-focused predictive supervision and an explicit target-selection interface, retaining ordinary text-conditioned operation for unambiguous tasks. problem |
| Core mechanism | Source description A shared visual latent stream predicts RGB and task-relevant masks, interacting with a separate action expert through MoT attention. jointencodingarchitecture |
| A key reported result | Language-ambiguous real-robot generalization: 84.9 overall; setting averages 92.9, 90.4, 74.6, 81.7. Execution success rate (%). Tasks 5–8; 60 trials per task per setting, spanning in-domain, distractors, novel instances and lighting. pi0-mask 51.7 overall; pi0-coord 32.6; FastWAM-coord 17.4. The 33.2-point advantage over pi0-mask is absolute, not a relative percent gain. Novel-instance transfer remains weaker than in-domain execution. ambiguousprotocol |
| Reading caution | Source description Training requires mask supervision; prompted deployment depends on segmentation quality. Reliable extraction in clutter remains difficult, and large-scale joint RGB-mask-action pretraining is deferred. limits |
Core contributions
- Source description
A shared visual latent stream predicts RGB and task-relevant masks, interacting with a separate action expert through MoT attention. jointencodingarchitecture
Figure 2. RGB and masks share a predictive latent stream; the action expert reads its intermediate state. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a): RGB and rendered masks pass through the frozen VAE, and their latents join along the channel dimension marked 2C. The trainable transformer predicts visual velocities while the action branch predicts action velocities. The separate τ_v and τ_a controls correspond to independent visual and action noise levels; RGB and masks share τ_v. In panel (b), read query rows against key columns: colored action-to-visual cells permit action queries to use visual context, while the reverse action/state columns are blank for visual queries. Panel (c) then reuses visual keys and values through KV-cache after partial visual denoising. architectureencodingtraininginference
What it supports. The source specifies one visual denoising step before action generation. This retains an inference-time connection to predictive visual features without requiring complete RGB or mask videos to be decoded. Separate visual and action experts remain visible in the architecture, despite the unified training objective and joint prediction formulation.
Where the evidence stops. The caption calls the mask block-wise causal, but its visual submatrix is dense, not frame-triangular. Follow the depicted block permissions. The figure does not specify the action sampler's step count or a measured latency gain.
2. Motivation
2.1 The problem and the proposed response
RGB prediction can spend representational capacity on irrelevant backgrounds, while language may not distinguish similar objects in clutter. The authors therefore seek both object-focused predictive supervision and an explicit target-selection interface, retaining ordinary text-conditioned operation for unambiguous tasks. problem
2.2 What this reading follows
A robot facing several similar bottles needs to know which bottle matters and how to move toward it. MaskWAM connects these two questions by putting object masks inside the same predictive representation as RGB video, then coupling that representation to an action expert. The key comparison is especially revealing: retaining a mask prompt but removing future-mask prediction sharply reduces success in the authors' ambiguous-task evaluation. Read the architecture alongside that ablation, then use the detailed generalization table to see where the gains hold and where they weaken. The reported control policy uses intermediate visual latents; the fully decoded future videos are offline visualizations. problemjointencodingablationambiguousinference
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Conflicts with the recorded classification
WAM and object/action grounding are supported. Dual-system has architectural support from distinct visual and action experts, even though they interact inside a unified MoT. However, Eq. (1) and the three denoising losses specify joint future/action prediction, not a separately defined inverse-dynamics mapping from completed future images to actions. Deployment conditions action denoising on partially denoised latents. Thus the recorded IDM/Q4 assignment is not supported as written; joint training alone also does not justify relabeling this a single undifferentiated model. jointarchitecturetraininginference
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the target channel consequential
A first-frame mask can be a weak side input if the policy can solve most training examples without using it. MaskWAM addresses this by requiring the model to predict future masks as well as future appearance and actions. Each rendered mask uses the same VAE as RGB, so its information enters aligned latent positions rather than a separately learned representation. The prediction head must then account for the mask channel throughout the future horizon. Reader interpretation: this creates a reason to preserve target identity inside the predictive representation. Table 4 provides the relevant control: removing the future-mask target and loss while retaining the prompt sharply reduces success. The result supports this mechanism within the studied model, but it does not establish that future prediction is the only way to train prompt sensitivity. encodingablationalternatives
Figure 6. Future-mask supervision depends on an annotation pipeline with human verification. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the episode frames on the left into language and mask annotation. The upper branch identifies task-relevant objects, which become text conditions for SAM3 in the lower branch. SAM3 produces a temporal mask sequence, and the human verification node either sends it to the labeled dataset or routes it through point-conditioned correction and another propagation pass. Appendix B.1 adds an essential distinction that is not fully represented by the generic diagram: language-clear tasks use Qwen3-VL to identify relevant objects, whereas language-ambiguous tasks begin with a human point on the first frame because text cannot select the intended target. Training-time propagation supplies future labels. annotationprompt
What it supports. The authors report that 91% of episodes need no correction and that annotating 50 episodes takes approximately three minutes, with an additional five to ten seconds per ambiguous episode for the initial point. These are reported pipeline costs; they do not eliminate the human verification stage or the extra target-selection input.
Where the evidence stops. Training-time tracking and correction should not be confused with deployment, where the prompt is generated once at episode start. The source does not give an objective threshold for deciding that annotation quality is satisfactory.
5.2 Separate semantic supervision from target selection
Masks play two roles that should be evaluated separately. Future masks supply supervision during training, including when the instruction already identifies the target. The initial mask supplies a spatial anchor when the instruction is ambiguous. Training replaces that initial mask with zeros half the time, so the same policy can receive or omit it at deployment. LIBERO tests the first role: the text-conditioned joint model improves over RGB-only without a test-time visual prompt. The ambiguous real-robot evaluation tests the second role in conjunction with predictive supervision. Appendix B.1 explains how the needed labels are obtained: language-clear data uses instruction-derived object identification, while ambiguous data receives a human first-frame point. This additional target information is part of the evaluated setup and must be preserved in a reproduction. promptliberoambiguousannotation
5.3 Use predictive features without decoding an entire imagined video
The training formulation jointly predicts action chunks, RGB futures and mask futures, but deployment uses a shorter computation. RGB and mask noise levels stay synchronized, while the action noise level is sampled independently during training. The action expert therefore learns with visual contexts at different denoising stages. At inference, the source specifies one visual denoising step and a cached visual context for generating actions, using current observations, language and proprioception. Figure 10 explicitly labels complete future sequences as offline visualizations. This distinction matters when interpreting the world model: auxiliary prediction shapes learned features, and intermediate predictive latents also enter inference, but the paper does not describe execution as selecting a fully rendered video plan. Exact action sampling steps, control cadence and measured latency tradeoffs remain unspecified. jointarchitecturetraininginferencelimits
5.4 Training and inference
During training
Fine-tune the prediction model while freezing the video VAE and T5. RGB and masks share noise timestep τ_v; actions use independent τ_a, exposing action denoising to multiple visual noise levels. Sum video, mask and action flow losses. Initial masks are zeroed with probability 0.5. trainingprompt
Reported inputs are 384 × 320 T-shape RGB frames with future horizon T = 8. Training labels use Qwen3-VL/SAM3 for language-clear episodes and human first-frame points for ambiguous episodes, followed by SAM3 propagation and human correction. trainingannotation
During inference
Obtain an optional SAM3 mask once at episode start. At a prediction call, condition on current RGB, language and proprioception, take one RGB-mask denoising step, and generate the action chunk using cached visual context. Subsequent predictions do not require repeated segmentation prompts. promptinferencearchitecture
Predicted action chunks drive real-robot evaluation. Full future RGB/mask decoding in Figure 10 is explicitly offline; those images do not establish test-time video rollout planning. Exact action execution cadence and sampler settings are unspecified. inferenceprotocollimits
5.5 Implementation flow
- Render and encode masks
Render task masks as three-channel images with fixed object/background colors. Encode masks and RGB using the same causal 3D VAE, preserving alignment in temporal horizon and spatial resolution. encoding
- Fuse in the video latent space
Concatenate RGB and mask latents by channel. Expand the pretrained patch embedding from C to 2C channels, preserving RGB weights and zero-initializing added mask weights; expand the prediction head correspondingly. encoding
- Couple visual and action experts
Wan 2.2 supplies the visual backbone. T5 language features enter its cross-attention; projected proprioception and noisy actions enter the action expert. Figure 2 permits actions to attend visual context while blocking visual queries from action/state keys. jointarchitecture
6. Experiments & results
MaskWAM couples RGB and mask future prediction with action denoising, using an optional first-frame target mask to resolve ambiguous instructions. Its central empirical distinction is between supplying a mask and training the policy to predict its future: the latter substantially improves mask-guided robot execution in the evaluated setup. Deployment uses partially denoised visual latents, while fully decoded future videos are offline illustrations (joint, inference, ablation).
6.1 Read the original evidence
Table 1, numerical and attention panels. Future-mask supervision improves the reported LIBERO average even without a test-time mask prompt. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the final three rows first. They compare RGB-only, Mask-only and joint prediction within the authors' model family. The columns separate Spatial, Object, Goal and Long before the average, so a better average need not mean a win on every suite. The joint model reaches 98.4% versus 97.3% for RGB-only and 97.6% for Mask-only. Next compare the external baseline rows: Motus has 97.7% average but a stronger Long score than MaskWAM. The attention images to the right show the qualitative grounding interpretation offered by the authors. The experiment text specifies that these LIBERO gains do not require visual prompts at deployment. libero
What it supports. The reported average improves by 1.1 percentage points over RGB-only. Spatial rises from 96.8% to 98.8% and Goal from 97.0% to 98.2%. This supports an auxiliary-supervision benefit beyond target selection at test time, while the suite breakdown prevents interpreting the average as universal dominance.
Where the evidence stops. No seed variability, confidence intervals or detailed evaluation counts accompany this table. Attention heatmaps have no quantitative grounding score and cannot establish that localization alone caused the execution improvement.
Table 5. Explicit mask prompting supports transfer, with novel instances remaining the hardest tested setting. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each horizontal block as a separate evaluation setting, using the Ours row and the Average column before inspecting individual tasks. Tasks 5–8 concern bowl, cup, bottle and cosmetic selection. The caption specifies 60 trials for each task under each setting. Within a block, compare Ours with π0-mask to compare the evaluated mask-guided systems; π0-coord and FastWAM-coord instead receive normalized target coordinates in text. The Ours averages are 92.9%, 90.4%, 74.6% and 81.7% in the displayed order. These blocks explain Figure 4's overall result, which averages in-domain and generalization settings together rather than reporting a separate unseen-task benchmark. ambiguousprotocolbaselines
What it supports. The source reports 84.9% overall for MaskWAM versus 51.7% for π0-mask, a 33.2-percentage-point difference. The detailed table adds an important boundary: success falls to 74.6% for novel instances. For Task 8 specifically, novel-instance performance is 65.0%, despite the much higher overall headline.
Where the evidence stops. The π0-mask baseline uses additive RGB/mask SigLIP embeddings, so this comparison covers that implementation. Appendix A.3's blanket failure wording for coordinate baselines on Tasks 7–8 is stronger than the nonzero entries visible here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO manipulation Four benchmark suites; language-conditioned deployment without visual prompts. | 98.4 average; Spatial 98.8, Object 100.0, Goal 98.2, Long 96.4. Execution success rate (%) | RGB-only 97.3; Mask-only 97.6; Motus 97.7; FastWAM 97.6. The RGB-only difference is 1.1 percentage points. Highest average does not mean winning every suite: Motus scores 97.6 on Long. libero |
| RoboTwin 2.0 manipulation Hammer, Bell, Card, Burger, Stand and Shoe; 500 randomized training episodes per task. | 92.2 average. Execution success rate (%) | RGB-only 87.3; Mask-only 88.8; FastWAM 87.7; pi0 72.8. Table 2 supports gains of 4.9 points over RGB-only and 4.5 over FastWAM. Figure 1 labels 92.1 instead; this report retains the detailed table's 92.2. robotwin |
| Language-clear real-robot manipulation Four Xtrainer tasks; approximately 100 demonstrations per task and 100 evaluation trials per task. | 84.3 average; Tasks 1–4: 91, 82, 81, 83. Execution success rate (%) | RGB-only 79.3; FastWAM 79.0; pi0.5 72.3. Mask prediction improves the reported rounded average by 5.0 points over RGB-only under this protocol. clearprotocol |
| Language-ambiguous real-robot generalization Tasks 5–8; 60 trials per task per setting, spanning in-domain, distractors, novel instances and lighting. | 84.9 overall; setting averages 92.9, 90.4, 74.6, 81.7. Execution success rate (%) | pi0-mask 51.7 overall; pi0-coord 32.6; FastWAM-coord 17.4. The 33.2-point advantage over pi0-mask is absolute, not a relative percent gain. Novel-instance transfer remains weaker than in-domain execution. ambiguousprotocol |
| Mask grounding ablation Language-ambiguous real tasks aggregated across in-domain and out-of-domain settings. | Full model 84.9; Ours-no-pred 21.6; Ours-coord 18.2. Execution success rate (%) | No-pred keeps mask prompting but removes the future-mask target/loss. Coord keeps mask prediction but substitutes coordinate text. This supports the future-mask objective within this architecture. It does not prove that every alternative policy must predict masks. ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4, numerical and attention panels. Supplying a mask alone is insufficient in the evaluated architecture. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the check marks before reading the success row. Ours-no-pred retains the mask prompt but removes future-mask targets and their loss, as specified by the caption. Ours-coord retains future-mask prediction but replaces the visual mask prompt with coordinate-augmented language. Ours combines prediction and mask prompting. Their average success rates are 21.6%, 18.2% and 84.9%, respectively, aggregated over the language-ambiguous in-domain and out-of-domain tasks. In the right panel, time advances from left to right, and the rows compare RGB observations with coordinate-conditioned and mask-conditioned attention. These examples illustrate the authors' grounding explanation but are separate from the execution success measurements. ablation
What it supports. The gap between Ours and Ours-no-pred is 63.3 percentage points, directly motivating the hypothesis that predicting masks helps the policy use its visual anchor. The coordinate variant's poor performance further indicates that the input representation matters even when future-mask supervision is retained in training.
Where the evidence stops. This ablation supports the claim within MaskWAM's training setup. It supplies neither uncertainty estimates nor a general impossibility result for other prompting methods. The attention images alone cannot isolate the causal mechanism.
Figure 8. Spatial mislocalization is a stronger failure mode than many boundary distortions in the plotted examples. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the black star at IoU 1.0, which denotes the original online SAM3 prompt. Move left along each colored curve to examine increasingly different prompts while keeping the trained checkpoint fixed. IoU is overlap with that original prompt, not with a ground-truth segmentation. The markers distinguish erosion, dilation, spatial shift and region dropout; the gray shift curves decline more sharply than several other perturbations. Compare the bowl-selection task on the left with the densely arranged cosmetics task on the right. Their vertical axes have different ranges and truncated lower bounds, so slope and vertical position should not be compared as if both panels shared an identical scale. noiseprotocol
What it supports. The plotted trend supports tolerance to some mask imperfections and sensitivity to target mislocalization. It also suggests a useful follow-up: compare different corruption types at matched IoU to separate mask-boundary quality from preservation of target location. The figure does not establish a universal safe-IoU threshold or performance against independent segmentation ground truth.
Where the evidence stops. Appendix B.2 states 20 trials per task/condition, yet some plotted rates fall between the five-point increments expected from that count. No aggregation explanation or uncertainty is given, so this edition does not infer exact per-condition rates.
7. Analysis & limitations
7.1 What the evidence leaves open
Training requires mask supervision; prompted deployment depends on segmentation quality. Reliable extraction in clutter remains difficult, and large-scale joint RGB-mask-action pretraining is deferred. limits
Tables report no seed variability or confidence intervals, and detailed simulation evaluation counts and training-budget parity are unstated. The customized pi0-mask baseline uses additive SigLIP embedding fusion; it does not represent every possible visual-prompted VLA. liberorobotwinclearambiguousbaselines
Appendix B.2 states 20 trials per task/condition, but Figure 8 contains finer-than-five-point success increments; aggregation is unexplained. IoU is measured against the original SAM3 prompt, not ground truth. Exact diagnostic rates should not be inferred. noise
Appendix C gives qualitative negative results without scores. Appendix A.3's blanket failure wording for coordinate baselines on Tasks 7–8 is stronger than Table 5, which includes nonzero successes. Attention examples support localization qualitatively, not a causal proof. alternativesbaselinesqualitative
7.2 Questions for discussion
- How much of mask prediction's benefit survives an equal-capacity, equal-budget multi-seed ablation?
- Does prompt location explain execution failure better than mask IoU alone?
- What latency–success tradeoff does one visual denoising step produce relative to deeper denoising?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include the Wan 2.2 visual backbone, frozen causal VAE/T5, expanded RGB-mask heads, action expert, annotated action-video episodes and SAM3 prompts. Physical replication requires the Xtrainer platform with D455 head and D405 wrist cameras. jointencodingtrainingannotationprotocol
Optimizer, learning rate, batch size, training duration, GPU configuration, exact K, action representation details and denoising schedule are missing. T = 8 does not specify the action chunk length. The paper supplies no measured latency comparison for its partial-denoising claim. limitsjointtraininginference
Reader-proposed checks: repeat the no-pred comparison with matched data, architecture and training budget, then test equal-IoU mask shifts versus boundary distortions with fixed checkpoints. Measure target-selection and execution failures separately; these checks have not been run. ablationnoise
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does future-mask supervision cause prompt sensitivity under matched training?
Reader-proposed experiment, not run: reproduce the full and no-pred conditions with identical initial weights, latent widths, annotated demonstrations, mask-dropout schedule, action horizon and optimization budget; change only the future-mask supervision target/loss. Repeat training across multiple seeds and evaluate Tasks 5–8 on matched layouts, preserving the paper's 60 trials per task/setting. On an additional matched subset, switch the prompted target while leaving the scene and ambiguous instruction fixed. Report target-choice accuracy separately from final execution success, with trial counts and intervals. The mechanism is supported if full-model target choice reliably follows the switched prompt and outperforms no-pred across seeds. A disappearing gap or equal prompt sensitivity would weaken the proposed explanation. ablationpromptprotocollimits
Check 2: At equal mask overlap, does displacement matter more than boundary error?
Reader-proposed experiment, not run: freeze a trained checkpoint and use the online SAM3 prompt as the reference on Tasks 5 and 8. Generate shifts, erosion/dilation and region dropout matched to the same IoU bins. Include the unmodified prompt, hold episode initialization and downstream settings fixed, and evaluate 60 matched-layout trials per task/condition. Record both success and whether the gripper approached the intended object; publish integer counts so the reported percentages are auditable. Larger failures from shifts than from boundary changes at matched IoU would support the spatial-anchor interpretation. Similar failures across corruption types would favor overlap loss as the simpler explanation. Repeat with independently annotated target masks to distinguish prompt-relative IoU from true segmentation quality. noiseprotocolprompt
8.3 Reading coverage
Visual audit: Visually inspected the title/author block; all method and result pages; all appendix hardware, baseline, annotation, mask-robustness and alternative-design pages; Figures 1–12; and Tables 1–5. All six final crops were opened and inspected, including the corrected annotation crop. Figure 2's attention permissions, noise controls and KV-cache arrow were cross-checked against Sections 3.1–3.3 and the inference text. Reference-only pages 9–12 were read in the supplied text but were not visually inspected; no scientific claim here depends on their typography. The full PDF body and appendices were read, but separate supplements remain unverified. Figure 1's RoboTwin label differs from Table 2, and Figure 8's success-rate granularity is not explained by its stated trial count; both discrepancies are retained in the report. All pages supporting retained method, training, evaluation, hardware and reproduction claims are included above.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract; 1 Introduction; 2 Related Work
- 3 Method; 3.1 Future Masks as Predictive Targets; 3.2 Initial Masks as Policy Conditions; 3.3 Training Objective
- 4 Experiments; 4.1 Real-world Tasks; 4.2 Ablations
- 5 Conclusion; 6 Limitations and Future Work
- References (pp. 8–12)
- Appendix A.1 Hardware; A.2 Tasks; A.3 Baselines
- Appendix B.1 Annotation; B.2 Mask Robustness; C Alternative Conditioning Designs
- Appendix qualitative figures through PDF p. 20
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting all scientific figure/table pages and all six final crops. All six supplied text chunks, including references and appendices, were read without truncation.
- Identity/version notes: exact title and all seven author names agree with the catalog. Reviewed arXiv:2606.13515v1; the sidebar says 11 June 2026 and the title block says 12 June 2026. No other revision was supplied.
- The title page assigns Chenghao Gu affiliation marker 4, but lists institutions only under 1–3. The institution names are verified collectively; his individual affiliation remains unresolved.
- No external project, code, checkpoints or separate supplements were inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv sidebar
The exact title and seven authors match the catalog. The sidebar identifies arXiv:2606.13515v1, 11 June 2026; the title block says 12 June 2026. Chenghao Gu has superscript 4, but the affiliation list contains only 1, 2 and 3.
Go to primary source ↓problemPDF pp. 1–2, Section 1 and Figure 1
The authors motivate explicit target supervision and first-frame spatial prompts by background distraction in RGB prediction and ambiguity in language-only object selection.
Go to primary source ↓jointPDF p. 3, Section 3 overview, Eq. (1)
The conditional distribution jointly models action chunks, future RGB frames and future masks given current RGB, first-frame mask, proprioception and language; the backbone is Wan 2.2 with an MoT action expert.
Go to primary source ↓encodingPDF p. 3, Section 3.1, Eq. (2) and Patch embedding expansion
Rendered mask images and RGB share a causal 3D VAE. Latents concatenate by channel; pretrained RGB patch weights are retained, added mask weights start at zero, and the output head expands for both latent velocities.
Go to primary source ↓architecturePDF p. 4, Figure 2(a–c) and Section 3.3
The model has interacting visual and action branches. The attention matrix lets action queries attend visual tokens, while visual queries do not attend action/state tokens. The inference panel passes cached visual context to action denoising.
Go to primary source ↓promptPDF p. 4, Section 3.2; p. 5, Implementation Details
The first-frame mask is optional, is zeroed with probability 0.5 in training, and can be generated by SAM3 once at episode start. Repeated segmentation prompts are not required during deployment.
Go to primary source ↓trainingPDF pp. 4–5, Section 3.3, Eq. (3) and Implementation Details
RGB and masks share visual timestep tau_v; actions use independently sampled tau_a. Three flow losses are summed. VAE and T5 remain frozen; input resolution is 384 by 320 in a T-shape layout and prediction horizon T is 8.
Go to primary source ↓inferencePDF p. 5, Implementation Details; p. 19, Figure 10 caption
Deployment performs one joint RGB-mask denoising step and generates actions from intermediate visual context. Figure 10's fully decoded future sequences are offline visualizations only.
Go to primary source ↓liberoPDF p. 5, Performance on LIBERO; p. 6, Table 1, suite and Avg columns
No test-time visual prompt is used in the stated LIBERO evaluation. Average success: full model 98.4, RGB-only 97.3, Mask-only 97.6, Motus 97.7, FastWAM 97.6. Full-model Spatial/Object/Goal/Long values are 98.8/100.0/98.2/96.4.
Go to primary source ↓robotwinPDF p. 6, Table 2, caption and all method rows; p. 2, Figure 1 RoboTwin bar
RoboTwin training uses 500 randomized episodes per task. Table 2 reports 92.2 average for Ours, 87.3 RGB-only, 88.8 Mask-only, 87.7 FastWAM and 72.8 pi0 over six tasks. Figure 1 instead labels Ours 92.1.
Go to primary source ↓protocolPDF p. 5, Section 4.1 protocol; p. 13, Appendix A.1–A.2 and Figure 5
Real-robot evaluation uses Dual-arm Xtrainer with a D455 head camera and D405 wrist cameras. Eight tasks average 100 demonstrations each. Language-clear evaluation uses 100 trials per task; ambiguous evaluation uses 60 per task per generalization setting.
Go to primary source ↓clearPDF p. 6, Table 3; p. 13, Appendix A.2, Tasks 1–4
Bowl stacking, mug hanging, drawer/pen interaction and towel folding yield 91/82/81/83 success, average 84.3. RGB-only averages 79.3, FastWAM 79.0, pi0.5 72.3 and pi0 55.8.
Go to primary source ↓ambiguousPDF pp. 6–7, Section 4.1 and Figure 4; p. 17, Table 5
Tasks 5–8 test bowl, cup, bottle and cosmetic target selection. Full-model averages are 92.9 in-domain, 90.4 distractors, 74.6 novel instances and 81.7 lighting; pi0-mask gives 62.9/52.9/44.6/46.3. Figure 4 overall values are 84.9 versus 51.7/32.6/17.4 for pi0-mask/pi0-coord/FastWAM-coord.
Go to primary source ↓ablationPDF p. 8, Section 4.2 Q2–Q3 and Table 4
Ours-no-pred retains the mask prompt but removes the future-mask target/loss and scores 21.6. Ours-coord retains future-mask prediction but replaces mask prompting with coordinate text and scores 18.2. Full MaskWAM scores 84.9.
Go to primary source ↓baselinesPDF pp. 13–14, Appendix A.3, Eq. (4); p. 17, Table 5
Coordinate baselines verbalize the target-mask centroid as normalized image coordinates. pi0-mask adds separately encoded RGB and mask SigLIP embeddings. Although Appendix A.3 describes failures on Tasks 7–8, Table 5 includes nonzero coordinate-baseline successes.
Go to primary source ↓annotationPDF pp. 14–15, Appendix B.1 and Figure 6
Qwen3-VL identifies task-relevant objects for language-clear data; ambiguous data receives a human first-frame point. SAM3 propagates masks, humans verify all first-round annotations and correct poor episodes with keyframe points. Reported annotation cost is about 3 minutes per 50 episodes, with 91% requiring no correction and 5–10 extra seconds per ambiguous episode.
Go to primary source ↓noisePDF pp. 15–16, Appendix B.2 and Figures 7–8
Only first-frame masks are synthetically perturbed with fixed model weights on Tasks 5 and 8. The stated protocol is 20 trials per task/condition. Figure 8 uses IoU relative to the original online SAM3 mask, not ground truth; shift curves degrade more strongly. Some plotted success rates are finer than five-percentage-point increments; aggregation is not explained.
Go to primary source ↓alternativesPDF pp. 16–17, Appendix C
Authors report negative results for direct mask downsampling, a separate 3D CNN encoder, ControlNet-style injection and zero-initialized gated VLA fusion, but provide no numerical comparison for these alternatives.
Go to primary source ↓limitsPDF p. 8, Section 6; pp. 3–5, method; pp. 13–17, Appendices A–C
Authors acknowledge mask-supervision and segmentation dependence and defer large-scale RGB-mask-action pretraining. Supplied implementation descriptions omit optimizer, learning rate, batch size, training steps, GPU configuration, exact action chunk length and action-sampling schedule.
Go to primary source ↓qualitativePDF p. 18, Figure 9; p. 19, Figure 10; p. 20, Figures 11–12
The appendix shows real-world rollouts, offline predicted RGB/mask futures and attention comparisons. These are qualitative examples without a quantitative attention-grounding metric.
Go to primary source ↓8.5 Primary sources
MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models ↗
PDF · 8,435 extracted words
Source fingerprint
84539c56017f173abff63d1ab31c8c0578794583e9a9a54f103476e8dae06b8a