ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
1. Paper overview
In one sentence: ST-WAM combines VAE dynamics, DINO future supervision and current-conditioned history to improve robustness under visual shifts, while paying additional inference cost without generating futures at deployment. e02e03e04e05e06e13e14e15e16e17
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Video-generative WAMs may encode distracting appearance alongside useful dynamics. The authors call the drift of shifted-scene predictions toward training-domain content Training-Distribution Hallucination; their manual audit flags 70.6% of 180 predictions. This motivates changing supervision and temporal context, but the audit itself does not establish a causal connection between hallucination and action failure. e02e04 |
| Core mechanism | Source description A controlled 290-triplet diagnosis compares identical task states across visual conditions and a different-state reference. DINOv3 has higher same-state similarity and better state discrimination than Wan-VAE. e03 |
| A key reported result | Zero-shot manipulation on LIBERO-Plus: 72.8% Overall success rate (%). LIBERO-trained policy; no fine-tuning; 10,030 test cases across seven perturbation dimensions. Fast-WAM 51.5%; Fast-WAM-Joint 59.0%; X-VLA 71.4%. The 21.3 percentage-point gain over Fast-WAM spans all seven categories. ST-WAM is not best in every category, and table baselines are imported from cited studies. e09e13 |
| Reading caution |
Core contributions
- Source description
A controlled 290-triplet diagnosis compares identical task states across visual conditions and a different-state reference. DINOv3 has higher same-state similarity and better state discrimination than Wan-VAE. e03
- Source description
Dual-Space Future Experts preserve VAE dynamics while adding DINO future supervision; CAIR adds current-conditioned semantic history. Training uses no additional embodied pretraining or task-specific semantic labels, while retaining pretrained visual, language and video models. e04e06e08e11
Figure 1. Prediction drift motivates a controlled comparison of representations. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) across time: the examples show background, illumination and camera-related prediction drift for LingBot-VA or Fast-WAM-Joint. These are generated futures. In panel (b), compare the clean and shifted initial frames, whose robot and object states match, with the different-state final frame. The DINO color maps illustrate representations, while the adjacent plot summarizes same-state cosine similarity across 290 triplets. Its labels give means of 0.904 for DINOv3 and 0.686 for Wan-VAE. Page 2 adds the separate state-discrimination comparison, so the figure should be read together with that diagnostic protocol. e02e03e06e20
What it supports. DINOv3 is more stable across the tested visual changes: the plotted mean similarity gap is 0.218. The text also reports that the shifted initial frame ranks its matched clean state above the final state in 95.2% of triplets, versus 60.0% for Wan-VAE. This motivates using DINO for future targets and history.
Where the evidence stops. The qualitative examples and similarity statistics do not prove that hallucination causes control failure. The paper defers additional diagnostic details to an unsupplied supplement; the plot’s confidence interval concerns feature similarity, not robot success.
2. Motivation
2.1 The problem and the proposed response
Video-generative WAMs may encode distracting appearance alongside useful dynamics. The authors call the drift of shifted-scene predictions toward training-domain content Training-Distribution Hallucination; their manual audit flags 70.6% of 180 predictions. This motivates changing supervision and temporal context, but the audit itself does not establish a causal connection between hallucination and action failure. e02e04
2.2 What this reading follows
A robot can predict a plausible future and still lose track of the scene it actually faces. ST-WAM studies this problem through visual perturbations, then changes how future supervision and recent observations enter an action policy. Its two future experts learn in VAE and DINO spaces; its history module retrieves DINO features using queries anchored in the current image and instruction. The crucial architectural detail is the attention mask: actions never read future tokens. The six visuals below connect that information flow to the diagnostic evidence, zero-shot transfer results and ablations, while separating representation stability from demonstrated robot success. e02e03e04e05e06e13e14e15e16e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Separate world and action expert parameters support the catalog’s coarse Dual-system label, although the actual architecture has three interacting branches. Other mechanisms/Outside quadrants fits deployment through predictive auxiliary training and retrieved history, without inverse-dynamics action extraction or generated-future conditioning. Joint training alone would not justify One Model. Latent prediction and memory are supported; the history is explicitly short-horizon, and neither JEPA-specific training nor broad long-horizon capability follows from those category labels. e04e05e06e07e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Understand how future targets can help an action-only policy
During training, ST-WAM observes demonstrated actions and future frames. Frozen encoders map those frames into VAE and DINO targets, which are corrupted with Gaussian noise alongside action targets. The future streams share a timestep so their mixed attention compares corresponding denoising stages; the action timestep is sampled independently. The loss asks each expert to estimate noise minus its clean target. Crucially, the action branch cannot read the noisy futures, and current anchors cannot relay them. My interpretation is that predictive learning shapes the parameters used to process present evidence, rather than supplying future content as an action input. At deployment the policy therefore retains current visual/semantic computation and CAIR but removes future-token streams. It integrates actions and replans after executing part of the chunk. e04e05e07e08
Figure 2. Two future spaces train the policy, while current-anchored history supplies action context. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start on the left with the current and future RGB observations. The frozen VAE and DINOv3 encoders send tokens into separate future experts alongside the action expert. The future RGB and DINO outputs are explicitly marked train only; the VAE target is latent even though the drawing illustrates RGB. On the right, current RGB and language enter frozen Qwen3-VL. A learnable query resampler creates current semantic queries, which attend to temporal DINO history as keys and values. Follow the resulting intent tokens back to the action expert’s cross-attention. Equations (6)–(7) confirm that this history context enters only the action branch. e04e05e06e08
What it supports. The design gives DINO two temporal roles: supervising future semantic states and representing recent observations for retrieval. CAIR produces eight intent tokens without task-specific intent labels. Separate branch parameters and heads remain inside the joint training system; the figure does not depict a single shared predictor for every modality.
Where the evidence stops. The overview omits the detailed routing restrictions and shared T5/proprioceptive contexts. Read it with Figure 3 and Eqs. (4)–(7): neither the future-output arrows nor the term intent means that deployment plans through an explicitly generated future.
5.2 Follow the current anchor into short semantic history
CAIR does not start by treating all old observations as equally relevant. It first forms queries from the current image and instruction through frozen Qwen3-VL and a learnable resampler. Frozen DINOv3 encodes the four selected past observations; projection and temporal embeddings turn their patches into the history sequence. Two cross-attention blocks let the current semantic queries retrieve that sequence, and an output projection produces eight intent tokens. The word intent describes a label-free summary trained through the action objective, not a separately supervised task-state variable. These tokens augment the action expert’s language and proprioceptive context. The history ablations matter here: current Qwen features alone, unanchored retrieval and VAE-based history all underperform full CAIR, and even underperform dual-space prediction without CAIR. e06e08e17
Figure 3. The mask makes future prediction a training signal without future-token access for actions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as the query stream and each column as the stream it can attend to. Colored cells allow attention; white cells block it. In the training panel, the current visual and semantic tokens, v0 and s0, read one another but no future or action tokens. Future rows exchange visual and semantic information while excluding actions. The purple action rows read only v0, s0 and other actions. The smaller inference panel retains exactly these current/action relationships after removing future rows and columns. This matches the structured-mask description on page 3, including the direction of every claim-relevant dependency. e05e07e16
What it supports. The direct action computation cannot use the noisy future targets because the corresponding attention entries are blocked, and current anchors cannot relay them. The future objectives can still influence learned parameters during training. At deployment, current visual and semantic streams remain, so removing future tokens does not remove all world-model computation.
Where the evidence stops. The diagram specifies permitted dependencies; it is not an implementation test of leakage or numerical equivalence. Future-supervision ablations support a training benefit, but the mask alone cannot establish why that benefit improves robustness.
5.3 Separate three kinds of robustness evidence
The frame-triplet study tests whether representations preserve state distinctions under appearance change. The simulation tables test executed task success, but with different exposure to shifts: LIBERO-Plus is zero-shot transfer from LIBERO, whereas RoboTwin uses mixed clean and randomized demonstrations during training. The real robot table adds physical execution, with matched task demonstrations and initializations but only five separately trained tasks. These results reinforce one another without being interchangeable. My reading is that the ablations provide stronger support for complementary supervision and retrieval than the illustrative hallucinations provide for a causal failure theory. The latency measurement adds a concrete tradeoff: complete chunk inference is 756.17 ms on the stated A100 setup, versus 609.30 ms for Fast-WAM. It does not by itself establish a robot control rate. e02e03e09e10e12e14e15e16e17
5.4 Training and inference
During training
The visual DiT is 5B; semantic and action DiTs are 1B each, initialized from Wan2.2. VAE, T5, DINOv3 ViT-S/16 and Qwen3-VL-4B-Instruct stay frozen. Future branches share a noise timestep; the action timestep and all branch noises are independent. CAIR learns through the action loss. e06e07e08
LIBERO/RoboTwin training uses 10/5 epochs and global batches 128/1,024. AdamW uses learning rate 0.0001, weight decay 0.01, cosine decay, BF16 and gradient clipping 1.0. The shifted flow schedule has shift 5.0; visual/semantic/action loss weights are 1.0/0.02/1.0. e08
During inference
Encode current inputs and history, construct intent context, and integrate the action flow for 10 steps while omitting future-token streams. Predict 32 actions, execute 10, then replan using updated observations. Current visual/semantic processing remains active: action-only output does not mean the action DiT is the only deployed component. e04e05e08e14
5.5 Implementation flow
- Represent current and future states twice
Frozen Wan2.2 VAE and DINOv3 encoders produce visual and semantic tokens, partitioned into clean current conditions and future targets. DSFE uses a pretrained visual DiT and a semantic DiT; a third DiT predicts actions. Each has separate parameters and a prediction head. e04e05
- Route information without future leakage
Mixed attention lets current VAE/DINO streams interact. Noisy futures read both current anchors and each other, but not actions. Actions read current streams and themselves, never future streams. Consequently future prediction supplies a training objective, rather than an imagined trajectory passed into a deployment planner. e05
- Retrieve history with a current anchor
Learnable queries first attend to frozen Qwen3-VL features of the current scene and instruction. These semantic queries retrieve temporally embedded DINO history through two cross-attention blocks. Eight resulting intent tokens condition only the action expert; all experts receive T5 language context and projected proprioception. Intent has no explicit annotation. e06e08
6. Experiments & results
ST-WAM trains separate visual-future, semantic-future and action experts together, then deploys an action-only policy. DINOv3 supplies both semantic future targets and recent-history features retrieved under current image/language context. The central result is improved manipulation under visual shifts, supported by component ablations; it does not establish that correcting generated videos is necessary for control.
6.1 Read the original evidence
Table 3. Zero-shot transfer improves substantially over Fast-WAM, with uneven strengths across perturbations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with Overall at the right, then compare ST-WAM with Fast-WAM and Fast-WAM-Joint category by category. ST-WAM’s 72.8% overall exceeds 51.5% and 59.0%, respectively. Camera rises from 16.4% to 55.4% against Fast-WAM; Noise rises from 37.7% to 79.5%. Emb. PT. indicates embodied pretraining, rather than all use of pretrained models. The protocol evaluates a LIBERO-trained policy without fine-tuning on 10,030 LIBERO-Plus cases. The caption credits baseline results to two prior studies, so the table is a reported benchmark comparison rather than evidence that every model was rerun within one controlled training pipeline. e08e09e11e13
What it supports. The strongest transfer comparison is a 21.3 percentage-point overall improvement over Fast-WAM, with gains in all seven perturbation categories. ST-WAM also exceeds the listed X-VLA overall score of 71.4%. The category columns show that this aggregate lead does not mean uniformly strongest performance.
Where the evidence stops. Language success is 79.3%, below Fast-WAM-Joint’s 88.9%, and other models lead several categories. No success-rate confidence intervals or seed variation are reported here; embodied pretraining and training pipelines differ across rows.
Table 4. The real robot results connect robustness to both semantic prediction and history retrieval. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two halves use different column groupings. Nominal Environment lists Flower, Drawer, Scoop, Fruit and Hang separately, then their average. Visual Distribution Shifts lists background, lighting, object appearance and their compound combination; each shift column averages all five tasks. Thus the two Avg. columns summarize different test conditions. All methods use the same 50 nominal demonstrations per task and are post-trained separately for each task. Evaluation uses 30 trials per task and condition with predefined initializations, without adaptation to the shifts. Compare the component-removal rows with ST-WAM to see how the future expert and CAIR each contribute under this protocol. e10e15e19e20
What it supports. Shifted success is 61.5% for ST-WAM versus 25.8% for Fast-WAM, a 35.7 percentage-point gain. Removing the semantic future expert yields 41.0%; removing CAIR yields 43.7%. Compound-shift success remains higher at 48.0% versus Fast-WAM’s 15.3%, supporting complementary benefits from both components.
Where the evidence stops. The shifted columns conceal task-specific variation, whose breakdown is deferred to the unsupplied supplement. The experiment covers five separately trained tasks on one robot, and compound success remains below half; transfer to new physical dynamics is untested.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot manipulation on LIBERO-Plus LIBERO-trained policy; no fine-tuning; 10,030 test cases across seven perturbation dimensions. | 72.8% Overall success rate (%) | Fast-WAM 51.5%; Fast-WAM-Joint 59.0%; X-VLA 71.4%. The 21.3 percentage-point gain over Fast-WAM spans all seven categories. ST-WAM is not best in every category, and table baselines are imported from cited studies. e09e13 |
| In-distribution manipulation on LIBERO Four suites, 40 tasks, 50 evaluation rollouts per task. | 98.7% Average success rate (%) | Fast-WAM 97.6%; ST-WAM Spatial/Object/Goal/Long: 99.0/100.0/99.0/96.8%. A high aggregate with a small margin over strong baselines; Table 1 supplies no uncertainty estimates. e09e11 |
| Bimanual manipulation on RoboTwin 2.0 2,500 clean plus 25,000 randomized training demonstrations; 100 trials per task in each setting. | 93.06% clean; 92.48% randomized; 92.77% average. Success rate (%) | Fast-WAM 91.83% average. The abstract rounds the average to 92.8%. Randomized evaluation follows mixed-domain training, unlike zero-shot LIBERO-Plus. e09e12 |
| Real-world manipulation under visual shifts Five Agilex Piper tasks; 50 shared nominal demonstrations per task; separate task training; 30 trials per task and condition. | 61.5% shifted; 79.3% nominal; 48.0% compound shift. Average success rate (%) | Fast-WAM 25.8% shifted, 64.7% nominal, 15.3% compound; pi0 32.8% shifted. The shifted gain over Fast-WAM is 35.7 percentage points. Each shift column averages five tasks; this does not establish generalization to new physical dynamics. e10e15e19 |
| Future-supervision and history ablation on LIBERO-Plus Table 5 controlled variants; LIBERO-trained models evaluated on LIBERO-Plus. | Full model 72.8%; dual-space without CAIR 66.4%. Success rate (%) | DINO-only 39.7%; VAE-only Fast-WAM 51.5%; parameter-matched semantic expert without future objective 62.9%; unanchored retrieval 56.5%. Results support complementary future spaces and carefully conditioned history. The capacity control strengthens the future-objective argument, but the table supplies no run-to-run uncertainty. e16e17 |
| Action-chunk inference latency on RoboTwin 2.0 Single NVIDIA A100-80GB, BF16, 10 flow steps; 20 synchronized full-call runs; 32-action output. | 756.17 ms Mean latency (ms; lower is better) | Fast-WAM 609.30 ms; reported 1.24× latency. Sub-second chunk generation incurs overhead. This measurement is not an end-to-end robot control-frequency guarantee. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. The controls distinguish semantic supervision from extra capacity and useful history from distracting context. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Future Prediction and Intent Conditioning columns before comparing success. The first group compares VAE-only, DINO-only and dual-space prediction without intent. The next group keeps CAIR but removes either the semantic expert or its future objective. The text identifies Semantic Expert w/o Future Obj. as parameter-matched: current-DINO conditioning, the semantic DiT and mixed attention remain, while future targets and loss are removed. The final group changes retrieval to unanchored DINO history, current Qwen features alone, or VAE history. LIBERO-Plus is the most revealing column because the full model’s benefit is largest under distribution shift. e16e17
What it supports. Dual-space prediction without CAIR reaches 66.4%, above VAE-only 51.5% and DINO-only 39.7%. The parameter-matched no-future-objective variant reaches 62.9%, below full ST-WAM’s 72.8%. All three alternative history-conditioning designs also fall below the 66.4% no-CAIR dual-space baseline, showing that adding context can hurt.
Where the evidence stops. These controls strengthen the proposed mechanism but do not provide seed-level uncertainty or a complete factorial experiment. In particular, the parameter-matched control changes future targets and loss together; a numerical advantage alone does not prove a unique causal explanation.
7. Analysis & limitations
7.1 What the evidence leaves open
Physical-dynamics and embodiment shifts remain future work. Real-world evidence covers five separately trained tasks with fixed nominal demonstrations; compound-shift success remains 48.0%. e10e15e19
Tables report point estimates without seed variation or confidence intervals. The diagnostic plot does show a confidence interval for a representation-similarity gap; that is not uncertainty on robot success. Attention heatmaps are qualitative, normalized within each map, and cannot prove causal feature use. e03e11e12e13e15e16e17e18
7.2 Questions for discussion
- Would correctly aligned DINO history outperform time-shuffled history when current inputs and compute are matched?
- How much of the semantic-future advantage persists across training seeds and under changes in physical dynamics?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruct the stated frozen checkpoints, separate experts and mask. Use future horizon K=8 sampled every four control steps; history indices are t−24,t−16,t−8,t−1. Match task data and evaluation initializations before comparing success or synchronized latency. e05e08e09e10e14
The supplied main paper leaves exact scheduler weighting, detailed preprocessing/action normalization, training hardware/time and seed-level variability unresolved. Additional implementation and task/subset results are deferred to an unsupplied supplement; a complete replication needs these specifications. e07e08e20
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test semantic prediction against matched capacity and corrupted targets
Reader-proposed experiment, not performed: repeat the LIBERO-to-LIBERO-Plus comparison for the full model and the parameter-matched Semantic Expert w/o Future Obj. control, holding CAIR, initialization, demonstrations, optimizer, training steps and evaluation cases fixed. Add a third variant that preserves the semantic loss and its stated 0.02 weight but permutes future DINO targets across demonstrations within a task. Use multiple matched seeds and report per-category success with uncertainty. The proposed mechanism predicts that correctly paired future targets outperform both controls. If the shuffled targets perform comparably, the claim that useful semantic transition supervision explains the advantage would weaken, even if extra training regularization still helps. e07e08e09e16
Check 2: Test whether CAIR uses recent progress rather than extra image context
Reader-proposed experiment, not performed: evaluate the same trained policy with correctly aligned history, history assigned to incorrect temporal slots, and four repeated current-frame features. Keep the current image, instruction, proprioception, query count, token budget, action noise and evaluation initializations fixed; preserve the stated retrieval architecture. Compare nominal and shifted success by task, alongside the trained Qwen Current Only and no-CAIR controls. A consistent advantage for aligned history would support retrieval of useful temporal evidence. Similar scores for repeated-current or shuffled histories would weaken the interpretation that recent progress, rather than additional semantic context, drives the gain. Any degradation from these interventions must also be distinguished from simple out-of-training-distribution sensitivity. e06e08e09e10e17
8.3 Reading coverage
Visual audit: All nine supplied PDF pages were rendered and visually inspected, alongside individual reading of all four text chunks. The title, all authors and affiliations were checked on p. 1. Figures 1–5 and Tables 1–5 were inspected; final crops retain Figure 1, Figure 2, Figure 3, Table 3, Table 4 and Table 5. All six final assets were viewed at their native crop resolutions. Figure 3 routing was checked against p. 3, and Figure 2 against Eqs. (4)–(7); no claim-relevant conflict was found. Pages 5–6 supply training, evaluation and latency details. No appendix is present; referenced supplementary material, code and task/subset breakdowns were not supplied or inspected. All numerical and method claims in this bundle are supported by the declared pages.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- Introduction (pp. 1–2)
- Related Work, all three subsections (pp. 2–3)
- Methodology: Problem Formulation, DSFE, CAIR and Joint Flow-Matching Objective (pp. 3–5)
- Experiments: setup, implementation, main results, ablations and qualitative analysis (pp. 5–7)
- Conclusion (pp. 7–8)
- References (pp. 8–9)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure omission was resolved by visually inspecting all nine PDF pages, Figures 1–5, Tables 1–5 and all six final crops.
- No supplement was supplied. Referenced diagnostic details, additional implementation details, task-wise shifted results and subset ablations remain outside this review.
- Code was not inspected and experiments were not reproduced.
- Identity note (e01): title and all 15 authors match. The observed artifact is arXiv:2607.28993v1, dated 31 July 2026; the catalog submittedDate is 30 July 2026. No other revision was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin
The title matches the supplied observed title. All 15 authors match the catalog after name-order normalization. The margin identifies arXiv:2607.28993v1 [cs.RO], 31 Jul 2026. Six affiliations are printed; Houde Liu and Tianlun Li are corresponding authors.
Go to primary source ↓e02PDF pp. 1–2, Introduction; p. 1, Figure 1(a)
The authors audit LingBot-VA and Fast-WAM-Joint predictions: 30 cases per model for each of background, illumination and camera shifts, totaling 180 predictions; 70.6% are judged to exhibit training-distribution hallucination.
Go to primary source ↓e03PDF p. 1, Figure 1(b); p. 2, Introduction, controlled representation diagnosis
Among 290 frame triplets, mean same-state cosine similarity is 0.904 for DINOv3 and 0.686 for Wan-VAE. The shifted initial frame is closer to the matched clean initial than to a different-state final frame in 95.2% versus 60.0% of triplets. The plot labels a mean gap of 0.218 and 95% CI [0.200, 0.235]; further diagnostic details are deferred to supplementary material.
Go to primary source ↓e04PDF p. 3, Problem Formulation, Eqs. (1)–(3); Dual-Space Future Experts
Current multi-view images, proprioception, language and short history condition action chunks. Frozen VAE and DINO encoders supply current tokens and future targets. Training models actions and both future spaces; deployment is an action-only policy.
Go to primary source ↓e05PDF p. 3, Three-Branch Mixture-of-Transformers and Structured Cross-Branch Attention Mask; p. 4, Figures 2(a) and 3
Visual, semantic and action DiTs have separate parameters and heads with mixed attention. Current streams read one another but not future/action tokens. Future streams read current and future streams, excluding actions. Actions read current streams and themselves, excluding futures. Figure 3 agrees with the textual routing and removes future tokens at inference.
Go to primary source ↓e06PDF pp. 3–4, Current-Anchored Intent Retrieval, Eqs. (4)–(7); p. 4, Figure 2(b)
Learnable queries attend to frozen Qwen3-VL current image/language features. The resulting anchors retrieve projected DINO history with temporal embeddings through cross-attention. Projected intent tokens condition only the action expert; all experts also receive T5 language and projected proprioception. Intent is label-free and trained through action flow matching.
Go to primary source ↓e07PDF p. 4, Joint Flow-Matching Objective, Eqs. (8)–(11); p. 5, opening continuation
Each clean branch target is interpolated with Gaussian noise; zero timestep is clean and one is noise. Target velocity is noise minus clean target. Future timesteps are shared, the action timestep is independent, and all noises are independent. Scheduler-weighted squared velocity errors are combined with branch loss weights.
Go to primary source ↓e08PDF p. 5, Implementation Details, both columns
The visual DiT has 5B parameters; semantic and action DiTs are 1B each, initialized from Wan2.2. Wan2.2 VAE/T5, DINOv3 ViT-S/16 and Qwen3-VL-4B-Instruct remain frozen. H=32, K=8, future sampling interval is four control steps; CAIR uses history indices t−24,t−16,t−8,t−1, two blocks and eight queries. Training uses 10/5 epochs and batch sizes 128/1024 for LIBERO/RoboTwin, AdamW at 1e−4, weight decay 0.01, cosine decay, BF16, clipping 1.0, schedule shift 5.0 and loss weights (1.0,0.02,1.0). Inference uses 10 integration steps and executes 10 actions before replanning.
Go to primary source ↓e09PDF p. 5, Experimental Setup, Benchmarks and Protocols
LIBERO covers 40 tasks across four suites with 50 evaluation rollouts per task. LIBERO-Plus uses the LIBERO-trained policy without fine-tuning on 10,030 cases across seven perturbation dimensions. RoboTwin training mixes 2,500 clean and 25,000 heavily randomized demonstrations; each task has 100 evaluation trials in each setting.
Go to primary source ↓e10PDF p. 5, Real-World Evaluation
An Agilex Piper 6-DoF single arm performs Arrange Flowers, Drawer Organization, Bean Scooping, Arrange Fruits and Hang Mug. Each method is post-trained separately per task on the same 50 nominal demonstrations. Evaluation uses 30 trials per task and condition with predefined initializations, nominal scenes and background, lighting, object-appearance and compound shifts without further fine-tuning.
Go to primary source ↓e11PDF p. 5, Table 1, ST-WAM and Fast-WAM rows; caption
LIBERO success for ST-WAM is 99.0/100.0/99.0/96.8% on Spatial/Object/Goal/Long, averaging 98.7%, versus Fast-WAM 97.6% average. Emb. PT. denotes large-scale robot-trajectory or embodied-video pretraining before adaptation; both rows are marked No.
Go to primary source ↓e12PDF p. 5, Table 2, ST-WAM and Fast-WAM rows; caption; p. 1, Abstract
Under mixed clean/randomized RoboTwin training, ST-WAM scores 93.06% clean and 92.48% randomized, averaging 92.77%; Fast-WAM averages 91.83%. The abstract on p. 1 rounds the ST-WAM average to 92.8%.
Go to primary source ↓e13PDF p. 6, Table 3, all rows and caption; Zero-Shot Generalization on LIBERO-Plus
ST-WAM reaches 72.8% overall versus Fast-WAM 51.5%, Fast-WAM-Joint 59.0% and X-VLA 71.4%. ST-WAM category scores are 55.4/60.1/79.3/93.0/74.2/79.5/74.3 for Camera/Robot/Language/Light/Background/Noise/Layout. It exceeds Fast-WAM in all seven, but Fast-WAM-Joint scores 88.9% on language. Baseline results are taken from the two studies cited in the caption.
Go to primary source ↓e14PDF p. 6, Inference Efficiency
Complete action-chunk inference, including all deployment components, is measured on one NVIDIA A100-80GB using BF16 and 10 integration steps. Over 20 synchronized runs, a 32-step chunk averages 756.17 ms for ST-WAM and 609.30 ms for Fast-WAM, reported as 1.24× latency.
Go to primary source ↓e15PDF p. 6, Real-World Generalization and Figure 4; p. 7, Table 4 and caption
ST-WAM nominal/shifted averages are 79.3/61.5%, versus Fast-WAM 64.7/25.8% and pi0 47.3/32.8%. ST-WAM background/light/object/compound success is 66.0/70.0/62.0/48.0%, with each shift averaged over five tasks. Compound applies all three shifts; Fast-WAM scores 15.3%. Removing the semantic future expert or CAIR yields shifted averages 41.0% or 43.7%.
Go to primary source ↓e16PDF pp. 6–7, Ablation Q1–Q2; p. 7, Table 5, future-representation and semantic-objective rows
LIBERO-Plus scores are 39.7% for DINO Future Only, 51.5% for Fast-WAM, 66.4% for Dual-Space w/o CAIR, 63.5% without the semantic expert, 62.9% for the parameter-matched semantic expert without future target/loss, and 72.8% for the full model. The parameter-matched control retains the semantic DiT, current-DINO conditioning and mixed attention.
Go to primary source ↓e17PDF p. 7, Table 5, history-conditioning rows; Ablation Q3
Naive unanchored DINO retrieval achieves 56.5% on LIBERO-Plus, Qwen Current Only 62.3%, and CAIR with VAE History 64.7%, versus dual-space without CAIR 66.4% and full DINO-based CAIR 72.8%. VAE history retains the Qwen anchor and retrieval architecture.
Go to primary source ↓e18PDF p. 7, Figure 5, caption and Cross-Branch Attention Visualization
Two task heatmaps show action queries attending to current DINO and VAE tokens. The authors describe DINO attention as more aligned with manipulated objects and interaction regions. Warm colors show relative attention within each map, not a shared calibrated scale.
Go to primary source ↓e19PDF pp. 7–8, Conclusion, especially p. 8 final sentence
Extension beyond visual shifts to changes in physical dynamics and embodiments is future work, rather than an established result.
Go to primary source ↓e20PDF p. 2, diagnosis paragraph; p. 5, Implementation Details ending; p. 6, Real-World Generalization ending; p. 7, Q3 ending; pp. 8–9, References
The paper refers to supplementary diagnosis details, implementation details, task-wise shifted results and subset ablations. The supplied nine-page PDF ends in references and contains no appendix or those supplemental breakdowns.
Go to primary source ↓8.5 Primary sources
ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts ↗
PDF · 5,243 extracted words
Source fingerprint
0c31090b27a4c604b596373354950b462cf387098c2bb993305dfcee6c155853