HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models
1. Paper overview
In one sentence: HarmoWAM uses a learned stage gate to combine video-guided transit with latent-conditioned action diffusion for precise interaction, gaining robustness to controlled distribution shifts at the cost of video generation and a multi-module training pipeline. e-motivatione-worlde-gatee-traininge-oode-ide-ablation
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Reaching an unfamiliar object and manipulating it precisely can fail for different reasons. In the authors' two-task study, generated-video inverse dynamics reaches targets reliably, whereas a latent-conditioned diffusion policy retains interaction precision when initialized nearby but often fails transit under distribution shift. These particular baselines motivate stage-dependent control; they do not establish a universal division between all WAMs. e-motivation |
| Core mechanism | Source description A shared Wan world model supplies explicit future frames and implicit dynamics features to complementary Transformer-based action experts. e-worlde-predictivee-reactive |
| A key reported result | Six-task zero-shot OOD manipulation: 0.82 overall; background 0.81, position 0.80, objects 0.85; reported relative drop from ID 7.9%. Table 3 Global Avg of task/sub-stage scores across OOD settings. Same downstream tasks with unseen backgrounds, spatially disjoint object positions, or substitute objects; no OOD fine-tuning. Wan+AnyPos 0.53, pi0.5 0.49, Cosmos-Policy 0.44. The headline 29 and 33 margins are percentage-point gains over the strongest listed WAM and VLA OOD averages. This tests controlled shifts in familiar tasks, not arbitrary new skills. e-oode-ood-settinge-protocol |
| Reading caution | Author claim Fixed video horizon limits temporal adaptation, and pixel-level future generation adds overhead. Adaptive horizons and latent-only prediction are proposed future work. e-limitations |
Core contributions
- Source description
A shared Wan world model supplies explicit future frames and implicit dynamics features to complementary Transformer-based action experts. e-worlde-predictivee-reactive
- Source description
A visual process classifier selects one expert using interaction labels derived from demonstration events, rather than averaging incompatible action proposals. e-gatee-gate-labels
- Source description
The evaluation combines six physical tasks, three controlled distribution shifts, stage-wise measurements and mechanism ablations. e-protocole-ood-settinge-ide-oode-ablation
Table 1. Separating target-reaching from contact exposes different baseline failure modes. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with each task's Transit and Interaction columns, not a single combined success rate. The rows contrast Wan plus AnyPos with a Wan-conditioned Action DiT on can stacking and flower placement. Each phase has ten trials per task and domain. The small sequences illustrate motion toward the object followed by contact. Crucially, the asterisk means the joint-model robot starts near the target for the OOD interaction test; this deliberately removes the reaching difficulty. Read those entries as an isolated manipulation diagnostic, not as successful completion after the corresponding transit trial. The retained caption carries that essential protocol qualification. e-motivation
What it supports. In the position-OOD rows, Imagine-then-Execute reaches both targets in 10/10 trials, yet interaction succeeds in 5/10 stacking and 2/10 flower trials. Joint Modeling reaches them in only 3/10 and 0/10 trials, while near-target interaction reaches 10/10 for both. This is the empirical motivation for specializing the action pathways.
Where the evidence stops. The comparison concerns two concrete implementations and small trial sets. Near-target initialization makes the interaction columns intentionally different from ordinary full-task execution; multiplying or averaging these columns would not reconstruct an observed episode-completion rate.
2. Motivation
2.1 The problem and the proposed response
Reaching an unfamiliar object and manipulating it precisely can fail for different reasons. In the authors' two-task study, generated-video inverse dynamics reaches targets reliably, whereas a latent-conditioned diffusion policy retains interaction precision when initialized nearby but often fails transit under distribution shift. These particular baselines motivate stage-dependent control; they do not establish a universal division between all WAMs. e-motivation
2.2 What this reading follows
A robot can know where to go yet fail at the final millimeters of contact. HarmoWAM starts from that separation: one baseline reaches unfamiliar object locations but struggles to manipulate them, while another manipulates well when placed nearby but often fails to reach them. Its solution shares one video world model between two specialized action experts and switches according to observed task progress. Read the figures as a chain from diagnosis to architecture to executed-robot evidence. The strongest reported gains concern familiar tasks under changed scenes; the headline scores average sub-stages, and the ablation prose contains a numerical conflict with its chart. e-motivatione-worlde-gatee-traininge-oode-ide-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The shared world model and separately structured diffusion and reactive action experts support the recorded Dual-system interpretation. The reactive branch explicitly maps imagined visual evolution to actions, supporting IDM. The label is incomplete if read exclusively: the predictive branch uses latent-conditioned action diffusion. A single overall framework is not evidence of a single unified action/video backbone. e-worlde-predictivee-reactivee-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a shared world model still needs two action mappings
The motivating experiment asks a more useful question than which baseline has the highest overall score: at what point does each policy lose the task? Reaching the target and establishing precise contact are tested separately. HarmoWAM turns the observed difference into an information-flow design. During transit, predicted future frames provide an explicit visual route, and future latent features enrich the reactive decoder's representation. During interaction, the predictive expert instead denoises an action sequence conditioned on current-step world features, current images and instruction. Both branches use learned world information, but neither merely executes a video. As reader analysis, the key hypothesis is that these mappings fail differently enough for observation-dependent selection to help. Table 1 motivates that hypothesis; it does not prove the split is optimal for every task or world model. e-motivatione-worlde-predictivee-reactivee-action
Figure 2. One video prediction supports two distinct ways to generate robot actions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue world-model band from instruction and image to latent dynamics and decoded future video. Then separate the orange latent path from the green frame path. On the left, the predictive DiT uses current-step latent conditioning, image features and text while denoising actions. On the right, DINOv2 extracts patches from predicted frames, which fuse with future latents before action decoding. Section 3.3 specifies token concatenation for this fusion symbol. The central test asks whether interaction confidence exceeds 0.5: Yes selects predictive control, while No routes to reactive control. These branch labels agree with Section 3.4 and its interaction-label convention. e-worlde-predictivee-reactivee-gatee-traininge-action
What it supports. The architectural division is about how the same world prediction becomes an action: iterative diffusion for interaction versus direct decoding of future visual evolution for transit. The world model therefore participates in inference, and its predictions condition executable action heads; video generation alone is not the robot policy's final output.
Where the evidence stops. The gate box also depicts routing of world outputs; Section 3.4 specifies the classifier input as current-image tokens. The figure is not a gradient-flow guarantee: Section 3.5 freezes the world model while training the experts and gate.
5.2 Turn demonstration events into an inference-time switch
The gating labels come from the demonstrations, not from a human specifying the active expert at deployment. Gripper opening or closing marks key events; task-dependent end-effector height thresholds add events for motions such as insertion or pouring. The pipeline labels the twenty frames before and after each event as interaction, and the remaining frames as transit. In dual-arm data, either arm can trigger the interaction window. A visual MLP then learns these labels through binary cross-entropy and selects predictive control when its confidence exceeds 0.5. The appendix reports 96.95% offline accuracy on 1,637 held-out test frame pairs. That is label-prediction evidence. As reader analysis, it does not measure the cost of a mistimed switch during a real contact, especially when heuristic labels do not match the best expert boundary. e-gatee-gate-labels
5.3 Follow the training stages through to the execution metric
The training recipe first adapts the video world model to demonstrations using flow matching, then freezes it while optimizing the action experts and gate. This matters when interpreting the authors' end-to-end description: the documented second stage does not send action-training updates into the world model. At evaluation time, the system produces actual robot commands, so the results go beyond visual plausibility. Yet the headline success metric compresses critical sub-stages. Flower manipulation illustrates the distinction: its ID headline is 0.85, while final insertion is 0.70 after earlier picking and handover stages. The same care applies to speed: 48 Hz action generation with twelve-action chunks and 4 Hz inference are different reported quantities. As reader analysis, both task aggregation and control timing must be reconstructed before attempting an independent reproduction. e-traininge-worlde-ide-stagese-speed
5.4 Training and inference
During training
Stage 1 fully fine-tunes the world model on task demonstrations with conditional flow matching. Stage 2 freezes it and trains the predictive expert with diffusion noise prediction, the reactive expert with Smooth L1 (beta 0.1), and the gate with binary cross-entropy. e-training
Training uses eight NVIDIA H20 GPUs and 100 demonstrations per task. World-model initialization includes broad robot pretraining: Table 4 lists 201,119 DROID, 3,017 AgiBot and 1,721,985 RoboMIND trajectories; additional closed-source data are mentioned without a count. e-pretraininge-protocol
During inference
A confidence above 0.5 selects the predictive expert; otherwise the reactive expert uses predicted frames and future latents. Repeated current-observation feedback supports closed-loop switching. Figure 2's Yes/No branches agree with Section 3.4. e-gatee-world
The introduction reports 48 Hz action generation with chunks of 12, while Table 9 reports 4 Hz inference at five world-model steps. Their arithmetic is compatible with chunk throughput, but the source does not establish a 48 Hz observation-to-action replanning loop or specify latency hardware. e-speed
5.5 Implementation flow
- Predict visual evolution
Wan2.2-TI2V-5B generates a 13-frame video at 256 × 320 using five denoising steps. Video predictions and latent features condition control; imagined frames are not executable commands. e-worlde-action
- Predictive expert for interaction
A 1B-parameter, 28-block Action DiT denoises action sequences. Cross-attention combines current-step world-model features with SigLIP image features and instruction features. The world feature has shape B × 80 × 3072. e-worlde-predictive
- Reactive expert for transit
DINOv2-base encodes each predicted future frame into 1369 patch tokens of width 768. Corresponding world features are pooled from 3072 to 768 channels and concatenated along tokens. A convolutional Orientation Decoder maps the fused representation to actions. e-reactive
- Route from observed progress
An MLP reuses current-image tokens to predict interaction confidence. Gripper transitions and task-specific height events label interaction windows spanning 20 frames before and after each keyframe; either arm can trigger a dual-arm interaction label. e-gatee-gate-labels
6. Experiments & results
HarmoWAM routes a shared video world model into two action experts: inverse-dynamics-style transit control and diffusion-based precise interaction. Real Franka experiments report strong generalization to changed backgrounds, positions and objects, but headline success rates average sub-stages and the implementation remains partly unspecified.
6.1 Read the original evidence
Table 2. High in-domain averages span both single-arm and bimanual tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across the highlighted row to see whether the benefit is confined to one task. Then compare down a fixed column, especially stacking and the two dual-arm tasks. The caption on the inspected source page defines each task score as an average over critical sub-stages. The final Avg column aggregates task scores; it should not be read as the fraction of complete episodes that finished every subgoal. Dashes mark unreported dual-arm evaluations for VPP and QwenVLA-OFT, so their averages cover a smaller task set. Section 4.1 specifies twenty evaluation episodes per task and human assessment. e-ide-protocol
What it supports. HarmoWAM reports 0.89 overall, compared with 0.78 for Cosmos-Policy, 0.74 for pi0.5 and 0.67 for Wan+AnyPos. Stacking reaches 0.90 and each dual-arm task reaches 0.85. The full six-task comparisons support a broad ID advantage within this platform and training regime.
Where the evidence stops. The authors report a three-repeat standard deviation of 0.03 for HarmoWAM, without matching uncertainty for every baseline. Baseline inputs and pretraining also differ: Appendix C.1 gives Wan+AnyPos a single third-person view while other policies use multiple views.
Table 3. Controlled scene changes widen the reported advantage over the compared policies. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each three-row block as one method evaluated under three separate shifts. The scenario Avg column summarizes tasks, while Global Avg combines the shift settings. Appendix C.3 defines background changes as distractors and lighting, position changes as regions outside demonstration coverage, and object changes as appearance/geometry substitutions. In the highlighted block, compare the position row with the same row for each baseline; it is especially revealing about reaching beyond familiar coordinates. The red percentages at the far right are relative drops from each method's ID score, not percentage-point losses and not comparisons against HarmoWAM. e-oode-ood-settinge-protocole-id
What it supports. The reported OOD global score is 0.82 versus 0.53 for Wan+AnyPos and 0.49 for pi0.5, giving 29- and 33-percentage-point gaps. HarmoWAM's position average is 0.80, whereas Cosmos-Policy reports 0.26. Its three scenario averages remain close to one another despite their different visual and spatial demands.
Where the evidence stops. Zero-shot here means no fine-tuning for these scene variations after training on the same task family. These remain task/sub-stage averages on one experimental platform, with missing dual-arm entries for some baselines; they do not establish generalization to arbitrary skills or embodiments.
Table 6. Final-stage results reveal the remaining gap between progress and task completion. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the arrow-linked columns as ordered subgoals. For flowers, Appendix B defines S1 as picking, S2 as the handover, and S3 as insertion into the vase. For the bag, the sequence progresses through item pickup, placement, bag stabilization, zipper grasp and pulling the zipper closed. These definitions matter because the earlier headline table compresses an entire sequence into a stage average. Compare each method's final column within a task before looking at the rightmost Avg. That reveals whether a method's early progress persists into the contact-intensive endpoint, while keeping the original sequential evaluation protocol visible. e-stagese-ide-protocol
What it supports. HarmoWAM's flower sequence is 0.95, 0.90 and 0.70, so final insertion is harder than its 0.85 stage average suggests. It still exceeds the best compared final insertion score, pi0.5's 0.40. The bag sequence ends at 0.80, versus 0.55 for Cosmos-Policy, supporting an advantage at a demanding endpoint.
Where the evidence stops. The successive columns describe sequentially evaluated stages, not independent Bernoulli events or conditional transition probabilities. Do not multiply them to estimate completion. This is an ID table; its late-stage values should not be substituted for the OOD headline averages.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Six-task in-domain manipulation Four single-arm and two dual-arm tasks; 20 independently initialized episodes per task, human stage evaluation. | 0.89; authors report ±0.03 standard deviation from three repeated evaluations. Reported average of task-level sub-stage success rates | Cosmos-Policy 0.78; pi0.5 0.74; Wan+AnyPos 0.67. The displayed differences are 11, 15 and 22 percentage points. These are physical execution results, but 0.89 is not an all-stages-completed episode rate. e-ide-protocol |
| Six-task zero-shot OOD manipulation Same downstream tasks with unseen backgrounds, spatially disjoint object positions, or substitute objects; no OOD fine-tuning. | 0.82 overall; background 0.81, position 0.80, objects 0.85; reported relative drop from ID 7.9%. Table 3 Global Avg of task/sub-stage scores across OOD settings | Wan+AnyPos 0.53, pi0.5 0.49, Cosmos-Policy 0.44. The headline 29 and 33 margins are percentage-point gains over the strongest listed WAM and VLA OOD averages. This tests controlled shifts in familiar tasks, not arbitrary new skills. e-oode-ood-settinge-protocol |
| Put Flowers in Vase: final insertion ID dual-arm task, sequential picking, handover and insertion. | 0.70; preceding stages 0.95 and 0.90. Final-stage success, Table 6 S3 | pi0.5 0.40; Cosmos-Policy 0.35; Wan+AnyPos 0.15. The 0.85 headline flower score averages stages; the harder final insertion succeeds less often. e-stagese-id |
| Two-task component and routing ablation Average over Pick Fruit to Plate and Put Flowers in Vase, position OOD; Figure 5. | Full model 90; without reactive expert 14; averaging 44; keyframe averaging 59. Success rate (%) | Full model exceeds averaging by 46 percentage points and keyframe averaging by 31. Routing and the reactive branch matter in this setting. The chart's without-predictive position score is 73, conflicting with 56 in the prose; that disputed value is not treated as resolved. e-ablation |
| Put Flowers in Vase: denoising-step ablation Appendix F.3; world-model denoising budget varied. | Five steps: 85%, 4 Hz. Reported task success (%) and inference frequency (Hz) | Three: 80%, 4 Hz; ten: 85%, 3.6 Hz; fifty: 87%, 3 Hz. Five steps is the authors' speed/quality compromise. The table does not specify an ID/OOD condition, repeated-run uncertainty or inference hardware. e-speed |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Routing helps on the two-task ablation, but a chart–text conflict remains unresolved. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend consistently: blue is ID, yellow background, pink position and peach objects. Panel (a) removes each expert; panel (b) compares the full model with unconditional averaging and interaction-stage averaging; panel (c) removes video latent conditioning, denoted by the '-vid' suffix. That suffix does not mean removing the reactive expert's predicted RGB frames. Compare like-colored bars within the relevant intervention, and remember these experiments average only fruit placement and flower placement, unlike the six-task results. The labeled bars are readable, but several claims depend on whether one follows the chart or its surrounding prose. e-ablation
What it supports. For position OOD, full HarmoWAM reaches 90%, removal of the reactive expert gives 14%, averaging gives 44%, and keyframe averaging gives 59%. The two routing gaps are therefore 46 and 31 percentage points. This supports the importance of expert selection and the reactive pathway on this particular two-task distribution shift.
Where the evidence stops. The without-predictive position bar reads 73%, whereas Section 4.4 says 56%; 56% is the plotted background value. The crop preserves the conflict. No error bars are supplied, and expert removal changes capacity and available pathways as well as the proposed mechanism.
7. Analysis & limitations
7.1 What the evidence leaves open
Fixed video horizon limits temporal adaptation, and pixel-level future generation adds overhead. Adaptive horizons and latent-only prediction are proposed future work. e-limitations
Failures include tilted stacking from placement offsets, vase-rim misalignment after grasp variation, and zipper slippage. The authors attribute the last failure partly to gripper contact mechanics. e-failures
Comparisons share downstream demonstrations but not all inputs or pretraining: Wan+AnyPos uses only a third-person view, while others use multiple views. VPP and QwenVLA-OFT omit dual-arm results. Attention images and expert-removal ablations do not independently prove learned physical understanding. e-protocole-ide-ablatione-attention
7.2 Questions for discussion
- Does routing still help with camera inputs and pretraining held equal across expert baselines?
- How sensitive is execution to the interaction-label window and height thresholds?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the robot-adapted Wan checkpoint, demonstration data, FR3 arms with UMI grippers, synchronized cameras and stage annotations. Appendix A distinguishes single-arm front/top/wrist views from dual-arm global/two-wrist views, clarifying the compressed main-text camera description. e-hardwaree-pretraininge-protocol
Unspecified items include closed-source pretraining volume, optimizer and learning-rate schedules, training duration, numeric task height thresholds, exact multi-view fusion, action denoising schedule and execution timing. The architecture is called end-to-end, but the documented action-training stage freezes the world model. e-pretraininge-traininge-gate-labelse-worlde-speed
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the learned switch chooses useful contact boundaries
Reader-proposed check, not performed: freeze one trained world model and both action experts, then compare the learned gate with fixed-expert policies, output averaging and manually annotated transit/interaction routing on matched ID and position-OOD fruit/flower episodes. Keep camera inputs, action timing, initial states and trial budgets identical. Record transit success, final-stage success, switch times and uncertainty across repeats. Separately vary the training label window around the source's twenty-frame setting while keeping the experts fixed. If manual routing helps but learned routing does not, gate timing or labels are the bottleneck; if averaging matches both, stage selection has not explained the claimed gain. e-gatee-gate-labelse-ablatione-protocole-stages
Check 2: Separate useful future latents from extra decoder capacity
Reader-proposed check, not performed: keep the world model, generated frames and routing fixed, and train matched reactive decoders with true future latents, temporally shuffled latents, or a parameter-matched visual-only projection. Use identical demonstrations, optimizer budgets and random-seed repetitions. Evaluate transit and final contact separately under the source's disjoint-position and object-substitution settings. A reproducible advantage for correctly aligned latents over both controls would support their temporal contribution; parity with shuffled features would weaken that interpretation. Report complete-episode and stage-average scores separately, and distinguish twelve-action throughput from observation-to-command latency. e-reactivee-traininge-ablatione-ood-settinge-protocole-speed
8.3 Reading coverage
Visual audit: Read all eight supplied text chunks, covering all 25 pages including references and appendices. Visually inspected the title/author/version page, Figures 1–11, Tables 1–9, the method and training equations, camera setup, dataset counts, gate-label construction, success criteria, denoising tradeoff, and failure/limitation pages. All six final crops were viewed; the ID table crop was tightened and viewed again. Table 1 retains its essential near-target-initialization qualification. Figure 2's threshold and branch directions agree with Section 3.4; its fusion symbol is interpreted using Section 3.3's token-concatenation definition. Figure 5's without-predictive position bar conflicts with the adjacent prose, as disclosed. Pages 10–14 were read as text and not visually rendered; no retained claim depends on their visual layout. Separate supplementary videos and code were not supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block, abstract and version stamp (p. 1)
- 1 Introduction (pp. 1–3)
- 2 Related Work (p. 3)
- 3.1–3.5: formulation, motivation, framework, gating and training (pp. 3–6)
- 4.1–4.4: setup, ID/OOD experiments and ablations (pp. 7–9)
- 5 Conclusion (pp. 9–10)
- References (pp. 11–14)
- Appendices A–C: hardware, demonstrations, baselines, training and OOD construction (pp. 15–17)
- Appendices D–E: inverse dynamics, motivation experiments, gating labels and visualizations (pp. 17–20)
- Appendix F: stage-wise results and denoising ablation (pp. 20–23)
- Appendices G–I: failures, broader impact and limitations (pp. 23–25)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The inspected title and eleven authors match the catalog. The artifact is arXiv:2605.10942v1 [cs.RO], dated 11 May 2026; no other revision or edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF pages and all final crops.
- Separate supplemental material availability has not been fully verified.
- No separate videos, code, checkpoints or datasets were inspected, and no experiments were reproduced.
- The catalog affiliation string contains a trailing '0.2cm' artifact absent from the inspected author block; verified metadata follows the title page.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block and arXiv margin stamp
Exact title and all eleven authors match the catalog; the inspected artifact is arXiv:2605.10942v1, 11 May 2026. Affiliations are Peking University, Simplexity Robotics and The Chinese University of Hong Kong, with the university laboratory and school named.
Go to primary source ↓e-actionPDF p. 3, Section 3.1; p. 17, Appendix D.1
Policy conditions on instruction and image; actions comprise relative position, Euler rotation and gripper state, concatenated for two arms. IDM converts video predictions into actions.
Go to primary source ↓e-motivationPDF p. 4, Table 1 and Section 3.2; p. 18, Appendix D.2
Ten trials per task/domain/phase compare Wan plus AnyPos with a Wan-conditioned Action DiT. Imagine-then-Execute transit is 10/10 throughout. OOD joint-model interaction entries use near-target initialization, as indicated by asterisks.
Go to primary source ↓e-worldPDF pp. 4–5, Section 3.3 and Figure 2
Wan2.2-TI2V-5B predicts 13 frames at 256 × 320 with five denoising steps; explicit video and latent features feed distinct action experts.
Go to primary source ↓e-predictivePDF pp. 4–5, Section 3.3, Overall Architecture and World Model to Predictive Expert
Predictive expert is a 1B, 28-block DiT using image, text and current-step world features, with world features shaped B × 80 × 3072 and cross-attention conditioning.
Go to primary source ↓e-reactivePDF p. 5, Figure 2 and World Model to Reactive Expert
DINOv2-base produces B × 1369 × 768 patch features. Future world latents are pooled in channels and concatenated with patch tokens before the Orientation Decoder.
Go to primary source ↓e-gatePDF p. 5, Figure 2; p. 6, Section 3.4 and Equation (1)
An MLP on current-image tokens predicts interaction probability. Positive interaction labels train BCE; score >0.5 selects predictive, score ≤0.5 selects reactive. Figure branch labels agree.
Go to primary source ↓e-trainingPDF p. 6, Section 3.5, Equations (2)–(4); p. 17, Appendix C.2, Equations (5)–(6)
Stage 1 fully fine-tunes the world model with conditional flow matching. Stage 2 freezes it and trains action experts/gate with loss weights 0.1 and 0.05; Smooth L1 beta is 0.1.
Go to primary source ↓e-pretrainingPDF pp. 16–17, Appendix C.2 and Table 4
Eight NVIDIA H20 GPUs are stated. Public robot pretraining counts are DROID 201,119, AgiBot 3,017 and RoboMIND 1,721,985, alongside unquantified closed-source data.
Go to primary source ↓e-hardwarePDF p. 15, Appendix A and Figure 6
FR3 arms use 3D-printed UMI grippers. Single-arm cameras are front D435, top D455 and wrist D435, at 640 × 480; dual-arm uses global and two wrist views. Observations also include proprioception.
Go to primary source ↓e-protocolPDF p. 7, Section 4.1; pp. 15–16, Appendix B and C.1
Six tasks have 100 SpaceMouse demonstrations each and 20 evaluation episodes per method/task. Stages have task-specific human success criteria. Baselines share downstream data/control spaces; Wan+AnyPos uses a third-person view, unlike multi-view policies. Only selected baselines are tested on dual-arm tasks.
Go to primary source ↓e-idPDF p. 7, Table 2 caption, Avg column and Section 4.2
Table 2 explicitly averages critical sub-stages. Global scores are 0.89 HarmoWAM, 0.78 Cosmos-Policy, 0.74 pi0.5 and 0.67 Wan+AnyPos. HarmoWAM has reported ±3% standard deviation over three repeats.
Go to primary source ↓e-ood-settingPDF p. 17, Appendix C.3; pp. 9 and 20, Figures 4 and 9
OOD backgrounds add 5–8 distractors and alter lighting; test object positions lie in spatially disjoint regions; substitute objects change appearance and geometry.
Go to primary source ↓e-oodPDF p. 8, Table 3, scenario Avg and Global Avg columns, Section 4.3
HarmoWAM reports background 0.81, position 0.80, objects 0.85 and global 0.82, with 7.9% relative drop from ID. Global baselines: Wan+AnyPos 0.53, pi0.5 0.49, Cosmos-Policy 0.44.
Go to primary source ↓e-stagesPDF p. 16, Appendix B, Dual-Arm Tasks; p. 21, Table 6
Flower stages are pick/handover/insertion, with HarmoWAM 0.95/0.90/0.70; final-stage baselines are 0.40 pi0.5, 0.35 Cosmos-Policy and 0.15 Wan+AnyPos. Bag stages end at 0.80 for HarmoWAM.
Go to primary source ↓e-ablationPDF p. 9, Figure 5 panels (a)–(c), legend and Section 4.4
Two-task ablation plots full HarmoWAM ID/background/position/objects as 93/87/90/90. Position bars are 14 without reactive, 44 averaging, 59 key averaging. Without-predictive position is 73 in the chart but 56 in prose; 56 is the chart's background bar. The -vid suffix removes latent conditioning, not predicted pixels.
Go to primary source ↓e-attentionPDF p. 7, Figure 3; p. 6, last paragraph of Section 3.4
Last-layer attention visualizations contrast reactive attention to gripper/surroundings with predictive attention to manipulated objects; the evidence is qualitative.
Go to primary source ↓e-gate-labelsPDF pp. 19–20, Appendix D.3
Gripper changes and task-specific height thresholds create keyframes; 20 frames on either side are labeled interaction. Either arm triggers dual-arm labels. Offline held-out accuracy is 96.95% on 1,637 test frame pairs.
Go to primary source ↓e-speedPDF pp. 2–3, Introduction; p. 23, Appendix F.3 and Table 9; p. 24, Figure 10
Introduction states 48 Hz action generation and chunk size 12. Flower-task denoising rows give steps/success/frequency: 3/80/4, 5/85/4, 10/85/3.6, 50/87/3. Figure 10 compares generated-video appearance.
Go to primary source ↓e-failuresPDF pp. 23–24, Appendix G and Figure 11
Authors discuss tilted stacking, insertion misalignment after grasp variation, and zipper grasp slip, with selected failure images.
Go to primary source ↓e-limitationsPDF p. 25, Appendix I
Fixed future horizon constrains adaptation; pixel-level generation adds overhead. Adaptive horizon and latent-level prediction remain future work.
Go to primary source ↓8.5 Primary sources
HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models ↗
PDF · 11,975 extracted words
Source fingerprint
b2a4b26d4cf8f6b7583157356f7ca0c1b5341a9c2dbd2e4023eb383529080974