Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
1. Paper overview
In one sentence: Zero-WAM learns to translate a human video prompt into robot futures and inverse-dynamics actions, gaining unseen-task transfer through synthetic paired data and auxiliary future prediction while retaining substantial execution failures. e02e04e06e07e08e11e12e13e15e16e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Language may omit manipulation order, intermediate states, and spatial constraints. A human video supplies these cues but lacks robot actions. Training must learn semantic correspondence across embodiments while preventing the policy from ignoring its prompt and extrapolating familiar robot history. e02e07e08 |
| Core mechanism | |
| A key reported result | Seven unseen RoboTwin 2.0 tasks: 46.95 ± 0.72 Macro-average task success, mean ± standard deviation (%). Clean setting; 43 seen/7 held-out tasks; reported 100 closed-loop rollouts per task per seed, three seeds. LingBot-VA: 17.45 ± 1.40; WAN-Action: 10.98 ± 1.07; gains of 29.50 and 35.97 percentage points. Zero-WAM leads all seven rows, but success ranges from 9.00 ± 2.16 for three-block stacking to 84.87 ± 0.18 for empty-cup placement. Training data and instruction modality differ across methods. e11e12 |
| Reading caution | Source description Experiments focus on stationary tabletop manipulation; mobile, dynamic, unstructured, and substantially longer-horizon settings remain future work. e18 |
Core contributions
Figure 1. A demonstrated task becomes a predicted robot future and then an executable action. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom: text passes through a language model, while the human demonstration supplies a visual task instruction. Both connect to the central task condition. Follow the observation into the video model, then the predicted future toward the action model. The data groups at left explain how this mapping is learned; the pictures at right identify the evaluation families. The method text sharpens this overview: the action Transformer does not directly attend to the human video. Instead, it conditions on robot history and the next robot video, so the visual prediction carries task information across the human–robot boundary. e02e06e07e08e10
What it supports. The future-video branch participates in deployed control: the policy first generates a robot video chunk and then decodes its aligned action chunk. This ordering supports the paper's inverse-dynamics description. Separate video and action parameters make the recorded Dual-system classification consistent with the architecture even though they form one policy.
Where the evidence stops. This overview does not depict the full attention mask or IFP modules. Training uses ground-truth next video for action prediction; deployment uses generated video, leaving an error-propagation boundary that the diagram alone cannot quantify.
2. Motivation
2.1 The problem and the proposed response
Language may omit manipulation order, intermediate states, and spatial constraints. A human video supplies these cues but lacks robot actions. Training must learn semantic correspondence across embodiments while preventing the policy from ignoring its prompt and extrapolating familiar robot history. e02e07e08
2.2 What this reading follows
A demonstration can specify which object moves where, in what order, without describing every motion in words. Zero-WAM turns that visual specification into a robot-domain future before decoding an action. Its central training problem is making the policy actually use the demonstration: familiar robot history can otherwise predict the next step well enough. Read the figures as a chain from paired-data construction to video-mediated control, then to experiments that separate prompt conditioning, data curation, and auxiliary prediction. The headline simulation gain is substantial, but the real-robot tests concern held-out configurations after adaptation, and the ablations do not isolate every competing explanation. e02e04e06e07e08e11e12e13e15e16e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system × IDM classification is supported by separate video/action Transformer parameters and video-first action decoding explicitly described as inverse dynamics. Shared attention and one overall policy do not erase that separation. IFP is auxiliary training, not an inference-time planner. Generalization/alignment and history/prefix-memory tags are supported; long-horizon capability remains weak in the reported stacking result. e06e07e08e10e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use the demonstration to specify outcomes across embodiments
Consider the three-object evaluation: a human video specifies which objects move and in which order, while the robot must execute with its own arms and scene geometry. The model is not given human joint trajectories as robot commands. During training, each human video is paired with a robot trajectory carrying executable action labels. Both videos enter a shared VAE latent space, but a positional offset distinguishes the human prefix from robot observations. The main video branch can attend to that prefix and predict a robot-domain continuation. The action branch then decodes from robot history and that continuation. Reader interpretation: this division assigns task correspondence to video prediction and embodiment-specific motion to inverse dynamics. Successful execution tests the complete chain; it does not separately prove that either intermediate representation is correct. e05e06e07e13
Figure 2. Robot trajectories supply action labels; generated human videos supply varied task instructions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top two rows as the provenance of the paired examples: public robot datasets, in-house embodiments, RoboTwin, and the real-world evaluation embodiment. Then follow the lower loop clockwise from task sampling. A VLM extracts the intended object-state changes, an image editor converts the first robot frame into a human scene, and a video generator animates that scene using the task description. Quality scoring precedes pairing. The arrows return to the human dataset because each accepted video is linked to its original robot trajectory. The robot actions remain the supervision; the synthetic human video is the task prompt. e03e04e05e07
What it supports. HumanGen reports 74.2K pairs spanning 8.6K task categories, with over 45 robot embodiments. This construction expands the instruction interface while preserving executable labels from robot data. Deliberate variations in viewpoint, objects, and background make the intended correspondence semantic rather than a requirement for matching pixels or human joint motions.
Where the evidence stops. The scoring boxes illustrate stricter semantic than physics acceptance, but provide no full rubric or acceptance statistics. The examples establish the pipeline's design, not independent validation of every generated video's physical plausibility or the accessibility of the in-house data.
5.2 Make longer futures supervise the representation used at deployment
The authors identify a shortcut in teacher-forced training: recent robot history can often explain the next chunk without consulting the demonstration. IFP raises the prediction horizon while routing supervision through the current video representation. Intermediate main-branch features are concatenated and projected into a fused feature, which conditions four auxiliary future predictors. With the paper's stride of two, their targets are chunks i+2, i+4, i+6, and i+8 when the main branch predicts i+1. Crucially, the auxiliary predictors cannot directly read the human prompt. The intended effect is to make the main branch carry task information needed for those futures. At deployment, all IFP modules disappear; only the trained video/action policy remains. The success ablation supports the training objective's usefulness without directly observing whether the proposed shortcut has been eliminated. e08e09e10e16
5.3 Locate exactly where the task becomes unseen
The evaluation has several boundaries that should remain separate. In simulation, 43 RoboTwin tasks supply robot demonstrations for post-training and seven are held out. Their test prompts are generated human videos constructed from corresponding robot trajectories, so the policy receives informative task demonstrations even though those tasks are excluded from robot training. In the real world, adaptation uses a small seen-configuration dataset on the evaluation embodiment, then tests held-out object/container combinations, manipulation sequences, or hole assignments. No parameter update is required for the new test configuration. Reader interpretation: this is a meaningful form of transfer with context, but it does not establish general deployment without embodiment data or equally strong transfer from arbitrary natural videos. The low stacking and insertion results further constrain any claim of open-ended reliability. e05e10e11e12e13e18
5.4 Training and inference
During training
Initialize from Wan-2.2-TI2V-5B; the video branch has 30 layers and width 3072, with action width 3072. Use video/action flow matching plus IFP on human-conditioned samples. Pretraining samples Task-diverse VA:HumanGen at 1:5, uses AdamW with peak learning rate 10⁻⁴ and weight decay 0.01, and costs 15,360 GPU-hours. e09
Training chunk size varies from 1–4. Human-video dropout is 0.1; ICL language dropout is 0.4 versus 0.1 for non-ICL samples. RoboTwin post-training uses 64 GPUs for 4,000 steps and Task-diverse VA:HumanGen:RoboTwin sampling of 2:10:3, with up to 160K packed tokens per GPU. e09e11
During inference
Remove IFP and cache the human-video prefix once. Generate the next robot video, decode its action chunk, and execute within the evaluated closed-loop policy. Implementation uses chunk size 2; video/ICL guidance is 5 and action guidance 1.0. Evaluated ICL mode disables language. Novel-task deployment requires no parameter update, but follows pretraining and seen-task embodiment adaptation. e07e10e11e13
5.5 Implementation flow
- Construct task-grounded prompts
Sample robot trajectories by action–object task. A VLM parses object-state changes; an image editor converts the first frame into a human scene; a video generator synthesizes the manipulation. A VLM screens semantics and physics before pairing the human video with the original robot actions. Viewpoint, scene, and object variation encourage semantic alignment. e03e04e05
- Encode and separate modalities
Wan-2.2 VAE encodes both human and robot videos; T5 encodes language. Human tokens precede robot history. A height-axis RoPE offset of 32 distinguishes human from robot latents. Video and action Transformers have separate projections, feed-forward networks, and output heads, interacting through shared attention. e06e07e09
- Predict video, then decode actions
The video branch attends to the human prompt and robot history. The action branch attends to robot history and the next robot video, without direct human-video attention. Teacher forcing supplies the ground-truth next video during training; generated video replaces it at inference. e06e07
- Make the current representation carry future intent
IFP fuses intermediate video-layer features through concatenation and an MLP. Four single-layer auxiliary Transformers, initialized from the last video layer, denoise strided future chunks in parallel. They receive the fused features, robot history, and language, but no direct human-video input. The authors argue this discourages prompt-ignoring shortcuts. e08e09
6. Experiments & results
Zero-WAM uses a human demonstration video to specify an unseen manipulation task, predicts the corresponding robot future, and decodes executable actions through inverse dynamics. Synthetic human–robot pairs and a training-only future-prediction objective support this interface. It reports 46.95% success on seven held-out RoboTwin tasks; real-robot evaluations test unseen configurations after embodiment adaptation. The evidence supports tabletop transfer under these protocols, with limited long-horizon reliability.
6.1 Read the original evidence
Table 2. Zero-WAM improves every reported held-out simulation task, with uneven absolute reliability. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a distinct task excluded from the 43-task RoboTwin post-training set. Values are success percentages, with the plus/minus term reported as standard deviation across three evaluation seeds; it is not a confidence interval. The final row is the task macro average. WAN-Action adapts the generic Wan prior, LingBot-VA adds its existing robotic pretraining, and Zero-WAM includes task-balanced data and human-video learning. Inspect the individual rows before the average: empty-cup placement is comparatively successful, while stacking three blocks remains difficult. The protocol specifies the clean simulation setting and 100 closed-loop rollouts per task per seed. e11e12
What it supports. The reported average is 46.95 ± 0.72% for Zero-WAM, versus 17.45 ± 1.40% for LingBot-VA and 10.98 ± 1.07% for WAN-Action. The gain over LingBot-VA is 29.50 percentage points. Nevertheless, the hardest task, three-block stacking, succeeds in only 9.00 ± 2.16% of reported trials.
Where the evidence stops. Training data and instruction modality differ across columns. Two tasks use modified success criteria. Some decimal means also remain unexplained by the stated trial counts; preserve the printed numbers pending raw counts and aggregation details.
Table 3. Real-robot transfer is evaluated after small-scale adaptation on seen configurations. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The middle columns matter as much as the final scores: the robot receives seen-task demonstrations to adapt to the bimanual Franka embodiment before evaluation. Placement uses 120 demonstrations from 30 combinations; sequential manipulation uses 96 from 16; insertion uses 36, with its combination count left as a dash. Test configurations are held out from those robot demonstrations. Zero-WAM receives a human video alone, whereas LingBot-VA receives detailed text. The table reports 30 real-robot trials for each task family. Read its percentages as executed-task outcomes, separately from the qualitative image sequences shown elsewhere in the paper. e13e14
What it supports. Zero-WAM reports 53.3% placement success, 33.3% three-object sequence success, and 16.7% two-leg insertion success. LingBot-VA reports 43.3%, 10.0%, and 0.0%, respectively. The pattern supports transfer of object choice, order, and target-hole information through human video, while showing that these configurations remain far from reliably solved.
Where the evidence stops. The table gives no uncertainty intervals or detailed failure taxonomy. The instruction modalities differ, and embodiment adaptation uses robot demonstrations. Qualitative insertion examples, including human–robot collaboration, should not be counted as additional autonomous trials.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Seven unseen RoboTwin 2.0 tasks Clean setting; 43 seen/7 held-out tasks; reported 100 closed-loop rollouts per task per seed, three seeds. | 46.95 ± 0.72 Macro-average task success, mean ± standard deviation (%) | LingBot-VA: 17.45 ± 1.40; WAN-Action: 10.98 ± 1.07; gains of 29.50 and 35.97 percentage points. Zero-WAM leads all seven rows, but success ranges from 9.00 ± 2.16 for three-block stacking to 84.87 ± 0.18 for empty-cup placement. Training data and instruction modality differ across methods. e11e12 |
| Object-to-container placement Bimanual Franka; unseen object/container configurations; 120 adaptation demonstrations from 30 combinations; 30 evaluation trials. | 53.3 Task success (%) | Language-conditioned LingBot-VA: 43.3; Zero-WAM uses human video alone. At least one object or container is held out per test configuration; no uncertainty interval is reported. e13 |
| Three-object sequential manipulation Franka; unseen objects/containers and video-specified order; 96 demonstrations from 16 combinations; 30 evaluation trials. | 33.3 Task success (%) | LingBot-VA: 10.0. Evidence of ordering transfer after adaptation, with most trials still unsuccessful. e13 |
| Two-table-leg insertion Franka; unseen color-to-hole configurations; 36 adaptation demonstrations; 30 evaluation trials. | 16.7 Task success (%) | LingBot-VA: 0.0. Both specified insertions must succeed. Training-combination count is a dash in Table 3; qualitative unseen-insertion examples do not add a measured benchmark. e13e14 |
| IFP ablation on seven unseen RoboTwin tasks Same 43/7 task split; full model versus removal of IFP during training. | 46.95 with IFP; 28.55 without Macro-average task success (%) | 18.40 percentage-point difference; stacking rises from 0.00 to 9.00. Supports auxiliary training utility; does not directly measure whether prompt information causes the gain. e16 |
| Human-video conditioning without Zero-WAM pretraining Wan-initialized variants trained on 43 seen RoboTwin tasks; evaluated on the seven held-out tasks. | 36.36 for Zero-WAM w/o pretrain Macro-average task success (%) | WAN-Action: 10.98; LingBot-VA: 17.45, retaining its robotic pretraining. The same-Wan comparison supports prompt usefulness. All three variants fail stacking; this is not an isolated test-time prompt swap. e15 |
| Text-only task-balanced pretraining HumanGen remains in the training mixture with human-video conditioning masked; same RoboTwin task split. | 39.44 Macro-average task success (%) | LingBot-VA: 17.45; difference 21.99 percentage points. Suggests data-curation value, but the corpora and training recipes are not fully matched, so sampling alone is not isolated. e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Human-video training helps even before the full Zero-WAM pretraining stage. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to distinguish the green Zero-WAM variant from the blue WAN-Action and beige LingBot-VA baselines. 'W/o pretrain' means without the additional large-scale Zero-WAM pretraining: the model still starts from the pretrained Wan video backbone. WAN-Action uses that same initialization and language instructions on the 43 seen RoboTwin tasks. LingBot-VA retains its robotic pretraining, so its bar has a different training history. Each group shows a held-out task, followed by the macro average. The labels are rounded integer percentages; the surrounding source text provides the more precise average values used in the report. e11e15
What it supports. The source gives 36.36% mean success for Zero-WAM w/o pretrain versus 10.98% for WAN-Action. This supports learning from human-video task specifications beyond the text-only Wan baseline. The green bars do not dominate every individual baseline row, and all three compared variants remain at zero on three-block stacking.
Where the evidence stops. This is a comparison of trained variants, not the same checkpoint with only its test-time prompt replaced. It therefore cannot attribute the entire gain to information supplied at inference, independent of the training changes.
Figure 6. Data curation and auxiliary future prediction contribute under different comparisons. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel as a text-only comparison and the right panel as a comparison within the human-video model. The left keeps HumanGen robot trajectories but masks their human-video condition, testing whether the revised pretraining recipe helps without that task signal. The right removes IFP during training. On both plots, inspect the numerical tick labels on each spoke: stacking tops out at 10%, empty-cup placement at 90%, and the average at 50%. Polygon area is therefore not a direct overall-success measure. Use the same spoke to compare variants, and use the source's prose for exact averages. e08e16e17
What it supports. The right-panel comparison reports 46.95% with IFP against 28.55% without, an 18.40-point difference; stacking changes from zero to 9.00%. The left-panel text-only variant reports 39.44% against LingBot-VA's 17.45%. These findings support both auxiliary future supervision and the broader data-curation recipe, under their respective experimental conditions.
Where the evidence stops. The left comparison does not match the entire corpus and training recipe, so it cannot isolate sampling alone. The right comparison measures success rather than prompt dependence directly; it leaves the proposed shortcut-suppression explanation open to targeted testing.
7. Analysis & limitations
7.1 What the evidence leaves open
Experiments focus on stationary tabletop manipulation; mobile, dynamic, unstructured, and substantially longer-horizon settings remain future work. e18
Simulation prompts are generated from held-out-task robot trajectories. This tests unseen-task execution with informative demonstrations, not unconstrained human-video transfer. Two task success criteria are modified identically for all methods but not specified precisely. e05e11
Reported trial counts do not explain some Table 2 decimal means, such as 69.14% from a stated three sets of 100 trials. Preserve the printed values; raw counts and aggregation details are needed to reconcile them. e11e12
7.2 Questions for discussion
- Does IFP increase reliance on the demonstrated order when prompts conflict with familiar robot history? [e08, e16]
- Does task-balanced sampling still help when corpus, tokens, optimizer, and conditioning are held fixed? [e03, e17]
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the public and in-house trajectories, generated pairs, task labels, filtering pipeline, Wan/VAE/T5 components, and exact modified RoboTwin tasks. Figure 2 illustrates semantic/physics acceptance examples, but full scoring rubrics, acceptance statistics, and task-specific sampling caps are absent. e03e04e05e11
The PDF omits IFP feature-layer selection and M, numerical global loss weights, action encoding/normalization, denoising-step counts, control frequency, frozen-module policy, and detailed real-robot optimization. These must be resolved before claiming faithful reproduction. e06e08e09e10e13
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether IFP makes the deployed policy depend on its human prompt
Reader-proposed check: train matched IFP and no-IFP variants with identical data, seeds, dropout, token budget, and optimizer settings. On the seven held-out RoboTwin tasks, repeat paired initial states under correct, task-mismatched, and temporally shuffled human prompts, keeping guidance settings fixed. Report per-task success and which demonstrated object or order the robot follows, using three seeds and 100 rollouts per task per condition. Record raw binary outcomes. If IFP's gain persists equally with irrelevant prompts, its benefit may arise from generic future-prediction regularization. A larger correct-prompt advantage, together with behavior tracking changed demonstrations, would support the proposed dependence on prompt semantics. e07e08e11e16
Check 2: Separate task balancing from corpus and evaluation changes
Reader-proposed check: use exactly the same available robot corpus in two text-only runs, masking human-video conditions in both. Change only trajectory-frequency sampling versus bounded task-level sampling; match initialization, total tokens, optimizer, and post-training mixture. Audit action–object task labels across the full corpus before applying the same 43/7 holdout. Run both original and paper-modified stamp-seal/stapler success criteria once those modifications are specified, retaining raw rollout counts and per-seed averages. A stable advantage for task balancing under both criteria would support the sampling mechanism. A disappearing gap would suggest that corpus composition, training exposure, or evaluation definitions explain part of the published comparison. e03e05e11e12e17
8.3 Reading coverage
Visual audit: Visually inspected the title, all twelve author credits, affiliations and v2 margin on page 1; Figures 1–6; Tables 1–3; and method/equation layouts on pages 7–9. All six final original-PDF crops were separately opened and checked for readable labels, complete legends, axes and table rows. Long captions and surrounding prose are excluded from the crops, with evaluation context retained in the explanations. Figure 6's unequal axis scales were explicitly checked. The full 21-page text, including references, was read; pages outside this list were not rendered for visual inspection. No separate supplement, external website, code, dataset, or moving-video playback was inspected.
PDF pages inspected for this edition: 1, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 2–3)
- 2 Data Curation, including 2.1–2.3 (pp. 3–6)
- 3 Zero-WAM: In-Context World Action Modeling, including 3.1–3.4 and Equations 1–20 (pp. 6–9)
- 4 Experiments, including 4.1–4.4 (pp. 9–14)
- 5 Conclusions and Discussions (pp. 14–15)
- 6 Related Works, including 6.1–6.3 (pp. 15–16)
- References 1–88 (pp. 16–21)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Version scope: the inspected title page identifies arXiv:2608.26103v2 [cs.RO], 27 Aug 2026. Its title and all twelve authors match the catalog after name-order normalization. The catalog does not specify a revision; no earlier version or revision history was supplied, so changes from v1 cannot be established.
- The title page supplies Robbyant, HKUST (GZ), and HKUST, whereas the catalog lists affiliations as not identified.
- Text extraction does not reconstruct figure images; this gap was addressed by inspecting original PDF pages and all six final crops. Figures 1–6 and Tables 1–3 were visually reviewed.
- Separate supplemental material availability has not been fully verified.
- The supplied PDF contains no appendix. Code, datasets, project-site material, and external references were not independently inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Observed title matches the supplied title; twelve authors are Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, and Yinghao Xu. Affiliations: Robbyant, HKUST (GZ), HKUST. The margin identifies arXiv:2608.26103v2 [cs.RO], 27 Aug 2026.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Figure 1, Section 1
The paper frames unseen-task manipulation as context-based specification, motivates human video for visual/temporal cues, and identifies paired-data scarcity and history-based shortcuts.
Go to primary source ↓e03PDF p. 3, Section 2.1
Task-diverse VA repartitions AgiBot, InternData-A1, OXE, RoboCOIN, and RoboMIND into action–object tasks; bounded per-task sampling yields over 6,000 tasks and about 400K trajectories per epoch. Numeric caps are not given.
Go to primary source ↓e04PDF pp. 4–5, Figure 2 and Section 2.2
The pipeline parses robot task/state changes with Gemini 3.1 Pro or Qwen3.6-Plus, edits the first frame with Nano Banana 2 or Qwen-Image-2.0, generates video with Wan 2.7 or Kling AI 3.0, and screens semantics/physics with a VLM. Figure 2 marks semantic score 5 accepted and 3/1 rejected; physics 5/3 accepted and 1 rejected, without a complete rubric.
Go to primary source ↓e05PDF pp. 5–6, Table 1 and Section 2.3
HumanGen reports 74.2K pairs, 8.6K tasks and over 45 embodiments. External pairs: 41,188/5,062 tasks; in-house: 30,247/3,522 tasks; simulation: 2,500 pairs/50 tasks, with 2,150 pairs/43 tasks for post-training; real-world: 252 pairs. Visual alignment varies across subsets.
Go to primary source ↓e06PDF pp. 6–7, Section 3.1, Equations 3–4 and Mixture-of-Transformers Design
Video prediction precedes inverse-dynamics action decoding. Separate video/action Transformers use modality-specific QKV projections, FFNs and heads, with shared attention over one sequence. Action tokens follow future-video tokens.
Go to primary source ↓e07PDF pp. 7–8, Section 3.2, Equations 5–10
Human video is prefix memory for video prediction, excluded from direct action attention. The action context uses ground-truth future video in training and generated video in inference. Shared VAE encoding is disambiguated by height-axis RoPE offsets.
Go to primary source ↓e08PDF p. 8, Section 3.3, Equations 11–14
IFP predicts K strided future video chunks from fused M-layer current-chunk features plus clean robot history and language. Modules copy a single video Transformer layer and are initialized from its last layer. No direct human-video conditioning is allowed; modules are removed at inference.
Go to primary source ↓e09PDF pp. 9–10, Section 3.4, Equations 15–20, and Section 4.1 implementation/training details
The source gives video/action flow objectives, ICL-only IFP supervision, Wan initialization, 30 video layers, width 3072 for both streams, RoPE offset 32, K=4, stride 2, weights (0.5,0.25,0.15,0.15), AdamW settings, mixture/dropout/chunk settings, 15,360 GPU-hours and 160K-token packing. Global loss weights and feature-layer choices are not numerically specified.
Go to primary source ↓e10PDF pp. 9–10, Sections 3.4 and 4.1, Inference paragraphs
Inference removes IFP, caches human-video tokens, generates video then actions, and supports language-only or ICL modes. Implementation disables language in ICL mode, uses video/ICL CFG 5, action CFG 1.0 and chunk size 2.
Go to primary source ↓e11PDF p. 10, Section 4.2 and footnote 1
RoboTwin clean evaluation uses 43 seen/7 held-out tasks, 64-GPU 4,000-step post-training, a 2:10:3 mixture and three evaluation seeds with 100 rollouts per task per seed. Human-video variants use generated prompts. Stamp-seal and stapler success criteria are modified equally for every method; exact changes are not supplied.
Go to primary source ↓e12PDF p. 11, Table 2, all task rows and Average row
Zero-WAM task rates are 24.67±2.05, 47.00±4.55, 59.00±2.83, 69.14±2.93, 35.00±3.74, 84.87±0.18 and 9.00±2.16 in table order. Macro averages are WAN-Action 10.98±1.07, LingBot-VA 17.45±1.40 and Zero-WAM 46.95±0.72. All values are percentages.
Go to primary source ↓e13PDF p. 11, Table 3 and Section 4.3; p. 13, sequential manipulation and insertion paragraphs
Real bimanual Franka evaluation reports 30 trials per task family and human-only conditioning for Zero-WAM versus text for LingBot-VA. Success pairs are 53.3/43.3, 33.3/10.0 and 16.7/0.0. Adaptation uses 120 demonstrations/30 combinations, 96/16 and 36/unspecified respectively, while evaluated configurations are held out.
Go to primary source ↓e14PDF p. 12, Figure 4; p. 13, two-table-leg insertion paragraph
Qualitative sequences show placement, ordered three-object manipulation, two-leg insertion with human–robot collaboration, and unseen lightbulb insertion. The figure does not report additional quantitative trial counts.
Go to primary source ↓e15PDF pp. 13–14, Figure 5 and Section 4.4, Effect of in-context human video prompts
The text gives 36.36% for Zero-WAM w/o pretrain, 10.98% WAN-Action and 17.45% LingBot-VA. The plot rounds values to integers. Zero-WAM/WAN-Action start from Wan; LingBot-VA retains robotic pretraining. All three have zero stacking success.
Go to primary source ↓e16PDF p. 14, Figure 6 right and Section 4.4, Effect of in-context future chunk prediction
The reported seven-task mean rises from 28.55% without IFP to 46.95% with IFP; stacking rises from 0.00% to 9.00%. Radar axes have task-specific scales.
Go to primary source ↓e17PDF p. 14, Figure 6 left and Section 4.4, Effect of task-balanced robotic data
Text-only Zero-WAM masks human-video conditions while retaining ICL examples as robot video/action data. It scores 39.44% against LingBot-VA's 17.45%; the text says most task-sampled data are covered by LingBot-VA's corpus, not that corpora and training are identical.
Go to primary source ↓e18PDF p. 15, Section 5, Discussions
The authors characterize experiments as primarily stationary tabletop manipulation and identify mobile manipulation, dynamic/unstructured environments and substantially longer horizons as future directions.
Go to primary source ↓8.5 Primary sources
Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization ↗
PDF · 11,303 extracted words
Source fingerprint
4f06ad8631437e2723e470c539327c9298477790779b88164f0a41f67b4d7ab7