World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis
1. Paper overview
In one sentence: WLA-0 trains language-guided dynamics features with a World Expert, allowing efficient action generation without future-image synthesis while leaving optional imagined-future selection and broad video transfer less fully established. e03e04e10e11e12e18e19
| At a glance | What to know |
|---|---|
| Research problem | Source description Visual next-state prediction alone offers a weak interface for semantic progress tracking. WLA represents the next state using both textual intention and physical dynamics, seeking language-guided long-horizon control without paying for future-image generation at every control step. e02e03e04 |
| Core mechanism | |
| A key reported result | RMBench memory-dependent manipulation: 56.5 Average success rate (%). M(n) subset, four tasks; one model/task trained 30k steps, batch 448; textual history used at inference. Mem-0: 28.5; Fast-WAM: 13.3; without language loss: 17.3 in Table 2 (17.25 in prose). Strong evidence for subtask supervision in this system; removing its loss does not separately isolate language reasoning from memory quality. WLA-0 does not win Blocks Ranking Try. e10 |
| Reading caution |
Core contributions
Figure 2. One shared dynamics representation supports two experts, while future-image generation remains optional during deployment. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom of panel (a): the vision encoder supplies historical and current observations, followed by memory, instruction and meta-queries. The backbone first predicts textual intention; the meta-query outputs form the physical-dynamics representation h_t. Follow its upward arrow into the Action Expert, which also receives robot state q_t, and its rightward arrow into the World Expert, which also receives the current observation. These are parallel conditioning routes, consistent with Eqs. 3.3–3.4: the action branch does not consume the generated image. The bracket marks the removable world branch. Panel (b) depicts optional future scoring; the highlighted 0.78 is an illustrative value, not a benchmark score. e03e04e05e06
What it supports. The World Expert can influence actions through training gradients without being called during efficient control. This makes world supervision a way to shape a useful action representation. The architecture supports an optional, more expensive imagination mode, but its default inference path only needs the backbone and Action Expert.
Where the evidence stops. Panel (b) omits the action-to-future pairing mechanism. Section 3.2 asserts candidate-specific futures, whereas Eq. 3.3 has no sampled-action input. Their precise coupling remains unspecified; the diagram alone does not resolve it.
2. Motivation
2.1 The problem and the proposed response
Visual next-state prediction alone offers a weak interface for semantic progress tracking. WLA represents the next state using both textual intention and physical dynamics, seeking language-guided long-horizon control without paying for future-image generation at every control step. e02e03e04
2.2 What this reading follows
A robot may need to remember how many times it pressed a button while still deciding precisely how to move its arm. WLA-0 separates these demands inside an autoregressive backbone: textual subtasks describe intended progress, and compact meta-query features describe the transition needed for control. A World Expert teaches those features through future-image supervision, while an Action Expert converts them into robot commands. The key deployment choice is to remove image synthesis from the normal control loop. Read the results as several distinct tests: memory-dependent manipulation, efficient physical execution, and learning unseen simulated tasks from videos. The last test succeeds for robot videos but deteriorates on average for human videos. e03e04e10e11e12e18e19
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded WAM / Dual-system / Other mechanisms / Outside quadrants assessment is defensible: distinct World and Action Experts share AR-backbone dynamics features, while efficient inference has no explicit image-to-action stage. Joint training alone does not establish One Model. Latent prediction, textual memory and efficient inference fit the subcategories, but the JEPA wording must not imply a JEPA objective: this source explicitly chooses VAE features. e03e04e06e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Treat textual intention as an evolving record of progress
On Press Button, the visible scene can look nearly unchanged after several correct presses, so the action decision depends on what has already happened. WLA addresses this by assigning demonstration segments to textual subtasks and training the backbone to predict the subtask window covering the next action horizon. Previous subtask text is accumulated as memory and used with the instruction and observations for the next prediction. The appendix's example distinguishes successive left-button presses, successive middle-button presses, and the final confirmation. Reader interpretation: this text can compress a relevant history more directly than repeatedly describing the entire image. It remains a predicted record, however, rather than a verified execution log. The strong language-loss ablation motivates the interface, but cannot prove that its memory remains correct after an unexpected execution failure. e03e10e20
5.2 Let future-image supervision teach a representation that actions can reuse
After predicting the textual window, the backbone processes meta-queries to obtain compact dynamics features. The World Expert receives those features together with the current observation, while the Action Expert receives them with robot proprioception. Joint world and action losses therefore train a shared conditioning signal. The World Expert predicts future-frame VAE features, taking responsibility for detailed appearance; actions need not wait for those pixels during deployment. Reader interpretation: the world objective is useful if the shared features retain transitions relevant to control, even when the image decoder itself is unnecessary online. Table 1 supports modest average gains from this supervision. Appendix B.2 also reports 98.2% for one target frame versus 94.2% for four targets at a fixed training budget. That ablation favors a compact target in its setting, without proving that richer supervision is always harmful. e04e05e08e09e16
5.3 Separate fast reactive control from selecting among imagined futures
Efficient mode generates actions without calling the World Expert. TTS adds a different decision procedure: draw multiple action chunks, imagine their corresponding future states, score those states, and execute the chunk with the largest predicted value. Longer imagination feeds generated frames back into the model. The value target combines binary rollout success with a discount for distance from episode end; it is learned from the fine-tuned policy's rollouts. Reader interpretation: this can help only if imagined futures distinguish the actual consequences of the sampled actions and the value model ranks those consequences reliably. The reported LIBERO change from 98.6% to 98.9% is compatible with a benefit, but gives neither a latency tradeoff nor a complete action-to-future coupling specification. Figure 10's plausible prediction examples do not establish candidate-ranking accuracy. e04e06e09e22
5.4 Training and inference
During training
WLA-0 reports 3.4B total parameters: RynnBrain-2B backbone (2.1B), SANA World Expert including VAE (900M), and a 390M action head. It uses 64 meta-queries and 28 layers per expert. 'No embodied pretraining' does not mean random initialization of the VLM. e03e07e08
Joint optimization combines action/world flow matching and subtask cross-entropy, with α=0.1 and β=0.005. AdamW uses weight decay 10⁻⁸, clipping 1.0, cosine learning rate 5×10⁻⁵ to 5×10⁻⁶ and 1,000 warm-up steps, using DeepSpeed. Main action chunks have length 8 on LIBERO and 32 elsewhere. e05e07
During inference
Default inference removes the World Expert; CUDA Graph replay, Triton operator fusion and caching reduce the reported latency from approximately 116 ms to under 40 ms. RoboTwin uses 32 flow steps and absolute end-effector positions; real-world control uses 10 steps and joint angles. e06e14e15e17
TTS samples K candidates with different seeds, predicts corresponding futures and executes the highest-valued chunk. Predicted frames can feed subsequent imagination steps. The value model learns discounted success from fine-tuned-policy rollouts, using task instruction and predicted frame as inputs. e06
5.5 Implementation flow
- Predict the next subtask window
The pretrained VLM backbone receives images, instruction and memory. Training subtasks have temporal segments; the predicted contiguous window covers the upcoming action horizon. Recursively accumulated subtask text provides long-horizon context. e03
- Compress the transition
Appended meta-queries aggregate context through causal attention after subtask prediction. Their outputs condition a World Expert together with the current observation. It predicts the target frame's VAE features, leaving visual detail generation outside the compact dynamics representation; depth prediction is optional. e04
- Generate and execute actions
The Action Expert conditions on dynamics features and proprioception. Environment execution supplies new observations for receding-horizon control; a predicted image is an imagined state, not an executed transition. e03e04
6. Experiments & results
WLA-0 combines language-based subtask memory with a compact dynamics representation shared by separate World and Action Experts. Future-image supervision improves the action representation during training; efficient deployment omits image generation. Its strongest mechanism-specific evidence concerns memory-dependent manipulation, while simulated video transfer succeeds unevenly and human-video transfer fails on average.
6.1 Read the original evidence
Table 1. World supervision gives modest average gains in the main simulation benchmarks; test-time scaling adds a small LIBERO increment. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the RoboTwin Clean and Rand. columns separately, then the four LIBERO suites and their Avg. column. Entries are success percentages; bold and italic formatting identify the paper's leading results. The row below WLA-0 removes the world-model loss, so it is the most direct comparison for the auxiliary prediction mechanism. The +TTS row turns imagination back on; dashes mean no reported RoboTwin TTS evaluation. The pretraining column concerns embodied pretraining: the backbone is still initialized from a pretrained VLM. Appendix B.1 adds a material protocol qualification outside this crop: Lingbot-VA uses seen instructions, while the other RoboTwin methods use unseen instructions. e03e06e07e08e09e15e23
What it supports. World-loss removal changes RoboTwin Clean from 92.94% to 90.98% and LIBERO average from 98.6% to 97.9%. TTS raises LIBERO to 98.9% using six candidates and horizon two. These averages support benefits in the reported settings, while Fast-WAM's 91.78% remains above WLA-0's 90.02% on randomized RoboTwin.
Where the evidence stops. No uncertainty is reported for the small LIBERO differences. The TTS row's nominal 2B count does not account explicitly for the reactivated world/value computation. Figure 1's single RoboTwin headline lacks the table's separate setting labels.
Figure 3(d). Low inference latency translates into faster Stack Cup completion in the paper's synchronous physical-control setup. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend before comparing bar heights: lavender completion time belongs to the left axis in seconds, while orange inference latency belongs to the right axis in milliseconds. Compare like-colored bars across methods; the two colors have different scales. All three policies run on one RTX 5090 with action chunk 32 and ten flow-matching steps; Motus predicts only one image per call. Completion time runs from motion onset to task completion, averaged over ten successful rollouts. Inference latency averages calls in those rollouts. Synchronous execution makes the robot wait for inference before executing actions, connecting computational delay directly to the control schedule. e04e11e14e17
What it supports. WLA-0 reports 37.7 ms inference latency and 25.8 s completion time, compared with 190.1 ms and 29.0 s for π0.5, and 1529.8 ms and 61.8 s for Motus. The deployment advantage combines the removable World Expert with graph capture, fused operators and caching described in Appendix A.
Where the evidence stops. These timings condition on successful rollouts and synchronous execution. They do not measure a general TTS latency or show what happens under asynchronous control. The comparison also does not isolate architecture from engineering optimizations.
Table 3. Unseen-task robot videos improve average action success, but transfer depends strongly on the task and embodiment. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Every cell contains Clean / Rand. success percentages, not a confidence interval. Begin with Seen-Action, which learns from action labels on 45 seen tasks. The next column adds videos of those same tasks. The last two columns add video supervision for the five unseen tasks, using either Aloha-AgileX or ARX-X5. Those unseen-task videos carry no action annotations, but action labels from seen tasks remain in training. Appendix D specifies an unseen-video loss weight of 0.1 and seen/unseen sampling at 1:1. Compare both the averages and individual rows: the cross-embodiment model improves bottle picking and basket placement, yet barely solves card moving. e12e18e19
What it supports. Same-embodiment videos raise average success from 13.0/11.6 to 34.4/30.0; cross-embodiment videos reach 28.8/27.4. Adding only seen-task videos gives 11.8/12.6. Thus the useful signal comes from exposure to novel-task videos in this experiment, and successful transfer is broader than the qualitative hammer example alone.
Where the evidence stops. 'Unseen' means unseen with action supervision, not absent from training videos. These results use simulated robot videos. They establish neither universal embodiment transfer nor learning from arbitrary human footage; the human-video diagnostic supplies a direct boundary.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0 multi-task manipulation 50 tasks; 2,500 clean plus 25,000 randomized training trajectories; 100k steps, batch 256; 100 trials/task. Subtask prediction disabled. | 92.94 / 90.02 Success rate, Clean / Rand. (%) | Without world loss: 90.98 / 89.34; Fast-WAM: 91.88 / 91.78. World supervision helps the average, but Fast-WAM leads randomized scenes. Lingbot-VA uses seen instructions, whereas the others use unseen instructions. e08e15 |
| LIBERO multi-suite manipulation One model across four suites, 10 tasks/suite and 50 demonstrations/task; 100k steps, batch 256; 50 trials/task. | 98.6; TTS: 98.9 Average success rate (%) | Without world loss: 97.9; Lingbot-VA: 98.5. TTS uses six candidates and imagination horizon two; the 0.3 percentage-point increase has no reported uncertainty or matched compute comparison. e09e06 |
| RMBench memory-dependent manipulation M(n) subset, four tasks; one model/task trained 30k steps, batch 448; textual history used at inference. | 56.5 Average success rate (%) | Mem-0: 28.5; Fast-WAM: 13.3; without language loss: 17.3 in Table 2 (17.25 in prose). Strong evidence for subtask supervision in this system; removing its loss does not separately isolate language reasoning from memory quality. WLA-0 does not win Blocks Ranking Try. e10 |
| LIBERO future-frame supervision ablation Preliminary n=32 experiment; single target t+32 versus targets t+8,t+16,t+24,t+32; both 30k steps, batch 256. | Single-frame: 98.2; multi-frame: 94.2 Average success rate (%) | Four-percentage-point advantage for single-frame supervision. This fixed-training-budget result is distinct from the main length-8 LIBERO setup; it does not establish multi-frame inferiority after equal convergence. e16e07 |
| Stack Cup synchronous real-world control Piper dual-arm platform; single RTX 5090; 10 successful rollouts; 10 flow steps and chunk 32 for all models; Motus generates one frame/call. | 25.8 s; 37.7 ms Completion time (s); inference latency (ms), lower is better | π0.5: 29.0 s / 190.1 ms; Motus: 61.8 s / 1529.8 ms. Measured deployment advantage under synchronous control, conditional on successful rollouts; architecture and implementation optimizations both contribute. e11e14 |
| Real-world standard and OOD manipulation Four tasks; 60 demonstrations/task; 50k training steps, batch 256; 10 trials per task/setting. | 7.5 / 5.25 / 4.75 Mean successful trials out of 10, Standard / OOD Object / OOD Scenario | π0.5: 7 / 4.5 / 4.75; Motus: 6 / 3.75 / 4. The standard 75% headline does not summarize OOD performance. WLA-0 ties π0.5 on the OOD-scenario mean. e11e17 |
| Unseen-task transfer from simulated robot videos 45 seen action-supervised tasks and five unseen tasks; 50 clean/500 randomized trajectories per task; 50k steps, batch 256; unseen video weight 0.1, seen/unseen sampling 1:1. | Same embodiment: 34.4 / 30.0; cross embodiment: 28.8 / 27.4 Mean unseen-task success, Clean / Rand. (%) | Seen-Action: 13.0 / 11.6; Seen-Action+Video: 11.8 / 12.6. Novel-task videos help without unseen-task action labels. Seen tasks still provide action supervision; cross-embodiment transfer remains uneven. e12e18 |
| Unseen-task transfer from human egocentric videos 100 real human videos for each of the five unseen tasks, mixed with seen-task data; evaluated on simulated RoboTwin tasks. | 7.8 / 7.8 Mean unseen-task success, Clean / Rand. (%) | Seen-Action: 13.0 / 11.6. Transfer degrades on average, although Place Object Basket improves. The authors' domain-gap explanation is a conjecture, not an isolated cause. e18e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Removing subtask supervision sharply reduces average success, especially on covering blocks and counting button presses. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each method for task-specific success, then compare the final Average column. WLA-0 trains a separate model for each of these four M(n) tasks. The final row removes the language-based subtask prediction loss; it is not a separately reported experiment that merely empties memory at inference. Cover Blocks requires remembering hidden block identities and uncovering them in a requested order. Press Button requires counting repeated presses before confirmation. The appendix's task illustrations make these demands concrete. Now compare these columns with Blocks Ranking Try: WLA-0 leads the average and three tasks, but Fast-WAM leads that particular trial-and-error task. e03e10e20
What it supports. WLA-0 reaches 56.5% average success versus Mem-0's 28.5% and Fast-WAM's 13.3%. Without language loss, the average is 17.3% in the table; the text gives 17.25%. Cover Blocks falls from 84% to 18%, and Press Button from 74% to 1%, strongly motivating the subtask interface.
Where the evidence stops. The ablation changes learned supervision, so it cannot separately attribute gains to language reasoning, memory quality or their interaction. Rollout counts, repeated-seed uncertainty and an independent memory-only ablation are not specified in this evaluation section.
Table 6. Human-video transfer fails on average, despite an improvement on basket placement. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two values in each cell as Clean / Rand. percentages, using the same five unseen simulated tasks as Table 3. The new supervision consists of one hundred real human egocentric videos per unseen task, collected using props chosen to resemble simulation objects. The comparison therefore changes both embodiment and visual domain. Start with the Average row, then inspect the exceptions: Place Object Basket improves, whereas Stack Bowls Three and Pick Diverse Bottles deteriorate. This prevents two misleading readings: that all human-video transfer succeeds because robot-video transfer worked, or that every task necessarily gets worse when human footage is added. e12e18e19
What it supports. The human-video model averages 7.8/7.8, below the action-only baseline's 13.0/11.6. Basket placement improves from 3/5 to 17/14, but stacking bowls falls from 52/43 to 21/21. The overall negative result narrows the paper's promise of exploiting broad egocentric video sources.
Where the evidence stops. The authors conjecture a real-to-simulation domain gap. This table does not isolate that explanation from embodiment, viewpoint, video quality or optimization effects. Improved alignment remains proposed future work, not a demonstrated remedy.
7. Analysis & limitations
7.1 What the evidence leaves open
Real-world evidence covers a small set of bimanual tasks on one platform. Successful video transfer uses simulated robot videos; Appendix D's human-video attempt limits the broader egocentric-learning promise. e13e18e19
Average gains are not uniform: Turn Switch drops from 54/46 to 39/32 Clean/Rand. with world supervision. Figures of predicted versus actual images are qualitative, without a quantitative dynamics-fidelity assessment. e21e22
TTS correspondence is under-specified: Eq. 3.3 conditions world prediction on h_t and o_t, not the sampled action, although Section 3.2 describes candidate-specific futures. Table 1 retains a nominal 2B active count for TTS without reconciling its reactivated World Expert and value model. e04e06e07e09
7.2 Questions for discussion
- Does subtask memory track executed progress reliably when the generated subtask window is wrong?
- How are different action samples coupled to their imagined futures in TTS?
- Would matched-domain human videos reverse the negative transfer result?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs temporally aligned subtask labels, action conventions, pretrained components and the deployment kernels. The PDF does not fully specify label construction, image preprocessing, frozen-module policy, training hardware/software versions, random seeds, value-model architecture, rollout volume or numerical discount γ. e03e06e07e14e15e17
Proposed checks: compare correct, empty and shuffled textual memory under matched supervision; separately test whether TTS predicted-value rankings track actual candidate outcomes against random selection at equal candidate budget. Neither check was performed here. e03e04e06e10
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate useful memory from the effect of language supervision
Reader-proposed check: reproduce the four RMBench task-specific models with the reported training setup, then evaluate the same full-model checkpoints using intact subtask memory, empty memory, and length-matched memory shuffled across episodes of the same task. Retain the paper's no-language-loss training ablation as a separate comparator. Hold observations, initialization seeds, action sampling and episode starts fixed where possible. Measure success by task and manually verify subtask-transition errors against executed actions, especially repeated presses and hidden-block recovery. A large drop only under corrupted memory would support useful history tracking; little change would weaken that explanation even if removing the training loss still hurts. Report multiple training seeds and uncertainty, explicitly as additions to the source protocol. e03e10e20
Check 2: Test whether TTS ranks action consequences rather than unrelated images
Reader-proposed check: first obtain an explicit specification of how each sampled action is coupled to its World Expert prediction; do not invent a missing conditional input. At resettable LIBERO states, use the reported six candidates and horizon two, save every predicted future/value, and execute each candidate from the identical restored simulator state to measure realized success. Compare value-based selection against random selection from the same candidate pool and efficient single-candidate control. Also permute candidate-to-prediction assignments as a diagnostic while keeping the compute budget fixed. Report ranking accuracy, selected-action success and end-to-end latency. If correct pairing performs no better than permuted pairing or random selection, the claimed consequence-based selection mechanism lacks support even when extra sampling occasionally improves success. e04e06e09
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all method and experiment pages, Appendix A–D evidence pages, Figures 1–10 and Tables 1–7, including uncropped ablation, training/configuration, task-decomposition, per-task and prediction-example pages. Six final original crops were individually viewed, retaining all table headers and chart legends/axes; narrow originals were rendered at higher DPI without upscaling. Figure 2's conditioning arrows and optional branch were checked against Eqs. 3.1–3.5 and Section 3.2; unresolved TTS pairing is disclosed. Reference-only pages 10–12 were read in the complete text chunks but were not visually inspected. Separate supplements, code and linked media remain outside this supplied-source review.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work, including 2.1–2.2 (pp. 2–3)
- 3 Methodology, including 3.1–3.2 (pp. 3–5)
- 4 Experiments, including 4.1–4.4 (pp. 5–8)
- 5 Conclusion and 6 Limitations (p. 9)
- References (pp. 9–12)
- A Acceleration Techniques (p. 13)
- B Simulation Benchmarks, including B.1–B.3 (pp. 13–14)
- C Real-World Experiments (p. 14)
- D Learning New Tasks from Videos (pp. 14–15)
- Appendix task illustrations, per-task results and prediction examples (pp. 15–19)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read individually. Original PDF Figures 1–10 and Tables 1–7 were separately inspected, resolving the extraction-only image omission for this PDF. Reference-only pages 10–12 were read as text but not rendered.
- Identity/edition note: p. 1 matches the catalog title and all twelve authors. The artifact is stamped arXiv:2606.05979v1 [cs.RO], 4 Jun 2026, while its title block says June 5, 2026. No later revision or separate edition was supplied or substituted. The catalog affiliation field contains unrelated prose; verified metadata retains only the title-page institutional abbreviations.
- Code, linked resources and separate supplements were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block and arXiv margin stamp
Exact title and twelve authors match the catalog identity. Affiliations are SJTU, SII, HUST, SCUT, ECUST, SHU and NJUPT. The stamp identifies v1, 4 Jun 2026; the title block prints June 5, 2026.
Go to primary source ↓e02PDF pp. 1–3, Sections 1–2 and Figure 1(a–c)
The authors motivate complementary textual intention and physical dynamics, contrasting AR language backbones with visual DiT-based WAMs.
Go to primary source ↓e03PDF p. 3, Sections 3–3.1, Textual Intention Learning and Eq. (3.1)
Inputs include historical/current observations, instruction and proprioception. A pretrained VLM predicts temporally aligned subtask windows using recursively updated textual memory; action chunks execute in receding horizon.
Go to primary source ↓e04PDF p. 4, Figure 2(a), Physical Dynamics Modeling and Eqs. (3.2)–(3.4)
Causal meta-queries produce h_t. World prediction consumes h_t and the current observation; action prediction consumes h_t and proprioception. Targets are single-frame VAE features, optionally depth. World-loss gradients shape shared features; the World Expert can be discarded at inference. JEPA/DINO features are explicitly not used as targets.
Go to primary source ↓e05PDF p. 4, Training Objective, Eq. (3.5)
Joint loss combines action/world flow matching and language cross-entropy, weighted by α and β.
Go to primary source ↓e06PDF p. 5, Section 3.2, Efficient Mode and TTS Mode
Efficient mode disables the World Expert. TTS samples action candidates, predicts corresponding frames, scores them and chooses the highest-valued action. Values use instruction/predicted frame with label y·γ^(T−t); no numeric γ or full value-model specification is given.
Go to primary source ↓e07PDF p. 5, Section 4.1, Implementation Details
Reports total/component parameter counts, 28 layers per expert, 64 queries, chunks 8 on LIBERO/32 elsewhere, DeepSpeed, AdamW settings, cosine learning rate and auxiliary-loss weights.
Go to primary source ↓e08PDF p. 5, Table 1, RoboTwin columns and WLA-0/−L_wm/Fast-WAM rows; p. 6, Section 4.2, RoboTwin 2.0
WLA-0 reports 92.94/90.02 versus 90.98/89.34 without world loss and 91.88/91.78 for Fast-WAM. Protocol uses 50 tasks, mixed 2,500/25,000 trajectories, 100k steps, batch 256, 100 trials/task and disabled subtask prediction.
Go to primary source ↓e09PDF p. 5, Table 1, LIBERO columns and parameter/pretraining columns; p. 6, Section 4.2, LIBERO
WLA-0/−L_wm/+TTS averages are 98.6/97.9/98.9; Lingbot-VA is 98.5. Four suites each have 10 tasks and 50 demonstrations/task; one model trains 100k steps at batch 256 with 50 trials/task. TTS uses six candidates and horizon two; its row still lists 2B active parameters.
Go to primary source ↓e10PDF p. 6, Table 2 and Section 4.2, RMBench
Four M(n) tasks use single-task training for 30k steps, batch 448, with textual history. Average success is 56.5%, Mem-0 28.5%, Fast-WAM 13.3%, and language-loss ablation 17.3% (prose 17.25%). Cover Blocks is 84% versus 18%; Press Button 74% versus 1%; Blocks Ranking Try is 23% versus Fast-WAM 37%.
Go to primary source ↓e11PDF p. 7, Figure 3(a–d) and Section 4.3
Four Piper tasks use 60 demonstrations/task, 50k steps/batch 256, standard/OOD settings and 10 trials/setting. Synchronous RTX 5090 Stack Cup evaluation averages 10 successful rollouts with chunk 32 and 10 flow steps: WLA-0 25.8 s/37.7 ms; π0.5 29.0 s/190.1 ms; Motus 61.8 s/1529.8 ms, one generated frame/call.
Go to primary source ↓e12PDF p. 8, Table 3, Section 4.4 and Figure 4
45 seen tasks retain action labels; five unseen tasks supply videos from Aloha-AgileX or ARX-X5. Clean/Rand. means are 13.0/11.6 for Seen-Action, 11.8/12.6 with seen videos, 34.4/30.0 with unseen same-embodiment videos and 28.8/27.4 with unseen cross-embodiment videos. Each task contains 50 clean/500 randomized trajectories.
Go to primary source ↓e13PDF p. 9, Section 6, Limitations
Authors restrict physical generality to a small bimanual task set on one platform and note simulated robot videos as the successful video-learning setting.
Go to primary source ↓e14PDF p. 13, Appendix A, Acceleration Techniques
CUDA Graph replay, custom Triton fusion and precomputation/caching reduce reported inference from approximately 116 ms to under 40 ms. Cached cross-attention keys/values are reused across denoising steps within a prediction.
Go to primary source ↓e15PDF p. 13, Appendix B.1, RoboTwin 2.0
Lingbot-VA uses seen instructions; other Table 1 methods use unseen instructions. WLA-0 uses 32 flow steps to avoid arm jitter and absolute end-effector positions.
Go to primary source ↓e16PDF p. 13, Appendix B.2 and Table 4, single-frame/multi-frame rows
At n=32, single-frame predicts t+32, multi-frame predicts t+8,t+16,t+24,t+32. With 30k steps and batch 256, average success is 98.2 versus 94.2; the authors suggest slower convergence/interference.
Go to primary source ↓e17PDF p. 14, Table 5 and Appendix C
Table 5 reports mean success counts out of ten: WLA-0 7.5/5.25/4.75, π0.5 7/4.5/4.75, Motus 6/3.75/4 for Standard/OOD Object/OOD Scenario. Physical actions use joint angles and 10 flow steps.
Go to primary source ↓e18PDF p. 14, Appendix D, Learning New Tasks from Videos
Unseen robot videos have loss weight 0.1 with seen/unseen sampling 1:1; models train 50k steps, batch 256. Authors also collect 100 human egocentric videos per unseen task, report failure to learn new tasks overall and conjecture real-to-simulation domain gap.
Go to primary source ↓e19PDF p. 15, Table 6 and Figure 5
Human-video Clean/Rand. mean is 7.8/7.8 versus 13.0/11.6 baseline. Place Object Basket improves 3/5 to 17/14, while Stack Bowls Three falls 52/43 to 21/21. Figure 5 pairs simulated task objects with human-video views.
Go to primary source ↓e20PDF pp. 13–14, Appendix B.3; p. 15 Figure 6; p. 16 Figures 7–8; p. 17 Figure 9
Authors attribute memory benefits to repeated executable-subtask inference. Illustrations show battery-orientation trials, block-order trials, ordered uncovering and counting button presses as explicit subtask sequences.
Go to primary source ↓e21PDF p. 18, Table 7, Turn Switch and Average rows
Turn Switch Clean/Rand. success is 39/32 with WLA-0 and 54/46 without world loss, despite the overall averages improving from 90.98/89.34 to 92.94/90.02.
Go to primary source ↓e22PDF p. 19, Figure 10 and caption
The figure pairs predicted and ground-truth observations from simulation and physical tasks; it provides visual examples rather than a quantitative fidelity metric.
Go to primary source ↓e23PDF p. 2, Figure 1(d); p. 5, Table 1, RoboTwin columns
Figure 1 labels a single RoboTwin WLA-0 score of 91.5 without a Clean/Rand. column; Table 1 explicitly separates 92.94 Clean and 90.02 Rand. This report retains the table's setting-specific values.
Go to primary source ↓8.5 Primary sources
World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis ↗
PDF · 8,511 extracted words
Source fingerprint
470577b0347c361b12b0daaf791197649a1157aacff494107639638e04abd442