ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation
1. Paper overview
In one sentence: ω-0 uses future visual supervision to improve directly denoised humanoid action latents, with controller grounding and action continuity carrying much of the practical burden. e02e03e06e12e13e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Wiping while stepping or carrying while turning requires coordinated legs, torso, arms and hands. The authors argue that separate locomotion/manipulation policies and video-centered action pipelines can impair this coordination. Their question is whether compact future representations can improve directly generated whole-body actions without decoding future videos for control. e02e03 |
| Core mechanism | |
| A key reported result | Eleven-task household loco-manipulation with egocentric input: ω-0 Ego: 79.1%; 35.8/41; 88.7%. Success rate; summed average subtask score /41; reported task progress. Real G1; one multi-task policy per method; ten trials per task; downstream humanoid demonstrations. ψ-0: 44.5%, 23.6, 59.6%; DiT4DiT: 43.6%, 23.1, 61.0%. Reported execution results, not video quality. The success advantage over ψ-0 is 34.6 percentage points (reader subtraction); baseline adaptations and progress ambiguity remain material. e11e12e17 |
| Reading caution | Reader analysis The main definition assigns progress m/n when the first failure occurs at subtask m, while Appendix E describes completed-stage fractions. Table 8 contains later successes after earlier failures, so the conventions differ. Treat progress as a reported, unresolved metric rather than an unambiguous measure of uninterrupted execution. e17e18 |
Core contributions
- Source description
ω-HOME contains 40.3 hours, 4,827 episodes and 24 tasks at 30 Hz, with synchronized ego RGB, exo RGB-D, motion references, states and action latents. e08
Figure 2. Future visual features condition motion queries before action denoising. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start on the left with Stage 1: discrete action tokens teach the VLM an action-aware representation. In the center, its features join V-JEPA image features and text in the joint predictor; the purple action-condition path reaches the DiT through condition fusion, while green output latents go to SONIC. Read the lower-right inset from bottom to top: separate self-attention, prefix attention, then action queries reading video features as keys and values. This direction agrees with Section 3.3. The 2D RoPE label applies to spatial visual-prefix tokens; the text excludes language, view and VLM summary tokens from spatial RoPE. e03e04e05e06e10
What it supports. The graphic locates the useful predictive connection inside the conditioning representation. Actions are denoised into the controller's latent interface, while the Wan decoder supplies optional video visualization. The architecture therefore supports future-aware action learning without requiring a decoded video trajectory to be inverted into commands.
Where the evidence stops. Appendix B says the future-visual branch is unnecessary for control, but does not reconcile that statement with the internal query interaction. Only the decoder's exclusion from inference is explicit; do not infer that every video-query computation is removed.
2. Motivation
2.1 The problem and the proposed response
Wiping while stepping or carrying while turning requires coordinated legs, torso, arms and hands. The authors argue that separate locomotion/manipulation policies and video-centered action pipelines can impair this coordination. Their question is whether compact future representations can improve directly generated whole-body actions without decoding future videos for control. e02e03
2.2 What this reading follows
A humanoid wiping a table must keep its hand in contact while its legs and torso reposition. ω-0 learns that coordinated behavior by combining an action-aware vision-language representation, compact future visual targets and a diffusion head that produces SONIC-compatible actions. Its central visual prediction is learned in latent space; decoding a future movie is optional. The paper reports substantial gains on physical household tasks, supported by component ablations and detailed trial annotations. Read those results with two distinctions in mind: the Omni variant changes the camera used for some tasks, and the published definitions of task progress do not consistently describe the same computation. e02e03e06e12e13e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Conflicts with the recorded classification
The WAM and latent-prediction labels fit. The recorded IDM/Q4 label does not: the source explicitly rejects test-time video-to-action inversion and directly denoises actions conditioned on learned future-aware queries. The predictor and action DiT are distinct modules, so their joint loss also cannot establish a single shared-backbone One Model architecture. This is an assessment of the unchanged catalog snapshot, not a replacement classification. e03e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn human motion into a controller's training targets
The staged pipeline solves two different representation problems. First, a language model needs discrete targets: the FAST tokenizer converts continuous whole-body motion into action tokens, and next-token training teaches the VLM to associate images and instructions with those tokens. The later policy consumes its hidden features rather than relying on that token decoder for deployment. Second, a humanoid controller needs executable targets: public human trajectories do not contain the required robot proprioception or SONIC latents. Simulation replay creates those quantities, with unsuitable motions discarded. Appendix A then standardizes the interface: 64 body-latent coordinates plus two hand commands form each action, while a 47-dimensional state includes continuous 6D orientation. This separation explains why semantic pretraining alone is insufficient to supply the final robot action labels. e04e06e08e09
Figure 3. Collection diversity supplies visual and controller supervision at different scales. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the ring's center first: the collection totals 40.3 hours, 24 tasks and 4,827 episodes. The adjacent bars describe task-level collection duration in hours, not task success. Their unequal heights show that demonstration coverage varies across behaviors. The lower montage illustrates why camera identity matters: a close hand/object view and a room-level body view expose different parts of the same activity, while depth adds scene geometry. Sections 4 and 5 connect these observations to synchronized robot states, motion references and action latents. Keep the full collection distinct from the 2,220 trajectories used for the eleven-task downstream fine-tuning setup. e08e09e14
What it supports. ω-HOME gives the learning pipeline paired visual and robot supervision beyond generic human motion. Its scale and modality alignment are the supported contribution here. The paper separately tests adding non-overlapping collection tasks during pretraining, so this figure alone is not evidence that every recorded modality or task improves control.
Where the evidence stops. The full 24-task collection is not an eleven-task held-out benchmark. Task exclusion is described for the additional pretraining pool, but detailed episode, room and demonstrator split manifests are absent. The montage documents data examples, not autonomous evaluation success.
5.2 Read the future through the conditioning pathway
The crucial coupling is the direction of attention, not the presence of a video-shaped output in the diagram. Current image features, language, viewpoint and the action VLM form a prefix. Motion and video queries read that prefix separately; motion queries then read video features. Wan-encoded future observations supervise those video features, while the action DiT receives the resulting motion representation together with text and state. My interpretation is that future supervision shapes a condition useful to action prediction, rather than furnishing a decoded movie that must be inverted. The formulation does not describe candidate-action search or a separate inverse-dynamics estimator. Table 4 supports the combined pathway, but removing tokens, loss and cross-attention simultaneously leaves open how much of the gain comes specifically from predicting correctly aligned futures. e03e05e06e15
5.3 Execute a short prefix, then refresh the prediction
A generated chunk is a proposed future sequence, not a commitment to execute every action without feedback. Appendix B predicts 25 actions and executes eight before refreshing perception. An unexecuted portion of the previous prediction initializes the next chunk; overlap blending smooths transitions before controller delivery according to the prose. Stage-3 training anticipates this dependence by supplying a clean prefix and scoring only the remaining actions. The reported 0.14-second forward pass describes inference latency, not the entire sensing-to-actuation schedule. Two implementation boundaries remain: the appendix does not specify its overlap length, and Algorithm 1 lists blending after execution even though the prose describes stitching first. A reproduction must resolve those details and record actual execution timing before attributing continuity gains solely to the learned predictive representation. e07e10e21
5.4 Training and inference
During training
Stage 1 uses ARCTIC, Xperience-10M and Motion-X after conversion to unified SMPL. Stage 2 replays public motions through SONIC to obtain robot states and action latents, discarding untrackable or unsuitable motions. Filtering thresholds and retained subset sizes are unspecified. e08
Stages 2 and 3 freeze the action VLM, V-JEPA and Wan encoders while optimizing the joint predictor, state encoder, condition fusion and action DiT. Stage 2 combines clean-action prediction and future-latent squared error. Stage 3 uses 2,220 real trajectories across eleven tasks and replaces a random 0–8-frame noisy prefix with clean actions; loss applies only to the remaining actions. e06e07e08
All three stages use eight NVIDIA H100 GPUs. This is training hardware, not a specified deployment benchmark configuration. e21
During inference
Refresh the image and state, reuse the episode instruction, and sample actions with DDIM. Appendix B specifies H=25 predicted actions and K=8 executed before refreshing perception, an unexecuted previous segment as the next prefix, linear overlap blending, denormalization and SONIC execution. It reports approximately 0.14 seconds per forward pass, over 7 Hz inference; this does not establish the complete control-loop frequency. e06e10
Video decoding is explicitly unnecessary. Appendix B also calls the future-visual branch unnecessary for control, but does not explain how that statement relates to motion queries attending to video queries in the main formulation. Pruning the internal query pathway cannot be assumed. e05e06e10
5.5 Implementation flow
- Learn an action vocabulary
A whole-body FAST tokenizer reconstructs continuous trajectories with an L1 objective. Qwen3-VL-2B-Instruct then predicts discrete action tokens autoregressively from language, an image and a view token; its hidden features become the later semantic prefix. e04e08
- Connect perception to future queries
Concatenate action-VLM features, T5 text features, a view token and frozen V-JEPA 2.1 image features. Prefix, motion and video tokens undergo separate self-attention; both query streams read the prefix, then motion queries read video queries. Visual-prefix, video and action tokens use 2D, 3D and temporal 1D RoPE respectively. e05
- Predict executable latents
Fuse future-aware motion features with text and encoded robot state. The 0.45B action DiT predicts clean action latents from noisy latents. A frozen Wan encoder supplies future visual targets; the optional Wan decoder is outside policy inference. e03e06
6. Experiments & results
ω-0 learns language-conditioned humanoid actions with auxiliary prediction of future visual latents. A pretrained action VLM and a joint query predictor condition a diffusion action head; SONIC executes its whole-body latents. Reported real-robot results favor this design, but camera differences, bundled ablations and inconsistent progress definitions limit causal interpretation.
6.1 Read the original evidence
Table 2. Physical execution improves substantially under the paper's downstream protocol. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row across three different summaries. Success requires complete task objectives; score sums the task-level average counts of completed subtasks, with a suite maximum of 41; progress retains the paper's reported label, whose definition is disputed elsewhere. The two bottom rows are different camera configurations, not repeated runs. All methods use one multi-task policy, with the main protocol specifying ten trials for each of eleven tasks. Compare Ego with the baseline rows first, then consult Section 6.4 before interpreting Omni: five of its tasks use an external camera. The colored cells mark the table's leading values, without supplying uncertainty. e11e12e13e17
What it supports. Ego reports 79.1% success against 44.5% for ψ-0, the highest baseline success in this table: a reader-computed gain of 34.6 percentage points. Its 35.8/41 score also exceeds ψ-0's 23.6. These are reported real-robot execution outcomes, not a proxy based on generated-video appearance.
Where the evidence stops. Backbones, controller interfaces and chunking differ across baselines. Omni additionally changes inference viewpoint. No confidence intervals are shown, and progress definitions conflict between the main text and appendix; preserve the reported numbers without treating every column as an independent causal test.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Eleven-task household loco-manipulation with egocentric input Real G1; one multi-task policy per method; ten trials per task; downstream humanoid demonstrations. | ω-0 Ego: 79.1%; 35.8/41; 88.7%. Success rate; summed average subtask score /41; reported task progress | ψ-0: 44.5%, 23.6, 59.6%; DiT4DiT: 43.6%, 23.1, 61.0%. Reported execution results, not video quality. The success advantage over ψ-0 is 34.6 percentage points (reader subtraction); baseline adaptations and progress ambiguity remain material. e11e12e17 |
| Eleven-task household loco-manipulation with task-dependent viewpoints Omni uses exocentric inputs on five specified locomotion-heavy tasks and egocentric inputs on the remainder. | ω-0 Omni: 81.8%; 36.7/41; 90.3%. Success rate; score /41; reported task progress | Ego: 79.1%; 35.8/41; 88.7%. The 2.7-point success gain changes inference viewpoint as well as training; it cannot establish an egocentric-only deployment benefit. e12e13 |
| Removing future prediction and continuity components Table 4 real-world task-suite ablations; compare with full Ego. | Without video queries: 64.5%, 30.6, 77.9%; without RTC: 71.8%, 33.4, 84.1%. Success rate; score /41; reported task progress | Full Ego: 79.1%, 35.8, 88.7%. Video removal reduces success by 14.6 points, but jointly removes queries, the visual loss and motion-to-video attention. It does not isolate predictive supervision alone. e15e07 |
| Additional non-overlapping ω-HOME pretraining Exclude eleven downstream tasks from the additional Stage-2 pool; then fine-tune on downstream training data. | With additional data: Ego 80.4%, 36.9, 89.7%; Omni 82.4%, 37.5, 91.2%. Success rate; score /41; reported task progress | Without: Ego 79.1%, 35.8, 88.7%; Omni 81.8%, 36.7, 90.3%. Reported gains are modest; uncertainty and denominators explaining these success percentages are not supplied. e14 |
| Generalization across objects, scenes and human-data transfer Three object-variation tasks, two scene-variation tasks, and one human-trained closet-closing task; object/scene models fine-tuned on full ω-HOME. | With video queries: objects 83.3%, 11.8/13, 90.8%; scenes 79.5%, 5.5/6, 91.7%; human transfer 60.0%, 2.2/3, 74.6%. Success rate; score; reported task progress | Without: objects 66.7%, 7.6, 63.3%; scenes 15.0%, 0.5, 15.0%; human transfer 20.0%, 1.2, 20.0%. These are adapted-policy evaluations. Trial denominators and uncertainty are unclear, especially for the reported 79.5% over two scene tasks; this is not broad zero-shot task generalization. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Future-query coupling helps, but the ablation removes several mechanisms together. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the check/cross columns to identify the intervention before reading performance. The full Ego row is the direct numerical reference for the ablation discussion; the full Omni row adds a different viewpoint configuration. Removing state yields 60.9% success, removing the VLM prefix 66.4%, and removing video queries 64.5%, versus Ego's 79.1%. The RTC row and Wan-encoder row probe continuity and current-image representation respectively. Section 6.5 is essential: 'w/o Video query' removes the query tokens, their future-latent loss and the attention connection into motion queries. Consequently, the intervention changes both supervision and computation rather than toggling one loss coefficient. e07e15
What it supports. The full Ego-to-no-video-query success drop is 14.6 percentage points by subtraction, supporting the utility of the combined predictive pathway. RTC removal gives 71.8%, while the Wan current encoder gives 63.6%. The authors describe better offline Wan reconstruction qualitatively; reconstruction quality alone does not establish better executed control.
Where the evidence stops. This table cannot separate aligned future supervision from added query capacity and cross-attention. It also does not quantify motion smoothness or offline latent reconstruction error. A matched-architecture supervision control is needed to sharpen the mechanism claim.
Table 5 and score-maxima note. The predictive pathway also helps the paper's specific transfer settings. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each pair vertically: the cross-marked row removes video queries, and the checked row retains them. Compare within a setting because the retained note gives different score maxima. Section 6.7 defines three object variations, two changed-scene tasks and one human-transfer task involving walking to a closet door and closing it. The object and scene evaluations follow fine-tuning on full ω-HOME; human transfer uses human demonstrations grounded in the robot action space. The source reports averages for each setting, but gives no explicit trial denominator here. In particular, keep the printed 79.5% cross-scene value rather than rounding it into an imagined count of successful trials. e16e15e17
What it supports. With video queries, reported success is 83.3% for object variations, 79.5% for scene variations and 60.0% for human transfer, compared with 66.7%, 15.0% and 20.0% after removal. The direction is consistent across these settings, extending the combined-pathway evidence beyond the main task-suite aggregate.
Where the evidence stops. These are narrow adaptation and variation tests, not evidence of unrestricted zero-shot household competence. The intervention still bundles tokens, loss and attention removal. Trial denominators, repeated-seed variability and the exact aggregation behind 79.5% remain unresolved.
Table 8. A trial can earn subtask credit after an earlier failure. Original paper, p. 24 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one column as a complete trial and one row as a physical milestone. Stage 1 is stable bin holding; the next three rows record target-object deposits; the final row requires all targets to remain in the bin without distractors. Trial 1 reads 1,0,1,0,0, while Trial 2 reads 1,0,1,1,0. Thus later successful collection does not imply the preceding target was collected. Only Trials 4 and 7–10 contain five ones. This is precisely the kind of record needed to distinguish whole-task success, total completed-stage credit and a strict completed prefix before the first failed stage. e18e11e17
What it supports. Counting the displayed columns gives five fully successful trials out of ten. Counting all ones gives 37 completed milestones, or 3.7/5 average score; both are reader calculations from this table. A strict completed-prefix calculation instead gives 60% average progress, whereas normalized total completion gives 74%, illustrating why the definition matters.
Where the evidence stops. Section 6.1 describes first-failure progress and prints m/n, while Appendix E uses completed-stage fractions. The two calculations above expose the distinction; neither is presented as a repaired official metric. The exact published progress convention requires clarification.
7. Analysis & limitations
7.1 What the evidence leaves open
The main definition assigns progress m/n when the first failure occurs at subtask m, while Appendix E describes completed-stage fractions. Table 8 contains later successes after earlier failures, so the conventions differ. Treat progress as a reported, unresolved metric rather than an unambiguous measure of uninterrupted execution. e17e18
Baseline action interfaces, backbones and chunking differ; GR00T uses sequential chunks without the same RTC. Shared downstream evaluation does not isolate world modeling from controller compatibility, capacity or training priors. e11e15
Figure 5 is a radar plot although its caption describes rollout sequences. Table 1 names fridge drink retrieval, while Table 16 and Figure 22 specify apple-and-pomegranate retrieval. These internal presentation differences remain unresolved. e19e20
The source reports no confidence intervals or repeated-training-seed uncertainty. Human-transfer evidence covers one specified task, and SONIC filtering restricts the motion distribution. The paper has no dedicated limitations section. e08e12e15e16
7.2 Questions for discussion
- Does aligned future supervision help when query capacity and attention topology are held fixed?
- Which internal video-query computations remain active at deployment, and how is progress computed from non-prefix trial annotations?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the pretrained encoders/VLM, SONIC-compatible labels and controller, and synchronized robot data. Appendix A specifies 66 action and 47 state dimensions, mean/standard-deviation normalization for the 64 body latents, min-max scaling for states/hands, z-up coordinates and initial-yaw subtraction. e04e08e09
Missing essentials include exact public-data filters and split manifests, optimization schedules, λvideo, DDIM step count, overlap length and deployment hardware. Appendix B's prose places stitching before execution, but Algorithm 1 lists execution before blending; scheduling and whether prefixes are clamped throughout denoising need clarification. e06e08e10e21
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Keep query capacity fixed while changing future supervision
Reader-proposed experiment, not performed: train three Ego variants with identical query tokens, attention topology, action DiT, data split, initialization protocol and compute. Use correctly aligned future Wan targets, targets shuffled within the same task/view, or no visual loss. Evaluate repeated matched starts on wiping and garbage collection, reporting task success, total stage score and separately defined strict-prefix progress. If aligned supervision reliably beats both controls while capacity and RTC remain fixed, that supports information in the future targets. If the controls match it, the published branch-removal ablation cannot attribute its gain specifically to predictive alignment. e05e06e11e15e17e18
Check 2: Separate learned prefix conditioning from runtime smoothing
Reader-proposed 2×2 experiment, not performed: cross RTC prefix training on/off with deployment warm-start plus overlap blending on/off, keeping the same Ego data, H=25, K=8, DDIM configuration and controller. Declare the overlap length and execution order explicitly. Repeat matched cleaning trials and log success, contact loss, action discontinuities at chunk boundaries, and measured image-to-controller latency. Compare deployment smoothing at each training setting and RTC training at each deployment setting. If smoothing alone accounts for the continuity gain, the training-specific claim weakens; an additional reproducible benefit from prefix training would support the intended train/deployment alignment. e07e10e11e15e21
8.3 Reading coverage
Visual audit: Actually viewed original PDF pages 1–18, 22–27 and 33, including the title/author/version block, all main method and experiment pages, Appendix A configuration and preprocessing, Appendix B timing and Algorithm 1, all per-trial tables, per-task result charts and the detailed fridge-task card. Figures 1–11 and 21–22 and Tables 1–16 were visually inspected. All six final crops were separately viewed; Table 5 retains its necessary score-maxima note. Figure 2 attention direction and RoPE labels were checked against Sections 3.3–3.4. Figure 5's caption mismatch, the progress-definition conflict, the fridge-task naming difference and the inference scheduling ambiguity are disclosed. The full 39-page text was read in eleven individual chunks; other task-gallery images and any separate supplemental material remain outside this visual pass. No video decoding, paper code or robot experiment was run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 22, 23, 24, 25, 26, 27, 33. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract; 1 Introduction
- 2 Related Work, including 2.1–2.3
- 3 Method, including 3.1–3.5
- 4 ω-HOME Dataset, including 4.1–4.3
- 5 Implementation and Experiments; 5.1 Datasets and Data Processing
- 6 Implementation Details and 6.1–6.7 evaluation, baselines, results, ablations and generalization
- 7 Conclusion; References
- Appendix contents; A.1–A.3 Data Format and Preprocessing
- B.1–B.2 Inference Details and RTC Deployment; Algorithm 1
- C.1–C.2 Real-World Evaluation Details; Tables 6–16
- E.1–E.3 Per-Task Baseline Comparison
- D Task Gallery and D.2 Full ω-HOME Task Cards, through PDF page 39
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The observed title page is arXiv:2608.06375v2 [cs.RO], dated 9 August 2026. The title and all eleven authors match the catalog. The catalog does not specify a revision; no earlier edition was supplied or compared.
- All eleven supplied text chunks were individually read, covering all 39 PDF pages. The acquisition warning that text extraction does not reconstruct figure images was addressed by inspecting original PDF pages and six final crops. Figure 3 chart text was garbled in extraction and checked visually.
- Separate supplemental material availability has not been fully verified.
- Visual inspection covers PDF pages 1–18, 22–27 and 33. Other task-gallery images were not visually inspected, although their complete supplied text was read. References were read without acquiring their cited works.
- Code, external project resources and datasets were not inspected; no training or robot experiment was reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin stamp
Observed title exactly matches the supplied primary title. Authors are Zhe Li, Zhenzhe Zhang, Yangyang Wei, Wenjie Zhang, Xichen Yuan, Peiyuan Zhi, Gen Li, Xinying Guo, Fengjie Gao, Jianfei Yang and Shanghang Zhang. The stamp is arXiv:2608.06375v2 [cs.RO], 9 Aug 2026; affiliations are MARS Lab, NTU; PKU; BAAI; HKUST(GZ).
Go to primary source ↓e02PDF pp. 2–3, Section 1, motivation and contributions
Concurrent loco-manipulation motivates learning whole-body coordination and compact predictive visual supervision rather than a video-centered action pipeline.
Go to primary source ↓e03PDF p. 4, Section 3.1; p. 5, Figure 2 and caption
The architecture has an action VLM, joint latent predictor and 0.45B action DiT, with SONIC execution. Figure 2 directs future action features into condition fusion; its Wan decoder visualizes future latents.
Go to primary source ↓e04PDF p. 5, Section 3.2, tokenizer and autoregressive-loss equations
FAST encodes action chunks, reconstructs them with L1 loss, and supplies discrete labels for Qwen3-VL-2B-Instruct next-token training. Hidden VLM features are reused in subsequent stages.
Go to primary source ↓e05PDF p. 6, Section 3.3, prefix construction, RoPE and final attention equations; p. 5, Figure 2 right panel
Current features come from V-JEPA 2.1; T5, a view token and action-VLM features form the prefix. Motion/video queries separately read the prefix before motion queries attend to video queries. Spatial RoPE excludes text, view and VLM summary tokens.
Go to primary source ↓e06PDF p. 7, Section 3.3, visual loss, condition fusion, x0 objective and frozen-module paragraphs
Wan future latents supervise squared error. The DiT predicts clean actions and uses DDIM at inference. V-JEPA, Wan and the VLM remain frozen. Video decoding is optional and outside real-time policy inference.
Go to primary source ↓e07PDF p. 7, Section 3.4, RTC equations; p. 8, Section 3.5; p. 16, Section 6.5, Training-time RTC
Fine-tuning replaces a clean action prefix and scores only the remaining actions. The ablation discussion specifies random prefix lengths between 0 and 8. Deployment reuses preceding predictions for continuity.
Go to primary source ↓e08PDF pp. 8–9, Sections 4–4.3 and Figures 3–4; p. 10, Section 5.1
ω-HOME reports 40.3 hours, 4,827 episodes, 24 tasks and 30 Hz multimodal recordings. Teleoperation uses Pico devices, ZED cameras and Inspire hands. Public training sources are ARCTIC, Xperience-10M and Motion-X; unsuitable motions are filtered before controller-supervised learning. Downstream fine-tuning uses 2,220 trajectories across eleven tasks.
Go to primary source ↓e09PDF p. 22, Appendix A.1–A.3, action/state definitions and normalization equations
Actions have 64 controller latent dimensions plus two bounded hand commands; state has 47 dimensions and excludes IMU acceleration/angular velocity. The appendix specifies 6D orientation, separate normalization, SMPL conversion, z-up coordinates and zero-yaw preprocessing.
Go to primary source ↓e10PDF p. 23, Appendix B.1–B.2, overlap equation and Algorithm 1 lines 4–12
A forward pass is approximately 0.14 s; H=25 and K=8. Unexecuted actions warm-start the next chunk; overlaps are linearly blended and latents denormalized. The prose says stitching precedes sending actions, whereas Algorithm 1 lists execution at line 10 and blending at line 12. The future-visual branch is described as unnecessary for real-time control.
Go to primary source ↓e11PDF pp. 10–12, Sections 6.1–6.4 and Table 1; p. 17, Section 6.6
The main evaluation uses one multi-task model per method and ten trials per task. Success requires all objectives; score counts completed subtasks. Baselines use adapted action heads and differing chunking/control interfaces. Section 6.6 states that G1 evaluation rollouts are autonomous, without teleoperation, replay or scripted intervention.
Go to primary source ↓e12PDF p. 13, Table 2, ω-0 Ego, ω-0 Omni, ψ-0 and DiT4DiT rows, all metric columns
Table 2 reports Ego 79.1/35.8/88.7, Omni 81.8/36.7/90.3, ψ-0 44.5/23.6/59.6 and DiT4DiT 43.6/23.1/61.0 for success percent, score out of 41 and progress percent. No uncertainty is shown.
Go to primary source ↓e13PDF pp. 13–14, Section 6.4, Ego/Omni viewpoint paragraphs
Ego uses egocentric observations for training and evaluation. Omni uses exocentric observations on clothes-to-basket, towel-to-washer, mopping, garbage collection and apple/drawer tasks; remaining tasks use egocentric observations.
Go to primary source ↓e14PDF p. 13, Table 3, all rows; p. 14, Section 6.4, additional ω-HOME data paragraph
Adding non-overlapping ω-HOME tasks to Stage 2 changes Ego SR/score/progress from 79.1/35.8/88.7 to 80.4/36.9/89.7 and Omni from 81.8/36.7/90.3 to 82.4/37.5/91.2. Downstream tasks are excluded from the additional pretraining pool.
Go to primary source ↓e15PDF p. 14, Table 4, all rows; pp. 15–16, Section 6.5
Success rates are 60.9 without state, 66.4 without VLM prefix, 64.5 without video queries, 71.8 without RTC and 63.6 with Wan as current encoder, versus Ego 79.1 and Omni 81.8. Video ablation removes queries, visual loss and motion-to-video attention together. Better offline Wan latent reconstruction is discussed qualitatively, without a numerical reconstruction table.
Go to primary source ↓e16PDF p. 17, Section 6.7, setting definitions and ablation; p. 18, Table 5 and score-maxima caption
Three object tasks, two scene tasks and one human-transfer task are described. Table 5 with/without-query SR is 83.3/66.7, 79.5/15.0 and 60.0/20.0 respectively; scores and progress are reported for each setting, with maxima 13, 6 and 3. Exact trial denominators and uncertainty are not given here.
Go to primary source ↓e17PDF p. 11, Section 6.1, Task Progress paragraph; p. 26, Appendix E.2; p. 27, Figure 10 caption
The main text defines progress from the first failing subtask m as m/n. Appendix E.2 and Figure 10 instead describe a normalized fraction of completed stages. These definitions are not equivalent for later successes following earlier failures.
Go to primary source ↓e18PDF pp. 24–25, Appendix C.1–C.2, Tables 6–16; specifically p. 24, Table 8 all rows and trials
Binary annotations mark completed physical milestones. In Table 8, Trial 1 is 1,0,1,0,0 and Trial 2 is 1,0,1,1,0, demonstrating non-prefix completion. All-five-stage columns occur in Trials 4 and 7–10.
Go to primary source ↓e19PDF p. 13, Figure 5 panels, legend and caption
The visual is three radar plots for score, success and progress. Its caption instead describes task instructions and orange markers over robot rollout sequences; the caption and graphic do not match.
Go to primary source ↓e20PDF p. 12, Table 1 row 11; p. 25, Table 16; p. 33, Figure 22 task card
The main task list labels fridge drink retrieval, whereas detailed annotations and the task card describe retrieving apple and pomegranate into a basket and closing the fridge.
Go to primary source ↓e21PDF p. 10, Section 6 Implementation Details; p. 23, Appendix B.1
Training uses eight NVIDIA H100 GPUs. The latency paragraph reports forward-pass time without specifying deployment hardware or full end-to-end control timing.
Go to primary source ↓8.5 Primary sources
ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation ↗
PDF · 18,057 extracted words
Source fingerprint
c56d4115741a7979bb63ed4ff8e76e25c8e6ca933d6b18f2d122d8a46ec4d02c