EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
1. Paper overview
In one sentence: Predicting semantic features or stabilized motion during training improves transfer from human demonstrations, while deployment retains only the action policy. e-framinge-jointe-maine-limitationse-sim-results
| At a glance | What to know |
|---|---|
| Research problem | Source description Human demonstrations contain useful object and scene diversity, but their grasp styles, body geometry and moving viewpoints can corrupt a shared action decoder. EgoWAM asks whether future-scene supervision provides a more transferable learning signal. Its target criteria are appearance abstraction, consistency of physical effects across embodiments, and separation of camera motion from environmental change. e-framinge-joint |
| Core mechanism | Source description A common HPT trunk and action decoder support swappable Pixel, Pixel-PT, DINO and 3D Flow heads, enabling comparison within the same action interface and data regimes. e-architecturee-targets |
| A key reported result | Fold-clothes: OOD generalization: 85% Full-task success rate. Real robot; DINO with EgoVerse; main protocol pools 10 unseen-object and 10 novel-scene rollouts. DINO robot-only: 15%; gain of 70 percentage points. Completion requires all three folds. Figure 3 includes 95% confidence intervals; exact endpoints are not tabulated. e-maine-evaluation |
| Reading caution | Source description The authors report context generalization, not acquisition of new motion primitives. They train one policy per task; multi-task scaling and the best general world representation remain open. e-limitations |
Core contributions
- Source description
A common HPT trunk and action decoder support swappable Pixel, Pixel-PT, DINO and 3D Flow heads, enabling comparison within the same action interface and data regimes. e-architecturee-targets
- Author claim
The authors attribute improved human-to-robot transfer to world-level supervision, with semantic targets favoring object/scene generalization and geometric targets favoring spatial control. e-maine-unalignede-modality
Figure 2. Two supervision channels shape one policy representation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom of panel (a). Blue stems serve both embodiments; the pink wrist stem is robot-only. Observation tokens enter the same transformer as the learned action and future queries. Follow black arrows upward to two separate readouts, then red dashed arrows back toward the shared transformer: these depict forward computation and training gradients, respectively. The right-hand panels change the world head and target, from VAE latents to DINO features to point-conditioned flow. The z labels denote trunk context; q supplies current-frame anchor positions for Flow. Equation (2) confirms parallel prediction conditioned on z, with no explicit action-to-world transition arrow. e-architecturee-jointe-targetse-flowe-objective-caveate-deploy
What it supports. The mechanism is representation learning through an extra loss. Both action fidelity and future prediction influence the same trunk, giving human data a second way to teach it. At deployment the authors remove the world head, so the architecture does not imply that a robot follows predicted pixels or flow.
Where the evidence stops. The sketch is not a complete loss specification. Section 4.2 uses noise-prediction formulas for Pixel/DINO, while Appendix B uses velocity-prediction language. DINO's RGB decoder is visualization-only. These textual qualifications matter when translating the diagram into an implementation.
2. Motivation
2.1 The problem and the proposed response
Human demonstrations contain useful object and scene diversity, but their grasp styles, body geometry and moving viewpoints can corrupt a shared action decoder. EgoWAM asks whether future-scene supervision provides a more transferable learning signal. Its target criteria are appearance abstraction, consistency of physical effects across embodiments, and separation of camera motion from environmental change. e-framinge-joint
2.2 What this reading follows
A human and a robot can move the same cup while looking different and using incompatible grasps. EgoWAM separates the supervision for that physical change from the supervision for executable actions. Its shared transformer learns both, but the auxiliary prediction target matters: reconstructing appearance can preserve the embodiment gap, whereas semantic features and camera-stabilized motion offer more transferable structure. Read the architecture first, then compare actual robot completion rates, and finally use the ablations to test the proposed explanation. The results support better context transfer within the studied tasks; they leave new skills, multi-task learning and precise insertion unresolved. e-framinge-jointe-maine-limitationse-sim-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Conflicts with the recorded classification
Joint prediction is supported by the factorization and simultaneous future/action supervision. Dual-system conflicts with the architecture: observation, action and future tokens share one transformer, followed by two heads. I assess One Model × Joint prediction, with training-only world supervision. Frozen target encoders are not a separate inference-time controller. DINO feature prediction here does not establish a JEPA objective. The catalog snapshot is preserved unchanged. e-architecturee-jointe-targetse-deploy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First align the action interface, then add another learning signal
Start with what a robot is actually asked to execute. EgoWAM places human and robot trajectories in a common end-effector representation, rather than mixing joint angles from unrelated bodies. It also compensates for moving human cameras and different demonstration speeds. This makes action-only co-training a meaningful baseline, but a shared numerical space does not make every human grasp executable by a parallel-jaw gripper. The auxiliary world head addresses the remaining gap through learning: the same observation representation must support both action prediction and a description of future scene change. Equation (3) sends gradients from both embodiments and both objectives into that representation. Equation (2) is equally important: the heads are parallel readouts of shared context. There is no inference-time inverse-dynamics stage that converts a predicted scene into actions. e-actionse-jointe-architecturee-deploy
5.2 Change the prediction target, change what the trunk must preserve
A pixel latent rewards reconstruction of appearance, so success at that auxiliary task can depend on visual details that differ between people and robots. DINO replaces that target with semantic patch features, but the patches still live on an image grid: camera motion changes their location. Flow instead measures tracked physical displacement after stabilization into the current camera frame. The intended result is near-zero background motion and retained manipulated-object motion. This is why camera poses and tracking thresholds are part of the method, not incidental preprocessing. Reader interpretation: these choices change the information the trunk is encouraged to preserve, which explains why richer-looking predictions need not imply better control. The comparison is nevertheless between target-and-head packages: the denoisers differ in architecture and capacity, and Pixel-PT receives a different training schedule. It is not a fully capacity-matched target-only experiment. e-framinge-targetse-flowe-training
5.3 Use the ablations to separate a useful pattern from a causal proof
Figure 9 asks whether careful demonstration alignment can repair transfer, while Figure 11 asks which human supervision channels matter. Their answers are complementary: manually aligning behavior and viewpoint helps BC, Pixel and DINO, but Flow already performs strongly; removing human action labels still leaves useful Flow transfer, while combining labels and Flow performs best. The authors describe action labels as providing intent and task relevance. Reader interpretation: the evidence supports complementary supervision, but neither the shared-trunk factorization nor the architecture drawing establishes a direct action-label conditioning input to the world predictor. Likewise, Table 1's lower prediction losses are descriptive corroboration without reported uncertainty. A gradient-detachment control could test whether the policy gains actually require world-loss updates to the trunk. Throughout, keep executed success distinct from prediction fidelity, since the world head is absent during deployment. e-alignmente-modalitye-world-losse-jointe-deploy
5.4 Training and inference
During training
Each co-training step draws 32 robot and 32 human samples, summing action and world losses with world weight 1. AdamW uses learning rate and weight decay 0.0001, cosine annealing and bf16. Most variants train 2000 epochs of 100 steps on one L40S; Pixel-PT uses two L40S GPUs for 1000 epochs. Reported duration is about two days per task/method. e-traininge-joint
Section 4.2 specifies noise prediction for Pixel/DINO, whereas Appendix B calls training velocity prediction while retaining a noise-target DINO formula. This unresolved distinction is preserved. e-objective-caveat
During inference
Only the trunk and action head run from the current observation. The action decoder uses 50 sampling steps; the authors report 30 Hz deployment on one RTX 4090. Future prediction is auxiliary supervision, not online imagination or a controller that follows generated flow. e-architecturee-deploy
5.5 Implementation flow
- Align actions before comparing supervision
Each arm contributes a 6-DoF pose and gripper command. Human trajectories are transformed into the device frame at the current observation; robot poses use the fixed ego-camera frame. Human 1-second and robot 1.5-second windows are resampled to 100 steps. First/99th-percentile normalization reduces tracking-outlier sensitivity. e-actionse-architecture
- Share a representation, retain separate readouts
Table 2 specifies ResNet-18 visual stems, a proprioceptive MLP, and a 256-dimensional, 16-block transformer. Its 64 learned action queries and 16 future queries share the trunk. A six-block flow-matching action decoder and a target-specific world decoder consume its features. These are parallel readouts, not an inverse-dynamics chain. e-architecturee-joint
- Choose what the future means
Pixel uses frozen Wan VAE latents; Pixel-PT adds a larger pretrained denoiser. DINO uses frozen DINOv2-B patch features and a wide RAE-style denoiser, with its RGB decoder used only for visualization. These image-grid targets contrast with 3D Flow, which predicts physical displacement after camera stabilization. e-targets
- Build the geometric target
Track4World estimates tracks; Aria VIO poses stabilize them to the current camera. Appendix B.2 specifies a 28-by-40 anchor grid and 100-by-1120-by-3 target. It discards low-motion anchors below 2 mm for robots and 10 mm for humans and skips the first/last 20 human frames. The decoder also receives anchor positions. e-flow
6. Experiments & results
EgoWAM studies which auxiliary future-prediction target helps a robot learn from egocentric human demonstrations. A shared transformer predicts actions alongside pixel latents, DINO features or stabilized 3D motion. DINO and 3D Flow produce stronger transfer than pixel prediction in the reported tasks. The world head shapes training representations and is discarded for deployment; the policy does not plan through generated futures.
6.1 Read the original evidence
Figure 3. Compare data regimes within a method before comparing representations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a task row, then distinguish normalized partial-task score from full success. Grey bars use robot data, pale yellow adds in-domain human data, and gold adds EgoVerse. Compare these colors within the same method to assess the effect of human data. The left half uses familiar objects and scenes with randomized poses; the right combines unseen-object and novel-scene tests. Green and red brackets annotate gains or losses between robot-only and EgoVerse. On success-rate axes, the annotated differences are percentage-point changes, not multiplicative improvements. The caption specifies 95% finite-sample-valid confidence intervals, which should be read alongside the bar heights. e-maine-evaluatione-targetse-traininge-deploy
What it supports. Fold-clothes OOD success rises from 15% for robot-only DINO to 85% with EgoVerse, a 70-point gain. Cup placement reaches 90% for Flow with EgoVerse versus 70% robot-only. These are executed task-completion results; they support useful transfer without requiring future-image generation during control.
Where the evidence stops. The main protocol gives 20 ID and 20 OOD trials per setting, with OOD split across object and scene changes. This pooled display does not isolate those changes. Exact interval endpoints are not tabulated, and head capacity and Pixel-PT training duration differ.
Table 4. Cross-robot transfer helps selected tasks but leaves precise mug hanging unsolved. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each EgoWAM block, s means training only on aloha-agilex and c means co-training across four robot embodiments. All methods use the same end-effector action interface and IK execution, with 100 shared held-out seeds. Read ACT-EE and DP-EE as single-embodiment architectural references, not as cross-trained controls. Bottle picking improves strongly for cross DINO. The dagger on stacking is essential: an incidental bowl-material change between demonstrations and the evaluated simulator creates an appearance shift, even though geometry and pose remain unchanged. The crop retains the source's explanatory footnote. Finally, read the hanging-mug row to test how far the transfer claim extends. e-sim-protocole-sim-results
What it supports. DINO bottle-picking success rises from 4% single to 28% cross; cross Flow reaches 16%. On shifted-material stacking, only cross DINO and Flow succeed, at 8% and 16%. These outcomes reproduce a transfer advantage in simulation, but overall success remains low and every method stays at or below 1% on mug hanging.
Where the evidence stops. The appendix's statement that world variants consistently exceed BC is broader than this table: Pixel ties BC at zero on stacking, and DINO/Flow tie BC at zero on hanging-mug. The incidental material shift also requires exact asset provenance for faithful reproduction.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Fold-clothes: OOD generalization Real robot; DINO with EgoVerse; main protocol pools 10 unseen-object and 10 novel-scene rollouts. | 85% Full-task success rate | DINO robot-only: 15%; gain of 70 percentage points. Completion requires all three folds. Figure 3 includes 95% confidence intervals; exact endpoints are not tabulated. e-maine-evaluation |
| Cup-on-saucer: in-domain placement Real robot; seen objects/scene with randomized poses; 20 rollouts per setting. | 3D Flow + EgoVerse: 90% Full-task success rate | 3D Flow robot-only: 70%; BC + EgoVerse: 15%. Executed bimanual placement improves by 20 percentage points over robot-only Flow. e-maine-evaluation |
| Bag-grocery: deliberately unaligned human data Figure 8 compares natural in-domain and deliberately unexecutable human demonstrations. | 3D Flow: 80% natural, 75% unaligned Full-task success rate | BC: 45% natural, 20% unaligned; robot-only: 60% Flow and 40% BC. Flow is more robust in this counterfactual; separate uncertainty estimates are absent. e-unaligned |
| Cup-on-saucer: human supervision modality Appendix A.2; ordered splits: in-domain / OOD object / OOD scene; human supervision modality is ablated. | Action + 3D Flow: 90% / 78% / 30% Full-task success rate | Flow-only: 35% / 44% / 10%; action-only: 15% / 11% / 0%. Joint supervision leads throughout. Separate ablation trial counts and uncertainty are not supplied; denominators cannot be inferred. e-modality |
| RoboTwin: pick-diverse-bottles demo_clean-50; aloha-agilex target, three additional training embodiments; 100 shared held-out seeds. | Cross-embodiment DINO: 28% Closed-loop success rate | Single DINO: 4%; cross BC: 6%; cross Pixel: 11%; cross Flow: 16%; ACT-EE/DP-EE: 2%/5%. Robot-to-robot simulation uses a common 14-D action interface and IK executor, separate from real human-to-robot rollouts. e-sim-protocole-sim-results |
| RoboTwin: appearance-shifted bowl stacking and mug hanging Table 4; 100 held-out seeds; bowl material differs from demonstration assets. | Stacking: cross Flow 16%, cross DINO 8%; hanging-mug: all methods at most 1%. Closed-loop success rate | All other stacking entries: 0%; only cross Pixel achieves 1% on hanging-mug. Abstraction helps appearance-shifted stacking but leaves fine insertion largely unsolved. e-sim-protocole-sim-results |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 9. Manual alignment helps image-based targets; Flow stays high in both conditions. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each line from natural in-domain demonstrations on the left to deliberately robot-aligned demonstrations on the right. Both use the cup-on-saucer task and a 1:1 human regime. The blue BC line crosses its pale 25% robot-only reference, showing that aligned human data can help action learning. Pixel and DINO also improve, while Flow's gold line is flat. The collection protocol explains what alignment means: the demonstrator mimics the robot's movement and grasp approach while matching its camera height and static viewpoint. Consequently, the horizontal comparison changes execution and sensing together; the axis is not a pure camera-motion intervention. e-alignmente-flow
What it supports. BC improves from 15% to 35%, Pixel from 35% to 65%, and DINO from 50% to 70%. Flow remains at 85%. Within these two collection regimes, Flow is less dependent on manually arranging human demonstrations to resemble the robot's observations and behavior.
Where the evidence stops. The caption attributes the contrast to isolated human head motion, but the collection description also changes grasp strategy and movement. The data support robustness to the combined alignment intervention. They do not prove universal viewpoint invariance or isolate the contribution of camera stabilization.
Figure 11. World supervision helps without human action labels; their combination performs best. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel as partial-task progress and the right as complete cup-placement success. Each split repeats three colors: action-only, Flow-only, and joint action-plus-Flow supervision. Here Flow-only means removing action supervision from human batches; it does not mean deploying a flow-following robot or eliminating robot action training. All three splits favor the joint case. For the novel-scene split, inspect both the nonzero partial scores and the stricter success values: a policy can make progress without completing placement. Appendix A.2 interprets action labels as useful context and task-relevance supervision; the architecture establishes shared training, rather than an explicit ground-truth-action input to the world head. e-modalitye-world-losse-jointe-deploy
What it supports. In-domain success is 15% with action-only human data, 35% with Flow-only, and 90% jointly. Novel-scene success is 0%, 10% and 30%. Table 1 also reports lower Flow prediction losses with joint supervision: 0.23 to 0.22 for humans and 0.20 to 0.19 for robots.
Where the evidence stops. The source gives no separate ablation trial counts or error bars. Object-split rates of 11%, 44% and 78% should remain reported percentages without reconstructed counts. Table 1 does not specify a held-out split or uncertainty, so its small loss differences do not establish significance.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report context generalization, not acquisition of new motion primitives. They train one policy per task; multi-task scaling and the best general world representation remain open. e-limitations
World-head capacity, initialization and Pixel-PT training duration differ. Figure 9 also changes grasp behavior and viewpoint together: BC rises 15% to 35%, Pixel 35% to 65%, DINO 50% to 70%, while Flow stays at 85%. This supports robustness but does not isolate camera motion causally. e-targetse-traininge-alignment
Appendix A.3 associates Pixel-PT's prematurely open predicted bag with skipped opening actions. Because prediction is discarded at deployment, these images diagnose a learned representation or prior; they do not establish an online generated-image-to-action failure pathway. e-pixel-failuree-deploy
7.2 Questions for discussion
- Would equal-capacity, equal-budget world heads preserve the DINO/Flow advantage?
- How much of Flow's benefit survives camera-pose errors or removal of human action labels?
8. Reproducibility audit
8.1 Requirements and known gaps
Collection requires camera-calibrated bimanual control, robot wrist/ego imagery and human hand/head poses. Table 3 reports 300/360/300 robot demonstrations for cup/fold/bag, two hours of in-domain human data each, and 20.5/21/7 hours of EgoVerse. Corpus scale is distinct from balanced 32/32 optimization batches. e-datae-training
Resolve the noise-versus-velocity objective and camera-frame convention. Specify how discarded low-motion anchors interact with the fixed full-grid target. Obtain exact simulation asset revisions and ablation trial counts: the bowl material mismatch and unreported ablation denominators matter. The illustrated edition proposes two controlled checks. e-objective-caveate-flowe-sim-resultse-modality
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does camera stabilization explain Flow's robustness?
Reader-proposed, not executed: on the same cup-on-saucer clips, compare two identical Flow policies trained with raw versus Aria-VIO-stabilized tracks. Keep action labels, data splits, anchor grid, loss weights, optimizer budget and training seeds fixed. Resolve the source's coordinate-frame ambiguity first and hold the anchor-validity mask constant across conditions so filtering cannot explain the difference. Measure static-background displacement, held-out world error and robot placement success, stratifying human clips by camera motion. The proposed mechanism predicts that stabilization reduces spurious background motion and improves transfer most for high-motion clips. If policy gains survive unchanged with raw tracks, or arise only when the mask changes, the camera-factoring explanation is weakened. e-flowe-objective-caveate-alignmente-training
Check 2: Do world-loss gradients, rather than the extra head alone, improve control?
Reader-proposed, not executed: repeat cup placement with full joint training, human Flow-only supervision, and a full-loss control that detaches trunk features only at the world-head inputs on both embodiments. In the detached control the world head still trains, but its loss cannot update the trunk; action gradients remain intact. Match initialization, robot/human batches, total steps and deployment action sampling. Evaluate identical ID/object/scene conditions and report trial counts plus intervals for success and held-out prediction loss. A full-versus-detached policy gap would support representation shaping; detached performance matching the full gain would weaken that explanation. The Flow-only comparison separately tests whether human action labels provide the claimed additional benefit. e-jointe-modalitye-world-losse-traininge-evaluatione-deploy
8.3 Reading coverage
Visual audit: Visually inspected the title/byline, Figures 1–13, Tables 1–4 and all pages supporting retained method, training, numerical, evaluation and reproduction details, including Appendix B hardware/configuration and Appendix D execution/material-shift details. All five final original-PDF crops were inspected; the Table 4 crop retains its dagger explanation. Page 2 and reference pages 10–14 were read in the complete text chunks but were not part of the visual pass. Separate project videos and supplements remain unverified; no code or experiments were executed.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, byline and abstract (p. 1)
- Sections 1–2: Introduction and Related Work (pp. 1–3)
- Sections 3.1–3.2: action alignment and world-level transfer (pp. 3–4)
- Sections 4.1–4.3: architecture, targets, training and inference (pp. 4–6)
- Sections 5.1–5.2: experiments and core findings (pp. 6–8)
- Sections 6–7, acknowledgments and references (pp. 8–14)
- Appendix A.1–A.3: alignment, modality and failure analysis (pp. 15–17)
- Appendix B.1–B.3: implementation and optimization (pp. 17–19)
- Appendix C.1–C.3: collection and evaluation (pp. 19–21)
- Appendix D.1–D.3: RoboTwin transfer (pp. 21–23)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Source extraction caution: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. Addressed here by direct inspection of the original PDF pages and five final crops.
- Separate supplemental material availability has not been fully verified.
- All seven supplied chunks were read, covering the complete 23-page PDF and its embedded supplementary materials. Linked project media, code, datasets and separate supplements were not inspected; no experiments were reproduced.
- Only arXiv:2607.08436v1, dated 8 July 2026, was reviewed. Title and all five authors match the catalog; EgoWAM capitalization is typesetting. The acquisition manifest identifies the earlier HTML as the same v1, but that HTML was not independently compared. No later revision was substituted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, affiliation and arXiv margin
The title matches the supplied observedTitle. Authors are Baoyu Li, Xinchen Yin, Mengying Lin, Yixin Zhang and Danfei Xu, Georgia Institute of Technology; Li and Yin share equal contribution. The margin identifies arXiv:2607.08436v1, 8 July 2026.
Go to primary source ↓e-framingPDF p. 1, abstract and Figure 1; p. 4, Section 4 desiderata
The paper motivates an auxiliary scene-evolution channel for human-to-robot transfer and proposes appearance abstraction, cross-embodiment consistency and ego-motion factoring.
Go to primary source ↓e-actionsPDF p. 3, Section 3.1 and Eq. (1)
Actions use two pose-plus-gripper vectors in a 14-D space; current-device coordinate alignment, 1 s human/1.5 s robot windows and first/99th-percentile normalization precede shared action training.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and Section 4.1; p. 17, Appendix B.1; p. 18, Table 2, backbone/action blocks
The single HPT trunk processes observation and learned action/future tokens. Table 2 specifies ResNet-18 stems, 256/16/8 trunk width/blocks/heads, 64 action and 16 future queries, 100-step chunks, and 50-step action sampling.
Go to primary source ↓e-jointPDF p. 4, Section 3.2, Eq. (2), Figure 2; p. 5, Section 4.3, Eq. (3)
Both heads condition on shared z; the joint density factorizes into world and action readouts. Human and robot action/world losses sum with lambda=1; Figure 2 routes both loss gradients into the trunk.
Go to primary source ↓e-targetsPDF p. 5, Section 4.2; p. 18, Table 2 and Appendix B.2; p. 19, RAE DINO Prediction Head
Targets are frozen Wan VAE latents, frozen normalized DINOv2-B patch features, or stabilized flow. Pixel uses a 6-block/384-wide decoder; Pixel-PT uses a 30-layer/1536-wide pretrained 1.3B decoder. DINO's wide head is 2048-dimensional and its RGB decoder is only for visualization.
Go to primary source ↓e-flowPDF p. 5, Section 4.2, 3D Flow; p. 19, Appendix B.2, 3D Flow Data Processing and Prediction Head
Track4World tracks are stabilized with Aria VIO. The appendix specifies a 28×40 anchor grid, 100×1120×3 target, 2 mm robot/10 mm human motion thresholds, removal of 20 frames at both ends of human clips, and anchor-conditioned flow matching.
Go to primary source ↓e-objective-caveatPDF p. 5, Section 4.2 Pixel/DINO losses and coordinate transform; pp. 18–19, Appendix B.2
Main-text Pixel/DINO formulas regress noise; appendix prose says v-prediction while its DINO equation still targets noise. Section 4.2 describes world-frame positions before applying a relative camera transform; Appendix B.2 describes re-expression into the current camera frame. These conventions are not reconciled in the supplied text.
Go to primary source ↓e-trainingPDF p. 18, Table 2 optimization block; p. 19, Appendix B.3 Training
Training uses 32 robot plus 32 human samples, AdamW lr/weight decay 1e-4, cosine T_max=1400 and eta_min=1e-5, bf16, and lambda=1. Non-PT methods use one L40S for 2000×100 steps; Pixel-PT uses two L40S for 1000×100 steps; approximately two days per task/method are reported.
Go to primary source ↓e-deployPDF p. 6, Section 4.3 Action-Only Inference; p. 19, Appendix B.3 Inference
The world head is detached for deployment; trunk/action decoding uses one current observation and runs at a reported 30 Hz on one RTX 4090.
Go to primary source ↓e-dataPDF p. 20, Appendix C.1–C.2, Figure 13 and Table 3; p. 21, EgoVerse Dataset
Robot demonstration counts are cup 300, fold 360, bag 300; robot hours 2.5/3/2.5, in-domain human hours 2 each, EgoVerse hours 20.5/21/7. Aria provides human hand/head poses; robot imagery includes ego and wrist cameras.
Go to primary source ↓e-evaluationPDF pp. 7–8, Section 5.1 Evaluation Protocol; p. 21, Appendix C.3
The main evaluation uses 20 ID and 20 OOD rollouts (10 unseen objects, 10 novel scenes), totaling 1800. Initial conditions are common across methods. Cup and fold score three stages; bag scores opening plus three ordered insertions. Success means full task completion.
Go to primary source ↓e-mainPDF p. 6, Figure 3, cup-on-saucer ID success and fold-clothes OOD success panels, legend and caption
Cup ID success: Flow robot-only 70%, EgoVerse 90%; BC EgoVerse 15%. Fold OOD: DINO robot-only 15%, EgoVerse 85%, with +70 annotated. Bars distinguish robot-only, in-domain human and EgoVerse. Caption identifies 95% finite-sample-valid confidence intervals.
Go to primary source ↓e-unalignedPDF p. 8, Figure 8 and Section 5.2 Q3; p. 20, Appendix C.2 Unaligned
Natural-to-deliberately-unaligned bag demonstrations change BC success 45%→20% and Flow 80%→75%; robot-only horizontal references are 40% and 60%. Human grasp strategies are deliberately made incompatible with the gripper.
Go to primary source ↓e-alignmentPDF pp. 15–16, Appendix A.1, Figure 9 and Figure 10; pp. 20–21, Appendix C.2 Aligned
Cup natural-to-aligned success is BC 15→35, Pixel 35→65, DINO 50→70, Flow 85→85 percent; BC robot-only is 25%. Collection aligns motion/grasp strategy and camera height/viewpoint together.
Go to primary source ↓e-modalityPDF p. 16, Appendix A.2 and Figure 11, both panels
For ID/object/scene, action-only success is 15/11/0%, Flow-only 35/44/10%, joint 90/78/30%. Normalized scores are .30/.26/.00, .43/.48/.13, and .93/.89/.43 respectively. No separate ablation counts or uncertainty are printed.
Go to primary source ↓e-world-lossPDF p. 16, Table 1, Human and Robot rows, and Appendix A.2
Flow-only versus action-plus-flow prediction loss is 0.23 versus 0.22 on the human stream and 0.20 versus 0.19 on the robot stream. The authors interpret this as complementary action context; the table does not define a held-out split or uncertainty.
Go to primary source ↓e-pixel-failurePDF pp. 16–17, Appendix A.3; p. 17, Figure 12
Pixel-PT robot-only predicts an already-open bag and is associated with skipped opening. Scratch Pixel is blurrier but preserves bag state; Pixel-PT with EgoVerse is described as sharper and faithful. Figure 12 illustrates predictions, not a deployed world-head computation.
Go to primary source ↓e-limitationsPDF p. 9, Section 7
Authors limit gains to context generalization, state that novel skills remain out of reach, use one policy per task, and leave multi-task scaling and the best world representation open.
Go to primary source ↓e-sim-protocolPDF pp. 21–22, Appendix D.1–D.2
RoboTwin 2.0/SAPIEN experiments use demo_clean-50, aloha-agilex as primary and arx-x5/franka/ur5 for cross-training. A common 14-D camera-frame end-effector interface feeds receding-horizon differential IK. Methods share 100 held-out seeds numbered at least 100000.
Go to primary source ↓e-sim-resultsPDF p. 22, Table 4 and Appendix D.3; p. 23, concluding precision limitation
Bottle pick: ACT-EE 2, DP-EE 5; BC s/c 2/6, Pixel 7/11, DINO 4/28, Flow 0/16 percent. Stacking: only cross DINO 8 and cross Flow 16 are nonzero; the dagger marks a bowl material shift. Hanging-mug: only cross Pixel is nonzero at 1%; authors identify a precision bottleneck.
Go to primary source ↓8.5 Primary sources
EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data ↗
PDF · 12,340 extracted words
Source fingerprint
dcda12fc4e871509ff95847ed92e64cfc71aaf021c3bf17e14312259e2229443