World Models for Learning Dexterous Hand-Object Interactions from Human Videos
1. Paper overview
In one sentence: DexWM learns finger-conditioned latent dynamics from human videos and uses them for robot trajectory search, but reliable manipulation still depends on simulation adaptation, planning initialization and contact handling. e02e03e04e06e07e11e15e19
| At a glance | What to know |
|---|---|
| Research problem | Source description Coarse camera, wrist or text actions omit finger articulation, while large dexterous robot datasets are scarce. The paper asks whether egocentric human videos and non-dexterous robot data can teach a state-transition model precise enough for image-goal manipulation. Its dynamics include agent movement, object response and camera appearance. e02e03 |
| Core mechanism | |
| A key reported result | Real-world image-goal grasping: 10/12 successes, reported as 83%. Manually judged secure grasp success. Franka Panda with Allegro; 12 trials using four objects; no real-world fine-tuning; two-step open-loop execution. Diffusion Policy and DexWM without pretraining: 0% each. A physical transfer demonstration with a small sample, not a measured real-world reaching/placing result. No uncertainty interval is supplied. e11e15e17e18 |
| Reading caution | Source description The model assumes deterministic, static scenes without external agents. Image goals are demonstrated; text goals and eliminating exploratory simulation adaptation remain future directions. e13 |
Core contributions
Figure 2. The learned transition predicts visual features; its two output branches make those features visible and geometrically testable. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the two observation images. Their encoder boxes produce state tokens; the blue freeze marks agree with the caption’s frozen DINOv2 description. The extra token group is initialized from the latest state, according to Section 3.2. Hand actions and camera changes condition the predictor; the detailed implementation uses AdaLN, so these schematic arrows should not be read as a complete attention specification. At the top, predicted state tokens branch left into image reconstruction and right into keypoint heatmaps. Appendix C.1 clarifies that the decoder is trained separately and used only for visualization. e03e05e06e14e16
What it supports. DexWM’s main learned output is a future latent scene, rather than a robot command. The hand head supplies a local training signal that global feature regression can underweight. Keeping these branches distinct explains why visually plausible reconstructed frames and accurate hand geometry require separate evaluation.
Where the evidence stops. The schematic contains no full attention-mask or temporal-target specification. Appendix C.1’s eight preceding frames and prediction indices through i=9 need clarification. Reconstructed images also depend on a separately trained decoder.
2. Motivation
2.1 The problem and the proposed response
Coarse camera, wrist or text actions omit finger articulation, while large dexterous robot datasets are scarce. The paper asks whether egocentric human videos and non-dexterous robot data can teach a state-transition model precise enough for image-goal manipulation. Its dynamics include agent movement, object response and camera appearance. e02e03
2.2 What this reading follows
DexWM tackles a practical mismatch: abundant human manipulation video and scarce dexterous robot experience. It encodes images into frozen DINOv2 features, predicts how those features change under detailed hand actions, and adds supervision that keeps fingertips and wrists recoverable. A separate optimizer then searches robot joint trajectories toward an image goal. This reading follows the path from representation to physical execution, keeping the auxiliary learning objective distinct from the planning cost. The results show meaningful transfer after exploratory simulation fine-tuning, while the appendix exposes open-loop real deployment, initialization assistance, expensive planning and frequent object drops. e02e03e04e06e07e11e15e19
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants classification fits a forward world model coupled to external CEM and low-level control. Actions condition state prediction and are searched at inference; this is neither joint action/future generation nor inverse dynamics. Joint state/heatmap training does not make it an integrated action policy. e03e05e06e07e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn motion into a shared action interface
DexWM separates what the camera sees from what the hand does. The visual state is a grid of frozen DINOv2 features; the action specifies changes in three-dimensional hand keypoints plus camera motion. Before differencing, future keypoints are transformed into the earlier camera frame, so camera movement is not inadvertently interpreted as finger articulation. This common representation permits training on human video and parallel-gripper robot data, but the compatibility is engineered: DROID receives synthetic hand points, and Allegro repeats one finger to fill the human five-finger layout. The predictor then learns the consequences of supplied actions. During robot planning, forward kinematics converts candidate joint configurations into this same action interface; the model does not generate motor commands directly. e03e04e05e15e17
Figure 4. Forward prediction becomes control through an external search over robot joint waypoints. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the green start image and follow its encoded state into the predictor. Above the predictor, CEM’s joint-angle candidates pass through G, the robot’s forward-kinematics mapping, to form candidate actions. The curved arrow represents repeated world-model prediction, not physical feedback by itself. Below, the terminal prediction is compared with the red goal image’s encoded state through cost C. Section 3.4 specifies both latent and keypoint distances, plus a grasp-orientation term. Appendix B supplies the missing execution step: a low-level controller interpolates the selected joint targets, with different replanning rules for simulation and the real robot. e07e15
What it supports. The model can evaluate candidate futures without directly learning a goal-conditioned action policy. Simulation uses three-step receding-horizon planning. The reported physical grasping experiment instead optimizes two steps and executes them open loop. Those are distinct deployment protocols, despite sharing the same forward model and search framework.
Where the evidence stops. The cost arrows are consistent with Eq. (6), but the diagram omits controller details and grasp initialization. Appendix B uses a teleoperated dummy grasp to seed search and reports substantial planning latency; the diagram alone cannot establish autonomous real-time control.
5.2 Keep a small hand region visible to the objective
The source’s motivation is spatial: hands occupy only a small portion of an image, so minimizing average feature error can leave their configuration inaccurate. HC adds fingertip and wrist heatmap prediction from the future latent state. Reader interpretation: this pressures the representation to retain information useful for localized control, rather than merely improving its overall visual match. Table 2 supports that interpretation with a large PCK gain. Table 3 supplies a complementary warning: PEVA* has lower embedding error than DexWM but worse PCK. Neither result makes heatmap accuracy equivalent to manipulation competence. At deployment, keypoint distance also enters the planning objective, giving the hand head a second role whose causal contribution is not isolated by the training-loss ablation alone. e06e07e09e10
5.3 Read zero-shot transfer through the complete execution chain
The transfer claim becomes clearer when training, search and execution are traced separately. Human and DROID pretraining is followed by exploratory RoboCasa fine-tuning; no real-world fine-tuning is used. CEM then evaluates candidate joint trajectories using the learned dynamics and a goal cost. The appendix specifies a dummy-grasp initialization, low-level interpolation, simulation replanning and open-loop physical execution. Reader interpretation: the successful transfer belongs to this complete pipeline, not to an unaided predictor. Table 4 shows strong gains under its evaluation, but simulation grasp success uses distance plus contact, whereas physical success is judged secure holding. Table 6 shows why that distinction matters: holding an object through placement remains difficult even when reaching and grasping scores improve substantially. e11e15e16e17e18e19
5.4 Training and inference
During training
Pretraining uses 829 hours of EgoDex and about 100 hours of DROID; EgoDex retains its original approximately 1% test split. Adam trains for 40 epochs, batch 4096, with learning rate decreasing from 1e-4 to 1e-7. Context frames are sampled nonuniformly within four seconds. e16e17
Fine-tuning uses approximately four hours of exploratory RoboCasa data for 50 epochs, batch eight, learning rate 1e-5, including multistep prediction. The default collection perturbs Lift trajectories and randomizes objects to avoid successful grasps. Fully programmatic random targets yield 49% mean simulation success versus 53% for the default. e11e16e17
During inference
CEM optimizes 23 robot joints through forward kinematics and latent rollouts. The objective combines terminal feature distance and heatmap-derived keypoint distance, weighted by 0.001, plus an orientation cost for grasping. Stationary-camera planning excludes camera-motion components. e07e15
Simulation uses horizon three, 512 candidates, ten elites and ten CEM iterations with receding-horizon feedback. Physical grasping uses horizon two and 256 candidates, retaining ten elites/iterations, and executes both waypoints open loop. Controllers interpolate joint targets. A teleoperated dummy grasp initializes the grasp-search mean. e15
5.5 Implementation flow
- Encode visual state
Frozen DINOv2-L maps 224 × 392 images to 448 patch vectors of dimension 1024. The default XL predictor has 456M parameters and 32 conditional transformer blocks. e03e14
- Align actions across embodiments
Express both hands’ 21 keypoints in the earlier camera frame before differencing; append camera translation and Euler-angle changes, giving 132 components. DROID uses concentric-circle dummy keypoints. Allegro duplicates its last finger for the human pinky; unused left-hand actions are zero. e04e17
- Regress future features
The deterministic CDiT-based predictor uses AdaLN action conditioning, initializes future tokens from the latest state and directly regresses features. Autoregressive reuse produces longer rollouts without diffusion denoising. e05
- Recover local hand geometry
A transformer head predicts 12 image heatmaps: ten fingertips and two wrists. Two-pixel Gaussian targets supervise this head and the predicted state. A separately trained image decoder serves visualization, not planning. e06e14e16
6. Experiments & results
DexWM learns an action-conditioned transition model in frozen visual features, using human hand motion to support dexterous robot planning. A hand-consistency objective preserves local finger information; CEM searches robot joint trajectories through the learned model. Simulation and small-scale physical grasping results support transfer after exploratory simulation fine-tuning, with substantial remaining placement failures and a strict distinction between predicted rollouts and executed actions.
6.1 Read the original evidence
Table 4. Transfer gains are large in the tested setup, with different protocols behind simulation and physical success. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the three RoboCasa columns from the single Real Robot column. Within each column, compare the pretrained DexWM row with both Diffusion Policy and DexWM without pretraining. Every entry is a success percentage, not a prediction metric. Section 4.5 reports 50 simulation trials for each task and 12 physical grasp trials, while Appendix C specifies four physical objects. Appendix B further distinguishes simulation MPC from two-step open-loop physical execution. The physical 83 therefore means ten successful grasps out of twelve, whereas simulation grasping uses distance and contact criteria. There are no physical reach or place columns. e11e15e17e18e22
What it supports. DexWM reaches 72%/28%/58% on simulation reach/place/grasp and 83% on physical grasping. It exceeds both listed baselines on every column. The results support transfer after human-plus-DROID pretraining and exploratory robot adaptation, while placement remains difficult and the physical evidence is confined to a small grasping study.
Where the evidence stops. No uncertainty intervals accompany the table. Removing pretraining removes both data sources, so this ablation does not isolate human videos. The abstract’s “over 50%” phrase should not be read as a greater-than-50-percentage-point mean simulation gain.
Figure 16. Physical execution examples show both successful grasps and a contact-induced miss. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each row from the green start frame through the execution snapshots to the blue reached frame; the red frame at the right is the goal reference. Black circles in the starting frames identify the target objects. The upper three rows are labeled successful, while the bottom row is a failure. The caption explains that the gripper collides with an upside-down bowl and moves it away before securing a grasp. It also notes small object displacements after successful grasps in two examples. These sequences depict actual robot execution, unlike the decoded human-video forecasts elsewhere in the paper. e20e11e15e18
What it supports. The figure makes the transfer result concrete without hiding a failure: a reasonable approach can still miss when contact changes the object’s position. Read alongside Table 4, it illustrates what a successful physical grasp and a contact-induced failure look like in this particular setup.
Where the evidence stops. The four pictured trajectories are selected examples, not the full evaluation sample. Success is manually judged secure holding. Since the physical protocol executes an optimized two-step trajectory open loop, the images do not demonstrate recovery through online replanning.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-world image-goal grasping Franka Panda with Allegro; 12 trials using four objects; no real-world fine-tuning; two-step open-loop execution. | 10/12 successes, reported as 83%. Manually judged secure grasp success | Diffusion Policy and DexWM without pretraining: 0% each. A physical transfer demonstration with a small sample, not a measured real-world reaching/placing result. No uncertainty interval is supplied. e11e15e17e18 |
| RoboCasa image-goal reaching, placing and grasping 50 randomized trials per task after exploratory simulation adaptation; MPC execution. | Reach 72; place 28; grasp 58. Task success rate (%) | Diffusion Policy: 16/8/0; DexWM without pretraining: 18/8/14, in the same task order. Human-pretrained DP separately reaches 4% average versus DexWM’s reported 53%. Reach requires mean hand-keypoint distance <15 cm; place object distance <10 cm; grasp wrist–object distance <20 cm plus contact, each sustained for ten steps. These criteria differ from physical secure-grasp judgment. e11e15e18 |
| EgoDex open-loop future prediction Original EgoDex test split; all compared models trained on EgoDex for 40 epochs with DINOv2; 20 frames at 5 Hz over four seconds. | DexWM PCK: 60 at four seconds / 68 average; L2: 0.67 / 0.51. PCK@20 (%) higher is better; embedding L2 error lower is better | PEVA*: PCK 56/63, L2 0.62/0.49; NWM*: PCK 34/48, L2 0.74/0.57. DexWM improves average PCK by five percentage points over PEVA*, while PEVA* has lower feature error. The starred baselines are adaptations with restricted action conditioning, not unchanged original systems. e10e17 |
| Hand-consistency loss ablation Table 2 evaluates EgoDex rollouts, comparing HC disabled/enabled; PCK uses a 20-pixel tolerance. | With HC: PCK 60/69 and L2 0.66/0.50, at four seconds/average. PCK@20 (%) and embedding L2 error | Without HC: PCK 26/52 and L2 0.85/0.61. The four-second gain is 34 percentage points, despite the prose calling it 34%. Table 2’s enabled row differs from the EgoDex-only row in Table 3; do not merge the configurations. e09e10 |
| Human-data contribution to RoboCasa prediction Zero-shot open-loop evaluation on RoboCasa Lift sequences, before exploratory adaptation. | EgoDex+DROID: L2 0.79/0.57; PCK 7/17. Embedding L2 and PCK@20 at four seconds / average | DROID only: L2 1.30/0.96; PCK 2/12. Human data improves cross-embodiment prediction, but absolute zero-shot hand accuracy remains low. Adding data changes both source diversity and training volume. e08e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Local heatmap supervision improves both reported feature prediction and hand-keypoint accuracy. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the cross and check rows vertically, keeping the four-second endpoint separate from the average over the rollout. Embedding error decreases; PCK@20 increases and counts predicted hand keypoints within a 20-pixel radius. The endpoint PCK changes from 26 to 60, while the average changes from 52 to 69. These are percentage-valued scores, so their differences are percentage points. The feature-error columns also improve, from 0.85 to 0.66 at the endpoint. Read this together with the loss definition on page 5: HC supervises heatmaps recovered from predicted latent states, rather than adding pixel reconstruction to the world-model objective. e06e09e10
What it supports. The cleanest mechanism evidence is the 34-percentage-point endpoint PCK improvement when the auxiliary loss is enabled. This supports the claim that scene-wide latent prediction alone loses useful local hand information. It does not, by itself, measure whether the object stays grasped during a robot trajectory.
Where the evidence stops. The prose calls the endpoint change 34%; the table supports 34 percentage points. Its enabled row also differs from Table 3’s EgoDex-only result. Table 2 does not explicitly identify its training mixture, so the two rows should not be silently equated.
Table 6. Different tasks fail at different parts of the manipulation pipeline. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task row horizontally before comparing tasks. Appendix C.6 defines Contact as complex interaction failures, Drop as losing the object before its destination, Last cm as small final-position inaccuracies, and State Drift as accumulating world-model error. The row totals are 28, 42 and 72, matching the complements of Table 4’s reach, grasp and place success rates. Accordingly, the entries are best read as contributions in percentage points of evaluated trials, not as percentages normalized within failed trials. Placement’s 70 under Drop is therefore a very different diagnosis from reaching’s 24 under Last cm. e11e18e19
What it supports. The dominant placement issue is retaining the object: dropping accounts for 70 of its 72 failure points. Grasping divides much of its failure between final-position accuracy and state drift. Reader interpretation: improving a global prediction score alone may leave the main placement bottleneck untouched.
Where the evidence stops. This is a categorized failure analysis, not a causal ablation of fixes. The paper does not provide uncertainty or an independent labeling-reliability analysis for the categories. Their numerical breakdown concerns simulation and should not be assigned to the twelve physical trials.
7. Analysis & limitations
7.1 What the evidence leaves open
The model assumes deterministic, static scenes without external agents. Image goals are demonstrated; text goals and eliminating exploratory simulation adaptation remain future directions. e13
Table 6 attributes 70 percentage points of place trials to dropping the object, versus 72 total failure points. Grasp failures include 20 points of final-centimeter error and 18 of state drift. Strong hand localization does not establish reliable sustained contact. e19
The pretraining ablation removes both human and DROID pretraining, so Table 4 alone does not isolate human data. Qualitative counterfactuals are selected decoded predictions, without a controlled physical counterfactual accuracy benchmark. e11e12e17
7.2 Questions for discussion
- Does hand-consistency supervision improve contact prediction beyond image-space keypoint accuracy?
- How much of grasp success depends on the dummy-grasp initializer?
8. Reproducibility audit
8.1 Requirements and known gaps
Required assets include DINOv2-L, annotated/calibrated EgoDex and DROID, exploratory RoboCasa trajectories, robot kinematics and the CEM initializer. Simulation planning is parallelized across eight H100 GPUs; reported planning time is 38–168 seconds per episode across tested settings, with 168 seconds the default. This is not a training-compute specification. e14e15e17
Clarify C.1’s eight-context-frame description versus target indices through i=9 before implementation. The supplied details leave the EgoDex 25-to-21-keypoint mapping, full grasp-orientation penalty, controller gains, software versions, seeds and training compute unspecified. Proposed checks should separate HC supervision from planning cost and test initialization dependence. e04e07e15e16e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate learned hand information from the hand-based planning cost
Reader-proposed check, not performed: train matched HC-enabled and HC-disabled predictors with the same data, encoder, schedule and random seeds. Fit an identical heatmap readout to detached features of each checkpoint so every condition can use a hand cost without introducing extra predictor supervision. Evaluate both predictors with and without the keypoint planning term, holding CEM budget and initialization fixed. Report EgoDex PCK and feature error plus paired RoboCasa success and drop rates. If HC improves held-out localization but not matched planning success, the claim that its gains explain control performance would weaken; a consistent success gain without the hand cost would support a representation-level effect. e06e07e09e15e16e18e19
Check 2: Measure dependence on the dummy-grasp initializer
Reader-proposed check, not performed: freeze the pretrained, simulation-adapted model and replay matched RoboCasa grasp scenes under the reported dummy-grasp mean, the starting-joint mean, and a task-independent feasible initialization. Keep covariance, controller, goal cost, sample count and iterations fixed, and repeat CEM seeds. Record success, final-centimeter errors, state drift and planning time; add a dummy-trajectory replay control without model-based refinement. A large collapse under neutral initialization, or little gain over direct replay, would limit attribution to learned dynamics. Comparable success across initializers would strengthen that attribution. Begin in simulation; do not mix these outcomes with the paper’s open-loop physical protocol. e07e11e15e18e19
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, every main-text figure and table, method equations, limitations, appendix architecture/training/planning/data/success definitions and all appendix figures. This covers Figures 1–17 and Tables 1–6. All six final original crops were separately viewed and retain their labels and table headers; narrow figures/tables were rendered at 400 DPI. Figure 2’s action arrows were checked against the AdaLN description and frozen-encoder caption; Figure 4’s flow was checked against Eq. (6) and Appendix B. The schematic omits physical feedback details, which are supplied in the explanation. Text reading covers the entire PDF; reference-list pages 11–14 were not visually inspected. No separate supplement, code, online project material or original earlier revision was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract
- 1 Introduction
- 2 Related Work
- 3 Proposed Methodology
- 3.1 State and Action Representation
- 3.2 Predictor
- 3.3 Hand Consistency Training Loss
- 3.4 Robot Task Planning
- 4 Experiments and Results
- 4.1 Datasets
- 4.2 Ablation Studies
- 4.3 Baselines
- 4.4 Open-Loop Trajectory Evaluation
- 4.5 Human Video To Robot Transfer
- 5 Limitations
- 6 Conclusion
- 7 Acknowledgement
- References
- Appendix
- A DexWM Architecture
- A.1 Encoder
- A.2 Predictor
- A.3 Keypoint Predictor
- A.4 Decoder
- B Planning Optimization
- C Implementation Details
- C.1 Training Details
- C.2 Datasets and Robotic Benchmarks
- C.3 Baselines
- C.4 Success Criteria for Simulation Tasks
- C.5 Success Criteria for Real-World Grasping
- C.6 Manipulation Failure Cases Breakdown
- D Additional Visualizations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected title page identifies arXiv:2512.13644v2 [cs.RO], 16 March 2026. Its title and all nine authors match the catalog. The catalog records submission on 15 December 2025; this report reviews the supplied v2, not the original submission. No earlier version or revision comparison was supplied.
- The acquisition noted that text extraction does not reconstruct figure images. This gap was resolved by visually inspecting the retained PDF: all 17 figures and all six tables, including appendix visuals, were reviewed.
- Separate supplemental material availability has not been fully verified; none was supplied.
- All seven supplied text chunks were read, covering the complete 23-page PDF including references and appendices A–D. Reference-list pages 11–14 were read as text; they were not visually rendered.
- Project links and code were not inspected, external source claims were not independently verified, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Exact catalog title; nine authors in catalog order; affiliations FAIR at Meta and New York University; arXiv:2512.13644v2 [cs.RO], 16 Mar 2026.
Go to primary source ↓e02PDF p. 2, Section 1, problem motivation and dynamics definition
Coarse action spaces and scarce dexterous robot data motivate human-video training; dynamics includes agent motion, environment response and camera appearance.
Go to primary source ↓e03PDF p. 3, Figure 2 and Section 3 problem formulation
Frozen image encoders supply latent histories to the action-conditioned predictor; its outputs feed the decoder and keypoint predictor.
Go to primary source ↓e04PDF p. 4, Section 3.1, Figure 3 and Eqs. (1)–(3)
Each hand has 21 3D keypoints; keypoint differences are aligned to the earlier camera frame and supplemented by translation/Euler-angle changes. The action is 44 × 3. DROID uses dummy keypoints; Allegro repeats its last finger.
Go to primary source ↓e05PDF p. 5, Section 3.2, Predictor and Multistep Prediction
Deterministic CDiT with AdaLN conditioning directly regresses DINOv2 features. Future tokens start from the latest state; predicted states are fed back autoregressively.
Go to primary source ↓e06PDF p. 5, Section 3.3, Eqs. (4)–(5) and training paragraph
State MSE is supplemented by 12 heatmap MSE targets for wrists/fingertips, weighted by lambda=100, with the image encoder frozen.
Go to primary source ↓e07PDF pp. 5–6, Section 3.4, Figure 4, Eq. (6) and Planning Cost
CEM optimizes joint angles via forward kinematics and predicted terminal state. Cost combines latent L2 with keypoint-pixel distance weighted 0.001; grasping adds an orientation cost.
Go to primary source ↓e08PDF p. 6, Table 1, RoboCasa columns and Section 4.2 Human Video
DROID-only RoboCasa L2 is 1.3/0.96 and PCK 2/12; EgoDex+DROID yields 0.79/0.57 and 7/17, respectively at four seconds/average. Evaluation uses Lift.
Go to primary source ↓e09PDF p. 7, Table 2, both HC rows and Section 4.2 Hand Consistency Loss
HC off/on: four-second PCK 26/60, average 52/69; four-second L2 0.85/0.66, average 0.61/0.50. The prose describes the PCK change as 34%.
Go to primary source ↓e10PDF p. 7, Table 3 and Sections 4.3–4.4; p. 8, Results
Table 3 compares EgoDex-only training for 40 epochs using DINOv2, with four-second 20-frame rollouts and shared keypoint evaluation. DexWM PCK 60/68, L2 0.67/0.51; PEVA* 56/63, 0.62/0.49; NWM* 34/48, 0.74/0.57. Starred models restrict conditioning.
Go to primary source ↓e11PDF p. 9, Section 4.5, Table 4 and surrounding training/evaluation/results paragraphs
Table 4 reports simulation reach/place/grasp and physical grasp percentages: DP 16/8/0/0; no-pretraining DexWM 18/8/14/0; DexWM 72/28/58/83. Physical success is 10/12. Random-target versus Lift-initialized exploration gives 49% versus 53% average; human-pretrained DP gives 4%.
Go to primary source ↓e12PDF p. 8, Figures 8–9 and comparison/controllability paragraphs; pp. 20–21, Figures 13–14
Selected decoded rollouts compare text conditioning and structured hand actions. Atomic right-hand actions move 1 cm per frame; examples depict cup displacement on contact. These are qualitative predictions.
Go to primary source ↓e13PDF p. 10, Section 5 Limitations
Only image goals are demonstrated; approximately four hours of exploratory simulation remain necessary. Static scenes without external agents are assumed; stochastic extensions are future work.
Go to primary source ↓e14PDF p. 15, Appendix A.1–A.4 and Table 5
DINOv2-L uses 14-pixel patches, d=1024, 224 × 392 inputs and 448 patches. XL has 456M parameters, 32 blocks and 16 heads. The heatmap head has six transformer blocks and 2-pixel Gaussian targets. A ViT-L decoder is separately trained for visualization.
Go to primary source ↓e15PDF pp. 15–16, Appendix B, Eq. (7), Figure 11 and implementation paragraphs
CEM searches 23 joints. Simulation T/N/K/L=3/512/10/10 with replanning; real robot 2/256/10/10 without replanning. Controllers interpolate joints. Camera motion is excluded. Simulation uses eight H100 GPUs; tested planning times span 38–168 seconds, default 168. Grasp initialization uses a teleoperated dummy sequence.
Go to primary source ↓e16PDF p. 16, Appendix C.1 Training Details
Pretraining: Adam, batch 4096, 40 epochs, learning rate 1e-4 to 1e-7. Eight preceding frames are selected within four seconds, while target indices are stated as i=1,...,9. Fine-tuning: 50 epochs, batch 8, learning rate 1e-5, multistep prediction. Decoder training is separate.
Go to primary source ↓e17PDF p. 17, Appendix C.2 and C.3
EgoDex has 829 hours and 25 annotated keypoints per hand with approximately 1% held out; about 100 DROID hours are used. Exploratory data derives from noisy Lift actions plus gripper-only sequences. Only the right arm acts. Real evaluation uses four objects over 12 trials. NWM*/PEVA* are implemented within the authors’ framework.
Go to primary source ↓e18PDF p. 18, Appendix C.3–C.5
DP uses pooled DINOv2 observations, context two, action chunk nine and future-frame goals. Simulation reach/place thresholds are 15/10 cm; grasp requires wrist distance under 20 cm and contact, sustained for ten steps. Real grasp success is manually judged secure holding.
Go to primary source ↓e19PDF pp. 18–19, Appendix C.6 and Table 6
Failure percentages by contact/drop/last cm/state drift/total: reach 0/0/24/4/28; grasp 4/0/20/18/42; place 0/70/0/2/72. Text defines the four categories.
Go to primary source ↓e20PDF p. 22, Figure 16 and caption
Three physical grasp successes and one failure are illustrated. Green/red/blue borders identify start/goal/reached frames. The failure is an upside-down bowl displaced by gripper collision.
Go to primary source ↓e21PDF pp. 6–7, Section 4.2, Figures 5–6 and encoder discussion
Predictor scaling improves reported feature error and PCK. Encoder comparisons use simulation success because feature scales differ; dimensions are aligned by projections when needed. Performance varies by task.
Go to primary source ↓e22PDF p. 1, abstract; p. 9, Table 4
The abstract says over 50% average improvement over Diffusion Policy. Reader arithmetic from Table 4’s simulation rows gives mean success of 52.67% for DexWM and 8% for DP; the report uses task-specific values rather than interpreting the abstract phrase as a percentage-point gain.
Go to primary source ↓8.5 Primary sources
World Models for Learning Dexterous Hand-Object Interactions from Human Videos ↗
PDF · 11,370 extracted words
Source fingerprint
caa494829e49b893346ae4fcedbe462e99f197192c7e84d27405a2776723a390