Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers
1. Paper overview
In one sentence: GALATEA preserves hand and object motion from generated video plans, then learns physical execution through simulation tracking, trading a structured, reusable controller for dependence on reconstruction quality and reference-following recovery. e02e04e09e13e17e18
| At a glance | What to know |
|---|---|
| Research problem | Source description Generated videos describe desired motion without robot actions, forces or guaranteed physical feasibility. GALATEA asks whether sufficiently reliable hand–object reconstruction and simulation tracking can convert diverse, noisy visual plans into dexterous execution with less manual reference curation. e02e03 |
| Core mechanism | |
| A key reported result | Real-world tracking of unseen video plans: Jar neck 6/10; jar top 6/10; mug rim 7/10; mug handle 8/10; overall 27/40. Successful trials. Distilled policy; four tasks, ten trials each; same completion thresholds, with FoundationPose object estimates and FK hand keypoints. No real-world baseline comparison is reported. Supports these physical tracking behaviors; small per-task samples and perception-based scoring limit broader conclusions. e16e17 |
| Reading caution | Source description Palm-friction lifts and near-table grasps of thin objects can be difficult for the robot despite plausible videos. Contact transitions expose sim-to-real mismatch, and recovery often fails once the reference is lost. In-hand manipulation is not systematically covered. e18 |
Core contributions
Figure 2. Generated videos supply both training references and deployment plans; a separate controller supplies executable actions. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the video-generation block in the upper left. Image and instruction inputs produce videos, which the reconstruction block converts into hand–object interaction (HOI) references. Follow the solid black arrows to RL Grounding: this is the training route that produces the Tracking Controller. Then follow the dashed gray arrows, identified as inference in the lower legend. A newly reconstructed reference goes directly to the learned controller, whose execution arrow points toward the real world. The two routes share a geometric interface, but use it differently: one learns how to act from simulation interaction, while the other asks the resulting policy to follow a plan. e02e06e09e16
What it supports. The useful separation is between specifying an interaction and learning the robot actions needed to realize it. Figure 2 supports a modular architecture with a reusable tracking layer. The source’s distillation stage explains how multiple object-category experts become the single controller shown in the deployment path.
Where the evidence stops. The schematic does not establish continuous video replanning or joint training of the generator and controller. Real-world closed-loop operation is described through object-pose feedback to the tracker; generation and reconstruction are separate stages.
2. Motivation
2.1 The problem and the proposed response
Generated videos describe desired motion without robot actions, forces or guaranteed physical feasibility. GALATEA asks whether sufficiently reliable hand–object reconstruction and simulation tracking can convert diverse, noisy visual plans into dexterous execution with less manual reference curation. e02e03
2.2 What this reading follows
A video can specify where an object should go and how fingers should interact with it, yet still leave the robot’s actions unknown. GALATEA builds an explicit bridge: reconstruct the hand and object together, use those trajectories to train tracking experts in simulation, and distill the experts into one feedback controller. Generated videos then serve as new plans for that controller. Read the evidence in three layers: reconstruction measures whether the reference is geometrically useful, simulation tests whether the robot can track it, and physical trials test transfer. The final layer reports 27 successful executions out of 40 unseen plans, with substantial limits on task breadth and recovery. e02e04e09e13e17e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is unassigned, so no recorded quadrant can be confirmed. Reader assessment: a modular video-planner, geometric-reconstruction and RL-controller architecture. The policy is reference-conditioned tracking, rather than joint future/action prediction or a learned inverse-dynamics decoder. No One Model architecture is established; the paper remains relevant to video-grounded action without forcing it into a quadrant. e02e04e06e09
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate absolute scale from relative contact
A visually plausible hand and a visually plausible object need not occupy a consistent 3-D scene. GALATEA therefore tackles two different ambiguities. First-frame stereo depth anchors MoGe-2 predictions to metric scale through background alignment, helping the object tracker place a known-size mesh. Joint optimization then reconciles the hand with that object using image evidence and contact intervals. It preserves initial object projections strongly because the authors find them relatively reliable, while allowing rigid hand corrections without changing finger articulation. Table I helps distinguish the effects: removing depth alignment especially harms object alignment on HO-Cap, whereas removing joint optimization strongly worsens relative and contact geometry. Reader interpretation: this division makes the reconstruction easier to diagnose, but the CDev tradeoff warns against treating any one aggregate rank as proof of a universally better reference. e03e04e11
Figure 3. Metric anchoring and joint refinement reconcile independently estimated hand and object motion. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper dashed boxes as inputs. Generated video fans out to depth, segmentation, hand-motion and contact-interval estimates. The separate first-frame-depth arrow enters Depth alignment, where the text specifies a scale-and-offset correction against a static background. The corrected depth and SAM2 masks feed FoundationPose; its object poses join WiLoR hand motion and HOI-DETR contact intervals at Joint optimization. Equations (1)–(2) explain what the gray block does beyond these arrows: it reconciles object projection, hand keypoints, visible silhouettes, contact proximity and temporal smoothness. Calibrated intrinsics and a metric object mesh are additional inputs specified in the method text. e03e04
What it supports. The diagram makes reconstruction a coupled geometry problem. Independent hand and object estimates are starting points; contact intervals help determine when their surfaces should meet. Joint refinement applies rigid corrections while retaining finger articulation, so it repairs relative placement without freely redesigning the generated grasp.
Where the evidence stops. Metric recovery assumes a fixed camera, static background, calibrated intrinsics and known object dimensions. Contact detection and retained finger articulation can still be wrong; the optimization is not evidence that every generated interaction is physically feasible.
5.2 Reward interaction without locking in noisy finger targets
The tracker receives current hand–object errors and shape information, but no retargeted hand-joint target. Its reward combines reference imitation with explicit contact and lift incentives. Object tracking becomes active when either the reference or rollout has contact. Dense hand tracking is suppressed when the reference has lifted the object beyond 5 cm but the rollout has not, preventing that reward component from remaining active during a missed lift. Augmentation also respects the contact boundary: approach perturbations vanish toward first contact, while later perturbations move hand and object together. SAPG supplies varied exploration across PPO agents. These are training mechanisms; at execution the MLP emits arm deltas and absolute hand targets with smoothing. The PPO and no-lift ablations test parts of this recipe, with success decreases on ten and seven benchmark objects respectively. e05e06e07e08e13e14
5.3 Follow what survives expert distillation into deployment
The unified controller is produced through an intermediate population of experts, not by an unspecified single training run over all references. Each object category has at most two experts, and behavior cloning followed by DAgger combines their behavior. The evaluation then changes one kind of novelty at a time: unseen reference trajectories for benchmark objects, and a separate five-object novel set in simulation. Physical testing adds perception and actuation mismatch, using the D455 for object feedback while the image-conditioning D435 is removed. Figure 7 illustrates preserved interaction choices; Table III quantifies completion for four tasks. Reader interpretation: these results support a reusable low-level execution interface, while the documented tendency to fail after leaving the reference limits any claim of general goal-directed recovery or systematic in-hand manipulation. e09e13e14e16e17e18
5.4 Training and inference
During training
PhysX in Isaac Gym simulates the Sharpa Wave Hand and FR3 arm. SAPG splits rollouts among PPO agents with different exploration settings and aggregates experience. Contact/lift shaping supplements imitation. PD gains, mass, friction, latency and object-pose observations are randomized; sampled-frame starts reset fingers open to avoid penetration. e07e08e16
At most two experts per object category cover about 40 source trajectories each, roughly 200 augmented. Training is direct or multi-object followed by category fine-tuning. Behavior cloning then DAgger distills the experts. The PDF describes no joint training of the video generator with the controller. e09e02
During inference
A newly generated plan is reconstructed and supplied to the learned tracker. Arm actions are 0.03-rad-scaled joint deltas; hand actions map to absolute joint-limit targets. EMA coefficients are 0.20/0.10 for arm/hand, followed by 30-Hz position PD control. These are robot control outputs, distinct from predicted video frames. e02e06
A RealSense D455 and FoundationPose provide online object feedback. The D435 used for the conditioning image is removed before execution. Hand–eye calibration is supplemented with a shared object-translation bias estimated while the robot rigidly grasps an object. e16
5.5 Implementation flow
- Generate controlled references
Seedance 2.0 uses real conditioning images because rendered images produced inconsistent interactions in the authors’ tests. Stereo depth anchors scale; the reconstruction assumes a fixed camera and static background. e03e04
- Recover and align geometry
MoGe-2 depth is scale/offset-aligned to first-frame stereo depth on background pixels. SAM2 provides masks, FoundationPose object poses, WiLoR hand estimates and HOI-DETR contact intervals. Joint rigid corrections retain finger articulation while reconciling projection, silhouette, contact and temporal constraints. e04
- Prepare executable variation
Each source trajectory receives five sampled variations. Approach perturbations decay toward contact; shared hand/object perturbations grow afterward, preserving relative contact geometry. A yaw transform, IK screening and interpolation add variation while respecting arm feasibility and joint-speed limits. e05
- Learn feedback actions
An MLP actor observes joint positions, previous action, wrist/fingertip geometry, noisy object pose, HOI errors, reference fingertip distances and BPS shape. It receives no force, mass, joint velocity or retargeted finger-joint target; the critic additionally receives clean simulator state. e06
6. Experiments & results
GALATEA turns image-and-language-conditioned manipulation videos into 3-D hand–object references, then learns simulation-based controllers that physically track them. Its central interface retains finger motion and object motion together. Multi-skill experts are distilled into one feedback policy; simulated transfer and 27/40 real-world successes on unseen plans support useful, but limited, generalization (e02, e09, e14, e17).
6.1 Read the original evidence
Figure 7. The reference specifies an interaction site and grasp style as well as object movement. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each of the four panels, compare the generated human reference on the left with robot execution on the right. The upper panels show a neck grasp and a top-down pinch of the jar; the lower panels show contact at the mug rim and handle. Map these visible labels to the evaluation tasks in Table III: jar-neck pose adjustment, jar top-down grasp-and-move, mug-rim pushing and mug-handle pulling. The source states that all displayed reference trajectories were unseen during policy training. Its discussion emphasizes that the neck grasp leaves the opening accessible, whereas the top-down grasp occupies that opening. e17e18
What it supports. These pairs illustrate why retaining hand motion matters to the intended interaction: different grasps can serve different purposes on the same object. Reader interpretation: the reference communicates contact strategy beyond object displacement. The physical success counts come from Table III; this figure supplies qualitative examples of the resulting behavior.
Where the evidence stops. Selected stills do not establish full-trajectory accuracy, robustness or a causal advantage over object-only references. No such real-world ablation is reported. The source separately records failures at contact transitions and after the controller departs from its reference.
Table III. Physical evaluation reports 27 successful trials across four unseen-plan tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task column together with its Type row: Jar neck is pose adjustment, Jar top is grasp-and-move, and both mug tasks are push-and-pull behaviors. Each success entry uses ten trials, so the overall 27/40 combines equally sized task groups. The surrounding text specifies unseen video plans conditioned on a scene image and new language instruction. Success means completing the reference while meeting the simulation section’s tracking thresholds, including 4-cm object-position and 30-degree object-rotation limits. In the physical trials, FoundationPose supplies object poses and forward kinematics from measured joints supplies hand keypoints. e13e16e17
What it supports. The controller succeeds in 6/10 jar-neck, 6/10 jar-top, 7/10 mug-rim and 8/10 mug-handle trials. This is direct physical execution evidence for the four listed tasks, complementing the larger simulation evaluation. It does not by itself measure general dexterous competence beyond this selected task set.
Where the evidence stops. Ten trials per task and no reported confidence intervals limit precision. Scoring depends on online pose estimation, with calibration described in the setup; an independent ground-truth pose system is not reported. No real-world baseline appears in this table.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Generated-video reconstruction preference Four reviewers; 120 uniformly sampled generated clips; 480 judgments per method, with ties allowed. | 1.17; 417/480 (87%). Mean rank, lower better; first-place judgments | VideoManip 2.16; DO AS I DO 2.88; EgoInfinity 3.47. Anonymized visual preference; these generated clips have no 3-D ground truth. e10 |
| Annotated HOI reconstruction and ablation 40 fixed-view clips each from H2O and HO-Cap, 150 frames/clip; shared object meshes. | Full ADD-S: 4.82/3.88; average metric rank: 1.67/1.67. ADD-S (cm), MRRPE/CDev (mm), lower better | Without joint optimization, HO-Cap MRRPE/CDev are 102.08/112.43 versus 54.87/41.36. Without depth alignment, HO-Cap ADD-S worsens to 6.01 but CDev improves to 39.34. Joint refinement improves relative geometry; depth anchoring does not improve every metric. Full-method sensing differs from monocular baselines. e10e11 |
| Simulation tracking and transfer 42 training objects; ten-object benchmark; distilled transfer to 300 unseen benchmark-object trajectories and five novel objects. Each method–object experiment uses 20,000 first-frame-start rollouts. | Experts: 78.6/80.5 across 42 objects; 77.4/81.0 on benchmark. Distilled unseen trajectories: 66.6/68.8; novel objects: 54.2/52.2. Object-macro mean/median completion success (%) | Distillation lowers training-object mean/median by 4.2/6.0 percentage points. Completion requires object position within 4 cm, rotation within 30 degrees and hand-group thresholds of 6/6/8/10/12/12 cm (thumb/index/middle/ring–pinky/level-1/level-2). Transfer uses the same embodiment. e13e14 |
| RL mechanism ablations Same ten benchmark objects and evaluation references; matched learned-policy training budgets. | PPO instead of SAPG: 10/10; removing lift reward: 7/10; two-stage training: 10/10. Objects with reduced success versus full experts | The two-stage GALATEA variant exceeds ManipTrans by 13.2 percentage points on average. Single-factor ablations support optimizer/lift choices; the ManipTrans comparison bundles reward and optimizer differences. e13e14 |
| Tracking accuracy during intent execution Segments after contact and a 5-cm lift until dropping below 2.5 cm for lifting tasks, or after contact for pushing; segments under eight steps discarded. | 10.2 mm / 17.4 degrees / 36.2 mm. Object position / rotation / hand-keypoint error | PPO: 11.6 / 26.4 / 39.3; DO AS I DO (floating): 35.3 / 7.1 / 36.9. Conditional errors exclude failed intent segments; the low DO AS I DO rotation error is not directly comparable because many pose-adjust plans are excluded. e13e15 |
| Real-world tracking of unseen video plans Distilled policy; four tasks, ten trials each; same completion thresholds, with FoundationPose object estimates and FK hand keypoints. | Jar neck 6/10; jar top 6/10; mug rim 7/10; mug handle 8/10; overall 27/40. Successful trials | No real-world baseline comparison is reported. Supports these physical tracking behaviors; small per-task samples and perception-based scoring limit broader conclusions. e16e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table I. Joint optimization improves relative geometry, while depth alignment creates a measurable metric tradeoff. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two dataset blocks separately: each contains 40 clips of 150 frames, with the image resolution printed in its header. ADD-S measures closest-point object alignment in centimeters; MRRPE measures wrist-to-object relative-position error in millimeters; CDev measures distances of ground-truth contact pairs under predicted poses, also in millimeters. Lower is better for all columns. Avg. Rank averages the three geometry-metric ranks and excludes runtime. Time is minutes per clip on one RTX 5880, as specified below the table in the source. Compare Ours with the two immediately following rows before comparing whole pipelines: these rows remove depth alignment or joint optimization. e04e10e11
What it supports. On HO-Cap, joint optimization reduces MRRPE from 102.08 to 54.87 mm and CDev from 112.43 to 41.36 mm. Depth alignment improves ADD-S from 6.01 to 3.88 cm. These changes support distinct roles for relative hand–object refinement and absolute metric anchoring, rather than one undifferentiated reconstruction gain.
Where the evidence stops. CDev is slightly better without depth alignment: 39.34 versus 41.36 mm on HO-Cap. DO AS I DO also has lower MRRPE. Full-method stereo input differs from monocular baselines; aggregate rank does not imply superiority on every metric or equal sensing.
Figure 5. Simulation tests both the RL recipe and how much performance survives distillation and transfer. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the vertical axis as trajectory-completion success, not image quality or average pose accuracy. Black circles represent the 42 training objects, colored circles the ten benchmark objects, and diamonds the five novel objects. The violins and boxes summarize variation across objects; the source describes interquartile ranges, medians and 1.5-times-IQR whiskers. Compare the benchmark expert column with PPO, w/o lift reward and Two-stage before reading the transfer columns on the right. The setup uses the same evaluation references and matched learned-policy training budgets, with 20,000 rollouts per method–object experiment. Numerical summaries are stated in the following page’s Results paragraph. e07e08e13e14e15
What it supports. Benchmark experts reach 77.4% mean and 81.0% median success. Replacing SAPG with PPO reduces success on all ten benchmark objects; removing lift reward reduces it on seven. The distilled policy reaches 66.6% mean on unseen trajectories and 54.2% on novel objects, establishing transfer with an evident remaining performance gap.
Where the evidence stops. Object distributions are not confidence intervals over training seeds. DO AS I DO retains a floating wrist, unlike the arm–hand methods. The source’s separate tracking-error table further conditions on intent-executing segments, so its error ordering cannot replace this completion comparison.
7. Analysis & limitations
7.1 What the evidence leaves open
Palm-friction lifts and near-table grasps of thin objects can be difficult for the robot despite plausible videos. Contact transitions expose sim-to-real mismatch, and recovery often fails once the reference is lost. In-hand manipulation is not systematically covered. e18
Reconstruction yield is not a controlled comparison against in-the-wild data because screening criteria differ. Simulated error averages are conditioned on intent execution; real-world testing covers four tasks without confidence intervals or independent ground-truth pose scoring. e12e13e17
7.2 Questions for discussion
- Would reconstruction gains persist when measured by downstream tracking on identical reference sets, rather than aggregate visual/geometry metrics (e11, e13)?
- How much unseen-plan failure comes from reference geometry versus recovery limitations after contact (e04, e18)?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include the proprietary generator, metric meshes, calibrated stereo capture, perception models, robot/simulator embodiment and expert-distillation pipeline. Table I reports reconstruction times of 12.33/6.73 minutes per H2O/HO-Cap clip on one RTX 5880; these are not policy-training costs. e03e04e09e10e16
The supplied PDF leaves reconstruction loss weights, screening criteria, network widths, SAPG exploration settings, training-step counts and domain-randomization ranges unspecified. A reproduction should document these before interpreting matched-budget claims or attributing gains to one component. e04e08e09e12e13
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does better reconstructed geometry actually improve tracking?
Reader-proposed check, not an executed experiment: apply a 2-by-2 depth-alignment/joint-optimization ablation to the same generated benchmark clips, meshes and initial estimates. Keep augmentation, feasibility screening, embodiment, training budget and evaluation thresholds fixed across controller runs. Report accepted-reference yield and tracking success per original input clip, counting rejected reconstructions separately and as failures in an end-to-end rate. Measure the geometry metrics separately on the fixed HO-Cap set. The claim is weakened if relative-geometry gains do not improve matched-clip tracking, or if apparent control gains arise only from discarding difficult references. Record the omitted loss weights and screening criteria explicitly. e04e05e11e12e13
Check 2: Separate exploration gains from lift shaping
Reader-proposed check, not an executed experiment: on the same ten benchmark objects, compare SAPG and PPO with the lift bonus enabled and disabled, using equal environment-step budgets and multiple recorded training seeds. Hold the other reward gates, observations, open-hand initialization and reference augmentation fixed. Report per-object completion and separate lifting from push-and-pull references; also retain failed-lift counts alongside the source’s conditional tracking errors. Lift shaping should preferentially improve lift-dependent completion. If SAPG helps only with the bonus, the evidence would favor an interaction between exploration and reward design rather than independent additive gains. e05e06e07e08e13e14e15
8.3 Reading coverage
Visual audit: All nine pages were rendered and visually inspected after reading all four complete text chunks. Page 1 verifies the title, authors, affiliations, v1 date and Figure 1. Pages 2–4 support Figures 2–3, reconstruction equations, contact-aware augmentation, observations, action mapping, rewards and RL setup. Page 5 supplies Figure 4, Table I, metric units, reconstruction timing and distillation. Pages 6–7 supply Figures 5–7, Tables II–III, simulation/real-world protocols, numerical results and perception/calibration details. Page 8 supplies failure modes and scope limits; pages 8–9 contain acknowledgments and references. All six final original crops were individually viewed, retaining their headers, labels and legends. Diagram arrows were checked against the method text; no claim-relevant arrow discrepancy was found. Table I metric units and Figure 5 evaluation details are supplied by the inspected accompanying text. No separate supplement, linked video, code or experiment execution was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations, version and Abstract (p. 1)
- I. Introduction (pp. 1–2)
- II. Related Work, A–B (pp. 2–3)
- III. Method: video generation, reconstruction, augmentation, RL and distillation (pp. 3–5)
- IV. Experiments: reconstruction, simulated tracking and real-world execution (pp. 5–8)
- V. Discussion and Conclusion (p. 8)
- Acknowledgment and References (pp. 8–9)
Outside the original text pass
- Identity/version: the inspected title and complete author list match the catalog. This report reads arXiv:2609.10050v1, dated 9 September 2026; no alternative revision or edition was supplied (e01).
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting all nine PDF pages, all seven figures and all three tables, plus all six final crops.
- Separate supplemental material availability has not been fully verified.
- No separate supplement or appendix was supplied; no appendix appears in this nine-page PDF.
- Linked code, project media and cited works were not opened. No experiments were run; code availability and reproduction are unverified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin
The title and all nine catalog authors match. The artifact is arXiv:2609.10050v1 [cs.RO], 9 Sep 2026. The affiliations are University of California, Berkeley; Sharpa Robotics; The University of Hong Kong.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Introduction; p. 2, Figure 2
GALATEA connects generated HOI videos, reconstructed references, simulation RL and a tracking controller; references serve both policy training and deployment.
Go to primary source ↓e03PDF p. 3, Section III-A.1 and opening of III-A.2
Seedance 2.0 uses a real scene image and language instruction. Reconstruction takes video, calibrated intrinsics, a metric object mesh and first-frame stereo depth; three manipulation types are defined.
Go to primary source ↓e04PDF p. 3, Figure 3; Section III-A.2, Metric Depth Estimation, Initial Motion Estimation and Eqs. (1)–(2)
Static-background alignment corrects MoGe-2 depth using first-frame stereo depth. SAM2, FoundationPose, WiLoR and HOI-DETR supply masks, poses and contact intervals. Joint rigid corrections retain finger articulation and combine projection, silhouette, contact and temporal losses.
Go to primary source ↓e05PDF pp. 3–4, Section III-B.1, Eq. (3) and following paragraph
Five sampled variations perturb approach and post-contact motion. A shared post-contact transform preserves relative hand–object pose; perturbations use 30% of source clearance/motion range. IK screening and interpolation enforce execution feasibility and joint-speed limits.
Go to primary source ↓e06PDF p. 4, Section III-B.2, Observation and control, Eq. (4)
An asymmetric actor–critic uses HOI tracking errors and object shape. The MLP emits seven arm and 22 hand actions, mapped to smoothed targets for 30-Hz PD control; the critic has privileged state.
Go to primary source ↓e07PDF p. 4, Section III-B.2, Reward, Eqs. (5)–(7) and contact/lift gates
Reward combines hand/object tracking, contact shaping, lift and power terms, minus regularization. Object tracking is contact-gated; hand tracking is suppressed when a reference lift is not reproduced.
Go to primary source ↓e08PDF p. 4, Section III-B.2, Policy optimization, Domain randomization, Episode initialization and termination
SAPG aggregates experience across PPO agents with different exploration settings. Dynamics, latency and actor object-pose observations are randomized. Training samples reference frames but resets hand joints open to avoid interpenetration.
Go to primary source ↓e09PDF p. 5, Section III-B.3, Policy Distillation
At most two experts per object category cover about 40 source trajectories each, roughly 200 after augmentation. Experts are trained directly or via multi-object training then category fine-tuning, followed by behavior cloning and DAgger distillation.
Go to primary source ↓e10PDF p. 5, Figure 4 and Section IV-A, Setup and Results
Four reviewers rank 120 generated clips with ties and randomized ordering: 480 judgments per method. GALATEA is first in 417 judgments, mean rank 1.17; competitors score 2.16, 2.88 and 3.47. Quantitative evaluation uses 40 clips each from H2O and HO-Cap, 150 frames per clip; metric units and single-RTX-5880 runtime are specified.
Go to primary source ↓e11PDF p. 5, Table I, all rows; pp. 5–6, Section IV-A ablations and interpretation
Full-method ADD-S is 4.82/3.88 cm on H2O/HO-Cap, with average rank 1.67 on each. Removing joint optimization increases relative and contact errors. Removing depth alignment worsens ADD-S but slightly improves CDev; DO AS I DO has lower wrist-relative error, whose definition cancels shared absolute-depth errors.
Go to primary source ↓e12PDF p. 2, Introduction, corpus totals; p. 6, Section IV-A, Remark: Usable reference yield
Of 2,500 generated clips, 83% pass reconstruction screening, described as about 2,000 usable references; over 1,500 are grounded by simulation experts. Comparison with in-the-wild yield uses different screening criteria.
Go to primary source ↓e13PDF p. 6, Section IV-B, Setup and Baselines and ablations; Figure 5 and Table II
Evaluation uses 20,000 first-frame-start rollouts per method–object experiment, with 4-cm object-position, 30-degree rotation and group-specific hand-error thresholds. Learned baselines have matched budgets; DO AS I DO retains a floating wrist. Tracking errors use filtered intent-executing segments.
Go to primary source ↓e14PDF p. 6, Figure 5; p. 7, Section IV-B, Results
Expert macro mean/median are 78.6/80.5% across 42 objects and 77.4/81.0% on ten benchmark objects. Distillation costs 4.2/6.0 percentage points. Unseen-trajectory and novel-object distilled means/medians are 66.6/68.8% and 54.2/52.2%. PPO, no-lift and two-stage variants lose success on 10/10, 7/10 and 10/10 benchmark objects.
Go to primary source ↓e15PDF p. 6, Table II and intent-executing segment definition; p. 7, Section IV-B, final Results paragraph
GALATEA tracking errors are 10.2 mm object position, 17.4 degrees rotation and 36.2 mm hand error. DO AS I DO has 7.1-degree rotation error, but failed pose-adjustment plans are excluded by the segment criterion, limiting comparison.
Go to primary source ↓e16PDF p. 3, Section III-B opening; p. 7, Section IV-C, Real-World Setup and Onboard Perception
The embodiment is a Sharpa Wave Hand on a Franka Research 3, simulated with PhysX in Isaac Gym. At deployment a D455 supports FoundationPose feedback; a D435 supplies the conditioning image and is removed. A rigidly grasped object calibrates shared translation bias.
Go to primary source ↓e17PDF p. 7, Figure 7, Table III and Section IV-C Results; p. 8, opening continuation
Forty unseen plans yield 6/10 jar-neck, 6/10 jar-top, 7/10 mug-rim and 8/10 mug-handle successes, 27/40 overall. Success uses simulation tracking thresholds with perceived object poses and FK hand keypoints. Figure 7 pairs generated references with robot executions; neck versus top grasps preserve different interaction choices.
Go to primary source ↓e18PDF p. 8, Section V.1–V.2
Failures include human-style palm-friction lifts, near-table thin-object grasps, contact-transition sim-to-real mismatch and poor recovery after losing the reference. In-hand manipulation is not systematically covered and subtle motion is obscured by reconstruction noise.
Go to primary source ↓8.5 Primary sources
Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers ↗
PDF · 6,757 extracted words
Source fingerprint
98b3cc4971b2994089c9a3a94263e8614a7f94875f50f2bba3912c5b7cb609b8