Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
1. Paper overview
In one sentence: Cross-embodiment video features and retargeted human actions enable robot skill transfer, but the evaluated skills remain represented in human training data. e-crosse-retargete-frameworke-skill-protocole-skill-resultse-pdcp
| At a glance | What to know |
|---|---|
| Research problem | Source description Teleoperation is expensive, and task labels omit spatial and temporal details useful for manipulation. The authors seek a policy that uses human video prompts to perform tasks without collecting new robot demonstrations or finetuning at deployment. The central challenge is transferring task information and action supervision across human and robot embodiments. e-probleme-skill-protocol |
| Core mechanism | Source description Cross-prediction trains a video generator to perform the prompted task with a target embodiment, creating a task-and-scene representation for policy conditioning. e-cross |
| A key reported result | Skills withheld from robot demonstrations: Grasp Bottle 0.47 / 65.3; Press Button 0.56 / 63.8; Close Drawer 0.65 / 66.5; Open Carton 0.46 / 50.8. Success rate (fraction); normalized task-progress score (0–100). Grasp Bottle, Press Button, Close Drawer and Open Carton; target-skill robot trajectories removed, human examples retained; wrist-circle annotations used. Without circles: 0.27 / 43.3, 0.25 / 36.3, 0.35 / 49.4, 0.23 / 36.9. XSkill, UniSkill and the robot-only language policy report zero success and score on all four. Circles add 20, 31, 30 and 23 percentage points of success. This supports the importance of visual alignment under human-supervised skill transfer. e-evaluatione-skill-protocole-skill-results |
| Reading caution | Reader analysis The held-out skills and objects are novel to robot demonstrations, not to all training data. Stage-1 exposure is not explicitly controlled. The experiments therefore do not isolate acquisition of a wholly unseen skill from one deployment prompt. e-datae-objecte-skill-protocol |
Core contributions
- Source description
Cross-prediction trains a video generator to perform the prompted task with a target embodiment, creating a task-and-scene representation for policy conditioning. e-cross
- Source description
Retargeted human video–action pairs expand policy supervision; a prototype-based auxiliary objective encourages skill-discriminative policy features. e-retargete-pdcp
Figure 2. Stage 1 learns the prompted task while varying the embodiment that performs it. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the two inputs, not the output films. The prompt video describes a task performed by a source embodiment; the initial image specifies the target embodiment and its scene. Follow the green branch to a robot video and the orange branch to a human video. These illustrate different- and same-embodiment prediction choices, respectively. Section III-A assigns the choices probabilities P and 1−P; the drawing does not encode their frequencies. Finally, follow the downward arrow from the inner blocks to the representation: this internal feature is what the next training stage uses to condition actions. e-crosse-frameworke-diagnostics
What it supports. The architecture explicitly trains embodiment transfer through video prediction. A shared task can be illustrated by a human prompt while the target image requests robot appearance and scene context. The representation is learned through this generation task before the action policy is trained.
Where the evidence stops. This figure describes the learning mechanism, not measured control success. The paper leaves P, cross-embodiment training-pair construction and exact feature extraction unspecified; generated robot imagery alone cannot establish executable robot actions.
2. Motivation
2.1 The problem and the proposed response
Teleoperation is expensive, and task labels omit spatial and temporal details useful for manipulation. The authors seek a policy that uses human video prompts to perform tasks without collecting new robot demonstrations or finetuning at deployment. The central challenge is transferring task information and action supervision across human and robot embodiments. e-probleme-skill-protocol
2.2 What this reading follows
A video prompt can show which object to manipulate and how the motion unfolds. This paper connects that information to dexterous robot control through two learning stages: a video generator first learns task representations across embodiments, then a separate action policy learns from robot demonstrations and reconstructed human hand motions. The strongest evidence is real-robot execution when the target skill is withheld from robot demonstrations but remains in human training data. Read the illustrations as a chain from representation learning to action supervision to controlled comparisons. Wrist annotations and incomplete training details matter when judging how much of the transfer can be reproduced. e-crosse-retargete-frameworke-skill-protocole-skill-resultse-pdcp
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is unassigned, so there is no existing quadrant judgment to validate. Architecturally this is a video representation model coupled to a separate diffusion action model; freezing and separate action heads support a multi-model reading. It does not jointly decode future video and actions, nor specify inverse dynamics from predicted frames. Its control mechanism is representation-conditioned action generation, which should remain distinct from either joint prediction or inverse-dynamics categories. e-frameworke-crosse-policy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the demonstrated task from the target embodiment
The first stage asks a video generator to reconcile two sources of information. The demonstration shows an action unfolding, while the initial target image specifies which embodiment and scene should realize it. For cross-prediction the embodiments differ; for same-embodiment prediction they match. This provides a training signal for transferring the task while preserving target appearance and context. The paper then extracts an internal representation instead of treating generated pixels as executable commands. At Stage 2 the video model is frozen, and its representation becomes a condition for another model. That boundary is essential: the system uses a video-prediction model inside a control pipeline, but does not describe one decoder jointly producing future frames and robot actions. e-crosse-frameworke-policy
Figure 3. Stage 2 trains actions and skill-sensitive features while the video backbone stays frozen. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the snowflake and Freeze marker first: the video backbone is held fixed here. A human prompt and the chosen initial image produce the human or dexterous representation. Green and orange paths then combine representation and observation information with the policy transformer. The CLS token feeds the prototype objective, while time/noise conditioning enters the DiT denoiser and predicted actions meet demonstration supervision at the action loss. The drawing groups the auxiliary objectives under one label; Section III-C separates contrastive, prototype cross-entropy and metric terms. Section III-B specifies the observation/representation conditioning token and separate action-component losses. e-frameworke-policye-retargete-pdcp
What it supports. The useful bridge is both visual and motor: frozen video features condition the policy, while reconstructed and retargeted human motions provide action targets alongside robot demonstrations. The prototype objectives shape policy features during training; they do not constitute a separate execution-time skill planner or robot controller.
Where the evidence stops. The explicit Freeze marker agrees with Section III-A, despite looser introduction wording about finetuning representations. The cartoon is not a complete implementation: it omits the rotation encoding and the adapted auxiliary-loss formulas.
5.2 Turn a human video into motor supervision
Video conditioning alone says what the policy should do; the second stage additionally needs action targets. The paper obtains these from human motion reconstruction. WiLoR predicts right-hand landmarks, a wrist-centered frame supplies pose, and coordinate conversion places the estimated motion in the robot's base frame. Retargeting maps finger configurations to the robot's hand joints. Human and robot demonstrations can then contribute to the same action-denoising objective. The policy combines current observation features with the frozen task representation, while PDCP trains a clustering-token feature to distinguish skills and organize prototypes. This makes human data useful beyond visual pretraining. Its practical dependency is the quality and consistency of the reconstructed actions, whose calibration and rotation encoding are incompletely specified. e-retargete-policye-pdcp
5.3 Read generalization through the training exclusions
Reader analysis: the strongest transfer claim becomes clearer when the held-out unit is explicit. In the object experiments, an object is removed from robot policy-training demonstrations but remains in human demonstrations. The skill experiments remove all robot trajectories involving a target skill and retain its human examples. Thus the policy can learn the relevant motor pattern before deployment, even though it has not observed the robot doing that skill. Table II then adds a useful local comparison: keeping the representation-plus-PDCP system, wrist circles improve every listed task. This supports the importance of embodiment alignment. It leaves open whether an inference prompt can teach a skill absent from both training modalities, and whether another equally informative wrist cue would match the circles. e-objecte-skill-protocole-skill-results
5.4 Training and inference
During training
Stage 1 uses over 250,000 videos from RH20T, Something-Something, HOI4D, Bridge and in-house Apple Vision Pro teleoperation. Stage 2 uses about 7,000 human/dexterous-hand videos, roughly 5:1, from in-house paired demonstrations and HOI4D. The evaluation suite covers 13 skills and over 100 concrete tasks. e-data
The video model is frozen for Stage 2, as stated in Section III-A and marked in Figures 1 and 3. The introduction's wording about finetuning the representation is less precise; the explicit freeze statements govern this reading. Action denoising uses validation-selected component weights plus three auxiliary losses whose detailed formulations are expressly omitted. e-frameworke-policye-pdcp
During inference
A human prompt yields a task representation, which is combined with real-time robot observations and passed to the RDT-1B DiT-based policy. The robot's built-in position controller executes actions at 10 Hz. This is a controller execution rate, not a reported end-to-end latency measurement; video-feature refresh and action horizons are unspecified. e-datae-policy
5.5 Implementation flow
- Learn cross-embodiment video features
A pretrained video model, called SVD in Section IV-B, receives a source demonstration and target initial image. Training chooses different embodiments with probability P and the same embodiment with probability 1−P. The intended representation retains task, objects and scene context while encoding embodiment transfer. P and the cross-embodiment sample-pairing recipe are unspecified. e-crosse-diagnostics
- Convert human motion into action supervision
WiLoR supplies 21 right-hand keypoints per frame. Wrist position and index/middle-finger landmarks define a local orientation frame; camera intrinsics and a coordinate transform place motion in the robot base frame. AnyTeleop-style optimization retargets finger motion to 12 robot joint angles. These reconstructed actions are supervision, not directly measured robot executions. e-retarget
- Condition a separate action denoiser
The policy models p(a_t | s_t, z): a pretrained ResNet-34 supplies the 1000-dimensional observation feature s_t, and the video model supplies the 4096-dimensional representation z. A transformer predicts action noise, with separate finger, rotation and position heads. The paper declares a 19-dimensional action vector but does not specify the wrist-rotation encoding that accounts for its dimensions. e-policy
- Structure policy features with PDCP
A learnable clustering token produces latent h. Same-skill examples are positives for NT-Xent contrastive learning; K-Means prototype labels supervise a cross-entropy loss; a Siamese metric loss attracts same-prototype examples and repels others. These are training objectives on policy features, not an inference-time planner. e-pdcpe-framework
6. Experiments & results
A human demonstration video conditions a frozen video model whose features guide a separate diffusion policy for an Xhand robot. Human hand motions are reconstructed and retargeted into the policy's action space, allowing human demonstrations to supervise action learning. The reported transfer concerns objects and skills absent from robot training demonstrations but present in human training data; it does not establish learning an entirely untrained skill from the inference prompt alone.
6.1 Read the original evidence
Table I. PDCP improves the reported success rate across all three basic variations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each setting has two columns: SR is the fraction of successful robot trials, whereas Score gives normalized subtask progress and is not a percentage success rate. Position changes target locations; Scene changes inoperable objects; Background changes visual surfaces such as tablecloths. Read the last two rows together to compare representation-conditioned policies trained with robot and human data, with and without the auxiliary objective. R. and H. denote those training modalities. The final row spells the objective DPCP, while the method section names it PDCP. Keep the three settings separate rather than pooling their scores into a new benchmark. e-variatione-evaluatione-pdcp
What it supports. Adding PDCP raises position success from 0.74 to 0.79, scene success from 0.67 to 0.69, and background success from 0.64 to 0.73. The associated scores also rise. The strongest success-rate gain in this comparison is nine percentage points under background changes.
Where the evidence stops. The table tests the combined PDCP objective, so it cannot attribute gains to a specific auxiliary loss. No confidence intervals or exact per-cell trial counts are given; the protocol states 60–80 trials per algorithm/generalization level.
Figure 6. Novel objects remain available in human demonstrations during policy training. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the training-data labels beneath the bars before comparing heights. The first three methods train their Stage-2 policies on robot data without the target objects. The final three can use human demonstrations that include those objects. Section IV-D defines novelty relative to the Stage-2 robot dataset and describes four evaluation tasks. The fifth and sixth bars offer the nearest PDCP comparison: both use learned representations and robot-plus-human supervision. The fourth bar instead uses language conditioning with both data modalities. The vertical axis is success rate in percent; the chart supplies an overall value per method, not separate object-specific results. e-objecte-evaluatione-overview-discrepancy
What it supports. The full method reports 82% success, compared with 73% for the representation policy without PDCP and 27% for language conditioning with both data modalities. These detailed bars support gains for the tested combined system, while comparisons to robot-only policies also reflect additional human action supervision.
Where the evidence stops. Human training exposure makes these objects familiar in another embodiment. No uncertainty bars or aggregation weights are supplied. Figure 1 labels the language-plus-human/robot object baseline 0.36; Figure 6 labels it 27%, without explaining the mismatch.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Position, scene and background generalization Real Xhand trials: move target objects, replace inoperable scene objects, or change backgrounds. The paper states 60–80 trials per algorithm/generalization level. | Position 0.79 / 85.3; scene 0.69 / 88.9; background 0.73 / 80.9. Success rate (fraction); normalized task-progress score (0–100) | Without PDCP, Repr.+R.+H.: 0.74 / 81.6, 0.67 / 84.4, 0.64 / 74.5, respectively. PDCP adds 5, 2 and 9 percentage points of success. This compares the combined auxiliary objective, not its three individual losses. e-evaluatione-variation |
| Object generalization with human training exposure Four tasks; target objects excluded from Stage-2 robot demonstrations but retained in human demonstrations. | 82% Success rate (%) | Representation+robot+human 73%; language+robot+human 27%; UniSkill 22%; XSkill 14%; language-conditioned diffusion policy 9%. The closest representation baseline improves by 9 percentage points with PDCP. Comparisons with robot-only Stage-2 baselines also change available action supervision. e-evaluatione-object |
| Skills withheld from robot demonstrations Grasp Bottle, Press Button, Close Drawer and Open Carton; target-skill robot trajectories removed, human examples retained; wrist-circle annotations used. | Grasp Bottle 0.47 / 65.3; Press Button 0.56 / 63.8; Close Drawer 0.65 / 66.5; Open Carton 0.46 / 50.8. Success rate (fraction); normalized task-progress score (0–100) | Without circles: 0.27 / 43.3, 0.25 / 36.3, 0.35 / 49.4, 0.23 / 36.9. XSkill, UniSkill and the robot-only language policy report zero success and score on all four. Circles add 20, 31, 30 and 23 percentage points of success. This supports the importance of visual alignment under human-supervised skill transfer. e-evaluatione-skill-protocole-skill-results |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5(a,b). Qualitative diagnostics probe what the learned video representation retains. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) links blue clusters to pushing, green to the button task and red to the orange task. Its dashed connectors associate representations with scene examples and object positions. The upper red inset shows a robot hand but says Human Orange, duplicating the lower human inset; this source labeling error leaves that embodiment annotation unreliable. Section IV-B.1 supplies the intended task/embodiment/position interpretation. In panel (b), follow the minus and plus signs using IV-B.2: robot pushing minus human pushing plus human grasping supplies a composite representation. The lower filmstrip shows the generated robot grasping sequence. The operation concerns internal representations, while the displayed frames illustrate its inputs and output. e-diagnosticse-crosse-diagnostic-label
What it supports. The examples suggest that the representation retains task, embodiment and spatial information, and that one embodiment offset can transfer to another illustrated task. They motivate examining whether the same organization helps the downstream policy, but provide no quantitative estimate of representation quality or transfer reliability.
Where the evidence stops. The original Human Orange labeling error in panel (a) is preserved. This is a qualitative diagnostic, not a controlled ablation: t-SNE separation does not establish original-space distances, and the generated sequence proves neither physical execution nor a universal linear skill algebra.
Table II. Wrist-circle annotations substantially improve the reported transfer of skills withheld from robot demonstrations. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task as a separate pair of SR and Score columns. Language+R.+H. reports 0.18 SR for Grasp Bottle and 0.08 SR for Open Carton; their adjacent scores are 27.3 and 17.7. Under this protocol, target-skill robot demonstrations are removed while human examples remain. Compare the final two rows to assess wrist circles within the representation-plus-PDCP policy. Section IV-E places circles on detected wrists in both embodiments, with size inversely related to camera distance. The first three rows use only robot data during Stage 2. Their zero results therefore involve a different supervision condition from the final two rows. e-skill-protocole-skill-resultse-evaluation
What it supports. Wrist circles increase success from 0.27 to 0.47 for bottle grasping, 0.25 to 0.56 for button pressing, 0.35 to 0.65 for drawer closing, and 0.23 to 0.46 for carton opening. The gain spans all four tasks and accompanies higher partial-progress scores.
Where the evidence stops. These skills are withheld from robot demonstrations, not from human training. The table gives no uncertainty estimates and does not compare other alignment cues, so it cannot isolate the circles' geometry from the benefit of explicit wrist-position information.
7. Analysis & limitations
7.1 What the evidence leaves open
The held-out skills and objects are novel to robot demonstrations, not to all training data. Stage-1 exposure is not explicitly controlled. The experiments therefore do not isolate acquisition of a wholly unseen skill from one deployment prompt. e-datae-objecte-skill-protocol
Generated-video examples and t-SNE/vector-arithmetic demonstrations are qualitative. They do not measure physical prediction accuracy or prove a generally linear, embodiment-independent skill space; no controlled removal of cross-prediction is reported. e-crosse-diagnosticse-variatione-objecte-skill-results
Tables and charts omit confidence intervals, exact cell-level trial counts and detailed success/partial-credit rubrics. Figure 1's object-generalization language+human+robot bar is 0.36, whereas Figure 6 reports 27%; the source does not explain the discrepancy. The detailed Figure 6 is used here. e-evaluatione-overview-discrepancy
Figure 5(a) labels an upper red inset containing a robot hand Human Orange, duplicating the lower human inset. This source labeling error makes that red example unreliable for assigning embodiment labels to clusters. The representation arithmetic interpretation follows IV-B.2 and Figure 5(b). e-diagnostic-labele-diagnostics
7.2 Questions for discussion
- Does cross-prediction improve robot control when data, optimization and feature extraction are held fixed?
- Do wrist circles help because they align embodiments, or because they provide a particularly easy position cue?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires calibrated human keypoint-to-robot transforms, retargeting, the composite video datasets, Xhand action data and target-skill exclusions. The paper describes wrist circles centered on detections with size inversely proportional to camera distance, but supplies no exact radius rule or calibration recipe. e-retargete-datae-skill-protocol
Missing details include the SVD checkpoint and feature-extraction layer/timestep, P, video sampling/pairing, action rotation encoding, loss weights, prototype count/update schedule, optimizer, training duration, compute hardware and diffusion inference settings. The three adapted contrastive losses are explicitly omitted. Proposed checks should publish these choices rather than claim exact reproduction. e-crosse-policye-pdcpe-datae-diagnostics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate cross-prediction while holding policy supervision fixed
Reader-proposed, not run: train matched Stage-1 models with same-embodiment prediction only versus a declared nonzero P, using identical videos, backbone initialization, updates and feature-extraction settings. Freeze both, then train identical Stage-2 policies on the same robot/human actions with PDCP and wrist annotations held fixed. Evaluate the same position, scene and background trials across multiple seeds, reporting exact counts and uncertainty. A repeatable control advantage would support cross-prediction's causal contribution; improved-looking videos without improved execution would weaken that explanation. e-crosse-frameworke-datae-variatione-diagnostics
Check 2: Distinguish wrist alignment from a generic visual marker
Reader-proposed, not run: repeat the four Table II skill exclusions with three matched annotation conditions: no circles, correctly centered distance-scaled wrist circles, and circles of matched size/color placed away from the wrist. Apply each condition consistently to human and robot inputs, keeping demonstrations, PDCP, optimization and evaluation prompts fixed. Report task-wise SR and Score with exact trial counts and uncertainty. If correctly centered circles outperform both controls, the position-alignment explanation gains support; similar gains from displaced circles would challenge it. e-skill-protocole-skill-resultse-evaluatione-pdcp
8.3 Reading coverage
Visual audit: All eight PDF pages were rendered and visually inspected: title/byline/version/affiliations (p. 1), overview (p. 2), cross-prediction (p. 3), policy architecture and equations (p. 4), losses/data/controller/evaluation and generated examples (p. 5), Table I and Figure 5 diagnostics (p. 6), object chart and object/skill protocols (p. 7), and Table II/conclusions/references (p. 8). All six final original crops were separately viewed. Figure 2 branches and Figure 3 Freeze/loss markers were checked against III-A–C and Equations (2)–(4). Figure 5 arithmetic was checked against IV-B.2; the upper red robot-hand inset mislabeled Human Orange in Figure 5(a) is explicitly disclosed. Page 8 was also inspected at 220 DPI and its final Table II crop at 520 DPI: Language+R.+H. has Grasp Bottle SR 0.18 and Open Carton SR 0.08. The introduction finetuning wording, PDCP/DPCP spelling and Figure 1/Figure 6 object-baseline discrepancy remain disclosed. Printed video frames were inspected, not complete videos. No separate supplement, code or experiments were reviewed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF pp. 1–8: complete supplied text, including references
- Abstract and I. Introduction
- II. Related Works, A–C
- III. Methods, A. VGCP, B. Human Video-Action Pair Boosting, C. PDCP
- IV. Experiments, A–E: video generation, representations, variation, object and skill generalization
- V. Conclusions
Outside the original text pass
- Identity: PDF p. 1 shows arXiv:2505.20795v2 [cs.RO], 29 May 2026. Title and authors match the catalog. This report reviews that revision, not the catalog's May 27, 2025 submission version; the earlier version was not supplied and revision changes cannot be compared. The catalog's ICRA 2026 venue label is not independently established by this PDF.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting all eight original PDF pages, Figures 1–7 and Tables I–II.
- Separate supplemental material availability has not been fully verified.
- No separate supplement or appendix was supplied; the eight-page PDF ends with conclusions and references.
- Code and external resources were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, affiliation footnote and arXiv margin
The exact title is Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt. Authors: Xiang Zhu, Yichen Liu, Hezhong Li, Jianyu Chen. Affiliations: Tsinghua University, China; Shanghai Qi Zhi Institute, China. Margin: arXiv:2505.20795v2 [cs.RO], 29 May 2026.
Go to primary source ↓e-problemPDF p. 1, Abstract and I. Introduction
The paper proposes human-video-prompted robot manipulation to reduce teleoperation and deployment finetuning while retaining spatial and temporal task information.
Go to primary source ↓e-frameworkPDF p. 1, Introduction; p. 2, Figure 1; p. 3, III-A final sentence; p. 4, Figure 3
The detailed method and both training diagrams explicitly freeze the video model in Stage 2 and train a separate diffusion policy. The introduction instead loosely describes finetuning the representation.
Go to primary source ↓e-crossPDF p. 3, Figure 2 and III-A, Cross-Prediction
Source task video plus target initial image supervise target-embodiment task generation. Different embodiments are chosen with probability P; identical embodiments with 1−P. No P value or precise pairing implementation is stated.
Go to primary source ↓e-retargetPDF pp. 3–4, III-B, Human Action Preprocessing
WiLoR reconstructs 21 right-hand keypoints. Wrist/index/middle landmarks define wrist pose; camera intrinsics and a transform map positions into robot coordinates. AnyTeleop-style optimization retargets hand joints to the dexterous hand's 12 joint angles.
Go to primary source ↓e-policyPDF p. 4, III-B, Representation Conditioned Diffusion Policy, Equations (1)–(3)
Policy conditioning uses ResNet-34 observation features in R^1000 and Stage-1 features in R^4096. The action vector is declared R^19, split into 12 finger angles, SO(3) rotation and R^3 position. Component noise losses receive validation-selected weights; the specific rotation encoding is absent.
Go to primary source ↓e-pdcpPDF pp. 4–5, III-C, Figure 3 and Equation (4)
A clustering token feeds skill-positive NT-Xent, K-Means prototype cross-entropy and Siamese prototype metric losses. Their adapted formulations are explicitly omitted. Equation (4) uses w_c, w_p, w_m; the preceding prose repeats w_p.
Go to primary source ↓e-dataPDF p. 5, IV. Experiments, dataset and inference paragraphs
Stage 1 uses over 250,000 videos from four named datasets plus in-house teleoperation; Stage 2 about 7,000 videos at approximately 5:1 human:robot. The suite has 13 skills and over 100 tasks. An RDT-1B DiT-based action policy conditions on representations and real-time observations; the Xhand position controller runs at 10 Hz.
Go to primary source ↓e-evaluationPDF p. 5, IV. Experiments, baseline list and final protocol paragraph
The source states 60–80 trials per algorithm/generalization level, matched evaluation conditions and video/text prompts. SR is fraction successful; Task Score is cumulative partial subtask credit normalized to 100. Baselines differ in Stage-2 use of human actions.
Go to primary source ↓e-diagnosticsPDF p. 5, Figure 4 and IV-A; p. 6, IV-A continuation, IV-B and Figure 5(a,b)
Figure 4 shows same/cross-embodiment generated sequences. IV-B names cross-prediction-finetuned SVD. Figure 5 shows t-SNE task/embodiment/position patterns and a robot-push minus human-push plus human-grasp representation example yielding generated robot grasp frames. These are qualitative demonstrations.
Go to primary source ↓e-variationPDF p. 6, Table I, all rows and columns, and IV-C definitions
Full method SR/Score is 0.79/85.3, 0.69/88.9, 0.73/80.9 for position/scene/background. Repr.+R.+H. gives 0.74/81.6, 0.67/84.4, 0.64/74.5. Position moves targets; scene replaces inoperable objects; background changes tablecloth/background. The table writes DPCP.
Go to primary source ↓e-objectPDF p. 7, Figure 6, all six bars, and IV-D
Figure 6 gives 9%, 14%, 22%, 27%, 73%, 82% for language-conditioned diffusion policy, XSkill, UniSkill, Language+R.+H., Repr.+R.+H., and Repr.+R.+H.+PDCP. Four tasks use objects absent from Stage-2 robot demonstrations but retained in human data; object-specific values and aggregation weights are not supplied.
Go to primary source ↓e-skill-protocolPDF p. 7, IV-E and Figure 7
Four skills/tasks are tested after removing target-skill robot trajectories while retaining corresponding human training demonstrations. Wrist circles appear in both embodiments, centered on detected wrists with size inversely proportional to camera distance.
Go to primary source ↓e-skill-resultsPDF p. 8, Table II, all rows and columns; especially Language+R.+H., Grasp Bottle and Open Carton SR columns
In Grasp Bottle/Press Button/Close Drawer/Open Carton order, full SR/Score is 0.47/65.3, 0.56/63.8, 0.65/66.5, 0.46/50.8; without circles 0.27/43.3, 0.25/36.3, 0.35/49.4, 0.23/36.9; Language+R.+H. 0.18/27.3, 0.22/28.9, 0.25/41.3, 0.08/17.7. In that language row, 0.18 is the Grasp Bottle SR; the Open Carton SR cell reads 0.08. XSkill, UniSkill and language-conditioned diffusion policy have all-zero entries. The table writes DPCP.
Go to primary source ↓e-overview-discrepancyPDF p. 2, Figure 1, middle object-generalization summary chart; p. 7, Figure 6, fourth bar
The overview labels LangPrompt H.+R. 0.36, while the detailed object-generalization chart labels Language+Robot+Human Data 27%. No explanation of this numerical discrepancy is supplied.
Go to primary source ↓e-diagnostic-labelPDF p. 6, Figure 5(a), upper and lower red orange-task insets; IV-B.1 and IV-B.2
The upper red inset visibly contains a robot hand but is labeled Human Orange, duplicating the lower inset that shows a human hand. This is an embodiment-label mismatch in the original figure. IV-B.1 describes task/embodiment/position clustering; IV-B.2 separately specifies robot push minus human push plus human grasp orange to generate robot grasp orange.
Go to primary source ↓8.5 Primary sources
Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt ↗
PDF · 6,137 extracted words
Source fingerprint
fc8b41c2681a16747796bd8e2f100a63ca8e061cf3afcb741284be86210d5698