Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
1. Paper overview
In one sentence: Gen2Act uses a generated human demonstration to guide a separate robot policy, gaining unseen-task coverage while remaining limited by both video plausibility and execution quality. e02e03e04e05e08e09e11
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations cannot easily cover every object, scene and manipulation motion. The paper asks whether a video generator trained on web data can supply task-specific motion information for situations absent from robot training. Its central assumption is that human videos can depict useful manipulation in the robot scene, and that a learned policy can translate those cues across the human–robot embodiment gap. e02e03 |
| Core mechanism | |
| A key reported result | Generalization across robot manipulation settings: Gen2Act: MG 83, G 67, OTG 58, MTG 30; reported average 60. Executed-task success rate (%). Physical kitchen, office and lab manipulation; MG changes configurations of seen instances, G uses unseen instances, OTG uses unseen object types, and MTG uses unseen motion types. Seen/unseen is relative to robot training data. Vid2Robot: 83, 38, 25, 0; average 37. RT1 average 22; RT1-GC average 26. Gen2Act exceeds Vid2Robot by 33 percentage points on OTG and 30 on MTG. Main-table rollout counts, uncertainty and detailed task splits are not reported. e07e08 |
| Reading caution | Source description The authors link failures under stronger generalization to implausible generated videos and identify unrealistic hand generation as a limit on dexterous tasks. Figure 6 also shows a plausible generation followed by failed robot execution, so generation quality alone is insufficient. e11e16 |
Core contributions
Figure 2. Human-video generation supplies the plan-like visual context; a separate policy translates that context into robot actions. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the left with the initial image and the instruction to drag the chair. The upper sequence is generated human video; the lower sequence is the robot observation history. Follow the solid arrows through the ViT feature extractors and the two Transformer encoders into the action branch. The dotted pathways lead to off-the-shelf tracking and the auxiliary predictors ψ_g and ψ_r. Those pathways teach the tokens to preserve motion information during training. The caption and Section III-C agree that track prediction is absent at inference; the video and observation features still feed action prediction. e02e04e05e06e12
What it supports. The architecture separates a pretrained generator from the learned translation policy. It does not jointly emit robot actions and future video from one model. Its motion supervision operates through policy representations, allowing deployment to use visual feedback without running the auxiliary track predictors.
Where the evidence stops. The diagram specifies information flow, not a complete training recipe. It does not establish loss weights, action horizon or which visual-encoder parameters are frozen. The prose contains inconsistent token symbols that require implementation clarification.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations cannot easily cover every object, scene and manipulation motion. The paper asks whether a video generator trained on web data can supply task-specific motion information for situations absent from robot training. Its central assumption is that human videos can depict useful manipulation in the robot scene, and that a learned policy can translate those cues across the human–robot embodiment gap. e02e03
2.2 What this reading follows
Imagine asking a robot to close an unfamiliar appliance: a language instruction names the goal, but a generated human demonstration can also suggest how the object should move. Gen2Act uses that demonstration as visual context for a policy trained on robot data. Its key addition is auxiliary point-track prediction, which encourages the policy’s internal tokens to retain motion information during training. At deployment, the policy uses video features and its own observation history to predict actions. This reading follows the separation between generation, representation learning and physical execution, then examines the reported gains, the limited chaining experiments and the failures that expose the remaining gap. e02e03e04e05e08e09e11
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The supplied classification is entirely unassigned, so there is no substantive catalog judgment to confirm. Architecturally this is a separate video generator followed by a learned action policy, not one model jointly predicting future video and actions. Video prediction supplies human motion guidance; the policy translates it with robot feedback. Auxiliary track prediction is training supervision, not an inference-time inverse-dynamics controller. This supports a modular video-to-action interpretation, but does not justify filling an unspecified catalog quadrant. e02e04e05
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn an instruction into task-specific visual context
The first stage is an interface between a broad video prior and a particular robot scene. The initial image anchors the relevant objects and background, while a person/task/static-camera prompt asks VideoPoet to depict human manipulation. During training, this image comes from the first frame of each robot demonstration, automatically creating a corresponding generated human video. The robot arm is reset mostly outside the camera view, helping the input resemble a scene for human interaction. This stage uses an existing generator without fine-tuning. The paired videos then train a separate translation policy. The automatic pairing reduces the need to film a matching human demonstration for every robot trajectory, but it does not make the robot policy itself training-free or establish that every generated demonstration is physically usable. e02e03e07
5.2 Teach compact tokens to retain motion
A video contains many frame features, so Gen2Act compresses the generated sequence and robot observation history into separate fixed-size token sets. Training uses 16 generated frames and eight robot frames, resized to 224 × 224, with 64 tokens per stream after the Perceiver-Resampler encoders. Compression alone need not retain the motion cues the policy needs. The auxiliary task therefore asks token-conditioned transformers to predict tracks obtained offline from an external tracker. Both human-video tokens and robot-history tokens receive this supervision alongside action cross-entropy. At test time, the auxiliary predictors are discarded while their trained representations remain. Figure 2’s dotted track branches and solid action pathway make this distinction explicit: learning to encode motion is different from executing predicted point tracks as robot waypoints. e04e05e06e12
5.3 Connect the ablation to the execution boundary
Reader analysis: the no-track comparison is the most direct test of the proposed motion supervision. Full Gen2Act and its ablation tie at 83% in mild generalization, but differ by 25 percentage points for unseen motion types. This pattern is consistent with motion-aware representations helping when robot training offers weaker support; it does not by itself establish the contribution of each track branch. The co-training result adds a complementary observation: more robot data still improves translation with the generator fixed. Finally, Figure 6 shows that even a plausible video can precede failed execution. Together these findings motivate testing motion sensitivity inside the policy and treating generation quality and robot controllability as distinct requirements. They do not support assuming that a better-looking generator will automatically produce reliable long-horizon behavior. e05e08e10e11
5.4 Training and inference
During training
Training uses existing offline robot demonstrations from the RT-1 work, some paired real human–robot demonstrations from Vid2Robot, and automatically generated human videos. The approximately 400 diverse teleoperated trajectories belong to the additional co-training experiment, not the entire robot training set. e07e10
An off-the-shelf tracker produces target tracks offline. Track transformers with six self-attention layers and eight heads learn to predict these tracks from policy tokens; their losses accompany cross-entropy behavior cloning on action sequences. The paper does not specify loss weights, optimizer, schedule or whether the ViT is frozen. e05e06e12
During inference
Generate a human video, then repeatedly condition the policy on that video and the latest robot observations. Track prediction is unnecessary at test time. The mobile manipulator is manually repositioned between scenes and operates through end-effector control at 3 Hz; autonomous navigation is not evaluated. e05e07
For multiple tasks, Gemini supplies subtask descriptions. After each rollout, its final scene image conditions generation for the next subtask. Appendix C reports subsequent video generation taking less than 10 seconds after the first video, without specifying first-generation latency or compute hardware. e09e15
5.5 Implementation flow
- Generate a human demonstration
VideoPoet receives a square scene image and a prompt of the form “A person task-name, static camera.” During dataset preparation, each robot trajectory supplies its first frame and task instruction; the corresponding human video is generated offline. e03e07
- Compress both visual streams
A ViT extracts frame features. Separate Perceiver-Resampler encoders process the generated video and robot observation history into 64 tokens per stream, using two layers each. Training uses 16 generated frames including the endpoints, eight recent robot frames, and 224 × 224 images. e04e12
- Predict executable actions
Cross-attention blocks combine the two token streams for action prediction. The policy learns the mapping from human motion cues to robot commands without explicit heuristic waypoint extraction. Each action dimension is discretized into 256 uniformly spaced bins; exact physical bounds are not given. e04e06e12
6. Experiments & results
Gen2Act turns a language instruction and an initial scene image into a generated human demonstration, then uses that video to condition a separate closed-loop robot policy. A pretrained VideoPoet supplies the demonstration without robot-specific fine-tuning. Auxiliary point-track prediction teaches policy representations to retain motion cues, while deployment predicts actions directly from video features and recent robot observations. Real robot results support improved generalization relative to the reported baselines, but plausible generation does not ensure correct execution, and long-horizon reliability remains limited.
6.1 Read the original evidence
Figure 4. The same kind of generated human demonstration guides different physical robot manipulations. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel from its red-bordered initial scene into two parallel sequences. The black-bordered upper frames show the generated human demonstration, while the blue-bordered lower frames show robot execution. Compare the intended object change across these rows rather than expecting matching limbs or poses. The pouring panel carries MTG, the scooper panel OTG, the microwave panel G and the apple panel MG. These labels refer to different kinds of novelty relative to robot training data. The mechanism being illustrated is visual conditioning across embodiments, with the policy also receiving its own recent camera observations. e02e07e08e13
What it supports. The panels illustrate the gap the translation policy must bridge: a human hand can depict the desired interaction while a robot follows a different physical trajectory. They show selected executions associated with several generalization categories, complementing the aggregate success rates in Table I.
Where the evidence stops. These are selected static frames, not an independently observed continuous rollout or an exhaustive evaluation set. They cannot establish success frequency, contact accuracy or timing; the quantitative table supplies the reported task-level evidence.
Table I. The strongest generalization gains appear where object or motion types are absent from robot training. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the columns as increasing departures from the robot dataset, not as universal difficulty scores. MG varies seen instances, G changes object instances, OTG changes object types and MTG changes motion types. First compare Gen2Act against the named baselines, then compare its last two rows to inspect track supervision. RT1 uses language and the same robot data; RT1-GC uses final generated goal frames; Vid2Robot uses paired real human–robot videos. Their conditioning and training inputs differ, so the no-track variant is the closer mechanism comparison. The yellow highlighting is part of the original table, not an uncertainty display. e05e07e08
What it supports. Gen2Act reports 58% OTG and 30% MTG success, compared with Vid2Robot’s 25% and 0%. Removing track supervision changes these to 50% and 5%. The 25-percentage-point MTG ablation gap is the clearest table evidence that auxiliary motion supervision matters for unseen motions.
Where the evidence stops. The paper supplies neither main-table rollout counts nor confidence intervals. The ablation removes track supervision without separating its human-video and robot-history branches, so it does not identify which branch causes the gain.
Table II. Sequential regeneration enables short activity chains, with substantial loss of success before the final stage. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the explicit stage list rather than the activity name. Gemini supplies these three descriptions, and each completed rollout’s final scene conditions the next human-video generation. Read the three numbers as cumulative completion through stages one, two and three, as clarified in Section IV-F. They are not three independently measured skills or conditional transition probabilities. Each activity has five trials. Thus equal second- and third-stage entries do not mean a separately established perfect final-stage policy. Appendix C explains why videos are generated sequentially: the actual object state after execution may differ from the initially imagined state. e09e16
What it supports. Reported final-stage completion is 60% for stowing an apple, 40% for cleaning the table, and 20% each for the coffee and soup sequences. These results demonstrate a limited chaining experiment. They do not show uniformly reliable completion of extended household activities.
Where the evidence stops. “Making Coffee” stops at closing the lid; “Heating Soup” stops at closing the microwave. Brewing and heating are outside these tests. Five trials per activity and no recovery-policy baseline sharply limit reliability conclusions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Generalization across robot manipulation settings Physical kitchen, office and lab manipulation; MG changes configurations of seen instances, G uses unseen instances, OTG uses unseen object types, and MTG uses unseen motion types. Seen/unseen is relative to robot training data. | Gen2Act: MG 83, G 67, OTG 58, MTG 30; reported average 60. Executed-task success rate (%) | Vid2Robot: 83, 38, 25, 0; average 37. RT1 average 22; RT1-GC average 26. Gen2Act exceeds Vid2Robot by 33 percentage points on OTG and 30 on MTG. Main-table rollout counts, uncertainty and detailed task splits are not reported. e07e08 |
| Effect of auxiliary point-track supervision Table I compares full Gen2Act with its variant without track loss across the same four generalization categories. | Full model: 83, 67, 58, 30; average 60. Executed-task success rate (%) | Without tracks: 83, 58, 50, 5; average 49. The largest category gain is 25 percentage points on MTG. This supports motion supervision, but does not isolate the two track branches or establish statistical significance. e05e07e08 |
| Additional teleoperation co-training Add approximately 400 diverse robot trajectories while retaining the fixed video generator. | With co-training: MG 85, G 75, OTG 62, MTG 35; average 64. Executed-task success rate (%) | Without co-training: 83, 67, 58, 30; average 60. The reported average rises four percentage points. Changing data quantity and diversity together does not isolate which drives the improvement. e10 |
| Three-stage manipulation through chaining Five trials per activity; Gemini provides three subtasks, with video regeneration after each preceding rollout. | Stowing Apple: 80, 60, 60; Making Coffee: 40, 20, 20; Cleaning Table: 60, 40, 40; Heating Soup: 40, 20, 20. Cumulative success through stages 1, 2 and 3 (%) | No long-horizon baseline or recovery-policy comparison is reported. These are cumulative completion rates, not conditional per-stage probabilities. Coffee ends at lid closure and soup at microwave closure; brewing and heating are not tested outcomes. e09 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table III. Adding roughly 400 diverse robot demonstrations improves the translation policy while the video generator stays fixed. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The top row repeats the full Gen2Act configuration from Table I. The lower row adds approximately 400 diverse teleoperated robot trajectories to the existing offline training data. Compare each column vertically to see where that additional data helps: the four generalization categories retain their robot-data-relative definitions. The video model is kept fixed, so this experiment changes the translation policy’s training support rather than fine-tuning the human-video generator. The caption’s “additional dataset” is essential context: these trajectories are not the whole training corpus, and the source does not provide the complete mixture size or sampling proportions. e07e08e10
What it supports. The reported average rises from 60% to 64%, with OTG increasing from 58% to 62% and MTG from 30% to 35%. The modular approach still benefits from more robot demonstrations; the video prior does not remove the need for adequate translation data.
Where the evidence stops. Co-training changes both the amount and diversity of robot data. There is no equal-size diversity control or reported uncertainty, so the table does not isolate a data-diversity mechanism or establish the gain’s statistical stability.
Figure 6. Failure can originate in the generated demonstration or in its translation to robot motion. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from the task-labeled scene at left through the generated human sequence in the middle and the blue-bordered robot sequence at right. The authors describe the first three rows as failures associated with incorrect generation. In the last row, lifting the scooper, they instead regard the generated video as plausible and attribute the execution problem to following the object trajectory after grasping. This distinction matters because it separates two necessary capabilities: depicting a useful manipulation and physically realizing it. All four illustrated tasks are classified as object-type generalization in the caption. e11e16
What it supports. The final row supplies a concrete boundary on the paper’s central idea: a plausible human demonstration is not sufficient for successful robot manipulation. The authors’ diagnosis points toward both improving generated motion and improving grasping or post-grasp execution, rather than treating all failures as generator errors.
Where the evidence stops. These selected examples are qualitative diagnostics, not a controlled correlation measurement or an exhaustive failure taxonomy. Static frame sequences cannot identify all contact errors or prove the causal origin of an individual failure.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors link failures under stronger generalization to implausible generated videos and identify unrealistic hand generation as a limit on dexterous tasks. Figure 6 also shows a plausible generation followed by failed robot execution, so generation quality alone is insufficient. e11e16
The generalization labels concern absence from robot training, not necessarily from the video model’s web pretraining. The comparison does not demonstrate novelty relative to all pretraining sources, and the qualitative failure examples cannot establish a causal failure rate. e07e11
Only five long-horizon trials per activity are reported, and recovery policies are future work. The activity names are broader than their operational three-step evaluations. e09e16
7.2 Questions for discussion
- Does temporal order matter after controlling for the generated frames and endpoints?
- Which track-supervision branch contributes most to unseen-motion execution?
- How often can the policy recover from a plausible but physically unsuitable generated demonstration?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the relevant VideoPoet model, offline robot and paired human–robot data, a point tracker, camera observations and end-effector action labels. Exact dataset sizes/mixture, checkpoints, seeds, compute budget, video sampling settings, point counts and action horizon are not specified here. e03e07e12
Section IV-B describes adapting VideoPoet to square-image conditioning, while Appendix A says no adaptation or fine-tuning. Both reject fine-tuning, but the interface adjustment is unexplained. The track text also mixes z_g/z_r with g_e/r_e notation and inconsistent history indices; an implementation must resolve these without assuming new architecture. e03e05e07e12
Proposed checks should isolate human versus robot track supervision at fixed data and compute, then perturb generated-video temporal order while preserving frames. These test whether motion information, rather than generic extra supervision or object appearance, explains the gains. e05e08e11
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate the two sources of track supervision
Reader-proposed experiment: train four matched policies using no track loss, generated-human tracks only, robot-history tracks only, and both. Hold demonstrations, generated videos, frame sampling, initialization schedule, action head and training updates fixed; match total auxiliary-loss scale where applicable. Evaluate the same predeclared MG/G/OTG/MTG tasks with repeated seeds and recorded rollout counts. Compare executed-task success and auxiliary prediction error. If the human-video branch supplies the transferable motion cue, removing it should disproportionately reduce MTG success. Comparable gains from either branch would instead support a more general representation-regularization explanation. This is a proposed test, not a reproduced result. e05e07e08e12
Check 2: Test temporal information while controlling appearance
Reader-proposed experiment: for a fixed trained policy, compare each normal generated sequence with versions that shuffle or reverse only its interior frames, preserving the first frame, last frame and frame set. Keep the scene, task, robot reset and observation feedback matched, and randomize rollout order. Report execution success separately for MG and MTG, with repeated trials and confidence intervals. A larger MTG drop would support dependence on temporal motion structure beyond appearance and goal information. Little change would weaken that interpretation. Since reordered clips are distribution shifts, include a matched training-time order-augmentation control before attributing any drop solely to motion reasoning. e05e07e08e11e12
8.3 Reading coverage
Visual audit: Inspected the title, byline, affiliations and v1 date on page 1; Figures 1–6 and Tables I–III on pages 1–6 and 11; and all method, training, evaluation, latency and configuration text supporting this edition on pages 2–6 and 10–11. All six final original crops were individually viewed. Figure 2’s solid action path and dotted track-supervision branches were checked against its caption and Section III-C. Appendix notation inconsistencies and VideoPoet adaptation wording are disclosed in the base report. Figure 5 visibly contains three activity strips although Section IV-F refers to four. Bibliography pages 7–9 were read in the complete text but not visually inspected. Project videos, external code and separate supplements remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 10, 11. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and abstract (PDF p. 1)
- I. Introduction (pp. 1–2)
- II. Related Works (pp. 2–3)
- III. Approach, A–D (pp. 3–4)
- IV. Experiments, A–H (pp. 5–6)
- V. Discussion and Conclusion (p. 6)
- Acknowledgements and References (pp. 7–9)
- Appendix A. Human Video Generation (p. 10)
- Appendix B. Closed-Loop Policy (p. 10)
- Appendix C. Chaining Gen2Act for Long-Horizon Manipulation (p. 10)
- Appendix D. Analysis of Failures (p. 11)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The inspected title page identifies arXiv:2409.16283v1 [cs.RO], 24 September 2024. Its title and all ten authors match the supplied catalog; affiliations were verified from the title-page footnotes. No other revision or edition was supplied or compared.
- The acquisition warning that text extraction does not reconstruct figures was addressed by inspecting the retained PDF: scientific pages 1–6 and 10–11, all six figures and all three tables. Bibliography pages 7–9 were read as text.
- Separate supplemental material availability has not been fully verified. The linked project videos and external resources were not inspected; only the PDF frame sequences were visually read.
- Code, model artifacts and datasets were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/byline, affiliation footnotes and arXiv margin
The title and ten authors match the catalog. The margin identifies arXiv:2409.16283v1 [cs.RO], 24 Sep 2024. Affiliations are Google DeepMind; The Robotics Institute, Carnegie Mellon University; Computer Science Department, Stanford University.
Go to primary source ↓e02PDF pp. 1–3, Section I; Figure 2 and caption (p. 2); Section III-A (p. 3)
The system factors language-conditioned manipulation into human video generation and a separate closed-loop policy using recent observations. Figure 2 distinguishes solid inference pathways from dotted training-specific track pathways.
Go to primary source ↓e03PDF p. 3, Section III-B; p. 10, Appendix A
Human videos are generated from the first robot-trajectory image and task language, without fine-tuning. Appendix A specifies a square image, an arm reset mostly outside camera view and a person/task/static-camera prompt. Expected improvement with future generators is an author projection.
Go to primary source ↓e04PDF p. 2, Figure 2; p. 4, Section III-C, Visual Feature Extraction; p. 10, Appendix B
Both video streams pass through ViT features and separate Perceiver-Resampler encoders. The diagram joins their tokens through cross-attention for action prediction; the prose rejects heuristic waypoint extraction.
Go to primary source ↓e05PDF p. 4, Section III-C, Point Track Prediction and BC Loss; p. 10, Appendix B
Offline tracked points supervise generated-video and robot-history token representations through auxiliary track predictors. The human-track loss is written as an L2 norm; track predictors are unused at test time. Some token symbols and history indices are inconsistent in the source prose.
Go to primary source ↓e06PDF p. 4, Section III-C, BC Loss; p. 10, Appendix B, action prediction
Action dimensions use 256 uniformly distributed bins and cross-entropy supervision for predicted action sequences. Appendix B names end-effector, gripper and termination predictions; it does not give physical bin bounds or a concrete action horizon.
Go to primary source ↓e07PDF p. 5, Section IV-A–C
Defines MG/G/OTG/MTG relative to robot interaction data, task completion success, 3 Hz end-effector manipulation and manual movement between scenes. Training combines prior offline robot and paired human–robot data with generated videos. RT1 uses the same robot data; RT1-GC uses goal frames; Vid2Robot uses paired real videos. Main rollout counts and exact data mixtures are unspecified.
Go to primary source ↓e08PDF p. 5, Table I, all rows and MG/G/OTG/MTG/Avg. columns
RT1: 68/18/0/0/22; RT1-GC: 75/24/5/0/26; Vid2Robot: 83/38/25/0/37; Gen2Act without tracks: 83/58/50/5/49; Gen2Act: 83/67/58/30/60. Values are reported task-success percentages, without uncertainty intervals.
Go to primary source ↓e09PDF p. 4, Section III-D; p. 6, Table II, Figure 5 and Section IV-F; p. 10, Appendix C
Gemini supplies subtasks and the last rollout image conditions the next generated video. Table II reports cumulative success across five trials: apple 80/60/60, coffee 40/20/20, cleaning 60/40/40, soup 40/20/20. Coffee ends at lid closure; soup ends at microwave closure. Figure 5 displays three activities, although IV-F refers to four.
Go to primary source ↓e10PDF p. 6, Table III and Section IV-G
Approximately 400 additional diverse teleoperated trajectories are used for co-training with the existing dataset, keeping video generation fixed. Without co-training the category rates and average are 83/67/58/30/60; with co-training they are 85/75/62/35/64.
Go to primary source ↓e11PDF p. 6, Section IV-H; p. 11, Figure 6 and Appendix D
The authors associate incorrect video generation with harder-generalization failures. Figure 6 shows fridge closing, spoon picking, fridge opening and scooper lifting; its last example has plausible generation but failed execution after grasping. These are qualitative examples rather than a quantified correlation study.
Go to primary source ↓e12PDF p. 10, Appendix B
Specifies 64 tokens per stream, two Perceiver-Resampler layers, 16 generated frames including first/last, eight robot frames, 224 × 224 resizing, and six-layer/eight-head track transformers. Optimizer, schedule, loss weights, hardware, encoder freezing and exact checkpoints are not specified.
Go to primary source ↓e13PDF p. 4, Figure 4 and caption
Four panels pair generated human frames with robot executions: pouring (MTG), closing a microwave (G), grabbing a scooper (OTG), and placing an apple in a drawer (MG). Red input frames, black generated sequences and blue robot sequences distinguish their roles.
Go to primary source ↓e14PDF p. 3, Figure 3 and Section III-B; p. 5, Section IV-D
Selected generated frame sequences depict banana picking, sink wiping and door closing. The text presents these as qualitative demonstrations of scene-preserving zero-shot human video generation; no aggregate video-quality benchmark is provided.
Go to primary source ↓e15PDF p. 10, Appendix C, generation-latency paragraph
The source reports less than 10 seconds to generate a new video after the first generation. It supplies no initial latency, compute configuration or end-to-end timing benchmark.
Go to primary source ↓e16PDF p. 6, Section V, Limitations and Future Work
The authors identify realistic-hand generation and dexterous tasks as limitations, propose denser motion information, and leave recovery policies for reliable long-horizon manipulation to future work.
Go to primary source ↓8.5 Primary sources
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation ↗
PDF · 8,259 extracted words
Source fingerprint
5d48dce0af6dc2a77904099c7454955c381c2e387ae38ed4acb6493bbcbd0295