Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow
1. Paper overview
In one sentence: Dream2Flow turns generated object motion into robot tracking goals, gaining flexibility across embodiments while inheriting errors from video generation, 3D reconstruction and domain-specific control. e-framinge-flowe-architecturee-controle-taskse-sac
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | Source description A modular video-to-control pipeline turns desired object motion into tracking targets for pushing and grasped manipulation, or rewards for simulated policy learning. e-architecturee-control |
| A key reported result | Open Oven: Dream2Flow 8/10 Success count; opening at least 60 degrees. Real robot; Veo 3; ten randomized oven placements facing the robot. Adapted AVDC 0/10; adapted RIGVID 6/10 The strongest real-task separation from AVDC; only two additional successes over RIGVID. e-taskse-real |
| Reading caution | Source description Rigid-grasp real-world dynamics restrict contact changes and deformation. Single-view occlusions can destroy tracks. Obtaining flow takes 3–11 minutes, mainly due to video generation. e-limits |
Core contributions
- Source description
A modular video-to-control pipeline turns desired object motion into tracking targets for pushing and grasped manipulation, or rewards for simulated policy learning. e-architecturee-control
- Reader analysis
The authors argue that this interface supports open-world manipulation without task-specific demonstrations. This claim should not be read as eliminating dynamics training or reinforcement learning. e-framinge-particlee-sac
Figure 2. Object trajectories connect video prediction to a separate robot controller. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the instruction and initial RGB-D observation on the left. The upper branch generates the human-interaction frames; the lower branch extracts an object mask. Follow the mask’s lower arrow into 2D tracking, while the generated frames supply both tracking and video depth. These two streams meet at the colored 3D point trajectories. Section III-B supplies a detail omitted from the drawing: predicted depths are aligned to the initial measured depth before camera geometry lifts the tracks. The final arrow enters a separate robot policy, whose implementation depends on whether the domain uses pushing, grasped motion or reinforcement learning. e-architecturee-flowe-controle-realplan
What it supports. The transferable target is object motion in the robot’s coordinate frame. The robot need not reproduce the generated hand trajectory to achieve that target. The figure therefore explains why the same interface can feed different control procedures, while leaving action feasibility to those procedures.
Where the evidence stops. This is a modular information-flow diagram. It does not establish joint video/action training, an action-conditioned video simulator, or universal executability of a generated trajectory. Real-world control still assumes a rigid grasp.
2. Motivation
2.1 The problem and the proposed response
A plausible human demonstration in pixel space does not specify executable robot actions. The paper asks whether reconstructed object motion can express the task while a separate controller handles the robot’s kinematics and dynamics. e-framinge-flow
2.2 What this reading follows
Imagine asking a robot to open an oven by first generating a video of a human doing it. The useful transferable signal is the oven door’s motion: the robot still needs to find a grasp and move within its own limits. Dream2Flow makes that separation explicit by reconstructing object-point trajectories in 3D. These trajectories guide either optimization or reinforcement learning. The experiments test both sides of this interface: which generated motions survive reconstruction, and which controllers can realize them. Read the real-robot tables alongside the simulated studies, because their supervision, success criteria and evidence for transfer differ substantially. e-framinge-flowe-architecturee-controle-taskse-sac
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is entirely unassigned, so there is no substantive label to confirm. Architecture evidence supports a modular video-generation, perception and control pipeline, not a single model jointly predicting future video and actions. Action recovery uses forward-dynamics optimization or RL; calling it a learned inverse-dynamics network would misdescribe the implementation. e-architecturee-flowe-controle-sac
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a visual prediction into a metric object target
Begin with the observation’s two different roles. The RGB image conditions a video model to imagine the task, while the measured initial depth anchors the generated geometry. Monocular video depth has scale and shift ambiguity, so aligning its first frame to the robot’s depth is necessary before interpreting a tracked pixel as a point in robot coordinates. Grounding DINO and SAM 2 select the relevant object, and CoTracker3 provides correspondences and visibility through time. For articulated objects, Appendix E further separates moving from stationary points and rejects tracks outside a propagated part mask. The result is a time-indexed set of object positions. Its usefulness depends on maintaining identity and geometric consistency; hallucinated objects and boundary-tracking errors can invalidate an otherwise plausible-looking demonstration. e-flowe-filtere-limits
5.2 Compile the same target into different physical actions
A target trajectory alone does not tell the robot how to make contact. In Push-T, the system samples pushes that contact the block, predicts their effects with learned dynamics, and chooses the push whose resulting points best approach a reference subgoal. It observes the outcome and replans, using a look-ahead of 20 video timesteps from the closest current shape. Physical grasped tasks use a different construction. AnyGrasp proposes contacts, generated thumb motion helps select one, and rigid-grasp dynamics link object-point movement to the end-effector. Joint optimization adds reachability, smoothness and manipulability costs before trajectory execution. Reader interpretation: the shared interface is the desired object motion, while the feasibility assumptions and feedback mechanisms remain specific to each controller. e-controle-pushplane-graspe-realplan
5.3 Separate zero-shot task guidance from policy training
The door experiment changes how the reference is used. Instead of optimizing a fresh action sequence against point error, SAC trains a policy with a reward combining reference progress and end-effector proximity. The closest point configuration selects a timestep; moving toward later reference timesteps earns more progress reward. Appendix H computes current particles by transforming initial particles with the simulator’s door angle, so the reward is not exposed to the full visual-tracking difficulty of the physical pipeline. Policies then learn strategies compatible with each embodiment. Reader interpretation: this supports generated motion as a reward specification, not training-free motor competence. Likewise, the strong Push-T result depends on a separately trained dynamics model. The off-the-shelf video prior and the learned control components should remain distinct when assessing reproducibility and taxonomy. e-sace-doore-controle-particlee-dynamics
5.4 Training and inference
During training
The video/perception components are used off the shelf. Push-T separately trains a small Point Transformer V3 with MLP projections on 500 random-push transitions. Its particle features contain position, RGB, normals and push parameters; training correspondences use simulator object poses. e-framinge-particle
Door-opening SAC policies learn from generated-flow rewards in simulation. Training uses 5000 iterations, or 10000 for GR1, with episode horizon 500. Current particles are computed from the simulator’s door angle, not online visual tracking. e-sac
During inference
Real execution optimizes joint configurations using flow error plus reachability, smoothness and manipulability costs; B-spline end-effector poses feed PyBullet IK and a Deoxys joint-impedance controller. Unlike Push-T, this description does not specify online flow replanning during execution. e-realplane-pushplan
In the RL branch, reference flow supplies the training reward and the learned policy supplies actions. This is distinct from calling the video generator as an action-conditioned simulator at every control step. e-controle-sac
5.5 Implementation flow
- Imagine the object motion
Generate video from the initial RGB image and instruction. Real prompts specify one hand and a still camera; these requirements support grasp selection and the fixed-camera depth assumption. e-flowe-prompts
- Recover calibrated flow
SpatialTrackerV2 estimates video depths; a global scale and shift align the first depth to measured RGB-D. Grounding DINO localizes the object, SAM 2 masks it, and CoTracker3 tracks sampled points and visibility. Camera geometry lifts valid tracks into 3D. e-flow
- Plan pushes with feedback
Random shooting evaluates contact-making push primitives through learned particle dynamics. After each executed push, tracking updates the object points. The next target lies 20 reference timesteps beyond the closest current shape, capped at the video end. e-controle-pushplan
- Select contact and optimize grasped motion
AnyGrasp candidates are matched to HaMer’s generated thumb location within 2 cm, with a movable-part fallback. Motion-based filtering and propagated SAM 2 masks reject inappropriate tracks. Rigid-grasp dynamics move grasped points with the end-effector while leaving other points unchanged. e-graspe-filtere-control
6. Experiments & results
Dream2Flow converts generated human-interaction videos into 3D object trajectories, then uses domain-specific optimization or reinforcement learning to make a robot realize them. Its central benefit is an object-level interface across embodiments; its reliability still depends on video geometry, tracking, contact assumptions and the downstream controller.
6.1 Read the original evidence
Table I. Real-robot gains are clearest for oven opening, while cloth covering remains difficult. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a separate ten-trial manipulation experiment, with methods compared across columns. The first two rows involve fake bread placement and an articulated oven door; the third uses a scarf. Their success criteria differ: bread must finish inside the bowl, the oven must open at least 60 degrees, and the scarf must cover at least 25% of the bowl top. Section IV-B specifies Veo 3 for these real-world experiments. Section IV-C is essential context: AVDC uses modified depth/transform estimation, while RIGVID is adapted to fit rigid transforms to reconstructed points instead of running its original pose tracker. e-taskse-real
What it supports. Dream2Flow reports 8/10 successes for both bread placement and oven opening, and 3/10 for covering. The oven comparison is 0/10 for AVDC and 6/10 for RIGVID. These counts support an advantage for the implemented interface, but the scarf result makes the remaining absolute difficulty visible.
Where the evidence stops. The baselines are adapted implementations, and each row has only ten trials. No confidence interval establishes the precision of the differences. Treat the table as executed task evidence under these settings, not a general ranking of complete original systems.
Table II. Flow-derived rewards approach the handcrafted reward’s observed door-opening performance. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each embodiment column to compare the two reward formulations. These entries are evaluations of separately trained SAC policies over 100 random door positions, not immediate execution of a generated video. The Franka and floating-base Spot attain 100/100 with flow rewards; the right-arm GR1 attains 94/100. Consult Appendix H when interpreting the reward labels: flow progress is the closest reference timestep divided by the final timestep, combined with a reaching term. Current object particles are transformed using the simulator’s door angle. This makes the table a test of an object-motion reward under state-assisted simulation. e-doore-taskse-controle-sac
What it supports. The flow reward gives 100/100, 100/100 and 94/100 for Franka, Spot and GR1, compared with 99/100, 99/100 and 96/100 from the handcrafted reward. The result supports comparable observed task performance across these embodiments, without showing a consistent advantage for either reward formulation.
Where the evidence stops. Policies are trained separately, and GR1 receives more training iterations. Simulator-derived particle positions remove a visual-tracking burden present in physical use. Neither statistical equivalence nor transfer of one shared policy across embodiments is established.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Open Oven Real robot; Veo 3; ten randomized oven placements facing the robot. | Dream2Flow 8/10 Success count; opening at least 60 degrees | Adapted AVDC 0/10; adapted RIGVID 6/10 The strongest real-task separation from AVDC; only two additional successes over RIGVID. e-taskse-real |
| Put Bread in Bowl Real robot; Veo 3; ten trials with randomized fake-bread and bowl positions. | Dream2Flow 8/10 Success count; bread finishes inside bowl | Adapted AVDC 7/10; adapted RIGVID 6/10 |
| Cover Bowl Real robot; Veo 3; ten trials with randomized folded scarf position/orientation. | Dream2Flow 3/10 Success count; at least 25% of bowl top covered | Adapted AVDC 2/10; adapted RIGVID 1/10 Deformable manipulation remains unreliable despite the relative improvement. e-taskse-real |
| Push-T dynamics ablation OmniGibson; Wan2.1 with start/goal images; ten initial states × ten random-shooting seeds. | Particle dynamics 52/100 Success count; within 2 cm translation and 15 degrees rotation | Pose dynamics 12/100; translation-only heuristic 17/100 Controller dynamics matter even with the same flow guidance; learned models share training data, but inputs/architectures differ. e-taskse-dynamicse-particle |
| Open Door reward comparison Robosuite; 100 random door positions per embodiment; separately trained SAC policies. | Flow reward: Franka 100/100, floating-base Spot 100/100, right-arm GR1 94/100 Success count; rotate handle and open door at least 17 degrees before timeout | Object-state reward: 99/100, 99/100, 96/100, respectively Comparable observed performance, with simulator-state assistance and no physical deployment established by this table. e-taskse-doore-sac |
| Video-generator comparison Push-T uses goal-image conditioning; real Open Oven uses ten trials per generator. | Wan2.1: Push-T 52/100, oven 2/10; Kling 2.1: 31/100, 4/10; Veo 3: oven 8/10 Downstream task success counts | Veo 3 Push-T is unreported because end-frame conditioning was unavailable. Generator ordering depends on domain; the missing cell is not a failure rate. e-taskse-generators |
| Real-world pipeline failure breakdown The authors’ combined set of 60 real-world Dream2Flow trials. | 40 successes; 12 video failures, 4 flow-extraction failures, 4 execution failures Stage-specific counts and completed executions | Figure 7 divides video failures into 6 morphing and 6 hallucination cases. Video generation is the largest recorded failure category; this pooled count is separate from the three ten-trial task rows. e-failures |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table III. The most successful generator changes with the task and conditioning protocol. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare generators vertically within a task column. Push-T is simulated and uses both start and goal images, with 100 controller trials; Open Oven is physical and uses ten trials. Wan2.1 leads Kling 2.1 on Push-T, while Veo 3 has the best observed oven count. The dash in Veo 3’s Push-T cell is an unavailable experiment: at evaluation time the model did not support the required end-frame prompt. Sections IV-E and Appendix A explain why the inputs matter, including the still-camera assumption and the articulation errors that can make generated trajectories unsuitable for execution. e-generatorse-taskse-prompts
What it supports. Wan2.1 reaches 52/100 on Push-T versus Kling’s 31/100. On the oven task, Veo 3 reaches 8/10 versus Kling’s 4/10 and Wan’s 2/10. These are downstream robot success counts, so they measure the generator’s usefulness to this entire pipeline rather than visual quality alone.
Where the evidence stops. The two columns use different domains, denominators and conditioning. Kling also receives a relevance setting and negative prompt. Missing Veo 3 conditioning support prevents a complete crossed comparison; it cannot be interpreted as zero success.
Table IV. Good object-flow targets still require a dynamics model that predicts useful pushes. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Hold the object-flow guidance conceptually fixed and read down the dynamics-model rows. The pose model predicts a block-pose change from its current pose and push parameters. The heuristic simply translates object points with the push and cannot model rotation. The particle model predicts scene-point displacements from geometric, appearance and push features. Section IV-F states that the learned pose and particle models share their training data; Appendix B identifies 500 random-push transitions for the particle model. Appendix C then explains how random shooting uses predicted points to select each executed push and replans from the resulting observation. e-dynamicse-particlee-pushplane-taskse-limits
What it supports. Particle dynamics succeeds in 52/100 trials, compared with 12/100 for pose dynamics and 17/100 for the heuristic. This supports the importance of downstream dynamics in the tested Push-T setup. The authors specifically attribute poor alternatives to difficulty accounting for required rotation.
Where the evidence stops. The ablation changes representation and model structure together. Equal training data does not imply equal capacity or optimization quality. These counts do not isolate particle coordinates as the sole cause, and no real-world particle-dynamics result is supplied.
Figure 7. The failure accounting locates losses before and after 3D reconstruction. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the bands from the 60 trials at the left toward robot execution on the right. Twelve trials branch into video-generation failures, leaving 48 video successes. Four further trials fail flow extraction, leaving 44, and four more fail execution, leaving 40 successes. The upper labels divide the video failures into six morphing and six hallucination cases. These counts are sequential: the four execution failures are drawn from the 44 flow successes. Section IV-B associates extraction failures with rotation or leaving the camera view, and the execution failures with unsuccessful grasping or insufficient motion during bowl covering. e-failurese-real
What it supports. Video failures are the largest recorded category, preceding both tracking and robot execution. The figure also shows why successful generation alone cannot establish completed manipulation: eight additional failures occur downstream. Its 40 successes describe the authors’ pooled real-world set, rather than any one row in Table I.
Where the evidence stops. Figure 7 labels six failures in each video category, whereas Section IV-B’s “for half the time” wording is ambiguous. Counts here follow the diagram. This descriptive breakdown does not quantify how many failures a particular intervention would prevent.
7. Analysis & limitations
7.1 What the evidence leaves open
Rigid-grasp real-world dynamics restrict contact changes and deformation. Single-view occlusions can destroy tracks. Obtaining flow takes 3–11 minutes, mainly due to video generation. e-limits
Table I evaluates adapted baseline interfaces, not untouched original systems. Ten-trial real tasks and five-trial robustness variants provide limited precision; reported counts lack confidence intervals or multi-training-seed statistics. e-reale-robustnesse-door
Chair, drawer, pasta and can demonstrations are qualitative case studies. They support task breadth but do not establish repeated-trial generalization rates. e-wild
7.2 Questions for discussion
- How much of particle dynamics’ advantage remains after matching model capacity, inputs and planning compute?
- Can visually estimated door particles preserve the reward comparison without simulator joint-angle access?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruct calibrated RGB-D geometry and pinned video/perception components. Push-T uses four virtual cameras and 1.5 cm point-cloud downsampling; real planning uses task/reachability/smoothness/manipulability weights 10/100/1/0.01. e-flowe-particlee-realplan
SAC’s recipe includes learning rate 3e-4, batch 256 and two 256-unit hidden layers. Missing details include the particle-model optimizer/loss schedule and held-out split, random-shooting sample count, maximum pushes, and a complete compute/software-version specification. e-particlee-pushplane-sac
Proposed checks should hold generated targets fixed when comparing dynamics and separately replace simulator-derived door particles with visual tracks. Measure executed success alongside rotation error or reward-progress error to test each mechanism. e-dynamicse-pushplane-sac
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does particle dynamics improve rotational control under matched planning?
Reader-proposed experiment, not run: use the same 500 random-push transitions, generated reference flows, initial states, candidate pushes and look-ahead L=20 for pose and particle dynamics. Match training budgets, report model capacity, and repeat training across seeds. Evaluate ten initial states with ten controller seeds each, stratified by required rotation. Record held-out one-push rotational prediction error, final translation/yaw errors and the paper’s 2 cm/15-degree success criterion. A persistent particle advantage concentrated in rotation would support the authors’ explanation; disappearance after capacity or planning controls would weaken the claim that representation drives the gain. e-particlee-pushplane-dynamicse-tasks
Check 2: How much does the flow reward depend on simulator-derived particles?
Reader-proposed experiment, not run: train matched Franka SAC policies with the same reference, hyperparameters, observation fields and training budget. Change only the flow-reward particle estimate: simulator-angle transformations versus RGB-D tracking. Retain a handcrafted-reward control and repeat training seeds. Evaluate 100 randomized door placements, adding controlled occlusions; record success, lost-track frequency and closest-reference-index error against simulator ground truth. A selective collapse of the tracked-flow condition would identify a perception dependency hidden by Table II. Disclose any door-angle inputs retained in policy observations: this comparison isolates reward estimation and would not by itself establish fully vision-based control. e-sace-doore-flowe-limits
8.3 Reading coverage
Visual audit: All six supplied text chunks were read individually, covering the complete 13-page source including references and Appendices A–J. The title/byline/version on page 1 and every declared page were rendered and actually viewed. Visual inspection covered Figures 1–12, Tables I–V, method equations, task definitions, baseline adaptations, particle training, planning, grasp/filtering details and SAC configuration/rewards. All six final crops were separately viewed: Figure 2, Tables I–IV and Figure 7. Figure 2 arrows agree with its caption and Sections III-A–C; Figure 7’s category counts are retained with the prose ambiguity disclosed. Bibliography-only pages 8–9 were text-read but not visually inspected. External videos, code and separate supplements were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, byline, abstract
- PDF pp. 1–2: I. Introduction
- PDF pp. 2–3: II. Related Works, A–C
- PDF pp. 3–4: III. Method, A–C
- PDF pp. 4–7: IV. Experiments, A–F
- PDF p. 7: V. Conclusion and Acknowledgments
- PDF pp. 7–9: References
- PDF pp. 10–13: Appendix A–J, including prompts, dynamics, planning, grasp selection, filtering, case studies, SAC rewards and limitations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure limitation was addressed by inspecting the original PDF pages and every final crop. Bibliography-only pages 8–9 were read as text, not rendered.
- No separate supplements, external project videos or code were inspected; no experiments were reproduced.
- Identity/version: the inspected title page is arXiv:2512.24766v1, 31 December 2025, matching the supplied title and complete five-author list. No revision or edition discrepancy was observed; no other version was supplied.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, affiliation, arXiv margin and advising footnote
The exact observed title and all five catalog authors match. The artifact identifies arXiv:2512.24766v1, 31 December 2025, Stanford University; Li Fei-Fei and Ruohan Zhang share advising credit.
Go to primary source ↓e-framingPDF pp. 1–2, Abstract and Section I
Generated human interactions supply desired object motion; reconstructing and tracking that motion separates task specification from embodiment-dependent action realization.
Go to primary source ↓e-flowPDF p. 3, Sections III-A and III-B, unnumbered optimization and depth-calibration equations
RGB-D and calibrated camera geometry anchor masked video point tracks in robot coordinates. SpatialTrackerV2 estimates depth, Grounding DINO and SAM 2 select objects, and CoTracker3 supplies tracks and visibility. The task cost sums squared point-target distances.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and caption
Instruction and initial observation feed video generation; a mask selects tracks, video depth and 2D tracks reconstruct 3D object flow, and a separate robot policy realizes the flow.
Go to primary source ↓e-controlPDF pp. 3–4, Section III-C, all three domain paragraphs
Push-T uses random shooting with learned particle dynamics; real manipulation uses grasp selection, rigid-grasp dynamics and optimization; simulated door opening trains SAC policies with an object-flow reward.
Go to primary source ↓e-tasksPDF p. 4, Sections IV-A and IV-B; p. 5, Section IV-B continuation
Task success definitions are 2 cm/15 degrees for Push-T, bread inside the bowl, oven opening of at least 60 degrees, scarf covering at least 25% of bowl top, and door opening of at least 17 degrees after handle rotation. Push-T uses ten initial states and ten random-shooting seeds each; real tasks use Veo 3 and ten trials each.
Go to primary source ↓e-realPDF p. 6, Table I, all rows/columns, and Section IV-C
Bread success is AVDC 7/10, RIGVID 6/10, Dream2Flow 8/10; oven 0/10, 6/10, 8/10; cover bowl 2/10, 1/10, 3/10. AVDC is modified to use video depths and initial-frame transforms; RIGVID is adapted to fit rigid transforms to the extracted 3D points instead of using a 6D pose tracker.
Go to primary source ↓e-doorPDF p. 6, Section IV-D, Table II, and Figure 6/caption
Across 100 randomized door positions per embodiment, object-state reward yields Franka 99/100, Spot 99/100 and GR1 96/100; flow reward yields 100/100, 100/100 and 94/100. Rollouts illustrate different strategies with a floating-base Spot and right-arm GR1.
Go to primary source ↓e-generatorsPDF p. 6, Table III and Section IV-E; p. 7, Section IV-E continuation; p. 10, Appendix A
Wan2.1/Kling 2.1 achieve 52/100 and 31/100 on Push-T; Wan2.1/Kling 2.1/Veo 3 achieve 2/10, 4/10 and 8/10 on Open Oven. Veo 3 lacks a Push-T result because it did not support end-frame conditioning at evaluation time. The text reports morphing, camera motion and incorrect articulation.
Go to primary source ↓e-dynamicsPDF p. 7, Table IV and Section IV-F
Push-T success is pose dynamics 12/100, translation-only heuristic 17/100 and particle dynamics 52/100. Learned pose and particle models use the same data; authors attribute failures to inadequate modeling of required rotation.
Go to primary source ↓e-failuresPDF p. 7, Figure 7/caption; p. 5, Section IV-B failure paragraph
Figure 7 partitions 60 trials into 12 video failures (six morphing, six hallucination), four later flow-extraction failures, four execution failures and 40 successes. Section IV-B attributes tracking failures to rotation/out-of-view motion and execution failures to bowl covering; its phrase about half the video failures is less clear than the diagram.
Go to primary source ↓e-promptsPDF p. 10, Appendix A and Figure 8
Real prompts request one hand and a still camera; cloth prompts additionally constrain motion. Push-T supplies start and goal images. Door prompts omit the hand. Kling adds relevance 0.7 and a negative prompt for fast motion, morphing and camera motion.
Go to primary source ↓e-particlePDF pp. 10–11, Appendix B and Figure 9
A small Point Transformer V3 with input/output MLPs maps 14-dimensional particle features to displacement. Four virtual RGB-D cameras supply a cropped point cloud, downsampled at 1.5 cm. Training uses 500 random-push transitions and simulator object poses for particle correspondence; optimizer, loss details and held-out split are not specified here.
Go to primary source ↓e-pushplanPDF p. 11, Appendix C
After every push, online tracking and depth update object points. Nearest-neighbor matching links tracked particles to scene particles. Random shooting targets reference time min(t-star + L, t-end), with L=20; sample count r and maximum push count are left unspecified.
Go to primary source ↓e-graspPDF p. 11, Appendix D and Figure 10
AnyGrasp proposes up to 40 top-down grasps from two coordinate frames. HaMer thumb proximity within 2 cm selects an early contact; if unavailable, selection falls back to proximity to the movable-part points.
Go to primary source ↓e-filterPDF pp. 11–12, Appendix E
Points moving at least one pixel per timestep on average define the movable part. SAM 2 uses positive movable and negative stationary point prompts, invalidating tracks outside the propagated part mask.
Go to primary source ↓e-realplanPDF p. 12, Appendix F, Eqs. (1)–(2) and execution paragraph
Real planning optimizes seven joint angles with reachability, pose smoothness and manipulability weights 100, 1 and 0.01, and task weight 10. End-effector poses are fit by a B-spline and sampled using 1 cm and/or 20-degree separation, then executed through PyBullet IK and Deoxys joint impedance on a Franka.
Go to primary source ↓e-wildPDF p. 12, Appendix G and Figure 11
Qualitative rollouts cover chair pulling, opening a partially open drawer, sweeping four pasta pieces and recycling a can. Success criteria are provided, but no repeated-trial quantitative table accompanies these case studies.
Go to primary source ↓e-sacPDF pp. 12–13, Appendix H, Table V and Eqs. (3)–(7)
SAC uses learning rate 3e-4, discount 0.99, batch 256, replay capacity 1e6, target update rate 0.005, two 256-unit hidden layers, 5000 iterations and horizon 500; GR1 uses 10000 iterations. Flow reward combines weighted reference progress and reaching. Object particles are transformed from door joint angle rather than visually tracked; the door angle is included in state.
Go to primary source ↓e-limitsPDF p. 13, Appendices I–J and Figure 12
Authors identify rigid-grasp restrictions, heavy single-view occlusion and 3–11 minutes to obtain flow, mainly spent on video generation. Figure 12 illustrates bread morphing into crackers and hallucination of another bowl.
Go to primary source ↓e-robustnessPDF p. 5, Section IV-B and Figures 4–5
Robustness testing uses five additional trials for each of six instance/background/viewpoint variants. The authors report a decline for the large-bread case; same-scene task changes are illustrated qualitatively. No formal significance test is reported.
Go to primary source ↓8.5 Primary sources
Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow ↗
PDF · 10,514 extracted words
Source fingerprint
873040c1844b4e34f493b1dff8ac4eb16948b23b2fd220adfba2f5dfcb05daad