Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
1. Paper overview
In one sentence: Dreamitate converts generated human tool-use videos into executable robot poses, gaining task-level generalization while relying on trackable rigid tools and slow open-loop inference. e02e03e05e07e09e10e11e12e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Human videos offer broad behavioral priors, but human hands and robot arms have incompatible embodiments. The paper asks whether a shared tool can provide the transferable interface: predict its interaction with objects visually, recover its pose geometrically, and let robot kinematics handle the arm. This also avoids collecting teleoperated robot-action demonstrations. e02e03e05 |
| Core mechanism | |
| A key reported result | Tabletop sweeping: 37/40 (92.5%). Success: move any particle within 50 mm of the target star.. 356 demonstrations; 40 trials with new particles and distractors, 2–4 beans and 5 distractors per scene. Diffusion Policy: 5/40 (12.5%); its encoder's final pooling layer was removed for better spatial resolution. The largest reported success-rate gap; success does not require collecting every particle or explicitly certify collision avoidance. e08e11 |
| Reading caution | Source description Heavy end-effector occlusion can break tracking; rigid tools restrict fine-grained manipulation. Video generation's computational cost prevents real-time closed-loop control in this implementation. e14 |
Core contributions
Figure 2. Training learns the visual behavior; inference extracts and executes the tool trajectory. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across the dotted divider as a change from training to deployment. On the left, two camera views of human demonstrations supervise the orange video-diffusion block. On the right, the initial scene image enters the adapted generator, whose output is a video of human tool use. The CAD-to-track arrow enters the tracking stage, and the final arrow leads to robot actions. This agrees with Equation (1): generation produces the frames and tracking converts their tool poses into actions. The drawing abbreviates stereo input as a single image stack; Section 3.2 and Appendix A specify how the two views are conditioned and paired. e03e04e06e07e14
What it supports. The robot does not need a learned mapping from human arm joints to robot joints. The transferable object is the tool trajectory, represented first in pixels and then as a 6D pose sequence. Video generation remains part of inference, rather than serving only as a pretraining objective for a separate action policy.
Where the evidence stops. The final arrow is a conceptual execution interface. It does not establish that every generated motion is physically realizable, or that the policy corrects errors continuously during execution; the source explicitly identifies closed-loop speed as a limitation.
2. Motivation
2.1 The problem and the proposed response
Human videos offer broad behavioral priors, but human hands and robot arms have incompatible embodiments. The paper asks whether a shared tool can provide the transferable interface: predict its interaction with objects visually, recover its pose geometrically, and let robot kinematics handle the arm. This also avoids collecting teleoperated robot-action demonstrations. e02e03e05
2.2 What this reading follows
Imagine teaching a robot to scoop by recording a person using a spoon. Dreamitate makes this possible through a carefully engineered interface: humans and robots manipulate corresponding tools whose geometry is known. A video model learns to imagine task execution from a new scene; stereo pose tracking then translates the imagined tool motion into robot actions. The interesting scientific question is how much this visual intermediate representation helps outside the demonstration scenes. Four real-world tasks show a substantial advantage over the tested Diffusion Policy baseline, but understanding that evidence requires keeping tracking assumptions, task-specific success rules, and rollout selection in view. e02e03e05e07e09e10e11e12e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is unassigned, so no existing quadrant can be confirmed. Architecturally this is a video predictor composed with a separate CAD-based pose tracker and robot execution stage. It derives actions from future video rather than jointly predicting actions and observations in one model. Calling tracking a learned inverse-dynamics policy would misdescribe the mechanism; no action-conditioned world-model planning loop is presented. e03e04e07e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the tool the transferable action representation
Begin with the scooping example. The human and robot have different joints, but both can place a spoon at a desired pose. Dreamitate therefore learns videos of human tool use and later recovers the tool's trajectory, rather than learning robot actions from teleoperation. Equation (1) exposes the two stages: the generator produces frames, and the tracking operator turns visible tool geometry into poses. Section 3.3 then uses those poses to control the mounted robot tool. My interpretation is that the method relocates much of the embodiment problem into the tool design, camera calibration, and kinematic interface. This explains both its appeal and its boundary: a known rigid object can be tracked, while hidden or deforming contact geometry cannot be assumed to yield an equally reliable action representation. e02e03e05e07e14
Figure 3. Known tool geometry supplies the interface between human demonstrations and robot execution. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the middle columns row by row: rotation uses paired contact tools, scooping a spoon, sweeping a brush, and Push-Shape a pushing surface. The human-facing handles and robot mounts need not look identical. The red arrows illustrate each spoon's use by its corresponding embodiment; they are not a learned transformation or an execution-feedback loop. Section 3.3 explains the required link: known CAD geometry supports recovery of the tool pose. Appendix D further specifies that the scooper handle is tracked, while handles are excluded for sweeping and pushing because the human hand hides them. e05e07e14
What it supports. The embodiment gap is reduced through physical interface design as well as machine learning. A useful video prediction must expose enough of the task tool for pose recovery, and the robot must carry the corresponding contact geometry. The tool therefore acts as both the manipulation device and the observable action representation.
Where the evidence stops. These are custom tools with known geometry, not evidence of arbitrary hand imitation or unseen-tool generalization. The authors identify rigid tools and heavy occlusion as restrictions, and the tracking choices differ across tasks.
5.2 Turn a video generator into a stereo pose source
The video model must produce views that support geometric recovery, not merely a plausible monocular movie. Dreamitate fine-tunes separate task models from Stable Video Diffusion, leaving its encoder and decoder frozen while adapting spatial and temporal attention. The per-frame image embedding identifies the camera view. Appendix A allocates thirteen outputs to the first camera and twelve to the second, discarding the repeated initial frame at inference. MegaPose then supplies each view's tool estimate; stereo center rays determine translation and the two rotations are averaged. This is a concrete path from pixels to robot poses, but it does not provide a demonstrated guarantee of cross-view physical consistency. Occlusion-specific tracking rules, such as using only the scooper handle, reveal how much the geometric stage depends on visible tool structure. e04e06e07e14
Table 3. The appendix fixes several reproducible settings and exposes a caveat in the data-scaling comparison. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three rows together to audit Figure 8, then compare the remaining tasks. Res means image/video resolution, Lr learning rate, and Train Steps the checkpoint used for evaluation. Clip Duration is the demonstration length in seconds; Fps is both the video subsampling rate and model fps parameter; MotScr is the model motion-score parameter. Every row uses 768×448 resolution, learning rate 10⁻⁵, and motion score 200. The rotation fractions share clip duration and fps, but the smallest fraction changes batch size and steps. Scooping and sweeping use three-second clips at five fps, while rotation and pushing use two seconds at six fps. e06e13
What it supports. This table provides concrete training settings for reproducing the task models and explains why the rotation scaling curve is not a perfectly controlled data-only intervention. Appendix A separately specifies the 13-plus-12 stereo-frame allocation and test-time removal of the first frame; those implementation details are necessary alongside the table.
Where the evidence stops. The table is not a complete training recipe: optimizer details, seeds, accelerator requirements, and measured runtimes remain unspecified. The supplied text also does not explain how every listed clip duration and sampling rate is mapped into the fixed stereo output slots.
5.3 Read generalization through the actual scoring rules
The experiments change objects and scene conditions while retaining the camera arrangement and task-specific tools. Both methods learn from the same demonstrations, although Diffusion Policy receives tracked pose targets and uses a different pretrained visual architecture. Table 2 establishes that the complete Dreamitate pipeline performs better under these conditions. Interpretation then depends on the task: transferring any particles is enough for scooping, while sweeping accepts any particle near the star. Push-Shape goes further in horizon but reports selected performance from four rollouts and uses the highest-IoU image to measure rotation error. My reading is that these are valuable demonstrations of transfer to new physical scenes, with limited evidence about unattended reliability. Figure 8 adds reduced-data evidence, but cannot independently explain whether video pretraining, representation, or other implementation differences cause the advantage. e08e09e10e11e12e13
5.4 Training and inference
During training
Train a separate generator for each task, initializing from pretrained Stable Video Diffusion. Freeze encoder and decoder; fine-tune only spatial and temporal attention. Equation (2) describes a stereo frame-prediction objective, while the source does not specify a complete diffusion denoising-loss implementation. e04e06
Table 3 uses 768×448 resolution, learning rate 10⁻⁵, and motion score 200. Full rotation, scooping, and sweeping use batch size 4 and 16,384 steps; Push-Shape uses 17,408. One-third rotation instead uses batch size 3 and 15,360 steps. e06
During inference
Generation uses 30 denoising steps and constant classifier-free guidance 1.0. Predicted videos serve directly as the action representation; there is no reported reward-based search over action-conditioned futures. The comparison is open-loop, and the authors identify generation cost as preventing real-time closed-loop control. e03e06e08e14
Tracking initialization uses background subtraction and skin-color hand removal. Rare failures receive human bounding-box corrections for both methods, so the evaluated pipeline includes occasional manual assistance. e08
5.5 Implementation flow
- Collect a transferable demonstration
Record humans using custom tools with two cameras. Separate task datasets contain 371 rotation, 368 scooping, 356 sweeping, and 727 Push-Shape demonstrations. Push-Shape targets come from the demonstrated object's final position, blended throughout the video. e03e05e08e12
- Generate paired future views
Adapt Stable Video Diffusion's 25 output frames: the first 13 represent one camera and the last 12 the other. View-specific image embeddings provide conditioning. Discard the first output, which matches the input, to leave paired views at test time. e04e06
- Recover a geometric trajectory
MegaPose tracks the known tool. Stereo center rays determine translation via the midpoint between the rays; rotations from both views are averaged. Scooping tracks the handle, whereas sweeping and Push-Shape exclude handles hidden by the human hand. e07
- Execute the tool poses
The robot carries the corresponding tool and executes the recovered pose sequence. Arm embodiment is handled through inverse kinematics. Sweeping and Push-Shape also impose end-effector height limits to avoid tabletop collision. e02e05e07e15
6. Experiments & results
Dreamitate learns task-specific video generation from human tool demonstrations, then extracts tool poses from synthesized stereo videos for robot execution. It outperforms the tested Diffusion Policy baseline on four physical manipulation tasks, while depending on calibrated cameras, known rigid tools, and slow open-loop generation. The central evidence concerns executed tool trajectories under specified object and scene shifts, rather than unrestricted manipulation or a general action-conditioned simulator.
The supplied paper's quantitative diagnostic is the rotation demonstration-count curve. It contains no controlled removal of video pretraining, stereo generation, or the tracking stage, and no uncertainty bars or point labels for the reduced-data results. Figure 8 therefore fills the diagnostic role without being presented as a component ablation. e04e08e13
6.1 Read the original evidence
Table 2. The advantage spans four tasks, whose metrics and evaluation protocols differ. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column vertically before comparing tasks. The first three columns count successful physical trials out of forty. Rotation requires maintaining contact through at least 25 degrees of counterclockwise rotation; scooping needs any particles in the empty bowl; sweeping needs any particle within 50 mm of the star. Push-Shape instead reports mask overlap, where higher is better, and angular error, where lower is better. Section 4.5 adds the essential qualification absent from the table cells: each of 32 trials scores the best of four rollouts. Appendix B also selects the highest-IoU image within a multistep trial for rotation-error measurement. e04e08e09e10e11e12
What it supports. Dreamitate leads the tested baseline in every displayed metric. Sweeping has the largest success-rate gap: 37/40 versus 5/40, equivalent to 92.5% versus 12.5%. Push-Shape's 8.0-degree error versus 48.2 degrees suggests better orientation alignment under the paper's selected-rollout evaluation, alongside its higher mask overlap.
Where the evidence stops. The table reports neither uncertainty intervals nor seed variation. These are task-specific system comparisons, and Push-Shape's selected results cannot be read as single-rollout reliability. The same demonstrations do not equalize the models' pretraining, architecture, resolution, or computational cost.
Figure 6. Generated-video insets and large robot frames distinguish imagination from physical execution. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row begins with the scene and then shows two execution snapshots for each method. White circles identify particles and green circles locate the target; the paper says these circles are illustrative overlays, not model input. The small insets in Dreamitate's columns show generated human-tool video, distinguished by colored bars. The larger frames show the robot execution. Follow the arrows to compare the illustrated displacement toward the target with the baseline's failed motion. The examples clarify what the success counts mean, while the full evaluation and its 50 mm acceptance radius are specified in Section 4.4 and Table 2. e03e11
What it supports. The qualitative evidence links a predicted human-tool sequence to a visibly different robot embodiment executing the task. It also makes the reported failure mode concrete: in these selected baseline examples, the motion fails to bring the marked particle to the target, while Dreamitate's execution is marked successful.
Where the evidence stops. Selected stills do not establish the frequency of obstacle collisions, smoothness, or tracking errors. The binary success criterion only needs one particle near the target, and these images do not replace the separate quantitative evaluation or continuous execution videos.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Object rotation 371 demonstrations, 31 training objects; 10 unseen test objects and 40 real-robot trials. | 37/40 (92.5%). Success: maintain contact while rotating at least 25° counterclockwise. | Diffusion Policy: 22/40 (55%). Supports better contact and rotation behavior in this test; qualitative failures include unstable grasps. No uncertainty intervals are reported. e08e09 |
| Granular material scooping 368 demonstrations with 17 bowls and 8 particle types; 40 trials with 8 unseen bowls, 4 new particle types, and 5 distractors per trial. | 34/40 (85%). Success: transfer any particles into the empty bowl. | Diffusion Policy: 22/40 (55%). Demonstrates physical transfer under clutter, but the binary metric does not measure transferred quantity or spillage. e08e10 |
| Tabletop sweeping 356 demonstrations; 40 trials with new particles and distractors, 2–4 beans and 5 distractors per scene. | 37/40 (92.5%). Success: move any particle within 50 mm of the target star. | Diffusion Policy: 5/40 (12.5%); its encoder's final pooling layer was removed for better spatial resolution. The largest reported success-rate gap; success does not require collecting every particle or explicitly certify collision avoidance. e08e11 |
| Push-Shape alignment 727 demonstrations on 26 letter shapes; 32 trials on 8 unseen shapes, scoring the best of 4 rollouts per trial. | mIoU 0.731; rotation error 8.0°. Mean intersection-over-union with target mask and average rotation error. | Diffusion Policy: mIoU 0.550; rotation error 48.2°. Camera 1 supplies mask evaluation; rotation error uses the highest-IoU image within a multistep trial. These selected outcomes do not establish single-rollout or final-step reliability. e08e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 8. A rotation data-scaling diagnostic supports robustness to fewer demonstrations, with confounded training settings. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis varies the fraction of rotation demonstrations, and the vertical axis is success rate. Blue squares identify Dreamitate; orange circles identify Diffusion Policy. Section 4.6 says the models were retrained at the reduced dataset sizes and evaluated over forty trials. The curves show a substantial advantage at every fraction, although Dreamitate's two-thirds point is slightly below its one-third point. Do not interpret the connecting segments as a fitted scaling law. Exact reduced-data values are not printed beside the markers, so the defensible reading is the visible ordering and broad trend rather than reconstructed trial counts. e08e13
What it supports. The plot supports a narrower claim than universal data efficiency: in this rotation experiment, the video-based system retains a large advantage as demonstrations are reduced. The authors attribute the result to internet-video pretraining, but this figure compares complete systems and does not turn that explanation into an isolated causal test.
Where the evidence stops. No uncertainty bars or repeated-seed distribution are shown. Table 3 changes the one-third setting to batch size 3 and 15,360 training steps, versus 4 and 16,384 elsewhere. This is a data-scaling diagnostic, not a controlled component ablation.
7. Analysis & limitations
7.1 What the evidence leaves open
Heavy end-effector occlusion can break tracking; rigid tools restrict fine-grained manipulation. Video generation's computational cost prevents real-time closed-loop control in this implementation. e14
Figure 8 shows a large reduced-data advantage on rotation, but supplies no point labels or uncertainty bars. Its smallest-data setting changes batch size and training steps. Neither this curve nor the main comparison isolates pretraining, stereo generation, or tracking quality. e08e13
Generalization is bounded by task-specific models, fixed camera arrangement, and known tools. Dreamitate and the baseline share demonstrations but differ in pretraining, architecture, and image resolution; the results establish a system comparison, not equal-compute superiority. e04e08e15
7.2 Questions for discussion
- How much of the advantage survives when pretraining and input resolution are controlled?
- Does the Push-Shape advantage persist under single-rollout, final-step scoring?
- Which failures originate in synthesized contact dynamics, stereo inconsistency, tracking, or execution?
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuild calibrated D435i stereo capture: approximately 660 mm camera spacing, 45° angle, and 760 mm height above the table. Use xArm 7 for rotation/scooping and UR5 for sweeping/Push-Shape, known tool CADs, and the black-cloth surface whose friction affects pushing. e07e15
Match the baseline: two pretrained ResNet-18 encoders, 384×224 stereo inputs, tracked demonstration poses, 200 epochs, and 12-step action predictions. Preserve manual initialization corrections and the task-specific success definitions when comparing runs. e08e09e10e11
The supplied implementation description leaves optimizer details, random seeds, training hardware, runtime measurements, calibration procedure, and correction frequency unspecified. Proposed checks should separate tracking error from generated-motion error and report single-rollout alongside best-of-four Push-Shape performance. e04e06e07e08e12e14e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate generated-motion errors from stereo tracking errors
Reader-proposed check, not performed: construct held-out scooping scenes with visible and partially occluded tool handles. For each scene, obtain an independently measured reference trajectory, record it from the calibrated cameras, and process both that real stereo video and Dreamitate's generated stereo video with the same MegaPose/CAD pipeline. Compare pose error on the real reference sequence, cross-view consistency on both sequences, correction frequency, and robot transfer success under fixed execution settings. Real-video pose failure despite a successful reference trajectory would implicate tracking. Accurate tracking of real videos but poor generated trajectories would instead implicate generation or its cross-view consistency. Report results by visibility rather than averaging occlusion failures away. e05e07e08e10e14e15
Check 2: Measure how rollout selection changes Push-Shape reliability
Reader-proposed check, not performed: use the same held-out shapes, initial placements, target masks, cameras, cloth surface, and execution limits for both methods. Run four independent rollouts per scene, recording every step. Report the first rollout's final-step IoU and rotation error, the distribution across all rollouts, and the paper's best-of-four/highest-IoU selection result separately. Keep per-rollout step budgets identical and retain all failures. If Dreamitate's advantage remains under first-rollout, final-step scoring, the claim extends beyond selected attainable performance. If it appears mainly after rollout or frame selection, the practical interpretation should focus on retry-assisted alignment, with the additional inference and execution cost measured explicitly. e08e12e14e15
8.3 Reading coverage
Visual audit: Visually inspected the title, byline, affiliation and v1 date on page 1; all method and experiment pages 2–8; and all appendix pages 12–17, including Figures 1–18 and Tables 1–3. Checked the Figure 2 generation-to-tracking-to-execution arrows against Equation (1), and the unsquared L2 norms in Equation (2). Checked Figure 6's generated-video insets and illustrative overlays against its caption. All six final original crops were viewed and retain the relevant diagram labels, table headers or plot axes and legend; surrounding prose and long captions are excluded. All pages supporting retained numerical, method, training, evaluation and proposed-reproduction claims are included here. Reference pages 9–11 were read in text but not visually inspected. Separate supplemental videos and external code/data remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 Approach
- 3.1 Overview
- 3.2 Video Generation
- 3.3 Track then Act
- 4 Experiments
- 4.1 Experimental Setup
- 4.2 Object Rotation Task
- 4.3 Granular Material Scooping Task
- 4.4 Table Top Sweeping Task
- 4.5 Push-Shape Task (Long Horizon)
- 4.6 Performance Scaling Curve
- 5 Limitations
- 6 Conclusion
- Acknowledgments
- References
- A Video Model Implementation
- B Experimental Setup
- C Data Collection
- D Object Tracking
- E Diffusion Policy Baseline
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: the inspected title page states arXiv:2406.16862v1, 24 June 2024. Its title and all eight authors match the catalog; Junbang Liang and Ruoshi Liu have equal-contribution marks. No revision or edition discrepancy is observed. Other versions were not supplied or compared.
- All four supplied text chunks were read, covering the complete 17-page PDF. Figures 1–18 and Tables 1–3 were visually inspected on pages 1–8 and 12–17. References on pages 9–11 were read as text.
- Acquisition omission preserved: text extraction does not reconstruct figure images; this reading separately inspected the retained PDF figures and equation/table layouts.
- Separate supplemental material availability has not been fully verified. External supplementary robot videos, code, and data were not inspected; experiments were not reproduced. The paper's stated intention to release code and data is not verified availability.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/byline, affiliations, equal-contribution footnote and arXiv margin
Title and eight authors match the catalog; the artifact is arXiv:2406.16862v1, 24 June 2024. Affiliations are Columbia University, Toyota Research Institute, and Stanford University.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1
The proposed transfer uses common tools, human demonstration videos, video prediction and 3D tracking; arm embodiment is addressed through inverse kinematics. Code/data release is stated as a future intention.
Go to primary source ↓e03PDF p. 3, Figure 2 and caption, Section 2 final paragraph, Section 3.1, Eq. (1)
Training and inference branches compose an initial-image-conditioned video model with tool tracking. Equation (1) maps synthesized frames to SE(3) actions; the text distinguishes this policy use from world-model simulation for planning.
Go to primary source ↓e04PDF p. 4, Section 3.2 and Eq. (2)
Stereo frame discrepancies are written as a sum of L2 norms. Each task has a separate SVD-initialized model; only spatial/temporal attention is fine-tuned, with frozen encoder/decoder and view-specific conditioning.
Go to primary source ↓e05PDF p. 4, Figure 3 and caption; Section 3.3
Custom human and robot tools serve rotation, scooping, sweeping and Push-Shape. Known CAD geometry enables tracking; tool poses are subsequently executed by the robot.
Go to primary source ↓e06PDF p. 12, Appendix A and Table 3, all rows and caption
The 25-frame model allocates 13 frames to view 1 and 12 to view 2, discarding the first at test time. Inference uses 30 denoising steps and guidance 1.0. Table 3 gives resolution, learning rate, batch sizes, checkpoint steps, clip lengths, fps and motion score; optimizer, seed, training hardware and runtime are not listed.
Go to primary source ↓e07PDF p. 4, Section 3.3; PDF pp. 12–13, Appendix D
MegaPose operates on 768×448 generated frames with CAD models. Stereo translation uses projected center rays and rotations are averaged. Tracked tool parts differ with occlusion. The same stereo tracker creates baseline demonstration targets.
Go to primary source ↓e08PDF p. 5, Section 4.1 and Table 1; PDF p. 13, Appendix E
Training/test objects do not overlap, environments vary with the same camera setup, and rare tracking-initialization failures receive manual box corrections for both methods. Demonstration counts are 371/368/356/727. The baseline uses tracked targets, two ResNet-18 encoders, 384×224 inputs, 200 epochs and 12-step predictions in an open-loop comparison.
Go to primary source ↓e09PDF p. 5, Section 4.2 and Figure 4; PDF p. 6, Table 2 Rotation column and Section 4.2 continuation
Rotation tests 10 unseen objects over 40 trials after training on 31 objects. Success requires contact throughout at least 25° counterclockwise rotation. Dreamitate records 37/40 versus 22/40; unstable grasps and slipping are discussed.
Go to primary source ↓e10PDF p. 6, Figure 5, Section 4.3 and Table 2 Scooping column
Scooping trains with 17 bowls, 8 particle types and one distractor per scene, then tests 8 unseen bowls, 4 new particle types and five distractors sampled from 15 objects. Any particle transfer to the empty bowl counts; results are 34/40 versus 22/40.
Go to primary source ↓e11PDF p. 6, Table 2 Sweeping column and Section 4.4; PDF p. 7, Figure 6 and Section 4.4 continuation
Sweeping tests new particles and distractors with 2–4 beans and five distractions. Success means any particle within 50 mm of the star. Dreamitate achieves 37/40 versus 5/40; the baseline drops its final pooling layer. Figure 6 separates execution frames from generated-video insets and identifies illustrative overlays.
Go to primary source ↓e12PDF p. 6, Table 2 Push-Shape columns; PDF p. 7, Section 4.5; PDF p. 8, Figure 7; PDF p. 12, Appendices B–C
Push-Shape uses 26 letter training shapes and eight unseen test shapes over 32 trials, scoring best of four rollouts. Results are 0.731 mIoU/8.0° versus 0.550/48.2°. Camera 1 supplies IoU; the highest-IoU image determines rotation error. Target masks are blended into demonstration videos.
Go to primary source ↓e13PDF p. 8, Section 4.6 and Figure 8; PDF p. 12, Table 3 rotation rows
Rotation is retrained at one-third, two-thirds and full data and tested over 40 trials. The plotted reduced-data advantage has no exact point labels or uncertainty bars. The one-third model uses batch size 3 and 15,360 steps versus 4 and 16,384 for the other rotation settings.
Go to primary source ↓e14PDF p. 8, Section 5
The authors identify visually trackable actions, heavy occlusion, rigid-tool restrictions and video-model computation as limits; real-time closed-loop control is described as infeasible.
Go to primary source ↓e15PDF p. 12, Appendix B; PDF p. 17, Figures 17–18 and captions
Two D435i cameras are about 660 mm apart, at 45°, about 760 mm above the table. Videos are recorded at 1280×720 before cropping/resizing. The black cloth changes friction. xArm 7 and UR5 handle specified task groups; sweeping and pushing have height limits. The photos show corresponding human and robot camera arrangements.
Go to primary source ↓8.5 Primary sources
Dreamitate: Real-World Visuomotor Policy Learning via Video Generation ↗
PDF · 6,916 extracted words
Source fingerprint
5cd15961814505b548a06b2374e43a07804c2b7a05619ff98f1330703303a990