Video Generators are Robot Policies
1. Paper overview
In one sentence: A video diffusion model can supply transferable manipulation features to a separate action denoiser, but the evidence depends on task-video adaptation and comes with costly inference. identityarchitectureobjectivestransferreal-resultsimplementation
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations are expensive, and behavior-cloning policies struggle when objects, appearances or tasks change. The paper asks whether predicting robot-execution videos can supply transferable representations, leaving a smaller action decoder to learn the interface to the robot. problemarchitecture |
| Core mechanism | Source description A modular pair of diffusion networks generates videos and executable actions together, while allowing video learning without action labels. architectureobjectives |
| A key reported result | RoboCasa validation: 24-task manipulation: 0.63; the separate 300-MimicGen-demonstration configuration reports 0.66. Average task success rate. 50 human demonstrations per task; standard validation, 50 rollouts per task across five scenes. DP-ResNet 0.41, DP-CLIP 0.43, UVA 0.50, DP-VLA 0.57. Using the printed average row, 0.63 versus 0.57 gives a 6-percentage-point difference. This is not reproduced by equally averaging the 24 displayed task values: Video Policy gives 0.6325, DP-VLA about 0.5858 and DP-CLIP about 0.4067, whereas the latter two printed averages are 0.57 and 0.43. The supplied paper explains no alternative weighting; these arithmetic checks do not establish an official correction. External baselines have different pretraining and demonstration budgets, and improvements are not uniform across tasks. robocasabaseline-budgets |
| Reading caution | Reader analysis The authors note modest simulation scale, one physical embodiment, dependence on SVD and costly inference, and attribute some physical failures to unrealistic predictions. Reader assessment: the alignment figures provide qualitative examples, not a quantitative video-to-action fidelity metric. limitationsreal-resultsqualitative |
Core contributions
- Source description
A modular pair of diffusion networks generates videos and executable actions together, while allowing video learning without action labels. architectureobjectives
- Author claim
The authors argue that video generation carries the policy and the decoder mainly provides an interface. Task-transfer and training ablations support this interpretation within the tested settings, rather than establishing general sufficiency of video prediction. problemtraining-ablationtransfer
Figure 2. Two coupled denoisers connect visual prediction to executable action. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the demonstration images at left, which indicate fine-tuning data rather than an extra deployment input. Follow video noise and task text into the upper video U-Net, then trace the downward feature connections through the CNN Video Adaptor into the lower action U-Net. Action noise enters that lower branch separately. Section 3.2 explains that five decoder-layer features are collected at every denoising step and compressed into the conditioning vector h_i. The two right-hand outputs are predicted video and robot actions. The arrows therefore represent feature conditioning during denoising, not tracking performed after a complete video has been rendered. architectureobjectivestraining-ablation
What it supports. The architecture supports the catalog’s Dual-system × Joint prediction classification: distinct video and action networks generate their outputs together. The feature interface also permits training the video network on trajectories without action labels, then learning an action decoder from the subset with robot supervision.
Where the evidence stops. The diagram does not show a stop-gradient marker. Section 3.3 explicitly blocks action-loss gradients into the video U-Net, and Section 4.4 describes freezing that U-Net in Two-Stage training. The shared diffusion enclosure should not be read as proof of unrestricted end-to-end gradients.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations are expensive, and behavior-cloning policies struggle when objects, appearances or tasks change. The paper asks whether predicting robot-execution videos can supply transferable representations, leaving a smaller action decoder to learn the interface to the robot. problemarchitecture
2.2 What this reading follows
Video Policy asks how much robot control can be learned by predicting what successful behavior looks like. Its SVD backbone imagines future camera views, while a smaller diffusion network converts intermediate video features into executable actions. The important separation is between learning from videos and learning from action labels: a task may contribute to the first stage without contributing to the second. The following visuals trace that interface, compare simulation results, and examine where the transfer claim holds. They also show why physical success, plausible prediction and real-time deployment must be assessed separately. This edition reads the supplied arXiv v1, including its implementation appendix. identityarchitectureobjectivestransferreal-resultsimplementation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × Joint prediction is supported by two distinct U-Nets connected through a feature adapter and coupled at every denoising step. Joint refers to future-video/action generation, not necessarily end-to-end optimization. This is a video-conditioned action diffusion policy, not an inverse-dynamics model fitted to completed frame pairs or action-conditioned rollout search. architectureobjectivestraining-ablation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow features through diffusion, not finished frames through a tracker
Begin with the current camera observations and the language instruction. The frozen SVD VAE provides image latents, and the text reaches the video U-Net through CLIP-based cross-attention. Video Policy arranges multiple camera streams along the temporal dimension, modifying the image conditioning for each view. At a denoising step, the video U-Net produces hidden spatiotemporal representations as it predicts future imagery. Five decoder-layer features are then collected by a CNN adapter, yielding the vector h_i that conditions the action U-Net. This exchange repeats throughout denoising, allowing video and action predictions to develop together. The robot ultimately receives the action sequence, not an instruction to imitate pixels. In simulation the actions encode gripper pose and opening state; physical deployment additionally models grasping force and uses the controller described in Appendix A.4. architectureobjectivesviewsimplementationrobocasacontroller
5.2 Separate representation learning from the action-supervision budget
The two-stage recipe turns the video model into a reusable representation before training the action decoder. First it learns robot-execution video prediction; then its U-Net is frozen while action learning uses that representation. The paper’s separate losses and stop-gradient statement explain the intended protection of the video network, but the exact gradient routing of the comparator called Joint is not fully resolved by its description. A reproduction should state that route explicitly. The ablation favors Two-Stage, although its longer schedule leaves a compute confound. Figure 4 adds a different test: allow videos from every task but actions from only half. Reader interpretation: this combination supports task knowledge entering through video training, while leaving open how much the decoder still learns task-specific behavior and how well the same mechanism transfers to tasks never shown in video. objectivestraining-ablationimplementationtransfer
5.3 Translate predictive quality into an executed-control claim carefully
A useful predictor must support actions that succeed in the environment. The horizon diagnostic keeps the action horizon fixed while changing how far video prediction looks ahead, providing evidence that future visual structure helps control. Its special MimicGen evaluation must remain separate from standard RoboCasa results. Physical deployment introduces another layer: the model predicts a sequence of relative poses and jaw positions plus absolute force, and impedance control executes twenty-four of thirty-two steps with a force-based closing correction. This means the reported policy includes a concrete execution interface, not just a generated movie. Reader interpretation: the approximate nine-second generation time reported for an A100 makes scheduling and feedback a central reproduction question. Qualitative video/rollout alignment cannot answer that question, and the ten-trial physical table demonstrates both useful transfer and pronounced failure cases. horizonhorizon-protocolcontrollerimplementationqualitativereal-results
5.4 Training and inference
During training
The two-stage variant first fine-tunes SVD on demonstration videos, freezes the video U-Net, then trains the action head. Equations (4)–(5) specify separate squared denoising losses; Section 3.3 explicitly blocks action-loss gradients into the video network. Section 4.4 separately evaluates a Joint variant, so simultaneous outputs must not be equated with a single training schedule. objectivestraining-ablation
Main-text learning rates are 1e-5 for video and 5e-5 for actions. Appendix Table 5 lists batch 32, mixed precision and 368,866 steps for Joint versus 368,866 × 2 for Two-Stage. Fine-tuning used eight A100 GPUs for about two weeks; real-world image resolution progressed from 256 × 192 to 448 × 320. viewsimplementation
During inference
Video and action denoising run together for 30 steps with classifier-free guidance 2.0. Simulation predicts 32 action steps and executes 16. Physical deployment predicts 32 steps of relative gripper pose, relative jaw position and absolute force, then follows 24 steps using impedance control; a closing correction applies when predicted force exceeds measured force by more than the source-stated 300 grams. implementationbaseline-implementationcontroller
A 25-frame 256 × 256 video takes about nine seconds on an A100. The paper does not resolve deployment scheduling around that latency; the reported rollout horizons alone do not establish real-time closed-loop operation. implementationcontroller
5.5 Implementation flow
- Condition the video generator
An SVD video U-Net uses CLIP task-text cross-attention and concatenates the frozen VAE encoding of the initial image with noisy video latents. Camera-specific image embeddings support multiple views. architectureimplementation
- Read hidden video features
At every denoising step, features from decoder layers 9, 14, 17, 20 and 23 pass through a CNN adapter into a global vector. A separate 1D action U-Net, adapted from Diffusion Policy, consumes that vector and noisy actions. It does not wait to decode a finished RGB movie. architectureobjectives
- Pack views and horizons
RoboCasa uses eight frames each from gripper, left and right cameras, plus one padded frame: 25 total. LIBERO10 uses two views with 12 frames each plus padding. The default video spans 32 action steps through stride-four subsampling. viewsimplementation
6. Experiments & results
Video Policy turns Stable Video Diffusion into a manipulation policy through a separate action diffusion U-Net conditioned on intermediate video features. Its main 50-demonstration RoboCasa configuration trains video prediction first and then freezes that representation for action learning. Simulation results support this approach, while sparse physical trials and slow video inference limit broader conclusions.
6.1 Read the original evidence
Table 1. RoboCasa gains are substantial on average and uneven across tasks. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the paper’s printed bottom row first, then move upward to inspect individual tasks; the DP-VLA and DP-CLIP averages do not reproduce the simple means of their displayed rows. The two rightmost columns distinguish Video Policy trained with 50 human demonstrations per task from the configuration using 300 MimicGen demonstrations. The caption and evaluation section specify 50 rollouts per task across five scenes. Compare DP-ResNet, DP-CLIP and UVA with the main Video Policy column, then inspect the broader published baselines separately. The pick-and-place block is particularly informative, but the drawer and knob rows prevent a blanket claim that the proposed method wins everywhere. Values are success fractions; the six-percentage-point comparison uses the printed averages 0.63 and 0.57, not a recalculation from all task rows. robocasabaseline-budgetsbaseline-implementation
What it supports. The 50-demonstration Video Policy reports 0.63 average success, compared with 0.41 for DP-ResNet, 0.43 for DP-CLIP, 0.50 for UVA and 0.57 for DP-VLA. The 300-demonstration configuration reaches 0.66. More data improves the overall score but does not improve every individual task.
Where the evidence stops. Reader arithmetic check: the 24 displayed rows average 0.6325 for Video Policy (50 demonstrations), about 0.5858 for DP-VLA and 0.4067 for DP-CLIP, whereas the latter two printed averages are 0.57 and 0.43. The source supplies no alternative weighting, so this is an unresolved aggregate discrepancy, not an official correction. The table also mixes retrained and published baselines with different demonstration budgets and sources; no seed variability or confidence intervals are supplied.
Table 4. Physical generalization depends strongly on the task and the visual change. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each row before comparing tasks. Each cell represents ten rollouts, following training with two hundred demonstrations per task. The first column changes positions of familiar objects, the second uses the task-specific novel object set, and the third changes background appearance. Those sets are documented in appendix photographs: for Open Drawer, object novelty is masking-tape coverage of the same cabinet, rather than a new mechanism. The paired video/rollout examples elsewhere in the paper illustrate alignment, but this table measures actual physical success. Small differences between adjacent cells correspond to very few trials and should be interpreted accordingly. real-setupreal-resultsobject-setsqualitative
What it supports. Pick and Place succeeds at 1.0, 0.9 and 0.8 across the three settings, but Stack Cups stays at 0.3, 0.2 and 0.2. M&Ms to Cup falls from 0.8 with varied locations to 0.2 under new backgrounds. These failures materially qualify the caption’s broad robustness claim.
Where the evidence stops. No real-world baseline or uncertainty estimates accompany the table. The source’s Stack Cups training-set count is unresolved: Section 4.5 says six cups, while Figure 12 says five. Better Upright Object performance under shifts is attributed to visibility, not established as a causal effect.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboCasa validation: 24-task manipulation 50 human demonstrations per task; standard validation, 50 rollouts per task across five scenes. | 0.63; the separate 300-MimicGen-demonstration configuration reports 0.66. Average task success rate | DP-ResNet 0.41, DP-CLIP 0.43, UVA 0.50, DP-VLA 0.57. Using the printed average row, 0.63 versus 0.57 gives a 6-percentage-point difference. This is not reproduced by equally averaging the 24 displayed task values: Video Policy gives 0.6325, DP-VLA about 0.5858 and DP-CLIP about 0.4067, whereas the latter two printed averages are 0.57 and 0.43. The supplied paper explains no alternative weighting; these arithmetic checks do not establish an official correction. External baselines have different pretraining and demonstration budgets, and improvements are not uniform across tasks. robocasabaseline-budgets |
| LIBERO10 manipulation 50 demonstrations per task; 50 trials per task following the UVA protocol. | 0.94 Average task success rate | UVA 0.90; pi_0 0.85; comparison values adopted from prior work. A strong reported aggregate, with no seed variance or confidence interval supplied. liberorobocasa |
| RoboCasa training-objective ablation Standard validation, 50 trials per task. | Two-Stage 0.63; Joint 0.57; No Video Tuning 0.09. Average task success rate | Two-Stage exceeds Joint by 6 percentage points. Task-video adaptation matters, but the longer two-stage schedule prevents attributing the entire gap to gradient isolation. training-ablationimplementation |
| Video prediction horizon diagnostic Separate sampled MimicGen environments; 50 trials per task. Actions always predict 1.6 seconds and execute 0.8 seconds. | 32-step video horizon 0.67; 16-step 0.55; zero-step reconstruction 0.30. Average task success rate | Table 8 compares Joint models; Figure 3 separates higher-shift pick-and-place tasks from lower-shift tasks. Future prediction helps under this diagnostic protocol; these averages cannot replace standard RoboCasa validation results. horizonhorizon-protocol |
| Action-free task transfer Video model sees all 24 RoboCasa tasks; action decoder and DP baseline see actions for the same 12 tasks. Evaluation covers all 24. | Video Policy 0.41 Average task success rate across all tasks | DP-ResNet trained on half the tasks: 0.21. This is an all-task average, not the held-out-task average. Transfer concerns missing action labels for tasks already observed in video. transfer |
| Real-world generalization across five tasks 200 demonstrations per task; 10 rollouts per task and condition. | Open Drawer 0.8/1.0/0.9; Pick and Place 1.0/0.9/0.8; M&Ms to Cup 0.8/0.9/0.2; Upright Object 0.3/0.7/0.8; Stack Cups 0.3/0.2/0.2. Success rate: varied location / unseen objects / unseen background | Within-method conditions only; Table 4 supplies no physical baseline. Physical execution is demonstrated, but stacking remains weak and background changes sharply hurt small-object localization. real-setupreal-results |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Robot-video adaptation matters; separated training performs best in this comparison. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The three rows vary how the video representation is obtained and used for action learning. Joint is the paper’s end-to-end comparator. Two-Stage first fine-tunes SVD on RoboCasa videos, then freezes the video diffusion U-Net while training the action denoiser. No Video Tuning leaves the video U-Net at its SVD initialization. Read each value as an average under standard RoboCasa validation; Appendix Table 6 supplies the corresponding per-task results and trial count. To interpret the training comparison, also consult Appendix Table 5: the listed Two-Stage schedule has 368,866 × 2 steps, whereas Joint lists 368,866. training-ablationobjectivesimplementation
What it supports. Two-Stage reaches 0.63 versus Joint’s 0.57, while using an untuned SVD representation yields 0.09. The strongest supported conclusion is that adapting video prediction to robot behavior is important in this implementation. The smaller Two-Stage–Joint gap motivates closer study of optimization and representation sharing.
Where the evidence stops. The comparison changes both the optimization schedule and representation updates. It does not isolate gradient blocking at matched compute. The caption’s claim that video learning is necessary and sufficient is broader than what three tested configurations can establish.
Figure 3. Predicting further ahead helps most for the more strongly shifted task group. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis changes the video prediction horizon, while the vertical axis measures executed-task success. Blue denotes the higher-shift group and orange the lower-shift group. Section 4.4 keeps action prediction at 1.6 seconds and executed action chunks at 0.8 seconds, so the manipulated variable is visual foresight. Appendix A.1 is essential context: these experiments use sampled MimicGen environments with training-like layouts and styles. Pick-and-place tasks introduce greater changes in object positions and categories than the remaining tasks. Appendix Table 8 lists the individual 0-, 16- and 32-step results; Figure 6 illustrates the environment comparison. horizonhorizon-protocol
What it supports. Both grouped curves improve with a longer video horizon, with the blue curve rising more sharply. For the complete task set, Table 8 reports average success of 0.30, 0.55 and 0.67 at zero, 16 and 32 video steps, respectively. These are diagnostic results under the separate MimicGen protocol.
Where the evidence stops. The labels do not mean that the orange tasks are universally in-distribution or that this is the standard validation split. Appendix A.1 describes relative shift within this diagnostic and misreferences its horizon figure as Figure 2; the plotted horizon figure is Figure 3.
Figure 4. Video exposure enables transfer to tasks without action labels. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The dotted vertical divider separates the twelve tasks used for action-head training from twelve tasks withheld from that supervision. Orange bars are DP-ResNet and blue bars are Video Policy. Crucially, the video generator was fine-tuned using videos from all twenty-four tasks. On the right, the thin colored extensions correspond to models trained with full action supervision; the legend calls them upper bounds. They are not error bars or confidence intervals. Read the solid bars to assess actual transfer, and compare them with those reference endpoints to see how much performance remains dependent on direct action labels. transfertraining-ablation
What it supports. Video Policy achieves nonzero success on many tasks omitted from action training, although transfer is uneven and some gaps to full supervision remain large. Appendix Table 6 reports an all-task average of 0.41 for this model versus 0.21 for the half-task DP baseline. That average includes both sides of the divider.
Where the evidence stops. These tasks are unseen to the action decoder’s supervision, not unseen to the video model. The experiment supports using action-free task videos; it does not establish generalization to tasks absent from all training data or a held-out-task average of 0.41.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors note modest simulation scale, one physical embodiment, dependence on SVD and costly inference, and attribute some physical failures to unrealistic predictions. Reader assessment: the alignment figures provide qualitative examples, not a quantitative video-to-action fidelity metric. limitationsreal-resultsqualitative
Source discrepancies remain unresolved: Section 4.5 states six Stack Cups training cups, while Figure 12 states five. Section 4.3 attributes UVA weakness to a single-camera setup, but Appendix A.2 describes adapting the evaluated baseline to three views. Neither discrepancy should be silently repaired. object-setsbaseline-budgetsbaseline-implementation
7.2 Questions for discussion
- Would Two-Stage still win with matched compute and seed replication?
- How much task transfer survives when held-out tasks are absent from both action labels and video fine-tuning?
- Can video inference latency be reduced without weakening precise physical manipulation?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the SVD checkpoint, demonstration trajectories, camera packing, action alignment and both training stages. Appendix A.3 specifies ResNet18/CLIP-Base DP baselines with three encoders, CLIP task-text context, batch 768, and respective image sizes 256 × 256/224 × 224. Match these rather than assuming every baseline has identical pixel inputs. architectureimplementationbaseline-implementation
The physical setup uses D435 side cameras, a Basler fisheye gripper camera, T265 pose tracking, ArUco jaw tracking and a uniaxial force sensor at 30 Hz. Exact software versions, random seeds and a complete action-head optimizer/adapter configuration are not supplied. Table 5’s video-model learning rate does not replace the separate action rate in Section 4.1. The exact gradient routing of the comparator labeled Joint is not fully reconciled with Section 3.3’s stop-gradient statement. controllerviewsimplementationobjectivestraining-ablation
Proposed checks: equalize training compute when comparing Joint and Two-Stage; independently vary whether held-out tasks appear in video pretraining while keeping action supervision fixed. These test representation isolation and the actual source of transfer. training-ablationimplementationtransfer
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does training separation help after controlling compute and gradient routing?
Reader-proposed check, not performed: use the same RoboCasa demonstrations, SVD initialization, adapter and action head for a staged frozen-video model and concurrent-training controls. Explicitly test concurrent training both with and without action-loss gradients entering the video U-Net, since the source’s Joint label does not fully specify that distinction. First reproduce the reported schedules, then run a comparison matched by measured training compute and report video/action update counts. Use fixed evaluation scenes, multiple seeds and 50 rollouts per task. A repeatable Two-Stage advantage after matching would support a representation or optimization benefit; disappearance of the gap would weaken the claim that isolation itself explains Table 3. robocasaobjectivestraining-ablationimplementation
Check 2: Is held-out-task transfer supplied by the extra task videos?
Reader-proposed check, not performed: retain Figure 4’s same twelve action-supervised tasks in every arm. Compare video pretraining on all twenty-four tasks against pretraining only on the twelve action-supervised tasks. Match total video samples and optimization steps by resampling the restricted set, freeze each resulting video U-Net, and train otherwise identical action heads. Include the half-task DP baseline and the fully supervised reference. Report the twelve action-unsupervised tasks separately, alongside the all-task mean, over repeated seeds and identical evaluation rollouts. A selective loss on action-unsupervised tasks when their videos are removed would support the proposed mechanism. Similar performance would suggest broader priors or shared behavior explain more of the transfer. transfertraining-ablationimplementationrobocasa
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author page, method text and equations, all Figures 1–15 and Tables 1–9, and appendix implementation, evaluation and controller details. All six final original-PDF crops were inspected for legibility, labels and completeness. Figure 2 feature arrows agree with the method; its omitted gradient marker is explained from Section 3.3. Figure 4 reference extensions were checked against its caption. Reference-only pages 10–13 were read as text. Separate supplemental videos and external artifacts were not supplied for inspection.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- 1 Introduction
- 2 Related Works
- 3 Methods
- 3.1 Overview
- 3.2 Architecture
- 3.3 Learning
- 4 Experiments
- 4.1 Implementation Details
- 4.2 Simulation Experimental Setup and Baselines
- 4.3 Quantitative Results
- 4.4 Analysis
- 4.5 Real-World Results
- 4.6 Generalization Analysis
- 5 Conclusion
- 6 Limitations
- Acknowledgments
- References
- A Appendix
- A.1 Video Model Implementation
- A.2 Unified Video Action Model Baseline
- A.3 Diffusion Policy Baseline
- A.4 Real-World Experiment Setup
Outside the original text pass
- Identity: the inspected title page states Video Generators are Robot Policies, arXiv:2508.00795v1 [cs.RO], 1 August 2025. All seven authors match the catalog in order. This report covers that supplied v1; no other revision was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting PDF pages 1–9 and 14–23, including every numbered figure and table. References and acknowledgments on pages 10–13 were read as text; cited external works were not independently inspected.
- Separate supplemental videos, code, checkpoints and datasets were not inspected; no experiments were reproduced. No missing implementation details are inferred from the project URL.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin
Title, seven authors, Columbia University and Toyota Research Institute affiliations; arXiv:2508.00795v1, 1 Aug 2025.
Go to primary source ↓problemPDF pp. 1–2, Abstract and Section 1
Demonstration cost and distribution shifts motivate video-based learning; the decoder-as-interface interpretation and future release promise are author claims.
Go to primary source ↓architecturePDF p. 3, Sections 3.1–3.2, Equations (1)–(3)
SVD video U-Net, frozen VAE, CLIP task text, five decoder features (9, 14, 17, 20, 23), CNN adapter and separate action U-Net.
Go to primary source ↓objectivesPDF p. 4, Figure 2 and Section 3.3, Equations (4)–(5)
Video features feed the action network; separate denoising losses; action gradients are stopped before the video U-Net; outputs directly control the end-effector.
Go to primary source ↓viewsPDF p. 4, Section 4.1
Three views with eight frames each or LIBERO10 two views with twelve each; one padded frame; video/action learning rates 1e-5/5e-5.
Go to primary source ↓robocasaPDF p. 5, Table 1 (all rows, especially average row), caption and Section 4.2
50 demonstrations and 50 evaluation rollouts per task across five scenes; 0.63/0.66 Video Policy averages and listed baseline averages; seven-dimensional simulation action space. Reader arithmetic check of the 24 displayed rows gives 0.6325 for Video Policy (50 demonstrations), approximately 0.5858 for DP-VLA and 0.4067 for DP-CLIP; the latter two do not reproduce the printed 0.57 and 0.43. The full supplied text gives no alternative aggregation weights. Preserve the printed values as reported, not independently verified aggregates.
Go to primary source ↓baseline-budgetsPDF p. 6, Section 4.3
GR00T uses 300 and DP-VLA 3000 generated demonstrations per task; source attributes UVA limitations to single-camera reliance.
Go to primary source ↓liberoPDF p. 5, Table 2; PDF p. 17, Table 9 and caption
LIBERO10 average 0.94 versus UVA 0.90 and pi_0 0.85; own per-task evaluation uses 50 trials; Table 2 baselines come from prior work.
Go to primary source ↓training-ablationPDF p. 6, Section 4.4 and Table 3; PDF p. 15, Table 6 and caption
Joint 0.57, Two-Stage 0.63, No Video Tuning 0.09; stage two freezes video weights; Table 6 supplies 50-trial per-task results.
Go to primary source ↓horizonPDF pp. 6–7, Section 4.4 and Figure 3; PDF p. 17, Table 8
Fixed 1.6-second action prediction and 0.8-second rollout; video horizons 0/16/32 steps yield Table 8 averages 0.30/0.55/0.67; graph shows greater horizon benefit for shifted tasks.
Go to primary source ↓horizon-protocolPDF p. 14, Appendix A.1, second paragraph; PDF p. 18, Figure 6
Horizon diagnostic samples MimicGen environments sharing training layout/style; pick-and-place has larger object shifts. Appendix’s Figure 2 reference refers to the horizon analysis actually plotted in Figure 3.
Go to primary source ↓transferPDF p. 7, Section 4.4 and Figure 4/caption; PDF p. 15, Table 6, Half Tasks and DP Half Tasks columns
Video training sees all 24 tasks, action supervision only 12; overall success 0.41 versus 0.21. Thin figure extensions denote fully action-supervised upper-bound references.
Go to primary source ↓real-setupPDF pp. 7–8, Section 4.5
Five physical tasks, 200 handheld-gripper demonstrations each and location/object/background shifts; task definitions include opening a drawer beyond half extension.
Go to primary source ↓real-resultsPDF pp. 8–9, Section 4.6, Figure 5 and Table 4/caption
All fifteen success rates come from ten rollouts per task/condition. Authors discuss unrealistic upright/stacking predictions and background-related M&M localization failures.
Go to primary source ↓limitationsPDF p. 9, Section 6
Limited simulation scale, one embodiment, one video-model family and diffusion inference cost restrict applicability.
Go to primary source ↓implementationPDF p. 14, Appendix A.1 and Table 5
Camera-aware embeddings, frame packing, stride four and 32-step horizon; eight A100 GPUs/about two weeks; 30 denoising steps, guidance 2.0 and about nine-second 256 × 256 generation; training schedules and resolutions.
Go to primary source ↓baseline-implementationPDF p. 19, Appendix A.2–A.3
UVA adapted to three views; DP uses three encoders, CLIP text context, ResNet18/CLIP-Base, specified image sizes and batch 768; predicts 32 steps and rolls out 16.
Go to primary source ↓controllerPDF p. 19, Appendix A.4; PDF p. 21, Figure 13
D435, Basler fisheye and T265 cameras, ArUco jaw tracking, force sensor; 30 Hz; physical 32-step prediction/24-step impedance rollout and force-based closing correction.
Go to primary source ↓object-setsPDF p. 8, Section 4.5; PDF p. 20, Figures 8–10; PDF p. 21, Figures 11–12
Object/background panels specify the physical test scope; drawer novelty is masking tape. Stack Cups count conflicts: six in main text versus five in Figure 12.
Go to primary source ↓qualitativePDF pp. 22–23, Figures 14–15 and captions
Three-view video-prediction/robot-rollout comparisons illustrate grasping and placement; these are qualitative examples.
Go to primary source ↓8.5 Primary sources
Video Generators are Robot Policies ↗
PDF · 9,162 extracted words
Source fingerprint
b816cc80f0368c3ab09df906de831b47c93d7f78b9a19af1ff698ae26eaa8165