Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
1. Paper overview
In one sentence: VPP trades full video generation for one pass through a frozen video predictor, then learns a separate action policy from its predictive features. e02e03e04e05e06e09e13e16e21
| At a glance | What to know |
|---|---|
| Research problem | Source description Static visual encoders may miss motion information needed for manipulation. The paper asks whether a pretrained video model supplies useful future structure before producing photorealistic video, and whether a policy can translate that structure into robot commands. Its implicit inverse-dynamics interpretation assumes predictive features encode robot motion that the downstream policy can exploit. e02e04 |
| Core mechanism | |
| A key reported result | CALVIN ABC→D long-horizon instruction following: 4.33; 0.769 Average completed chain length; five-task success rate. Language-annotated ABC training; unseen D; five chained tasks; Table 1 full-data rows. RoboUniview: 3.65; 0.507. GR-1: 3.06; 0.401. The 18.6% relative length gain uses RoboUniview; the introduction’s 41.5% uses GR-1. These are different comparators. Scores mix imported and rerun baselines. e08e09 |
| Reading caution |
Core contributions
Figure 2. Predictive video features become the conditioning signal for a separate action policy. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left with the instruction and image-conditioned video model. Follow the downward arrows from its upsampling blocks: Section 4.2 says these feature maps are interpolated and concatenated across channels. The circle-plus graphics accompany stacking; the text specifies concatenation, not elementwise addition. Follow the long arrow into frame-indexed tokens on the right. The attention inset uses learnable tokens as queries and visual features as keys and values. Spatial attention precedes temporal attention and the feed-forward network, matching Equation (5). The resulting tokens and language condition the DiT blocks. Section 4.1 establishes that TVP is frozen during this second training stage. e03e04e05e06e19
What it supports. The architecture separates predictive representation learning from action learning. During policy inference, the video model supplies internal features after one forward pass; the right-hand action denoiser still performs its own iterative sampling. Therefore, one video pass should not be read as one total denoising step for the entire policy.
Where the evidence stops. The image-like thumbnails at the action head are schematic: Equations (5)–(6) define action generation from tokens. Equation (6) targets clean actions, despite adjacent prose saying noise prediction; that implementation ambiguity is unresolved.
2. Motivation
2.1 The problem and the proposed response
Static visual encoders may miss motion information needed for manipulation. The paper asks whether a pretrained video model supplies useful future structure before producing photorealistic video, and whether a policy can translate that structure into robot commands. Its implicit inverse-dynamics interpretation assumes predictive features encode robot motion that the downstream policy can exploit. e02e04
2.2 What this reading follows
A manipulation policy must decide how the scene should change, not merely recognize what is present. VPP addresses this by adapting Stable Video Diffusion to manipulation and reading its internal features before a finished video exists. A Video Former compresses those features, and a separate diffusion policy translates them into actions. The central question is whether approximate future structure is already sufficient for control. The figures below connect that design to CALVIN chains, physical dexterous tasks and targeted ablations. They also expose reporting inconsistencies that matter for reproduction, so the observed gains are kept separate from the authors’ explanation of why generalization occurs. e02e03e04e05e06e09e13e16e21
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × IDM is supported by a separately adapted, frozen video predictor feeding a separately trained action diffusion policy. The authors call the latter implicit inverse dynamics. This is representation-conditioned IDM rather than an explicit two-image transition regressor, and it is not joint future/action generation in one network. e02e03e04e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Train a useful future space before aligning it with actions
The first training stage gives heterogeneous videos a common supervision format: reconstruct a clean sequence conditioned on its initial observation and language. Equation (3) does not require robot action labels, so human manipulation and robot videos can train the same predictor. Equation (4) weights data sources, and Table 8 supplies the actual sampling mixture and held-out data rules. After adaptation, the video model is frozen. Downstream demonstrations then teach a separate policy how the resulting visual features relate to a particular action space. This division explains why the CALVIN 10%-annotation result is meaningful without implying that VPP used only a small video corpus. The dexterous collection and mixture counts still disagree, so an exact data reconstruction requires clarification before this training recipe can be reproduced faithfully. e03e07e09
Figure 4. A coarse one-pass prediction can expose motion without reconstructing clean textures. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. For each task, begin with the isolated yellow-bordered input. Move left to right through the green ground-truth sequence, then compare the red and blue rows at similar positions. The legend identifies red as 30-step denoising and blue as one-step direct prediction. In the drawer example, the blue row retains broad arm movement while blurring its appearance; the orange example also shows distorted colors and object shapes. These are visualizations of the predictive representation, whereas the policy consumes internal feature maps. The red rows illustrate what further video denoising can produce, not the computation performed at every policy update. e04e18e20
What it supports. The examples support the narrower observation that the first pass preserves approximate scene evolution even when textures are poor. This makes VPP’s choice of internal predictive features plausible. The separate CALVIN ablation, where two passes score 4.19 versus 4.33 for one, provides control evidence beyond these selected pictures.
Where the evidence stops. These are qualitative examples, not calibrated trajectory-error measurements. The graphic alone cannot show that the action head tracks robot motion, nor that coarse predictions remain reliable across all unseen objects, viewpoints or contact events.
5.2 Keep video encoding and action denoising separate
At a policy update, the current image and instruction enter TVP alongside a noisy video latent. A single forward pass exposes upsampling features across predicted times; interpolation and channel concatenation retain several levels of representation. The Video Former compresses these features through spatial attention followed by temporal attention, yielding tokens for the action model. The DiT then denoises an action chunk conditioned on those tokens and language. Thus VPP avoids repeatedly refining a video, while still sampling actions iteratively. Table 13 lists ten action sampling steps and task-dependent action shapes. Equation (6) prints clean-action reconstruction despite prose describing noise prediction, so that target should be checked in any implementation. The diagrams explain information flow, but they do not define every controller delay or the exact feedback schedule during a chunk. e04e05e06e11e19
5.3 Evaluate the mechanism through several kinds of evidence
Reader analysis: the strongest case for VPP comes from connecting complementary measurements. CALVIN tests executed instruction chains in a held-out environment; the dexterous tables test physical actions; video visualizations show what the predictive features may contain. Component removals connect representation processing to performance, while the temporal ablation asks whether cross-frame aggregation matters. None of these alone identifies the learned action head as a motion tracker. Encoder substitutions also change pretraining and architecture, and the two-pass comparison changes computation time. The proposed mechanism is plausible, but its causal test should preserve appearance information and policy capacity while disrupting temporal structure. Likewise, the real-world averages should be reconstructed from trial records before treating the paper’s headline percentage as one comparable effect across task groups. e08e09e12e13e14e16e18e20e21
5.4 Training and inference
During training
Stage one minimizes video reconstruction losses weighted across human, robot and collected data. Table 8 lists 375,192 trajectories, dataset sampling ratios and validation holdouts. Stage two freezes TVP and trains aggregation and action modules on downstream demonstrations. e03e07
Reported compute is 2–3 days on eight A100s for video fine-tuning and 6–12 hours on four A100s for policy training. CALVIN uses language-annotated ABC; MetaWorld uses 50 oracle trajectories per task. e08e11
During inference
Each new observation requires one TVP pass followed by action diffusion. Reported rollout is 7–10 Hz on RTX 4090, with TVP under 160 ms. Table 13 uses ten action sampling steps; CALVIN/Panda/Xhand action horizons are ten, MetaWorld four. These counts differ from video denoising steps and do not specify every sensor-to-actuator delay. e04e11e19
5.5 Implementation flow
- Adapt the video predictor
Start from 1.5-billion-parameter SVD. Add CLIP language cross-attention and concatenate the initial observation with each predicted frame as conditioning. Train on mixed manipulation videos; the main description uses 16 frames at 256 by 256 resolution. e03
- Extract predictive features
Run frozen TVP once on current observation, language and noisy latent. Interpolate upsampling features to a common spatial grid and concatenate channels. Separate camera views produce separate predictive volumes, each retaining time. e04
- Compress time, space and views
Learnable Video Former queries attend to matching-frame features across views. Temporal attention and feed-forward layers combine frames; the resulting tokens condition the action head through cross-attention. e05e06
- Generate executable actions
The DiT reconstructs actions from noisy actions. This is action generation conditioned on predicted features, not explicit geometric tracking or optimization over candidate action-conditioned world rollouts. e02e06
6. Experiments & results
Video Prediction Policy (VPP) turns a manipulation-adapted video diffusion model into a predictive visual encoder. A separate diffusion policy learns actions from its internal future features. One video forward pass replaces full video generation during control. CALVIN and physical-robot results support the approach, while conflicting values and incomplete protocol details constrain causal and reproducibility conclusions.
6.1 Read the original evidence
Table 1. CALVIN shows the largest advantage at longer chains, under the paper’s reported comparison protocol. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the annotation column before comparing rows. The main block uses all language-annotated ABC data for action learning and evaluates in unseen environment D. Columns one through five describe completion of progressively longer task chains, while the final column is average completed length. Thus VPP’s 0.769 belongs to completing five tasks, whereas 4.33 measures length. Compare its highlighted row with RoboUniview and GR-1 separately. The bottom two rows change the downstream annotation budget to 10%. They do not state that the full video pretraining mixture shrinks to 10%, so those rows support action-label efficiency under the pretrained representation. e07e08e09
What it supports. VPP reports 4.33 average length versus RoboUniview’s 3.65, an approximately 18.6% relative improvement. Against GR-1’s 3.06, the relative gain is approximately 41.5%, explaining the different introduction headline. At five tasks, VPP reports 0.769 versus RoboUniview’s 0.507; these are success fractions rather than length scores.
Where the evidence stops. Section 5.1 combines imported and rerun baseline scores and supplies no seed-level uncertainty. Its provenance paragraph mentions 3D Diffuser Actors while this table names RoboUniview. Treat this as the reported benchmark comparison, not a matched-compute rerun of every method.
Table 7. Executed dexterous tasks provide evidence beyond plausible generated futures. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three blocks independently: seen manipulation, unseen manipulation and tool use. Unseen is defined in Section 5.3 by new objects or backgrounds in the collected-data setting. The four method columns use success fractions. Inspect category rows before the bold averages: VPP’s unseen unplug score is only 0.20 even though its overall unseen average is 0.605. The retained footnote gives different trial counts for pick/place and other categories, and just ten trials per tool task. Those unequal counts matter when interpreting aggregate results. The bottom block concerns physical spoon, hammer, drill and pipette tasks, not merely predicted videos. e14e16
What it supports. The table reports VPP averages of 0.749 for seen tasks, 0.605 for unseen tasks and 0.68 for tool use, compared with Susie’s 0.450, 0.328 and 0.23. Tool-specific VPP results range from 0.4 for pipette to 0.9 for spoon. The evidence therefore shows substantial gains with uneven capability across tasks.
Where the evidence stops. Table 5 gives Susie’s unseen average as 0.28, conflicting with 0.328 here. The abstract’s 31.6% aggregate improvement lacks an explicit weighting formula. These reported summaries should remain distinguishable; ten trials per tool and absent uncertainty estimates also limit precision.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN ABC→D long-horizon instruction following Language-annotated ABC training; unseen D; five chained tasks; Table 1 full-data rows. | 4.33; 0.769 Average completed chain length; five-task success rate | RoboUniview: 3.65; 0.507. GR-1: 3.06; 0.401. The 18.6% relative length gain uses RoboUniview; the introduction’s 41.5% uses GR-1. These are different comparators. Scores mix imported and rerun baselines. e08e09 |
| CALVIN with reduced action annotation ABC→D; 10% language-annotated ABC data for downstream training. | 3.25 Average completed chain length | GR-1 at 10%: 1.41; GR-1 at 100%: 3.06. Shows reduced downstream annotation requirements, not training with only 10% of all pretraining videos. e07e09 |
| MetaWorld 50-task manipulation One language-conditioned policy; 50 oracle demonstrations per task. | 0.682 Average task success rate | GR-1: 0.574; Susie: 0.410. A 10.8-percentage-point gain over GR-1, not a 10.8% relative gain. This multi-task evaluation differs from CALVIN unseen-environment transfer. e08e10 |
| CALVIN component and temporal ablations ABC→D; Tables 4 and 12. | Full VPP: 4.33, about 140 ms. Average completed chain length; latency | Without Video Former: 3.86, about 450 ms; without temporal attention: 4.18; two TVP passes: 4.19. Supports aggregation and one-pass encoding in this configuration. No uncertainty is reported; feature-aggregation removal has conflicting values. e11e13e18 |
| Franka Panda seen and unseen manipulation Table 6; seen pick/place 50 trials each, others 20; unseen counts 25 and 10 respectively. | 0.856 / 0.737 Reported seen / unseen success averages | Susie: 0.563 / 0.463. Uses detailed appendix values; Table 5 reports coarser values. Unseen means object/background novelty. e14e15 |
| Xhand dexterous manipulation and tool use Table 7; seen/unseen categories and four tool tasks, ten trials per tool. | 0.749 / 0.605 / 0.68 Reported seen / unseen / tool-use success averages | Susie: 0.450 / 0.328 / 0.23 in Table 7. Table 5 instead gives Susie unseen 0.28. Conflicting summaries and unspecified headline aggregation prevent adopting the abstract’s 31.6% as one verified effect. e14e16 |
| Bridge video prediction Appendix B.2 references Seer’s evaluation setup; comparator scores are imported. | 41.4 Fréchet Video Distance, lower is better | Seer: 246.3. Supports reported prediction quality, not robot success or the quality of one-pass features. e17e20 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Data adaptation and feature compression both contribute to the reported control performance. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row as the common VPP reference, then distinguish data removals from architectural changes. Removing internet data retains SVD initialization; the two-line row removes both internet data and SVD pretraining. Removing CALVIN video instead probes the role of adaptation to the downstream visual domain. The final two rows modify how video features reach the policy. Read both numerical columns: average length should increase, while latency should decrease. In particular, eliminating Video Former lowers task completion and increases latency. Section 5.2 identifies this timing comparison as evaluation of the complete VPP model on one RTX 4090. e11e13
What it supports. VPP scores 4.33 at about 140 ms; without Video Former it scores 3.86 at about 450 ms. Removing internet data gives 3.97, whereas removing CALVIN video gives 3.31. These outcomes support both domain adaptation and learned compression, although the ablations probe different interventions rather than a single common causal factor.
Where the evidence stops. The feature-aggregation row says 3.60, but the adjacent prose and Table 10’s final-layer result say 4.05. Preserve both values rather than resolving them by assumption. The 1.63 row also removes two factors together, so it cannot isolate the effect of SVD initialization alone.
Table 12. One video pass remains competitive with two in the tested architecture. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Decode the row labels using Appendix C.2 on PDF pages 15–16: Single-view retains only the static camera, Ablation.1 removes temporal attention, and Ablation.2 uses two TVP denoising steps. The final column is average chain length; the preceding five columns are success through progressively longer chains. Compare each intervention with the same full-model row. Avoid confusing these one-versus-two forward passes with Table 11’s diffusion-time settings, or with the separate ten-step action sampler in Table 13. Those quantities govern different parts of the inference pipeline. The table itself supplies neither latency measurements nor uncertainty bars for these variants. e18e19
What it supports. Full VPP reaches 4.33 average length, versus 4.18 without temporal attention and 4.19 with two TVP passes. Static-view-only input reaches 3.58, with five-task success 0.518. These findings support the temporal and multi-view design and provide no observed accuracy benefit for the extra video pass in this setup.
Where the evidence stops. The nearby prose mistakenly calls 3.58 a Task-5 success rate; the table identifies it as average length. Without repeated-run uncertainty or matched feedback timing, the small one-versus-two-pass difference cannot establish a universal advantage for single-pass representations.
7. Analysis & limitations
7.1 What the evidence leaves open
Feature-aggregation removal is 3.60 in Table 4 but 4.05 in prose and the final-layer Table 10 result. Single-view prose mislabels 3.58 as Task-5 success; Table 12 distinguishes length 3.58 from success 0.518. Both discrepancies remain explicit. e13e18
Encoder substitutions change architecture and pretraining; the 1.63 ablation removes both internet data and SVD initialization. Neither uniquely isolates future-motion information. Qualitative prediction/execution alignment is consistent with, rather than proof of, implicit motion tracking. e12e13e20e21
Unseen real tasks concern collected-data object/background novelty; absence from broad pretraining is not established. Tables give no seed-level uncertainty, and tool use has ten trials per task. Broad reliability and causal generalization claims exceed the measured scope. e14e16e21
7.2 Questions for discussion
- Would temporal shuffling preserve performance after controlling current appearance and token count?
- Does one-pass encoding remain advantageous with matched observation-feedback timing?
- Which aggregation reproduces the dexterous summaries and headline improvement?
8. Reproducibility audit
8.1 Requirements and known gaps
Use Table 8’s mixture and holdouts, but resolve 2,476 dexterous video trajectories versus 4,000 collected demonstrations. The human counts 193,690 and 191,642 differ by the stated 2,048 holdout; this does not resolve the dexterous discrepancy. e07
For CALVIN, Table 13 specifies six Video Former layers, four encoder/four decoder DiT layers, policy batch 76, 12 epochs and learning rate 1e-4. Remaining settings reference prior work. Exact checkpoints, diffusion schedules, controller timing and seed-level evaluation records require clarification. e11e19
Proposed focused checks compare intact versus temporally shuffled features with matched policy capacity, and one versus two TVP passes under matched feedback timing. These target predictive content and latency, respectively; neither was run here. e04e05e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does ordered future information improve action learning?
Reader-proposed check, not executed: freeze the same adapted TVP and train otherwise identical CALVIN policies on intact features or future-frame features shuffled across temporal positions, preserving the current-frame features, language, feature dimensions and token budget. Apply each transformation consistently during its policy’s training and evaluation to avoid testing only a surprise distribution shift. Use the same ABC annotations, D task chains, action sampler and several seeds. Report chain length and five-task success. A repeatable degradation under shuffling would support the importance of ordered future information; unchanged performance would weaken that specific explanation without disproving the value of video pretraining. e04e05e08e18e19
Check 2: Separate extra prediction computation from feedback delay
Reader-proposed check, not executed: compare one and two TVP passes with the same CALVIN policy architecture, feature aggregation, ten-step action sampler and ten-action chunk. Train each configuration consistently. Evaluate at native speed and at a shared slower observation-update schedule, delaying the one-pass condition to match the two-pass latency. Record end-to-end latency, actual observation times and chain success over paired task sequences and seeds. If two passes help only after feedback timing is matched, latency plausibly explains the native-speed result. If two passes still provide no gain, the evidence favors sufficiency of one-pass features in this configuration. e04e11e18e19
8.3 Reading coverage
Visual audit: Visually inspected the title and affiliation block; Figures 1–10; Tables 1–13; Equations (3)–(6); and the supporting method, training, evaluation and appendix text on these pages. Every final crop was separately viewed. Figure 2 attention arrows and feature-stacking symbols were checked against Section 4.2 and Equation (5). Equation (6), feature-aggregation scores, single-view metric wording and real-world summary discrepancies are disclosed rather than silently repaired. Appendix PDF pages 13–19 have printed pages 1–7. Reference-only PDF pages 10–12 were read in the complete text but not rendered. No external videos, code or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author credits and abstract (PDF p. 1)
- Sections 1–3: Introduction, Related Works and Preliminaries (PDF pp. 1–3)
- Sections 4.1–4.2: video training, predictive features, Video Former and action generation (PDF pp. 3–5)
- Sections 5.1–5.3: simulation, ablations and physical robot evaluation (PDF pp. 5–9)
- Conclusion, acknowledgements, impact statement and references (PDF pp. 9–12)
- Appendix A.1–A.2: real-world protocols (PDF pp. 13–14; appendix pp. 1–2)
- Appendix B.1–B.4: data mixture, FVD and predictions (PDF pp. 14–15, 18–19)
- Appendix C.1–C.3: architecture, extra ablations and baselines (PDF pp. 15–17)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting all scientific figure/table pages and six final original crops. References on PDF pp. 10–12 were read as text; external cited works were not opened.
- Identity/version: the supplied ICML 2025 / PMLR 267 title and all nine authors match the catalog. No numbered revision or alternative edition was supplied; cross-version equivalence is not claimed.
- All six supplied text chunks were read individually. Code, external project videos and separate supplements were not inspected; experiments were not reproduced. Appendix printed pagination restarts at 1 on PDF page 13.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author line, affiliation footnote and proceedings imprint
The title and all nine catalog authors match. Yucheng Hu and Yanjiang Guo share equal contribution. The imprint identifies ICML 2025, PMLR 267; no numbered revision is printed. Affiliations: IIIS, Tsinghua University; Shanghai AI Lab; Shanghai Qi Zhi Institute; RobotEra; University of California, Berkeley.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Section 1 and Figure 1
Predictive representations contain current and future frame features. The authors describe downstream action learning as implicit inverse dynamics and motivate it by limitations of static visual encoders.
Go to primary source ↓e03PDF p. 3, Section 4.1, Equations (3)–(4)
A 1.5-billion-parameter SVD receives CLIP language cross-attention and initial-image channel conditioning. The main configuration predicts 16 frames at 256 by 256. Video reconstruction combines human, robot and collected data; TVP is frozen for action learning.
Go to primary source ↓e04PDF p. 4, Figure 2 and Section 4.2, TVP Model as Vision Encoder
One forward pass processes current image, language and noisy latent. Upsampling features are interpolated to a common spatial size and concatenated by channel; static and wrist views are predicted independently.
Go to primary source ↓e05PDF p. 4, Figure 2 attention inset, Video Former paragraph and Equation (5)
Learnable queries attend spatially to corresponding-frame visual features across views, then use temporal attention and a feed-forward network. The inset routes learned tokens through query projection and visual features through key/value projections.
Go to primary source ↓e06PDF pp. 4–5, Action Generation and Equation (6)
The DiT head is conditioned on aggregated tokens and language. Equation (6) minimizes squared error to clean actions a_0, although adjoining prose describes approximating noise epsilon; the inconsistency remains unresolved.
Go to primary source ↓e07PDF p. 5, VPP Training Details; PDF p. 15 (appendix p. 3), Table 8 and footnote; PDF pp. 8 and 13, dexterous collection descriptions
Table 8 lists 375,192 trajectories. Sampling ratios are Something-Something-v2 0.30, RT-1 0.15, Bridge 0.15, BC-Z 0.08, Taco-Play 0.01, Jaco-Play 0.01, CALVIN-ABC 0.10, MetaWorld 0.05, Panda 0.05, dexterous hand 0.10. The footnote reserves 5,558 Bridge and 2,048 human trajectories and 3% elsewhere. Its 191,642 human count differs from the main-text 193,690 by the stated holdout; 2,476 dexterous trajectories are not reconciled with 4,000 collected demonstrations.
Go to primary source ↓e08PDF pp. 5–6, Section 5.1 benchmark setups and Quantitative Results; PDF p. 16, Section C.3
CALVIN trains on language-annotated ABC and tests unseen D with five-task chains. MetaWorld uses 50 oracle trajectories per task for 50 tasks. CALVIN comparisons mix imported scores and rerun baselines; the provenance paragraph mentions 3D Diffuser Actors although Table 1 names RoboUniview in its 3D row.
Go to primary source ↓e09PDF p. 5, Table 1; PDF p. 1, Abstract; PDF p. 2, final introduction paragraph
VPP average length is 4.33 versus RoboUniview 3.65 and GR-1 3.06; five-task success is 0.769 versus 0.507 and 0.401. At 10% annotated data VPP achieves 3.25 versus GR-1 1.41. The abstract states 18.6% relative improvement; the introduction states 41.5% against the GR-1 citation.
Go to primary source ↓e10PDF p. 5, Table 2 and MetaWorld Benchmark paragraph
One language-conditioned VPP policy reports success 0.682 across 50 tasks, versus GR-1 0.574 and Susie 0.410. Easy/middle/hard VPP success is 0.818/0.493/0.526 across 28/11/11 tasks.
Go to primary source ↓e11PDF p. 6, training continuation and Policy Roll-out Details; PDF p. 7, Table 4 and Video Former ablation paragraph
Video fine-tuning takes 2–3 days on eight A100s; policy training takes 6–12 hours on four A100s. Reported rollout is 7–10 Hz on RTX 4090, with a TVP pass under 160 ms and action chunking. Table 4 gives about 140 ms for VPP and 450 ms without Video Former, evaluated on one RTX 4090.
Go to primary source ↓e12PDF pp. 6–7, Section 5.2 and Table 3
Encoder substitution gives average lengths 2.58 for Stable-VAE, 1.23 for VC-1 and 1.54 for Voltron, versus VPP 4.33. VC-1 and Voltron receive additional video-dataset fine-tuning; Stable-VAE does not. Feature resampling is used, but architecture and pretraining objective change.
Go to primary source ↓e13PDF p. 7, Table 4 and feature-aggregation paragraph; PDF p. 16 (appendix p. 4), Table 10
Table 4 reports 3.97 without internet data, 3.31 without CALVIN video, 1.63 without both internet data and SVD initialization, 3.86 without Video Former and 3.60 without feature aggregation. The prose gives 4.05 for final-layer features, matching Table 10 layer 12; the 3.60 versus 4.05 discrepancy is unresolved. Layer 9 gives 4.29.
Go to primary source ↓e14PDF p. 8, Figure 5 and Section 5.3; PDF p. 13 (appendix p. 1), Sections A.1–A.2 and Figure 7
Real platforms are Franka Panda and Xarm with a 12-DoF Xhand. Unseen denotes new objects or backgrounds relative to collected training data. Collection uses a space mouse for Panda and retargeted Vision-Pro hand movements for Xhand. A low-level PD controller smooths VPP dexterous trajectories.
Go to primary source ↓e15PDF p. 13 (appendix p. 1), Table 6 and caption; PDF p. 8, Table 5
Table 6 gives Panda seen/unseen averages 0.856/0.737 for VPP and 0.563/0.463 for Susie; Table 5 gives coarser 0.85/0.73 and 0.56/0.46. Trials per seen pick/place category are 50, versus 20 for others; unseen counts are 25 and 10 respectively.
Go to primary source ↓e16PDF p. 14 (appendix p. 2), Table 7 and caption; PDF p. 8, Table 5; PDF p. 1, Abstract
Table 7 gives VPP seen/unseen/tool-use averages 0.749/0.605/0.68 and Susie 0.450/0.328/0.23. Table 5 instead gives Susie unseen 0.28, beyond rounding. VPP spoon/hammer/drill/pipette success is 0.9/0.6/0.8/0.4, ten trials per tool. The headline 31.6% dexterous improvement lacks an explicit aggregation formula.
Go to primary source ↓e17PDF p. 14, Section B.2; PDF p. 15 (appendix p. 3), Table 9 and caption
Bridge video FVD is 41.4 for VPP versus 246.3 for Seer, 501.2 for VideoFusion and 515.7 for Tune-A-Video; lower is better. Baselines are copied from Seer, whose evaluation setup is referenced. This is video evaluation, not executed control.
Go to primary source ↓e18PDF pp. 15–16, Section C.2; PDF p. 16 (appendix p. 4), Tables 11–12
Table 12 average lengths are 4.33 for VPP, 3.58 for static-only, 4.18 without temporal attention (Ablation.1), and 4.19 with two TVP steps (Ablation.2). Single-view five-task success is 0.518, although prose mislabels 3.58 as Task-5 success. Table 11 diffusion-time settings 10/20/30 give 4.21/4.33/4.25; these settings differ from one versus two forward steps.
Go to primary source ↓e19PDF p. 17 (appendix p. 5), Table 13; PDF p. 15, Section C.1
CALVIN uses video length 16, action shape 10 by 7, six Video Former layers, four encoder and four decoder DiT layers, ten action sampling steps, policy batch 76, 12 epochs and learning rate 1e-4. MetaWorld uses video length 8 and action shape 4 by 4; Panda/Xhand action shapes are 10 by 7 and 10 by 18. Remaining settings are delegated to prior SVD and MDT work.
Go to primary source ↓e20PDF p. 7, Figure 4 and caption; PDF p. 15, Section B.4; PDF p. 18, Figure 9
Figure 4 compares yellow input, green ground truth, red 30-step predictions and blue one-step predictions. One-step results show rough textures and approximate motion. Figure 9 adds examples; neither directly measures the action head tracking robot motion.
Go to primary source ↓e21PDF pp. 8–9, Figure 6 and Generalization Analysis
Examples compare red predicted futures with green executed trajectories for tennis-ball relocation, pouring and spoon use. The authors attribute generalization to predictive video knowledge and implicit inverse dynamics; this is their mechanism interpretation.
Go to primary source ↓e22PDF p. 15, Section B.3; PDF p. 18, Figure 8; PDF p. 19, Figure 10
Human and robot validation predictions use 30 denoising steps and classifier-free guidance 7.5; eight frames are shown from a 16-frame horizon. Green denotes ground truth and red predicted futures. These full video-generation examples differ from the one-pass policy encoder.
Go to primary source ↓8.5 Primary sources
Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations ↗
PDF · 9,844 extracted words
Source fingerprint
dd668ffe00f01bfa8683e4457b692e539712434e1cfb1e9b3195c0dac9500e89