Scalable Policy Evaluation with Video World Models
1. Paper overview
In one sentence: Action-conditioned video rollouts can rank robot policies, but their usefulness depends on behavioral coverage, faithful dynamics and reliable success judgments. e02e04e06e09e12e14e15e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Real robot evaluation requires repeated resets, equipment and labor; handcrafted simulation demands assets and physics/rendering alignment. The target is an observation-conditioned simulator that preserves performance differences between policies. Agreement with actual policy performance, rather than visual plausibility alone, is the central criterion. e02e03 |
| Core mechanism | |
| A key reported result | Real-world policy evaluation in the Bridge setup: Reported Cosmos: Pearson 0.687; MMRV 0.171. Pearson correlation ↑ / MMRV ↓. OCTO-Small, OCTO-Base and OpenVLA; lifting a pot, carrot, eggplant or cup; Bridge V2 and 300 local trajectories train the video model. Reported IRASim: Pearson 0.613; MMRV 0.611. Figure 6 reports positive association with physical performance in this adapted setup. Its MMRV values are retained as printed. Under Equations (1)–(2), MMRV cannot exceed the range of the actual scores used. If these inputs are the plotted rates, visually about 0.10–0.70, IRASim’s 0.611 is not readily reconciled with that bound. Raw scores and sufficient aggregation or normalization details are absent, so the size of this MMRV difference is not independently reconstructed evidence. Trial counts and confidence intervals are also missing. e14e03 |
| Reading caution | Source description Hallucinations can remove graspable geometry or place an eggplant into the gripper, biasing success downward or upward. View disagreement and repeated near-static frames further break interaction. Errors compound autoregressively, especially in Tool Hang rollouts of 700 transitions. e15 |
Core contributions
Figure 2. Actions enter the video denoiser through its time-conditioning pathway. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper path from corrupted image tokens through self-attention, cross-attention and the MLP to denoised tokens and decoding. Then trace the lower action branch: Fourier feature mapping feeds an action encoder, and the plus sign combines its embedding with the time embedding. The caption and Section III-B explicitly confirm this addition; the scale, shift and gate block distributes the conditioning to the transformer layers. Read the legend before interpreting training: blue modules start from pretrained weights, pink identifies the new action module, and gray marks frozen encoding and decoding. The generated output is an observation; the separately evaluated policy supplies actions. e05e06
What it supports. The adaptation reuses a pretrained video model while introducing a learned route for robot actions to affect predicted observations. This architecture supports an interactive visual simulator: its output can be returned to a policy for the next decision. It does not depict a shared head that predicts both future images and control actions.
Where the evidence stops. Section III-B calls training end-to-end, while the figure explicitly freezes encoding/decoding. Preserve that qualification when interpreting the phrase. The supplied main PDF also defers detailed architecture settings to Appendix VII, which was not supplied.
2. Motivation
2.1 The problem and the proposed response
Real robot evaluation requires repeated resets, equipment and labor; handcrafted simulation demands assets and physics/rendering alignment. The target is an observation-conditioned simulator that preserves performance differences between policies. Agreement with actual policy performance, rather than visual plausibility alone, is the central criterion. e02e03
2.2 What this reading follows
A simulator for evaluating a robot policy must respond to the policy’s mistakes as well as its successful actions. This paper adapts Cosmos to that role: actions condition future video, the policy reacts to generated observations, and a vision-language model labels task completion. The key question is whether this loop preserves the performance differences measured in actual environments. The figures below connect the conditioning mechanism to synthetic ranking results, rollout-data ablations and a Bridge comparison with physical performance. They also expose the boundary: convincing video can contain impossible grasps, missing objects or incompatible camera views, and the success judge introduces its own errors. e02e04e06e09e12e14e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
Neural policy evaluation and manipulation simulator categories fit the contribution. The world model predicts observations conditioned on a separate policy’s actions; it is not a One Model controller, joint future/action predictor or inverse-dynamics action extractor. Controller architecture, prediction paradigm and quadrant are therefore not applicable, rather than unknown. The sim-to-real evaluation tag denotes comparison with physical performance, not policy training in this model followed by robot transfer. e04e05e06e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make policy evaluation a conditional-dynamics problem
Begin with the boundary between the policy and the simulator. Section III-B gives the policy responsibility for choosing end-effector actions from observation history. The learned model instead predicts the next observation conditional on that history and the actions. Repeating this process creates a closed interaction loop: a prediction error can change the policy’s next decision, which then changes subsequent model inputs. Figure 2 adds action information to an existing video denoiser, while Figure 1 holds the policy and world model fixed during evaluation. Afterward, the VLM labels completion and those labels become estimated policy performance. Equations (1)–(2) test whether relative performance is preserved, penalizing a reversed ordering more strongly when the true policy gap is large. This explains why evaluating generated images alone would miss the central objective. e03e04e05e06
5.2 Train the simulator to encounter imperfect behavior
The paper starts from successful teleoperated demonstrations, yet its simulator must evaluate policies of different skill levels. Those policies can produce behaviors that successful demonstrations rarely contain. Section III-C therefore adds policy rollouts, and Section IV-A keeps the rollout-generating policies separate from the later evaluation policies. Figure 5 connects this choice to both visual artifacts and ranking metrics; Table II tests video prediction separately. Reader analysis: the useful mechanism is broader coverage of the interactions that an evaluator will encounter, but the presented augmentation also increases data volume. A causal claim about diversity alone requires a matched-size comparison. Similarly, Figure 7 and the pretrained-weight ablation support initialization benefits without proving that the model has learned complete physical constraints. Figure 8 shows that object permanence and view consistency can still fail. e07e08e11e12e13e15
5.3 Separate simulated behavior from the success judge
The final score passes through two imperfect stages. First, generated dynamics determine what the policy appears to do; hallucinations can manufacture a grasp or remove geometry needed for success. Second, a VLM interprets subsampled rollout frames using a completion prompt. Section IV-D reports that VLM annotations often reach only 65–80% accuracy, with further details outside the supplied PDF. Reader analysis: a change in the final ranking can therefore come from better dynamics, better labels, or both. Figure 6 provides an end-to-end comparison with physical performance, but does not disentangle these contributions. The same distinction matters when reading Table I: high Pearson correlation and low MMRV support useful comparisons between policies, while the underprediction visible in Figure 3 warns against interpreting those metrics as calibrated success probabilities. e03e04e09e10e14e15e16
5.4 Training and inference
During training
The backbone is pretrained on text-to-video and image-to-video data; action-conditioned training uses EDM. Figure 2 distinguishes pretrained trainable modules, an action module trained from scratch, and frozen image encoding/decoding. The prose description of end-to-end training should therefore not be read as unfreezing every module. e06
Synthetic training starts from 1,700 successful RoboMimic demonstrations and adds transformer-policy rollouts. The introduction describes expansion from 0.3 million to 1.1 million transitions. Training, validation and test sets are mentioned; rollout-generating policies are excluded from later evaluation. Split sizes and detailed dataset composition are deferred to the absent appendix. e07e08
Bridge training uses Bridge V2 plus 300 locally collected trajectories to align the replicated environment. This adaptation limits claims about evaluating unseen setups without additional data. e14
During inference
Figure 1 depicts the policy and world model as frozen during evaluation. Synthetic diffusion policies use gripper/agent images from two timesteps and predict eight actions; different training durations produce different skill levels. The supplied paper does not specify the execution schedule linking action chunks to video generation. e04e08
5.5 Implementation flow
- Represent visual dynamics
The model predicts the next observation from observation and action histories. In the two-view setting, gripper and agent images are concatenated and both next views are predicted; no explicit view-consistency loss is used. e05e15
- Inject actions into denoising
Cosmos-Predict2-Video2World-2B supplies the diffusion-transformer backbone. Raw actions pass through Fourier feature mapping and an MLP; their embeddings are added to time embeddings. Figure 2 routes conditioning through scale, shift and gate modulation of self-attention, cross-attention and MLP blocks. e06
- Close the evaluation loop
The evaluated policy chooses actions, the video model generates observations, and those observations feed the next policy decision. After a rollout, a VLM receives subsampled frames and a task-completion prompt. Labels aggregate into estimated performance for comparison with actual rollouts. e04e05
6. Experiments & results
This paper turns a pretrained video diffusion model into an action-conditioned simulator for evaluating existing robot policies. A policy acts on generated observations; a vision-language model judges the resulting video to estimate success rates. Policy-rollout augmentation and pretrained weights improve agreement with simulator evaluations. Bridge experiments extend the test to physical performance, but hallucinations, inconsistent camera views and imperfect labels limit reliability. The contribution is a learned evaluation environment, not a jointly learned action-producing controller.
6.1 Read the original evidence
Table I. Synthetic evaluation tests agreement with a simulator’s policy performance. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task as a pair of columns: higher Pearson means stronger linear association, while lower MMRV means smaller performance-weighted ranking violations. Start with the full model, then compare the two immediately following ablations within the same task. On Square, removing policy rollouts changes Pearson from 0.847 to 0.505 and MMRV from 0.165 to 0.384; removing pretrained weights gives 0.523 and 0.245. The lower rows compare different backbones and adaptation choices. Section IV-A uses RoboMimic simulator dynamics as ground truth, with evaluation policies distinct from those used to generate training rollouts. These columns measure evaluator fidelity, not robot success percentages. e03e08e09e10
What it supports. The complete Cosmos-Predict2-2B configuration leads all four tasks on both reported metrics. Lift reaches Pearson 0.879 and MMRV 0.015; Tool Hang reaches 0.833 and 0.217. The ablations make a particularly large difference on Square, supporting the practical importance of both rollout data and pretrained initialization in this setup.
Where the evidence stops. No uncertainty intervals accompany these values, and cross-backbone comparisons change several factors. Figure 3 on the same page shows residual underprediction on harder tasks. Use Equations (1)–(2) for MMRV: its expansion in the Table I caption differs from Section III-A.
Table II. Video quality improves with rollout augmentation and pretrained weights. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two column groups are different evaluation distributions, not two training stages. The caption specifies autoregressive generation of 64 future frames. Within each group, PSNR and SSIM increase with reconstruction or structural similarity, whereas latent L2 and FVD decrease with their respective errors or distribution mismatch. Compare the full model against each ablation using the same column. For example, policy-rollout FVD is 250.234 for the full model, 285.001 without rollout augmentation and 291.158 without pretrained weights. The full model’s corresponding RoboMimic FVD is 98.455, showing that the easier distribution and the policy-generated distribution remain materially different. e08e11
What it supports. Both rollout augmentation and pretrained initialization improve every listed video metric relative to their ablations. However, policy-rollout prediction remains harder: the full model’s PSNR falls from 20.622 on RoboMimic to 15.996 on policy rollouts, while FVD rises from 98.455 to 250.234. This diagnostic supports, but cannot replace, the separate ranking evaluation.
Where the evidence stops. The source labels policy-rollout evaluation out-of-distribution despite training with other policy rollouts. Its split descriptions and unsupplied dataset appendix do not fully specify that boundary. These image/video metrics are not physical-success measurements or a guarantee of action-faithful dynamics.
Figure 6. Bridge evaluation compares generated-rollout scores with physical policy performance. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In panel (a), the first column is the initial observation and the remaining columns are generated rollout frames. The cup and pot rows illustrate predicted success and failure for Octo-Small; they are not photographs documenting those physical executions. Panel (b) places actual success rate horizontally and predicted success rate vertically. Marker shapes distinguish OCTO-Small, OCTO-Base and OpenVLA; colors distinguish Cosmos and IRASim. Read the numeric legend for aggregate agreement rather than estimating values from individual points. Section IV-C specifies four lifting tasks and explains that the video model was trained on Bridge V2 plus 300 trajectories from the replicated setup. e03e14
What it supports. Figure 6 reports Cosmos Pearson 0.687 and MMRV 0.171, compared with IRASim’s 0.613 and 0.611. Pearson supports positive correspondence in the adapted Bridge setup. The MMRV comparison needs an additional qualification: Equations (1)–(2) bound MMRV by the actual-score range, while the plotted rates visually span about 0.10–0.70. If these are the metric inputs, 0.611 is not readily reconciled. The main PDF does not resolve whether different aggregation or normalization was used. Preserve the printed values, but do not draw additional strength from the size of the MMRV gap without the underlying scores and protocol.
Where the evidence stops. Local training trajectories help align this setup, so the figure does not establish transfer without adaptation. The main PDF omits trial counts and uncertainty intervals and defers detailed real-world protocols to Appendix VI-A. Generated success examples must remain distinct from physical trial evidence.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Synthetic policy evaluation on RoboMimic Lift, Can, Square and Tool Hang; evaluated CNN/transformer diffusion policies differ from rollout-generating training policies; simulator dynamics provide ground truth. | Reported full model: Lift 0.879 / 0.015; Can 0.860 / 0.141; Square 0.847 / 0.165; Tool Hang 0.833 / 0.217. Pearson correlation ↑ / MMRV ↓ | Square: without policy rollouts 0.505 / 0.384; without pretrained weights 0.523 / 0.245; Wan-2.1-14B-LoRA 0.801 / 0.179; IRASim 0.213 / 0.708. The full model leads all Table I task/metric columns. Figure 3 still shows underprediction on harder tasks. These are evaluation-agreement scores, not executed task success rates; uncertainty intervals are not provided. e08e09e10 |
| Autoregressive action-conditioned video prediction 64 future frames; RoboMimic in-distribution trajectories versus policy-rollout trajectories labeled out-of-distribution by the source. | Reported full model: in-distribution 20.622 / 0.868 / 0.315 / 98.455; out-of-distribution 15.996 / 0.793 / 0.458 / 250.234. PSNR ↑ / SSIM ↑ / latent L2 ↓ / FVD ↓ | Out-of-distribution FVD is 285.001 without rollouts and 291.158 without pretrained weights. Both interventions improve video metrics, but the supplied split description cannot reconstruct exactly what is held out. Video similarity alone does not validate policy ranking. e08e11 |
| Real-world policy evaluation in the Bridge setup OCTO-Small, OCTO-Base and OpenVLA; lifting a pot, carrot, eggplant or cup; Bridge V2 and 300 local trajectories train the video model. | Reported Cosmos: Pearson 0.687; MMRV 0.171. Pearson correlation ↑ / MMRV ↓ | Reported IRASim: Pearson 0.613; MMRV 0.611. Figure 6 reports positive association with physical performance in this adapted setup. Its MMRV values are retained as printed. Under Equations (1)–(2), MMRV cannot exceed the range of the actual scores used. If these inputs are the plotted rates, visually about 0.10–0.70, IRASim’s 0.611 is not readily reconciled with that bound. Raw scores and sufficient aggregation or normalization details are absent, so the size of this MMRV difference is not independently reconstructed evidence. Trial counts and confidence intervals are also missing. e14e03 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Adding policy behavior broadens training beyond successful demonstrations. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) from left to right in time, comparing the row without policy rollouts against the row with them. The source uses disappearing gripper geometry to illustrate degradation when the model encounters behavior outside demonstration coverage. In panel (b), blue denotes Lift and orange denotes Square. Both horizontal axes count thousands of policy-rollout transitions, not complete trajectories. Improvement points downward for MMRV and upward for Pearson. Square’s Pearson curve changes more strongly than Lift’s, while both tasks show an overall reduction in MMRV across the displayed range. The source links this experiment to training on policies with different manipulation performance. e07e08e12
What it supports. The overall trends support adding rollout data: a model exposed to a broader range of policy behavior gives more useful evaluation signals. Square benefits strongly in Pearson correlation, consistent with its harder grasping and alignment requirements. The examples also connect evaluation quality to a concrete failure mechanism: the simulated gripper itself can become unreliable.
Where the evidence stops. The caption says performance improves consistently, but the plotted MMRV curves contain small intermediate increases; they are not strictly monotonic. No error bars are shown. Adding rollouts changes both behavioral diversity and transition count, so this comparison does not isolate diversity alone.
Figure 8. Diagnostic examples show how generated dynamics can bias policy scores. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) contains two separate temporal examples, indicated by horizontal arrows. In the synthetic scene, compare the square object’s protruding geometry before and after generation; the caption identifies the disappearance as hallucination. In the Bridge example, the source identifies an eggplant appearing in the gripper as an implausible grasp that could inflate estimated success. Panel (b) is a different comparison: its images are labeled Agent View and Gripper View, not successive timesteps. The caption reports that the can appears grasped in one view but is missing in the other. Section IV-D explains that concatenating two views does not itself enforce consistency. e15
What it supports. Errors can bias evaluation in opposite directions. Removing task-relevant geometry can induce failure, whereas hallucinating an object into the gripper can create apparent success. The policy also receives incompatible visual evidence when the cameras disagree. These examples explain why a plausible-looking rollout is insufficient evidence of a reliable evaluation environment.
Where the evidence stops. This is a qualitative diagnostic, not an ablation with measured failure frequencies. Section IV-D additionally reports replicated frames during small motions and compounding errors over long rollouts; the figure does not quantify how much each mechanism contributes to ranking error.
7. Analysis & limitations
7.1 What the evidence leaves open
Hallucinations can remove graspable geometry or place an eggplant into the gripper, biasing success downward or upward. View disagreement and repeated near-static frames further break interaction. Errors compound autoregressively, especially in Tool Hang rollouts of 700 transitions. e15
VLM annotation accuracy is described as often only 65–80%; detailed analysis is deferred to Appendix VI-A.5. World-model error and judge error are therefore distinct unresolved contributors to evaluation reliability. e16
Rollout augmentation changes both data volume and behavioral coverage, so it does not isolate diversity alone. Figure 5’s overall improvement is not strictly monotonic in MMRV. Backbone comparisons also vary size, initialization and adaptation, weakening attribution solely to pretraining-domain alignment. e08e09e12e13
7.2 Questions for discussion
- Would rollout diversity still improve ranking after transition count and optimization budget are matched?
- How much remaining ranking error comes from generated dynamics versus VLM task-completion judgments?
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include the Cosmos checkpoint, action encoding, EDM training, separated rollout/evaluation policies, RoboMimic splits, initial frames and VLM judge. Architecture settings, optimizer schedule, compute, split counts, sampling configuration and judging prompts are absent or deferred to unsupplied appendices. The paper plans a pipeline release; this does not establish available artifacts. e06e08e17e04
Reader-proposed checks: match transition counts and training updates when comparing demonstrations with mixed policy rollouts; separately compare VLM and blinded human labels on identical generated videos. Measure Pearson/MMRV against fixed simulator evaluations to distinguish behavioral coverage gains from annotation artifacts. e03e12e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does rollout diversity help at a fixed data budget?
Reader-proposed experiment, not performed: on Lift and Square, initialize both world models from the same Cosmos checkpoint. Train one on demonstrations only and the other on a mixture containing policy rollouts, matching the number of transition presentations, optimizer updates and other settings. Keep rollout-generating policies disjoint from evaluation policies and use identical evaluation initial states and judge settings. Repeat across seeds and report Pearson, MMRV and uncertainty against fixed simulator evaluations. A persistent mixed-data advantage, particularly on Square, would support behavioral coverage as a mechanism; disappearance of the advantage after budget matching would weaken that interpretation. The absent appendix prevents claiming an exact original-protocol reproduction. e03e06e08e12e17
Check 2: How much ranking error comes from the VLM judge?
Reader-proposed experiment, not performed: freeze generated rollouts for a held-out synthetic policy set and preregister explicit task-success criteria. Give the same sampled frames to a fixed VLM and blinded human annotators, then recompute policy success estimates and Pearson/MMRV against simulator evaluations with matched starting states. Adjudicate ambiguous cases using the complete generated sequence as a separate diagnostic. If human relabeling substantially improves ranking while videos stay fixed, annotation error is consequential. If ranking remains poor despite strong label agreement, inspect hallucinated grasps, missing objects and view conflicts as dynamics errors. Record prompts and sampling choices explicitly because the source defers those details to an unsupplied supplement. e03e04e08e15e16e17
8.3 Reading coverage
Visual audit: Visually inspected every supplied PDF page: title, author credits and version on p. 1; metric equations and supplement disclosure on p. 2; action-conditioning architecture and training text on p. 3; synthetic tasks, policy separation, Table I and Figure 3 on p. 4; policy inputs, Table II and rollout ablation on p. 5; Bridge results, local data and pretraining illustration on p. 6; failure cases, VLM limitations and conclusion on p. 7; references on pp. 7–8. All six final original crops were separately viewed and checked for legibility, complete labels and faithful bounds. Figure 2’s addition and modulation arrows were cross-checked against its caption and III-B; its frozen-module legend qualifies the prose’s end-to-end wording. Figure 5’s MMRV curves are not strictly monotonic despite the caption. No external appendix, supplementary images or code were supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not read.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, version stamp and abstract
- PDF pp. 1–2: I. Introduction
- PDF p. 2: II. Related Work
- PDF pp. 2–3: III-A. Evaluating Robot Policies
- PDF p. 3: III-B. Learning Action-conditional Video Prediction Models
- PDF pp. 3–4: III-C. Policy Rollout Data Augmentation
- PDF pp. 4–5: IV-A. Synthetic Setting
- PDF pp. 5–6: IV-B. Ablation Study, all three subsections
- PDF p. 6: IV-C. Policy Evaluation in Real-World Setup
- PDF pp. 6–7: IV-D. Failure Case Analysis
- PDF p. 7: V. Conclusion
- PDF pp. 7–8: References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks were read individually in full. All eight PDF pages were rendered and visually inspected, resolving the extraction-only image omission for this PDF.
- The eight-page PDF contains the complete main paper and references, but no appendix. The project-site supplement, including cited Appendices VI-A, VI-A.2, VI-A.5, VII and VIII, was not supplied or read. Supplement availability remains unverified.
- Identity/version: the title and identifier match the catalog. The PDF identifies arXiv:2511.11520v3, 4 December 2025. All seven author identities agree; the PDF prints Yen-Chen Lin where the catalog lists Lin Yen-Chen. The catalog submission date is 14 November 2025. This report reviews the supplied v3 artifact; earlier versions and their scientific differences were not supplied or compared.
- Code and external project resources were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, affiliation footnote and arXiv margin stamp
Title and seven authors are verified, ending with Yen-Chen Lin. Affiliations: Nvidia Research, University of Toronto, Vector Institute. Stamp: arXiv:2511.11520v3 [cs.RO], 4 Dec 2025.
Go to primary source ↓e02PDF p. 1, Abstract, I. Introduction and Figure 1 caption
Motivates cheaper policy evaluation through action-conditioned video rollouts, VLM success judgments and correlation with actual policy performance.
Go to primary source ↓e03PDF p. 2, III-A, Equations (1)–(2); PDF p. 3, III-A opening paragraph
Defines actual and simulator policy performances, pairwise rank violation, Mean Maximum Rank Violation and Pearson correlation as evaluation metrics.
Go to primary source ↓e04PDF p. 1, Figure 1(a–c); PDF p. 3, III-A, paragraph beginning Once we have a world model
Frozen policy/world-model symbols accompany the evaluation loop; rollout frames are subsampled for a VLM with a task-completion prompt whose details are deferred to the supplement.
Go to primary source ↓e05PDF p. 3, III-B, first three paragraphs
Actions are end-effector poses in world coordinates; the policy consumes observations; learned dynamics condition on observation/action histories; generated observations are reused autoregressively.
Go to primary source ↓e06PDF p. 3, Figure 2, caption and legend; III-B, Video Model Architectures and Implementation
Cosmos-Predict2-Video2World-2B uses Fourier features and an action MLP added to time embeddings. Figure 2 routes scale/shift/gate conditioning, identifies pretrained/new modules and freezes encoding/decoding. Text specifies EDM and end-to-end training, referring details to Appendix VII.
Go to primary source ↓e07PDF p. 2, I. Introduction first paragraph; PDF p. 3, III-C
The introduction describes 0.3 million to 1.1 million transitions; III-C motivates policy rollouts because teleoperated demonstrations rarely contain failure.
Go to primary source ↓e08PDF p. 4, IV-A, tasks, dataset paragraphs and footnote 1; PDF p. 5, IV-A, policy evaluation paragraph
Four RoboMimic tasks use 1,700 demonstrations plus transformer-policy rollouts and training/validation/test splits. Rollout-generating policies are not reused in evaluation. Evaluated diffusion policies use two views at two timesteps, predict eight actions and vary in training duration.
Go to primary source ↓e09PDF p. 4, Table I, all rows and task columns
Full-model Pearson/MMRV: Lift 0.879/0.015, Can 0.860/0.141, Square 0.847/0.165, Tool Hang 0.833/0.217. Square ablations: no rollouts 0.505/0.384, no pretraining 0.523/0.245. Wan LoRA: 0.801/0.179; IRASim: 0.213/0.708. The caption expands MMRV differently from III-A.
Go to primary source ↓e10PDF p. 4, Figure 3, axes, legend and Square/Tool Hang panels; PDF p. 5, IV-A concluding paragraph
Predicted versus actual success plots distinguish policy families, models and the oracle diagonal. Full-model trends on Square and Tool Hang fall below that diagonal; text describes weaker correlation on longer tasks.
Go to primary source ↓e11PDF p. 5, Table II, caption and all rows; IV-A, video-metric paragraph
64-frame autoregressive PSNR/SSIM/latent-L2/FVD: full model 20.622/0.868/0.315/98.455 on RoboMimic and 15.996/0.793/0.458/250.234 on policy rollouts. OOD FVD ablations are 285.001 and 291.158.
Go to primary source ↓e12PDF p. 5, Figure 5(a–b), axes, legend and caption; IV-B.1
Rollouts reduce illustrated gripper artifacts and improve overall Pearson/MMRV trends for Lift and Square. The caption says consistently enhances, but plotted MMRV has small intermediate increases. Horizontal axes count thousands of policy-rollout transitions.
Go to primary source ↓e13PDF pp. 5–6, IV-B.2–3; PDF p. 6, Figure 7; PDF p. 4, Table I alternative-backbone rows
Pretraining is compared with random initialization; Figure 7 illustrates reduced object disappearance. Alternatives include Cosmos-Predict1, Wan2.1 with LoRA and IRASim. Benefits from physical-AI pretraining are hypothesized; model-specific training details are deferred.
Go to primary source ↓e14PDF p. 6, IV-C and Figure 6(a–b), including correlation legend
Bridge evaluation covers OCTO-Small, OCTO-Base and OpenVLA on four lifting tasks; training includes Bridge V2 and 300 local trajectories. Figure 6 shows generated cup success/pot failure; Cosmos MMRV/Pearson is 0.171/0.687 versus IRASim 0.611/0.613.
Go to primary source ↓e15PDF pp. 6–7, IV-D, Hallucinations, Multi-View Inconsistency and Replicated Prediction; PDF p. 7, Figure 8 and caption
Describes hallucination, inconsistent concatenated camera views, absent view-consistency loss, replicated frames under small displacement and compounding error over 700 Tool Hang transitions. Figure 8 shows missing square geometry, a hallucinated eggplant grasp and inconsistent can visibility.
Go to primary source ↓e16PDF p. 7, IV-D, VLM Annotation Errors
VLM annotations often achieve only 65–80% accuracy; detailed analysis is assigned to Appendix VI-A.5.
Go to primary source ↓e17PDF p. 2, final contribution bullet and supplement paragraph; PDF pp. 3–4 and 6–7, appendix referrals
Plans release of the evaluation pipeline, initial frames, action-conditioned model and task-completion VLM; points to external qualitative results and appendices for architecture, data and experimental details.
Go to primary source ↓8.5 Primary sources
Scalable Policy Evaluation with Video World Models ↗
PDF · 6,406 extracted words
Source fingerprint
dc1b311d5cfd1bf340b78c3a7a4dea25597d357d8584a01b05d85d1e9f11d991