WorldEval: World Model as Real-World Robot Policies Evaluator
1. Paper overview
In one sentence: WorldEval turns policy-internal action embeddings into videos whose judged outcomes can rank robot policies, but simulator familiarity and imperfect action fidelity constrain that proxy. goalarchitectureverifiersetuppairedsimulationencodingcheckpoints
| At a glance | What to know |
|---|---|
| Research problem | Source description Real-robot evaluation becomes costly as policies, checkpoints and tasks multiply. WorldEval asks whether a video simulator can preserve their relative performance. Its stated target is a useful ranking proxy, without requiring exact trajectory replication or replacing physical evaluation. goal |
| Core mechanism | Source description Policy2Vec reuses policy-internal action representations to condition video generation, avoiding a separately trained external action encoder. architecture |
| A key reported result | Policy ranking versus real-to-sim on three RoboTwin tasks: WorldEval averages: MMRV 0.044; r 0.942. Mean Maximum Rank Violation (lower better); Pearson r (higher better). Real-trained policies; Place Cup, Strike Block and Handover Block; paired real/proxy evaluation. Real-to-sim averages: MMRV 0.261; r 0.411. Table 1 favors WorldEval under this comparator. It is not a direct SIMPLER benchmark: Appendix C transforms simulated observations using visual models and MidJourney. simulationmetricssetup |
| Reading caution | Source description The authors acknowledge hallucinations, object deformation, arm ghosting and incomplete latent-action fidelity, especially for weak policies or novel scenes. Appendix A's action-only description conflicts with Section 3.2's explicit language/image conditioning. limitations |
Core contributions
- Source description
Policy2Vec reuses policy-internal action representations to condition video generation, avoiding a separately trained external action encoder. architecture
- Source description
An automated video-verification pipeline is compared with paired real trials and a RoboTwin-based real-to-sim evaluator. verifierpairedsimulation
- Source description
Checkpoint, distribution-shift, encoding and frequency diagnostics probe when generated behavior reflects policy differences. encodingcheckpointsshiftfrequency
Figure 2. Policy2Vec supplies the bridge between a robot policy and a separately adapted video simulator. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left with the policy's vision, language and action-decoder components. The checkered strip is a latent action representation, which passes upward through the projection and then into the addition node. The other input comes from the text branch labeled umT5; the initial image follows a separate encoder route. Section 3.2 adds a detail absent from the schematic: a learnable alpha scales the projected action representation. The projection and LoRA blocks identify the adaptation sites. In the lower panel, different policy latents produce different videos, which then flow to a success detector and an illustrative success-rate chart. architectureverifierlimitations
What it supports. The mechanism makes a policy's internal representation observable through generated behavior. The training description caches those representations before adapting the simulator, so this figure does not establish end-to-end joint learning of policy actions and future frames. Its rightmost bars illustrate the evaluation flow rather than reporting a benchmark result.
Where the evidence stops. Appendix A says control exclusively uses action outputs and proposes adding language, despite the explicit text and image paths here and in Section 3.2. The latter supplies the verifiable mechanism; the discrepancy remains unresolved.
2. Motivation
2.1 The problem and the proposed response
Real-robot evaluation becomes costly as policies, checkpoints and tasks multiply. WorldEval asks whether a video simulator can preserve their relative performance. Its stated target is a useful ranking proxy, without requiring exact trajectory replication or replacing physical evaluation. goal
2.2 What this reading follows
A robot policy can look competent in a demonstration and still fail when its objects, checkpoint or environment changes. WorldEval asks whether a learned video simulator can help compare such policies before repeated physical testing. Its key interface is Policy2Vec: internal policy features condition a pretrained image-to-video model, and a separate video judge estimates success. Read the evidence in two layers. The paired success plots and simulation comparison test relative ranking; the embedding and checkpoint diagnostics examine what the simulator may actually be responding to. The reported correlations are promising for the tested bimanual tasks, while source inconsistencies and unmeasured verifier errors leave stronger behavioral claims open. goalarchitectureverifiersetuppairedsimulationencodingcheckpoints
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is entirely unassigned. Architecture evidence shows a robot policy, a separate video simulator and a success verifier. This is an action-conditioned evaluation system, without joint prediction of executable actions and futures or inverse-dynamics action extraction. It therefore does not establish a One Model world-action policy classification; an evaluation-focused placement is a reader suggestion. architecturegoalverifier
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the policy, the simulator and the judge
A useful way to follow WorldEval is to ask which component supplies each piece of information. The existing robot policy processes the observation and instruction and supplies an internal action representation. Policy2Vec projects that representation into a condition that can be fused with text for WAN 2.1. The video model then supplies predicted visual outcomes, and Gemini-2.0 supplies the task-success decision. During simulator training, cached latents allow the policy network to be absent from optimization; projection training and LoRA adaptation connect the representations to videos. At evaluation, the policy is needed to provide the candidate embedding, but the reported pipeline does not specify a recurring generated-frame feedback loop. Generated robot motion is the simulator's prediction, and the binary label is the judge's interpretation of that prediction. architectureverifiertraining
5.2 Evaluate the evaluator using paired performance
Reader analysis: success in a generated clip becomes useful for model selection only when it predicts how policies compare physically. Figure 4 therefore compares real and proxy success for each policy within a task. Pearson correlation summarizes association, while MMRV asks whether the evaluator reverses policy ordering. Neither alone proves that a generated trajectory follows the policy's actual motor commands. Table 1 adds a practical comparison against the paper's real-to-sim implementation, with better displayed metrics for WorldEval. However, the comparator includes observation transformation, and the Strike Block MMRV differs between the table and plot. A careful reading keeps those source values separate and treats the paired evidence as support for relative evaluation in this setup, without turning it into a universal simulator ranking or a guarantee of calibrated success. goalpairedmetricssimulationlimitations
5.3 Ask what a poor generated video is diagnosing
Reader analysis: a simulator trained using final-checkpoint embeddings can react poorly to an earlier checkpoint because the policy is weak, because its latent distribution is unfamiliar, or both. Appendix B makes this ambiguity concrete, and Table 3 shows that FID changes do not always bring success improvements. Table 2 supports the usefulness of Policy2Vec relative to two encodings, but it does not fully separate semantic action information from feature-distribution effects. The same issue matters for the safety claim: mosaic video accompanies one hazardous behavior, yet a broken video is not a validated hazard label. Finally, Figure 10's reversed row descriptions caution against reading temporal fidelity directly from selected stills. Independent action measurements and separately audited success labels would help distinguish policy failure, simulator failure and verifier failure. checkpointsencodingsafetylimitationsfrequencyverifier
5.4 Training and inference
During training
Cache policy latents with observations and instructions; the policy network is unnecessary during simulator fine-tuning. Train the projection and lightly adapt DiT linear layers with LoRA. The source gives no explicit optimization-loss equation or complete module-freezing list. architecture
The simulator uses 1,400 real trajectories, sampled three times into 81-frame, 480×480 clips; learning rate 1e-4, batch size 8, LoRA alpha/rank 16 and 30 epochs. Appendix C reports eight H800 GPUs and approximately 11 hours. training
For policy fine-tuning, the paper lists 100 Bussing Table trajectories and 50 each for Place Cup, Handover Block and Strike Block. Their overlap with simulator-training data is unspecified. Collect Toy is described as unseen. Appendix B says final-checkpoint latents train the video model, an important conditioning-distribution constraint. setuptrainingcheckpoints
During inference
Extract a candidate policy's latent, synthesize a video using 50 diffusion steps, and obtain a task-success judgment. The paper does not document an iterative generated-frame feedback loop, denoising guidance settings, or video-generation latency. architectureverifiertraining
Evaluate on an AgileX bimanual robot with a top RealSense 457 camera and 14-dimensional state/action spaces. The source reports 40 rollouts per task and over 1,000 real trials, but does not fully itemize all policy/checkpoint/task trial allocations. setup
5.5 Implementation flow
- Extract policy representations
Read latent features before the action decoder emits its final action sequence. The same observation and instruction can produce different embeddings for different policies. The paper does not specify exact extraction layers or dimensions for every evaluated architecture. architecture
- Condition the video simulator
Project the embedding to the required dimension, multiply by learnable alpha, and add it to the language embedding. Feed this fused condition and the initial image into WAN 2.1's DiT-based image-to-video model. Figure 2 shows separate image and language pathways. architecture
- Judge generated outcomes
Ask Gemini-2.0 a task-specific binary completion question about the generated video, then compare aggregate success with real evaluation. This is predicted visual behavior followed by classification; the pipeline does not decode generated frames into robot commands. verifiergoalarchitecture
6. Experiments & results
WorldEval estimates robot-policy rankings by turning internal policy embeddings into generated manipulation videos and judging their outcomes. Policy2Vec conditions a separately adapted WAN 2.1 simulator; Gemini-2.0 supplies success labels. Paired experiments support relative-ranking usefulness on the tested tabletop setup, while imperfect action fidelity, checkpoint-distribution dependence and inconsistent source reporting limit stronger conclusions.
6.1 Read the original evidence
Figure 4. Paired success rates show strong association, with a visible policy-order reversal in Collect Toy. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each panel fixes a task and plots four policies: green squares denote pi0, pink diamonds DexVLA, orange stars Diffusion Policy and blue crosses OpenVLA. Read the horizontal axis as real success and the vertical axis as generated-video success. The sloped gray lines do not trace equal success rates; the caption and text do not specify their construction. Compare each point's horizontal and vertical coordinates to assess absolute agreement. The headers report Pearson r and MMRV, whose preferred directions are up and down respectively. Compare policy ordering within a panel before comparing tasks. In Collect Toy, pi0 lies to the right of DexVLA but below it, showing a reversal between real and generated rankings. pairedmetricssimulation
What it supports. The printed correlations span 0.885 for Collect Toy to 0.980 for Handover Block. These paired observations support a relationship between real and generated performance for the evaluated policies. Reader interpretation: high correlation alone does not establish calibrated success probabilities, and the Collect Toy reversal demonstrates that strong association can coexist with ranking errors.
Where the evidence stops. Do not interpret distance from the gray lines as calibration error. Strike Block prints MMRV=0.000 here but 0.133 in Table 1; the paper does not reconcile them. Four policy points per panel and absent uncertainty bars limit robustness conclusions.
Table 1. WorldEval gives better ranking metrics than the paper's RoboTwin-based comparator. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this table in two horizontal blocks. The upper block measures rank violations with MMRV, so smaller values favor the evaluator. The lower block measures linear association with real success using Pearson r, so larger values favor it. Within each block, compare WorldEval with Real-to-Sim for a single task, then inspect the Avg. column. Keep the evaluation setting attached: the policies were trained on real data and evaluated in RoboTwin with a matching embodiment. Appendix C says simulation observations underwent sim-to-real visual transformation, including MidJourney, before being passed to the policy. This is a specific implemented comparator. simulationmetricspaired
What it supports. Across the three tasks, Table 1 reports average MMRV 0.044 for WorldEval versus 0.261 for real-to-sim, and average Pearson r 0.942 versus 0.411. WorldEval is favored on both metrics in every displayed task. These are evaluator-quality metrics, not robot success-rate improvements.
Where the evidence stops. A direct SIMPLER comparison was infeasible. The visual-transformation comparator and its incompletely specified settings limit generalization to other simulators. Table 1's Strike Block MMRV also conflicts with Figure 4, so its printed averages remain table-specific.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Policy ranking versus real-to-sim on three RoboTwin tasks Real-trained policies; Place Cup, Strike Block and Handover Block; paired real/proxy evaluation. | WorldEval averages: MMRV 0.044; r 0.942. Mean Maximum Rank Violation (lower better); Pearson r (higher better) | Real-to-sim averages: MMRV 0.261; r 0.411. Table 1 favors WorldEval under this comparator. It is not a direct SIMPLER benchmark: Appendix C transforms simulated observations using visual models and MidJourney. simulationmetricssetup |
| Cross-policy correlation across five tasks Figure 4: pi0, DexVLA, Diffusion Policy and OpenVLA, real versus generated success. | Bussing Table 0.935; Collect Toy 0.885; Place Cup 0.958; Handover Block 0.980; Strike Block 0.887. Pearson r | Each panel compares four policies against their real success; no uncertainty intervals are shown. Correlation supports relative comparisons but does not imply calibrated success probabilities. Strike Block MMRV is inconsistent: Figure 4 prints 0.000, Table 1 prints 0.133. pairedsimulationmetrics |
| Action-encoding ablation Table 2, reported across five tasks. | Policy2Vec: 0.939 / 0.192 / 61.33. Pearson r / MMRV / FID | VQVAE: -0.862 / 0.292 / 71.79; one-hot: -0.333 / 0.416 / 75.91. All three metrics favor Policy2Vec. Reader arithmetic gives MMRV reductions 0.100 and 0.224; Section 4.3 prints 2.24 for the latter. This table's aggregation is not reconciled with Figure 4. encodingpaired |
| Checkpoint diagnostic for brown-bowl placement Table 3, pi0 Task A within Bussing Table; final-checkpoint latents train the simulator. | 5k: 81.62 and 0.725; final 60k: 74.65 and 0.925. FID and reported Succ. fraction | Diffusion Policy has zero success at all listed checkpoints and nonmonotonic Task A FID. Selected improvements do not establish a general scaling law. Table 3 has no separate real/proxy success columns or uncertainty estimates. checkpoints |
| Evaluation under unseen backgrounds Appendix B: office, living room and kitchen; cup placement, striking and handover. | 0.047; 0.927. MMRV; Pearson r | Laboratory MMRV 0.044. Reported correlation remains high. The prose calls 0.047 lower than 0.044; numerically it is slightly higher and worse under the stated metric. Per-environment breakdowns are absent. shiftmetrics |
| FID as a lightweight performance proxy Figure 7, cross-policy comparisons within each task. | Place Cup 0.999; Handover Block 0.993; Bussing Table 0.495. Absolute Pearson correlation |r| between FID and real success | Collect Toy 0.760; Strike Block 0.805. The plotted relationship is negative: higher real success generally accompanies lower FID. FID is less informative for complex table bussing and does not itself certify task completion. fid |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Policy2Vec improves both video quality and policy-ranking metrics in the reported encoding comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each metric column to compare the encoding choices while keeping their distinct roles in mind. Pearson r evaluates association with real performance; MMRV evaluates ranking violations; FID evaluates generated-image distribution quality. The preferred arrows are printed in the headers. VQVAE quantizes continuous actions, while the one-hot baseline uses threshold-based categorization according to Section 4.3. Policy2Vec instead uses policy-internal representations. The caption describes results across five tasks, but the table does not expose task-level measurements or a complete aggregation procedure. Its final Rank column summarizes the comparison and should not replace the individual metrics. encodingmetrics
What it supports. Policy2Vec reports r=0.939, MMRV=0.192 and FID=61.33, compared with negative correlations and higher MMRV/FID for both alternatives. Thus the displayed advantage is not confined to image quality. The ablation supports this representation choice within the reported setup, without isolating every possible cause of the difference.
Where the evidence stops. Section 4.3 prints a 2.24 MMRV reduction against one-hot; the table implies 0.416−0.192=0.224. This edition retains the original cells and discloses the arithmetic conflict. Training-budget and representation-capacity matching are not fully documented.
Table 3. Checkpoint trends expose both the promise and the distribution dependence of the evaluator. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the caption's task mapping: A is brown-bowl placement, B green-plate placement and C brown-mug placement. Each task has an FID column and a reported success fraction, Succ. Compare checkpoints within a policy before comparing different architectures, whose listed training steps differ. Appendix B states that final-checkpoint latent actions were used to fine-tune the video model. Earlier checkpoints therefore test representations that may differ from its training distribution. The source discusses real-policy performance but supplies only one success column per task here, so this table cannot independently display paired real-versus-generated agreement at every checkpoint. checkpoints
What it supports. For pi0 on Task A, 5k to final 60k changes FID from 81.62 to 74.65 and success from 0.725 to 0.925. The pattern is not universal: Diffusion Policy stays at zero success, while Task A FID falls and then rises. The table supports selected checkpoint trends, not a general scaling law.
Where the evidence stops. Video quality may track familiarity with final-checkpoint embeddings as well as policy competence. No independent latent-distribution control separates those effects, and the table provides no uncertainty estimates or paired real/proxy success columns.
Figure 10. A frequency diagnostic is informative only if row identity and temporal alignment are preserved. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the labels printed on the graphic: the upper row is 50 Hz and the lower row is 10 Hz. Within each half, the columns show the twentieth, thirtieth and fortieth frames; the left half is real robot imagery and the right half generated imagery. Appendix D describes two policies trained separately on datasets with equal trajectory counts and evaluated using one WorldEval model. It then reverses the row order in its prose. Follow the visible row labels when locating examples, but keep the inconsistency explicit. Frame indices identify selected images; they do not establish matched elapsed time across collection rates. frequency
What it supports. The figure documents a qualitative comparison involving policies trained at different data-collection frequencies. The authors interpret the 50 Hz condition as producing smoother, less segmented generated behavior. These stills show selected scene configurations and support inspecting the comparison, but they do not themselves measure smoothness or temporal fidelity.
Where the evidence stops. Appendix D says the first row is 10 Hz and the second 50 Hz, opposite the graphic. No timestamps, smoothness metric or frame-count/duration controls resolve this diagnostic, and data-collection frequency is not generation latency.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge hallucinations, object deformation, arm ghosting and incomplete latent-action fidelity, especially for weak policies or novel scenes. Appendix A's action-only description conflicts with Section 3.2's explicit language/image conditioning. limitations
The unsafe DexVLA example produces mosaic video, but no threshold, recall or false-alarm analysis establishes a dependable safety detector. Generation failure can reflect simulator distribution shift as well as hazardous behavior. safetylimitationscheckpoints
Figure 10 labels the upper/lower rows 50/10 Hz; Appendix D reverses them. Static frames cannot verify the claimed temporal smoothness. Equal trajectory counts also leave duration and frame-count matching unspecified. frequency
Final-checkpoint latent training can favor familiar embedding distributions. High correlations over four policies, without uncertainty bars or independent verifier-accuracy measurements, leave ranking robustness and error attribution unresolved. checkpointspairedverifier
7.2 Questions for discussion
- Would rankings survive simulator training on disjoint policy checkpoints?
- How much ranking error originates in generation versus the success verifier?
- Can action fidelity be measured independently of task completion and image quality?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires policy checkpoints and exact latent-extraction hooks, aligned videos, the adapted WAN model, and fixed verifier prompts. The source omits a full split manifest, per-policy feature dimensions, detailed FID computation, complete success rubrics and seeds; these must remain unresolved. architecturetrainingverifierfidsetup
Reader-proposed checks should hold observations, prompts, training data and generation budgets fixed when perturbing policy latents, and independently audit generated-video judgments against human labels. Checkpoint-held-out evaluation would test whether ranking survives unfamiliar latent distributions. architectureencodingcheckpointsverifier
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Hold out checkpoints and perturb only their latent condition
Reader-proposed experiment, not performed: choose Place Cup with identical initial observations and instructions across checkpoints. Compare a simulator adapted only on final-checkpoint latents with one adapted on a disjoint checkpoint mixture, holding training clips, updates and generation seeds fixed. For each, evaluate correct embeddings, within-policy shuffled embeddings and a zeroed projected-action condition. Compare rankings with paired physical success using Pearson r and MMRV, and annotate whether cup motion follows the tested policy. If rankings are unchanged by shuffling or depend mainly on which checkpoints trained the simulator, the action-semantic interpretation is weakened. Report uncertainty over scenes and seeds, not only a single correlation. architectureencodingcheckpointssetupmetrics
Check 2: Separate video-prediction error from verifier error
Reader-proposed experiment, not performed: build a held-out, task-balanced set of paired real and generated clips spanning completion, ordinary failure and known abnormal behavior. Have blinded human annotators apply fixed task-success rubrics to each clip, then compare the fixed Gemini-2.0 prompt's labels with those annotations. Compute confusion matrices separately for real and generated videos, and recompute policy rankings using human versus automated labels. Also count mosaic/artifact flags in safe and unsafe cases, including safe but visually unfamiliar scenes. If human-labeled generated videos rank policies correctly but Gemini labels do not, verifier error dominates; disagreement already present in human real/generated labels implicates simulation. Report unsafe-case recall and false alarms separately. verifierpairedqualitativesafetylimitationsshift
8.3 Reading coverage
Visual audit: Read all six supplied text chunks, including references and Appendices A–D. Visually inspected the title/version/author block, introductory motivation on p. 2, Figures 1–10, Tables 1–3 and all pages supporting retained method, training, evaluation, limitation and reproduction details, including hardware and comparator configuration on p. 19. All six final original PDF crops were regenerated at their recorded bounds and visually inspected. Figure 2 pathways were checked against Sections 3.2–3.3. Figure 4's sloped lines were checked against the axes and caption: they are not equality references, and their construction is unspecified. The language-conditioning conflict, Strike Block MMRV discrepancy, ablation arithmetic, background-shift wording and Figure 10 row reversal remain explicit. Supplemental videos were not supplied for viewing, so motion descriptions remain author interpretations of static PDF evidence. Bibliography-only pages and introductory/related-work text on p. 3 were read as text; no independent external references, code or experiments were inspected.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 9, 10, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2 (Introduction and Related Work)
- Sections 3.1–3.3 (formulation, Policy2Vec, verifier)
- Sections 4.1–4.3 (paired evaluation, real-to-sim, ablations)
- Section 5 and References
- Appendices A–D, including all tables and figure captions
Outside the original text pass
- Reviewed arXiv:2505.19017v1 dated 25 May 2025, marked Preprint, Under review. Title and all five authors match the catalog; no revision or edition difference was observed. Other versions were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by inspecting the supplied PDF: all ten figures, all three tables and supporting method/result/appendix pages were visually reviewed.
- Separate supplemental material availability has not been fully verified.
- Supplemental videos, project website and code were not inspected; no experiments were reproduced. References were read as bibliography, not independently verified sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author block, affiliations, arXiv margin and footnotes
The title and five authors match the catalog. The artifact identifies arXiv:2505.19017v1, 25 May 2025, and is marked Preprint, Under review. Affiliations are Midea Group and East China Normal University; Yaxuan Li and Yichen Zhu share equal contribution.
Go to primary source ↓goalPDF p. 2, Section 1 opening motivation; p. 4, Section 3.1
The introduction motivates an evaluator by the difficulty of physical testing across tasks and checkpoints. The objective is correlation of relative policy performance between real evaluations and a world simulator, explicitly without requiring one-to-one behavioral replication or replacing real evaluations.
Go to primary source ↓architecturePDF p. 2, Section 1 Policy2Vec rationale; pp. 4–5, Figure 2 and Section 3.2
Policy2Vec extracts internal action embeddings, projects them, scales them by learnable alpha and adds them to language embeddings conditioning WAN 2.1. Initial images also condition generation. Cached policy embeddings permit simulator fine-tuning without the policy network; projection and DiT linear layers are adapted using LoRA on a 14B image-to-video model.
Go to primary source ↓verifierPDF p. 5, Section 3.3
The classifier C_T approximates the binary task-success function T over image states S. Gemini-2.0 judges generated videos using task questions; an apple-on-plate yes/no question is illustrative.
Go to primary source ↓setupPDF p. 6, Figure 3 and robot/task setup; p. 20, Appendix C task descriptions
An AgileX ALOHA-style bimanual robot uses two 6-DoF arms, a top RealSense 457 camera, 14-dimensional state/action spaces and 50 Hz teleoperation. Bussing Table uses 100 policy-training trajectories; Place Cup, Handover Block and Strike Block use 50 each. Collect Toy is described as unseen. The paper reports 40 rollouts per task and over 1,000 real-world trials. Appendix C describes randomized object positions and task instructions.
Go to primary source ↓trainingPDF p. 6, implementation details; pp. 18–19, Appendix C implementation details
WAN 2.1 14B is adapted with 1,400 trajectories, three samples per trajectory, 81-frame 480×480 clips, learning rate 1e-4, batch size 8, LoRA alpha/rank 16 and 30 epochs. Training uses eight H800 GPUs with data parallelism for approximately 11 hours; video generation uses 50 diffusion steps.
Go to primary source ↓metricsPDF p. 7, evaluation-metric bullets
Pearson r measures linear association of real and proxy performance; higher is better. MMRV measures severity of pairwise ranking violations; lower is better. Its detailed definition is referred to SIMPLER rather than provided here.
Go to primary source ↓pairedPDF p. 7, Figure 4, task headers, axes and policy legend; Section 4.1
Figure 4 plots real versus generated success for pi0, DexVLA, Diffusion Policy and OpenVLA. Pearson r is 0.935, 0.885, 0.958, 0.980 and 0.887 for Bussing Table, Collect Toy, Place Cup, Handover Block and Strike Block. Its Strike Block header prints MMRV=0.000, whereas Table 1 prints 0.133. The sloped gray lines are not equality references; their construction is not specified in the caption or text. In Collect Toy, pi0 has higher real success but lower generated success than DexVLA.
Go to primary source ↓simulationPDF p. 7, Table 1, all rows and Avg. column; pp. 7–8, Section 4.2; p. 19, Appendix C Evaluation on Robotwin
Table 1 reports average WorldEval MMRV/r of 0.044/0.942 versus real-to-sim 0.261/0.411 across Place Cup, Strike Block and Handover Block. The comparator uses RoboTwin with real-trained policies; a direct SIMPLER comparison was infeasible. Appendix C says simulation observations are transformed via sim-to-real visual models and the MidJourney API before policy inference.
Go to primary source ↓encodingPDF p. 7, Table 2, all encoding rows; p. 9, Section 4.3 encoding comparison
Policy2Vec reports r=0.939, MMRV=0.192, FID=61.33; VQVAE reports -0.862/0.292/71.79; one-hot reports -0.333/0.416/75.91. The caption specifies five tasks. Section 4.3 describes vector quantization and threshold-based one-hot encoding, but says the one-hot MMRV reduction is 2.24; subtraction of the table values gives 0.224.
Go to primary source ↓qualitativePDF p. 8, Figure 5 and caption; p. 9, Section 4.3 action fidelity; pp. 18–19, Figures 8–9
Figure 5 pairs real and generated pi0 examples, with Bussing Table, Place Cup and Handover Block labeled successes and Collect Toy and Strike Block labeled failures. Appendix figures supply additional success and failure examples. Section 4.3 describes Collect Toy as a novel-task generalization example.
Go to primary source ↓safetyPDF p. 8, Figure 6; p. 9, Section 4.3 safety detector paragraph
One hazardous DexVLA example raises both arms while its corresponding generated video becomes a mosaic. Poorly learned latent features are offered as a hypothesis; the presented evidence does not supply a safety-detector operating threshold, false-positive rate or recall.
Go to primary source ↓fidPDF p. 9, Figure 7 task headers and axes, Section 4.3 FID paragraph; p. 10, continuation
Figure 7 labels absolute correlations |r| of 0.495 for Bussing Table, 0.760 for Collect Toy, 0.999 for Place Cup, 0.993 for Handover Block and 0.805 for Strike Block. The plotted trend is negative: higher real success generally accompanies lower FID. The text limits FID's utility especially for complex table bussing.
Go to primary source ↓checkpointsPDF p. 17, Appendix B policy-checkpoint paragraph and Table 3, all rows
The video model is fine-tuned on latent actions from final policy checkpoints. Table 3 defines A/B/C as brown bowl/green plate/brown mug placement. For pi0 Task A, 5k to final 60k changes FID 81.62 to 74.65 and Succ. 0.725 to 0.925. Diffusion Policy reports zero success throughout; Task A FID changes 83.60, 80.99, 82.96. The table supplies a single Succ. column per task, not separate real/proxy columns.
Go to primary source ↓shiftPDF pp. 17–18, Appendix B distribution-shift paragraphs
Policies trained in a laboratory are evaluated in an office, living room and kitchen on cup placement, block striking and block handover. The text reports MMRV=0.047 and Pearson r=0.927, comparing with laboratory MMRV=0.044. Its wording calls 0.047 lower than 0.044, contrary to those printed numbers.
Go to primary source ↓limitationsPDF p. 17, Appendix A; PDF p. 5, Section 3.2 for conditioning contrast
The authors acknowledge hallucinations, object deformation/disappearance, overexposure, arm ghosting, incomplete action fidelity and difficulties in substantially novel scenarios. Appendix A says control exclusively uses action outputs and proposes adding language; Section 3.2 already includes language and an initial frame, leaving inconsistent descriptions.
Go to primary source ↓frequencyPDF p. 20, Figure 10, row/frame labels, and Appendix D final paragraph
Figure 10 labels its upper row 50 Hz and lower row 10 Hz, with real/generated samples at frames 20, 30 and 40. Appendix D reverses the row order in prose. It describes separate policies trained on equal trajectory counts, evaluated by one WorldEval model, and claims smoother generated behavior for 50 Hz.
Go to primary source ↓8.5 Primary sources
WorldEval: World Model as Real-World Robot Policies Evaluator ↗
PDF · 8,985 extracted words
Source fingerprint
9ba537bf8a9bae5223465c26858d1c55489253e107b6dca53015f0370d30a043