Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
1. Paper overview
In one sentence: WoW-World-Eval connects generated robot videos to human judgment and IDM-mediated execution, exposing weak planning while making assessment depend on learned judges and calibration. e-scopee-calibratione-human-rubrice-execution
| At a glance | What to know |
|---|---|
| Research problem | Source description Standard video realism can conceal incorrect object interactions, missing sub-goals or unusable robot motion. The authors ask whether video foundation models both convince observers and provide useful embodied priors, using the minimal initial-image-plus-instruction interface. e-scope |
| Core mechanism | |
| A key reported result | Real-world execution through GC-IDM: WoW-wan 40.74%; WoW-cosmos2 18.52%. Real-world success rate (%). Generated videos decoded by a real-world-trained IDM; nine manipulation tasks. Kling 9.88%; Hailuo 2.47%; Cosmos-Predict2 8.64%; CogVideoX, Cosmos-Predict1 and Wan2.1 each 0.00%. Video-score leadership does not ensure action transfer. Trial denominators and uncertainty are not supplied for these generated-video results. e-executione-main-table |
| Reading caution |
Core contributions
- Source description
Two complementary Turing tests connect video assessment to human real/fake discrimination and to actions decoded by an independently trained inverse dynamics model. e-human-turinge-execution
Figure 2. The benchmark combines curated robot demonstrations with edited and artistic generalization inputs. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left half as data preparation. Public RoboMIND/DROID examples and in-house scenes enter preliminary GPT selection; the blue downward arrow leads to human labeling. The right-hand branches illustrate edited OOD images, cleaned examples and visual annotations for grounding metrics. The donut charts describe different distributions, so their percentages do not all share one denominator. The ability chart shows where most evaluation material lies; the other charts divide physical interactions, single-object material types and perception attributes. Section 3.2 supplies the entry format: initial image, instruction, reference video when available, and annotations for relevant prediction metrics. e-datae-generalization
What it supports. Prediction and perception dominate the displayed ability distribution at 50.57% and 40.89%. This supports reading the benchmark primarily as a detailed examination of generated robot perception and dynamics, with a much smaller planning component. The curated mixture broadens input variety without making all capability estimates equally well supported.
Where the evidence stops. Figure 2 labels planning 3.61%, whereas Section 3.1 reports 25 planning samples out of 609; this discrepancy is unresolved. Generalization cases can lack reference videos, so not every case supports the same metrics.
2. Motivation
2.1 The problem and the proposed response
Standard video realism can conceal incorrect object interactions, missing sub-goals or unusable robot motion. The authors ask whether video foundation models both convince observers and provide useful embodied priors, using the minimal initial-image-plus-instruction interface. e-scope
2.2 What this reading follows
A robot video can look convincing while skipping the requested manipulation or suggesting motion that an action decoder cannot execute. WoW-World-Eval makes these distinctions measurable. It starts from an image and an instruction, evaluates the resulting video with complementary metrics, and adds two separate tests: whether people mistake generated footage for reality, and whether a real-world-trained inverse dynamics model can turn it into successful robot behavior. Read the figures as an evaluation pipeline, not a proposed robot policy. The headline rankings are useful, but their meaning depends on normalization, human-rating conventions and the decoder used for execution. e-scopee-calibratione-human-rubrice-execution
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The catalog snapshot remains Not assigned. A One Model or prediction/action quadrant is not applicable to this benchmark itself: it evaluates external video generators and uses a separate IDM for action extraction. Evaluator fine-tuning is not evidence of a joint world-action architecture. e-scopee-evaluatore-execution
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one manipulation through the evaluation branches
Begin with an initial image containing a robot and a target object, plus an instruction. The candidate generator predicts a video; WoW-World-Eval then asks several different questions of that same prediction. Appearance metrics compare visual content to reference footage. Semantic evaluation describes initial, processing and final states, then checks actions and objects. Region tracking separates a stable background from a potentially unstable gripper or object. Finally, planning evaluation converts the depicted steps into a DAG whose dependencies permit different valid orderings. An apparently smooth clip can therefore score poorly for leaving out an action or losing an object. If a generalization input has no reference video, caption comparison is unavailable and instruction evaluation uses the remaining scores. The benchmark never makes one shared-reference assumption valid for every sample. e-generalizatione-semantice-regionse-planning
Figure 5. Separate evaluators inspect motion, action dependencies and physical plausibility. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: human annotations initialize SAM2 tracking, while the DINOv3 branch represents region-feature analysis. Follow the upper-right video strips into generated DAGs and then a DAG comparison score; the arrows encode dependencies between atomic actions, not robot actuator commands. The bottom row moves from video-QA data through first-stage GRPO, then human scoring annotations and second-stage GRPO. Section 9.5 explains that this trains Qwen-2.5-VL as an evaluator. Although the final lower arrows point toward both a video strip and JSON, the video is an evaluation example; the trained model scores it rather than synthesizing it. e-regionse-trajectoriese-planninge-evaluatore-physics-prompt
What it supports. The design can distinguish a stable scene from a drifting gripper, and a visually plausible clip from a missing or wrongly ordered task step. The physics judge adds learned commonsense assessment. These are complementary evidence channels, while the generator itself remains an external system being evaluated.
Where the evidence stops. The lower schematic is not a video-generation architecture. Four scoring dimensions supervise stage two, but inference requests six physical categories. Sections 9.5 and 12 specify that change; the illustrated score is an example, not a benchmark-wide result.
5.2 Understand what the physics judge learns before trusting its score
The physics evaluator has its own training pipeline. First, Qwen-2.5-VL learns from multiple-choice video questions through group-relative rewards: correct answers receive binary credit, and eight sampled outputs establish the comparison group. Second, human ratings teach four structured scoring dimensions using a reward based on normalized score error. At evaluation, the prompt narrows attention to six categories of physical plausibility, with null allowed when a category is absent. This is a change in requested scoring structure, not six separately reported training objectives. Automatic scores are then direction-corrected, monotonically transformed and aggregated. Because the transform is fitted to human ratings, understanding the calibration split is part of understanding the metric itself. A high correlation is meaningful evidence of alignment only within its documented evaluation conditions. e-evaluatore-physics-prompte-calibration
5.3 Separate the two meanings of passing a Turing test
The human Turing test asks whether a person mistakes generated footage for real footage; the expert-rating study instead asks how good that footage is across several dimensions. The IDM test asks a third operational question: can a learned inverse dynamics model turn the video into actions that succeed on a robot? Reader interpretation: these tests probe different bottlenecks, so their leaders need not coincide. Hailuo leads the aggregate video score but WoW-wan leads GC-IDM-mediated execution. The genuine-video replay control supports the decoder’s competence on real demonstrations, yet it cannot prove equal competence on synthetic inputs. Treat the execution outcome as a property of the generator–decoder–robot combination under the stated tasks, rather than a universal physical-validity certificate for the generated video. e-humane-human-turinge-main-tablee-executione-replay
5.4 Training and inference
During training
The learned component is an evaluator: Qwen-2.5-VL 7B receives GRPO on about 50,000 video-QA examples, with eight sampled answers and binary rewards, then 1,297 human-scored examples. Stage two rewards agreement across four JSON scoring keys through normalized absolute error. Stage-one held-out accuracy reportedly rises from 60.83% to 71.51%. These stages do not jointly train video generation and action prediction. e-evaluator
During inference
Evaluated generators produce five-second clips at differing resolutions. The trained physics judge switches to six physical categories, allowing inapplicable categories to be null. In the separate execution experiment, GC-IDM converts generated video into gripper-centric actions for real-world replay; the benchmark does not specify a closed-loop replanning controller. e-modelse-evaluatore-physics-prompte-executione-replay
5.5 Implementation flow
- Curate embodied cases
GPT-4o screens public and in-house data, followed by human verification and initial-frame/keypoint annotation. Edited images and paintings probe generalization. Reference-free cases omit caption-to-GT comparison. The collection is dominated by perception and prediction, with only 25 stated long-horizon planning examples. e-datae-generalization
- Measure appearance and task meaning
FVD, PSNR, SSIM, DINO and DreamSim assess complementary video properties. Structured captions compare states, actions and objects; sequence and execution-quality judgments assess the depicted instruction. These execution-quality scores describe videos and are distinct from measured robot success. e-semantice-main-tablee-execution
- Localize consistency failures
GroundingSAM-2 propagates human-initialized masks. Frozen DINOv3 features separately measure object, gripper and background stability; missing regions receive zero pairwise consistency. The detailed trajectory formulation uses normalized mask centroids, temporal subsampling and L2Norm/DTW/Fréchet comparisons, plus camera-motion diagnostics. e-regionse-trajectories
- Evaluate causal plans
Reference and generated videos become DAGs: nodes contain skill, object and arguments; edges encode dependencies. Node correctness and MLLM-estimated task completion reward valid decompositions without imposing a single total action order. The completion component is a lightweight language-model rollout, not physical execution. e-planning
- Calibrate and aggregate
Direction-correct raw metrics, apply fixed-anchor scaling and a selected monotone transform, then average within groups and combine available groups. Mapping parameters are fitted to human ratings and frozen. Thus the tables report desirability scores: their FVD column is higher-is-better after transformation. e-calibratione-main-table
6. Experiments & results
WoW-World-Eval tests whether instruction-conditioned robot videos are visually convincing, task-correct, physically plausible and usable for action extraction. Its 609-sample benchmark combines automated metrics, human judgments and a separate GC-IDM robot-execution test. Hailuo leads the reported aggregate video score, while WoW-wan leads physical execution. The central lesson is that a plausible imagined manipulation and an executable one are distinct outcomes; calibration and evaluator dependence constrain how broadly the scores can be interpreted.
This work proposes a benchmark rather than a new video generator. Figure 5 documents evaluator construction; Table 4 is a prompt intervention and Figure 6 is a metric-validation diagnostic. They should not be read as controlled component ablations of a joint world-action model. The paper’s real-video replay table displays only four of its nine described tasks. e-scopee-evaluatore-densee-humane-replay
6.1 Read the original evidence
Table 2. Good visual scores coexist with weak long-horizon planning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each model before comparing winners down a column. The first block measures video quality; the next measures caption, sequence and depicted execution agreement; the final column measures DAG planning. Each “Overall” inside this table belongs to its local block, rather than the complete four-group benchmark score, which appears in Table 3. The paper has already transformed the raw measurements to higher-is-better scores, including FVD. In particular, “Exec. Quality Score” is an assessment of the action shown in a video. Real robot success appears in a different table and uses a different protocol. e-main-tablee-calibratione-modelse-datae-planninge-execution
What it supports. Hailuo leads the displayed planning column at only 17.27, with Cosmos-Predict2 at 13.41 and WoW-cosmos2 at 12.27. WoW-cosmos2 nevertheless slightly exceeds Hailuo in instruction understanding, 70.36 versus 70.11. The comparison shows why instruction agreement and plan completeness should be examined separately rather than treated as interchangeable abilities.
Where the evidence stops. These are calibrated scores, not raw FVD, PSNR in decibels, or execution percentages. Section 8 uses differing output resolutions. No uncertainty intervals accompany these rankings, and the small planning subset limits broad conclusions.
Table 5. The best video-score model is not the best source of executable action. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This table changes the endpoint: its entries are real-world success percentages, not transformed video metrics. A generated video is passed to GC-IDM, which infers gripper-centric actions for execution across nine manipulation tasks. Read WoW-wan’s highlighted row against both WoW-cosmos2 and the commercial models. Then consult the separate reference-video control in Section 10.3: the authors first test whether GC-IDM can replay genuine demonstrations. That control matters because a failure after decoding could originate in the generated scene, the action decoder, or their interaction. It should inform interpretation without being treated as a perfect guarantee. e-executione-main-tablee-replay
What it supports. WoW-wan reaches 40.74%, compared with WoW-cosmos2 at 18.52%, Kling at 9.88% and Hailuo at 2.47%. Hailuo led the concise-prompt aggregate benchmark score, yet does poorly here. The result directly demonstrates a gap between strong video evaluation and this particular route from generated video to executed manipulation.
Where the evidence stops. Generated-video trial denominators and confidence intervals are absent. Table 7 displays only four task columns for the claimed 90% real-video replay control. It cannot rule out GC-IDM domain shift on synthetic videos or establish decoder-independent action quality.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Overall benchmark score with concise prompts WoW-World-Eval; reported four-group aggregate over eligible metric subsets. | Hailuo 52.55; WoW-cosmos2 50.74. Normalized overall score, higher is better | Kling: 37.93 overall despite the highest physical-law aggregate, 68.02. Balanced video competence changes the ranking relative to any single group. These scores are neither raw metric values nor success percentages. e-main-tablee-calibration |
| Long-horizon video planning Concise prompts; Section 3.1 states 25 planning samples. | Hailuo 17.27. Planning DAG score, 0–100 | Cosmos-Predict2 13.41; WoW-cosmos2 12.27. Planning remains weak under this evaluator; the score does not directly measure robot task success. e-datae-planninge-main-table |
| Dense-prompt intervention Same benchmark with InternVL3-78B-expanded instructions; Tables 2–4. | Cosmos-Predict2 instruction: 56.80 → 75.96; WoW-cosmos1 overall: 46.70 → 53.20. Normalized instruction, planning and overall scores | WoW-wan instruction falls 62.16 → 50.83; WoW-cosmos2 planning falls 12.27 → 9.09. Prompt enrichment helps selectively. The tabulated regressions contradict a universal-improvement reading of the prose. e-dense |
| Human preference alignment 15 domain experts; over 1,200 real and generated videos rated along four dimensions. | Overall: 0.93 / 0.91. Pearson r / Spearman rho | Planning: 0.43 / 0.51; physical law: 0.81 / 0.83. Strong aggregate agreement coexists with weaker planning agreement. The calibrated metric requires independent held-out validation. e-humane-calibration |
| Human Turing test 13 participants distinguish real from generated videos. | Overall-score Pearson r = 0.679. Correlation with deceive-human ratio | Video quality r = 0.874; physical law r = 0.753. Perceptual deception is a separate endpoint from expert preference; the 0.93 preference correlation cannot be substituted here. e-human-turinge-human |
| Real-world execution through GC-IDM Generated videos decoded by a real-world-trained IDM; nine manipulation tasks. | WoW-wan 40.74%; WoW-cosmos2 18.52%. Real-world success rate (%) | Kling 9.88%; Hailuo 2.47%; Cosmos-Predict2 8.64%; CogVideoX, Cosmos-Predict1 and Wan2.1 each 0.00%. Video-score leadership does not ensure action transfer. Trial denominators and uncertainty are not supplied for these generated-video results. e-executione-main-table |
| Ground-truth video replay control Nine tasks described with ten real videos each; Table 7 displays four tasks. | Authors claim 90% overall; displayed GC-IDM counts total 36/40. Replay success | Displayed ResNet-MLPs counts total 22/40; AVDC 13/40 (reader sums). The control supports real-video decoding ability, but the full nine-task breakdown is absent and generated-video domain shift remains possible. e-replay |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Prompt enrichment changes the balance of strengths and weaknesses. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Expand the headers as Video Quality, Instruction Understanding, Physical Law and Planning Reasoning. These are dense-prompt scores, so compare each row to the same model in Tables 2 and 3, not to another model under a different prompt condition. InternVL3-78B supplies the expanded instructions while the initial image stays fixed. The appendix template asks for environment, robot and camera details, sub-goals and a post-action state. That makes the intervention informative about conditioning, but not a pure test of word count: it can introduce additional inferred scene details and procedural guidance at the same time. e-densee-main-tablee-dense-prompt
What it supports. Cosmos-Predict2 instruction understanding increases from 56.80 to 75.96, while WoW-cosmos1 overall increases from 46.70 to 53.20. Gains are not universal: WoW-wan instruction understanding drops from 62.16 to 50.83, and WoW-cosmos2 planning drops from 12.27 to 9.09. The supported finding is selective benefit, not uniformly improved reasoning.
Where the evidence stops. The prose claims every model benefits in video quality, but Cosmos-Predict1 falls from 39.06 to 35.48. Preserve this table/prose discrepancy. The experiment changes several prompt attributes together and does not isolate an architectural mechanism.
Figure 6. Aggregate alignment is stronger than the planning evaluator’s alignment. Original paper, p. 24 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each point is a plotted score pair, and each colored line is a least-squares fit, not an uncertainty band. The top row shows quality, instruction following and physical law; the lower row shows planning and the overall score. Read the axes carefully: component human ratings use a 1–5 rubric, while the overall human score sums four components to a 4–20 scale. The annotation boxes report Pearson r for linear association and Spearman rho for rank association. These plots belong to the expert-rating study; they do not plot the separate 13-participant real-versus-generated Turing-test decisions. e-humane-human-rubrice-human-turinge-calibration
What it supports. The overall panel reports r = 0.93 and rho = 0.91, whereas planning reports only 0.43 and 0.51. Physics has the strongest component association at 0.81 and 0.83. Thus aggregate agreement supports usefulness as a summary while leaving substantial room for mistakes in the specific capability the benchmark identifies as weakest.
Where the evidence stops. The transformations were selected using human ratings. Although the paper describes development-set cross-validation and freezing, the exact separation behind these plotted validation scores is not specified. Correlation alone does not establish accurate judgments on unseen model families.
7. Analysis & limitations
7.1 What the evidence leaves open
The benchmark is unevenly distributed and does not establish clean separation from every generator’s training data. Figure 2 labels planning as 3.61%, inconsistent with 25/609; the allocation remains unresolved. e-datae-models
Calibration uses human supervision, but the exact partition behind the reported validation correlation is unclear. Human rubrics also encode conventions: static video gets maximum physics credit, while generated human task execution gets minimum physics credit. e-calibratione-humane-human-rubric
Caption scoring is 1–5 in the main text versus a mean of 0/0.5/1 components in the appendix; sequence scoring is discrete in Section 9.2 but proportional in the prompt. Camera correction also leaves pixel-versus-normalized-coordinate conversion unclear. These are unresolved protocol differences. e-semantice-trajectories
The authors attribute generated-video replay failures to video realism after the real-video control. Reader assessment: that control cannot exclude IDM sensitivity to synthetic appearance, viewpoint or timing; it does not isolate the generator as the sole cause. e-replaye-execution
7.2 Questions for discussion
- Does calibrated agreement persist on unseen generator families and manipulation tasks?
- How much of the execution ranking persists when the inverse dynamics decoder changes?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the curated samples, masks/keypoints, evaluator checkpoints, fixed model versions, prompts and calibration partitions. Section 8 specifies resolutions, but full optimization settings, hardware/runtime, all scaling anchors, robot/controller configuration and generated-video denominators remain missing. e-modelse-regionse-evaluatore-calibratione-reproduction
Proposed checks: freeze calibration before testing unseen task/model groups; separately compare GT and generated-video replay under matched rendering, timing and a second IDM. Preserve raw metrics and per-task trial counts so ranking changes and decoder dependence can be tested. e-calibratione-humane-replaye-execution
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test calibration on unseen tasks and generator families
Reader-proposed check, not performed: recover the raw metric outputs and human ratings, then partition by manipulation task and generator family before selecting any transformation. Fit the published mapping families only on development groups and freeze them. On held-out groups, compare Pearson/Spearman alignment against simple direction-corrected scaling; report component scores, aggregate scores and bootstrap intervals with task-level resampling. Keep videos, human rubrics and missing-metric handling identical across the two mappings. If the calibrated advantage disappears or rankings reverse on held-out groups, the reported aggregate alignment is sensitive to the calibration population rather than a stable improvement across models. e-calibratione-humane-human-rubric
Check 2: Measure how much execution ranking depends on the IDM
Reader-proposed check, not performed: use the nine listed manipulation tasks with matched initial scenes and fixed robot success criteria. Run genuine reference videos and generated videos through GC-IDM and a separately trained alternative IDM, giving both decoders the same real-video training pool. Match input resolution and temporal sampling; include a real-video resampling control to measure preprocessing sensitivity. Record attempts, successes and failure stages separately for each task, generator and decoder. A generator ranking that changes substantially between decoders, despite comparable real-video replay accuracy, would falsify the assumption that the table isolates video actionability independently of the inverse dynamics model. e-executione-replaye-modelse-trajectories
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version page; main benchmark diagrams and Tables 1–5; supplementary model settings, metric equations, evaluator objectives and Table 6; human rubrics, Figure 6 and Table 7; and the retained scoring/dense-prompt evidence pages. All six original final crops were inspected for legibility and complete relevant labels. Figure 5 arrow directions were cross-checked against Sections 3.3 and 9.3–9.5: its lower video strip is an evaluated example, not a video synthesized by the physics scorer. All numerical and method claims, including facts motivating proposed checks, have supporting pages in this list. Text reading covered all 35 pages via all 11 supplied chunks. Reference pages 11–15 and pages 26–29 were read in text only; Figures 7–10 and any underlying videos were not visually reviewed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 30, 31, 32, 33, 34, 35. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–6 (PDF pp. 1–10)
- References (PDF pp. 10–15)
- Supplementary Sections 7–8: related work and model details (PDF pp. 16–17)
- Supplementary Sections 9.1–9.6: all metric definitions, evaluator training and calibration (PDF pp. 17–22)
- Supplementary Sections 10.1–10.3: human rubrics, correlations and real-video replay (PDF pp. 22–25)
- Supplementary Sections 11–12: case descriptions and all evaluation/data/dense-caption prompts (PDF pp. 25–35)
Outside the original text pass
- Identity/version: the title and all 21 authors agree with the catalog; the inspected artifact is arXiv:2601.04137v1, 7 January 2026. No different revision was supplied or substituted. Supplementary printed page numbering restarts at 1 on PDF page 16; evidence uses absolute PDF pages.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- That extraction limitation was addressed by inspecting original PDF pages and all six final crops. Figures 7–10 on PDF pp. 26–28 were read as extracted captions only; their images and underlying videos were not inspected.
- Separate supplemental material availability has not been fully verified.
- All 11 supplied text chunks were read, including the supplement embedded in the PDF. External references, source code, benchmark files, evaluator checkpoints and separate media were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, complete byline, affiliations and arXiv margin stamp
The title and 21 authors match the catalog. The source is arXiv:2601.04137v1 [cs.RO], 7 January 2026. The title page names Peking University, Beijing Innovation Center of Humanoid Robotics, and The Hong Kong University of Science and Technology.
Go to primary source ↓e-scopePDF pp. 1–4, Abstract, Sections 1 and 3.1, Figure 1
WoW-World-Eval evaluates image-and-instruction-conditioned video generation through five ability dimensions and 22 metrics, including separate human and inverse-dynamics Turing tests; it is a benchmark, not a newly proposed joint action/video generator.
Go to primary source ↓e-dataPDF pp. 4–5, Figure 2 and Sections 3.1–3.2
The benchmark combines RoboMIND, DROID, in-house trajectories and AI-created OOD inputs, with GPT-4o selection and human verification. It reports 609 samples, 249 perception samples and 25 planning samples. Figure 2 labels prediction 50.57%, perception 40.89%, planning 3.61%, and execution 1.48%; its planning percentage does not reconcile with 25/609.
Go to primary source ↓e-generalizationPDF p. 5, Generative Generalization and Section 3.3, Without Ground-Truth
Style-transferred or edited in-house images and paintings provide generalization inputs. When GT video is absent, instruction evaluation reports sequence match and execution quality, omitting caption comparison.
Go to primary source ↓e-semanticPDF pp. 5, 17–18 and 30–31, Sections 3.3, 9.2 and semantic-evaluation prompts
GPT-4o/VLM evaluation compares initial, processing and final states, action and object descriptions; also scores action order and depicted execution quality. Main-text caption scale 1–5 differs from the appendix mean of 0/0.5/1 components. The appendix describes discrete sequence scoring, while the prompt specifies a proportional ordered-pair score.
Go to primary source ↓e-regionsPDF pp. 18–19, Figure 5 and Section 9.3 equations
Human contour points initialize GroundingSAM-2 masks; frozen DINOv3-Large produces mask-weighted region features. Object, gripper and background are scored separately. Missing masks yield zero features and zero pairwise consistency; first-frame and previous-frame similarities receive equal weight.
Go to primary source ↓e-trajectoriesPDF pp. 19–20, Section 9.4, mask-to-point, camera correction and distance equations
Detailed trajectory construction uses SAM2 mask centroids normalized by image size, uniform sampling to the shorter sequence, L2Norm, DTW and Fréchet distance. Camera motion uses boundary features, optical flow and affine estimation, with ATE/RPE comparisons. The correction subtracts a camera offset from normalized points, but camera offsets are later described in pixels; conversion and drift-clipping threshold are not explicit.
Go to primary source ↓e-planningPDF p. 6, Section 3.3, Planning and Task Decomposition and LongHorizon equation; PDF p. 18, Figure 5
Reference and generated videos are parsed into DAGs with atomic skill/object/args nodes and dependency edges. Node correctness and MLLM-estimated task completion are added and multiplied by 50. This scores depicted plans, not robot trials.
Go to primary source ↓e-evaluatorPDF pp. 21–22, Section 9.5, equations (1)–(5) and Final Inference for Evaluation; PDF p. 18, Figure 5
Qwen-2.5-VL 7B undergoes two GRPO stages: about 50,000 multiple-choice video-QA samples, group size 8 and binary correctness rewards; then 1,297 human-rated examples with four scoring keys. Stage-1 held-out accuracy is reported as 60.83% to 71.51%. Stage-2 reward is one minus matched-key normalized absolute error, clipped to [0,1]; invalid JSON or no matched keys scores zero. Final inference scores six physical categories.
Go to primary source ↓e-physics-promptPDF pp. 31–34, Section 12, Stage-2 scoring prompt and 6-Dimension Evaluation Prompt
Training requests quality, instruction following, physical consistency and planning logic. Final physical scoring requests object interaction, physical properties, temporal consistency, lighting/reflections, fluids/particles and local anomalies, permitting null for inapplicable categories.
Go to primary source ↓e-calibrationPDF p. 6, Section 3.4; PDF p. 22, Section 9.6 and Table 6
Metrics are direction-corrected with fixed anchors, transformed monotonically and scaled to 0–100 before group averaging and weighted aggregation over available groups. PSNR anchors are 0 and 50; FVD anchors are 0 and 2000. Parameters are selected using human ratings and cross-validation, then frozen. Table 6 includes FVD gamma 1.52, caption gamma 0.12 and planning simple mapping; all other absolute anchors and the exact development/evaluation partition are not enumerated.
Go to primary source ↓e-modelsPDF p. 7, Section 4, Models; PDF pp. 16–17, Section 8
Versions include Kling 2.1, Hailuo-02, CogVideoX1.5-I2V-5B, Cosmos-Predict1-7B, Cosmos-Predict2-2B, Wan2.1-I2V-14B and WoW variants. Section 8 describes five-second outputs with differing resolutions: 720p for Kling/Cosmos, 768p for Hailuo, 1360×768 for CogVideoX and 832×480 for Wan; WoW follows its backbone settings.
Go to primary source ↓e-main-tablePDF p. 7, Tables 2–3, named model rows and Overall / Planning DAG / Physical Law columns; PDF p. 8, Sections 4.1–4.2
With concise prompts, Hailuo has overall 52.55, quality 56.09, instruction 70.11 and planning 17.27. WoW-cosmos2 has overall 50.74, instruction 70.36 and planning 12.27. Cosmos-Predict2 planning is 13.41. Kling has physical-law aggregate 68.02 but overall 37.93. Scores are transformed, not raw errors or success percentages.
Go to primary source ↓e-densePDF pp. 7–8, Tables 2–4 and Section 4.2
InternVL3-78B expands concise prompts. Cosmos-Predict2 instruction changes 56.80 to 75.96; WoW-cosmos1 overall changes 46.70 to 53.20. Regressions include WoW-wan instruction 62.16 to 50.83 and WoW-cosmos2 planning 12.27 to 9.09. Cosmos-Predict1 quality falls 39.06 to 35.48, contradicting the prose claim that every model benefits in quality.
Go to primary source ↓e-dense-promptPDF pp. 34–35, Section 12, Dense Prompts Extension Prompt
The recaptioning template requests scene, environment, robot, camera, sub-goal/action breakdown and post-action state, aiming at 100–150 words. It permits inferable details, so the intervention changes semantic and planning information, not just length.
Go to primary source ↓e-humanPDF p. 9, Section 4.3 and Figure 3b; PDF p. 24, Figure 6 and Section 10.2
Fifteen domain experts rate over 1,200 real/generated videos. Overall metric/human correlation is Pearson r=0.93 and Spearman rho=0.91. Dimension correlations (r/rho) are quality 0.66/0.73, instruction 0.75/0.71, physics 0.81/0.83 and planning 0.43/0.51.
Go to primary source ↓e-human-rubricPDF pp. 22–24, Section 10.1
Human ratings score four dimensions from 1 to 5, summed to 4–20. Static video gets physical-law 5, instruction 1 and planning 1. Generated human task execution receives physical-law 1 regardless of physical plausibility.
Go to primary source ↓e-human-turingPDF p. 3, 2AFC description; PDF p. 9, Section 4.4, Human Turing Test and Figure 4
Thirteen participants distinguish real and generated videos. Deceive-human ratio correlates with overall score at r=0.679, video quality at 0.874 and physical law at 0.753. This is separate from the expert preference-rating study.
Go to primary source ↓e-executionPDF p. 10, Section 4.4, Inverse Dynamic Model Turing Test and Table 5
Generated videos are passed through real-world-trained GC-IDM for nine manipulation tasks. Real-world success percentages are Kling 9.88, Hailuo 2.47, CogVideoX 0.00, Cosmos-Predict1 0.00, Wan2.1 0.00, Cosmos-Predict2 8.64, WoW-wan 40.74 and WoW-cosmos2 18.52. Generated-video trial counts and confidence intervals are not supplied here.
Go to primary source ↓e-replayPDF p. 25, Section 10.3 and Table 7
GC-IDM real-video replay is described for nine tasks with ten GT videos each; overall 90% is claimed. Table 7 displays only four tasks: GC-IDM 10/10, 9/10, 9/10, 8/10 versus ResNet-MLPs 6/10, 7/10, 5/10, 4/10 and AVDC 3/10, 4/10, 4/10, 2/10. The shown GC-IDM subtotal is 36/40, not the complete nine-task breakdown.
Go to primary source ↓e-reproductionPDF pp. 16–22, Sections 8–9.6; PDF p. 25, Section 10.3
Versions, resolutions, evaluator stages and transformations are partly specified. These sections do not supply a complete runnable recipe: hardware/runtime, full optimization settings, all anchors, calibration partitions, exact robot/controller configuration and full per-task generated-video denominators remain unspecified.
Go to primary source ↓8.5 Primary sources
Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test ↗
PDF · 18,312 extracted words
Source fingerprint
aef22be60a1ee060d8a74ff0c157f9b69090af532086b0dcd145a206684f70fc