EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models
1. Paper overview
In one sentence: EWMBench separates scene stability, motion consistency, and semantic alignment to expose failures hidden by plausible robot videos, but its offline scores remain proxies whose calibration and reporting need careful scrutiny. e02e03e06e07e09e10e12
| At a glance | What to know |
|---|---|
| Research problem | Source description A robot video can preserve its background while failing to perform the requested manipulation. EWMBench asks whether generated scenes retain their structure, end-effector motion follows a plausible task sequence, and language-described goals agree with the depicted interactions. General perceptual quality alone does not answer these questions. e02e03 |
| Core mechanism | |
| A key reported result | Overall embodied video quality on EWMBench: EnerVerse_FT 4.7010; LTX_FT 4.5493. Table 2 Overall aggregate; higher is better, not a percentage.. Section 4 image-text-to-video evaluation: ten Agibot-World tasks, ten ground-truth episodes per task, three candidates per model and episode, selected by Hausdorff distance. Kling 3.8698; Hailuo 3.4125; COSMOS 3.2872; OpenSora 3.1392; LTX 2.9676. These are Table 2 values; its group and overall aggregates are consistent with sums up to rounding. Figure 4 assigns the last four scores differently. The table favors domain adaptation but gives no uncertainty estimates; unequal training exposure and missing recipes prevent a clean causal attribution. e08e09e10e20 |
| Reading caution | Source description The authors restrict current evaluation to end-effector trajectories, fixed viewpoints, and manipulation; whole-arm configuration, moving cameras, navigation, and mobile manipulation remain future work. e17 |
Core contributions
Figure 2. Initialization feeds generation, and generated frames feed three distinct evaluations. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the green World Initialization block. Its scene image and task instruction define the manipulation context; the left/right trajectory icons represent an optional conditioning input. Follow the horizontal arrow into Generative Models, then the downward arrow to Generated Video Frames, and finally the upward arrow to Evaluation Metrics. The arrows describe a one-way evaluation pipeline. Equation (1) supplies preprocessing and video normalization omitted from the drawing. Scene features, extracted motion tracks, and semantic descriptions are different views of the generated output. Although the diagram lists action-conditional generators, Section 4 evaluates only image-text-to-video models. e03e06e07e08
What it supports. The architectural contribution is a common benchmark interface, with separate tests of what stays stable, how the end-effector moves, and whether the depicted behavior matches language. The optional input trajectory is distinct from the trajectory detected afterward for scoring; neither arrow depicts action execution on hardware.
Where the evidence stops. The diagram is not a robot controller or a shared world-action network. Action-conditioned evaluation is deferred, and there is no depicted or reported feedback path that corrects a physical robot using these scores.
2. Motivation
2.1 The problem and the proposed response
A robot video can preserve its background while failing to perform the requested manipulation. EWMBench asks whether generated scenes retain their structure, end-effector motion follows a plausible task sequence, and language-described goals agree with the depicted interactions. General perceptual quality alone does not answer these questions. e02e03
2.2 What this reading follows
Consider a generated robot video that keeps the workbench perfectly still while never completing the requested handover. EWMBench treats that as a reason to evaluate several properties independently. It combines a manipulation dataset with scene features, detected end-effector trajectories, and language-based judgments. This reading follows the path from initialization to scoring, then examines what the model comparison and deliberate trajectory corruptions actually establish. Domain-adapted models lead the main results table, but the paper also contains conflicting score displays and incomplete evaluation details. The useful contribution is a structured diagnostic framework; the reported scores do not demonstrate successful execution on a robot. e02e03e06e07e09e10e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The recorded snapshot is entirely unassigned. EWMBench is an evaluation benchmark whose outputs are scores of other models. Its collection of detectors, visual encoders, and an MLLM does not establish a One Model or Two Models world-action architecture, joint future/action prediction, or inverse-dynamics control. The optional action input is an interface capability, untested here. e03e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one sub-action through the evaluator
The evaluation unit connects a scene to an instruction describing the current sub-action. The dataset decomposes high-level tasks into aligned segments, while the initialization interface allows up to four reference images and an optional sequence of 6D poses. In the tested setting, image-text-conditioned generators produce videos. Preprocessing fixes the image resolution and adds viewpoint and first-frame constraints; output normalization sets a common resolution and frame rate for scene and semantic scoring. The evaluator then splits into distinct representations: DINOv2 patch embeddings for scene structure, detected and tracked end-effector positions for motion, and MLLM descriptions for semantic comparison. These branches measure generated content. The paper's evaluator training concerns its feature extractor and detector, whereas the compared generators have their own training histories. e03e04e05e07e08e14
5.2 Why one trajectory score is insufficient
A trajectory contains both a set of visited positions and an order in which they are visited. HSD emphasizes the spatial set: reversing a path need not change its spatial coverage. nDTW can respond to the changed order, which explains the reversal diagnostic. DYN instead compares velocity and acceleration distributions through Wasserstein distance, with amplitude ratios intended to avoid misleading amplification from small movements. The still-frame experiment reveals why distributional dynamics and sequence alignment are not interchangeable: repeated alignment can raise nDTW while prolonged inactivity damages DYN. My interpretation is that the three measures act as checks on one another's blind spots. This remains an evaluation of one selected projected end-effector track; none of the three establishes whole-arm collision avoidance or object contact. e06e12e15e16e17
5.3 Separate a benchmark ranking from demonstrated capability
Read the overall ranking only after identifying the sampling rule and the measurement scale. Section 4 generates three candidates for every model and reference episode, and the motion criterion chooses the best candidate using ground truth. Table 2 therefore measures quality under that selection budget. Its aggregate values also combine unlike metric families, and the cells labeled averages are consistent with sums up to rounding. My reading is that the domain-adapted models show a promising advantage under this protocol, but the result does not establish a single-sample guarantee or a controlled causal effect of adaptation. The human comparison offers a second, narrower check of ordering among four models. Conflicting score displays, unspecified annotation details, and incomplete recipes should be resolved before treating the numerical ranking as a reproducible measurement standard. e06e08e09e10e11e20
5.4 Training and inference
During training
The evaluator fine-tunes dinov2-vitb14-reg4 on Agibot-World for 20,000 unsupervised iterations. It trains yolov8s-worldv2 for 100 epochs on 1,451 manually annotated frames, holding out Freezer Restocking and Factory Packing for validation; reported recall is 0.91667 and precision 1.0. These are evaluator-training details, not a new control-policy objective. e05e14
EnerVerse_FT and LTX_FT are evaluated as domain-adapted generators. The supplied paper does not give a complete generator fine-tuning recipe, evaluator optimization loss, frozen-module specification, or compute budget. e08e20
During inference
Each tested model generates three candidates for each of ten episodes in each of ten tasks: seven models yield 2,100 videos. The selection criterion is Hausdorff distance. This best-of-three evaluation uses a ground-truth-dependent selector and should not be interpreted as single-sample performance or a deployable action-selection loop. e06e08
The benchmark interface permits action conditioning, but Section 4 explicitly leaves its evaluation to future work. There is no reported robot execution or online feedback loop. e03e08
5.5 Implementation flow
- Curate ordered and varied tasks
Ten tasks cover toaster retrieval, pouring water, cutlery placement, freezer restocking, ice production, detergent packing, bottle cleaning, microwave heating, showerhead installation, and drawer storage. Tasks contain 4–10 atomic sub-actions. Appendix A.2 samples 100 instances per task, voxelizes left/right end-effector positions, and greedily selects ten trajectories per task by low average 3D voxel IoU. e04
- Normalize inputs and outputs
Resize reference images to 640 × 480 and append constraints to preserve the robot viewpoint and first frame. Use the current sub-action caption. Generated videos are resized to 640 × 480 and resampled to 30 FPS for scene and semantic evaluation. e03e14
- Measure scene structure
Fine-tuned DINOv2 extracts patch embeddings; cosine similarities against consecutive and initial frames measure layout and viewpoint consistency. Appendix feature maps illustrate the intended focus on agents and tools, without establishing a quantitative localization improvement. e05
- Measure motion through a detector
Fine-tuned YOLO-World detects both end-effectors and BoT-SORT associates them over time. Select the hand with the largest convex-hull diameter. Compare its generated and reference 2D tracks using HSD for spatial deviation, nDTW for ordered alignment, and DYN for velocity/acceleration distributions. e06e14e15e16
- Measure semantics separately
A video MLLM produces global captions scored against instructions with BLEU, step descriptions compared with MLLM-generated reference descriptions using CLIP, and penalties for logical violations. CLIP video-feature diversity is computed as one minus similarity; feature variation alone does not demonstrate successful task generalization. e07
6. Experiments & results
EWMBench evaluates instruction-conditioned robot videos through scene consistency, end-effector motion, and semantics. Its seven-model comparison favors domain-adapted generators, while controlled trajectory corruptions expose why static-looking plausibility is insufficient. These are offline video-evaluation results, with unresolved reporting inconsistencies, rather than demonstrations of executed robot control.
The paper supplies trajectory-corruption diagnostics and qualitative feature comparisons, rather than a controlled generator-training ablation or robot-execution success study. The ablation visuals therefore illustrate metric sensitivity and evaluator representation changes; they cannot quantify a control-policy improvement. e03e05e08e12
6.1 Read the original evidence
Table 2. The main table favors domain-adapted generators under Hausdorff-selected best-of-three evaluation. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each model row across the three families before looking at Overall. SceneC summarizes scene consistency; HSD, Dyn, and nDTW address different trajectory properties; the semantic columns mix diversity, caption alignment, and logic. The Avg. columns and Overall are consistent with sums up to rounding, rather than arithmetic means or success percentages. For example, OpenSora's displayed semantic components sum to 1.8740, versus its printed 1.8739. Its displayed group values sum to 3.1391, versus Overall 3.1392. All models use the Section 4 protocol of three generated candidates per reference episode, selected by Hausdorff distance. Compare LTX_FT with LTX for a within-family contrast, while retaining the missing training-control caveat. e08e09e10e20
What it supports. Table 2 reports EnerVerse_FT at 4.7010, LTX_FT at 4.5493, and Kling at 3.8698 overall. LTX_FT also exceeds LTX on HSD, DYN, and nDTW. This supports the usefulness of domain adaptation in this reported comparison; it does not isolate the effect of a particular training component.
Where the evidence stops. Figure 4(B), on page 6, assigns the four lower scores to different models. This reading retains Table 2's assignments and flags the conflict. Missing uncertainty estimates and generator-training details limit statistical and causal interpretation.
Figure 5. A stable scene can conceal a failed handover. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally from the reference image to the bad and good generated frame strips, keeping each task label attached to its row. The first row emphasizes scene preservation; the toast row presents trajectory scores; the bottom showerhead handover combines both. On that bottom row, compare the bad example's high SceneC with its low HSD, DYN, and nDTW, then inspect the more visibly changing good example. This is the paper's clearest illustration of why scene and motion judgments should remain separate. The displayed frames support the comparison, while Section 4.1 provides the interpretation that apparently plausible static videos can lack meaningful motion. e13e06e17e18
What it supports. The bad handover has SceneC 0.994 yet DYN 0.005; the good example has SceneC 0.878 and DYN 0.614. The scene score alone would favor the wrong example. Combining complementary diagnostics exposes the difference between preserving an appearance and depicting the required behavior.
Where the evidence stops. These are selected examples with printed frame sequences, not a measured failure frequency or continuous-video inspection. A high trajectory score still does not verify depth, gripper contact, whole-arm feasibility, or successful physical execution.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Overall embodied video quality on EWMBench Section 4 image-text-to-video evaluation: ten Agibot-World tasks, ten ground-truth episodes per task, three candidates per model and episode, selected by Hausdorff distance. | EnerVerse_FT 4.7010; LTX_FT 4.5493. Table 2 Overall aggregate; higher is better, not a percentage. | Kling 3.8698; Hailuo 3.4125; COSMOS 3.2872; OpenSora 3.1392; LTX 2.9676. These are Table 2 values; its group and overall aggregates are consistent with sums up to rounding. Figure 4 assigns the last four scores differently. The table favors domain adaptation but gives no uncertainty estimates; unequal training exposure and missing recipes prevent a clean causal attribution. e08e09e10e20 |
| Motion consistency of LTX and LTX_FT Same aggregate evaluation and candidate selection as Table 2. | LTX_FT: 0.4758 / 0.6197 / 0.5208. HSD / DYN / nDTW consistency scores; higher is better. | LTX: 0.1575 / 0.1002 / 0.1425. All three motion measures improve in the domain-adapted variant. For these two rows, Motion Avg. matches the displayed component sums: 1.6163 versus 0.4002. More generally, Table 2 aggregates are consistent with sums up to rounding. e08e09 |
| Agreement with human model rankings Four-model overall-quality ranking with multiple review rounds, Section 4.2 and Figure 6(A–B). | Human and EWMBench order: LTX_FT, Kling, Hailuo, OpenSora. Ordinal model ranking, not a reported correlation coefficient. | VBench order in Figure 6(B): Hailuo, Kling, LTX_FT, OpenSora. This supports agreement for the displayed comparison only. Annotator count, confidence intervals, and scoring of the third-ranked item are not specified; the prose assigns 3, 2, and 0 points to best, second, and worst among four models. e11e20 |
| Sensitivity to controlled trajectory corruption Sequence reversal, outlier insertion, and repeated still frames, Section 4.3 and Figure 6(C). | Reversal markedly lowers nDTW; an outlier lowers HSD and DYN; extensive still-frame repetition lowers DYN while increasing nDTW. Changes in HSD, DYN, and nDTW consistency. | Before-corruption scores and the other trajectory metrics in each panel. The metrics have complementary failure sensitivities. These are diagnostic manipulations, not a component-removal ablation or physical-safety validation. e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. Human rankings and controlled corruptions test different aspects of metric validity. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top-left podium as an ordinal human comparison, then compare the model order in the two top-right bar groups. The WMBM-labeled group follows the human order, whereas VBench prefers Hailuo. In the lower panels, use the before/after legends rather than comparing colors across experiments. Reversal chiefly changes temporal ordering, an outlier changes spatial extent and dynamics, and still-frame insertion changes temporal progression. The last legend includes 5, 15, and 2000 inserted-frame conditions. Compare each metric against its own unmodified baseline: the relevant evidence is its response to the controlled corruption, not a universal threshold shared across panels. e10e11e12e15e16e20
What it supports. nDTW is strongly sensitive to reversal; HSD and DYN react to the outlier; extensive repetition depresses DYN even while nDTW increases. These complementary responses motivate keeping all three motion views. Separately, the four-model human ranking agrees with the EWMBench order shown here.
Where the evidence stops. The WMBM scores in panel B differ from Table 2 and are not merged with it. Human annotation counts and uncertainty are unstated. Panel C is a corruption diagnostic, not a generator-training ablation or a validation of physical safety.
Figure 8. Adapting the scene evaluator changes the features used to measure consistency. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each four-image group independently: Input, VBench, DINOv2, and WMBM. Appendix A.3.1 identifies WMBM here with the authors' DINOv2 fine-tuned on embodied data. Compare where the foreground agent and manipulated tool appear in the input with the spatial structure in each feature map. The paper interprets the adapted maps as concentrating more coherently on task-relevant regions. The surrounding method explains why this matters: scene consistency uses patch-level cosine similarity, so its behavior depends on which visual structure the encoder represents. The adapted encoder starts from dinov2-vitb14-reg4 and receives 20,000 unsupervised iterations on Agibot-World. e05e20
What it supports. The examples make evaluator adaptation visible: the benchmark does not simply reuse a generic visual representation unchanged. They support the authors' rationale for adapting scene features, while the actual scene score remains an embedding-similarity measurement rather than a direct test of successful manipulation.
Where the evidence stops. The colors have no supplied quantitative or class legend. These qualitative maps do not establish localization accuracy, uncertainty, or the numerical benefit of fine-tuning in isolation; they should not be interpreted as calibrated probabilities.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors restrict current evaluation to end-effector trajectories, fixed viewpoints, and manipulation; whole-arm configuration, moving cameras, navigation, and mobile manipulation remain future work. e17
Projected tracks can miss depth, contact, and unobserved-arm failures. Figure 5 shows a bad showerhead example with SceneC 0.994 but DYN 0.005; Appendix A.4 also documents empty grasping, viewpoint drift, human-hand substitutions, and jitter. Printed frames do not establish continuous dynamics or executed success. e06e13e18
Reporting conflicts remain unresolved: the introduction says 30 candidate samples, unlike the ten-by-ten experiment and appendix selection; Figure 4 swaps Hailuo/COSMOS and OpenSora/LTX score assignments relative to Table 2; Figure 6(B) uses WMBM scores unlike Table 2. These displays are not merged. e19e10
7.2 Questions for discussion
- How much of the reported ranking survives removal of ground-truth-dependent best-of-three selection?
- Can motion scores distinguish successful contact from a visually similar empty grasp when only one 2D end-effector track is evaluated?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful replication needs exact selected episodes, voxel-grid settings, evaluator checkpoints, detection/tracking thresholds, prompt suite and MLLM version, generation settings, metric normalization and aggregation, and zero-distance conventions. The PDF supplies some preprocessing and training counts but omits this complete specification, software/hardware configuration, and full online prompts. e04e07e14e15e16e20
Proposed checks: compare automatic with manually verified end-effector tracks under the same three trajectory corruptions; then compare first-candidate, mean-of-three, and Hausdorff-selected scores on identical generated samples. These isolate evaluator error and selection benefit without claiming full-system reproduction. e06e08e12e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate motion-metric sensitivity from tracking error
Reader-proposed check, not performed: use a fixed set of generated/reference clips from several benchmark tasks and manually verify both end-effector tracks. Compute HSD, nDTW, and DYN on those tracks and on the YOLO-World/BoT-SORT outputs, holding normalization, candidate choice, and the selected hand constant. Apply the same reversal, outlier, and still-frame corruptions to both sets of tracks. Report task-level score changes and the fraction of clips whose ranking changes after correction. The metric-complementarity interpretation predicts the same directional corruption responses on clean tracks; disappearance of those responses or large ranking reversals after annotation correction would implicate the extraction pipeline. Explicitly record missing detections and zero-distance handling. e06e12e14e15e16e20
Check 2: Measure the benefit and bias of best-of-three selection
Reader-proposed check, not performed: retain all three candidates for every model and episode in the stated ten-task evaluation. Keep prompts, reference data, evaluator versions, and score aggregation fixed. Compare the first generated candidate, the mean over all three candidates, and the Hausdorff-selected candidate on each individual metric as well as Overall. Use paired episode-level differences, retain task groupings, and add a blinded human ranking of the same candidates. If the domain-adapted advantage reflects consistently better generation, it should persist beyond the selected samples. If it shrinks substantially or reverses under first-candidate or mean scoring, the published best-of-three ranking is sensitive to selection. Check separately whether optimizing HSD sacrifices semantic or dynamic quality. e06e08e09e11e20
8.3 Reading coverage
Visual audit: Visually inspected the title, all main-body scientific pages, and all appendix pages: Figures 1–14, Tables 1–2, Equations (1)–(7), protocols, training details, and limitations. Every final crop was separately inspected; labels, legends, axes and complete table headers are retained. Figure 2 arrow directions agree with its caption and Equation (1). Figure 4/Table 2 model-score assignments and Figure 6/Table 2 score differences remain explicitly unresolved. Reference-only pages 9–11 were read as text; all five supplied text chunks were read in full. Printed frame strips were inspected, not external videos or separate supplements.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Works (2.1–2.2)
- 3 The EWMBench Benchmark (3.1–3.4)
- 4 Experiments (4.1–4.4)
- 5 Conclusions and Limitations
- References
- A.1 Additional Details on World Specification
- A.2 Additional Details on Dataset Curation (A.2.1–A.2.2)
- A.3 Additional Details on Metrics (A.3.1–A.3.2)
- A.4 Visual Examples of Model Generation Results
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The complete supplied 17-page PDF text, including references and appendix, was read. The extraction limitation above was addressed by visually inspecting all scientific pages and all figures and tables; reference-only pages 9–11 were read as text.
- Identity: the title page identifies arXiv:2505.09694v2 [cs.RO], 18 May 2025. The catalog submission date is 14 May 2025; no v1 artifact or revision history was supplied, so inter-version scientific changes cannot be established. The title agrees; the first author is printed Yue Hu, whereas the catalog says Hu Yue. The other seven authors agree in order.
- External project material, full online MLLM prompts, code, datasets, checkpoints, and videos beyond the printed frame sequences were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block, affiliation footnote, arXiv margin
The title matches EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models. The eight authors begin Yue Hu; the margin identifies arXiv:2505.09694v2, 18 May 2025. Affiliations are AgiBot, SJTU, MMLab-CUHK, and HIT.
Go to primary source ↓e02PDF p. 2, Section 1, motivation and evaluation dimensions
Structured manipulation motivates separate scene, motion, and semantic evaluations beyond general video quality.
Go to primary source ↓e03PDF p. 4, Figure 2, Section 3.1 and Equation (1)
Initial images, language, and optional 6D-pose trajectories feed a preprocessed generator; generated frames are normalized and scored by separate evaluator modules. Up to four initial images are supported.
Go to primary source ↓e04PDF p. 4, Section 3.2; pp. 12–13, Appendix A.2.1–A.2.2 and Equation (2)
Ten named tasks are decomposed into 4–10 sub-actions. Appendix curation samples 100 instances per task, constructs left/right voxel grids, and greedily selects ten low-IoU trajectories per task.
Go to primary source ↓e05PDF p. 5, Section 3.3(A); pp. 13–14, Appendix A.3.1, Figures 8–9
Scene consistency compares DINOv2 patch embeddings. The evaluator starts from dinov2-vitb14-reg4 and uses 20,000 unsupervised iterations; qualitative feature maps compare VBench, pretrained DINOv2, and the adapted model labeled WMBM.
Go to primary source ↓e06PDF p. 5, Section 3.3(B); p. 12, Appendix A.1, Trajectory Extraction
HSD, nDTW, and DYN evaluate detected end-effector motion; candidates are selected by Hausdorff distance. YOLO-World and BoT-SORT yield two 2D tracks, and the hand with the greatest convex-hull diameter is selected.
Go to primary source ↓e07PDF pp. 5–6, Sections 3.3(C) and 3.4
CLIP feature diversity uses one minus similarity; MLLM global captions use BLEU, step captions use CLIP, and logical violations are penalized. Full prompts are deferred to the project page.
Go to primary source ↓e08PDF p. 6, Section 4, Models and evaluation protocol
Seven models are evaluated in image-text-to-video mode: OpenSora 2.0, LTX, COSMOS-7B, Kling-1.6, Hailuo I2V-01-live, LTX_FT, and EnerVerse. Ten tasks times ten episodes times three candidates times seven models gives 2,100 videos; action-conditioned evaluation is future work.
Go to primary source ↓e09PDF p. 7, Table 2, all model rows; Motion, Semantics and Overall columns
Overall scores are EnerVerse_FT 4.7010, LTX_FT 4.5493, Kling 3.8698, Hailuo 3.4125, COSMOS 3.2872, OpenSora 3.1392, LTX 2.9676. LTX_FT motion scores are 0.4758/0.6197/0.5208 versus LTX 0.1575/0.1002/0.1425. The Avg. columns and Overall are consistent with sums up to rounding. OpenSora's displayed semantic components sum to 1.8740 versus printed 1.8739; its displayed group values sum to 3.1391 versus Overall 3.1392.
Go to primary source ↓e10PDF p. 6, Figure 4(B); p. 7, Table 2; p. 8, Figure 6(B)
Figure 4 assigns 3.4125 to COSMOS, 3.2872 to Hailuo, 3.1392 to LTX, and 2.9676 to OpenSora, unlike Table 2. Figure 6(B) displays WMBM scores 5.49/4.75/4.30/4.03 for LTX_FT/Kling/Hailuo/OpenSora, unlike their Table 2 Overall values.
Go to primary source ↓e11PDF p. 7, Section 4.2; p. 8, Figure 6(A–B)
The human podium and EWMBench scores order LTX_FT, Kling, Hailuo, OpenSora; VBench orders Hailuo, Kling, LTX_FT, OpenSora. The text specifies 3/2/0 points for best/second/worst but leaves the third-place assignment and annotator count unstated.
Go to primary source ↓e12PDF p. 7, Section 4.3; p. 8, Figure 6(C), all three panels
Reversal chiefly reduces nDTW; outliers reduce HSD and DYN; repeated still frames eventually reduce DYN while nDTW increases. Legends retain before/after and before/5/15/2000-frame conditions.
Go to primary source ↓e13PDF p. 7, Figure 5, showerhead bottom row; p. 6, Section 4.1
The bad showerhead sequence has SceneC 0.994, HSD 0.178, DYN 0.005, nDTW 0.123; the good sequence has 0.878, 0.520, 0.614, 0.558 respectively. Section 4.1 explains that static plausibility can hide absent motion.
Go to primary source ↓e14PDF p. 12, Appendix A.1, preprocessing, normalization, trajectory extraction and detector training
Images and generated videos use 640 × 480; normalized videos use 30 FPS. Prompt constraints preserve viewpoint/first frame. yolov8s-worldv2 uses 1,451 annotated frames, 100 epochs, two held-out tasks, recall 0.91667 and precision 1.0.
Go to primary source ↓e15PDF pp. 14–15, Appendix A.3.2, Equations (3)–(4)
HSD and nDTW scores are reciprocals of distances between ground-truth G and prediction P. The printed equations contain no zero-distance safeguard.
Go to primary source ↓e16PDF p. 15, Appendix A.3.2, Equations (5)–(7)
DYN combines inverse Wasserstein distances of velocity and acceleration, amplitude ratios VR/AR, epsilon 10^{-8}, and weights alpha 0.007 and beta 0.003.
Go to primary source ↓e17PDF p. 8, Section 5, Limitations and Future Work
Whole-arm state, dynamic viewpoints, navigation, and mobile manipulation are beyond the present end-effector/fixed-view manipulation evaluation.
Go to primary source ↓e18PDF pp. 15–17, Appendix A.4, Figures 10–14 and captions
The paper presents empty grasping by adapted models, COSMOS viewpoint drift, LTX scene transitions, human-hand substitutions by LTX/COSMOS, and OpenSora jitter as qualitative failure examples.
Go to primary source ↓e19PDF p. 2, Section 1, dataset paragraph; p. 6, Section 4; pp. 12–13, Appendix A.2.2
The introduction says 30 candidate samples across ten tasks; Section 4 evaluates ten episodes per task, while the appendix selects ten trajectories per task from 100 instances per task. Their relationship is not explained.
Go to primary source ↓e20PDF pp. 5–6, Sections 3.3–4.1; p. 7, Table 2 and Section 4.2; pp. 12–15, Appendix A.1–A.3.2
The PDF provides evaluator model names and some training/preprocessing parameters, but no complete generator fine-tuning recipes, MLLM identity/full prompts, metric normalization/aggregation algorithm, zero-distance conventions, software/hardware specification, or quantitative uncertainty protocol. Table 2 group aggregates exceed one despite the individual-score normalization description.
Go to primary source ↓8.5 Primary sources
EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Models ↗
PDF · 6,995 extracted words
Source fingerprint
42b02ed4949bdd4377830927b42205a4a1f5eab48a2e491bce065319eb5e6d01