RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation
1. Paper overview
In one sentence: RoboWM-Bench turns generated manipulation videos into executable robot trajectories, gaining an operational test of physical feasibility while inheriting errors from action extraction and simulation. e03e08e09e10e11e13
| At a glance | What to know |
|---|---|
| Research problem | Source description A generated grasp can look convincing while lacking the contact geometry needed to lift an object. The benchmark makes this gap operational: generate a manipulation video from an initial observation and task description, recover actions, execute them, and test intermediate interactions and final completion. It targets scalable evaluation of human-hand and robot videos across rigid, articulated, deformable, compositional and bimanual tabletop tasks. e02e03e06 |
| Core mechanism | |
| A key reported result | Robotic Close Drawer: Cosmos-FT: 90%. Task success (%). Real-to-sim robot-video evaluation; Cosmos-FT receives 50 task-specific real trajectories for fine-tuning. Cosmos: 0%; Wan 2.6: 50%; Wan 2.2: 30%; Veo 3.1: 20%. Fine-tuning improves this evaluated task; the extra supervision means this is not an equal-data generalization comparison. e07e08 |
| Reading caution | Source description Action extraction can change contact-sensitive outcomes. Human videos are judged through robot retargeting; fingertip and gripper contacts differ. The suite is limited to selected tabletop settings, and manually authored key nodes/checkers constrain expansion. e13 |
Core contributions
Figure 2. Two video-to-action interfaces connect generated imagery to a shared execution test. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the green scene block: reconstruction produces the environment in which actions will be judged. Then follow the colored arrows from the observed scenes through World Models to the two video strips. The upper branch uses hand geometry to produce a gripper pose; the lower branch uses an inverse dynamics model to produce joint-space actions. Section 3.3.2 attributes that IDM architecture to reference [23], identified as DreamGen in the bibliography on page 11. Both branches terminate in simulated executions on the right. Section 3.3.1 clarifies the upper branch: the thumb–index midpoint specifies position, while contact-plane geometry specifies orientation. The arrows connect separate components for generating videos, recovering actions and simulating their consequences. e03e04e05e06e10e11
What it supports. The benchmark tests a concrete interface between visual prediction and action execution. Its human and robot routes share the final execution criterion but use different extraction mechanisms. Consequently, successful imagery and successful replay are separate observations, and a failure can originate in the video, its conversion, or the execution environment.
Where the evidence stops. The graphic is a pipeline illustration, not evidence for a unified learned architecture or an online feedback controller. Its checkmarks are illustrative; quantitative reliability comes from the separate experiments in Section 4.4.
2. Motivation
2.1 The problem and the proposed response
A generated grasp can look convincing while lacking the contact geometry needed to lift an object. The benchmark makes this gap operational: generate a manipulation video from an initial observation and task description, recover actions, execute them, and test intermediate interactions and final completion. It targets scalable evaluation of human-hand and robot videos across rigid, articulated, deformable, compositional and bimanual tabletop tasks. e02e03e06
2.2 What this reading follows
A video can show a banana rising with a hand even when the depicted contact would not support a real grasp. RoboWM-Bench tests that missing link by recovering robot actions from generated human-hand or robotic videos and executing them in matched simulation scenes. This reading follows the two conversion pathways, the task and step success tables, and the controls that test whether the evaluation machinery is trustworthy. The central distinction is between judging the generated behavior and validating the interfaces used to judge it. Strong replay reliability on real demonstrations supports the protocol, but does not eliminate contact-sensitive errors or prove physical deployment readiness. e03e08e09e10e11e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The catalog snapshot is unassigned. This work contributes a benchmark, not a single world-action model: video generators, human retargeting, a separate IDM and an external physics simulator form the evaluation pipeline. The robot pathway uses inverse dynamics, but that does not justify assigning the benchmark a One Model or joint-prediction quadrant. e03e05
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Translate visible contact into the robot's geometry
Follow the human-hand branch from keypoints to a gripper command. HaMeR supplies a 3D hand estimate; the benchmark places the gripper at the thumb–index fingertip midpoint and orients it with a plane fitted through thumb and index keypoints. Gripper opening uses the minimum distance from the thumb tip to the other fingertips, accommodating contact that is not dominated by the index finger. Smoothing stabilizes the resulting trajectory, but it cannot make an unsupported object lift physically valid. The robot branch instead infers joint-space action chunks from consecutive images through a separately trained IDM. Thus, both branches introduce embodiment-specific interpretations between generated pixels and simulator commands. Appendix F's depth investigation further clarifies that a generated RGB future does not come with measured future depth; the final retargeting pipeline excludes estimated depth. e03e04e05e15
5.2 Validate the measuring instrument in two separate experiments
A low execution score is informative only if the evaluation machinery is sufficiently reliable. The paper addresses this with two controls that should be read separately. First, identical low-level real robot actions are executed in the corresponding reconstruction: agreement tests the scene and physics interface without introducing video-to-action estimation. Second, successful real videos pass through retargeting or IDM before simulation replay: failures now include action-recovery errors. Table 2's simulation-pretrained IDM improves replay compared with the real-only variant using the same real trajectories, supporting that training choice. The reader's remaining concern is distribution shift: successful real videos provide valid robot geometry, whereas generated videos may distort links or imply impossible contact. These controls establish useful local reliability, but cannot completely identify which component caused each generated-video failure. e05e09e10e11e13
5.3 Read success as a chain of physical requirements
Section 3.5 requires all predefined key nodes and the final objective to pass. This makes the step panels especially useful. In robotic Put in Drawer, Cosmos-FT records 60% contact but only 20% lift, with later stages and final success also at 20%. A reader can locate a grasp-to-lift bottleneck without assuming that each stage is an independent probability or multiplying the displayed rates. Figure 4 adds a different diagnostic: high video-based domain scores can accompany poor execution. Appendix J reveals what that comparison asks the video judge, while Appendix D keeps its domain and quality scores separate. Together, these observations motivate recording final success, intermediate failure location and interface reliability alongside perceptual scores. They do not establish a universal ranking of visual metrics or eliminate uncertainty from the limited evaluation configurations. e06e07e08e09e14e18
5.4 Training and inference
During training
IDM training first uses action-labeled Franka simulation trajectories with backgrounds masked to retain only the arm, then adapts to real Franka observations without masking. The reliability comparison uses 50 real trajectories per task for both real-only and simulation-plus-real variants. e05e11
Separately, Cosmos-Predict2.5 is fine-tuned on 50 real manipulation trajectories per task, producing Cosmos-Finetune/Cosmos-FT. Evaluated generators also include Veo 3.1, Wan 2.2, Wan 2.6 and, for human tasks, LVP. The source does not specify full optimization objectives, training schedules or frozen-module choices. e07e05
During inference
The reported flow generates a future video, converts it to actions, then executes and scores it; online replanning from execution feedback is not described. Ten initial object configurations are shared across models, with randomized poses, fixed evaluation cameras and standardized descriptions and seeds. e03e07
Qwen3-VL-Flash produces task-specific prompts. Appended constraints request a static camera and physically consistent hands or robot structure; human prompts also constrain visibility and unrelated object motion. These are generation inputs, not enforced simulator guarantees. e17
5.5 Implementation flow
- Build the execution scene
LeHome supports rigid, articulated and deformable objects. The real-to-sim pipeline reconstructs backgrounds with 4D Gaussian representations, obtains rigid geometry through 3D segmentation/reconstruction, constructs articulated/deformable assets, estimates object poses and calibrates camera poses with FEEPE, averaging multiple runs. e03
- Retarget human contact geometry
HaMeR reconstructs 3D hand keypoints. The gripper position is the thumb–index fingertip midpoint. A fitted thumb/index plane determines orientation: the projected fingertip line gives the x-axis and the plane normal the z-axis. Opening uses the minimum thumb-to-other-fingertip distance. Smoothing and temporal denoising stabilize trajectories. Estimated depth is excluded from the final pipeline. e04e15
- Recover robot actions
Section 3.3.2 adopts the IDM architecture from reference [23], identified as DreamGen in the bibliography on page 11. Two consecutive image frames predict the intermediate joint-space action chunk. This is a separate learned video-to-action interface, not joint future-video/action prediction inside the evaluated generator. e05
- Execute and check
Actions run in the corresponding simulator. Manually specified key nodes test meaningful contact or stable-pose stages. Overall success requires every key-node check and the final objective to pass; merely reaching an object in the generated image is insufficient. e06e13
6. Experiments & results
RoboWM-Bench evaluates whether generated manipulation videos can be converted into robot actions that complete tasks in simulation. Separate human-hand retargeting and robot inverse-dynamics interfaces expose failures hidden by plausible imagery. Reliability tests support these interfaces on real demonstrations, but execution scores still depend on action extraction, reconstructed physics and task-specific checkers.
6.1 Read the original evidence
Table 1. Task completion falls short of contact success, even when task-specific fine-tuning helps. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four panels independently: human task outcomes, human intermediate steps, robot task outcomes, and robot intermediate steps. All entries are percentages, with higher success better. For human Put on Plate, Wan 2.6 reaches 100% contact, 75% lift and 70% place. For robotic Put in Drawer, Cosmos-FT reaches 60% contact but only 20% lift and final completion. The step columns help locate where execution stops working, whereas the task panels answer whether the full objective is met. Cosmos-FT additionally receives 50 real trajectories per task; its advantage should be interpreted with that supervision difference in view. e07e08e13
What it supports. Cosmos-FT raises robotic Close Drawer success from Cosmos's 0% to 90%, and Put in Drawer from 0% to 20%. The remaining contact-to-lift gap shows why a single final success score can hide useful information about grasping failures. The human and robot panels also demonstrate substantially different performance profiles.
Where the evidence stops. The source specifies ten initial configurations, but does not explain all trial denominators or aggregation behind fractional rates such as 23% and 75%. No uncertainty intervals are supplied, and the human/robot task sets and interfaces are not matched causal controls.
Figure 4. A high video-based domain score can coexist with failed embodied execution. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In the left plots, the horizontal axis is PAI-Bench domain score and the vertical axis is RoboWM-Bench execution accuracy. Colors and markers distinguish models; the dashed diagonal is the reference line labeled y = x, not a fitted relationship. Points crowd toward high domain scores while their execution scores spread vertically. On the right, read the generated banana-lifting images above the PAI-Bench judgment, then compare the simulated gripper images below. Appendix J defines the domain test through task completion, contact, anatomical or mechanical structure, object integrity and plausible motion questions. This figure concerns domain scores; Appendix D separately reports quality scores. e09e14e18
What it supports. The example and scatter plots support the paper's claim that visual plausibility and executability supply different information. A depicted object can appear to move with a hand even when the recovered robot grasp fails. Evaluation through physical replay therefore reveals failures that this video-questioning protocol may miss.
Where the evidence stops. This is the paper's adapted PAI-Bench evaluation on RoboWM-Bench videos. It does not establish that all perceptual metrics fail. Table 6 additionally contains an unresolved Wan 2.6 average of 99.5 despite eight displayed task scores of 100.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Human Put on Plate Generated human-hand videos retargeted to robot execution in reconstructed simulation; shared initial configurations. | Wan 2.6: 70% task success; contact/lift/place: 100%/75%/70%. Task and step success (%) | Task success: LVP 70%, Wan 2.2 55%, Veo 3.1 30%, Cosmos 15%. Contact success can substantially exceed completed placement. These are robot-retargeted outcomes, not direct human-hand dynamics. e07e08e13 |
| Robotic Close Drawer Real-to-sim robot-video evaluation; Cosmos-FT receives 50 task-specific real trajectories for fine-tuning. | Cosmos-FT: 90%. Task success (%) | Cosmos: 0%; Wan 2.6: 50%; Wan 2.2: 30%; Veo 3.1: 20%. Fine-tuning improves this evaluated task; the extra supervision means this is not an equal-data generalization comparison. e07e08 |
| Robotic Put in Drawer Real-to-sim robot-video evaluation, using the same generation-to-IDM execution protocol. | Cosmos-FT: 20% task success; contact/lift/above drawer/in drawer/close drawer: 60%/20%/20%/20%/20%. Task and step success (%) | All four non-fine-tuned robot baselines have 0% task success. The contact-to-lift transition remains a bottleneck even after fine-tuning. e07e08 |
| Video-to-action replay reliability Videos from successful real manipulation demonstrations converted to actions and replayed in reconstructed simulation. | Simulation-plus-real IDM: 95.7%; human retargeting: 97.1%. Mean replay success across seven tasks (%) | Real-only IDM: 71.4%, using the same 50 real trajectories per task. Simulation pretraining improves the reported robot replay average by 24.3 percentage points (reader subtraction). Human and robot averages cover different task sets; neither measures generated-video quality. e11 |
| Real-to-sim outcome consistency Identical low-level actions from 10 successful and 10 failed real trajectories per task, across seven robotic tasks. | 10/10 in both categories for every listed task; reported averages 100%/100%. Agreement with real success and real failure outcomes | Real outcomes are the reference, not another world model. Supports outcome fidelity on these trajectories, not universal simulator accuracy or a guarantee for generated actions. e10 |
| Simulation-native complex robotic manipulation Appendix B's purely simulated generation and execution protocol. | All four models: 0% on Turn Off Faucet, Assemble Burger and Fold Clothes. Task success (%) | Wan 2.6 achieves 40% on Cut Sausage; Veo 20%, Cosmos and Wan 2.2 10%. Difficulty persists without real-to-sim reconstruction, although task complexity is not experimentally isolated. e12 |
| Bimanual human-video execution Additional human-hand tasks converted to embodied execution. | Wan 2.6: Cook 50%; Lift Large Box 60%. Task success (%) | Veo: 40%/50%; Wan 2.2 and LVP: 30%/30%; Cosmos: 10%/10%. Bimanual results broaden coverage but do not establish a controlled causal effect of coordination difficulty. e12b |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Replaying identical commands tests whether reconstruction preserves task outcomes. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left groups illustrate successful and failed real executions alongside their reconstructed-simulation counterparts. The right table separates agreement on real successes from agreement on real failures. This distinction matters: a simulator that turns every input into a success would not pass the failure column. Section 4.4.1 uses ten successful and ten failed real robotic trajectories for each listed task and replays the same low-level actions in simulation. Because the actions are held fixed, this experiment primarily probes the reconstruction and execution environment. It intentionally leaves human retargeting and robot video-to-action prediction to the separate replay test in Table 2. e10e13
What it supports. Every listed task has 10/10 agreement in both columns, giving the reported 100% success and failure consistency averages. That is useful evidence that the reconstructed scenes retain task-relevant constraints for these trajectories. It supports using simulation as an execution diagnostic within the tested setup.
Where the evidence stops. Agreement in task outcomes does not establish pixelwise, trajectory-level or universal physical fidelity. The control uses selected real robotic trajectories; its perfect agreement does not automatically extend to generated videos, unseen scenes, deformable tasks or retargeted human behavior.
Table 2. Action recovery is tested against successful real demonstrations before judging generated videos. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top block as a check of the human-hand conversion path and the lower block as a robot-IDM training comparison. Despite the caption's term accuracy, Section 4.4.2 defines the test through successful execution after recovering actions from real demonstration videos. The two IDM rows use the same fifty real trajectories per task; the lower row first receives simulation pretraining. Compare individual tasks as well as the average column: robotic Put in Drawer increases from 50% to 90%, while human Stack Cups and Pour Water remain at 90%. These residual failures are attributed to contact-sensitive grasp geometry. e05e11e13
What it supports. The robot average rises from 71.4% to 95.7%, a reader-computed increase of 24.3 percentage points. Human retargeting averages 97.1%. These results make gross interface failure a less likely explanation for every low generator score, while still showing residual replay failures near difficult contacts.
Where the evidence stops. The comparison changes simulation exposure and training work, so it does not isolate masking or motion priors alone. Replay inputs are successful real videos; reliability on structurally distorted generated videos need not match this result. Human and robot averages also cover different tasks.
Figure 8. Depth alignment improves the illustrated estimate without establishing better action replay. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row runs from ground truth to estimated depth to an error map. Row (a) uses predicted absolute depth; row (b) aligns estimated relative depth using the first frame's ground-truth depth. The color bars, hand-region boxes and mean-error annotations belong to the original figure. The top annotation is 0.7645 m and the bottom is 0.0512 m. Appendix F explains the underlying constraint: generated later frames have no measured depth. The authors tested Video Depth Anything as a substitute, but report that neither strategy improved downstream pose tracking and retargeting, so the final pipeline excludes depth information. e15
What it supports. The aligned estimate is substantially closer in this illustrated comparison, yet the authors still report no downstream benefit. The distinction is scientifically useful: reducing an intermediate depth error does not by itself establish better contact reconstruction or more successful robot execution.
Where the evidence stops. The numbers annotate the displayed diagnostic, not a reported benchmark-wide depth average. The source does not fully define their aggregation region or provide a quantitative downstream depth ablation. Better mean depth and better grasp geometry therefore remain distinct hypotheses.
7. Analysis & limitations
7.1 What the evidence leaves open
Action extraction can change contact-sensitive outcomes. Human videos are judged through robot retargeting; fingertip and gripper contacts differ. The suite is limited to selected tabletop settings, and manually authored key nodes/checkers constrain expansion. e13
Ten initial configurations are specified, but exact trial denominators and aggregation behind entries such as 23%, 57% or 75% are not explained. Tables omit uncertainty intervals; do not derive counts or significance from rounded percentages. Explicit train/test separation and overlap controls are also missing. e07e08
Domain-score tables contain unresolved average inconsistencies: Table 6 shows eight 100s for Wan 2.6 but a 99.5 average; Table 5 reports LVP's average as 87.0, whereas its displayed scores average 89.75 (reader calculation). Neither average is silently corrected. Domain and eight-component quality scores remain distinct; Appendix D evaluates RoboWM-Bench videos, not the original PAI-Bench corpus. e14
7.2 Questions for discussion
- Would model rankings survive alternative action extractors and contact-check thresholds?
- How much of the human-versus-robot gap remains with matched objects, data and embodiment?
- Can inferred depth improve grasp-region geometry without introducing temporally unstable actions?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include calibrated LeHome scenes, HaMeR retargeting, the two-stage IDM, generation prompts and per-task checkers. The paper leaves simulator/software versions, simulation-training scale, optimizer settings, smoothing parameters, action-chunk length and checker thresholds insufficiently specified for exact replication. e03e04e05e06e07e13e17
Appendix G reports representative processing only: human retargeting of 1280×720, 15-FPS video up to 25 seconds takes about 112 seconds and 5.5 GB on one RTX 4090. Robot IDM processing of 1280×720, 30-FPS, 5-second video takes about 1.5 seconds and 4 GB on one A800. These exclude a complete training or end-to-end evaluation cost accounting. e16
Proposed checks: reproduce the real-only versus simulation-pretrained IDM comparison with controlled data and compute, and test depth-assisted retargeting on held-out contact-sensitive clips. Measure action errors and execution outcomes separately to identify whether an interface improvement changes model rankings. e05e11e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate simulation exposure from extra IDM optimization
Reader-proposed check, not an experiment performed here: fix a held-out set of real Franka videos and reconstructed scenes, and keep the fifty real training trajectories per task identical across variants. Compare real-only IDM, masked-simulation pretraining plus real adaptation, and a real-only control matched for total optimization steps; add unmasked simulation pretraining to isolate masking. Use identical architecture, evaluation clips and action execution. Report paired task replay outcomes, contact-to-lift failures and joint-action error where recorded commands are available, with repeated seeds. A persistent simulation-pretraining advantage over the compute-matched control would support useful simulation supervision; disappearance of the advantage or improvement only in nominal joint error would weaken the proposed execution benefit. e05e10e11e13
Check 2: Test depth where small grasp errors change success
Reader-proposed check, not an experiment performed here: on held-out contact-sensitive Stack Cups and Pour Water demonstrations with measured depth for evaluation, compare the final depth-free pipeline, predicted absolute depth, and relative depth aligned only to the measured first frame. Hold keypoint estimation, smoothing, retargeting geometry and simulator settings fixed. Measure depth error near fingertip–object contact, gripper pose deviation and paired execution success; do not supply later measured depth to the prediction-based variants. Then repeat the available comparisons on generated clips, reporting the absence of future ground truth. If alignment reduces depth error but not replay failures, the paper's downstream observation is supported; a repeatable success gain would motivate revisiting the depth-free choice. e04e11e15
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all eight numbered figures and all twelve tables, plus the method, evaluation, limitation, depth, compute, prompt and VQA pages supporting retained claims. Page 5's architecture attribution to [23] was checked against the visible DreamGen bibliography entry on page 11; both pages are included for verification. The six final original crops were separately viewed with readable labels and complete relevant boundaries. Figure 2's two paths were checked against its caption and Section 3.3; Figure 4's axes distinguish domain from quality scores; Figure 8's error values were assigned from the original row labels. Pages 10, 12 and 13 were read as text only and supply no unique method or numerical claim in this edition. Separate supplementary videos, code and cited external papers remain uninspected. Figure 5 and Table 2 are reliability diagnostics; no new experiments are implied. The Table 5 and Table 6 average inconsistencies remain explicit in the base evidence record.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1 Introduction
- 2 Related Work, including 2.1–2.3
- 3 RoboWM-Bench, including 3.1–3.5 and both video-to-action pathways
- 4 Experiments, including 4.1–4.4.2 and 4.3 Perceptual Plausibility vs. Embodied Executability
- 5 Conclusion
- References
- Appendices A–J, including additional experiments, depth, compute, impacts, prompts and VQA questions
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The inspected title page identifies arXiv:2604.19092v2 [cs.RO], 14 May 2026. Its title and all eleven authors match the catalog. The catalog submission date is 21 April 2026; this report reviews v2, and changes from the earlier version were not supplied or compared.
- The supplied extraction does not reconstruct figure images. Original PDF figures and tables were therefore inspected separately; all six final crops were also viewed.
- Separate supplemental material availability has not been fully verified. No separate supplementary videos or other external material were inspected.
- Code was not inspected and no experiments were reproduced. Appendix G's statement about released evaluation scripts is an author statement, not independently verified availability.
- All six supplied text chunks, covering all 24 PDF pages, were read. Visual inspection covered pages 1–9, 11 and 14–24, including reference [23] on page 11 to verify the IDM's DreamGen attribution. The conclusion continuation and remaining bibliography on pages 10, 12 and 13 were read as text. Cited external papers were not separately inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin
The exact catalog title and eleven authors are visible; affiliations are Peking University, Tsinghua University and Lightwheel. The artifact is arXiv:2604.19092v2 [cs.RO], dated 14 May 2026.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1; p. 2, Figure 1
The benchmark motivates execution-grounded evaluation of predicted human and robotic manipulation, using retargeting or inverse dynamics and simulation validation.
Go to primary source ↓e03PDF p. 4, Figure 2 and Sections 3.1–3.3
Separate scene-reconstruction and video-to-action paths feed embodied validation. LeHome supports rigid, articulated and deformable objects; backgrounds use 4D Gaussians, with object reconstruction, pose estimation and FEEPE calibration.
Go to primary source ↓e04PDF p. 5, Section 3.3.1
HaMeR keypoints determine a thumb–index midpoint, contact-plane orientation and minimum thumb-to-fingertip gripper opening, followed by smoothing and temporal denoising.
Go to primary source ↓e05PDF p. 5, Section 3.3.2; p. 11, References, entry [23]
Section 3.3.2 explicitly adopts the IDM architecture from [23]; the bibliography identifies [23] as Jang et al., Dreamgen: Unlocking generalization in robot learning through video world models (2025). The IDM maps consecutive image pairs to intermediate joint-space action chunks; training uses masked simulation Franka data then unmasked real Franka data. This attribution is verified within RoboWM-Bench; the cited DreamGen paper was not separately inspected.
Go to primary source ↓e06PDF pp. 5–6, Sections 3.4–3.5
The suite spans object types, horizons and coordination demands. Key-node interaction checks and achievement of the final objective jointly define task success.
Go to primary source ↓e07PDF p. 6, Sections 4.1.1–4.1.2
Evaluation uses 10 shared initial configurations, fixed cameras and standardized descriptions/seeds. Listed baselines include Veo 3.1, Wan 2.6, Wan 2.2, Cosmos-Predict2.5 and LVP; Cosmos fine-tuning uses 50 real trajectories per task.
Go to primary source ↓e08PDF p. 7, Table 1, all four panels
Human Put on Plate task rates are 15/55/30/70/70 for Cosmos/Wan 2.2/Veo 3.1/Wan 2.6/LVP. Robot Cosmos-FT achieves 90% Close Drawer and 20% Put in Drawer; the latter has 60/20/20/20/20 step rates, while all other robot models have 0% final success.
Go to primary source ↓e09PDF p. 7, Sections 4.2–4.3; p. 8, Figures 3–4
Qualitative cases show unsuccessful contact and robot geometry distortion. Figure 4 juxtaposes high PAI-Bench domain scores with variable execution and a banana that appears held but is not securely grasped during replay.
Go to primary source ↓e10PDF pp. 8–9, Section 4.4.1; p. 9, Figure 5 consistency table
Identical real actions are replayed in reconstruction. Each of seven listed tasks has 10 successful and 10 failed real trajectories; all outcomes agree, yielding reported 100% success and failure consistency.
Go to primary source ↓e11PDF p. 9, Table 2 and Section 4.4.2
Successful real-video replay averages are 97.1% for retargeting, 71.4% for real-only IDM and 95.7% for simulation-plus-real IDM. Both IDM variants use the same 50 real trajectories per task; remaining failures are described as contact-sensitive cases.
Go to primary source ↓e12PDF p. 14, Appendix B and Table 3
Four models all score 0% on simulated Turn Off Faucet, Assemble Burger and Fold Clothes. Cut Sausage rates are Cosmos 10%, Wan 2.2 10%, Wan 2.6 40% and Veo 20%.
Go to primary source ↓e12bPDF pp. 14–15, Appendix C; p. 15, Table 4
For Cook/Lift Large Box, Wan 2.6 scores 50%/60%, Veo 40%/50%, Wan 2.2 and LVP 30%/30%, and Cosmos 10%/10%. Step columns provide bimanual interaction diagnostics.
Go to primary source ↓e13PDF p. 14, Appendix A
Authors acknowledge intermediate-interface errors, especially near contact boundaries; robot retargeting does not directly simulate human-hand dynamics. The suite is nonexhaustive and key-node checkers are manually specified.
Go to primary source ↓e14PDF pp. 14–16, Appendix D; p. 15, Figure 6; p. 16, Tables 5–8, especially Table 5 LVP row and Table 6 Wan 2.6 row
Quality-score scatter clusters near 0.78 while execution varies. Domain and quality tables use videos generated for this benchmark. Table 6 displays eight 100s for Wan 2.6 and an average of 99.5. Table 5 displays LVP scores 100, 78, 100, 78, 100, 100, 86 and 76, with a reported average of 87.0; their arithmetic mean is 89.75 (reader calculation). Both mismatches are unresolved.
Go to primary source ↓e15PDF p. 18, Appendix F and Figure 8 panels (a)–(b)
Only initial-frame ground-truth depth is available. Video Depth Anything absolute prediction and first-frame-aligned relative prediction are tested; example mean errors are 0.7645 m and 0.0512 m respectively. Authors report no downstream retargeting improvement and omit depth from the final pipeline.
Go to primary source ↓e16PDF p. 18, Appendix G
Representative human processing uses one RTX 4090, about 112 s and 5.5 GB for 1280×720/15-FPS/up-to-25-s video. IDM uses one A800, about 1.5 s and 4 GB for 1280×720/30-FPS/5-s video. Evaluation scripts are described as released.
Go to primary source ↓e17PDF pp. 19–23, Appendix I.1–I.2 and Tables 9–12
Qwen3-VL-Flash generates scene-conditioned task prompts. Human prompts constrain hand visibility, camera and unrelated motion; robot prompts constrain gripper/link geometry and camera. Tables provide tasks, initial images and prompt examples.
Go to primary source ↓e18PDF pp. 23–24, Appendix J.1–J.2
PAI-Bench domain evaluation uses five targeted VQA questions per suite: task completion, contact, hand/robot structure, object integrity and plausible object motion. Only the first is task-dependent; all list yes as the target answer.
Go to primary source ↓8.5 Primary sources
RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation ↗
PDF · 10,328 extracted words
Source fingerprint
4450ceffa72850511da6c91b3ce6f73bc070d372104dc8f60ce6fbf8f84ed566