Interactive World Simulator for Robot Policy Training and Evaluation
1. Paper overview
In one sentence: A separately learned visual simulator can supply demonstrations and rank robot policies, but its usefulness depends on the fidelity and coverage of its task-specific interaction data. e02e03e07e08e13e15
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | Real-world imitation learning from simulator demonstrations: Simulator-only: DP 87.9%; ACT 76.2%; π₀.₅ 73.1%. Average normalized task score, reported as percent. Four real tasks, 100 demonstrations per mixture, ten evaluations per task/policy/mixture; Figure 5 also includes a separate MuJoCo task evaluated 100 times. Real-only: DP 90.3%; ACT 73.6%; π₀.₅ 88.8%. DP and ACT support useful transfer from synthetic demonstrations. π₀.₅ improves by 15.7 percentage points with real-only data, qualifying the broad parity claim. Scores combine partial-credit tasks and are not uniform binary success rates. e12e13 |
| Reading caution |
Core contributions
Figure 2. Learn a renderer first, then predict action-conditioned futures in its frozen latent space. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left: the CNN encoder maps an RGB image into a spatial latent, while the consistency decoder receives both that latent and a noisy image. The middle panel advances time by appending a noisy latent, predicting it, and recycling the result as history. Decoder snowflakes mark frozen weights; the caption and Section III-B specify that the encoder is frozen too. At right, action and noise-level embeddings enter FiLM alongside convolutional features before spatial and temporal attention. Cross-checking these arrows with Equations (2)–(5) confirms that actions condition visual prediction; the diagram does not show an action-output branch. e03e04e05e06e07
What it supports. The architecture separates the image reconstruction problem from temporal prediction. Once the representation is fixed, dynamics training can operate on compact latents, and generated images can provide feedback to an external controller. Context corruption is the authors’ additional strategy for tolerating errors that accumulate when predictions become future inputs.
Where the evidence stops. The schematic does not specify the actual context length, layer count, noise schedule, or denoising step count. Its frozen modules and conditioning arrows explain the information flow, but they do not establish which component causes the reported stability.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations require hardware and operator time; repeatable physical policy evaluation also requires controlled resets. The paper asks whether a learned RGB simulator can provide both services while remaining fast and stable during feedback-driven interaction. e02e07
2.2 What this reading follows
Imagine collecting a mug-grasping demonstration while the mug exists only in generated images. Interactive World Simulator makes that interface possible by learning how observations change under robot actions. Its encoder compresses images, its dynamics model predicts the next latent state, and its decoder renders feedback for a human or a separate policy. The paper then asks two practical questions: can demonstrations collected this way train a real robot, and can simulated evaluations identify good policies? The results are promising within the tested setups, but the distinctions between visual accuracy, transferred task performance, and reliable policy ranking matter throughout this reading. e02e03e07e08e13e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational neural-simulator and evaluation-protocol labels fit. Although the simulator has multiple modules, the catalog architecture/prediction quadrants concern world–action policy mechanisms: here actions condition future observations and come from separate humans or policies. This supports the recorded Not applicable fields, rather than assigning One Model, joint future/action prediction, or inverse dynamics. e03e05e07e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why learn reconstruction before dynamics?
The model first needs a representation that can return to pixels. Its CNN encoder preserves a spatial latent grid, while the decoder learns to remove image noise conditioned on that grid. Equations (1)–(3) describe the same underlying noise applied at a higher and a lower scale, with the lower-noise image used as the regression target. After this stage, both appearance modules are frozen. The dynamics learner therefore predicts within an established representation rather than changing the representation and transition model simultaneously. That division is a useful interpretation of the training design, not a measured ablation benefit. Figure 2’s snowflakes and Section III-B agree on freezing. The paper supplies the loss structure but leaves the exact noise schedule, loss weighting and model dimensions unspecified. e03e04e05
5.2 How one predicted latent becomes the next interaction
At inference, encode the starting image and append a noisy latent to the available history. The action-conditioned consistency model denoises that appended frame, which then joins the context for another prediction. Older context is discarded beyond a threshold, keeping the history used at each step bounded. The decoder renders the new latent so a human or policy can respond. This feedback loop explains why the authors inject small noise into training contexts: a rollout will eventually condition on its own imperfect predictions. Robustness to that corruption is the proposed bridge between next-frame supervision and sustained interaction. Actions still come from the operator or a separate policy; the world model does not infer motor commands from a visual goal. During simulator teleoperation, the displayed manipulation occurs in generated observations. e05e06e07
5.3 Three empirical questions require three different controls
The video benchmark compares predictions with ground-truth recordings under matched actions and 192-step horizons. It tests visual agreement. The mixture experiment instead holds the policy dataset at 100 demonstrations and changes its real/synthetic composition, then evaluates task performance. That tests whether generated demonstrations can teach a useful controller, while retaining the world model’s original training-data cost. Finally, the correlation experiment keeps policies and initial configurations matched across real and simulated evaluation, testing whether simulator scores track policy quality. These controls answer different questions: better PSNR alone cannot establish policy transfer, and high correlation alone cannot establish unbiased scores. The π₀.₅ mixture gap and positive evaluation bias demonstrate why those distinctions matter. Likewise, the ten-minute interaction claim should remain separate from the much shorter quantitative video benchmark. e08e09e12e13e14e15e16
5.4 Training and inference
During training
World-model data cover six real ALOHA tasks: mug grasping, rope routing, rope collecting, T pushing, box packing and pile sweeping, plus MuJoCo T pushing. Each real task uses about 600 play episodes of 200 steps; MuJoCo uses 10,000 scripted random-interaction episodes. Default images are 128×128. Some controls are restricted, including planar bimanual T pushing. e08
Collection takes about six person-hours per real task. Typical stage-1/stage-2 training takes six/twelve hours, respectively, on one H200 each. The mug-grasping model occupies 176.02 MB; this is a storage size, not a parameter count. e08
During inference
Encode the initial observation, append a noisy latent, denoise it under recent actions and latent context, and append the prediction to the history. Drop old latents after a context threshold, keeping per-step context bounded. Decode predictions for human or policy feedback. Training-time context corruption is intended to tolerate prediction errors recycled during this loop. e05e06
5.5 Implementation flow
- Learn a spatial representation and renderer
A CNN encoder E_ϕ maps an image to a compact 2D latent. A consistency decoder D_θ reconstructs the image conditioned on that latent. Training maps a higher-noise image to a lower-noise target using the same underlying noise; CTM motivates this alternative to unstable one-step consistency training. e03e04
- Freeze appearance modules and learn dynamics
Freeze both encoder and decoder, then train F_ψ on latent sequences and action histories. Fully noise the final latent and lightly corrupt observation context. Conv3D features receive action and noise-level conditioning through FiLM, followed by spatial and temporal attention. The authors motivate stochastic prediction by multimodal interaction outcomes. e03e05
- Connect a human or policy to the simulator
The action source remains external. A human can teleoperate generated observations to collect demonstrations; during policy evaluation, a separately trained policy consumes generated frames and supplies the next actions. The world model predicts consequences, with no inverse-dynamics action extraction or joint action-generation head described. e07
6. Experiments & results
Interactive World Simulator learns action-conditioned visual dynamics from robot play data, then acts as an interactive environment for collecting demonstrations and evaluating separate policies. Consistency models decode images and predict latent futures. The evidence supports useful simulation within the studied task distributions, with less support for broad equivalence to real demonstrations or unrestricted physical fidelity.
The supplied paper has no component-removal ablation of consistency decoding, latent dynamics or context noise. Figures 3 and 6 serve as qualitative and demonstration-count diagnostics, not substitutes for that missing experiment. Section IV-B and Table I refer per-task video results to an appendix/supplement, but the twelve-page PDF ends with references and contains none. This edition therefore preserves the available aggregate table and six original visuals without inventing per-task tables or mechanism-isolating results. e05e10e14e18
6.1 Read the original evidence
Table I. Across-task video metrics favor IWS under the reported 192-step comparison protocol. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row using its arrow: lower is better for MSE, LPIPS, FID and FVD; higher is better for PSNR, SSIM and UIQI. Compare columns within a row rather than comparing numeric magnitudes across metrics. The table aggregates six real tasks and one MuJoCo task, with 192-step rollouts from matched initial conditions. It therefore summarizes visual prediction, not the later policy-training experiments. Section IV-B also matters: DINO-WM and community Dreamer4 train from scratch, UVA uses pretrained components, and Cosmos is finetuned. These are the paper’s baseline recipes, without a reported matched training-compute budget. e08e09e10e18
What it supports. IWS reports PSNR 25.82 ± 2.72 versus the strongest baseline mean, Dreamer4’s 20.81 ± 2.21. Its FVD is 243.20 ± 103.58 versus Cosmos’s 799.34 ± 220.07. Every displayed metric favors IWS, supporting better aggregate visual agreement under this protocol rather than directly measuring robot task completion.
Where the evidence stops. The source does not define what the ± values summarize, so they should not be relabeled as confidence intervals or standard deviations. Per-task results are referenced to an appendix/supplement that is absent from the supplied PDF.
Figure 5. Keep demonstration count fixed and vary where the demonstrations come from. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each group contains four policy colors: blue DP, orange ACT, green π₀ and purple π₀.₅. Read the paired horizontal labels carefully: Real % rises from left to right while WS % falls. Every training set still contains 100 episodes, so this experiment changes data source rather than dataset size. The first panel is MuJoCo T pushing; four panels cover real tasks, and the last aggregates the real tasks. The vertical scale is task score, including partial credit. The plotted shapes represent Bayesian posteriors under a uniform Beta prior, based on 100 evaluations per MuJoCo point and ten per real point. e08e12e13
What it supports. The accompanying text reports simulator-only versus real-only average scores of 87.9% versus 90.3% for DP and 76.2% versus 73.6% for ACT. Those endpoints support practical transfer. For π₀.₅, however, the corresponding 73.1% and 88.8% indicate a substantial real-data benefit, so parity should not be generalized across all policy families.
Where the evidence stops. The reported real-world evaluation is small per data point, and the paper does not establish statistical equivalence. Simulator-only demonstration training still relies on a world model trained from interaction data; it does not eliminate that initial collection cost.
Figure 7. Simulator evaluation tracks broad performance differences, while bias and nearby-policy ambiguity remain. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is real-world task score and the vertical axis is the corresponding world-simulator score. Each point summarizes one policy/checkpoint evaluated from the same twenty initial configurations in both environments; colors distinguish DP, ACT, π₀ and π₀.₅. These initial configurations come from the simulator’s training distribution. Inspect both the point ordering and the fitted trend rather than reading the correlation label alone. The paper identifies the horizontal and vertical bars as Clopper–Pearson confidence intervals. T pushing appears at upper left, rope routing at upper right, mug grasping at lower left, and pile sweeping at lower right. e12e15
What it supports. The reported correlations are 0.8553, 0.8455, 0.8869 and 0.9908 for those four tasks, respectively. They support using simulator results to separate broadly different policy quality within the studied distribution. The authors also observe a positive simulator bias outside T pushing, so agreement in relative performance can coexist with optimistic absolute scores.
Where the evidence stops. The confidence level and the construction of intervals for normalized partial-credit scores are not detailed. Correlation across pooled policy families and checkpoints does not guarantee reliable ordering of close candidates or generalization to unseen initial-state distributions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Action-conditioned video prediction Six real tasks plus MuJoCo T pushing; matched initial conditions and 192-step/19.2-second rollouts. Results aggregate tasks; evaluation split and rollout count are unspecified. | 25.82 ± 2.72; 0.051 ± 0.019; 243.20 ± 103.58 PSNR ↑; LPIPS ↓; FVD ↓ | Strongest baseline means: Dreamer4 PSNR 20.81 ± 2.21 and LPIPS 0.163 ± 0.052; Cosmos FVD 799.34 ± 220.07. All seven Table I metrics favor IWS. The ± statistic is undefined. These video metrics measure visual agreement, not executed robot success; baseline pretraining and training procedures differ. e09e10e11 |
| Real-world imitation learning from simulator demonstrations Four real tasks, 100 demonstrations per mixture, ten evaluations per task/policy/mixture; Figure 5 also includes a separate MuJoCo task evaluated 100 times. | Simulator-only: DP 87.9%; ACT 76.2%; π₀.₅ 73.1%. Average normalized task score, reported as percent | Real-only: DP 90.3%; ACT 73.6%; π₀.₅ 88.8%. DP and ACT support useful transfer from synthetic demonstrations. π₀.₅ improves by 15.7 percentage points with real-only data, qualifying the broad parity claim. Scores combine partial-credit tasks and are not uniform binary success rates. e12e13 |
| Demonstration-count scaling in MuJoCo T pushing ACT and DP; 5, 10, 20, 40 or 100 demonstrations from MuJoCo or the world simulator; 100 random evaluation trials. | Broad improvement with more demonstrations; exact curve values are not tabulated. Task-score curves | World-simulator-trained policies remain competitive with MuJoCo-trained policies. Figure 6 supports a general trend, with visible dips and plateaus. It does not establish monotonic improvement or scaling of world-model training data. e08e14 |
| Simulator-to-real policy evaluation correlation DP, ACT, π₀ and π₀.₅ trained on real data; intermediate/final checkpoints; twenty matched initial configurations per task from the simulator training distribution. | T pushing 0.8553; rope routing 0.8455; mug grasping 0.8869; pile sweeping 0.9908. Reported correlation coefficient r | Normalized scores from the same policies in the simulator and real world; Clopper–Pearson intervals shown without a stated confidence level. Supports relative evaluation within the tested distribution. Positive simulator bias outside T pushing limits absolute calibration; correlation does not guarantee the ordering of closely matched policies. e12e15 |
| Interactive long-horizon simulation Author-reported operation on one RTX 4090. | 15 FPS; more than ten minutes. Rendering throughput and stable interaction duration | Quantitative baseline rollouts last 192 steps, a separate protocol. An author-reported capability; the supplied PDF does not give ten-minute aggregate prediction-error statistics. e09e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Selected rollouts expose different ways a visually plausible simulator can lose the interaction. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left half horizontally to follow pile sweeping from the initial frame through the displayed late steps. Then compare vertically against the ground-truth row, keeping robot pose and swept objects in view. The right half fixes the terminal step and changes the task: box packing, rope routing, T pushing, rope collecting, and mug grasping. The red circles and labels belong to the original figure. They direct attention to dynamics errors for Cosmos, robot-position errors for UVA, artifacts for Dreamer4, and missing detail for DINO-WM. This is a qualitative diagnostic comparison of complete systems, not a component-removal experiment. e11e09e05e16e18
What it supports. In these selected examples, IWS preserves robot–object relationships more closely than the displayed baselines. The grid helps interpret why a better aggregate video score may matter to a controller: wrong robot position or disappearing objects can change the observation on which its next action depends. That causal consequence is reader analysis.
Where the evidence stops. These are selected frames from short benchmark rollouts, not a failure-frequency estimate or a ten-minute quantitative test. The figure cannot isolate context corruption, consistency training, or the encoder as the cause of improvement.
Figure 6. The demonstration-count diagnostic asks whether synthetic data remain useful as policy datasets grow. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use color and shade together: the two blue series are DP, and the two orange/brown series are ACT; lighter and darker shades distinguish MuJoCo from world-simulator demonstrations as shown in the legend. The horizontal labels give five dataset sizes, from 5 to 100 episodes. At each size, compare sources for the same policy before comparing policies. All are evaluated over 100 random trials. The experiment belongs to the MuJoCo T-pushing setting and varies the imitation policy’s demonstration budget. It does not vary how much interaction data were used to train the world model itself. e08e14e18
What it supports. Both sources support higher policy scores in the larger-data regime than at the smallest dataset size, and simulator demonstrations remain useful throughout the displayed range. The synthetic-data series are not uniformly worse than their MuJoCo counterparts. This supports a limited demonstration-scaling result without establishing a general scaling law.
Where the evidence stops. The caption describes consistent improvement, but individual curves visibly dip or plateau. Read the conclusion as an overall trend, not strict monotonicity. This diagnostic also cannot substitute for an ablation of the world model’s architecture or training losses.
7. Analysis & limitations
7.1 What the evidence leaves open
The experiments cover constrained task setups and mostly low-resolution images. Correlation is tested on initial configurations from the training distribution; novel objects, viewpoints and unrestricted controls remain unestablished. e08e15
No component ablation isolates the decoder, consistency dynamics or context corruption. The per-task appendix is absent, preventing inspection of task-specific metric variation. The authors leave environment diversity and world-model data/compute scaling to future work. e18e17
7.2 Questions for discussion
- Does context corruption improve long-horizon task fidelity when one-step reconstruction quality is controlled?
- How often does simulator checkpoint selection choose a worse real-world policy, especially outside the training distribution?
8. Reproducibility audit
8.1 Requirements and known gaps
Preserve task scoring: T pushing uses maximum target IoU within 600 steps; rope routing counts threaded clips, mug grasping awards grasp/place points, and pile sweeping counts pieces in the tray within 200 steps. Match initial states and distinguish video, mixture-training and policy-correlation protocols. e12e15
A faithful implementation needs the latent dimensions, history length, optimizer and policy-training settings, noise schedules and context-noise magnitude, loss weights, denoising steps, action encoding, count-score normalization, and dataset split manifest, which this PDF does not fully specify. Baseline comparisons also need the exact community Dreamer4 implementation and pretrained UVA/Cosmos checkpoints. e04e05e06e08e09e12e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate the contribution of noisy history training
Reader-proposed check, not performed: use MuJoCo T pushing, a fixed frozen autoencoder, an identical dynamics architecture, and the same interaction train/test episodes. Train matched dynamics runs with zero context noise and with several explicitly documented small noise levels, leaving final-frame corruption, optimization, and compute unchanged. Because the paper omits its noise magnitude, this would test the mechanism rather than claim an exact reproduction. Measure one-step error, the reported 192-step video metrics, and object-pose/target-IoU drift over longer matched action sequences; also compare fixed-policy task scores. The hypothesis predicts slower rollout degradation with noisy contexts, beyond any one-step improvement. Equal or worse long-horizon behavior across matched runs would weaken that explanation of stability. e04e05e06e08e09e12e18
Check 2: Test whether simulator checkpoint selection survives distribution shift
Reader-proposed check, not performed: freeze the simulator and compare the same intermediate and final policy checkpoints on paired real/simulated trials. First match the paper’s twenty-configuration, in-distribution protocol; then predefine a second set of held-out object placements outside the training range while preserving the task and action interface. Keep the paper’s task scores and horizons, documenting score normalization and uncertainty construction. Report signed score bias, pairwise policy-order agreement, and the real-world score lost by choosing the simulator’s best checkpoint instead of the real-world best. Use paired resampling over initial configurations. Strong pooled correlation accompanied by frequent reversals or poor selected-checkpoint performance would falsify the stronger interpretation that correlation alone makes simulator-based selection dependable. e08e12e15
8.3 Reading coverage
Visual audit: All twelve PDF pages were rendered and visually inspected, including title/authors/version (p. 1), motivation (pp. 1–2), architecture and equations (pp. 3–4), task/data/compute and evaluation details (pp. 5–6), Table I and Figure 5 (p. 7), Figures 6–7 and correlation protocol (p. 8), conclusion (p. 9), and the final references (pp. 9–12). Figures 1–7 and Table I were inspected. All six final original crops were separately viewed, retaining labels, legends and uncertainty bars. Figure 2 arrows and freezing markers were checked against its caption and Section III-B. No component-ablation page or appendix exists in this supplied PDF; the referenced separate per-task results and project videos remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations, version stamp and Abstract (p. 1)
- I. Introduction (pp. 1–2)
- II. Related Works, A–C (pp. 2–3)
- III. Method, A–D, Equations (1)–(5) (pp. 3–5)
- IV. Experiment, A–D (pp. 5–8)
- V. Conclusion (pp. 8–9)
- Acknowledgement and References (pp. 9–12)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only image limitation above was resolved by inspecting all twelve PDF pages and six original crops; all five supplied text chunks were read in full.
- The reviewed edition is arXiv:2603.08546v1, dated 9 March 2026. Its exact title and ten authors match the catalog. No other revision was supplied; the catalog BibTeX venue, RSS Workshop, is not established by this PDF.
- The per-task appendix/supplement referenced in Section IV-B and Table I is not present in the supplied PDF. No separate supplement was supplied.
- Project videos, code, datasets and external links were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin stamp
The title is Interactive World Simulator for Robot Policy Training and Evaluation. The ten authors match the catalog in order. The artifact is arXiv:2603.08546v1 [cs.RO], dated 9 March 2026; the institutions are Columbia University, Toyota Research Institute, Amazon, and University of Illinois Urbana-Champaign.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section I
The paper motivates fast action-conditioned simulation by the cost of robot demonstrations and real-world policy evaluation, presenting synthetic demonstration collection and policy evaluation as its two applications.
Go to primary source ↓e03PDF p. 3, Figure 2 and caption; p. 4, Section III-B
A CNN encoder and consistency decoder form the first-stage autoencoder. Both are frozen for second-stage latent dynamics training. Figure 2 shows action/time embeddings modulating Conv3D features through FiLM, followed by spatial and temporal attention; decoder snowflakes denote freezing.
Go to primary source ↓e04PDF p. 4, Section III-B.1, Equations (1)–(3)
The encoder produces a spatial latent. The same noise is applied at two image noise scales, sigma_t greater than sigma_s, which is nonnegative. The conditional decoder regresses the lower-noise target using a noise-weighted squared loss. CTM motivates the design; exact sampling schedules and weights are not supplied.
Go to primary source ↓e05PDF p. 4, Section III-B.2, Equations (4)–(5) and context-noise paragraph
The frozen autoencoder supplies latent sequences. The dynamics consistency model conditions on action history, receives full noise only on the last frame, and uses weighted latent regression. Small noise is injected into observation contexts to improve robustness to generated histories. No noise magnitude or component ablation is given here.
Go to primary source ↓e06PDF p. 4, Section III-B.3
Inference encodes the initial image, appends a noisy latent, denoises a new frame conditioned on history and actions, recycles predicted latents, discards old context above a threshold, and decodes predicted latents. The threshold and denoising step count are not numerically specified.
Go to primary source ↓e07PDF p. 4, Sections III-C–D; p. 5, continuation of III-D; p. 6, Figure 4 and Section IV-C teleoperation paragraph
A human supplies keyboard or kinematic teleoperation actions to collect simulator observation/action demonstrations for separate imitation policies. In evaluation, policy actions drive the world model and predicted images feed back to the policy. Figure 4 depicts a simulated mug interaction, not manipulation of a real mug on that table.
Go to primary source ↓e08PDF p. 5, Section IV-A; p. 6, Section IV-A continuation
Six real tasks use the ALOHA bimanual setup, alongside MuJoCo T pushing. World-model training uses about 600 real play episodes per task, 200 steps each, or 10,000 scripted MuJoCo episodes. Default images are 128×128; some action spaces are constrained. Collection takes about six person-hours per real task. The mug model is 176.02 MB; training typically takes six hours for stage 1 and twelve for stage 2, each on one H200.
Go to primary source ↓e09PDF p. 6, Section IV-B, comparison protocol and baseline-training paragraphs
Video comparisons use all seven tasks, matched initial conditions, and 192-step (19.2-second) rollouts. DINO-WM and community Dreamer4 train from scratch; UVA starts from a pretrained VAE and MAR model; Cosmos uses default finetuning. Video evaluation sample counts, split details, and matched compute budgets are not specified.
Go to primary source ↓e10PDF p. 7, Table I, all rows and caption
Across-task aggregates favor IWS on all seven metrics. IWS PSNR is 25.82±2.72 versus Dreamer4 20.81±2.21; LPIPS is 0.051±0.019 versus Dreamer4 0.163±0.052; FVD is 243.20±103.58 versus Cosmos 799.34±220.07. These are the strongest baseline means for the respective metrics. The caption does not define the ± statistic and refers per-task results to supplementary material.
Go to primary source ↓e11PDF p. 5, Figure 3, panels (a)–(b) and caption
The selected grid compares pile-sweeping frames through step 190 and other tasks at step 191. Author annotations identify Cosmos dynamics errors, UVA robot-position errors, Dreamer4 artifacts, and missing DINO-WM details. IWS examples visually track the ground-truth interactions more closely.
Go to primary source ↓e12PDF p. 6, Section IV-C task definitions; p. 7, Figure 5 and evaluation paragraph
Mixture experiments keep 100 demonstrations per dataset and compare DP, ACT, pi0, and pi0.5 on MuJoCo T pushing and four real tasks. Evaluation uses 100 trials per MuJoCo point and ten per real point. Figure 5 represents task-score Bayesian posteriors under a uniform Beta prior. Scores are maximum T-block target IoU within 600 steps; rope clips threaded, mug grasp/place points, or pile pieces swept within 200 steps.
Go to primary source ↓e13PDF p. 7, Section IV-C mixture-results paragraph and Figure 5, Average [Real] panel
Reported simulator-only versus real-only average task scores are DP 87.9% versus 90.3%, ACT 76.2% versus 73.6%, and pi0.5 73.1% versus 88.8%. The paper describes overall comparability while acknowledging pi0.5 improves as real data increase.
Go to primary source ↓e14PDF p. 7, Section IV-C final paragraph; p. 8, Figure 6 and continuation
ACT and DP are trained using entirely MuJoCo or entirely world-simulator demonstrations, at 5, 10, 20, 40, and 100 episodes, and evaluated over 100 random trials. The curves broadly improve, although individual displayed series have dips or plateaus. Exact curve values are not tabulated.
Go to primary source ↓e15PDF p. 8, Section IV-D and Figure 7, all panels
Four policies trained on real data are evaluated at final and intermediate checkpoints from twenty matched initial configurations per task sampled from the simulator training distribution. Figure 7 plots normalized real versus simulator scores with Clopper–Pearson intervals: r=0.8553 for T pushing, 0.8455 for rope routing, 0.8869 for mug grasping, and 0.9908 for pile sweeping. The text notes positive simulator bias except for T pushing; the confidence level is not specified.
Go to primary source ↓e16PDF p. 1, Abstract and Figure 1 caption; p. 6, Section IV-B final paragraph
The authors report stable interactions lasting more than ten minutes at 15 FPS on one RTX 4090. The quantitative baseline benchmark is separately specified as 192 steps; long video examples are referred to the project website.
Go to primary source ↓e17PDF p. 9, Section V continuation, future-work paragraph
Future directions include more diverse environments, more complex tasks, and studying world-model scaling with interaction data and compute.
Go to primary source ↓e18PDF pp. 5–9, Section IV, Figures 3–7, Table I and transition to Section V; pp. 9–12, References
The supplied experimental body contains video comparisons, demonstration mixtures, demonstration-count scaling, and policy correlation, but no component-removal ablation. Section IV-B and Table I refer to per-task appendix/supplement results; this PDF instead ends with references and contains no appendix.
Go to primary source ↓8.5 Primary sources
Interactive World Simulator for Robot Policy Training and Evaluation ↗
PDF · 8,357 extracted words
Source fingerprint
6038aefdcdd3ec7b6e50304c32438e98cd106ba653f282af78f30c11bef39f22