Ctrl-World: A Controllable Generative World Model for Robot Manipulation
1. Paper overview
In one sentence: Ctrl-World makes existing robot policies interact with generated multi-view observations, enabling targeted improvement from selected synthetic successes while retaining a substantial gap in precise physical execution. e02e03e08e12e13e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Generalist manipulation policies require expensive physical rollouts for evaluation and additional corrective demonstrations for improvement. Ctrl-World seeks to move both processes into generated experience, requiring camera compatibility, precise action response and memory of scene content that disappears from view. e02 |
| Core mechanism | Source description A multi-view, action-conditioned video simulator combines sparse visual memory with pose embeddings, allowing an external policy to repeatedly consume predicted observations. e03 |
| A key reported result | Post-training π0.5-DROID on synthetic successes: 38.7% → 83.4%, an increase of 44.7 percentage points. Reported average downstream success rate. Downstream spatial, size/shape, directional towel-folding and novel-object tasks; base versus synthetic-data-fine-tuned policy. Figure 9 labels category pairs 0.29→0.88, 0.44→0.91, 0.57→0.80 and 0.25→0.75. The source calls this a 44.7% improvement; the endpoints establish an absolute percentage-point gain. It supports targeted instruction-following improvement. Post-training trial counts, test split and plotted-whisker definition are not provided. e08e13e14 |
| Reading caution | Author claim The authors report inaccurate collisions, sliding and rotations, missing retries, sensitivity to the initial observation, and failures on precise interactions or long-horizon reasoning. They do not expect the present model to improve low-level success on familiar instructions. e14 |
Core contributions
- Source description
A multi-view, action-conditioned video simulator combines sparse visual memory with pose embeddings, allowing an external policy to repeatedly consume predicted observations. e03
Figure 2. Pose-associated history and future action poses turn video denoising into a controllable prediction step. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the top timeline: selected historical observations accompany noisy future observations. At left, three camera views contribute tokens to each frame; the legend defines P as the product of camera count N, latent height H and latent width W. Follow the green historical poses and red future action poses into the spatial transformer. The enlarged panel at right aligns each frame with its own pose embedding. The caption and Section 4.1 explain that visual tokens attend to those embeddings; the double-headed schematic arrows should not be interpreted as a separate bidirectional dynamics model. Temporal processing then carries information between frames, and generated futures feed the next interaction. e02e03e06
What it supports. The mechanism supplies complementary information: multiple views reveal scene content, history preserves earlier appearances, and pose conditioning ties future appearance to proposed movement. Appendix A makes the schematic concrete: seven history frames accompany five future model steps after downsampling a one-second, 15-action chunk.
Where the evidence stops. The figure reuses H for latent height and temporal horizon in different places. Read each label locally. Its CLIP semantic-token block is shown, but the surrounding formulation does not fully specify that conditioning path. No attention mask or stop-gradient rule is established here.
2. Motivation
2.1 The problem and the proposed response
Generalist manipulation policies require expensive physical rollouts for evaluation and additional corrective demonstrations for improvement. Ctrl-World seeks to move both processes into generated experience, requiring camera compatibility, precise action response and memory of scene content that disappears from view. e02
2.2 What this reading follows
Imagine testing a robot policy by returning generated camera images after each proposed action chunk. Ctrl-World builds this loop around an existing policy and a video diffusion model. Its central design problem is keeping the resulting images responsive to actions, consistent across cameras and stable when the wrist camera revisits a scene. The paper combines joint view prediction, sparse visual history and frame-specific pose conditioning, then uses human judgments to evaluate or select imagined trajectories. The figures below connect those mechanisms to video-quality ablations, real-versus-imagined policy behavior and downstream post-training gains. The key reading boundary is between following the requested instruction and accurately simulating task completion. e02e03e08e12e13e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The neural-world-simulator classification and non-applicable action/world-model quadrant fit the architecture: an external policy supplies actions to a separately trained predictive simulator. This is neither joint future/action generation nor inverse-dynamics action extraction. The datasets/data-collection tag fits synthetic trajectory collection, but the paper does not establish a new public benchmark dataset. e02e03e07e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Connect a joint-velocity policy to a Cartesian visual simulator
The policy/world-model boundary matters because the evaluated policies and the simulator speak different action languages. Appendix B says the official DROID policies consume joint angles and two camera views, then produce joint velocities. Ctrl-World conditions its visual future on Cartesian end-effector poses. A learned two-layer adapter first converts current joints and predicted velocities into future joint configurations; Panda forward kinematics then produces the required poses. The world model generates images and returns an observation for the next policy query. Thus the action trajectory originates in the policy, while the world model predicts its visual consequences. Algorithm 1 alternates those components without executing physical actions inside imagination. Reproducing only the video backbone would leave out the learned interface that makes this closed loop possible. e02e07e08
5.2 Separate scene memory from the current command
A moving wrist camera can lose sight of an object that remains visible in another camera, or revisit a location seen earlier. Ctrl-World supplies both kinds of context: jointly processed views address the first case, and sparse historical frames address the second. Pose embeddings provide a way to associate each appearance with an arm configuration. In training, the network denoises future visual targets while attending to history and the associated future action poses. My interpretation is that pose conditioning makes historical appearance easier to reuse selectively, but the experiment does not isolate selective retrieval from simply having more context. Table 2 supports the complete design through removal ablations. Figure 4's nonzero initial-frame attention is suggestive; its larger 3-second bar prevents treating the picture as proof that pose matching dominates. e03e04e06e10e11
5.3 Understand what synthetic success teaches the policy
The improvement procedure searches the existing policy's behavior space before changing its weights. Instruction rephrasing and arm resets create alternative imagined attempts; human judgments retain successful trajectories; supervised fine-tuning then makes those selected behaviors easier to produce. Section 5.4 reports 400 attempts per task and 25–50 retained successes. My reading is that this pipeline can repair which object or direction the policy chooses even when the simulator has imperfect contact physics. That interpretation matches the authors' focus on novel instructions and objects, their downstream gains, and their warning about low-level execution. Table 3 makes the distinction concrete: approaching the laptop lid transfers much better into imagination than completing the closure. The gain therefore motivates controlled tests of selection and instruction diversity, rather than an assumption of uniformly accurate simulated physics. e08e12e13e14
5.4 Training and inference
During training
The 1.5B SVD backbone initializes the world model; only the action-projection MLP is newly initialized. The PDF says other parameters are unchanged at initialization, which does not establish a frozen backbone during fine-tuning. Diffusion training predicts clean future observations from noisy futures and history. e04
DROID supplies 95,599 trajectories from 564 scenes, including approximately 76k successes and 19k failures. Appendix A specifies 100k steps, learning rate 1e-5, batch 64 and 16 H100 GPUs for about 2–3 days. Its 'all 95k' wording must be reconciled with the main text's 2% validation holdout. e05e06
For policy improvement, instruction paraphrases or randomized arm resets diversify imagined behavior. Each task yields 400 candidates, with 25–50 human-selected successes retained. The separate π0.5-DROID policy is fine-tuned for 2k steps on four H100 GPUs. e08
During inference
Start from a validation observation or a real setup snapshot. Alternate policy action prediction, action-to-pose conversion and conditional video generation. Humans label completed trajectories for evaluation or select synthetic successes for later training; an automated reward model is future work. e02e07e08
5.5 Implementation flow
- Translate the policy's action interface
The evaluated DROID policies consume two camera views and joint angles and predict joint velocities. A two-layer learned adapter predicts future joint configurations; Panda forward kinematics converts them to Cartesian poses for the world model. This interface is distinct from the world's image predictor. e07
- Build a shared visual context
A VAE encodes all three views; their tokens are concatenated. Seven noisy history frames provide sparse context. Appendix A specifies 192×320 images, 24×40 latent grids, and temporal downsampling of a one-second, 15-action chunk to five model steps. e03e06
- Associate frames with poses
Spatial-transformer cross-attention links each historical frame to its observed pose and each future frame to its action-derived pose. Temporal processing can reuse visual information from past states. This is learned attention over supplied history, without a separately described nearest-neighbor retrieval stage. e03
- Close the imagined loop
The final generated observation is returned to the policy for another action chunk. Short chunks are padded, with only valid predictions retained. The loop predicts consequences; physical robot execution is a separate evaluation setting. e02e05e12
6. Experiments & results
Ctrl-World turns a pretrained video diffusion model into an action-conditioned simulator for existing robot policies. Joint camera prediction, pose-associated history and frame-specific action conditioning support imagined interactions. Human-selected synthetic successes then improve instruction following, while imperfect contact physics limits the simulator's ability to predict actual task completion.
6.1 Read the original evidence
Table 1. Joint prediction improves most reported video metrics, with a visible FID tradeoff. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the arrow beside each metric before comparing rows: higher PSNR and SSIM are preferred, while lower LPIPS, FID and FVD are preferred. Every row is scored on third-camera output, even when the method uses multiple views. The caption specifies 256 ten-second validation clips generated through ten successive one-second action chunks. First compare Ctrl-World with its single-view variant to see the effect of the complete multi-view configuration. Then inspect the earlier baselines, remembering that changing available camera information changes the comparison. Dashes in the two multi-view baseline rows are missing reported metrics, not zero scores. e05e09e15
What it supports. Full Ctrl-World reports FVD 97.4 against 127.5 for its single-view variant, with PSNR increasing from 21.27 to 23.56. However, FID increases from 23.47 to 25.00. The source's general superiority language therefore needs a metric-specific reading; the table does not show improvement in every column.
Where the evidence stops. These numbers measure agreement or distributional similarity of generated videos, not robot success. The single-view variant also has lower PSNR than single-view IRASim. The PDF gives no uncertainty estimates or detailed configuration for the added multi-view baseline variants.
Table 3. Behavioral ranking and completion calibration are different tests of the world model. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as two separate comparisons: instruction following on the left and full task success on the right. The source's human criteria explain why they can differ. For close-laptop, approaching the lid counts as following the instruction, whereas fully closing it counts as success. Compare real and imagined rates within one policy before comparing policies. Section 5.3 states that the two rollout settings share their initial observations and that the new DROID setup uses novel camera placements. The table covers pick-place, towel folding, drawer placement, wiping, laptop closing, tissue pulling and stacking; Appendix B supplies the corresponding task-specific judging rules. e07e12e14
What it supports. For π0.5 on close-laptop, instruction following is 0.80 in reality and 0.70 in imagination, while success is 0.70 and 0.05. Drawer success is likewise 0.80 versus 0.30. The simulator can preserve useful behavioral ordering while badly underestimating completion on particular interactions.
Where the evidence stops. The table does not give trial denominators or confidence intervals. Its matched starts do not establish generalization to every scene or contact regime. A ranking useful for instruction-following evaluation is insufficient evidence that imagined success rates are calibrated physical probabilities.
Figure 9. Selected imagined successes support large targeted downstream gains, with incomplete statistical reporting. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to distinguish the base policy from the policy fine-tuned on selected synthetic trajectories. Each pair represents a task family: spatial references, size/shape distinctions, towel-folding direction or unfamiliar objects. The rightmost pair summarizes the average. Section 5.4 explains the intervention: generate varied imagined rollouts, retain human-judged successes and fine-tune π0.5-DROID. Appendix C supplies task examples, per-task tables and training hardware. Read the bar labels as the figure's displayed approximations; use the results paragraph for the stated overall endpoints. This plot summarizes downstream policy performance rather than a video-quality score. e08e13e14
What it supports. The paper reports the average rising from 38.7% to 83.4%, which is a 44.7-percentage-point increase. Every displayed task family improves. The authors frame the benefit as better instruction following for unfamiliar instructions and objects, and explicitly limit claims about improving low-level execution on familiar tasks.
Where the evidence stops. The whiskers are not defined, and evaluation trial counts and a post-training test split are unspecified. Appendix Table 5 prints 0.4374 although its four base cells average to 0.4375. The source's '+44.7%' wording denotes an absolute gain here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Ten-second action-conditioned video prediction DROID 2% trajectory holdout; 256 clips; ten autoregressive one-second chunks; third-view scoring. | 23.56; 0.828; 0.091; 25.00; 97.4. PSNR ↑; SSIM ↑; LPIPS ↓; FID ↓; FVD ↓ | Ctrl-World-Single-View: 21.27; 0.793; 0.110; 23.47; 127.5. Joint prediction improves four listed metrics over the single-view variant, while FID worsens. Single-view Ctrl-World also falls slightly below IRASim-Single-View in PSNR (21.27 versus 21.36); superiority is not uniform. These are video metrics. e09 |
| Memory, frame conditioning and joint prediction ablations Table 2 uses the world-model quality evaluation, separately scoring third-person and wrist views. | Full model: 97.4 third-view, 127.1 wrist-view. FVD ↓ | Without memory: 105.5 / 133.1; without frame conditioning: 122.7 / 179.1; wrist-view without joint prediction: 158.1. Each component helps the reported metric. The larger frame-conditioning drop does not by itself prove physical accuracy or statistically reliable differences; no uncertainty is supplied. e09e10 |
| Policy evaluation on close-laptop π0.5 in matched real/world-model initial observations; human-defined instruction-following and completion criteria. | Real: 0.80 and 0.70; world model: 0.70 and 0.05. Instruction-following rate; task success rate | The same policy loses 0.10 in instruction-following rate but 0.65 in completion rate inside the model. This concrete case exposes a large execution-calibration gap despite broadly preserved behavioral ranking across tasks. Trial counts and confidence intervals are unspecified. e12 |
| Post-training π0.5-DROID on synthetic successes Downstream spatial, size/shape, directional towel-folding and novel-object tasks; base versus synthetic-data-fine-tuned policy. | 38.7% → 83.4%, an increase of 44.7 percentage points. Reported average downstream success rate | Figure 9 labels category pairs 0.29→0.88, 0.44→0.91, 0.57→0.80 and 0.25→0.75. The source calls this a 44.7% improvement; the endpoints establish an absolute percentage-point gain. It supports targeted instruction-following improvement. Post-training trial counts, test split and plotted-whisker definition are not provided. e08e13e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Frame-specific conditioning has the largest listed wrist-view FVD degradation among these removals. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the horizontal divider as a change of evaluated camera, not a continuation of one score series. Within each block, the full model is the reference. Compare a removed-component row with that reference using the same metric column. The wrist block includes an additional removal of joint prediction, allowing a direct look at the importance of information from other cameras for the moving wrist view. Read the table alongside the architecture: memory removes historical context, frame conditioning removes the per-frame pose connection, and joint prediction changes access to shared visual information. Table 1 supplies the surrounding ten-second prediction protocol. e03e09e10
What it supports. Wrist-view FVD rises from 127.1 to 133.1 without memory, 179.1 without frame-level conditioning, and 158.1 without joint prediction. Wrist PSNR also falls from 19.18 to 15.69 without frame conditioning. These measurements support using all three components in the tested model, especially precise pose alignment.
Where the evidence stops. Removal ablations change the information available to the network; they do not isolate every implementation choice. No seed variation, confidence interval or compute-matched replacement is reported. Video-metric degradation cannot be converted directly into a physical-control error.
Figure 4. Action branches and return motions illustrate controllability, with an attention diagnostic that needs careful interpretation. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each arrow from the shared initial observation to the right. The top three branches use different translations, rotations and gripper commands, illustrating distinct generated consequences. For a closer ablation comparison, follow the green, blue and purple rows: each uses the same sequence of two negative-X movements followed by two positive-X movements. The colored circles draw attention to different appearances during the return. The inset plots attention scores for the last frame against time. The caption highlights attention to the initial frame with a matching pose; visually, the largest bar is instead at 3 seconds, while the 0-second bar remains nonzero. e03e10e11
What it supports. The examples show action-dependent visual changes and qualitatively different results when history or frame-level pose conditioning is removed. Together with Table 2, they motivate the proposed mechanism. The inset supports some reuse of earlier information, but does not demonstrate that the matching initial pose receives the greatest attention.
Where the evidence stops. The caption's emphasis on the 0-second matching pose is stronger than the plotted ranking supports; the 3-second bar is tallest. Preserve this distinction. These selected images do not provide an aggregate centimeter-error measurement or a causal test of retrieval.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report inaccurate collisions, sliding and rotations, missing retries, sensitivity to the initial observation, and failures on precise interactions or long-horizon reasoning. They do not expect the present model to improve low-level success on familiar instructions. e14
Figure 4's text highlights attention to the matching initial pose, but its tallest plotted bar is at 3 seconds, not 0 seconds. The diagnostic is suggestive, without establishing that matching-pose retrieval dominates attention. e11
Table 5 prints base average 0.4374; its four displayed values average to 0.4375. This apparent arithmetic error is preserved as a source discrepancy. Missing trial denominators and undefined Figure 9 whiskers prevent a statistical interpretation of the policy gains. e12e13
7.2 Questions for discussion
- Would pose-matched history still help when distractor frames share the same arm pose but depict a changed object state?
- How much of the downstream gain comes from instruction rephrasing, state resets, human filtering or simulator fidelity?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs DROID, SVD initialization, the action MLP, joint-velocity adapter, Panda kinematics and the evaluated DROID policy checkpoints. Exact split identifiers, optimizer and sampler settings, adapter training and detailed policy fine-tuning configuration remain unspecified in the PDF; resolve these before asserting comparable results. e05e06e07e15
Proposed checks: shuffle history-pose associations while keeping frames and compute fixed to test pose-guided memory; compare success-filtered and equally sized unfiltered synthetic datasets under matched rollout and training budgets to isolate selection effects. e03e08e10e11e13
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does correct pose association improve return-view prediction?
Reader-proposed experiment, not performed: keep the seven history images, future action poses, architecture, training budget and held-out ten-second clips fixed. Train/evaluate one condition with correct historical pose-image pairs and another with historical poses shuffled among those same frames. Use shared sampling seeds, report wrist-view PSNR/FVD, and separately score moments when the camera returns to an earlier arm pose. A selective return-view advantage for correct pairing would support pose-guided reuse beyond extra visual context. Equal performance, or an advantage confined to unrelated frames, would weaken that explanation. Include the original no-memory ablation as a reference, not as the sole control. e03e06e09e10e11
Check 2: Is the policy gain caused by selecting successful synthetic behavior?
Reader-proposed experiment, not performed: use the same base π0.5-DROID checkpoint, initial snapshots, instructions and fixed 400-candidate budget per task. Compare equal-sized datasets of human-selected successes and randomly selected unfiltered candidates; retain a no-fine-tuning baseline. Keep paraphrasing/reset proportions and the 2k-step fine-tuning budget identical. Evaluate on a predeclared held-out physical setup with separate instruction-following and completion labels, blinded raters and reported trial counts. A reproducible advantage for filtered data would support the selection mechanism; equal gains would implicate extra training or exposure, while gains only on familiar snapshots would weaken generalization claims. e08e12e13e14
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version, Figures 1–9, Tables 1–7, Algorithm 1, all retained method and experiment pages, the conclusion, and Appendices A–C including adapter details and hardware. All six final original crops were inspected for readable labels, complete bounds and relevance. Figure 2 was checked against Section 4.1 and Appendix A; Figure 4's attention interpretation was checked against its plotted bars and caption. The Table 5 average discrepancy and undefined Figure 9 whiskers remain disclosed. Reference pages 11–15 were read in the supplied text but not rendered; they support no retained technical or numerical claims. Separate videos, code and supplemental artifacts were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/version block
- Sections 1–3: Introduction, Related Works and Problem Formulation
- Sections 4.1–4.2: world-model learning, evaluation and improvement; Algorithm 1
- Sections 5.1–5.4: setup, model quality, policy evaluation and improvement
- Section 6: Conclusion and limitations
- References
- Appendices A–C: architecture/training, policy interface/evaluation and policy improvement
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The PDF visual pass resolved the extraction-only image limitation for all nine figures and seven tables; separate videos and supplementary code were not inspected.
- All six supplied text chunks were read. No code was inspected and no experiments were reproduced.
- The inspected edition is arXiv:2510.10125v3, dated 1 March 2026, marked ICLR 2026; title and full author list match the catalog. Earlier revisions were not supplied, so cross-version scientific changes are unverified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, author affiliations and arXiv version stamp
Title and authors match the catalog: Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen and Chelsea Finn. Guo and Shi contributed equally. Affiliations are Stanford University and Tsinghua University. The artifact is arXiv:2510.10125v3, 1 March 2026, marked published at ICLR 2026.
Go to primary source ↓e02PDF pp. 1–3, Introduction, Figure 1 and Section 3, Eqs. (1)–(2)
The problem is costly policy evaluation and corrective data collection. A separate generalist policy consumes images, robot pose and an instruction, proposes an action chunk, and receives the final predicted observation from an action-conditioned world model for the next imagined interaction.
Go to primary source ↓e03PDF pp. 3–4, Section 4.1, Multi-View Joint Predictions, memory/action conditioning and Figure 2
Camera-view tokens are concatenated for joint prediction. Sparse history images and associated poses provide memory; each future frame attends to its corresponding Cartesian action-pose embedding through spatial-transformer cross-attention. Figure 2 also labels CLIP semantic tokens entering the temporal transformer.
Go to primary source ↓e04PDF p. 4, Section 4.1, Training Objective, Eq. (3)
The world model starts from pretrained 1.5B SVD, newly initializes the action-projection MLP, and is fine-tuned with squared clean-future prediction error under diffusion noise. Other parameters are unchanged at initialization; this statement does not specify that they remain frozen during fine-tuning.
Go to primary source ↓e05PDF p. 5, Section 5.1; p. 6, Section 5.2 validation protocol; p. 16, Appendix A, Training Datasets
DROID has 95,599 trajectories in 564 scenes, about 76k successful and 19k failed. The platform uses a Panda arm, Robotiq gripper, two external cameras and one wrist camera. Section 5.2 holds out 2% of trajectories, while Appendix A says all 95k trajectories are used. Section 5.1 specifies seven history frames separated by 1–2 seconds and pads action chunks shorter than 15 steps, keeping only valid predictions.
Go to primary source ↓e06PDF p. 16, Appendix A, Model Architecture and Training Process
A three-layer MLP maps 7D Cartesian actions to 1024D embeddings. An 8×8 spatial VAE maps each 192×320 view to 24×40 latent features. Seven noisy history frames and five future steps form B×(7+5)×(3×24×40) tokens; 15 actions over one second are temporally downsampled to five. Training uses learning rate 1e-5, 100k steps, batch 64 and 2×8 H100 GPUs over approximately 2–3 days.
Go to primary source ↓e07PDF p. 16, Appendix B, policy/world-model interface and equations
Official DROID policies take joint angles and two camera views and output joint velocities. A learned two-layer MLP maps current joint angles and future joint velocities to future joint configurations; Panda forward kinematics then supplies Cartesian poses to the world model.
Go to primary source ↓e08PDF pp. 4–5, Section 4.2 and Algorithm 1; p. 9, Section 5.4; p. 18, Appendix C, Finetuning Process
Imagined rollouts start from observations and instructions and are judged by people. Policy improvement diversifies rollouts by instruction paraphrases or initial-arm resets, generates 400 trajectories per task, retains 25–50 judged successful, and fine-tunes π0.5-DROID for 2k steps. Appendix C specifies four H100 GPUs. Algorithm 1 summarizes supervised action fitting on the selected synthetic dataset.
Go to primary source ↓e09PDF p. 6, Table 1, all rows and metric columns, caption and Section 5.2
On 256 held-out ten-second clips, with ten autoregressive one-second action chunks, Ctrl-World reports third-view PSNR 23.56, SSIM 0.828, LPIPS 0.091, FID 25.00 and FVD 97.4. Its single-view variant reports 21.27, 0.793, 0.110, 23.47 and 127.5. IRASim-Single-View PSNR is 21.36. WPE-Multiview and IRASim-Multiview omit SSIM, LPIPS and FID entries.
Go to primary source ↓e10PDF p. 6, Table 2, third-view and wrist-view ablation blocks
Third-view FVD is 97.4 for full Ctrl-World, 105.5 without memory and 122.7 without frame-level conditioning. Wrist-view FVD is respectively 127.1, 133.1 and 179.1, and is 158.1 without joint prediction. Wrist-view PSNR is 19.18 for the full model and 15.69 without frame-level conditioning. The table supplies no uncertainty estimates.
Go to primary source ↓e11PDF p. 7, Figures 4–5, captions and controllability/consistency discussion
Figure 4 contrasts action branches and memory/pose-conditioning ablations. Its text highlights attention to the matching initial pose when predicting t=4 s, but the plotted attention bar is tallest at 3 s, with a smaller nonzero 0 s bar. Figure 5 attributes green-box wrist content to other camera views and red-box content to history. These are illustrative examples rather than aggregate physical-accuracy measurements.
Go to primary source ↓e12PDF p. 8, Section 5.3 and Figures 6–7; p. 17, Table 3 and task criteria; p. 18, Stack criterion
Three policies are evaluated across seven tasks from matched real/world-model initial observations in a new DROID setup. Table 3 gives π0.5 close-laptop instruction-following rates 0.80 real and 0.70 imagined, but success rates 0.70 and 0.05. For drawer, π0.5 success is 0.80 real and 0.30 imagined. Human criteria distinguish approaching/attempting the requested interaction from completion; no trial denominators accompany Table 3.
Go to primary source ↓e13PDF p. 9, Section 5.4, Figures 8–9 and Results; p. 18, Appendix C and Tables 4–7
The reported downstream policy success rises from 38.7% to 83.4% across spatial understanding, size/shape understanding, directional towel folding and novel-object tasks. Figure 9 displays rounded category bars and whiskers without defining the whiskers. Table 5 prints base average 0.4374 although its four cells 0.40, 0.45, 0.40 and 0.50 average to 0.4375. Tables 4–7 report post-training averages 0.875, 0.9125, 0.80 and 0.75. Trial counts and an explicit post-training test split are not specified.
Go to primary source ↓e14PDF pp. 9–10, end of Section 5.3 and Section 6, limitations
The authors describe inaccurate collisions, sliding and rotations, missed retry behavior, sensitivity to initial observations and failures in precise interactions or long-horizon reasoning. They focus on improving instruction following and expect the model to be insufficiently accurate for improving low-level success on previously seen instructions.
Go to primary source ↓e15PDF p. 16, Appendix A–B; p. 18, Appendix C
The PDF gives model dimensions, world-model training rate and steps, compute, adapter structure and policy fine-tuning steps. It does not specify exact data-split identifiers, optimizer details, diffusion sampling configuration, adapter-training hyperparameters, software versions or detailed policy fine-tuning hyperparameters. The appendix points to supplementary code, whose availability is not established by this reading.
Go to primary source ↓8.5 Primary sources
Ctrl-World: A Controllable Generative World Model for Robot Manipulation ↗
PDF · 8,728 extracted words
Source fingerprint
5acee75221001b53171078e8b02f85151147b6dc629b36cf086d91d1c921ba0f