Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
1. Paper overview
In one sentence: A video diffusion model can learn executable actions through latent frame injection, while rollout-refined prediction improves action selection at a substantial inference cost. e-injectione-jointe-planninge-decodinge-robocasa-ablatione-latency
| At a glance | What to know |
|---|---|
| Research problem | Source description How can a pretrained video generator produce executable robot controls while retaining useful predictive structure? The authors replace separate action modules with a shared latent representation and ask whether future-observation and value prediction improve both imitation learning and action selection. e-motivatione-injection |
| Core mechanism | Source description Latent frame injection accommodates proprioception, continuous action chunks, values and multiple camera views without architectural changes to the pretrained diffusion model. e-injectione-implementation |
| A key reported result | ALOHA planning on challenging candy tasks: Model-based 72.0: bowl 60.0, bag 84.0. Average task-completion score (0–100). Harder initial conditions for bowl and ziploc-bag tasks after refinement on 648 rollouts; a separate evaluation setting from the direct-policy suite. Direct 59.5: bowl 49.0, bag 70.0; Q-based 61.3: bowl 52.0, bag 70.7. A 12.5-point gain over direct control supports this planning configuration. Separate planning trial counts and uncertainty are not reported. e-rolloutse-planning-resultse-aloha-protocol |
| Reading caution | Source description Planning takes 4.9 seconds on eight H100 GPUs per chunk, versus 0.95 seconds for direct ALOHA inference on one H100. Rollout requirements, one-layer search and slow reactions constrain dynamic manipulation; deeper planning and locomotion are future directions. e-latencye-discussion |
Core contributions
- Source description
Latent frame injection accommodates proprioception, continuous action chunks, values and multiple camera views without architectural changes to the pretrained diffusion model. e-injectione-implementation
- Source description
Conditioning masks turn the same denoiser into a joint policy, world model or value predictor. Optional rollout refinement enables best-of-N planning through dual checkpoint deployment. e-jointe-planning
- Author claim
The authors report the highest aggregate performance among their listed comparisons in LIBERO, RoboCasa and ALOHA. These claims concern the reported benchmarks and protocols, not unrestricted robot generalization. e-liberoe-robocasae-aloha
Figure 8. Actions enter the diffusion sequence as repeated numerical vectors in reserved latent frames. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: the first blank image handles the tokenizer’s special first-frame rule. Subsequent camera images appear four times each because temporal compression must yield one latent per view at one timestep. Follow the downward VAE arrow, then the horizontal injection arrow: current proprioception, the action chunk, future proprioception and value overwrite their placeholder latents. In the bottom row, the current observation s remains clean while a, s′ and V(s′) are noised prediction targets. This is the policy-training configuration. Appendix A.1 explains how repeated vectors are averaged and unnormalized to recover controls; action extraction does not pass through the image decoder. e-backbonee-state-valuee-injectione-implementatione-aloha-protocol
What it supports. The mechanism places executable controls and predicted observations in one denoiser’s representational space. The ordering supports both joint generation and conditional prediction. Crucially, the repeated pictures are tokenizer inputs for a single timestamp, so the figure is not claiming a densely predicted future video trajectory.
Where the evidence stops. Section 4.1 defines s′ at the action-chunk endpoint and uses observations as an approximation to state. This graphic shows an illustrative camera arrangement; the ALOHA setup uses two wrist views and one overhead view.
2. Motivation
2.1 The problem and the proposed response
How can a pretrained video generator produce executable robot controls while retaining useful predictive structure? The authors replace separate action modules with a shared latent representation and ask whether future-observation and value prediction improve both imitation learning and action selection. e-motivatione-injection
2.2 What this reading follows
Cosmos Policy asks a practical question: can a pretrained video generator become a robot controller without adding a separate action network? Its answer is to place actions, proprioception and return targets directly into the video model’s latent sequence. This creates a common representation for acting, predicting and evaluating. The distinction between those roles matters: the strong direct-policy results discard predicted futures at execution time, whereas planning uses an additional checkpoint refined on rollout experience. The visuals below trace that mechanism, examine the real and simulated control evidence, and show where auxiliary prediction helps—and where the paper leaves causal and reproducibility questions open. e-injectione-jointe-planninge-decodinge-robocasa-ablatione-latency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded One Model × Joint prediction classification fits the architecture: one denoiser jointly models actions, future observations and values without an inverse-dynamics module. The deployment qualification is substantial: reported planning uses two checkpoints with different roles. Joint supervision benefits direct control even when its predicted future/value outputs are discarded. e-injectione-jointe-planninge-decoding
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make a video latent carry a robot action
The representation change comes before policy learning. A video tokenizer normally turns images into latent volumes, but an action chunk is already a numerical array with a controller-defined scale. Cosmos Policy normalizes that array, flattens it and repeats it until it fills a latent volume. During training, this volume replaces the encoded blank placeholder, so the existing denoiser learns an action distribution using its usual reconstruction machinery. At deployment, the process reverses: repeated copies are averaged and converted back to the action scale. This is why generated actions are directly executable and do not require an inverse-dynamics network to interpret generated pictures. The future camera observations provide an additional target at the chunk endpoint; they are not a sequence of intermediate controls or a dense simulation of every movement. e-injectione-implementatione-backbone
Figure 12. The conditioning boundary changes the function learned by a shared diffusion model. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the clean sequence at the top as the reference layout. In the first red panel, only s is available and the model jointly reconstructs actions, future observations and value. In the middle panel, the action is also clean, so the target becomes the future and its value. In the final panel, all observations and the action are supplied, leaving only the value target noisy. Compare these boundaries with Section 4.2’s conditional distributions rather than treating the panels as three separate networks. Auxiliary supervision is visible in the first two panels: policy training includes future/value targets, and world-model training includes a value target. e-jointe-training-maske-decodinge-planning
What it supports. The figure explains how one architecture supports three functions through conditioning. It also explains why direct control can benefit from future supervision even when future predictions are discarded at deployment. The auxiliary target affects learned parameters during training; it need not be used for online search.
Where the evidence stops. Panel labels say percentages of 'Training Batches'; Section 4.2 and the caption instead split samples within each batch. This explanation follows the text’s sample allocation. The graphic depicts initial training, before rollout refinement changes proportions and value-input masks.
5.2 Separate auxiliary prediction from online planning
The training masks explain two different uses of a world model. When only the current observation is clean, the shared model predicts actions together with future observations and value. These additional targets can shape the policy during training, even though direct evaluation throws away their outputs. When the action is also clean, the same architecture instead learns the action-conditioned future needed for planning. The base value task receives the complete prefix. Optional refinement later changes its inputs so that V(s′) depends on the future, or Q(s,a) depends on current observation and action. Reader interpretation: RoboCasa’s auxiliary-target ablation is evidence about representation learning for control, while Figure 7 is evidence about using predicted outcomes to select actions. They should not be treated as interchangeable demonstrations of planning. e-jointe-training-maske-decodinge-robocasa-ablatione-planning-results
5.3 Learn failures before trusting an imagined future
Successful ALOHA demonstrations initially expose the model to a narrow outcome distribution. The paper therefore refines prediction on rollout experience that includes mistakes, and deploys the original checkpoint for proposing actions alongside the refined checkpoint for evaluating them. The actual refinement pool contains rollouts from several policies plus additional Cosmos bag-task episodes. At each decision, the planner evaluates multiple possible action chunks, samples several futures and values, and selects the highest aggregated value. Only then does the robot execute a chunk and obtain fresh observations. Reader interpretation: Figure 6 illustrates why a model trained only on demonstrations can miss grasp failures, but that example does not quantify calibration. Figure 7 supplies control evidence; the measured 4.9-second planning pause defines an important limit on when that evidence is useful. e-jointe-planninge-rolloutse-world-diagnostice-planning-resultse-latency
5.4 Training and inference
During training
All base Cosmos-Predict2 weights are fine-tuned. LIBERO uses 40K steps, batch 1920 and 64 H100 GPUs; RoboCasa uses 45K, batch 800 and 32 H100s; ALOHA uses 50K, batch 200 and 8 H100s. Each run lasts 48 hours. ALOHA combines 185 demonstrations across four tasks. e-training
Noise sampling mixes the original log-normal distribution with probability 0.7 and Uniform[1,85] with probability 0.3. The original log-noise mean and standard deviation are 1.39 and 1.2. Inference uses noise bounds 4 and 80. The authors report improved precision; the mixture probabilities were not tuned. e-noise
Planning refinement uses 10/45/45 policy/world/value sampling. Its 648-rollout pool combines 505 evaluations across all compared policies and 143 additional Cosmos bag-task rollouts. Although the deployment motivation invokes on-policy prediction, this reported pool contains data from multiple policies. e-planninge-rollouts
During inference
Direct evaluation jointly generates all targets and discards future/value predictions. It uses five denoising steps in simulation and ten on ALOHA. LIBERO executes all 16 predicted actions; RoboCasa predicts 32 and executes 16; ALOHA executes 50 actions at 25 Hz, then queries again. e-traininge-decoding
Planning keeps the original action checkpoint and uses the refined checkpoint for prediction. Eight action proposals each receive three future predictions and five values per future. Values are aggregated by the majority success/failure group and its mean. Model-based value prediction masks current state/action; the Q variant masks the future. Search uses ten action-denoising steps and five each for future/value. e-planninge-rolloutse-decodinge-latency
5.5 Implementation flow
- Pack observations and controls into video latents
Images pass through the Wan2.1 VAE. Placeholder latents are overwritten with flattened, normalized and repeated proprioception, action or value vectors. The sequence is ordered as current observation, action, endpoint observation and value. The illustrative three-camera layout has 11 latents, including an initial blank. Four copies of each image accommodate temporal compression; they represent one timestep. e-backbonee-injectione-implementation
- Recover controls directly
After denoising, repeated action-vector copies are averaged and unnormalized. Value extraction averages its entire latent volume and rescales to [0,1]. Neither operation requires VAE decoding. The predicted future state is only a camera/proprioception approximation of state at t+K, where K is chunk length; no intermediate future trajectory is predicted. e-state-valuee-injectione-implementation
- Change the conditioning boundary
Initial training allocates 50% of samples to joint action/future/value prediction, 25% to future/value prediction given an action, and 25% to value given the full prefix. Policy examples use successful demonstrations; world/value examples also retain failed replays when available. ALOHA initially has only successful demonstrations. e-jointe-training-mask
6. Experiments & results
Cosmos Policy adapts Cosmos-Predict2-2B to generate robot action chunks, future observations and values in one latent diffusion sequence. Direct control benefits from joint supervision; optional planning adds rollout refinement and a separate planning checkpoint. Experiments support strong manipulation performance, while slow search and uneven OOD performance limit broader conclusions.
6.1 Read the original evidence
Table 2. RoboCasa combines strong reported success with relatively few target-task demonstrations. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the demonstration-count column before comparing success rates. Cosmos Policy uses the 50 human demonstrations available per task, with unsuccessful replays excluded from policy training but retained for world/value training. Many listed methods use 300 demonstrations; others use more, and DreamGen also lists synthetic data. The 67.1% result averages 24 kitchen tasks, five evaluation scenes per task and three random seeds, totaling 3600 trials. Section 5.1 states that evaluation object instances are unseen and two of the five scene styles per task were absent from training. This table concerns direct policies, so its score does not include best-of-N planning. e-sim-protocole-robocasae-decoding
What it supports. Cosmos Policy’s 67.1% exceeds the table’s 66.4% FLARE and HAMLET entries, each trained with 300 demonstrations per task, and UVA’s 50.0% with 50 demonstrations. This supports the paper’s target-demonstration-efficiency claim within the reported benchmark and training settings.
Where the evidence stops. These methods differ in pretraining, architecture and training data. Demonstrations per task are not total lifetime data or compute. The table gives no uncertainty intervals, so the small lead should not be called statistically significant.
Table 3. The aggregate ALOHA advantage coexists with a weaker OOD average than π₀.₅. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the three horizontal blocks to keep distribution shifts separate. Each method is evaluated on the same initial conditions, with 101 total trials distributed across the four tasks. Entries are completion scores: the candy-bowl rubric awards 20 points per candy, and the bag rubric awards 20 per completed stage. They are not simply the fraction of entirely successful episodes. The final block combines in-distribution and OOD trials within each task, then reports an average across tasks. Appendix A.3.2 describes concrete shifts such as an unseen orange bowl, a different bag and unfamiliar shirt/distractor configurations. Compare the same block and task column before interpreting an advantage. e-aloha-protocole-alohae-training
What it supports. Cosmos Policy has the highest full average, 93.6 versus 88.6 for π₀.₅. In the OOD block, however, π₀.₅ scores 92.5 versus Cosmos Policy’s 89.3. The OOD bowl column shows a particularly clear reversal: 90.0 for π₀.₅ and 74.0 for Cosmos Policy.
Where the evidence stops. The 93.6 aggregate is a partial-completion score, despite some prose calling it success rate. The limited tabletop shifts and small task-specific trial counts cannot establish broad OOD superiority. No uncertainty intervals are supplied.
Figure 7. On harder candy-task starts, prediction-based search improves average completion by 12.5 points. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the legend: gray is direct Cosmos Policy, blue selects using Q(s,a), and purple uses a predicted future followed by V(s′). The first two groups are the bowl and ziploc-bag tasks; the right group is their average. These trials use more challenging initial conditions than the earlier four-task comparison. Section 5.3 describes refinement on 648 rollouts, while Appendix A.4.2 specifies eight candidate actions, three futures per candidate and five value predictions per future. The model-based variant masks current state and action during value prediction; the Q variant masks the future, allowing it to rank an action without first generating that observation. e-planninge-rolloutse-planning-resultse-aloha-protocole-latency
What it supports. The model-based score rises from 49.0 to 60.0 on the bowl task and from 70.0 to 84.0 on the bag task, averaging 72.0 against the direct policy’s 59.5. The Q-based average is 61.3. The result supports this rollout-refined, one-layer search procedure in the tested conditions.
Where the evidence stops. The gain is 12.5 score points, not a relative percentage or a comparison with the earlier 93.6 aggregate. Planning costs 4.9 seconds on eight H100 GPUs. Separate planning trial counts, uncertainty and a compute-matched V-versus-Q comparison are not supplied.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO direct-policy manipulation Four ten-task suites; 50 demonstrations per task before failure filtering. Evaluation: 50 episodes per task over three seeds, 6000 trials. | 98.5 overall; Spatial 98.1, Object 100.0, Goal 98.2, Long 97.6. Average success rate (%) | CogVLA 97.4 and OpenVLA-OFT 97.1 overall; Video Policy reports 94.0 on Long. Strong reported control success, especially on Long. These are benchmark comparisons; the table gives no uncertainty intervals. e-sim-protocole-libero |
| RoboCasa direct-policy manipulation 24 kitchen tasks; 50 human demonstrations per task before filtering. Five evaluation scenes, unseen object instances, three seeds, 3600 trials. | 67.1 Average success rate (%) | FLARE and GR00T-N1.5 + HAMLET: 66.4 with 300 demonstrations/task; Video Policy: 66.0 with 300; UVA: 50.0 with 50. Supports efficiency in target-task demonstrations. Unequal pretraining, architectures and data budgets prevent a controlled attribution solely to video priors. e-sim-protocole-robocasa |
| ALOHA direct-policy bimanual manipulation Four tasks with 101 trials per method, shared initial conditions, and in-distribution/OOD subsets; 185 training demonstrations. | 93.6 overall; in-distribution 96.3; OOD 89.3. Average task-completion score (0–100) | π₀.₅: 88.6 overall, 87.8 in-distribution, 92.5 OOD. Highest aggregate score does not imply the best OOD score. The rubric awards partial credit, so 93.6 is not a binary episode-success percentage. e-aloha-protocole-aloha |
| ALOHA planning on challenging candy tasks Harder initial conditions for bowl and ziploc-bag tasks after refinement on 648 rollouts; a separate evaluation setting from the direct-policy suite. | Model-based 72.0: bowl 60.0, bag 84.0. Average task-completion score (0–100) | Direct 59.5: bowl 49.0, bag 70.0; Q-based 61.3: bowl 52.0, bag 70.7. A 12.5-point gain over direct control supports this planning configuration. Separate planning trial counts and uncertainty are not reported. e-rolloutse-planning-resultse-aloha-protocol |
| LIBERO mechanism ablations Two independent changes, each trained for the full policy's gradient-step count. | Full 98.5; auxiliary losses removed 97.0; pretrained initialization removed 94.6. Average success rate (%) | Drops of 1.5 and 3.9 percentage points respectively; Long falls from 97.6 to 94.0 and 88.6. Both ingredients help under this training budget. Equal steps do not establish that a randomly initialized model has converged. e-libero-ablation |
| RoboCasa cumulative objective ablations Same training hyperparameters/compute as the full model; 3600 trials over 24 tasks and three seeds. | Full 67.1; remove value-training samples 66.6; remove world/value-training samples 64.0; also remove auxiliary value 62.5; also remove auxiliary future state 44.4. Average success rate (%) | The final adjacent change reduces success by 18.1 percentage points. Future supervision matters in that already-reduced configuration. The full-to-bare-policy gap is not an isolated future-prediction effect. e-robocasa-ablation |
| RoboCasa denoising-speed tradeoff Direct policy; one H100 GPU; latency per predicted 32-action chunk. | One denoising step: 66.4 and 0.16 s. Success rate (%) and inference latency (seconds) | Five steps: 67.1 and 0.61 s. Reported rows imply a 0.7-percentage-point drop; Table 5's caption instead says −0.5%. This internal discrepancy remains unresolved. e-robocasa-ablatione-latency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. The strongest adjacent ablation drop occurs when the reduced policy loses its future-state target. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read rows (1) through (4) as a cumulative sequence. First, value-function training samples disappear; next, only policy-training samples remain. The third row additionally removes the policy’s auxiliary value target. The fourth also removes its future-state target, leaving action-only prediction. Thus the 62.5-to-44.4 comparison tests the future target after other objectives have already been removed. All variants use the full model’s RoboCasa training hyperparameters and compute and the 3600-trial protocol. The separate bottom row changes inference to one denoising step rather than removing another training objective. Appendix A.4.2 supplies its latency comparison with five-step sampling. e-robocasa-ablatione-training-maske-latency
What it supports. The final adjacent ablation loses 18.1 percentage points, supporting the usefulness of future supervision in the reduced policy. Separately, one-step inference reaches 66.4% at 0.16 seconds per chunk on one H100, compared with 67.1% and 0.61 seconds for five steps. This is a measured accuracy–latency tradeoff.
Where the evidence stops. The full-to-action-only gap combines several changes, including sample allocation; it cannot isolate future prediction alone. The caption says one-step inference loses −0.5%, whereas the tabulated 67.1 and 66.4 imply 0.7 percentage points. The discrepancy is unresolved.
7. Analysis & limitations
7.1 What the evidence leaves open
Planning takes 4.9 seconds on eight H100 GPUs per chunk, versus 0.95 seconds for direct ALOHA inference on one H100. Rollout requirements, one-layer search and slow reactions constrain dynamic manipulation; deeper planning and locomotion are future directions. e-latencye-discussion
Figure 6 illustrates improved future prediction but supplies no aggregate world-model accuracy metric. Narrow tabletop OOD tests and single-endpoint observation prediction do not establish general physical simulation or long-horizon planning accuracy. e-world-diagnostice-aloha-protocole-injection
Section 5.3 says 'three' value formulations but explicitly specifies and plots only V(s′) and Q(s,a). Its explanation of Q-model overfitting is a hypothesis, not a measured diagnosis. e-rolloutse-planning-results
7.2 Questions for discussion
- Does future-observation supervision still help when action examples per optimizer step are held fixed across objective ablations?
- Would V(s′) retain its advantage over Q(s,a) with matched inference cost and more held-out rollout data?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful reproduction needs the stated initialization, successful/failed demonstration separation, latent packing, mixed-noise sampler, conditioning masks and platform-specific chunk execution. The paper announces checkpoints, data and scripts; this is an author release statement. e-backbonee-implementatione-jointe-noisee-traininge-release
The supplied PDF leaves optimizer/learning-rate settings, image resolution, normalization statistics, numerical discount and majority-mean threshold unspecified. It also lacks a separate planning-refinement compute schedule and explicit planning train/evaluation partition. These gaps need resolution before claiming exact reproduction. e-implementatione-state-valuee-planninge-rolloutse-training
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate future supervision while fixing policy sample exposure
Reader-proposed check, not performed: compare the full RoboCasa objective with a version that masks only auxiliary future-observation losses on policy-training examples. Preserve successful demonstrations, action-example count per update, initialization, chunk execution, noise distribution, total updates and compute; keep conditional world/value training unchanged. Separately repeat Table 5’s adjacent 62.5-to-44.4 comparison. Evaluate the same 24 tasks, scenes and seeds, reporting paired task differences and uncertainty. A drop in both comparisons would support a robust auxiliary-supervision effect. A drop only in the cumulative setting would indicate interaction with the removed objectives or changed sampling, weakening attribution of the full-to-bare-policy gap to future supervision alone. e-jointe-sim-protocole-traininge-noisee-robocasa-ablation
Check 2: Test whether rollout refinement improves action ranking
Reader-proposed check, not performed: hold out entire bag and bowl rollouts and initial-condition groups before planning refinement. Keep the original policy’s eight candidate chunks fixed across three rankers: the base future/value model, the rollout-refined V(s′) model, and the rollout-refined Q(s,a) variant. Record ranking agreement with subsequently observed task completion, calibration of predicted returns, executed scores and wall-clock cost. Report both the paper’s ensemble settings and an equal-time comparison; specify the missing majority-mean threshold and discount. If refinement improves image appearance without improving held-out ranking or executed scores, the proposed planning mechanism would lack the discriminating evidence this check seeks. e-planninge-rolloutse-state-valuee-world-diagnostice-planning-resultse-latency
8.3 Reading coverage
Visual audit: The title/authors/version and all Figures 1–12 and Tables 1–5 were visually inspected, together with every supporting method, training, scoring and latency page. All six final original-PDF crops were viewed and checked for legible labels, complete boundaries and retained legends/headers. Figure 8 injection arrows and Figure 12 conditioning boundaries were checked against Sections 4.1–4.3 and Appendices A.1/A.4.1; Figure 12's batch-versus-sample wording difference is disclosed. Table 5's one-step score discrepancy is preserved. Reference-only pages 12–14 were read as text. Separate supplements, external artifacts and experiments remain outside this review.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Section 1 Introduction (pp. 1–2)
- Section 2 Related Work (pp. 2–3)
- Section 3 Preliminaries (pp. 3–4)
- Sections 4.1–4.3: latent injection, joint training and planning (pp. 4–6)
- Sections 5.1–5.3: experimental setup, direct policies, ablations and planning (pp. 7–10)
- Sections 6–7: discussion and reproducibility (pp. 10–11)
- Acknowledgments and References (pp. 11–14)
- Appendix A.1: latent injection implementation (pp. 15–16)
- Appendix A.2.1: noise distribution (pp. 15–16)
- Appendices A.2.2–A.2.4: training details (p. 17)
- Appendices A.3.1–A.3.2: evaluation and scoring (pp. 17–19)
- Appendices A.4.1–A.4.2: further ablations and latency (pp. 20–22)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The text-extraction image omission was addressed by inspecting the supplied PDF: all scientific figures and tables, the title page, and the claim-supporting method, evaluation and appendix pages were visually reviewed. Reference-only pages 12–14 were read as text.
- Identity and edition scope: the supplied title page identifies arXiv:2601.16163v1 [cs.AI], 22 January 2026. Its title and all 11 authors match the catalog. This report concerns that v1 artifact; no separate ICLR 2026 proceedings edition or later revision was supplied or compared. The printed project address uses /labs/dir/cosmos-policy/, whereas the catalog uses /labs/cosmos-lab/cosmos-policy/; neither was fetched.
- All seven supplied text chunks were read in full. External code, model checkpoints, datasets, project pages and separate supplements were not inspected; no experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
Title and all eleven catalog authors match; affiliations are NVIDIA and Stanford University. The margin identifies arXiv:2601.16163v1 [cs.AI], 22 Jan 2026.
Go to primary source ↓e-motivationPDF pp. 1–3, Abstract and Sections 1–2
The proposal adapts pretrained video diffusion to robot control without adding a separate action module; prior physical and temporal structure motivates this design.
Go to primary source ↓e-backbonePDF p. 3, Section 3, Cosmos video model paragraph and denoising objective
Initialization is Cosmos-Predict2-2B-Video2World, using Wan2.1 VAE tokens, T5-XXL text conditioning and an EDM reconstruction objective.
Go to primary source ↓e-state-valuePDF pp. 3–4, Section 3, MDP and world-model/value definitions
The finite-horizon formulation has sparse terminal reward and Monte Carlo discounted returns. Observations approximate state; numerical discount is not given.
Go to primary source ↓e-injectionPDF pp. 4–5, Figure 2 and Section 4.1
New modalities occupy normalized, repeated latent volumes; the example has 11 frames including a placeholder. Observations are at t and t+K only, with no input history.
Go to primary source ↓e-jointPDF p. 5, Section 4.2 and empirical-return footnote
Initial sample fractions are 50/25/25 for policy/world/value tasks. Conditioning defines their functions; auxiliary targets extend policy/world prediction. V(s′) denotes an empirical return target.
Go to primary source ↓e-planningPDF p. 6, Section 4.3
Refinement changes sampling to 10/45/45. Original and refined checkpoints serve as policy and planning models. Best-of-N uses three futures and five values per future, majority-mean aggregation and full-chunk execution.
Go to primary source ↓e-sim-protocolPDF p. 7, Section 5.1, LIBERO and RoboCasa paragraphs
LIBERO supplies 50 demonstrations per task. RoboCasa uses 24 tasks, 50 human demonstrations per task, five evaluation scenes and unseen objects; failures are filtered only for policy training.
Go to primary source ↓e-liberoPDF p. 8, Table 1, all suite/average columns and caption
Cosmos scores 98.1/100.0/98.2/97.6, average 98.5; CogVLA averages 97.4 and OpenVLA-OFT 97.1. Cosmos evaluation totals 6000 trials over three seeds.
Go to primary source ↓e-robocasaPDF p. 8, Table 2, training-demonstration and average-SR columns
Cosmos obtains 67.1 with 50 demonstrations/task; FLARE and HAMLET obtain 66.4 with 300; Video Policy 66.0 with 300; UVA 50.0 with 50. Caption reports 3600 trials.
Go to primary source ↓e-aloha-protocolPDF p. 7, Section 5.1 ALOHA paragraph; p. 18, Appendix A.3.2
ALOHA uses three cameras, 14 joint angles and 25-Hz control. Trials total 30/20/25/26 by task with shared initial conditions; scoring assigns partial completion credit and defines limited OOD variations.
Go to primary source ↓e-alohaPDF p. 8, Figure 4; p. 19, Table 3, In-distribution/OOD/Full blocks
Cosmos averages 96.3/89.3/93.6 across these blocks, versus π₀.₅ at 87.8/92.5/88.6. Full-task Cosmos scores are 100.0, 99.5, 89.6 and 85.4.
Go to primary source ↓e-world-diagnosticPDF p. 9, Figure 6 and caption
Selected base/refined predictions are juxtaposed with actual bag-task observations, illustrating a base-model grasp-error mismatch and a refined-model match; no aggregate predictive-accuracy statistic accompanies the example.
Go to primary source ↓e-rolloutsPDF p. 10, Section 5.3, rollout collection and value-function formulations
Refinement uses 505 rollouts across policies plus 143 Cosmos bag rollouts. V(s′) masks current state/action; Q(s,a) masks future state. The prose says three formulations but lists two.
Go to primary source ↓e-planning-resultsPDF p. 10, Figure 7 bars/legend and Section 5.3 results
Bowl/bag/average scores: direct 49.0/70.0/59.5, model-free 52.0/70.7/61.3, model-based 60.0/84.0/72.0. Evaluation uses harder initial conditions; separate trial counts and intervals are absent.
Go to primary source ↓e-libero-ablationPDF p. 9, Section 5.2 ablation paragraph; p. 20, Table 4
Independent auxiliary-loss and pretrained-initialization removals use the same gradient-step count. Average scores are 98.5, 97.0, 94.6; Long scores 97.6, 94.0, 88.6.
Go to primary source ↓e-discussionPDF pp. 10–11, Section 6
The authors identify slow model-based search, substantial rollout needs and one-layer search as limitations; deeper and longer-horizon prediction is proposed future work.
Go to primary source ↓e-releasePDF p. 11, Section 7 Reproducibility Statement
The authors state that model checkpoints, training data and code are released through their project website; the statement itself does not verify those artifacts.
Go to primary source ↓e-implementationPDF p. 15, Appendix A.1; p. 16, Figure 8 and caption
Blank-image latents are overwritten with repeated normalized vectors. Averaging and unnormalization recover actions/values without VAE decoding. A first blank and four identical image copies accommodate the tokenizer.
Go to primary source ↓e-noisePDF pp. 15–16, Appendix A.2.1 and Figure 9
Training uses 0.7 log-normal with log parameters 1.39/1.2 plus 0.3 uniform over [1,85]. Inference bounds are σmin=4 and σmax=80; mixture weights were not tuned.
Go to primary source ↓e-trainingPDF p. 17, Appendices A.2.2–A.2.4
Full model fine-tuning uses LIBERO 40K/1920/64 H100, RoboCasa 45K/800/32 H100, ALOHA 50K/200/8 H100, all 48 hours. Predicted/executed chunk sizes are 16/16, 32/16, 50/50. ALOHA uses 185 demonstrations.
Go to primary source ↓e-decodingPDF pp. 17–18, Appendix A.3.1
Direct generation is parallel with five simulation or ten ALOHA denoising steps; future/value outputs are discarded. Planning generates action, future and value autoregressively using 10/5/5 steps and 3-by-5 ensembles.
Go to primary source ↓e-training-maskPDF pp. 20–21, Appendix A.4.1 and Figure 12/caption
The fixed latent sequence is conditioned differently for 50/25/25 initial training. During refinement, value masking yields V(s′) or Q(s,a).
Go to primary source ↓e-robocasa-ablationPDF p. 22, Table 5, complete table and caption
Cumulative objective removals score 67.1, 66.6, 64.0, 62.5 and 44.4 under matching compute; one-step inference scores 66.4. The caption's −0.5% loss disagrees with the 67.1-to-66.4 rows.
Go to primary source ↓e-latencyPDF pp. 21–22, Appendix A.4.2
Direct latencies are 0.61/0.95/0.16 seconds on one H100 for 5/10/1 steps. ALOHA best-of-8 uses eight H100s, three future predictions and five values per future, totaling 4.9 seconds.
Go to primary source ↓8.5 Primary sources
Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning ↗
PDF · 12,126 extracted words
Source fingerprint
05696475c9b5481afbf88673a65f80a46cf3fb87a5755c1410e83d0f468cc6e0