Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
1. Paper overview
In one sentence: Cosmos-Transfer1 combines independently trained video-control branches through spatial and temporal weights, trading appearance fidelity against variation in selected regions. controlnetfusionsalient-maprobot-protocolscope-boundary
| At a glance | What to know |
|---|---|
| Research problem | Source description Simulator renderings provide reliable scene layouts but differ visually from real imagery. Transfer1 aims to change appearance while retaining selected structure, using blur, edges, depth or segmentation as controls. Driving-specific HD maps and LiDAR extend this idea to enriching rare scenarios. identitymodalitiesav-preprocessing |
| Core mechanism | Source description A modality-specific ControlNet design permits independent branch training and inference-time composition, including different weights across image locations and time. controlnetfusion |
| A key reported result | Robotics Sim2Real video generation: 0.63; 0.58; 10.42, respectively. Foreground mask mIoU ↑; Diversity-LPIPS ↑; Quality Score ↑. Twenty simulated kitchen scenarios, six prompts each, 120 videos. Setting 2 applies Edge to foreground robots and Seg to background. Seg-only: 0.54; 0.60; 9.29. Setting 1, adding foreground Vis: 0.61; 0.57; 9.57. Setting 2 improves robot-mask alignment and visual quality relative to Seg-only while slightly lowering diversity. This measures generated imagery, not physical task success. robot-protocolrobot-resultsmetricsscope-boundary |
| Reading caution | Reader analysis The study contains no downstream robot-policy or closed-loop driving evaluation. Appearance realism and mask consistency do not establish correct contact dynamics, retained action labels or improved deployment success. robot-protocolrobot-resultsdriving-protocolscope-boundary |
Core contributions
- Source description
A modality-specific ControlNet design permits independent branch training and inference-time composition, including different weights across image locations and time. controlnetfusion
- Source description
TransferBench and regional-control experiments quantify how appearance preservation competes with diversity; robotics and driving studies test structural preservation in generated videos. benchmarkregional-resultsrobot-resultsdriving-results
- Source description
Head-parallel inference reaches faster-than-playback generation of a five-second clip on 64 B200 GPUs. runtime
Figure 2. Separately trained modality branches meet inside the denoiser. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the left column from noisy video tokens to noise prediction. Each right-hand column processes one modality through three transformer blocks. The horizontal paths pass branch outputs through linear projections and multiply them by the corresponding spatial and temporal weight map before adding them to the main stream. The repeated weight symbols show that a modality's map is applied at several injection sites. Figure 1 and Section 2 specify that the output projections are zero-initialized and the backbone is frozen during branch training. Section 3 then combines independently trained branches at inference. These connections represent feature-level control, not a blend of finished videos. controlnetfusion
What it supports. The useful control interface is a map over where and when each modality influences generation. A reader can understand the modularity directly from the separate branches and shared addition sites: different condition streams can be enabled or weighted without retraining them jointly. The diagram's final prediction remains diffusion noise.
Where the evidence stops. The schematic abstracts away feature-channel dimensions and exact map-to-token resampling. Section 3 specifies normalization only where summed weights exceed one. It does not show an action decoder, execution loop or guarantee of physical consistency.
2. Motivation
2.1 The problem and the proposed response
Simulator renderings provide reliable scene layouts but differ visually from real imagery. Transfer1 aims to change appearance while retaining selected structure, using blur, edges, depth or segmentation as controls. Driving-specific HD maps and LiDAR extend this idea to enriching rare scenarios. identitymodalitiesav-preprocessing
2.2 What this reading follows
A simulator can supply a robot's motion and scene layout while its rendered appearance remains far from real imagery. Cosmos-Transfer1 uses those renderings as structured conditions for a diffusion video generator. Its central decision is where each condition should matter: edges and blurred RGB can preserve a foreground object, while depth and segmentation allow a more varied background. The six visuals below explain that information flow and examine the resulting tradeoffs on general video transfer, kitchen-robot scenes and driving data. The evidence supports controllable video generation; whether the generated training data improves an acting system requires a separate experiment. controlnetfusionsalient-maprobot-protocolscope-boundary
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational-work classification fits a controllable video-generation backbone and neural simulator. Its control branches condition pixels; they do not jointly predict actions or infer actions through inverse dynamics. Robot motion is supplied externally, so the recorded architecture, prediction-paradigm and quadrant axes appropriately remain Not applicable for a world-action policy taxonomy. controlnetfusionrobot-protocolscope-boundary
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Train a reusable control branch, then compose it during generation
Begin with the pretrained Cosmos-Predict1 video denoiser. A condition video enters an added branch whose three transformer blocks inherit backbone weights. Zero-initialized output projections connect that branch to the main stream. The initialization therefore starts with no added output from those projections; training learns useful residual control while the backbone stays frozen. Repeat this process independently for each modality. At inference, the branches participate together, and spatial and temporal weights scale their contributions. The authors motivate this separation by training-memory savings, the ability to use different modality datasets and flexible modality selection. Reader interpretation: the modular interface is the key contribution, while the experiments establish useful composition without providing a direct comparison against jointly training all branches. controlnetfusion
Figure 6. Appearance constraints follow the foreground; sparse conditions guide the background. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the four modalities on the left, then inspect their masks in the middle. The caption defines white as weight 0.5 and black as zero. Vis and Edge therefore constrain the selected foreground, while Depth and Segmentation constrain its complement. The prompt supplies the bicycle-shop description, and the right image shows the generated scene. Compare the person's blue shirt and silhouette with the coarse visual input, then look at the changed tools, colors and textures around him. Section 5.2 obtains foreground labels by asking a VLM to classify segment phrases using video frames and the generation caption. The arrows summarize the control routing; Figure 2 supplies its feature-level implementation. salient-mapfusionmodalities
What it supports. This example makes the design goal concrete: preserve appearance where dense visual information matters, while letting the generator vary surrounding details under weaker geometric and instance-layout constraints. The output retains the salient person's appearance while the shop's background changes. It illustrates spatial selectivity within one generated scene.
Where the evidence stops. The masks encode a chosen foreground/background heuristic, not an automatically optimal control policy. This static example cannot establish temporal consistency, and the apparent combination symbol should not be read as literal averaging of condition-image pixels.
5.2 Choose which parts of the scene must remain recognizable
In the bicycle-shop example, preserving the foreground person's appearance and varying the room are different requirements. SalientObject uses detected segment phrases, reference frames and the prompt to assign foreground membership. Vis and Edge then receive weight 0.5 within that foreground, while Depth and Seg receive weight 0.5 outside it. Table 2 swaps this allocation and shows the expected movement of appearance preservation and diversity between regions. Figure 7 separately varies modality weights over 0, 0.333 and 0.5; the reported correlations are 0.93 for foreground Vis alignment and −0.92 for background depth error. Reader interpretation: this is evidence that the weights provide useful control, not proof that the chosen masks or weights are optimal for every task. salient-mapregional-resultsmetrics
5.3 Follow the evidence from generated pixels toward an acting system
The robotics pipeline already has robot motions from task-and-motion planning before Transfer1 generates any video. Its reported foreground mIoU asks whether a segmentation pipeline recovers a similar robot region, while DOVER-technical and LPIPS assess quality and variation. The driving study similarly checks boxes, lanes and reprojection agreement. These measurements are useful intermediate tests: a badly distorted robot or road would undermine augmentation. They do not determine whether simulator action labels remain valid through every contact or whether a policy trained on the output succeeds in reality. Reader interpretation: the next discriminating experiment must hold policy training and data budgets fixed and evaluate behavior. Likewise, the 4.2-second runtime for a five-second clip establishes generation throughput, not a tested feedback-control latency. robot-protocolmetricsdriving-protocolscope-boundaryruntime
5.4 Training and inference
During training
Control branches are trained separately while the base stays frozen, allowing unpaired modality datasets. The general model uses the referenced Cosmos-Predict1 finetuning data; each branch reportedly requires 1024 H100 GPUs for 2–4 weeks. The PDF does not give a complete optimization recipe. fusiongeneral-training
Sample-AV uses RDS-HQ: 65K twenty-second surrounding-view clips, approximately 360 hours, with LiDAR, maps, boxes and dense captions. General-control augmentation randomizes blur parameters, edge thresholds and segmentation colors. av-trainingmodalities
Appendix A trains Pixtral-12B to expand short prompts using condition videos. Gemma-2-9B-it supplies shortened descriptions; one million paired videos per modality are jointly trained for one epoch with FSDP2. The 4K branch separately learns recovery of corrupted high-resolution patches. prompt-upsamplerupscaler
During inference
Choose branches and maps, then fuse their residuals during denoising. The inherited causal tokenizer and patchification yield 56,320 tokens for the stated output. Optional 4K generation uses a 3×3 overlapping grid and averages overlaps at each denoising step. fusiongeneral-trainingupscaler
The accelerated implementation distributes tokens outside attention and uses all-to-all communication to distribute attention heads. Positive and negative prompt denoising run in separate GPU groups. These are generation operations; robot motions in the application dataset come from an external task-and-motion planner. runtimerobot-protocol
5.5 Implementation flow
- Build complementary condition streams
Vis preserves coarse colors through bilateral blur; Edge retains boundaries; DepthAnything2 supplies depth normalized to [0,1]. GroundingDINO detects captioned objects in the first frame and SAM2 propagates masks; randomized mask colors denote instances rather than fixed semantic classes. modalities
- Inject learned residual controls
Each modality receives three transformer blocks initialized from corresponding base-model blocks. Their output linear layers start at zero and add control activations into the frozen generator. The denoiser predicts noise; no robot-action head or inverse-dynamics stage is described. controlnetfusion
- Route control by location and time
A modality's map multiplies its branch activations before addition. Maps may be manual or heuristic; a learned map predictor is suggested as an alternative. Normalize across modalities only at sites whose total weight exceeds one. SalientObject classifies segment phrases using a VLM, reference frames and the generation caption. fusionsalient-map
- Adapt the driving representation
Sample-AV uses a dashcam-adapted base. HDMap includes road geometry and tracked boxes. LiDAR conditioning combines the nearest scan with four neighbors, compensates dynamic points through interpolated boxes, and fills projection holes before generation. av-trainingav-preprocessing
6. Experiments & results
Cosmos-Transfer1 turns structured condition videos into visually varied RGB videos using separately trained control branches around a frozen diffusion transformer. Spatial and temporal weights decide which constraints dominate each region. Its experiments establish a controllability–diversity tradeoff and video-generation throughput; practical policy-transfer benefits remain untested.
6.1 Read the original evidence
Table 1. Uniform fusion improves quality while different controls retain different strengths. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first four rows as single-modality controls, the next four as leave-one-modality-out mixtures, and the final row as four controls each weighted 0.25. The six metric columns should be read separately: arrows point upward for every score except depth error. Bold and underlining mark the source's best and second-best values. TransferBench contains 600 examples spanning manipulation, driving and egocentric activity. Section 5 measures alignment after extracting comparable representations from input and generated videos; diversity instead compares outputs prompted differently. Consequently, a higher diversity score does not mean better reconstruction of the input. Start with the last row, then locate the specialist that beats it in each column. benchmarkmetricsuniform-resultsdepth-discrepancy
What it supports. The full mixture has Quality Score 8.54, but its Blur SSIM of 0.87 is below Vis-only's 0.96 and its Edge F1 of 0.20 is below Edge-only's 0.28. Seg-only permits much more variation, with diversity 0.42 versus 0.22. Fusion offers a compromise, rather than winning every objective.
Where the evidence stops. Section 5.1's prose calls full fusion best for depth, but the original table gives no-Seg 0.43 versus full fusion 0.47, where lower is better. Preserve this discrepancy. The table reports no confidence intervals.
Table 3. Region-specific control improves the robot's visual preservation. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first four rows again use single modalities. Setting 1 combines foreground Vis and Edge with background Seg; Setting 2 removes foreground Vis, permitting robot appearance changes while retaining edge guidance. The scenario set contains twenty kitchen manipulation scenes, each generated under six prompts. Read foreground mask mIoU separately from whole-scene mask mIoU: the former is the column aimed specifically at robot preservation. Then compare it with diversity and quality, because preserving every background pixel is not the goal of augmentation. Finally inspect the depth-error column to see that improvements in selected objectives can coexist with poorer overall geometric alignment. fusionrobot-protocolrobot-resultsmetricsscope-boundary
What it supports. Against Seg-only, Setting 2 raises foreground mask mIoU from 0.54 to 0.63 and Quality Score from 9.29 to 10.42, while diversity drops slightly from 0.60 to 0.58. It does not dominate all columns: depth error reaches 1.41, compared with Vis-only's 0.82. The reported benefit is selective.
Where the evidence stops. Setting 1 lists two raw unit foreground weights; applying Section 3's normalization would make each 0.5, although effective weights are not restated beside the table. Robot-motion labels and real execution success are not validated by these visual metrics.
Table 4. HD maps improve lanes while LiDAR remains strongest on two other measures. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The three rows compare HDMap-only, LiDAR-only and their combination in Sample-AV. Read box mAP and lane mIoU upward, but reprojection error downward. Section 5.4 evaluates 3D boxes with a detector at IoU threshold 0.2 and extracts lane areas with grounded SAM2. Reprojection tests agreement between generated frames using LiDAR-derived depths, camera poses and box annotations, after retaining frames aligned to the original 10 fps scans. This is structural evaluation of generated videos. Compare the fused row against each single-control row individually rather than collapsing the columns into one score: their preferred control inputs differ. driving-protocoldriving-resultsscope-boundary
What it supports. Fusion gives lane mIoU 51.55, above HDMap-only's 50.37 and LiDAR-only's 48.19. However, its box mAP of 44.66 is below LiDAR-only's 46.50, and its reprojection error of 8.67 is slightly above 8.60. The result supports complementary constraints with a measurable tradeoff.
Where the evidence stops. The evaluation sample count, split and uncertainty are not stated. Table 4 does not label percentage units, so values are retained as printed. These metrics do not measure whether a driving policy becomes safer after training on the videos.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multimodal video transfer on TransferBench 600 examples: 200 each from AgiBot World, OpenDV and Ego-Exo-4D. Four modalities have spatially uniform weights of 0.25. | 0.87; 0.20; 0.47; 0.72; 0.22; 8.54, respectively. Blur SSIM ↑; Edge F1 ↑; depth si-RMSE ↓; mask mIoU ↑; Diversity-LPIPS ↑; DOVER-technical Quality Score ↑ | Vis-only: SSIM 0.96 and quality 5.94. Edge-only: F1 0.28. Seg-only: diversity 0.42. No-Seg: depth error 0.43. Fusion has the highest quality score among these configurations, but specialized controls retain advantages. Table 1 contradicts the prose claim that full fusion has the best depth reconstruction. benchmarkmetricsuniform-resultsdepth-discrepancy |
| Foreground–background control allocation TransferBench with SalientObject masks; Vis/Edge each weighted 0.5 on foreground, Depth/Seg each 0.5 on background; compare swapping the pairs. | SSIM 0.81/0.71; diversity 0.01/0.33; quality 8.29. Foreground/background Blur SSIM and Diversity-LPIPS; overall Quality Score | Swapped map: SSIM 0.68/0.93; diversity 0.12/0.03; quality 8.08. Appearance constraints localize preservation while sparse controls permit variation. The comparison changes both regions simultaneously and does not isolate every modality's contribution. benchmarkmetricssalient-mapregional-results |
| Robotics Sim2Real video generation Twenty simulated kitchen scenarios, six prompts each, 120 videos. Setting 2 applies Edge to foreground robots and Seg to background. | 0.63; 0.58; 10.42, respectively. Foreground mask mIoU ↑; Diversity-LPIPS ↑; Quality Score ↑ | Seg-only: 0.54; 0.60; 9.29. Setting 1, adding foreground Vis: 0.61; 0.57; 9.57. Setting 2 improves robot-mask alignment and visual quality relative to Seg-only while slightly lowering diversity. This measures generated imagery, not physical task success. robot-protocolrobot-resultsmetricsscope-boundary |
| Autonomous-driving video structural consistency Sample-AV: HDMap-only, LiDAR-only and fused control. Box mAP uses IoU 0.2; lane masks and LiDAR-based photometric reprojection assess other structure. Evaluation-set size and split are unspecified. | Fusion: 44.66; 51.55; 8.67. 3D-box mAP ↑; lane mIoU ↑; reprojection error ↓, in the table's reported units | HDMap: 41.89; 50.37; 9.46. LiDAR: 46.50; 48.19; 8.60. Fusion improves lane alignment but gives up box accuracy and a little reprojection consistency versus LiDAR-only. The table does not establish an across-metric winner. driving-protocoldriving-results |
| Five-second video generation throughput Cosmos-Transfer1-7B on B200 GPUs in the GB200 NVL72 setting; reported five-second 720p output. | 64 GPUs: 4.2 s end-to-end; 3.5 s diffusion-only. End-to-end generation time ↓; diffusion-only time ↓ | One GPU: 141.7 s end-to-end; 141.0 s diffusion-only. Clip-generation throughput exceeds playback speed at rack-scale compute. Feedback latency, preprocessing inclusion and continuously interactive simulation are not demonstrated. runtime |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Swapping the control map moves the region where variation is allowed. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Decode the eight weight columns before comparing scores. In the first row, foreground receives Vis and Edge at 0.5 each; background receives Depth and Seg at 0.5 each. The second row reverses that allocation. Every paired metric then distinguishes foreground, marked FG, from background, marked BG. Compare the same region across rows: foreground Blur SSIM decreases from 0.81 to 0.68 when appearance controls leave it, while its Diversity-LPIPS rises from 0.01 to 0.12. Background shows the opposite tradeoff. The final Quality Score is a whole-video measure, so it should not be mistaken for another foreground/background pair. salient-mapregional-resultsmetrics
What it supports. The first allocation allows substantial background variation, with BG diversity 0.33, while tightly preserving foreground appearance. Reversing it reduces BG diversity to 0.03 and increases BG SSIM from 0.71 to 0.93. These numbers demonstrate that the location of constraints matters, beyond merely choosing which modalities are active.
Where the evidence stops. Both regional allocations change together, so this is not a clean single-modality causal test. The table supplies no uncertainty estimates. Figure 7 provides a separate weight sweep, but its displayed distributions should not be interpreted as specified confidence intervals.
7. Analysis & limitations
7.1 What the evidence leaves open
The study contains no downstream robot-policy or closed-loop driving evaluation. Appearance realism and mask consistency do not establish correct contact dynamics, retained action labels or improved deployment success. robot-protocolrobot-resultsdriving-protocolscope-boundary
Main quantitative comparisons are internal control variants, with no reported confidence intervals. Sparse conditions can admit artifacts; stronger controls reduce diversity. DOVER-technical is a video-quality proxy, and the mask metric discards correspondences below IoU 0.1. metricsuniform-resultsregional-resultsrobot-resultsdriving-results
Section 5.1 calls full fusion best for depth, while Table 1 gives 0.43 to no-Seg versus 0.47 to full fusion. This report preserves the table values and flags the disagreement. depth-discrepancy
7.2 Questions for discussion
- Can region-specific control preserve action-relevant geometry well enough to improve real robot performance?
- How sensitive are the reported gains to foreground-mask errors and discarded segmentation matches?
- What feedback latency remains when condition extraction and prompt expansion are included?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful implementation needs the base checkpoint/tokenizer, branch training data, preprocessing models and weight maps. The paper leaves optimizer settings, learning rates, full denoising configuration and augmentation distributions unspecified; referring to earlier work does not supply those details here. controlnetfusiongeneral-trainingmodalities
Recover exact benchmark samples, prompts, random seeds and the TransferBench diversity prompt count K before comparing scores. AV evaluation requires calibrated LiDAR, poses, boxes and the detector/segmentation pipelines; its test-set size and split are not stated. benchmarkmetricsdriving-protocoldriving-results
Reader-proposed checks should compare correct versus swapped or shuffled regional maps under fixed prompts and seeds, then test whether matched-budget translated simulation data improves a held-out robot policy. Neither check was performed in this reading. regional-resultsrobot-protocolrobot-results
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does correct regional routing outperform equally strong misplaced control?
Reader-proposed experiment, not performed: use a fixed TransferBench subset, identical prompts, seeds, branches and sampling settings. Compare the published SalientObject map with its foreground/background inversion, masks shuffled between examples, and uniform control. Preserve each regional pair's summed weight and record effective weights after normalization. Measure foreground Blur SSIM, Edge F1 and mask mIoU together with background Diversity-LPIPS and whole-video quality; bootstrap differences over source clips. The routing explanation predicts that correct masks preserve foreground structure better than misplaced masks at a comparable diversity level. If shuffled masks perform equally well, the result would weaken the claim that semantic region selection drives the benefit. benchmarkfusionmetricssalient-mapregional-results
Check 2: Does visual improvement translate into robot-policy success?
Reader-proposed experiment, not performed: generate matched training sets from the same kitchen trajectories using raw simulator RGB, Seg-only transfer and Setting 2 transfer. Keep trajectory count, simulator action labels, prompt count, policy architecture, optimization budget and random seeds matched. Audit robot pose, manipulated-object position and contact timing in translated videos before accepting their original action labels. Train identical policies and evaluate successful cabinet and pick-and-place tasks in held-out real scenes under a fixed trial protocol, reporting uncertainty. If Setting 2 improves visual metrics but not real task success, its claimed augmentation value would remain unsupported; consistent behavioral gains would supply the missing downstream evidence. robot-protocolrobot-resultsscope-boundary
8.3 Reading coverage
Visual audit: The title and revision markings were inspected on p. 1 and contributor credits on p. 23. Pages 2–7 cover the architecture, modality preparation, training scale, driving data and upscaler; pp. 7–12 cover the benchmark, metric definitions, Table 1/prose discrepancy and regional ablations. Pages 12–18 cover robotics and driving protocols, Tables 3–4 and Figures 8–12. Pages 18–19 supply the inference hardware, parallelism and Table 5 timings. Page 20 contains the conclusion, and pp. 21–22 contain prompt-upsampler training and examples. All twelve numbered figures and five tables were visually inspected. Every final crop was separately inspected and retains its original labels. The architecture paths and map values were cross-checked against Sections 2–3 and the Figure 6 caption. Reference-only pp. 24–28 were read as text; no playable videos or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Preliminary
- 3. Method
- 4. Modality and Training
- 5. Evaluations
- 5.1. Unimodal versus Multimodal
- 5.2. Case Study for Spatiotemporal Control Maps
- 5.3. Case Study for Robotics Sim2Real Data Generation
- 5.4. Case Study for Autonomous Driving Data Enrichment
- 6. Real-time Inference
- 7. Related Work
- 8. Conclusion
- A. Prompt Upsampler
- B. Contributors and Acknowledgements
- B.1. Core Contributors
- B.2. Contributors
- B.3. Acknowledgments
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by visually inspecting all twelve numbered figures and all five tables.
- Separate supplemental material availability has not been fully verified.
- The supplied artifact is arXiv:2503.14492v2, marked 1 Apr 2025 in the margin and 2025-4-3 in its title-page header. The catalog gives an initial submission date of 2025-03-18. Only this v2 artifact was supplied; changes from v1 and the reason for the two PDF date markings are unverified.
- The exact title matches. The title-page byline is NVIDIA; Appendix B.1–B.2 lists the same 39 individual contributors represented in the catalog. Its isolated ':' entry is not a credited author. Metadata preserves the collective byline. Individual affiliations are not supplied.
- The full supplied text, including references, was read. PDF pages 1–23 were visually inspected; reference-only pages 24–28 were read as text. Linked code, external cited works and playable videos were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author credit, header and arXiv margin; p. 23, Appendix B.1–B.2
The exact observed title matches the catalog. The byline is NVIDIA. The margin identifies arXiv:2503.14492v2, 1 Apr 2025; the header reads 2025-4-3. Appendix B distinguishes core contributors from other contributors.
Go to primary source ↓controlnetPDF pp. 2–3, Section 2, Figure 1 and Eqs. (1)–(2)
The conditional denoiser predicts noise from noisy video tokens, noise deviation and condition tokens. Three copied transformer blocks and zero-initialized output linear layers form a trainable branch; the base model stays frozen.
Go to primary source ↓fusionPDF pp. 3–4, Section 3 and Figure 2
Each modality has a separately trained control branch. Spatial and temporal weights multiply branch activations before addition into the generator. Weights are normalized to sum to one only where their sum exceeds one; branches are combined at inference.
Go to primary source ↓general-trainingPDF p. 4, Section 4 and footnote 2
The general model is post-trained from Cosmos-Predict1-7B-Video2World using its referenced high-quality finetuning data. Each control branch uses 1024 H100 GPUs for 2–4 weeks. A call produces a 5-second 1280×704 video at 24 fps; footnote 2 computes 56,320 tokens using 121 frames and the causal tokenizer.
Go to primary source ↓modalitiesPDF pp. 4–5, Section 4 modality bullets and Figure 3
Vis uses bilateral blur; Edge uses framewise Canny edges; Depth uses DepthAnything2 normalized to [0,1]; Seg uses first-frame GroundingDINO detection and SAM2 video masks with randomized instance colors. Blur parameters and edge thresholds are randomized during training.
Go to primary source ↓av-trainingPDF pp. 4–6, Section 4, Sample-AV initialization and RDS-HQ description
Sample-AV starts from the dashcam-finetuned base model. RDS-HQ contains 65K 20-second surrounding-view clips, approximately 360 hours, with 10 Hz LiDAR, dense captions, HD maps and 3D box annotations.
Go to primary source ↓av-preprocessingPDF p. 6, Section 4, HDMap and LiDAR bullets; Figure 4
HDMap control includes road annotations and tracked 3D boxes. LiDAR is aligned to 30 fps imagery using the nearest scan and four adjacent scans, box-based interpolation for dynamic points, and a size-4 interpolation kernel.
Go to primary source ↓upscalerPDF pp. 6–7, Section 4, 4KUpscaler description and Figure 5
The upscaler learns to recover high-quality patches from corrupted high-resolution video crops. At inference, a 3×3 overlapping grid produces 4K output, averaging overlaps at each denoising step.
Go to primary source ↓benchmarkPDF p. 7, Section 5, TransferBench paragraph
TransferBench contains 600 examples: 200 each from AgiBot World, OpenDV and Ego-Exo-4D, covering robot manipulation, driving and egocentric daily activity.
Go to primary source ↓metricsPDF pp. 8–9, Section 5 metric definitions and footnote 3
Alignment uses blurred-video SSIM, Canny pixel F1, estimated-depth scale-invariant RMSE and matched-mask mIoU. Matches below IoU 0.1 are discarded. Diversity averages pairwise LPIPS across differently prompted outputs; quality uses DOVER-technical. Edge F1 requires strict pixel alignment.
Go to primary source ↓uniform-resultsPDF p. 10, Table 1, all rows and metric columns
Uniform four-way weights give SSIM 0.87, Edge F1 0.20, depth error 0.47, mask mIoU 0.72, diversity 0.22 and quality 8.54. Vis-only has SSIM 0.96 and quality 5.94; Edge-only F1 is 0.28; Seg-only diversity is 0.42. No-Seg has the lowest depth error, 0.43.
Go to primary source ↓depth-discrepancyPDF p. 9, Section 5.1, final results paragraph; p. 10, Table 1, depth column
The prose says uniform four-modality fusion has the best depth reconstruction. The table instead marks no-Seg at 0.43 as best and the full model at 0.47 as second-best.
Go to primary source ↓salient-mapPDF pp. 9–10, Section 5.2 and Figure 6 with caption
SalientObject uses reference frames, a generation caption and segment phrases to classify masks as foreground or background with a VLM. In Figure 6, white means weight 0.5 and black 0: Vis/Edge constrain foreground, Depth/Seg background.
Go to primary source ↓regional-resultsPDF pp. 11–12, Figure 7, Table 2 and Section 5.2
Table 2's first map yields FG/BG SSIM 0.81/0.71 and diversity 0.01/0.33, with quality 8.29. Swapping modality pairs gives SSIM 0.68/0.93, diversity 0.12/0.03 and quality 8.08. Figure 7 tests weights 0, 0.333 and 0.5; reported Vis and depth correlations are 0.93 and −0.92.
Go to primary source ↓robot-protocolPDF pp. 12–13, Section 5.3, dataset, settings and metric description
Twenty kitchen manipulation scenarios from Omniverse/Isaac Lab each use six prompts, giving 120 generated videos per evaluated setting. Task-and-motion planning supplies robot motion. Setting 1 assigns raw unit Vis and Edge weights to foreground and Seg to background; Setting 2 uses only Edge on foreground and Seg on background.
Go to primary source ↓robot-resultsPDF p. 12, Table 3, Seg, Setting1 and Setting2 rows; pp. 13–14, Figure 8 and interpretation
Setting 2 has foreground mask mIoU 0.63, diversity 0.58 and quality 10.42; Seg-only gives 0.54, 0.60 and 9.29. Setting 1 gives 0.61, 0.57 and 9.57. Setting 2 depth error is 1.41 versus 0.82 for Vis-only. Figure 8 illustrates foreground artifacts under single control.
Go to primary source ↓driving-qualitativePDF pp. 14–18, Section 5.4 and Figures 9–12
Depth/Seg examples use equal weights; HDMap/LiDAR examples use 0.3/0.7. Figures illustrate lane-layout and object-orientation differences, prompt-driven appearance changes, and generation conditioned on synthetic LiDAR. These examples are qualitative.
Go to primary source ↓driving-protocolPDF pp. 16–17, Section 5.4, quantitative metric definitions
3D-box mAP is measured at IoU 0.2 using a detector built upon StreamPetr and Hydra-MDP. Grounded SAM2 produces lane masks. Reprojection uses generated frames synchronized to 10 fps LiDAR, known camera poses and boxes, then L1 photometric discrepancy.
Go to primary source ↓driving-resultsPDF p. 18, Table 4, all rows and columns
HDMap-only yields box mAP 41.89, lane mIoU 50.37 and reprojection error 9.46; LiDAR-only yields 46.50, 48.19 and 8.60; fusion yields 44.66, 51.55 and 8.67. Table 4 does not label percentage units.
Go to primary source ↓runtimePDF pp. 18–19, Section 6 and Table 5
The implementation shards tokens outside attention and uses all-to-all communication for head-parallel attention. Positive and negative prompt denoising are split across GPU groups. Table 5 reports 141.7 seconds end-to-end on one B200 and 4.2 seconds on 64 for a 5-second video; diffusion-only times are 141.0 and 3.5 seconds.
Go to primary source ↓prompt-upsamplerPDF pp. 21–22, Appendix A, training description and examples
Pixtral-12B is finetuned to expand a short prompt using its condition video. Gemma-2-9B-it creates short prompts from existing descriptions; the paired dataset has 1M videos per modality, jointly trained for one epoch with FSDP2. Fidelity, completeness and distribution consistency are objectives.
Go to primary source ↓scope-boundaryPDF pp. 12–18, Sections 5.3–5.4; p. 20, Section 8
The application studies report video-generation metrics and visual examples. The conclusion claims benefits for bridging the simulation-to-real gap, but no trained-policy deployment, executed-action success rate or closed-loop driving evaluation is reported.
Go to primary source ↓8.5 Primary sources
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control ↗
PDF · 11,970 extracted words
Source fingerprint
c9d0256d1a066ed7a165cb1f62b1fee097f15e776a7f792d1998379c6fb0a6d9