NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation
1. Paper overview
In one sentence: OmniDreams turns a video backbone into a responsive camera simulator, trading substantial compute and chunk-level feedback for controllable visual generation. e-probleme-modele-servinge-speede-wame-qualitye-incidents
| At a glance | What to know |
|---|---|
| Research problem | Source description Driving evaluation needs observations that respond to the policy's actions, including conditions absent from a captured scene. OmniDreams supplies AlpaSim's visual environment; policy, traffic and physics services retain responsibility for trajectories and state updates. e-probleme-serving |
| Core mechanism | Source description Structured conditioning, causal video generation, multi-view attention, Self Forcing distillation and stateful serving make a generative renderer usable in closed-loop simulation. e-controlse-modele-multiviewe-distille-serving |
| A key reported result | Real-time camera generation: SV: 118 ms / 68 FPS on one GPU. Four-view MV: 151 ms / 105 FPS per camera on sixteen GPUs. Latency and effective FPS. 704×1280; two-step model; NVIDIA GB300; per-chunk timings. MV on one GPU: 1,289 ms / 12 FPS. Measured renderer throughput; totals exclude asynchronous cache updates and do not establish complete policy-loop latency. e-speede-serving |
| Reading caution | Source description Figure 4 places cross-view attention before text cross-attention; its caption and Section 4.1 say after. Section 9.1 says LightVAE decoder, whereas Table 5 and Section 5.1 specify LightTAE. These discrepancies remain unresolved. e-multiviewe-decoder |
Core contributions
- Source description
Structured conditioning, causal video generation, multi-view attention, Self Forcing distillation and stateful serving make a generative renderer usable in closed-loop simulation. e-controlse-modele-multiviewe-distille-serving
Figure 4. Camera streams share scene information while retaining causal visual histories. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the four colored streams downward from noisy video and conditioning to denoised tokens. View and time embeddings feed the shift, scale and gate modulation paths. The triangular causal-mask marker points to self-attention: visual context is restricted to the current and earlier frames. The red horizontal connection identifies exchange between camera views at a common time; the green text connection supplies appearance conditioning. Section 3.2 explains why these operations are factorized instead of applying full attention across every view and time. Read the layer order cautiously: the drawing places cross-view attention above text cross-attention, whereas the caption and Section 4.1 explicitly place it after text cross-attention. e-modele-multiviewe-wame-serving
What it supports. The architecture supplies two complementary constraints: causal attention carries each camera's temporal context, and cross-view attention couples simultaneous observations. This makes four-view generation a coordinated prediction problem. The diagram depicts the sensor generator; the driving-policy extension uses a different, additional trajectory head described in Section 7.
Where the evidence stops. The caption and p. 11 agree on a layer order that conflicts with the drawing. The crop is unchanged; the actual implemented order cannot be resolved from this PDF. The causal mask does not imply continuous mid-chunk action feedback.
2. Motivation
2.1 The problem and the proposed response
Driving evaluation needs observations that respond to the policy's actions, including conditions absent from a captured scene. OmniDreams supplies AlpaSim's visual environment; policy, traffic and physics services retain responsibility for trajectories and state updates. e-probleme-serving
2.2 What this reading follows
A driving simulator must show the policy what its latest actions would make the cameras see. OmniDreams addresses that rendering problem with a causal video generator conditioned on structured scene state, text and visual history. AlpaSim still manages trajectories, traffic and physics. The paper also asks whether the learned backbone can become a driving policy, and reports a separate trajectory-prediction variant. Read the evidence in three layers: the architecture and training needed for streaming generation, the cost and quality of the fast implementation, and the extent to which simulator measurements preserve useful conclusions about policies. The tables expose several qualifications to the paper's broader claims. e-probleme-modele-servinge-speede-wame-qualitye-incidents
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational neural-simulator/video-backbone classification fits the main system. Its renderer couples to separate policy and physics services, supporting the catalog's not-applicable quadrant for that contribution. Section 7 adds a WAM: shared causal visual features feed a separate trajectory flow head, with joint training and trajectory-only inference. Joint losses alone do not establish a unified future-video/action predictor at deployment. e-probleme-modele-servinge-wam
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow an action through the simulator before interpreting its video
The controlling input is a trajectory translated into camera-aligned world-scenario renders. Static map lines and actor cuboids constrain the scene, while text and the first RGB image constrain appearance. AlpaSim chooses the ego and traffic trajectories, interpolates them over the requested chunk and sends the corresponding conditions to the video service. OmniDreams then synthesizes observations using the recent visual cache and returns them to the policy loop. This separates the learned image-generation problem from the services that update abstract state. Reader interpretation: the model's causal attention makes streaming possible, but the operational feedback boundary is the chunk, not each rendered frame. A policy cannot revise a committed trajectory halfway through a generated chunk without invalidating its observations. That distinction is essential when interpreting the throughput table. e-controlse-modele-servinge-speed
5.2 Distinguish causal training, exposure-bias correction and deployment speed
Converting the bidirectional Cosmos model to causal attention changes which context it can use. Diffusion Forcing trains with independent frame noise levels, but the resulting student still encounters its own imperfect history during deployment. Self Forcing addresses that mismatch by training on generated rollouts; the DMD objective uses frozen real and learned fake score networks, while gradients through historical cached embeddings are stopped. Continued training with a longer bidirectional teacher targets artifacts that emerge when generation outlasts the shorter teacher's context. Table 6 supports reduced degradation across twenty seconds, although it also shows remaining drift. Deployment then makes another tradeoff: fast LightTAE decoding loses quality relative to the original VAE. Reader interpretation: the strongest quality number, the longest-horizon claim and the fastest throughput should not be combined into one unqualified configuration claim. e-causale-distille-longe-runtimee-decodere-speed
5.3 Separate learning to render from learning to choose a trajectory
The WAM experiment begins with a causal single-view checkpoint before map-control finetuning, then adds visual conditioning and an ego-history token. That token can read current and previous video tokens and feeds a separate flow-matching trajectory MLP. Training combines video and trajectory denoising losses, but policy inference runs the DiT once and samples only the trajectory head. The model therefore benefits from a video-trained representation without rendering candidate futures for planning. The 574-scene experiment reports lower collision at ten-hertz replanning; the 501-scene experiment instead compares policy ordering across two renderers at a slower chunk cadence. Reader interpretation: the former is evidence for a useful policy backbone, while the latter tests a simulator's evaluation behavior. Neither alone proves that joint video/action training causes the gain or that simulation ranks policies correctly on real roads. e-wame-wam-evale-eval-protocole-incidents
5.4 Training and inference
During training
RDS supplies 16,600 hours and RDS-HQ-1M 4,944 hours. The latter has 1,142,285 clips; 5,000 held-out clips are scenario-balanced rather than sampled proportionally. Filtering removes unreliable sensors/annotations, visual artifacts and repetition; exact thresholds are not supplied. e-data
AV/multi-view adaptation precedes causal Diffusion Forcing on RDS, then map-controlled training on RDS-HQ-1M. Self Forcing uses generated history, detached historical KV embeddings, a frozen real score network and learned fake score network. Two-step distillation uses [1000,450]; a 58k-video subset and subsequent longer-teacher training target rollout artifacts. e-causale-distille-multiview
WAM training sums video and trajectory flow losses with independent noise times. The diffusion-fixer variant learns degraded-render-to-clean-image correction from paired targets. e-wame-fixer
During inference
Rendering uses two denoising steps, bounded static caches, CUDA graph replay and LightTAE decoding. Local windows are six latent frames for SV and eight for MV; initial-image attention sinks supplement recent history. MV uses pixel-shuffle conditioning encoding. e-runtimee-speed
WAM inference processes four slightly noised video latent frames and one history token in one DiT pass. Only the trajectory MLP performs four sampling steps. It does not generate future video or search imagined rollouts. e-wam
5.5 Implementation flow
- Render actions into conditioning
Maps and actor cuboids are projected using camera calibration. Ego motion changes the requested viewpoint. Text controls appearance; a small MLP converts structured control into tokens concatenated with visual tokens. e-controls
- Predict observations causally
First-frame latents initialize a Cosmos-derived DiT; attention keys and values persist across chunks. SV emits eight RGB frames from two latent frames; MV emits sixteen from four per camera. Temporal attention stays within a camera; cross-view attention exchanges simultaneous information. e-modele-multiview
- Keep simulation time consistent
AlpaSim commits ego and traffic trajectories for each chunk before requesting frames. Session identifiers select caches. Returned frames enter the timeline at their capture times. Rendering throughput therefore does not imply mid-chunk policy feedback. e-serving
- Repurpose the backbone as a policy
The WAM initializes before map-control tuning. A token embeds 1.6 seconds of ego history and attends to current/past video tokens, excluding other history tokens. Its output conditions a 12-layer U-Net-shaped trajectory MLP; DINOv2-initialized and telescope features add visual conditioning. e-wam
6. Experiments & results
OmniDreams adapts Cosmos into a causal, action-conditioned camera simulator with few-step diffusion and reusable visual history. Its strongest systems result is four-camera rendering at 105 effective FPS per camera on sixteen GB300 GPUs. A separately post-trained trajectory policy also improves simulated collision rates. These findings establish rendering and policy capabilities under distinct protocols, with compute, decoder quality and chunk-level feedback tradeoffs.
6.1 Read the original evidence
Table 3. The reported 105 FPS per camera requires sixteen GB300 GPUs. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read columns as GPU counts and rows as stages of one four-camera generation request. Every request produces sixteen RGB frames per camera, corresponding to four latent frames, at 704 by 1280 resolution with two denoising steps. The caption specifies that effective FPS is expressed per camera. Do not add the KV-cache row to Total: the authors exclude it because cache maintenance runs on a side thread outside the critical path. At sixteen GPUs, the listed total combines 121 ms of DiT work and 30 ms of decoding. The conditioning encoder uses pixel shuffle and is described as having negligible latency. e-speede-runtimee-servinge-eval-protocol
What it supports. The measured four-view total falls from 1,289 ms on one GB300 to 151 ms on sixteen, with reported throughput increasing from 12 to 105 effective FPS per camera. The model crosses the paper's 30 FPS real-time threshold in the four-GPU configuration, which reports 48 FPS.
Where the evidence stops. These are renderer timings, not measurements of policy inference, network transport, traffic and physics combined. Nor does 105 FPS permit 105 policy corrections each second: the evaluated multi-view loop commits trajectories for 533 ms chunks.
Figure 13. Policy ranking transfers across renderers more closely than absolute incident rates. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend first: orange denotes NuRec and green denotes OmniDreams. Each pair evaluates the same policy, in the order WAM, full Alpamayo 1.5, two-camera Alpamayo and one-camera Alpamayo. The left panel summarizes all incidents; the remaining panels separate front, lateral and rear collisions from offroad events. Their scales differ, and the front-collision and offroad panels contain broken axes, so bar heights cannot be compared across panels. The figure reports means over three trials. The supporting protocol uses 501 compatible scenes, twenty-second rollouts and 533 ms replanning, counting incidents only while the ego stays within four meters of the recorded trajectory. e-eval-protocole-incidentse-wam-eval
What it supports. Both renderers order the policies from WAM through four-, two- and one-camera Alpamayo. Absolute rates differ: WAM rises from 4.7% all incidents in NuRec to 10.9% in OmniDreams, and full Alpamayo rises from 10.1% to 18.1%. The figure supports ranking agreement in this setup, rather than numerical interchangeability.
Where the evidence stops. The plot provides no error bars despite averaging trials. The trajectory gate and compatible-scene subset restrict the inference. This figure is distinct from the 574-scene, 10 Hz collision comparison and supplies no direct real-road safety validation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-time camera generation 704×1280; two-step model; NVIDIA GB300; per-chunk timings. | SV: 118 ms / 68 FPS on one GPU. Four-view MV: 151 ms / 105 FPS per camera on sixteen GPUs. Latency and effective FPS | MV on one GPU: 1,289 ms / 12 FPS. Measured renderer throughput; totals exclude asynchronous cache updates and do not establish complete policy-loop latency. e-speede-serving |
| Training-stage simulation quality SV; 1,000 clips from the 5,000-clip RDS-HQ-1M holdout; Table 4. | Self Forcing: 24.8; 1.90; 0.400; 0.828. FVD ↓; Temporal Sampson ↓; LET-AP ↑; lane F1 ↑ | Bidirectional: 26.8; 1.83; 0.378; 0.823. Diffusion Forcing: 31.7; 1.87; 0.221; 0.775. Most columns improve; Temporal Sampson worsens. Data and optimization also change, so this does not isolate the loss. e-qualitye-distill |
| Fast-decoder quality tradeoff Distilled SV; Section 9.1 held-out evaluation; Table 5. | LightTAE: 45.4; 0.376; 0.813. FVD ↓; LET-AP ↑; lane F1 ↑ | Original VAE: 24.8; 0.400; 0.828. The fast deployment should not inherit original-decoder quality scores. Table 5 omits paired decoder latencies. e-decodere-quality |
| Long-rollout teacher ablation Twenty-second front-wide rollouts, four five-second windows against a common real-video reference; Table 6. | Progressive long-context teacher: 268.4 / 179.4. Final-window / mean segmented FVD ↓ | Short-context teacher: 409.2 / 240.0. Drift remains: progressive-teacher FVD starts at 95.5. Table 6 does not specify sample count or uncertainty. e-long |
| WAM policy collision comparison 574 Physical AI Autonomous Vehicles NuRec scenes excluding WAM training scenes; 20-second closed-loop rollouts; 10 Hz replanning. | 4.2% with roughly 2B parameters. Collision rate ↓ | Alpamayo 1.5: 6.9%, roughly 10B parameters. A 2.7-percentage-point simulated reduction. This preliminary comparison does not isolate pretraining from architecture, data or camera inputs; Section 7.2 supplies no uncertainty. e-wame-wam-eval |
| Policy ranking across sensor simulators 501 compatible NuRec scenes; 20 seconds; 533 ms replanning; incidents count within 4 m of the recorded trajectory; mean over three trials. | OmniDreams: WAM 10.9%; Alpamayo 1.5 18.1%; two-camera 24.5%; one-camera 51.3%. All Incidents (Collision + Offroad) ↓ | NuRec: 4.7%; 10.1%; 20.9%; 51.9%, respectively. Ordering agrees while absolute rates shift. This differs from the 574-scene/10 Hz protocol and is not real-world safety validation. e-eval-protocole-incidents |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Self Forcing recovers most quality measures after conversion to causal generation. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the row transition: an AV-adapted bidirectional generator becomes a causal Diffusion Forcing model, then a few-step Self Forcing student. Section 9.1 evaluates single-view outputs on 1,000 clips sampled from the scenario-balanced 5,000-clip RDS-HQ-1M holdout. The column groups measure different properties. FVD compares video distributions with real footage; Temporal Sampson measures cross-frame geometric consistency. BEVFormer supplies the vehicle-detection scores, while LATR supplies lane metrics on synthesized images. Follow each arrow separately. The bold 24.8 in the distilled row is best FVD, but the bold 1.83 in the bidirectional row is best Temporal Sampson. e-datae-qualitye-distill
What it supports. FVD deteriorates from 26.8 to 31.7 after causal conversion, then improves to 24.8 after Self Forcing. Distilled LET-AP reaches 0.400 and lane F1 reaches 0.828. Temporal Sampson instead rises to 1.90, so the prose's broad claim of strongest fidelity needs this explicit exception.
Where the evidence stops. The rows change training stage, optimization and data selection together; the result does not isolate Self Forcing's loss alone. Detector performance on synthetic images and distribution-level FVD also cannot certify the simulator's physical accuracy.
Table 5. Decoder acceleration carries a measurable quality cost. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows within each metric rather than comparing this table's LightTAE row to an unrelated model. The distilled backbone is presented with its original VAE and with a LightTAE decoder, under the Section 9.1 quality evaluation. The first two columns reward lower values; detection and most lane columns reward higher values. Table 5 labels the fast decoder LightTAE, consistent with the inference description in Section 5.1. Section 9.1's preceding sentence instead calls it LightVAE, an unresolved naming inconsistency. The distinction matters when associating quality measurements with the fast rendering system, because Table 4's best FVD belongs to the original decoder configuration. e-qualitye-decodere-runtime
What it supports. FVD increases from 24.8 to 45.4 with LightTAE. The changes also reach driving-relevant perception proxies: LET-AP falls from 0.400 to 0.376 and lane F1 from 0.828 to 0.813. The table therefore supports a quality-versus-speed tradeoff, although it does not measure downstream policy degradation.
Where the evidence stops. Table 5 contains no paired decoder latency measurements, so it cannot quantify the speed gained per unit of quality lost. Preserve the LightTAE table label and disclose the conflicting LightVAE wording instead of silently harmonizing them.
Table 6. A longer teacher reduces the growth of rollout error without eliminating it. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read left to right as time advances through a twenty-second generated video. Each five-second segment is compared with the same real front-wide video reference distribution, so the rising numbers expose degradation over the rollout. Compare rows within a window to assess the progressive teacher. The final column, delta, is defined in the caption as final-window FVD minus first-window FVD; it is not an uncertainty estimate. Figure 9 on the preceding page supplies matching qualitative examples with visible shifting artifacts after short-teacher distillation. Section 4.3 explains the intervention as continued distillation using a longer-context bidirectional teacher after the short-context stage. e-longe-distill
What it supports. The progressive teacher lowers final-window FVD from 409.2 to 268.4 and mean FVD from 240.0 to 179.4. Its final-minus-first gap falls from 299.9 to 172.9. Nevertheless, its first and last windows still differ substantially, at 95.5 and 268.4; improved stability is a narrower conclusion than drift-free generation.
Where the evidence stops. The table does not specify its evaluation sample count, confidence intervals or equalized extra training compute. These twenty-second measurements do not quantify the paper's broader claim of maintaining decent quality for minutes.
7. Analysis & limitations
7.1 What the evidence leaves open
Figure 4 places cross-view attention before text cross-attention; its caption and Section 4.1 say after. Section 9.1 says LightVAE decoder, whereas Table 5 and Section 5.1 specify LightTAE. These discrepancies remain unresolved. e-multiviewe-decoder
Chunk commitments constrain reaction frequency; the authors acknowledge higher compute than reconstruction-based simulation. FVD and detector outputs measure visual/statistical fidelity, not physical correctness. Trajectory-gated ranking agreement does not validate hazards outside that regime. e-servinge-coste-qualitye-eval-protocole-incidents
Figure 14 supports better visual extrapolation along fixed off-log trajectories, not their safety. Scenario edits, unusual objects and diffusion-fixer examples lack quantitative control or safety tests. e-deviatione-editse-fixer
7.2 Questions for discussion
- Would policy rankings persist without the 4 m incident gate?
- How much of the longer-teacher gain survives equalizing extra distillation compute?
- Does decoder acceleration change policy decisions more than aggregate FVD suggests?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs exact checkpoints, curated data manifests, calibrated maps/tracks, teachers, decoder and AlpaSim configuration. The PDF lacks a complete optimizer/batch/step schedule, training hardware budget, package versions and WAM training-set specification. Its log-normal versus logit-normal time descriptions also need clarification. e-datae-controlse-causale-distille-runtimee-wam
Proposed checks: equalize additional optimization when comparing teacher context lengths; separately compare both sensor backends using fixed scenes and paired incident-rate differences as well as ranking. e-longe-eval-protocole-incidents
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does teacher context help beyond simply training longer?
Reader-proposed experiment: branch the same short-teacher student into two continued-distillation runs, one retaining the short teacher and one using the longer teacher. Match the 58k-video mixture, update count, optimizer, two-step schedule, cache window, decoder and seeds; report training compute as well. Generate identical held-out twenty-second scenarios and measure all four segmented FVD windows against one fixed reference distribution. A persistent reduction in final-minus-first FVD under these controls would support the context explanation; a gain that vanishes with matched training would weaken it. Repeat across seeds and report variability. e-distille-runtimee-long
Check 2: Does ranking agreement survive a test of incident calibration?
Reader-proposed experiment: repeat the 501-scene, twenty-second comparison using the same policy checkpoints, 533 ms replanning, initial states, traffic/physics services and camera subsets for both renderers. Pair scenes and seeds, retain the original four-meter incident gate, and measure both policy ranking and per-policy incident-rate differences with paired uncertainty estimates. As a separately labeled sensitivity analysis, also count events without the gate and stratify by trajectory deviation. Stable ranking with large systematic rate shifts would support comparative selection but challenge absolute-risk calibration; changed rankings would narrow even the selection claim. Do not pool these results with the 574-scene protocol. e-eval-protocole-incidentse-deviatione-wam-eval
8.3 Reading coverage
Visual audit: The title and corporate byline on p. 1 and research contributors on p. 30 were visually verified. Figures 1–15 and Tables 1–6 were inspected on the original PDF pages, along with the method, training, inference and evaluation text supporting retained claims. All six final crops were separately opened and inspected. Figure 4's layer-order conflict and the Table 5/Section 9.1 decoder-name conflict are preserved. Static figure panels were reviewed; linked videos, code, external releases and separate supplements were not inspected. Contents, remaining related-work/conclusion pages, references and the glossary were read as supplied text but were not visually inspected; no additional numerical or implementation claims rely on those pages.
PDF pages inspected for this edition: 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and contents (pp. 1–3)
- 1. Introduction (pp. 4–5)
- 2. Data, all subsections (pp. 5–7)
- 3. Model Architecture (pp. 8–9)
- 4. Training, all subsections and equations (pp. 10–13)
- 5. Training-free Model Inference Optimization (pp. 13–15)
- 6. Closed-Loop Simulation Integration (pp. 15–17)
- 7. World-Action Model post-training (pp. 17–18)
- 8. Diffusion Fixer (pp. 18–19)
- 9. Experiments and Results, all subsections (pp. 19–26)
- 10. Related Work (pp. 26–28)
- 11. Conclusion (p. 29)
- A. Contributors and Acknowledgments (p. 30)
- References (pp. 31–35)
- B. Glossary (pp. 36–39)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/edition note: the exact catalog title and identifier match the supplied v2 PDF. Its arXiv margin is dated 23 July 2026, matching the catalog submittedDate; the report dateline is 27 July 2026. No earlier edition was supplied, so further revision differences are not established.
- Authorship note: the title-page byline is NVIDIA. The catalog combines that corporate byline with the research contributors in Appendix A.1; those names agree. No individual affiliation mapping is provided.
- All 11 supplied text chunks were read. The extraction does not reconstruct figure images; this limitation was addressed by inspecting the original PDF pages and all fifteen figures and six tables.
- Separate supplemental material availability has not been fully verified; none was supplied.
- Code, linked model weights and external resources were not inspected. No experiments were reproduced. Visual inspection covers static PDF panels, not linked videos.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, dateline and arXiv margin; p. 30, Appendix A.1
The exact title matches. The byline is NVIDIA; A.1 lists the catalog's research contributors. The margin identifies arXiv:2606.03159v2, 23 July 2026; the report dateline is 2026-7-27.
Go to primary source ↓e-problemPDF p. 4, Section 1 and Figure 1
The policy supplies actions, AlpaSim updates abstract state, and OmniDreams returns camera observations. Reconstruction-based scenes are constrained by captured content.
Go to primary source ↓e-dataPDF pp. 5–7, Sections 2.1–2.5 and Table 1
RDS supplies 16,600 hours; RDS-HQ-1M supplies 4,944 hours and 1,142,285 clips. A scenario-balanced 5,000-clip holdout includes 300 clips with 60-second versions. Curation filters unreliable data, annotations, artifacts and repetition.
Go to primary source ↓e-controlsPDF p. 6, Section 2.2 and Figure 2; p. 9, Section 3.1
Calibrated maps and actor cuboids are rendered into camera-aligned conditioning videos. Ego actions change these renders. Text supplies appearance conditions; a lightweight MLP produces control tokens.
Go to primary source ↓e-modelPDF pp. 8–9, Sections 3–3.2 and Figure 3
A Cosmos-Predict 2.5-derived causal transformer uses first-frame RGB, text, world-scenario control and cached history. SV generates eight RGB frames per step; MV generates sixteen per camera across four views.
Go to primary source ↓e-multiviewPDF pp. 9–11, Section 3.2, Figure 4 diagram and caption, Section 4.1
Temporal attention is per view; cross-view attention connects simultaneous views. View embeddings modulate AdaLN. Figure 4 draws cross-view attention before text cross-attention; its caption and p. 11 place it after text cross-attention, before the MLP.
Go to primary source ↓e-flowPDF p. 10, Section 4.1, Preamble
Rectified flow interpolates clean video latents and Gaussian noise, predicting noise minus clean latents with squared error. Auxiliary conditioning includes text, initial images and control.
Go to primary source ↓e-causalPDF pp. 11–12, Section 4.2, Causal Masking and Equation (1)
Diffusion Forcing uses causal attention and independently sampled frame noise levels. RDS training precedes controlled RDS-HQ-1M training. Section 4.2 calls its time distribution log-normal; Section 4.1 calls the bounded scalar distribution logit-normal.
Go to primary source ↓e-distillPDF pp. 12–13, Section 4.3 and Equation (2)
Self Forcing uses generated history, two denoising steps at [1000,450], detached historical KV embeddings and DMD with frozen real and learned fake score networks. Continued distillation uses a longer bidirectional teacher; its data subset has 58k videos.
Go to primary source ↓e-runtimePDF p. 13, Section 5.1; p. 21, Section 9.2
Inference uses static caches, CUDA graphs, six/eight latent-frame local windows for SV/MV, LightTAE decoding and pixel-shuffle conditioning encoding for MV. Section 9.2 identifies first-RGB-frame tokens as attention sinks.
Go to primary source ↓e-speedPDF p. 14, Tables 2–3 and Sections 5.2–5.3
At 704×1280 with two denoising steps, SV takes 118 ms for eight frames on one GB300. Four-view MV takes 1,289/330/209/151 ms on 1/4/8/16 GB300 GPUs, reporting 12/48/74/105 effective FPS per camera. Totals exclude side-thread cache updates.
Go to primary source ↓e-servingPDF pp. 15–17, Figure 6 and Sections 6.1–6.4.2
A stateful gRPC service retains session caches. AlpaSim uses pre-fetch: trajectories are committed at chunk boundaries and interpolated for rendering; frames enter the timeline in order. Mid-chunk trajectory changes invalidate a chunk.
Go to primary source ↓e-wamPDF pp. 17–18, Section 7 and Section 7.1
The policy starts from the SV causal checkpoint before map-control tuning. DINOv2-initialized features, telescope-camera features and ego-history tokens condition a trajectory flow MLP. Video and trajectory losses train jointly; inference runs the DiT once and the trajectory MLP takes four flow steps.
Go to primary source ↓e-wam-evalPDF p. 18, Section 7.2
On 574 NuRec dataset scenes excluding WAM training scenes, 20-second rollouts at 10 Hz yield collision 4.2% for roughly 2B WAM versus 6.9% for roughly 10B Alpamayo 1.5. Front/lateral/rear values are 0.9/0.4/3.0% versus 1.0/0.6/5.3%.
Go to primary source ↓e-fixerPDF pp. 18–19, Section 8 and Figure 7
A separate variant learns from degraded-render/clean-target pairs, starting its correction flow from the degraded rendering. Figure 7 supplies qualitative correction examples.
Go to primary source ↓e-qualityPDF p. 20, Section 9.1 and Table 4, all rows
On 1,000 clips sampled from the 5,000-clip RDS-HQ-1M holdout, bidirectional/causal/distilled FVD is 26.8/31.7/24.8 and Temporal Sampson is 1.83/1.87/1.90. Distilled LET-AP is 0.400 and lane F1 is 0.828.
Go to primary source ↓e-decoderPDF p. 21, Table 5; p. 20, Section 9.1; p. 13, lightweight encoders and decoders
Original-VAE versus LightTAE-decoder FVD is 24.8 versus 45.4; LET-AP is 0.400 versus 0.376; lane F1 is 0.828 versus 0.813. Section 9.1 says LightVAE decoder, but Table 5 and Section 5.1 identify LightTAE.
Go to primary source ↓e-longPDF pp. 21–22, Section 9.2, Figure 9 and Table 6
Twenty-second rollouts are divided into five-second windows against a common real front-wide reference. Progressive-teacher final-window FVD is 268.4 versus 409.2; mean is 179.4 versus 240.0; final-minus-first delta is 172.9 versus 299.9.
Go to primary source ↓e-editsPDF pp. 22–24, Sections 9.3.1–9.3.2, Figures 10–11
Prompts, seed images and scenario maps support edits. A separate dynamic-cuboid-dropout variant propagates unusual inserted objects without explicit cuboid trajectories; static cuboids are retained. Evidence is qualitative.
Go to primary source ↓e-eval-protocolPDF pp. 23–24, Section 9.4.1, Policies compared and Evaluation set and metrics
Orchestration, traffic, physics and initial state are held fixed. The comparison uses 501 compatible NuRec scenes excluding WAM training scenes, runs 20 seconds and replans every 533 ms. Incidents count only within 4 m of the recorded trajectory.
Go to primary source ↓e-incidentsPDF p. 25, Figure 13, All Incidents panel and Results and implications
Mean over three trials: WAM, Alpamayo 1.5, two-camera and one-camera variants have incident rates 4.7/10.1/20.9/51.9% in NuRec and 10.9/18.1/24.5/51.3% in OmniDreams. Ranking agrees; absolute rates differ. Front and offroad panels have broken axes.
Go to primary source ↓e-deviationPDF pp. 25–26, Visual Realism and Figure 14
Closed-loop trajectories are frozen and replayed through both renderers. FVD is grouped by mean deviation. The 4+ m bin labels NuRec 207 and OmniDreams 125; its gap is labeled 82.2.
Go to primary source ↓e-costPDF p. 26, Controllability and Compute
The authors explicitly state that generative simulation requires substantially more compute than reconstruction-based simulation.
Go to primary source ↓8.5 Primary sources
NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation ↗
PDF · 18,180 extracted words
Source fingerprint
8cd616c033a02efadb62a0564d22981c0dd8ab123db5d207a951a2927d264afa