DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
1. Paper overview
In one sentence: DrivingWorld couples pose and video prediction through separate temporal and within-frame autoregression, improving reported generation while leaving important decoding and evaluation details unresolved. e01e04e06e07e08e11e14e15e17
| At a glance | What to know |
|---|---|
| Research problem | Author claim A flattened video-token stream makes sparse pose signals compete with hundreds of image tokens and repeatedly attends over a long history. The authors target annotation-light driving prediction with stronger temporal dependencies, useful trajectory conditioning and lower attention cost. e02e03 |
| Core mechanism | |
| A key reported result | NuScenes video generation with NuScenes training: 6.5 / 86.0 FID / FVD; lower is better. Table 1 blue group; evaluation describes 150 NuScenes test clips and equal generated clip length following Vista, without specifying that length. Vista: 6.9 / 89.4. Maximum-generation rows list 40 s / 400 frames for DrivingWorld and 15 s / 150 frames for Vista. Best listed generation distances in this group; maximum duration is separate from the matched-length metric protocol. Training data and model resources differ. e11e12 |
| Reading caution |
Core contributions
Figure 2. Pose and appearance share a next-state predictor while retaining separate representations. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read left to right. Red orientation tokens and blue location tokens come from the vehicle-pose tokenizer; gray image tokens come from the video tokenizer. The central world-model box contains the next-frame and internal-frame modules. On the right, separate decoders turn predicted tokens back into pose and appearance. The caption names historical orientations, locations and images as inputs, while the method text specifies that the next pose is predicted before the next image. Section 3.1 further explains that the pose representation actually uses changes between neighboring timesteps, with orientation and two-dimensional location encoded separately. e03e04e05e09
What it supports. The model can connect a proposed change in ego motion to a predicted visual future. This is more specific than unconditioned video generation: pose information participates in the state representation, and Figure 1 also demonstrates generating futures under two supplied trajectories.
Where the evidence stops. The vehicle drawings depict pose coordinates, not executed controls. Equation (7)'s printed y decoder is inconsistent with its claimed inverse of Equation (1), so exact pose reconstruction needs clarification before implementation.
2. Motivation
2.1 The problem and the proposed response
A flattened video-token stream makes sparse pose signals compete with hundreds of image tokens and repeatedly attends over a long history. The authors target annotation-light driving prediction with stronger temporal dependencies, useful trajectory conditioning and lower attention cost. e02e03
2.2 What this reading follows
DrivingWorld treats a driving scene as a sequence of ego poses and front-view images. The central design problem is that two pose tokens must influence hundreds of image tokens without losing useful history. Its solution separates temporal processing from within-frame fusion, then autoregressively produces a new pose and image. Balanced attention favors sparse modalities, and random masking exposes training to corrupted histories. The six visuals below connect these choices to generation, planning and ablation evidence. This edition reads the supplied arXiv v3, including its conflicting long-video timings; benchmark results remain distinct from evidence of physical vehicle execution. e01e04e06e07e08e11e14e15e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The neural-world-simulator placement is supported by explicit next-state pose/video generation. The recorded Not applicable architecture, prediction paradigm and quadrant are preserved as the foundational-resource scope. Mechanistically, one multimodal predictor generates pose then appearance, with separate tokenizers/decoders; there is no demonstrated inverse-dynamics action extractor. This is architectural evidence, not merely joint training, and does not establish an executed world-action control system. e03e04e06e09e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 A driving state has two kinds of prediction error
Begin with the state in Figure 2. A vehicle's orientation and planar location describe ego motion, while the image describes what the front camera sees. Section 3.1 encodes changes relative to the previous timestep, so the motion representation does not grow with global travel distance. Orientation and the location pair occupy two tokens; an image occupies 512. The temporal-aware image tokenizer adds causal attention around quantization to improve consistency before world-model prediction begins. Table 2 tests this encoding–decoding stage and reports better reconstruction metrics than fine-tuned Llama-Gen. Reader interpretation: this separates two possible failure sources. Poor reconstruction can damage the visible frame even when predicted token dynamics are reasonable, while a strong tokenizer alone cannot guarantee accurate future motion or scene evolution. e04e05e07e13
Figure 5. Temporal context and within-frame token dependence are handled in separate stages. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a): orientation, location and image tokens are grouped by time. Equation (2) defines the temporal layer as following one token position through its past, yielding g features; Equation (3) mixes those features within a timestep to form f features. In panel (b), these f features accompany a shifted prefix beginning with sos, then the next orientation, location and image tokens. Outputs below correspond to the next state, with losses drawn beneath them. Equation (4) retains preceding-token dependence, so the diagram's parallel layout does not establish independent simultaneous generation of all future tokens. e06e07e08
What it supports. The architectural separation preserves temporal context while allowing pose and image tokens to interact within each state. The internal-frame module then restores dependencies among outputs of that state. This is an information-flow argument supported by the equations, rather than a claim that autoregressive decoding has disappeared.
Where the evidence stops. Panel (a)'s upper fan into the rightmost box in the middle feature row appears to include other token positions, unlike Equation (2)'s fixed-position history. Connectivity remains unresolved. Panel (b) also draws pose losses that Equation (6)'s image-token sum does not specify.
5.2 Temporal attention supplies context; internal autoregression orders the outputs
Equation (2) follows one token position across time, and Equation (3) combines modalities within the same timestep. The resulting features therefore carry temporal and spatial information without the formulation requiring every token to attend to a completely flattened history. The next stage adds these features to a shifted token prefix. The stated prediction order places next-state orientation and location before image tokens, allowing the image to depend on the generated pose. Reader interpretation: parallel loss computation with available prefixes is compatible with sequential dependencies at generation time; the paper's parallel-processing language does not remove Equation (4)'s preceding-token dependence. Figure 5a's upper connections also appear broader than Equation (2), so the equation is the verifiable guide to intended temporal connectivity, with the diagram discrepancy left explicit. e04e06e07
5.3 Test conditioning and rollout stability separately
Balanced attention and RMS address different pressures. Equation (5) adds a larger per-token adjustment for a smaller modality before softmax; it does not explicitly equalize total attention mass. RMS instead corrupts some teacher-forced histories, aiming to reduce the gap between clean training prefixes and imperfect generated prefixes. Table 4 improves when both are enabled, but its metric is video distance rather than direct trajectory adherence. Figure 7 supplies a qualitative long-rollout example with conflicting frame-rate descriptions. Table 3 evaluates trajectories, yet changes in inputs and supervision complicate comparisons with other planners. Reader interpretation: a convincing mechanism test should measure commanded-pose adherence and visual degradation separately under identical histories and data, rather than treating one favorable FVD or collision-rate column as proof of every claimed benefit. e07e08e14e15e17
5.4 Training and inference
During training
The 70M-parameter tokenizer uses a 16,384-entry codebook, 1,000K steps, batch 128 and 32 NVIDIA 4090 GPUs, with Openimages, COCO, YoutubeDV and NuPlan. Its losses are Charbonnier, perceptual and codebook losses. e10
The 1B-parameter world model conditions on 15 frames. Training uses over 3,456 hours: 120 NuPlan hours and 3,336 private hours; 450K iterations, batch 64, 64 NVIDIA A100 GPUs and 12 days. NuScenes fine-tuning takes one day. e10
RMS selects sequences with probability epsilon = 0.5; each masked token has eta = 0.3 probability of arbitrary replacement. The fraction selected for masking is not given. Equation (6) shows image-token cross-entropy conditioned on poses, while Figure 5b draws losses on pose tokens too; their complete loss formulation is unspecified. e06e08e10
During inference
The stated order predicts the next pose and then its image, conditioned on history. Preset trajectories provide an alternative conditioning mode. These are predicted or supplied pose changes, not measured steering/throttle commands, and the paper does not describe deployment in a physical closed loop. e03e04e09e14
5.5 Implementation flow
- Encode local motion and images
Relative orientation and location changes are measured between adjacent timesteps in an ego-centered coordinate system; the first changes are zero. Orientation becomes one token and the discretized x/y pair another. A VQGAN-derived tokenizer adds causal temporal attention before and after quantization; each image supplies 512 tokens. e05e07
- Separate temporal and multimodal processing
Image tokens are flattened in zig-zag order. Equation (2) applies causal temporal attention to each fixed token position across history; Equation (3) fuses all modalities within a timestep. Figure 5a's upper fan of connections appears broader than this fixed-position definition, so exact diagram connectivity remains unresolved. e06
- Generate the next state internally
Temporal/fusion features are added to shifted prefixes beginning with a start token, followed by next-state orientation, location and image tokens. Equation (4) retains preceding-token dependence. The paper's parallel-processing wording should not be read as independent simultaneous inference of all tokens. e04e06e07
- Decode and continue
Predicted image tokens retrieve codebook features for temporal attention and convolutional decoding. Pose tokens map back to relative motion. Repeated next-state prediction yields a video rollout; Figure 1 additionally illustrates supplying different future trajectories. No actuator interface or feedback controller is specified. e03e04e09
6. Experiments & results
DrivingWorld predicts driving poses and front-view video through a multimodal autoregressive model. It separates temporal attention from within-frame fusion, then generates pose and image tokens in sequence. Balanced attention protects sparse pose information; corrupted training histories target rollout drift. Reported generation and planning results are promising, but private training data, incomplete implementation details and inconsistent long-video timing limit reproducibility.
6.1 Read the original evidence
Table 1. The reported generation advantage depends on whether NuScenes was seen during training. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the color legend before comparing columns: blue means NuScenes was included in training, and orange means zero-shot evaluation. Within each group, smaller FID and FVD are preferred. DrivingWorld appears twice because these are different training conditions. Its blue column reports 6.5 FID and 86.0 FVD; its orange column reports 7.4 and 90.9. Now read the last two rows separately: they state each method's maximum generation duration and frame count. Section 4.1 says FID and FVD use equal-length generated videos following Vista; it does not say the metric values were measured on those maximum-length outputs. e10e11e12
What it supports. In the seen group, DrivingWorld's 6.5/86.0 improves on Vista's 6.9/89.4. In the zero-shot group, its 7.4/90.9 is better than GenAD (OpenDV)'s 15.4/184.0. These comparisons support generation quality and transfer in the reported setting, with different training distributions explicitly separated.
Where the evidence stops. The exact shared metric clip length is omitted, training resources differ, and no uncertainty is reported. The 40-second row is a capacity statement, not proof that the listed FVD evaluates a full 40-second rollout.
Figure 7. Selected frames illustrate extended generation, with unresolved timing in the source. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the red frame numbers across each row and then downward. They sample the sequence sparsely: adjacent tiles do not represent equal time increments. Road lanes, roadside buildings, vehicles and the forward viewpoint remain recognizable across the displayed images, which motivates the authors' qualitative consistency claim. The final tile is labeled 640. Interpret these labels as frame indices only. The caption on this page says the sequence runs at 5 Hz and lasts 128 seconds, while Section 4.3 on the next page describes 640 generated frames at 10 Hz and 64 seconds after conditioning on 15 frames. e11e17
What it supports. The supplied stills support the narrower observation that the model can produce recognizable driving imagery at widely separated indices of an extended rollout. They give a concrete visual example beyond Table 1's 400-frame maximum entry, but the source leaves the relationship between those reported settings unspecified.
Where the evidence stops. The 5-Hz/128-second caption conflicts with the 10-Hz/64-second text. Neither duration is treated as resolved here. Sparse stills also cannot verify intervening temporal continuity, geometric accuracy or the frequency of failed long rollouts.
Table 3. Front-camera planning is competitive under lighter supervision, but does not lead both average metrics. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First inspect the input and supervision columns. The star on DrivingWorld's Camera entry means front camera only, and its auxiliary supervision is listed as None. Other methods list resources such as maps, boxes, motion or occupancy. Then separate the two numerical blocks: L2 measures trajectory error in meters, while collision rate is a percentage. Each has 1-, 2- and 3-second columns plus an average. DrivingWorld's average is 1.26 m and 0.38%. Comparing averages avoids selecting only its strongest horizon, although the differences in sensing and supervision mean this is not a controlled ablation of architecture alone. e14e18
What it supports. The table's UniAD row has lower average L2 and collision rate, 1.03 m and 0.31%. GenAD has lower average L2, 0.91 m, but higher average collision rate, 0.43%. DrivingWorld therefore offers a reported supervision tradeoff, rather than an across-the-board numerical lead.
Where the evidence stops. These benchmark values do not demonstrate physical closed-loop driving. The paper gives limited planning protocol detail, and UniAD's citation [25] resolves to an unrelated database-system bibliography entry; the intended baseline reference remains unresolved.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| NuScenes video generation with NuScenes training Table 1 blue group; evaluation describes 150 NuScenes test clips and equal generated clip length following Vista, without specifying that length. | 6.5 / 86.0 FID / FVD; lower is better | Vista: 6.9 / 89.4. Maximum-generation rows list 40 s / 400 frames for DrivingWorld and 15 s / 150 frames for Vista. Best listed generation distances in this group; maximum duration is separate from the matched-length metric protocol. Training data and model resources differ. e11e12 |
| Zero-shot NuScenes video generation Table 1 orange group: NuScenes not seen in training. | 7.4 / 90.9 FID / FVD; lower is better | GenAD (OpenDV): 15.4 / 184.0; DriveGAN: 73.4 / 502.3. Supports transfer within the reported driving-video setting, not unrestricted out-of-distribution generalization. e11e12 |
| NuPlan tokenizer reconstruction Encoding–decoding comparison of VQVAEs; tokenizer-specific split is not separately detailed. | 14.66 / 4.29 / 23.82 / 0.1828 FVD_12 / FID / PSNR / LPIPS | Fine-tuned Llama-Gen: 20.33 / 5.19 / 22.71 / 0.1909. Reconstruction improves across all four measures; this does not measure autoregressive future prediction. e13 |
| NuPlan balanced-attention and masking ablation Table 4; FVD subscripts 10, 25 and 40 retained as printed because their units are not explicitly defined. | 122.37 / 144.26 / 159.46 FVD_10 / FVD_25 / FVD_40; lower is better | Without BA: 240.25 / 319.35 / 361.94; without RMS: 214.69 / 283.71 / 324.58; without both: 463.21 / 603.88 / 665.15. Both components improve reported FVD; no uncertainty or control-specific ablation metric is supplied. e15 |
| NuScenes trajectory planning Table 3 and Section 4.6; front camera only, no auxiliary supervision; evaluation at 1, 2 and 3 seconds. | L2: 0.68, 1.20, 1.90; average 1.26. Collision: 0.03, 0.22, 0.90; average 0.38. L2 error (m) / collision rate (%) | UniAD row: averages 1.03 m / 0.31%; GenAD row: 0.91 m / 0.43%. Both list auxiliary supervision. Competitive with different supervision, but not best on either average metric. These benchmark collision rates do not establish real-world safety. e14e18 |
| Human preference against Vista Figure 8; NuPlan and NuScenes comparisons; participant and trial counts are not reported. | Visual quality: 87.3% / 71.7%; controllability: 93.2% / 73.1% (NuPlan / NuScenes). Percentage favoring DrivingWorld | Complementary Vista preferences: 12.7% / 28.3% and 6.8% / 26.9%. Reported preference supports perceived quality and conditioning; statistical precision cannot be assessed. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Balanced attention and random masking each contribute to the reported video-distance improvement. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first row removes both balanced attention (BA) and random masking strategy (RMS). The second removes only BA, retaining RMS; the third removes only RMS, retaining BA. The final row retains both. Read downward within a column before comparing columns, because that holds the printed evaluation label fixed. For FVD_40, the full model scores 159.46, compared with 361.94 without BA, 324.58 without RMS and 665.15 without both. The source labels the columns FVD_10, FVD_25 and FVD_40 but does not explicitly define the subscript units, so they are preserved without conversion to seconds or frames. e07e08e15
What it supports. Either component alone improves all three reported columns relative to removing both, and the full model improves further. This is evidence that both changes matter to the reported generation metric. It does not by itself isolate whether balanced attention improves pose adherence, appearance quality or both.
Where the evidence stops. No seed variation, uncertainty or control-specific ablation metric is supplied. RMS's token-selection fraction is also unspecified. The table therefore supports a component comparison but leaves mechanism attribution and exact replication incomplete.
7. Analysis & limitations
7.1 What the evidence leaves open
Long-rollout timing conflicts: Figure 7 says 640 frames at 5 Hz, or 128 seconds; Section 4.3 says 10 Hz, or 64 seconds. Table 1 separately lists 400 frames/40 seconds. Selected stills cannot resolve the rate or prove continuous 3D consistency. e11e17
Equation (7)'s printed y decoder contains an extra multiplication by v_t relative to the claimed inverse of Equation (1). Pose-bin counts/ranges, optimizer, learning rate, sampling settings, freezing schedule and full pose loss are not supplied. This prevents an unambiguous implementation. e05e06e08e09e10
Most world-model training hours are private. Tables omit uncertainty, and video distances cannot establish collision avoidance. Multi-view integration is future work. Table 3's UniAD reference [25] points to a database-system bibliography entry, leaving that baseline citation inconsistent. e10e11e13e14e15e18e19
7.2 Questions for discussion
- Does balanced attention improve measured trajectory adherence when rollout quality is held comparable?
- How much of the reported transfer survives removal of the private driving corpus?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires synchronized front-view video and ego pose, the specified tokenizer datasets and substantial GPU resources; a public-NuPlan-only run would be a reduced-data experiment, not a replication of the reported training distribution. e05e10
Before training, resolve pose quantization/decoding, attention connectivity, pose loss and masking-rate ambiguities. Before evaluation, fix clip length, frame rate and sample selection; the source delegates matched-length details to Vista and leaves the long-rollout rate inconsistent. e05e06e08e09e12e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Cross balanced attention with masking under matched rollout conditions
Reader-proposed experiment, not run: train the four BA/RMS combinations from Table 4 on the same documented data subset, with identical tokenizer, initialization schedule, optimization budget and 15-frame histories. Resolve and record RMS's missing token-selection fraction first. Use paired predefined trajectories and fixed generation settings, measuring both video quality and deviation from the supplied pose path. Declare frame counts and frame rate explicitly instead of guessing the units of Table 4's FVD subscripts. The mechanism claim predicts better pose adherence from BA and less degradation under corrupted/generated histories from RMS. Improvement in FVD without corresponding adherence or drift changes would weaken that attribution. Repeat across seeds; a public-only run must be labeled reduced-data. e03e07e08e10e15e17
Check 2: Check whether the printed pose decoder actually inverts the encoder
Reader-proposed check, not run: after obtaining the missing bin counts and coordinate ranges, enumerate valid orientation and planar location bins. Encode their lower-edge representatives with Equation (1), decode them literally with Equation (7), and compare the resulting bin identities. Include zero initial motion, adjacent y bins and boundary cases; assess upper-bound handling separately because it is unspecified. A systematic failure to recover the y bin would falsify the printed inverse claim. Test any proposed mathematical repair as a separately labeled candidate, not as the authors' verified implementation. Only after this check should decoded trajectories be used to interpret planning errors or control-conditioned generation. e05e09e14
8.3 Reading coverage
Visual audit: All 15 pages of the supplied PDF were rendered and visually inspected, including the title/authors/version, Equations (1)–(7), training/data/hardware descriptions, Figures 1–9, Tables 1–4 and the bibliography. All four supplied text chunks were read individually. Six final crops were viewed; Table 1 retains its training/zero-shot legend and Table 3 retains its front-camera footnote. Figure 5 was checked against Equations (2)–(6), the pose decoder against Equation (1), and Figure 7 against Section 4.3; unresolved source discrepancies are disclosed. Table 2, Figure 8 and Figure 9 were inspected but not cropped. No appendix is present, and separate supplements were not supplied or verified. Code and dynamic video files were not inspected; still-image inspection is not a playback evaluation.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1 Introduction (pp. 1–2)
- 2 Related Work, including 2.1–2.3 (pp. 3–4)
- 3 Method, including 3.1 Tokenizer, 3.2 Prelimilaries of Next-Token Prediction, 3.3 World Model, 3.4 Decoder (pp. 4–9)
- 4 Experiments, including 4.1–4.6 (pp. 9–13)
- 5 Conclusion and Future Work (p. 13)
- References (pp. 13–15)
Outside the original text pass
- Verified source: arXiv:2412.19505v3 [cs.CV], 16 August 2026. The title and all six authors match the catalog. The catalog records an ICPR 2026 Springer chapter; that separate publication artifact and earlier arXiv revisions were not supplied, so their textual equivalence or revision differences cannot be established.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks and all 15 PDF pages were read. The extraction's missing figure images were addressed by visual inspection of the retained PDF and six final original crops. No appendix is present in this PDF.
- Code, project links and external cited works were not inspected. No training, inference or reproduction experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, affiliations and arXiv margin
Exact title and six authors match the catalog; margin identifies arXiv:2412.19505v3 [cs.CV], 16 Aug 2026. Affiliations are HKUST, Horizon Robotics and Nanjing University.
Go to primary source ↓e02PDF p. 2, Introduction, Control Signal Degradation and Computational Inefficiency
Motivates sparse-pose attention dilution, weak temporal connections and the cost of flattened autoregressive attention.
Go to primary source ↓e03PDF p. 3, Figure 1 and caption
One conditioning video and two predefined trajectories illustrate different generated driving futures.
Go to primary source ↓e04PDF pp. 4–5, Figure 2/caption and Section 3 opening
Historical orientations, locations and front-view images are tokenized; next pose is predicted before next image, with decoders returning pose/image outputs.
Go to primary source ↓e05PDF pp. 5–6, Section 3.1 and Equation (1)
Adds causal temporal attention before and after image quantization. Ego-centered relative changes are discretized using orientation alpha and x/y beta/gamma bins; first changes are zero. Numerical bin counts and ranges are absent.
Go to primary source ↓e06PDF pp. 6–8, Figures 4–5, Section 3.3 and Equations (2)–(4)
Defines fixed-position temporal attention, within-timestep multimodal fusion and shifted-prefix internal autoregression. Figure 5a's upper fan appears to connect different token positions to the rightmost box in the middle feature row, despite Equation (2)'s fixed-position history; Figure 5b depicts losses on pose and image outputs.
Go to primary source ↓e07PDF p. 8, Internal-Frame AR Module and Equation (5)
Uses 512 image tokens and two pose tokens. Balanced attention adds 1/(m n_j) to the score for a token in modality j before softmax; internal-frame prediction depends on preceding tokens.
Go to primary source ↓e08PDF pp. 8–9, Random Masking Strategy and Equation (6)
Sequence masking is selected with epsilon; a masked token is randomly replaced with probability eta. Token-selection fraction is not specified. Displayed cross-entropy sums image-token probabilities conditioned on historical/current pose tokens; explicit pose-loss terms are absent.
Go to primary source ↓e09PDF p. 9, Section 3.4 and Equation (7), compared with p. 6 Equation (1)
Image decoding uses codebook lookup, causal attention and convolutions; pose decoding claims an inverse quantizer. The printed y expression contains (v_t/gamma) v_t minus floor(v_t/gamma), inconsistent with the stated encoding inverse.
Go to primary source ↓e10PDF pp. 9–10, Section 4.1, Tokenizer and Decoder / World Model
Reports 70M tokenizer parameters, codebook 16,384, 1,000K steps, batch 128 and 32 NVIDIA 4090 GPUs; tokenizer datasets/losses. World model: 1B parameters, 15-frame conditioning, epsilon 0.5, eta 0.3, over 3,456 hours including 120 public and 3,336 private, 450K iterations, batch 64, 64 A100 GPUs, 12 days and one-day NuScenes fine-tuning.
Go to primary source ↓e11PDF p. 8, Table 1, color legend and FID/FVD/maximum-duration/maximum-frames rows
NuScenes-seen DrivingWorld 6.5/86.0 versus Vista 6.9/89.4; zero-shot DrivingWorld 7.4/90.9 versus GenAD (OpenDV) 15.4/184.0 and DriveGAN 73.4/502.3. Maximum rows list DrivingWorld 40 s/400 frames and Vista 15 s/150 frames.
Go to primary source ↓e12PDF pp. 10–11, Section 4.1 Evaluation Dataset and Metrics; Section 4.2
Evaluation uses 200 NuPlan test clips and 150 NuScenes test clips. Generated videos for FID/FVD have equal frame length following Vista, without a numerical length. Describes both NuScenes-trained and untrained variants.
Go to primary source ↓e13PDF p. 11, Table 2, Temporal-aware and Llama-Gen Finetuned rows; Section 4.4, continuing p. 12
NuPlan encoding–decoding metrics FVD_12/FID/PSNR/LPIPS are 14.66/4.29/23.82/0.1828 for temporal-aware and 20.33/5.19/22.71/0.1909 for fine-tuned Llama-Gen; tokenizer comparison split is not separately specified.
Go to primary source ↓e14PDF p. 12, Table 3, input/auxiliary-supervision columns and Ours/UniAD/GenAD rows; Section 4.6, pp. 12–13
Ours uses only the front camera and no auxiliary supervision. L2 at 1/2/3 s is 0.68/1.20/1.90 m, average 1.26; collisions 0.03/0.22/0.90%, average 0.38%. UniAD averages 1.03 m/0.31%; GenAD averages 0.91 m/0.43%, with auxiliary supervision.
Go to primary source ↓e15PDF p. 13, Table 4, all rows/columns; Section 4.5 on p. 12
NuPlan FVD_10/25/40: full 122.37/144.26/159.46; no BA 240.25/319.35/361.94; no RMS 214.69/283.71/324.58; neither 463.21/603.88/665.15. The supplied text does not explicitly define subscript units or report uncertainty.
Go to primary source ↓e16PDF p. 10, Figure 8 and caption; p. 11, Section 4.2
DrivingWorld preference: visual quality 87.3% NuPlan and 71.7% NuScenes; controllability 93.2% and 73.1%, with complementary Vista shares. No participant/trial counts are given.
Go to primary source ↓e17PDF p. 10, Figure 7 and caption; p. 11, Section 4.3
Figure 7 shows selected frames through 640; its caption states 5 Hz/128 seconds while Section 4.3 states 15 conditioning frames and 640 generated frames at 10 Hz/64 seconds. The source does not reconcile these timings.
Go to primary source ↓e18PDF p. 12, Table 3 UniAD [25] and Section 4.6; p. 14, Reference 25
The planning comparison cites UniAD [25], but Reference 25 is 'Uniad: A unified ad hoc data processing system' in a database journal. The supplied bibliography therefore does not resolve the intended driving baseline.
Go to primary source ↓e19PDF p. 13, Section 5 Conclusion and Future Work
Proposes future multimodal and multi-view integration; no physical closed-loop driving experiment is reported in the experiments.
Go to primary source ↓8.5 Primary sources
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT ↗
PDF · 5,489 extracted words
Source fingerprint
db17a90b4c3b3ca3f63c31410c9cbdae661fed66900fc7b9c046c07a9cfce82e