OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
1. Paper overview
In one sentence: OccWorld jointly predicts semantic occupancy and ego motion through shared world tokens, but its strongest results depend on oracle occupancy and deteriorate with forecast horizon. e01e03e04e06e10e11e12e14e15
| At a glance | What to know |
|---|---|
| Research problem | Source description Object-box prediction describes traffic participants but omits fine scene geometry and static structure. OccWorld asks whether a dense semantic occupancy world model can forecast the surrounding scene and ego trajectory together, reducing reliance on separate box/map supervision. Its planning formulation assumes a satisfactory downstream controller; predicting waypoints does not establish executed driving. e02e03e07 |
| Core mechanism | |
| A key reported result | Oracle 4D occupancy forecasting: Reported averages: 17.14 mIoU and 26.63 IoU. mIoU at 1/2/3 seconds: 25.78/15.14/10.51. mIoU (%) and IoU (%), higher is better. Occ3D; OccWorld-O receives ground-truth occupancy, with 2-second history and 3-second prediction. The supplied body does not identify the evaluation split; Avg. covers 1, 2, and 3 seconds. Copy&Paste: 11.33 average mIoU and 20.52 IoU. Exceeds static persistence with the same occupancy-input setting, but semantic agreement declines sharply with horizon. The 0-second columns measure reconstruction, not forecasting. e07e09e10 |
| Reading caution |
Core contributions
Figure 2. One predictive pathway advances the scene and the ego trajectory together. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each time column from bottom to top. The occupancy encoder supplies scene tokens beside a distinct ego token. Peach spatial aggregation mixes their information before the yellow temporal block; blue aggregation then produces the tokens decoded into occupancy and the displacement pair (Δx, Δy). The red dashed arrows lead from one prediction toward the next input token set. The caption resolves the apparent coexistence of observations and feedback: training uses tokens from ground-truth scenes, whereas inference uses predicted tokens. Section 3.4 confirms token feedback, so the drawing should not be read as requiring a decode-and-reencode loop at every forecast step. e03e05e06
What it supports. Scene evolution and ego motion emerge from interacting world tokens in the same generative framework. That architectural coupling supports the catalog's joint-prediction classification. The separate output decoders make the two products inspectable, but their existence alone would not establish coupling without the shared spatial-temporal pathway shown here.
Where the evidence stops. The temporal mask is named rather than drawn as a matrix; its direction is specified by Equation (6). The displacement branch produces a planned trajectory. Section 3.1 assumes a downstream controller and provides no executed-control evaluation.
2. Motivation
2.1 The problem and the proposed response
Object-box prediction describes traffic participants but omits fine scene geometry and static structure. OccWorld asks whether a dense semantic occupancy world model can forecast the surrounding scene and ego trajectory together, reducing reliance on separate box/map supervision. Its planning formulation assumes a satisfactory downstream controller; predicting waypoints does not establish executed driving. e02e03e07
2.2 What this reading follows
OccWorld asks a driving model to predict both what the surrounding scene will become and where the ego vehicle will move. It represents the scene as semantic 3D occupancy, learns a discrete vocabulary through reconstruction, and forecasts scene and ego tokens together. This reading follows the supplied November 2023 arXiv v1 through its architecture, two-stage training, and occupancy/planning experiments. The central evidence is a gain over occupancy persistence and ablations supporting scene–ego coupling. The boundaries are equally useful: camera perception weakens the results, longer-horizon planning deteriorates, and the two reported planning metric conventions must remain separate. e01e03e04e06e10e11e12e14e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The shared world-token transformer jointly evolves occupancy and ego tokens, supporting One Model × Joint prediction at the predictive-core level. Separate perception, tokenizer, and decoding modules remain. The modeled visual state is semantic 3D occupancy rather than RGB video; ego outputs are trajectory waypoints rather than low-level executed actions. e03e04e05e06e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a vocabulary before learning how the scene changes
Start with a semantic occupancy grid: each voxel records free space or an occupied category. OccWorld embeds these categories and folds height into channels, allowing a 2D convolutional encoder to produce a smaller BEV feature grid. Quantization then assigns features to a learned vocabulary, and a decoder reconstructs the original occupancy. This first stage trains with softmax and Lovász-softmax reconstruction losses. It does not yet predict motion. The second stage can supervise future scene tokens as code classifications rather than predict the full occupancy grid directly. The decoder remains available to turn those predictions into a spatially interpretable output. Table 3 supplies a useful empirical warning: increasing latent resolution improves reconstruction but worsens downstream forecasting, so the best vocabulary for reconstruction need not be the best one for a world model. e04e06e13
Figure 3. Reconstruction learns the discrete vocabulary used for future-scene prediction. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper arrows from the input occupancy to the encoder and the pink quantization block, then move downward to the token grid and left through the decoder. The codebook enters quantization from above; it is learned together with reconstruction according to the caption. Section 3.2 supplies the hidden implementation steps: category embeddings are stacked along height, a 2D CNN compresses the resulting BEV features, and deconvolution restores voxel predictions. This diagram represents reconstruction of the same scene during tokenizer learning. Temporal forecasting begins only when the subsequent transformer predicts these tokens, which the learned scene decoder can convert back into occupancy. e04e06e13
What it supports. The tokenizer converts dense voxel labels into a compact prediction target while retaining an occupancy decoder. Its purpose is not merely compression: the paper expects the representation to expose scene structure that is easier to forecast. Table 3 tests the practical tradeoff between reconstruction fidelity and future prediction.
Where the evidence stops. Equation (4) prints a minimum distance although the prose specifies nearest-code selection; adjacent latent and codebook shapes are also inconsistent. The arrows agree with reconstruction, but they do not resolve these mathematical or codebook-update ambiguities.
5.2 Separate spatial context, temporal prediction, and rollout
A token at one spatial position cannot describe every interaction that matters to the next scene. OccWorld first aggregates spatial information, including an ego token, and merges neighboring scene tokens to form multiple scales. Temporal attention then predicts each position's next token from its history; U-net aggregation combines the scales. The causal mask blocks future-to-past information during training, when tokenizer-produced tokens supply the inputs. At inference, those future inputs are unavailable, so the model feeds back its own predicted tokens. Scene decoding yields occupancy, while ego decoding yields displacement relative to the current frame. These steps explain why joint prediction is an architectural property here. They also locate the experiment's boundary: generating a sequence of planned displacements is different from evaluating those displacements through a vehicle controller. e03e05e06e07
5.3 Read the experiments as three different tests of the proposal
The oracle-versus-persistence comparison asks whether the model learns useful temporal evolution when occupancy input is supplied. The camera rows ask a harder system-level question that also includes perception error and changes in supervision. The ablation tables ask which design choices help the reported implementation. Keeping these questions separate prevents an oracle forecasting improvement from becoming an unsupported claim about end-to-end driving reliability. Planning adds another distinction: the non-dagger and dagger blocks use different metric computations. My reading is that OccWorld establishes a promising shared prediction architecture and measurable gains over the listed persistence and occupancy-planning baselines, while leaving deployment quality open. Its strongest mechanistic evidence is the deterioration after spatial, temporal, or ego components are removed, although those replacements do not preserve all capacity and history controls. e07e09e10e11e12e14e15
5.4 Training and inference
During training
Stage one trains occupancy reconstruction using softmax and Lovász-softmax losses. Stage two uses tokenizer-produced scene tokens as targets, supervises next-code classification, and adds an L2 ego-displacement objective weighted by λ2. The latter still requires ego-trajectory supervision; “None” in auxiliary-supervision columns excludes that supervision. e06e10e11
Temporal masks prevent future information from entering earlier predictions during teacher-forced training. AdamW uses initial learning rate 10⁻³, weight decay 0.01, cosine annealing, and one sample per GPU on eight NVIDIA GeForce RTX 4090 GPUs. The paper reuses the learned tokenizer but does not explicitly enumerate stage-two freezing flags. e06e08
During inference
Start from observed history, then autoregressively feed predicted tokens into later steps. This is joint scene/ego prediction, without a specified search over candidate controls or an inverse-dynamics action extractor. The evaluated outputs are occupancy and trajectory estimates, not throttle, steering, or braking executions. e03e06e07
5.5 Implementation flow
- Encode and quantize occupancy
Embed voxel categories, concatenate the height dimension into BEV channels, and downsample with 2D convolutions. Assign spatial features to nearest codebook entries; a deconvolution decoder reconstructs voxel semantics. The default tokenizer downsamples by four and uses 512 codes with 128-dimensional features. e04e08
- Couple scene and ego tokens
Add an ego-position token to the scene representation. Spatial aggregation exchanges information; merging 2×2 scene-token windows builds coarser scales. Position-wise causal temporal attention predicts the next token set, and U-net aggregation combines scales. The implementation uses three scales, six scene temporal-attention layers per scale, and two layers of spatial and temporal cross-attention for ego planning tokens. e05e08
- Decode and extend the future
Reuse the scene decoder for occupancy and learn an ego decoder for displacement relative to the current frame. Append predicted world tokens to the history for later predictions. The shared prediction pathway provides architectural evidence for coupling, beyond simply training two independent heads on related data. e03e06
6. Experiments & results
OccWorld compresses semantic 3D occupancy into discrete scene tokens and jointly forecasts those tokens and an ego-motion token. Its shared spatial-temporal transformer supports both scene forecasting and trajectory planning. Oracle occupancy gives substantially stronger results than camera-derived occupancy, while long-horizon error and collision rates limit the driving conclusions.
6.1 Read the original evidence
Table 1. Learned dynamics beat persistence with oracle occupancy; camera perception leaves a large gap. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the Input and Aux. Sup. columns. O uses ground-truth occupancy; D, T, and S use cameras with different perception-training supervision. “None” excludes the ego-trajectory supervision used by the model. Read the 0-second column as reconstruction, then compare the 1-, 2-, and 3-second future columns. The shaded averages exclude 0 seconds. Copy&Paste is the relevant persistence baseline for O because it copies current ground-truth occupancy forward. D, T, and S change the perception setting, so their differences mix occupancy-input quality with the downstream forecasting problem. Section 4.1 identifies Occ3D and the two-second input history. e02e07e09e10
What it supports. OccWorld-O improves average mIoU from Copy&Paste's 11.33% to 17.14%, and IoU from 20.52% to 26.63%. The D/T/S average mIoUs are 8.62%, 3.56%, and 0.26%, respectively. The strong oracle result therefore does not transfer unchanged to camera-derived occupancy.
Where the evidence stops. The introduction says 17.13 mIoU; this table says 17.14. The edition retains the table value. Exact split and metric details are deferred to an unsupplied supplement, and no uncertainty estimates accompany these rows.
Table 2. Planning comparisons depend on both the supervision setting and the metric convention. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the ordinary rows from the gray dagger-marked rows at the bottom. The retained note defines dagger as VAD's metric computation; it does not mark a newer model or a confidence interval. Within either block, read Input and Aux. Sup. before comparing the shaded averages. OccWorld-O receives occupancy, whereas UniAD and VAD receive cameras with additional annotation types. Within the ordinary block, compare O with the occupancy-input OccNet row, and then examine how O changes from the 1-second to the 3-second L2 column. Finally compare dagger rows only with other dagger rows. Lower is better for both error and collision. e03e07e11e12
What it supports. In the ordinary block, OccWorld-O averages 1.17 m L2 and 0.60% collision, versus 2.25 m and 0.69% for occupancy-input OccNet; UniAD reaches 1.03 m and 0.31%. In the dagger block, O reaches 0.64 m and 0.24%, while VAD-Base reaches 0.72 m and 0.22%.
Where the evidence stops. These rows do not hold input modality or auxiliary supervision constant across all methods. The two metric conventions cannot be pooled. The source reports predicted-trajectory performance, not a closed-loop road test or evidence that the predicted motion was executed.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Oracle 4D occupancy forecasting Occ3D; OccWorld-O receives ground-truth occupancy, with 2-second history and 3-second prediction. The supplied body does not identify the evaluation split; Avg. covers 1, 2, and 3 seconds. | Reported averages: 17.14 mIoU and 26.63 IoU. mIoU at 1/2/3 seconds: 25.78/15.14/10.51. mIoU (%) and IoU (%), higher is better | Copy&Paste: 11.33 average mIoU and 20.52 IoU. Exceeds static persistence with the same occupancy-input setting, but semantic agreement declines sharply with horizon. The 0-second columns measure reconstruction, not forecasting. e07e09e10 |
| Camera-based 4D occupancy forecasting Same reported forecasting task; dense-supervised (D), sparse-semantic-LiDAR-supervised (T), and self-supervised (S) camera perception. Evaluation split unspecified in the supplied body. | D: 8.62/16.53; T: 3.56/8.34; S: 0.26/5.00. Average mIoU (%) / IoU (%) at 1, 2, and 3 seconds | Oracle O: 17.14/26.63. Perception and supervision substantially affect the system. These rows change the inputs and their training, so they do not isolate the generative transformer's causal contribution. e07e09e10 |
| Motion planning under the non-dagger metric convention nuScenes; 2-second context, 1/2/3-second trajectory evaluation; non-dagger Table 2 rows. Split and detailed metric computation are deferred to the unavailable supplement. | OccWorld-O: average 1.17 m / 0.60%; 1-second L2 0.43 m and 3-second L2 1.99 m. L2 error (m) and collision rate (%), lower is better | UniAD: 1.03 m / 0.31% averages, 1.65 m at 3 seconds. Occupancy-input OccNet: 2.25 m / 0.69%. OccWorld improves on the listed occupancy-input OccNet configuration but does not beat UniAD overall. Input modalities and auxiliary annotations differ. These are trajectory metrics, without closed-loop deployment evidence. e03e07e11e15 |
| Motion planning under the VAD metric convention nuScenes; dagger-marked rows in Table 2 use VAD metric computation, separately from the non-dagger rows. Exact computation and split remain unspecified here. | OccWorld-O†: 0.64/0.24; OccWorld-D†: 0.77/0.32. Average L2 (m) / collision rate (%) | VAD-Base†: 0.72/0.22. Oracle OccWorld has lower L2 but slightly higher collision rate than VAD-Base under this convention. The 0.64 m value is not interchangeable with the non-dagger 1.17 m result. e07e12 |
| Spatial-temporal and ego-token ablations OccWorld-O ablations; averages over 1, 2, and 3 seconds. | Full model: 17.14/1.17; no spatial attention: 10.07/1.42; no temporal attention: 8.98/2.06. Removing ego prediction yields 15.13 mIoU; replacing ego temporal attention yields 12.07 mIoU and 5.89 m. Forecast mIoU (%) and planning L2 (m) | Temporal-attention removal substitutes a convolution using the current tokens; ego temporal removal substitutes an MLP. Results support spatial context, history, and ego/scene coupling in this implementation. Replacement modules also change capacity and accessible history, limiting mechanistic isolation. e14 |
| Tokenizer reconstruction versus forecastability Table 3 changes latent resolution while keeping 128 channels and 512 codes. | 50² tokens: 66.38, 17.14, 1.17, 18.0. 100² tokens: 78.12, 12.38, 1.36, 6.7. Reconstruction mIoU (%), average forecast mIoU (%), planning L2 (m), FPS | The finer representation improves reconstruction but worsens forecasting, planning, and reported speed. Reconstruction alone is an inadequate tokenizer-selection criterion. The authors' high-level-concept explanation is plausible but not directly measured by this table. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. A tokenizer that reconstructs better can make the world model harder to predict. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Decode the Setting tuple using the accompanying analysis: latent spatial resolution, channel dimension, then codebook size. The first row is the default. Compare it with the 100² row to change resolution while retaining 128 channels and 512 codes. Read across reconstruction, forecasting, planning, and speed together; the bold reconstruction maximum is not the best forecast. The 256- and 1024-code rows instead alter vocabulary size at the default spatial resolution. Treat the 25² row cautiously because it also raises the channel count to 256. This table therefore offers several useful comparisons, but its rows do not all isolate one factor. e13
What it supports. Increasing resolution from 50² to 100² raises reconstruction mIoU from 66.38% to 78.12%, yet average forecast mIoU falls from 17.14% to 12.38%. Planning L2 worsens from 1.17 m to 1.36 m and reported FPS drops from 18.0 to 6.7. Reconstruction quality alone would select the wrong setting for these downstream results.
Where the evidence stops. The authors explain the tradeoff through high-level concepts and forecast difficulty. This table does not directly measure concept abstraction, match computation across resolutions, or report seed variation. It supports the observed tradeoff more strongly than that causal explanation.
Table 4. Both scene context and ego history matter to the shared forecast. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row as the reference and read the four replacements using Section 4.3, not their shortened names alone. “w/o spatial attn” removes spatial aggregation. “w/o temporal attn” substitutes a convolution that uses the current world tokens, changing the access to history as well as the operator. “w/o ego” removes ego prediction, so planning entries are intentionally absent. “w/o ego temporal” replaces the ego temporal module with an MLP. Forecast scores increase upward; planning L2 and collision improve downward. All scores average the 1-, 2-, and 3-second horizons, so this crop does not reveal which horizon causes each deterioration. e14
What it supports. Removing spatial attention lowers mIoU from 17.14% to 10.07%; removing temporal attention lowers it to 8.98%. Removing ego prediction also reduces mIoU, to 15.13%. Replacing ego temporal attention yields 5.89 m planning error and 12.07% mIoU, consistent with the two prediction branches influencing one another.
Where the evidence stops. These ablations change modules, capacity, or accessible history together. They support the chosen implementation but do not isolate a unique causal mechanism. The claim that erroneous ego predictions mislead scene forecasting is the authors' interpretation.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge failures to predict newly entering vehicles, weaker collision performance, and rapid long-horizon deterioration. Their attribution to missing safety supervision or diverse futures is explanatory conjecture, not a controlled diagnosis. e16e15
The introduction reports 17.13 average mIoU and 1.16 m L2; Tables 1–3 instead report 17.14 and 1.17. This report uses the table values and preserves the discrepancy. The tables provide no error bars or repeated-seed uncertainty. e02e10e11e13
7.2 Questions for discussion
- Does ego/scene coupling still improve forecasting when model capacity and available history are matched?
- Would choosing the tokenizer by future prediction quality remain preferable after equalizing optimization budget across latent resolutions?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs Occ3D occupancy, nuScenes trajectories, the appropriate perception variant, and the stated tokenizer/transformer configuration. Recover the omitted dataset split, metric definitions, voxel preprocessing, training duration, loss weights, and software versions before claiming an exact replication. e04e06e07e08e09
The tokenizer description leaves implementation ambiguities: Equation (4) prints a minimum distance where its prose describes selecting the nearest code; its codebook and latent-shape dimensions are inconsistent. The supplied objectives do not specify commitment/codebook-update details or exact freezing behavior. These require clarification, not silent correction. e04e06
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test ego–scene coupling with a matched history control
Reader-proposed, not performed: after recovering the missing data split and preprocessing, compare the full oracle model with an otherwise matched model whose scene tokens cannot receive information from ego tokens. Retain both prediction heads, their losses, historical window, widths, layers, tokenizer, optimizer, and training budget. Use several seeds and report horizon-specific occupancy mIoU plus both planning metrics under one declared convention. If disconnecting ego-to-scene information consistently lowers forecasting quality while the architecture and history remain matched, that strengthens the coupling interpretation of Table 4. If forecasting is unchanged, the original no-ego result may reflect other consequences of removing the branch. e05e06e07e08e14
Check 2: Test whether finer tokens hurt rollout or merely need more optimization
Reader-proposed, not performed: reproduce Table 3's 50² and 100² settings with 128 channels and 512 codes on the same recovered split. First hold training steps fixed; then repeat with a declared matched-compute budget. For each setting, measure reconstruction, one-step prediction with observed history, and autoregressive 1/2/3-second forecasting; also record planning L2 and inference speed under the same measurement conditions. If the finer grid retains superior reconstruction yet worse future-token prediction across budgets, the forecastability tradeoff is more convincing. If the deficit appears only in rollout or disappears with optimization, the explanation should shift toward error accumulation or training budget. e04e06e07e08e13
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page; Figures 1–5; Equations (1)–(8) in their page layouts; Tables 1–4; and the method, training, hardware, evaluation, and analysis text on pages 1–8. Cross-checked Figure 2 feedback arrows against its caption and Sections 3.1/3.4, Figure 3 against Section 3.2/Equation (4), and Figure 4 against the temporal-mask definition in Equation (6). The tokenizer's printed nearest-code/dimension ambiguities are disclosed. Inspected every final original crop after boundary corrections; Table 2 retains its metric-convention and auxiliary-supervision note. References on pages 9–11 were read in the supplied text; they support no retained method or numerical claim. No separate supplement or code was supplied for this visual pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Proposed Approach
- 3.1. World Model for Autonomous Driving
- 3.2. 3D Occupancy Scene Tokenizer
- 3.3. Spatial-Temporal Generative Transformer
- 3.4. OccWorld: a 3D Occupancy World Model
- 4. Experiments
- 4.1. Task Descriptions
- 4.2. Implementation Details
- 4.3. Results and Analysis
- 5. Conclusion
- References (PDF pp. 9–11)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: the observed title and all six authors match the catalog. The inspected artifact is arXiv:2311.16038v1 [cs.CV], dated 27 November 2023 (p. 1). The catalog identifies ECCV 2024; its proceedings edition was not supplied or compared, and equivalence between editions is not established.
- All seven supplied text chunks were read individually, including all eleven PDF pages. Original PDF pages 1–8 and Figures 1–5/Tables 1–4 were visually inspected; reference pages 9–11 were read as text.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed for the supplied PDF through visual inspection.
- Separate supplemental material availability has not been fully verified. No supplement was supplied; Section 4.1 delegates dataset and evaluation-metric details to it. No appendix appears in this eleven-page artifact.
- Code and linked resources were not inspected. Experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, equal-contribution footnote, and arXiv margin
Title and six authors match the catalog. The PDF identifies arXiv:2311.16038v1, 27 November 2023, and the Automation and Electronic Engineering departments at Tsinghua University, China.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1, final introductory result paragraph
Motivates semantic occupancy and joint scene/ego forecasting. The introduction prints 17.13 average mIoU, 26.63 IoU, and 1.16 planning L2.
Go to primary source ↓e03PDF p. 3, Figure 2/caption and Section 3.1, Equations (1)–(3)
Shared scene/ego prediction is autoregressive. The caption distinguishes ground-truth tokens during training from predicted tokens at inference. A downstream controller is assumed; the formal outputs are future positions.
Go to primary source ↓e04PDF p. 4, Figure 3/caption and Section 3.2, Equation (4)
Class embeddings and height concatenation create BEV features; 2D convolution, nearest-code quantization, and deconvolution reconstruct occupancy. Equation (4) prints min distance despite nearest-code prose, and adjacent codebook/latent dimensions are inconsistent.
Go to primary source ↓e05PDF pp. 4–5, Section 3.3, Equations (5)–(6), Figure 4/caption
An ego token joins scene tokens. Spatial aggregation and 2×2 merging create multiple scales; position-wise masked temporal attention predicts future tokens, followed by U-net aggregation.
Go to primary source ↓e06PDF p. 5, Section 3.4, Equations (7)–(8) and training/inference paragraphs
Stage one uses softmax plus Lovász-softmax reconstruction. Stage two supervises scene-code classification and ego displacement with L2 loss. Learned tokenizer/decoder are reused; predicted past tokens drive inference. Numerical loss weights, commitment/update rules, and freezing flags are not specified.
Go to primary source ↓e07PDF p. 6, Sections 4.1 and 4.2 opening
Occupancy forecasting uses Occ3D and mIoU/IoU; planning uses nuScenes, 2D BEV waypoints, L2 and collision rate. Two seconds of history predict three seconds ahead. Dataset and metric details are deferred to supplementary material.
Go to primary source ↓e08PDF pp. 6–7, Section 4.2
Tokenizer: downsampling four, 512 codes, 128 features. Transformer: three scales, six scene temporal-attention layers per scale, two spatial/temporal cross-attention layers for ego tokens. AdamW, cosine annealing, learning rate 10⁻³, decay 0.01, batch one per GPU, eight RTX 4090 GPUs.
Go to primary source ↓e09PDF p. 7, Section 4.3, '4D occupancy forecasting' and footnotes
O uses ground-truth occupancy. D/T use TPVFormer trained with dense occupancy/sparse semantic LiDAR; S uses self-supervised perception associated with SelfOcc. Copy&Paste persists current occupancy, and 0-second metrics are reconstruction.
Go to primary source ↓e10PDF p. 6, Table 1, all rows, horizon and Avg. columns
Oracle averages are 17.14 mIoU and 26.63 IoU versus 11.33/20.52 for Copy&Paste. D/T/S average pairs are 8.62/16.53, 3.56/8.34, and 0.26/5.00. Avg. covers 1/2/3 seconds; auxiliary supervision excludes ego trajectories.
Go to primary source ↓e11PDF p. 7, Table 2, non-dagger OccWorld, UniAD, and occupancy-input OccNet rows
O averages 1.17 m and 0.60% collision; UniAD 1.03 m and 0.31%; occupancy-input OccNet 2.25 m and 0.69%. O L2 is 0.43/1.08/1.99 m at 1/2/3 seconds. Inputs and auxiliary supervision vary.
Go to primary source ↓e12PDF p. 7, Table 2, dagger definition and dagger-marked rows
Dagger marks VAD metric computation. Average L2/collision: O† 0.64 m/0.24%, D† 0.77 m/0.32%, VAD-Base† 0.72 m/0.22%; these belong to a distinct metric convention.
Go to primary source ↓e13PDF p. 8, Table 3 and 'Analysis of the scene tokenizer'
Settings encode latent resolution, channels, and codebook size. Changing 50² to 100² at 128 channels/512 codes changes reconstruction mIoU 66.38→78.12, average forecast mIoU 17.14→12.38, planning L2 1.17→1.36, FPS 18.0→6.7. The 25² row also changes channels to 256.
Go to primary source ↓e14PDF p. 8, Table 4 and 'Analysis of the spatial-temporal generative transformer'
Full/no-spatial/no-temporal mIoU and L2 pairs are 17.14/1.17, 10.07/1.42, 8.98/2.06. No ego gives 15.13 mIoU. Ego temporal attention replaced by an MLP gives 12.07 mIoU, 5.89 m L2, 6.23% collision. No temporal attention uses a current-token convolution.
Go to primary source ↓e15PDF pp. 7–8, Section 4.3 'Motion planning' and continuation before tokenizer analysis
Authors acknowledge UniAD's overall advantage, collision shortcomings, and declining longer-horizon planning. They suggest supervision and future diversity as possible explanations.
Go to primary source ↓e16PDF p. 1, Figure 1/caption; PDF pp. 6–7, Figure 5/caption and 'Visualizations'
Examples show predicted scene and ego evolution. Figure 1 acknowledges failures for vehicles newly entering view; its claim of more reasonable drivable areas is qualitative. Figure 5 compares O/D/T outputs.
Go to primary source ↓8.5 Primary sources
OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving ↗
PDF · 8,168 extracted words
Source fingerprint
d6201710786323f341479decceddff379df6aebbfd0c9b76c16be468ef51b61c