OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving
1. Paper overview
In one sentence: A shared occupancy-language-action generator improves several driving benchmarks, but its spatial attention helps scene forecasting more consistently than trajectory planning. ev-motivationev-architectureev-generationev-forecastev-planningev-components
| At a glance | What to know |
|---|---|
| Research problem | Source description The authors seek one model that describes driving scenes, forecasts their evolution, and produces trajectories. Semantic occupancy supplies geometry and class labels, but sparse voxel grids require efficient tokenization and spatial structure does not fit ordinary token-by-token generation. ev-motivationev-tokenizerev-generation |
| Core mechanism | Source description The architecture combines a sparse occupancy tokenizer, discrete waypoint vocabulary, and a shared LLaMA generator with scene-specific attention. ev-architectureev-vocabularyev-generation |
| A key reported result | 4D occupancy forecasting with ground-truth occupancy: Reported averages: 19.93 / 29.17; at 3 s: 15.26 / 24.41. Forecast mIoU / IoU (%), higher is better. NuScenes/Occ3D, -O protocol; two seconds of history predict three seconds ahead. OccWorld-O: averages 17.14 / 26.63; at 3 s 10.51 / 20.18. Longer-horizon gains are clearer: at 1 s, Ours-O mIoU 25.05 is below OccWorld-O 25.78. Reconstruction columns are separate from forecast performance. ev-dataev-forecast |
| Reading caution | Reader analysis The authors identify data diversity and inference delay from model size as future work, proposing quantization and distillation. No measured latency accompanies the claimed advantage of one-step scene generation. ev-conclusionev-generation |
Core contributions
- Source description
The architecture combines a sparse occupancy tokenizer, discrete waypoint vocabulary, and a shared LLaMA generator with scene-specific attention. ev-architectureev-vocabularyev-generation
- Reader analysis
The authors position this combination as a possible driving foundation model; the experiments establish benchmark multitask capability, not that broader deployment claim. ev-motivationev-forecastev-planningev-vqa
Figure 2. Three token vocabularies share one generator, with spatial attention reserved for scene positions. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: occupancy is grouped into pillars, encoded, and represented by scene codes. The two reconstruction heads separate occupied geometry from semantic classes; the downward mask arrow goes from Vox Head to Cls Head. Follow the middle branches from text, quantized scene features, and binned trajectories into the unified vocabulary. At the bottom left, blue cells allow scene-to-scene attention beyond the gray causal pattern. The grouped query arrows indicate one scene prediction, while single arrows indicate next-token prediction. The schematic sends a latent-grid arrow directly into the decoder; Section 3.2 instead describes reconstruction after quantization, so that text supplies the intended decoding order. ev-architectureev-tokenizerev-vocabularyev-generationev-motivation
What it supports. The One Model interpretation has architectural support: scene and action tokens pass through the same generative block. Occupancy prediction is part of the generation sequence rather than merely a separate training loss. A specialized tokenizer and external occupancy perception still remain distinct modules around that shared generator.
Where the evidence stops. The upper action sequence repeats <act>, while Section 3.3 defines </act> as the closing boundary. Algorithm 1 also declares scene length S but slices with undefined K. These source ambiguities are preserved; the diagram is not a complete implementation specification.
2. Motivation
2.1 The problem and the proposed response
The authors seek one model that describes driving scenes, forecasts their evolution, and produces trajectories. Semantic occupancy supplies geometry and class labels, but sparse voxel grids require efficient tokenization and spatial structure does not fit ordinary token-by-token generation. ev-motivationev-tokenizerev-generation
2.2 What this reading follows
OccLLaMA asks what a driving language model should see and predict. Its answer is semantic occupancy: a three-dimensional grid that records both occupied space and object classes. The model compresses this grid into scene tokens, combines them with language and discretized ego waypoints, and learns to generate across all three modalities. The key modification is that an entire scene is predicted together, while words and actions remain sequential. Read the figures as a progression from representation to generation to evidence: longer-horizon occupancy gains are substantial, planning gains are smaller, and the ablations show that better scene prediction does not automatically improve every control metric. ev-motivationev-architectureev-generationev-forecastev-planningev-components
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The shared generator predicts scene and action tokens within one multimodal sequence, supporting One Model × Joint prediction. Separate tokenizer/front-end modules do not make this an inverse-dynamics pipeline. World prediction is used during inference, not only as an auxiliary loss. The catalog's video-action label should be read broadly: the generated visual modality is semantic 3D occupancy, not RGB video. ev-architectureev-vocabularyev-generationev-planning
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make a sparse semantic scene into reusable tokens
The tokenizer solves two distinct representation problems. Air occupies much of the grid, so the encoder first discards air voxels and collects the remaining height/class pairs into BEV pillars. Pillar embeddings and a Swin block compress those observations, and nearest-codebook quantization provides discrete identities that a language-style model can consume. Reconstruction then separates occupied geometry from semantic classification, with the occupied mask restricting semantic supervision. Equation (2) reflects that separation through geometry and semantic losses plus a codebook term. Once trained, this tokenizer is frozen. Table 4 shows why its capacity matters: the reported resolution-50, dimension-256, size-2048 setting reconstructs better than the tested alternatives, including a larger codebook. The authors' overfitting explanation is plausible but is not independently isolated by reconstruction scores. ev-tokenizerev-stagesev-tokenizer-ablation
5.2 Alternate between individual tokens and whole scenes
A scene's flattened tokens are spatial positions, whereas language and waypoint sequences naturally have an ordering. OccLLaMA retains causal next-token prediction for language/actions but changes scene positions to spatial attention. After an occupancy opening token, learned queries generate all scene positions in one forward step; ordinary sequential prediction then resumes. Modality delimiters keep the combined stream interpretable. Pretraining aligns occupancy with actions through a world-model objective and with language through captions, followed by LoRA instruction tuning. This makes scene generation available at inference rather than only during training. As a reader interpretation, that is the essential basis for the joint-prediction classification. It does not imply a search over candidate actions or a physical feedback loop, neither of which the supplied method specifies. ev-vocabularyev-generationev-stagesev-planning
5.3 Test whether forecasting quality transfers to planning
Read the results as separate tests of scene forecasting, trajectory prediction, and QA. Table 1 offers the clearest late-horizon occupancy improvement, but Table 2's trajectory-only variant asks a different question: does including scene prediction help planning? Its average metrics improve with the full model, while individual collision horizons do not all improve. Table 5 sharpens the distinction: spatial attention strongly helps forecast mIoU yet slightly worsens planning, whereas action tokenization helps both. My interpretation is that the shared generator benefits from different structures for different outputs, and a single world-model quality score cannot summarize their interaction. QA pretraining gains in Table 6 add another capability test, but do not determine whether language alignment, dynamics learning, or their combination supplies the benefit. ev-forecastev-planningev-componentsev-pretrainingev-stages
5.4 Training and inference
During training
First train the tokenizer, then freeze it. Full-parameter generative pretraining combines a world-model objective aligning occupancy/actions with a scene-caption objective aligning occupancy/language. Finally, use LoRA for instruction tuning on scene understanding and planning. ev-stages
NuScenes provides 700 training and 150 validation videos, each 20 seconds at 2 Hz keyframes; Occ3D supplies semantic occupancy and NuScenes-QA supplies questions. A collected caption dataset pairs occupancy with object positions, classes, states, and future trajectories; its size is not reported. ev-data
Most comparisons use LLaMA-3.1-8B with tokenizer settings Res.=50, Dim.=256, Size=2048. The matched VQA comparison uses LLaMA-2-7B and 25×25 resolution. AdamW training uses tokenizer/pretraining learning rate 10⁻⁴ and batches 4/1; instruction tuning uses 5×10⁻⁵ and batch 4. Tokenizer training lasts 100 epochs on eight RTX 4090s; generative pretraining/tuning last 10/5 epochs on eight V100s. ev-implementationev-tokenizer-ablation
During inference
Language/action positions retain causal next-token prediction. At an occupancy boundary, learned queries and spatial attention predict an entire scene in one forward step; generation resumes sequentially afterward and stops at the end token or length limit. Algorithm 1 declares scene length S but slices with an undefined K, leaving exact implementation ambiguous. ev-generation
Planning can alternate scene and trajectory predictions; Table 2 also evaluates trajectory-only output. The source does not describe a downstream actuator controller, deployed feedback loop, or candidate-trajectory search. Generated waypoints therefore remain predictions rather than demonstrated vehicle execution. ev-generationev-planning
5.5 Implementation flow
- Compress occupied geometry
Discard air voxels and organize non-air voxels into BEV pillars of height/semantic-label pairs. Pillar embeddings and a Swin-transformer block produce a downsampled BEV feature map; nearest-codebook quantization turns its features into discrete scene indices. ev-tokenizer
- Reconstruct geometry and semantics
Convolutions and upsampling restore dense voxel features. Separate voxel and class heads decode occupancy and semantics; the voxel head masks semantic supervision to occupied voxels. Section 3.2 describes decoding after quantization, although Figure 2 does not explicitly connect its quantizer branch back to the decoder. ev-tokenizerev-architecture
- Unify modalities
Scene indices extend the language vocabulary. Waypoint coordinates map to nearest bins chosen from trajectory statistics, giving dedicated action tokens. Functional tokens delimit occupancy/actions and introduce learnable scene queries; task-dependent sequences can mix modalities. ev-vocabulary
6. Experiments & results
OccLLaMA makes semantic 3D occupancy, language, and ego waypoints share a generative vocabulary. A sparse scene tokenizer feeds a modified LLaMA that predicts text/actions sequentially and all tokens within each occupancy scene in parallel. NuScenes-based results support multitask forecasting, planning, and question answering, while the component ablation exposes a forecasting–planning tradeoff.
6.1 Read the original evidence
Table 1. Forecasting gains become clearer at later horizons, with input quality kept explicit. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First pair methods by suffix: -O uses ground-truth semantic occupancy, whereas -F uses FBOCC occupancy predicted from cameras. Then choose a metric block: mIoU on the left and IoU on the right are percentages, with larger values preferred. Within each block, Recon. measures the scene tokenizer's reconstruction and must not be read as a future forecast. The next three columns evaluate one, two, and three seconds into the future; Avg. summarizes those forecast columns. The source supplies two seconds of historical frames. Comparing matched suffixes prevents the advantage of ground-truth perception from being mistaken for an architectural improvement. ev-forecastev-dataev-tokenizer-ablation
What it supports. Ours-O reaches 19.93% average forecast mIoU versus 17.14% for OccWorld-O, and its three-second mIoU is 15.26% versus 10.51%. The one-second values reverse that ordering, 25.05% versus 25.78%. With camera-derived occupancy, average mIoU improves from 6.16% to 8.66%, but remains well below the ground-truth-input setting.
Where the evidence stops. These scores concern occupancy forecasts on NuScenes-derived data, not executed driving. Reconstruction and forecast gains change together, so this comparison alone does not isolate the generator from its improved tokenizer.
Figure 3. Two examples make the late-horizon reconstruction differences visible. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each three-row group separately. Ground truth is the first row, OccWorld-O the second, and Ours-O the third; the dashed horizontal divider separates two different examples. Time advances left to right from 0.5 to 3.0 seconds. Use the bottom legend to distinguish magenta driveable surface, green vegetation, light-green terrain, and the other semantic classes. The red dashed boxes direct attention to three-second regions where spatial structure diverges. Compare a method vertically against ground truth at the same time before judging changes across a row, since the desired scene itself evolves over time. ev-qualitativeev-forecast
What it supports. The displayed OccLLaMA forecasts retain more of the local structure visible in the ground-truth rows at the highlighted three-second regions. This is qualitative support for the authors' long-horizon interpretation and is consistent with Table 1's stronger late-horizon scores. It supplies a visual example of the effect, not another measured performance estimate.
Where the evidence stops. Only two selected examples are shown, both in the -O setting. Small semantic objects are difficult to judge from these views, and the figure cannot establish classwise accuracy, failure frequency, or robustness to camera-perception errors.
Table 2. Alternating scene prediction improves average planning metrics, with exceptions at individual horizons. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with Input and Sup. rather than the bold scores: rows mix LiDAR, camera, and occupancy inputs and use different supervision. The -F pair is the camera-derived occupancy comparison; the bottom -O rows use occupancy directly. Lower values are better for both L2 trajectory error in meters and collision percentage. The retained footnote defines the dagger row as trajectories generated without scene predictions. Compare that row with full Ours-O to examine scene prediction's contribution, and inspect each horizon as well as Avg. The table's None entry refers to its supervision column, not an absence of semantic information in occupancy input. ev-planningev-dataev-generation
What it supports. Full Ours-O reports 1.14 m average L2 and 0.49% collision, compared with trajectory-only 1.18 m and 0.53%. Yet the dagger variant has lower collision at one and two seconds; the full model is better at three seconds. UniAD's 1.03 m and 0.31% averages use a different input/supervision configuration.
Where the evidence stops. The source does not specify whether the dagger variant is separately trained or only changed at inference. These offline trajectory comparisons neither establish a deployed feedback controller nor isolate architecture across baselines with unequal supervision.
Table 3. Occupancy supports stronger aggregate QA scores while leaving uneven question-type performance. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The five central groups are question categories, each divided into h0, h1, and all: zero-hop, one-hop, and their combined evaluation. The far-right acc column is overall Top-1 accuracy; it is not a sixth question category. Start with Ours-LLaMA2 against LiDAR-LLM and LLaVA because the text identifies a shared LLaMA-2-7B backbone for that comparison. Inputs still differ: occupancy, LiDAR, and cameras. LLaVA-D uses depth generated from point clouds. Keep Ours-LLaMA3.1 as a separate backbone result rather than treating its higher overall score as a controlled effect of the occupancy representation. ev-vqaev-dataev-implementation
What it supports. The matched-backbone OccLLaMA result is 53.4% overall accuracy versus LiDAR-LLM's 48.6%, a 4.8-percentage-point difference. The LLaMA3.1 variant reaches 54.5%. Performance is uneven: counting remains at 19.2% for that variant, and the LLaMA2 occupancy model trails LiDAR-LLM on zero-hop status questions, 48.0% versus 53.4%.
Where the evidence stops. Semantic occupancy already supplies category information that differs from raw sensor inputs. These comparisons support QA capability under the reported representations, but do not isolate language reasoning or demonstrate generalization beyond NuScenes-QA.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| 4D occupancy forecasting with ground-truth occupancy NuScenes/Occ3D, -O protocol; two seconds of history predict three seconds ahead. | Reported averages: 19.93 / 29.17; at 3 s: 15.26 / 24.41. Forecast mIoU / IoU (%), higher is better | OccWorld-O: averages 17.14 / 26.63; at 3 s 10.51 / 20.18. Longer-horizon gains are clearer: at 1 s, Ours-O mIoU 25.05 is below OccWorld-O 25.78. Reconstruction columns are separate from forecast performance. ev-dataev-forecast |
| 4D occupancy forecasting with camera-derived occupancy Same forecasting horizon; -F uses FBOCC occupancy predicted from cameras. | Reported 8.66 / 22.99. Average forecast mIoU / IoU (%) | OccWorld-F: 6.16 / 18.99. Improvement holds with predicted input, but the gap to -O shows that perception quality remains consequential. ev-forecast |
| Motion planning with occupancy input NuScenes-based Table 2; occupancy input, 1–3 s trajectory evaluation; no additional supervision in the table's Sup. column. | Reported Ours-O: 1.14 / 0.49; trajectory-only Ours-O†: 1.18 / 0.53. Average L2 (m) / collision (%) | OccWorld-O: 1.17 / 0.60. Camera-based Ours-F: 1.20 / 0.70 versus OccWorld-F 1.34 / 0.73. Scene prediction helps averages but not every horizon: trajectory-only collision is lower at 1/2 s. UniAD reports 1.03 / 0.31 with different inputs and richer supervision; these are not controlled architecture comparisons. ev-dataev-planning |
| Visual question answering on NuScenes-QA Table 3; matched LLaMA-2-7B comparison, with different input representations. | Reported Ours-LLaMA2: 53.4. Overall Top-1 accuracy (%) | LiDAR-LLM: 48.6; LLaVA: 47.4. Separate Ours-LLaMA3.1 result: 54.5. The matched-backbone advantage over LiDAR-LLM is 4.8 percentage points. Semantic occupancy, LiDAR and cameras provide different information, so the comparison does not isolate the tokenizer. ev-vqa |
| Spatial-attention and action-token ablation Table 5; components removed individually from the occupancy forecasting/planning model. | Reported full: 19.93 / 1.14 / 0.49; without spatial attention: 15.78 / 1.12 / 0.48. Forecast mIoU (%) / planning L2 (m) / collision (%) | Without action tokenization: 18.05 / 1.19 / 0.54. Action tokens benefit both tasks here. Spatial attention improves forecasting while slightly worsening planning; no repeated-run uncertainty is supplied. ev-components |
| Tokenizer capacity and caption-pretraining diagnostics Table 4 reconstruction; Table 6 QA category accuracy. These are separate ablations. | Reported default tokenizer: 75.20 mIoU; caption pretraining plus tuning: 66.6 comparison accuracy. Reconstruction mIoU (%) and comparison-question accuracy (%) | Codebook size 4096 instead of 2048: 70.94 mIoU; resolution 25 instead of 50: 59.04. Tuning without pretraining: 51.2 comparison accuracy. More codebook entries are not automatically better. Pretraining helps every listed QA category, but the experiment does not separate dynamics learning from caption alignment. ev-tokenizer-ablationev-pretrainingev-stages |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Spatial attention favors forecasting; dedicated action tokens help both reported tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The retained note expands s.a. as spatial attention and a.t. as action tokenization. Read the checkmarks carefully: the first row enables both, the second keeps only spatial attention, and the third keeps only action tokenization. Forecasting mIoU/IoU increase with improvement; planning L2/collision decrease. Removing spatial attention restores causal attention in the flattened scene order. Removing action tokenization represents waypoint numbers using the original language vocabulary. Compare each incomplete row with the first row to isolate the reported intervention, then compare the direction of change across the two task blocks rather than looking only at boldface. ev-componentsev-generationev-vocabulary
What it supports. Removing spatial attention reduces forecast mIoU from 19.93% to 15.78%, while planning improves slightly from 1.14 m/0.49% to 1.12 m/0.48%. Removing action tokenization lowers forecast mIoU to 18.05% and worsens planning to 1.19 m/0.54%. Better world prediction and better trajectory metrics are therefore not interchangeable outcomes in this ablation.
Where the evidence stops. There is no neither-component row or seed uncertainty. The authors attribute the planning tradeoff to spatial attention disrupting global causal attention, but this table does not directly test that explanation or establish statistical significance.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify data diversity and inference delay from model size as future work, proposing quantization and distillation. No measured latency accompanies the claimed advantage of one-step scene generation. ev-conclusionev-generation
All reported tasks use NuScenes-derived resources. Offline forecast, trajectory and QA scores do not establish closed-loop driving safety, out-of-domain generalization, or statistically reliable gains. Tables lack confidence intervals and seed variation. ev-dataev-forecastev-planningev-vqaev-components
7.2 Questions for discussion
- Does spatial attention's planning tradeoff persist across seeds and matched inference budgets?
- Would scene prediction still improve planning if its tokens were temporally misaligned?
- How much QA performance comes from semantic labels versus learned spatial reasoning?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs Occ3D/NuScenes preprocessing, FBOCC predictions for -F, NuScenes-QA, and the collected occupancy-caption pairs. It must preserve backbone and tokenizer differences between the main and matched VQA settings. ev-dataev-implementationev-forecast
The source leaves action-bin counts/edges, caption count/split construction, LoRA rank, detailed generative loss weighting, and software versions unspecified. Resolve Algorithm 1's K/S mismatch and the schematic quantizer connection before implementation. ev-vocabularyev-generationev-architectureev-stagesev-dataev-implementation
Proposed checks: repeat spatial-attention ablations with fixed tokenizer/data and multiple seeds; compare alternating-scene planning against trajectory-only and scene-shuffled controls to test whether aligned scene information drives the gain. ev-componentsev-planning
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the spatial-attention tradeoff with controlled training
Reader-proposed, not executed: train full and no-spatial-attention variants using the same frozen tokenizer, action vocabulary, data split, initialization policy, and optimizer-update budget, with at least three seeds. Preserve the source's two-second-history/three-second-forecast setting. Record per-horizon mIoU/IoU and planning L2/collision, plus measured inference time on identical hardware. Report confidence intervals and decoding costs rather than assuming equal speed. If the forecast gain persists while the small planning difference changes sign across seeds, the evidence would support a forecasting benefit but weaken the proposed causal-attention explanation for planning degradation. A stable tradeoff would justify a more targeted attention-mask experiment. ev-componentsev-generationev-forecastev-stagesev-implementation
Check 2: Separate useful scene feedback from extra generation
Reader-proposed, not executed: first document whether Table 2's dagger setting changes training or only inference. At a fixed checkpoint, compare normal alternating scene/action generation with trajectory-only output and with a control that replaces generated scene tokens by scene tokens from another validation example while preserving token count and modality boundaries. Keep the observed history, prompts, action decoder, and evaluation cases fixed; repeat sampling consistently. Measure each horizon separately. If normal scene tokens outperform the length-matched shuffled control, aligned scene content plausibly matters. If shuffling leaves planning unchanged, the reported average gain would not establish useful scene-conditioned feedback. Distribution shift from replacement must be reported as a limitation of this diagnostic. ev-planningev-generationev-vocabularyev-components
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page (1), task overview (2), full architecture and attention graphic (3), tokenizer equations and generation algorithm (4), training/data/hardware settings and forecasting table (5), qualitative forecasts and planning table including dagger note (6), and QA, tokenizer/component/pretraining ablations plus conclusion (7). All six final crops were separately viewed, with labels and relevant notes retained. Figure 2's mask direction was checked against Section 3.2; its missing explicit quantizer-to-decoder connection and repeated upper <act> label, and Algorithm 1's S/K mismatch, are disclosed. References on pages 8–9 were read in the supplied text; no scientific claim relies on uninspected external references. No appendix is present, and separate supplements remain unverified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, arXiv:2409.03272v1 [cs.CV], 5 September 2024; Abstract
- Sections 1–2.3: Introduction and all Related Works, pp. 1–3
- Sections 3.1–3.4: Overview, Scene Tokenizer, Generative World Model, Train Stage, pp. 3–5
- Sections 4.1–4.3: Experiment Settings, Results and Analysis, Ablation Study, pp. 5–7
- Section 5: Conclusion, p. 7
- References, pp. 8–9
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Reviewed the supplied v1 PDF; its title and six authors match the catalog. No later revision or venue edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by visually inspecting PDF pages 1–7 and all six final crops.
- Separate supplemental material availability has not been fully verified.
- No appendix appears in the nine-page supplied PDF. References on pages 8–9 were read as text; cited external works were not opened.
- Code was not inspected and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title block and left-margin arXiv version stamp
Exact catalog title and six authors: Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, Wenchao Ding. Affiliations are Fudan University's Academy for Engineering & Technology and Tsinghua University's Institute for AI Industry Research. Stamp: arXiv:2409.03272v1 [cs.CV], 5 Sep 2024.
Go to primary source ↓ev-motivationPDF p. 1, Abstract and Section 1; p. 2, Figure 1 and caption
The proposal unifies semantic occupancy, language and action for forecasting, motion planning and scene understanding. Figure 1 places a Camera2Occ module before OccLLaMA.
Go to primary source ↓ev-architecturePDF p. 3, Figure 2 and caption; p. 4, Section 3.2 Decoder
Figure 2 joins text, scene and action vocabularies in one OccLLaMA block; the attention diagram contains a blue scene-to-scene block and gray causal positions. Vox Head supplies a mask to Cls Head. The decoder arrow comes directly from the latent grid while quantization branches downward; Section 3.2 describes decoding after quantization. The upper action sequence repeats <act>, while Section 3.3 defines </act> as the closing boundary.
Go to primary source ↓ev-tokenizerPDF p. 4, Section 3.2 Encoder, Quantification, Decoder, Loss; Equations (1)–(2)
Non-air voxels become height/label pillar sets, followed by pillar embedding, Swin features, nearest-codebook quantization, and geometry/semantic heads. Loss sums weighted cross-entropy and Lovász terms for geometry/semantics plus embedding loss.
Go to primary source ↓ev-vocabularyPDF p. 4, Section 3.3 Unified Vocabulary
Code indices form scene vocabulary; waypoint coordinates map to nearest bins derived from trajectory statistics. Text, scene, action and functional tokens share a vocabulary. Numerical bin count and boundaries are not given.
Go to primary source ↓ev-generationPDF p. 4, Section 3.3 Next Token / Scene Prediction and Algorithm 1
Text/action retain causal next-token prediction; scene positions use spatial attention and learned queries to predict the whole scene in one forward pass. Algorithm 1 stops at end token or maximum length and branches when the latest token is <occ>; scene length is S, but replacement slices use undeclared K.
Go to primary source ↓ev-stagesPDF p. 5, Section 3.4 continuation: all three training stages
Tokenizer is trained then frozen. Full-parameter pretraining uses world-model and scene-caption objectives. Instruction tuning uses LoRA for understanding/planning prompts.
Go to primary source ↓ev-dataPDF p. 5, Section 4.1 Dataset
NuScenes has 700 training/150 validation 20-second videos at 2 Hz; Occ3D supplies occupancy; NuScenes-QA has five question types with zero/one-hop categories. A collected caption dataset matches occupancy to object positions, classes, states and future trajectories; no count or detailed split construction is supplied.
Go to primary source ↓ev-implementationPDF p. 5, Section 4.1 Implementation Details, including continuation at top of right column
Most comparisons: LLaMA-3.1-8B and 50×256×2048 tokenizer setting; matched VQA: LLaMA-2-7B and 25×25 resolution. AdamW; tokenizer 100 epochs, eight RTX 4090s, learning rate 10⁻⁴, batch 4, loss weights 10/5/10/5/5; generator pretraining 10 epochs at 10⁻⁴/batch 1 and tuning 5 epochs at 5×10⁻⁵/batch 4, on eight V100s. LoRA rank and software versions are absent.
Go to primary source ↓ev-forecastPDF p. 5, Table 1, all rows and mIoU/IoU horizon, average and reconstruction columns; Section 4.2 4D Occupancy Forecasting
Two-second history predicts three seconds. -O uses ground-truth occupancy; -F uses camera-derived FBOCC occupancy. Ours-O averages 19.93/29.17 versus OccWorld-O 17.14/26.63 mIoU/IoU; 3-second pairs 15.26/24.41 versus 10.51/20.18; 1-second mIoU 25.05 versus 25.78. -F averages 8.66/22.99 versus 6.16/18.99. Recon. means tokenizer reconstruction.
Go to primary source ↓ev-qualitativePDF p. 6, Figure 3 and caption
Two examples show GT, OccWorld-O and Ours-O across 0.5–3 seconds, with eight semantic color labels and red boxes at three seconds. The caption claims better long-term temporal modeling; examples are qualitative, not additional aggregate metrics.
Go to primary source ↓ev-planningPDF p. 5, Section 4.2 Motion Planning; p. 6, Table 2, Input/Sup., L2 and Coll. columns, and dagger footnote
Average L2/collision: Ours-O 1.14 m/0.49%, OccWorld-O 1.17/0.60, Ours-O† 1.18/0.53, Ours-F 1.20/0.70, OccWorld-F 1.34/0.73, UniAD 1.03/0.31 with richer supervision. Dagger removes scene predictions; dagger/full collision at 1 and 2 s is 0.02/0.04 and 0.17/0.24, respectively. Full/dagger 3-second collision is 1.20/1.39.
Go to primary source ↓ev-vqaPDF p. 5, Section 4.1 VQA configuration; pp. 6–7, Section 4.2 Visual Question Answering; p. 7, Table 3, accuracy and question-type columns
Top-1 accuracy: Ours-LLaMA2 53.4%, LiDAR-LLM 48.6%, LLaVA 47.4%, Ours-LLaMA3.1 54.5%. LLaMA2 uses the same 7B backbone as comparison baselines. h0/h1 indicate zero/one-hop questions. LLaMA3.1 count-all is 19.2%; LLaMA2 status-h0 is 48.0 versus LiDAR-LLM 53.4.
Go to primary source ↓ev-tokenizer-ablationPDF p. 7, Table 4 and Section 4.3 Scene Tokenizer Parameters
Res./Dim./Size 50/256/2048 gives 75.20 mIoU and 63.76 IoU reconstruction; size 4096 gives 70.94/61.03, size 1024 gives 68.26/58.81, resolution 25 gives 59.04/49.25, dimension 128 gives 65.93/57.66. Authors attribute a larger codebook's decline to overfitting/inefficient utilization and discuss resolution's token cost.
Go to primary source ↓ev-componentsPDF p. 7, Table 5, checkmark columns and all three rows; Section 4.3 Generative Model Components
Full s.a.+a.t.: forecast mIoU/IoU 19.93/29.17 and planning L2/collision 1.14/0.49. Spatial attention only: 18.05/28.55 and 1.19/0.54. Action tokenization only: 15.78/27.84 and 1.12/0.48. Removing spatial attention restores flattened causal attention; removing action tokenization uses language tokens for waypoints. Authors suggest spatial attention locally disrupts global causal attention.
Go to primary source ↓ev-pretrainingPDF p. 7, Table 6 and Section 4.3 Benefits of Pretraining
Pretraining plus fine-tuning versus fine-tuning alone: existence 80.9/70.5, count 19.2/12.4, object 46.3/41.2, status 47.8/32.7, comparison 66.6/51.2. Caption identifies P as pretraining and F as fine-tuning.
Go to primary source ↓ev-conclusionPDF p. 7, Section 5 Conclusion
Future work includes more diverse data and quantization/distillation to address inference delay caused by the parameter count.
Go to primary source ↓8.5 Primary sources
OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving ↗
PDF · 6,304 extracted words
Source fingerprint
0867aeb5a73d5096c6672cd6737ce8be62ee2c3d16400eab09e2522ecdde9c00