LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
1. Paper overview
In one sentence: LMGenDrive uses joint action and future-video training to improve a language-conditioned driving policy, then removes video generation from online control while retaining it for offline rollouts. e02e04e05e07e09e11
| At a glance | What to know |
|---|---|
| Research problem | Author claim Direct vision-language policies may miss scene evolution, while video generators alone do not establish instruction-following control. The authors seek a shared representation supporting both future imagery and closed-loop driving, particularly for rare or ambiguous situations. e02 |
| Core mechanism | |
| A key reported result | Closed-loop instruction-following driving on LangAuto: LangAuto: DS 62.2±3.3, RC 74.5±4.1, IS 0.85±0.04. Short DS: 77.1±4.1. Tiny DS: 84.1±3.6. Driving score (DS), route completion (RC), infraction score (IS); higher is better.. CARLA 0.9.10.1; LangAuto, Short and Tiny test tracks, spanning eight towns and varied weather/instructions; Table 1 reports three evaluation runs. BEVDriver DS: 48.9/66.7/70.2; AD-H: 44.0/56.1/77.5; LMDrive: 10.7±3.8/14.2±4.4/20.1±4.1, in the same track order. LMGenDrive leads the listed methods on all three tracks. The table retains different baseline uncertainty reporting, and its ± statistic is undefined; these are simulator results, not physical-road validation. e09e10 |
| Reading caution | Reader analysis Evaluation is confined to CARLA. No physical deployment, dedicated rare-event breakdown, statistical significance test or measured online latency is supplied. Figure 4 illustrates selected generated scenes; it does not quantify instruction fidelity or certify cross-view geometry. e09e10e15 |
Core contributions
Figure 2. Shared LLM context feeds two outputs, with generated feedback reserved for the autoregressive mode. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the lower left: the navigation instruction enters through a tokenizer, while camera views enter through the vision module. The inset expands that module into a vision encoder, Q-Former and adapter. Distinguish the current Action input from the learnable Action Query; they have different roles. Above the LLM, the orange branch supplies the world generator and the green branch predicts future action. Trace the dashed optional path from future video/action to the next input on the right. The caption identifies this as offline feedback; online operation instead receives new observations. Section 3.3 supplies the waypoint-to-PID step that the action icon compresses. e02e03e04e05
What it supports. The design connects action prediction and future imagery through the LLM, but preserves separate output mechanisms. Its optional loop explains how the same trained system can generate extended offline sequences. For online driving, the stated policy discards diffusion generation, so the world-model contribution is learned before deployment.
Where the evidence stops. The illustrated action icon does not mean direct pixel-to-actuator execution: the text specifies waypoint regression followed by PID controllers. The repeated right-hand portion is schematic and does not specify attention masks or all implementation details.
2. Motivation
2.1 The problem and the proposed response
Direct vision-language policies may miss scene evolution, while video generators alone do not establish instruction-following control. The authors seek a shared representation supporting both future imagery and closed-loop driving, particularly for rare or ambiguous situations. e02
2.2 What this reading follows
A driving policy must connect a command such as turning at the next intersection to what nearby vehicles and road geometry will do next. LMGenDrive asks whether learning to generate those futures can improve the policy's internal representation. Its LLM produces action-query features for waypoints and world-query features for a separate diffusion generator. The distinction between the two deployment modes matters: CARLA driving uses observed camera feedback without generating video, whereas offline synthesis consumes its own predictions. Read the diagrams as a training and information-flow design, then use the ablations to assess what the reported driving gains actually establish. e02e04e05e07e09e11
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × Joint prediction is supported by the separate LLM policy and diffusion generator linked through world-query conditioning, with action and video objectives trained together. The paper's word 'unified' does not make this a single shared generative backbone. It is not inverse dynamics, and online driving does not use video rollouts for action selection. e02e04e05e06e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Compress observations without losing the distinction between perception and planning
The camera-only choice serves a specific purpose: future generated frames will not come with LiDAR, so the encoder uses BEV positional queries instead of LiDAR-derived inputs. ResNet and transformer processing produce a spatial representation plus waypoint and traffic-light tokens. Pretraining gives these tokens driving-related supervision before the prediction heads are removed. A Q-Former then reduces roughly two thousand tokens per frame to eight queries, allowing recent observations to fit alongside a language instruction in the LLM. Separate action queries produce features for waypoint regression; world queries supply a different output interface for video conditioning. Figure 2 also shows a current-action input, which must not be confused with these learnable queries. The final driving command still comes from tracking the predicted waypoints with longitudinal and lateral PID controllers. e03e04e05
Figure 3. Current appearance and LLM context meet before per-view video denoising. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the two arrows into Multi-view World Fusion: the left input comes from the LLM and the right input comes from camera images through CLIP. The downward arrow carries the fused embedding to the white cross-attention block inside the denoising network. The legend distinguishes orange self-attention, white cross-attention and blue temporal attention. The lower curved arrow denotes repeated denoising, while the Views label indicates per-view video outputs. Read this together with Section 3.4: image conditioning supplies appearance and initial state, while world-query conditioning carries scene context. The source describes noise-to-video generation; the diagram alone is not a complete sampling algorithm. e06e07
What it supports. The authors deliberately give the generator access to the last observed images as well as the LLM representation. This makes appearance information available without requiring the compressed LLM path to encode every visual detail. Their claim that this encourages dynamic representations is a design rationale, rather than a measured disentanglement result.
Where the evidence stops. The snowflakes on the E/D blocks have no stage-specific freezing legend. Follow Section 3.5 for trainability: Stage 2 trains the generator; Stage 3 freezes its parameters while gradients propagate. Do not infer that the whole generator is always frozen.
5.2 Understand why a frozen generator can still teach the LLM
Stage 2 jointly adapts the LLM and generator using waypoint, instruction-completion and diffusion losses. Stage 3 changes both the horizon and parameter update pattern: generated video becomes subsequent input, generator parameters are frozen, and the LLM remains trainable. The paper explicitly says gradients still propagate. Reader interpretation: freezing weights is therefore different from detaching the generator's output from the computation graph; the fixed generator can still transmit a learning signal to its conditioning representation. However, the source does not fully specify the gradient path through denoising, image reuse and the frozen vision encoder, so the exact implementation remains a reproduction question. This training mechanism also explains why the online agent can benefit from a branch that it later discards. Its claimed advantage is acquired representation quality, rather than online video-based trajectory search. e02e03e06e07e08
5.3 Separate three kinds of evidence before accepting the central claim
First, Table 1 measures executed CARLA driving: the policy receives observations, issues controls and accumulates route-completion and infraction outcomes. Second, Tables 2 and 4 ask whether architecture and training choices matter for that policy, although the world-generator removal changes queries too and iteration matching does not match all computational factors. Third, Tables 3 and 5 assess generated videos rather than vehicle behavior. Reader interpretation: together these experiments support a useful training connection between understanding and generation, but they do not establish that realistic videos are accurate simulators for choosing actions. Figure 4 provides examples of instruction-associated futures, not a quantitative instruction-consistency test. The distinction becomes especially important at longer horizons, where the authors report drift and instruction deviation even though the model can keep generating additional frames. e09e10e11e12e13e14e15
5.4 Training and inference
During training
Stage 1 pretrains perception on 3M expert CARLA frames with object detection, waypoint regression and traffic-light classification. Its prediction heads are discarded and the encoder stays frozen. Stage 2 jointly fine-tunes the LLM and generator. Stage 3 feeds generated videos into subsequent steps, freezes generator parameters while retaining gradient propagation, and keeps the LLM trainable. e03e07
Stages 2–3 use L1 waypoint, binary completion and diffusion losses. The main schedule is 20k/20k/10k iterations. Implementation names Vicuna-7B, Stable Diffusion 1.5 and AnimateDiff; AdamW starts at 0.00001, using eight H800 GPUs with DeepSpeed ZeRO-2 for roughly two days. Multi-step training is limited to one–three timesteps by memory. e07e08
During inference
Online planning discards diffusion generation and refreshes the visual history with observed simulator frames. Offline generation reuses the last generated frame and predicted action. There is no specified online search that scores candidate trajectories with generated videos. e02e04e05e09
5.5 Implementation flow
- Encode a camera-only scene
ResNet features from multiple cameras are fused by a transformer. BEV positional queries replace LiDAR-derived inputs, keeping future generated imagery usable. The encoder supplies BEV, four waypoint and one traffic-light token; a Q-Former compresses roughly 2,000 visual tokens into eight queries per frame, followed by an MLP adapter. e03
- Predict executable motion
The LLM combines instruction tokens, compressed visual history, action queries and world queries. A two-layer MLP maps action-query features to waypoints and an instruction-completion flag. Two PID controllers translate waypoints into longitudinal and lateral control; generated pixels are not vehicle commands. e03e04e05
- Condition future imagery
LLM world-query outputs provide dynamic context. CLIP encodes the last multi-view images; self-attention fuses views and cross-attention injects LLM guidance. Per-view embeddings condition a diffusion U-Net with spatial and temporal attention, producing future video through iterative denoising. e05e06
6. Experiments & results
LMGenDrive couples a language-based driving policy to a multi-view diffusion generator during training. Separate action and world queries connect instruction-grounded planning to future-video supervision. At deployment, online driving uses observed camera feedback and discards the generator; offline synthesis rolls generated views and predicted actions forward. The paper reports improved CARLA LangAuto driving scores, while its horizon experiment exposes accumulating video errors. This is evidence for a training-time generative contribution to simulated driving, with important implementation and evaluation gaps.
6.1 Read the original evidence
Table 1. The strongest listed driving scores occur across all three LangAuto tracks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each track as its own three-column block. DS is the primary driving score, RC measures route completion, and IS penalizes infractions; all arrows point upward. Section 4.1 defines DS as RC multiplied by IS. Start with the Ours row and compare vertically within a single track, rather than treating route lengths as interchangeable settings. The caption retained above the table states three evaluation runs and notes that AD-H uses an additional OPT-350M model for low-level control. These evaluations run in CARLA 0.9.10.1 with visual observations and natural-language commands, using recent sensor history during testing. e08e09e10
What it supports. On LangAuto, LMGenDrive reports DS 62.2±3.3 versus BEVDriver's 48.9, alongside RC 74.5±4.1 and IS 0.85±0.04. Its DS is also highest in the displayed Short and Tiny comparisons, at 77.1±4.1 and 84.1±3.6. The evidence concerns executed simulator control, not merely attractive generated video.
Where the evidence stops. The paper does not define the ± statistic, and AD-H/BEVDriver entries lack comparable uncertainty. The table does not establish matched training data or compute across methods, statistical significance, physical-road transfer or a separate rare-event success rate.
Table 5. Autoregressive extension degrades both image and video distribution metrics. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally to increase the prediction horizon, while keeping the metric row fixed. The authors generate multi-view videos autoregressively and compare them with ground-truth videos on the same evaluation split. FID is used for visual realism and FVD for video quality and temporal consistency; lower is better in each row. Both rows worsen at every reported horizon, including the intermediate 16- and 24-frame settings. Section 4.4 attributes long-horizon problems to accumulating errors and describes distant-object drift, inconsistent agent motion and deviation from instructions. These columns describe offline video length, not the route lengths in the closed-loop driving benchmark. e09e12e14
What it supports. FID/FVD increase from 6.3/286 at 8 frames to 15.1/981 at 128 frames. The model therefore supports autoregressive extension, but the reported scores do not establish stable long-horizon simulation. The accompanying qualitative failure descriptions make the degradation relevant to dynamics and instruction alignment, beyond a single aggregate image metric.
Where the evidence stops. FID and FVD use different feature spaces and scales, so compare changes within each metric rather than equating their raw increments. The paper omits evaluation sample counts and exact split construction; this table measures neither action-video consistency nor driving safety directly.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Closed-loop instruction-following driving on LangAuto CARLA 0.9.10.1; LangAuto, Short and Tiny test tracks, spanning eight towns and varied weather/instructions; Table 1 reports three evaluation runs. | LangAuto: DS 62.2±3.3, RC 74.5±4.1, IS 0.85±0.04. Short DS: 77.1±4.1. Tiny DS: 84.1±3.6. Driving score (DS), route completion (RC), infraction score (IS); higher is better. | BEVDriver DS: 48.9/66.7/70.2; AD-H: 44.0/56.1/77.5; LMDrive: 10.7±3.8/14.2±4.4/20.1±4.1, in the same track order. LMGenDrive leads the listed methods on all three tracks. The table retains different baseline uncertainty reporting, and its ± statistic is undefined; these are simulator results, not physical-road validation. e09e10 |
| Planning-module ablations Table 2 baseline matches the LangAuto result; §4.3 describes each intervention. | Full 62.2±3.3; without world generator 53.4±2.2; without action queries 58.7±3.1; without visual pretraining 54.9±4.5; without Stage 3 55.6±4.5. DS, higher is better. | Removing the generator also removes world queries; removing action queries substitutes autoregressive action prediction. The coupled generation branch is helpful, but this does not isolate video supervision from query architecture. Contrary to the prose ranking, visual-pretraining removal causes a larger DS drop than Stage-3 removal. e11 |
| Generation-module ablations Table 3; generation evaluation sample count and exact split construction are not specified. | Full: 6.3/286; no multi-view fusion: 7.8/371; 32 world queries: 10.1/318; 16 queries: 11.6/424. FID/FVD, lower is better. | Full configuration uses 64 world queries. View fusion and query capacity improve reported distributional video metrics. They do not directly measure calibrated dynamics, instruction success or collision risk. e12e09 |
| Training curriculum under controlled iteration counts Table 4 assigns 30k iterations across Stages 2–3; reports Tiny and Short. | Full: 84.1/77.1; Stage 3 only: 78.0/67.9; Stage 2 only: 80.1/72.3, in Tiny/Short order. DS, higher is better. | Full uses 20k Stage-2 plus 10k Stage-3 iterations; alternatives allocate 30k to one stage. Both stages outperform either alone. Equal iteration counts do not establish equal compute; Table 4 does not restate Stage-1 handling or supply uncertainty. e13e08 |
| Autoregressive video generation across horizons Table 5 compares generated and ground-truth videos on the same evaluation split at 8, 16, 24, 32, 64 and 128 frames. | 8 frames: 6.3/286; 32: 9.2/435; 64: 12.8/610; 128: 15.1/981. FID/FVD, lower is better. | Both metrics worsen monotonically across all six reported horizons. The authors observe distant-object drift, inconsistent agent motion and instruction deviation. This diagnoses offline generation, without measuring closed-loop driving at these video horizons. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Tables 2 and 3. Planning and video metrics diagnose different parts of the coupled system. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the two tables separate when interpreting an intervention. Table 2 on the left reports driving metrics: the baseline has DS 62.2, and the no-world-generator row has DS 53.4. Section 4.3 specifies that this removal also deletes world queries; the no-action-query row instead substitutes autoregressive action prediction. Table 3 on the right reports lower-is-better video metrics, with a 64-world-query baseline. Its arrows from 64 to 32 or 16 denote reduced query counts. The two sets of metrics answer different questions: improved video distributions do not by themselves demonstrate improved vehicle control, and the left table does not isolate all factors changed by each removal. e02e11e12
What it supports. All displayed removals degrade their respective baseline. Without multi-view fusion, FID/FVD worsen from 6.3/286 to 7.8/371. The planning table supports a contribution from the coupled generation branch during training, especially because the full model also omits diffusion generation during online planning. It does not prove that generating videos online would improve decisions.
Where the evidence stops. The prose calls generator and Stage-3 removal the largest planning drops, but Table 2 places visual-pretraining removal second: 54.9 is below 55.6. Preserve the table values. Generator removal also removes queries, leaving the specific causal contribution unresolved.
Table 4. Keeping both grounding and autoregressive training helps under the reported iteration budget. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Setting column: the baseline allocates 20k iterations to Stage 2 and 10k to Stage 3, while each alternative assigns 30k to the remaining stage. The source table abbreviates these allocations as 2w, 1w and 3w; Section 4.3 provides the iteration counts. Compare DS vertically in the Tiny block, then repeat in the Short block. The adjacent RC and IS columns show that the differences concern both progress and infractions. To understand the intervention, return to Section 3.5: Stage 2 jointly trains planning and generation on single steps; Stage 3 reuses generated observations over multiple steps. e07e08e11e13
What it supports. The combined schedule yields DS 84.1 on Tiny and 77.1 on Short, above either single-stage alternative. Removing Stage 3 gives 80.1/72.3; removing Stage 2 gives 78.0/67.9. These numbers favor the curriculum, and show that multi-step training alone does not replace initial single-step grounding.
Where the evidence stops. Equal iteration counts do not guarantee equal compute or data exposure when sequence lengths and trainable components differ. Table 4 gives no uncertainty and does not restate how Stage 1 is handled. This is a different control from simply omitting Stage 3 in Table 2.
7. Analysis & limitations
7.1 What the evidence leaves open
Evaluation is confined to CARLA. No physical deployment, dedicated rare-event breakdown, statistical significance test or measured online latency is supplied. Figure 4 illustrates selected generated scenes; it does not quantify instruction fidelity or certify cross-view geometry. e09e10e15
The training setup says eight future frames from t+0.1 to t+0.9 seconds in 0.1-second increments, which lists nine timestamps if both endpoints are included. The source does not resolve the indexing. It also lacks a dedicated limitations section; long-horizon degradation is discussed in §4.4. e08e14
7.2 Questions for discussion
- Would correctly aligned video supervision outperform an architecture-matched no-video-loss control? (e11)
- How much does Stage 3 depend on allowing gradients through the frozen generator versus simply exposing the LLM to generated observations? (e07)
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include CARLA expert perception labels, instruction-aligned driving sequences, the named pretrained models, three 224×224 RGB views at 10 Hz, an eight-frame history and four waypoint targets at offsets 0.2/0.4/0.6/0.8 seconds. e03e07e08
A reproducible implementation still needs exact train/test manifests, Stage-2/3 dataset construction, batch size, loss weights, diffusion sampler/steps, action encoding, PID gains and the Stage-3 gradient path through generated-image reuse. CLIP trainability and the frame-index ambiguity also need resolution. The supplied paper does not establish a verified code release. e04e05e06e07e08e09
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate the learning signal from the presence of world queries
Reader-proposed check, not performed: start from the same pretrained vision encoder and retain the same LLM, action/world queries and generator in every arm. Compare normal video-loss gradients into the LLM with an arm that stops only that loss's gradient at the LLM conditioning interface; keep planning losses active. Match data, stage schedule, generated-feedback policy and iteration budget, and report compute and multiple seeds. Evaluate both agents online with the generator disabled, using identical LangAuto routes and DS/RC/IS reporting. A repeatable advantage for the connected-gradient arm would support the proposed generative teaching signal beyond query capacity; no advantage would weaken that explanation of Table 2. Specify the gradient implementation before interpreting the result. e02e05e07e08e09e11
Check 2: Locate the source of long-horizon video drift
Reader-proposed check, not performed: on a fixed held-out set of instruction-aligned clips, compare normal autoregressive generation with an oracle-reset condition that replaces generated camera inputs with ground-truth views at each eight-frame boundary. Use the same checkpoint, prompts, random seeds, horizons and predicted-action feedback in both arms. Record FID/FVD at the paper's 8–128-frame horizons and separately annotate instruction deviations and agent-motion discontinuities. A substantially reduced horizon penalty after resets would implicate visual-feedback error accumulation; persistent degradation would point toward action feedback, conditioning or within-segment dynamics. Disclose the reset intervention as a diagnostic using privileged observations, and publish the exact clip split and metric implementation rather than presenting it as deployable driving performance. e05e08e12e14e15
8.3 Reading coverage
Visual audit: Rendered and visually inspected the title/authors/affiliations/version on p. 1; motivation and deployment modes on p. 2; Figure 1 on p. 3; encoder and policy details on p. 4; Figure 2, PID control and world conditioning on p. 5; Figure 3, training losses, freeze schedule, timing and hardware on p. 6; Tables 1–3 and evaluation/implementation details on p. 7; Figure 4, ablation definitions and horizon protocol on p. 8; and Tables 4–5, failure descriptions and conclusion on p. 9. Every final crop was actually viewed, with corrected framing for both architecture figures and Table 4. Figure 2's optional feedback and Figure 3's conditioning arrows/attention legend were checked against the captions and method; the freeze-symbol interpretation is explicitly bounded in the generator caution. All retained method, training, numerical, evaluation and proposed-check premises are covered by these pages. The complete supplied text, including references on pp. 9–12, was read in five chunks. Reference-only pages 10–12 were not visually inspected; separate supplements and code were not inspected, and no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations, abstract and arXiv version stamp (p. 1)
- 1. Introduction (pp. 1–2)
- 2. Related works, including 2.1–2.3 (pp. 3–4)
- 3. Method, including 3.1–3.5 (pp. 4–6)
- 4. Experiments, including 4.1–4.5 (pp. 6–9)
- 5. Conclusion (p. 9)
- References (pp. 9–12)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting PDF pages 1–9 and all six final crops.
- Separate supplemental material availability has not been fully verified.
- Only the supplied arXiv:2604.08719v1, dated 9 April 2026, was reviewed; no other revision was supplied or compared. The observed title and all eight authors match the catalog.
- No appendix is present in this 12-page PDF. Code, external references and separate supplements were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block and arXiv margin stamp
The exact catalog title and all eight authors appear in the same order. Affiliations are CUHK MMLab, University of Toronto and UC Berkeley. The artifact identifies itself as arXiv:2604.08719v1 [cs.CV], 9 April 2026.
Go to primary source ↓e02PDF pp. 1–2, Abstract and §1, especially the two operating modes on p. 2
The authors motivate integrating semantic understanding with future-scene generation. Online planning discards diffusion generation; offline generation autoregressively reuses generated video and predicted control.
Go to primary source ↓e03PDF p. 4, §3.2 and §3.3, Instruction and visual tokenization
ResNet and transformer features feed BEV positional queries, waypoint tokens and a traffic-light token. Pretraining heads use detection, L1 waypoint and traffic-light classification losses. A frozen encoder and eight-query Q-Former compress about 2k tokens per frame; recent history is buffered.
Go to primary source ↓e04PDF p. 4, §3.3 Action prediction; p. 5, continuation and Figure 2 caption
Learnable action queries produce waypoint features and a completion flag through a two-layer MLP. Separate longitudinal and lateral PID controllers produce brake, throttle and steering. Figure 2 includes a current-action input and optional predicted-action feedback.
Go to primary source ↓e05PDF p. 5, Figure 2 and §3.4 World Query Conditioning
Instruction, visual, action and world-query inputs feed the LLM. World-query outputs condition a separate generator. The dashed autoregressive path returns the generated last frame and predicted action; the caption distinguishes offline generation from observed online feedback.
Go to primary source ↓e06PDF p. 5, §3.4 Multi-view Image Conditioning and World Generator; p. 6, Figure 3 and caption
CLIP image features and LLM world embeddings are fused with attention. View-specific embeddings enter a diffusion U-Net through cross-attention, with spatial/temporal transformers. Figure 3's legend identifies self, cross and temporal attention; its E/D blocks carry snowflakes without a stage-specific freezing legend.
Go to primary source ↓e07PDF p. 6, §3.5, all three stages and the unnumbered diffusion-loss equation
Stage 1 uses 3M expert CARLA frames and freezes the pretrained vision encoder. Stage 2 jointly trains LLM and generator. Stage 3 extends to 2–3 steps with generated feedback, freezes generator parameters while retaining gradient propagation, and trains the LLM. The last two stages use waypoint, completion and diffusion losses.
Go to primary source ↓e08PDF pp. 6–7, §4.1 Training Details
Three 224×224 cameras are sampled at 10 Hz with eight-frame context. Four waypoint offsets are specified; the eight-video-frame claim conflicts with the stated inclusive 0.1–0.9-second range. Training uses AdamW at 10^{-5}, eight H800 GPUs, ZeRO-2, roughly two days, 20k/20k/10k iterations, Vicuna-7B, Stable Diffusion 1.5 and AnimateDiff. Stage 3 is memory-limited to one–three timesteps.
Go to primary source ↓e09PDF p. 7, §4.1 Benchmark and Metric; §4.2 sensor-history description; Table 1 caption
CARLA 0.9.10.1 and LangAuto test routes across eight towns, varied weather and misleading language are described. DS is RC times IS; IS penalizes infractions. FID/FVD assess generated-video quality. Table 1 reports three runs and notes AD-H's additional OPT-350M controller, but does not define the ± statistic.
Go to primary source ↓e10PDF p. 7, Table 1, all rows and LangAuto/Short/Tiny DS, RC and IS columns
LMGenDrive reports LangAuto 62.2±3.3 DS, 74.5±4.1 RC, 0.85±0.04 IS; Short 77.1±4.1 DS, 87.9±3.5 RC, 0.88±0.03 IS; Tiny 84.1±3.6 DS, 92.5±4.0 RC, 0.92±0.04 IS. DS comparators are BEVDriver 48.9/66.7/70.2, AD-H 44.0/56.1/77.5, and LMDrive 10.7±3.8/14.2±4.4/20.1±4.1.
Go to primary source ↓e11PDF p. 7, Table 2; pp. 7–8, §4.3 Ablation Study on Module Design
DS values are full 62.2±3.3, no generator 53.4±2.2, no action queries 58.7±3.1, no visual pretraining 54.9±4.5, and no Stage 3 55.6±4.5. The first removes world queries too; the second substitutes autoregressive action prediction. The prose calls generator and Stage-3 removal the largest drops, although visual-pretraining removal has the second-largest numerical drop.
Go to primary source ↓e12PDF p. 7, Table 3; p. 8, §4.3 Ablation Study on Generation Module Design
FID/FVD are 6.3/286 for the 64-query baseline, 7.8/371 without multi-view fusion, 10.1/318 with 32 queries, and 11.6/424 with 16. Sample counts, exact evaluation split construction and uncertainty are not reported here.
Go to primary source ↓e13PDF p. 8, §4.3 Ablation Study on training stages; p. 9, Table 4
With 30k iterations allocated across Stages 2–3, baseline DS is 84.1 Tiny and 77.1 Short; Stage 3 only gives 78.0/67.9; Stage 2 only gives 80.1/72.3. Table labels use 2w/1w/3w, corresponding to the textual 20k/10k/30k allocation. Stage-1 treatment is not restated.
Go to primary source ↓e14PDF pp. 8–9, §4.4; p. 9, Table 5, all horizon columns
Against ground-truth video on the same evaluation split, horizons 8/16/24/32/64/128 yield FID 6.3/7.4/7.8/9.2/12.8/15.1 and FVD 286/340/371/435/610/981. The authors identify distant-object drift, inconsistent agent motion and instruction deviation at longer horizons.
Go to primary source ↓e15PDF p. 8, Figure 4; p. 9, §4.5 and §5
Figure 4 shows initial multi-view observations and three generated future steps for turning, slowing/stopping and maintaining course. The authors interpret examples as spatially consistent and instruction-aligned. The conclusion frames real-world scaling as future work.
Go to primary source ↓8.5 Primary sources
LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving ↗
PDF · 7,894 extracted words
Source fingerprint
cc02913ac3e9645d8f430f1d8199b3415e2770d697e9efcc0d7c2dde761a3b6f