ADriver-I: A General World Model for Autonomous Driving
1. Paper overview
In one sentence: ADriver-I links language-based control prediction to action-conditioned video generation, enabling recurrent imagined driving while leaving physical fidelity and long-term reliability unverified. e03e05e11e13e15e16e17e18
| At a glance | What to know |
|---|---|
| Research problem | Source description Can a driving system predict controls directly from visual history and imagine their consequences without explicit boxes or HD maps? The authors propose interleaved image/action tokens as an interface between control prediction and future-scene generation, replacing the usual sequence of task-specific perception and planning modules. e02e03 |
| Core mechanism | |
| A key reported result | Current-frame control prediction on nuScenes: 0.072 m/s speed L1; 0.091 rad steering L1; steering A_0.01 = 0.411. Speed and steering L1 error (lower); steering A_0.01 (higher). Three historical vision-action pairs plus current image; supervised control evaluation. Exact validation split and sample count are not specified. ViT-based: 0.103 m/s, 0.092 rad and 0.299. MLP-only: 0.122 m/s and 0.101 rad; CNN-based: 0.106 m/s and 0.095 rad. Speed and tight-tolerance steering improve over the constructed baselines, while mean steering error barely changes against ViT. The comparison does not isolate model size or pretraining effects. e07e09e10e11 |
| Reading caution | Source description The authors identify degraded generated frames under rapid control changes, inadequate deployment performance, separate training, and missing route information. Feeding imperfect generated observations back into the policy can propagate errors. e18 |
Core contributions
Figure 1. Predict an action, generate its visual continuation, and reuse the next image. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the purple image tokens and orange action tokens across the top. Three historical pairs and the current image enter the MLLM; the current predicted action then travels upward into diffusion. The history also conditions diffusion directly. The green path returns the next generated image to the MLLM for another action prediction. Dashed downward arrows to the two historical predictions indicate extra training supervision. Section 3.2 supplies an omitted interface detail: GPT3.5 turns numerical controls into motion language for the VDM. The diagram therefore summarizes information flow between two models rather than one jointly optimized predictor. e02e03e04e05e06
What it supports. The architecture supports action-first forecasting with feedback through generated observations. It connects an action predictor and a forward generator without requiring boxes or HD maps. The operative evidence is the explicit sequence of predictions; the caption's 'self-learning' language is not accompanied by online optimization in the training description.
Where the evidence stops. The next-action box also has a dashed outline, although Section 3.1 explicitly generates that action during inference. Follow the green arrows and textual procedure; the caption's training-only description of dashed marks does not consistently explain this outline.
2. Motivation
2.1 The problem and the proposed response
Can a driving system predict controls directly from visual history and imagine their consequences without explicit boxes or HD maps? The authors propose interleaved image/action tokens as an interface between control prediction and future-scene generation, replacing the usual sequence of task-specific perception and planning modules. e02e03
2.2 What this reading follows
ADriver-I asks a language model to predict what a car should do, then asks a video model to depict what happens next. Its central interface is a history of interleaved front-view images and numerical actions. This reading follows that interface from tokenization, through intermediate action supervision, into motion-language conditioning and recurrent generation. The experiments offer useful evidence for speed prediction and action encoding, but the generation comparison mixes forecasting protocols. The final driving sequence is an illustration of the proposed loop. Read it alongside the authors' warning that poor generated frames can disturb the next control prediction. e03e05e11e13e15e16e17e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system classification is supported by distinct MLLM and VDM architectures and separate training. Actions are predicted from past/current observations, then condition forward video generation; they are not recovered by inverse dynamics from a desired future. 'Other mechanisms' and 'Outside quadrants' reasonably describe this staged action-first loop rather than a single backbone jointly predicting video and action. The WAM placement reflects coupled control and forecasting, with demonstrated deployment capability remaining limited. e03e04e06e18
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn the driving history into supervised conversation
Start with the distinction between an observation and its associated control value. ADriver-I encodes the image with CLIP and an adapter, while speed and steering become language tokens. The current image appears after three historical image/action pairs, so the model has visual context and recent motion history when completing the next action. Training also asks it to predict intermediate actions. This makes the history contribute supervised targets as well as context. The representation ablations explain why the numeric interface matters: absolute values beat English number words and adjacent-frame differences. Reader interpretation: the conversation structure supplies a practical training format, but these comparisons alone do not demonstrate a general numerical-reasoning ability. They combine different target and tokenization choices. e03e04e05e07e13e15
5.2 Separate predicting an action from predicting its consequences
The control model and video model solve different conditional problems. The MLLM predicts the current action from the observed history. VDM then receives history together with action-derived language and predicts future images. GPT3.5 bridges these interfaces because the authors found the diffusion text encoder poor at interpreting raw steering values. This bridge preserves coarse motion categories, but its quantitative fidelity is not established. The example descriptions even characterize changing numerical histories as nearly stable motion. Reader interpretation: two nearby control sequences could conceivably produce similar language conditions, weakening precise action sensitivity. That is a testable possibility rather than a reported result. Separate training also means video-generation errors do not automatically train the control predictor through an end-to-end objective. e03e04e05e06e18e20
5.3 Treat recurrent imagination as a new evaluation problem
A four-frame prediction and a recurrent driving system require different evidence. In the basic forecasting task, the model begins from supplied observations and its videos receive FID/FVD scores. In the recurrent procedure, its own first predicted image becomes the next observation. Any visual error can therefore influence the next action and the following generated scene. Figure 7 shows how to repeat this procedure, but a finite example with a continuation marker does not establish indefinite stability. The authors explicitly recognize degradation when actions change rapidly. Reader interpretation: a useful evaluation must separate short-horizon visual quality, quantitative action response and error growth under repeated feedback. Matching context and output horizon is also necessary before attributing Table 6's score differences to the model itself. e03e09e16e17e18
5.4 Training and inference
During training
The two models are trained separately. MLLM pretraining uses nearly 1.4M private highway vision-action pairs: the LLM is frozen while the visual encoder and adapter learn. During SFT, only the visual encoder is frozen. Separate SFT runs use nuScenes and private data; cross-entropy supervises text outputs. e06e07
VDM inherits pretrained diffusion weights, pretrains on private data and fine-tunes on about 23K nuScenes videos. Training uses eight-frame clips at 256×512. The paper invokes a prior latent-video-diffusion training scheme without presenting its own diffusion-loss equation or a complete noise configuration. e04e06e08
During inference
The current control prediction conditions VDM through motion language; DDIM uses 50 sampling steps. Figure 1's green feedback path is inference recurrence. Its caption's 'self-learning' wording is not accompanied by an online weight-update procedure; Section 3.3 instead specifies separate offline training. e03e05e06e08
Initialization is incompletely reconciled: Section 3.1 needs three history pairs plus a current image, while Section 4.5 says the demonstration receives only three history pairs. The missing-current-frame detail remains unresolved. e03e17
5.5 Implementation flow
- Encode a driving conversation
CLIP-ViT-Large extracts visual features, a two-layer MLP adapter maps them into language space, and Vicuna-7B-1.5 processes interleaved image/action tokens. The adapter is initialized from LLaVA-7B-1.5. The implementation represents three-decimal controls as integers after multiplication by 1000. e04e05e07
- Predict actions with intermediate supervision
Training supervises actions at t−2, t−1 and t; inference requests the current action. Historical prediction outputs in Figure 1 should not be confused with executing a sequence of earlier controls. e03e15
- Translate action into a video condition
GPT3.5 converts consecutive numerical controls into descriptions such as turning or decelerating because the VDM text encoder struggles with raw numerical motion semantics. Stable Diffusion 2.1 is extended with temporal modules and reference-frame concatenation; motion text supplies action guidance. e04e05
- Feed imagination back into control
The VDM predicts four future frames. Selecting the first as the next current image permits repeated action prediction and generation. This is feedback through a learned image generator; no physical actuator execution or external simulator validation is reported in the recurrent demonstration. e03e17
6. Experiments & results
ADriver-I couples a multimodal language model that predicts speed and steering with an action-conditioned video diffusion model. Generated frames can feed the next control prediction. Its strongest evidence concerns supervised control prediction and short-horizon video quality; the recurrent demonstration does not establish reliable autonomous driving. The inspected source is the original arXiv v1.
6.1 Read the original evidence
Table 2. Strong speed gains coexist with nearly unchanged mean steering error. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read speed and steering as separate targets with different units. Within each block, smaller L1 is better, while larger tolerance accuracy is better. The four thresholds count predictions whose absolute error falls within the stated tolerance; these entries are proportions, not errors. Compare the first four rows for the nuScenes experiment. The dagger row is a separate private-data result, as the retained dataset note explains. For the strongest visual baseline, compare ViT-Based against ADriver-I: speed error changes substantially, whereas steering L1 barely moves. The steering tolerance columns expose a difference hidden by that near tie. e07e09e10e11e12
What it supports. On nuScenes, ADriver-I reports speed L1 of 0.072 m/s against ViT-Based's 0.103 m/s, but steering L1 is 0.091 versus 0.092 rad. Steering accuracy at tolerance 0.01 rises from 0.299 to 0.411. The evidence favors better small-error control prediction without a comparable reduction in mean steering error.
Where the evidence stops. The private-data row changes the dataset and training exposure. It cannot establish transfer or a causal scaling benefit. The baselines also do not isolate MLLM capacity and private pretraining, and no confidence intervals are reported.
Table 6. The lowest generation distances come with a different forecasting protocol. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read Input→Output before comparing the metric columns. F counts frames; B refers to boxes and M to HD maps. The original note calls B a 'bouncing box', while the surrounding method text consistently discusses bounding boxes. All three methods are conditioned on controls. ADriver-I receives four frames and predicts four, whereas the other rows predict twelve from one observed frame; DriveDreamer additionally receives box and map inputs. FID summarizes frame-level image-distribution similarity and FVD assesses video quality. Neither column directly reports action accuracy, collision avoidance, or the correctness of a particular predicted future. e04e09e16e19
What it supports. Table 6 reports FID 5.5 and FVD 97.0 for ADriver-I, compared with 73.4/502.3 for DriveGAN and 52.6/452.0 for DriveDreamer. These are the lowest reported distances in the table, but they accompany longer visual context and a shorter prediction horizon. The introduction gives the more precise FID value 5.52.
Where the evidence stops. Context length, output horizon and spatial priors differ across rows. The comparison therefore cannot isolate architecture quality. Evaluation sampling details and uncertainty are not provided, and distributional realism does not establish physically accurate responses to control interventions.
Figure 7. A finite illustrated rollout shows the proposed feedback cycle. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the images left to right across successive rows, using the timestamps to maintain temporal order. The blue bars correspond to speed on the left axis; orange bars correspond to steering angle on the right axis. Do not compare their heights as if they shared a unit or scale. The scene turns toward a roadway while the steering bars approach zero. Section 4.5 attributes frames to VDM and controls to MLLM. The final panel is a continuation marker, not another measured scene. Section 3.1 explains the mechanism: reuse the first predicted frame as the next control input. e03e17e18
What it supports. The figure illustrates a finite sequence of coupled video and control outputs, with labels extending to +3.5 seconds. It makes the intended feedback procedure concrete. It does not supply ground-truth future frames, a success rate, or an intervention showing how the same initial scene responds to different actions.
Where the evidence stops. Section 4.5 calls all displayed frames generated and mentions only three initial history pairs, while Section 3.1 also requires a current image. Initialization is underspecified. The authors explicitly warn that rapid control changes can degrade frames and subsequent predictions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Current-frame control prediction on nuScenes Three historical vision-action pairs plus current image; supervised control evaluation. Exact validation split and sample count are not specified. | 0.072 m/s speed L1; 0.091 rad steering L1; steering A_0.01 = 0.411. Speed and steering L1 error (lower); steering A_0.01 (higher) | ViT-based: 0.103 m/s, 0.092 rad and 0.299. MLP-only: 0.122 m/s and 0.101 rad; CNN-based: 0.106 m/s and 0.095 rad. Speed and tight-tolerance steering improve over the constructed baselines, while mean steering error barely changes against ViT. The comparison does not isolate model size or pretraining effects. e07e09e10e11 |
| Current-frame control prediction on private highway data Private-data SFT and evaluation; distinct from the nuScenes comparison. | 0.035 m/s and 0.015 rad. Speed and steering L1 error | The same table's nuScenes result is 0.072 m/s and 0.091 rad. This is a cross-dataset observation. The authors cite lower highway action variance and more SFT data; it is not evidence of transfer or an isolated scaling benefit. e12 |
| Action representation and supervision ablations Section 4.3 control ablations; no separate split, seed variation or uncertainty is given. | Absolute Number: 0.072/0.091; Multi Round: 0.072/0.091. Speed/steering prediction error, in m/s and rad | Num2English: 2.094/0.536; Special Token: 0.094/0.106; Relative Diff: 0.081/0.096. Single Round: 0.078/0.094; Temporal Fusion: 0.078/0.092. Two decimals: 0.073/0.091. The results favor absolute numeric controls and intermediate supervision. Near equality between two and three decimals suggests limited benefit from the final decimal here. Single-step errors do not prove reduced long-rollout error accumulation. e13e14e15 |
| Future-scene generation on nuScenes ADriver-I predicts four frames from four observed frames, conditioned on actions. Table 6 gives different contexts and horizons for its comparators. | FID 5.5; FVD 97.0, as printed in Table 6. Frame-wise FID and FVD (lower) | DriveGAN: 73.4/502.3 with 1F→12F. DriveDreamer: 52.6/452.0 with 1F+12B+12M→12F. All rows use control conditioning. Lower reported distances accompany longer visual context and a shorter prediction horizon, so the table cannot attribute the gap solely to architecture. The introduction prints FID 5.52; no extra precision is inferred from Table 6. e09e16e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The numerical representation is a consequential part of the control model. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each target column to compare how the action value is represented. Num2English translates the number into English words. Special Token bins integers and treats the target as a classification problem. Relative Diff predicts the change between adjacent frames. Absolute Number uses the current value itself and is the highlighted row. Section 4.1 adds that the chosen implementation keeps three decimal places and scales by 1000 before tokenization. Thus the last row should not be read as arbitrary free-form numerical prose. The table compares action representations; it does not measure the separate GPT3.5 motion-description interface used by diffusion. e05e07e13
What it supports. Absolute Number reports speed/steering errors of 0.072/0.091, compared with 0.081/0.096 for Relative Diff and 0.094/0.106 for Special Token. Num2English performs much worse at 2.094/0.536. Within this implementation, direct numerical targets are the strongest of the four tested encodings.
Where the evidence stops. Tokenization, quantization and target transformation change together across rows, so this is not a pure test of numeric reasoning. Bin specifications and uncertainty are not supplied. The decimal examples in Figure 2 also differ from the implementation's integer-scaling description.
Table 5. Supervising intermediate actions improves the final prediction in this comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The row names encode different uses of history and supervision. Temporal Fusion merges the multi-frame visual and action tokens. Single Round supervises the current action only. Multi Round adds intermediate action targets, which Section 3.1 identifies as the predictions at t−2 and t−1 alongside t. Lower values are better in both columns. Focus on Single Round versus Multi Round to understand the contribution of extra supervision, and keep Temporal Fusion as a separate way of combining history. Nothing in the table measures how long a generated driving sequence remains stable: these are control-prediction errors. e03e09e15
What it supports. Multi Round reports 0.072 m/s speed error and 0.091 rad steering error, versus Single Round's 0.078 and 0.094. The comparison supports the usefulness of intermediate targets in this setting. It gives a concrete training explanation for the dashed historical prediction arrows in Figure 1.
Where the evidence stops. Extra targets change the amount and placement of supervision. The table does not report a matched supervised-token budget or uncertainty. The authors' explanation in terms of reduced cumulative error is broader than the final-frame errors measured here.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify degraded generated frames under rapid control changes, inadequate deployment performance, separate training, and missing route information. Feeding imperfect generated observations back into the policy can propagate errors. e18
Figure 7 displays a finite example through +3.5 seconds and a continuation marker. It provides neither collision/success statistics nor an intervention study separating action influence from visual-history influence. FID/FVD also do not measure driving safety. e09e16e17
Figures 2–3 describe nearly stable motion despite increasing speeds and changing steering in the displayed history. This illustrates an unresolved calibration issue in the numeric-to-language bridge, rather than validating precise action-conditioned dynamics. e20
7.2 Questions for discussion
- How much of the control gain remains after matching pretraining exposure and capacity?
- Does action-conditioned generation preserve quantitative speed and steering, or only coarse motion categories?
- How does control error grow when generated observations replace measured images?
8. Reproducibility audit
8.1 Requirements and known gaps
MLLM stages use two epochs each, batch 16, 336×336 images, AdamW at 2×10⁻⁵ and eight A100 80GB GPUs. VDM pretraining/fine-tuning each run 40K steps, with batches 128/32, 32/16 A100 80GB GPUs and learning rates 4×10⁻⁴/3.2×10⁻⁴. e07e08
A faithful reconstruction requires the private pretraining corpus, dataset partitions, control synchronization and scaling conventions, GPT3.5 prompt-generation settings, and full diffusion configuration. These are not fully specified. Decimal examples also differ from the integer-scaling implementation description. NuScenes-only training would test a modified setting. e05e06e07e08e09
Proposed checks: equalize supervised-token budgets when testing intermediate action losses; separately hold history and diffusion noise fixed while intervening on motion descriptions. These would test supervision and action-conditioning mechanisms more directly than the published aggregate comparisons. e05e15e16e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does intermediate supervision help beyond adding more training targets?
Reader-proposed check, not performed: start from the same MLLM checkpoint, data split, three-pair context and absolute-number encoding. Compare current-action-only supervision with the paper's intermediate-plus-current supervision. Include a current-only control with matched total supervised action-token exposure, and report the extra optimization steps or repeated examples needed for that match. Evaluate speed and steering L1 and all four tolerance accuracies over multiple seeds. The mechanism receives stronger support if the multi-round advantage survives the exposure control; disappearance of the gap would support extra supervision quantity as an alternative explanation. Resolve the printed accuracy-index ambiguity before scoring. e03e07e09e13e15
Check 2: Does the motion-language bridge preserve action-dependent consequences?
Reader-proposed check, not performed: hold a four-frame history, VDM checkpoint, diffusion seed and sampling configuration fixed. Construct plausible alternative control histories representing opposite steering directions and acceleration versus deceleration, then record their GPT3.5 motion descriptions and resulting four-frame videos. Add unchanged-description and shuffled-description controls. Predefine blinded judgments of turn direction and motion change, and separately record malformed or visibly degraded generations. Reliable conditioning should make counterfactual descriptions produce corresponding motion changes more often than shuffled controls. Identical descriptions for materially different controls would locate information loss in the bridge; identical videos despite distinct prompts would implicate VDM conditioning. This tests sensitivity, not physical driving safety. e04e05e08e16e18e20
8.3 Reading coverage
Visual audit: Inspected the title, authors, affiliations and v1 stamp on page 1; Table 1 and introductory metric precision on page 2; architecture arrows and Section 3.1 on page 3; both prompt figures and component descriptions on page 4; baseline architecture, Table 2, training stages and hardware settings on page 5; control examples, metric equations, Table 3 and inference settings on page 6; future-video examples and Tables 4–5 on page 7; recurrent rollout, Table 6 and discussion on page 8; the discussion continuation on page 9. Every final crop was viewed at its output resolution. Tables 2 and 6 retain source notes needed to identify the private-data row and differing input/output protocols. All other crops exclude surrounding captions and body prose. Figures 2–6 and Tables 1 and 4 were also read on their full pages even where not selected as crops. Reference-only pages 10–12 were read in the complete text chunks; no appendix is present. No separate supplement was supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Works (2.1–2.4)
- 3. Method (3.1 Architecture, 3.2 Prompt Construction, 3.3 Model Training)
- 4. Experiments (4.1–4.5)
- 5. Conclusion and Discussion
- References (PDF pp. 9–12)
Outside the original text pass
- Identity/version scope: the inspected title page identifies arXiv:2311.13549v1, 22 November 2023. The exact title and all eight authors match the supplied catalog. No later revision or alternate edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction's figure-image omission was addressed by inspecting the supplied PDF: pages 1–9, Figures 1–7 and Tables 1–6. Reference-only pages 10–12 were read in text, not rendered.
- No appendix is present in the supplied 12-page PDF. No separate supplement was supplied.
- Code and external resources were not inspected; no training, evaluation or reproduction experiment was run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, affiliation lines and arXiv margin stamp
The title is ADriver-I: A General World Model for Autonomous Driving. Authors, in order: Fan Jia; Weixin Mao; Yingfei Liu; Yucheng Zhao; Yuqing Wen; Chi Zhang; Xiangyu Zhang; Tiancai Wang. Affiliations: MEGVII Technology; Waseda University; University of Science and Technology of China; Mach Drive. The stamp identifies arXiv:2311.13549v1 [cs.CV], 22 Nov 2023.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1; p. 2, Table 1, ADriver-I row
The proposed system combines control prediction and future-scene generation from interleaved vision-action pairs without bounding-box or HD-map inputs. The motivation contrasts this with a perception–prediction–planning–control pipeline.
Go to primary source ↓e03PDF p. 3, Figure 1 and caption; Section 3.1, Overall Pipeline
Three historical image/action pairs and the current image condition the system. Training supervises predicted actions at t−2, t−1 and t. The current predicted action conditions four future frames. At inference, the first generated future frame can become the next current observation. Dashed historical output arrows denote training-only predictions; green arrows denote recurrent inference.
Go to primary source ↓e04PDF p. 4, Section 3.1, Multi-modal Large Language Model and Video Diffusion Model
The MLLM uses Vicuna-7B-1.5, CLIP-ViT-Large and a two-layer MLP visual adapter pretrained by LLaVA-7B-1.5. VDM builds on Stable Diffusion 2.1 with temporal modules, reference frames concatenated to diffusion input, and retained text conditioning.
Go to primary source ↓e05PDF pp. 4–5, Section 3.2, Prompt Construction; p. 4, Figures 2–3
Front-view images become visual tokens and numerical actions become language tokens. System prompts organize an interleaved conversation. GPT3.5 converts consecutive control signals into motion descriptions for VDM; the text defines negative steering as right and positive as left. Figures 2–3 display decimal-valued control examples.
Go to primary source ↓e06PDF p. 5, Section 3.3, Model Training
MLLM and VDM are trained separately and combined during inference. MLLM pretraining uses nearly 1.4M private highway vision-action pairs, freezes the LLM, and updates the visual encoder and adapter. SFT freezes the visual encoder and updates the remainder, separately on nuScenes and private data. VDM is pretrained on private data and fine-tuned on about 23K nuScenes video samples.
Go to primary source ↓e07PDF p. 5, Section 4.1, Multi-modal Large Language Model
MLLM pretraining and SFT each use two epochs, batch 16, 336×336 images, three history pairs plus the current image, and cross-entropy supervision. Numbers use three decimal places and multiplication by 1000 to form integers. AdamW learning rate is 2×10⁻⁵; hardware is eight A100 80GB GPUs.
Go to primary source ↓e08PDF pp. 5–6, Section 4.1, Video Diffusion Model
VDM pretraining uses 40K steps, batch 128, 32 A100 80GB GPUs and learning rate 4×10⁻⁴. Fine-tuning uses 40K steps, batch 32, 16 A100 80GB GPUs and learning rate 3.2×10⁻⁴. Both use eight-frame videos at 256×512. Inference uses DDIM with 50 sampling steps.
Go to primary source ↓e09PDF p. 6, Section 4.2, Equations (1)–(2) and metric definitions
Control is measured by L1 error and accuracy within thresholds 0.01, 0.03, 0.05 and 0.07. Equation (1) accepts absolute errors less than or equal to the threshold. Equation (2) prints a sum from i=0 to N divided by N, while N is defined as the number of validation samples. Frame-wise FID and FVD evaluate generation.
Go to primary source ↓e10PDF p. 5, Figure 4 and caption; p. 6, Section 4.3, baseline descriptions
The action-only baseline uses three fully connected layers on three historical actions. CNN and ViT baselines add globally pooled visual features concatenated with action features; the latter replaces the CNN with ViT-B.
Go to primary source ↓e11PDF p. 5, Table 2, nuScenes rows and all metric columns; p. 6, Section 4.3
On nuScenes, ADriver-I speed/steering L1 errors are 0.072 m/s and 0.091 rad; MLP-only reports 0.122/0.101, CNN-based 0.106/0.095, and ViT-based 0.103/0.092. Steering A0.01 is 0.411 for ADriver-I and 0.299 for ViT-based. Speed A0.01 is 0.237 versus 0.200. The authors suggest large steering-variance cases may dominate mean steering error.
Go to primary source ↓e12PDF p. 5, Table 2, ADriver-I dagger row and caption; pp. 6–7, Section 4.3, cross-dataset comparison
The private-data result is 0.035 m/s speed L1 and 0.015 rad steering L1. The dagger identifies a different dataset. The authors attribute the lower errors to highway scenes with smaller action variance and larger SFT data scale, quoted as 1.4M versus 23K.
Go to primary source ↓e13PDF p. 6, Table 3 and caption; p. 7, Section 4.3, Ablation Study
Speed/steering errors for Num2English, Special Token, Relative Diff and Absolute Number are respectively 2.094/0.536, 0.094/0.106, 0.081/0.096 and 0.072/0.091. The caption defines English number expressions, integer bins and adjacent-frame differences.
Go to primary source ↓e14PDF p. 7, Table 4 and caption; Section 4.3, decimal-place ablation
Zero, one, two and three decimal places give speed/steering errors of 0.212/0.099, 0.094/0.093, 0.073/0.091 and 0.072/0.091. Two decimals almost match three in this experiment.
Go to primary source ↓e15PDF p. 7, Table 5 and caption; Section 4.3, final ablation paragraph; p. 3, Section 3.1
Temporal Fusion, Single Round and Multi Round give speed/steering errors of 0.078/0.092, 0.078/0.094 and 0.072/0.091. Temporal fusion merges multi-frame visual/action tokens. Single-round supervises the current frame; multi-round adds intermediate action supervision. Section 3.1 specifies predicted training actions at t−2, t−1 and t.
Go to primary source ↓e16PDF p. 8, Table 6 and caption; pp. 7–8, Section 4.4 and Figure 6
On nuScenes, ADriver-I uses 4F→4F and reports FID 5.5/FVD 97.0. DriveGAN uses 1F→12F with 73.4/502.3; DriveDreamer uses 1F+12B+12M→12F with 52.6/452.0. All rows are action-conditioned; F denotes frames, B boxes and M HD maps. Figure 6 shows four historical frames followed by predictions at +0.5, +1, +1.5 and +2 seconds.
Go to primary source ↓e17PDF p. 8, Figure 7 and caption; Section 4.5, Joint Control&Generation; p. 3, Section 3.1
The recurrent demonstration alternates MLLM control predictions and VDM scenes. Figure 7 labels frames through +3.5 seconds and then shows 'To be continued'; blue bars denote speed and orange bars steering. Section 4.5 says the demonstration is initialized with three historical pairs and describes all displayed frames as generated, whereas the general pipeline also requires a current image.
Go to primary source ↓e18PDF pp. 8–9, Section 5, Discussion, limitations (1)–(4)
The authors warn that rapid control changes can produce low-quality frames that disturb subsequent action prediction; performance is insufficient for deployment. Separate training prevents end-to-end optimization, and missing routing information limits long-distance driving.
Go to primary source ↓e19PDF p. 2, Section 1, final paragraph; p. 8, Table 6, ADriver-I row
The introduction reports FID 5.52 and FVD 97; the experiment table reports FID 5.5 and FVD 97.0. The report retains the table's precision and discloses the difference.
Go to primary source ↓e20PDF p. 4, Figures 2–3, numeric action histories and generated descriptions
The illustrative history has speeds 2.948, 3.217, 3.424 and 3.63 and changing steering values. The accompanying generated language describes stable motion and little or no steering change. These examples do not establish calibrated correspondence between numerical controls and motion descriptions.
Go to primary source ↓8.5 Primary sources
ADriver-I: A General World Model for Autonomous Driving ↗
PDF · 7,909 extracted words
Source fingerprint
d830a893408eeb9757fe03efbfe55321c8e902ea2ef74f4de863a2673ec6a7dd