Doe-1: Closed-Loop Autonomous Driving with Large World Model
1. Paper overview
In one sentence: Doe-1 learns images, descriptions and ego motion in one autoregressive stream, gaining several driving capabilities while leaving the safety and efficiency of driving with live feedback unestablished. e02e03e06e10e11e13e16
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that fixed intermediate scene representations discard potentially relevant information, prediction without ego-action conditioning misses interactions, and long-horizon plans contain work discarded at the next observation. Their proposed alternative learns observation–description–action evolution directly. These are motivations for scalability and interaction modeling, not experimentally established scaling laws or efficiency gains. e02 |
| Core mechanism | |
| A key reported result | End-to-end motion planning (standard Table 3 metric): At 1/2/3 s: L2 0.50/1.18/2.11 m, average 1.26 m; collision 0.04/0.37/1.19%, average 0.53%. L2 trajectory error (m) and collision rate (%), lower is better. nuScenes validation scenes; five-frame/two-second history; front camera and QA auxiliary supervision; upper, unmarked Table 3 block. VAD-Base averages 1.22 m/0.53%; GenAD averages 0.91 m/0.43%, with surround-camera inputs and different auxiliary supervision. Doe-1 is competitive but does not lead overall. Its 1 s collision rate ties VAD-Base. These dataset trajectory metrics do not establish executed closed-loop safety. e07e08e13 |
| Reading caution | Source description The authors restrict Doe-1 to front-view images because multi-view inputs are inefficient, while acknowledging that surrounding views matter for safety. e16 |
Core contributions
Figure 5. Separate tokenizers feed a shared predictor of the driving sequence. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the three data types on the left, then follow the reorganization arrow into the right-hand sequence. Purple O tokens represent observations, yellow D tokens descriptions and teal A tokens actions. The lower row enters one next-token prediction block; the upper row depicts the shifted targets. Repeated boxes illustrate token runs within modalities. The time arrow orders the image, description, action and following image. Cross-check this ordering against Equation (5): the next image depends on earlier actions, its description adds that image to the conditioning context, and its action adds the description. The model retains access to preceding observations through attention. e03e04e05e06
What it supports. The architectural basis for joint world-and-action modeling is the shared autoregressive predictor. The image tokenizer does not independently predict the future, and the action tokenizer is not a separately trained policy. Description generation also lies on the inference path used for planning, so it is more than an auxiliary output attached only during training.
Where the evidence stops. The caption calls action data the next-frame position, while Section 3.2 defines multi-motion displacement-and-yaw actions. Equation (7) then retains the first motion for subsequent observations. The figure omits this masking detail, and Equation (7)'s horizon indexing is internally inconsistent.
2. Motivation
2.1 The problem and the proposed response
The authors argue that fixed intermediate scene representations discard potentially relevant information, prediction without ego-action conditioning misses interactions, and long-horizon plans contain work discarded at the next observation. Their proposed alternative learns observation–description–action evolution directly. These are motivations for scalability and interaction modeling, not experimentally established scaling laws or efficiency gains. e02
2.2 What this reading follows
A driving model can describe a road, propose a trajectory or imagine a future image, but connecting those abilities requires deciding what each prediction conditions on. Doe-1 makes that connection explicit: an observation precedes a description, the description precedes an action, and the action precedes the next observation. One transformer learns this sequence, while prompts choose which parts are supplied and which are generated. Read the architecture first, then the separate perception, video and planning results. The central distinction is between a loop inside a generative model and a vehicle responding to an independently evolving environment. This paper's quantitative evidence primarily assesses the component tasks. e02e03e06e10e11e13e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The architecture supports One Model × Joint prediction: a single shared transformer generates future observation and action tokens, together with descriptions, within one autoregressive distribution. Separate tokenizers do not make separate dynamics and policy predictors. Joint means sequentially factorized multimodal generation, not simultaneous emission or inverse dynamics; descriptions also participate during planning inference. e03e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn driving supervision into one conditional sequence
Read Doe-1 as a model of a serialized record. An image is converted into discrete observation tokens, scene language becomes description tokens, and ego motion becomes action tokens. Figure 5 supplies the shared predictor; Equation (5) supplies its conditioning order. A description can use the image that precedes it, and an action can use both the description and earlier images. The next image can then depend on the action. Figure 6 exposes different portions of this learned sequence through prompts: observations invite descriptions, observations plus motion history invite planned actions, and supplied actions invite future images. The paper claims these capabilities without additional task-specific fine-tuning after shared training. The world-model and policy roles therefore arise from where generation continues within the same model, rather than from passing a latent prediction to a separately specified controller. e03e06
5.2 Make numerical motion usable, then hide the unexecuted tail
Descriptions may contain distances, while actions contain displacement and yaw. Doe-1 treats such numbers differently from ordinary text: distances are quantized at a stated resolution and assigned fixed sinusoidal embeddings so nearby numerical values remain nearby in the embedding space. Motion uses a similar encoding with adjusted scales. A second design choice governs temporal reuse. The model generates a trajectory, but only its first motion should condition the next predicted observation; the unused tail is replaced with mask tokens. This separates the trajectory being evaluated from the motion advancing the simulated scene. Preserve the source's ambiguity when implementing it: the prose describes p motions and p−1 masks, whereas Equation (7) writes p+1 motions and p masks. Table 4 supports masking as a recipe but does not determine which indexing convention an implementation used. e04e05e09e15
5.3 Ask which part of the loop each experiment actually tests
The three quantitative task families test different links. VQA evaluates generated language against reference answers and counterfactual keywords. Video evaluation assesses image-distribution quality using FID, while Figure 8 supplies selected qualitative evidence of conditioning. Planning compares trajectories with dataset references and reports collision metrics under two aggregations. Reader analysis: combining successful component tests does not automatically validate the composed loop. Description errors can affect actions; action errors can affect generated images; later decisions may then rely on those images. Section 4.4 illustrates that composition but provides no numerical live-feedback driving benchmark. Table 4 helps by testing description removal and plan masking, yet it does not measure the entire chain under an independently reacting environment. A faithful conclusion is therefore multi-task generative capability with useful planning evidence, while robustness of the composed driving loop remains open. e03e06e08e10e11e12e13e14e15e16
5.4 Training and inference
During training
The experimental split uses 700 nuScenes training scenes and 150 validation scenes for testing, with 2 Hz keyframes. OmniDrive-nuScenes supplies GPT-4-generated QA supervision. Despite the dataset section mentioning six cameras, the experiments and limitations explicitly specify front-camera-only Doe-1. e07
Initialize from Lumina-mGPT 7B, fine-tune five epochs on BDD100k for driving-image generation, then train 16 epochs with eight A100 GPUs and total batch size 24. Images are 672×384. AdamW uses cosine scheduling, initial learning rate 10⁻⁵ and weight decay 0.1; action-token loss is weighted fivefold and Z-loss by 10⁻⁵. Action resolution is 0.02 m, with displacement/yaw scales 10,000/1,000. Only the numerical embeddings are explicitly described as non-learnable; a complete module-freezing specification is absent. e04e09
During inference
VQA generates descriptions or answers from observations. Planning uses observations and motion history to generate a description followed by future motions. Video generation starts with the image at time zero and supplied actions, then generates later images iteratively. Combining these modes produces an internally simulated observation–description–action loop. e06
The supplied work does not specify candidate-trajectory search, reward-based rollout selection or a low-level vehicle controller. Generated images are available for simulation, but the planning prompt itself does not require scoring imagined futures before generating actions. e03e05e06
5.5 Implementation flow
- Serialize the driving sequence
Reorganize sensor images, perception text and trajectories into temporal observation–description–action blocks. A Chameleon-style multimodal autoregressive transformer models their concatenation. Equation (5) conditions the next observation on preceding blocks, its description on that observation, and its action on both; previous image tokens remain accessible through attention. e03
- Tokenize vision and numerical language
The pretrained Lumina-mGPT image tokenizer is described as producing 1,024 tokens from an 8,192-entry codebook, plus image-shape tokens. Text uses Chameleon BPE. Numerical distances in descriptions are rounded at 0.02 m resolution into 4,000 tokens with non-learnable sinusoidal embeddings, intended to preserve proximity between nearby values. e04
- Represent motion and restrict its influence
Each motion contains two horizontal displacements and yaw. An action concatenates future motions, uses similar positional encoding and has boundary tokens. Before subsequent observation generation, the prose says to retain the first motion and mask the other p−1 motions. Equation (7), however, enumerates p+1 motions and p masks; its horizon indexing is inconsistent with that prose. e05
6. Experiments & results
Doe-1 puts front-camera images, free-form scene descriptions and discretized ego motion into one autoregressive token stream. Different prompts expose perception, action-conditioned image prediction and trajectory planning. Its nuScenes experiments support this shared capability, while the claimed closed loop is demonstrated through model-generated scene evolution rather than a quantitative driving evaluation with live environment feedback.
6.1 Read the original evidence
Table 1. Caption quality and counterfactual recall tell different stories. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the counterfactual and caption groups separately. Counterfactual AP and AR are percentages computed from keywords in answers about a proposed trajectory; they are not measured collision frequencies. METEOR, CIDEr and ROUGE compare generated language with reference answers. Every arrow points upward because larger scores are preferred. The asterisk beside Doe-1 matters: it uses only the front camera, whereas Section 4.5 describes OmniDrive's surrounding-camera inputs. Compare each column, rather than counting bold entries as a universal win. In particular, the recall column checks a different failure tendency from the precision column, even though both concern the same counterfactual questions. e07e08e10
What it supports. Doe-1 reaches 54.5% AP against OmniDrive-3D's 52.3%, but its 54.0% recall falls below 59.6%. Its CIDEr and ROUGE scores are higher, at 72.8 and 35.6, while METEOR is slightly lower at 37.6 versus 38.0. The evidence supports competitive language-based perception, with a visible precision–recall tradeoff.
Where the evidence stops. OmniDrive-nuScenes uses GPT-4-generated QA supervision. Matching those answers or their keywords does not establish grounded causal reasoning or safe action execution. Camera coverage also differs, and the table does not report uncertainty or an input-matched architecture ablation.
Table 2. A shared autoregressive model approaches several diffusion baselines on FID. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each method column: model family, output resolution, then FID. Lower FID is preferred, so the bold 6.9 belongs to Vista rather than Doe-1. The source evaluates Doe-1 with accurate BEV displacements as conditions. Its tabulated resolution is 384×672; the implementation paragraph writes the same image dimensions in the opposite order. Compare resolution alongside FID because the columns do not establish matched image-generation conditions. Also preserve the citation numbers: GenAD [90] in this video table and GenAD [100] in the planning table are differently cited methods sharing an abbreviation, so their results should not be fused into one baseline profile. e08e09e11e13
What it supports. Doe-1 reports FID 15.9, close to Drive-WM's 15.8 and the video-generation GenAD's 15.4. Vista remains substantially better at 6.9. The table demonstrates that the shared model can generate driving images at a competitive distributional quality level among some comparators; it does not show that autoregression is the best generator.
Where the evidence stops. FID does not itself measure whether changing an action changes the future correctly. Training data, compute and evaluation details are not matched by this table, and resolutions vary. An architectural advantage cannot be isolated from these reported scores.
Figure 8. Selected rollouts show qualitative changes consistent with different motion conditions. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. For each row, begin inside the black dashed Inputs box and follow the arrows through the Predictions box. The rows illustrate going straight, turning left, turning right, overtaking and following. Compare road geometry, viewpoint change and nearby vehicles across the row, not merely the appearance of individual images. Section 3.3 says that video generation supplies the initial image and actions, then iteratively generates later observations. Section 4.6 identifies the conditioning as accurate BEV displacements. Thus the maneuver labels summarize examples; they do not establish that these sequences were produced using only a natural-language command. No action coordinates or timestamps are printed in this figure. e03e06e11e12e16
What it supports. The sequences offer visual support for the authors' claim that the generated viewpoint evolution can follow an action condition. They also show why image generation belongs in the model's loop: an action is followed by another observation that can supply context for subsequent description and action generation.
Where the evidence stops. These are selected qualitative examples without paired counterfactual ground truth or error bars. Plausible viewpoint movement does not verify reactive behavior of other traffic participants. The figure cannot establish long-horizon stability or collision avoidance in an independently evolving environment.
Table 3. Keep the two evaluation aggregations separate when judging planning quality. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with Input and Auxiliary Supervision before reading the numerical columns. Doe-1 uses front-camera input and QA auxiliary supervision; many camera baselines have surrounding views and map, box or motion labels. For the upper block, inspect errors at each horizon and then the shaded averages. Now move to the lighter lower block: the dagger means the VAD-style aggregation, which Section 4.7 describes as averaging across preceding frames at each time step. This is a different metric calculation, not another Doe-1 checkpoint. The source caption explains the star and dagger but leaves OmniDrive's double-dagger undefined, so avoid inferring its meaning. e07e08e13e14e16
What it supports. In the upper block, Doe-1 averages 1.26 m L2 and 0.53% collision, versus VAD-Base's 1.22 m and 0.53%. GenAD [100] is better on these averages at 0.91 m and 0.43%. Doe-1's 0.04% one-second collision rate is encouraging within this protocol, but its three-second value rises to 1.19%.
Where the evidence stops. The lower Doe-1† averages of 0.70 m and 0.21% cannot be compared directly with the upper row. These are dataset-based trajectory evaluations with differing inputs and supervision across methods; the paper supplies no quantitative live-feedback driving trial establishing deployment safety.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| End-to-end motion planning (standard Table 3 metric) nuScenes validation scenes; five-frame/two-second history; front camera and QA auxiliary supervision; upper, unmarked Table 3 block. | At 1/2/3 s: L2 0.50/1.18/2.11 m, average 1.26 m; collision 0.04/0.37/1.19%, average 0.53%. L2 trajectory error (m) and collision rate (%), lower is better | VAD-Base averages 1.22 m/0.53%; GenAD averages 0.91 m/0.43%, with surround-camera inputs and different auxiliary supervision. Doe-1 is competitive but does not lead overall. Its 1 s collision rate ties VAD-Base. These dataset trajectory metrics do not establish executed closed-loop safety. e07e08e13 |
| End-to-end motion planning (VAD-style cumulative metric) Same dataset evaluation; dagger-marked Table 3 block averages performance over preceding frames at each horizon. | Doe-1†: L2 0.37/0.67/1.07 m, average 0.70 m; collision 0.02/0.14/0.47%, average 0.21%. VAD-style L2 (m) and collision rate (%) | VAD-Base† averages 0.72 m/0.22%; GenAD† averages 0.52 m/0.19%. Keep this aggregation separate from the upper block. The two Doe-1 rows are not evidence of a model improvement. e14 |
| Visual question-answering and counterfactual reasoning OmniDrive-nuScenes evaluation; Doe-1 uses the front camera, OmniDrive variants use surrounding cameras. | Doe-1: 54.5/54.0 AP/AR; 37.6 METEOR, 72.8 CIDEr, 35.6 ROUGE. Counterfactual AP/AR (%); caption METEOR, CIDEr and ROUGE, higher is better | OmniDrive-3D: 52.3/59.6 AP/AR; 38.0 METEOR, 68.6 CIDEr, 32.6 ROUGE. Higher precision and some caption scores coexist with lower recall. Keyword-based counterfactual scoring and language similarity do not directly measure safe driving. e08e10 |
| Action-conditioned video generation nuScenes; accurate BEV displacements provide conditioning; initial image followed by autoregressive future generation. | 15.9 at the table's 384×672 resolution. FID, lower is better | Drive-WM: 15.8 at 192×384; GenAD: 15.4 at 256×448; Vista: 6.9 at 576×1024. Doe-1 approaches two diffusion baselines but trails Vista. Different resolutions and unspecified matched training/evaluation details limit architectural attribution. FID alone does not validate conditional dynamics. e06e08e11e12 |
| Planning-strategy ablation Table 4 uses the standard planning metrics. Its Description row removes intervening description prompts; its Mask row disables masking during training and evaluation. | Full model: 1.26/0.53; Description: 1.79/1.07; Mask: 2.83/2.72. Average L2 (m) / collision rate (%) | Both removal variants worsen both averages relative to the full model. Supports the complete description-and-masking recipe. The masking comparison changes training and evaluation together, so it does not isolate inference-time error accumulation. No uncertainty estimates accompany these rows. e15 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Removing descriptions or exposing old future plans degrades planning. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the retained caption before the row labels. Description means that no description prompt intervenes between observations and actions; it does not mean that descriptions were added. Mask means that preceding plans are left unmasked during both training and evaluation; it does not mean that masking is enabled. Full Model includes both mechanisms. Compare each removal row with Full Model at the same horizon and in the same metric. Connect the masking row to Section 3.2: later observations should see the first motion of a prior action and mask tokens for its unused future tail. The table evaluates that complete recipe, rather than an attention-mask visualization. e05e15
What it supports. Average L2/collision worsens from 1.26 m/0.53% to 1.79 m/1.07% without intervening descriptions, and to 2.83 m/2.72% without masking. The direction is consistent across the shown horizons. This is the clearest evidence that the chosen sequence contents matter for planning, beyond simply sharing a transformer.
Where the evidence stops. Disabling masking at training and evaluation together does not distinguish learning effects from rollout-time contamination. No repeated-run uncertainty is provided. The p versus p+1 inconsistency in Equation (7) must also be resolved before claiming an exact implementation match.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors restrict Doe-1 to front-view images because multi-view inputs are inefficient, while acknowledging that surrounding views matter for safety. e16
The closed-loop evidence is qualitative and internally generated; no quantitative reactive-environment driving benchmark, physical deployment, runtime comparison or statistical uncertainty is reported. Figure 8 illustrates selected action-conditioned sequences, not calibrated predictions of other agents' responses. e06e11e12e13e15e16
The source contains editorial inconsistencies: Figure 1's caption swaps the planning/prediction names relative to its arrows, Equation (4) and Section 4.4; Section 4.2 calls the evaluated model GaussianAD although the result rows identify Doe-1. These slips are preserved as discrepancies, not treated as another model or edition. e17
7.2 Questions for discussion
- Does observation generation improve planning under matched supervision, or mainly add another capability?
- Does masking help when both generated descriptions and generated observations drift over long rollouts?
- How much of the planning gap changes under matched front-camera inputs and auxiliary supervision?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful attempt needs the Lumina-mGPT initialization and image tokenizer, Chameleon text tokenizer, BDD100k adaptation, nuScenes histories and OmniDrive-nuScenes QA, with the reported weighting and compute configuration. The paper alone leaves sequence packing, decoding settings, full freezing policy, numerical token ranges/bias and precise yaw discretization unresolved. e03e04e05e07e09
Proposed first check: separate training-time and evaluation-time masking in a controlled factorial ablation, preserving the split, initialization, history, description prompts and metric implementation. Resolve the p versus p+1 indexing first; report per-horizon errors and collisions rather than combining the two Table 3 protocols. This is a proposed experiment, not a reproduction result. e05e08e14e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate training and evaluation masking effects
Reader-proposed experiment, not performed: construct a two-by-two comparison with masking enabled or disabled independently during training and evaluation. Hold the initialization, training split, compute, description prompts, history and action resolution fixed, and first document a consistent interpretation of the p-motion horizon. Use repeated seeds and report the standard Table 3 per-horizon L2 and collision metrics, with VAD-style metrics reported separately. If evaluation-time masking reduces deterioration within each fixed trained model, that supports an immediate conditioning effect. If the benefit appears only after masked training, the published ablation's gain is more plausibly tied to the learned sequence distribution than to a standalone inference intervention. e05e07e08e09e14e15
Check 2: Test action sensitivity separately from image quality
Reader-proposed experiment, not performed: for the same held-out initial images, compare continuations conditioned on logged actions with continuations conditioned on actions shuffled among clips with similar initial motion. Hold rollout horizon and decoding seeds fixed. Report FID separately from measured ego-viewpoint change, using a fixed annotation or motion-estimation procedure and explicit uncertainty. The logged-action branch can be compared with the recorded continuation; the shuffled branch has no observed counterfactual ground truth and tests sensitivity only. If outputs barely change when actions change, even while FID remains good, that weakens the claim that the model learned useful action-conditioned evolution. A responsive generator would still require a separate test of reactive-agent accuracy. e06e07e08e11e12
8.3 Reading coverage
Visual audit: Visually inspected the title, authors, affiliation and v1/date on page 1; motivation and paradigm diagrams on pages 2–4; architecture, tokenization and equations on page 5; action masking, task prompts and dataset description on page 6; QA table, qualitative example, evaluation and training configuration on page 7; video table, generated sequences and closed-loop discussion on page 8; both planning protocols, ablation caption and limitations on page 9. All eight figures and all four tables were inspected on these pages. All six final original crops were separately viewed; the narrow ablation was rendered at 300 DPI, the others at 200 DPI. Its caption was retained because it defines the removal conditions. Figure 5 was cross-checked against Equations (4)–(7) and Figure 6; Figure 8 arrows were checked against Section 3.3. Internal source discrepancies are disclosed in the report. Pages 10–13 contain references and were read as text only. No appendix or separate supplement was supplied; external material and code remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Proposed Approach
- 3.1. Closed-Loop Autonomous Driving
- 3.2. Doe-1: A Large Driving World Model
- 3.3. Applications of Doe-1
- 4. Experiments
- 4.1. Dataset
- 4.2. Task Descriptions
- 4.3. Implementation Details
- 4.4. Close-Loop Autonomous Driving
- 4.5. Visual Question-Answering
- 4.6. Action-Conditioned Video Generation
- 4.7. End-to-End Motion Planning
- 4.8. Analysis
- 5. Conclusion, including Limitations
- References, entries 1–101
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by visually inspecting PDF pages 1–9 and every final crop.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks, covering all 13 PDF pages, were read. No appendix appears in this PDF, and no separate supplement was supplied. Reference pages 10–13 were read as text, not visually inspected.
- Identity and edition: the title and all six authors match the catalog. The inspected title page identifies arXiv:2412.09627v1 [cs.CV], 12 December 2024. No other revision was supplied, so differences from other editions cannot be assessed.
- Code, project links and external resources were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block, contribution footnote and arXiv margin
The title is Doe-1: Closed-Loop Autonomous Driving with Large World Model. Authors are Wenzhao Zheng, Zetian Xia, Yuanhui Huang, Sicheng Zuo, Jie Zhou and Jiwen Lu, affiliated with Department of Automation, Tsinghua University, China. Zheng and Xia contributed equally; Zheng is project leader. The margin states arXiv:2412.09627v1 [cs.CV], 12 Dec 2024.
Go to primary source ↓e02PDF p. 2, Section 1; p. 4, Section 3.1, Figures 3–4 and Equation (4)
The authors motivate avoiding restrictive intermediate representations, conditioning scene evolution on ego actions and retaining immediate planning. Equation (4) orders observation, description, action and next observation through Doe-1.
Go to primary source ↓e03PDF pp. 4–5, Section 3.2 Overview, Figure 5 and Equation (5)
A Chameleon-style unified autoregressive model consumes concatenated observation, description and action tokens. Figure 5 depicts a single next-token predictor. Equation (5) conditions later observation, description and action blocks sequentially on earlier tokens.
Go to primary source ↓e04PDF p. 5, Section 3.2 Observation Tokenizer and Description Tokenizer; Equation (6)
Lumina-mGPT supplies an image tokenizer described as 1,024 image tokens with an 8,192-entry codebook. Chameleon BPE tokenizes text. Numerical values are rounded to 0.02 m, mapped into 4,000 tokens and assigned non-learnable sinusoidal features. Equation (6) gives T(x)=round(x/r)+b, with resolution r and codebook-dependent bias b.
Go to primary source ↓e05PDF p. 5, Section 3.2 Action Tokenizer; p. 6, Generative Architecture and Equation (7)
Motion z=(dx,dy,theta) comprises two displacements and yaw; action a is described as p motions with boundary tokens. The text retains the first motion and masks p−1 others for subsequent observations. Printed Equation (7) instead lists z_t through z_(t+p) and m_1 through m_p, leaving an indexing inconsistency.
Go to primary source ↓e06PDF p. 6, Section 3.3 and Figure 6
Prompt settings select VQA, motion planning or action-conditioned video generation. Planning uses observations and motion history and generates descriptions and motions. Video generation receives the initial image and supplied actions, then iteratively generates future images. The section explicitly describes simulation without real-world interaction feedback.
Go to primary source ↓e07PDF p. 6, Sections 4.1–4.2; p. 8, Section 4.5; p. 9, Section 4.7 and Limitations
Experiments use 700 training and 150 validation nuScenes scenes with 2 Hz keyframes. OmniDrive-nuScenes adds GPT-4-generated QA. Although the dataset paragraph refers to six cameras, the experimental comparisons and limitations consistently identify Doe-1 as front-camera-only.
Go to primary source ↓e08PDF p. 7, Section 4.2 Task Descriptions
VQA uses METEOR, ROUGE and CIDEr; counterfactual AP/AR uses extracted safety, collision, red-light and drivable-area keywords. Video generation reports FID. Planning uses five history frames spanning two seconds and reports L2 and collision rates at one, two and three seconds.
Go to primary source ↓e09PDF p. 7, Section 4.3 Implementation Details
Lumina-mGPT 7B is adapted for five epochs on BDD100k. The configuration specifies 672×384 images, 0.02 m action resolution, displacement/yaw scales 10,000/1,000, AdamW with cosine scheduling, 10⁻⁵ initial learning rate, 0.1 weight decay, fivefold action-token loss, 10⁻⁵ Z-loss weight, 16 training epochs, eight A100 GPUs and batch size 24.
Go to primary source ↓e10PDF p. 7, Table 1, Doe-1 and OmniDrive-3D rows; p. 8, Section 4.5
Doe-1 reports AP/AR 54.5/54.0%, METEOR 37.6, CIDEr 72.8 and ROUGE 35.6. OmniDrive-3D reports 52.3/59.6%, 38.0, 68.6 and 32.6. The star means front-camera-only; the comparison text specifies surrounding-camera OmniDrive inputs.
Go to primary source ↓e11PDF p. 8, Table 2, Resolution and FID rows; Section 4.6
Doe-1 FID is 15.9 at 384×672, Drive-WM 15.8 at 192×384, GenAD 15.4 at 256×448 and Vista 6.9 at 576×1024. Doe-1 is autoregressive; those three comparators use diffusion. The section states accurate BEV actions provide conditioning.
Go to primary source ↓e12PDF p. 8, Figure 8 and Section 4.6 Visualizations
Figure 8 separates one input image from four prediction panels per row, labeled Go Straight, Turn Left, Turn Right, Overtake and Follow. Arrows run forward through the image sequences. The authors claim action and structural consistency from these selected visualizations; the figure supplies no quantitative interaction or control-error measure.
Go to primary source ↓e13PDF p. 9, Table 3 upper block, Doe-1, VAD-Base and GenAD rows; Section 4.7
Doe-1 L2 is 0.50/1.18/2.11 m (average 1.26) and collision rate 0.04/0.37/1.19% (average 0.53). VAD-Base averages 1.22 m/0.53% and also has 0.04% at one second. GenAD averages 0.91 m/0.43%. Input and supervision columns differ between Doe-1 and these baselines.
Go to primary source ↓e14PDF p. 9, Table 3 dagger-marked lower block and caption; Section 4.7 first paragraph
The lower block averages performance across previous frames at each time step following VAD. Doe-1† gives 0.37/0.67/1.07 m, average 0.70, and 0.02/0.14/0.47%, average 0.21. VAD-Base† averages 0.72/0.22 and GenAD† 0.52/0.19. The caption defines star and dagger but does not explain the double-dagger attached to OmniDrive.
Go to primary source ↓e15PDF p. 9, Table 4 caption and all rows; Section 4.8
Description means no intervening prompt between observations and actions; Mask means prior plans are not masked during training and evaluation. Average L2/collision is 1.79 m/1.07% for Description, 2.83 m/2.72% for Mask and 1.26 m/0.53% for Full Model. No uncertainty estimates are shown.
Go to primary source ↓e16PDF p. 6, Section 3.3; p. 8, Section 4.4 referring to Figure 1 on p. 1; p. 9, Section 5 Limitations
The source's closed-loop demonstration is the qualitative Figure 1 and the simulation described in Section 3.3. Section 4.4 supplies no closed-loop driving score. The stated limitation is inefficient multi-view processing, resulting in front-view-only inputs despite the safety relevance of surrounding information.
Go to primary source ↓e17PDF p. 1, Figure 1 arrows and caption; p. 4, Equation (4); p. 7, Section 4.2 End-to-End Motion Planning; p. 8, Section 4.4; p. 9, Table 3
Figure 1's caption lists perception, prediction, planning against observation→description, description→action, action→observation, swapping the latter two names relative to the arrows and Section 4.4. Section 4.2 says GaussianAD, while the paper title and evaluated rows say Doe-1. These are internal source discrepancies.
Go to primary source ↓8.5 Primary sources
Doe-1: Closed-Loop Autonomous Driving with Large World Model ↗
PDF · 9,428 extracted words
Source fingerprint
7582f1a747ab56671cefb9f97ccba6cbd1ca788e28eddf20e856da2c50ca9e4d