ACT-Bench: Towards Action Controllable World Models for Autonomous Driving
1. Paper overview
In one sentence: ACT-Bench measures action fidelity through a learned video-to-motion evaluator, revealing TERRA's higher aggregate controllability while leaving evaluator transfer and comparison fairness unresolved. e-purposee-metricse-estimatore-terra-coree-iece-alignmente-failures
| At a glance | What to know |
|---|---|
| Research problem | Source description Visually plausible driving video can ignore its requested maneuver. ACT-Bench makes action fidelity measurable through an annotated test set, a learned video-to-motion evaluator and a trajectory-conditioned baseline. Here, execution means motion inferred from synthetic frames; the experiments do not execute a driving policy in a vehicle. e-purposee-experiment-scope |
| Core mechanism | |
| A key reported result | Instruction-execution consistency of generated driving video: TERRA: 44.11% IEC, percent. ACT-Bench comparison; 2,286 videos generated per model, with different instruction frequencies. Exact scored-category reconciliation is unspecified. Vista: 30.72%; a calculated difference of 13.39 percentage points. TERRA has higher aggregate agreement, but most estimated actions still mismatch. Training data, architecture and conditioning frequency all differ; this is not an isolated frequency ablation. e-protocole-iece-terra-training |
| Reading caution | Reader analysis Benchmark and estimator labels differ: shifting categories are present in Table 2 but absent from Figure 6/Table 4, whereas stopped and speed subdivisions appear in the estimator. The mapping and scored denominator are unresolved, so the reported IEC should not be treated as verified coverage of every benchmark pair. e-datasete-estimator-datae-iece-alignment |
Core contributions
- Source description
ACT-Estimator jointly learns maneuver classification and trajectory regression. TERRA adds a frame-conditioned autoregressive video baseline trained with an additional trajectory-annotated dataset. e-estimatore-terra-coree-terra-training
Figure 11. The evaluator turns one video into two complementary motion descriptions. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrows downward from Generated Video. I3D first extracts features across space and time. Flattening, positional encoding and the two Transformer encoder layers provide a shared representation. The left branch pools that representation and applies linear layers with GELU to predict a high-level action, illustrated here by stopping. The right branch attends to the shared features while a GRU emits successive trajectory coordinates; recurrent arrows carry information between prediction steps. Section 3.3 identifies classification and trajectory regression as jointly trained tasks. At benchmark evaluation, both branches read the generated frames, and their outputs are compared with the supplied instruction. e-estimatore-estimator-datae-datasete-estimator-validatione-experiment-scope
What it supports. The architecture supports two distinct questions: whether the video depicts the requested maneuver, and how far its inferred motion deviates from the requested path. Sharing visual features can support both tasks, but the paper does not isolate the benefit of this sharing with a single-task ablation.
Where the evidence stops. The estimator's predicted classes exclude lateral shifts, although the benchmark contains shifting templates. Its real-video validation also does not establish accuracy on generated video. Neither issue is resolved by the two-head design.
2. Motivation
2.1 The problem and the proposed response
Visually plausible driving video can ignore its requested maneuver. ACT-Bench makes action fidelity measurable through an annotated test set, a learned video-to-motion evaluator and a trajectory-conditioned baseline. Here, execution means motion inferred from synthetic frames; the experiments do not execute a driving policy in a vehicle. e-purposee-experiment-scope
2.2 What this reading follows
A generated car journey can look convincing while missing the requested turn or traveling the wrong distance. ACT-Bench asks two complementary questions: does the inferred maneuver match the instruction, and does the inferred trajectory follow the intended path? Its evaluator reads generated video and returns both answers. The accompanying TERRA baseline interleaves trajectory instructions with image tokens, allowing instructions to change every frame. Read the evaluator architecture before the model comparison: every reported fidelity score passes through that learned instrument. Then compare the confusion matrices with geometric errors, because better turning behavior does not guarantee lower displacement error or physically correct interactions with other vehicles. e-purposee-metricse-estimatore-terra-coree-iece-alignmente-failures
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator category fits the central contribution: a dataset, metrics and learned evaluator for driving-video controllability. Architecture and prediction quadrant remain not applicable at benchmark level. TERRA separately predicts future observations conditioned on supplied actions; ACT-Estimator infers motion for scoring. Joint estimator training does not establish a unified future-and-action policy. e-purposee-estimatore-terra-core
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build the measuring instrument before trusting the benchmark
The benchmark provides an intended trajectory, but a generated video has no measured ego pose to compare with it. ACT-Estimator supplies that missing observation by predicting both a maneuver label and a path from the frames. Its supervision comes from real nuScenes video: ego poses define trajectories, and the sequential rules in Table 6 assign action labels. Joint classification and regression then make one visual representation serve both outputs. During evaluation, the instruction is compared with these predictions rather than directly with physical motion. Reader interpretation: the benchmark's validity therefore depends on the estimator transferring from real to synthetic imagery. The strong real-video validation is relevant evidence, but it cannot by itself calibrate that transfer. The mismatched benchmark and classifier label sets are a separate coverage problem. e-metricse-estimatore-estimator-datae-estimator-validatione-datasete-iec
Figure 3. TERRA inserts a trajectory instruction between successive blocks of image tokens. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the red-bordered context frames at left; the time arrow points upward through the displayed images. The image tokenizer converts each frame into discrete codes. Above the Transformer, the action embedder supplies trajectory embeddings that are interleaved with image-token blocks. Follow the rollout arrow to the longer sequence below and then the pale blue connections to the frame-wise decoder. Blue outlines mark predicted future frames, while red outlines mark context reconstructions. Appendix D.1 makes the timing precise: after predicting a frame's image tokens, the model appends that frame's trajectory input before predicting the next frame. Equation (5) places loss only on image codes. e-terra-coree-protocole-refiner
What it supports. TERRA's action interface accepts a trajectory at every frame, permitting more frequent instruction updates than the Vista setup used here. The actions are provided conditions rather than generated policy outputs. The rollout predicts what the camera could see under those conditions, which is subsequently evaluated by ACT-Estimator.
Where the evidence stops. Figure 3's caption says a video decoder is omitted, but Appendices D.2–E describe a separate SVD Video Refiner after frame decoding. This simplified drawing does not show that stage or establish its effect on fidelity.
5.2 Trace a supplied trajectory through TERRA's next frame
A trajectory instruction contains future positions and their timestamps relative to the vehicle at the current frame. TERRA embeds those vectors and inserts them after the current image-code block. The Transformer then predicts the next image's codes autoregressively, using earlier image codes and trajectory inputs; after that image is complete, another supplied trajectory is inserted. Training follows this ordering while applying prediction loss only to image tokens. It does not train the model to output a control policy. The frame-wise decoder converts predicted codes into coarse images, and the separately trained SVD refiner uses those images as conditioning. This explains why the main drawing is incomplete as an account of final video production. It also explains the benchmark's practical interface: new instructions can arrive every frame, although this experiment follows precomputed ideal trajectories rather than a closed-loop planner. e-terra-coree-refinere-protocole-dataset
5.3 Separate what the scores show from why the models differ
Begin with IEC: it tests equality between requested and estimated maneuver labels. Then examine ADE and FDE, which penalize position errors even when the label is correct. TERRA's higher aggregate IEC and lower reported average displacement errors support improved fidelity under the authors' comparison. The category rows and trajectory plots qualify that conclusion: Vista has lower curving ADE, and both models can miss the intended travel distance. Reader interpretation: those results motivate more than one diagnostic axis, not a single winner for every behavior. They also leave the cause of the aggregate difference unresolved. TERRA and Vista differ in architecture, training data and instruction-update frequency. Finally, the qualitative failures show that matching ego motion does not establish physically appropriate behavior for surrounding traffic or validate a generated collision. e-metricse-iece-alignmente-diagnostice-protocole-terra-traininge-failures
5.4 Training and inference
During training
Estimator supervision uses 44-frame windows with stride one and rule-labeled ego-pose trajectories. Table 5 lists 46,412 samples over eleven labels, but shifting classes are excluded from classification, leaving nine classes. A random 8,407-sample validation subset is withheld. e-estimator-datae-estimator-training
ACT-Estimator trains for 30,850 iterations on four H100 GPUs, effective batch 96, using AdamW and OneCycleLR with maximum learning rate 1.2e-4. Cross-entropy and smooth L1 losses are mixed by beta, whose numerical value is not specified. e-estimator-traininge-estimator
TERRA uses 25-frame, 10 Hz clips, 576 image tokens per frame and six trajectory points. Its randomly initialized Llama-based Transformer learns image-token prediction from OpenDV-YouTube, nuScenes and CoVLA; missing trajectories receive an empty placeholder. The tokenizer supplies precomputed codes, and the SVD refiner trains separately. e-terra-traininge-terra-coree-refiner
During inference
After context tokens, TERRA samples a complete frame, appends that frame's supplied trajectory and repeats. Only image tokens are predicted. The evaluation describes three context frames and 44 generated frames over two rounds; Vista receives one instruction per round, TERRA one per frame. Generated videos are then passed to ACT-Estimator for scoring. e-terra-coree-protocole-estimator
5.5 Implementation flow
- Construct compatible scene–instruction pairs
The benchmark provides ten-frame contexts from nuScenes validation and 36 CoVLA-derived trajectory variants across nine maneuvers. It retains 2,286 pairs after a 10 km/h initial-speed tolerance and visual filtering of interfering foreground interactions. Successive instructions use the ideal vehicle orientation, rather than feedback from an executed vehicle. e-datasete-templates
- Estimate motion from video
I3D extracts spatiotemporal features; positional encoding and two Transformer encoder layers refine them. A pooled MLP predicts the action class, while an attention-equipped GRU autoregressively regresses 2D trajectory points. This inverse motion estimate is an evaluation instrument, not a policy selecting the next driving command. e-estimatore-purpose
- Generate TERRA futures
A pretrained image tokenizer produces discrete codes. Image-token embeddings alternate with linearly embedded trajectory points containing future position and time. An autoregressive Transformer predicts subsequent image codes, and the tokenizer decoder reconstructs frames. A separately trained SVD refiner processes those coarse frames; it is absent from the simplified architecture drawing. e-terra-coree-refiner
6. Experiments & results
ACT-Bench tests whether generated driving videos follow supplied ego trajectories. Its learned evaluator estimates a maneuver label and a path, separating instruction agreement from geometric error. TERRA improves the reported aggregate scores over Vista, but unequal conditioning frequency and unresolved evaluation details limit causal interpretation of that comparison.
The supplied paper contains estimator validation, a two-model comparison and qualitative diagnostics, but no controlled component ablation. Figure 7 therefore serves as a trajectory diagnostic in the ablation section. Neither the benefit of joint estimator training, the added CoVLA data, the refiner, nor instruction frequency is isolated. Still-frame causal-misalignment examples provide no event rate, and the referenced supplementary movies were not supplied. e-experiment-scopee-failurese-protocole-terra-training
6.1 Read the original evidence
Table 3. Validate the measuring instrument before interpreting generated-video scores. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a real-video trajectory-estimation test. Ours denotes ACT-Estimator; DS denotes DROID-SLAM adapted with Metric3D monocular depth. Each metric has its own pair of columns, and the downward arrows mean smaller errors are better. ADE averages displacement across trajectory points, whereas FDE compares their final positions. HS and LS denote high-speed and low-speed constant motion. The bottom row is the paper's reported Average, whose weighting is not specified here. After checking that row, inspect stopped: the bold entries move to DS. That reversal matters when assessing whether the evaluator is uniformly reliable across motion categories. e-estimator-validatione-estimator-traininge-metricse-experiment-scope
What it supports. The reported averages strongly favor ACT-Estimator: ADE/FDE are 0.81/1.59 meters, compared with 7.52/13.75 for the adapted baseline. These results support its use as a candidate motion estimator on the tested real-video split, while the stopped row demonstrates that its advantage is not universal.
Where the evidence stops. The caption claims lower errors across all categories, but stopped favors DS: 0.06/0.09 versus 0.76/1.48. The authors also acknowledge a baseline domain gap. No independent calibration on generated video is shown.
Figure 6. Aggregate instruction agreement improves, but the pattern depends strongly on maneuver. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the vertical axis as the conditioned action and the horizontal categories as the action estimated from the generated video. A diagonal cell is a match; off-diagonal mass identifies what the model appears to do instead. The shared color scale ranges from zero to one. TERRA's curving-right diagonal reaches 1.00 in the displayed matrix, compared with Vista's 0.49. The pattern reverses for accelerating, where the diagonals are 0.04 and 0.23 respectively. Slow-straight also favors Vista. Use those rows to avoid turning the headline improvement into an all-category claim. These are generated-video instruction matches, distinct from the estimator's real-video classification accuracy. e-iece-protocole-datasete-estimator-datae-estimator-validation
What it supports. TERRA's reported overall IEC is 44.11%, versus 30.72% for Vista, a calculated gain of 13.39 percentage points. The matrices show stronger turning agreement alongside weaknesses in other maneuvers. Even the better aggregate leaves a majority of instructed actions unmatched according to this estimator.
Where the evidence stops. Shifting categories are absent and the stopped row is all zero; the paper does not reconcile this display with Table 2's categories and scored denominator. Instruction frequency also differs between the models.
Table 4. Geometric alignment reveals exceptions hidden by the average and by action labels. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First compare the bottom Average row separately for ADE and FDE. Then move upward to curving: Vista has the lower ADE for both turn directions, while TERRA has the lower FDE. Starting is another clear exception, favoring Vista on both metrics. These mixed rankings are possible because ADE aggregates the full path and FDE uses only its endpoint. HS and LS retain the high-speed and low-speed meanings from the estimator evaluation. The stopped cells are explicitly N/A, not zero error. Read these values alongside Figure 7's trajectory shapes rather than assuming that a correct turn classification implies the closest path. e-alignmente-diagnostice-metricse-protocole-estimator-validation
What it supports. TERRA improves the reported average ADE from 4.50 to 3.98 meters and FDE from 8.66 to 8.21 meters. However, its starting ADE/FDE is 4.52/14.01, versus Vista's 3.23/10.32. The table establishes aggregate improvement under this protocol, with substantive category-specific exceptions.
Where the evidence stops. Section 5.2 says TERRA has lower curving ADE, but the table shows higher values: 3.82 versus 3.59 for left and 3.78 versus 3.73 for right. These crops preserve the reported values.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| High-level action classification on real validation video ACT-Estimator's randomly selected 8,407-sample validation subset; nine predicted classes. | 94.03% Classification accuracy | No competing classifier is reported; class diagonals include 0.77 for decelerating and 0.83 for accelerating. This validates the estimator on its real-video split, not its calibration on synthetic videos or scene-disjoint generalization. e-estimator-validatione-estimator-training |
| Trajectory estimation on real validation video Same estimator validation split; adapted DROID-SLAM uses Metric3D monocular depth. | ACT-Estimator: 0.81 / 1.59 Reported average ADE / FDE, meters | DROID-SLAM + Metric3D: 7.52 / 13.75. The mean favors ACT-Estimator, but stopped scenes favor the baseline (0.06/0.09 versus 0.76/1.48), contradicting Table 3's universal caption claim. The authors acknowledge a baseline domain gap. e-estimator-validation |
| Instruction-execution consistency of generated driving video ACT-Bench comparison; 2,286 videos generated per model, with different instruction frequencies. Exact scored-category reconciliation is unspecified. | TERRA: 44.11% IEC, percent | Vista: 30.72%; a calculated difference of 13.39 percentage points. TERRA has higher aggregate agreement, but most estimated actions still mismatch. Training data, architecture and conditioning frequency all differ; this is not an isolated frequency ablation. e-protocole-iece-terra-training |
| Trajectory alignment of generated driving video Same generated-video comparison; Table 4 reports action-category and aggregate estimates, with stopped marked N/A. | TERRA: 3.98 / 8.21 Reported average ADE / FDE, meters | Vista: 4.50 / 8.66. TERRA improves the reported averages but has higher curving ADE and worse starting ADE/FDE. The prose reverses the curving-ADE comparison; retain the table values. Turning more convincingly can coexist with travel-distance error. e-alignmente-diagnostice-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. A trajectory can turn in the right direction and still miss the requested distance. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare models vertically within a column: TERRA occupies the top row and Vista the bottom. The light-blue lines are individual estimated trajectories, blue dots are their mean, and orange dots are the instructed trajectory. Both axes measure position in meters; compare within panels carefully because their ranges differ. For curving, inspect lateral displacement and the separation between the blue and orange paths. For starting, inspect the longitudinal distance reached. The starting means remain well below the instruction's endpoint. The plot therefore helps explain why maneuver agreement and displacement error should be read together, rather than collapsed into one notion of controllability. e-diagnostice-alignmente-iece-protocole-experiment-scope
What it supports. The displayed mean curves turn more strongly for TERRA than for Vista, but neither set of estimates consistently coincides with the requested paths. The starting panels expose a distance shortfall even though starting can be recognized as an action category. The visual supports a diagnostic distinction between direction and distance.
Where the evidence stops. This is a diagnostic comparison, not a controlled ablation or a confidence interval. The caption's broad favorable framing does not override Table 4's curving-ADE exceptions, and the plots do not isolate instruction frequency as their cause.
7. Analysis & limitations
7.1 What the evidence leaves open
Benchmark and estimator labels differ: shifting categories are present in Table 2 but absent from Figure 6/Table 4, whereas stopped and speed subdivisions appear in the estimator. The mapping and scored denominator are unresolved, so the reported IEC should not be treated as verified coverage of every benchmark pair. e-datasete-estimator-datae-iece-alignment
Stride-one estimator windows combined with random sample validation may permit overlapping scenes across splits; this is a leakage risk, not demonstrated leakage. No generated-video calibration or uncertainty intervals are reported. Foreground-interaction filtering also narrows the benchmark's traffic-interaction coverage. e-estimator-datae-estimator-traininge-datasete-experiment-scope
Qualitative examples show abrupt transitions and other agents apparently responding to ego instructions in both models. Crash generation is illustrated, but neither its physical validity nor a causal-misalignment rate is established. No controlled ablation separates joint losses, CoVLA data, conditioning frequency or video refinement. e-failurese-experiment-scope
7.2 Questions for discussion
- Would the evaluator retain its accuracy with scene-disjoint splits and independent generated-video motion labels?
- How much of TERRA's IEC advantage survives matched instruction frequency and training-data controls?
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuilding the benchmark requires nuScenes camera/ego-pose data, selected CoVLA templates, the sequential labeling thresholds in Table 6 and manual scene filters. Preserve context identity when constructing splits. Exact category mapping, estimator beta and dataset sampling proportions need clarification; a release promise does not establish downloadable artifacts. e-datasete-templatese-estimator-datae-estimatore-terra-traininge-purpose
Full TERRA training is substantial: the text states 40k steps on 56 H100 80GB GPUs, followed by separate refiner training for 800k steps on eight H100 80GB GPUs. Table 7 instead lists scheduler num_training_steps=172440. Preserve both entries until their relationship is resolved. e-terra-traininge-settings
Generation settings include Vista seed 23 and 50 diffusion steps, and TERRA temperature 0.9, top_p 1.0 and top_k -1. Table 8's n_conds=1 is not reconciled with the main text's three context frames. Proposed checks should first fix these conventions, then test scene-disjoint evaluator reliability and matched instruction frequency. e-settingse-protocole-estimator-datae-estimator-training
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the evaluator survives scene separation and generated video
Reader-proposed check, not performed: reconstruct the rule-labeled estimator data using Table 6 and compare the reported random-window split with a split that holds out entire nuScenes scenes. Keep architecture, training budget and class balance fixed; report per-class accuracy and ADE/FDE, including stopped. Separately audit a fixed, stratified set of Vista and TERRA videos with blinded maneuver labels and an independent motion estimate, disclosing the latter's uncertainty. Compare evaluator disagreement by model and maneuver. A large scene-disjoint performance drop or model-dependent errors on generated video would weaken the interpretation of IEC as a generator-only score. Resolve shifting-class mapping and the scored denominator before aggregation. e-estimator-datae-estimator-traininge-estimator-validatione-iece-datasete-alignment
Check 2: Isolate trajectory-update frequency within TERRA
Reader-proposed check, not performed: use the same TERRA checkpoint, contexts, trajectory templates, sampling seeds and refiner for two conditions: instructions updated every frame, and instructions refreshed only at the two Vista round boundaries. Keep sequence length fixed and explicitly document how repeated or masked trajectory slots emulate the sparse interface. First reconcile the three-context-frame description with Table 8's n_conds field. Compare per-class IEC, ADE/FDE and estimated motion discontinuity at the round boundary, using paired samples. If denser updates improve these outcomes consistently, they support a frequency mechanism within TERRA; if not, that explanation for its advantage is weakened. This check would still not isolate the effects of architecture or training data in the original Vista comparison. e-terra-coree-protocole-settingse-iece-alignmente-failures
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block on p. 1; all Figures 1–15 and Tables 1–8; metric definitions on pp. 4–5; estimator training and validation on pp. 5–6; comparison protocol, matrices, errors and diagnostic trajectories on p. 7; failure examples on pp. 8 and 18–19; dataset rules and estimator design on pp. 12–14; and TERRA token ordering, loss, refiner, hardware, schedules and inference settings on pp. 15–17. Read all seven text chunks, including reference pages 9–11, which were not rendered. All six final original crops were viewed; the TERRA crop was widened to retain its time label. Checked the diagram flows against Sections 3.3 and D.1–D.2, and retained the Figure 3 refiner omission and numerical/prose discrepancies in the explanations. Separate supplemental movies were not supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–2: motivation and related work
- Sections 3.1–3.4: benchmark, metrics, estimator and TERRA
- Sections 4.1–4.2: estimator validation
- Sections 5.1–5.3 and 6: controllability, diagnostics and conclusion
- References, PDF pp. 9–11
- Appendices A–C: estimator dataset, design, thresholds and instruction templates
- Appendices D–F: TERRA architecture, refiner, training and generation settings
- Appendix G: visualizations
Outside the original text pass
- Verified artifact: arXiv:2412.05337v1, 6 December 2024. The four authors match the catalog. The supplied observedTitle contains small-cap extraction spacing, preserved in provenance and metadata. The catalog's ICLR Workshop 2025 BibTeX is not established by this PDF; no later edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure omission was resolved by inspecting the retained PDF: all numbered figures and tables were visually reviewed. Reference pages 9–11 were read as text but not rendered.
- No separate supplemental movies were supplied or viewed. Code, checkpoints and external release availability were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block and arXiv margin
The title is ACT-Bench: Towards Action Controllable World Models for Autonomous Driving; Hidehisa Arai, Keishi Ishihara, Tsubasa Takahashi and Yu Yamaguchi are credited at Turing Inc. The margin identifies arXiv:2412.05337v1, 6 December 2024. The supplied observedTitle preserves extraction spacing.
Go to primary source ↓e-purposePDF pp. 1–3, Abstract, Introduction, Figure 1 and contribution list
ACT-Bench evaluates action fidelity with context/trajectory pairs and a video-based motion estimator; TERRA is its trajectory-conditioned baseline. The abstract promises future public release of all benchmark components.
Go to primary source ↓e-datasetPDF pp. 3–4, Section 3.1 and Table 2
The benchmark uses nuScenes validation CAM_FRONT contexts of ten frames paired with trajectory templates. Speed differences over 10 km/h and visually identified interfering foreground interactions are filtered. Table 2 totals 2,286 pairs, including 115 left-shift and 192 right-shift pairs, but no stopped category.
Go to primary source ↓e-templatesPDF p. 4, Section 3.1; pp. 12–14, Appendix C and Figure 12
Nine maneuver categories contain 36 instruction-trajectory variants manually curated from CoVLA and paired with compatible scenes; trajectories are adjusted for the ideal vehicle orientation at successive frames.
Go to primary source ↓e-metricsPDF pp. 4–5, Section 3.2, Equations (1)–(3)
IEC is the sample fraction whose instructed and estimated high-level actions match. Trajectory alignment compares instructed and estimated paths using ADE over points and FDE at the last point; smaller distances indicate closer alignment.
Go to primary source ↓e-estimatorPDF p. 5, Section 3.3 and Equation (4); p. 12, Appendix B and Figure 11
ACT-Estimator uses I3D features, flattened features with positional encoding and two Transformer encoder layers, a pooled MLP classification head, and a GRU trajectory head with attention. Training combines cross-entropy and smooth L1 losses weighted by beta; no numerical beta is provided in these descriptions.
Go to primary source ↓e-estimator-dataPDF pp. 12–13, Appendix A, Tables 5–6; p. 5, Section 3.3
Estimator data uses 44-frame windows with stride one and ego-pose trajectories in initial-frame local coordinates. Table 5 lists 46,412 samples across eleven labels; the two shifting classes are excluded from classification, leaving nine prediction classes including stopped and separate fast/slow constant-speed classes. Table 6 gives sequential rule conditions and thresholds.
Go to primary source ↓e-estimator-trainingPDF pp. 5–6, Section 4, Training procedure and Validation dataset
Estimator training uses 30,850 iterations, four H100 GPUs, per-GPU batch 12, accumulation two, AdamW and OneCycleLR with maximum learning rate 1.2e-4. The effective batch is 96. Validation randomly selects 8,407 samples excluded from training; scene-disjointness is not specified.
Go to primary source ↓e-estimator-validationPDF p. 6, Sections 4.1–4.2, Figures 4–5 and Table 3
Figure 4 reports 94.03% classification accuracy on 8,407 validation samples, with diagonal values 0.77 for decelerating and 0.83 for accelerating. Table 3 reports average ADE/FDE 0.81/1.59 for ACT-Estimator and 7.52/13.75 for DROID-SLAM combined with Metric3D. The stopped row favors the baseline: 0.06/0.09 versus 0.76/1.48. This contradicts the caption's claim of lower errors across all categories. Figure 5 labels trajectory axes in meters; Section 4.2 acknowledges the baseline's domain gap.
Go to primary source ↓e-protocolPDF p. 7, Section 5 experimental setup and Section 5.1; p. 17, Appendix F
Each world model generates 2,286 videos. Section 5 specifies three conditioning frames, two rounds of 22 generated frames and 44 generated frames (4.4 seconds). Vista receives one trajectory per round; TERRA receives one per frame. Vista is modified to use the corrected trajectory at the start of each round.
Go to primary source ↓e-iecPDF p. 7, Figure 6 and Section 5.1
IEC is 30.72% for Vista and 44.11% for TERRA. The displayed nine-class matrices omit shifting actions and contain an all-zero stopped row. Curving-right diagonal entries are 0.49/1.00 for Vista/TERRA, while accelerating is 0.23/0.04 and slow-straight is 0.89/0.54. No explicit reconciliation with all Table 2 categories or scored denominator is supplied.
Go to primary source ↓e-alignmentPDF p. 7, Table 4 and Section 5.2
Reported average ADE/FDE are 4.50/8.66 for Vista and 3.98/8.21 for TERRA. Curving-left ADE is 3.59/3.82 and curving-right ADE 3.73/3.78 for Vista/TERRA, contradicting the prose saying TERRA has lower curving ADE. Starting ADE/FDE is 3.23/10.32 versus 4.52/14.01. All stopped entries are N/A; shifting rows are absent.
Go to primary source ↓e-diagnosticPDF p. 7, Figure 7, caption and Section 5.2
The scatter plots compare estimated trajectories, mean estimates and instructed trajectories for curving left, curving right and starting. TERRA's displayed means turn more strongly than Vista's, while travel-distance errors remain. Both coordinate axes are in meters.
Go to primary source ↓e-terra-corePDF p. 5, Figure 3 and Section 3.4; pp. 13–15, Appendix D.1 and Equation (5)
TERRA tokenizes images, embeds supplied trajectory vectors, and interleaves image tokens with trajectory inputs. At inference it predicts the next frame token by token, appends that frame's trajectory and continues. Equation (5) conditions image tokens on preceding frames, preceding same-frame tokens and earlier trajectories; only image tokens incur prediction loss. Text conditioning is omitted.
Go to primary source ↓e-refinerPDF pp. 15–16, Appendix D.2 and E
A separately trained Stable Video Diffusion-based post-hoc Video Refiner conditions on the coarse decoded frame sequence to improve temporal consistency and resolution. Frames are upscaled from 288×512 to 384×640 before SVD encoding. This component is absent from Figure 3's simplified drawing, whose caption says a video decoder is omitted.
Go to primary source ↓e-terra-trainingPDF pp. 15–16, Appendix E.1–E.2 and Table 7
TERRA uses a pretrained Open-MAGVIT2 tokenizer, 576 image tokens per frame, six trajectory points, 25-frame clips at 10 Hz and 14,550-token training sequences. The Llama-based world model is randomly initialized. OpenDV-YouTube, nuScenes and CoVLA contribute 1.67 million, 25,000 and 0.23 million non-overlapping clips. World-model training is stated as 40k steps on 56 H100 80GB GPUs, batch one per GPU and accumulation four; the refiner uses 800k steps on eight H100 80GB GPUs.
Go to primary source ↓e-settingsPDF p. 16, Table 7 and Appendix E.2; p. 17, Table 8 and Appendix F
Table 7 specifies AdamW learning rate 1e-4, cosine scheduling and num_training_steps 172440, while Appendix E.2 says 40k training steps. Table 8 gives Vista n_conds=1, n_frames=25, n_rounds=2, seed 23, 576×1024 resolution, cfg_scale=2.5 and 50 steps. TERRA generation uses vLLM, temperature 0.9, top_p 1.0 and top_k -1. The relationship of n_conds to the three context frames in Section 5 is not explained.
Go to primary source ↓e-failuresPDF p. 8, Section 5.3 and Figures 8–10; pp. 17–19, Appendix G and Figures 13–15
The authors show Vista round-transition discontinuities, other vehicles changing motion under ego-action conditioning in both models, and a TERRA crash example. Appendix G acknowledges imperfect trajectory adherence. These are qualitative frame sequences; supplementary movies are referenced on p. 8.
Go to primary source ↓e-experiment-scopePDF pp. 6–8, Sections 4–5, Figures 4–10 and Tables 3–4; pp. 17–19, Appendix G
The reported empirical evidence consists of estimator validation, a Vista/TERRA comparison and qualitative diagnostics. No controlled component ablation, uncertainty intervals, causal-misalignment frequency, or executed-vehicle evaluation is reported in these experiment sections.
Go to primary source ↓8.5 Primary sources
ACT-B ENCH : Towards Action Controllable World Models for Autonomous Driving ↗
PDF · 12,817 extracted words
Source fingerprint
560190990e9f2d2f89add329668ce653c758d9975447e6f52db7ebed6bbe11c5