AdaWorld: Learning Adaptable World Models with Latent Actions
1. Paper overview
In one sentence: Pretraining a world model on continuous latent actions makes new control interfaces easier to learn, while leaving action selection to an external planner. e02e03e05e06e08e09e10
| At a glance | What to know |
|---|---|
| Research problem | Source description Passive video pretraining provides visual knowledge but leaves new control interfaces expensive to learn. AdaWorld asks whether unlabeled transitions can supply reusable action conditions despite heterogeneous controls and changing appearances. e02 |
| Core mechanism | |
| A key reported result | Procgen goal-reaching with visual MPC: 56.67±2.16% with finetuning; 44.83±1.37% without finetuning. Average success rate ± standard error. 30 selected scenes each from Heist, Jumper, Maze and CaveFlyer; 100 samples per action per scene; 500 finetuning steps; success within 20 executed steps; five random seeds. Action-agnostic 26.00±0.98%; random 26.17±2.55%; Q-learning 27.17±1.27%; ground-truth-simulator MPC 80.67±2.11%. Demonstrates executed simulator planning. Unfinetuned AdaWorld still uses collected interactions to initialize controls. Selected reachable scenes limit generality; Jumper declines from 68.00% without finetuning to 58.67% with it. e10e15e22 |
| Reading caution | Source description Authors report non-real-time inference, difficulty generating beyond the initial scene, long-rollout degradation and failures in physics, dynamic agents and large viewpoint changes. e20 |
Core contributions
Figure 2. The earlier frame supplies context while a compact latent variable describes the transition. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two images at left: both the earlier frame and the later frame feed the encoder. Follow the middle path through the continuous latent space to the sampled action, then into the decoder. The long upper connection carries only the earlier frame directly to that decoder. This distinction agrees with Section 2.1 and Equation (2): the encoder may observe the transition being represented, while reconstruction conditions on the earlier frame and compressed action. The later frame is the reconstruction target. The decoder therefore has a separate route for existing appearance, reducing the need to transmit unchanged scene details through the action variable. e03e04e13e19e20
What it supports. The bottleneck is designed to spend limited information on change. AdaWorld uses a 32-dimensional action representation, and its β-weighted objective balances reconstruction against compression. This gives the subsequent world model a common conditioning format across videos whose original action labels may be absent or incompatible.
Where the evidence stops. A small code does not guarantee that every encoded change is an agent action. The design assumes actions often dominate transitions; the source acknowledges blocked actions and imperfect physics. The diagram itself provides no causal-identification guarantee.
2. Motivation
2.1 The problem and the proposed response
Passive video pretraining provides visual knowledge but leaves new control interfaces expensive to learn. AdaWorld asks whether unlabeled transitions can supply reusable action conditions despite heterogeneous controls and changing appearances. e02
2.2 What this reading follows
A video model can generate plausible motion without knowing how to respond to a new environment's controls. AdaWorld addresses that gap by first learning a compact description of the transition between two frames. A separate diffusion model then learns to predict the next frame given that description and recent history. At adaptation time, demonstrations or labeled interactions connect unfamiliar controls to the learned interface. The illustrations below follow that information flow, then distinguish three tests: transferring motion between videos, improving prediction after finetuning, and reaching goals through executed simulator actions. The strongest evidence concerns adaptation under the paper's particular data and planning budgets. e02e03e05e06e08e09e10
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational-work tags for latent action pretraining and neural world simulators fit. The encoder infers transitions from frame pairs; a separate model predicts futures conditioned on controls, and an external planner selects executed actions. This is not joint future/action policy prediction. The recorded architecture/paradigm/quadrant 'Not applicable' is appropriate for this foundational classification, rather than evidence of a unified One Model controller. e03e05e06e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why the earlier frame bypasses the action bottleneck
Imagine two demonstrations of the same movement with different colors and backgrounds. An unconstrained encoding of their later frames could store those appearances rather than a reusable movement. AdaWorld gives the decoder the earlier frame directly, then makes the latent action compact and penalizes excess information through β. The source's rationale is that reconstruction can reuse existing context and reserve the latent channel for critical variation. My interpretation is that this is an architectural incentive, not a guarantee of semantic action recovery. Figure 7 supports a qualitative tradeoff: reducing β separates representations more finely but weakens overlap of corresponding actions across environments. The acknowledged blocked-action cases also matter: a command can produce little visible change, so a transition-derived representation need not uniquely identify the intended command. e03e04e19
Figure 3. Training extracts action conditions from known transitions; generation rolls predicted frames back into memory. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper path as diffusion training: the target next frame receives noise, enters the world model with the history representation, and is compared with the prediction through the red dashed loss connection. The middle encoder path uses the current and next frames to extract a training action condition. At the bottom, noise augmentation modifies historical frames during training. The gray dashed arrow runs from the predicted next frame back to short-term memory and indicates autoregressive inference. Section 2.2 clarifies that the actual implementation operates with encoded image latents and an SVD-initialized model; the pictorial noisy image is a schematic, not an additional pixel-space architecture specification. e03e05e06
What it supports. Action recognition and high-quality prediction serve different roles. The small autoencoder decoder makes coarse predictions, so AdaWorld learns a separate diffusion simulator. Its training objective teaches the simulator to respond to the extracted action while maintaining recent visual context, rather than using the reconstruction decoder for long rollouts.
Where the evidence stops. The next-frame input to the action encoder belongs to training or an observed demonstration. Future target observations are not available during deployment; control then comes from transferred latent actions or a learned command interface. No stop-gradient or attention-mask rule is specified by this schematic.
5.2 Follow the change from training target to inference condition
During autoencoder learning, both frames are observed and the later frame supplies the reconstruction target. During world-model pretraining, that observed transition also supplies a latent action, while a separate diffusion model learns next-frame prediction from the action and recent history. This ordering explains why the action encoder can see the next frame in Figure 3 without implying access to future deployment observations. At transfer time, the observed demonstration supplies the action sequence and a new initial image supplies the context. At adaptation time, labeled interactions initialize discrete embeddings or an MLP that maps continuous commands. Generated frames then enter short-term memory. The autoencoder decoder is not the rollout engine: Section 2.2 says its coarse single-pass predictions degrade, motivating the independent diffusion model for generation. e03e05e06
5.3 Ask which resource each adaptation result saves
The experiments support several distinct efficiency claims. Video transfer reuses a demonstration without additional training, but measures generated videos. Table 2 then fixes a small adaptation dataset and evaluates prediction fidelity after 800 world-model steps, with extra MLP initialization for nuScenes. Table 3 goes further: a planner uses the adapted predictions to choose actions that are actually executed in Procgen. Even its unfinetuned variant requires interaction-derived embeddings, so that row saves model updates rather than eliminating interaction data. Finally, VP2 uses only 1K adaptation steps but thousands of training trajectories: 5K for Robosuite and 35K for RoboDesk. My reading is that AdaWorld most clearly improves reuse of a learned control interface; its experiments do not establish one uniform notion of few-shot, zero-interaction or low-total-compute control. e08e09e10e11e15
5.4 Training and inference
During training
Training mixes roughly two billion frames from Gym Retro, Procgen, Open X-Embodiment, Ego4D, SSv2 and MiraData. Procgen uses 9,000 training start levels and 1,000 held-out levels in hard mode. Mixture weights follow video counts, not frame counts. e07e12
The 500M-parameter autoencoder has 32-dimensional actions and trains from scratch for 200K steps, batch 960, AdamW learning rate 2.5×10⁻⁵ and β=2×10⁻⁴. The 1.5B-trainable-parameter world model defaults to 256×256 inputs, 80K steps, batch 64, learning rate 5×10⁻⁵ and 16 NVIDIA A100 GPUs. Comparative experiments instead specify 50K steps for every world model; these budgets must remain distinct. e13e07
During inference
Transfer extracts a demonstration's action sequence and reuses it with a new initial image, appending predicted frames to memory. Default generation uses five sampling steps and guidance 1.05. Historical frames remain uncorrupted although the augmentation-level condition is 0.1. e06e14
Procgen MPC scores predicted trajectories by maximum RGB cosine similarity to a goal image. Two CEM iterations each sample 100 length-15 action sequences and retain ten elites; five actions are then executed before replanning. Planning uses three denoising steps without guidance and a 20-step execution limit. VP2 instead uses MPPI. e15e10e11
5.5 Implementation flow
- Compress the transition
A spatiotemporal Transformer processes two frames as 16×16 patches plus learned tokens. Spatial attention stays within each frame; temporal attention connects matching spatial positions. The later learned token parameterizes the latent-action posterior. A spatial decoder receives the sampled action and earlier frame to reconstruct the later frame. e03
- Learn a predictive control interface
After autoencoder training, its encoder supplies action conditions. An SVD-initialized 3D UNet denoises one future frame. The action joins timestep and CLIP image embeddings; encoded history joins the noisy latent map. The latest history frame supplies SVD's image condition. Up to six historical frames and training-time history corruption encourage temporal robustness. e05e13
- Bind controls to the interface
For discrete actions, average inferred latent actions sharing a label and initialize one embedding per control before whole-model finetuning. For continuous controls, initialize a lightweight MLP on raw-action/latent-action pairs. This learns how commands enter the simulator; it does not directly learn a goal-conditioned policy. e06e09e15
6. Experiments & results
AdaWorld learns a continuous description of change between video frames, then uses that description to condition a separate next-frame diffusion model. Demonstration actions can be replayed in a new visual context, or mapped to an environment's controls with limited adaptation. Its central contribution is pretraining an adaptable control interface; planning still requires an external optimizer and environment feedback.
6.1 Read the original evidence
Table 1. Continuous latent actions lead all three reported transfer measures in both datasets. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each dataset has three columns. Lower FVD indicates closer video-distribution similarity; higher ECS measures better frame-level I3D embedding similarity; the Human column reports judged action-transfer success. Compare within each column rather than combining their incompatible scales. Evaluation pairs a demonstration with a different initial context and generates 20 frames, using 1,300 pairs across the datasets. The human columns come from a smaller sample: four volunteers judge 50 pairs per dataset. The other rows replace AdaWorld's continuous condition with zeros, optical flow, or eight discrete codes under the aligned comparison training setting. The shaded row is AdaWorld, not a separate finetuned variant. e07e08e12
What it supports. On LIBERO, AdaWorld reaches FVD 767.0 and human success 70.5%, compared with 1545.2 and 0% for action-agnostic pretraining. SSv2 shows the same ordering: 473.4 FVD and 61.5% human success versus 847.2 and 1%. This agreement between automatic and human measures supports better demonstrated-motion transfer in generated videos.
Where the evidence stops. These are video-generation outcomes, not executed manipulation success. No error bars are reported. Although the text calls the evaluation datasets unseen, SSv2 also appears in the training mixture; exact split independence is unresolved in the supplied source.
Table 2. Prediction quality improves after a fixed adaptation budget across discrete and continuous controls. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first three environment groups use discrete action spaces, while nuScenes uses continuous displacements. Read each pair of columns together: higher PSNR and lower LPIPS both favor AdaWorld. The table's source caption specifies 800 finetuning steps with 100 samples per discrete action or 100 continuous trajectories. Section 3.2.1 separately identifies 300 validation samples per environment. Those are different quantities and should not be treated as one shared sample count. For the continuous interface, a two-layer MLP also receives 3K initialization steps on action/latent-action pairs before world-model adaptation. The table therefore compares the resulting adapted pipelines, including their prescribed initialization methods. e09e16
What it supports. Minecraft improves from 19.44 PSNR and 0.532 LPIPS for action-agnostic pretraining to 21.59 and 0.457 for AdaWorld. The discrete latent baseline is closer at 21.33 and 0.465. All four domains support action-aware pretraining, with continuous latent actions providing the best reported values under this particular budget.
Where the evidence stops. Prediction fidelity is not a success rate, and no uncertainty estimates appear here. The table combines pretraining and interface-initialization choices; the adjacent ablation is needed to assess their separate contributions. Continuous-interface initialization adds optimization beyond the 800 world-model steps.
Table 3. A better adaptable simulator supports more successful goal-reaching, with substantial room below oracle MPC. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each game column summarizes 30 selected scenes, and the rightmost column averages across the four games. Entries report success rates and standard errors across five seeds, as specified in the source caption. Compare the two AdaWorld rows first: 'without finetuning' still initializes action embeddings from collected interactions. The finetuned row additionally adapts for 500 steps. The action-agnostic row uses an adapted model, while the bottom oracle uses ground-truth simulation within the planning strategy. Appendix B.4 describes CEM search over candidate action sequences and execution of the first five actions before replanning. Success requires reaching the goal within 20 executed steps, rather than merely generating a goal-like frame. e10e15e22
What it supports. Finetuned AdaWorld averages 56.67±2.16%, versus 26.00±0.98% for the action-agnostic model; even the unfinetuned variant reaches 44.83±1.37%. Oracle MPC reaches 80.67±2.11%, exposing remaining model-related headroom within this planner. Finetuning is not uniformly beneficial: Jumper falls from 68.00% to 58.67% while the overall average rises.
Where the evidence stops. These are simulator results on deliberately reachable scenes, not all Procgen tasks. Ground-truth search selected the scenes and survival-oriented games were excluded. The oracle is a reference for this planning strategy, not a proof of globally optimal control.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Demonstration action transfer 1,300 paired videos across LIBERO and the ten most frequent SSv2 labels; 20 predicted frames. Human evaluation: 50 pairs per dataset, four volunteers; no transfer finetuning. | LIBERO: 767.0 / 0.804 / 70.5%; SSv2: 473.4 / 0.639 / 61.5%. FVD ↓; I3D embedding cosine similarity ↑; human transfer success ↑ | Action-agnostic: 1545.2 / 0.702 / 0% and 847.2 / 0.592 / 1%, respectively. Flow and eight-code discrete conditions also score worse in every column. Supports visual action transfer, not robot execution. No uncertainty is supplied; SSv2 split independence remains unclear because it also appears in training. e08e07e12 |
| Adaptation to four unseen environments 800 finetuning steps; 100 samples per discrete action or 100 nuScenes trajectories; 300 validation samples per environment. | Habitat 23.58 / 0.327; Minecraft 21.59 / 0.457; DMLab 22.92 / 0.335; nuScenes 21.60 / 0.436. PSNR ↑ / LPIPS ↓ | Action-agnostic: 20.34 / 0.450; 19.44 / 0.532; 20.96 / 0.386; 20.86 / 0.475, respectively. Best reported fidelity across all four settings, without uncertainty estimates. These image metrics do not directly measure control success. e09 |
| Procgen goal-reaching with visual MPC 30 selected scenes each from Heist, Jumper, Maze and CaveFlyer; 100 samples per action per scene; 500 finetuning steps; success within 20 executed steps; five random seeds. | 56.67±2.16% with finetuning; 44.83±1.37% without finetuning. Average success rate ± standard error | Action-agnostic 26.00±0.98%; random 26.17±2.55%; Q-learning 27.17±1.27%; ground-truth-simulator MPC 80.67±2.11%. Demonstrates executed simulator planning. Unfinetuned AdaWorld still uses collected interactions to initialize controls. Selected reachable scenes limit generality; Jumper declines from 68.00% without finetuning to 58.67% with it. e10e15e22 |
| VP2 robot-task planning 64×64 models, 1K adaptation steps, 5K Robosuite and 35K RoboDesk trajectories; four evaluation runs; MPPI control. | 63.50±1.71%; aggregate 21.54. Robosuite push success ± standard error; simulator-normalized aggregate score | Action-agnostic 17.50±0.50%; aggregate 5.03. Simulator robot tasks, not physical deployment. The aggregate is normalized, not an ordinary raw percentage. Flat block and Open drawer results are omitted by the authors because scores were not meaningful under this adaptation budget. e11 |
| Control-interface initialization ablation Minecraft and nuScenes, varying interaction counts and 0–800 finetuning steps. | Source reports that randomly initialized AdaWorld surpasses the action-agnostic baseline after 200 steps. PSNR curves | Data-initialized controls usually start stronger; random initialization initially loses this advantage. Supports contributions from both pretraining and interface initialization. Figure 6 has no uncertainty bands and shows nonmonotonic improvement, including an initially weaker initialized nuScenes result at 100 samples. e16 |
| Transfer of the pretraining recipe to iVideoGPT OpenX checkpoint; tokenizer fixed; 27K additional pretraining steps; 1K BAIR adaptation steps; 256 test videos predicting 15 future frames. | 17.40 / 0.204. PSNR ↑ / LPIPS ↓ | Action-agnostic iVideoGPT: 16.59 / 0.220. Evidence that the recipe extends beyond diffusion, under this specific backbone and adaptation protocol. e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. Initialization buys an early advantage, while pretrained controllability also helps random interfaces learn. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis counts finetuning steps and the vertical axis is PSNR. Read one panel at a time because its title fixes the available data. Minecraft's upper row uses 50, 100, 150 or 200 samples per action; nuScenes' lower row uses 100, 200 or 300 samples. Blue uses the proposed embedding or MLP initialization, green omits it, and red is action-agnostic pretraining. Blue versus green tests initialization within AdaWorld. Green versus red asks whether action-aware pretraining helps even after discarding that initialization. The two comparisons answer different questions, so blue versus red alone should not be credited entirely to a single mechanism. e09e16
What it supports. The green curves recover quickly and the source reports overtaking the baseline after 200 steps, supporting a benefit from the pretrained interface itself. Blue generally provides an earlier advantage. The plots also qualify a simple 'more finetuning is better' story: low-data nuScenes predictions deteriorate after an early peak.
Where the evidence stops. There are no uncertainty bands. Section 3.2.1 says AdaWorld starts better in all cases, but the 100-sample nuScenes panel places blue below red initially. Preserve this exception; neither the graphic nor its text supports uniformly superior starting performance.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors report non-real-time inference, difficulty generating beyond the initial scene, long-rollout degradation and failures in physics, dynamic agents and large viewpoint changes. e20
The bottleneck encourages context separation but does not establish identifiable causal actions. Figure 7's UMAP supports a qualitative β tradeoff; blocked actions and uncontrolled changes can complicate interpretation. e03e19e20
SSv2 is called unseen in evaluation while Table 7 includes it in pretraining; the supplied text does not resolve exact split overlap. Procgen excludes survival-oriented tasks and selects reachable scenes using the ground-truth simulator. e08e12e22
7.2 Questions for discussion
- Does cross-context transfer survive independently moving distractors?
- How much adaptation benefit remains when initialization data and total optimizer steps are matched?
- Would improved PSNR predict better control on less selectively chosen Procgen scenes?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the staged autoencoder/world-model training, SVD initialization, documented data mixture, approximate 10 Hz preprocessing, brightness jitter, center cropping and history noise augmentation. Exact OpenX subset removals, biased action-sampling probabilities and periods are not enumerated. e05e12e13
Preserve each evaluation budget: simulation adaptation uses batch 32, learning rate 5×10⁻⁵ with a 0.1 multiplier for pretrained weights; nuScenes adds 3K MLP initialization steps. Report this overhead separately. Simulator versions, checkpoint identifiers and full software configuration remain insufficiently specified in the implementation appendix. e09e13e14e15e11
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate pretrained controllability from embedding initialization
Reader-proposed check, not performed: repeat Minecraft adaptation with the same 100 labeled samples per action and independent held-out validation samples. Compare action-agnostic pretraining, AdaWorld with random embeddings, and AdaWorld with averaged embeddings, using matched 50K-step pretraining runs and the source's batch size, learning rates and 0–800-step adaptation range. Repeat across seeds and report PSNR, LPIPS and uncertainty at identical checkpoints. If random-interface AdaWorld reliably overtakes the action-agnostic baseline near 200 steps, the pretrained interface contributes beyond initialization. If only averaged initialization wins, the stronger mechanism claim is weakened. Keep sample identities and embedding initialization costs explicit. e07e09e16
Check 2: Test whether the β tradeoff reflects action or context information
Reader-proposed check, not performed: train matched latent-action autoencoders at the source's β values, 2×10⁻⁴ and 2×10⁻⁶, holding architecture, data and updates fixed. On Habitat, Minecraft and DMLab, collect the same labeled commands across varied contexts, separating successful movements from obstacle-blocked attempts. Measure cross-environment action retrieval and how well a probe recovers environment identity from the codes; add independently moving distractors as a controlled stress condition. Evaluate transferred actions with the corresponding world-model conditions rather than relying only on UMAP. The bottleneck explanation predicts better action transfer with less recoverable context at the larger β. Failure under distractors or blocked transitions would delimit that interpretation. e03e04e13e19e20
8.3 Reading coverage
Visual audit: Inspected the title/author page; Figures 1–9 and 21; Tables 1–7 and 9; all method equations; and the appendix pages supporting training, hardware, preprocessing, sampling, planning, evaluation selection and proposed checks. Page 16's dense Table 8 was inspected as a layout, while its full extracted list was read without claiming visual validation of every entry. Figures 10–20 remain outside visual inspection although their captions and accompanying text were read. All six final original crops were individually viewed and retain the relevant arrows, axes, legends, headers and complete data rows. Figure 3's training and inference paths were checked against Sections 2.1–2.3 and Equations (2)–(3). Figure 6's initially weaker 100-sample nuScenes result qualifies the broader prose claim, as disclosed in its caution. Separate supplements, code and linked project media were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 18, 19, 28. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–4, including all method, experiment, ablation and limitation subsections
- Acknowledgements, Impact Statement and References
- Appendix A.1–A.2: collection, mixture and complete extracted environment list
- Appendix B.1–B.6: architecture, training, sampling, Procgen, VP2 and iVideoGPT
- Appendix C.1–C.3: additional results, failures and clustering
- Appendices D–F: SSv2 categories, selected scenes and related work
- All nine supplied text chunks covering PDF pages 1–28, including figure and table captions
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity: the observed title and all five authors match the catalog. The title page identifies the ICML 2025 proceedings, PMLR 267. No revision history or alternative edition was supplied; no cross-version equivalence is established.
- Text extraction does not reconstruct figure images; the retained PDF was inspected separately on the pages declared in the illustrated edition.
- Separate supplemental material availability has not been fully verified.
- Figures 10–20 were read through their supplied captions and accompanying text, but their page images were not inspected. Table 8's complete extracted environment list was read; its dense individual entries were not visually validated.
- Code, external project material and checkpoints were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author line, affiliations and proceedings footer
Observed title: AdaWorld: Learning Adaptable World Models with Latent Actions. Authors: Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, Chuang Gan. Affiliations: HKUST; Harvard; UMass Amherst; MIT-IBM Watson AI Lab. PMLR 267, 2025, ICML proceedings.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1
Motivates action-aware pretraining to reduce the action-label and finetuning burden when adapting world models to new control spaces.
Go to primary source ↓e03PDF p. 3, Figure 2, Section 2.1 and Eq. (1)
Two-frame spatiotemporal encoder, 16×16 image patches, learned tokens and posterior sampling; decoder predicts the next frame from earlier frame plus latent action. Spatial and temporal attention operate over the stated token groupings.
Go to primary source ↓e04PDF p. 4, Section 2.1 continuation and Eq. (2)
Introduces β multiplying the KL term to balance action expressiveness and context disentanglement; equation displays expected log likelihood minus β KL.
Go to primary source ↓e05PDF p. 3, Figure 3; pp. 4–5, Section 2.2 and Eq. (3)
Previously trained action encoder provides conditions to an independent SVD-initialized world model. It denoises one frame, injects actions through timestep and image embeddings, conditions on up to six historical frames with training noise augmentation, and optimizes squared diffusion prediction error.
Go to primary source ↓e06PDF p. 5, Section 2.3 and Figure 5; p. 6, Section 2.3 continuation
Demonstration transfer reuses extracted action sequences. Discrete labels use averaged latent embeddings; continuous controls use an initialized MLP. Whole-model finetuning specializes controls. Figure 5 illustrates averaging right and jump actions.
Go to primary source ↓e07PDF p. 6, Section 3 baseline definitions and final setup paragraph
Baselines use zero action conditions, flattened 16×16 UniMatch flow or eight VQ-VAE codes. Other settings are aligned and comparison world models train for 50K iterations. Data comprise approximately two billion frames and 1,016 game environments plus public video datasets.
Go to primary source ↓e08PDF p. 6, Table 1 all rows/columns and Section 3.1; p. 19, Appendix D
Reports 1,300 video pairs, 20-frame generation, FVD and I3D ECS, plus four volunteers evaluating 50 pairs per dataset. AdaWorld LIBERO values are 767.0, 0.804, 70.5%; SSv2 values are 473.4, 0.639, 61.5%. Action-agnostic values are 1545.2, 0.702, 0% and 847.2, 0.592, 1%. The prose calls the evaluation datasets unseen; Appendix D lists ten SSv2 categories without resolving train/test overlap.
Go to primary source ↓e09PDF p. 7, Table 2, Section 3.2.1 setup/results
Four environments excluded from training; 300 validation samples each, 100 training samples per discrete action or 100 nuScenes trajectories, 800 adaptation steps, batch 32 and learning rate 5×10⁻⁵ with pretrained-weight factor 0.1. nuScenes uses two-layer MLP initialization for 3K steps. Table 2 gives the reported PSNR/LPIPS values for every method.
Go to primary source ↓e10PDF p. 8, Table 3 and Section 3.2.2
Thirty scenes per game, 100 samples per action per scene and 500 adaptation steps. Five-seed average success/standard error: AdaWorld finetuned 56.67±2.16%, unfinetuned 44.83±1.37%, action-agnostic 26.00±0.98%, random 26.17±2.55%, Q-learning 27.17±1.27%, oracle 80.67±2.11%. Jumper unfinetuned/finetuned means are 68.00%/58.67%.
Go to primary source ↓e11PDF p. 8, Table 4 and Section 3.2.3; p. 18, Appendix B.5
VP2 uses MPPI and 64×64 models, adapted 1K steps using 5K Robosuite and 35K RoboDesk trajectories. Four runs produce Robosuite push 63.50±1.71% versus 17.50±0.50%. Ground-truth-normalized aggregate scores are 21.54 versus 5.03. Two RoboDesk tasks are omitted. Robosuite success requires cost below 0.05.
Go to primary source ↓e12PDF p. 15, Appendix A.1–A.2, Table 7 and Figure 8
Describes manual OpenX subset filtering, biased random action sampling, 9,000 Procgen training levels versus 1,000 held-out levels, hard mode and mixture weighting by video counts. Table 7 includes 7M SSv2 frames at 3% mixture weight. Specific filtering subset identities and action-bias schedules are not listed.
Go to primary source ↓e13PDF p. 16, Appendix B.1–B.2; p. 17, Appendix B.2 continuation
Autoencoder: 500M parameters, 16 encoder/16 decoder blocks, width 1024, 16 heads, 32-dimensional actions; 200K steps, batch 960, AdamW 2.5×10⁻⁵, weight decay 0.01, β=2×10⁻⁴. World model: 1.5B trainable parameters, memory six, 256×256, 80K steps, batch 64, learning rate 5×10⁻⁵, 16 NVIDIA A100 GPUs, cosine schedule and 10K warmup. EMA, noise levels 0–0.7, jitter, center crops and approximate 10 Hz preprocessing are specified.
Go to primary source ↓e14PDF p. 17, Appendix B.3
Default generation uses five sampling steps, classifier-free guidance 1.05, uncorrupted history with augmentation condition 0.1, and timestep shifting.
Go to primary source ↓e15PDF pp. 17–18, Appendix B.4, numbered MPC procedure and parameter paragraphs
Goal scoring uses maximum RGB cosine similarity along trajectories. Two CEM iterations draw 100 length-15 sequences and retain ten elites. First five actions of the selected sequence are executed, with a 20-step search limit. Planning uses three denoising steps and disables classifier-free guidance.
Go to primary source ↓e16PDF p. 7, Figure 6, all seven panels; pp. 8–9, Section 3.3 interface initialization
Blue curves use data-initialized embeddings/MLP, green omit initialization, red use action-agnostic pretraining. Text reports random-initialized AdaWorld overtakes the baseline after 200 steps. Curves show no uncertainty bands; low-sample nuScenes PSNR can decline after initial improvements, and initialized AdaWorld starts below the baseline in its 100-sample panel.
Go to primary source ↓e17PDF p. 9, Table 5 and Section 3.3 data diversity
After 40K latent-autoencoder training steps, Procgen decoder PSNR/LPIPS are 25.51/0.318 for OpenX, 26.43/0.250 for Retro and 26.62/0.234 for their mixture. This is decoder prediction, not full diffusion planning.
Go to primary source ↓e18PDF p. 9, Table 6 and Section 3.3 method generality; p. 18, Appendix B.6
Action-aware iVideoGPT yields BAIR PSNR 17.40 and LPIPS 0.204 versus 16.59 and 0.220. Both resume the OpenX checkpoint with fixed tokenizer, train 27K additional steps and adapt 1K steps; 256 test videos have 15 predicted future frames.
Go to primary source ↓e19PDF p. 9, Figure 7 and Section 3.3 hyperparameter choice
UMAP compares β=2×10⁻⁴ with 2×10⁻⁶ using 1,000 samples per action from Habitat, Minecraft and DMLab. Authors interpret lower β as more expressive but less context-disentangled, and note noise when actions cannot execute, such as moving into obstacles.
Go to primary source ↓e20PDF p. 9, Section 4 Limitations; p. 18, Appendix C.1 failure study; p. 28, Figure 21 and caption
Authors acknowledge non-real-time inference, difficulty leaving the initial scene, extremely long rollouts and imperfect physics/dynamics. Failure strips illustrate physics, dynamic-agent, long-rollout and large-view-change limitations.
Go to primary source ↓e21PDF p. 18, Appendix C.3 and Table 9
K-means over game-training latent actions generates a selectable number of control centers. Decoder ΔPSNR evaluates sensitivity to action conditions for 4–12 controls; eight-control AdaWorld scores 6.26 versus the fixed discrete model's 6.47. The full metric formula is not restated.
Go to primary source ↓e22PDF p. 19, Appendix E
Exhaustive ground-truth search selects 120 scenes solvable by goal-image matching. Survival-oriented games such as BigFish and BossFight are excluded.
Go to primary source ↓8.5 Primary sources
AdaWorld: Learning Adaptable World Models with Latent Actions ↗
PDF · 13,263 extracted words
Source fingerprint
7c460f4327ec664c1300d1e4bc1db8f4ec5acf7da880a1c91da3fe7c78a6a633