Mastering diverse control tasks through world models

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Danijar Hafner; Jurgis Pasukonis; Jimmy Ba; Timothy Lillicrap

Affiliations: Google DeepMind, San Francisco, CA, USA; University of Toronto, Toronto, Ontario, Canada

Source: Nature 2025 · ref-8fa0ebc722d8d35eaf75 ↗ · Project page ↗ · Catalog record

Reading: 396 / 558 · 6 original figures & tables · ~20 min ·

1. Paper overview

In one sentence: DreamerV3 makes latent-imagination policy learning robust across diverse tasks through balanced losses and scale control, while retaining benchmark-specific resource and environment choices. e-world e-kl e-actor e-transform e-ablation e-scores e-protocol-table e-minecraft e-minecraft-setup

At a glance	What to know
Research problem	Source description RL methods often need substantial retuning when observations, reward scales or action spaces change. Dreamer seeks a reusable learning algorithm across these changes, while each agent still learns from its own environment experience. e-problem e-availability
Core mechanism	Source description Combines categorical latent dynamics with KL balancing, free bits, signed target transformations and percentile return normalization to stabilize world-model and actor–critic learning. e-kl e-transform e-actor
A key reported result	Minecraft Diamond: 9.1 return; 10/10 runs discover diamonds Episode return; runs discovering diamonds. Modified MineRL; 100M environment steps; ten seeds; no human data or curriculum. PPO 5.1, Rainbow 6.3, IMPALA 7.1 return; none discovers diamonds. Discovery across runs does not imply reliable episode success: the plotted diamond episode mean remains below 1%. e-scores e-minecraft e-evaluation e-minecraft-setup
Reading caution	Reader analysis The Minecraft interface combines images with structured state, milestone and health rewards, abstract crafting and accelerated mining. Its success does not establish learning under the unmodified keyboard/mouse game or VPT protocol. e-minecraft-setup

Core contributions

Source description
Combines categorical latent dynamics with KL balancing, free bits, signed target transformations and percentile return normalization to stabilize world-model and actor–critic learning. e-kl e-transform e-actor
Author claim
The authors report strong performance across eight domains, diamond discovery without demonstrations, and improved interaction efficiency when scaling model size or replay. e-scores e-minecraft e-scaling

Two training diagrams: observation encoders and decoders shape discrete recurrent states on the left; a single observed start state branches into imagined states with action, reward and value outputs on the right. — **Figure 1.** A learned latent simulator supplies the trajectories that train the actor and critic. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Source description

RL methods often need substantial retuning when observations, reward scales or action spaces change. Dreamer seeks a reusable learning algorithm across these changes, while each agent still learns from its own environment experience. e-problem e-availability

2.2 What this reading follows

A world model helps control only if its predictions support useful behavior and remain learnable as the task changes. DreamerV3 tackles both requirements: it learns compact action-conditioned dynamics, then uses imagined experience to improve a separate actor and critic. The paper's central contribution is a set of normalization, balancing and transformation choices that work across markedly different domains. Read the architecture alongside the gradient ablations, then interpret the benchmark scores through their protocols. Minecraft provides the clearest warning about metrics: every training run discovers diamonds, yet diamonds remain rare within individual episodes, under an explicitly modified game interface. e-world e-kl e-actor e-transform e-ablation e-scores e-protocol-table e-minecraft e-minecraft-setup

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	Foundational work
Architecture	Not applicable
Prediction paradigm	Not applicable
Quadrant	Not applicable

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The foundational model-based RL classification fits: a learned action-conditioned world model trains distinct actor and critic networks. Concurrent training does not establish a single joint future/action predictor. Architecture, prediction-paradigm and quadrant labels remain not applicable within the supplied catalog taxonomy; this is neither inverse dynamics nor an unverified identity. e-world e-critic e-problem

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
Observation sequences, previous actions, rewards and continuation flags (e-world). Minecraft additionally exposes inventory, milestone, equipped-item and health-related state alongside RGB (e-minecraft-setup).	Predicted latent trajectories, rewards and continuation probabilities; observation reconstructions (e-world). A separately parameterized actor distribution and critic return distribution (e-critic).

4.2 Equations and their role

h_t=f_\phi(h_{t-1},z_{t-1},a_{t-1}),\qquad z_t\sim q_\phi(z_t\mid h_t,x_t),\qquad \hat z_t\sim p_\phi(\hat z_t\mid h_t)

Here x is the observation, a the action, h recurrent memory, z a stochastic latent, and phi the world-model parameters. q encodes observed data; p supplies imagined latents. e-world

\begin{aligned}\mathcal L_{\mathrm{dyn}}&=\max\!\left(1,\mathrm{KL}\!\left[\operatorname{sg}(q_\phi(z_t\mid h_t,x_t))\,\|\,p_\phi(z_t\mid h_t)\right]\right),\\\mathcal L_{\mathrm{rep}}&=\max\!\left(1,\mathrm{KL}\!\left[q_\phi(z_t\mid h_t,x_t)\,\|\,\operatorname{sg}(p_\phi(z_t\mid h_t))\right]\right).\end{aligned}

sg stops gradients. The two terms target different sides of the same KL divergence; weights are 1 and 0.1. Below one nat, their gradients vanish, prioritizing prediction. e-kl

\operatorname{symlog}(x)=\operatorname{sign}(x)\log(1+|x|),\qquad \operatorname{symexp}(x)=\operatorname{sign}(x)(e^{|x|}-1)

These inverse transforms compress large signed scalar magnitudes while remaining approximately identity near zero. They underpin observation preprocessing and the exponentially spaced prediction bins. e-transform

S=\operatorname{EMA}\!\left(\operatorname{Per}(R_t^\lambda,95)-\operatorname{Per}(R_t^\lambda,5),0.99\right)

R with superscript lambda denotes bootstrapped returns; Per is a batch percentile and EMA an exponential moving average. The actor divides its return advantage by max(1,S), retaining a fixed entropy scale. e-actor

5. Method in detail

5.1 First learn what an observation must preserve

Source description

Dreamer's model state combines recurrent memory with a sampled categorical representation of the current observation. This separates information accumulated over time from newly observed detail. Reconstruction makes the state informative even when rewards are sparse, while the dynamics predictor learns which latent states can follow an action. Those objectives can conflict: a representation that preserves every visual detail may be difficult to predict, while a perfectly predictable constant state is useless. The paired KL losses regulate the trade-off by stopping gradients on different sides of the encoder/prior comparison. Their one-nat floor stops demanding further agreement once the mismatch is small. The smaller representation-loss weight limits pressure to discard information. Figure 6 then tests a consequence of this design: removing reconstruction gradients is particularly damaging to performance. e-world e-kl e-ablation

Dreamer defaults grouped into general optimization settings, world-model loss scales, and actor–critic settings including imagination horizon, entropy and return normalization. — **Extended Data Table 5.** The stable recipe couples asymmetric KL regularization with bounded return normalization. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

5.2 Turn simulated experience into a policy that can act directly

Source description

After a replayed observation supplies a starting state, the actor proposes an action and the world model predicts the next latent, reward and continuation signal. Repeating this produces imagined experience without decoding and observing an actual future environment. A critic supplies bootstrapped return estimates beyond the rollout, and also receives a lower-weight learning signal from replay rewards. The actor uses REINFORCE with a stopped return advantage, normalized by a robust percentile range. The denominator cannot fall below one, so very small predicted returns do not become large noisy policy updates. Entropy can therefore continue to drive exploration. At interaction time, the learned actor samples an action directly from the current state. Imagination trains that behavior; it is not a fresh online search at every decision. e-world e-critic e-actor

5.3 Separate broad capability from protocol-dependent conclusions

Reader analysis

The benchmark table supports a strong claim about reusing an algorithm across tasks, but it requires several narrower comparisons. DMLab experts receive more data; Atari and Atari100K aggregate different statistics; model size and replay ratio vary in the protocol table. The Minecraft figure adds another distinction: discovering a diamond at least once across a long training run is different from obtaining one reliably in an episode. Reading these together, my interpretation is that Dreamer demonstrates robust learning and difficult exploration under the supplied interfaces, with clear remaining reliability limits. The reconstruction ablation provides mechanism evidence for the learned representation, but cannot turn the result into reward-free learning or establish physical robot performance. Likewise, a common hyperparameter recipe is not evidence for one trained model that transfers across all domains. e-scores e-evaluation e-atari-protocol e-protocol-table e-minecraft e-minecraft-setup e-ablation e-problem

5.4 Training and inference

During training

Source description

World model, actor and critic train concurrently during interaction. Uniform replay incorporates new online trajectories and refreshes stored latent states. Defaults include 16 sequences of length 64, learning rate 0.00004, LaProp and adaptive gradient clipping; no external pretraining dataset is used. e-world e-implementation e-hyperparameters e-availability

Reader analysis

A single A100 trains each agent. Table 2 varies replay ratio and environment parallelism, and uses 1M parameters for proprioceptive control versus 200M elsewhere. Thus the common loss/optimizer configuration does not imply identical compute allocations. e-evaluation e-protocol-table

During inference

Source description

Update the recurrent state using the preceding action, encode the new observation, and sample the actor action from that state. The environment executes it and supplies feedback. There is no inference-time lookahead or decoded-video-to-action inverse model. e-world e-critic

5.5 Implementation flow

Build a predictive state
An image CNN or vector MLP encodes observations into categorical stochastic latents. A recurrent state summarizes preceding latents and actions; concatenating both supports decoding and reward/continuation prediction. Imagination substitutes the learned latent prior for future observation encodings. e-world e-implementation
Keep representations informative and predictable
Reconstruction supplies task-agnostic supervision. Separate KL terms train the prior toward stopped encoder targets and train the encoder toward a stopped prior. Their asymmetric weights and one-nat floor preserve information without forcing identical regularization pressure across tasks. e-kl
Learn behavior in imagination
Rollouts start from replayed states. Predicted rewards and continuation flags support bootstrapped lambda returns; the critic also learns from replay rewards. REINFORCE uses stopped, normalized return advantages, with entropy encouraging exploration. This produces a policy for subsequent interaction. e-critic e-actor
Control numerical scale
Symlog transforms vector observations. Detailed prediction sections specify symexp two-hot reward/value heads, which classify between neighboring exponentially spaced bins. Return normalization only scales down large ranges, avoiding amplification of small noisy advantages. e-transform e-actor

6. Experiments & results

DreamerV3 learns latent dynamics from experience and trains an actor through imagined trajectories. Its contribution is robust learning across diverse control domains using shared algorithm settings, with benchmark-specific compute choices. Minecraft diamond discovery is demonstrated under a modified MineRL interface. Evidence: e-world, e-actor, e-protocol-table, e-minecraft-setup, e-minecraft.

6.1 Read the original evidence

Table of eight benchmarks with their aggregation metric, Dreamer and PPO scores, and two tuned expert comparisons per row. — **Extended Data Table 1.** Dreamer improves the reported aggregate across every row, but each row has its own metric and protocol. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Minecraft return curve and six item-discovery plots, separating percentage of agents from percentage of episodes for iron ore, iron pickaxe and diamond. — **Figure 5.** All ten runs find diamonds, while diamonds remain rare at the episode level. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Benchmark table listing task counts, environment-step budgets, action repeat, number of environments, replay ratio, GPU days and model size. — **Extended Data Table 2.** The shared learning recipe is evaluated with explicit, benchmark-specific resource allocations. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
Minecraft Diamond Modified MineRL; 100M environment steps; ten seeds; no human data or curriculum.	9.1 return; 10/10 runs discover diamonds Episode return; runs discovering diamonds	PPO 5.1, Rainbow 6.3, IMPALA 7.1 return; none discovers diamonds. Discovery across runs does not imply reliable episode success: the plotted diamond episode mean remains below 1%. e-scores e-minecraft e-evaluation e-minecraft-setup
DMLab 30 tasks; 100M environment steps; five seeds.	71 Capped mean score	PPO 36; IMPALA 66 and R2D2+ 65 at 1B steps. Higher reported aggregate despite a tenfold data advantage for expert baselines; compute is not matched. e-scores e-evaluation e-protocol-table
ProcGen 16 games; hard difficulty, unlimited levels; 50M steps; five seeds.	72 Normalized mean score	PPO 43; PPG 65; Rainbow 55. Tests procedural variation under this training protocol, without establishing a separately held-out finite-level result. e-scores e-evaluation e-protocol-table
Atari 57 games with sticky actions; 200M frames; five seeds.	830 Gamer-normalized median score	PPO 180; MuZero 693; Rainbow 223. Aggregate benchmark performance; individual-game scores are outside the supplied PDF. e-scores e-evaluation e-protocol-table
Atari100K 26 games; 400K frames, or 100K after action repeat; five seeds.	125%; 49% Gamer-normalized mean; median	IRIS 105%/29%; TWM 96%/51%; EffMuZero 190%/109% (mean/median). Dreamer improves the mean over IRIS and TWM. Its median of 49% exceeds IRIS's 29% but falls below TWM's 51%. EffMuZero uses a different protocol; Dreamer does not use separate evaluation episodes. e-scores e-atari-protocol e-evaluation
BSuite 23 environments with 468 configurations; ten seeds; benchmark-specific budgets.	66 Task mean score	PPO 49; Boot DQN 60; DQN 54. Aggregate evidence across targeted RL challenges; Table 2 does not supply one common step budget. e-scores e-evaluation e-protocol-table
Visual Control Suite 20 simulated continuous-control tasks; image observations; 1M steps; five seeds.	802 Task mean return	PPO 206; DrQ-v2 705; TD-MPC2 634. Simulation control performance, not physical robot deployment. e-scores e-evaluation e-protocol-table
Proprioceptive Control Suite The same 20 tasks with vector inputs; 1M steps; 1M-parameter Dreamer; five seeds.	843 Task mean return	PPO 205; DMPO 834; TD-MPC2 825. Numerically close to strong experts; the aggregate table has no uncertainty intervals to establish significance. e-scores e-evaluation e-protocol-table

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Four panels compare robustness removals, representation-learning gradient removals, six model sizes, and seven replay ratios on aggregate tasks, Crafter and DMLab goals. — **Figure 6.** Reconstruction gradients are central to the learned representation, and added compute improves interaction efficiency in the scaling tasks. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

The Minecraft interface combines images with structured state, milestone and health rewards, abstract crafting and accelerated mining. Its success does not establish learning under the unmodified keyboard/mouse game or VPT protocol. e-minecraft-setup

Reader analysis

Figure 6 supports reconstruction-driven representation learning, but removes KL balancing and free bits together. Aggregate curves cannot identify every technique independently. Scaling is tested on Crafter and one DMLab task, not every domain. e-ablation e-scaling

Reader analysis

Shared settings demonstrate algorithmic generality, not a single trained agent transferring across domains. Open-loop video examples are qualitative predictions; physical deployment and internet-video pretraining remain unestablished. e-problem e-availability e-video

Reader analysis

Aggregate score tables lack uncertainty intervals. The main text and hyperparameter table disagree on imagination horizon (16 versus 15); the initial reward-loss description also differs from the later two-hot specification. Neither discrepancy is silently resolved here. e-scores e-source-discrepancies

7.2 Questions for discussion

How much of the reconstruction ablation effect depends on task-irrelevant visual complexity (e-ablation)?
Would the same robustness settings remain effective under an unmodified Minecraft motor interface (e-minecraft-setup)?

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Reproduction needs the listed model dimensions, batch/replay settings, optimizer and environment wrappers. Minecraft requires MineRL v0.4.4, Minecraft 1.11.2, the 25-action mapping and termination/mining modifications. The PDF does not quantify the material-specific block-breaking speed multiplier. e-implementation e-scaling e-hyperparameters e-minecraft-setup

Reader analysis

Authors state that code and curve datapoints are released. Before implementation, resolve the horizon convention and reward-loss wording; use the detailed two-hot specification as the reported design, without claiming the discrepancy is verified away. e-availability e-source-discrepancies

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Does reconstruction improve control through the learned state?

Reader-proposed experiment, not run: on Crafter and DMLab goals, compare full Dreamer with two gradient interventions: stop reconstruction gradients into the representation, or stop reward/value gradients into it. Retain the prediction heads and reward-based actor objective in every condition. Use the same 25M architecture, replay ratio 32, environment budgets, optimizer and five matched seeds; declare the horizon convention explicitly. Measure environment return and held-out latent-prediction loss. A substantially larger control loss after removing reconstruction would support the paper's mechanism. If only prediction loss changes while returns remain similar, the representation-to-control interpretation would be weaker on these tasks. e-ablation e-scaling e-implementation e-critic e-evaluation e-hyperparameters e-source-discrepancies

Check 2: Does return normalization preserve exploration under changed reward units?

Reader-proposed experiment, not run: select one sparse-reward and one dense-reward BSuite configuration and record their exact identifiers before testing. Multiply rewards by 0.1, 1 and 10 without changing transitions or reward timing. Compare full Dreamer against the advantage-normalization substitution shown in Figure 6, fixing model, replay, budgets and matched seeds. Report unscaled task scores, policy entropy and return-normalization denominators. The prediction is relative stability under larger reward units, with continued exploration in the sparse case. Do not demand exact invariance below the denominator limit: that asymmetry is deliberate. Collapse under larger units would challenge the stated normalization benefit. e-actor e-ablation e-evaluation e-protocol-table e-hyperparameters

8.3 Reading coverage

Visual audit: Directly inspected every PDF page: title/author/affiliation block; all method, training, evaluation and implementation pages; Figures 1–6; Extended Data Figures 1–2 and Tables 1–7. All six final crops were separately viewed, with legible headers, axes and legends and without body prose or long captions. Diagram arrows were checked against the state definitions and gradient equations. The repeated final reward/value labels, horizon discrepancy and reward-loss wording difference are preserved. The uncropped video figures also have caption/axis timing ambiguity, so no exact video forecast length is inferred. Extended Data Table 3 on page 15 was directly checked alongside the benchmark crop: IRIS has a 29% median, compared with Dreamer at 49% and TWM at 51%. External Supplementary Information, code and videos remain outside this reading. Visuals are cropped excerpts of Hafner et al., Nature (2025), licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/); only cropping was applied.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Title, abstract and introduction (p. 1)
Learning algorithm: world model, critic, actor and robust predictions (pp. 1–4)
Evaluation, Minecraft, ablations and scaling (pp. 4–6)
Previous work, conclusion and references (pp. 6–7)
Methods: baselines, protocols, implementation and Minecraft environment (pp. 8–10)
Availability and author statements (p. 10)
Extended Data Figures 1–2 and Tables 1–7 (pp. 11–19), including direct visual table reading

Outside the original text pass

Text extraction does not reconstruct figure pixels or preserve equation/table layout; downstream reading must inspect the retained original PDF.
Extended Data Tables 1, 2, 4, 5, 6 and 7 on PDF pages 13, 14, 16, 17, 18 and 19 are largely absent from plain-text extraction. Direct visual reading of the original table pixels is required for benchmark, hyperparameter, model-size and action-space claims.
Availability of external supplements, code, datasets and videos beyond the retained PDF has not been fully verified.
The extraction gaps above were addressed by directly inspecting all 19 PDF pages, including every Extended Data table. No supplied text chunk was skipped.
The reviewed edition is the Nature 2025 article. Title and all four authors match the catalog; no earlier manuscript or other revision was supplied for comparison. The earlier selected-sections acquisition label refers to the same PDF hash, not a different edition.
External Supplementary Information, per-task supplementary curves and reference-score lists were not supplied. Included Methods and Extended Data were reviewed; external supplements were not.
Code, external datasets and videos were not inspected; no experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title, author and affiliation blocks; publication informationInspect

Observed title: Mastering diverse control tasks through world models. Authors: Danijar Hafner, Jurgis Pasukonis, Jimmy Ba and Timothy Lillicrap. Affiliations: Google DeepMind and University of Toronto. Nature 640, 17 April 2025, p. 647; published online 2 April 2025; DOI 10.1038/s41586-025-08744-2.

Go to primary source ↓

e-problemPDF p. 1, abstract and introduction; p. 7, ConclusionInspect

DreamerV3 targets learning across over 150 tasks with fixed algorithm hyperparameters. A single world model across domains and learning from internet videos are future directions.

Go to primary source ↓

e-worldPDF p. 2, Figure 1 and World model learning, state definitionsInspect

The recurrent state depends on previous state, latent and action. The encoder conditions on current observation and recurrent state; the dynamics predictor generates categorical latents without future observations. Decoder, reward and continuation heads read the model state.

Go to primary source ↓

e-klPDF p. 2, World model learning, objective and three loss definitions; p. 3, opening paragraphInspect

Prediction, dynamics and representation losses have weights 1, 1 and 0.1. Dynamics and representation KL terms use opposite stop-gradient placements and a one-nat floor. Categorical distributions mix 1% uniform probability with 99% network output.

Go to primary source ↓

e-criticPDF p. 3, Critic learning, actor/critic definitions and return equationsInspect

The actor samples environment actions without lookahead. Imagined states train return prediction; the text states horizon T=16 and discount 0.997. Critic losses use imagined and replay trajectories at scales 1 and 0.3, with EMA regularization and zero-initialized output weights.

Go to primary source ↓

e-actorPDF p. 3, Actor learning, surrogate objective and percentile normalization; p. 4, continuationInspect

REINFORCE trains discrete and continuous policies. The actor divides its return advantage by max(1,S), where S is an EMA of the 95th-minus-5th return percentiles with decay 0.99; entropy scale is 0.0003.

Go to primary source ↓

e-transformPDF p. 4, Robust predictions, symlog/symexp and two-hot equations; p. 9, DistributionsInspect

Symlog compresses signed magnitudes. Rewards and critic returns use categorical predictions on exponentially spaced bins and two-hot targets. Expected predictions are weighted bin averages; output weights start at zero. Positive and negative bin contributions require careful summation.

Go to primary source ↓

e-implementationPDF pp. 8–9, Implementation: Model sizes, Networks, Optimizer and Experience replayInspect

The implementation uses block-diagonal GRUs with eight blocks, RMSNorm, SiLU, adaptive gradient clipping and LaProp. Uniform replay includes an online queue; latent states are stored and refreshed. Replay ratio counts trained time steps relative to collected steps before action repeat.

Go to primary source ↓

e-evaluationPDF pp. 5 and 8, Benchmarks, Protocols, Seeds and error bars, Computational choicesInspect

Atari uses sticky actions; ProcGen uses hard difficulty and unlimited levels. BSuite includes 23 environments with 468 configurations. DMLab expert baselines receive ten times the data. Dreamer/PPO use five seeds per benchmark except ten for BSuite and Minecraft. Curves show mean and one standard deviation; each agent trains on one Nvidia A100.

Go to primary source ↓

e-scoresPDF p. 13, Extended Data Table 1, all benchmark rows and score columnsInspect

Dreamer/PPO scores are Minecraft 9.1/5.1, DMLab 71/36, ProcGen 72/43, Atari 830/180, Atari100K 125/11, BSuite 66/49, DMC Vision 802/206 and DMC Proprio 843/205. Tuned expert values and row-specific aggregation labels are printed beside them; no uncertainty intervals appear in this table.

Go to primary source ↓

e-protocol-tablePDF p. 14, Extended Data Table 2, benchmark rows and all protocol columnsInspect

Budgets are Minecraft/DMLab 100M, ProcGen 50M, Atari 200M, Atari100K 400K and both DMC variants 1M environment steps; BSuite has a dash. Replay ratios range from 32 to 1024. DMC Proprio uses 1M parameters; other rows use 200M. Minecraft uses 64 environments and 8.9 GPU days.

Go to primary source ↓

e-atari-protocolPDF p. 15, Extended Data Table 3, Gamer score (%) and Gamer median (%) rows, IRIS/TWM/Dreamer/EffMuZero columns and protocol rows; p. 5, Atari100k paragraphInspect

Mean/median gamer scores are Dreamer 125%/49%, IRIS 105%/29%, TWM 96%/51% and EffMuZero 190%/109%. The table marks early resets and life information for EffMuZero and IRIS, life information for TWM, and no separate evaluation episodes for Dreamer. Protocols differ across methods.

Go to primary source ↓

e-minecraftPDF pp. 5–6, Minecraft and Figure 5; p. 8, Seeds and error bars; p. 12, Extended Data Figure 2Inspect

All ten Dreamer runs discover diamonds within 100M environment steps; compared PPO, Rainbow and IMPALA runs do not. Figure 5 separates the fraction of agents ever discovering an item from episode success; the diamond episode axis spans 0–1%, with mean below 1%.

Go to primary source ↓

e-minecraft-setupPDF pp. 9–10, Minecraft: Learning environment, Observations and rewards, Action space, Break speed; p. 19, Extended Data Table 7Inspect

Minecraft 1.11.2 runs on MineRL v0.4.4 with modified termination and faster block breaking. Inputs include 64×64 RGB plus inventory, milestone, equipped-item and health-related vectors/scalars. Twelve once-per-episode item rewards are +1, with health changes rewarded at ±0.01. Twenty-five actions include abstract crafting.

Go to primary source ↓

e-ablationPDF p. 5, Ablations; p. 6, Figure 6a–b and captionInspect

Across 14 tasks, removing robustness techniques reduces mean performance. KL balancing and free bits are removed together. Stopping reconstruction gradients harms performance much more than stopping reward/value gradients into representations; exact per-task curves are referred to Supplementary Information.

Go to primary source ↓

e-scalingPDF pp. 5–6, Scaling properties and Figure 6c–d; p. 16, Extended Data Table 4Inspect

Crafter and DMLab goals experiments compare six sizes from 12M to 400M parameters and replay ratios from 1 to 64. Larger sizes and more replay improve performance and interaction efficiency. Table 4 additionally specifies a 1M model; the 200M model has hidden size 1024 and 8192 recurrent units.

Go to primary source ↓

e-hyperparametersPDF p. 17, Extended Data Table 5, General, World Model and Actor Critic blocksInspect

Defaults include replay capacity 5M, batch size 16, batch length 64, learning rate 0.00004, AGC(0.3), LaProp, KL scales 1/0.1 and one free nat. The table lists imagination horizon H=15, lambda=0.95, entropy scale 0.0003 and return normalization limit 1.

Go to primary source ↓

e-baselinePDF p. 8, Baselines: PPO; p. 18, Extended Data Table 6Inspect

PPO uses an Acme implementation and an IMPALA network, with configuration checked against published ProcGen performance. Its table specifies observation/reward and advantage normalization, entropy scale 0.01, discount 0.997 and learning rate 0.0003.

Go to primary source ↓

e-availabilityPDF p. 10, Data availability, Code availability and Supplementary informationInspect

Experience is generated by interacting with simulated environments, without external datasets. Authors state that code, experiment implementations and curve datapoints are available in the linked repository; additional supplementary material is linked separately.

Go to primary source ↓

e-source-discrepanciesPDF p. 2, Figure 1 and prediction-loss prose; pp. 3–4, Critic learning and Robust predictions; p. 9, Distributions; p. 17, Table 5Inspect

Figure 1 repeats r2 and v2 above the final h3/latent column. The main text says prediction horizon T=16, whereas Table 5 lists H=15. Page 2 describes reward prediction with symlog squared loss; pages 4 and 9 specify symexp two-hot reward prediction. These differences are preserved rather than silently reconciled.

Go to primary source ↓

e-videoPDF p. 4, Figure 3 and caption; p. 11, Extended Data Figure 1 and captionInspect

Open-loop video illustrations compare true and predicted frames after context input, conditioned on supplied actions. Both captions describe five context frames and 45 future frames/steps, while displayed frame labels extend to 60; no quantitative prediction-error evaluation is provided in these figures.

Go to primary source ↓

8.5 Primary sources

Mastering diverse control tasks through world models ↗

PDF · 10,717 extracted words

Source fingerprint

b34e999951cb5b8b84594f3936b1c395e9d51e563e0932273e9197cee66e37ef