Mastering Atari with Discrete World Models
1. Paper overview
In one sentence: Discrete latent predictions and balanced KL learning make a separately trained world model useful for Atari policy learning, while leaving important per-game failures and configuration ambiguities. e-identitye-cyclee-rssme-imaginatione-maine-ablatione-failurese-ranking
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Can a separately learned visual world model become accurate enough for behavior learned inside it to compete with strong model-free Atari agents? The target is useful latent prediction for control: reconstructing attractive frames alone would not demonstrate that imagined rewards and transitions support effective actions. e-cyclee-rssme-main |
| Core mechanism | Source description DreamerV2 replaces Gaussian stochastic states with multiple categorical variables and balances the prior/posterior sides of the KL gradient. Its actor and critic remain separate from world-model optimization. e-discretee-losse-imagination |
| A key reported result | Atari 55-game aggregate performance: Default DreamerV2: 2.15 / 11.33 / 0.44 / 0.28. Gamer-normalized median / mean; record-normalized mean / clipped mean (higher is better). Separate agents per game, 200M frames, action repeat 4, sticky actions, full action space, no life information or frame stacking, and a 108,000-frame episode limit. Dopamine supplies baseline scores. IQN: 1.29 / 8.85 / 0.21 / 0.21; Rainbow: 1.47 / 9.12 / 0.17 / 0.17. Scheduled DreamerV2: 2.64 / 10.45 / 0.43 / 0.28. Table 1 favors default DreamerV2 on all four aggregates. These are dimensionless normalized scores, not success percentages. The scheduled variant improves median but not clipped mean; Table 1 gives no numerical uncertainty intervals. e-evale-maine-metrics |
| Reading caution | Reader analysis Aggregate superiority is not per-game dominance. Table K.1 gives Video Pinball 41,860 versus IQN 415,833 and Rainbow 466,895, and Venture 2 versus 1,313 and 1,529. The one-pixel-ball explanation for Video Pinball is an author hypothesis, not a tested diagnosis. e-failures |
Core contributions
- Source description
DreamerV2 replaces Gaussian stochastic states with multiple categorical variables and balances the prior/posterior sides of the KL gradient. Its actor and critic remain separate from world-model optimization. e-discretee-losse-imagination
- Author claim
The authors report the first human-level Atari aggregate performance from behavior learned purely inside a separately trained world model, and recommend clipped world-record normalization to reduce domination by outlier games. e-identitye-maine-metrics
Figure 2. Observed images teach a recurrent model to predict the state it will need when images are unavailable. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column upward from the observed image x to the blue encoder and posterior state z. The purple recurrent state h carries previous information and actions horizontally; it also feeds the prior state marked with a hat. The dashed green connection labeled min KL compares posterior and prior distributions, rather than sending an observed image into the prior. Brown and orange branches reconstruct images and predict rewards from the posterior model state. The right inset depicts categorical samples; its labels specify 32 variables with 32 classes each. Equation (1) confirms that the recurrent update uses the previous stochastic sample and action. e-rssme-discretee-loss
What it supports. The important distinction is informational: the posterior can see the current frame, while the prior must predict without it. KL training connects those two distributions so the same learned state space can later support imagination. The discrete code is a component of the state; deterministic recurrent memory is retained.
Where the evidence stops. The discount predictor is defined in Equation (1) but omitted from this diagram. The dashed min KL link also omits gradient asymmetry: Algorithm 2 stops posterior gradients in one term and prior gradients in the other. Those training details come from the text.
2. Motivation
2.1 The problem and the proposed response
Can a separately learned visual world model become accurate enough for behavior learned inside it to compete with strong model-free Atari agents? The target is useful latent prediction for control: reconstructing attractive frames alone would not demonstrate that imagined rewards and transitions support effective actions. e-cyclee-rssme-main
2.2 What this reading follows
DreamerV2 asks whether an agent can learn effective behavior from a world it predicts internally. Its answer has two stages: compress observed images into recurrent discrete states, then learn an actor and critic from trajectories generated by the learned dynamics. The figures below separate those stages before examining Atari scores and the ablations that support the design. The headline result is an aggregate over independently trained games, so it should be read alongside failures and evaluation details. This edition follows the supplied February 2022 arXiv v4 copy of the ICLR 2021 paper and preserves its unresolved reporting differences. e-identitye-cyclee-rssme-imaginatione-maine-ablatione-failurese-ranking
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational model-based RL classification is supported. DreamerV2 has a separately optimized world model and actor/critic; it predicts action-conditioned latent transitions and learns actions by return optimization. It is neither a unified joint future/action predictor nor an inverse-dynamics action extractor. The catalog’s Not applicable architecture, prediction paradigm, and quadrant fields are retained as scope decisions, not treated as missing verification. e-cyclee-rssme-imaginatione-policy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the latent prior learn from a better-informed posterior
Reader interpretation: the central modeling difficulty is an information mismatch. During training, the posterior sees the current image and can encode changes that the temporal prior has not yet learned to anticipate. During imagination, only that prior is available. Equation (2) ties them together while reconstruction and reward prediction make the representation useful. A symmetric KL update can reduce disagreement partly by making the posterior less informative; Algorithm 2 deliberately favors learning the prior instead. Its stop-gradient placement allocates more weight to prior learning without simply multiplying every model loss. Categorical states then provide the discrete representation being predicted. The paper tests the usefulness of both choices, but its proposed explanations involving sparsity, aggregate-posterior matching, and discontinuous game events remain hypotheses rather than isolated causal findings. e-rssme-losse-discretee-ablation
Figure 3. After initialization from experience, actor actions drive latent trajectories used to train behavior. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the only image, x1, at the lower left. It produces a posterior state; subsequent green circles are hatted prior samples. Red branches emit actions, which bend into the next recurrent update together with the preceding stochastic state. Orange branches predict rewards and yellow branches estimate values. There is no image decoder along the imagined rollout. The drawing shows a short example, while Section 2.2 uses a horizon of 15. The world model is fixed during this stage. The actor and critic consume the concatenated recurrent and stochastic state, even where the equations abbreviate the compact state notation. e-cyclee-rssme-imaginatione-policye-discrete
What it supports. Latent imagination supplies many trajectories without rendering future images. The critic combines imagined rewards with bootstrapped future value, and the actor learns to improve those returns. This architecture separates learning a predictive environment representation from optimizing behavior within it; actor and critic gradients do not update world-model parameters during behavior learning.
Where the evidence stops. The caption permits REINFORCE, straight-through policy gradients, or mixtures. The default Atari configuration specifically uses REINFORCE only; this differs from straight-through training of discrete world-model states. Real environment execution still requires observations and actor actions, as described in Section 2.
5.2 Turn short imagined rollouts into a long-horizon policy
Once the world model has been updated from replay, its parameters are fixed for behavior learning. Posterior states initialize imagined trajectories; sampled actor actions and the learned prior generate later states. Reward and discount predictors provide the learning signals that the environment would otherwise supply. A critic extends the horizon by estimating rewards beyond the explicitly imagined sequence, and lambda-returns combine intermediate predictions with these value estimates. The actor optimizes the resulting returns with entropy regularization. For default Atari, its gradient estimator is REINFORCE, even though categorical model states were learned using straight-through differentiation. The actor is then executed in the actual environment to collect more data. Thus, a successful imagined policy is tested through real game interaction; a latent rollout or a reconstructed frame is not itself an executed action. e-cyclee-imaginatione-policye-discretee-changes
Table D.1. The appendix exposes the training recipe and a discount mismatch that matters for reproduction. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three blocks as different parts of one training loop. World Model specifies replay capacity, sequence sampling dimensions, discrete state size, RSSM width, KL scale beta, balancing alpha, and reward transformation. Behavior sets imagination horizon H, discount gamma, lambda-return mixing, actor-gradient mixing rho, entropy scale eta, and actor/critic learning rates. Common lists update frequency and optimizer controls. The row labeled Policy steps per gradient step connects real interaction to learning workload. This is an Atari table; Appendix A changes the actor distribution and settings for Humanoid Walk, so continuous-control results should not inherit these values automatically. e-hypere-replaye-losse-policye-humanoide-montezuma
What it supports. The table makes important reproduction choices concrete: 50 sequences of length 50, 32 categorical variables with 32 classes, horizon 15, and different learning rates for model, actor, and critic. It also demonstrates why a methods summary cannot safely substitute for the full configuration: the listed discount is 0.995.
Where the evidence stops. Section 2.1 instead states discount 0.999. The source does not resolve which value generated each result. Appendix B separately uses 0.99 for Montezuma’s Revenge. The table also lacks a complete software-version specification; this edition preserves those gaps.
5.3 Separate aggregate strength from mechanism and coverage
Reader interpretation: the results answer three different questions. Table 1 asks whether the complete agent competes with strong model-free baselines under the selected Atari protocol. Table 2 asks which removals damage an earlier implementation. Table K.1 asks where the aggregate hides failures. These questions should not be collapsed into a claim that every game is mastered or that one modification explains the entire improvement. The appendix strengthens this reading: Figure G.1 identifies the smaller seed count for latent/KL ablations, Figure I.1 shows task-specific policy-gradient effects, and Figure J.1 compares actor-driven collection with random data. Humanoid Walk demonstrates continuous simulated control under changed settings, while Montezuma’s Revenge improves under a separately adjusted discount. Neither extension establishes transfer between games or deployment on physical robots. e-maine-ablatione-failurese-task-scorese-abl-seedse-policy-ablatione-extra-ablatione-humanoide-montezumae-scope
5.4 Training and inference
During training
Replay batches contain 50 sequences of length 50. Sequence starts are sampled within episodes and clipped to retain enough episode endings. Atari uses KL scale 0.1; the default actor uses REINFORCE only, with mixing parameter rho = 1 and entropy scale 0.001. This does not remove straight-through gradients from world-model training. e-replaye-losse-policye-discrete
The paper reports under 10 days per 200M-frame run on one NVIDIA V100 and one environment instance, with 468B imagined compact states versus 50M observed inputs after action repeat. The model has 20M parameters and the actor and critic have 1M each. e-computee-losse-imagination
During inference
During collection, the current image updates the posterior state, the actor chooses an environment action, and the resulting experience grows replay. Imagination is used for policy learning; actual rewards and observations arrive from the environment. Appendix C describes policy-entropy exploration instead of added external action noise. e-cyclee-rssme-imaginatione-changes
5.5 Implementation flow
- Encode experience into recurrent state
A CNN and MLP infer the posterior stochastic state from the current image and GRU state. The GRU carries the previous deterministic state, stochastic sample, and action forward. Concatenating deterministic and stochastic states provides the representation used by predictors and behavior networks. e-rssm
- Learn a discrete predictive state
The stochastic component contains 32 categoricals with 32 classes each: a flattened sample has 1024 entries and 32 active bits. Straight-through differentiation keeps discrete forward samples while using probability gradients. The temporal prior predicts the stochastic state without the current image. e-discretee-rssm
- Fit predictions and balance representation regularization
The model jointly minimizes image, reward, and discount negative log-likelihoods plus a scaled posterior-to-prior KL. Image and reward likelihoods are unit-variance Gaussians; discounts use a Bernoulli likelihood. KL balancing weights prior learning more strongly than posterior regularization. e-loss
- Learn behavior from imagined trajectories
Freeze world-model parameters, start from replay-derived posterior states, and roll out actor actions through the prior for 15 steps. Predicted rewards and discounts feed a bootstrapped lambda-return. The critic regresses stopped-gradient targets; cumulative predicted discounts weight actor and critic losses to account for termination. e-imaginatione-policy
6. Experiments & results
DreamerV2 learns a categorical latent dynamics model from images, then trains a separate actor and critic entirely on imagined latent trajectories. Its reported Atari advantage concerns aggregate performance under a 200M-frame, sticky-action protocol. Discrete latents, KL balancing, and image-derived representation gradients are important ingredients; the experiments do not establish universal game mastery or physical robot control.
6.1 Read the original evidence
Table 1. Four aggregation rules favor DreamerV2 in Table 1, but they measure different aspects of Atari performance. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across one row before comparing methods. The first two columns normalize against a professional gamer and summarize games by median or mean. The last two use world-record normalization, with the rightmost capping scores at one before averaging across games. That cap prevents exceptional performance on a few games from indefinitely increasing the aggregate. Keep the default DreamerV2 row separate from the scheduled variant. Section 3 specifies 55 single-task agents, sticky actions, full actions, and 200M frames with action repeat four. The baseline cells come from Dopamine, not necessarily from the original baseline papers’ deterministic Atari evaluations. e-evale-maine-metricse-rankinge-task-scores
What it supports. Default DreamerV2 reaches gamer median 2.15 and clipped record mean 0.28, compared with IQN’s 1.29 and 0.21, and Rainbow’s 1.47 and 0.17. Scheduling raises the gamer median to 2.64 but leaves clipped mean at 0.28. These are normalized aggregates, not the percentage of games solved or a guarantee of per-game superiority.
Where the evidence stops. Read these cells literally: p. 8’s prose and Figure F.1’s baseline ordering do not fully agree with Table 1/Figure 4. No correction is inferred. Table 1 lacks numerical confidence intervals; Table K.1 requires aggregation before seed averaging, and two record values were substituted.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Atari 55-game aggregate performance Separate agents per game, 200M frames, action repeat 4, sticky actions, full action space, no life information or frame stacking, and a 108,000-frame episode limit. Dopamine supplies baseline scores. | Default DreamerV2: 2.15 / 11.33 / 0.44 / 0.28. Gamer-normalized median / mean; record-normalized mean / clipped mean (higher is better) | IQN: 1.29 / 8.85 / 0.21 / 0.21; Rainbow: 1.47 / 9.12 / 0.17 / 0.17. Scheduled DreamerV2: 2.64 / 10.45 / 0.43 / 0.28. Table 1 favors default DreamerV2 on all four aggregates. These are dimensionless normalized scores, not success percentages. The scheduled variant improves median but not clipped mean; Table 1 gives no numerical uncertainty intervals. e-evale-maine-metrics |
| Atari mechanism ablations 200M frames; a slightly earlier DreamerV2 implementation, with one component removed per ablation. Figure G.1 reports two seeds for latent/KL comparisons. | Reference 0.25; no discrete latents 0.19; no KL balancing 0.16; no policy REINFORCE 0.15; no image gradients 0.01; no reward gradients 0.24. Clipped record-normalized mean (higher is better) | Compare against the local 0.25 reference, not Table 1’s 0.28. No layer normalization also scores 0.25. Image gradients are critical here. Stopping reward gradients means blocking their effect on representations, not deleting the reward predictor. The ablations establish utility, not why categorical states help. e-ablatione-abl-seedse-representation |
| Humanoid Walk from pixels DeepMind Control Suite simulation; 21-dimensional continuous actions and a truncated-normal actor. | The authors report learning both stand-up and subsequent walking; no exact endpoint score is tabulated. Reported stand-up and walking behavior; reward learning curve | Appendix A provides one DreamerV2 curve, without a comparator curve or a stated seed count. This extends applicability to continuous simulated control. It is not evidence of physical robot deployment. e-humanoid |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Image-derived representations, categorical states, and KL balancing are consequential in the earlier ablation implementation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row as the reference for every removal. In the rightmost column, follow the progression from 0.25 to 0.19 without discrete latents, 0.16 without KL balancing, and 0.15 without policy REINFORCE. Then contrast the image-gradient and reward-gradient rows: stopping image gradients gives 0.01, while stopping reward gradients gives 0.24. Section 3.2 defines these interventions as stopping gradients before they reach model states. The reward predictor still supplies imagined rewards. The original caption identifies this as an earlier implementation than Table 1; its 0.25 baseline must not be silently replaced with the later 0.28. e-ablatione-representatione-abl-seedse-changes
What it supports. The strongest contrast concerns representation learning: high-dimensional image prediction supplies a learning signal that reward gradients alone fail to replace in this setup. Discrete latents and KL balancing also improve aggregate control performance. Neither removal establishes which hypothesized property of categorical distributions causes the advantage.
Where the evidence stops. Figure G.1 reports only two seeds for latent/KL ablations. The table does not report uncertainty intervals, and all changes were not comprehensively ablated. Small differences, such as 0.25 versus 0.24, should not be treated as proof of statistical equivalence or improved transfer.
7. Analysis & limitations
7.1 What the evidence leaves open
Aggregate superiority is not per-game dominance. Table K.1 gives Video Pinball 41,860 versus IQN 415,833 and Rainbow 466,895, and Venture 2 versus 1,313 and 1,529. The one-pixel-ball explanation for Video Pinball is an author hypothesis, not a tested diagnosis. e-failures
The cause of the categorical advantage remains open: posterior matching, sparsity, optimization, and non-smooth dynamics are proposed explanations. Appendix C explicitly says a comprehensive ablation of all changes was computationally infeasible. Transfer, multitask learning, and physical robotics remain prospective directions. e-ablatione-changese-scope
Reporting inconsistencies remain unresolved: p. 8’s IQN/Rainbow prose conflicts with Table 1’s gamer mean, and Figure F.1’s record-panel ordering conflicts with Table 1/Figure 4. Numerical comparisons here follow Table 1. Two games also use substituted world-record values. e-rankinge-metrics
7.2 Questions for discussion
- Does KL balancing improve control through better prior prediction, or primarily through changing representation regularization?
- Would the image-gradient advantage persist under visual distractors or changed reward functions?
8. Reproducibility audit
8.1 Requirements and known gaps
Table D.1 specifies a 2M-entry FIFO dataset, 600 RSSM units, model/actor/critic learning rates of 0.0002/0.00004/0.0001, tanh reward transformation, and four policy steps per gradient step. Reproduce environment preprocessing and separate per-game training before comparing aggregates. e-hypere-eval
Resolve configuration differences explicitly: Section 2.1 gives discount 0.999, while Table D.1 gives 0.995. General continuous-control settings give beta = 1 and eta = 0.0001, but Humanoid Appendix A uses beta = 2 and eta = 0.00001. Appendix B’s Montezuma result changes discount to 0.99 and is not the default benchmark result. e-replaye-hypere-losse-policye-humanoide-montezuma
Use seed-level data: Table K.1 requires aggregation before seed averaging. Figure F.1 reports five seeds and additional ten-episode averaging for IQN/Rainbow; Figure G.1 uses two seeds. The supplied paper does not pin an executable software environment or fully explain these evaluation asymmetries. e-task-scorese-uncertaintye-abl-seedse-hyper
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does KL balancing improve prediction that matters for control?
Reader-proposed check, not executed: on Assault, Seaquest, and Video Pinball, train categorical world models on identical fixed replay splits with Algorithm 2 balancing enabled or with the standard KL control. Hold architecture, beta, optimizer budget, and preprocessing fixed, and document gradient scales. Across five paired seeds, measure held-out prior prediction loss and imagined-versus-environment return error for fixed action sequences. Then train otherwise identical actors in each frozen model. Consistently better return calibration and environment scores would support the proposed dynamics explanation; lower training KL without those gains would weaken it. This isolates a mechanism and is not a full online Atari reproduction. e-losse-ablatione-rssme-policy-ablatione-failurese-task-scores
Check 2: Resolve the discount discrepancy without mixing protocols
Reader-proposed check, not executed: compare discounts 0.995 and 0.999 on Montezuma’s Revenge and a dense-reward control game, with a third 0.99 arm matching Appendix B. Hold all other Atari settings, five seeds, and the 200M-frame budget fixed. Record undiscounted environment scores and value-target variability, retaining seed-level results. A repeatable sparse-game benefit at 0.99 would support the appendix’s stabilization hypothesis; overlapping score distributions would not. Report each configuration separately and aggregate tasks before averaging seeds, rather than reconstructing aggregate metrics from Table K.1’s already averaged cells. e-replaye-hypere-montezumae-policye-evale-task-scorese-uncertainty
8.3 Reading coverage
Visual audit: All eight text chunks were read. Visually inspected the title and revision stamp; Figures 1–5, A.1–A.2, B.1–B.2, E.1–J.1; Tables 1–3, D.1, and K.1; and all method, evaluation, compute, and configuration pages supporting retained claims. Figure 2 and Figure 3 arrows were checked against Equation (1), Algorithms 1–2, and Section 2.2; omitted discount heads and gradient choices are explained rather than inferred from the diagrams. Five final original crops were individually inspected for complete labels and legibility. The narrow hyperparameter crop was rendered at 400 DPI. References on pp. 12–15 were read as text only; separate supplements, code, videos, and raw runs remain outside this review. Configuration and table/prose/plot disagreements are preserved in the report and visual cautions.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (p. 1)
- 1 Introduction (pp. 1–2)
- 2 DreamerV2; 2.1 World Model Learning; 2.2 Behavior Learning (pp. 2–6)
- 3 Experiments; 3.1 Atari Performance; 3.2 Ablation Study (pp. 6–10)
- 4 Related Work; 5 Discussion (pp. 10–11)
- References (pp. 12–15)
- Appendices A–B: Humanoid from Pixels; Montezuma’s Revenge (pp. 16–17)
- Appendices C–D: Summary of Modifications; Hyper Parameters (pp. 18–19)
- Appendices E–K: agent comparisons, latent/representation/policy/additional ablations, and Atari task scores (pp. 20–26)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Recovered-edition limitation: arXiv v4 author-deposited copy. Page 1 says Published as a conference paper at ICLR 2021. This arXiv revision is dated 2022; byte-for-byte equivalence to the blocked ICLR PDF was not checked. Title and all four authors match.
- The extraction limitation was addressed by visually inspecting all scientific pages, pp. 1–11 and 16–26, and five final original crops. References on pp. 12–15 were read as text only.
- Identity notes: the observed title and four authors match the catalog; the title page explicitly identifies arXiv:2010.02193v4, 12 February 2022, and ICLR 2021. This report reviews that revision, not a verified byte-identical copy of the venue PDF.
- No separate supplements, code, project videos, or raw seed-level training logs were inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author/affiliation block, arXiv margin stamp, and Abstract
Title and authors are Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba; affiliations are Google Research, DeepMind, Google Research, and University of Toronto. The stamp identifies v4, 12 Feb 2022; the header identifies ICLR 2021.
Go to primary source ↓e-cyclePDF pp. 1–2, Abstract, Introduction, and Section 2 opening paragraph
The algorithm alternates world-model learning from experience, actor/critic learning in imagination, and actor execution to grow the dataset.
Go to primary source ↓e-replayPDF p. 2, Section 2.1, Experience dataset
Batches use 50 sequences of length 50; sequence starts are clipped near episode ends. The prose specifies discount 0.999 within episodes and zero at terminals.
Go to primary source ↓e-rssmPDF pp. 2–3, Model components; Figure 2 and caption; Equation (1); Neural networks
A CNN/MLP posterior receives the image and recurrent state; the GRU uses previous state and action. The prior lacks the current image. Predictors reconstruct images and predict rewards/discounts; inputs are resized to 64 × 64.
Go to primary source ↓e-discretePDF p. 3, Figure 2 and Algorithm 1; p. 4, Distributions; p. 9, Categorical latents
The stochastic state comprises 32 categorical variables with 32 classes; its flattened sample has 1024 entries with 32 active bits. Algorithm 1 supplies straight-through probability gradients.
Go to primary source ↓e-lossPDF p. 4, Distributions, Equation (2), KL balancing, and Algorithm 2
Unit-variance Gaussian image/reward likelihoods and Bernoulli discount likelihood train with a KL penalty. Atari beta is 0.1, general continuous-control beta is 1; alpha is 0.8. Stop-gradient placement separates prior and posterior updates. The world model has 20M parameters.
Go to primary source ↓e-imaginationPDF pp. 4–5, Section 2.2, Figure 3 and caption, Equations (3)–(4)
The fixed world model supports 2500 parallel imagined trajectories starting from posterior states, horizon 15, predicted rewards/discounts, actor and critic MLPs of 1M parameters each, and lambda = 0.95 targets.
Go to primary source ↓e-policyPDF p. 6, Equations (5)–(6) and surrounding critic/actor text
The critic regresses stopped targets using a target network updated every 100 gradients. The actor combines REINFORCE and dynamics gradients with entropy regularization; default Atari rho = 1, eta = 0.001; general continuous control rho = 0, eta = 0.0001.
Go to primary source ↓e-computePDF p. 6, Section 3 opening paragraph; p. 10, Table 3, DreamerV2 row
The authors report one NVIDIA V100, one environment, under ten days to 200M frames, 468B imagined states and 50M real inputs after repeat. Table 3 lists 22M total parameters.
Go to primary source ↓e-evalPDF p. 7, Experimental setup and Model-free baselines
The benchmark uses 55 separately trained games, 200M frames, action repeat 4, full actions, sticky actions, no life information, no frame stacking, and a 108,000-frame time limit. Baselines come from Dopamine.
Go to primary source ↓e-mainPDF p. 7, Table 1, all rows and four metric columns
Default DreamerV2 scores 2.15, 11.33, 0.44, 0.28; scheduled variant 2.64, 10.45, 0.43, 0.28; IQN 1.29, 8.85, 0.21, 0.21; Rainbow 1.47, 9.12, 0.17, 0.17; C51 1.09, 7.70, 0.15, 0.15; DQN 0.65, 2.84, 0.12, 0.12. No intervals are tabulated.
Go to primary source ↓e-metricsPDF p. 8, Section 3.1 aggregation bullets, schedules paragraph, and Individual games
Gamer median/mean and world-record mean have different outlier sensitivities; record scores are clipped above at 1 before task averaging. The authors substitute values for two games lacking registered records; schedules anneal actor entropy and gradient mixing.
Go to primary source ↓e-ablationPDF pp. 8–9, Figure 5 and caption; Table 2, all rows; Section 3.2
Earlier-version ablations remove one component. Clipped means are 0.25 reference, 0.25 without layer norm, 0.24 without reward gradients, 0.19 without discrete latents, 0.16 without KL balancing, 0.15 without policy REINFORCE, and 0.01 without image gradients. Explanations for categorical benefits are hypotheses.
Go to primary source ↓e-abl-seedsPDF p. 22, Figure G.1 caption
Latent/KL ablations use an earlier agent version and show mean and standard deviation across two seeds.
Go to primary source ↓e-representationPDF p. 9, Model gradients; p. 23, Figure H.1 and caption
The interventions stop image or reward gradients before model states. Reward prediction remains available when its representation gradients are stopped; the image signal is critical in these experiments.
Go to primary source ↓e-uncertaintyPDF p. 21, Figure F.1 caption
Curves show mean and standard deviation over five seeds; IQN and Rainbow additionally average each point over ten evaluation episodes.
Go to primary source ↓e-rankingPDF p. 7, Figure 4 and Table 1; p. 8, paragraph beginning From Figure 4; p. 21, Figure F.1 record panels and legend
The prose says IQN exceeds Rainbow in the remaining aggregations, but Table 1 gamer means are 8.85 and 9.12. Figure F.1 places the green Rainbow trace above orange IQN on record aggregates, unlike Figure 4 and Table 1. The supplied source does not reconcile these differences.
Go to primary source ↓e-humanoidPDF p. 16, Appendix A, Figures A.1–A.2 and text
Humanoid Walk is a simulated Control Suite task with 21 continuous action dimensions and pixel inputs. A truncated-normal actor uses rho = 0, eta = 0.00001, beta = 2. Stand-up and walking are reported; the curve has no comparator, tabulated endpoint, or stated seed count.
Go to primary source ↓e-montezumaPDF p. 17, Appendix B and Figure B.2; p. 26, Table K.1, Montezuma Revenge row
Appendix B changes discount to 0.99 and compares against curiosity-enhanced Rainbow; the default Table K.1 score is 81, versus 500 for both IQN and Rainbow. These are different configurations.
Go to primary source ↓e-changesPDF p. 18, Appendix C, modification lists and final paragraph
Changes include categorical latents, KL balancing, REINFORCE, increased model size, and entropy-based exploration. Full ablation of every change was infeasible; schedules and layer normalization yielded marginal or task-dependent benefits.
Go to primary source ↓e-hyperPDF p. 19, Table D.1, World Model, Behavior, and Common blocks
The table supplies dataset capacity, 50 × 50 batches, 32 × 32 latent structure, 600 RSSM units, tanh rewards, learning rates, horizon 15, discount 0.995, lambda 0.95, update ratio and optimizer settings. It is not a complete software-version manifest.
Go to primary source ↓e-failuresPDF p. 8, Individual games; p. 26, Table K.1, Video Pinball and Venture rows
Video Pinball: DreamerV2 41860, IQN 415833, Rainbow 466895. Venture: 2, 1313, 1529. The tiny-ball explanation is explicitly hypothesized.
Go to primary source ↓e-task-scoresPDF p. 26, Table K.1 caption
Individual-game scores are already seed-averaged; the caption requires task aggregation before seed averaging.
Go to primary source ↓e-scopePDF p. 7, Experimental setup; p. 11, Discussion
Experiments use single-task training; transfer, multitask learning, physical robots, and uncertainty-driven exploration are discussed as future avenues.
Go to primary source ↓e-policy-ablationPDF p. 10, Policy gradients; p. 24, Figure I.1 and caption
REINFORCE is important across tasks; combining it with straight-through gradients especially helps some games such as Seaquest. Bias/variance explanations are conjectures.
Go to primary source ↓e-extra-ablationPDF p. 25, Figure J.1 and caption
Layer-normalization effects depend on the task. Random-policy data collection is compared to actor-driven collection to expose exploration-sensitive tasks.
Go to primary source ↓8.5 Primary sources
MASTERING ATARI WITH DISCRETE WORLD MODELS ↗
PDF · 14,675 extracted words
Source fingerprint
037dce26de328dd86ce4ab71dfb54f44513fef49a02f2995a1c05848ccef158f