Dream to Control: Learning Behaviors by Latent Imagination
1. Paper overview
In one sentence: Dreamer turns short latent rollouts into a policy-learning signal through bootstrapped values, gaining efficient visual control while remaining dependent on the learned dynamics and representation. e-frameworke-flowe-returnse-updatese-scorese-discrete
| At a glance | What to know |
|---|---|
| Research problem | Source description A policy that maximizes only rewards inside a finite model rollout can miss delayed consequences. Dreamer asks whether differentiable latent dynamics and learned state values can train farsighted image-based control efficiently, while avoiding expensive pixel prediction during behavior learning. e-probleme-frameworke-returns |
| Core mechanism | |
| A key reported result | DeepMind Control Suite aggregate: Dreamer: 823.39 after 5 × 10⁶ environment steps. Reported average episode return; higher is better. Online training on 20 visual tasks; randomized initial states, 1,000-step episodes, rewards in [0,1], R = 2; Figure 6 reports five-seed averages. PlaNet: 332.97 at 5 × 10⁶; D4PG: 786.32 and proprioceptive A3C: 243.70 at 10⁸. The table supports the reported aggregate advantage. Baseline budgets and input modalities differ, and D4PG/A3C omit both quadruped tasks. These averages therefore do not compare identical task coverage; the table supplies no uncertainty interval. e-setupe-scores |
| Reading caution | Reader analysis The evaluation concerns simulated control and game tasks. It does not establish physical deployment, cross-task transfer, or robustness to unfamiliar visual domains. The authors identify representation learning for more complex environments as future work. e-setupe-discretee-conclusion |
Core contributions
- Author claim
The authors report better data efficiency, training time, and aggregate continuous-control performance than their selected baselines. Representation learning is a separable design choice, tested with reconstruction, contrastive estimation, and reward-only prediction. e-setupe-scorese-representation
Figure 3. One learned world model serves different roles during model fitting, imagined behavior learning, and interaction. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right as three operations, not three consecutive steps of one imagined trajectory. In panel (a), blue observation encoders feed green latent states; gray decoders reconstruct images and yellow symbols represent predicted rewards. In panel (b), only the initial observation anchors the displayed rollout. Predicted actions drive latent transitions, while rewards and trophy-shaped values supply the actor-learning signal. In panel (c), observations enter at every displayed step before the next action is chosen. The green arrows show forward state dependence; the analytic gradient path is specified by Equations (7–8) and their accompanying text, rather than by a separate backward arrow in this diagram. e-frameworke-flowe-updatese-reconstruction
What it supports. The crucial separation is between training a predictive model from collected data and training behavior through that model. Panel (c) then shows feedback from actual observations. Together with Algorithm 1, this supports an actor that executes directly from an inferred state, without online trajectory optimization at every action.
Where the evidence stops. Figure 3 is a component schematic, not a layer-level RSSM specification. Separate θ, φ, and ψ parameters and the fixed-world-model behavior update are established in Algorithm 1 and the objective text.
2. Motivation
2.1 The problem and the proposed response
A policy that maximizes only rewards inside a finite model rollout can miss delayed consequences. Dreamer asks whether differentiable latent dynamics and learned state values can train farsighted image-based control efficiently, while avoiding expensive pixel prediction during behavior learning. e-probleme-frameworke-returns
2.2 What this reading follows
Imagine training a controller without rendering an image for every hypothetical move. Dreamer first learns to compress experience into a predictive latent state, then uses that model as a differentiable training environment for an actor and a value estimator. The value estimator matters because useful consequences may lie beyond the rollout that fits in memory. Execution returns to fresh observations and the learned actor. The figures below connect this information flow to horizon ablations, representation choices, and simulator returns. They also expose the limits: prediction examples are qualitative, baseline budgets differ, and success on selected discrete tasks does not establish suite-wide competitiveness. e-frameworke-flowe-returnse-updatese-scorese-discrete
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/classical-world-model category fits an RSSM-based reinforcement-learning agent with separately parameterized world model, actor, and critic. Action-conditioned state prediction and policy optimization do not constitute one joint future/action generator or inverse dynamics. Architecture, prediction-paradigm, and quadrant labels remain Not applicable within the supplied catalog framework; their joint use does not justify a One Model designation. e-frameworke-flowe-updates
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Make the latent state useful for both prediction and control
Begin with a sequence from replay, not an isolated image. The representation model combines the previous latent state, the preceding action, and the current observation. A transition model learns to predict the next state when that observation is unavailable. Reconstruction of images and rewards supplies supervision, and a KL term constrains the observation-conditioned representation relative to the transition prediction. Appendix B motivates this regularization as encouraging reuse of information from preceding time steps. The observation decoder is a training component: imagined behavior learning can proceed through states and predicted rewards without decoding future images. This distinction explains why the visual prediction grid is a useful diagnostic of the model, while the representation ablation tests the more consequential question of whether its learned states support control. e-frameworke-reconstructione-derivatione-predictionse-representation
Figure 5. Latent predictions retain recognizable motion while accumulating visible pose differences. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the five images grouped under Context. The caption states that the representation model observes those frames, then the dynamics predict the following 45 steps using only the actions. Each pair of rows compares the real trajectory with decoded model states; columns after Context display selected times, not every predicted frame. Track both broad motion and precise limb configuration. The upper example increasingly diverges at later displayed times, and the lower example also develops pose differences. These visible deviations matter when assessing the caption's claim of accurate long-term prediction. The decoder makes latent evolution inspectable here, although imagined actor–critic training does not require generating these pictures. e-predictionse-frameworke-reconstructione-updates
What it supports. The examples show that an RSSM can generate coherent action-conditioned continuations from short observed context. They also visibly expose imperfections. This is useful evidence about what the model represents, while the return tables and behavior ablations supply the separate evidence that the learned policy can exploit it for control.
Where the evidence stops. Only two held-out trajectories are shown, with no aggregate image, reward, or state-prediction error. The supplied action sequence conditions each continuation; these frames do not demonstrate autonomous execution of an imagined plan.
5.2 2. Bootstrap beyond the rollout, then differentiate through it
A short rollout supplies rewards only up to its endpoint. Dreamer augments these with the critic's prediction at a later state, forming the k-step targets in Equation (5), then mixes return lengths with the lambda weighting in Equation (6). The actor seeks actions whose resulting imagined states have higher value estimates. Reparameterized continuous actions and latent samples make this path differentiable; the world model's parameters remain fixed during the behavior update. Meanwhile, the critic regresses onto the return targets with target gradients stopped. These are different gradient roles: holding model weights fixed still permits gradients through its state transitions. The horizon ablation compares this mechanism with reward-only actor learning and online planning. Its task-specific exceptions should temper any claim that value learning eliminates horizon sensitivity. e-returnse-updatese-horizon
5.3 3. Separate imagined learning from measured behavior
At collection time, the actor receives a state inferred from the actual observation history and supplies the action to execute. New experience then refreshes replay and the world model. Thus, the measured control returns evaluate behavior in the simulator, while the prediction grid evaluates selected action-conditioned model rollouts. Reader interpretation: these two tests answer complementary questions and cannot substitute for one another. The score table should next be read through its protocol rows: Dreamer and the rerun PlaNet share the listed interaction budget, whereas the model-free references use more steps and A3C has a different input modality. Missing quadruped entries further limit aggregate comparability. Finally, the discrete-task plots extend the mechanism to categorical actions and termination, but their selected tasks and disparate reference budgets support applicability rather than a general ranking. e-flowe-hypere-predictionse-scorese-discrete
5.4 Training and inference
During training
The default continuous setup uses 50 sequences of length 50 per batch, imagination horizon H = 15, discount γ = 0.99, and λ = 0.95. Adam learning rates are 6 × 10⁻⁴ for the world model and 8 × 10⁻⁵ for actor and critic; gradient norms are capped at 100. KL weight β = 1 with a lower clip of 3 free nats. e-hyper
Five random seed episodes initialize replay. Training alternates 100 updates with one collected episode. No latent overshooting, actor entropy bonus, or critic target network was found necessary in the reported setup. e-hyper
During inference
At each environment step, update the latent state using the new observation and execute the actor's action. Appendix A specifies the predicted mode plus Normal(0, 0.3) exploration noise during collection, with action repeat R = 2. Fresh observations close the feedback loop; no online trajectory-search stage appears in this action-selection procedure. e-flowe-hyper
For discrete tasks, a categorical actor uses straight-through sampling gradients during imagination. A learned discount predictor handles early termination by weighting objectives with cumulative predicted discounts; these gradients train behavior rather than differentiating through the external environment. e-updatese-discrete
5.5 Implementation flow
- Infer a compact state
The representation model conditions on the previous state, previous action, and current image. A convolutional encoder feeds the recurrent state-space model (RSSM); the transition model can advance without receiving the next image. e-frameworke-reconstruction
- Learn dynamics from experience
Optimize image and reward likelihoods plus a KL regularizer aligning observation-conditioned states with the transition prior. The decoder supplies supervision; it is unnecessary for imagined actor–critic rollouts. Appendix B derives reconstruction and contrastive objectives through information-bottleneck bounds. e-reconstructione-derivation
- Imagine and improve behavior
Start trajectories at states inferred from replay sequences. Sample actor actions, transition forward, predict rewards and values, and form bootstrapped returns. Keep world-model parameters fixed during these updates, but differentiate through its computations to train the actor. Regress the critic onto stopped-gradient targets. e-flowe-returnse-updates
6. Experiments & results
Dreamer learns a visual world model from experience, then trains a separate actor and value model through imagined latent trajectories. Bootstrapped values extend the learning signal beyond a short rollout; environment interaction uses the learned actor with freshly inferred states. The continuous-control results support this approach, subject to unequal baseline budgets and incomplete baseline task coverage.
6.1 Read the original evidence
Appendix G: Continuous Control Scores (unnumbered table). Read the protocol rows before comparing returns across methods. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin above the task rows. Dreamer, PlaNet, and D4PG use pixels, whereas A3C uses proprioception; the table assigns five million steps to Dreamer/PlaNet and one hundred million to the model-free references. The footnote explains that PlaNet was rerun with fixed action repeat R = 2, so its scores differ from its original publication. Next compare specific tasks before the final Average row. Acrobot Swingup and Hopper Hop favor Dreamer; Finger Spin favors D4PG. The quadruped dashes denote missing baseline scores, not zeros. Figure 6 states that the reported performance uses five-seed averages, but this table itself provides no dispersion or explanation of its boldface criterion. e-scorese-task-scorese-setupe-inconsistencies
What it supports. Dreamer's reported mean is 823.39 versus PlaNet's 332.97 at the same listed step budget. D4PG reports 786.32 at a larger budget. The task rows show both the gains and exceptions: Acrobot is 365.26 versus D4PG's 91.70, whereas Finger Spin is 498.88 versus 985.70.
Where the evidence stops. The aggregate columns have different task coverage because D4PG/A3C lack quadruped scores. Several learning-curve legends elsewhere label those baselines at 10⁹ steps; this table uses 10⁸. These protocols remain explicitly distinct.
Figure 9. Discrete actions and early termination are supported, with uneven performance across the selected tasks. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel independently because both the return range and environment-step axis vary. The blue trajectory is Dreamer; dashed lines identify reference methods and a gray random baseline. The legend explicitly lists different training budgets for SimPLe, DQN, Rainbow, and IMPALA. Boxing shows a comparatively strong learned behavior, whereas Pong and several other games remain far from the strongest reference lines. The last two panels concern DeepMind Lab rather than Atari. Appendix C specifies image observations and sticky actions for Atari; Appendix A explains the categorical actor, straight-through sampling gradient, reward bounding, and learned discount needed to extend the continuous-control method to this setting. e-discretee-updatese-conclusion
What it supports. The plots establish that latent-imagination behavior learning can produce useful actions with categorical choices and early episode endings. They also show why the paper presents these experiments as applicability evidence: the results vary substantially, and strong continuous-control averages do not imply uniformly competitive Atari or DeepMind Lab performance.
Where the evidence stops. This is a selected subset, not a full-suite evaluation. Reference budgets differ greatly, and the caption does not define the blue shading's uncertainty statistic. No suite-wide ranking or numerical uncertainty is inferred.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| DeepMind Control Suite aggregate Online training on 20 visual tasks; randomized initial states, 1,000-step episodes, rewards in [0,1], R = 2; Figure 6 reports five-seed averages. | Dreamer: 823.39 after 5 × 10⁶ environment steps. Reported average episode return; higher is better | PlaNet: 332.97 at 5 × 10⁶; D4PG: 786.32 and proprioceptive A3C: 243.70 at 10⁸. The table supports the reported aggregate advantage. Baseline budgets and input modalities differ, and D4PG/A3C omit both quadruped tasks. These averages therefore do not compare identical task coverage; the table supplies no uncertainty interval. e-setupe-scores |
| Acrobot Swingup Appendix G visual-control comparison; Dreamer/PlaNet at 5 × 10⁶ steps, D4PG at 10⁸. | Dreamer: 365.26. Episode return | PlaNet: 3.21; pixel-input D4PG: 91.70. A concrete delayed-control example where Dreamer substantially exceeds these baselines; this is simulator reward, not a robot success rate. e-setupe-task-scorese-behavior |
| Imagination-horizon ablation Figure 4: Cartpole Swingup, Cheetah Run, Quadruped Walk, Walker Walk; reconstruction models and R = 2. | Dreamer is generally less horizon-sensitive; Walker Walk deteriorates at the longest displayed horizon. Final episode return across imagination horizons | An actor maximizing finite imagined rewards without a value model, and PlaNet online planning. Supports the utility of bootstrapped values, without establishing complete horizon invariance. Figure 4 supplies no uncertainty bands. e-horizone-returns |
| Representation-learning ablation Figure 8 and Appendix E compare reconstruction, contrastive, and reward-only objectives with Dreamer; Appendix E shows five-seed means and standard deviations. | Reconstruction leads on most tasks; contrastive learning is stronger on Finger Spin; reward-only learning is insufficient in these experiments. Episode-return learning curves | The behavior-learning approach is retained while representation supervision changes. The benefits depend on the state representation. A universal superiority of pixel reconstruction is not established. e-reconstructione-representation |
| Discrete-action applicability Appendix C: selected Atari games and DeepMind Lab levels, 64 × 64 × 3 inputs, 3–18 actions; Atari uses sticky actions. | Successful behaviors appear on selected tasks, with uneven performance across the displayed subset. Episode-return learning curves | Figure 9 supplies SimPLe, DQN, Rainbow, IMPALA, and random references with markedly different budgets. Evidence of applicability to discrete actions and termination, not competitive performance across entire suites. e-discrete |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Bootstrapped values usually reduce dependence on how far the model explicitly imagines. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each panel varies imagination horizon horizontally and measures final episode return vertically. The legend identifies Dreamer with V_lambda, while the no-value actor and PlaNet optimize finite imagined rewards V_R. Compare methods within each task first: Cartpole Swingup and Cheetah Run show large benefits for Dreamer over much of the sweep. Quadruped Walk needs enough imagined steps even with values. Walker Walk is a useful counterexample to a simple universal story: the methods are close across intermediate horizons, and Dreamer falls at the longest one. The caption specifies shared pixel-reconstruction representation learning and action repeat R = 2. Equations (5–6) explain the critic's bootstrapped contribution. e-horizone-returnse-updatese-inconsistencies
What it supports. The plot supports a practical role for value prediction beyond explicit rollout rewards: strong performance can be reached with relatively short imagination. It also shows that horizon remains a consequential setting. Increasing rollout length does not monotonically improve every method or task, particularly in Walker Walk.
Where the evidence stops. No uncertainty bands accompany this sweep. Equation (2) prints a finite value sum despite the beyond-horizon motivation; the verified bootstrapping rule is in Equations (5–8). The report preserves this notation tension.
Figure 8. Changing how the world model learns its representation changes the resulting policy. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the three solid colors: reconstruction is blue, contrastive learning green, and reward-only prediction purple. The horizontal axis measures collected environment steps; the vertical scale varies by task, so compare curves within a panel rather than panel heights. The main pattern is strong blue performance, substantial green learning on some tasks, and weak purple performance. Finger Spin is the instructive exception: green exceeds blue. Dashed D4PG and A3C lines are reference scores at the budgets printed in this legend, not learning curves under the same collection budget. Appendix E supplies the complete task grid and identifies its shaded areas as standard deviations across five seeds. e-reconstructione-representatione-inconsistencies
What it supports. The same Dreamer behavior-learning approach can benefit very differently from different representation objectives. Reconstructing observations usually supplies a useful learning signal beyond rewards alone. However, the visible Finger Spin exception means reconstruction should be treated as a strong tested choice, not an objective that necessarily dominates every control task.
Where the evidence stops. These objectives change representation supervision and associated model components; the result is not an isolated test of one loss coefficient. The legend's 10⁹-step references differ from Appendix G's 10⁸-step headers.
7. Analysis & limitations
7.1 What the evidence leaves open
The evaluation concerns simulated control and game tasks. It does not establish physical deployment, cross-task transfer, or robustness to unfamiliar visual domains. The authors identify representation learning for more complex environments as future work. e-setupe-discretee-conclusion
Figure 5 offers only two held-out action-conditioned prediction examples, without a quantitative dynamics-error metric. Visually plausible rollouts cannot by themselves validate policy gradients or long-term reward accuracy. e-predictions
Source inconsistencies remain unresolved: several curve legends label D4PG/A3C at 10⁹ steps, while Figure 6 and Appendix G use 10⁸. Section 6 says both 19 and 20 tasks for behavior curves and specifies horizon 20, whereas Appendix A defaults to 15. Equation (2) prints a finite-horizon value sum despite the surrounding beyond-horizon motivation; this report follows the explicit bootstrapping in Equations (5–8). e-inconsistencies
Appendix F's action-repeat comparison uses two seeds and an earlier hyperparameter setting; its preference for R = 2 is not an otherwise identical ablation of the final configuration. e-repeat
7.2 Questions for discussion
- Would bootstrapped values retain their advantage with shared fixed replay data and matched behavior-update budgets? [e-horizon, e-updates]
- Does better action-conditioned prediction accuracy imply more reliable policy gradients, or can those criteria diverge? [e-predictions, e-updates]
8. Reproducibility audit
8.1 Requirements and known gaps
Architecture prerequisites include the cited convolutional encoder/decoder and RSSM, 30-dimensional diagonal-Gaussian latent distributions, and three 300-unit ELU layers for other functions. The source delegates component details to earlier work. TensorFlow Probability is named without a version; each run uses one Nvidia V100 and 10 CPU cores. e-hypere-setup
The authors report about three training hours per million control-suite steps for Dreamer, versus eleven for PlaNet, and twenty-four hours for D4PG to reach similar performance. These are implementation-specific timing statements, not a fully controlled hardware comparison. e-setup
A reproduction must distinguish the default continuous configuration from the discrete setup: H = 10, β = 0.1, tanh-bounded rewards, and epsilon-greedy exploration decaying from 0.4 to 0.1 over 200,000 gradient steps. The exact software/environment versions and the earlier Appendix F configuration require clarification. e-hypere-discretee-repeat
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does value bootstrapping help when experience and compute are controlled?
Reader-proposed experiment: on Acrobot Swingup and Walker Walk, train one reconstruction world model per seed from a shared fixed replay dataset. With identical starting states, actor architecture, and model weights, compare Dreamer's bootstrapped actor–critic with the no-value reward-sum actor at proposed horizons 5, 15, and 30. First match actor-update counts, then repeat with matched simulated-transition budgets so longer horizons do not receive an unreported compute advantage. Evaluate executed return without collection noise across held-out simulator initializations and report seed variability. The hypothesis predicts a larger short-horizon benefit on Acrobot than on Walker Walk. If that advantage disappears under shared experience or compute matching, the original online comparison cannot attribute the whole gain to bootstrapping alone. This is a proposed controlled variant, not a claimed reproduction. e-horizone-behaviore-updatese-flowe-hyper
Check 2: Can visually good predictions conceal unhelpful control representations?
Reader-proposed experiment: train reconstruction, contrastive, and reward-only world models on identical fixed replay from Cheetah Run and Finger Spin, using matched parameter and update budgets as closely as the objectives allow. Train the same Dreamer actor–critic against each frozen model. On held-out action sequences, measure multi-step reward error for all models and image error only where a decoder exists; separately measure executed policy return in the simulator. Use the same rollout horizon, state-start sampling, and random seeds across conditions, and disclose residual architecture differences. If reconstruction improves image appearance without improving reward accuracy or control, qualitative video quality is an unreliable selection criterion. If the Finger Spin advantage of contrastive learning persists under these controls, a claim of universal reconstruction superiority is falsified. No such controlled comparison was run here. e-reconstructione-representatione-predictionse-updates
8.3 Reading coverage
Visual audit: Visually inspected the title and affiliations, Figures 1–12, Algorithm 1, Equations (1–16), the evaluation text, Appendix A configuration, and Appendix G's complete score table and footnote. All six final crops were separately viewed. Figure 3's forward arrows and panel roles agree with the caption and Algorithm 1; gradient behavior is checked against p. 5 rather than inferred from arrow colors. Source discrepancies in value notation, baseline budgets, behavior-curve task count, and horizon settings are preserved. References on pp. 10–13 were read as text, not rendered. Separate supplements, linked code, and videos remain outside this pass. The table crop retains original blank space because its relevant footnote is at the page bottom.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–7 (PDF pp. 1–9)
- References (PDF pp. 10–13)
- Appendix A: Hyper Parameters (p. 14)
- Appendix B: Derivations (p. 15)
- Appendix C: Discrete Control (p. 16)
- Appendix D: Behavior Learning (p. 17)
- Appendix E: Representation Learning (p. 18)
- Appendix F: Action Repeat (p. 19)
- Appendix G: Continuous Control Scores (p. 20)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Recovered-edition limitation: arXiv v3 author-deposited copy. Page 1 says Published as a conference paper at ICLR 2020. Byte-for-byte equivalence to the blocked OpenReview PDF was not checked. Title and all four authors match.
- The extraction limitation was addressed by visually inspecting Figures 1–12, Algorithm 1, the score table, and all pages supporting retained scientific claims. References were read as supplied text.
- Identity: the title and four authors match the catalog after normalizing the title's typography. The title page identifies arXiv:1912.01603v3, 17 March 2020, and ICLR 2020. This report describes that recovered edition.
- Code, linked videos, cited predecessor implementations, and separate supplements were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author affiliations, conference header, and arXiv margin stamp
Dream to Control: Learning Behaviors by Latent Imagination credits Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Affiliations are University of Toronto, Google Brain, and DeepMind as assigned on the title page. The stamp is arXiv:1912.01603v3, 17 March 2020; the header identifies ICLR 2020.
Go to primary source ↓e-problemPDF pp. 1–2, Introduction and contribution bullets
Finite imagined rewards can yield shortsighted behavior; Dreamer proposes learning actions and state values with analytic gradients in latent imagination.
Go to primary source ↓e-frameworkPDF p. 2, Section 2, Agent components and Latent dynamics, Eq. (1)
Representation conditions on previous state/action and current observation; the transition prior predicts without a new image; the reward model conditions on state. Dynamics learning, behavior learning, and interaction are distinct operations.
Go to primary source ↓e-flowPDF p. 3, Figure 3 panels (a–c), Section 3 Imagination environment, and Algorithm 1
Experience trains dynamics; replay-inferred states seed imagined behavior training; online interaction encodes observations and executes actor actions. Model parameters θ, actor φ, and critic ψ are separate.
Go to primary source ↓e-returnsPDF p. 4, Section 3, Eqs. (3–6) and Value estimation
The continuous actor samples a tanh-transformed Gaussian by reparameterization. V_R sums finite rewards; k-step estimates bootstrap from a value prediction; V_lambda mixes return lengths.
Go to primary source ↓e-updatesPDF p. 5, Learning objective, Eqs. (7–8) and following paragraphs
The actor maximizes value estimates through differentiable dynamics; the critic regresses stopped-gradient targets. World-model parameters are fixed during behavior learning. Continuous samples use reparameterization; discrete actions use straight-through gradients; predicted discounts weight objectives under termination.
Go to primary source ↓e-reconstructionPDF pp. 5–6, Section 4, Reward prediction, Reconstruction, Contrastive estimation, Eqs. (9–12)
The default model jointly learns observation/reward reconstruction and KL regularization using a CNN, RSSM, transposed CNN, and dense reward network. Contrastive estimation replaces observation generation with image-conditioned state prediction and a minibatch contrastive term.
Go to primary source ↓e-derivationPDF p. 15, Appendix B, Eqs. (13–16)
The information-bottleneck formulation yields reconstruction and contrastive lower bounds and a transition-versus-representation KL regularizer. The stated rationale is to encourage reliance on previously extracted information.
Go to primary source ↓e-predictionsPDF p. 5, Figure 5 and caption
Two held-out trajectories use the first five images as context and then predict 45 steps conditioned only on actions. The visual compares true and decoded model frames; it provides no numerical prediction-error metric.
Go to primary source ↓e-setupPDF p. 8, Section 6, Control tasks, Implementation, and Baseline methods
Twenty control-suite tasks use 64 × 64 × 3 images, 1–12 action dimensions, rewards 0–1, 1,000-step episodes, randomized initial states, and R = 2. The authors name TensorFlow Probability, one V100 and ten CPU cores, and report Dreamer/PlaNet timing plus a D4PG time-to-similar-performance comparison. PlaNet is rerun; other baseline scores come from prior reports.
Go to primary source ↓e-scoresPDF p. 6, Figure 6 and caption; p. 20, Appendix G, header, Average row, Quadruped rows, and footnote
Average scores are Dreamer 823.39, PlaNet 332.97, D4PG 786.32, A3C 243.70. Budgets are 5 × 10^6 for Dreamer/PlaNet and 10^8 for D4PG/A3C. A3C is proprioceptive; the others use pixels. A3C/D4PG quadruped entries are missing. Figure 6 states five-seed averages; the table footnote identifies PlaNet's fixed R = 2 rerun.
Go to primary source ↓e-task-scoresPDF p. 20, Appendix G, Acrobot Swingup, Hopper Hop, and Finger Spin rows
Acrobot scores are Dreamer 365.26, PlaNet 3.21, D4PG 91.70. Hopper Hop scores are 368.97, 0.37, and 242.00 respectively. Finger Spin scores are Dreamer 498.88 and D4PG 985.70.
Go to primary source ↓e-horizonPDF p. 4, Figure 4 panels and caption; p. 9, Section 6, Long horizons
The horizon sweep compares Dreamer, a no-value actor, and PlaNet using pixel reconstruction and R = 2. Dreamer is generally more robust, with a visible Walker Walk decline at the longest horizon. The displayed sweep contains no uncertainty bands.
Go to primary source ↓e-behaviorPDF p. 7, Figure 7 and caption; p. 17, Appendix D, Figure 10 and caption
Behavior-learning curves compare Dreamer, no-value learning, and PlaNet across control tasks. Acrobot and hopper motivate long-term credit assignment; Appendix D specifies means and standard deviations across five seeds.
Go to primary source ↓e-representationPDF p. 8, Figure 8 and caption; p. 18, Appendix E, Figure 11 and caption
Reconstruction usually gives the strongest learning signal; contrastive learning can outperform it on Finger Spin. Reward-only prediction is insufficient in these experiments. Appendix E specifies means and standard deviations across five seeds.
Go to primary source ↓e-hyperPDF p. 14, Appendix A, Model components, Learning updates, and Environment interaction
The appendix specifies Gaussian latent size, dense-network architecture, minibatches, learning rates, gradient clipping, free nats, H = 15, gamma/lambda, random initialization episodes, collection schedule, exploration noise, and fixed action repeat. Encoder/decoder and RSSM details refer to earlier papers.
Go to primary source ↓e-discretePDF p. 14, Appendix A, Discrete control; p. 16, Appendix C and Figure 9 including legend
Discrete training uses categorical straight-through actions, scheduled epsilon-greedy exploration, H = 10, beta = 0.1, tanh rewards, and a discount classifier. Evaluation uses selected Atari/DMLab tasks, 3–18 actions, and sticky-action Atari protocol. Reference budgets in Figure 9 vary; complete-suite competitiveness is not claimed.
Go to primary source ↓e-repeatPDF p. 19, Appendix F, Figure 12 and caption
Action repeats 1, 2, and 4 are compared using a previous hyperparameter setting and two seeds; shading is one standard deviation. The authors favor R = 2 across tasks but do not enumerate that earlier configuration.
Go to primary source ↓e-inconsistenciesPDF p. 4, Eq. (2) versus Eqs. (5–6); p. 5, Eqs. (7–8); pp. 6–9, Figures 6–8 and Long horizons; pp. 14, 17–20, Appendix A and Figures 10–12/table headers
Equation (2)'s printed finite sum contrasts with the beyond-horizon explanation and subsequent bootstrapping. Curve legends use 10^9 D4PG/A3C steps while Figure 6 and Appendix G use 10^8. Section 6 refers to both 19 and 20 tasks and horizon 20; Appendix A gives default H = 15.
Go to primary source ↓e-conclusionPDF p. 9, Section 7; p. 16, Appendix C discussion and Figure 9 caption
The authors emphasize continuous-control performance and discrete-task applicability while identifying representation learning for higher visual complexity and broader discrete-suite competitiveness as future work.
Go to primary source ↓8.5 Primary sources
DREAM TO CONTROL : L EARNING BEHAVIORS BY LATENT IMAGINATION ↗
PDF · 9,937 extracted words
Source fingerprint
318891d59ba0065f57eb0e735735700e5eb528f53bf08b81c773437385586c64