DayDreamer: World Models for Physical Robot Learning
1. Paper overview
In one sentence: A learned latent simulator trains an actor while robots collect real experience, enabling learning within hours under carefully engineered control interfaces. e-loope-modele-baselinese-a1-resultse-ur5e-spheroe-limits
| At a glance | What to know |
|---|---|
| Research problem | Source description Physical trial and error is expensive, while simulation can miss real dynamics. The paper asks whether Dreamer can learn locomotion, manipulation and navigation directly from rewards, without simulators or demonstrations. Each robot supplies its own ongoing experience; there is no common offline dataset or held-out benchmark split. e-loope-baselinese-a1-setupe-ur5e-xarme-sphero |
| Core mechanism | Reader analysis The authors deploy the existing Dreamer algorithm across continuous/discrete action spaces, dense/sparse rewards and proprioceptive/visual inputs. Shared hyperparameters demonstrate breadth across these four tasks, rather than cross-robot transfer of one trained policy. e-loope-baselines |
| A key reported result | UR5 multi-object visual pick and place: 2.5 objects/minute within 8 hours. Average objects transferred per minute. Three objects; RGB and proprioception; discrete 2 Hz control; sparse rewards and automatic destination-bin opening. Approaches the human reference; Rainbow and PPO mostly grasp and release in the source bin. Rainbow’s plotted trace ends earlier. Successful transfer exceeds the local grasp/release behavior under this interface. Replicate counts and uncertainty-band meaning are not specified here; equal final training durations cannot be assumed. e-ur5 |
| Reading caution | Source description Hardware wear can require intervention or repair, and longer-budget behavior of Dreamer and its baselines remains insufficiently explored. e-limits |
Core contributions
- Reader analysis
The authors deploy the existing Dreamer algorithm across continuous/discrete action spaces, dense/sparse rewards and proprioceptive/visual inputs. Shared hyperparameters demonstrate breadth across these four tasks, rather than cross-robot transfer of one trained policy. e-loope-baselines
- Author claim
An asynchronous actor and learner let training continue while hardware moves. The authors report releasing this robotics infrastructure; that release statement alone does not establish reproducibility. e-loope-baselines
Figure 3. Reconstruction teaches the latent model; imagined latent trajectories teach the policy. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel from the sensory inputs x along the bottom. Blue encoders map observations to stochastic codes z; green recurrent states h propagate information forward using those codes and actions a. Orange decoders reconstruct observations for world-model training. In the right panel, only the initial observation is encoded. Later states are generated through the action-conditioned dynamics, while actor, reward and value outputs support behavior learning. The missing decoders on this side agree with the caption and Section 2: policy training uses latent rollouts. Use Eqs. (1)–(3) for the temporal formulation, because the diagram repeats some reward/value subscripts. e-modele-actore-diagram-indices
What it supports. The computational distinction is that imagined policy improvement need not render future camera images. Reconstruction still supplies representation-learning supervision, while real commands come from the trained actor. Actor and critic gradients do not update world-model parameters; differentiating an action through dynamics and optimizing the dynamics parameters are separate operations.
Where the evidence stops. The final h_3 branch still labels its outputs r_2 and v_2. This repeated indexing is preserved from the source and is not reconciled with Eqs. (1)–(3). Treat the panel as a structural schematic, not an exact timestep specification.
2. Motivation
2.1 The problem and the proposed response
Physical trial and error is expensive, while simulation can miss real dynamics. The paper asks whether Dreamer can learn locomotion, manipulation and navigation directly from rewards, without simulators or demonstrations. Each robot supplies its own ongoing experience; there is no common offline dataset or held-out benchmark split. e-loope-baselinese-a1-setupe-ur5e-xarme-sphero
2.2 What this reading follows
DayDreamer asks whether a world model can make trial-and-error learning practical on physical robots. Its answer comes from four deployments of Dreamer, using the same reported hyperparameters across locomotion, manipulation and navigation. The important separation is between learning a simulator from experience, improving a policy inside that simulator, and executing the policy on hardware. The figures below show how this loop works, where its learning curves improve on baselines, and why model introspection is not a dynamics guarantee. Read the results with their interfaces attached: action constraints, reward design and continuing human involvement shape what the demonstrations establish. e-loope-modele-baselinese-a1-resultse-ur5e-spheroe-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/classical world-model and model-based-RL category fits the architecture. A distinct actor critic learns from action-conditioned latent dynamics; this is neither inverse dynamics nor a unified joint future/action predictor. Shared training infrastructure does not make it a One Model architecture. Retaining the catalog’s not-applicable quadrant and prediction-paradigm labels is appropriate for this foundational entry. e-loope-modele-actore-position
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate imagined learning from physical action selection
Start with one real observation. The encoder combines the sensors available on that robot with the recurrent history to update the latent state. The actor chooses a command, the physical robot moves, and the resulting experience enters replay. Independently, the learner draws replayed sequences to improve its world model. It then generates many latent trajectories to train the actor and critic, using predicted rewards rather than waiting for all those trajectories to happen on hardware. The decoder provides reconstruction supervision during model learning but is absent from the imagined behavior loop. This separation explains the asynchronous design: the actor must meet a control deadline, while the learner can continue updating. It also identifies what is deployed: a policy acting from an updated latent state, rather than a freshly decoded video plan for each motor command. e-loope-modele-actore-position
Appendix D, unnumbered hyperparameter table. The shared configuration is concrete, but the horizon needs reconciliation. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the table by its four groups. General settings specify the FIFO replay, sequence batch size B and sequence length T, plus network size and activation. World Model lists the RSSM size, categorical latent dimensions and KL balancing. Actor Critic supplies imagination horizon H, discount gamma, return lambda and a target update interval. All Optimizers gives clipping, learning rate and Adam epsilon. The distinction between B = 32 sequences and latent behavior batches matters: the main text separately discusses typical 16K behavior batches. Do not silently equate those quantities. This is the paper’s configuration table, not a table of experimental performance. e-confige-horizone-actore-baselinese-repro-gaps
What it supports. The appendix makes several implementation choices explicit: 32 sequences of length 32, an RSSM size of 512, and 32 latents with 32 classes each. Discount and return lambda are both 0.95. Those details support a concrete reconstruction of the learner, while the reported shared settings still need robot-specific interfaces.
Where the evidence stops. Appendix D prints H = 15; Section 2 says H = 16. The source does not resolve the discrepancy or state units for the start-learning threshold. The table also supplies no per-experiment compute, throughput or software-version specification.
5.2 Read cross-bin transfer as a long-term reward problem
The UR5 reward design makes the behavioral distinction concrete. A detected grasp gives +1, releasing in the original bin gives -1, and placing in the opposite bin gives +10. The paper reports that the model-free baselines tend to grasp and release locally, while Dreamer learns productive transfer. In Dreamer, the reward model predicts consequences along an imagined action sequence, and the critic bootstraps value beyond the finite rollout through lambda-returns. This provides a mechanism by which a grasp can be valued for enabling a later placement. As a reader interpretation, the result is consistent with useful longer-term credit assignment. It does not isolate that mechanism: the algorithms also differ in representation learning and optimization. Preserve the action constraints and the unequal plotted baseline durations before treating the throughput gap as a controlled architectural comparison. e-ur5e-actore-model
5.3 Distinguish recovery, invariance and faithful prediction
The sunrise example begins with a policy learned under nighttime camera conditions. A strong lighting change then coincides with a sharp loss of pick-and-place performance, followed by recovery during continued learning in approximately five hours. That sequence is evidence of recovery, not invariance to the changed appearance. The imagined-rollout panel adds a second distinction: a useful latent model can still decode an object with the wrong color. As a reader interpretation, these observations suggest evaluating task-relevant predictive information separately from visual fidelity. They do not show that reconstruction errors are harmless or identify which network adapts to lighting. A controlled study would need repeatable illumination and matched starting checkpoints, with selected model and policy parameters frozen. Without those controls, encoder adaptation, changed policy behavior and changing environmental conditions remain intertwined explanations. e-adaptatione-imaginatione-model
5.4 Training and inference
During training
Continuous actions use reparameterization gradients through differentiable dynamics; discrete actions use Reinforce. Entropy promotes exploration. Actor/critic updates leave world-model parameters unchanged, even though the model can transmit action gradients. A slowly updated critic supplies return targets. e-actor
Appendix D specifies replay capacity 10^6, batches of 32 sequences of length 32, RSSM size 512, 32 categorical latents with 32 classes each, KL balancing 0.8, and Adam learning rate 10^-4. The source summarizes rather than fully expands the world-model loss. e-confige-repro-gaps
During inference
For real control, incoming observations update the latent state and the actor supplies a command. Imagination trains the policy; the described deployment does not search over decoded future videos at every action. Environment rewards then provide feedback for further online learning. e-loope-modele-actore-position
Execution includes robot-specific machinery: A1 motor-angle commands pass through filtering and a PD controller, while arm commands use constrained discrete movements and gripper logic. These interfaces are part of the demonstrated system. e-a1-setupe-ur5e-xarm
5.5 Implementation flow
- Collect and replay
The current actor interacts with the robot and appends experience to replay. A separate learner samples subsequences and updates networks continuously, without a fixed training-to-interaction ratio. e-loope-actor
- Learn a compact simulator
An encoder fuses observations into discrete stochastic codes. The RSSM propagates recurrent state and predicts future codes using actions. Reconstruction and reward prediction supply learning signals; all world-model components are trained together. e-model
- Optimize behavior in latent space
Starting from encoded experience, the actor generates actions and the dynamics model predicts subsequent states. Learned rewards and critic values construct lambda-returns. The critic regresses these targets; the actor maximizes them without reconstructing future camera images. e-modele-actor
6. Experiments & results
DayDreamer applies Dreamer to online learning on four physical robots. A learned latent dynamics model supplies imagined experience for actor-critic training while the policy continues collecting real data. Locomotion and manipulation improve within hours, but navigation only matches its baseline. The contribution is practical deployment across robot interfaces with shared hyperparameters; it does not establish a new algorithm or a general robot policy.
The supplied v1 paper presents quantitative performance as learning curves and contains no numerical results table. The included original table is Appendix D’s configuration, explicitly labeled as such. There is also no controlled component ablation: the two visuals assigned to the ablation section are adaptation and imagination diagnostics, with their causal limits stated. These are properties of this source edition, not claims about later editions. e-visual-scopee-adaptatione-imaginatione-config
6.1 Read the original evidence
Figure 4, learning-curve panel. A1 learns recovery and walking during a single hour-long run. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue Dreamer and orange SAC curves over training minutes, using average reward on the vertical axis. This is a shaped reward that combines upright posture, joint-pose terms and forward velocity; it is not a percentage of successful walks. The filled circles indicate falls onto the robot’s back. Read those markers alongside the later recovery in reward, rather than treating every dip as terminal failure. The caption defines shading as one standard deviation within each time bin of a single training run. Section 3.1 supplies the execution context: proprioceptive observations, motor-angle commands at 20 Hz, a PD controller and command filtering. e-a1-resultse-a1-setupe-a1-reward
What it supports. The accompanying text reports that Dreamer rolls over, stands and walks within one hour, while SAC only learns to roll over in that budget. A subsequent pushing phase elicits adaptation within ten minutes. The plotted hour supports the initial learning comparison; the later push result is reported separately in Figure 8 and Section 3.1.
Where the evidence stops. The bands are not confidence intervals over repeated training runs. SAC required help with a leg deadlock, and the setup allows spatial-boundary intervention. Eq. (5) also leaves an apparent upright-reward sign/gating inconsistency unresolved.
Figure 5, learning-curve panel. UR5 progresses from grasping to sustained cross-bin transfer. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the vertical axis: objects per minute measures productive pick-and-place throughput, not grasp detection or accumulated reward. The horizontal axis measures hours of online robot learning. Dreamer’s blue curve rises after an initially slow phase, approaching the dashed human reference. Rainbow and PPO stay near the bottom, consistent with the reported behavior of grasping and releasing objects in the original bin. The baseline curves have different plotted lengths: the orange Rainbow trace ends before eight hours. Section 3.2 supplies the common task interface, including discrete 2 Hz commands, restricted vertical motion, and automatic gripper opening above the destination bin. e-ur5
What it supports. The source reports 2.5 transferred objects per minute within eight hours for Dreamer. That is a concrete physical manipulation result under sparse rewards, stronger than simply detecting a grasp. The human comparison comes from three joystick demonstrators recorded for twenty minutes; the paper describes Dreamer as approaching their performance.
Where the evidence stops. The caption does not define the uncertainty bands or replicate count, and Rainbow is not plotted for the full eight hours. The control interface constrains the task, so this result cannot establish unrestricted arm manipulation or equal-compute algorithm superiority.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| A1 quadruped rollover, standing and walking Online physical A1 learning from proprioception at 20 Hz, starting on its back; no reset policy, with spatial-boundary intervention. | Walking within 1 hour; push adaptation within 10 further minutes. Training time to reported behaviors | SAC rolls over but does not stand or walk in the reported budget and needed assistance with a leg deadlock. These are executed behaviors. Figure 4 shows one training run; its shaded standard deviation is within time bins, not uncertainty across seeds. e-a1-setupe-a1-results |
| UR5 multi-object visual pick and place Three objects; RGB and proprioception; discrete 2 Hz control; sparse rewards and automatic destination-bin opening. | 2.5 objects/minute within 8 hours. Average objects transferred per minute | Approaches the human reference; Rainbow and PPO mostly grasp and release in the source bin. Rainbow’s plotted trace ends earlier. Successful transfer exceeds the local grasp/release behavior under this interface. Replicate counts and uncertainty-band meaning are not specified here; equal final training durations cannot be assumed. e-ur5 |
| XArm visual pick and place RGB, depth and proprioception at approximately 0.5 Hz; a soft object is tethered to the gripper. | 3.1 objects/minute in 10 hours. Average objects transferred per minute | Comparable to the human reference; Rainbow fails to learn successful transfer. The result includes the string-assisted setup, sometimes exploited to extract the object from corners; it does not establish unrestricted deformable-object manipulation. e-xarm |
| Sphero visual navigation RGB-only fixed-goal navigation at 2 Hz; 100-step episodes followed by random motor actions. | 0.15 after 2 hours. Time-averaged distance to goal in units of area size | DrQv2 achieves similar performance. Lower is better. This is normalized distance, not meters or a success percentage, and provides no demonstrated superiority over DrQv2. e-sphero |
| XArm adaptation to sunrise Continued online learning after training at night and encountering strong sunlight. | Approximately 5 hours. Time to recover prior transfer performance | The authors describe recovery as faster than initial learning; no controlled frozen-agent comparison is shown. An observed adaptation case, not an isolated test of which network changes cause recovery. e-adaptation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure A.1. Continued learning recovers from a large visual change. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two observation images before reading the curve. The nighttime view resembles the training conditions; the sunrise view introduces strong light across the robot workspace. In the plot, the orange dashed line marks sunrise and the green dashed line marks recovery, separated by roughly five hours. The vertical axis remains objects per minute, so the curve concerns executed pick-and-place behavior rather than image reconstruction error. The experiment keeps Dreamer learning as conditions change. Appendix A reports recovery and subsequent improvement, but does not separate changes in the encoder, dynamics model and actor critic. This is an adaptation diagnostic, not a component ablation. e-adaptation
What it supports. The reported recovery takes approximately five hours of additional online learning after the distribution shift. This demonstrates that the particular system can regain useful behavior despite a substantial change in its camera inputs. It also shows that performance can initially collapse even after the nighttime task has been learned.
Where the evidence stops. No frozen-agent control or repeatable lighting intervention is reported. Changing illumination and continued learning occur together, so the figure cannot isolate a causal adaptation mechanism or establish robustness without additional experience.
Figure B.1. Decoded imagination reveals both intended motion and prediction defects. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row independently from left to right. These are decoded latent trajectories, with every second frame displayed; the upper two rows concern UR5 and the lower two concern XArm. The source presents them as a way to inspect the policy’s imagined behavior. They are not time-aligned recordings of robot execution. Pay particular attention to the second UR5 row: the caption identifies a stationary orange ball that becomes green as the rollout proceeds. This visible change illustrates why a plausible movement sequence should not be treated as an exact scene forecast. The decoder exposes the learned representation for inspection, although it is unnecessary during latent policy optimization. e-imaginatione-modele-ur5e-xarm
What it supports. The visual supports a qualitative claim: Dreamer can generate interpretable imagined arm trajectories while retaining noticeable visual errors. Together with the actual manipulation results, it motivates studying which prediction errors matter for control. The paper does not measure the causal effect of this color error on the learned policy.
Where the evidence stops. These are selected model-generated examples, not executed successes, calibrated forecasts or a quantitative dynamics benchmark. No reconstruction ablation is shown. Their existence does not establish object identity preservation, long-horizon accuracy or reliable action outcomes.
7. Analysis & limitations
7.1 What the evidence leaves open
Hardware wear can require intervention or repair, and longer-budget behavior of Dreamer and its baselines remains insufficiently explored. e-limits
Task engineering matters: A1 uses filtered PD control; UR5 restricts vertical movement and automates release; XArm uses a tether. These successful interfaces do not establish unattended general-purpose learning. e-a1-setupe-ur5e-xarm
There is no controlled component ablation. Figure B.1 even shows a stationary ball changing color in imagination; visual reconstruction quality and executed task success are distinct evidence. e-visual-scopee-imagination
7.2 Questions for discussion
- How much of UR5’s advantage survives matched interaction and learner-update budgets?
- Does lighting recovery require world-model adaptation, actor-critic adaptation, or both?
8. Reproducibility audit
8.1 Requirements and known gaps
A replication needs the specified robot interfaces, reward events and asynchronous learner, with both elapsed time and interaction/update counts recorded. The paper gives typical 16K latent behavior batches on one GPU, but no per-experiment GPU model, throughput, package versions or complete camera/control settings. e-loope-a1-setupe-ur5e-confige-repro-gaps
Resolve the 16-versus-15 imagination horizon and the table’s unspecified start-learning units. Also clarify Eq. (5): the printed upright term subtracts one before dividing by two, while subsequent terms require preceding rewards of at least 0.7. The source does not explain the apparent sign/gating inconsistency. e-horizone-confige-a1-reward
Proposed checks: repeat UR5 with H = 15 versus 16 under matched data and update budgets; separately compare frozen and adapting XArm agents under repeatable lighting changes. The illustrated edition specifies controls and falsifiable observations. e-horizone-ur5e-adaptation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Measure whether the 15-versus-16 horizon discrepancy matters
Reader-proposed check, not performed: repeat UR5 training with H = 15 and H = 16 while holding initialization, replay preparation, architecture, rewards, camera processing and action constraints fixed. Use repeated runs and matched robot-interaction and learner-update budgets; record wall time and control latency because the original learner has no fixed update ratio. Compare transfer rate at matched steps and after eight hours, plus return-prediction errors. If the between-setting effect exceeds run-to-run variation, the source discrepancy materially affects replication. Similar results would reduce that concern but would not reveal which horizon produced the published run. e-horizone-confige-loope-actore-ur5
Check 2: Identify what must adapt after the lighting shift
Reader-proposed check, not performed: start XArm trials from the same nighttime-trained checkpoint and impose a repeatable illumination change that remains stable. Compare four conditions: all weights frozen, only world-model weights updating, only actor-critic weights updating, and both updating. Keep sensing, tether, actions and reward events identical; allow normal recurrent-state updates in every condition. Measure transfer rate, recovery time and reward-prediction error over the next five hours across repeated trials. A benefit unique to model updates would support representation/dynamics adaptation; recovery in the fully frozen condition would instead expose environmental or evaluation confounding. e-adaptatione-xarme-modele-actor
8.3 Reading coverage
Visual audit: Inspected the title/author/version page; all method and experiment pages, including Figures 1–8 and Eqs. (1)–(5); Appendix A adaptation, Appendix B imagination, Appendix C positioning and Appendix D configuration. Every retained method, numerical result and proposed-check source fact is supported on these pages. Inspected all six final original crops. Figure 3 indexing was checked against Eqs. (1)–(3); its repeated labels remain visible. Reference pages 9–12 were read completely as text but not rendered. No separate supplements, external code or project videos were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block and Abstract (p. 1)
- 1 Introduction (p. 2)
- 2 Approach, including Eqs. (1)–(4) (pp. 3–4)
- 3 Experiments, implementation and baselines (pp. 4–5)
- 3.1 A1 Quadruped Walking, including Eq. (5) (pp. 5–6)
- 3.2 UR5 Multi-Object Visual Pick and Place (p. 6)
- 3.3 XArm Visual Pick and Place (p. 7)
- 3.4 Sphero Navigation (pp. 7–8)
- 4 Related Work; 5 Discussion; Acknowledgements (pp. 7–8)
- References (pp. 9–12)
- A Adaptation; B Imagination (p. 13)
- C Detailed Related Work (p. 14)
- D Hyperparameters (p. 15)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting original PDF pages and every final crop. All five supplied text chunks were read completely. Reference pages 9–12 were read as text; their cited works were not opened.
- Identity/edition note: this report reads arXiv:2206.14176v1 (28 June 2022), with Ken Goldberg preceding Pieter Abbeel. The catalog reverses those final two authors, lists CoRL 2022 and publication year 2023, and has a 2022 BibTeX year. Title and all five author identities agree. The venue edition was not supplied or compared; equivalence to it is not asserted. The first three PDF authors share equal-contribution credit.
- Code, project videos and separately hosted supplements were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block and arXiv margin
The title is DayDreamer: World Models for Physical Robot Learning. The author order is Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel; the first three have equal-contribution marks. The affiliation is University of California, Berkeley. The artifact is arXiv:2206.14176v1, dated 28 June 2022.
Go to primary source ↓e-loopPDF p. 2, Figure 2 and Introduction contributions; p. 3, Section 2 opening paragraph
Real interaction supplies replay, replay trains the world model, imagined latent trajectories train an actor critic, and the policy collects further experience. Separate learner and actor threads support continuous training and low-latency control. The contribution is applying Dreamer to four physical robots without a new algorithm.
Go to primary source ↓e-modelPDF p. 3, Figure 3 caption, Section 2 World Model Learning and Eq. (1)
The encoder fuses sensory inputs into stochastic discrete representations; recurrent dynamics predict future representations conditioned on actions. Decoder reconstruction and reward prediction train the model. All world-model components are jointly optimized by stochastic backpropagation; sensory decoding is unnecessary for latent behavior learning.
Go to primary source ↓e-actorPDF p. 4, Actor Critic Learning, Eqs. (2)–(4)
The actor maps latent states to action distributions; the critic predicts returns. Lambda-returns bootstrap beyond the imagination horizon. Continuous tasks use reparameterization gradients, discrete tasks use Reinforce, and actor entropy encourages exploration. Actor and critic gradients do not update world-model parameters. A slowly updated critic supplies targets; learning runs without a training-frequency rate limit.
Go to primary source ↓e-configPDF p. 15, Appendix D, complete unnumbered hyperparameter table
The table lists FIFO replay capacity 10^6, start learning 10^4, batch size 32, batch length 32, MLP size 4 × 512, LayerNorm + ELU, RSSM size 512, 32 latents with 32 classes each, KL balancing 0.8, imagination horizon 15, discount and return lambda 0.95, target update interval 100, gradient clipping 100, learning rate 10^-4 and Adam epsilon 10^-6. Units for start learning and target update interval are not supplied in the table.
Go to primary source ↓e-baselinesPDF p. 5, Implementation and Baselines paragraphs
The implementation builds on DreamerV2 with asynchronous acting and learning and reports identical hyperparameters across experiments. SAC is the quadruped baseline; Rainbow is used for both arms, PPO additionally for UR5, and DrQv2 for Sphero. Arm baselines receive proprioception as broadcast image planes. Human operators use the robot control interface.
Go to primary source ↓e-a1-setupPDF p. 5, Section 3.1, final paragraph
The A1 has 12 motors controlled at 20 Hz by continuous motor-angle commands realized by a hardware PD controller. Inputs are motor angles, orientations and angular velocities. A Butterworth filter suppresses high-frequency commands. At the training-area boundary, humans intervene without changing joint configuration or orientation.
Go to primary source ↓e-a1-rewardPDF p. 6, continuation of Section 3.1, reward paragraph and Eq. (5)
Five reward terms concern upright posture, hip/shoulder/knee pose and forward velocity. A term is active only when preceding terms reach at least 0.7. The printed upright expression subtracts one from the up-vector dot product before dividing by two; the source does not reconcile this sign with the stated gating condition.
Go to primary source ↓e-a1-resultsPDF p. 5, Figure 4 and Figure 8 captions; p. 6, Section 3.1 outcome paragraph
Dreamer learns rollover, standing and forward walking in one hour, followed by adaptation to pushes within ten minutes of further learning. SAC rolls over but does not stand or walk within the reported budget and needed assistance with a deadlocked leg configuration. Figure 4 is a single training run; shading is one standard deviation within each time bin, and circles mark falls onto the back. Maximum reward is stated as 14.
Go to primary source ↓e-ur5PDF p. 6, Figure 5 and Section 3.2
UR5 uses third-person RGB plus proprioception, three objects, discrete motion and gripper actions at 2 Hz, rewards +1 for grasping, -1 for release in the original bin and +10 for opposite-bin placement. Vertical movement requires holding an object and opening above the destination bin is automatic. Three human demonstrators are recorded for 20 minutes. Dreamer reaches 2.5 objects/minute within eight hours; Rainbow and PPO tend to grasp and release in the same bin. The Rainbow trace ends earlier than the other plotted traces; this caption does not define its uncertainty band or replicate count.
Go to primary source ↓e-xarmPDF p. 7, Figure 6 and Section 3.3
The seven-DOF XArm is controlled at approximately 0.5 Hz. It adds depth to the UR5-style observations and manipulates a soft object connected to the gripper by a string to reduce corner trapping. Dreamer reaches 3.1 objects/minute in ten hours, comparable to the plotted human baseline; Rainbow does not learn successful transfer. The string is sometimes used to pull the object out of a corner.
Go to primary source ↓e-spheroPDF p. 7, Section 3.4; p. 8, Figure 7 and caption
Sphero Ollie receives RGB images and continuous motor commands at 2 Hz; heading is ambiguous from one image. Reward is negative L2 goal distance. Episodes end after 100 steps, followed by random high-power motor actions to randomize position. Dreamer reaches average distance 0.15 in units of area size, averaged over time steps, after two hours; DrQv2 performs similarly.
Go to primary source ↓e-adaptationPDF p. 13, Appendix A and Figure A.1
XArm training occurs after sundown. Sunrise substantially changes image observations and performance drops; continued learning recovers and then exceeds prior performance in approximately five hours. The figure includes night/sunrise observations and sunrise/recovery markers, but no frozen-agent or controlled illumination comparison.
Go to primary source ↓e-imaginationPDF p. 13, Appendix B, Figure B.1 and caption
Each row visualizes a decoded imagined trajectory, showing every second frame. Upper rows are UR5 and lower rows XArm. The caption identifies a stationary orange ball turning green in the second trajectory, illustrating imperfect visual prediction.
Go to primary source ↓e-limitsPDF p. 8, Section 5 Discussion, Limitations paragraph
The authors identify hardware wear and possible human intervention or repair, insufficient exploration of longer training budgets, and harder tasks as limitations or future work.
Go to primary source ↓e-positionPDF p. 14, Appendix C, Model-based RL paragraph
The paper distinguishes dynamics models used for online action planning from models used as simulators to train a policy, placing Dreamer in the latter category. It contrasts latent rollouts with action-conditioned video prediction and mentions reconstruction-free representation learning as future work.
Go to primary source ↓e-horizonPDF p. 4, Actor Critic Learning paragraph before Eq. (3); p. 15, Appendix D, Actor Critic / Imagination horizon row
The main text states H = 16 steps; the hyperparameter table states H = 15. No reconciliation of these values is supplied.
Go to primary source ↓e-diagram-indicesPDF p. 3, Figure 3 right panel and Eq. (1); p. 4, Eqs. (2)–(3)
The right panel repeats reward/value subscripts r_2 and v_2 above the later states, including the state labeled h_3. Eq. (1) maps the reward network at s_(t+1) to r_t, and Eqs. (2)–(3) define state-indexed values and returns. The schematic does not provide a consistent full temporal indexing specification.
Go to primary source ↓e-visual-scopePDF pp. 5–8, Figures 4–8 and Sections 3.1–3.4; p. 13, Figures A.1–B.1; p. 15, Appendix D
This edition reports quantitative outcomes through learning curves, with adaptation and imagined-rollout diagnostics in the appendices. The supplied paper contains no numerical performance table or controlled component-removal ablation; its unnumbered table lists hyperparameters.
Go to primary source ↓e-repro-gapsPDF pp. 3–4, Section 2; p. 5, Implementation and Section 3.1; p. 6, Section 3.2; p. 15, Appendix D
The source specifies algorithm components, robot interfaces and shared hyperparameters, but does not give a complete expanded world-model loss, camera preprocessing configuration, PD gains or Butterworth settings, package versions, or a per-experiment learner hardware and update-throughput specification. Page 4 describes typical latent behavior batches of 16K on a single GPU, without identifying the experimental GPU model.
Go to primary source ↓8.5 Primary sources
DayDreamer: World Models for Physical Robot Learning ↗
PDF · 7,717 extracted words
Source fingerprint
2ee10ae0961bef97ca522bb7ee5b7514e7a918a637dedad569b572b18bccb35e