DeepMind Control Suite
1. Paper overview
In one sentence: A standardized physics benchmark makes continuous-control scores interpretable, while its baselines show why observation modality and training budget must remain part of every comparison. e-designe-rewardse-aggregatee-pixelse-table-time
| At a glance | What to know |
|---|---|
| Research problem | Source description Continuous-control comparisons need stable physics, solvable tasks and interpretable scores. The suite organizes MuJoCo models into reusable domains and tasks, and tests them with learning agents to expose unstable simulation or unintended solutions. This establishes an empirical benchmark, rather than a learned predictor of future worlds. e-designe-domains |
| Core mechanism | |
| A key reported result | manipulator:bring_ball after 100 million training steps: D4PG state: 895.9 ± 3.7 Mean episode return ± standard error across seeds; higher is better. Online training and same-task simulated evaluation; Table 1 uses 100 episodes per seed, each 1,000 steps. State A3C/DDPG: 15 seeds; D4PG: 5. A3C: 0.4 ± 0.0; DDPG: 0.6 ± 0.1; D4PG pixels: 0.5 ± 0.1. A large advantage on this configured manipulation task, including its exploration-aiding initializations; not evidence for unrestricted manipulation or physical deployment. e-table-stepse-protocole-manipulator |
| Reading caution | Reader analysis Aggregate D4PG superiority has an early-data exception: DDPG leads before 1e7 environment steps. Claims that fewer actors improve D4PG data efficiency refer to experiments not shown. No isolated ablation separates distributional learning, replay changes and parallelism. e-aggregatee-d4pg |
Core contributions
- Source description
State-feature A3C, DDPG and D4PG baselines are accompanied by pixel D4PG and encoder-gradient comparisons. Both sample use and elapsed time are reported, exposing a tradeoff hidden by either budget alone. e-protocole-aggregatee-pixels
Figure 2. The reward-building primitive preserves a common scale while allowing different penalties outside the target interval. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the gray vertical lines at x=0 and x=1: every illustrated function gives one between them. Move outward by the displayed margin of one. In the upper panel, the magenta guides reach x=-1 and x=2, where the functions take value 0.2; their tails continue beyond that margin. The lower panel instead reaches zero at those locations and stays zero outside. The colored legends identify alternative shapes, not competing learned policies. The caption confirms the role of bounds, margin and value_at_margin, so the graphic should be read as a reward definition rather than an empirical learning curve. e-rewardse-protocol
What it supports. Averaging or multiplying terms confined to [0,1] keeps a compound reward in that range. This makes the suite's 1,000-step evaluation scores share a nominal 0–1,000 scale. The figure explains the mechanism behind comparable axes, while leaving each task's reward geometry configurable.
Where the evidence stops. Equal numerical ranges do not imply equal difficulty or equal attainable returns. Travel or swing-up time consumes part of an episode, and this figure does not measure how a particular tail shape affects learning.
2. Motivation
2.1 The problem and the proposed response
Continuous-control comparisons need stable physics, solvable tasks and interpretable scores. The suite organizes MuJoCo models into reusable domains and tasks, and tests them with learning agents to expose unstable simulation or unintended solutions. This establishes an empirical benchmark, rather than a learned predictor of future worlds. e-designe-domains
2.2 What this reading follows
The Control Suite turns a collection of simulated bodies into a common experimental interface: choose a domain and task, supply continuous controls, and collect observations and bounded rewards. Reading this paper well means separating that benchmark design from the agents used to demonstrate it. D4PG performs strongly across tasks, but the same agent can succeed with state features and fail with camera images. A distributed implementation also changes the relationship between environment steps and elapsed time. The five visuals below follow that chain from reward construction to aggregate comparisons, exact task results and a diagnostic of how visual representations are trained. e-designe-rewardse-aggregatee-pixelse-table-time
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator classification is supported: the contribution standardizes physical tasks and evaluates agents. MuJoCo supplies the transition dynamics; the documented agents learn policies and values. A world/action-model architecture or prediction quadrant is therefore not applicable, rather than an unverified One Model assignment. e-designe-mdpe-d4pge-domains
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one control step, then distinguish its endpoint from a terminal state
Choose cart-pole swing-up as a concrete entry point. The domain supplies the body and joints; the task starts the pole downward and specifies the observations and reward. A reset samples an initial condition, the policy produces an action array, and step advances MuJoCo before returning observation, reward, discount and step type. The low-level stepping order matters because positions and pixels are synchronized with the new state while force-dependent signals refer to the previous transition. Next distinguish the experiment's endpoint from the task's objective. The benchmark sums 1,000 rewards for evaluation, but a non-LQR LAST step still carries continuation discount one. Reader interpretation: the cutoff limits what was observed, not the value of every possible future action. Preserving that distinction is necessary when adapting an RL implementation to the suite. e-domainse-mdpe-apie-physicse-protocol
5.2 Read D4PG as a policy-and-value baseline with a distributed training system
The suite's transition comes from MuJoCo. Within that environment, DDPG learns an actor that produces actions and a critic that evaluates them. D4PG changes both learning and execution organization: its critic represents a categorical distribution of returns, its targets use five-step returns, replay is prioritized, and experience comes from 32 CPU actors while a GPU learner updates the networks. These choices explain why the two columns of the aggregate figure answer different questions. Environment-step curves measure how much interaction was consumed; time curves also reflect the rate at which the system gathers and processes it. Reader interpretation: a better time curve cannot by itself demonstrate a better learning rule. Because the paper changes several mechanisms together, the baseline comparison supports the configuration's performance, while leaving their individual causal contributions unresolved. e-mdpe-ddpge-d4pge-aggregate
5.3 Trace images through the actor without confusing shared weights with shared gradients
Pixel D4PG receives a stack of three camera frames rather than the default state features. Two convolution layers extract a representation, followed by the described normalized 50-unit transformation and the actor and critic pathways. Sharing convolution weights lets both pathways use the same visual representation, but does not require both losses to update it. In the critic-only variant, the actor still consumes visual features and learns to choose actions; its policy gradient simply stops before the shared convolution weights. The paper reports this as the best pixel setting and plots the competing gradient routes. Reader interpretation: this is evidence about how to train a control representation, not evidence of future-image prediction. Its limits are visible in the same figure: fish, humanoid and manipulation failures persist even when the critic trains the encoder. e-pixelse-pixel-ablatione-table-time
5.4 Training and inference
During training
A3C uses 32 workers and a shared actor–critic MLP, with RMSProp and an annealed learning rate. DDPG uses a single actor/learner, separate 300→200 actor and 400→300 critic MLPs, Adam at 1e-4, discount 0.99, replay capacity 1e6 and batches of 64. Hyperparameters are shared across tasks. e-a3ce-ddpge-protocole-limits
D4PG extends DDPG with a categorical return critic: 101 categories on [-150,150]. It uses 32 CPU actors, one GPU learner, five-step returns, prioritized replay sampling/eviction, batches of 256 and hard target updates every 100 steps. These changes are evaluated together, so their individual contributions are not isolated. e-d4pge-ddpg
Pixel D4PG stacks three 84×84 RGB frames from camera 0. Two 3×3, 32-channel ELU convolutions, first stride 2, precede a 50-unit layer with normalization and tanh. Actor and critic can share convolutions; the best variant reported by the authors updates those shared weights only through the critic. e-pixels
During inference
A3C samples Gaussian actions; deterministic-policy baselines map observations to control arrays. Training exploration noise is distinct from evaluation: DDPG's periodic evaluations remove it. Actions are executed in the simulator and followed by new observations. The documented baselines do not plan through a learned transition model. e-a3ce-ddpge-d4pge-api
Critic-only encoder training does not remove visual input from the deployed actor. It blocks the actor's training gradient into shared convolution weights; those weights still encode the images used to choose actions. e-pixels
5.5 Implementation flow
- Define the task
A domain supplies a physical model; its task specifies initialization, observations and rewards. Initial states vary to discourage rote solutions. Non-LQR actions occupy the unit box, and default features expose the state except in point-mass:hard, whose randomized actuator mapping requires memory. e-mdpe-domains
- Compose bounded rewards
Tolerance terms equal one inside a desired interval and optionally decay outside it. Averaging or multiplication preserves their [0,1] range. This produces comparable score scales, although time spent reaching the goal can make the attainable return lower than the nominal maximum. e-rewardse-protocol
- Advance physics and return feedback
The agent supplies an action to step; MuJoCo integrates the dynamics and the interface returns a TimeStep. State-dependent observations match the current state, whereas force-dependent sensors refer to the previous transition. Resetting state requires synchronization of derived quantities through reset_context. e-apie-physics
- Separate continuation from evaluation
Non-LQR tasks return discount 1 even at LAST. Their continuing objective is evaluated through a 1,000-step reward sum. Analysis: treating that observation cutoff as terminal would change the learning problem. The agent's own discount is distinct from the environment's continuation signal. e-apie-mdpe-protocole-ddpg
6. Experiments & results
DeepMind Control Suite standardizes simulated continuous-control tasks, interfaces and bounded rewards so reinforcement-learning agents can be compared across physical domains. Its baseline study favors distributed D4PG in aggregate, while revealing that pixel observations, training budgets and encoder updates strongly affect the conclusions.
This benchmark paper has no separate neural-network architecture diagram: Section 6 describes the baseline networks in prose, while Figure 2 supplies the original method visual for reward construction. Figures 6–7 diagnose pixel-encoder gradient routing, but omit the separate-encoder variant and do not isolate D4PG's other changes. The source reports baseline control returns, not learned-world prediction metrics or physical-robot experiments. e-rewardse-a3ce-ddpge-d4pge-pixelse-pixel-ablatione-limits
6.1 Read the original evidence
Figure 3. D4PG's aggregate advantage depends on whether the budget is measured in data or elapsed time. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the columns as different resource budgets: environment training steps on the left and elapsed training time on the right. The upper row compares orange A3C, dark-blue D4PG and light-blue DDPG using state-derived features. The lower row contains only pixel D4PG, as the original caption specifies; the shared legend appearing there is not evidence of pixel A3C or DDPG runs. Compare the early upper-left crossing with the sustained upper-right separation. Finally compare rows to see the aggregate cost of replacing state features with images. These are averages across tasks, so they must be checked against individual-task plots and tables. e-aggregatee-ddpge-d4pge-repro-gaps
What it supports. The authors identify D4PG as the strongest aggregate agent, with DDPG more data-efficient before 10 million environment steps. The time view favors the distributed configuration more strongly. Thus, reporting only a final return or only one budget would hide a material part of the baseline comparison.
Where the evidence stops. These configurations use different parallelization, and exact CPU/GPU models are not provided. The claim that reducing D4PG actors improves data efficiency refers to experiments not shown. The aggregate curves do not isolate that intervention.
Table 1. A fixed data budget exposes both D4PG's manipulation advantage and tasks where pixel control is competitive. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a domain and task pair first, then stay within that row while comparing algorithm columns. The retained note fixes the budget at 100 million training steps and specifies 100 evaluation episodes for each seed. Section 6 supplies the remaining conventions: 1,000 steps per evaluation episode, 15 seeds for state A3C and DDPG, five for D4PG, and table standard errors across seeds. The manipulator:bring_ball row shows a striking separation between state D4PG and all other columns. The ball_in_cup:catch row supplies a useful counterexample to any claim that camera-based control always fails. e-table-stepse-protocole-manipulator
What it supports. On bring_ball, state D4PG reaches 895.9 ± 3.7 while pixel D4PG reaches 0.5 ± 0.1 and DDPG 0.6 ± 0.1. On catch, pixel D4PG reaches 980.5 ± 0.5, close to state D4PG's 981.2 ± 0.7. The observation gap depends strongly on the task.
Where the evidence stops. These values are returns, not success percentages. Bring_ball includes exploration assistance: in 10% of episodes the object starts in the gripper or at the target. This is part of the evaluated task, and the result does not establish unrestricted manipulation.
Table 2. The time-budget table preserves the large state–pixel gaps on manipulation and swimming. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the column meanings from Table 1 but change the budget: the note now specifies 24 hours of training. The evaluation still uses 100 episodes per seed, and Section 6 retains the 1,000-step episode and seed conventions. Begin with manipulator:bring_ball to check whether the strong state D4PG result survives this alternative resource budget. Then read fish:swim across the two D4PG columns to compare observation modalities without changing the named algorithm. A cross-table comparison describes two reported budget conditions; it should not be treated as a controlled estimate of the effect of extra training on identical hardware. e-table-timee-protocole-limitse-d4pge-repro-gaps
What it supports. State D4PG obtains 903.7 ± 3.9 on bring_ball, versus pixel D4PG's 0.6 ± 0.1. For fish:swim, the corresponding values are 852.5 ± 2.7 and 76.8 ± 4.2. High state-feature performance therefore coexists with substantial visual-control failures under the same nominal time budget.
Where the evidence stops. The authors suggest camera placement as a possible contributor to fish and swimmer failures, but provide no camera ablation. Algorithm parallelization differs, and the source does not specify hardware models, limiting portable wall-clock conclusions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| manipulator:bring_ball after 100 million training steps Online training and same-task simulated evaluation; Table 1 uses 100 episodes per seed, each 1,000 steps. State A3C/DDPG: 15 seeds; D4PG: 5. | D4PG state: 895.9 ± 3.7 Mean episode return ± standard error across seeds; higher is better | A3C: 0.4 ± 0.0; DDPG: 0.6 ± 0.1; D4PG pixels: 0.5 ± 0.1. A large advantage on this configured manipulation task, including its exploration-aiding initializations; not evidence for unrestricted manipulation or physical deployment. e-table-stepse-protocole-manipulator |
| manipulator:bring_ball after 24 hours Table 2; 24-hour training budget, 100 evaluation episodes per seed and 1,000 steps per episode; 15 A3C/DDPG seeds and 5 D4PG seeds. | D4PG state: 903.7 ± 3.9 Mean episode return ± standard error across seeds | A3C: 0.6 ± 0.1; DDPG: 1.2 ± 0.4; D4PG pixels: 0.6 ± 0.1. The advantage also appears under the time budget, but different parallelization and unspecified hardware models prevent treating this as a hardware-neutral comparison. e-table-timee-protocole-d4pge-ddpge-repro-gaps |
| ball_in_cup:catch after 100 million training steps Table 1; 100 evaluation episodes per seed, 1,000 steps each; D4PG state and pixels each use 5 seeds. | D4PG pixels: 980.5 ± 0.5 Mean episode return ± standard error across seeds | D4PG state: 981.2 ± 0.7; DDPG state: 984.5 ± 0.3. Pixel control nearly matches the state-feature baseline here; pixel failure is task-dependent rather than universal. e-table-stepse-protocol |
| fish:swim after 24 hours Table 2; 100 evaluation episodes per seed, 1,000 steps each; 5 seeds for each D4PG modality. | D4PG state: 852.5 ± 2.7; D4PG pixels: 76.8 ± 4.2 Mean episode return ± standard error across seeds | DDPG state: 666.8 ± 6.8; A3C state: 395.7 ± 7.3. A substantial observation-modality gap. The authors suggest camera framing as a contributor, but do not intervene on the camera to establish causality. e-table-timee-protocole-limits |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Changing which loss trains the shared visual encoder is a mechanism-relevant diagnostic, with clear task-dependent limits. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the upper-left legend and the explanation on page 16 together. Dotted green means actor-only gradients enter the shared convolution layers; dashed green means critic-only; solid green means both. Blue is the state-feature reference. These line styles describe training updates, not whether the actor receives images at inference. Follow a column downward to the shared time labels, which span one day. The vertical axis is labelled Mean Return; Section 6 specifies that plotted lines summarize medians across seeds and shaded regions their 5th–95th percentiles. Compare catch and finger turning, where gradient routing matters, with fish and humanoid panels, where pixel variants remain weak. e-pixel-ablatione-pixelse-protocole-d4pg
What it supports. The source reports the best pixel configuration as shared convolutions trained only through the critic. Several panels visibly show actor-only training failing where critic-involving variants learn. Other panels show that changing gradient routing alone does not close the gap to state features, so representation updates are only part of the pixel-control problem.
Where the evidence stops. Critic-only is an overall author conclusion, not uniform dominance in every panel. The separately encoded actor/critic variant was tried but is absent from the plots. This comparison does not isolate D4PG's distributional critic, replay prioritization or actor parallelism.
7. Analysis & limitations
7.1 What the evidence leaves open
Aggregate D4PG superiority has an early-data exception: DDPG leads before 1e7 environment steps. Claims that fewer actors improve D4PG data efficiency refer to experiments not shown. No isolated ablation separates distributional learning, replay changes and parallelism. e-aggregatee-d4pg
Shared hyperparameters and limited grid searches make these starting baselines, not performance ceilings. Initialization sequences vary across runs. Plot bands are 5th–95th percentiles, whereas table errors are standard errors; neither is a reported significance test. e-protocole-limits
The source deliberately excludes rich task/model distributions such as complex-terrain locomotion. Pixel failures and omitted separate-encoder curves limit architectural conclusions. No held-out task/model generalization or physical robot evaluation is established by these experiments. e-limitse-pixel-ablation
7.2 Questions for discussion
- Would a camera intervention reduce the fish/swimmer pixel gap while leaving state-feature control unchanged?
- How much of D4PG's wall-clock advantage remains when actor count and hardware are controlled?
8. Reproducibility audit
8.1 Requirements and known gaps
Recreate the v1 task definitions, reward functions, observation timing and truncation semantics. Preserve manipulator's 10% assisted initializations. Use the reported encoder, replay and actor/learner configuration, and report step-limited and time-limited budgets separately with the original seed and episode counts. e-apie-physicse-manipulatore-pixelse-d4pge-protocole-table-stepse-table-time
Exact reproduction still needs software versions, hardware models and full referenced algorithm details. Selected A3C unroll length and second-layer width are not unambiguously identified from the listed search alternatives. Episode-initialization sequences are uncontrolled; network seeds alone cannot reconstruct every trajectory. e-a3ce-protocole-repro-gaps
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the encoder-gradient comparison with explicit controls
Reader-proposed check, not executed: train pixel D4PG on ball_in_cup:catch and finger:turn_hard using actor-only, critic-only and both-gradient shared encoders, plus the separately encoded variant omitted from the paper's plots. Hold camera 0, three-frame 84×84 input, layer sizes, initialization seed set, actor count, optimizer settings, evaluation episodes and training-step budget fixed. Use several paired seeds and report individual runs as well as aggregate returns and encoder gradient norms. Verify directly that blocked actor gradients do not reach the shared convolutions. The falsifiable expectation is that critic-only or critic-involving updates reproduce the strong advantage over actor-only training; comparable actor-only returns would weaken the reported architectural lesson. e-pixelse-pixel-ablatione-protocole-d4pg
Check 2: Test the camera explanation for swimming failures
Reader-proposed check, not executed: compare the original camera-0 pixel D4PG setup with a camera selected in advance to keep both swimmer body and target visible on fish:swim and swimmer:swimmer6. Change only the camera; retain image resolution, frame stack, rewards, critic-only shared-encoder updates, actor count, training-step budget and paired seed protocol. Include state-feature D4PG as a control and record target visibility alongside evaluation return. The camera hypothesis predicts improved pixel returns when visibility improves, with state-feature performance unaffected by rendering choice. If better visibility leaves pixel learning poor, the result would point toward additional representation or optimization limitations rather than establishing camera placement as the sufficient explanation. e-limitse-pixelse-pixel-ablatione-table-timee-domains
8.3 Reading coverage
Visual audit: All 24 pages of the supplied v1 PDF were rendered and visually inspected, including the title and byline, domain examples, APIs, equations, architecture and hardware descriptions, all seven numbered figures, both result tables and references. Pages 17–19 contain full benchmark plots despite sparse extracted text. All five final original crops were individually viewed and retain their legends, axes or table protocol notes. Figure 2 was checked against its caption and reward definition; Figures 6–7 were checked against the page 15–16 gradient-routing description. The same legend in Figure 3 does not indicate additional pixel baselines. No scientific graphic was reconstructed. Separate supplements, external implementations and later revisions are outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/byline/version (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Structure and Design (pp. 2–4)
- 3 Domains and Tasks, including CMU motion-capture tools and LQR (pp. 4–7)
- 4 Reinforcement learning API (pp. 7–9)
- 5 MuJoCo Python interface (pp. 9–12)
- 6 Benchmarking: protocols, architectures, state and pixel results (pp. 12–15; continuation p. 16; plots/tables pp. 17–22)
- 7 Conclusion and future work (p. 16)
- References (pp. 23–24)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Pages with little extracted text require visual inspection: 17, 18, 19
- The extraction-only figure and low-text-page cautions above were addressed by visually inspecting all 24 PDF pages, including pages 17–19, and all five final crops. Separate supplemental availability remains unverified.
- Version scope: the reviewed PDF is arXiv:1801.00690v1, dated 2 January 2018 in the arXiv strip and 3 January 2018 on the title page. Title and all 12 authors match the catalog. The supplied provenance identifies the previous HTML as the same v1; that HTML and any later editions were not independently compared.
- No external code, linked videos, motion-capture files or cited papers were inspected; no experiments were executed. The title page supplies no explicit affiliations.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, complete byline, printed date and rotated arXiv strip
The observed title is DeepMind Control Suite. The byline lists Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap and Martin Riedmiller. The strip identifies arXiv:1801.00690v1, 2 January 2018; the printed date is January 3, 2018. No affiliation block is printed.
Go to primary source ↓e-designPDF pp. 1–3, Abstract; Sections 1–2, Model and Task verification
The suite supplies Python continuous-control tasks backed by MuJoCo and MJCF physical models. The authors iterated on stability, exploitability and solvability using learning agents; the benchmarking set contains tasks solved by some agent.
Go to primary source ↓e-mdpPDF p. 3, Section 2, State, Action, Dynamics, Observation and Termination and Discount
The source defines state s, action a, transition f, observation o and reward r. Initial states are non-singleton; actions are bounded except in LQR. State-derived features are strongly observable except for point-mass:hard. Dynamics are numerically integrated; agents use discounted returns as a practical alternative to the continuing average-return objective.
Go to primary source ↓e-rewardsPDF pp. 3–4, Section 2, Reward and Evaluation; Figure 2 and caption
Non-LQR rewards lie in [0,1]. The tolerance function gives one within bounds; a margin supplies decreasing tails. Figure 2 uses bounds (0,1) and margin 1, with value_at_margin 0.2 for infinite-support examples and 0 for finite-support examples. Averaging or multiplying bounded terms preserves the reward range.
Go to primary source ↓e-domainsPDF pp. 4–7, Section 3, domain/task distinction, BENCHMARKING/EXTRA tags, Point-mass and LQR entries
A domain is a physical model and a task its MDP configuration. EXTRA contains particularly difficult or nonstandard tasks. Point-mass:hard randomizes the action gain matrix and requires memory. LQR uses unbounded controls and quadratic rewards and lies outside the benchmarking set. The suite includes locomotion, reaching and manipulation tasks.
Go to primary source ↓e-manipulatorPDF p. 6, Section 3, Manipulator entry
Manipulator has state, control and observation dimensions (22,5,37). It rewards bringing an object to a target; in 10% of episodes the object starts in the gripper or at the target to aid exploration. Only bring_ball among the four named manipulator tasks belongs to the benchmarking set.
Go to primary source ↓e-apiPDF pp. 8–9, Section 4, RL Environment class, suite module and Pixel observations
Actions are NumPy arrays and observations are ordered dictionaries. reset and step return step_type, reward, discount and observation. All non-LQR tasks return discount 1 even at a LAST step, indicating continuing-task truncation. A pixel wrapper can replace or supplement state features.
Go to primary source ↓e-physicsPDF pp. 10–11, Section 5, Physics class, reset_context and Running the simulation
Physics loads MJCF models and renders RGB or depth images. reset_context synchronizes derived quantities after setting state. The normal stepping path calls mj_step2 followed by mj_step1; state-dependent quantities match the current state, while force/acceleration-dependent sensors refer to the previous transition. The RK4 footnote specifies a concluding mj_step1.
Go to primary source ↓e-protocolPDF pp. 12–13, Section 6, evaluation, seeds and uncertainty; p. 15, Results: Learning from state features; pp. 21–22, Tables 1–2 captions
Evaluation sums rewards over 1,000 steps. Tables use 100 episodes per seed and means/standard errors across seeds; state A3C/DDPG use 15 seeds and D4PG/pixel runs use 5. Curves use medians and 5th–95th percentiles across seeds. Initialization sequences are not fixed across runs. DDPG is evaluated without exploration noise for 10 episodes per 100,000 steps; A3C has a concurrent evaluator.
Go to primary source ↓e-a3cPDF p. 13, Section 6, Algorithm and Architecture Details, A3C
A3C uses 32 workers, two shared MLP layers, a diagonal-Gaussian action distribution and RMSProp. The starting learning rate is 5e-5, linearly annealed to zero, and entropy regularization has weight 3e-3. The grid includes alternative second-layer widths and unroll lengths.
Go to primary source ↓e-ddpgPDF pp. 14–15, Section 6, DDPG
DDPG uses one actor/learner, ReLU actor layers of 300 and 200 units and critic layers of 400 and 300 units. Independent Adam optimizers use learning rate 1e-4, discount 0.99, soft target updates, replay capacity 1e6, minibatches of 64 and temporally correlated exploration noise.
Go to primary source ↓e-d4pgPDF p. 15, Section 6, D4PG
D4PG uses a categorical return critic with 101 categories on [-150,150], 32 CPU actors and one GPU learner, five-step returns and prioritized replay sampling/eviction with exponents 0.6. It uses minibatches of 256, hard target updates every 100 steps and actor-dependent Gaussian exploration standard deviations from 1/32 to 1. Algorithm details are delegated to cited work.
Go to primary source ↓e-pixelsPDF pp. 15–16, Section 6, Results: Learning from pixels
Pixel D4PG stacks three 84×84 RGB frames from camera 0. Two 3×3, 32-channel ELU convolutions (first stride 2) feed a 50-unit normalized tanh layer. The tested variants separate actor/critic encoders or share convolution weights with actor-only, critic-only or both gradient sources. The authors report best performance with shared convolutions updated only by the critic.
Go to primary source ↓e-aggregatePDF pp. 13–14, Section 6 aggregate comparison; Figure 3, four panels and caption
Figure 3 plots task-average returns against environment steps and wall-clock time. D4PG leads the aggregate comparison except that DDPG is more data-efficient before 1e7 steps. The lower panels show pixel D4PG only despite the shared legend listing all three algorithms. The claimed benefit of reducing D4PG actor count is explicitly based on experiments not shown.
Go to primary source ↓e-state-curvesPDF pp. 17–18, Figures 4–5, legends, axes and manipulator:bring_ball panels; p. 15, state-feature results
The state-feature plots compare A3C, DDPG and D4PG by environment steps and one day of training. D4PG learns manipulator:bring_ball while the other shown state baselines remain near zero. The figures show broad differences in learning speed and variability across tasks.
Go to primary source ↓e-pixel-ablationPDF pp. 19–20, Figures 6–7, legend and ball_in_cup:catch, finger:turn_easy/turn_hard, fish:swim and humanoid panels; pp. 15–16, pixel-results text
Dotted green is actor-only convolution training, dashed green critic-only, solid green both, and solid blue state D4PG. Actor-only learning is poor on several tasks where critic-involving updates succeed. All pixel variants remain poor on some domains. The separate-encoder variant is described as worse but omitted from the plots. The curve uncertainty convention is stated on p. 13.
Go to primary source ↓e-table-stepsPDF p. 21, Table 1, header, caption and manipulator/bring_ball, ball_in_cup/catch, fish/swim rows
After 1e8 training steps, manipulator:bring_ball returns are A3C 0.4±0.0, D4PG 895.9±3.7, pixels 0.5±0.1 and DDPG 0.6±0.1. Ball-in-cup returns are 104.7±7.8, 981.2±0.7, 980.5±0.5 and 984.5±0.3 in the same column order. Fish:swim returns are 81.3±1.1, 844.3±3.1, 72.2±3.0 and 492.7±9.8. The caption specifies 100 evaluation episodes per seed.
Go to primary source ↓e-table-timePDF p. 22, Table 2, header, caption and manipulator/bring_ball, fish/swim and ball_in_cup/catch rows
After 24 hours, manipulator:bring_ball returns are A3C 0.6±0.1, D4PG 903.7±3.9, pixels 0.6±0.1 and DDPG 1.2±0.4. Fish:swim returns are 395.7±7.3, 852.5±2.7, 76.8±4.2 and 666.8±6.8. Ball-in-cup D4PG state/pixel returns are 979.8±0.7 and 980.5±0.6. The caption specifies 100 evaluation episodes per seed.
Go to primary source ↓e-limitsPDF p. 12, Section 6 baseline caveats; p. 16, pixel failures and Section 7, Future work
Hyperparameters are shared across tasks and not exhaustively optimized. The authors suggest unsuitable camera views as a likely contributor to fish/swimmer pixel failures, without a camera intervention. Rich manipulation and complex-terrain task distributions are outside the release's intended simple benchmark scope. Quadrupeds and several interface/platform features are future work.
Go to primary source ↓e-repro-gapsPDF pp. 12–16, Section 6, benchmark configuration; p. 23, Anonymous 2017a/2017b references
The paper provides baseline architectures, optimizer settings, actor/learner counts, observations and evaluation budgets, but defers complete algorithm specifications to references, including submissions listed anonymously. It does not identify exact CPU/GPU models or software versions in this configuration description, and does not fix episode-initialization sequences.
Go to primary source ↓8.5 Primary sources
DeepMind Control Suite ↗
PDF · 8,654 extracted words
Source fingerprint
cdd89fa0165267b7a248ff694a578bee58df12293cdcafebf6918cb241160377