Diversity is all you need: Learning skills without a reward function
1. Paper overview
In one sentence: DIAYN turns state–skill discriminability into an intrinsic reward, building reusable stochastic policies whose downstream usefulness still depends on what the discriminator can distinguish. e-loope-objectivee-transfere-hierarchye-prior
| At a glance | What to know |
|---|---|
| Research problem | Source description Sparse rewards and expensive human feedback make task-specific exploration difficult. The paper asks whether an agent can first interact without knowing its eventual reward, acquiring reusable behaviors that reduce later learning effort. This assumes access to environment interaction; it removes external reward supervision during pretraining, not the need for an optimization signal. e-motivatione-objective |
| Core mechanism | Source description A variational information objective combines state–skill discriminability with stochastic control. A fixed categorical prior prevents early successful skills from monopolizing training. e-objectivee-prior |
| A key reported result | Hierarchical half-cheetah hurdle control: Approximately 4–4.5 reward near 9 hours, read from the solid DIAYN curve in Figure 7. Task reward versus training hours; higher is better. Modified HalfCheetah-v1 with repeated hurdles; meta-controller selects a skill every 10 steps. TRPO and VIME fluctuate substantially lower, roughly around 1–2 over this interval; SAC remains near zero. Graph-read estimates, not tabulated scores. The dashed DIAYN continuation is not treated as additional measured samples; uncertainty and seed aggregation are not specified for this plot. e-hierarchye-environments |
| Reading caution | Source description Discriminability need not match downstream usefulness. Ant skills often follow arcs rather than running straight; Figure 15 reports no learned skill above 1,000 on that benchmark. More entropy broadens exploration but excessive entropy makes skills harder to distinguish. e-skillse-entropy |
Core contributions
- Source description
A variational information objective combines state–skill discriminability with stochastic control. A fixed categorical prior prevents early successful skills from monopolizing training. e-objectivee-prior
- Source description
The authors demonstrate emergent locomotion and three downstream uses: policy initialization, temporal composition and retrieval for imitation. These are simulated control experiments, not physical robot deployment. e-skillse-transfere-hierarchye-imitation
Figure 1 and Algorithm 1. Environment interaction supplies the states; skill classification supplies the reward. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the green prior block: one categorical skill z is sampled for an entire episode. The blue policy receives z and the current state, then sends action aₜ into the green environment. Its returned state sₜ₊₁ feeds both continued control and the blue discriminator. Green means fixed, blue means learned; the transition distribution inside the environment box is therefore not a learned predictor. Read the adjacent pseudocode to distinguish execution from optimization. The policy is updated with SAC using the discriminator-derived reward, while the discriminator is updated to identify the sampled skill. These updates reinforce each other rather than forming an adversarial game. e-loope-objective
What it supports. The method learns behavior by rewarding distinguishable consequences of executed actions. It does not train a future-state generator or infer an expert's missing actions. The fixed prior preserves opportunities to train each skill; stochasticity is introduced through the SAC objective described in Section 3.2.
Where the evidence stops. The feedback arrow is labeled qφ(z | sₜ), but the discriminator block and Algorithm 1 use sₜ₊₁. This edition follows the algorithm's next-state timing. Section 3.1 also prints I(A;Z) where Equation 1 and the prose specify state–skill information.
2. Motivation
2.1 The problem and the proposed response
Sparse rewards and expensive human feedback make task-specific exploration difficult. The paper asks whether an agent can first interact without knowing its eventual reward, acquiring reusable behaviors that reduce later learning effort. This assumes access to environment interaction; it removes external reward supervision during pretraining, not the need for an optimization signal. e-motivatione-objective
2.2 What this reading follows
A reward-free agent still needs a reason to do one thing rather than another. DIAYN supplies that reason through a classification problem: after observing a state, can a discriminator identify the skill that produced it? The policy earns intrinsic reward for making the answer clear, while action entropy encourages exploration and a uniform skill prior keeps every option in training. The figures below follow this loop into reward-based adaptation, hierarchical tasks and imitation. They also expose its boundaries: faster fine-tuning excludes pretraining cost, ant navigation benefits strongly from added state information, and a large sampled skill count is not itself proof of useful behavioral diversity. e-loope-objectivee-transfere-hierarchye-prior
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The catalog's foundational-policy placement and not-applicable world-action-model axes are supported. DIAYN learns an action policy and a state-to-skill discriminator; its environment transition is fixed. Neither joint future/action prediction nor inverse dynamics appears. Multiple learned components alone do not make this a world-model/action-model architecture. e-loope-objectivee-hierarchy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a classifier can teach a policy
Imagine fixing a skill label before an episode without telling the agent to run, jump or balance. The label initially has no behavioral meaning. DIAYN creates meaning by training qφ(z | s) to recover that label from visited states, then rewarding the policy when recovery is easy. Equation 1 uses state–skill information because action differences can be invisible in their environmental consequences. Equation 2 exposes the remaining ingredient: high action entropy within each skill. The policy should therefore remain stochastic while producing recognizable states. Reader interpretation: this encourages distinguishable regions of behavior, but does not establish that those regions align with future goals. The hallway example makes another subtle point: skills can temporarily share states if their later consequences become distinguishable. Recognition is rewarded cumulatively, not imposed as a hard separation at every step. e-objectivee-loope-overlap
5.2 Separate choosing a skill from learning a task
Once a repertoire exists, DIAYN offers several interfaces to it. Fine-tuning evaluates skills with the new task reward, selects the strongest and transfers policy and value weights into further learning. Hierarchical control instead learns a meta-controller whose actions select a skill for a fixed interval. This changes the temporal action space while retaining low-level feedback from observed states. The distinction matters when interpreting Figure 7: hurdle performance tests composition, whereas Figure 5 tests initialization. Neither means that downstream reward has disappeared. The ant experiment then shows how representation shapes the repertoire itself: supplying center-of-mass information to the discriminator biases discovery toward spatial movement. Reader interpretation: that improvement reveals a useful intervention, but also a dependence on what an observer is allowed to recognize as a difference between skills. e-transfere-hierarchye-loop
5.3 Ask what the diversity evidence actually measures
The fixed-prior plot, entropy sweep and imitation diagnostic address different links in the argument. Figure 17 measures how evenly the algorithm samples labels; its constant DIAYN line follows from uniform sampling. Appendix D.2 shows trajectory shapes as entropy changes, but gives no numeric coverage score. Figure 22 asks whether the resulting repertoire can match another seed's demonstrated behavior. These observations are complementary, yet none alone proves broad downstream competence. The theory has similarly specific scope: Appendix B assumes stationary visitation and proves a two-skill unregularized optimum, followed by an entropy-regularized near-optimality statement. Reader interpretation: the main text's stronger uniqueness wording should not be transferred to continuous-control neural policies. A convincing reproduction should connect sampling balance to measured state coverage and held-out task performance while preserving these distinct definitions. e-priore-entropye-imitatione-grid
5.4 Training and inference
During training
The main implementation uses entropy coefficient α = 0.1. Appendix C specifies 300 hidden units for policy, Q and value networks, rather than the referenced SAC implementation's 128, and episodes of at most 1,000 steps. Other SAC hyperparameters are inherited by citation, not fully restated. e-implementation
The pseudo-reward baseline affects termination incentives. Appendix A motivates retaining −log p(z) to discourage early termination, under its better-than-chance discriminator assumption; its footnote allows removal when rapid termination is desirable. This is a task-dependent choice, not a universal nonnegativity guarantee. e-baseline
During inference
For direct execution, fix z and repeatedly map the observed state to an action. For fine-tuning, evaluate skills with the task reward, select the best and initialize both policy and value networks before reward-based learning. e-loope-transfer
A downstream meta-controller chooses skills for k steps: k = 10 for cheetah hurdles and 100 for ant navigation. The low-level policy still uses state feedback within each interval. The ant DIAYN+prior variant supplies the discriminator with center-of-mass information. e-hierarchy
For imitation from states without expert actions, enumerate skills and choose the one maximizing the product of discriminator probabilities along the demonstration. Execute that existing feedback policy; no inverse-dynamics model is trained and no new skill is synthesized. e-imitation
5.5 Implementation flow
- Sample a skill and execute it
Draw z from a fixed uniform categorical prior at the beginning of an episode. Concatenate z with the current state for the policy, Q function and value function. Sample an action, execute it in the environment and observe the next state; the environment transition is not a learned world-model prediction. e-loope-implementation
- Turn classification into intrinsic reward
Train the discriminator to infer the sampled skill from individual visited states. Algorithm 1 evaluates the resulting next state and assigns log qφ(z | sₜ₊₁) − log p(z). The policy and discriminator cooperate: both benefit when the generated states identify the skill. e-loope-objective
- Preserve stochasticity and sampling coverage
SAC maximizes the intrinsic return plus action entropy. Random actions must remain compatible with state discriminability, encouraging broader yet distinguishable behavior. A uniform prior keeps every skill eligible for training. State overlap is allowed when shared transitions eventually reach distinguishable regions. e-objectivee-priore-overlap
6. Experiments & results
DIAYN learns a repertoire of stochastic, state-feedback skills without an external task reward. A discriminator rewards states that reveal which skill generated them, while action entropy and a fixed skill prior discourage narrow behavior and neglected skills. The repertoire supports reward-based adaptation, hierarchical control and state-only imitation, but task usefulness depends on what states become distinguishable (e-objective, e-loop, e-transfer, e-hierarchy, e-imitation).
This supplied preprint contains no numerical results table: its quantitative comparisons are learning curves, distributions and diagnostics. Six original figures therefore cover method, results and ablations. Plot-derived values are explicitly approximate; missing tabulated scores, unspecified uncertainty and absent timing hardware are preserved rather than reconstructed. e-visual-formate-transfere-hierarchye-implementatione-imitation
6.1 Read the original evidence
Figure 5. Pretrained policy and value weights improve early downstream learning. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel as a separate benchmark: their reward scales and time spans differ. Blue starts from the DIAYN skill with the highest evaluated task reward; orange starts from random weights. Both subsequently learn with the task reward, and the initialization includes the value networks as well as the policy. The source states that the architecture is matched and the plotted curves average five random seeds. Follow the early portions of the curves to see the initialization advantage; do not interpret the three panels as interchangeable numerical scores. The horizontal axis is labeled hours, even though the appendix discusses training budgets in epochs and steps. e-transfere-implementation
What it supports. The blue curves reach useful rewards earlier in all three tasks, supporting DIAYN as an initialization for subsequent reward-based learning. The advantage is most clearly about the learning trajectory: the endpoints can become close, especially for hopper and ant, without erasing the earlier difference.
Where the evidence stops. Section 4.2.1 explicitly excludes unsupervised pretraining under a free-or-amortized-cost assumption. These curves therefore do not show total training cost. Seed dispersion is not displayed, and hardware needed to interpret the hours axis is not specified.
Figure 7. Composing skills helps hurdle control; ant navigation exposes the value of task-relevant state information. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. On the left, the red DIAYN curve rises to approximately 4–4.5 reward near nine hours, above the fluctuating TRPO and VIME curves and nearly flat SAC line. Only the solid segment is used for that estimate. On the right, distinguish red DIAYN from purple DIAYN+prior: the latter gives the discriminator center-of-mass information. Around fifteen hours, purple is approximately 2.2–2.4 while red remains about 0.4–0.5. The meta-controller selects a skill for ten cheetah steps or one hundred ant steps. Appendix C defines the ant score as up to five ordered waypoint rewards, so the plotted range is not a success percentage. e-hierarchye-environments
What it supports. The strongest hurdle result supports using discovered skills as temporal actions. Ant provides a different lesson: its largest gain comes from an additional prior about useful state features. Even the stronger curve remains below the maximum of five, so this figure does not establish dependable completion of the whole route.
Where the evidence stops. These are approximate plot readings without reported seed aggregation or uncertainty for Figure 7. The dashed cheetah continuation is not counted as further measurements. DIAYN+prior must not be reported as the fully task-agnostic variant.
Figure 22. The discriminator can help retrieve a matching skill and diagnose poor matches. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Move rightward toward less negative discriminator log scores; downward means smaller L2 distance between the retrieved skill's trajectory and the expert trajectory. Blue denotes DIAYN, orange uses only five skills, green reduces entropy, and red learns the prior. All other variants use fifty skills. The source aggregates six hundred synthetic imitation tasks whose expert trajectories come from another random seed of DIAYN. This prevents exact reuse of a current skill's trajectory, but keeps experts within the same broad behavior-generating procedure. Read the blue-versus-green separation mainly in the lower-score region, since they approach each other near the high-score end. e-imitation
What it supports. Higher discriminator scores are associated with smaller imitation error in this experiment, and DIAYN generally achieves the strongest matching curve. Red and orange illustrate costs of concentrating sampling or reducing repertoire size. The method retrieves an existing policy using expert states, so successful matching need not require expert action labels.
Where the evidence stops. The source does not report a tabulated aggregate error, confidence intervals or a calibrated probability of safe execution. Synthetic experts restrict the conclusion. Figure 9's failed handstand demonstrates that retrieval cannot guarantee a skill for every requested behavior.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Hierarchical half-cheetah hurdle control Modified HalfCheetah-v1 with repeated hurdles; meta-controller selects a skill every 10 steps. | Approximately 4–4.5 reward near 9 hours, read from the solid DIAYN curve in Figure 7. Task reward versus training hours; higher is better | TRPO and VIME fluctuate substantially lower, roughly around 1–2 over this interval; SAC remains near zero. Graph-read estimates, not tabulated scores. The dashed DIAYN continuation is not treated as additional measured samples; uncertainty and seed aggregation are not specified for this plot. e-hierarchye-environments |
| Hierarchical ant waypoint navigation Modified Ant-v1; five ordered waypoint rewards, maximum +5; k = 100. | DIAYN+prior reaches approximately 2.2–2.4 near 15 hours; plain DIAYN is roughly 0.4–0.5. Accumulated waypoint reward; higher is better | The added center-of-mass prior accounts for the pronounced separation from plain DIAYN, TRPO, VIME and SAC. Approximate Figure 7 readings do not establish reliable completion of all five waypoints. The stronger variant includes task-relevant prior knowledge. e-hierarchye-environments |
| Reward-based fine-tuning of locomotion skills Half-cheetah, hopper and ant; best pretrained skill versus random initialization, same architecture; five-seed averages. | Figure 5 shows faster early learning with DIAYN initialization in all three environments; exact endpoint values are not tabulated. Benchmark reward versus downstream training hours | Both policy and value networks are initialized from pretraining; the baseline starts from scratch. Pretraining cost is explicitly omitted under a free-or-amortized-pretraining assumption. This does not demonstrate lower total interaction or compute cost. e-transfere-implementation |
| Skill-prior collapse diagnostic Inverted pendulum and mountain car; three seeds per environment in Figure 17; fixed versus learned prior. | Fixed uniform sampling retains 50 effective skills; learned-prior curves fall to roughly 1–3 late in training. Effective sampled skills, exp(H[Z]); higher indicates broader sampling | Appendix E reports an order-of-magnitude reduction; the half-cheetah comparison appears separately in Figure 4. This measures sampling entropy, not independently verified behavioral diversity. The VIC-labeled comparator is implemented by learning the prior within DIAYN. e-prior |
| State-only imitation by skill retrieval 600 synthetic imitation tasks using expert trajectories from another DIAYN seed; classic-control evaluation. | Figure 22 shows generally lower distances for DIAYN; curves approach the low-entropy variant at high scores. No aggregate numerical error is tabulated. L2 trajectory distance versus discriminator log score; lower distance is better | Lower entropy, learned prior, and five skills instead of 50; qualitative half-cheetah examples succeed on three of four demonstrations. Synthetic expert generation limits generalization claims. The handstand fails; score–distance association is not calibrated execution-safety evidence. e-imitation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Appendix D.2, unnumbered entropy-sweep panels. Action entropy trades narrow distinguishable motion for broader exploration. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read from left to right as the entropy coefficient α increases, keeping in mind that these are trajectory examples rather than reward curves. The point mass controls its orientation and forward velocity inside a box. At α = 0.01, paths travel along relatively narrow directions. At α = 1, trajectories spread through wider regions. At α = 10, substantial overlap appears. The continuation of Appendix D.2 on the next page explains that very high entropy makes skills difficult to distinguish. The panels have different displayed coordinate ranges, so compare trajectory structure rather than estimating a common occupied-area percentage from the image. e-entropye-objective
What it supports. The sweep illustrates why discriminability and exploration must be balanced. Low entropy can produce distinct but narrow behavior; stronger entropy creates more varied paths while eventually making classification harder. It motivates a tunable entropy coefficient, not a conclusion that increasing randomness always improves skill discovery.
Where the evidence stops. The source gives no quantitative coverage measure, uncertainty or downstream test for this sweep. The three shown coefficients differ from the main implementation's α = 0.1; these images do not establish a universal optimal coefficient.
Figure 17. A fixed uniform prior prevents collapse in how often skills are sampled. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First read the vertical axis carefully: effective skill count is exp(H[Z]), not the number of visually verified behaviors. With fifty uniformly sampled skills it remains fifty by construction, producing the orange dashed line. Blue curves show three runs of the learned-prior comparator in each environment; their effective counts fall toward roughly one to three late in training. Appendix E constructs this VIC-labeled comparison by adapting the prior within DIAYN according to each skill's discriminator score. The two panels here are inverted pendulum and mountain car. The corresponding half-cheetah comparison appears separately in Figure 4, despite Appendix E.2 referring broadly to all three. e-prior
What it supports. The plot supports the proposed feedback problem: skills sampled more often get more opportunities to improve, while others can be neglected. Keeping the prior uniform removes that route to sampling collapse. The learned-prior runs provide an empirical contrast to a fixed-count reference that is guaranteed by the algorithm.
Where the evidence stops. A constant effective count does not prove fifty distinct or useful skills. The comparator is a DIAYN-based implementation of learning the prior, not evidence that every implementation of VIC collapses. Figure 17 reports three seeds; Figure 4 uses five.
7. Analysis & limitations
7.1 What the evidence leaves open
Discriminability need not match downstream usefulness. Ant skills often follow arcs rather than running straight; Figure 15 reports no learned skill above 1,000 on that benchmark. More entropy broadens exploration but excessive entropy makes skills harder to distinguish. e-skillse-entropy
Section 3.3 claims a unique evenly partitioned gridworld optimum, but Appendix B proves an unregularized two-skill optimum and only near-optimality with entropy, under exact stationary visitation. It supplies neither the stated uniqueness guarantee nor continuous-control convergence. e-grid
Source notation is inconsistent: Section 3.1 names state–skill information but prints I(A;Z), whereas Equation 1 uses I(S;Z). Figure 1's feedback label uses sₜ while its discriminator block and Algorithm 1 use sₜ₊₁. The explanation follows Equation 1 and Algorithm 1. e-objectivee-loop
7.2 Questions for discussion
- Does fixing the prior improve behavioral coverage beyond the sampling entropy it guarantees? (e-prior)
- When does center-of-mass supervision improve reusable skills rather than specializing them to one navigation task? (e-hierarchy)
- How many downstream tasks are needed to amortize DIAYN pretraining? (e-transfer)
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruct the listed Gym versions and custom tasks. Ant uses joint gear ratio 30, square-corner goals at coordinates ±2 m and a 0.5 m acceptance radius; hurdle boxes are 0.25 m high, 0.1 m wide and 1 m deep, spaced every 3 m. Match these before comparing curves. e-environments
Appendix C says 1,000 episodes per epoch but also equates 1,000 epochs with at most one million steps; Figure 5 instead labels time in hours. This accounting is unresolved. Hardware, complete discriminator configuration and exact baseline entropy settings are not supplied, so precise timing replication requires additional information. e-implementatione-transfere-imitation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the fixed prior preserve behavior, beyond preserving label entropy?
Reader-proposed check, not performed: use the listed HalfCheetah-v1 setup, fifty skills and α = 0.1, comparing fixed uniform sampling with the Appendix E learned-prior update. Match networks, interaction budget and initial seeds. Log label frequencies and exp(H[Z]), but also evaluate every skill equally on held-out rollouts, measuring discriminator accuracy, visited-state coverage under one fixed discretization, and the reward distribution. Report results across five seeds and equalize evaluation counts. The sampling argument predicts that the learned prior concentrates; the stronger behavioral argument additionally predicts lost coverage or fewer distinct useful behaviors. If effective count collapses while coverage and downstream scores remain unchanged, the proposed mechanism's practical consequence is weakened rather than confirmed by entropy alone. e-priore-implementatione-skills
Check 2: When does fine-tuning repay the pretraining bill?
Reader-proposed check, not performed: repeat Figure 5's best-skill initialization comparison with matched architectures and three initializations: random policy/value, pretrained policy with random value, and pretrained policy/value. Count unsupervised interaction, skill-selection evaluations and downstream learning separately. Report time-to-reward thresholds chosen before running, using both downstream-only and total-interaction accounting; then amortize the same repertoire over a declared set of distinct task rewards. Hold hardware fixed and report it because the paper's hours are otherwise difficult to compare. The paper predicts a downstream learning advantage, while the total-cost advantage may require reuse across tasks. Failure of pretrained value initialization to improve the matched policy-only control would challenge that specific transfer explanation. e-transfere-implementation
8.3 Reading coverage
Visual audit: All 22 supplied PDF pages were rendered and actually inspected, including the title/author page, equations and Algorithm 1, all main-text plots, gridworld proofs, Appendix C configuration details, entropy and prior diagnostics, locomotion images on pp. 19–21 and the imitation experiment on p. 22. All six final original crops were inspected; the imitation crop was widened to retain the complete rightmost axis tick. Figure 1's state-index discrepancy was checked against Algorithm 1 and Equation 3. The original plots contain no numerical results table, and no separate code, videos, references or supplemental artifacts were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2: Introduction and Related Work
- Sections 3.1–3.3: Objective, Implementation, Stability
- Sections 4.1–4.2.3: Skill analysis, initialization, hierarchical RL, imitation
- Section 5: Conclusion; References
- Appendix A: Pseudo-reward
- Appendix B: Optimum for Gridworlds
- Appendix C.1–C.2: Experimental details and environments
- Appendix D.1–D.4: Training objectives, entropy, reward distributions, exploration
- Appendix E.1–E.2: Learning the skill prior
- Appendix F.1–F.2: Classic control and simulated robot skills
- Appendix G–G.1: Imitation formulation and experiments
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- PDF p. 20 was visually checked during source acquisition and contains only Figure 20 plus its extracted caption; figure images are not supplied to the text-only reading pass.
- The preceding acquisition omissions are preserved. This reading additionally inspected all 22 supplied PDF pages, including Figure 20, and all six final original crops. All six text chunks were read individually; no supplied main-text or appendix section was omitted.
- Identity/version: the title and four authors match the catalog work. The inspected artifact is arXiv:1802.06070v6, 9 October 2018, not an asserted byte-identical copy of the catalog's ICLR 2019 proceedings edition. The catalog BibTeX orders Ibarz before Gupta; the inspected title page orders Gupta before Ibarz.
- Linked code, videos, external references and any separate supplements were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
Title matches the observed manifest title. Authors, in order: Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine. Affiliations shown are Carnegie Mellon University, UC Berkeley and Google Brain. Margin identifies arXiv:1802.06070v6, 9 Oct 2018.
Go to primary source ↓e-motivationPDF pp. 1–2, Abstract and Introduction
Motivates reward-free skill acquisition for sparse rewards, expensive feedback and hierarchical reuse; describes subsequent reward-based applications.
Go to primary source ↓e-loopPDF p. 3, Figure 1 and Algorithm 1
Fixed prior and environment are green; policy and discriminator are learned blue blocks. Algorithm samples a skill per episode, executes actions, scores next states, updates policy with SAC and discriminator with SGD. Diagram feedback label uses s_t despite s_{t+1} in its block and algorithm.
Go to primary source ↓e-objectivePDF pp. 3–4, Sections 3.1–3.3, Equations 1–3 and variational bound
Defines state–skill mutual information, action entropy and conditional action–skill penalty; uses a discriminator lower bound, uniform categorical prior, SAC and intrinsic log-posterior-minus-log-prior reward. Section 3.1's I(A;Z) notation conflicts with its prose and Equation 1.
Go to primary source ↓e-implementationPDF p. 4, Section 3.2; p. 14, Appendix C opening paragraph and C.1
Sets α = 0.1; specifies 300 hidden units for policy/Q/value, concatenated skill inputs, maximum 1,000-step episodes, inherited SAC settings and Gym environment versions. Epoch/episode/step statements are internally inconsistent; full hardware and discriminator configuration are absent.
Go to primary source ↓e-baselinePDF p. 12, Appendix A and footnote 5
Motivates the log-prior baseline using absorbing states and survival incentives, under a better-than-chance assumption; permits removal for tasks favoring early termination.
Go to primary source ↓e-gridPDF p. 4, Section 3.3; pp. 12–13, Appendix B, Lemmas B.1–B.2 and Figures 10–11
Main text states uniqueness. Appendix assumes exact stationary visitation, constructs an unregularized two-skill optimum and states an entropy-regularized gap of log(4)/(2N), with N the grid side length; it discusses short borders and bottlenecks.
Go to primary source ↓e-overlapPDF p. 5, Figure 2b; p. 6, Section 4.1, Question 3
Skills share a hallway then separate in a room; cumulative return allows temporarily indistinguishable states.
Go to primary source ↓e-skillsPDF p. 5, Figure 3 and Question 1; p. 16, Appendix D.3–D.4 and Figure 15; pp. 18–21, Appendix F and Figures 18–21
Shows diverse simulated locomotion and classic-control behaviors. Ant often moves in arcs; Appendix D.3 states no learned ant skill exceeds 1,000 benchmark reward. Figures 19–21 depict additional half-cheetah, hopper and ant skills.
Go to primary source ↓e-transferPDF p. 6, Figure 5 and Section 4.2.1; p. 7, opening paragraph
Five-seed average curves compare reward versus hours for DIAYN and random initialization. The highest-reward skill initializes both policy and value networks. Pretraining steps are excluded because pretraining is assumed free or amortized.
Go to primary source ↓e-hierarchyPDF p. 7, Section 4.2.2, Figures 6–7; p. 8, Question 7
Meta-controller chooses skills every 10 cheetah or 100 ant steps. Figure 7 solid cheetah DIAYN curve approaches 4–4.5 near 9 hours; ant DIAYN+prior is about 2.2–2.4 near 15 hours and plain DIAYN about 0.4–0.5. Center-of-mass discriminator input supplies the additional prior. Figure 6 uses 25 goals and five seeds.
Go to primary source ↓e-environmentsPDF p. 14, Appendix C.1–C.2
Lists Gym versions, hurdle geometry and ant modifications. Ant gear ratio is 30, ordered square-corner goals are at ±2 m, acceptance radius is 0.5 m and rewards sum to at most +5. Point navigation uses squared-distance goal reward and can be solved by choosing one skill.
Go to primary source ↓e-entropyPDF p. 15, Appendix D.2, unnumbered three-panel sweep; p. 16, first paragraph
For a point mass controlling orientation and forward velocity, compares α = 0.01, 1 and 10. Low entropy produces narrow trajectories; more entropy broadens coverage; very high entropy makes skills harder to distinguish. No coverage statistic or uncertainty is reported for this sweep.
Go to primary source ↓e-priorPDF p. 6, Figure 4 and Question 4; pp. 17–18, Appendix E.1–E.2 and Figure 17
VIC-labeled comparison learns the prior within DIAYN using p(z) proportional to exp of expected discriminator log score. Fixed prior gives 50 effective skills. Figure 17 shows three learned-prior seeds per classic-control task approaching roughly 1–3 effective skills; Figure 4 separately shows half-cheetah. Effective count is exp(H[Z]).
Go to primary source ↓e-imitationPDF p. 8, Section 4.2.3 and Figure 9; p. 22, Appendix G–G.1, Equation 5 and Figure 22
Enumerates skills by product of state discriminator scores; relates retrieval to an M-projection under a perfect discriminator and fixed prior. Figure 9 shows three successes and failed handstand. Figure 22 aggregates 600 synthetic tasks from another seed; baseline variants lower entropy, learn the prior, or use five instead of 50 skills. Distance is L2 in state space; no aggregate numerical error or calibrated safety test is given.
Go to primary source ↓e-visual-formatPDF pp. 3–8 and 12–22, method and experiment figures throughout the main text and appendices
The supplied paper presents its quantitative evidence in plots and its method in Figure 1 with Algorithm 1. It contains no numerical results table; entropy and prior comparisons are graphical diagnostics.
Go to primary source ↓8.5 Primary sources
DIVERSITY IS ALL YOU NEED : LEARNING SKILLS WITHOUT A REWARD FUNCTION ↗
PDF · 9,881 extracted words
Source fingerprint
af9965836b917def393670da0258d76c31787076c19ad7cbf46a7a7eda510624