Behaviour Suite for Reinforcement Learning

The exact title is Behaviour Suite for Reinforcement Learning. Fourteen authors and DeepMind are credited. The artifact is arXiv:1908.03568v3, 14 February 2020, published as an ICLR 2020 conference paper; the title page uses Csaba Szepesvari, Richard Sutton and Hado Van Hasselt.

The suite targets interpretable, scalable behavioural experiments; it is complementary to grand-challenge benchmarks and does not claim broad coverage of all important RL issues. An experiment fixes environments, interaction and analysis.

Scores map behaviour to [0,1]; standard analysis does not inspect agent internals. Experiments should be targeted, simple, challenging, scalable and fast, with a design target below 30 minutes on a standard CPU.

Memory length presents a uniformly sampled binary context only initially, then zero context with normalized time. The final action is compared with the initial context. Lengths span 1–100 exponentially; the main text specifies 10k episodes and a pass threshold below 75% of random-policy average regret.

The prose describes recurrent A2C as good for N≤30 and essentially random beyond 30, associated with backpropagation through time of length 30. Figure 2 shows a transition near 30, with the first failing marker around that tick. Feedforward DQN and Bootstrapped DQN fail for lengths above 1. The plot labels the criterion average_regret < 0.75 and its horizon as 10k episodes.

The N×N one-hot grid begins top left and descends each step, with fixed randomized action-to-direction mappings. Right incurs −0.01/N; always going right earns an additional +1. Uniform random actions reach reward with probability 2^−N. Main-text evaluation uses N=10,12,…,50, 10k episodes, and regret below 0.9 sooner than 2^N episodes.

Bootstrapped DQN's summary bar reaches 1, while recurrent A2C and DQN are at 0. Its points are solved throughout the displayed size sweep; the other agents remain unsolved at the 10,000-episode ceiling. The dashed curve is the 2^N reference. Scaling beyond N=50 is suggested, not measured.

The paper describes automatic environment logging and notebook analysis, a policy/update agent interface, optional integration into existing run loops, dm_env and Gym interfaces, observation specifications and optional linear interpolation. Figure 5 aggregates seven capability categories. Code and notebooks are linked and described as released.

The authors disclaim comprehensive RL coverage and explicitly note missing targeted hierarchical-RL experiments in bsuite2019. Future releases and committee review are proposed.

Basic tasks are deterministic bandit, MNIST contextual-bandit classification, Catch, Cartpole and Mountain car. Tables specify 20 seeds and 10k episodes with normalized regret. The MNIST entry does not identify a dataset split; Appendix A directs full documentation to the code.

Reward-noise standard deviations are 0.1, 0.3, 1, 3 and 10; reward multipliers are 0.01, 0.1, 1, 10 and 100. Each sweep allocates four seeds to each level.

Appendix tables give deep-sea sizes N=[5..50], 10k episodes and the percentage of runs with average regret below 90% of random; the stochastic variant adds stochastic transitions and N(0,1) reward noise. These summaries differ from the main text's size range and omit its faster-than-2^N qualification.

The suite includes sparse-reward cartpole swingup, umbrella length/features and discounting chain. These tables specify 1k episodes. Umbrella tasks isolate a consequential first decision followed by distracting variables; length or feature count varies logarithmically from 1 to 100.

Memory-length and memory-bits tables specify 1k episodes, normalized regret and the issue label credit assignment. Memory length varies 1–100 with one binary context; memory bits fixes length 2 and varies bits 1–100. Both adjacent diagrams depict multiple context components. The memory-length table conflicts with Section 2.1's episode budget and score definition.

A one-page report template combines automatically generated scores with author-written agent definitions and commentary, with deeper analyses linked separately.

Baseline defaults compare random actions, DQN, Bootstrapped DQN with prior networks, and a recurrent actor critic. The commentary identifies DQN's feedforward MLP and 5%-greedy exploration, Bootstrapped DQN's exploration advantage, and the recurrent agent's memory advantage.

DQN optimizers are tuned on basic tasks over learning rates {1e−1,1e−2,1e−3}; SGD selects 1e−2 and RMSProp/Adam 1e−3, with other defaults held constant. Adaptive methods generally outperform SGD, especially in generalization and scale. D.3 claims superiority in every category and refers to Figure 11, although optimizer results are Figures 8–9; Figure 11 on p. 19 concerns ensembles.

Bootstrapped DQN with prior networks is evaluated with ensemble sizes 1, 3, 10 and 30. The commentary describes poor large-deep-sea performance below 10, reliable solution with larger ensembles, and diminishing gains from 10 to 30. Not every cartpole-swingup instance is solved; instability and a possible Double DQN remedy are hypotheses.

Implementation descriptions defer to repository defaults and tutorials. The paper does not supply a complete layer/replay/target-update configuration, software version, hardware model/count or raw numerical result tables and uncertainty estimates sufficient for exact reconstruction.