The Arcade Learning Environment: An Evaluation Platform for General Agents
1. Paper overview
In one sentence: ALE turns varied Atari games into a shared evaluation protocol, showing that representation, simulator access, and score aggregation can each change which agent appears strongest. e02e03e05e06e09e13e18
| At a glance | What to know |
|---|---|
| Research problem | Source description A method repeatedly refined on a few familiar tasks can appear general while depending on substantial domain-specific engineering. ALE proposes testing algorithm design on games withheld from that design process. The target is competence across independently developed Atari games, with perception grounded in screens or RAM. Generality concerns the reusable algorithm and its parameter choices: each learning agent still trains separately inside each evaluation game. e02e03e06 |
| Core mechanism | |
| A key reported result | Zaxxon planning: UCT: 22,610.0 Mean raw game score; higher is better. Held-out testing game; exact-emulator planning, 10 episodes, at most 18,000 frames per episode, decisions every five frames; learner comparator uses 30 trials of 500 evaluation episodes after training. Full tree: 0.0; best learner (LSH): 3,365.1; best simple baseline: 2.0. A large simulator-planning gain under different information and computation budgets. It does not measure learned-model accuracy or real-time control. e06e10e22e24 |
| Reading caution |
Core contributions
Figure 5. From a detailed game screen to sparse colour-location features. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the pair from left to right. The left panel is the source Freeway screen in the SECAM palette; the right panel shows colour occupancy on a coarse spatial grid after background subtraction. Cars and the player need not be named as objects for their colours to activate features in particular regions. Appendix A.1 then adds pairwise combinations of these colour-location features, allowing a linear value function to respond to their joint presence. Those pairwise combinations are described in the text; the image displays the underlying spatial encoding, not a complete feature vector or a search tree. e05e17e22
What it supports. The method preserves coarse locations and colour relationships while discarding much of the screen's detail. This makes the representation relevant to game interaction without requiring DISCO-style object classes. The figure explains a feature-construction choice; its usefulness for reward maximization must be assessed in the separate learning results.
Where the evidence stops. Basic is not simply BASS with pairwise features removed: it also changes from eight to 128 colours. Any explanation of BASS's advantage must preserve that confound. The graphic contains no future-state prediction or action output.
2. Motivation
2.1 The problem and the proposed response
A method repeatedly refined on a few familiar tasks can appear general while depending on substantial domain-specific engineering. ALE proposes testing algorithm design on games withheld from that design process. The target is competence across independently developed Atari games, with perception grounded in screens or RAM. Generality concerns the reusable algorithm and its parameter choices: each learning agent still trains separately inside each evaluation game. e02e03e06
2.2 What this reading follows
A common interface does not make game competence a single, easily measured quantity. This paper connects three decisions: what an agent observes, what computation it may perform before acting, and how its performance is summarized across games. The illustrated reading starts with BASS's coarse colour features, compares the learning and planning tables, and then examines two ways a convincing result can mislead: faulty object classes and an aggregate dominated by one game. Read every score with its protocol attached. Algorithms are designed on five games, then evaluated on fifty others; learning agents still receive training episodes within each testing game. e02e03e05e06e09e13e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator classification is supported by the shared Atari interface, evaluation protocol, and diagnostic metrics. Architecture, prediction paradigm, and quadrant are not applicable to ALE itself. Separate model-free learners and an emulator-driven planner do not constitute a unified learned world/action model, joint future/action prediction, or inverse dynamics. e03e05e09e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn perception into an action-value representation
Begin with the distinction between a feature and a learned prediction. BASS's grid encodes colours that are present now. Pairwise features let a linear value function assign value to co-occurrences, such as a player colour appearing near a collectible colour. The learner then improves action selection through SARSA and reward feedback; this pathway does not forecast future images. DISCO tries a different abstraction: discover object classes first, then represent spatial and motion relationships among their instances. Its Seaquest failure illustrates why adding semantic-looking structure can lose useful distinctions. Reader interpretation: a representation is useful only insofar as its invariances preserve distinctions needed for reward-sensitive action choices. The reported Basic/BASS comparison cannot identify the effect of interactions alone, because colour resolution changes too. e05e17e18
5.2 Use the emulator to ask counterfactual questions
ALE's save/restore interface enables the planner to explore alternative actions from the same emulator state. Breadth-first search distributes effort across a rapidly growing tree and backs up discounted rewards. UCT concentrates visits on selected branches and supplements the tree with rollouts. The implementation also treats actions with identical resulting emulator states as duplicates, and retains the selected subtree for the next decision. Algorithm 2 chooses the most-visited root action, which is then executed before planning resumes. Appendix C fixes the search resources, while the main text reports substantial time per decision. Reader interpretation: this separates the challenge of choosing useful simulated trajectories from the challenge of learning a simulator. The experiments address the former; the printed pseudocode ambiguities remain reproduction questions. e03e09e10e21e22
5.3 Keep algorithm generalization and score normalization separate
The five design games are where representations and parameters are refined. The fifty testing games ask whether those choices remain useful elsewhere, even though each learning agent still trains on episodes from each testing game. That separation guards against tuning the algorithm to evaluation tasks; it does not impose zero-shot policy transfer. Once scores are obtained, a second problem arises: different games use different units and difficulty scales. Baseline normalization supplies an external reference, but an exceptional normalized score can dominate its mean. Inter-algorithm normalization bounds comparisons but can award a perfect relative score to a weak best agent. Medians, distribution curves, and per-game paired tests therefore complement the raw tables, with each retaining a different part of the evidence. e02e06e12e13e14e15
5.4 Training and inference
During training
Design and hyperparameters use Asterix, Beam Rider, Freeway, Seaquest, and Space Invaders. Fifty testing games are sampled from 123 eligible titles filtered from a 381-game list. Each learning trial contains 5,000 training episodes followed by 500 episodes without learning; results average 30 trials. Episodes stop at game end or 18,000 frames, with an action every five frames. e06
Visual preprocessing is additional game-specific data use: background estimation uses 18,000 screens, and DISCO class discovery uses 36,000. Sample trajectories follow a human-provided trajectory for a random duration before random actions. These agents are therefore not uniformly free of demonstration-derived preprocessing. e05e22
Appendix C sets discount γ = 0.999 and exploration ε = 0.05. Basic/BASS use learning rate α = 0.5 and trace decay λ = 0.9. DISCO uses α = 0.1, λ = 0.9; RAM uses α = 0.2, λ = 0.5; LSH uses α = 0.5, λ = 0.5. e22
During inference
Learning evaluation freezes updates. Planning instead searches online at each decision: UCT returns the most-visited root action and advances its root to that child. Planning uses 10 episodes per game, about 15 seconds per action selection, and the same game-frame cap and action interval. Appendix C specifies 500 UCT simulations per action, a 300-frame maximum depth, exploration constant 0.1, and 133,000 emulated frames per full-tree decision. e06e10e21e22
5.5 Implementation flow
- Expose a common environment
The game-handling layer reads score changes, usually as rewards, and detects episode termination. Save/restore includes RAM, registers, and address counters, allowing counterfactual simulator rollouts. The paper's agents use all 18 actions even though ALE can provide a smaller game-specific action set. e03
- Construct features for value learning
Basic records colour presence in a 16 × 14 grid using 128 colours. BASS uses eight colours and adds pairwise feature conjunctions. DISCO discovers object classes and tile-codes positions and relative positions/velocities. LSH uses random projections and hashing. RAM supplies individual bits and every pairwise logical AND; its input excludes extra cartridge RAM. e05e17e18e19e20e22
- Separate learning from simulator search
SARSA uses linear function approximation, replacing eligibility traces, and epsilon-greedy exploration without a dynamics model. Breadth-first search backs up discounted rewards. UCT allocates visits among branches, performs rollouts, merges actions producing identical emulator states, and reuses the selected subtree. This is planning with a supplied simulator, not learning a future-prediction model. e05e09e21
- Compare games through explicit reference ranges
Normalize each score against a per-game range, then report means, medians, score distributions, or paired comparisons. Random-reference, baseline-set, and inter-algorithm normalization answer different questions; bounded relative scores alone cannot establish competent play. e12e13e14e15
6. Experiments & results
The Arcade Learning Environment makes Atari games a common testbed for general-purpose agents. Its contribution combines an emulator interface, a separation between design games and evaluation games, and metrics for comparing incompatible score scales. Linear reinforcement-learning agents and planners using the exact simulator establish useful baselines, while exposing representation failures, sparse rewards, and misleading aggregate rankings.
This benchmark paper supplies feature-construction examples rather than a unified neural architecture diagram. Its Basic/BASS comparison changes both palette size and pairwise interactions, and it reports no controlled correction of DISCO's detector or removal of UCT's duplicate-state optimization. Figures 3 and 7 serve as measurement and perception diagnostics, not isolated causal ablations. e05e09e13e17e18
6.1 Read the original evidence
Table 1. Different representations win on different games under the same SARSA framework. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each row across the five learning representations before looking at the simple policies and human column. The scores are rounded raw game scores, so compare agents within a row rather than comparing score magnitudes across different games. Asterix and Seaquest are design games; Boxing, H.E.R.O., and Zaxxon are testing games. BASS leads the learner columns on Seaquest and H.E.R.O., RAM leads on Boxing, and LSH leads on Asterix and Zaxxon. The learners use 5,000 training episodes and 500 evaluation episodes per trial, averaged across 30 trials. Human scores come from a different, much smaller evaluation. e06e07e17e23
What it supports. RAM's 44 in Boxing and BASS's 6,459 in H.E.R.O. illustrate why a single example cannot identify a generally superior representation. Direct access to console-memory bits does not make RAM uniformly strongest. This selected table is a useful entry point, while Appendix D supplies the complete per-game record.
Where the evidence stops. The human column represents five episodes from a beginner, not expert performance. No uncertainty is shown in this table. Basic/BASS also differ in palette size, and visual preprocessing uses additional sampled trajectories.
Table 2. Exact-simulator search substantially improves several scores, at a substantial decision-time cost. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read Full Tree and UCT as two search procedures with access to emulator states, then use Best Learner and Best Baseline as contextual references. Asterix and Seaquest were used for design; the remaining rows were held out from design. UCT reaches 22,610 in Zaxxon, compared with zero for Full Tree and 3,365 for the best learner in this rounded table. Both planners reach 100 in Boxing, so selective search is not uniquely beneficial in every row. Planning uses ten episodes per game. Its decisions occur every five game frames, but the reported computation takes roughly fifteen seconds per decision. e06e09e10e22e24
What it supports. The Zaxxon and H.E.R.O. rows show how selective simulator search can find rewards that these learners or breadth-first search miss. The result demonstrates the value of online search under an exact transition model. It does not show that any system learned that transition model from observations.
Where the evidence stops. These are different information and compute budgets, and the table mixes design and testing games. Five minutes of simulated gameplay is an episode limit, not five minutes of wall-clock evaluation.
Figure 4. Read the fraction of games clearing a threshold, rather than only one average. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a threshold on the horizontal axis and move upward to an agent's curve: the vertical coordinate is the fraction of games achieving that score or better. Crucially, both threshold axes decrease from left to right, so the curves rise as the required score becomes easier. The baseline panel has nonuniform tick spacing, including 2,000, 50, and 10 on its left side. The right panel runs from an inter-algorithm score of one down to zero. Labels and leader lines identify each curve. Compare vertical positions at the same threshold; do not interpret a steeper segment as a learning rate or a temporal improvement. e06e12e14
What it supports. The distributions show why BASS is generally preferred to DISCO across this set without reducing every game to one average. Relative curves expose how often an agent approaches the best learner, while baseline-normalized thresholds relate performance to simple policies. Their complementary references preserve information that a single ranking obscures.
Where the evidence stops. The figure pools all games, including design games. It is not a learning curve and shows no uncertainty. A score above one exceeds the best baseline; exactly one can tie it. Nonuniform horizontal spacing prevents a naive area comparison.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zaxxon planning Held-out testing game; exact-emulator planning, 10 episodes, at most 18,000 frames per episode, decisions every five frames; learner comparator uses 30 trials of 500 evaluation episodes after training. | UCT: 22,610.0 Mean raw game score; higher is better | Full tree: 0.0; best learner (LSH): 3,365.1; best simple baseline: 2.0. A large simulator-planning gain under different information and computation budgets. It does not measure learned-model accuracy or real-time control. e06e10e22e24 |
| H.E.R.O. model-free learning Held-out testing game; 5,000 learning episodes and 500 frozen evaluation episodes per trial, averaged across 30 trials. | BASS: 6,458.8 Mean raw game score; higher is better | Basic: 6,053.1; DISCO: 2,719.8; LSH: 3,835.8; RAM: 3,281.1; Random: 712.0. The representation matters within the same learning framework. This row supplies no uncertainty and does not isolate pairwise features from palette changes. e06e17e23 |
| BASS versus Basic across games All 55 games, including five design games; per-game two-tailed t-tests with 99% confidence intervals as described in Section 4.3. | BASS: 32 better, 18 worse than Basic. Games with statistically better versus worse scores | BASS versus DISCO: 48 better, 5 worse. Counts distinguish cross-game consistency from effect size. The paper prints 'Welsh’s t-test'; the report preserves the stated protocol without reconstructing unprovided trial data. e15 |
| Reported breadth of baseline improvement Combined five design games and 50 testing games; separate learning and exact-simulator planning protocols. | Learning agents: 40/55; search methods together: 49/55. Author-reported count of games with improvement | Learning is compared with simple policies; search is compared with learners and simple policies. These are author aggregates, not a single agent's held-out success rate. Table 5 prints a Skiing tie at 0.0 among full tree, best learner, and baseline, so the search count should not be read as 49 demonstrated strict wins. e08e11e24 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. An object representation can compress the wrong entities together. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the original screen on the left with the detected class instances on the right. Here colour indicates a discovered class rather than the original game's palette. Follow the red detections: several fish share this class, but so do a life icon near the top and the oxygen bar near the bottom. Appendix A.3 explicitly identifies this confusion. DISCO first discovers classes in preprocessing, then builds features from detected instances, including their positions and pairwise relative positions and velocities. The visual therefore exposes an error upstream of value learning, where superficially similar shapes acquire an inappropriate shared identity. e05e08e18
What it supports. DISCO's intended benefit is to reuse experience across instances of the same object class. The displayed grouping shows the corresponding failure mode: entities with different game roles can contribute to the same class representation. This is a concrete perceptual diagnostic that helps interpret, but does not explain completely, DISCO's weak overall results.
Where the evidence stops. This is one Seaquest frame from a design game. It neither measures detection accuracy across the testing set nor isolates class confusion as the cause of lower returns. No corrected-detector control is reported.
Figure 3. Changing the summary statistic changes the apparent winner. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each row before comparing vertically. The top row holds baseline normalization fixed and changes the aggregate from average to median. LSH's orange bar dominates the average, but its median no longer leads. The bottom row repeats the average/median comparison after each game's scores are normalized against the five learners themselves. BASS's green bars are strongest there. The colours identify the same methods throughout, while the vertical scales differ by panel. Section 4.2.2 attributes LSH's striking mean-baseline advantage to its Zaxxon performance; the visual is therefore a diagnostic of the measurement procedure, not an additional training experiment. e12e13e23
What it supports. A large mean can reflect an exceptional game rather than consistently good play. The figure supports reporting more than one aggregate and checking influential games. BASS looks stronger under bounded relative comparisons, but that result describes its position among these agents and does not certify that it solves the games.
Where the evidence stops. The panels use different normalization and axis scales; bar heights cannot be compared directly across panels. There are no error bars here. Inter-algorithm normalization changes when the comparator set changes and removes an absolute competence reference.
7. Analysis & limitations
7.1 What the evidence leaves open
Basic/BASS comparison changes both colour resolution and pairwise interactions. DISCO's Seaquest example groups fish with a life icon and oxygen bar, but one diagnostic image cannot establish the cause of its entire cross-game deficit. e17e18e08
Simple policies can obtain strong scores by refusing to play games with negative rewards. Human comparisons are only five-episode averages from a beginner on selected games, not an expert benchmark. Search retains sparse-reward failures despite its exact model. e07e08e11
LSH looks strongest under mean baseline normalization because of Zaxxon, while medians and bounded comparisons change the ranking. Inter-algorithm normalization can label an objectively weak agent best. The finite, filtered game set and mixed design/test aggregates limit broader claims of general intelligence. e06e12e13e16
7.2 Questions for discussion
- How much of BASS's advantage survives when palette size is fixed and only pairwise conjunctions change?
- How should a benchmark report distinguish withheld algorithm-design games from the interaction data used to train a policy inside each test game?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires matching ALE/Stella behavior, game ROMs and reward/termination rules, the listed game split, preprocessing trajectories, and Appendix C parameters. The PDF does not pin an emulator commit, ROM fingerprints, random seeds, or experimental hardware specification; its approximate planning time therefore lacks a portable hardware context. e03e04e05e06e10e22
Resolve UCT pseudocode ambiguities before implementation: Algorithm 3 prints a square root of log child visits divided by parent visits and no exploration coefficient, whereas Appendix C lists 0.1. Algorithm 2 assigns 'immediate-return' but Algorithm 3 reads 'immediate-reward'. These printed details do not establish what the released code executed. e21e22
LSH prose specifies a 7 × 210 × 160 bit vector, but Algorithm 1's binarizeScreen assigns a single index x + y*h + Ixy per pixel. The intended encoding needs clarification. DISCO has several listed thresholds, but exact shape-merging and small-region filtering criteria are not quantified here. e18e19e22
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate BASS's pairwise features from its palette change
Reader-proposed check, not an experiment performed here: run a two-factor comparison of eight versus 128 colours and pairwise conjunctions off versus on. Hold the 16 × 14 grid, background samples, game split, episode budget, and SARSA settings fixed; pair random seeds across conditions and report feature memory as well as returns. Choose settings only on the original design games, then evaluate the frozen designs on the listed testing games. Compare conjunctions at fixed palette and palette changes without conjunctions. An advantage confined to a palette change, with no repeatable gain from conjunctions at either palette, would undermine an interactions-only explanation of BASS's reported advantage. e05e06e17e22
Check 2: Test whether duplicate-state merging buys useful search depth
Reader-proposed check, not an experiment performed here: compare UCT with duplicate-state merging enabled and disabled, first holding subtree reuse fixed. Use Beam Rider and Seaquest as diagnostic design games, retaining the published action interval and depth limit. Record distinct children, reached depths, emulated frames, decision time, and episode return. Run both the published simulation-count setting and a second comparison matched by total emulated frames. Fewer children without deeper useful exploration or improved returns would weaken the proposed explanation for merging's benefit. Resolve and explicitly document the printed selection-bonus and exploration-constant ambiguity before either comparison; changing that choice between conditions would confound the test. e09e10e21e22
8.3 Reading coverage
Visual audit: Visually inspected the title/author page and every body/appendix page from 1 through 25, including Figures 1–7, Tables 1–5, Algorithms 1–3, and all Appendix C parameters. All six final crops were separately viewed. Figure 4's decreasing threshold axes and nonuniform baseline ticks were checked against Section 4.2.3; Figure 7's class confusion was checked against Appendix A.3. The printed LSH and UCT ambiguities are retained in the base report. References on pages 26–27 were read in the complete six-chunk text, without a separate visual pass. No separate supplements were supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; title and author block
- 1. Introduction
- 2. Arcade Learning Environment, including 2.1–2.3
- 3. Benchmark Results, including 3.1.1–3.1.3 and 3.2.1–3.2.3
- 4. Evaluation Metrics, including 4.1.1–4.1.3, 4.2.1–4.2.3, and 4.3
- 5. Related Work, including 5.1–5.2
- 6. Final Remarks; 7. Conclusion; Acknowledgments
- Appendix A. Feature Set Construction, A.1–A.5
- Appendix B. UCT Pseudocode, Algorithms 2–3
- Appendix C. Experimental Parameters
- Appendix D. Detailed Results, D.1–D.2
- References
Outside the original text pass
- Identity: the inspected title and four authors match the catalog. The supplied artifact is arXiv:1207.4708v2 [cs.AI], dated 21 June 2013, and bears the JAIR 47 (2013), 253–279 masthead, submitted February 2013 and published June 2013. Earlier revisions and a separately obtained publisher PDF were not supplied; revision-to-revision equivalence was not assessed.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation above was addressed by visually inspecting PDF pages 1–25, including all seven figures, all five numbered tables, algorithms, and Appendix C. References on pages 26–27 were read as text.
- Separate supplemental material availability has not been fully verified.
- No separate supplements, code, executable environment, ROM artifacts, or raw experimental trials were supplied or inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1 (journal p. 253), title/author block, masthead, and arXiv margin stamp
Exact title and authors are Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling; affiliations are University of Alberta and Empirical Results Inc. The artifact identifies arXiv v2, 21 June 2013, and JAIR 47 (2013), 253–279.
Go to primary source ↓e02PDF pp. 1–2 (journal pp. 253–254), Section 1
The motivation is avoiding method overfitting and testing reusable representations/parameters on games withheld from algorithm development.
Go to primary source ↓e03PDF pp. 2–3 (journal pp. 254–255), Sections 2.1–2.2
ALE wraps Stella; screen observations are 160 × 210 seven-bit pixels, actions number 18, rewards and termination are game-specific, and emulator save/restore includes RAM, registers, and address counters.
Go to primary source ↓e04PDF p. 4 (journal p. 256), Section 2.3
The paper describes a GPL release, C++ implementation, language interfaces, and benchmark-agent code availability, without pinning a release or commit.
Go to primary source ↓e05PDF pp. 4–5 (journal pp. 256–257), Sections 3.1–3.1.1
SARSA uses linear approximation, replacing traces, and epsilon-greedy exploration. Five feature sets are described. Background and class preprocessing use 18,000 and 36,000 sampled observations with human-trajectory prefixes.
Go to primary source ↓e06PDF pp. 5–6 (journal pp. 257–258), Section 3.1.2
The five named design games and 50 testing games are specified; the latter come from 123 eligible games. Per-game RL has 5,000 training and 500 no-learning evaluation episodes per trial, 30 trials, a five-frame action interval, and an 18,000-frame cap.
Go to primary source ↓e07PDF p. 6 (journal p. 258), Section 3.1.2 baseline and human paragraphs
Random acts every frame; Const selects the best fixed-action policy; Perturb repeats a fixed action with probability 0.95. Human scores average five episodes from a beginner and are explicitly nonexhaustive.
Go to primary source ↓e08PDF pp. 6–7 (journal pp. 258–259), Section 3.1.3
The authors report learners beating simple baselines in 40 of 55 games, no uniformly strong representation, nonplaying policies benefiting from negative rewards, and DISCO performing worse on unseen games.
Go to primary source ↓e09PDF pp. 7–8 (journal pp. 259–260), Sections 3.2–3.2.1
Search uses the emulator as a perfect generative model. Breadth-first search backs up discounted rewards; UCT balances exploration and return, rolls out, merges duplicate resulting states, and reuses a selected subtree.
Go to primary source ↓e10PDF p. 8 (journal p. 260), Section 3.2.2
Planning uses the same five tuning and 50 testing games, ten episodes per game, 18,000-frame caps, five-frame decisions, and roughly 15 seconds per action selection.
Go to primary source ↓e11PDF p. 8 (journal p. 260), Section 3.2.3
The authors report the two search methods together outperforming learners and baselines on 49 of 55 games; Freeway, Private Eye, Montezuma's Revenge, and Venture exemplify sparse-reward failures.
Go to primary source ↓e12PDF pp. 9–10 (journal pp. 261–262), Figure 2 and Sections 4.1–4.1.3
The normalization formula uses per-game score ranges. References are a random score, 37 baseline policies, or algorithm extrema. Near-zero references distort scale, and relative normalization loses objective performance information; no general zero-range rule is given.
Go to primary source ↓e13PDF pp. 10–11 (journal pp. 262–263), Sections 4.2.1–4.2.2 and Figure 3
Four panels compare average/median baseline and inter-algorithm scores. The text attributes LSH's striking mean-baseline lead to Zaxxon and explains median robustness.
Go to primary source ↓e14PDF pp. 11–12 (journal pp. 263–264), Section 4.2.3 and Figure 4
Score distributions show fractions of games reaching thresholds. Both plotted threshold axes decrease left to right; the baseline axis has nonuniform tick spacing. BASS is generally stronger and DISCO weaker among these learners.
Go to primary source ↓e15PDF p. 12 (journal p. 264), Table 3, BASS row/Basic and DISCO columns; Section 4.3
BASS has 32–18 better–worse games against Basic and 48–5 against DISCO. The stated protocol uses per-game two-tailed 'Welsh’s t-test' with 99% confidence intervals; the text warns that counts magnify small significant differences.
Go to primary source ↓e16PDF pp. 13–15 (journal pp. 265–267), Sections 5.2, 6, and 7
The paper emphasizes perceptual grounding and Atari's programming artifacts; progression toward real-world general competency is a proposed research trajectory, not an evaluated deployment result.
Go to primary source ↓e17PDF pp. 16–17 (journal pp. 268–269), Appendix A.1–A.2 and Figure 5
Figure 5 shows Freeway in SECAM colours and its coarse BASS encoding. BASS adds pairwise colour/location features; Basic omits pairs but uses 128 colours, so the comparison changes two representation factors.
Go to primary source ↓e18PDF pp. 17 and 19 (journal pp. 269 and 271), Appendix A.3, Figures 6–7
DISCO separates class-discovery preprocessing from value learning, filters and merges classes, then tile-codes object relations. Figure 7 and its discussion identify fish sharing a class with a life icon and the oxygen bar; some filtering/merging criteria remain qualitative.
Go to primary source ↓e19PDF pp. 18–19 (journal pp. 270–271), Algorithm 1 and Appendix A.4
LSH uses random projections and hashes. The prose gives bit-vector size 7 × 210 × 160; Algorithm 1 prints binarizeScreen as s[x + y*h + Ixy] = 1, leaving an encoding ambiguity.
Go to primary source ↓e20PDF p. 20 (journal p. 272), Appendix A.5 and footnote 7
RAM features include 1,024 console-memory bits and pairwise ANDs. The footnote explicitly excludes additional cartridge RAM from the approach.
Go to primary source ↓e21PDF pp. 21–22 (journal pp. 273–274), Appendix B, Algorithms 2–3
UCT returns the most-visited root action and can reroot at its child. The printed selection bonus uses log child visits over parent visits without an exploration multiplier. The algorithms use different immediate-return/immediate-reward names.
Go to primary source ↓e22PDF p. 23 (journal p. 275), Appendix C, all parameter groups
The table specifies preprocessing sizes and thresholds, representation and SARSA parameters, 500 UCT simulations per action, 300-frame depth, exploration constant 0.1, and 133,000 full-tree emulated frames per action. It provides no hardware configuration or random seeds.
Go to primary source ↓e23PDF p. 6, Table 1; PDF p. 24 (journal p. 276), Table 4, H.E.R.O., Boxing, and Zaxxon rows
Table 1 provides rounded selected results. Table 4 gives H.E.R.O. Basic 6053.1, BASS 6458.8, DISCO 2719.8, LSH 3835.8, RAM 3281.1, Random 712.0; Boxing RAM 44.0; Zaxxon LSH 3365.1.
Go to primary source ↓e24PDF p. 8, Table 2; PDF p. 25 (journal p. 277), Table 5, Zaxxon, Boxing, and Skiing rows
Zaxxon has full-tree 0.0, UCT 22610.0, best learner 3365.1, and baseline 2.0. Boxing has both planners at 100.0. Skiing prints full tree, best learner, and baseline at 0.0, with UCT −0.8. Table 2 rounds the selected results.
Go to primary source ↓8.5 Primary sources
The Arcade Learning Environment: An Evaluation Platform for General Agents ↗
PDF · 10,841 extracted words
Source fingerprint
924135b3905b17aa195918620dff175db6c0d8463ecae20aac69e2bea95c4431