Transformers are Sample-Efficient World Models
1. Paper overview
In one sentence: IRIS turns compressed Atari images into an autoregressive training environment for a separate policy, trading real interaction for simulated experience whose usefulness depends on visual fidelity and event coverage. ev-loopev-dynamicsev-imaginationev-resultsev-evaluationev-explorationev-token-figure
| At a glance | What to know |
|---|---|
| Research problem | Source description Can an image-based agent learn useful behavior from only 100,000 real actions per game? IRIS treats Atari as a partially observable process with image observations, discrete actions, scalar rewards and episode ends. Learning in imagination reduces demands on real interaction, but policy quality then depends on whether the simulator preserves consequential objects and game mechanics. ev-problemev-benchmarkev-exploration |
| Core mechanism | Source description IRIS combines a learned discrete image vocabulary with token-level autoregressive dynamics and a separate policy trained entirely on imagined trajectories. ev-loopev-dynamicsev-imagination |
| A key reported result | Atari 100k aggregate performance: IRIS: mean 1.046; median 0.289; IQM 0.501; optimality gap 0.512; 10/26 superhuman games. Human-normalized mean, median, IQM; optimality gap (lower is better); superhuman-game count. 26 games; 100k actions per game. IRIS uses five runs, evaluating 100 episodes per game at training end. Comparison data use five SimPLe runs, 100 CURL/DrQ/SPR runs and three MuZero/EfficientZero runs. SPR: 0.616, 0.396, 0.337, 0.577 and 6/26 respectively. Search-based EfficientZero: mean 1.943 and 14/26 superhuman games; its IQM and gap are N/A in Table 1. Strong mean/IQM performance does not imply superiority on the median or over search-based methods. Figure 5 supplies stratified-bootstrap intervals, without tabulated endpoints. The score normalizes agent-minus-random return by human-minus-random return. ev-benchmarkev-resultsev-evaluation |
| Reading caution | Reader analysis The authors identify double exploration: a rare transition must first occur in real data, then be rediscovered by the policy in imagination. Frostbite illustrates failure to acquire later levels; Krull has more frequent transitions. This explanation is observational rather than a controlled causal ablation. ev-exploration |
Core contributions
- Source description
IRIS combines a learned discrete image vocabulary with token-level autoregressive dynamics and a separate policy trained entirely on imagined trajectories. ev-loopev-dynamicsev-imagination
- Author claim
The authors report a new Atari 100k state of the art among the compared methods without lookahead search, specifically for mean, IQM, optimality gap and superhuman-game count; this is a historical, protocol-bounded claim. ev-resultsev-evaluation
Figure 1. The policy acts inside the simulator; the Transformer predicts the consequences of those actions. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the real frame x₀ above the leftmost block. The green encoder E produces image tokens z₀; the green decoder D reconstructs the frame consumed by policy π. Follow the purple arrow upward to action a₀, then the blue G arrow toward next-frame tokens. The blue branches also predict reward r̂₀ and termination d̂₀. Dashed boxes group one frame’s tokens, while nested solid boxes show the accumulated observation/action context. Repeat this loop to the right. The caption and Equation (1) clarify that each next frame is generated token by token; the diagram compresses those individual sampling operations into one arrow. ev-loopev-dynamicsev-imaginationev-benchmarkev-rlev-algorithm
What it supports. Action generation and dynamics prediction belong to separate components. G provides a training environment conditioned on policy actions, while π learns to select those actions from decoded images. This separation explains why the paper can learn with a world model yet execute its real-environment policy without lookahead search.
Where the evidence stops. Section 2.3 specifies stopping at predicted termination; Algorithm 1 omits that branch, while Equation (4) masks terminal continuation. The figure is an information-flow sketch, not a complete rollout implementation.
2. Motivation
2.1 The problem and the proposed response
Can an image-based agent learn useful behavior from only 100,000 real actions per game? IRIS treats Atari as a partially observable process with image observations, discrete actions, scalar rewards and episode ends. Learning in imagination reduces demands on real interaction, but policy quality then depends on whether the simulator preserves consequential objects and game mechanics. ev-problemev-benchmarkev-exploration
2.2 What this reading follows
IRIS asks whether an agent can learn to play from a small amount of real experience by doing most of its behavioral learning inside a learned simulator. An autoencoder supplies the visual vocabulary, a Transformer predicts how that vocabulary changes after actions, and a recurrent actor-critic learns from the resulting images and rewards. The central achievement is measured Atari performance without decision-time search. The central qualification is unevenness: strong aggregate scores coexist with weak games, lost visual details and rare events that imagination cannot reliably recover. Read the architecture first, then the score distribution, and finally the token-count intervention that probes one bottleneck. ev-loopev-dynamicsev-imaginationev-resultsev-evaluationev-explorationev-token-figure
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/classical model-based RL category is supported: a learned environment model supplies training trajectories for a separate actor-critic. The WAM quadrant fields remain not applicable. G conditions on actions and predicts future observations, rewards and termination; it neither jointly generates the policy actions nor recovers them through inverse dynamics. Multiple alternating components do not establish a One Model architecture. ev-loopev-dynamicsev-algorithm
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 A vocabulary is useful only if it preserves the game
The first modeling decision is what information reaches temporal prediction. IRIS learns a codebook rather than assigning a token to every pixel. Its encoder finds nearby learned embeddings, and its decoder turns the selected tokens back into images. Reconstruction, commitment and perceptual losses train this stage; there is no adversarial discriminator in the implemented autoencoder. The Transformer then sees a sequence in which a frame’s tokens are followed by an action token. Equation (1) conditions each new visual token on previous frames, actions and the already generated part of its own frame. Figure 9 explains why this division matters: a player erased during encoding cannot be faithfully tracked from that representation alone. This final observation is a reader deduction from the documented compression errors, not a separate measured causal result. ev-tokensev-autoencoderev-dynamicsev-token-figure
5.2 A short dream still needs a value beyond its endpoint
An imagined trajectory begins with real replay data, but subsequent rewards, terminations and observations come from the world model. The policy consumes decoded images through a recurrent actor-critic; Appendix A initializes its hidden state by burning in preceding frames. Because imagination stops after a finite horizon, the final value estimate stands in for later rewards. Equation (4) propagates that estimate backward through lambda-returns and removes continuation when termination is predicted. The critic fits these targets with stopped gradients. The actor uses REINFORCE with the value as a baseline and adds entropy regularization. At real collection time, frames still pass through the autoencoder before action selection, preserving the policy’s reconstructed-image input convention. Algorithm 1 separates model updates from behavior updates; this is how real feedback improves the training environment without supplying a direct real-reward policy objective. ev-imaginationev-configev-rlev-autoencoderev-algorithm
5.3 Separate missing pixels from missing experience
Two distinct failure mechanisms can produce a weak policy. In Alien, the supplied reconstruction comparison shows that small visual details may be lost, and Table 7 tests a larger token budget. In Frostbite, the authors instead emphasize a rare transition into later levels: the world model first needs real examples, after which the policy must encounter and exploit that transition in imagination. The contrasting Krull example motivates their double-exploration explanation. As reader analysis, increasing representation capacity addresses the first bottleneck but need not fix the second. More real data also is not isolated cleanly by Appendix F, because its larger-budget experiment reduces the update-to-data ratio. A discriminating reproduction should therefore measure visual fidelity, transition coverage and executed return separately, instead of treating every gain as evidence for one mechanism. ev-token-figureev-token-tableev-explorationev-scaling
5.4 Training and inference
During training
Alternate real collection, world-model updates and behavior updates. Train the autoencoder with equally weighted L1 reconstruction, commitment and perceptual terms using a straight-through estimator; the VQGAN discriminator is removed. G uses cross-entropy for transitions/termination and either squared error or cross-entropy for rewards, depending on the reward function. ev-algorithmev-tokensev-dynamicsev-autoencoder
The critic regresses to stopped-gradient lambda-returns. The actor uses REINFORCE with a stopped-gradient advantage, a value baseline and entropy regularization. Model parameters are updated in the model phase; actor/critic parameters in the behavior phase. The policy objective uses imagined rewards, rather than differentiating action choices through G. ev-rlev-algorithm
Table 5 specifies 600 epochs, collection during 500 epochs at 200 environment steps each, and component starts after epochs 5/25/50 for autoencoder/Transformer/actor-critic. It lists 200 training steps per epoch, Adam at 0.0001 and gradient clipping at 10.0. ev-config
During inference
In the real environment, pass each observed frame through D(E(x)) before the recurrent policy chooses an action. Execute that action in Atari and collect feedback. G provides imagined training experience; real decision-making uses no lookahead search. ev-autoencoderev-algorithmev-benchmark
Collection epsilon-greedy is 0.01 and evaluation sampling temperature is 0.5. Appendix H separately lowers Freeway sampling temperature from 1 to 0.01 to encourage sustained movement during exploration; this is a game-specific exception. ev-configev-freeway
5.5 Implementation flow
- Build an image vocabulary
The CNN encoder maps each 64 × 64 frame to 16 nearest-codebook tokens from 512 entries. A CNN decoder reconstructs pixels. The codebook embedding dimension is 512; its discrete symbols are learned representations, not hand-labeled objects. ev-tokensev-autoencoder
- Predict action-conditioned dynamics
G receives interleaved frame and action tokens. It predicts the next frame one token at a time, conditioning on earlier tokens of that frame, and predicts reward and termination. The Transformer uses 20 timesteps, 10 layers, four heads and embedding width 256. ev-dynamicsev-config
- Turn predictions into policy experience
Start imagination from a replayed real frame, decode it, sample a policy action, then generate reward, termination and the next frame. Rollouts last up to 20 steps. The actor-critic shares a CNN/LSTM backbone, with a 512-dimensional recurrent state initialized using 20 preceding frames. ev-imaginationev-configev-rl
6. Experiments & results
IRIS learns an Atari policy inside a world model that compresses images into discrete tokens and predicts their evolution with an autoregressive Transformer. Real experience trains the simulator; imagined rewards train a separate recurrent actor-critic. Its Atari 100k mean and interquartile mean exceed the reported no-search baselines, while its median trails SPR. Reconstruction errors, rare events and substantial training compute delimit the result.
6.1 Read the original evidence
Table 1. IRIS leads the listed no-search methods on several aggregates, with a lower median than SPR. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the search columns, MuZero and EfficientZero, from the five no-search columns. Individual game rows contain raw returns; the bottom mean, median and IQM rows use human-normalized scores. The random and human reference columns define that normalization. Read the arrows beside the aggregate names: larger is better except for optimality gap. Bold entries identify the best no-search values, whereas underlining identifies overall best values, as explained in the original caption. Compare IRIS with SPR across all aggregate rows before drawing a conclusion. The median row is essential because it prevents the large mean from becoming a claim of uniform superiority. ev-benchmarkev-resultsev-evaluationev-resources
What it supports. IRIS reports mean 1.046, IQM 0.501 and 10 superhuman games, versus SPR’s 0.616, 0.337 and six. Its median is 0.289 versus SPR’s 0.396. EfficientZero remains stronger on mean at 1.943, but uses lookahead search. The supported result is substantial progress within the listed no-search comparisons.
Where the evidence stops. Baseline run counts and computational setups differ. IRIS uses five runs and 100 evaluation episodes per game per run. The table does not provide per-game uncertainty or an equal-compute comparison.
Figure 5. Uncertainty and choice of aggregate change how the performance claim should be read. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel independently because the horizontal scales differ. Rows align across the mean, median and interquartile-mean panels, with blue for IRIS and orange for SPR. Horizontal colored intervals display the stratified-bootstrap uncertainty described in the caption; the short vertical marks identify point estimates. IRIS sits well to the right of the other methods in the mean and IQM panels, but its median interval spans a much broader region and overlaps competitors. The caption specifies five runs for IRIS and SimPLe, versus 100 for SPR, CURL and DrQ. Interval width therefore should not be interpreted as a pure measure of architectural stability. ev-evaluationev-results
What it supports. The mean and IQM panels reinforce the aggregate improvement, while the median panel qualifies its breadth. In reader terms, a high average can coexist with substantial differences between games. The figure supports reporting several summaries together instead of treating one point estimate as a complete ranking of these agents.
Where the evidence stops. The paper does not tabulate interval endpoints or state a confidence level in this caption. No numerical endpoints are inferred here, and visual overlap alone is not a formal pairwise significance test.
Figure 3. A selected action-conditioned rollout preserves ball motion, paddle motion and a score change. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the top and bottom images within each column, then move from left to right. The top row comes from the real environment. The bottom row begins with two real conditioning frames, outlined in green; later frames are generated while the model receives the actions from the real trajectory. This is a reenactment with a fixed action sequence, not a comparison between independently acting policies. Follow the ball and paddle locations through successive columns. The blue outline highlights the scoreboard change after an exchange, which the caption identifies as evidence that the model captures a consequence of the game’s dynamics as well as movement. ev-pongev-dynamics
What it supports. The authors describe this example as pixel-perfect prediction after 120 training games. The paired rows make their qualitative claim inspectable: the generated sequence tracks the shown real sequence and its score event. This illustrates that a token-based simulator can preserve task-relevant consequences in a favorable game.
Where the evidence stops. This is a selected successful reenactment conditioned on real actions. It does not establish long-horizon error rates, uncertainty calibration, or the return of a policy choosing a new action sequence.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Atari 100k aggregate performance 26 games; 100k actions per game. IRIS uses five runs, evaluating 100 episodes per game at training end. Comparison data use five SimPLe runs, 100 CURL/DrQ/SPR runs and three MuZero/EfficientZero runs. | IRIS: mean 1.046; median 0.289; IQM 0.501; optimality gap 0.512; 10/26 superhuman games. Human-normalized mean, median, IQM; optimality gap (lower is better); superhuman-game count | SPR: 0.616, 0.396, 0.337, 0.577 and 6/26 respectively. Search-based EfficientZero: mean 1.943 and 14/26 superhuman games; its IQM and gap are N/A in Table 1. Strong mean/IQM performance does not imply superiority on the median or over search-based methods. Figure 5 supplies stratified-bootstrap intervals, without tabulated endpoints. The score normalizes agent-minus-random return by human-minus-random return. ev-benchmarkev-resultsev-evaluation |
| Token-count ablation on Alien, Asterix and BankHeist Appendix E compares 16 versus 64 tokens per frame in three Atari games; variant-specific seed counts and evaluation-episode counts are not separately restated. | 64 tokens: Alien 570.0; Asterix 1890.4; BankHeist 282.5. Reported final game return | 16 tokens: 420.0, 853.6 and 53.1 respectively. More tokens improve all three point estimates, but gains vary markedly. Figure 9 shows better Alien reconstruction. No uncertainty estimates or equal-compute control accompany Table 7. ev-token-figureev-token-table |
| Scaling real experience to 10 million environment steps Appendix F increases experience from 100k to 10M steps while reducing optimization-to-environment-step ratio from 1:1 to 1:50. | 10M: mean 7.488; median 1.207; IQM 2.239; 15/26 superhuman games. Human-normalized mean, median and IQM; superhuman-game count | 100k reference: mean 1.046; median 0.289; IQM 0.501; 10/26. Measured improvements support scaling potential, but this changes both data and optimization schedule. It is not a controlled data-only scaling law; the appendix supplies no dedicated uncertainty estimates. ev-scaling |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 9. The visual vocabulary can discard objects before the Transformer ever predicts their motion. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward within a column rather than interpreting the rows as time. Each top image is an original 64 × 64 Alien frame; the middle image reconstructs it with 16 tokens, and the bottom with 64. Compare the small player, the larger colored enemies and the locations of reward objects. The caption identifies player erasure, color changes and misplaced rewards in the middle row. The lower row preserves these details more faithfully in the displayed examples. These are autoencoder reconstructions of the same inputs, so the visual isolates what survives compression; it does not show the Transformer generating a future frame. ev-token-figureev-token-tableev-tokens
What it supports. The source examples locate a plausible failure upstream of temporal prediction: the representation itself can lose consequential visual information. Increasing the token count improves the displayed reconstructions, which motivates checking downstream returns in Table 7. Better-looking frames alone, however, do not measure how much policy performance must improve.
Where the evidence stops. The original observations are low-resolution game images, and this figure supplies selected examples rather than a dataset-wide reconstruction metric. More tokens also require more compute and memory, as Appendix E states.
Table 7. Extra visual tokens improve all three reported scores, but the benefit differs substantially by game. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Focus first on the two rightmost columns: they are the intervention of interest, changing the frame encoding from 16 to 64 tokens. Read across each game separately because these are raw game returns with different scales. The random and human columns provide context, while SimPLe, CURL, DrQ and SPR are reference baselines, not additional token-count variants. Then connect the Alien row to Figure 9: better reconstruction accompanies a return increase, but that increase is smaller than in the other two games. This pairing tests whether a visual improvement is accompanied by a behavioral one, without claiming they are equivalent. ev-token-tableev-token-figure
What it supports. Asterix rises from 853.6 to 1890.4 and BankHeist from 53.1 to 282.5. Alien rises from 420.0 to 570.0 despite the clearer reconstruction examples. The pattern supports token capacity as one bottleneck, with room for exploration or behavioral-learning difficulties to remain after the visual representation improves.
Where the evidence stops. Appendix E gives neither dedicated uncertainty estimates nor an equal-compute control, and does not separately restate variant seed or evaluation-episode counts. Three selected games do not establish a universal benefit from increasing token count.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify double exploration: a rare transition must first occur in real data, then be rediscovered by the policy in imagination. Frostbite illustrates failure to acquire later levels; Krull has more frequent transitions. This explanation is observational rather than a controlled causal ablation. ev-exploration
Selected Pong reenactments and reward/termination examples demonstrate local capabilities, not comprehensive rollout accuracy. The experiments concern Atari control; they establish neither robot execution nor real-world safety. Tokenization can erase small objects before dynamics learning begins. ev-pongev-rewardsev-benchmarkev-token-figure
Section 2.3 says imagined rollouts stop at predicted termination, but Algorithm 1 omits that branch. Equation (4) explicitly masks continuation by termination. The simplified pseudocode should not be treated as a complete implementation specification. ev-imaginationev-algorithmev-rl
7.2 Questions for discussion
- Would equal-compute token-count comparisons preserve the large Asterix and BankHeist gains?
- How much later-level real experience is needed before a policy can rediscover that level in imagination?
8. Reproducibility audit
8.1 Requirements and known gaps
The reported experiments used eight Nvidia A100 40GB GPUs. Two environments sharing one GPU took about seven days; the authors express this as an average 3.5 days per environment. This amortized figure is not a demonstrated single-environment wall-clock time, and sample efficiency is distinct from compute efficiency. ev-resources
Reproduction needs the tokenizer, autoregressive sampler, recurrent-state burn-in, alternating schedule, termination handling and game-specific exploration settings. The PDF does not specify exact emulator/ROM/software versions or the complete preprocessing configuration. Its appendices sometimes say frames where the benchmark definition says actions; do not infer an undocumented frame conversion. ev-benchmarkev-configev-algorithmev-scalingev-freeway
The authors state that code and models are released and describe configuration-based training/evaluation. This source statement alone does not verify artifact availability or successful reproduction. Proposed checks in the illustrated edition target tokenization and rare-event coverage. ev-identityev-codeev-token-tableev-exploration
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does token capacity still help when compute is controlled?
Reader-proposed check, not executed: compare 16 and 64 tokens on Alien, Asterix and BankHeist using matched real-data buffers and multiple seeds. First hold update counts, temporal context and actor-critic settings fixed; then add an equal-GPU-time comparison. Evaluate held-out object reconstruction, action-conditioned rollout errors and real-environment return, reporting intervals and actual compute. If reconstruction improves while the Asterix/BankHeist return advantage disappears under equal compute, Table 7 alone cannot attribute the behavioral gain to better visual information. A persistent gain in both comparisons would strengthen that explanation. ev-token-figureev-token-tableev-configev-evaluationev-resources
Check 2: Can the policy rediscover a level the model already knows?
Reader-proposed check, not executed: use Frostbite buffers of equal total size with controlled amounts of real level-transition experience. Keep architecture and model-update counts fixed. For each learned model, compare ordinary replay-based imagination starts with starts deliberately placed near the transition, without adding direct real-reward policy updates. Measure held-out transition fidelity, imagined later-level visitation and real later-level completion. The double-exploration account predicts that real transition coverage repairs model fidelity, while suitable imagined starts help a policy exploit it. If model fidelity is already high and both start schemes yield the same low visitation, that account needs further explanation. ev-explorationev-imaginationev-algorithmev-config
8.3 Reading coverage
Visual audit: Visually inspected the title/author/revision page, every original figure (1–10), all eight tables, Algorithm 1 and the method/objective pages. Checked Figure 1 arrows against its caption and Equations (1)–(3); checked terminal handling against Section 2.3, Equation (4) and Algorithm 1. Final crops of Figures 1, 3, 5 and 9 and Tables 1 and 7 were each opened and inspected. Appendix pages 15–21 supply architecture, optimization, ablation, scaling, compute and Freeway details used by the report and proposed checks. Reference pages 11–14 were read in the complete text chunks, not visually inspected. External code, videos and separate supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, abstract and Section 1: Introduction (pp. 1–2)
- Sections 2.1–2.3: tokenization, dynamics and learning in imagination (pp. 3–5)
- Sections 3.1–3.3: benchmark, results and world-model analysis (pp. 5–8)
- Sections 4–5: related work and conclusion (p. 9)
- Reproducibility, ethics and acknowledgments (p. 10); references (pp. 11–14)
- Appendix A: architectures and hyperparameters (pp. 15–16)
- Appendices B–D: actor-critic objectives, optimality gap and Algorithm 1 (pp. 17–18)
- Appendices E–H: token counts, data scaling, compute and Freeway exploration (pp. 19–21)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only visual omission above was resolved by inspecting the original PDF figures, tables and equations. All six supplied text chunks were read in full. Reference pages 11–14 were read as text; all other pages were also visually inspected.
- Identity: the title page matches the catalog title and all three authors; it identifies arXiv:2209.00588v2 [cs.LG], 1 March 2023, published at ICLR 2023. The supplied observedTitle retains extraction spacing and capitalization. No different title or author list was observed; earlier revisions were not supplied or compared (ev-identity).
- Code, linked models, external references and separate supplements were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title/author block, arXiv margin and abstract
Title and authors match the catalog; Vincent Micheli and Eloi Alonso share equal contribution, followed by François Fleuret. All list University of Geneva. The artifact is arXiv v2, 1 March 2023, ICLR 2023; the abstract links code/models.
Go to primary source ↓ev-loopPDF p. 2, Figure 1 and caption
Green encoder/decoder arrows map pixels and tokens; purple policy arrows produce actions; blue G arrows predict next tokens, reward and termination. Solid boxes accumulate context.
Go to primary source ↓ev-problemPDF p. 3, Section 2, POMDP formulation and three-step procedure
Observations are images, actions discrete; real experience trains dynamics, while behavior updates occur in imagination.
Go to primary source ↓ev-tokensPDF p. 4, Section 2.1
CNN outputs are quantized to nearest codebook embeddings; decoder reconstructs images. Training combines reconstruction, commitment and perceptual losses with straight-through estimation.
Go to primary source ↓ev-dynamicsPDF p. 4, Section 2.2, Equations (1)–(3) and training paragraph
G models interleaved observation/action sequences, next-frame tokens autoregressively, and reward/termination distributions; listed losses differ by prediction target.
Go to primary source ↓ev-imaginationPDF p. 5, Section 2.3
Replay initializes imagined rollouts; the policy sees decoded frames, simulation stops at horizon or predicted termination, and a value network bootstraps returns.
Go to primary source ↓ev-benchmarkPDF p. 6, Section 3.1
Atari 100k permits 100k actions in each of 26 games, roughly two hours of gameplay; the comparison distinguishes methods with and without decision-time search.
Go to primary source ↓ev-resultsPDF p. 6, Table 1, aggregate rows and IRIS/SPR/EfficientZero columns
IRIS mean/median/IQM/gap are 1.046/0.289/0.501/0.512 with 10 superhuman games; SPR has 0.616/0.396/0.337/0.577 and six. EfficientZero mean is 1.943 with 14 superhuman games.
Go to primary source ↓ev-evaluationPDF p. 7, Section 3.2 and Figures 5–6; p. 8, median discussion
IRIS averages 100 end-of-training episodes over five runs; baseline run counts differ. Human normalization and stratified-bootstrap intervals are described; median intervals overlap.
Go to primary source ↓ev-pongPDF p. 4, Figure 3 and caption
The top row is a real test trajectory; the lower rollout conditions on two frames and then real actions. The authors describe pixel-perfect prediction after 120 training games, including the scoreboard update.
Go to primary source ↓ev-rewardsPDF p. 5, Figure 4 and caption
Breakout/Gopher examples mark predicted positive rewards in yellow and terminations in red, tied to bricks, holes, enemies and failure events.
Go to primary source ↓ev-explorationPDF p. 8, Figure 7, Section 3.2 and Section 3.3
The authors discuss rare level transitions, double exploration, visually difficult games and the consequences of flawed world models; Frostbite and Krull illustrate differing coverage.
Go to primary source ↓ev-codePDF p. 10, Reproducibility Statement
The authors state that code is open-source with configuration files, training/evaluation commands and visualization scripts.
Go to primary source ↓ev-autoencoderPDF p. 15, Appendix A.1, Tables 2–3 and experience-collection note
Frames are 64 × 64, vocabulary 512, tokens per frame 16, code embedding width 512. The discriminator is removed; collection still reconstructs policy inputs.
Go to primary source ↓ev-configPDF pp. 15–16, Appendix A.2–A.3 and Tables 4–5
The Transformer embeds interleaved sequences; Table 4 specifies L=20, width 256, ten layers and four heads. A CNN/LSTM actor-critic uses 512 hidden units and 20-frame burn-in. Table 5 provides schedule, optimizer and sampling settings.
Go to primary source ↓ev-rlPDF p. 17, Appendix B, Equations (4)–(6) and Table 6
Lambda-returns mask future value after termination; value targets and policy advantages stop gradients. The actor uses REINFORCE plus entropy; H=20, gamma=0.995, lambda=0.95, eta=0.001.
Go to primary source ↓ev-algorithmPDF p. 18, Appendix D, Algorithm 1
Collection reconstructs real frames before acting. Model updates and behavior updates are separate; the displayed imagination loop lacks an explicit predicted-termination break.
Go to primary source ↓ev-token-figurePDF p. 19, Appendix E, Figure 9 and caption
Alien columns show original frames above 16-token and 64-token reconstructions. The text identifies missing players, changed colors and misplaced rewards with 16 tokens, and greater compute/memory with more tokens.
Go to primary source ↓ev-token-tablePDF p. 19, Appendix E, Table 7, last two columns
16/64-token returns are Alien 420.0/570.0, Asterix 853.6/1890.4 and BankHeist 53.1/282.5. The appendix does not separately give variant run counts, uncertainty or compute-matched controls.
Go to primary source ↓ev-scalingPDF p. 20, Appendix F, opening paragraph and Table 8 aggregate rows
The 10M-step setting changes update ratio to 1:50 from 1:1; mean/median/IQM become 7.488/1.207/2.239 with 15 superhuman games. No dedicated uncertainty estimates appear.
Go to primary source ↓ev-resourcesPDF p. 21, Appendix G, first paragraph
The authors report five seeds per Atari environment, eight A100 40GB GPUs and seven days for two environments sharing a GPU, averaged as 3.5 days per environment.
Go to primary source ↓ev-freewayPDF p. 21, Appendix H and Figure 10
Freeway requires sustained upward movement for sparse reward. The authors lower sampling temperature from 1 to 0.01 while retaining fixed epsilon-greedy exploration; the text calls the budget 100k frames.
Go to primary source ↓8.5 Primary sources
TRANSFORMERS ARE SAMPLE -E FFICIENT WORLD MODELS ↗
PDF · 9,257 extracted words
Source fingerprint
6b583542ba35dc99a36127dd3b7650aac5d28479f3fe0e5e47f741768ae587b4