Genie: Generative Interactive Environments
1. Paper overview
In one sentence: Genie turns unlabeled video into an interactive world model by learning a small latent action vocabulary, trading broad video supervision for uncertain action semantics and short-horizon consistency. e-designe-lame-inferencee-metricse-bce-limitations
| At a glance | What to know |
|---|---|
| Research problem | Source description Internet gameplay contains transitions without the button presses that caused them. Genie asks whether a model can recover a reusable control interface while learning to generate environments from video alone. Text-derived prompts enter through externally generated images; the described interactive model consumes image prompts and latent actions. e-designe-inferencee-qualitative |
| Core mechanism | Source description A VQ-based latent action bottleneck supplies eight discrete controls without ground-truth action supervision during world-model training. e-lam |
| A key reported result | Latent-action input ablation: Pixel input: Platformers 40.1 / 1.91; Robotics 136.4 / 2.07. FVD ↓; Δ₄PSNR ↑. Table 2: Platformers models of 2.3B/2.5B parameters and Robotics models of 1B; evaluation split details are not specified alongside the table. Token input: Platformers 38.8 / 1.33; Robotics 257.8 / 1.65. Pixels improve action sensitivity in both datasets, but slightly worsen Platformers FVD. Unequal Platformers model sizes weaken causal attribution; no uncertainty is reported. e-lam-ablatione-metrics |
| Reading caution | Source description The authors report unrealistic hallucinated futures, only 16 frames of memory, and approximately 1 FPS interaction. Qualitative motion from varied prompts does not establish persistent long-horizon environments. e-limitations |
Core contributions
- Source description
A VQ-based latent action bottleneck supplies eight discrete controls without ground-truth action supervision during world-model training. e-lam
- Source description
Causal spatiotemporal attention supports video tokenization and action-conditioned dynamics at scale, with separate training and interactive-generation pathways. e-designe-tokenizere-dynamicse-inference
- Source description
Frozen latent-action labeling supports imitation in CoinRun after a small labeled calibration set maps latent codes to real actions. e-bc
Figure 3. Two representations of the same video teach a controllable predictor. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the split after the stack of T input frames. The blue branch produces discrete video tokens z, while the yellow branch infers transition embeddings, shown as tilde a. Both reach the orange dynamics model, but they carry different information: tokenized visual history versus a compact description of change. Section 2.1 adds the training order missing from this overview: learn the tokenizer first, then co-train the latent action and dynamics models. Figure 5 and its text explain that the action encoder can inspect the next frame during training. That future observation supervises a transition code; it is unavailable during interactive generation. e-designe-lame-tokenizere-dynamics
What it supports. The diagram explains how action-free video can support an action-conditioned predictor. The training signal for the action model comes from reconstructing observed transitions, while the dynamics model learns to predict video tokens conditioned on inferred transition embeddings. The components share data without becoming a single joint action-and-video predictor.
Where the evidence stops. This overview omits loss and gradient details. Page 5 explicitly stops gradients through the action inputs to dynamics; the connecting line must not be interpreted as unrestricted end-to-end optimization. Interactive inference also removes most of the LAM.
2. Motivation
2.1 The problem and the proposed response
Internet gameplay contains transitions without the button presses that caused them. Genie asks whether a model can recover a reusable control interface while learning to generate environments from video alone. Text-derived prompts enter through externally generated images; the described interactive model consumes image prompts and latent actions. e-designe-inferencee-qualitative
2.2 What this reading follows
A gameplay video shows what changed but usually omits the button that caused it. Genie learns a compact set of transition codes from those missing-action examples, then lets a user choose the codes while a dynamics model generates the next frame. The crucial separation is between discovering controls during training and supplying controls during play. Its evidence also separates three questions: whether videos look plausible, whether changing a code changes the future, and whether inferred codes help a policy act in a real simulator. This reading follows those distinctions through the architecture, two ablations and a calibrated CoinRun imitation experiment. e-designe-lame-inferencee-metricse-bce-limitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
Foundational work, neural world simulation and latent action pretraining fit the architecture and experiments. Retaining Not applicable for the action-prediction quadrants is appropriate: Genie combines separate tokenizer, inverse-transition latent inference and forward dynamics modules, while its downstream imitation policy is trained separately. Co-training is insufficient to classify this as one model jointly predicting executable actions and futures. e-designe-lame-dynamicse-bc
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a control bottleneck from a future frame
The missing action label is replaced by a learned reconstruction problem. During training, the latent action encoder receives past frames plus the next observed frame. Its output must pass through a small discrete vocabulary before a separate decoder can reconstruct that future from the history. Because history is already available to the decoder, the code is encouraged to carry information about the transition. For the main system, eight codes make the interface small enough to explore manually. This objective supplies useful pressure toward change information, but the paper does not prove that it isolates the original controller command. The pixel-versus-token ablation is therefore especially relevant: it asks whether a compressed visual representation has already removed information the action learner needs. Its results favor pixels on the proposed controllability proxy, with a Platformers fidelity tradeoff. e-lame-lam-ablatione-metrics
Figure 8. During play, the user supplies controls and generated tokens supply the next visual context. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at x_1 on the left. The encoder produces z_1; a user chooses a discrete action and the retained codebook supplies its embedding. The orange model generates the next frame's tokens, and the dashed upper loop feeds predictions back into the input history. The blue decoder converts those tokens to visible frames. The pale yellow entries indicate continued action inputs, not actions predicted by the dynamics model. Section 2.2 verifies this one-action/next-frame sequence. Main-model sampling uses 25 MaskGIT iterations at temperature 2 for each frame, so one environment step involves repeated token refinement before temporal generation continues. e-inferencee-lame-dynamicse-model-config
What it supports. This is an autoregressive simulator with an external controller. The next image depends on both the generated past and the selected latent code. The LAM reconstruction machinery is needed to learn the interface, but human interaction retains only its codebook rather than inferring actions from an unseen future.
Where the evidence stops. Figure 8 labels pending actions tilde a_{2:T}, whereas the text describes T−1 transition actions for T frames. This indexing inconsistency remains unresolved; the verified sequential procedure supplies one action per new frame. The diagram does not establish joint action prediction.
5.2 Generate a world by alternating controls and predictions
Once training finishes, interactive generation no longer has an observed next frame from which to infer an action. The user instead selects an index in the learned codebook. The prompt's video tokens and that embedding condition a dynamics transformer, which iteratively fills the next frame's tokens using MaskGIT. The tokenizer decoder makes the prediction visible, while generated tokens become part of the context for later steps. Thus temporal autoregression and within-frame token refinement are different loops. Causal spatial/temporal attention makes the model usable with a growing history, but the implementation still has a 16-frame memory limit. The reported roughly one-frame-per-second operation is also distinct from the 10-FPS training videos. These limits explain why frame-level control and persistent, responsive environments remain separate achievements in the reviewed version. e-designe-inferencee-model-confige-limitations
5.3 Test transfer only after accounting for calibration
The CoinRun experiment supplies a more demanding test than changing a generated picture. A frozen Internet-trained LAM labels expert transitions, and a separate imitation policy learns to predict those latent labels from observations. Real environment actions still need a bridge: a small labeled expert set populates a dictionary from each latent code to actual actions. Evaluation then closes the loop through CoinRun, not through Genie's video generator. As reader analysis, success with an observation-independent dictionary is evidence that some code meanings transfer across states. It does not imply zero-label imitation, nor demonstrate a general agent trained by exploring generated worlds. The most informative follow-up is to hold imitation data fixed and test whether calibration remains effective on unseen levels, with shuffled mappings and an oracle under matched action-sampling rules. e-bc
5.4 Training and inference
During training
Train the tokenizer first, then co-train LAM and dynamics. Tokenizer and LAM use VQ-VAE objectives; dynamics uses token cross-entropy. Its random masking rate is sampled uniformly from 0.5 to 1. Training sequences contain 16 frames at 10 FPS, distinct from the collected 16-second clips. e-designe-lame-dynamicse-datae-model-config
The filtered corpus contains 6.8M clips, approximately 30,000 hours, from an initial 55M clips. The nominal 11B system is reported as 10.7B total parameters with a 10.1B dynamics model, trained for 125k updates at batch 512 on 942B tokens. Page 7 specifies 256 TPUv5p; Appendix D says 256 TPUv5. The website decoder additionally produces 360p output. e-datae-scaling
The tokenizer uses 1,024 codes and patch size 4; the main LAM uses eight codes and patch size 16. Both code embeddings have dimension 32. Dynamics training uses bfloat16 and QK normalization. e-model-config
During inference
Encode the initial image, map a chosen integer to its codebook embedding, predict the next frame's tokens, decode them, and feed generated tokens into subsequent predictions. The LAM encoder and reconstruction decoder are discarded for human interaction; only its codebook remains. Main-model sampling uses 25 MaskGIT iterations per frame at temperature 2. e-lame-inferencee-model-config
CoinRun uses a different execution path: freeze the pretrained LAM, label expert transitions, train a policy by cross-entropy, and build a dictionary from latent codes to observed real actions using labeled expert transitions. The policy samples a latent action and the dictionary supplies an executable environment action. e-bc
5.5 Implementation flow
- Represent video with causal spatial and temporal attention
Each ST block applies within-frame spatial attention, causal temporal attention at corresponding spatial locations, then one feed-forward layer. The dominant spatial-attention cost grows linearly with frame count; this is not a claim that every attention term is linear. e-design
- Learn transition codes from pixels
The LAM encoder sees the history and next frame. Its VQ bottleneck restricts the transition representation; a decoder reconstructs the next frame using the history and latent action. This reconstruction objective encourages informative changes, but does not identify a ground-truth motor command. e-lam
- Predict discrete future frames
The video tokenizer compresses frames with temporal context. A decoder-only MaskGIT dynamics model predicts future tokens using past tokens and additive action embeddings. Actions are stop-gradient inputs to the dynamics loss; co-training therefore does not imply unrestricted end-to-end gradient flow. e-tokenizere-dynamics
6. Experiments & results
Genie learns a small latent control interface from unlabeled video, then generates future frames from an image prompt and user-selected codes. A separate video tokenizer, latent action model and dynamics transformer make this possible. Its ablations distinguish visual fidelity from action sensitivity; its CoinRun experiment transfers inferred actions into a separately trained imitation policy. This is evidence for interactive neural simulation and latent-action pretraining, with limited evidence for persistent worlds or general-purpose control.
6.1 Read the original evidence
Figure 17. Repeated latent codes have recognizable effects across selected game scenes. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as separate interventions on the prompt in its first column. The four following columns show the result of repeating a selected latent action five times; they are not consecutive frames of one trajectory. Then scan vertically down a colored column to ask whether the same code has a related effect in different scenes. The paper labels these effects left, right, jump and no-op after learning from action-free videos. Distinguish avatar position from scrolling scenery when judging movement. No-op should be read as the proposed control meaning, not as a guarantee that the entire generated scene remains pixel-identical. e-qualitativee-lame-metricse-limitations
What it supports. These examples support a qualitative claim of reusable action semantics: the learned interface is more structured than an arbitrary switch between unrelated images. The figure complements the PSNR-based action-sensitivity metric by showing what some of the visible changes look like across four prompts.
Where the evidence stops. The panel shows selected examples and four semantic labels from the main eight-code interface. It does not establish universal meanings for every code, quantify failure frequency, or verify long-horizon persistence. Apparent movement can include camera motion.
Figure 15. Latent-action imitation approaches the oracle after calibrating codes to real actions. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the easy and hard panels, then read the horizontal axis as labeled expert samples used to map latent actions to real actions. It is not the total amount of video used to train the LAM or imitation policy. The vertical axis is the percentage of levels solved. Purple traces show the Genie-LAM policy and 95% confidence intervals; cyan and red dashed lines mark the oracle behavioral-cloning and random baselines. The caption reports 100 evaluation samples and five seeds. Appendix E explains the dictionary mapping and the held-out evaluation, which are necessary to interpret the rising curves as simulator performance. e-bc
What it supports. The curves rise rapidly above the random baseline and approach the oracle with relatively little labeled calibration. The text claims matching the oracle with as few as 200 samples. This supports useful transferable structure in latent codes, but the plotted values should not be replaced by invented exact success percentages.
Where the evidence stops. The plot does not show exact equality at 200 samples in both modes. Appendix E also gives the oracle 10% random actions. This experiment uses real CoinRun interaction for evaluation and expert videos for imitation; it is not reinforcement learning inside Genie.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Latent-action input ablation Table 2: Platformers models of 2.3B/2.5B parameters and Robotics models of 1B; evaluation split details are not specified alongside the table. | Pixel input: Platformers 40.1 / 1.91; Robotics 136.4 / 2.07. FVD ↓; Δ₄PSNR ↑ | Token input: Platformers 38.8 / 1.33; Robotics 257.8 / 1.65. Pixels improve action sensitivity in both datasets, but slightly worsen Platformers FVD. Unequal Platformers model sizes weaken causal attribution; no uncertainty is reported. e-lam-ablatione-metrics |
| Tokenizer architecture ablation Table 3: patch size 10, batch 128, sequence length 16; the same downstream dynamics/LAM setup is trained with each tokenizer. Dataset/split is not explicitly restated in the table. | ST-ViViT: 81.4 / 1.66 / 0.9GB, 205M parameters. FVD ↓; Δ₄PSNR ↑; memory | ViT: 114.5 / 1.39 / 0.3GB, 230M; C-ViViT: 272.7 / 1.37 / 1.6GB, 225M. ST-ViViT provides the best reported fidelity and action sensitivity, with memory between the alternatives. These are downstream video metrics, not tokenizer reconstruction scores. e-tokenizer-ablatione-metrics |
| CoinRun imitation transfer Easy and hard CoinRun; held-out test set, 100 evaluation samples, five seeds and 95% confidence intervals. | The authors report oracle-level performance with as few as 200 labeled expert samples; exact success percentages are not tabulated. Percentage of levels solved versus labeled expert calibration samples | Figure 15 compares a Genie-LAM policy with oracle behavioral cloning and a random agent. The plotted curves approach the oracle; they do not establish exact equality at 200 samples in both modes. This is calibrated imitation in the simulator, not agent training through Genie-generated rollouts. e-bc |
| Robotics video generation Separate 2.5B model on the Robotics mixture; reported test split. | 82.7 FVD ↓ | No matched baseline is given for this result; Table 2's 1B Robotics ablation is a different configuration. Figures show generated arm motion and object deformation. Neither FVD nor these examples measure physical robot task completion. e-roboticse-lam-ablation |
| Platformers dataset curation Appendix B, Table 4; both models have 580M parameters; split and uncertainty are not specified there. | 54.8 with 6.8M curated clips. FVD ↓ | 61.4 with 55M original clips. Filtering improves the reported fidelity despite reducing data volume. The result does not isolate a particular filtering criterion. e-datae-curation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Direct pixel input improves action sensitivity, with a fidelity tradeoff on Platformers. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare within each dataset pair before comparing across datasets. Lower FVD means better reported video fidelity; higher delta-PSNR means a larger reconstruction advantage for inferred over random actions, evaluated at the fourth frame. In Platformers, the pixel-input model improves delta-PSNR from 1.33 to 1.91, while FVD changes from 38.8 to 40.1 in the unfavorable direction. In Robotics, pixels improve both measures: FVD falls from 257.8 to 136.4 and delta-PSNR rises from 1.65 to 2.07. Finally check the parameter column: the Platformers comparison uses different total sizes, while both Robotics rows are marked 1B. e-lam-ablatione-metrics
What it supports. The main design choice targets controllability rather than optimizing video fidelity alone. Raw pixels provide a stronger action-sensitivity result in both datasets, consistent with the authors' hypothesis that tokenization can discard movement information. Robotics additionally shows a large fidelity improvement under the compared configurations.
Where the evidence stops. Platformers uses 2.3B versus 2.5B models, so input representation is not the only changed factor. The table gives no uncertainty or detailed split specification, and delta-PSNR does not establish correct motor semantics or physical execution.
Table 3. Temporal tokenization helps when architecture and resource costs are considered together. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the last two columns as downstream video-generation measurements after training dynamics and latent-action models with each tokenizer. They are not reconstruction PSNR scores for the tokenizer itself. ST-ViViT has FVD 81.4 and delta-PSNR 1.66, compared with 114.5 and 1.39 for spatial-only ViT. C-ViViT reports 272.7 and 1.37. Now return to the resource columns: ST-ViViT's reported 0.9GB lies between ViT's 0.3GB and C-ViViT's 1.6GB. The text fixes patch size 10, batch size 128 and sequence length 16 for this comparison, with similar but unequal tokenizer parameter counts. e-tokenizer-ablatione-metrics
What it supports. ST-ViViT offers the best measured combination of fidelity and action sensitivity in this ablation while using less memory than C-ViViT. The spatial-only alternative is cheaper in the reported memory column, so the result supports a quality/resource tradeoff rather than minimum memory use.
Where the evidence stops. The table does not establish a universal tokenizer ranking or quantify uncertainty. Page 10 suggests C-ViViT overfitting and regularization difficulties as an explanation; this is not a separately isolated causal result. Detailed split and memory-accounting conventions are not given here.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report unrealistic hallucinated futures, only 16 frames of memory, and approximately 1 FPS interaction. Qualitative motion from varied prompts does not establish persistent long-horizon environments. e-limitations
Δ₄PSNR tests sensitivity to inferred versus random actions. It cannot by itself separate useful control from action-dependent visual disruption. The tokenizer comparison also changes parameter counts; the authors suggest C-ViViT overfitting as a possible explanation rather than demonstrating it conclusively. e-metricse-tokenizer-ablation
The reviewed version explicitly withholds checkpoints, training data and examples from that data. Its limited CoinRun recipe does not replace access to the original large-scale training assets. e-releasee-small-case
7.2 Questions for discussion
- Would action sensitivity still predict useful control after matching model sizes and measuring held-out movement semantics?
- How much does CoinRun transfer depend on expert calibration coverage rather than the stability of the latent action vocabulary?
8. Reproducibility audit
8.1 Requirements and known gaps
A smaller check can follow Appendix F: collect 10M random-policy CoinRun-hard transitions; train a tokenizer for 300k steps at batch 48×16 frames, then LAM/dynamics for 200k steps at batch 36×16. The stated target is 16G memory on one TPU; under-one-week TPU/GPU feasibility is the authors' claim, not an independently measured result. This case uses six action codes and sampling temperature 1.0. e-small-case
Implementation gaps remain: Table 17 repeats num_layers with values 12 and 8, so the second entry cannot safely be relabeled num_heads. Table 8 lists equal maximum/minimum learning rates despite prose describing cosine decay. The filter confidence threshold, precise evaluation splits/sample counts for FVD, and complete BC training duration are not supplied in the relevant protocols. e-small-casee-model-confige-datae-metricse-bc
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does pixel input improve useful control at a matched budget?
Reader-proposed check; not run. Use the small CoinRun setup in Appendix F to compare pixel-input and token-input LAMs with the same tokenizer, transition data, vocabulary, downstream dynamics architecture and training budget, matching trainable parameter counts as closely as possible and reporting any remaining difference. Keep true actions hidden during model training and reserve them for evaluation. Measure FVD, Δ₄PSNR and agreement between latent codes and held-out transition effects across multiple seeds. Include shuffled action embeddings at evaluation as a negative control. The mechanism is weakened if the pixel advantage vanishes after matching capacity, or if higher Δ₄PSNR accompanies worse semantic consistency. Table 17's ambiguous architecture entry must be documented before implementation. e-lam-ablatione-metricse-small-case
Check 2: Does a small latent-to-real mapping generalize beyond its calibration set?
Reader-proposed check; not run. In a reconstruction of the Appendix E transfer experiment, freeze the LAM and latent imitation policy, then vary only the labeled calibration set used for the dictionary. Compare 0, 50, 100, 200 and 1,000 expert transitions on disjoint CoinRun levels; use independent calibration samples and report level-solved rates with intervals. At zero labels use an explicitly declared random mapping. Controls are a shuffled dictionary, a random policy, and an oracle with the same inference exploration rule, including a separate test of the reported 10% random-action choice. Record missing-code coverage. If shuffled mappings perform similarly, or the 200-sample benefit disappears on held-out levels, stable transferable action semantics would be less well supported. e-bc
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page; all numbered Figures 1–17 and Tables 1–17; method, metric, result and limitation pages 1–11; and appendix figures, data filtering, optimizer/hardware settings, behavioral cloning and the small reproduction case on pages 20–27. All six final original crops were viewed and checked for readable labels, legends and complete table rows. Narrow page-5/page-9 crops were rendered at 400 DPI; other crops used 200 DPI. The training overview was cross-checked against the stop-gradient statement on page 5, and Figure 8's action-index discrepancy is disclosed. Pages 12–19 (contributions and references) were read as text only. Separate supplements and external video demonstrations were not supplied or inspected; static figures cannot establish temporal quality beyond the depicted frames.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 20, 21, 22, 23, 24, 25, 26, 27. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, abstract and introduction (pp. 1–3)
- 2. Methodology, including model components and inference (pp. 3–5)
- 3. Experimental Results: datasets, metrics, training, scaling, qualitative results, agent training and ablations (pp. 5–10)
- 4. Related Work (pp. 10–11)
- 5. Conclusion and Future Work; Broader Impact; Acknowledgements (p. 11)
- Author Contributions and References (pp. 12–19)
- Appendix A: Additional Example Trajectories (pp. 20–21)
- Appendix B.1: Platformers Dataset (pp. 21–22)
- Appendix C: Training details (pp. 22–24)
- Appendix D: Scaling Experiments Details (pp. 23–25)
- Appendix E: Behavioral Cloning Details (pp. 24–26)
- Appendix F: Reproducible Case Study (pp. 26–27)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only image omission above was addressed by visually inspecting PDF pages 1–11 and 20–27, including all numbered figures and tables. Pages 12–19 were read as text; no separate supplement was supplied.
- Identity/version note: the title matches exactly and all 25 authors have compatible identities. The supplied artifact is arXiv:2402.15391v1, marked 23 Feb 2024, with a title-page date of 2024-2-26. The catalog cites ICML 2024; that venue edition was not supplied, so equivalence and revision changes cannot be established. Observed author forms include Michael Dennis, Yuge (Jimmy) Shi, Sarah Bechtle, Stephanie Chan and Nando de Freitas, differing from catalog expansions or capitalization. Metadata preserves the observed title-page names.
- External project demonstrations, the venue publication, code and other linked resources were not inspected. No installation or experimental reproduction was attempted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, complete author line, affiliations, header date and arXiv margin stamp
Exact title and 25 credited authors; affiliations Google DeepMind and University of British Columbia. The artifact displays arXiv:2402.15391v1, 23 Feb 2024, and header date 2024-2-26.
Go to primary source ↓e-designPDF pp. 2–4, §§1–2.1; Figures 3–4; Table 1
Video-only training, three model components, staged training and ST blocks with spatial attention, causal temporal attention and a single subsequent feed-forward layer.
Go to primary source ↓e-lamPDF pp. 3–4, §2.1 Latent Action Model; Figure 5
The encoder uses past and next frames; a VQ-VAE bottleneck limits the main vocabulary to eight actions. A history-conditioned decoder supplies reconstruction supervision. Interactive inference retains only the LAM codebook.
Go to primary source ↓e-tokenizerPDF p. 4, §2.1 Video Tokenizer; Figure 6
A VQ-VAE with ST-transformer encoder/decoder maps frames to discrete video tokens. Causal token encodings contain information from previously seen frames.
Go to primary source ↓e-dynamicsPDF p. 5, §2.1 Dynamics Model; Figure 7
Decoder-only MaskGIT predicts next-frame tokens with token cross-entropy, stop-gradient action inputs, Bernoulli token masking with rate sampled from 0.5–1, and additive action embeddings.
Go to primary source ↓e-inferencePDF p. 5, §2.2 and Figure 8, including footnotes 1–2
Prompt frames are encoded; a discrete user choice indexes the action codebook; generated tokens are recursively fed to dynamics and decoded to frames. The code meanings must be learned by the user.
Go to primary source ↓e-dataPDF p. 5, §3 Datasets; pp. 21–22, Appendix B.1
Initial 55M 16-second clips at 10 FPS and 160×90 resolution are filtered to 6.8M clips, about 30k hours. Keyword screening and a classifier trained on human quality ratings are described, but the final confidence decision threshold is not.
Go to primary source ↓e-metricsPDF p. 6, §3 Metrics and displayed ΔtPSNR equation
FVD measures video fidelity. ΔtPSNR subtracts reconstruction PSNR with random actions from PSNR with inferred actions; all reported experiments use t=4. Detailed FVD test-set sizes and uncertainty are not provided here.
Go to primary source ↓e-model-configPDF p. 6, Training Details; pp. 22–24, Appendix C and Tables 5–9
Main tokenizer: about 200M parameters, 1,024 codes, patch 4. LAM: about 300M, eight codes, patch 16. Both embeddings are 32-dimensional. Sequences are 16 frames at 10 FPS; main sampling uses 25 steps and temperature 2. Dynamics uses bfloat16 and QK norm. Tokenizer training is 300k AdamW steps; Table 8 has max_lr=min_lr=3e-4 despite cosine-decay prose.
Go to primary source ↓e-scalingPDF pp. 6–7, §3.1 and Figure 9; pp. 23–25, Appendix D and Tables 10–12
Dynamics-size and batch-size sweeps improve reported training loss. The final model has 10.1B dynamics parameters, 10.7B total, batch 512 and 125k steps, with 942B tokens reported on p. 7. That page specifies 256 TPUv5p, whereas p. 24 says 256 TPUv5. Table 12 reports 6.6×10^22 FLOPs. A larger website decoder produces 360p videos.
Go to primary source ↓e-qualitativePDF p. 7, §3.2 and Figure 10; pp. 20–21, Appendix A, Figures 16–17
Examples start from generated images, sketches and photographs. Figure 17 compares four prompts under latent codes described as left, right, jump and no-op; each output follows five repetitions of a code. These are selected qualitative examples.
Go to primary source ↓e-roboticsPDF p. 6, Robotics dataset paragraph; p. 8, §3.2 Robotics-trained model and Figures 11–13
The Robotics mixture includes RT1 demonstrations, simulation and earlier real-robot episodes, used without actions. A 2.5B model obtains test FVD 82.7; generated examples illustrate motion, deformation and consistent down/up/left controls.
Go to primary source ↓e-bcPDF p. 9, §3.3 and Figure 15; pp. 24–26, Appendix E and Tables 13–14
A frozen LAM labels expert video; a separate policy learns latent-action imitation. Labeled expert transitions populate a latent-to-real-action dictionary. CoinRun easy/hard results use a held-out test set, 100 samples, five seeds and 95% confidence intervals. The text claims matching the oracle with 200 calibration samples; the plot provides no exact numerical table. BC uses four-frame sequences, batch 16, sampled logits; the oracle benefits from 10% random actions. The appendix's later pi(s_t) notation leaves s_t undefined.
Go to primary source ↓e-lam-ablationPDF p. 9, §3.4 and Table 2, all rows and FVD/ΔtPSNR columns
Platformers token input (2.3B): 38.8/1.33; pixel input (2.5B): 40.1/1.91. Robotics token input (1B): 257.8/1.65; pixel input (1B): 136.4/2.07. Uncertainty and detailed split definitions are not listed.
Go to primary source ↓e-tokenizer-ablationPDF p. 9, §3.4 and Table 3; p. 10, opening two paragraphs
With patch 10, batch 128 and 16 frames, ViT reports 230M/0.3GB/114.5/1.39; C-ViViT 225M/1.6GB/272.7/1.37; ST-ViViT 205M/0.9GB/81.4/1.66 (parameters/memory/FVD/ΔtPSNR). The same downstream architecture is trained per tokenizer. C-ViViT overfitting is suggested as an explanation.
Go to primary source ↓e-curationPDF p. 22, Appendix B.1 and Table 4
At 580M parameters, the original 55M-clip dataset yields FVD 61.4 and the curated 6.8M-clip dataset yields 54.8; no uncertainty is given.
Go to primary source ↓e-limitationsPDF p. 11, §5 Conclusion and Future Work
Authors identify hallucinated unrealistic futures, a 16-frame memory limit, difficulties maintaining consistency over long horizons, and interaction at around 1 FPS.
Go to primary source ↓e-releasePDF p. 11, Broader Impact: Training Data and Weights; Reproducibility
The reviewed version explicitly does not release trained checkpoints, training data or examples from that data, and motivates the smaller Appendix F example.
Go to primary source ↓e-small-casePDF pp. 26–27, Appendix F.1–F.3 and Tables 15–17; p. 24, Table 9
CoinRun-hard random-policy collection produces 10M transitions. Tokenizer: batch 48 of length 16, 300k steps, 16G TPU memory and three days reported. LAM/dynamics: batch 36 of length 16, 200k steps using Table 9. The small LAM has six codes; sampling temperature is 1.0 with 25 steps. Table 17 duplicates num_layers as 12 and 8 without resolving the second label; exact mid-range device and software versions are not stated.
Go to primary source ↓8.5 Primary sources
Genie: Generative Interactive Environments ↗
PDF · 11,199 extracted words
Source fingerprint
94c63a5ced89355326706c88236591c2972ced12f3dece4f8982f020110c81f3