Factored Latent Action World Models
1. Paper overview
In one sentence: FLAM gives each learned scene factor its own latent action while sharing interaction-aware dynamics, improving multi-entity video modeling but still requiring action supervision to train an executable policy. e-probleme-dynamicse-protocole-policy
| At a glance | What to know |
|---|---|
| Research problem | Source description A single scene-level latent action must compress simultaneous changes from several independently moving entities. FLAM asks whether learning separate state factors and action bottlenecks makes this problem easier without action labels or a separately pretrained object-centric representation. The intended factors follow action independence, so two visually distinct entities can share a factor when their actions correlate. e-probleme-correlated |
| Core mechanism | Source description The model jointly learns a slot factorizer, shared inverse dynamics, shared forward dynamics and a feature aggregator under a next-feature prediction objective. Factorization is learned through dynamics after separate image-tokenizer pretraining. e-encodere-dynamicse-objective |
| A key reported result | Limited-label Procgen behavior cloning: 10k labels: Bigfish 8.8; Starpilot 9.1. At 1k: both 1.8. Mean episodic return ↑. One million expert demonstration frames per environment; 1k or 10k action-labeled frames; pseudo-labeled full set; evaluation on disjoint, unseen levels. Vanilla BC at 10k: 3.6 / 6.4; at 1k: 1.0 / 1.9. Random: 0.9 / 1.1, respectively. Useful gains with 10k labels; Starpilot shows no improvement at 1k. No uncertainty or episode count accompanies Table 4. e-policye-policy-protocol |
| Reading caution | Source description The authors identify dataset-specific tokenizers and the pretrained visualization decoder as limits to a universal, higher-fidelity model. Diffusion and flow-matching decoders are suggestions, not evaluated improvements. e-limitations |
Core contributions
- Source description
The model jointly learns a slot factorizer, shared inverse dynamics, shared forward dynamics and a feature aggregator under a next-feature prediction objective. Factorization is learned through dynamics after separate image-tokenizer pretraining. e-encodere-dynamicse-objective
- Reader analysis
The experiments connect prediction accuracy to representation diagnostics and limited-label policy learning. This makes factor quality testable separately from image quality rather than assuming good reconstructions imply usable controls. e-resultse-dcie-policye-ablation
Figure 3. The bottleneck is per factor; scene context and network weights are shared. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) upward: the encoder produces features, FSQ quantizes them, and the decoder reconstructs the image. In panel (b), the snowflakes mark frozen tokenizer components. Follow the current frame on the left and next frame on the right into their factorizers. The central block repeats across K slots with shared IDM/FDM weights. Equations (4)–(5) establish the crucial asymmetry: the IDM receives all current slots and one next slot; the FDM receives all current slots and one action. Finally, follow the long current-feature bypass into the aggregator, which combines existing visual context with predicted slot changes. e-encodere-factorizere-dynamicse-objectivee-protocol
What it supports. The architecture makes each factor responsible for transmitting its own unpredictable transition information. At the same time, shared current-slot context lets both dynamics networks account for interactions. The aggregator's current-feature input reduces the need to encode unchanged appearance inside the action-bearing slots; prediction training can concentrate those slots on dynamics.
Where the evidence stops. The caption's phrase “all modules trained jointly” must be read with the frozen snowflakes and Sections 4.2/5: the four LAM components are optimized jointly in stage two, not the entire pretrained tokenizer. The future-frame branch is available for action inference, not future-free generation.
2. Motivation
2.1 The problem and the proposed response
A single scene-level latent action must compress simultaneous changes from several independently moving entities. FLAM asks whether learning separate state factors and action bottlenecks makes this problem easier without action labels or a separately pretrained object-centric representation. The intended factors follow action independence, so two visually distinct entities can share a factor when their actions correlate. e-probleme-correlated
2.2 What this reading follows
Imagine a grid scene in which several agents turn and move independently. A single latent action has to describe their combined transition; FLAM instead learns slots and gives each slot its own constrained action channel. Crucially, each dynamics network still sees the current scene, so factoring actions does not discard interactions. This reading follows the architecture through prediction, representation diagnostics and behavior cloning. The main prediction scores use actions inferred from future ground-truth frames. The separate policy experiment asks a different question: whether a small labeled subset can turn video-inferred actions into useful training labels. Both support the method, under different evidence boundaries. e-probleme-dynamicse-protocole-policy
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned, so there is no existing quadrant claim to support or contradict. Architecture evidence shows distinct inverse- and forward-dynamics modules plus a separate downstream policy. Latent actions are inferred from observed transitions; they are not jointly predicted with futures by one policy/world-model backbone. Joint optimization of the four LAM components alone does not establish a One Model classification. e-dynamicse-objectivee-utilization
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Factor the action channel without removing interactions
The key bottleneck is what information reaches the forward model from the future. During training, the IDM sees a transition and must describe its unpredictable part through a restricted latent action. FLAM repeats this channel for every slot, using shared networks and a shared normal prior. Each IDM still sees all current slots, which can explain interactions already apparent in the scene, but it sees only its target slot at the next time. The corresponding FDM gets all current slots and only that target action. This makes per-factor action responsibility explicit without assuming that entities never interact. The KL term discourages copying the next state wholesale. These actions describe visible transition information, however, not uniquely recoverable motor commands: blocked forward movement can be indistinguishable from staying still. e-probleme-dynamicse-objectivee-action-ambiguity
5.2 Test what the learned slots actually organize
A useful slot must persist long enough for an inferred action to refer to the same entity across a transition. FLAM's causal temporal attention provides that continuity, while prediction loss trains the factorizer together with dynamics. The controlled correlated-agent experiment gives the claim a sharper meaning: two agents sharing actions can occupy one factor when K=3. With four independent agents, too few factors hurt prediction, while the reported K=4,8,16 results are similar. Reader interpretation: the representation is organized by the dynamics that the task needs to transmit, not by a requirement to segment every visible object. The global-coupling ablation strengthens this distinction. Giving dynamics access to all action channels preserves image quality but weakens D and I, so image reconstruction alone would be an inadequate test of the intended factorization. e-factorizere-objectivee-correlatede-factor-counte-ablatione-dci-protocol
5.3 Separate video reconstruction from a usable policy
The paper uses learned latent actions in different information regimes. In the prediction benchmark, the IDM extracts actions from ground-truth future frames before the autoregressive rollout. In controllable generation, a user changes selected actions while other factors can follow the original video. Neither procedure alone defines an agent that chooses task-solving physical actions from a current image. The policy experiment adds that bridge: a small labeled subset trains an action decoder; FLAM labels the remaining expert video; and a separate behavior-cloning policy learns from those pseudo labels. At execution, the described policy uses observations to choose environment actions rather than consulting future frames. Table 4 measures the resulting policy return. Its stronger 10k-label gains and weak 1k-label result show that learning the action mapping remains a substantive supervision requirement. e-utilizatione-protocole-horizone-policye-policy-protocol
5.4 Training and inference
During training
After tokenizer pretraining, optimize the four latent-action modules jointly with feature prediction error plus per-slot KL penalties. Table 5 specifies AdamW at 10⁻⁴ in both stages, tokenizer/LAM batch sizes 64/32, and default β=2×10⁻⁴. It uses four factors for default MultiGrid and sixteen for Procgen and nuPlan. e-objectivee-config
Comparisons share each dataset's pretrained encoder and applicable component architectures. Total latent-action dimension is matched: a monolithic d-dimensional action is compared with K actions of dimension d/K. This controls a major capacity difference without proving identical compute. e-protocol
During inference
For video editing, select an entity and sample or select its latent actions from the prior, then roll out the FDM and aggregator autoregressively. Other entities retain actions inferred from the original video. These are generated trajectories, not executed physical interventions. e-utilizatione-control
For policy learning, train an action decoder on labeled demonstrations, infer and decode actions for the larger video set, then behavior-clone those pseudo labels. The deployed policy maps observations to environment actions; the described procedure does not plan by searching world-model rollouts. e-utilizatione-policye-policy-protocol
5.5 Implementation flow
- Pretrain a visual tokenizer
A CNN encoder maps pixels to patch features z; finite scalar quantization (FSQ) and a decoder reconstruct the image. The paper calls this a VQ-VAE. Each dataset receives its own tokenizer, and the encoder is frozen for subsequent latent-action learning. e-encodere-protocol
- Bind slots across time
Learned slot embeddings compete for image patches through iterative Slot Attention. Each iteration also includes causal temporal self-attention to the same slot's past values, encouraging consistent entity bindings. e-factorizer
- Infer factor actions with shared context
For slot i, the IDM sees all current slots and only that slot's next value. Spatial attention captures interactions; temporal cross-attention extracts the transition. A Gaussian posterior produces the latent action, regularized toward a unit normal prior. Separate factors therefore do not imply isolated dynamics. e-dynamics
- Predict factors and restore visual detail
The shared FDM sees all current slots but only the target factor's action. Predicted slots become keys for aggregator cross-attention, whose queries are current patch features. This bypass lets slots focus on changes while current features retain static appearance; predicted features are quantized and decoded for visualization. e-dynamicse-objectivee-encoder
6. Experiments & results
FLAM learns a video world model whose slots each carry a latent action while sharing interaction-aware dynamics networks. Prediction-trained factorization improves rollouts supplied with future-inferred actions and can supply pseudo action labels for behavior cloning. Its strongest evidence concerns multi-entity video modeling; downstream control is evaluated separately in Procgen.
6.1 Read the original evidence
Table 1. Strong average prediction scores still depend on access to future-inferred latent actions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the arrows: larger PSNR and SSIM are better, while smaller LPIPS and FVD are better. Read each dataset block separately before the reported Average block. Recon is the pretrained tokenizer's reconstruction reference, not a competing dynamics predictor. The evaluation first extracts actions from ground-truth video and then rolls predictions autoregressively; Appendix C.1 specifies ten steps. The default data split is 80/10/10 by frame count, and comparisons share the pretrained encoder. Notice the World Model entries marked N/A for nuPlan: this physical-action baseline is evaluated only where action labels exist. e-resultse-protocole-horizone-data
What it supports. FLAM reports average PSNR 34.9 and FVD 419.6, versus AdaWorld's 24.7 and 749.5. On nuPlan, its PSNR 19.7 and FVD 1946 improve on AdaWorld's 18.1 and 2309, while remaining behind the reconstruction reference. Superiority is not universal: PlaySlot ties on MultiGrid and AdaWorld slightly wins Bigfish FVD.
Where the evidence stops. These metrics assess rollouts supplied with future-informed latent actions, not autonomous forecasts or driving behavior. Physical-action baselines also differ in available information: Procgen labels describe only the player, while inferred actions can encode other entities' changes. The table supplies no uncertainty estimates.
Figure 6. Per-factor action channels provide a visible interface for editing one agent's trajectory. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read time from left to right within each strip. Every pair repeats the original trajectory above an edited trajectory named by its controlled agent color. The original first frame supplies the starting image, which explains the blank leading space in edited rows. Compare the named triangle's orientation and location with its original counterpart, then check the other triangles at the same column. The procedure changes the selected entity's latent actions while retaining the other entities' original inferred actions. This is why the unchanged context has a concrete source: the original video provides actions for those other factors. e-controle-utilization
What it supports. The examples show alternative trajectories for the selected blue, green, purple and red agents while the remaining trajectories are visually preserved. This is qualitative evidence that factor-specific latent actions can localize edits. It complements the numerical alignment diagnostics by showing the interface through which a user can alter generated scenes.
Where the evidence stops. The preserved actors use future-informed original actions; they are not independently forecast from the starting image. Selected rollout strips provide no intervention success rate or measured spillover distribution. They demonstrate video manipulation, not physical action execution or a task-solving policy.
Table 4. Pseudo labels help most in the larger of the two tested label budgets. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each dataset has a 1k-label row and a 10k-label row. The displayed numbers are mean episodic returns, not percentages or success rates. The FLAM column uses the labeled subset to train an action decoder, then trains behavior cloning on a million demonstration frames with inferred action labels. Vanilla BC trains only on the labeled subset; policy architecture and training setup are otherwise shared. The random reference is one value per environment, spanning its two rows. Appendix C.3 specifies disjoint demonstration levels for training, validation and testing, with test levels unseen by the cloned policy. e-policye-policy-protocole-utilization
What it supports. With 10k labels, Bigfish return is 8.8 versus 3.6 for vanilla BC, and Starpilot is 9.1 versus 6.4. At 1k labels, FLAM reaches 1.8 in both environments; Starpilot's vanilla BC is slightly higher at 1.9. The evidence supports conditional label efficiency, not an improvement at every budget.
Where the evidence stops. The comparison combines inferred labels with access to many more training frames. It does not isolate FLAM against alternative pseudo-labeling models or report uncertainty. Experts were trained on all Procgen levels, so “unseen levels” refers to the behavior-cloning split, not necessarily the expert's training exposure.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Ten-step video prediction across five datasets Unseen videos from MultiGrid, Bigfish, Leaper, Starpilot and nuPlan; actions inferred from ground-truth frames before autoregressive prediction; default 80/10/10 split by frame count. | 34.9 / 0.890 / 0.051 / 419.6 Reported average PSNR ↑ / SSIM ↑ / LPIPS ↓ / FVD ↓ | AdaWorld: 24.7 / 0.862 / 0.096 / 749.5; Genie: 28.1 / 0.859 / 0.091 / 717.5. Best reported averages among compared predictors, not forecasting without future access. FLAM ties PlaySlot on MultiGrid; AdaWorld has slightly lower Bigfish FVD (63.3 versus 63.6). e-protocole-datae-horizone-results |
| nuPlan front-camera video prediction 224×224 front-facing images; K=16; same ground-truth-action-inference rollout protocol. | 19.7 / 0.577 / 0.229 / 1946 PSNR ↑ / SSIM ↑ / LPIPS ↓ / FVD ↓ | AdaWorld: 18.1 / 0.529 / 0.292 / 2309; tokenizer reconstruction: 21.7 / 0.650 / 0.163 / 1025. A real-video improvement with substantial residual visual error. It establishes neither driving-policy quality nor closed-loop driving safety. e-datae-protocole-horizone-results |
| MultiGrid factor–agent correspondence Unseen four-agent MultiGrid; K=4; linear position probes trained with permutation-invariant matching and evaluated on a separate validation partition. | 0.91 / 0.91 / 0.93 Disentanglement / completeness / informativeness ↑ | PlaySlot: 0.81 / 0.81 / 0.86; SlotFormer: 0.85 / 0.85 / 0.90. Supports factor–agent alignment under the paper's position-probe construction; it is not a general causal-identification guarantee. e-dcie-dci-protocol |
| Limited-label Procgen behavior cloning One million expert demonstration frames per environment; 1k or 10k action-labeled frames; pseudo-labeled full set; evaluation on disjoint, unseen levels. | 10k labels: Bigfish 8.8; Starpilot 9.1. At 1k: both 1.8. Mean episodic return ↑ | Vanilla BC at 10k: 3.6 / 6.4; at 1k: 1.0 / 1.9. Random: 0.9 / 1.1, respectively. Useful gains with 10k labels; Starpilot shows no improvement at 1k. No uncertainty or episode count accompanies Table 4. e-policye-policy-protocol |
| MultiGrid architecture ablations Original FLAM versus previous-slot initialization without factorizer temporal attention, and a variant coupling all future slots/actions. | Original: 56.5; 0.91 / 0.91 / 0.93. PSNR ↑; disentanglement / completeness / informativeness ↑ | Without temporal attention: 51.6; 0.33 / 0.23 / 0.20. Global-Coupled: 56.3; 0.79 / 1.00 / 0.58. Temporal consistency matters. Global coupling largely preserves prediction while weakening disentanglement and informativeness; completeness actually increases, so not every representation metric declines. e-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Increasing the number of agents exposes the difficulty of a single scene-level action. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each colored method across the shared entity counts 2, 4, 8 and 16. The upper panels reward high values; the lower panels reward low values. In particular, the FVD axis is logarithmic, so vertical gaps represent multiplicative differences. FLAM is blue, AdaWorld orange, Genie green, World Model red, PlaySlot purple and SlotFormer brown. The appendix says total action capacity is matched: its eight-agent example gives FLAM eight 32-dimensional actions and AdaWorld one 256-dimensional action. This comparison asks whether distributing a fixed total action representation across factors helps as simultaneous behaviors multiply. e-scalinge-probleme-horizon
What it supports. FLAM stays comparatively stable as the scene grows, whereas several baselines deteriorate; PlaySlot exhibits a particularly sharp collapse at sixteen entities. The result supports the paper's motivation for a factored action representation under this controlled MultiGrid scaling experiment. It does not establish the same scaling trend for crowded real-world driving video.
Where the evidence stops. Dimension matching does not establish equal optimization difficulty, bitrate or compute. Genie uses a codebook-based capacity rule in this sweep. Exact point values and uncertainties are not tabulated here; use the curves for trends and keep this experiment distinct from Table 1.
Table 7. Accurate images can coexist with less useful factor–agent alignment. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read D, C and I as disentanglement, completeness and informativeness. Appendix C.2 trains linear position probes with Hungarian matching because slot order is not a fixed agent identity. D measures whether a slot concentrates on one agent; C measures whether an agent concentrates on one slot; I is matched position-classification accuracy. Compare the no-temporal-attention column with original FLAM to test persistent bindings. Then compare Global-Coupled FLAM, which broadens IDM access to all future slots and FDM access to all actions. Keep the complete three-metric pattern: global coupling improves C while reducing D and I. e-ablatione-dci-protocol
What it supports. Original FLAM scores 0.91/0.91/0.93, versus 0.33/0.23/0.20 without temporal attention and 0.79/1.00/0.58 with global coupling. The companion Table 6 on the inspected page reports PSNR 56.5 for FLAM and 56.3 for global coupling. Almost unchanged image prediction can therefore conceal a loss of disentanglement and informativeness.
Where the evidence stops. This ablation does not show every DCI metric worsening: global completeness is 1.00. Removing temporal attention also changes how slots are initialized, so it tests that replacement design. These diagnostics concern four-agent MultiGrid position information, not universal object discovery.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify dataset-specific tokenizers and the pretrained visualization decoder as limits to a universal, higher-fidelity model. Diffusion and flow-matching decoders are suggestions, not evaluated improvements. e-limitations
The action-conditioned World Model baseline has only player actions in Procgen, whereas inferred actions can capture other entities' motion. In MultiGrid its action token is monolithic. Its lower scores therefore do not isolate an advantage over a fully observed, factored physical-action model. e-protocol
The β sweep supports stability from 10⁻⁵ through 10⁻³ on MultiGrid, but β=10⁻² gives PSNR 24.9 and FVD 479.9. The broad robustness wording must retain this failure. Observation-equivalent blocked movement and staying still also remain ambiguous latent actions. e-betae-action-ambiguity
7.2 Questions for discussion
- Would FLAM retain its advantage under episode-disjoint world-model splits and latent actions generated without future frames?
- Does improved factor disentanglement predict more localized interventions, beyond better position-probe scores?
8. Reproducibility audit
8.1 Requirements and known gaps
Recreate the stated image preprocessing and frame-count split, with separate level splits for policy evaluation. Table 5 gives IMPALA tokenizers for simulation, MAGVIT-v2 for nuPlan, and attention dimensions, heads and layers. Exact world-model corpus sizes, training duration, hardware, software versions and random-seed uncertainty are not provided; the frame-count rule alone does not establish episode-level separation. e-datae-confige-policy-protocol
A minimal proposed check is the four-agent temporal-attention/global-coupling ablation with fixed data, tokenizer and capacity, scoring both rollouts and matched position probes. Reporting only image metrics could miss the intended representation benefit. e-ablatione-dci-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does restricted action access produce more localized edits?
Reader-proposed experiment: train original FLAM and Global-Coupled FLAM on the same four-agent MultiGrid split, using the same tokenizer, total action dimension, optimizer budget and repeated seeds. Reproduce PSNR and all three matched-position DCI scores. Then replace one factor's inferred action while holding the current state and other factors' inferred actions fixed. Measure the selected agent's trajectory change and changes to the remaining agents, matching slots to agents before intervention. The discriminating outcome is similar PSNR but more spillover under global coupling. If DCI declines without increased spillover, the reported representation metrics would be weaker evidence for independently usable controls. e-ablatione-dci-protocole-controle-config
Check 2: Do pseudo-label improvements survive controlled data and decoder comparisons?
Reader-proposed experiment: reproduce the 1k/10k-label Bigfish and Starpilot protocol with fixed million-frame videos and disjoint BC levels. Compare labeled-only BC, FLAM pseudo labels, and an AdaWorld-based pseudo-label pipeline with the same decoder and BC architecture. Measure held-out true-action decoding accuracy and episodic returns across repeated seeds; optionally include full-label BC as an upper reference. Record which levels the expert saw. A FLAM-specific benefit requires better action recovery or returns than the matched alternative, especially at 10k labels. Gains shared equally by both pseudo-label methods would support video augmentation but would not isolate factorization as the cause. e-policye-policy-protocole-utilizatione-protocol
8.3 Reading coverage
Visual audit: The title/byline and version, method equations and Algorithm 1, evaluation protocols, limitations, Appendix A/B configuration, Appendix C diagnostics, every Figure 1–14 and Table 1–8 were visually inspected on these pages. All six final original crops were individually inspected; narrow assets were rendered at higher native PDF DPI. Architecture connections and frozen markers were checked against Eqs. (4)–(6), Algorithm 1 and the training text. Figure 3's broad caption is qualified by its frozen tokenizer markers. References-only page 10 was read as text. No external videos, code or separate supplements were inspected, and no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Preliminaries
- 4. Factored Latent Action Model (FLAM)
- 4.1. Pretrained Encoder
- 4.2. Factored Latent Action Model (FLAM)
- 4.3. Learned Latent Actions Utilization
- 5. Experiments
- 5.1. World Model Accuracy
- 5.2. FLAM State Representation
- 5.3. Latent Action Policy Learning
- 5.4. Ablation Studies
- 6. Conclusion and Limitation
- Impact Statement and Acknowledgments
- References
- A. Dataset Details
- B. Implementation Details
- C.1. World Model Accuracy
- C.2. FLAM State Representation
- C.3. Latent Action Policy Learning
- C.4. Ablation Studies
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Version scope: the inspected title page identifies arXiv:2602.16229v2, 25 May 2026, and prints an ICML 2026/PMLR 306 proceedings line. Title and all seven authors match the catalog. The catalog submission date is 18 February 2026; this report reads the supplied v2, not an independently recovered February edition. Earlier-version differences were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF figures and tables.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read. PDF pages 1–9 and 11–23 were visually inspected; the references-only page 10 was read as text.
- Code, external project videos and other linked resources were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/byline, affiliation footnote and arXiv margin
Exact title and seven-author byline match the catalog; Wang and Shi have equal-contribution marks. Affiliations are University of Texas at Austin and Sony AI. The margin states v2, 25 May 2026.
Go to primary source ↓e-problemPDF pp. 1–2, Introduction and Figure 1/caption
Monolithic actions encode joint multi-entity changes; FLAM uses a common action space across factors. Figure 1 contrasts d with d/K action dimensions.
Go to primary source ↓e-encoderPDF pp. 3–4, Section 4.1, Eq. (1), Figure 3
Image features are quantized by FSQ and decoded for reconstruction. Figure 3 shows tokenizer pretraining followed by frozen tokenizer modules around learned LAM components.
Go to primary source ↓e-factorizerPDF p. 4, Section 4.2, Eqs. (2)–(3)
Iterative slot attention is interleaved with causal temporal attention to each slot's history.
Go to primary source ↓e-dynamicsPDF p. 4, Section 4.2, Eqs. (4)–(5) and IDM/FDM attention definitions
Shared IDM uses all current slots and the target next slot; shared FDM uses all current slots and the target action. The action posterior is Gaussian with mean and standard-deviation projections, regularized toward a unit normal prior.
Go to primary source ↓e-objectivePDF p. 5, Algorithm 1, Aggregator and Eq. (6)
The aggregator queries predicted slots with current patch features. Factorizer, IDM, FDM and aggregator jointly minimize squared feature prediction error plus summed action KL penalties.
Go to primary source ↓e-utilizationPDF p. 5, Section 4.3 and Eq. (7)
Prior actions control generation. A supervised action decoder creates pseudo labels from video-inferred actions, followed by behavior cloning.
Go to primary source ↓e-protocolPDF p. 6, Implementations, Baselines and Section 5.1
Dataset-specific encoders are frozen and shared in comparisons; total action dimensions are matched. Test actions are inferred from ground-truth videos. Procgen physical actions cover only the player; MultiGrid baseline actions form a monolithic token.
Go to primary source ↓e-resultsPDF p. 7, Table 1, Average, MultiGrid, Bigfish and nuPlan rows
Reports the retained prediction metrics and tokenizer Recon reference. MultiGrid ties PlaySlot; Bigfish FVD is 63.6 for FLAM and 63.3 for AdaWorld. nuPlan has no World Model baseline.
Go to primary source ↓e-dciPDF p. 8, Table 2; p. 7, Section 5.2 Independent Entities
Four-agent DCI scores are FLAM 0.91/0.91/0.93, PlaySlot 0.81/0.81/0.86 and SlotFormer 0.85/0.85/0.90.
Go to primary source ↓e-dci-protocolPDF pp. 13–15, Appendix C.2, Independent Entities and Eq. (8)
Linear position probes use Hungarian slot–agent matching. Likelihood-normalized entropy defines D/C and optimally matched classification accuracy defines I, averaged on validation frames.
Go to primary source ↓e-policyPDF p. 8, Section 5.3 and Table 4
Policies use a 1M-frame expert set with 1k/10k labeled subsets. Table 4 reports Bigfish FLAM/BC 1.8/1.0 and 8.8/3.6; Starpilot 1.8/1.9 and 9.1/6.4, without uncertainty or evaluation episode count.
Go to primary source ↓e-policy-protocolPDF p. 15, Appendix C.3
Phasic Policy Gradient experts train for 25M environment steps on all levels; demonstration train/validation/test levels are disjoint. BC architecture and training setup are shared.
Go to primary source ↓e-ablationPDF p. 16, Tables 6–7 and Appendix C.4; p. 18, Figure 9
Removing factorizer temporal attention changes initialization to previous slots and worsens predictions and DCI. Global-Coupled uses all future slots/all actions, with PSNR 56.3 and D/C/I 0.79/1.00/0.58 versus 56.5 and 0.91/0.91/0.93.
Go to primary source ↓e-betaPDF p. 16, Table 8 and KL regularization paragraph
PSNR remains 56.5, 56.5 and 56.3 for β=10⁻⁵,10⁻⁴,10⁻³, then falls to 24.9 at 10⁻²; corresponding final FVD is 479.9.
Go to primary source ↓e-limitationsPDF p. 9, Limitation paragraph
The authors identify separate tokenizers and decoder expressiveness as limitations and propose shared pretrained tokenizers and more expressive decoders.
Go to primary source ↓e-dataPDF p. 11, Appendix A
Default splits are 80/10/10 by frame count with [0,1] normalization. MultiGrid uses 128×128 renderings of 8×8 grids; Procgen uses 224×224 with backgrounds disabled; nuPlan uses 224×224 front-camera frames.
Go to primary source ↓e-configPDF p. 11, Appendix B and Table 5
Lists tokenizer families, FSQ settings, AdamW learning rates and batches, attention configuration, factor counts and β. Does not specify hardware, software versions, training duration or full world-model corpus sizes.
Go to primary source ↓e-horizonPDF p. 12, Appendix C.1, Prediction Accuracy
States T=10 across datasets and actions inferred from ground-truth frames before autoregressive generation.
Go to primary source ↓e-scalingPDF p. 12, Figure 4 and Scaling with the number of entities
Sweeps 2,4,8,16 MultiGrid entities; matches action capacity, illustrating eight 32-dimensional FLAM actions versus one 256-dimensional AdaWorld action. FLAM's curves remain comparatively stable.
Go to primary source ↓e-controlPDF pp. 12–14, Controllable video generation and Figure 6/caption
Single-entity latent-action edits generate alternative MultiGrid trajectories; other agents use original inferred actions.
Go to primary source ↓e-action-ambiguityPDF p. 13, Figure 5/caption and explanatory paragraph
UMAP colors use action labels only for visualization; blocked forward movement can look identical to staying still and overlap in latent space.
Go to primary source ↓e-correlatedPDF p. 7, Correlated Entities; p. 13, Appendix C.2; p. 15, Figure 7
In a controlled four-agent dataset with two agents sharing actions, K=3 groups those agents into one factor.
Go to primary source ↓e-factor-countPDF p. 8, Table 3; p. 17, Figure 8/caption
With four independent agents, K=2 has PSNR 53.4; K=4,8,16 each report 56.5. Figure 8 illustrates separate entity assignments when enough slots exist.
Go to primary source ↓e-rolloutsPDF pp. 19–23, Figures 10–14 and captions
Qualitative multi-domain rollouts show both consistent bindings and failures. Figure 14 specifies nuPlan history w=5; Figure 13 notes that constant-speed entities can be predicted despite less consistent bindings.
Go to primary source ↓8.5 Primary sources
Factored Latent Action World Models ↗
PDF · 10,140 extracted words
Source fingerprint
9f9d7a7f51564cdd4c7eb54fee8ee3b8c1922aae5b4dd1307b3ef484c36e0af3