PAN: A World Model for General, Interactable, and Long-Horizon World Simulation
1. Paper overview
In one sentence: PAN combines language-conditioned latent prediction with causal video diffusion to support iterative simulation, while its evidence leaves physical validity and individual component benefits unresolved. architectureconditioningwindowinferenceplanningplanning-resultsflowwindow-details
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | Source description PAN combines pretrained vision-language representations, continuous next-state queries and a diffusion decoder into an action-conditioned simulator. architectureconditioning |
| A key reported result | Long-horizon Forecast: 53.6% / 64.1%. Transition Smoothness / Simulation Consistency (%). Multi-action rollouts; optical-flow smoothness and progressively penalized WorldScore-derived consistency; exact horizon unspecified. Best alternatives: Gen-3 smoothness 47.8%; KLING and Cosmos2 consistency 59.2%. Both aggregate scores favor PAN. No uncertainty or horizon-specific breakdown establishes the stability of the advantage. forecast-protocolforecast-resultsvisual-scope |
| Reading caution | Reader analysis No controlled ablation isolates latent conditioning, generated-history feedback, causal masking or noise augmentation. Cross-model comparisons cannot establish their individual contributions, and reconstruction supervision alone does not demonstrate physically correct dynamics. glpvisual-scope |
Core contributions
- Source description
PAN combines pretrained vision-language representations, continuous next-state queries and a diffusion decoder into an action-conditioned simulator. architectureconditioning
- Author claim
Causal Swin-DPM carries partially denoised history across chunks while masking future-action context, aiming to reduce discontinuities and error accumulation. window
- Reader analysis
Evaluation separates action fulfillment, long-horizon quality, atomic prediction and external-agent planning; these measure different capabilities. fidelity-protocolforecast-protocolatomic-protocolplanning
Figure 1. PAN links a long-history latent predictor to a decoder that maintains local video continuity. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the initial observation in the upper left and follow its downward path through the vision encoder. Blue tokens represent latent world states, while green tokens represent language actions. The yellow backbone predicts successive states using historical context; the pale blue decoder turns those states into video observations. Horizontal decoder connections represent local continuity, while long backbone arcs represent access to earlier context. The diagonal feedback paths should be read alongside Section 5: generated observations are re-encoded and appended to the next prediction context. This diagram summarizes information flow rather than specifying the exact tensor layout or training updates. architectureinference
What it supports. The separation lets PAN combine a compact state sequence with detailed video rendering. Crucially, simulation history includes both predicted latent states and the appearance of what the decoder actually generated. That is the source-defined feedback loop; it does not incorporate new measurements from a physical environment.
Where the evidence stops. The final state–decoder link is drawn bidirectionally, whereas Equation 11 specifies state-to-observation generation. Section 5 separately defines observation re-encoding for feedback. Use those equations for implementation; the schematic does not define an inverse-dynamics or action-output module.
2. Motivation
2.1 The problem and the proposed response
An interactive simulator must respond to changing actions while preserving local motion and longer-term scene history. PAN addresses this with Generative Latent Prediction (GLP): abstract state prediction coupled to observation-space supervision. problemglp
2.2 What this reading follows
PAN treats a simulated future as both a sequence of compact world states and a video that can be inspected. An autoregressive backbone uses visual history and natural-language actions to predict what comes next, while a diffusion decoder supplies detailed observations and feeds generated appearance back into the history. The central design question is how to preserve continuity when actions arrive one at a time and future details remain uncertain. This reading follows the state pathway, the decoder's separate conditioning streams, and its causal denoising window before examining the benchmark. It distinguishes simulated planning gains from physical execution and preserves the paper's unresolved noise-schedule convention. architectureconditioningwindowinferenceplanningplanning-resultsflowwindow-details
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational neural-world-simulator/video-backbone classification is supported. PAN predicts observations from supplied actions; an external VLM proposes and selects planning actions. There is no joint executable-action head or inverse-dynamics controller. The catalog's WAM architecture/paradigm/quadrant fields being not applicable is appropriate despite PAN's multi-module architecture. architectureconditioninginferenceplanning
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Make a latent prediction answer to observable data
Start with the GLP objective: the encoder maps an observation into a state, the predictor advances that state under an action, and the decoder reconstructs the next observation. The discrepancy is imposed on the generated observation through flow matching, rather than only comparing two feature vectors. The authors argue that this anchors transitions to realizable sensory changes. PAN implements this idea with Qwen vision-language components, learnable query embeddings and a Wan diffusion decoder. The important implementation boundary is the freeze policy: in Stage 2, the VLM remains frozen while queries and the decoder are trained. Thus the phrase joint training does not mean every module learns new weights. Nor does the observation-space loss by itself demonstrate that all generated transitions obey physical laws; that stronger interpretation needs independent tests. glparchitectureflowstage2visual-scope
Figure 2. The diffusion block has separate pathways for current action text and predicted world state. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this block from top to bottom. Hidden states first pass through self-attention, then feed two cross-attention streams. On the left, the action enters the existing text-conditioning path; Section 3.3.2 identifies its encoder as umT5. On the right, latent world-state features enter the copied cross-attention block. Its output passes through the explicitly marked zero-initialized linear projection before being added to the action-conditioned output. The combined representation continues into the feed-forward network. The surrounding repetition marker indicates this design is used across decoder blocks. The text also specifies an input projection for state features that the schematic does not show separately. conditioningvisual-scope
What it supports. PAN preserves a direct action-conditioning route while adding state-dependent information from the autoregressive backbone. The zero-initialized output projection is a stabilization device: initially, the added pathway contributes no output through that projection. Learning can subsequently make world-state features affect the generated observation.
Where the evidence stops. The two input routes create an opportunity to use history, but the diagram does not establish how much the trained decoder relies on each. The paper supplies no isolated ablation removing the state branch.
5.2 2. Separate simulation feedback from causal video decoding
At inference, the predicted latent state is only part of the next context. PAN also encodes the video it has just generated and appends that representation to its history. The next prediction can therefore use both the intended state and its visual realization. Locally, Causal Swin-DPM keeps partially noised chunks in a sliding window instead of restarting every segment from a single sharp image. Its block-causal mask prevents earlier content from seeing later context, and cached denoising outputs allow the next action to arrive after the current chunk finishes. These are two different continuity mechanisms: backbone history and decoder overlap. Neither supplies corrective measurements from the real world. Implementing the decoder also requires resolving the inconsistent apparent direction of the interpolation variable k across Equation 7 and the subsequent schedule description. inferencewindowwindow-detailsflow
Figure 3. Causal Swin-DPM carries partially denoised context forward while hiding future chunks. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the queue from right to left: fresh noise enters, partially denoised content remains in the window, and completed frames leave. The drawing labels the noise scale T, whereas Section 3.3.3 calls the total scale K; both show a half-scale separation. Green blocks in the mask permit attention within each chunk and from the later chunk to the earlier chunk. The white upper-right block prevents the earlier chunk from accessing the future one. Section 5.1 adds an operational detail absent from the picture: DiT outputs are cached at each denoising step so the current chunk can finish before the next action is supplied. windowwindow-detailsflowinferencevisual-scope
What it supports. The intended mechanism combines temporal overlap with causal interaction. Partially denoised history provides more transition context than a single final frame, while future-action masking avoids requiring an action that has not arrived. The authors argue this reduces accumulated artifacts, but cross-model results do not isolate that mechanism.
Where the evidence stops. Schedule ambiguity remains: Equation 7 makes larger k cleaner, but Section 3.3.3 assigns k+0.5 to the later, supposedly noisier chunk; Section 3.3.5 also calls k=0.055 mild noise. The mask is interpretable, but the exact noise convention requires clarification.
5.3 3. Ask what the planner actually gains from imagination
The planning experiment wraps the simulator in an external OpenAI-o3 agent. The agent proposes actions, inspects their predicted consequences and chooses the one that appears to advance the goal. Figure 4 makes that division of labor visible: the blue path is selected from gray alternatives, while the simulator renders the candidate outcomes. Reader interpretation: better-looking futures help only if their differences correctly rank the actions. A visually compelling but incorrect transfer of an object could otherwise mislead the search. The reported planning improvements are encouraging within the 15 Agibot scenarios and 46 Language Table cases, but the chart does not supply absolute success rates or clarify the improvement denominator. Treat this as evidence about a particular simulation-assisted decision process, with physical transfer still requiring a separate experiment. planningplanning-results
Figure 4. The external agent searches over simulated consequences to choose a language action. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the stated goal at the upper left: match yellow-can counts between the right blue tray and middle white tray. The initial observation is below it. Moving right, each column expands candidate language actions and displays their simulated consequences. Gray arrows identify alternatives that were explored; blue arrows mark the selected sequence, including grasping and releasing cans. The caption assigns proposal and selection to a VLM, while the world model supplies the imagined outcomes. Section 7.2 identifies the planning agent as OpenAI-o3. The final check mark denotes completion in this illustrated reasoning trajectory, not a separately reported hardware trial. planningplanning-results
What it supports. This example explains how a predictive model can assist planning without generating an executable policy itself. The agent compares imagined futures, selects an action and repeats from the chosen outcome. PAN therefore acts as an internal simulator within a larger decision process whose quality also depends on the external agent.
Where the evidence stops. This is one selected search trace. It does not disclose the complete action budget, candidate-selection prompts or calibration of predicted goal progress against real outcomes. Robot imagery alone does not establish that the pictured plan was executed.
5.4 Training and inference
During training
Stage 1 adapts the diffusion decoder; Qwen requires no additional adaptation, while Wan-VAE and the text encoder remain frozen. Five epochs use 960 H200 GPUs, BF16 AdamW, learning rate 10^-5, cosine decay, 5% warm-up and gradient clipping at 0.05. stage1
Stage 2 is called joint training, but explicitly freezes the VLM and updates only query embeddings and the diffusion decoder with flow matching. Training uses ground-truth history, limited to ten rounds, and stops after one epoch based on validation convergence. stage2architecture
Public videos are segmented, merged and filtered for motion, aesthetics, text and undesirable content. Dense VLM captions emphasize temporal change. The resulting corpus size and exact filtering thresholds are not enumerated. data
During inference
Encode the initial image, predict a state for the current action, render a chunk, and re-encode that generated observation. Concatenate it with predicted states and action history for the next step. Feedback closes within simulation; there are no fresh real-world observations. inference
Causal decoding caches DiT outputs at each denoising step and uses classifier-free guidance scale four. Eight-way sequence parallelism and modified SageAttention2++ accelerate inference, but absolute interactive latency is unreported. inferenceacceleration
5.5 Implementation flow
- Encode and predict
Qwen2.5-VL-7B-Instruct supplies the vision encoder and language backbone. Conversation-style turns alternate visual states and actions with 256 learnable assistant-query embeddings. The backbone produces 256 continuous state tokens; these are not motor commands. architecture
- Condition the decoder twice
Wan2.1-T2V-14B receives umT5 action features through text cross-attention. A separate world-state cross-attention stream adds its zero-initialized projected output to that pathway. Action text therefore reaches both predictor and renderer. architectureconditioning
- Carry a denoising window forward
The decoder window contains one conditioning frame and two ten-frame latent chunks. Earlier content is less noisy than later content; completed content exits and fresh noise enters. Chunk-wise masking blocks future context while retaining within-chunk attention. Conditioning-frame noise is intended to damp accumulated artifacts. windowwindow-details
6. Experiments & results
PAN turns an initial image and successive language actions into simulated video futures. A Qwen-based backbone predicts compact latent states; a Wan-based causal diffusion decoder renders them and returns generated visual history. Its strongest evidence concerns comparative video continuity and prediction; planning uses an external VLM and does not establish physical execution success.
The reviewed v3 PDF contains no quantitative table and no controlled component ablation or ablation figure. Figure 5 is the original numerical evidence, so the edition preserves its eight bar charts instead of inventing a table. Architecture and qualitative figures fill the remaining visual roles; none is labeled as an ablation. The cross-model results cannot isolate Causal Swin-DPM, state conditioning or noise augmentation. visual-scopeforecast-resultsconditioningwindow
6.1 Read the original evidence
Figure 5. The result chart separates video quality, action effects and planning assistance. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three rows as distinct evaluations. The top-left panel reports atomic next-observation accuracy on Agibot after domain fine-tuning; the other top panels show improvement over the same VLM agent, not absolute success. The middle row separates environment interventions, agent behavior and their overall action-fulfillment score, using VLM judgment of generated rollouts. The bottom row measures optical-flow transition smoothness and WorldScore-derived simulation consistency with later-step penalties. Blue bars are PAN; gray bars are named baselines. Comparisons belong within a panel: the percentages do not share one definition. The planning sets contain 15 Agibot scenarios and 46 Language Table cases, whereas the other test counts and precise forecast horizon are unspecified. atomic-protocolatomic-resultsplanningplanning-resultsfidelity-protocolfidelity-resultsforecast-protocolforecast-resultsvisual-scope
What it supports. PAN reaches 56.1% atomic prediction accuracy versus Cosmos2's 31.1%, and leads the forecast panels at 53.6% smoothness and 64.1% consistency. Its 58.6% overall action fulfillment exceeds the plotted open-source alternatives but trails MiniMax's 62.0%. Planning panels report +26.7 and +23.4 improvements, with their percent-versus-percentage-point convention unresolved.
Where the evidence stops. No error bars or component ablations are shown. Human scoring for generated atomic predictions differs from latent-similarity scoring for V-JEPA 2. These comparisons support the reported benchmark ranking, not a proof of causal fidelity, physical execution or uniform superiority across all panels.
Section 8.1, unnumbered driving sequence. A selected rollout changes environments while retaining a recognizable vehicle and viewpoint. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row left to right, then continue to the next instruction. The car first exits a mountain tunnel, then travels through a lit tunnel, a flower valley, snow, a volcanic landscape, an underground passage and a futuristic city. The text above each row is the source's instruction for that segment. Follow the vehicle silhouette and rear-view framing across environmental changes, and inspect the boundary between the last frame of one row and the first of the next. These are sampled stills from a qualitative simulation, so they can illustrate visual persistence and action-conditioned scene change but cannot directly reveal the smoothness of every intervening frame. qualitativeforecast-results
What it supports. The example makes PAN's interaction format concrete: successive instructions can change the simulated setting while a recognizable red car remains central. It illustrates the breadth claimed for the simulator and complements the aggregate forecast scores. It is one selected trajectory rather than a measured distribution of long-horizon successes.
Where the evidence stops. The page supplies no frame rate, full duration, physical ground truth or failure denominator. Changing from snow to lava through tunnels is an authored imagined scenario; it cannot establish accurate real driving dynamics or safe vehicle control.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Action Simulation Fidelity Initial images and GPT-4o-proposed actions; VLM-judged agent/environment rollouts; test count and split unspecified. | Agent 70.3%; environment 47.0%; overall 58.6%. Action fulfillment score (%) | Overall: Cosmos2 51.0%, WAN 2.2 50.2%, MiniMax 62.0%. PAN leads the plotted open-source models; MiniMax scores higher overall. These judge scores are not physical-control accuracy. fidelity-protocolfidelity-results |
| Long-horizon Forecast Multi-action rollouts; optical-flow smoothness and progressively penalized WorldScore-derived consistency; exact horizon unspecified. | 53.6% / 64.1%. Transition Smoothness / Simulation Consistency (%) | Best alternatives: Gen-3 smoothness 47.8%; KLING and Cosmos2 consistency 59.2%. Both aggregate scores favor PAN. No uncertainty or horizon-specific breakdown establishes the stability of the advantage. forecast-protocolforecast-resultsvisual-scope |
| Step-Wise Simulation Agibot next-observation selection with three distractors; all models domain-fine-tuned; human scoring for generators, latent similarity for V-JEPA 2. | 56.1%. Prediction accuracy (%) | Cosmos2 31.1%; V-JEPA 2 15.5%; Cosmos1 7.8%. Strongest plotted atomic score, under differing output-assessment procedures; no executed robot policy is measured. atomic-protocolatomic-results |
| Open-Ended Simulation and Planning Same OpenAI-o3 agent; 15 Agibot scenarios; simulated search and blinded human trajectory assessment. | +26.7, plotted as improvement (%); absolute success rate not supplied. Reported improvement over VLM-only agent (%) | Cosmos1 +6.7; Cosmos2 0.0; V-JEPA 2 -6.7. Supports usefulness for this protocol. Relative percent versus percentage-point improvement is not explicitly defined; neither is silently inferred. planningplanning-results |
| Structured Simulation and Planning Same agent–simulator loop on 46 Language Table cases, with minimally rearranged goal observations. | +23.4, plotted as improvement (%); absolute success rate not supplied. Reported improvement over VLM-only agent (%) | Cosmos1 -2.1; Cosmos2 +4.3; V-JEPA 2 +10.6. PAN yields the largest reported improvement; the denominator convention and physical transfer remain unresolved. planningplanning-results |
6.3 Ablations and diagnostic examples
No ablation or diagnostic visual is included in this edition. The source-specific visual limitations are recorded above.
7. Analysis & limitations
7.1 What the evidence leaves open
No controlled ablation isolates latent conditioning, generated-history feedback, causal masking or noise augmentation. Cross-model comparisons cannot establish their individual contributions, and reconstruction supervision alone does not demonstrate physically correct dynamics. glpvisual-scope
Evaluation leaves prompts, several sample counts, exact horizons, metric weights and uncertainty unspecified. Different scoring procedures for generative and embedding models further limit direct interpretation. fidelity-protocolforecast-protocolatomic-protocolplanning-results
Kitchen, driving, fantasy-world, rover and hazard sequences demonstrate range but do not measure failure frequency, calibrated physics or real execution. No dedicated limitations section is supplied. qualitativevisual-scope
7.2 Questions for discussion
- Which benefit survives a matched ablation of latent-state conditioning with decoder and data fixed?
- Does simulated goal progress predict independent environment success under a fixed planning budget?
- How much consistency comes from suppressing motion rather than preserving correct action effects?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the Qwen/Wan initializations, caption pipeline and stage-specific freeze policy. HSDP, checkpointing and sequence parallelism are described, but corpus size, batch size, wall time and complete evaluation recipes are missing. architecturestage1stage2datafidelity-protocolforecast-protocol
Preserve the 21-latent/81-real-frame window and VAE padding of 0–122 prior frames. First resolve the schedule: assigning k+0.5 to the later chunk makes it cleaner under Equation 7; describing k=0.055 as slight noise also suggests a different convention. Neither statement can be silently repaired. flowwindowwindow-details
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate world-state conditioning under matched training
Reader-proposed check, not performed: compare the full decoder with an otherwise matched decoder whose added world-state cross-attention contribution is disabled. Train both on the same fixed video/action subset with the same initialization, optimizer schedule, updates, causal window and direct umT5 action pathway. Use held-out action scripts with matched sampling seeds and report action fulfillment, transition smoothness and late-step consistency separately. Repeat across seeds and report uncertainty. If the state-conditioned variant does not improve history-dependent outcomes, the claimed benefit of the latent backbone is weakened even if visual quality remains high. Preserve the VLM freeze policy in both arms. conditioningstage1stage2fidelity-protocolforecast-protocolvisual-scope
Check 2: Test whether predicted goal progress survives independent execution
Reader-proposed check, not performed: on a fixed set of Language Table-style tasks, compare the same planning agent with and without PAN under identical candidate-action and total decision budgets. Use identical initial states and goals, record each predicted outcome, then evaluate selected actions in an independently observed environment with a fixed execution interface. Score actual binary goal completion as well as predicted completion, and report raw counts, absolute rates and percentage-point differences. A high simulated success rate accompanied by unchanged or lower actual success would falsify the proposed transfer benefit. Keep this proposed execution test distinct from the paper's reported imagined trajectories. planningplanning-resultsatomic-protocol
8.3 Reading coverage
Visual audit: PDF pages 1–27 were rendered and actually viewed, including title/byline, contributor roster, every method/training/inference/evaluation page and all qualitative pages. Figures 1–5 and the six final crops were visually inspected; the architecture crop was expanded to retain complete action labels and viewed again. Figure 1's final bidirectional state–decoder link is interpreted using Section 5 rather than as inverse dynamics. Figure 2's branch merge and zero-initialized projection and Figure 3's mask were checked against their text. The unresolved k/noise convention is disclosed. References on pages 28–33 were read in text only. Full-motion videos, code and separate supplements were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Overview and Section 1: Introduction
- Sections 2.1–2.3: GLP and parameterization
- Sections 3.1–3.3.5: encoder, backbone, diffusion, causal window and VAE details
- Sections 4.1–4.2: both training stages
- Sections 5–5.2: inference and acceleration
- Sections 6.1–6.3: segmentation, filtering and captions
- Sections 7.1–7.3: baselines, protocols and results
- Sections 8.1–8.3: all qualitative sequences
- Sections 9–10: related work and conclusion
- Appendix A.1–A.2: contributors
- References, pp. 28–33
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Pages with little extracted text require visual inspection: 21, 22, 24
- All nine text chunks were read. PDF pages 1–27 were visually inspected, addressing the extraction cautions, including low-text pages 21, 22 and 24. References were read as text; cited works were not opened.
- Separate supplements were not supplied or verified. Code, weights and full-motion demonstration videos were not inspected; experiments were not reproduced.
- Identity/revision note: the observed version is arXiv:2511.09057v3, dated 15 November 2025; the IFM report date is 14 November 2025. The title matches the catalog, and the team byline plus all 33 named contributors were verified on pages 1 and 27. Earlier revisions were not compared. The manifest identifies the retained HTML as the same v3; that HTML was not re-read.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title/byline and arXiv margin; p. 27, Appendix A.1–A.2
Exact catalog title; arXiv:2511.09057v3 dated 15 November 2025; IFM Technical Report dated 14 November 2025. PAN Team, Institute of Foundation Models is the byline, with Mohamed bin Zayed University of Artificial Intelligence. All 33 named catalog contributors appear in Appendix A, grouped by role.
Go to primary source ↓problemPDF pp. 1–3, Overview and Section 1
PAN targets history- and language-action-conditioned future simulation, combining latent dynamics with observable video generation across domains.
Go to primary source ↓glpPDF pp. 3–4, Sections 2.1–2.2, Eqs. (1)–(6); p. 11, Section 4.2, Eq. (9)
Encoder h, predictor f and decoder g define generative latent prediction. The discrepancy supervises generated observations. The authors contrast this with latent matching and argue it avoids collapse and ungrounded transitions.
Go to primary source ↓architecturePDF p. 5, Figure 1 and Section 2.3; pp. 6–7, Sections 3.1–3.2.2
Qwen2.5-VL-7B-Instruct supplies the vision encoder and autoregressive backbone; Wan2.1-T2V-14B supplies the decoder. The backbone uses 256 learnable query embeddings and outputs 256 continuous next-state tokens. Training uses ground-truth states in conversational turns.
Go to primary source ↓flowPDF p. 8, Section 3.3.1, Eqs. (7)–(8)
The clean VAE latent x_1 and Gaussian noise x_0 are interpolated as x_k=k x_1+(1-k)x_0; velocity is x_1-x_0. Training samples from 1,000 discrete denoising values with a shifted schedule.
Go to primary source ↓conditioningPDF p. 8, Figure 2 and Section 3.3.2
umT5 action features use the original text cross-attention. Projected world-state tokens use an added cross-attention stream; its zero-initialized output projection is summed with the action-conditioned stream before the feed-forward network.
Go to primary source ↓windowPDF p. 9, Figure 3 and Section 3.3.3; p. 10, Section 3.3.3 continuation
The earlier chunk starts half-noised and the later chunk fully noised; the earlier chunk is dequeued after half the denoising process and fresh noise is enqueued. Green blocks permit within-chunk and later-to-earlier attention; white blocks mask future context. Figure 3 uses total noise scale T, whereas the prose uses K.
Go to primary source ↓window-detailsPDF p. 10, Sections 3.3.3–3.3.5
Training assigns k in [0,0.5] to the first chunk and k+0.5 to the second. The 21-latent-frame window corresponds to 81 real frames and contains one conditioning frame plus two chunks of ten. VAE encoding uses 0–122 preceding padding frames, later discarded. Conditioning noise is described as small at k=0.055; no loss is applied to that frame.
Go to primary source ↓stage1PDF p. 11, Section 4.1
Stage 1 adapts the decoder to Causal Swin-DPM; Qwen requires no adaptation, and the Wan VAE and text encoder are frozen. Training uses BF16, AdamW, learning rate 10^-5, cosine decay, 5% warm-up, gradient norm cap 0.05, five epochs and 960 NVIDIA H200 GPUs. HSDP, checkpointing, FlashAttention-3 and FlexAttention are described.
Go to primary source ↓stage2PDF pp. 11–12, Section 4.2
The explicit implementation freezes the VLM and trains only query embeddings and the diffusion decoder, using at most ten history rounds. BF16 AdamW, learning rate 10^-5, cosine decay and 5% warm-up are reused; early stopping occurs after one epoch rather than five. Intra-node sequence-parallel group size is four.
Go to primary source ↓inferencePDF p. 12, Section 5, Eqs. (10)–(11); pp. 12–13, Section 5.1
The backbone receives initial state, actions, predicted states and re-encoded generated observations. The decoder receives predicted state and previous observation. Feedback uses generated history rather than new ground truth. DiT outputs are cached at every denoising step; CFG scale is four.
Go to primary source ↓accelerationPDF p. 13, Section 5.2
Inference uses sequence-parallel group size eight with uneven head sharding. Modified SageAttention2++ gives reported acceleration of 30.3% over FlexAttention/FlashAttention3; absolute per-chunk latency is not reported.
Go to primary source ↓dataPDF pp. 13–15, Sections 6.1–6.3
Public videos are segmented, merged and filtered using motion/appearance rules, aesthetic/text detectors and a custom VLM. Dense VLM captions emphasize events and changes as action inputs. Corpus size, exact constituent sources, filter thresholds and a complete captioning recipe are not enumerated.
Go to primary source ↓fidelity-protocolPDF pp. 16–17, Section 7.2, Action Simulation Fidelity
GPT-4o proposes feasible action sequences from images. A VLM judges generated rollouts for faithfulness and precision, separating agent behavior from environment interventions. Exact test counts, splits and judge prompts are absent.
Go to primary source ↓forecast-protocolPDF p. 17, Section 7.2, Long-horizon Forecast
Transition Smoothness uses the inverse exponential of optical-flow acceleration magnitude. Simulation Consistency uses WorldScore-derived measures and increasing late-step penalties. Exact horizons, weights and full metric configuration are not given.
Go to primary source ↓atomic-protocolPDF p. 17, Section 7.2, Step-Wise Simulation
Agibot instances supply a correct next observation and three distractors. All models are fine-tuned on Agibot; V-JEPA 2 gains UMT5 language conditioning. Generative outputs are human-assessed; embedding predictions use latent similarity.
Go to primary source ↓planningPDF p. 18, Figure 4/caption and Section 7.2 planning settings
OpenAI-o3 proposes actions, the world model simulates alternatives, and the agent selects predicted goal progress. Fifteen Agibot scenarios use blinded human assessment; 46 Language Table cases use minimally rearranged goals. Goal completion is binary over a trajectory. Blue arrows show the selected can-transfer path; gray arrows show alternatives.
Go to primary source ↓fidelity-resultsPDF p. 19, Figure 5 middle row and Section 7.3
PAN scores 70.3% agent, 47.0% environment and 58.6% overall action fulfillment. Cosmos2 overall is 51.0%, WAN 2.2 is 50.2%; MiniMax leads overall at 62.0% (72.3% agent, 51.7% environment).
Go to primary source ↓forecast-resultsPDF p. 19, Figure 5 bottom row; p. 20, Section 7.3
PAN scores 53.6% smoothness and 64.1% consistency. The best plotted alternatives are Gen-3 at 47.8% smoothness and KLING/Cosmos2 at 59.2% consistency.
Go to primary source ↓atomic-resultsPDF p. 19, Figure 5 top-left; p. 20, Section 7.3
Step-Wise Simulation accuracy is PAN 56.1%, Cosmos2 31.1%, V-JEPA 2 15.5%, Cosmos1 7.8%.
Go to primary source ↓planning-resultsPDF p. 19, Figure 5 top-middle/top-right; p. 20, Section 7.3
PAN improvement over the VLM agent is +26.7 open-ended and +23.4 structured, on an axis labeled percent. Cosmos1/Cosmos2/V-JEPA 2 score +6.7/0.0/-6.7 and -2.1/+4.3/+10.6 respectively. Absolute success rates and an explicit relative-percent-versus-percentage-point formula are absent.
Go to primary source ↓qualitativePDF pp. 20–25, Sections 8.1–8.3 and unnumbered sequences
Selected simulations show kitchen manipulation, a red-car journey, alternative worlds behind a door, rover actions and rare road hazards, without failure-rate or quantitative physical-validity tests.
Go to primary source ↓visual-scopePDF pp. 15–20, Section 7 and Figures 4–5; pp. 20–26, Sections 8–10; p. 27, Appendix A
Empirical reporting consists of cross-model bars and qualitative examples. No quantitative table, controlled component ablation, uncertainty interval, dedicated limitations section or technical appendix is supplied; Appendix A lists contributors.
Go to primary source ↓8.5 Primary sources
PAN: A World Model for General, Interactable, and Long-Horizon World Simulation ↗
PDF · 13,524 extracted words
Source fingerprint
81201c0b7046011620abb614d78e6b2ecaa02e1e116ecf374dd98a10a907d16f