WorldAgen: Unified State-Action Prediction with Test-Time World Model Training
1. Paper overview
In one sentence: WorldAgen adapts a shared action-and-observation predictor through exploratory observation supervision, gaining simulated task success at the cost of target-environment interaction and additional training. e02e03e08e12e13e14e15
| At a glance | What to know |
|---|---|
| Research problem | Source description A policy trained on static demonstrations may encounter unfamiliar scenes or dynamics at deployment. The paper asks whether self-supervised prediction of newly observed transitions can improve task execution without new task-specific action labels. Exploration still requires interaction with the target environment. e02e08 |
| Core mechanism | |
| A key reported result | LIBERO-10 long-horizon manipulation: WorldAgen-TTT: 79.0%. Average task success (%). Ten downstream tasks in the LIBERO-100 protocol; LIBERO-90 is described as pretraining data. TTT is performed separately in each test scene. WorldAgen: 75.5%; Seer: 78.7%. The gain is 3.5 percentage points over the unadapted model, but only 0.3 over Seer. Microwave success rises 35% to 60%, while mug-and-pudding falls 60% to 50%; gains are not universal. e11e13e18 |
| Reading caution | Reader analysis The experiment called Simulator to Real World only adds Gaussian observation noise with standard deviation 0.1. It provides corruption robustness evidence, not physical deployment. The stated single-architecture limitation is qualified by an appendix GPT2/Qwen3 comparison; neither establishes broad model-size scaling. e16e21 |
Core contributions
- Source description
LoRA adaptation uses only observation loss while freezing the policy head; action behavior can nevertheless change through the updated shared representation. e08
Figure 1(a). One shared representation supports action prediction and observation prediction. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left: blue tokens represent the instruction, yellow tokens observations, peach tokens robot state, and green tokens actions. Gray placeholders mark outputs to be predicted. These modalities enter the single central WorldAgen block. On the right, the upper branch combines instruction, observations and state to predict an action chunk. The lower branch predicts future observations using the current observations and predicted actions. The lower cartoon omits proprioception and history; both are included in the formal world-model conditional on page 3. The architecture therefore shares a backbone while retaining different prediction targets and intended conditioning. e03e07e08
What it supports. The shared backbone gives observation supervision a route to affect action prediction. During TTT, the paper freezes the policy head and adapts backbone LoRA parameters using observation loss. A frozen head can produce different actions when its input representation changes; that is the architectural rationale, not a measured decomposition of the performance gain.
Where the evidence stops. The diagram is not a planning or action-execution trace. The formal world conditional excludes instruction, while the two-step inference prose includes it. The report follows the formal conditional and preserves this unresolved inconsistency.
2. Motivation
2.1 The problem and the proposed response
A policy trained on static demonstrations may encounter unfamiliar scenes or dynamics at deployment. The paper asks whether self-supervised prediction of newly observed transitions can improve task execution without new task-specific action labels. Exploration still requires interaction with the target environment. e02e08
2.2 What this reading follows
WorldAgen asks whether a robot policy can benefit from learning what its actions actually do in a new scene. A shared Transformer predicts both actions and future observations; after exploration, observation reconstruction supplies the adaptation signal while the policy head stays frozen. This makes the shared representation the proposed bridge between world learning and better control. The figures below explain that bridge, the attention restrictions intended to support it, and the measured gains on CALVIN and LIBERO. Read the ablations separately from the headline comparison: the source contains conflicting baseline values, and larger adaptation budgets do not consistently help. e02e03e08e12e13e14e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded catalog is unassigned, so there is no substantive label to confirm. Architectural evidence supports a One Model assessment: action and future-observation heads share a Transformer. The mechanism is joint state-action prediction with action-first factorization and observation-based TTT, not inverse dynamics or a separate planner. This is a reader assessment, not a catalog update. e03e07e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Predict actions first, then predict their consequences
Begin with a trajectory unit containing a task instruction, observations and proprioception. The encoders place these modalities into a common token space. Action and observation chunk lengths can differ: the main CALVIN configuration retains one observation per chunk while grouping five actions. The policy predicts the action chunk first; the world model then predicts a future observation chunk conditioned on those actions. During offline training, ground-truth actions replace predicted ones through teacher forcing. During deployment, executing predicted actions advances the environment and supplies subsequent observations. This ordering distinguishes the model from a method that first imagines a target image and then infers an action between images. The paper specifies neither search over candidate actions nor a reward-scored rollout planner. Its chunk-sampling formula and diagram disagree, so exact preprocessing needs clarification. e03e04e07e09e10
Figure 3. The mask is intended to prevent action-target leakage and direct instruction conditioning of observation prediction. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the repeated token order shown on the left to interpret the two trajectory-unit blocks; the source does not separately label columns. White cells permit attention, pale green cells mark local restrictions, and blue cells mark global restrictions. In the action-placeholder row, the current action position is locally masked, implementing the body text’s rule against reading the target action. In the observation-placeholder row, the instruction position is globally masked. Compare these cells with the prose on page 4: the Figure 3 caption mistakenly calls the placeholder the action chunk and the action chunk the placeholder, reversing their names. e03e05
What it supports. The grid explains the intended separation between task-conditioned policy learning and task-agnostic observation prediction inside one backbone. This is an architectural proposal about information access. The paper does not present a dedicated mask ablation or an invariance test demonstrating that the implemented multilayer network satisfies every intended restriction.
Where the evidence stops. The caption’s reversed labels remain in the source; the crop is unchanged. Directly blocked instruction attention alone does not prove complete independence through intermediate token representations. That question motivates a proposed language-perturbation check.
5.2 Understand why a frozen policy head can still adapt
The TTT intervention begins with target-scene interaction, not a labeled demonstration of the evaluation task. CALVIN exploration uses instructions from the training set; collected trajectories then receive the generic no-lang label. The observation objective trains the system to account for the recorded transitions. The paper describes updating LoRA parameters in the shared backbone while freezing the policy head. Reader interpretation: the head’s weights may stay fixed while its input features change, allowing action outputs to change as well. This explains a plausible transfer route from observation learning to control without claiming it has been causally isolated. The source complicates implementation by defining the displayed update parameters as world-head parameters. Likewise, the mask’s intended task independence needs to be checked against the actual token computation, not inferred from the shared-loss story. e03e05e08e18
5.3 Separate three experimental questions
The first question is whether joint observation prediction helps the starting policy; Table 3 addresses it by removing image-prediction tokens. The second is whether observation-based TTT helps an already trained WorldAgen model; the paired bottom rows in Tables 1 and 2 address that question. The third is how to constrain adaptation: the data-budget sweep and the LoRA/full-fine-tuning comparison address selected budgets and update methods. Reader interpretation: none of these comparisons alone establishes that better physical-dynamics modeling caused the action improvement. Improved visual features are another possible route, and the paper supplies no direct dynamics-accuracy decomposition. The noise experiment remains simulated, and missing uncertainty matters particularly for the small advantage over Seer. A useful reproduction should measure both prediction error and executed-task success under controlled exploration. e12e13e14e15e16e19
5.4 Training and inference
During training
Offline training jointly optimizes action and observation prediction with ground-truth actions supplied by teacher forcing. The stated 370M-parameter model has 120M trainable parameters, 24 layers and 12 attention heads. Pretraining reportedly takes about 60 hours for CALVIN and five hours for LIBERO on four H100 GPUs. e06e10
For TTT, instruction-guided exploratory trajectories are relabeled no-lang and LoRA updates use observation loss only. The text specifies shared-backbone adaptation with a frozen policy head, but the update equation calls the updated parameters world-head parameters; the exact trainable set remains ambiguous. e08
During inference
The agent fills a zero action placeholder, then predicted actions condition the world-model step. Executing actions produces real simulator transitions for subsequent units and adaptation. No candidate-action search, reward ranking or inverse-dynamics action extraction is specified. The formal world conditional excludes language, although the inference prose includes it. e03e07
5.5 Implementation flow
- Encode multimodal trajectories
MAE-pretrained ViT-B and a Perceiver Resampler encode/compress visual tokens; CLIP ViT-B/32 encodes text. Separate MLPs process arm and gripper components. A Qwen3-based Transformer shares representations across modalities; the image decoder predicts patches and the action decoder produces seven values, thresholding the gripper at 0.5. e09e10
- Separate observation and action horizons
Each unit contains instruction, observation/state chunks and action/observation placeholders. The first unit has n actions; later units have m actions and n sampled observations/states. CALVIN uses n=1,m=5 and trajectory length 16; LIBERO uses n=1,m=3 and length 7. Sampling definitions differ across formula, diagram and algorithm. e04e10
- Restrict information access
The intended local mask prevents the action predictor from seeing its current action targets. The global mask blocks direct instruction access for observation prediction. Figure 3 reverses the action/placeholder names in its caption; the body and matrix provide the clearer interpretation, but do not establish implementation-level isolation. e05
6. Experiments & results
WorldAgen shares a Transformer between action prediction and future-observation prediction, then adapts the shared representation using exploratory target-environment transitions. Simulated manipulation improves after observation-only test-time training (TTT), while implementation ambiguities and unreported uncertainty limit the strength of the generalization claim.
6.1 Read the original evidence
Table 1. CALVIN improves after adaptation, with a small increase in average completed sequence length. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read T1 through T5 as success rates for completing progressively longer sequences, expressed in percent. The final column instead measures average sequence length, so its entries must not be read as percentages. Compare the two bottom rows first to see the change associated with TTT in WorldAgen. Then compare with Seer-Large and the other baselines, keeping in mind that TTT adds target-environment exploration and optimization. Use the actual values rather than boldface to determine column leaders: Seer-Large’s T4 value of 68.9 exceeds WorldAgen-TTT’s 68.7. The appendix explains reuse of adaptation from the first sample’s scenario. e11e12e18
What it supports. WorldAgen rises from 3.87 to 3.93 average sequence length, a table-implied gain of 0.06. Five-task success rises from 59.1% to 60.5%, or 1.4 percentage points. The adapted model has the highest average sequence length among the listed methods, while it does not lead every success column.
Where the evidence stops. The paper supplies no confidence intervals or evaluation trial count here, and does not specify its chosen CALVIN environment split. The small differences should be treated as reported point estimates, with target-scene adaptation included in the setting.
Table 2. The average LIBERO gain hides both strong improvements and an individual-task regression. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The table wraps horizontally into two stacked blocks. Keep the method order from the top block when reading the unlabeled rows below: MT-ACT, MVP, MPI, OpenVLA, Seer, WorldAgen and WorldAgen-TTT. The average column summarizes the ten tasks, rather than only the four tasks alongside it. First compare the last two rows, then follow the same task columns to inspect variation. For example, the final microwave column improves after TTT, whereas the lower block’s mug-and-pudding column declines. Appendix benchmark and adaptation details identify LIBERO-10/Long as the downstream suite and describe separate adaptation in each test scene. e11e13e18
What it supports. Average success increases from 75.5% to 79.0%, a gain of 3.5 percentage points. This is 0.3 points above Seer’s reported 78.7%. Microwave success rises from 35% to 60%, but mug-and-pudding success falls from 60% to 50%. The aggregate improvement therefore does not imply uniformly beneficial adaptation.
Where the evidence stops. The comparison includes extra scene interaction for TTT, and no uncertainty accompanies the reported means. Its narrow lead over Seer does not establish statistical superiority. Preserve the lower block’s row alignment to avoid attributing a task score to the wrong method.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN sequential manipulation Simulated sequences of up to five language-conditioned tasks; target-scenario TTT. Exact training/test environment split and evaluation trial count are not specified. | WorldAgen-TTT: 3.93 and 60.5%. Average sequence length (higher is better); five-task success (%) | WorldAgen: 3.87 and 59.1%; Seer-Large: 3.83 and 60.3%. TTT adds 0.06 completed tasks on average and 1.4 percentage points at five tasks. Seer-Large remains higher at T4 (68.9% versus 68.7%); no uncertainty is supplied. e11e12e18 |
| LIBERO-10 long-horizon manipulation Ten downstream tasks in the LIBERO-100 protocol; LIBERO-90 is described as pretraining data. TTT is performed separately in each test scene. | WorldAgen-TTT: 79.0%. Average task success (%) | WorldAgen: 75.5%; Seer: 78.7%. The gain is 3.5 percentage points over the unadapted model, but only 0.3 over Seer. Microwave success rises 35% to 60%, while mug-and-pudding falls 60% to 50%; gains are not universal. e11e13e18 |
| Contribution of image prediction Models with versus without image-prediction tokens, before TTT. | With world modeling: 3.87; 75.5% (Table 3). CALVIN average sequence length; LIBERO success (%) | Without: 2.96; 46.5%. The table supports a strong association with joint image prediction. The adjacent paragraph instead says 78.0% on LIBERO; retain the table value and do not merge the conflicting figures. Token removal does not isolate representation learning from every architectural change. e14 |
| CALVIN TTT data-budget ablation Table 5 defines data volume as samples multiplied by repetitions. | 6/90/204/340 samples: 3.871/3.922/3.928/3.917. Average sequence length | The largest reported budget is below the 204-sample result. Benefits are nonmonotonic. Overfitting is an author interpretation, not a directly measured cause; independent sample diversity and repetitions are not separated. e15 |
| CALVIN LoRA versus full fine-tuning Matched comparison using Adam, learning rate 0.0005 and weight decay 0.01; LoRA rank 128. | LoRA: 3.93; full fine-tuning: 3.85. Average sequence length | Unadapted model: 3.87. This configuration favors LoRA; the comparison uses different optimizer settings from the main TTT recipe and does not establish general superiority over tuned full fine-tuning. e18e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Joint image prediction has a larger reported association with baseline performance than the subsequent TTT increment. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each cross/check pair within one dataset. The intervention removes or includes image-prediction tokens to test the contribution of world modeling to policy learning. Although the column is generically headed average success, CALVIN’s entries are average sequence lengths, while LIBERO’s entries are success percentages. This comparison concerns the base joint model, not the additional test-time adaptation step shown in the preceding result tables. It therefore addresses whether learning to predict observations helps the starting policy. To understand the intervention, connect this table to the shared backbone and joint action-plus-observation objective rather than treating the check mark as a separate planning module. e03e06e14
What it supports. Using the table entries, image prediction is associated with an increase from 2.96 to 3.87 on CALVIN and from 46.5% to 75.5% on LIBERO. Those imply gains of 0.91 sequence-length units and 29.0 percentage points. The results support the usefulness of joint prediction under the reported setup.
Where the evidence stops. The adjacent paragraph reports LIBERO 78.0% and a 67.7% relative increase, conflicting with the table’s 75.5%. This edition retains the table value. Removing tokens also changes the model’s computation, so the comparison does not isolate every possible cause.
Table 5. More adaptation data helps up to a point in this reported sweep. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first column using the caption’s definition: the data volume is the product of the number of samples and repetitions. It is not necessarily a count of independent exploratory rollouts. The second column is CALVIN average sequence length. Follow the rows in increasing volume to see a rise through 204 and a decline at 340. The nearby discussion attributes the decline to excessive emphasis on observation prediction, but the table measures action-task performance rather than a diagnostic of overfitting. Keep this sweep separate from the appendix’s five-chunk experiment, which uses a different configuration and contains an unresolved table/figure discrepancy. e15e20
What it supports. The highest listed score is 3.928 at volume 204, compared with 3.871 at 6 and 3.917 at 340. The result motivates treating the adaptation budget as a hyperparameter rather than assuming monotonic gains. It does not show that 204 is an optimum outside these four evaluated settings.
Where the evidence stops. No uncertainty is reported, so the small decline does not establish overfitting as its cause. Samples and repetitions are combined in the volume definition, leaving diversity and repeated exposure confounded. Appendix Table 9 and Figure 6 should not be merged into this sweep.
7. Analysis & limitations
7.1 What the evidence leaves open
The experiment called Simulator to Real World only adds Gaussian observation noise with standard deviation 0.1. It provides corruption robustness evidence, not physical deployment. The stated single-architecture limitation is qualified by an appendix GPT2/Qwen3 comparison; neither establishes broad model-size scaling. e16e21
Reported numbers lack confidence intervals. Appendix Table 7 uses a LIBERO baseline of 78.0%, versus 75.5% in Table 2. Table 9 and Figure 6 disagree on the data volume associated with 3.67 for the five-chunk setting. These differences prevent treating all ablations as one uniform protocol. e12e13e17e20
The selected before/after trajectory illustrates a possible recovery, but it is not a frequency estimate or a dynamics-accuracy metric. Improved action success alone does not identify whether adaptation learned physical dynamics, appearance features or both. e22e12e13
7.2 Questions for discussion
- Would observation-loss improvements predict held-out action success after controlling for exploration coverage?
- Does the implemented mask make observation predictions invariant to language when actions and state histories are fixed?
8. Reproducibility audit
8.1 Requirements and known gaps
Main TTT uses 204 CALVIN trajectories from 34 instructions with six samples each, and 36 LIBERO trajectories per scene from six samples and six seeds. Exploration lasts 60 frames. Ranks are 128/64, AdamW learning rate 0.005 and weight decay 0.01, with the source wording single-epoch single-step. Its batching interpretation is unspecified. e18
Resolve the action-loss definition, lambda, exact frozen modules, sampling rule and CALVIN split before reproducing. Reported RTX 4090 sampling-plus-adaptation times are about eight minutes per CALVIN task and two per LIBERO scene; the per-task CALVIN timing unit is not reconciled with first-scenario adaptation reuse. e04e06e08e09e11e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test the intended attention invariances
Reader-proposed check, not performed: with a fixed checkpoint, hold observations, proprioception, actions and histories constant while swapping language instructions. Compare world-output tensors and layerwise changes; separately perturb current teacher-forced action targets and check that the agent prediction cannot read them. Use a deliberately unmasked implementation as a positive leakage control. A language-dependent world output with fixed action inputs, or target-dependent agent output, would falsify the intended isolation for that implementation. Resolve Figure 3’s caption reversal and document the exact row/column mask before interpreting the test; ordinary action changes caused by a new instruction must not be mistaken for leakage. e03e04e05e06e07
Check 2: Isolate adaptation from exploration coverage and repeated exposure
Reader-proposed check, not performed: collect the same CALVIN exploration pool for every comparison, then evaluate no update, shared-backbone LoRA with observation loss, and observation-decoder-only adaptation with the backbone frozen. Begin with the stated main TTT recipe, explicitly documenting what single-epoch single-step means. Separate distinct trajectory count from repetitions, and evaluate paired held-out initial states with several seeds. Measure held-out observation error, executed sequence length and total interaction/training time. If decoder-only fitting reduces prediction error without improving actions while shared-backbone updates improve both, that would support the proposed transfer route. If action gains disappear against the identical-exploration no-update control, the adaptation explanation would weaken. e08e12e15e18e19
8.3 Reading coverage
Visual audit: Actually inspected the title/author/version page, all method pages, Figures 1–6, Tables 1–12, Algorithm 1, appendix implementation and TTT settings, and the qualitative-example explanation. The six final original crops were each viewed after extraction; narrow crops were rendered from the PDF at higher DPI without upscaling. Figure 1(a) is the selected architecture panel; its other panels were inspected on the full page. Cross-checked mask colors and placeholder names against the method body, caption and model conditionals, retaining the source inconsistencies. All supporting evidence pages are declared, including uncropped appendix pages. Reference pages 8–9 were read as text; separate supplements, code and project media were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Introduction (pp. 1–3)
- Method: task formulation, joint prediction, chunking, masking, pretraining, inference and TTT (pp. 3–5)
- Experiments, results and ablations (pp. 5–7)
- Related Work; Conclusion; Limitations and Future Work (p. 7)
- References (pp. 8–9)
- Complete Appendix: implementation, Algorithm 1, baseline/TTT ablations, comparisons and notation (pp. 10–15)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity verified against p. 1: arXiv:2609.08162v1, 8 September 2026; title and all five authors match the catalog. No different revision was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF figures and tables.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read. PDF pages 1–7 and 10–15 were visually inspected; reference pages 8–9 were read as text.
- Code and the linked project website were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block and arXiv margin
The exact catalog title and five authors appear: Chi Wan, Kangrui Wang, Yuan Si, Pingyue Zhang, Manling Li; all list Northwestern University. The artifact is arXiv:2609.08162v1 [cs.AI], 8 Sep 2026.
Go to primary source ↓e02PDF p. 1, Abstract and Introduction
The authors motivate adapting a jointly trained world model and action predictor to deployment environments using exploratory transitions.
Go to primary source ↓e03PDF p. 2, Figure 1(a) and caption; p. 3, Joint State–Action Prediction
One Transformer backbone supports task-conditioned action and task-agnostic observation prediction. The formal world conditional includes observations, proprioception, predicted actions and world history, but no instruction.
Go to primary source ↓e04PDF p. 3, Trajectory Splitting and Action Chunking; p. 4, Figure 2 and sampling definition; p. 11, Algorithm 1; p. 15, Table 12
The initial unit contains n actions; later units contain m actions and n sampled observations/states. Figure 2 uses observations 2,4 for n=2,m=3, whereas the floor-index formula yields 2,3. The prose defines sampling without replacement; Algorithm 1 invokes that routine.
Go to primary source ↓e05PDF p. 4, Mixed Unidirectional Attention Mask; p. 5, Figure 3 and caption
White, green and blue cells indicate unmasked, local-mask and global-mask positions. The action placeholder is blocked from current actions; the observation placeholder is blocked from instructions. The caption reverses the names of action chunk and placeholder. Columns are not explicitly labeled.
Go to primary source ↓e06PDF p. 4, Pretraining and displayed loss
Teacher forcing uses ground-truth actions. The joint objective is action cross-entropy plus lambda times observation reconstruction loss; lambda is not numerically specified.
Go to primary source ↓e07PDF p. 3, Task Formulation and model conditionals; pp. 4–5, Two-Step Inference; p. 4, Figure 2(b)
The agent predicts actions first, then the world model predicts observations conditioned on predicted actions; executed actions advance the environment. The inference prose includes instruction conditioning, unlike the task-agnostic formal conditional. Figure 2 places the dotted zero box on a0,1, while its caption names Ap.
Go to primary source ↓e08PDF p. 2, Figure 1(b) and caption; p. 5, Test-Time Training and Implementation Details
Collected trajectories receive a no-lang label. Stage 2 describes LoRA updates to the shared backbone with policy head frozen; the displayed update instead defines theta_w as world-head parameters. LoRA targets Qwen3 attention and MLP projections. Sampling plus adaptation is reported at about eight minutes per CALVIN task and two minutes per LIBERO scene on RTX 4090.
Go to primary source ↓e09PDF p. 10, Model Architecture: Encoder and Decoder
MAE-pretrained ViT-B encodes static/wrist RGB, a Perceiver compresses image tokens, and CLIP ViT-B/32 encodes text. MLPs encode and decode seven-dimensional state/actions; images are decoded to patches. Gripper decoding thresholds at 0.5: zero closed, one open.
Go to primary source ↓e10PDF p. 11, Baseline Results: Pretraining and Image and Action Chunk Lengths
The backbone has 12 attention heads and 24 layers; total and trainable parameter counts are 370M and 120M. Pretraining uses four H100 GPUs for approximately 60 hours on CALVIN and five on LIBERO. CALVIN uses image/action chunks 1/5 and trajectory length 16; LIBERO uses 1/3 and length 7.
Go to primary source ↓e11PDF p. 10, Implementation Details: Benchmark; p. 12, TTT Details
LIBERO-90 is described as policy-pretraining data and LIBERO-10/Long as downstream evaluation. CALVIN has environments A–D and evaluates sequences of up to five tasks; the paper does not explicitly identify its chosen CALVIN training/test environment split. TTT interacts with target scenarios.
Go to primary source ↓e12PDF p. 5, Table 1, WorldAgen, WorldAgen-TTT and Seer-Large rows
Average sequence length is 3.87, 3.93 and 3.83 respectively. WorldAgen-TTT T1–T5 are 96.6, 88.5, 78.5, 68.7 and 60.5 percent; baseline values are 96.3, 87.7, 76.8, 67.3 and 59.1. Seer-Large T4 is 68.9.
Go to primary source ↓e13PDF p. 6, Table 2, both stacked blocks
Average success is 75.5% for WorldAgen, 79.0% for WorldAgen-TTT and 78.7% for Seer. Microwave success rises 35% to 60%; mug-and-pudding falls 60% to 50%. The lower block continues the same seven-method row ordering.
Go to primary source ↓e14PDF p. 6, Table 3 and World Modeling Ability paragraph
Removing versus including image-prediction tokens gives CALVIN 2.96 versus 3.87 and LIBERO 46.5% versus 75.5% in the table. The adjacent paragraph instead reports LIBERO 78.0% and a 67.7% relative increase.
Go to primary source ↓e15PDF p. 7, Tables 4–5 and associated ablation paragraphs
CALVIN rank scores range from 3.918 to 3.930 for ranks 16–256. Table 5 gives data volumes 6,90,204,340 with scores 3.871,3.922,3.928,3.917. Its caption defines volume as samples multiplied by repetitions. Overfitting is the authors’ explanation for the decline.
Go to primary source ↓e16PDF p. 7, Table 6, Simulator to Real World, and Limitations and Future Work
Gaussian observation noise with standard deviation 0.1 is used to simulate sensor corruption. Scores without TTT / TTT without noise / TTT with noise are CALVIN 3.87/3.93/3.90 and LIBERO 75.5/79.0/78.0. This is not a physical-robot transfer experiment. The limitations paragraph states a single model size and architecture.
Go to primary source ↓e17PDF p. 11, Figure 4; p. 12, Table 7 and Number of Chunks continuation
Figure 4 peaks at three chunks, with image/action lengths 1/3. Table 7 reports CALVIN 3.87 at lengths 1/5 and LIBERO 78.0% at 1/3; the latter differs from the main baseline table’s 75.5%. Longer chunks need not improve performance.
Go to primary source ↓e18PDF p. 12, TTT Details
CALVIN collects 204 trajectories through 34 training-set instructions, 60-frame exploration and six samples per instruction in the first sample’s scenario. LIBERO uses six samples and six random seeds per scene, yielding 36. Main TTT uses ranks 128/64, AdamW, learning rate 0.005, weight decay 0.01 and the wording single-epoch single-step.
Go to primary source ↓e19PDF p. 12, LoRA vs. Full Fine-tuning; p. 13, Table 8
The matched LoRA/FFT comparison uses rank 128 for LoRA, Adam, learning rate 0.0005 and weight decay 0.01. CALVIN scores are baseline 3.87, LoRA 3.93 and FFT 3.85; these optimizer settings differ from main TTT details.
Go to primary source ↓e20PDF p. 13, Table 9, Figure 6 and Test-Time Training Data Size
For N=5,n=1,m=3, Table 9 lists rank 256/data 204 and 3.50 to 3.67. Figure 6 with rank 256 and learning rate 0.0005 places 3.67 at 170 samples and 3.59 at 204; the paper does not reconcile these configurations/results.
Go to primary source ↓e21PDF p. 13, Table 10 and Comparison Between Backbones; Table 11 and LIBERO Subsets
GPT2/Qwen3 scores are CALVIN 3.32/3.87 and LIBERO-10 75.0/75.5. This backbone comparison qualifies the single-architecture limitation on p. 7. Table 11 reports Spatial/Object/Goal/LIBERO-10 as 93.0/79.5/91.0/79.0.
Go to primary source ↓e22PDF p. 12, Figure 5 and caption; p. 14, Comparison Visualization
Selected simulator trajectories show green success and red failure regions before and after TTT. The accompanying task sequence is rotate blue block right, move slider right, lift red block slider and place in slider; the authors describe the unadapted policy stopping after the first task.
Go to primary source ↓8.5 Primary sources
WorldAgen: Unified State-Action Prediction with Test-Time World Model Training ↗
PDF · 8,742 extracted words
Source fingerprint
0e289d8fa0b1a84ada62f7c2993cc24267eaee9d7bfd772698baf92aba4ae5ab