RLVR-World: Training World Models with Reinforcement Learning
1. Paper overview
In one sentence: Rewarding decoded transition predictions improves separate language and video world models, but the gains remain specific to the chosen metrics, data and downstream evaluation. e02e04e05e06e08e09e11e13e14
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Teacher-forced maximum likelihood rewards correct tokens, whereas a useful world model needs accurate action-conditioned transitions over complete outputs. Discrete visual tokenization also separates token prediction from image-space quality. The authors propose using verifiable prediction metrics to close this objective mismatch; verification means agreement with available ground truth, not guaranteed physical correctness. e02e04e05e20 |
| Core mechanism | |
| A key reported result | RT-1 multi-step video prediction: LPIPS 13.4 ± 0.02; MSE 0.486 ± 0.003; repetition 9.9%. LPIPS ×100, MSE ×100 and repetition rate; lower is better. Same RT-1 split; seven predicted frames with supplied actions and context; three sampling runs. Base: 14.8 ± 0.02, 0.659 ± 0.006, 48.6%. Repetition-rejection base: LPIPS 14.4 ± 0.01 at 0.0% repetition. RLVR improves prediction beyond removing repeated frames. An explicit repetition-penalty variant reaches 0.0% repetition but LPIPS worsens to 13.7 ± 0.02. Table 3 reports the default LPIPS gain as 9.2% relative; rounded displayed means need not reproduce that percentage exactly. e11e13e16e17 |
| Reading caution | Reader analysis Best-of-100 ground-truth-ranked samples let the base model surpass RLVR on single-step LPIPS, despite RLVR's stronger one-shot result. Reward specialization also trades between metrics; neither proves general physical understanding. OOD actions, diffusion-model post-training and richer temporal/physical rewards remain open. e13e20 |
Core contributions
Figure 2. A shared training recipe places reward verification after modality-specific decoding. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each row from left to right. Both the state and the supplied action become input tokens below a world-model block. Green response tokens represent a sampled group, not an action sequence selected for execution. The solid arrows then pass through detokenization or a visual decoder to predictions on the right. Ground truth enters the comparison box separately. The dashed GRPO path marks training feedback based on these comparisons; Eq. (3) defines the reward's direction. The diagram lists possible metric families. The actual RT-1 experiments use negative L1 plus LPIPS, and Appendix A.4.1 explicitly keeps the visual tokenizer frozen during this post-training stage. e03e04e05e16
What it supports. The reward can judge an entire decoded output even when tokenization prevents direct optimization of that output metric through the usual prediction loss. This makes exact structured correctness and perceptual image quality usable training signals. The unification concerns the learning formulation; the language and video models remain separate implementations.
Where the evidence stops. This is a training diagram. It does not establish joint prediction of future states and control actions, and its list of candidate metrics is broader than the tested default reward. Ground-truth verification is available during post-training, not ordinary deployment.
2. Motivation
2.1 The problem and the proposed response
Teacher-forced maximum likelihood rewards correct tokens, whereas a useful world model needs accurate action-conditioned transitions over complete outputs. Discrete visual tokenization also separates token prediction from image-space quality. The authors propose using verifiable prediction metrics to close this objective mismatch; verification means agreement with available ground truth, not guaranteed physical correctness. e02e04e05e20
2.2 What this reading follows
A world model can become better at predicting tokens without becoming equally useful at predicting what an action will do. RLVR-World addresses that gap by generating several possible transitions, decoding them, measuring their agreement with observed outcomes and reinforcing the better responses. The same training recipe is demonstrated on text-game states, web-page changes and robot videos. These illustrations trace the reward loop, then separate transition accuracy from planning success and simulated policy evaluation. The central evidence is a consistent improvement over each model's supervised starting point; the qualifications include imbalanced data, metric tradeoffs, test-based checkpoint selection and failures outside the reported drawer tasks. e02e04e05e06e08e09e11e13e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work classification, with neural simulators and training optimization, fits this post-training framework. Architecture, prediction paradigm and quadrant marked Not applicable are defensible for a world-action-model taxonomy: separate language/video predictors consume actions, while distinct policies select them downstream. A common autoregressive formulation does not establish one joint future-and-action model or inverse-dynamics action extraction. e04e05e10e16e19
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Move the training target from tokens to decoded consequences
Begin with one recorded transition: a state, an action and its observed next state. Maximum likelihood fits the target response token by token. RLVR-World instead samples several responses from that same input, decodes each and assigns a score for agreement with the next state. GRPO compares those scores within the sample group, so a better complete output receives favorable learning credit across its generated tokens. Language verification can demand a fully correct game update or exact web-item changes. Video verification can combine pixel and perceptual discrepancies after decoding. This is why the reward need not be differentiated through the visual tokenizer. The paper's sign convention simply turns a smaller error into a larger reward. This reinforcement-learning problem trains a predictor; selecting actions for the original environment remains a separate operation. e03e04e05e06e08e16
5.2 Understand what the seven-frame video reward can see
The RT-1 implementation uses two different temporal arrangements. Single-step prediction reads four observed state/action pairs and predicts one new frame with an independent image tokenizer. Multi-step prediction compresses shared scene information into 1,280 context tokens, then represents subsequent frames with 80 tokens each. A 138M-parameter autoregressive Transformer receives the current frame, context and supplied actions; generation alternates predicted frame tokens with those action tokens. The visual tokenizer stays fixed while RLVR sums L1 and LPIPS errors across seven decoded future frames. A trajectory that freezes into repetitions can therefore receive a poor score even if copying previous tokens was locally plausible under likelihood training. Table 3's rejection control is particularly informative: removing repeated samples without reinforcement still leaves worse LPIPS than default RLVR. An explicit repetition penalty changes the balance again, improving that artifact metric while slightly worsening LPIPS. e11e13e16e17
5.3 Require a separate test before calling a predictor useful for control
Reader analysis: a more accurate transition model helps a controller only if its improvements affect the decisions that controller makes. The web experiment tests this by retaining external DeepSeek-V3 components while replacing the SFT world model with its RLVR version. Three candidate actions receive predicted outcomes, summaries and averaged value scores before one action is executed. PushT uses a different route: CEM planning measures goal distance in DINOv2 embeddings of generated frames, so the planning objective is distinct from the default image reward. Drawer evaluation reverses the question again: fixed robot policies interact with the learned simulator, and the resulting success estimates are compared with real reference outcomes. These examples support specific downstream uses. The failed grasp simulations show why improved frame metrics alone cannot establish reliable behavior under poorly represented actions or failure states. e09e10e14e19e20
5.4 Training and inference
During training
Language experiments start from DeepSeek-R1-Distill-Qwen-1.5B, with a 7B text-game variant. Text-game SFT uses LoRA and 4,237 rejection-filtered DeepSeek-R1 responses; web SFT combines WMA's GPT-4o-mini reasoning with extracted changes. RLVR uses exact correctness or a property-based reward for games and exact-item F1 for web states. Table 5 gives group size 5 and KL coefficient 0.001. e06e07e08
RT-1 visual tokenizers and separate 138M-parameter LLaMA-style predictors are trained from scratch. Finite scalar quantization replaces VQ. Independent frames use 320 tokens; compression uses 1,280 context tokens and 80 per subsequent frame. RLVR freezes the tokenizer and optimizes negative L1 plus LPIPS with GRPO group size 16; the reward sums over predicted frames. e16e17
During inference
Ordinary prediction samples and decodes the trained model without ground-truth reward feedback. Web MPC samples 20 policy actions, evaluates the three most frequent, summarizes predicted changes and averages 20 value scores per action. DeepSeek-V3 supplies policy, summarization and value functions; the selected action is executed in WebArena. e05e10
Real2Sim instead feeds generated frames back to an existing robot policy and advances a sliding prediction window, up to 112 frames. This evaluates policies inside a learned simulator; it is not new physical deployment or training an action policy within RLVR-World. e14e19
5.5 Implementation flow
- Serialize transitions
Templates map states and actions to question tokens and future states to response tokens. Text uses language tokenization; videos use learned visual tokenizers; each continuous action dimension is discretized into 256 bins. Actions condition prediction rather than being inferred from future frames. e04e16
- Sample, decode and verify
For one question, sample a group of responses, extract structured text changes or decode visual tokens, then compare each prediction with the observed target. GRPO normalizes rewards within the group and applies a clipped policy objective with reference-policy KL regularization; it needs no value network. e03e05
- Preserve temporal conditioning
RT-1 single-step prediction uses four observed state/action pairs. Multi-step prediction generates seven future frames from the current frame, supplied actions and an earlier context frame. Predicted frame tokens and supplied action tokens are appended alternately; only predicted-frame tokens contribute to the multi-step training loss. e16
6. Experiments & results
RLVR-World post-trains autoregressive world models by scoring decoded next-state predictions against ground truth and reinforcing better samples. Separate language and video implementations improve transition metrics and selected downstream applications, while remaining dependent on the base model, reward and evaluation protocol (e02–e05, e09–e14).
6.1 Read the original evidence
Table 1, numerical panel. Overall accuracy conceals a persistent gap on actions that change the state. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the 1.5B and 7B model blocks. Within a block, SFT is the immediate starting point for RLVR, whereas Base precedes that supervised adaptation. Next read the Changed column before Overall: a correct unchanged prediction can be easier than generating all effects of an intervention. The split refers to whether the true next state differs from the input state. Models predict structured state differences, evaluated for complete correctness on the official 2,954-transition test set. Finally compare the 7B binary row with the cited GPT-4 row. The aggregate ranking reverses when attention shifts specifically to state-changing examples. e06e09
What it supports. For the 1.5B model, task-specific RLVR raises overall accuracy from 32.87% to 63.24%, a calculated 30.37-point gain. Changed-case accuracy improves from 24.21% to 33.80%, but remains much lower than unchanged accuracy. The 7B result narrowly exceeds GPT-4 overall while trailing its changed-case score by 11.27 points.
Where the evidence stops. Binary versus task-specific RLVR also changes the training changed:unchanged ratio from 100:40 to 100:5. That comparison cannot isolate reward design. The table supplies no seed uncertainty, and the GPT-4 row is a result cited by the authors from prior work.
Table 2, numerical panel. Better exact state-change prediction accompanies a smaller gain in end-task web success. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The vertical divider separates two evaluations. Precision, recall and F1 score predicted accessibility-tree changes; correctness requires exact item strings, including fields and formatting. The special None item represents no change. These predictions are tested on the one-percent holdout of a length-filtered WMA subset. The rightmost column instead measures a WebArena agent using a policy, world model, summarizer and value model. It therefore tests a complete decision pipeline. Read the bottom delta row as relative change from SFT, as the source caption specifies. Subtracting the percentage-valued entries gives percentage-point changes, which are different quantities from those green relative gains. e02e07e08e09e10
What it supports. F1 rises from 49.94% to 65.11%, a calculated increase of 15.17 percentage points, while the table reports 30.3% relative improvement. Web-agent success rises from 12.06% to 14.29%. Improved state verification therefore transfers to this planning setup, but the two gains have different denominators and meanings.
Where the evidence stops. Appendix A.2 selects the world-model checkpoint using test-set reward. Table 6 reports SFT overall success as 12.07%, rather than this table's 12.06%, and shows a CMS regression. The introduction's '+15.1% F1' is not the table's relative F1 gain.
Table 3. Prediction improves beyond a reduction in repeated frames. MSE, SSIM and LPIPS are shown ×100; uncertainties are sampling-run standard deviations. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the two task blocks separate: single-step prediction conditions on four observed steps, while multi-step prediction rolls out seven frames with supplied actions and context. They use separately trained predictors and different tokenizers. Arrows in the headers indicate which direction is better. As the source caption specifies, MSE, SSIM and LPIPS values are multiplied by 100; the displayed plus/minus quantities are standard deviations over three sampling runs. In the lower block, compare RLVR with the repetition-rejection baseline before inspecting the penalty-trained variant. Rejection removes repeated outputs through additional sampling; the penalty variant instead changes the training reward. The delta rows denote relative improvements. e11e13e16e17
What it supports. Default multi-step RLVR reduces repetition from 48.6% to 9.9% and LPIPS ×100 from 14.8 to 13.4. Rejection alone reaches zero repetition but only 14.4 LPIPS ×100, so avoiding repeats does not explain the whole prediction gain. Penalizing repetition during RLVR reaches zero repeats at 13.7, exposing a tradeoff with the default metric result.
Where the evidence stops. These metrics compare generated frames with recorded observations; they do not measure executed robot success. The uncertainty covers sampling rather than independent training runs. Displayed means are rounded, so recomputed percentages may differ slightly from the authors' delta row.
Table 13. A simulator is useful when its success estimates approach the real-policy reference. Original paper, p. 31 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a policy column, then compare each simulated estimate with the Real row within the same task block. A larger predicted success rate is not automatically better: the goal is accurate evaluation of that fixed policy. SIMPLER-VA and SIMPLER-VM denote Variant Aggregation and Visual Matching baselines. The learned-model estimates come from policy interaction with generated frames, initialized with real RT-1 observations. Appendix A.5 uses 30 initial frames per task, six drawer tasks and four policies for each of two world models. Each rollout is capped at 112 frames. A single annotator receives only its final frame and task description, with model and policy identity hidden. e14e19
What it supports. For RT-1 Converged, RLVR predicts 62.2% opening success against 81.5% real success, improving on the base estimate of 48.9%. Closing improves from the base estimate of 81.1% to 88.9%, against 92.6% real success. Calibration improves for this policy, although opening remains substantially underestimated and the beginning policy still has false positive estimates.
Where the evidence stops. These are predicted policy outcomes, not newly measured physical deployments. The authors focus on drawers because grasp simulations often hallucinated success despite poor actions. Single-annotator final-frame judgments and task selection limit generalization to other behaviors and failure modes.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Text-game state prediction ByteSized32-State-Prediction: 76,369 transitions from 31 games; official 2,954-transition test set. | 1.5B: 57.01% with binary reward; 63.24% with task-specific reward. Exact prediction accuracy; higher is better | SFT: 32.87%. Task-specific changed/unchanged accuracy: 33.80%/83.66%, versus 24.21%/38.88% for SFT. The task-specific overall gain is 30.37 percentage points, calculated from Table 1. The 7B binary model reaches 65.53% overall versus cited GPT-4's 64.76%, but remains below GPT-4 on changed cases: 40.33% versus 51.60%. e06e09 |
| Web-page state-change prediction WMA/WebArena data filtered to approximately 7,000 samples of at most 5,000 prompt-plus-target tokens; 99% training and 1% testing. | 65.11% Exact-item-change F1; higher is better | SFT 49.94%; base 11.83%. A calculated 15.17-point gain over SFT; Table 2 reports 30.3% relative improvement. The introduction's '+15.1% F1' should not be read as the table's relative F1 gain. Reformulated exact changes prevent direct comparison with WMA's descriptions. e02e07e08e09 |
| WebArena model predictive control Five WebArena domains; shared DeepSeek-V3 planning components with SFT or RLVR world model. | 14.29% Task success rate; higher is better | Table 2: SFT 12.06%; Table 6 instead gives 12.07%. Table 2 implies +2.23 percentage points and reports +18.4% relative gain. The source discrepancy is preserved. CMS decreases from 11.54% to 10.99%; improvement is not uniform across domains. e09e10 |
| RT-1 single-step video prediction 87,212 RT-1 trajectories, 99%/1% train/test; fixed test segment and four-step observed history. Mean ± standard deviation over three sampling runs. | LPIPS 12.2 ± 0.01; MSE 0.287 ± 0.001. LPIPS ×100 and MSE ×100; lower is better | Base: LPIPS 13.0 ± 0.04; MSE 0.336 ± 0.002. Table 3 reports 6.0% and 14.3% relative improvements respectively. Uncertainty measures sampling variability, not independently trained seeds. e11e16e17 |
| RT-1 multi-step video prediction Same RT-1 split; seven predicted frames with supplied actions and context; three sampling runs. | LPIPS 13.4 ± 0.02; MSE 0.486 ± 0.003; repetition 9.9%. LPIPS ×100, MSE ×100 and repetition rate; lower is better | Base: 14.8 ± 0.02, 0.659 ± 0.006, 48.6%. Repetition-rejection base: LPIPS 14.4 ± 0.01 at 0.0% repetition. RLVR improves prediction beyond removing repeated frames. An explicit repetition-penalty variant reaches 0.0% repetition but LPIPS worsens to 13.7 ± 0.02. Table 3 reports the default LPIPS gain as 9.2% relative; rounded displayed means need not reproduce that percentage exactly. e11e13e16e17 |
| Single-step prediction on PushT, Rope and Granular PushT: 18,685/21 train/test trajectories and three-step history. Rope and Granular: 900/100 trajectories each and one-step history. | PushT 0.70; Rope 2.08; Granular 2.42. LPIPS ×100; lower is better | Own base: 0.83/3.03/3.14. Reported DINO-WM: 0.7/0.9/3.5; its public PushT checkpoint re-evaluated by the authors gives 3.39. Results are mixed against reported DINO-WM: comparable PushT, worse Rope, better Granular. Published baseline numbers and the authors' checkpoint evaluation are distinct evidence sources within this table. e12e18 |
| PushT model predictive control CEM planner following DINO-WM configuration, with decoded predictions embedded by DINOv2 for goal comparison. | 0.86 Success rate; higher is better | Own base 0.80; DINO-WM 0.86. A separate simulated control result supports downstream utility. The appendix does not report Rope/Granular MPC because the baseline configurations could not be replicated. e19 |
| Real2Sim drawer-policy evaluation Four existing policies, six drawer tasks, 30 initial frames per task and model-policy pairing; one annotator labels final frames blind to model/policy identity. | RT-1 Converged: open 62.2%, close 88.9%. Simulated success rate compared with reported real success | Real: 81.5%/92.6%; base simulator: 48.9%/81.1%. For this policy, RLVR brings estimates closer to real outcomes but still underestimates success. These are simulator estimates; the paper does not show an increase in the policy's physical success rate. e14e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Sampling budget, GRPO group size and reward choice reveal the scope of the gain. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) as an oracle selection experiment: among N generated candidates, the best LPIPS against ground truth is retained. Panel (b) changes GRPO group size G and plots test LPIPS against gradient steps; it does not normalize total sampled tokens or GPU time. In panel (c), columns specify the training reward and rows specify the evaluation metric. Cell numbers retain their original units, while the color bar is normalized so higher is better. Do not compare raw values across different metric rows. The authors' metric-matching pattern is general rather than universal: the MSE row is slightly better under PSNR training than under MSE training. e13e16e21
What it supports. RLVR's one-shot LPIPS beats the base model's best-of-five result, yet the base model becomes better at best-of-100. Larger training groups improve convergence when measured by steps. Reader interpretation: RLVR improves the probability of useful ordinary samples, but these plots do not establish that it expands the best achievable predictions at every sampling budget.
Where the evidence stops. Appendix A.4.1 identifies all three panels as single-step RT-1 experiments, normally with group size 16. Oracle best-of-N needs ground truth unavailable in normal deployment. Group-size curves do not by themselves establish superior efficiency at equal compute.
7. Analysis & limitations
7.1 What the evidence leaves open
Best-of-100 ground-truth-ranked samples let the base model surpass RLVR on single-step LPIPS, despite RLVR's stronger one-shot result. Reward specialization also trades between metrics; neither proves general physical understanding. OOD actions, diffusion-model post-training and richer temporal/physical rewards remain open. e13e20
The text-game reward comparison also changes the changed:unchanged sampling ratio from 100:40 to 100:5, so it does not isolate reward design. Web checkpoint selection explicitly maximizes test-set reward, limiting an untouched-test interpretation. Table 1 and Table 2 do not provide training-seed uncertainty. e06e08e09
Policy evaluation focuses on drawers after grasp tasks produced false predicted successes, attributed by the authors to missing failed grasps in expert data. A single final-frame annotator and remaining real/sim discrepancies constrain the result. Cross-domain Rope/Granular transfer is preliminary and later degrades with overfitting. e14e18
7.2 Questions for discussion
- Would task-specific game rewards still outperform binary rewards with identical sample balance and independent checkpoint selection? (e06, e08)
- Can metric gains survive a fixed sampling-compute budget and improve action ranking on failed or counterfactual trajectories? (e13, e14, e20)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduce filtering, expert game rules, state-difference extraction, exact-string matching and the special None item, then freeze a validation-based checkpoint rule. Language SFT uses LoRA rank 32/alpha 16; Table 5 supplies response limits, batches and sampling settings. Exact split membership and software versions are not specified in these appendix descriptions. e06e07e08e21
For RT-1, retain tokenizer checkpoints, action-bin ranges and context sampling before the predicted segment. Table 8 gives AdamW, RLVR batch 128, mini-batch 32, learning rate 0.00005, KL coefficient 0.001 and top-k 100. Resolve the multi-step vocabulary discrepancy: the sequence description says 9,006, while Table 8 says 9,008. e16e17
Budget the pretrained system as well as RLVR: multi-step tokenizer/transformer training costs 480/500 GPU-hours; 200 RLVR steps take 10 hours on four 40GB A100s. Single-step equivalents are 360/530 GPU-hours and 3.5 hours on four A100s. Thus a small gradient-step count is not a matched-compute comparison. The paper names accelerate and verl but supplies no version pins here. e21
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate game-reward shaping from sample-balance changes
Proposed check, not performed: start all runs from the same 1.5B SFT checkpoint and cross the two rewards with both changed:unchanged ratios, 100:40 and 100:5. Keep prompts, group size, update budget and decoding fixed, and select checkpoints on a new validation split while reserving the official test set for one final evaluation. Repeat training with independent seeds and report changed, unchanged and overall accuracy separately. Reward shaping would have stronger causal support if it improves changed-case accuracy at each matched ratio. If its aggregate advantage disappears when the ratio is fixed, the original comparison mainly supports a combined reward-and-resampling intervention. e06e07e09
Check 2: Test whether video gains survive equal sampling cost
Proposed check, not performed: use identical RT-1 held-out segments and frozen tokenizer checkpoints to compare the base model, default RLVR, base repetition rejection and repetition-penalty RLVR. Match total generated tokens or measured GPU time, counting rejected samples, and report seven-frame LPIPS, MSE and repetition with uncertainty across independent training seeds. Separately repeat the single-step best-of-N diagnostic at N = 1, 5, 10 and 100, labeling it as ground-truth oracle selection. A persistent RLVR advantage over rejection at matched cost would strengthen the claim that reward tuning improves transition predictions beyond removing repetitions. Loss of that advantage would identify inference budget as an important explanation. e11e13e16e17e21
8.3 Reading coverage
Visual audit: Actually viewed the title/author page, every numbered figure and table, the equations and method pages, and all appendix pages supporting retained training, evaluation, compute and reproduction details. All six final original-PDF crops were individually viewed and retain readable headers, labels and legends. Figure 2's forward and GRPO feedback arrows were checked against Sections 3–4 and the frozen-tokenizer statement in Appendix A.4.1. Figure 4's row/column orientation, metric directions and normalized color legend were checked against the source text; the MSE/PSNR exception is retained. The complete text, including checklist and prompt examples, was read in all 11 chunks. Pages 11–21 and 33–39 were read as text but are outside the page-image inspection pass; they supply no additional retained numerical or implementation claims. Separate supplements, external links and code remain uninspected. Internal differences between Tables 2/6 and between the multi-step vocabulary prose/Table 8 are explicitly preserved in the base report.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–3: introduction, related work, visual tokenization and GRPO
- Section 4: formulation, sequence modeling and verifiable rewards
- Section 5: text games, web prediction and web-agent model predictive control
- Section 6: video prediction, model analysis and Real2Sim evaluation
- Section 7: discussion and limitations
- Acknowledgements, references and complete NeurIPS checklist
- Appendix A.1–A.5: all implementation details and extended experiments, including A.4.2
- Appendices B–C: computational resources and broader impact
- Appendix D: complete prompt examples through PDF page 39
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All 11 supplied text chunks were individually read, covering the complete 39-page PDF. Text extraction alone does not reconstruct figure images; the retained PDF was additionally inspected for every numbered figure and table and all pages supporting retained scientific details.
- Separate supplemental material availability has not been fully verified; none was supplied separately.
- Code, linked resources and external baseline papers were not inspected, and no experiments were reproduced.
- The observed title and all four authors match the catalog. The title page identifies the NeurIPS 2025 proceedings edition; no numbered revision or revision date is stated there, and no comparison with another edition is established.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and conference footer
Exact title matches the catalog; authors are Jialong Wu, Shaofeng Yin, Ningya Feng and Mingsheng Long. Affiliations are School of Software, BNRist, Tsinghua University and Zhili College, Tsinghua University. Footer identifies NeurIPS 2025; this page supplies no numbered revision/date.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Figure 1 and Section 1
Authors motivate RLVR by mismatch between likelihood and transition metrics, discrete components and accumulated errors. Introduction reports '+15.1% F1' for web prediction, whereas Table 2 distinguishes the actual values and relative gains.
Go to primary source ↓e03PDF p. 3, Section 3, GRPO paragraph and Eq. (1)
GRPO samples G responses, standardizes response rewards within their group, avoids a value function and uses a clipped probability-ratio objective with reference-policy KL regularization.
Go to primary source ↓e04PDF p. 4, Figure 2 and Sections 4.1–4.2
Separate illustrated language/video pathways encode states and actions, generate response groups and compare decoded outputs with ground truth. Question tokens represent state/action inputs; response tokens represent next states. The generic diagram lists example metric families rather than one fixed reward.
Go to primary source ↓e05PDF p. 5, Eqs. (2)–(3), Section 4.3 and Remarks
MLE-pretrained models generate response groups whose decoded predictions are compared with target next states. sign(D) reverses lower-is-better metrics. RL optimizes the prediction process, not the original environment's control MDP.
Go to primary source ↓e06PDF p. 5, Section 5.1; p. 22, Appendix A.1, Eq. (4) and Figure 7
Text games use 76,369 transitions, 31 games and a 2,954-transition official test set, expert rules and state differences. Bases are R1-Distill-Qwen-1.5B/7B. Detailed 1.5B SFT uses 4,237 correct R1 responses, LoRA and epoch 10. Property-reward weights are 0.1/1/0.2; changed:unchanged ratios differ between binary (100:40) and task-specific (100:5) training.
Go to primary source ↓e07PDF p. 23, Table 5 and Appendix A.2, Dataset detail/Supervised fine-tuning
Filtering at 5,000 prompt-plus-target tokens retains about 7,000 of 14,000 web trajectories, split 99%/1%. SFT combines WMA GPT-4o-mini CoT with extracted changes. Table 5 gives LoRA rank 32/alpha 16; RLVR group 5, learning rate 10^-6, KL coefficient 10^-3, and dataset-specific batch/response settings.
Go to primary source ↓e08PDF pp. 23–24, Appendix A.2, reward definition and checkpoint selection
F1 uses exact matching across item fields and formatting, with explicit empty-set cases and a valid None item for no change. The checkpoint with highest test-set reward is selected for subsequent MPC evaluation.
Go to primary source ↓e09PDF p. 6, Tables 1–2, all model rows and metric columns; Section 5.2 and footnote 2
Table 1 reports 1.5B SFT/binary/task-specific overall accuracy 32.87/57.01/63.24%; 7B binary 65.53% versus cited GPT-4 64.76%, with changed accuracy 40.33 versus 51.60%. Table 2 reports F1 49.94 to 65.11% and web success 12.06 to 14.29%; delta is relative. The changed-item target differs from WMA's natural-language target. No uncertainty is given in these tables.
Go to primary source ↓e10PDF pp. 6–7, Section 5.3; p. 24, Appendix A.3 and Table 6
Web MPC keeps the top three of 20 sampled actions, summarizes top-ten predicted changes and averages 20 value scores on a 1–5 scale. DeepSeek-V3 implements policy/summarization/value with top-p 0.95. Table 6 gives overall SFT 12.07%, unlike Table 2's 12.06%; CMS falls from 11.54% to 10.99%.
Go to primary source ↓e11PDF p. 7, Table 3, single-step/multi-step rows and caption; Figure 3; p. 8, Section 6.2
Table 3 reports means and standard deviations over three sampling runs; MSE, SSIM and LPIPS are multiplied by 100. Single-step LPIPS is 13.0±0.04 to 12.2±0.01; multi-step 14.8±0.02 to 13.4±0.02. Rejection yields 14.4±0.01 at zero repetition; penalty-trained RLVR yields 13.7±0.02 at zero repetition. Figure 3 uses different pre/post-training step scales; continued MLE reaches LPIPS×100 of 14.5.
Go to primary source ↓e12PDF p. 8, Table 4 and caption
RLVR LPIPS×100 on PushT/Rope/Granular is 0.70/2.08/2.42 versus own base 0.83/3.03/3.14 and reported DINO-WM 0.7/0.9/3.5. Authors separately re-evaluate a public DINO-WM PushT checkpoint at 3.39. These comparisons are single-step prediction.
Go to primary source ↓e13PDF pp. 8–9, Section 6.3 and Figure 4(a–c); p. 26, Model analysis
All Figure 4 experiments are single-step, normally group 16. RLVR one-shot beats base best-of-five LPIPS, but base wins at best-of-100. Larger GRPO groups improve step-based convergence. Heatmap rows are test metrics and columns training metrics; color is normalized higher-is-better while cell values retain metric units. Metric-matched training is generally strongest, with exceptions including MSE under PSNR reward. Repetition penalty is negative consecutive-identical-frame rate.
Go to primary source ↓e14PDF p. 9, Figure 5/Section 6.4; p. 31, Appendix A.5, Task selection, Trajectory generation, Success judgment and Table 13
Drawer evaluation uses two world models, four policies, six tasks and 30 trajectories per pairing (1,440 total), capped at 112 frames. A single annotator sees final frame/task only. Grasp predictions produced false successes; authors attribute this to expert data lacking failed grasps. Converged-policy open/close estimates: RLVR 62.2/88.9%, base 48.9/81.1%, real 81.5/92.6%. Table 13 also contains both SIMPLER variants and other policy checkpoints.
Go to primary source ↓e15PDF p. 10, Figure 6; p. 28, Figure 9; p. 30, Figure 10
Original image sequences illustrate selected base/RLVR predictions, repeated frames, drawer-policy simulations and single-step PushT/Rope/Granular predictions. They are qualitative examples rather than additional aggregate success measurements.
Go to primary source ↓e16PDF p. 8, World model paragraph; p. 26, Appendix A.4.1, Eqs. (7)–(10) and footnotes 6–7
Actions use 256 bins per dimension; ranges are determined over the entire dataset. Single-step uses four prior state/action pairs; multi-step predicts seven frames with context. Separate 138M LLaMA-style Transformers are pretrained. Context precedes the segment; predicted frames alone receive multi-step loss; final action is a causally irrelevant placeholder. Tokenizers remain frozen in RLVR. Multi-step prose codebook size is 9,006.
Go to primary source ↓e17PDF p. 25, Appendix A.4.1 and Table 7; p. 27, Table 8 and Figure 8
RT-1 has 87,212 trajectories, 256×320 observations, 13-dimensional actions and a 99%/1% trajectory split with fixed test segments. FSQ tokenizers use 320 per-frame tokens or 1,280 context plus 80 per-frame tokens. Table 8 gives 12 layers, hidden size 768, GRPO group 16, batch 128, mini-batch 32, AdamW, learning rate 5×10^-5, KL 10^-3, top-k 100 and temperature 1. It lists multi-step vocabulary 9,008, conflicting with p. 26.
Go to primary source ↓e18PDF p. 28, Appendix A.4.2, Datasets/Implementation details; p. 29, Tables 9–11 and Extended analysis
PushT uses 18,685/21 trajectories and three-step history; Rope/Granular each use 900/100 and one-step history. PushT adds proprioception MSE to reward; Rope/Granular use six-layer models; all use group 32. Joint pretraining improves Rope's later individually tuned LPIPS×100 from 2.08 to 1.65. Cross-task gains are small and later degrade with overfitting.
Go to primary source ↓e19PDF pp. 29–30, Appendix A.4.2, Model-predictive control and Table 12; pp. 30–31, Appendix A.5
PushT planning follows DINO-WM's CEM settings and embeds generated frames with DINOv2; success is 0.80 for base and 0.86 for RLVR/DINO-WM. Rope/Granular control was not run because official planner details were lacking. Real2Sim uses external RT-1/RT-1-X policies interacting with generated frames through a sliding window.
Go to primary source ↓e20PDF p. 10, Section 7
Authors discuss post-training saturation, untested OOD action generalization, video models pretrained and post-trained on the same dataset, diffusion outside the study, and visual metrics that fail to fully capture physical rules or temporal consistency.
Go to primary source ↓e21PDF pp. 31–32, Appendix B, Computational Resources
Text-game SFT/RLVR: four 80GB A100s for 6.5h/eight for 22.5h. Web: eight 40GB A100s for 17h/eight 80GB H100s for 25h. RT-1 single-step tokenizer/Transformer: 360/530 GPU-hours; 200 RLVR steps: four 40GB A100s for 3.5h. Multi-step: 480/500 GPU-hours and four A100s for 10h. Frameworks are accelerate and verl; version pins are absent here.
Go to primary source ↓8.5 Primary sources
RLVR-World: Training World Models with Reinforcement Learning ↗
PDF · 18,234 extracted words
Source fingerprint
9e8a406d32adcc869bf2890ca55ec83cb37856a8d45c59b204b1c840232bb6db