Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Ruicheng Zhang; Mingyang Zhang; Jun Zhou; Xiaofan Liu; Zunnan Xu; Zhizhou Zhong; Puxin Yan; Haocheng Luo; Xiu Li

Affiliations: Tsinghua University; X Square Robot; Sun Yat-sen University; HKUST

Source: 2512.06628 ↗ · Catalog record

Reading: 314 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: MIND-V turns language into geometric video conditions and searches over generated futures, improving reported subtask success while adding inference cost and dependence on learned judges. e-probleme-srhe-bsbe-mvge-pfce-rolloute-longe-policy-resulte-scale

At a glanceWhat to know
Research problem
Source description

Long manipulation sequences amplify grounding mistakes, disappearing objects, and errors carried between subtasks. The authors seek a scalable source of coherent robot videos that accepts an image and instruction without requiring a user to draw each trajectory. e-probleme-srh

Core mechanism
Source description

A Semantic Reasoning Hub (SRH), Behavioral Semantic Bridge (BSB), and Motor Video Generator (MVG) separate task decomposition, geometric conditioning, and video synthesis. e-srhe-bsbe-mvg

A key reported resultLong-horizon manipulation video generation: MIND-V: 0.613; 0.462; 49%.

Average subtask success; PFC; user preference. 108 evaluation samples mixing Bridge V2 test scenes and unseen web scenes; long tasks contain 2–4 subtasks. Autonomous-model comparison.

Best baseline per column: WoW-1-Wan-14B success 0.347; Dreamdojo PFC 0.424 and preference 18%. Success improves by 26.6 percentage points, not 76.7 points; the latter is the approximate relative gain. Success averages subtasks rather than requiring an entire sequence to succeed. Preference is best-video voting from 30 volunteers, not robot execution. e-protocole-longe-user

Reading caution
Reader analysis

The authors identify rollout overhead and propagation of affordance-localization errors. A fallback uses VLM-inferred coordinates as segmentation prompts. Arbitrarily long operation remains an architectural claim: evaluated tasks have 2–4 subtasks, and timing covers only 1–3. e-limite-protocole-scale

Core contributions

  • Source description

    A Semantic Reasoning Hub (SRH), Behavioral Semantic Bridge (BSB), and Motor Video Generator (MVG) separate task decomposition, geometric conditioning, and video synthesis. e-srhe-bsbe-mvg

  • Source description

    PFC-guided GRPO post-training and staged candidate rollouts address different failure points: generator alignment during training and continuation selection during inference. e-pfce-grpoe-rollout

Figure 2. Task reasoning, geometric conditioning, and candidate evaluation form a loop around video synthesis. Original paper, p. 3 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start at the upper left with the scene and the instruction to clear the desktop. The SRH breaks this into object-specific subtasks, then grounds grasp points and plans trajectories below. Move right to the BSB's three numbered elements: object representations, collaborative trajectories, and phase-transition points. These feed the MVG, which produces candidate clips. The rightmost VLM assigns C1 for completion, C2 for physical plausibility, and C3 for visual quality, then sums them. Follow the dotted feedback arrow back toward planning: a judged continuation can advance the sequence or trigger another plan. e-srhe-bsbe-rolloute-ablation

What it supports. The architecture makes task-level decisions explicit before rendering pixels. Its feedback acts on candidate videos and revised geometric plans. This supplies a mechanism for correcting a misplaced endpoint or unsuitable continuation, consistent with the substantial success reduction when staged rollouts are removed.

Where the evidence stops. The figure's scores are illustrative, and the acceptance threshold and retry limit are unspecified. This is a loop over simulated visual outcomes; it does not depict sensing and acting on a physical robot.

2. Motivation

2.1 The problem and the proposed response

Source description

Long manipulation sequences amplify grounding mistakes, disappearing objects, and errors carried between subtasks. The authors seek a scalable source of coherent robot videos that accepts an image and instruction without requiring a user to draw each trajectory. e-probleme-srh

2.2 What this reading follows

Imagine asking a robot to clear a table: choosing the spoon, grasping it, and placing it in a pot must remain consistent across several visual transitions. MIND-V approaches this problem as hierarchical video synthesis. A reasoning model chooses subtasks, a structured bridge supplies masks and motion paths, and a diffusion model renders candidate continuations. Physical-coherence rewards shape training, while a separate inference loop judges and revises candidates. The paper reports stronger video subtask success and an additional policy-learning gain in simulation. Read those outcomes separately, and keep the source's inconsistent architectural details and experimental values visible when considering reproduction. e-probleme-srhe-bsbe-mvge-pfce-rolloute-longe-policy-resulte-scale

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryFoundational work
ArchitectureNot applicable
Prediction paradigmNot applicable
QuadrantNot applicable

This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded foundational neural-world-simulator placement is supported. SRH and MVG are separate modules connected through BSB; the core output is video. The downstream VLA is separately trained with visual losses, so this paper does not establish one network jointly predicting futures and actions or an inverse-dynamics action head. The recorded action-model architecture, paradigm, and quadrant being not applicable is appropriate. Dataset and training subcategories correspond to BSB annotation and GRPO. e-srhe-mvge-policye-datae-grpo

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Initial scene image and long-horizon language instruction; internally constructed masks, trajectories, and phase timings.
  • Generated manipulation-video sequences; a separate OpenVLA-OFT experiment uses generated visual goals to supervise a robot policy.

4.2 Equations and their role

hnew=h+norm⁡(G)⋅Gh_{\mathrm{new}}=h+\operatorname{norm}(G)\cdot G
Equation 1 fuses intermediate DiT state h with motion representation G. The text defines norm as Group Normalization, in conflict with Figure 3's Layer Normalization label. e-mvg
LSFT(θ)=Eϵ∼N(0,I),t[∥ϵ−ϵθ(xt,t,BSB)∥2]\mathcal{L}_{\mathrm{SFT}}(\theta)=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,I),t}\left[\|\epsilon-\epsilon_\theta(x_t,t,\mathrm{BSB})\|^2\right]
Equation 2 trains the parameterized noise predictor εθ on noised video xt at diffusion timestep t, conditioned on BSB; ε is the sampled Gaussian noise. e-sft
si=sim⁡cos⁡ ⁣(Pv(Ev(xcontext(i))),Ev(xtarget(i))),Rphysics(x0)=∑i=1Nwe(1−si)/τ∑j=1Nwe(1−sj)/τsis_i=\operatorname{sim}_{\cos}\!\left(P_v(E_v(x_{\mathrm{context}}^{(i)})),E_v(x_{\mathrm{target}}^{(i)})\right),\qquad R_{\mathrm{physics}}(x_0)=\sum_{i=1}^{N_w}\frac{e^{(1-s_i)/\tau}}{\sum_{j=1}^{N_w}e^{(1-s_j)/\tau}}s_i
Equations 4–5 compare V-JEPA2 encoder Ev and predictor Pv outputs over video windows. Nw counts windows; temperature τ controls emphasis on low-similarity windows. The target is another part of the generated video, not a measured physical future. e-pfc
R(x0)=wpRphysics(x0)+waRaesthetic(x0),wp=0.2,wa=1R(x_0)=w_pR_{\mathrm{physics}}(x_0)+w_aR_{\mathrm{aesthetic}}(x_0),\qquad w_p=0.2,\quad w_a=1
Equation 3 combines physical-coherence and VLM aesthetic rewards for generated video x0. These are training weights, distinct from the VLM's inference-time candidate score. e-pfce-rollout

5. Method in detail

5.1 Turn an instruction into a path for the active agent

Source description

The bridge is easiest to understand by separating semantic choice from temporal placement. SRH first decides which object should move and where it should go. Affordance grounding then identifies a useful interaction point, rather than treating every point on an object's mask as equally suitable for grasping. BSB organizes the motion into arm approach, object interaction, and arm retraction. During training, this structure is derived from demonstrations: noun extraction identifies the target, segmentation and tracking recover masks and centroid paths, and object movement defines phase boundaries. During inference, the planner supplies the corresponding conditions. The same representation therefore connects observed demonstrations to planned futures. The pipeline is largely automated, but Appendix B explicitly reserves failed annotations for human correction; autonomy at generation time does not imply annotation-free training. e-srhe-bsbe-data

Figure 3. BSB conditioning enters the video generator through structured motion features. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the bottom panel first. Encoded object and robot appearance is placed along the planned paths, with blue phase boundaries separating approach, interaction, and retraction. The resulting spatiotemporal guidance passes upward through motion embedding. In the right panel, spatial convolution, rearrangement, and temporal convolution produce features that enter a normalization-and-multiplication branch. Orange arrows then add motion guidance to the DiT stream alongside instruction and scene conditioning. Equation 1 writes this as h_new = h + norm(G)·G. The symbol G denotes motion features here, rather than the GRPO candidate-group size used elsewhere. e-bsbe-mvge-arch-appe-grpo

What it supports. This figure shows how geometric intent can repeatedly influence denoising instead of relying on the language prompt alone. It also exposes implementation details that matter: the normalization branch scales the motion features before addition, and Appendix D limits injection to alternating transformer blocks.

Where the evidence stops. Preserve two source conflicts: Figure 3 labels Layer Normalization but Equation 1 defines Group Normalization; Section 3.1 says each block, whereas Appendix D specifies even-numbered blocks. The supplied paper does not resolve either.

5.2 Understand what the physics reward can actually measure

Reader analysis

PFC compares two latent descriptions of the same generated clip. One is V-JEPA2's predicted future from context frames; the other is its encoding of the clip's target frames. Cosine similarity measures their agreement. Equation 5 weights windows with larger disagreement more heavily, so a locally implausible transition can lower the aggregate even when most frames look stable. GRPO combines this score with aesthetic judgments and learns from relative rewards within an eight-video group, while a KL penalty keeps the policy near the SFT reference. The frozen referee supplies a learned regularity signal, not a symbolic contact solver. Reader analysis: because PFC is also an evaluation metric, improvement should be checked against independent contact, collision, and object-permanence judgments before interpreting it as broad physical reliability. e-pfce-grpoe-grpo-appe-protocole-user

5.3 Separate candidate selection from policy execution

Reader analysis

At a subtask boundary, the system can spend more computation before choosing a continuation. SRH proposes BSB variants, MVG renders them, and a VLM evaluates their outcomes. Failure feedback changes the plan; a passing candidate advances the generated sequence. This explains why disabling staged rollouts and lowering K are related but distinct interventions: the former removes a mechanism, while the latter limits its candidate set. The K sweep supports a practical tradeoff, with most success gains appearing by three candidates. Reader analysis: an equal-generation-budget control is needed to establish how much comes from replanning itself. Keep the downstream policy experiment separate: there, generated videos become visual supervision for another trainable model. The reported MimicGen gain does not turn the video-generation loop into a demonstrated physical closed-loop controller. e-rolloute-ke-ablatione-policye-policy-result

5.4 Training and inference

During training

Source description

Bridge V2 annotations use noun-phrase extraction, Grounded SAM2, a robot template mask, and SAM2 centroid tracking. Object motion determines phase boundaries; 10% is randomly held out. Failed annotations receive manual correction, qualifying claims of fully autonomous data preparation. e-data

Source description

SFT uses 37-frame, 480×640 videos, AdamW, 30,000 steps, and learning rate 2×10⁻⁵. GRPO then runs 1,500 iterations at 5×10⁻⁵. Four NVIDIA H200 GPUs are reported. e-sfte-setup

Source description

Frozen V-JEPA2 scores predicted-versus-observed future latents; a VLM supplies aesthetic scores. GRPO normalizes rewards within eight-video groups and uses clipping and KL regularization toward the SFT reference. Appendix E gives clipping 0.2, KL coefficient 0.01, PFC temperature 0.1, and 50 DDIM steps. e-pfce-grpoe-grpo-app

During inference

Source description

At each transition SRH proposes K BSB variants, MVG generates candidate clips, and a VLM judges task completion, physical plausibility, and visual quality. A successful best candidate advances the sequence; otherwise textual feedback triggers replanning. Default K is 3. This feedback concerns generated scenes rather than physical robot observations. e-rolloute-k

5.5 Implementation flow

  1. Plan and ground

    Gemini-2.5 Pro decomposes the instruction into ordered action/object/destination tuples. Affordance-R1 localizes object masks and functional interaction points; the VLM plans smooth trajectories discretized into frame-aligned waypoints. e-srhe-setup

  2. Build the bridge

    BSB contains object and robot masks, collaborative trajectories, and phase-transition frame indices. Arm approach, object interaction, and arm retraction specify which agent's appearance follows which path. e-bsb

  3. Render conditioned futures

    The CogVideoX-5B-based MVG encodes appearance into a spatiotemporal guidance tensor. Spatial and temporal convolutions produce motion features fused into the DiT. Appendix D specifies 30 blocks, width 1920, and injection in even-numbered blocks; the main text says each block. Figure 3 labels Layer Normalization, while Equation 1 defines Group Normalization. These implementation discrepancies remain unresolved. e-mvge-setupe-arch-app

  4. Use visual goals for policy learning

    A separate OpenVLA-OFT policy receives pose supervision plus pixel L1 and LPIPS losses against successful MIND-V rollouts. Figure 6 marks MIND-V frozen and the VLA trainable. This is auxiliary policy training, not evidence that MVG directly outputs executable robot commands. e-policy

6. Experiments & results

MIND-V generates manipulation videos by converting language into subtasks, grounding object and arm trajectories, and rendering candidate futures with a conditioned diffusion model. RL adjusts a learned physical-coherence proxy; inference searches over generated alternatives. Its strongest evidence concerns video subtask success, with a separate, incompletely specified policy-learning experiment in simulation.

6.1 Read the original evidence

Table 2. The strongest headline result concerns successful subtasks in generated videos. Original paper, p. 8 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read each column with its own meaning. PFC measures agreement with the learned V-JEPA2 referee. Task Success Rate is defined in Section 4.1 as the average over subtasks, so 0.613 is not an all-or-nothing completion rate for the whole sequence. User preference is the percentage of pooled best-video choices in a blinded study. Compare MIND-V with the strongest baseline in each column: Dreamdojo for PFC and preference, and WoW-1-Wan-14B for success. The evaluation mixes Bridge V2 test scenes with unseen web scenes within a 108-sample set; long tasks contain two to four subtasks. e-protocole-longe-pfce-usere-reward-sensitivitye-k

What it supports. MIND-V raises reported subtask success from the strongest baseline's 0.347 to 0.613, a 26.6-percentage-point difference. PFC rises from 0.424 to 0.462, while preference reaches 49% against 18%. These are three different indicators of generated-video quality and instruction following.

Where the evidence stops. PFC also guides training, so it is not an independent physical test. Appendix F gives 30 participants but no valid-vote count or uncertainty. The nominal default PFC differs in Tables 5 and 7.

Table 3. Generated visual goals improve the reported mean policy success in MimicGen simulation. Original paper, p. 8 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. All entries are percentages, as specified by the original table caption. Read downward within a task to distinguish the base OpenVLA-OFT policy from continued imitation learning and from continued learning augmented with MIND-V visual supervision. Section 4.3 adds pixel L1 and LPIPS losses against a successful generated rollout to the end-effector pose objective. Figure 6 marks the generator frozen while the policy is trainable. The final Mean column aggregates the three tasks; it does not reuse the video subtask metric from Table 2. Coffee, StackThree, and Square remain separate simulated manipulation evaluations. e-policye-policy-result

What it supports. The mean rises from 33.4% under continued IL to 43.5% with MIND-V supervision, a 10.1-percentage-point gain. Improvements appear in all three columns: the augmented policy reports 51.7% on Coffee, 48.3% on StackThree, and 30.4% on Square. This is the paper's direct evidence for downstream policy utility.

Where the evidence stops. Section 4.3 mentions 128 demonstrations and 300 expert trajectories per task without explaining their relationship. It omits trial counts, uncertainty, loss weights, and a precise action-to-frame optimization path; physical-robot transfer is not established.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
Long-horizon manipulation video generation

108 evaluation samples mixing Bridge V2 test scenes and unseen web scenes; long tasks contain 2–4 subtasks. Autonomous-model comparison.

MIND-V: 0.613; 0.462; 49%.

Average subtask success; PFC; user preference

Best baseline per column: WoW-1-Wan-14B success 0.347; Dreamdojo PFC 0.424 and preference 18%.

Success improves by 26.6 percentage points, not 76.7 points; the latter is the approximate relative gain. Success averages subtasks rather than requiring an entire sequence to succeed. Preference is best-video voting from 30 volunteers, not robot execution. e-protocole-longe-user

Video visual quality

Table 1 separates short- and long-horizon evaluation; trajectory-guided short-horizon baselines receive privileged manual conditions.

MIND-V: short 0.684; long 0.658.

V-Bench imaging quality, higher is better

RoboMaster short 0.688; Dreamdojo long 0.660.

These exceptions contradict the blanket claim of outperforming every baseline across all metrics. Favorable dynamics scores do not imply the best score on every visual-quality dimension. e-protocole-visual-quality

MimicGen policy learning

OpenVLA-OFT on Coffee, StackThree, and Square. Section 4.3 mentions both 128 demonstrations and base-policy training on 300 expert trajectories per task without reconciling them.

MIND-V + IL: 51.7 / 48.3 / 30.4; mean 43.5.

Simulated task success (%)

Base mean 27.7; continued IL mean 33.4.

The reported mean improves by 10.1 percentage points over continued IL. This supports a visual-supervision benefit in simulation; trial counts and uncertainty are not reported. e-policye-policy-result

Component contribution on long-horizon videos

Table 4 variants of the full system.

Full: 0.613 success, 0.462 PFC.

Average subtask success and PFC

Without staged rollouts: 0.327 success; without GRPO: 0.582 success, 0.429 PFC; replacement localizer: 0.455 success; Qwen3-VL replacement: 0.567.

Removing staged rollouts produces the largest listed success decline. The comparison changes inference work, so it does not isolate search strategy from extra sampling compute. e-ablation

Rollout count versus computational cost

Appendix A.2, K varied from 1 to 5; K=3 is the stated default.

K=3: 61.3%, 181.6 s, 70.1 GB.

Video subtask success (%), time (s), peak VRAM (GB)

K=1: 35.2%, 144.5 s, 31.8 GB; K=5: 62.5%, 223.4 s, 122.0 GB.

The additional gain from three to five candidates is 1.2 percentage points. K=1 is a separate reported condition from disabling staged rollouts and should not be merged with Table 4. e-ke-ablation

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Table 4. Staged rollouts have the largest listed effect on subtask success. Original paper, p. 8 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Use the final column to track instruction completion and the PFC column to track the learned physical-coherence proxy. Row (a) removes GRPO, retaining SFT. Row (b) abbreviates the replacement as YOLO; the accompanying text specifies YOLO-World plus SAM2 in place of the affordance localizer. Row (c) removes staged rollouts. Row (d) replaces Gemini with Qwen3-VL. Compare each row with the full-system row at the bottom, while keeping visual-quality columns separate from functional correctness. This distinction matters because improved rendering quality is not itself evidence that the requested object reached its destination. e-ablatione-visual-qualitye-reward-sensitivitye-k

What it supports. Success falls from 0.613 to 0.327 without staged rollouts, the largest reduction among these variants. Removing GRPO reduces PFC from 0.462 to 0.429 and success to 0.582. Replacing the affordance localizer reduces success to 0.455, supporting the importance of functional grounding.

Where the evidence stops. Removing rollouts also changes inference compute, so the table does not isolate replanning from additional sampling. Full-model visual scores differ from Table 1, and its PFC differs from Tables 5/7; retain each table's reported setting.

Table 7. Three candidates capture most of the reported success gain, with substantial memory cost. Original paper, p. 13 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with K on the left: it counts candidate futures at a subtask transition, not the number of subtasks. Read the two cost columns before moving across to success and quality. The highlighted K=3 row is the authors' default, not the row with the highest success. Compare K=1 to K=3 to see the main gain, then K=3 to K=5 to see diminishing returns. Appendix A's separate task-length analysis reuses memory across sequential subtasks; that claim does not mean memory stays fixed when more candidate videos are generated in parallel. e-ke-scalee-longe-ablatione-setup

What it supports. Moving from one to three candidates raises success from 35.2% to 61.3%, with peak memory rising from 31.8 to 70.1 GB. Five candidates yield 62.5% at 122.0 GB. Thus the final two candidates add 1.2 percentage points of success while requiring 51.9 GB more peak memory.

Where the evidence stops. The K=3 PFC value is 0.445, versus 0.462 in Tables 2/4. Section 4.1 reports about 50 GB, and Table 6 has inconsistent time averages. Network transmission latency is excluded from the appendix timing protocol.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

The authors identify rollout overhead and propagation of affordance-localization errors. A fallback uses VLM-inferred coordinates as segmentation prompts. Arbitrarily long operation remains an architectural claim: evaluated tasks have 2–4 subtasks, and timing covers only 1–3. e-limite-protocole-scale

Reader analysis

PFC is both an optimized reward and an evaluation metric, so higher PFC alone cannot certify physical correctness. OOD demonstrations are selected frame sequences without a separate quantitative OOD breakdown or physical-robot deployment. e-pfce-protocole-oode-policy

Reader analysis

PFC at the nominal default is 0.462 in Tables 2/4 but 0.445 in Tables 5/7. Full-model visual-quality values also differ between Tables 1 and 4. Section 4.1 reports about 50 GB; Appendix A reports about 70 GB. Table 6's 30.14–30.85-second averages conflict with its totals and the prose's approximately 60 seconds per subtask. These values are retained separately. e-longe-ablatione-reward-sensitivitye-ke-visual-qualitye-setupe-scale

7.2 Questions for discussion

  1. Does replanning still help when every comparison receives the same video-generation budget?
  2. Do improvements in PFC survive evaluation by independent human contact and object-permanence judgments?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Reproduction requires the specified video, planning, affordance, segmentation, and V-JEPA2 models plus the annotated Bridge V2 split. Appendix D specifies 16 VAE latent channels, temporal/spatial compression 4/8, and 2×2×2 patches; padding for 37-frame inputs is not described. Resolve the block-injection and normalization contradictions before implementation. e-setupe-datae-arch-appe-mvg

Open question

Missing operational details include the retained training-set size, object-motion threshold, candidate acceptance threshold, retry limit, and exact V-JEPA2 checkpoint. The 108-sample composition and valid user-vote count are unspecified. Policy reproduction additionally needs the loss weights, a defined action-to-frame training path, evaluation trial counts, and clarification of the 128/300 trajectory descriptions. e-datae-rolloute-pfce-protocole-usere-policye-policy-result

Reader analysis

Proposed checks: compare replanning with equal-budget candidate selection, then compare PFC-aligned and SFT-only generators using blinded physical-violation ratings. These would separate extra inference compute and reward-specific improvements from the intended mechanisms. e-rolloute-ablatione-ke-pfce-user

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Does replanning beat equal-budget candidate selection?

Reader-proposed check: freeze the same SFT/RL checkpoint and evaluate the same tasks under the full propose–verify–refine loop, best-of-K selection without plan revision, and single-candidate continuation. Give the first two conditions the same total generated-frame budget, judge calls, and maximum retries; record both independent subtask and whole-sequence success alongside latency and VRAM. Predeclare acceptance rules because the source omits them. The replanning claim is weakened if equal-budget selection matches the full loop; it is strengthened if revised BSB plans reliably recover failures that selection alone cannot. e-rolloute-ablatione-ke-protocol

Check 2: Does PFC alignment improve independent physical judgments?

Reader-proposed check: compare the SFT checkpoint, aesthetic-only GRPO, and the reported combined reward, holding training updates, candidate count, and BSB plans fixed. Use held-out scenes and blinded, shuffled clips rated specifically for contact, collision, and object permanence; report PFC separately from human violation rates and instruction success. Include uncertainty over tasks and raters. If PFC rises without fewer independently judged violations, the reward is improving its proxy more than the intended behavior. If both improve without reduced task success, the physical-alignment interpretation gains support. e-sfte-pfce-grpoe-grpo-appe-ablatione-user

8.3 Reading coverage

Visual audit: Visually inspected the title/version/author block; Figures 1–12; Tables 1–8; method equations, training and evaluation text; limitations; and Appendices A–F, including hardware, architecture, GRPO, and user-study details. Every final crop was viewed at its helper-produced dimensions. Checked Figure 2's feedback direction and critic labels against Section 3.2; Figure 3's fusion path against Equation 1 and Appendix D; Figure 4's frozen encoder/predictor against the PFC equations; and Figure 6's frozen generator against policy-learning text. Preserved normalization, injection-frequency, score, trajectory-count, memory, and timing inconsistencies in the accompanying explanations. Reference-only pages 10–11 were read in the complete text but not visually inspected. No external videos or separate supplements were supplied or inspected.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1 Introduction
  • 2 Related Work (2.1–2.2)
  • 3 Method (3.1–3.3), Equations 1–6
  • 4 Experiments (4.1–4.4), Equation 7
  • 5 Conclusion
  • 6 Limitations
  • References
  • Appendix A: Computational Cost and Hyperparameters
  • Appendix B: Dataset Construction
  • Appendix C: Additional Visual Results
  • Appendix D: Network Architecture Details
  • Appendix E: Key GRPO Hyperparameters
  • Appendix F: User Study Protocol

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Version scope: the inspected title page identifies arXiv:2512.06628v4, 5 July 2026. The title and nine-author sequence match the catalog author field, including Puxin Yan. The supplied BibTeX instead includes Zhangrui Guo and changes author order; that credit is not on this PDF. Earlier editions were not supplied, so their contents and authorship changes cannot be reconstructed.
  • Acquisition noted: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by inspecting the original PDF pages and final crops.
  • Separate supplemental material availability has not been fully verified.
  • All seven supplied text chunks were read individually, including references and Appendices A–F. PDF pages 1–9 and 12–19 were visually inspected; reference-only pages 10–11 were read as text.
  • No external sources or code were inspected, and no experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title, author/affiliation block and arXiv marginInspect

Exact title; nine authors in order: Ruicheng Zhang, Mingyang Zhang, Jun Zhou, Xiaofan Liu, Zunnan Xu, Zhizhou Zhong, Puxin Yan, Haocheng Luo, Xiu Li. Affiliations: Tsinghua University, X Square Robot, Sun Yat-sen University, HKUST. The margin identifies arXiv:2512.06628v4, 5 July 2026.

Go to primary source ↓
e-problemPDF pp. 1–2, Abstract and Section 1; Figure 1Inspect

Motivates coherent long-horizon video synthesis from high-level instructions and introduces the three-component hierarchy, rollout search, and RL alignment.

Go to primary source ↓
e-srhPDF p. 3, Section 3.1, SRH and Figure 2(a–d)Inspect

Initial image and instruction produce action/object/destination tuples; affordance grounding supplies masks and interaction points; VLM plans waypoints. Figure 2 critic sums completion, plausibility, and visual-quality scores.

Go to primary source ↓
e-bsbPDF pp. 3–4, Section 3.1, Behavioral Semantic BridgeInspect

BSB specifies object/robot masks, pre-interaction/interaction/post-interaction trajectories, and phase frame indices.

Go to primary source ↓
e-mvgPDF p. 4, Figure 3 and Section 3.1, Equation 1Inspect

Guidance tensor enters convolutional motion embedding and additive DiT fusion. Equation 1 uses h + norm(G)·G and defines Group Normalization; diagram labels Layer Normalization. Prose says each block.

Go to primary source ↓
e-rolloutPDF pp. 4–5, Section 3.2; PDF p. 3, Figure 2(d)Inspect

K candidate BSB plans produce videos; a VLM judges them. A best candidate passing a predefined threshold advances the task; otherwise textual feedback refines masks and trajectories. Threshold and retry budget are unspecified.

Go to primary source ↓
e-sftPDF p. 5, Section 3.3, Stage 1 and Equation 2Inspect

BSB-conditioned noise-prediction objective adapts a pretrained video model using short subtask videos and supplies an SFT reference policy.

Go to primary source ↓
e-pfcPDF p. 5, Figure 4(b), Section 3.3 and Equations 3–5Inspect

Frozen V-JEPA2 encodes context and target frames and predicts future latents. Cosine similarity is aggregated with weights proportional to exp((1−s)/τ); physical and aesthetic reward weights are 0.2 and 1.

Go to primary source ↓
e-grpoPDF pp. 5–6, Section 3.3, GRPO Optimization and Equation 6Inspect

Group-normalized rewards determine advantages; a clipped objective and KL penalty constrain the denoising policy relative to the SFT reference. No separate value network is used.

Go to primary source ↓
e-setupPDF p. 6, Section 4.1, Architecture and TrainingInspect

Gemini-2.5 Pro, Affordance-R1, CogVideoX-5B, Bridge V2, 37 frames at 480×640; SFT 30,000 steps at 2×10⁻⁵; GRPO 1,500 at 5×10⁻⁵; four H200 GPUs; approximately 50 GB for 111 frames.

Go to primary source ↓
e-protocolPDF p. 6, Section 4.1, Evaluation Protocol and Baselines; Section 4.2Inspect

108 Bridge V2/web samples; long tasks contain 2–4 subtasks and success is averaged across subtasks. Trajectory-guided short-horizon baselines receive manual conditions; long-horizon comparisons use autonomous models. Section 4.2 claims universal superiority.

Go to primary source ↓
e-visual-qualityPDF p. 8, Table 1, Imaging Quality column and MIND-V rowsInspect

Short-horizon imaging: MIND-V 0.684, RoboMaster 0.688. Long-horizon imaging: MIND-V 0.658, Dreamdojo 0.660. MIND-V long aesthetic is 0.512.

Go to primary source ↓
e-longPDF p. 8, Table 2, MIND-V, WoW-1-Wan-14B and Dreamdojo rowsInspect

MIND-V PFC 0.462, subtask success 0.613, preference 49%; WoW-1-Wan-14B success 0.347; Dreamdojo PFC 0.424 and preference 18%.

Go to primary source ↓
e-policyPDF p. 7, Section 4.3, Figure 6 and Equation 7Inspect

MimicGen Coffee/StackThree/Square experiment uses OpenVLA-OFT, pose loss plus pixel L1 and LPIPS visual supervision; Figure 6 marks MIND-V frozen. Text mentions 128 demonstrations and 300 expert trajectories per task without reconciliation; loss weights and action-to-frame optimization details are absent.

Go to primary source ↓
e-policy-resultPDF p. 8, Table 3, all rowsInspect

Base/IL/MIND-V+IL mean success is 27.7/33.4/43.5%. MIND-V+IL task values are 51.7/48.3/30.4%; continued IL values are 37.4/36.7/26.1%. No uncertainty or trial count appears.

Go to primary source ↓
e-ablationPDF pp. 7–8, Section 4.4; PDF p. 8, Table 4Inspect

Full aesthetic/imaging/PFC/success: 0.504/0.684/0.462/0.613. Without GRPO PFC/success 0.429/0.582; without rollout success 0.327; replacing affordance with YOLO-World+SAM2 success 0.455; Qwen3-VL replacement 0.567.

Go to primary source ↓
e-reward-sensitivityPDF p. 8, Table 5 and Reward Sensitivity AnalysisInspect

At wa=1.0, wp=0.2, aesthetic/imaging/PFC/success are 0.504/0.658/0.445/0.613. At wa=0.5 PFC is 0.449; at wa=2.0 PFC is 0.434 and success 0.562.

Go to primary source ↓
e-limitPDF p. 9, Section 6, LimitationsInspect

Authors acknowledge candidate-generation overhead and upstream localization errors; fallback VLM coordinates serve as segmentation point prompts.

Go to primary source ↓
e-scalePDF p. 12, Appendix A.1, Figure 7 and footnote 1; PDF p. 13, Table 6Inspect

For 1/2/3 subtasks, totals are 60.24/123.02/181.55 s with peak VRAM 70.12 GB. Listed averages 30.14/30.60/30.85 s conflict with totals and approximately 60-second prose. Timings exclude network transmission latency.

Go to primary source ↓
e-kPDF pp. 12–13, Appendix A.2, Figure 8 and Table 7Inspect

K=1/3/5: success 35.2/61.3/62.5%, time 144.5/181.6/223.4 s, VRAM 31.8/70.1/122.0 GB. Default K=3 has PFC 0.445. K=3 is presented as the preferred tradeoff.

Go to primary source ↓
e-dataPDF pp. 12–15, Appendix B.1–B.2 and Figure 9Inspect

Bridge V2 BSB annotations use VLM noun extraction, Grounded SAM2 object masks, robot template mask, SAM2 tracking and motion phases; random 10% test split. Page 15 specifies manual correction. Total retained count and object-motion threshold are unstated.

Go to primary source ↓
e-oodPDF pp. 15–17, Appendix C, Figures 10–12 and captionsInspect

Selected frames compare long/short tasks and illustrate cluttered/stylized scenes, articulated doors, and cloth folding. These are qualitative generated-video examples rather than quantitative physical-robot tests.

Go to primary source ↓
e-arch-appPDF pp. 16–18, Appendix D and Table 8Inspect

3D VAE has 16 channels with temporal/spatial compression 4/8; DiT has 30 blocks, width 1920, 2×2×2 patches, 30 attention heads. Guidance enters even-numbered blocks only, unlike Section 3.1. Table 8 specifies a 50-step DDIM scheduler.

Go to primary source ↓
e-grpo-appPDF p. 18, Appendix EInspect

GRPO group size 8, clipping 0.2, KL coefficient 0.01, temperature 0.1, weights wa=1/wp=0.2; 50 DDIM steps and 37-frame 480×640 videos. AdamW moments 0.9/0.999 and weight decay 10⁻⁴ are specified.

Go to primary source ↓
e-userPDF pp. 18–19, Appendix FInspect

30 unpaid volunteers with relevant technical backgrounds view randomized subsets of the 108 tasks and anonymized, shuffled videos. Best-overall choices are pooled as preference percentages; replay is allowed and incomplete responses excluded. Valid vote count is absent.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.