PAPER REPORTENAll readings ↗

Motus2: A Self-Evolving General World Model for Dexterous Manipulation

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Hongzhe Bi, Zihao Zhou, Yihang Tang, Jingrui Pang, Shuhe Huang, Haitian Liu, Runqing Wang, Shuai Huang, Yichen Wang, Yiming Cheng, Ruowen Zhao, Zhenghua Li, Hengkai Tan, Xiaolong Liu, Jinhui Wan, Jiabao Liu, Min Zhao, Fan Bao, Jun Zhu

Affiliations: GensPI; Tsinghua University; BUAA; BIT

Source: 2608.30237 ↗ · Project page ↗ · Catalog record

Reading: 5 / 558 · 6 original figures & tables · ~23 min ·

1. Paper overview

In one sentence: Motus2 makes action generation, future simulation, and progress evaluation conditional interfaces of one model, enabling optional planning and policy improvement while memory and tactile feedback add distinct deployment costs. E02E03E04E08E09E13E15E16E17

At a glanceWhat to know
Research problem
Source description

Dexterous manipulation combines expensive embodiment-specific demonstrations, visual occlusion, and contact states that vision cannot reliably reveal. The paper seeks to exploit human interaction data and unsuccessful robot experience while preventing undesirable actions from becoming imitation targets and preventing future observations from leaking into deployed action prediction. E02E03E18

Core mechanism
Source description

A shared transformer exposes policy, action-conditioned simulator, and value interfaces through token dependencies and supervision gates. Architectural sharing is implemented through common computation, beyond merely optimizing several losses together. E02E03

A key reported resultFive-task main manipulation suite: Midtrain-SFT: Place Ball 100%, Multi-Finger 70%, Attach Eraser 100%, Screw Bulb 90%, Put Phone 60%; average 84%.

Task success rate and task-macro-average. Physical robot; 20 rollouts per task; matched target-task SFT data and evaluation. WAN-SFT, Pretrain-SFT, and Midtrain-SFT differ in initialization.

Pretrain-SFT averages 51%; WAN-SFT and π0.5 each average 0%. Mid-training adds 33 percentage points over Pretrain-SFT. Reported executed-robot results support the staged initialization recipe on this suite. The 84% result belongs to supervised Midtrain-SFT, not a combined demonstration of all proposed extensions. E12E13

Reading caution
Author claim

The authors identify wearable tactile noise and morphology mismatch as the primary limitation: glove deformation can produce signals without external contact, and human gloves do not transfer directly across differently shaped robot hands. E18

Core contributions

  • Source description

    A shared transformer exposes policy, action-conditioned simulator, and value interfaces through token dependencies and supervision gates. Architectural sharing is implemented through common computation, beyond merely optimizing several losses together. E02E03

  • Author claim

    The authors frame imagined branch evaluation followed by DiffusionNFT action updates as self-evolution. The implemented update freezes the video backbone and evaluator, making this a specific policy-improvement loop rather than continual updating of every interface. E06E07

  • Source description

    The work studies a monocular-to-stereo-to-robot curriculum, alternative visual memories, and a tactile expert that refines executable actions using recent contact measurements. E08E09E10E20E21

Figure 1. One shared core supports three queries, while tactile refinement forms an additional pathway. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the left half as the modeling interface. Language, action, stereo-video, value-query, and tactile tokens enter their indicated pathways; action and video generation share causal transformer blocks. The tactile expert is drawn separately and refines action sub-chunks. Read the right half as a sequence of conditional queries: the policy proposes action branches Aᵢ, the simulator produces corresponding future latents Zᵢ, and the evaluator assigns values Vᵢ. At the bottom, Maximum selects an action for test-time execution, whereas RL Loss returns a learning signal to the policy. These paths share the core model’s parameters but answer different conditional questions, so the drawing should not be interpreted as three independently trained networks. E02E04E06E09iv01

What it supports. The overview connects a world-action policy to an executable model-based loop: propose, predict, score, then select or improve. It also exposes the boundary of ordinary deployment. The action factor can run alone; imagined futures and evaluator calls are activated when planning or model-based policy optimization needs them.

Where the evidence stops. The diagram combines optional extensions in one overview. It does not mean every result uses planning, full history, and tactile feedback together, or that every control step generates future video.

2. Motivation

2.1 The problem and the proposed response

Source description

Dexterous manipulation combines expensive embodiment-specific demonstrations, visual occlusion, and contact states that vision cannot reliably reveal. The paper seeks to exploit human interaction data and unsuccessful robot experience while preventing undesirable actions from becoming imitation targets and preventing future observations from leaking into deployed action prediction. E02E03E18

2.2 What this reading follows

Dexterous manipulation demands more than a plausible action sequence: the hands can hide the object, contact can change rapidly, and a candidate movement can make the task worse. Motus2 builds a shared video–action model that can propose actions, simulate their consequences, and score progress. Its action-first dependency structure allows ordinary control to stop after generating actions; planning activates the additional interfaces. The illustrated evidence separates the large benefit of egocentric pretraining and robot grounding from the smaller measured gains of model-based policy improvement. Separate memory and tactile studies then examine what information the controller needs when recent images are insufficient. These are different controlled studies, with different tasks and operating settings. E02E03E04E08E09E13E15E16E17

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryWAMs
ArchitectureOne Model
Prediction paradigmOther mechanisms
QuadrantOutside quadrants

3.1 Evidence-based assessment

Insufficient evidence to decide

Reader analysis

WAM and One Model are supported for the core: action, video, and read-only value tokens share a transformer with explicit causal masks. WM-RL, memory, and tactile subcategories are also supported; the tactile expert has separate parameters, so sharing should not be interpreted as covering every module. Other mechanisms/Outside quadrants remain uncertain without catalog quadrant definitions: the paper combines joint video–action pretraining with action-first conditional control, simulation, and evaluation, rather than inverse-dynamics action recovery. The supplied snapshot is preserved. E02E03E06E08E09

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Language instruction, current proprioception, and encoded visual observation history.
  • Stereo egocentric video and aligned human or robot actions during training.
  • Optional recent tactile forces and Sharpa deformation images.
  • Executable action chunks or tactile-refined sub-chunks.
  • Optional action-conditioned future video latents.
  • A categorical progress distribution and its expectation for branch ranking.
  • Future-force predictions used as auxiliary training outputs.

4.2 Equations and their role

pθ(At,Zt,Ytct)=πθ(Atct)pθwm(Ztct,At)pθvm(Ytct,At,Zt),Vθ(ct,At,Zt)=EYtpθvm(ct,At,Zt)[Yt].\begin{aligned}p_\theta(A_t,Z_t,Y_t\mid c_t)&=\pi_\theta(A_t\mid c_t)\,p_\theta^{\mathrm{wm}}(Z_t\mid c_t,A_t)\,p_\theta^{\mathrm{vm}}(Y_t\mid c_t,A_t,Z_t),\\V_\theta(c_t,A_t,Z_t)&=\mathbb E_{Y_t\sim p_\theta^{\mathrm{vm}}(\cdot\mid c_t,A_t,Z_t)}[Y_t].\end{aligned}
At is an executable action chunk, Zt its future observation latents, Yt a numerical progress-bin center, and ct the available context. The factorization specifies the action-first dependencies enforced by the mask. E02E03
xσ=(1σ)x+σϵ,v=ϵx,L=wzvθzvz22+λawavθava22+λvwvCE(pθvm,Yt).\begin{aligned}x_\sigma&=(1-\sigma)x+\sigma\epsilon,\qquad v^*=\epsilon-x,\\\mathcal L&=w_z\|v_\theta^z-v_z^*\|_2^2+\lambda_a w_a\|v_\theta^a-v_a^*\|_2^2+\lambda_v w_v\operatorname{CE}(p_\theta^{\mathrm{vm}},Y_t).\end{aligned}
For video or action targets x, independent noise levels and gates select policy, simulation, or evaluation training. The respective gate triples are (1,1,0), (1,0,0), and (0,0,1). E03
rt={ΔtTt,successful segments,ΔtTt,negative segments.r_t=\begin{cases}\dfrac{\Delta t}{T-t},&\text{successful segments},\\-\dfrac{\Delta t}{T-t},&\text{negative segments}.\end{cases}
Δt is segment length, T trajectory length, and t segment start. This target measures signed progress relative to the remaining recorded horizon; it is not a Bellman return or calibrated completion probability. E05E24
r^i=0.5+0.5clip ⁣(Vimean(V)max(std(V),ϵ),1,1),vi+=(1β)vref,i+βvθ,i,vi=(1+β)vref,iβvθ,i,LNFT=Ei ⁣[r^ivi+vitar22+(1r^i)vivitar22].\begin{aligned}\hat r_i&=0.5+0.5\operatorname{clip}\!\left(\frac{V_i-\operatorname{mean}(V)}{\max(\operatorname{std}(V),\epsilon)},-1,1\right),\\v_i^+&=(1-\beta)v_{\mathrm{ref},i}+\beta v_{\theta,i},\\v_i^-&=(1+\beta)v_{\mathrm{ref},i}-\beta v_{\theta,i},\\\mathcal L_{\mathrm{NFT}}&=\mathbb E_i\!\left[\hat r_i\|v_i^+-v_i^{\mathrm{tar}}\|_2^2+(1-\hat r_i)\|v_i^--v_i^{\mathrm{tar}}\|_2^2\right].\end{aligned}
Generated branch scores determine attraction or repulsion around a fixed reference velocity. Population statistics exclude the ground-truth anchor, whose weight is one. Predicted futures affect optimization through scores and are never inputs to the action-velocity predictor. E06

5. Method in detail

5.1 Make failed experience useful without imitating it

Reader analysis

Motus2 factorizes its shared model as p(A,Z,Y | c) = π(A | c) p(Z | c,A) p(Y | c,A,Z). The context c contains instruction, proprioception, and accessible visual history; A is an action chunk, Z its future observations, and Y a categorical progress target. The action-first mask enforces this order, but supervision gates determine which trajectories teach which factor. Successful curated demonstrations activate action and video learning. Simulation mode conditions on clean recorded actions while predicting future video; evaluation mode conditions on clean actions and video while learning progress. Failed or irrelevant behavior can therefore teach transitions and unfavorable outcomes without becoming an imitation target. My interpretation is that this separation is central to the “self-evolving” claim: information about what happened and preference for what should happen enter different supervised pathways. E02E03E04E05

Figure 2. The decisive mask change prevents action queries from reading their own predicted future. Original paper, p. 6 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Take a row as the querying group and a column as the group it can read. In the joint mask on the left, A₁ can read Z₁ and Z₁ can read A₁; their within-chunk modeling is bidirectional. In the action-first mask on the right, the A₁-to-Z₁ cell is white, but the Z₁-to-A₁ cell remains colored. U₁ can then read the action and its predicted consequence, while its column is hidden from other token groups. Repeat this inspection for later chunks. Context groups C provide clean observed history under the windowed causal structure. The muted U positions on the left indicate that value queries are inactive during joint pretraining. E03E04iv02

What it supports. The mask realizes the factorization policy → simulator → evaluator. Actions depend on available context, future predictions can depend on those actions, and values can inspect both. Removing the reverse future-to-action edge is what lets the adapted policy produce an action chunk without materializing that chunk’s future video during ordinary control.

Where the evidence stops. A mask specifies permitted access, not whether learned features exploit it well. Loss routing is a separate requirement: only successful trajectories supervise action imitation, while other trajectories can still supervise transitions or evaluation.

5.2 Turn imagined progress into selection or a policy update

Source description

For a candidate action Aᵢ, the simulator predicts Zᵢ and the evaluator estimates a progress value Vᵢ. Best-of-N planning executes the highest-valued branch and replans after a real observation. MBRL instead uses the same values to change action-related parameters through DiffusionNFT. Scores are centered and normalized within the generated candidate group, clipped, and mapped into weights between zero and one. Higher weights emphasize a positive velocity field around an EMA reference; lower weights emphasize its negative counterpart. A ground-truth action–future pair anchors each group positively and is excluded from score normalization. The predicted future affects the action update through its score, not as an input to the action predictor. This distinction explains Table 3: candidate selection and proposal learning can help separately, but both depend on the reliability of learned simulation and evaluation. E05E06E07E15E21

5.3 Separate missing history from missing contact information

Reader analysis

A hand can obscure an object that was visible earlier, or conceal a contact force that was never observable in RGB. Motus2 addresses these as separate information problems. Global autoregression retains every past clean visual latent; hybrid memory trades some of that detail for persistent compressed tokens and a recent window. Their task-specific comparison favors global history, with cost growing over time. The tactile expert instead operates at a shorter temporal scale: the backbone denoises a 48-action chunk to an intermediate noise level once, and a small expert finishes successive six-action sub-chunks using newly acquired touch. Detached K/V features are reused across the eight refinements. Future force is predicted during training only. My reading is that these extensions support different causal explanations—remembering earlier evidence and measuring current contact—and their separate ablations should remain separate when interpreting the overall system. E08E09E16E17E22E23

5.4 Training and inference

During training

Source description

Initialization uses Wan 2.2-TI2V-5B. Video-only monocular training runs for 500K low-resolution and 340K high-resolution steps; joint stereo video–action training follows for 450K steps. The approximately 130K corpus hours describe raw recordings before filtering, not effective training hours. E10E20

Source description

Mid-training uses more than 100 hours of robot and supplementary alignment data. Simulation and evaluation receive 10% and 5% of samples; the remainder use policy mode. The evaluator uses 201 bins and value-loss weight 0.05. Target-robot supervised adaptation retains the action-first mask. E21

Source description

MBRL generates eight candidates per prefix plus a ground-truth anchor, with a one-chunk imagined horizon. DiffusionNFT updates only action-related parameters; the video backbone and evaluator stay frozen. A versioned EMA reference supports asynchronous rollout and optimization, with stale or excessively delayed samples discarded. E06E07

Source description

Long-history variants train on complete episodes packed with masks that prevent cross-episode attention. Tactile losses update the expert and its modality encoders/decoder using detached backbone states. Force prediction has weight 0.1 and is auxiliary to action refinement. E08E22E23

During inference

Source description

Ordinary control queries only the action factor; default deployment uses five action-denoising steps. Future video and values need not be generated. After execution, actual observations refresh the policy context. E04E08E21

Source description

Planning proposes N chunks, simulates and scores each, executes the highest-valued chunk, and replans after receiving the real observation. Tactile deployment outputs refined actions; it does not require future-force prediction. E06E09

5.5 Implementation flow

  1. Standardize human interaction data

    Synchronize recordings, wrist poses, and language annotations. A 134-dimensional representation combines bilateral wrist poses with wrist-local finger keypoints. Human fingers are retargeted offline to first-generation 20-DoF Wuji joints using constrained geometric optimization; target-robot post-training instead uses recorded native joint angles. E10E11

  2. Construct the shared causal computation

    Language enters through cross-attention; stereo video, proprioception, actions, and value queries participate in the shared transformer. Pretraining allows mutual video–action visibility within a chunk. Robot adaptation blocks action tokens from reading that chunk’s future video or value query. Video can read actions, and the read-only value query can read both. E02E03

  3. Route supervision by trajectory quality

    Curated successful trajectories supply action targets. Simulation mode conditions on clean recorded actions and learns future video; evaluation mode conditions on clean actions and video and learns progress. Failed, suboptimal, and task-irrelevant interactions enter applicable simulation or evaluation modes without supervising imitation. E03E04

  4. Learn a branch evaluator

    Successful segments receive positive relative-progress targets and negative trajectories receive their signed counterpart. A categorical readout predicts discretized targets, and its expectation ranks candidate action–future pairs. These values are explicitly not calibrated success probabilities. E02E05E24

  5. Retain observation history

    The default sliding window stores actual observed latents, evicts old entries, and rebases temporal RoPE. Global autoregression retains every preceding observation at growing cache and attention cost. Hybrid memory retains initial anchors and recent frames at full resolution while compressing intermediate history into persistent tokens. E08E22

  6. Refine actions with tactile feedback

    The backbone partially denoises a 48-action chunk to σc = 0.2 and supplies detached layer-wise caches. At 30 Hz action execution, the tactile expert refines eight successive six-action sub-chunks using the latest tactile window. Future-force tokens can read actions, but action refinement cannot read noised future forces. E09E23

6. Experiments & results

Motus2 adapts a jointly pretrained video–action transformer into an action-first controller: propose an action chunk, optionally simulate its visual consequences, and evaluate the branch using learned relative progress. Best-of-N planning selects branches, while DiffusionNFT updates action parameters using the same scores. Separate studies examine full-history context and tactile refinement. The main physical-task suite reaches 84% average success after robot-domain mid-training; policy improvement and the other extensions are evaluated separately. [E02, E06, E13, E15–E17]

6.1 Read the original evidence

Table 2. Egocentric pretraining and robot-domain grounding are separate steps in the main comparison. Original paper, p. 12 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. The three rows ending in SFT use the same target-task supervised adaptation procedure but start from different checkpoints. WAN-SFT begins from the video foundation model, Pretrain-SFT adds the two-stage egocentric curriculum, and Motus2 begins after robot-domain mid-training. Each task entry is a success rate over twenty matched configurations; partial completion counts as failure. Read across the five tasks before comparing Avg. SR, which gives equal weight to each task. Place Ball and Put Phone test placement, Attach Eraser and Screw Bulb involve alignment and sustained contact, and Multi-Finger tests coordinated manipulation. The π0.5 row uses the matched target-task data and evaluation interface described in the setup. E12E13iv03

What it supports. The initializations produce 0%, 51%, and 84% mean success for WAN-SFT, Pretrain-SFT, and Motus2, respectively. Mid-training adds 33 percentage points over egocentric pretraining alone. Even the strongest row remains uneven: Place Ball and Attach Eraser reach 100%, while Put Phone reaches 60%.

Where the evidence stops. Zero success for π0.5 and WAN-SFT is specific to these tasks and adaptation conditions. This table is not a universal ranking, and it does not itself isolate planning, tactile feedback, or memory extensions.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
Five-task main manipulation suite

Physical robot; 20 rollouts per task; matched target-task SFT data and evaluation. WAN-SFT, Pretrain-SFT, and Midtrain-SFT differ in initialization.

Midtrain-SFT: Place Ball 100%, Multi-Finger 70%, Attach Eraser 100%, Screw Bulb 90%, Put Phone 60%; average 84%.

Task success rate and task-macro-average

Pretrain-SFT averages 51%; WAN-SFT and π0.5 each average 0%. Mid-training adds 33 percentage points over Pretrain-SFT.

Reported executed-robot results support the staged initialization recipe on this suite. The 84% result belongs to supervised Midtrain-SFT, not a combined demonstration of all proposed extensions. E12E13

Policy optimization and test-time selection

Put Phone and Multi-Finger; physical robot; 20 rollouts per task; factorial comparison from the same target-robot checkpoint.

Baseline: 60%/70%, average 65.0%; planning: 65%/70%, average 67.5%; MBRL: 65%/80%, average 72.5%; both: 70%/80%, average 75.0%.

Success rate

Planning adds 2.5 percentage points with either policy; MBRL adds 7.5 points with either inference setting.

The measured improvements distinguish changing candidate selection from changing policy weights. Their statistical reliability is unresolved because uncertainty estimates are not reported. E12E15

Long-horizon context probes in simulation

Find Square and Press Button; 25 rollouts per task; matched training and paired configurations.

Global autoregression: 84%/72%, average 78%.

Success rate

Hybrid memory: 64%/40%, average 52%.

Full observation retention outperforms the tested compressed representation in simulation. This table does not include the default sliding-window baseline. E16E22

Long-horizon context probes on the physical robot

Find Square and Press Button; 20 rollouts per task; matched training and paired configurations.

Global autoregression: 65%/50%, average 57.5%.

Success rate

Hybrid memory: 30%/20%, average 25.0%.

The ordering agrees with simulation, but these physical execution rates must remain separate from the simulation results. E12E16

Contact-sensitive manipulation

Pull Out Paper Cup with Sharpa Wave and Tear Paper with Wuji Hand 2; 20 physical rollouts per task; matched checkpoint and post-training data.

With tactile expert: 75%/70%, average 72.5%.

Success rate

Without tactile expert: 65%/55%, average 60.0%; average gain 12.5 percentage points.

This establishes a benefit for the complete tactile package. It does not isolate the contributions of refinement, auxiliary force prediction, or added model capacity. E17E23

Stereo human-action prediction scaling

Nested 2k, 4k, 10k, and 20k raw-hour subsets; common trajectory-disjoint held-out set; best validation epoch selected at each scale.

Reported fitted relation: L*val(D) ≈ 0.101 − 0.005 ln D.

MSE between the mean sampled action prediction and normalized ground-truth wrist/hand actions

The text reports monotonically decreasing best validation error across the four scales.

This is a fitted human-action prediction trend over the measured range, not an executed-robot success result. Individual curve values are omitted because the figure was not visually inspected. E14

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Table 3. Planning changes candidate selection; model-based learning changes the proposal policy. Original paper, p. 14 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. The four rows form a two-by-two design from the same target-robot checkpoint. Compare the first and second rows to add planning without updating the policy; compare the first and third to add MBRL under direct inference. The last row activates both. Put Phone and Multi-Finger each have twenty evaluation rollouts, so one additional success changes a task’s entry by five percentage points and the two-task average by 2.5 points. Read the task columns as well as Avg. SR: planning raises Put Phone in this sample but leaves Multi-Finger unchanged. MBRL updates action-related parameters, whereas the simulator and evaluator remain fixed during that optimization. E06E07E15E21iv04

What it supports. Planning adds 2.5 points to the two-task mean both before and after MBRL. MBRL adds 7.5 points with or without planning, bringing the combined system to 75.0%. The table supports distinct observed contributions from selecting among candidates and improving the distribution that generates them.

Where the evidence stops. The planning increment corresponds to one additional successful rollout across forty trials. The table provides neither confidence intervals nor a matched wall-clock comparison, so broader reliability and compute-efficiency claims remain unresolved.

Table 4. Keeping all observations wins these memory probes, with a growing history cost. Original paper, p. 14 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Keep the Simulation and Real robot blocks separate: they use twenty-five and twenty rollouts per task, respectively. Within each block, compare hybrid memory against global autoregression under matched training and paired configurations. Both variants are trained on complete episodes, but their retained information differs. Global autoregression keeps all prior clean visual latents. Hybrid memory retains initial anchors and recent frames at full resolution while compressing older observations into memory tokens. Find Square and Press Button were selected because useful evidence can be delayed or occluded. Their averages diagnose these long-history mechanisms; the table does not include the bounded sliding-window default as another row. E08E16E21E22iv05

What it supports. Global autoregression reaches a 78% simulation mean and 57.5% real-robot mean, compared with 52% and 25% for hybrid memory. The direction agrees across both tasks and settings. The results favor retaining full visual history on these probes, while leaving the deployment cost of that additional context to a separate measurement.

Where the evidence stops. Global attention and KV storage grow with episode length. These variants also use fifty denoising steps in the reported comparison, unlike the default five-step controller; their success rates should not be paired with default-runtime assumptions.

Table 5. Fresh tactile refinement helps on two contact-sensitive tasks. Original paper, p. 15 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. The rows begin from the same mid-trained checkpoint and use the same target-task post-training data. The upper row removes the tactile expert; the lower row includes both its action-refinement and future-force prediction objectives. Read the two task columns separately: Pull Out Paper Cup uses the Sharpa Wave hand, and Tear Paper uses Wuji Hand 2. Each rate comes from twenty rollouts, with complete task success as the criterion. The average weights both tasks equally. The mechanism behind the lower row uses recent tactile measurements to finish denoising short action sub-chunks while reusing detached backbone features; future-force prediction supplies a training target rather than a deployed output. E09E17E23iv06

What it supports. Tactile feedback raises paper-cup success by 10 percentage points and tear-paper success by 15 points, giving a 12.5-point mean gain. These outcomes support the complete tactile extension on the evaluated contact tasks. They do not yet identify how much of the gain comes from fresh sensing versus the auxiliary force-prediction objective.

Where the evidence stops. The comparison removes or adds the entire tactile expert. It does not separate sensor information, extra model capacity, refinement cadence, and force prediction, or establish transfer between human and robot tactile interfaces.

7. Analysis & limitations

7.1 What the evidence leaves open

Author claim

The authors identify wearable tactile noise and morphology mismatch as the primary limitation: glove deformation can produce signals without external contact, and human gloves do not transfer directly across differently shaped robot hands. E18

Reader analysis

The physical studies use 20 rollouts per task without reported confidence intervals or repeated-training-seed estimates. Separate task suites test MBRL, memory, and tactile sensing, so the reported gains do not establish the performance of a single configuration combining every extension. E12E13E15E16E17

Reader analysis

Self-evolution is demonstrated through policy updates with frozen dynamics and value components. The experiments do not quantify simulator error, imagined-branch ranking accuracy, or sustained improvement across repeated real-world data-collection cycles. Shared model biases could therefore influence both imagined outcomes and their scores. E07E15E24

Reader analysis

Signing targets according to trajectory outcome can label useful prefixes of failed executions negatively. Whether this penalizes recoverable behavior is untested. The qualitative evaluator examples use executed actions and observed video, which does not directly establish reliability on generated futures. E05E24

Reader analysis

The memory comparison lacks a sliding-window baseline and measured latency or memory costs. The tactile comparison changes both refinement and prediction machinery, leaving the auxiliary prediction objective's independent effect unresolved. E08E16E17E22

Reader analysis

Qualitative evidence has internal documentation mismatches. Section 5.4 describes Screw Bulb examples that differ from Appendix F's four-task account. Appendix F also describes phone-box closure and separate-container sorting, whereas Appendix G specifies a phone stand and an orange basket. Those trajectories cannot be assumed to validate the main-suite task checkers. E25

7.2 Questions for discussion

  1. Would segment-level recovery labels improve over signing every segment by trajectory outcome? [E05, E24]
  2. How accurately do evaluator rankings on imagined branches predict real execution outcomes before and after NFT optimization? [E06, E15]
  3. Can hybrid memory approach full-history success under matched latency, and does force prediction help beyond tactile refinement alone? [E16, E17, E22]

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

The supplied title, complete author list, and identifier match the catalog; the cover identifies arXiv:2608.30237v1, dated 31 August 2026. This review uses the supplied artifact provenance. No code was inspected and no experiments were reproduced. E01

Reader analysis

Reproducing the full curriculum requires the Wan initialization, access to the mixed open and procured corpus, synchronized pose processing and retargeting, robot/alignment trajectories, and the relevant bimanual hardware. Source-specific filtering thresholds, exact target-task data quantities, and complete training compute are not specified sufficiently for an exact rerun. E10E11E19E20E21

Source description

Several settings are concrete: default action sampling uses five steps, while the memory comparison uses 50; MBRL rollout action/video sampling uses four/eight steps, β = 0.1, EMA decay 0.99, publication every ten updates, and a 600-second sample-age cutoff. Comparisons must preserve these distinct protocols rather than silently sharing one sampler configuration. E07E21E22

Reader analysis

A focused reproduction would repeat the policy-optimization/planning factorial study from one checkpoint with paired physical configurations, then test whether branch rankings agree with actual outcomes. This directly probes the proposed feedback mechanism without requiring a new large-scale pretraining run. E06E15

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Does predicted progress rank real consequences?

Proposed check: freeze the policy and generate the same candidate set for every matched task prefix. Compare learned-value ranking with random ranking, shuffled values, and an offline oracle based on realized outcomes in resettable evaluation states. Separately score real future observations and model-predicted futures to localize simulator versus evaluator error. Report pairwise ranking accuracy, calibration, success, and compute at a fixed candidate count. A learned ranking advantage that survives matched sampling would support the planning mechanism; systematic high scores for failed imagined branches would reveal exploitable model error. E05E06E15E24

Check 2: What history benefit survives a fixed inference budget?

Proposed check: compare the bounded sliding-window default, hybrid memory, and global autoregression using identical demonstrations, denoising steps, and paired Find Square/Press Button resets. Vary the delay between informative observation and required action, recording peak KV memory and control latency alongside success. Repeat with a strict per-decision compute budget. The falsifiable prediction is that full history helps increasingly after evidence leaves the recent window, but may lose its advantage when attention growth reduces feedback frequency. This would distinguish information retention from the operating budget used in the reported table. E08E16E21E22

8.3 Reading coverage

Visual audit: Original PDF pages 4, 6, 12, 14, and 15 and all six final crops were visually inspected. Figure 1 interfaces, Figure 2 attention cells, Table 2 main results, Table 3 factorial planning/MBRL comparison, Table 4 memory study, and Table 5 tactile study were read with their source protocols. Every table header and denominator context was checked; cropped graphics retain their original content. The PDF SHA-256 was recomputed and matches the base report. Other figures, external demonstrations, code execution, value calibration, and robot replication remain outside this illustrated pass.

PDF pages inspected for this edition: 4, 6, 12, 14, 15. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1. Introduction
  • 2. Related Work
  • 3. Method
  • 3.1. Overview
  • 3.2. General World Modeling
  • 3.3. Value-Guided Closed-Loop Self-Evolution
  • 3.3.1. Progress-Based Value Learning
  • 3.3.2. Planning and Policy Optimization
  • 3.4. Working Memory
  • 3.5. Tactile Expert: Tactile Refinement and Prediction
  • 4. Large-Scale Egocentric Dataset
  • 5. Experiments
  • 5.1. Experiment Setup
  • 5.2. Main Results
  • 5.3. Scaling Laws of Stereo Egocentric Human Data
  • 5.4. Model-Based Policy Improvement and Test-Time Planning
  • 5.5. Long-Horizon Context Mechanisms
  • 5.6. Effect of Tactile Feedback
  • 6. Conclusion
  • 7. Limitations and Future Work
  • Acknowledgments
  • References
  • A. Robot System and Teleoperation Stack
  • A.1. Robot Platforms
  • A.2. Teleoperation
  • B. Human–Robot Alignment Data Collection
  • C. Egocentric Data Curation and Offline Annotation
  • C.1. Pose Standardization
  • C.2. Quality-Aware Temporal Processing
  • C.3. Language Annotation
  • D. Hand Retargeting Algorithm
  • E. Model and Training Details
  • E.1. Pre-training
  • E.2. Mid-training
  • E.3. Post-training
  • F. Qualitative Value-Model Trajectories
  • G. Evaluation Tasks

Outside the original text pass

  • Review covers the supplied text, including appendix prose and figure captions; no PDF figure images were visually inspected.
  • Numerical curves were not reconstructed from extracted figure labels.
  • Separate supplemental material availability and completeness remain unverified.
  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

E01PDF p. 1, title, author block, and arXiv version lineInspect

The title and ordered author list match the catalog. The cover identifies arXiv:2608.30237v1, 31 August 2026, and lists a project-page link.

Go to primary source ↓
E02PDF pp. 4–5, Section 3.1, Eqs. (1)–(3)Inspect

The POMDP observation includes vision, proprioception, and optional tactile sensing. A shared density factors into action policy, action-conditioned future prediction, and categorical progress evaluation.

Go to primary source ↓
E03PDF pp. 5–6, Section 3.2, Eqs. (4)–(7)Inspect

Token masks enforce action-first dependencies; all modalities use the shared transformer. Independent noise levels and loss gates select policy, simulation, and evaluation supervision.

Go to primary source ↓
E04PDF p. 6, Section 3.2, supervision routing and Action-first inferenceInspect

Only curated successes activate action supervision. Ordinary deployment reads the action factor, while planning or optimization can additionally generate future video and values.

Go to primary source ↓
E05PDF p. 7, Section 3.3.1, Eqs. (8)–(9)Inspect

Successful segments receive Δt/(T−t); failed and task-irrelevant segments receive its negative. Targets are discretized for categorical value learning.

Go to primary source ↓
E06PDF p. 8, Section 3.3.2, Model-Based Planning and Eqs. (10)–(13)Inspect

Planning executes the highest-valued candidate and replans with real observations. NFT normalizes generated scores, excludes the ground-truth anchor from statistics, and updates action velocities without future-video conditioning.

Go to primary source ↓
E07PDF p. 9, Asynchronous MBRL Training; p. 25, Appendix E.3, MBRL and planningInspect

The asynchronous pipeline freezes video and value components. It uses eight generated candidates plus an anchor, a one-chunk horizon, four/eight action/video denoising steps, β=0.1, EMA decay 0.99, ten-update reference publication, and a 600-second age cutoff.

Go to primary source ↓
E08PDF pp. 9–10, Section 3.4Inspect

Sliding memory retains observed latents with eviction and RoPE rebasing; global autoregression retains complete history; hybrid memory compresses older observations. Episode packing prohibits cross-episode attention.

Go to primary source ↓
E09PDF p. 10, Section 3.5, Eqs. (14)–(16)Inspect

A tactile expert completes partially denoised actions using detached backbone caches. Auxiliary future-force prediction cannot feed into action-refinement tokens and is used only during training.

Go to primary source ↓
E10PDF pp. 10–12, Section 4 and Table 1Inspect

The corpus totals approximately 130,000 raw recording hours before filtering and annotation, combining open-source and procured monocular/stereo data. Alignment collection is separate.

Go to primary source ↓
E11PDF pp. 22–24, Appendices C–D, Eqs. (20)–(25)Inspect

Data use a 134-dimensional bilateral pose representation, quality-aware segmentation, and language annotation. Human keypoints are retargeted to 20-DoF Wuji joints; target-robot data retain native recorded joints.

Go to primary source ↓
E12PDF p. 12, Section 5.1Inspect

Physical evaluations use 20 configurations per task and the simulation memory study uses 25. Each configuration receives one rollout; all task predicates must be met, and averages weight tasks equally.

Go to primary source ↓
E13PDF p. 12, Table 2, all method rows; p. 13, Section 5.2 ResultsInspect

Midtrain-SFT reports 100/70/100/90/60% across the five tasks and 84% average. Pretrain-SFT averages 51%; WAN-SFT and π0.5 report zero throughout.

Go to primary source ↓
E14PDF p. 13, Section 5.3, Eqs. (17)–(18)Inspect

Nested 2k/4k/10k/20k raw-hour subsets share a trajectory-disjoint validation set. Best-epoch normalized human-action MSE is fitted by 0.101−0.005 ln D.

Go to primary source ↓
E15PDF p. 14, Section 5.4 and Table 3, all rowsInspect

Baseline, planning, MBRL, and combined variants report Put Phone/Multi-Finger success of 60/70, 65/70, 65/80, and 70/80%, respectively.

Go to primary source ↓
E16PDF p. 14, Section 5.5 and Table 4, simulation and real-robot rowsInspect

Hybrid/global averages are 52/78% in simulation and 25/57.5% on the robot. The comparison includes Find Square and Press Button but no sliding-window row.

Go to primary source ↓
E17PDF p. 15, Section 5.6 and Table 5Inspect

The complete tactile variant improves cup extraction from 65 to 75% and paper tearing from 55 to 70%, using matched starting checkpoint and post-training data.

Go to primary source ↓
E18PDF pp. 15–16, Section 7Inspect

The stated limitations concern noisy glove signals from deformation and the inability to transfer one glove geometry directly between human and robot embodiments.

Go to primary source ↓
E19PDF pp. 21–22, Appendices A–BInspect

The paper describes Tianji dual-arm systems with Wuji, Wuji Hand 2, or Sharpa Wave hands and platform-specific teleoperation. Human alignment records stereo views and tracker-relative wrist poses without commanding a robot.

Go to primary source ↓
E20PDF p. 24, Appendix E.1Inspect

Wan 2.2-TI2V-5B initializes video-only training for 500K and 340K steps, followed by 450K joint stereo video–action steps. Stage 2 uses 12 latent frames and two-frame chunks.

Go to primary source ↓
E21PDF p. 25, Appendix E.2 and E.3, Sliding-window SFTInspect

Mid-training uses over 100 hours, simulation/evaluation fractions 0.10/0.05, 201 value bins, and value weight 0.05. Default SFT uses an eight-frame window, two-frame chunks, a two-chunk cache, and five deployment denoising steps.

Go to primary source ↓
E22PDF p. 25, Appendix E.3, Global autoregression and Hybrid working memoryInspect

Hybrid memory uses two anchor frames, four recent frames, and eight memory tokens per latent frame. Both long-context variants use 10K training steps and 50 evaluation denoising steps.

Go to primary source ↓
E23PDF pp. 25–26, Appendix E.3, Tactile expertInspect

The 30-layer expert uses width 128. A 48-action chunk at 30 Hz is divided into eight six-action sub-chunks after denoising to σc=0.2. Forces are sampled at 90 Hz; auxiliary prediction weight is 0.1.

Go to primary source ↓
E24PDF pp. 26–27, Appendix F, Protocol and trajectory descriptionsInspect

Qualitative values condition on executed actions and observed video, without the eventual outcome as input. Values are relative-progress scores, explicitly not calibrated success probabilities.

Go to primary source ↓
E25PDF p. 14, Section 5.4, Qualitative value trajectories; pp. 26–27, Appendix F; p. 30, Appendix GInspect

Section 5.4 references Screw Bulb trajectories, while Appendix F discusses Cut Reeds, Make Tea, Multi-Finger, and Put Phone. Appendix F's box-closure and separate-container requirements differ from Appendix G's phone-stand and orange-basket descriptions.

Go to primary source ↓
iv01PDF p. 4, Figure 1; original page and final crop visually inspected at 200 DPIInspect

Figure 1 was visually inspected: shared Video-Action Causal DiT blocks sit beside a tactile expert; its right chain explicitly labels Policy (WAM), Simulator (AC-WM), Evaluator (VM), RL Loss, and Maximum. Candidate action, frame, and value branches are retained in the crop.

Go to primary source ↓
iv02PDF p. 6, Figure 2; original page and final crop visually inspected at 200 DPIInspect

Figure 2 was visually read row by row for the first action/video/value block and checked against later blocks. The right A1 row masks Z1 and U1; Z1 reads A1; U1 reads A1 and Z1; other queries cannot read U1. The left joint mask permits A1/Z1 mutual attention and leaves value groups inactive.

Go to primary source ↓
iv03PDF p. 12, Table 2; original page and final crop visually inspected at 200 DPIInspect

Table 2 was visually inspected, preserving all five tasks and the average column. π0.5 and WAN-SFT are zero throughout; Pretrain-SFT is 60/35/90/55/15, mean 51; Motus2 Midtrain-SFT is 100/70/100/90/60, mean 84. Source setup states twenty configurations per task and matched SFT.

Go to primary source ↓
iv04PDF p. 14, Table 3; original page and final crop visually inspected at 200 DPIInspect

Table 3 was visually inspected: baseline 60/70/65.0; planning 65/70/67.5; MBRL 65/80/72.5; both 70/80/75.0 for Put Phone/Multi-Finger/average. Caption and surrounding protocol specify the same checkpoint and twenty trials per task.

Go to primary source ↓
iv05PDF p. 14, Table 4; original page and final crop visually inspected at 200 DPIInspect

Table 4 was visually inspected: Simulation hybrid/global results are 64/40/52 and 84/72/78; Real robot results are 30/20/25.0 and 65/50/57.5. The original table contains no sliding-window row. Appendix E.3 specifies matched long-history training and fifty denoising steps.

Go to primary source ↓
iv06PDF p. 15, Table 5; original page and final crop visually inspected at 200 DPIInspect

Table 5 was visually inspected with all task and average headers: without tactile, 65/55/60.0; with tactile, 75/70/72.5. The source protocol identifies matched checkpoint/data, twenty trials per task, and the complete refinement-plus-prediction extension.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.