Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

N₀-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: NeoteAI Team & Fudan TEAI Team

Affiliations: NeoteAI; Institute of Trustworthy Embodied AI, Fudan University

Source: 2607.23783 ↗ · Project page ↗ · Catalog record

Reading: 113 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: N₀-TWAM predicts scene and contact before acting, then adds current-touch feedback; its average manipulation gains come with sensor-dependent representations and task-specific deployment choices. e-probleme-cascadee-maske-observede-schedulere-univtace-reale-ablatione-pose

At a glanceWhat to know
Research problem
Source description

A visually aligned gripper can still slip, miss an object or jam during insertion. The paper asks how to anticipate these hidden contact states while retaining the scene prediction of a video world-action model. e-problem

Core mechanism
Source description

The proposed combination is a three-expert generative backbone, predicted and observed tactile pathways, and contact-event scheduling for multi-stage tasks. e-cascadee-observede-scheduler

A key reported resultReal-robot contact-rich manipulation: 46.3

Macro success rate (%). Eight InTac S1 tasks on Flexiv/PiPER; 20 randomized trials per task; operator-judged success.

π0.5 30.0; LingBot-VA 21.9; FastWAM 14.4. The reported average gain over π0.5 is 16.3 percentage points. Cup Stacking and Bag Packing favor π0.5; Making Lemon Tea reaches only 15%. e-protocole-real

Reading caution
Reader analysis

Real-robot per-task uncertainty is substantial: the authors estimate up to approximately 11% binomial standard error. Selected tactile/flow images illustrate plausibility but do not quantify forecasting accuracy; simulation pathway ablations do not isolate the physical scheduler. e-reale-qualitativee-ablatione-scheduler

Core contributions

  • Source description

    The proposed combination is a three-expert generative backbone, predicted and observed tactile pathways, and contact-event scheduling for multi-stage tasks. e-cascadee-observede-scheduler

  • Author claim

    The authors claim large-scale tactile-native pre-training is central; the reduced-data and tactile-pathway ablations support average gains within the evaluated protocols. e-datae-ablation

Figure 2. Future touch and current touch reach action through different pathways. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read upward from the raw video and tactile inputs into their encoded tokens, then across the shared self-attention bar. Green and blue experts generate the future scene and contact; the action stream consumes the committed pair before producing commands. On the right, observed tactile takes a shorter route through an encoder and cross-attention near the action output. That route carries current evidence rather than a denoising target. The bar represents communication at each transformer layer, not completion of the entire inference procedure. Section 2.3 still requires iterative future prediction followed by iterative action denoising. On real robots the observed branch reads a force-space representation. e-cascadee-observede-streaming

What it supports. The architectural claim is more specific than simply adding another sensor input: touch is both a generated intermediate forecast and an observed action condition. Private expert weights separate modality capacity, while shared attention integrates the backbone. The model can therefore consume predicted contact without requiring a separate standalone tactile predictor.

Where the evidence stops. The drawing explicitly routes its text arrow only into video; its caption also specifies language cross-attention into action and excludes tactile. Follow that caption for routing. The main simulation observed branch uses a scratch-trained latent encoder, not the real-robot force encoder.

2. Motivation

2.1 The problem and the proposed response

Source description

A visually aligned gripper can still slip, miss an object or jam during insertion. The paper asks how to anticipate these hidden contact states while retaining the scene prediction of a video world-action model. e-problem

2.2 What this reading follows

A gripper can look correctly positioned while feeling the wrong contact. N₀-TWAM gives a video world-action model a tactile future to predict and a separate tactile present to read. Follow the architecture and attention mask first: they explain how three experts share information while action waits for a forecast. The contact-event diagram then shows how touch also advances a sub-task queue. Finally, compare simulation success, physical execution and pathway ablations. These results support the combined system, while task-level reversals, absolute-pose exceptions and small physical trial counts constrain how broadly its advantage should be interpreted. e-probleme-cascadee-maske-observede-schedulere-univtace-reale-ablatione-pose

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryWAMs
ArchitectureDual-system
Prediction paradigmIDM
QuadrantQ4 · Dual-system × IDM

3.1 Evidence-based assessment

Conflicts with the recorded classification

Reader analysis

WAM, tactile modeling and event-based long-horizon control are supported. The predict-then-act factorization supports an IDM-style mechanism: action conditions on a predicted future. At backbone level, however, this is one integrated MoT with shared attention at every layer, not a separately coupled world model and standalone inverse-dynamics policy. Private expert weights and sequential denoising alone do not establish Dual-system. The external sub-task scheduler is distinct from that architecture judgment; the catalog snapshot is preserved. e-cascadee-maske-observede-scheduler

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Language instruction; multi-view RGB and tactile observation history; past actions and chunk-start pose [e-cascade, e-action]
  • Current tactile readings, represented differently for real robots and main simulation experiments [e-observed]
  • Future video and tactile latents; decoded end-effector and gripper action chunks [e-cascade, e-action]

4.2 Equations and their role

pθ(x1:Kv,x1:Kt,x1:Ka∣c)=∏k=1Kpθ(xkv,xkt∣X<k,c) pθ(xka∣xkv,xkt,X<k,c)p_\theta(x^v_{1:K},x^t_{1:K},x^a_{1:K}\mid c)=\prod_{k=1}^{K}p_\theta(x^v_k,x^t_k\mid X_{<k},c)\,p_\theta(x^a_k\mid x^v_k,x^t_k,X_{<k},c)
Eq. (1): c is the instruction, k indexes chunks through K, and v/t/a identify video, tactile and action streams. X below k collects past streams. The second factor conditions action on the predicted future pair. e-cascade
x^k,jm=(1−σk,jm)xk,jm+σk,jmϵk,jm,uk,jm=ϵk,jm−xk,jm\hat{x}^{m}_{k,j}=(1-\sigma^{m}_{k,j})x^{m}_{k,j}+\sigma^{m}_{k,j}\epsilon^{m}_{k,j},\qquad u^{m}_{k,j}=\epsilon^{m}_{k,j}-x^{m}_{k,j}
Eq. (2): m selects modality, j its temporal group, x the clean target, sigma the sampled group noise level and epsilon Gaussian noise. The model regresses velocity u under the weighted loss of Eq. (3). e-flow
Δit=xit−x0t,xit=x0t+Δit\Delta_i^t=x_i^t-x_0^t,\qquad x_i^t=x_0^t+\Delta_i^t
Eq. (6): future tactile latent x at step i is represented by its residual Delta from initial latent x at step zero, emphasizing contact changes. e-touch

5. Method in detail

5.1 Learn a tactile future before adding the tactile present

Source description

A tactile sensor produces a gel image, so the model can encode it with the same frozen video VAE used for the scene. Future touch is represented as a change from its initial latent, concentrating the target on contact onset, deformation and release. Pre-training predicts that residual alongside video and action, while the observed-touch branch stays disabled. This ordering is intended to make foresight useful before a current-contact shortcut becomes available. Post-training then opens the observed branch. On real robots its force estimator stays frozen while the NeoForce encoder adapts under action loss, and a zero-initialized cross-attention output initially leaves the pretrained policy unchanged. The main simulation experiments use a different, scratch-trained latent encoder at that same port, so their results should not automatically be credited to pretrained force-space features. e-touche-flowe-observede-pretraine-posttrain

Figure 3. The mask makes future vision and touch available before action. Original paper, p. 5 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Rows are queries and columns are the keys they may read. Within each chunk, locate the noisy video and tactile rows: their solid green and blue cells allow mutual co-generation. The noisy action row instead reaches the hatched clean video and tactile columns of its chunk, alongside its own noisy action block. Moving into the second chunk adds access to clean history from the first. The empty upper-right region blocks future chunks. Each displayed video/tactile block collapses two latent frames, so this is a schematic of the temporal ordering rather than a complete token-resolution mask. These access rules agree with the three mask rules in Section 2.1. e-maske-cascadee-streaming

What it supports. This one-way dependence explains why inference can cache video/tactile keys and values after prediction. Subsequent action-denoising steps do not change those forecasts through current-action feedback, and only the lightweight action expert must repeat. The mask implements an IDM-style action dependency inside shared transformer attention.

Where the evidence stops. Clean history during training is ground truth, with occasional light visual/tactile re-noising; at inference, generated entries can be imperfect until observations replace them. The schematic establishes permitted information flow, not forecast accuracy or a measured latency improvement.

5.2 Follow one chunk through prediction, execution and confirmation

Source description

At a chunk boundary, history contains the scene, contact and actions already committed. The model first denoises a future scene/contact pair, then caches its attention keys and values. Because the mask blocks current-action feedback into those prediction streams, action denoising can reuse the cached pair while repeating only the slim action expert. The robot executes the decoded chunk while the next is predicted; arriving camera and tactile readings replace predicted history. Current-touch conditioning can refresh between these chunk predictions. For a multi-stage task, another clock sits above this loop: predicted contact prepares a prompt advance and observed contact confirms it. These three time scales are described as deployment mechanisms; the paper does not provide measured rates. It also specifies different training and evaluation action horizons without fully explaining their reconciliation. e-maske-streaminge-schedulere-posttraine-protocol

Figure 5. Contact events provide training boundaries and an execution clock. Original paper, p. 9 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with the paired traces on the left. A gripper-aperture dip can occur in both successful and failed grasps, whereas tactile magnitude rises when contact actually loads the fingers. The colored intervals become clips with local instructions instead of one unchanging episode instruction. On the right, the queue supplies the active prompt to N₀-TWAM, which executes before feedback advances the queue. The feedback box summarizes a more specific rule in the accompanying prose: predicted contact initiates an advance, but an observed onset or release must confirm it before the prompt switch is committed. The next instruction then re-seeds streaming context. e-schedulere-ablatione-real

What it supports. Touch has a third role beyond prediction and immediate action conditioning: it marks stage transitions. This is particularly relevant when a limited observation window cannot disambiguate which part of a long task is underway. The diagram explains the intended coordination of demonstration segmentation and inference-time prompt scheduling.

Where the evidence stops. The traces are explanatory rather than a calibrated detector specification: event thresholds and confirmation timing are not given. The paper does not separately ablate this scheduler, and a local contact event alone does not establish successful completion of an entire task.

5.3 Separate useful contact prediction from a useful training target

Reader analysis

Reader interpretation: the pathway ablations are stronger mechanism evidence than the headline comparison against vision-only policies, because they retain much of the same model and adaptation setup. Even so, zeroing the tactile prediction target changes the learning problem as well as the information eventually presented to action. A lower success rate could reflect lost training regularization, worse joint scene features, less useful future contact at inference, or a combination. The paper's qualitative forecast examples cannot distinguish those explanations. Keep action parameterization fixed when testing them: the absolute-pose analysis shows that this configuration alone can substantially change precision-task performance. The physical results also leave room for a separate scheduler contribution. A discriminating reproduction should therefore test forecast use and event timing with controlled interventions rather than treating the aggregate benchmark margin as their individual effect. e-ablatione-qualitativee-posee-reale-scheduler

5.4 Training and inference

During training

Source description

NeoData exceeds 30,000 hours across six embodiments and 450 tasks, with tactile on a large fraction of episodes. Cached training windows contain 33 latent frames with stride 24. Full collection statistics are deferred to another report. e-data

Source description

Pre-training disables observed conditioning and fits video/tactile/action flow velocities with equal modality weights plus SNR and validity weighting. LingBot-VA initializes video; the other experts start from scratch. AdamW uses learning rate 1e-4, weight decay 0.1, batch 512 and 30,000 steps on 128 H800 GPUs with FSDP2/bf16; 3% of each task is held out. e-flowe-pretraine-observed

Source description

Post-training fine-tunes all transformer parameters and the observed encoder, retaining all three denoising losses. It normally runs 1000 AdamW steps at 1e-4 with four-step accumulation and clipping at 2.0. Language/tactile dropout each use probability 0.1; missing tactile removes tokens. Evaluation leaves tactile guidance at w=1. Appendix C shows an UMI-related video-loss increase without a comparable tactile-loss jump. e-posttraine-training-dynamics

During inference

Source description

Denoise future vision/touch, cache their keys/values, then denoise action using only the slim action expert. Execute asynchronously while predicting the next chunk; incoming sensor observations replace predicted history. Current-touch conditioning can refresh between chunk predictions. e-streaming

Source description

Long demonstrations are segmented at contact events, with aperture fallback and human checks. Sub-task prompts condition each clip. At execution, a predicted event initiates an advance, observed contact confirms it, and the scheduler re-seeds context with the next prompt. e-scheduler

5.5 Implementation flow

  1. Encode synchronized streams

    A frozen video VAE encodes scene and gel-camera tactile streams; frozen umT5 encodes language. Tactile targets are residuals from the initial latent, with sensor-ID embeddings. The shared latent format supports attention across modalities. e-datae-touch

  2. Predict before acting inside one backbone

    Video, tactile and action experts retain private weights but exchange information through shared attention at every layer. The 30-layer, 7.16 B backbone uses private widths 3072/1024/1024. Its approximately 15 B full-width comparator is an estimate excluding frozen encoders. e-cascadee-capacity

  3. Control information with a causal mask

    Same-position noisy vision and touch co-generate. Action follows their clean predictions; future chunks remain inaccessible. Training teacher-forces history and lightly re-noises visual/tactile history with probability 0.5. This ordering implements the prediction-to-action dependency. e-mask

  4. Add observed touch during adaptation

    On real robots, a frozen estimator converts tactile images to force maps; a fine-tuned NeoForce encoder feeds action cross-attention through a zero-initialized output. Main simulation instead trains a lightweight tactile-latent encoder from scratch. The observed branch is conditioning, while predicted touch remains a denoising target. e-observed

  5. Decode the robot interface

    Each arm has position, 6-D orientation and gripper targets, totaling 20 dimensions bimanually; absent-arm losses are masked. Default position/orientation targets subtract the chunk-start coordinates, while gripper commands remain absolute. Four precision tasks use absolute poses instead. e-actione-pose

6. Experiments & results

N₀-TWAM generates future camera and tactile observations, then conditions robot actions on that forecast and current touch. Its integrated transformer cascade improves average contact-rich task success, but transfer varies across tasks and sensors.

6.1 Read the original evidence

Table 2. The strongest simulation average is accompanied by task-level exceptions. Original paper, p. 12 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the task columns before the final average. Rows are grouped by model family, and boldface marks the highest reported success in each column, including ties. The bottom row is N₀-TWAM; its average is compared against all listed methods, not only the world-action group. The evaluation uses 100 randomized trials per task and an equal-task macro average. These are the main clean-tactile simulation conditions, where the observed tactile branch is trained from scratch. Keep Table 5 alongside this table: the authors adopt absolute end-effector poses for Lift Can, Lift Bottle and Put Bottle in Shelf, despite the default delta-action description. e-univtace-protocole-observede-realisme-posee-ablation

What it supports. N₀-TWAM reaches 84.5% average success against the strongest listed baseline, InternVLA-A1 at 67.1%, a 17.4-percentage-point difference. The advantage is not universal: Lift Bottle is 58% for N₀-TWAM versus 100% for π0.5. The average establishes broad gains within this benchmark rather than dominance on every contact task.

Where the evidence stops. Baseline rows differ in pre-training and tactile access, so this table alone cannot attribute the margin to tactile foresight. Task-specific pose choices also matter. The within-model pathway ablations provide a more focused comparison, though they still change the training information.

Figure 7. Physical execution improves on average, while long tasks remain difficult. Original paper, p. 14 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Use the legend to compare the blue N₀-TWAM bar with the three baselines within each task. The suite spans four single-arm Flexiv tasks and four dual-arm PiPER tasks, using InTac S1 tactile sensors. Each percentage comes from 20 randomized trials, with success judged against a fixed task criterion by the operator. Thus one additional successful trial changes a task score by five percentage points. The caption reports the macro average across these eight groups. Read the low bars as carefully as the largest gaps: Making Lemon Tea, Cup Stacking and Bag Packing qualify the overall story of contact-sensitive improvement. e-protocole-reale-schedulere-ablation

What it supports. The reported macro success is 46.3%, compared with 30.0% for π0.5, 21.9% for LingBot-VA and 14.4% for FastWAM. N₀-TWAM reaches 70% on Bottle Standing and Socket Plugging, but only 15% on Making Lemon Tea. π0.5 remains ahead on Cup Stacking and Bag Packing.

Where the evidence stops. These are executed robot trials, but only 20 per task. The authors note a binomial standard error of up to approximately 11%; small per-task gaps are consequently uncertain. No scheduler-only real-robot ablation separates contact prediction from stage-transition logic.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
UniVTAC contact-rich manipulation

Eight simulation tasks; 100 randomized trials each; benchmark success checks; task-specific post-training.

84.5

Macro success rate (%)

InternVLA-A1 67.1; FastWAM 48.0; LingBot-VA 31.4.

Highest reported average, 17.4 percentage points above the strongest comparator. This includes task-specific absolute-pose settings, and tactile access differs from vision-only baselines. e-protocole-univtace-pose

NeoSim contact-rich manipulation

Twelve in-house simulation tasks, four single-arm and eight dual-arm; 100 trials each.

49.4

Macro success rate (%)

π0.5 45.8; LingBot-VA 32.1.

A smaller average advantage; Unplug & Plug Charger and Place Gears remain at zero. FastWAM was not evaluated here. e-protocole-neosim

Real-robot contact-rich manipulation

Eight InTac S1 tasks on Flexiv/PiPER; 20 randomized trials per task; operator-judged success.

46.3

Macro success rate (%)

π0.5 30.0; LingBot-VA 21.9; FastWAM 14.4.

The reported average gain over π0.5 is 16.3 percentage points. Cup Stacking and Bag Packing favor π0.5; Making Lemon Tea reaches only 15%. e-protocole-real

Real-robot distribution shifts

Novel objects on Bottle Standing/Bag Packing, novel placements on Cup Stacking, visual perturbations on Fruit Collection.

65 / 45 / 45; reported axis average 51.7

Success rate (%) by shift axis

π0.5: 80 / 45 / 25, average 50.0.

Touch is most helpful under visual perturbation; the model does not lead on novel objects. The object-column aggregation across its two tasks is not detailed. e-generalization

Tactile pathways and pre-training scale

Matched downstream tactile ablations on both simulation suites; 20%-data pre-training tested only on UniVTAC.

Full / no predicted / no observed: UniVTAC 84.5 / 71.8 / 70.5; NeoSim 49.4 / 41.1 / 29.6.

Macro success rate (%)

20%-pretrain UniVTAC: 65.4.

Both tactile paths help on average. Predicted-path removal zeros targets before noising, preserving layout and loss; it is not deletion of the tactile expert. e-ablation

Action parameterization on precision tasks

Lift Can, Lift Bottle, Put Bottle in Shelf and Grasp Chip; Section 4.6 task-specific settings.

Absolute 82.5

Mean success rate (%)

Delta 50.0.

Absolute poses are adopted for these tasks. This large configuration effect must be preserved when comparing or reproducing main results. e-pose

Simulated tactile rendering and NeoForce

UniVTAC training and evaluation with clean or gel-rendered tactile; contact physics unchanged.

Clean 84.5; gel-rendered 82.4; gel-rendered + NeoForce 88.1.

Macro success rate (%)

NeoSim single-arm: 63.8 / 58.5 / 64.8.

These are adapted simulation conditions, not zero-shot physical transfer. NeoSim dual-arm NeoForce results are absent. e-realism

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Figure 9. The two tactile roles contribute differently across simulation suites. Original paper, p. 17 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Compare variants within each benchmark before reading the rightmost average, which is the mean of the two suite averages rather than a pooled twenty-task score. Orange appears only for UniVTAC because the reduced-data checkpoint was not evaluated on NeoSim. Purple denotes the predicted-tactile information ablation: targets are zeroed before noising while token layout, mask and diffusion loss are retained. Pink removes the observed-conditioning branch. The tactile variants share their starting checkpoint and downstream training setup; the orange variant instead changes pre-training data. Appendix A provides the per-task rows needed to assess whether these average losses are uniform. e-ablation

What it supports. UniVTAC drops from 84.5% to 71.8% without informative predicted tactile and to 70.5% without observed conditioning; the reduced-data variant reaches 65.4%. NeoSim falls from 49.4% to 41.1% and 29.6%, respectively. These comparisons support contributions from both pathways, with a larger observed-path effect on NeoSim.

Where the evidence stops. Average gains do not imply every task benefits: Appendix A reports Cup Handover increasing from 14% to 65% without predicted tactile. Zeroing the prediction target also changes learning, so this comparison does not isolate the inference-time value of an otherwise identically trained forecast.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

Real-robot per-task uncertainty is substantial: the authors estimate up to approximately 11% binomial standard error. Selected tactile/flow images illustrate plausibility but do not quantify forecasting accuracy; simulation pathway ablations do not isolate the physical scheduler. e-reale-qualitativee-ablatione-scheduler

Reader analysis

The paper supplies no measured latency/control-rate benchmark, sensor-failure success table or event-detector thresholds. Optional reduced precision and claims of a sensor-rate reflex therefore lack a complete deployment specification. Broader sensor coverage and longer horizons remain author-identified future work. e-streaminge-posttraine-generalizatione-schedulere-future

7.2 Questions for discussion

  1. Does predicted tactile still help when forecast conditioning is perturbed only at inference, while the trained auxiliary objective is held fixed? [e-ablation]
  2. How much long-horizon improvement comes from sub-task labeling versus prediction-assisted event timing? [e-scheduler, e-real]

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Required resources include the LingBot-VA initialization, NeoData, frozen Wan/umT5 encoders, tactile calibration and task demonstrations. NeoForce separately uses 20,000 demonstrations over 30 tasks and 100,000 pre-training steps on eight A100 GPUs. The paper promises code/checkpoints rather than establishing release. e-datae-pretraine-future

Open question

Resolve the 24-step post-training versus 16-step evaluation horizon, task-specific absolute poses, denoising schedules, post-training data counts, gripper quantiles and contact thresholds before replication. Appendix C adds pure-real continuation after UMI mixing beyond Section 4.1; exact curriculum timing and split granularity remain unclear. e-posttraine-protocole-posee-streaminge-datae-schedulere-training-dynamics

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Does the trained action expert use the generated tactile future?

Reader-proposed check, not performed: on UniVTAC Lift Bottle and Put Bottle in Shelf, evaluate the same full checkpoint with its normal tactile forecast versus a temporally shuffled generated tactile forecast supplied to action. Hold video forecasts, observed touch, history, token layout, absolute-pose settings and action horizon fixed; perturb only the predicted-tactile conditioning passed to the action loop. Use paired initial states and noise seeds, at least the paper's 100 trials per task, and uncertainty intervals. Compare this inference intervention with the paper's target-zeroing training ablation. A reliable loss under both interventions supports direct forecast use; a loss only after altered training weakens that explanation. Report distribution-shift effects of shuffling rather than treating the intervention as perfectly selective. e-ablatione-maske-streaminge-posee-protocol

Check 2: Does forecast-assisted scheduling beat observed events alone?

Reader-proposed check, not performed: keep the same post-trained policy, sub-task clips, instruction queue and action decoding on Making Lemon Tea and Hanoi Tower. Compare fixed-duration switching, observed-contact-only switching and predicted-trigger plus observed-confirmation switching. Predefine detector thresholds and confirmation windows on separate development trials, then freeze them. Include matched normal trials and controlled empty grasps or delayed releases. Measure full-task success, premature switches, repeated stages and transition delay, with repeated randomized trials and uncertainty intervals. The prediction-assisted scheduler should reduce delay without increasing unconfirmed transitions; otherwise its extra forecast trigger has not demonstrated value beyond observed tactile events. e-schedulere-protocole-real

8.3 Reading coverage

Visual audit: The title/author/version page, all twelve figures and all ten tables were visually inspected. The six final crops were each viewed; the narrow attention-mask crop was rendered at 400 DPI and adjusted to retain the complete Query label. Figure 2 routing was cross-checked against its caption, Figure 3 against the mask rules and Eq. (1), and Figure 5 against the predicted-trigger/observed-confirmation prose. Declared pages include training hardware and settings, evaluation protocols, absolute-pose exceptions, generalization, tactile-rendering comparisons, all empirical appendices and every supporting method page used here. All nine text chunks, including references, were read; reference-only pages and p. 19 were not visually inspected. No separate supplements, code or external resources were inspected, and no proposed check was run.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 27, 28. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1 Introduction
  • 2 Model, including 2.1–2.3.3
  • 3 Data, including 3.1–3.2
  • 4 Training and Experiments, including 4.1–4.6
  • 5 Related Work
  • 6 Conclusion and Future work
  • Contributors
  • References
  • Appendix A: Per-task ablation results
  • Appendix B: Per-task tactile-realism results
  • Appendix C: Per-modality training dynamics

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.
  • All nine supplied text chunks were read. The extraction-only image limitation was addressed by inspecting all twelve figures and all ten tables in the retained PDF.
  • Identity notes: reviewed arXiv:2607.23783v1, dated 26 July 2026 in the margin and 25 July internally. The catalog submission date matches the margin. Team author names agree after catalog name-order normalization. Supplied observedTitle uses N_0 and World-Action where the title page typesets N₀ and World Action; no different revision or edition was substituted.
  • No separate supplements, linked foundation/data report, code, checkpoints or external references were inspected; no experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title block and arXiv margin; p. 20, ContributorsInspect

The title identifies N₀-TWAM and credits NeoteAI Team & Fudan TEAI Team. The margin specifies arXiv:2607.23783v1, 26 Jul 2026; the internal date is July 25, 2026. The printed title uses “World Action”; the supplied observedTitle normalizes this to “World-Action” and the model name to N_0. Contributors are listed by role on p. 20.

Go to primary source ↓
e-problemPDF p. 1, Introduction; p. 3, Model openingInspect

Contact states decisive for manipulation can be visually ambiguous. The proposed model forecasts scene and contact before generating action, with a separate current-touch input.

Go to primary source ↓
e-cascadePDF p. 3, Section 2.1, Eq. (1); p. 4, Figure 2 and Eqs. (4)–(5)Inspect

The joint distribution factors into future video/tactile prediction followed by action conditioned on that pair and history. Three private-weight experts interact through one shared self-attention at each layer. Figure 2 caption specifies language cross-attention into video and action, but not tactile.

Go to primary source ↓
e-flowPDF p. 4, Section 2.1, Eqs. (2)–(3)Inspect

Each modality has two temporal groups per chunk. Independent group noise levels interpolate clean targets with Gaussian noise; the target velocity is noise minus clean data. Weighted squared velocity error combines modality weights, SNR weighting and absent-arm masking.

Go to primary source ↓
e-capacityPDF p. 5, Table 1 and Parameter-efficient expertsInspect

The 30-layer backbone has private widths 3072/1024/1024 and video/action/tactile parameters 5.00/1.13/1.03 B, totaling 7.16 B. Shared attention width is 3072. The approximately 15 B all-full-width comparator is estimated, and frozen VAE/text encoders are excluded.

Go to primary source ↓
e-maskPDF p. 5, Figure 3 and Diffusion-forcing cascade; p. 6, Section 2.2.1 final paragraphInspect

Rows are queries and columns keys, with noisy and clean copies. Same-position noisy vision and touch co-generate; later action reads clean predicted vision/touch. Future chunks are masked. Training uses ground-truth history and lightly re-noises video/tactile history with probability 0.5; action history remains clean.

Go to primary source ↓
e-touchPDF p. 6, Section 2.2.1, Eq. (6)Inspect

Tactile images use the video VAE and (1,2,2) patches with sensor IDs for up to four sensors. The target is future tactile latent minus the initial tactile latent. Tactile and video share temporal positions, while the tactile expert has private weights and no text cross-attention.

Go to primary source ↓
e-observedPDF pp. 6–8, Section 2.2.2, Eq. (7) and Figure 4Inspect

On real robots a frozen estimator produces three-axis force maps and contact masks; the NeoForce encoder is warm-started and fine-tuned with action loss. Observed tokens enter action through a zero-initialized cross-attention output. Main simulation uses a scratch-trained current-tactile-latent branch instead. Observed conditioning is off in pre-training and enabled in post-training; a force reconstruction anchor is optional.

Go to primary source ↓
e-actionPDF p. 6, Action representation; p. 10, Unified action spaceInspect

Each arm contributes three position coordinates, six orientation coordinates and one gripper command. Bimanual actions have 20 dimensions; the absent arm is masked. Default targets subtract the chunk-start position and 6-D orientation coordinate-wise, keep gripper commands absolute, normalize per robot and clip gripper quantiles.

Go to primary source ↓
e-streamingPDF pp. 8–9, Sections 2.3.1–2.3.2Inspect

Future vision/touch are denoised first, their per-layer keys/values cached, and only the slim action expert is repeated during action denoising. Arriving observations replace predicted history. Prediction overlaps execution, and observed conditioning can refresh at sensor rate. Reduced-precision deployment is optional; no measured latency or control-rate table is supplied.

Go to primary source ↓
e-schedulerPDF p. 9, Section 2.3.3 and Figure 5; p. 10, Contact-event stage segmentationInspect

Contact changes segment demonstrations, with gripper-aperture fallback and human checks. Short sub-task instructions condition clips. During execution predicted contact triggers an early advance, observed onset/release confirms it before commitment, and the next prompt re-seeds context. Figure 5 shows grasp closure without tactile rise as a failed grasp.

Go to primary source ↓
e-dataPDF p. 10, Sections 3.1–3.2Inspect

NeoData exceeds 30,000 hours across six embodiments and 450 tasks; a large fraction, not all episodes, carries synchronized tactile. Collection and full statistics are referred to another report. Frozen VAE and umT5 encoding is cached. Windows contain 33 latent frames with stride 24, from 387 raw frames at 30 fps subsampled to 129 at 10 fps; incomplete required-modality windows are filtered.

Go to primary source ↓
e-pretrainPDF pp. 10–11, Section 4.1, Pre-training and Tactile encoderInspect

The video expert is warm-started from LingBot-VA, with scratch action/tactile experts. Pre-training uses equal modality weights, AdamW at 1e-4 with weight decay 0.1, 30,000 steps, about 2.2 epochs, effective batch 512, 128 NVIDIA H800 GPUs, FSDP2/bf16 and a 3% per-task validation holdout. Section 4.1 describes real-robot training followed by roughly 60% UMI mixing. NeoForce uses a DINOv2-initialized ViT-B, separately pretrained on 20,000 demonstrations across 30 tasks for 100,000 steps on eight A100 GPUs at 2e-5.

Go to primary source ↓
e-posttrainPDF pp. 11–12, Section 4.1, Post-training, Condition dropout and OptimizationInspect

Post-training fine-tunes all transformer parameters and the observed encoder, preserves three denoising objectives, enables observed conditioning and specifies 24 action steps, 12 per latent frame. Language and tactile dropout each have probability 0.1; tactile dropout removes tokens rather than zeroing them. Evaluations use tactile guidance weight w=1. AdamW uses 1e-4, weight decay 0.1, bf16, gradient clipping 2.0, one sample per GPU, four-step accumulation and normally 1000 optimizer steps.

Go to primary source ↓
e-protocolPDF pp. 12–13, Section 4.2; p. 16, Figure 8Inspect

Success is macro-averaged over eight UniVTAC, twelve NeoSim or eight real tasks. Simulation has 100 trials per task; real robots have 20, with randomized initial states and operator-judged fixed criteria. The execution description specifies 16-step chunks and a 1.5-times-demonstration-length budget. Real experiments use InTac S1 sensors, four Flexiv and four PiPER tasks; six tasks come from NeoReal.

Go to primary source ↓
e-univtacPDF p. 12, Table 2, all rows, especially Avg., Lift Bottle and N₀-TWAM rowInspect

UniVTAC average success is 84.5% for N₀-TWAM, 67.1% InternVLA-A1, 56.1% StarVLA-α, 54.3% Xiaomi-Robotics-0, 41.4% π0.5, 48.0% FastWAM, 31.4% LingBot-VA and 16.5% GigaWorld-Policy. N₀-TWAM has 58% Lift Bottle success versus π0.5 at 100%.

Go to primary source ↓
e-neosimPDF p. 13, Table 3, Average and task rowsInspect

NeoSim averages are 49.4% for N₀-TWAM, 45.8% π0.5 and 32.1% LingBot-VA. N₀-TWAM scores zero on Unplug & Plug Charger and Place Gears. FastWAM and ACT were not run on this suite.

Go to primary source ↓
e-realPDF p. 14, Figure 7 and Overall orderingInspect

Real-robot macro averages are N₀-TWAM 46.3%, π0.5 30.0%, LingBot-VA 21.9% and FastWAM 14.4%. N₀-TWAM task rates are Fruit Collection 60, Hanoi Tower 40, Bottle Standing 70, Socket Plugging 70, Making Lemon Tea 15, Cup Stacking 45, Bag Packing 15 and Board Wiping 55. π0.5 scores 50 on Cup Stacking and 20 on Bag Packing. The authors note up to approximately 11% binomial standard error per task.

Go to primary source ↓
e-generalizationPDF pp. 15–16, Section 4.4 and Table 4Inspect

Object shifts use Bottle Standing/Bag Packing, position shifts use Cup Stacking and lighting/background shifts use Fruit Collection. N₀-TWAM scores 65/45/45 across the three axes, average 51.7, versus π0.5 at 80/45/25, average 50.0, and LingBot-VA at 75/35/30, average 46.7.

Go to primary source ↓
e-ablationPDF pp. 16–17, Section 4.5 and Figure 9; p. 27, Appendix A, Tables 7–8Inspect

UniVTAC full/20%-pretrain/no-predicted/no-observed scores are 84.5/65.4/71.8/70.5. NeoSim full/no-predicted/no-observed scores are 49.4/41.1/29.6. The reduced-data variant is UniVTAC-only. Predicted-path ablation zeros tactile targets before noising while preserving tokens, mask and loss; observed-path ablation omits the branch. Tactile variants use matched checkpoint/data/hyperparameters; reduced-data pre-training necessarily changes the checkpoint. Table 8 Cup Handover improves from 14 to 65 without predicted tactile.

Go to primary source ↓
e-posePDF p. 17, Section 4.6, Table 5Inspect

Absolute poses are adopted for Lift Can, Lift Bottle, Put Bottle in Shelf and Grasp Chip. Delta/absolute scores are 24/93, 52/58, 86/87 and 38/92; averages are 50.0/82.5. This qualifies the default delta-action description.

Go to primary source ↓
e-realismPDF pp. 17–18, Section 4.6, Figure 10 and Table 6; pp. 27–28, Appendix B, Tables 9–10Inspect

Clean and gel-rendered tactile share contact physics and differ in image rendering. Training/evaluation on gel-rendered tactile gives 82.4% versus 84.5% clean UniVTAC success; gel plus pretrained NeoForce gives 88.1%. NeoSim single-arm scores are 63.8/58.5/64.8 and dual-arm scores 42.3/39.3/not reported. Dual-arm NeoForce results are marked with slashes.

Go to primary source ↓
e-qualitativePDF p. 14, Figure 6; p. 18, Figure 11Inspect

Selected predicted/ground-truth visual and tactile rollouts illustrate Insert Hole and a lemon-tea grasp. Figure 11 also shows per-finger contact-flow maps. These are qualitative examples, without a quantitative contact-prediction error metric.

Go to primary source ↓
e-training-dynamicsPDF p. 28, Appendix C and Figure 12; p. 11, Section 4.1 curriculum paragraphInspect

Figure 12 covers only the first 18,000 steps of a 30,000-step run. At UMI introduction video loss rises, tactile loss changes little and action loss steps slightly. Appendix C describes UMI mixing followed by pure real-robot continuation, whereas Section 4.1 stops its recipe description at UMI mixing; the final recipe is not fully reconciled.

Go to primary source ↓
e-futurePDF p. 20, Section 6, Future work; p. 1, release statementInspect

The authors identify faster inference, longer prediction horizons and broader tactile-sensor coverage as future work. Code and checkpoints are promised for future public availability; the source does not establish their release.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.