Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention
1. Paper overview
In one sentence: Directional tactile attention improves contact-rich action generation and preserves an RGB-only visual trajectory, while the separate contributions of contact supervision and attention bias remain unresolved. e-identitye-probleme-architecturee-attentione-proxye-drifte-consistencye-pixel-discrepancy
| At a glance | What to know |
|---|---|
| Research problem | Source description Contact-rich manipulation can require corrections for slip, jamming or misalignment before these become clear in external RGB. Adding sparse tactile tokens to a visually pretrained predictor may disturb its visual dynamics. The paper asks how to preserve visual prediction while using contact information for executable actions. e-probleme-architecture |
| Core mechanism | Source description VideoClean blocks video queries from tactile keys while retaining action access to touch; the remaining causal multimodal paths stay available. e-attention |
| A key reported result | Physical contact-rich manipulation: 59/120 (49.2%) Execution success. Five physical tasks, with standard/dim power insertion counted separately; six conditions, 20 trials each. π₀.₅: 29/120 (24.2%); DreamZero: 26/120 (21.7%); VT-WAM: 19/120 (15.8%). Tactile-WAM leads all six conditions. Dim-light power insertion is 10/20 versus DreamZero’s 4/20; this is a limited lighting comparison, without reported success-rate confidence intervals. e-real |
| Reading caution | Reader analysis The motivating diagnostics contain internal conflicts: Figure 3 reports Spearman ρ = 0.810 despite the text’s weak-correlation description, and Figure 2(b) gives +11.2% MSE rather than Section 3’s 50%. These observations do not justify repeating the stronger prose claims as verified findings. e-pixel-discrepancye-mse-discrepancy |
Core contributions
- Source description
VideoClean blocks video queries from tactile keys while retaining action access to touch; the remaining causal multimodal paths stay available. e-attention
- Reader analysis
The strongest supported mechanism result is the cumulative VideoClean control gain, paired with reduced drift from an RGB-only reference; independent benefits of proxy supervision and bias remain unresolved. e-ablatione-drift
Figure 4. One shared predictor, with different rules for video and action access to touch. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the colored modality streams into the central Wan DiT, then separate the future video, future tactile and action outputs on its right. Language enters through cross-attention. In the enlarged TAAM grid, rows are queries and columns are keys: the red cross removes the video-query/tactile-key connection, while the green bias mark strengthens the action-query/tactile-key connection. The diagram shows modality routing; Appendix B supplies the additional temporal causal mask. The virtual-force branch has two uses in the text: observed contact changes drive the bias, while decoded future tactile predictions receive auxiliary supervision. These uses involve different time intervals and different gradient paths. e-architecturee-tokenizatione-attentione-proxye-probleme-freeze-discrepancy
What it supports. The architecture supports a single model that jointly predicts futures and actions. Its main design choice is directional access to touch: action prediction can use tactile information even when direct video attention to tactile keys is blocked. This is narrower than isolating every possible indirect tactile influence on video.
Where the evidence stops. The diagram’s frozen symbols on action/state encoders conflict with Appendix B’s jointly trained projections. The detailed training paragraph is followed here. The virtual-force label denotes a deformation proxy, and does not establish calibrated force prediction.
2. Motivation
2.1 The problem and the proposed response
Contact-rich manipulation can require corrections for slip, jamming or misalignment before these become clear in external RGB. Adding sparse tactile tokens to a visually pretrained predictor may disturb its visual dynamics. The paper asks how to preserve visual prediction while using contact information for executable actions. e-probleme-architecture
2.2 What this reading follows
A robot can see an object near a socket while still missing the contact error that determines its next correction. Tactile-WAM adds predicted touch to a model that already predicts video and actions, then restricts how those modalities interact. The central idea is that action queries should use tactile evidence even when video queries cannot directly read it. This reading follows that information flow, distinguishes observed-touch routing from future-touch supervision, and separates executed-task success from visual-reference agreement. The supplied v3 also contains figure/text and protocol inconsistencies; the original visuals below make those evidence boundaries visible. e-identitye-probleme-architecturee-attentione-proxye-drifte-consistencye-pixel-discrepancy
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
One shared DiT jointly predicts future video, tactile latents and actions, supporting One Model × Joint prediction (Q1). The directional mask does not create separate world and policy models. Actions are jointly denoised, not extracted by inverse dynamics. Proxy supervision is auxiliary training, while observed-touch routing operates during action generation. e-architecturee-attentione-proxye-flow
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Trace one control call through the shared predictor
Begin with RGB history, bilateral tactile observations, proprioception and a language instruction. RGB and the complete left-right tactile mosaic enter the same frozen VAE separately, producing latent streams that receive their own token treatment. Action and state representations then join these streams in one Wan DiT. The model’s temporal organization aligns two future latent frames with each action chunk, while the first latent frame stays clean as conditioning. VideoClean changes the permitted direction of attention inside this shared computation; it does not introduce a second policy network. At inference, denoising generates action chunks alongside future latents. The ManiFeel protocol executes eight of twenty-four predicted actions before replanning, so subsequent observed contact provides feedback rather than requiring the entire imagined future to unfold. e-tokenizatione-architecturee-attentione-flowe-protocol
5.2 Separate the cue available now from the future training target
The same deformation operator serves two temporally different roles. For routing, it compares observed tactile images at the current block anchor and eight frame indices earlier. Its detached six-component output passes through a magnitude threshold, temperature scaling and a cap to create the action-to-tactile attention bias. For supervision, the operator instead evaluates eight consecutive future transitions per block, both in the reference tactile sequence and in decoded predictions. SmoothL1 penalizes their difference and backpropagates through the frozen decoder into predicted latents. Thus, the future target teaches a representation during training, while the observed pair supplies a causal cue during action generation. The proxy has no learned parameters and summarizes image deformation; its “virtual force” terminology does not convert it into a calibrated force sensor. e-attentione-proxye-problem
5.3 Ask which experiment identifies which claim
Reader analysis: the evidence is strongest when the outcome and intervention are stated narrowly. The paired visual diagnostic asks whether VideoClean keeps generated video close to DreamZero under matched conditions and noise seeds. Its positive result does not imply more accurate ground-truth dynamics, because the companion ground-truth intervals include zero. The cumulative ManiFeel ablation asks a different question about executed control and identifies a large increment after enabling VideoClean. Its last row cannot separate proxy supervision from attention bias because both switches change together. Finally, the physical bars establish higher observed success in the six reported conditions, while selected bulb-insertion images illustrate a possible correction mechanism. Keeping these claims separate avoids using a plausible attention story as a substitute for an independently measured component effect. e-drifte-groundtruthe-ablatione-real
5.4 Training and inference
During training
Flow matching predicts noise-minus-clean-target velocities. Each block samples one of 1000 training noise levels shared across its aligned modalities. Video, action and tactile losses are normalized by valid features; the proxy adds 0.05 SmoothL1 in the main objective. e-flowe-proxye-training
Appendix B jointly fine-tunes the DiT and new projections/heads while freezing VAE and language encoder. AdamW uses peak learning rate 10⁻⁵, 5% warmup and cosine decay; training uses eight A100s, global batch eight, BF16 and ZeRO-2. Appendix C’s “from scratch” wording and Figure 4’s frozen projection symbols leave implementation ambiguities. e-traininge-consistencye-freeze-discrepancy
During inference
Conditioned on observed histories, denoising jointly produces future latents and an action chunk. The reported ManiFeel policy predicts 24 actions, executes eight, then replans from feedback. Proxy supervision is a training objective; the observed proxy still drives inference-time attention. Predicted video is not itself an executed action. e-flowe-attentione-proxye-protocol
5.5 Implementation flow
- Encode and align modalities
The frozen Wan VAE separately encodes RGB and a horizontal bilateral tactile mosaic, each resized to 320×160. Tactile patches receive learnable projections and modality embeddings; state/action tokens join the shared Wan2.2-TI2V-5B DiT, and UMT5-XXL language features enter through cross-attention. e-tokenizatione-architecture
- Organize causal prediction blocks
Thirty-three image frames become nine latent frames. One remains clean; the other eight form four two-frame blocks, each aligned with a 24-step action chunk and a state token. Temporal alignment uses block identifiers and modality-specific positional encodings. e-tokenization
- Route observed touch
VideoClean removes direct tactile-key access for visual queries. An observed tactile pair eight frame indices apart produces a detached deformation cue; a thresholded, temperature-scaled and capped score boosts only same-block action-to-tactile logits already allowed by the causal mask. Future targets do not compute this bias. e-attention
- Supervise future contact representations
Grayscale temporal differences multiplied by normalized spatial gradients yield a deformation field. Spatial means of horizontal motion, vertical motion and divergence give three components per sensor. Eight consecutive future transitions per block supervise decoded predictions; gradients traverse the fixed estimator and frozen decoder into predicted latents. e-proxy
6. Experiments & results
Tactile-WAM jointly generates visual futures, tactile futures and action chunks in one Wan-based Transformer. Its directional attention mask protects the direct video pathway while an observed deformation cue strengthens action access to touch. Control gains are strongest relative to RGB-only DreamZero; visual-prior preservation and physical accuracy are distinct findings.
6.1 Read the original evidence
Table 2. Tactile-WAM improves over DreamZero across the suite, while π₀.₅ retains the highest total. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task column contains successes out of fifty trials after the reported 60K-step training budget. Read down a column to compare methods on the same task, then use the Overall column to see the equally weighted nine-task aggregate. Bold entries identify task leaders. Peg, gear and bulb insertion and bolt-nut assembly favor Tactile-WAM, but USB insertion and object search remain particularly weak. Comparing only those favorable contact tasks would conceal the broader task tradeoff. Also keep the initialization caveat from Appendix C in view: the paper describes π₀.₅ as starting from released weights, while its descriptions of WAM initialization are inconsistent. e-manifeele-protocole-consistency
What it supports. Tactile-WAM reaches 147/450 successes, or 32.7%, versus DreamZero’s 70/450, or 15.6%: a reported 17.1-percentage-point improvement. Its 34/50 bulb insertion result contrasts with only 1/50 USB insertion. π₀.₅ achieves 165/450, or 36.7%, so the table does not establish an overall lead for Tactile-WAM.
Where the evidence stops. Fixed trial budgets are reported without success-rate confidence intervals or variation across training seeds. The conclusion’s state-of-the-art wording exceeds the overall comparison shown here, and equal training-step counts do not resolve initialization differences.
Figure 5, quantitative panels. Five physical tasks yield six evaluated conditions because power insertion is tested under two lighting settings. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to identify gray π₀.₅, blue DreamZero, orange VT-WAM and green Tactile-WAM. Each task-condition panel reports a success count over twenty trials, with the corresponding percentage above the bar. The lower-right Overall panel pools all six conditions. Power insertion appears twice, so this aggregate weights that task under two lighting settings rather than representing six distinct manipulation tasks. The small image sequences beneath the bars show example execution stages; they do not supply additional trials. Compare the two power-insertion panels directly to see how the reported policies behave when the visual setting changes while the task identity stays fixed. e-reale-ablatione-consistencye-qualitative
What it supports. Tactile-WAM leads every displayed condition and totals 59/120 successes, or 49.2%, versus 29/120 for π₀.₅ and 26/120 for DreamZero. Under dim lighting its power-insertion success is 10/20, compared with DreamZero’s 4/20. Its own standard-light result is 13/20, so illumination robustness is partial.
Where the evidence stops. These trials do not isolate the attention bias or establish broad sensor and robot generalization. Appendix Figure S22’s phone pick-and-place example is outside this six-condition count; it must not be added to the reported physical success total.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| ManiFeel manipulation Nine simulation tasks; 60K training steps; 50 trials per task. | 147/450 (32.7%) Execution success | DreamZero: 70/450 (15.6%); π₀.₅: 165/450 (36.7%). The reported gain over DreamZero is 17.1 percentage points. Four contact tasks favor Tactile-WAM, but π₀.₅ leads overall; USB insertion and object search remain weak. e-manifeele-protocol |
| UniVTAC manipulation Eight simulation tasks; 100K steps; 20 trials per task. | 48/160 (30.0%) Execution success | Without VideoClean: 42/160 (26.3%); DreamZero: 34/160 (21.3%); π₀.₅: 69/160 (43.1%). Tube insertion reaches 17/20, but HDMI insertion is 0/20. Gains are task-dependent rather than an overall simulation lead. e-univtac |
| Physical contact-rich manipulation Five physical tasks, with standard/dim power insertion counted separately; six conditions, 20 trials each. | 59/120 (49.2%) Execution success | π₀.₅: 29/120 (24.2%); DreamZero: 26/120 (21.7%); VT-WAM: 19/120 (15.8%). Tactile-WAM leads all six conditions. Dim-light power insertion is 10/20 versus DreamZero’s 4/20; this is a limited lighting comparison, without reported success-rate confidence intervals. e-real |
| Preservation of the RGB-only visual trajectory 84 paired held-out UniVTAC samples; matched 20K checkpoints, conditions and diffusion seeds. | 0.001227 with VideoClean versus 0.001568 without (21.8% reduction). Prediction-to-DreamZero MSE; lower is closer to reference | Reported paired improvement 0.000342; 95% bootstrap CI [0.000117, 0.000555]. H4 reduction: 38.5%. This establishes closer agreement with DreamZero. Ground-truth MSE slightly increases, and every ground-truth metric interval includes zero; improved physical prediction accuracy is not established. e-drifte-groundtruth |
| Cumulative tactile-component ablation ManiFeel; 60K steps and 50 trials per task. | RGB-only 15.6%; naive touch 15.1%; +VideoClean 28.4%; full model 32.7%. Overall execution success rate | VideoClean adds 13.3 percentage points; proxy supervision plus bias add 4.3 points jointly. The largest measured increment is routing isolation. The final increment changes two components and cannot identify either component’s independent effect. e-ablatione-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. The displayed diagnostic contains both a strong rank association and an example of local mismatch. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as the norm of the change in the paper’s virtual-force proxy, and the vertical axis as tactile-image RGB RMS change; both axes are logarithmic. The cloud rises across the plot, and its printed Spearman coefficient is 0.810. The orange marker selects a transition near the lower-right edge, where relatively small pixel change accompanies a larger proxy change. Follow the arrow to the paired left/right tactile images and the six before/after component bars. These are image-derived proxy components. The inset illustrates a selected transition, while the density plot describes an aggregate relationship; they answer different questions. e-pixel-discrepancye-proxye-problem
What it supports. The selected example supports the possibility that a small image change can accompany a meaningful proxy change. However, the printed rank correlation indicates substantial monotonic association across the plotted sample. The visual therefore supports a more qualified motivation than the introduction’s broad statement that the correlation is weak.
Where the evidence stops. Spearman association does not establish linearity or physical calibration. The caption says adjacent images, but the inset labels frames 29 to 32; the prose’s colored regions and dashed lines also do not match the supplied graphic.
Table 3. VideoClean preserves the RGB-only reference trajectory more closely; this table does not measure agreement with ground truth. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the reference: all four metrics compare predicted video with DreamZero predictions. The two method columns compare tactile fusion without and with VideoClean. The paired-improvement column uses signs so positive values favor VideoClean, while the final column reports the percentage of samples on which it is closer. The detailed protocol uses eighty-four held-out UniVTAC samples with matched conditioning and diffusion seeds at 20K checkpoints. Read this table together with Table 5 on the same page, which switches the reference to ground truth. A gain in this table concerns preservation of a learned trajectory, whereas a gain in Table 5 would concern prediction accuracy. e-drifte-groundtruthe-consistency
What it supports. Prediction-to-DreamZero MSE falls from 0.001568 to 0.001227, a reported 21.8% reduction, and VideoClean is closer on 84.5% of samples. The paired MSE improvement has a positive 95% bootstrap interval. Table 4 further reports a 38.5% MSE reduction at the fourth rollout chunk.
Where the evidence stops. All ground-truth metric intervals in Table 5 include zero; this is neither an accuracy improvement nor an equivalence proof. The abstract assigns the diagnostic to ManiFeel, conflicting with the detailed UniVTAC protocol used here.
Table 6. The ablation isolates the VideoClean increment, but adds proxy supervision and attention bias together. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows as a cumulative sequence of interventions. The RGB-only row has no future-touch target. Naive VT-WAM adds that target, the next row also enables VideoClean, and the final row additionally checks both Force and Bias. Force means the virtual-force auxiliary supervision, while Bias means the observed contact-aware attention bias; they are not two names for one mechanism. Success is reported as a fraction in this table, so 0.284 means 28.4%. The associated protocol is ManiFeel at 60K steps and fifty trials per task. To identify a component’s contribution, look for rows differing only in that component. e-ablatione-protocole-proxye-attention
What it supports. Naive future-touch prediction gives 15.1%, slightly below the 15.6% RGB-only baseline. Adding VideoClean raises success to 28.4%, the largest measured increment at 13.3 percentage points. The full model reaches 32.7%, but the remaining 4.3-point gain belongs to the combined supervision-and-bias intervention.
Where the evidence stops. There is no force-only or bias-only row, so neither independent contribution nor their interaction can be estimated. The qualitative bulb-insertion comparison in Figure 7 does not replace a controlled real-robot success-rate ablation.
7. Analysis & limitations
7.1 What the evidence leaves open
The motivating diagnostics contain internal conflicts: Figure 3 reports Spearman ρ = 0.810 despite the text’s weak-correlation description, and Figure 2(b) gives +11.2% MSE rather than Section 3’s 50%. These observations do not justify repeating the stronger prose claims as verified findings. e-pixel-discrepancye-mse-discrepancy
The abstract places the drift experiment on ManiFeel, whereas the detailed protocol says UniVTAC. Appendix B’s eight-dimensional action width conflicts with Appendix C’s seven-dimensional interface. Seven qualitative physical examples include phone pick-and-place outside the six-condition quantitative evaluation. These inconsistencies remain unresolved. e-consistency
Simulation comparisons do not demonstrate overall state of the art: π₀.₅ leads both totals, and initialization is not consistently described. Neither a qualitative bias comparison nor pooled trial counts establish generalization across robots, sensors or training seeds. e-consistencye-univtace-manifeele-ablatione-real
7.2 Questions for discussion
- Would proxy supervision and observed-touch bias each improve control under identical initialization and seeds?
- Can the deformation proxy identify independently labeled contact events beyond what tactile pixel change already predicts?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires synchronized RGB/tactile/state/action data, the stated Wan and UMT5 components, bilateral mosaic preprocessing, causal masks and percentile-normalized valid action dimensions. The source supplies optimization details but not bias constants, tactile loss weight, inference solver/steps, complete splits or physical data counts. e-tokenizatione-attentione-traininge-missing-details
The physical setup is pictured, but exact control frequency, reset rules, success criteria and sensor-installation details referred to supplementary material are not supplied as an operational protocol. Initialization and action-width discrepancies must also be resolved before claiming a faithful implementation. e-reale-missing-detailse-consistency
Proposed checks are a matched factorial proxy/bias ablation and a trajectory-disjoint pixel/proxy audit with independent contact labels. Both should report falsifiable comparisons rather than infer success from selected execution images. e-ablatione-pixel-discrepancye-qualitative
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate proxy supervision, bias and their interaction
Reader-proposed check, not performed: keep future tactile prediction and VideoClean enabled, and train a 2×2 design with proxy supervision off/on and observed-touch bias off/on. Use identical, explicitly documented Wan initialization, data splits, action adapters, 60K-step budgets and paired evaluation seeds; repeat training over several seeds. Evaluate all nine ManiFeel tasks with fifty trials per task per run, reporting per-task counts and uncertainty across runs. The bias-only and supervision-only cells identify individual gains; the combined cell tests interaction. If neither single component improves on VideoClean alone and the combination is unstable across seeds, the claimed joint 4.3-point gain would not support a reliable independent mechanism. e-ablatione-protocole-consistencye-training
Check 2: Audit whether the image-derived proxy adds contact information
Reader-proposed check, not performed: reconstruct the specified fixed deformation operator and evaluate it on trajectory-disjoint tactile sequences with independently annotated slip, contact establishment and misalignment events. Compare a simple event detector using RGB RMS change with one using the six proxy components and with one using both, under identical training splits and model capacity. Report rank and linear correlation separately, event precision/recall, and performance under brightness-only image perturbations as a negative control. Also compare adjacent-frame pairs with the eight-frame anchor interval used for routing. Failure to improve held-out event detection, or strong response to brightness changes without contact, would weaken the rationale for the proxy even if the Figure 3 correlation is reproducible. e-pixel-discrepancye-proxye-attentione-problem
8.3 Reading coverage
Visual audit: The title/author page and Figures 1–7, Tables 1–6, and method/training/protocol pages 10–12 were visually inspected. Page 3 was additionally read at higher resolution to verify Figure 2(b)'s 11.2% MSE annotation and Figure 3's rho = 0.810. Selected supplementary images were inspected: S3 (p. 15), S18–S19 (p. 26), S22–S23 (p. 28) and S24 (p. 29). All supplied appendix text and captions were read, but the remaining supplementary trajectory images were not visually inspected. All six final crops were viewed; the physical-results crop was corrected to retain the complete title. The declared pages include the uncropped ground-truth table and appendix details used in the visual explanations. No code or experiment was run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 15, 26, 28, 29. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and Abstract, p. 1
- 1 Introduction and 2 Preliminaries and Notation, p. 2
- 3 Key Observations, pp. 2–3
- 4 Method, pp. 3–5
- 5 Experiments and 6 Conclusion, pp. 5–7
- References, pp. 8–9
- Appendix A Related Work, p. 10
- Appendix B Model Details, pp. 10–12
- Appendix C Experiments Details and all Figures S1–S24 captions, pp. 12–29
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read in full. The visual pass inspected PDF pages 1–7, 10–12, 15, 26, 28 and 29; it did not inspect every supplementary trajectory image.
- No code, external source, separately hosted supplement or earlier revision was inspected; no experiment was reproduced.
- Identity note (e-identity): this report reads v3, 27 August 2026. Earlier editions were not supplied, so revision differences cannot be established. The title and all twelve authors agree substantially with the catalog; observed name order is retained for Huang Kaixiang and Chen Yonghang. The title page additionally verifies Ant Group, omitted from the catalog affiliation string.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author/affiliation block and arXiv margin
The title matches the supplied identity. The margin identifies arXiv:2606.26663v3, 27 August 2026. Twelve authors are credited, including names displayed as Huang Kaixiang and Chen Yonghang. Affiliations are Ant Group, Institute of Automation of the Chinese Academy of Sciences, The University of Hong Kong, and Zhejiang University. Siyu Wu and Linjing You contributed equally.
Go to primary source ↓e-problemPDF p. 2, Introduction and Section 2, Vision-Based Tactile Sensing
Slip, jamming and misalignment can be difficult to observe in external RGB. Optical tactile images reveal deformation but do not directly give calibrated physical forces.
Go to primary source ↓e-flowPDF p. 2, Section 2, Eqs. (1)–(3); p. 4, Joint Visual-Tactile-Action Modeling
Conditional flow matching transforms Gaussian noise into future observation latents and actions. Tactile history extends the conditioning context and tactile futures become an additional prediction target.
Go to primary source ↓e-architecturePDF pp. 3–4, Section 4 and Figure 4; p. 10, Appendix A, Cross-modal interference
Video, tactile, action and state tokens are concatenated inside one Wan DiT. Figure 4 shows joint future video, tactile and action outputs, the VideoClean restriction and a tactile bias for action queries.
Go to primary source ↓e-tokenizationPDF pp. 10–11, Appendix B, Backbone, Multimodal Tokenization and Temporal Block Organization
Wan2.2-TI2V-5B has 30 blocks, width 3072 and 24 heads. The frozen shared VAE encodes RGB and a complete left-right tactile mosaic resized to 320×160. It has 48 latent channels and spatial/temporal reductions of 16/4. Thirty-three image frames yield nine latent frames: one clean condition and four two-frame future blocks, each aligned with 24 actions and one state token. UMT5-XXL language features enter through cross-attention. Appendix B specifies eight-dimensional padded action/state representations and percentile normalization.
Go to primary source ↓e-attentionPDF p. 4, Eqs. (9)–(13); p. 11, Appendix B, Eqs. (21)–(23), (28)
VideoClean sets visual-query/tactile-key logits to negative infinity. A bounded bias from the detached observed proxy is added only for same-block action queries and tactile keys already permitted by the causal mask. The observed cue compares anchor frames eight indices apart and is independent of future targets and denoising predictions.
Go to primary source ↓e-proxyPDF p. 5, Eqs. (14)–(17); p. 11, Appendix B, Eqs. (24)–(30)
A fixed differentiable operator uses grayscale temporal differences and centered spatial gradients, with epsilon 10^-6 specified in the main text. Each sensor contributes mean horizontal deformation, mean vertical deformation and mean divergence. Bilateral concatenation gives six components. Future supervision covers eight consecutive transitions per block, with SmoothL1 weighted by 0.05 in Eq. (16); gradients pass through the frozen VAE decoder to predicted latents.
Go to primary source ↓e-trainingPDF pp. 11–12, Appendix B, Training Configuration
The DiT, modality projections, embeddings, modified attention and heads are jointly fine-tuned; VAE and language encoder are frozen. AdamW uses peak learning rate 10^-5, betas 0.95/0.999, epsilon 10^-8, eligible weight decay 10^-5, 5% warmup and cosine decay. Training uses 1000 noise levels with a shared level per aligned causal block, BF16, gradient clipping 1.0, seed 42, eight A100 GPUs, effective batch eight and ZeRO-2.
Go to primary source ↓e-protocolPDF p. 5, Experimental Setup; p. 12, Appendix C, Experimental Setup
UniVTAC uses eight tasks, 100K steps and 20 trials/task; ManiFeel uses nine tasks, 60K steps and 50 trials/task. ManiFeel samples observations at 10 fps, runs simulation at 60 Hz with decimation four, predicts 24 actions and executes eight before replanning. Appendix C describes six-dimensional actions padded to a seven-dimensional interface. It calls WAM training from scratch and π₀.₅ initialization released base weights.
Go to primary source ↓e-univtacPDF p. 5, Table 1, all rows and Overall/Tube ins./HDMI ins. columns
UniVTAC totals are Tactile-WAM 48/160 (30.0%), without VideoClean 42/160 (26.3%), DreamZero 34/160 (21.3%) and π₀.₅ 69/160 (43.1%). Tube insertion is 17/20 for Tactile-WAM versus 4/20 for DreamZero and π₀.₅; Tactile-WAM has 0/20 HDMI insertion successes.
Go to primary source ↓e-manifeelPDF p. 5, Table 2, all rows
ManiFeel totals are 147/450 (32.7%) for Tactile-WAM, 70/450 (15.6%) for DreamZero and 165/450 (36.7%) for π₀.₅. Tactile-WAM leads peg insertion (8/50), gear insertion (19/50), bulb insertion (34/50) and bolt-nut assembly (20/50), but achieves 1/50 USB insertion and 2/50 object search.
Go to primary source ↓e-realPDF pp. 5–6, Real-Robot Results and Figure 5; p. 12, Real-robot protocol
Five named tasks, with power insertion repeated under dim lighting, give six conditions and 120 trials per method. Figure 5 reports totals 59/120 (49.2%), 29/120 (24.2%), 26/120 (21.7%) and 19/120 (15.8%) for Tactile-WAM, π₀.₅, DreamZero and VT-WAM. Tactile-WAM power insertion changes from 13/20 to 10/20 under dim light; DreamZero changes from 11/20 to 4/20.
Go to primary source ↓e-driftPDF p. 6, Tactile Pollution and VideoClean; p. 7, Tables 3–4
The diagnostic uses 84 paired held-out UniVTAC samples, matched conditioning and diffusion seeds, and 20K checkpoints. Prediction-to-DreamZero MSE falls from 0.001568 to 0.001227 (21.8%); the reported paired improvement is 0.000342 with 95% bootstrap CI [0.000117, 0.000555]. Table 4 reports MSE reductions of -1.7%, 3.3%, 22.4% and 38.5% over H1–H4. Table 3 reports MAE 0.020904 versus 0.017190, and VideoClean closer on 84.5% of samples by MSE.
Go to primary source ↓e-groundtruthPDF p. 7, Table 5, all rows
Against ground truth at 20K, MSE is 0.001748 without VideoClean and 0.001777 with it; paired improvement is -0.000029, 95% CI [-0.000257, 0.000202]. All four reported metric intervals include zero.
Go to primary source ↓e-ablationPDF p. 7, Table 6, Ablation Study and Figure 7; p. 6, Effect of Contact-Aware Attention Bias
The cumulative ManiFeel ablation gives 0.156 RGB-only, 0.151 naive future touch, 0.284 with VideoClean, and 0.327 with both Force and Bias. The source explicitly states that Force and Bias are not separately identified. Figure 7 shows a bulb-insertion comparison, but no bias-only real-robot success-rate variant is supplied.
Go to primary source ↓e-pixel-discrepancyPDF pp. 2–3, Introduction, Section 3 and Figure 3
The prose calls pixel/contact correlation weak and discusses 5,000 image pairs and colored regions/dashed lines. Figure 3 is a log-axis density plot annotated Spearman rho = 0.810, with a selected low-pixel-change transition and before/after virtual-force bars. The inset identifies episode 51, frames 29 to 32; the caption describes adjacent images. The claimed dashed-line annotations are absent from the displayed plot.
Go to primary source ↓e-mse-discrepancyPDF p. 3, Section 3, Tactile Pollution and Figure 2(b)
Section 3 states that naive tactile fusion yields 50% higher prediction MSE. Figure 2(b), labeled decoded video at 20K and n=84, instead reports RGB-only 0.001571, without VideoClean 0.001748, and +11.2%. These magnitudes cannot be silently treated as identical.
Go to primary source ↓e-consistencyPDF p. 1, Abstract; pp. 5–7, results and Conclusion; pp. 10–12, Appendices B–C; p. 28, Figure S22
The abstract attributes the 21.8% drift result to ManiFeel while the detailed protocol says UniVTAC. The conclusion calls simulation results state of the art despite π₀.₅ leading both overall tables. Appendix C calls WAM training from scratch although Appendix B describes fine-tuning a Wan backbone. Appendix B uses width eight whereas Appendix C says seven. Appendix C describes seven qualitative physical experiments and Figure S22 adds phone pick-and-place outside the six quantitatively evaluated conditions.
Go to primary source ↓e-freeze-discrepancyPDF p. 4, Figure 4, snowflake legend and action/state encoder icons; p. 11, Training Configuration
Figure 4 places frozen-module symbols on the action and state encoders, while Appendix B says multimodal projections and prediction heads are jointly fine-tuned. The report follows the detailed training paragraph but preserves this diagram/text discrepancy.
Go to primary source ↓e-qualitativePDF p. 12, Qualitative Execution Trajectories; pp. 13–29, Figures S1–S24 captions; inspected images on pp. 15, 26, 28–29
The appendix presents selected successful simulation and physical executions. Inspected examples include ManiFeel bulb insertion, real bulb insertion and gear meshing, phone pick-and-place, and power insertion under dim and standard lighting. These are success illustrations, without additional quantitative evaluation for phone pick-and-place.
Go to primary source ↓e-missing-detailsPDF pp. 4–5, bias/objective definitions; pp. 10–12, Appendices B–C; pp. 13–29, qualitative appendix
The source defines bias threshold, temperature, amplification and cap without numerical settings and leaves the tactile flow-loss weight unspecified. The supplied appendix lacks an inference solver/step count, measured inference latency, complete dataset splits and real-robot data counts. Appendix C refers physical control frequency, success/reset rules and sensor-installation detail to supplementary material, but the remaining supplied pages are qualitative execution figures.
Go to primary source ↓8.5 Primary sources
Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention ↗
PDF · 6,024 extracted words
Source fingerprint
d38f4fd97442390a0fd6096d0282c5efca0fb57cf8dd0e8f1c8495373dfc6aa5