HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation
1. Paper overview
In one sentence: HiTac-WAM uses a directed contact–deformation–slip forecast to rank action candidates and monitor execution, improving three-task robot success while leaving generated-candidate forecast accuracy and slip calibration unresolved. e-motivatione-contexte-hierarchye-selectione-verificatione-rankinge-limitations
| At a glance | What to know |
|---|---|
| Research problem | Source description Visually plausible grasping, erasing, or insertion rollouts can conceal missed contact, insufficient pressure, slip, and jamming. The problem is to predict each unexecuted action chunk’s tactile consequences, choose among candidates, and notice when the chosen forecast diverges from physical execution. e-motivation |
| Core mechanism | Source description A shared multimodal backbone supports a directed tactile pathway with deterministic contact, deformation, and slip heads, imposing explicit downstream conditioning rather than merely sharing features. e-contexte-hierarchy |
| A key reported result | Fixed-budget tactile-forecast versus task-progress ranking: Tactile ranking 61.1% [50.8,70.5]. Success rate; two-sided 95% Wilson interval. K=4 in both arms, matched candidate seeds and validity rules; independent executions, 90 trials per arm. Tactile arm reuses Table I Selection trials. Task-progress ranking 35.6% [26.4,45.8], a 25.5-percentage-point difference. This descriptive comparison better isolates the ranking signal from candidate count. It is not an additional independent replication of Selection. e-protocole-ranking |
| Reading caution | Source description The authors explicitly leave forecast quality on model-generated action chunks unevaluated separately and predicted slip risk uncalibrated for online detection. e-limitations |
Core contributions
- Source description
A shared multimodal backbone supports a directed tactile pathway with deterministic contact, deformation, and slip heads, imposing explicit downstream conditioning rather than merely sharing features. e-contexte-hierarchy
- Source description
The same temporally aligned forecast is used prospectively for candidate ranking and subsequently as a reference for corrective replanning. e-selectione-verification
Figure 2. Each action keeps its tactile forecast through selection and execution. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a): visual, action, tactile, state and language inputs enter the shared context. The modality attention diagram allows tactile queries to read video and actions while blocking the reverse tactile-to-video/action route. Move to panel (b), where the three physical outputs are ordered rather than predicted as unrelated targets. Finally, follow panel (c): paired actions and forecasts receive a score, the lowest-score rollout is chosen, and its forecast becomes the reference for later observations. The residual density score determines whether execution continues or a persistent mismatch causes replanning. e-contexte-hierarchye-selectione-verification
What it supports. The architecture supports a shared-model interpretation and gives tactile prediction an explicit inference-time control role. Its contribution is candidate evaluation and monitoring: measured touch can change which candidate is executed through ranking, even though video and action queries are prevented from reading tactile tokens directly.
Where the evidence stops. Panel (b) is schematic: Equation (3b) conditions deformation on a detached contact transition feature, not simply the displayed scalar contact estimate. Equation (6) also defines the residual and task standardization more precisely than panel (c).
2. Motivation
2.1 The problem and the proposed response
Visually plausible grasping, erasing, or insertion rollouts can conceal missed contact, insufficient pressure, slip, and jamming. The problem is to predict each unexecuted action chunk’s tactile consequences, choose among candidates, and notice when the chosen forecast diverges from physical execution. e-motivation
2.2 What this reading follows
Imagine two visually convincing USB-insertion rollouts: one makes productive contact, while the other jams sideways. HiTac-WAM asks what each proposed action will feel like before choosing it. A tactile branch reads the candidate’s action and predicted visual context, then forecasts contact, deformation and slip in a physical order. The controller uses this forecast twice: first to rank candidates, then to check whether measured touch follows the chosen prediction. Read the architecture before the results, and keep held-out prediction accuracy separate from executed robot success. The fixed-budget ranking experiment provides the clearest comparison of tactile versus visual progress scoring. e-motivatione-contexte-hierarchye-selectione-verificatione-rankinge-limitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Insufficient evidence to decide
The shared backbone and directed attention support the recorded One Model architecture; this judgment rests on information flow, not joint training alone. The tactile contribution performs forward forecasting, candidate ranking and online verification, making Other mechanisms/Outside quadrants plausible for that layer, although the underlying WAM jointly generates video and actions. Multimodal tactile sensing is directly supported. Policy post-training & WM-RL is only partly supported by task-specific supervised adaptation: no reinforcement-learning objective is described. e-contexte-hierarchye-losse-selectione-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the candidate and its imagined touch aligned
At one replanning instant, all candidates share the observed RGB history, robot state, language instruction and bilateral tactile history. Independent sampling produces different action chunks with associated visual futures. The tactile branch reads each candidate’s aligned action tokens and predicted visual context, so its output answers a candidate-specific question: what contact, deformation and slip would follow this action? Temporal alignment matters: the action at forecast offset h−1 is paired with the tactile target at h, matching what can be observed after execution. The attention mask lets tactile queries consult the generated future without allowing tactile tokens into video/action queries. Reader interpretation: this arrangement makes forecast-to-candidate correspondence essential, which is why measuring forecast quality on generated chunks remains a consequential open evaluation gap. e-contexte-selectione-limitations
Figure 7. The selected forecast becomes a time-aligned reference for detecting execution deviations. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue anomaly score across the executed prefix, whose configured length is eight steps. The dashed red line is the task-calibrated threshold, and three successive points above it trigger correction. The controller then abandons the remaining prefix, retreats to a safe state, observes again and generates fresh candidates. The photographs connect this recorded trace to one physical interaction. The score comes from the density of standardized forecast–observation residuals fitted on successful validation episodes; it is not the raw predicted slip risk. Read the post-replanning point as a new observation after intervention. e-verificatione-protocole-monitor-resultse-limitations
What it supports. This example makes the execution-time use of the forecast concrete. Across the ninety Full trials, post-hoc inspection found eighteen triggers: sixteen coincided with anomalies, and twelve of those were followed by success. The aggregate result supports investigating this monitoring mechanism beyond the single illustrated recovery.
Where the evidence stops. A selected successful trace cannot establish detector reliability or recovery causality. Anomalies were not exhaustively annotated, and the paper leaves controlled-anomaly evaluation for future work. No universal numerical threshold should be inferred from this example plot.
5.2 Separate the deformation target from the deformation used for control
The deformation head learns an increment from the current sensor grid to a future grid. Current deformation itself is measured relative to an undeformed baseline; adding the predicted increment therefore reconstructs an ungated future deformation. Predicted contact then gates this physical output. This distinction prevents confusing a change prediction with an absolute tactile state. The slip head receives detached contact and the magnitude of the ungated future deformation, while deformation receives the contact head’s detached transition feature. Training supervises increments only on ground-truth-contact frames and applies weighted binary cross-entropy to the ungated slip probability. Stop-gradient blocks direct downstream updates of the upstream heads, while the shared tactile representation still learns from the joint objective. The diagnostic interventions test whether upstream signals carry useful information. e-hierarchye-losse-conditioning
5.3 Use the same forecast before contact and after execution starts
Candidate ranking combines visual progress with three normalized tactile costs. Because productive manipulation needs contact, the contact term penalizes phase-inconsistent contact; deformation and slip terms discourage excessive or imbalanced loading and predicted sliding. After choosing the minimum-cost valid candidate, Full retains its entire tactile forecast but executes an eight-step prefix. Each new causal measurement is compared with the forecast for that exact executed offset. Successful validation episodes define what ordinary residuals look like through task-specific standardization and a KDE. Three consecutive unusually low-density residuals trigger retreat, reobservation and renewed generation. This is a residual-based monitor, not direct thresholding of slip probability. The real-robot comparison measures the combined effect of adding verification and corrective opportunities; controlled anomaly trials would better isolate detection quality. e-selectione-verificatione-protocole-monitor-results
5.4 Training and inference
During training
Each task has 200 episodes, split into 160/20/20 training/validation/test episodes before window extraction. At 30 Hz, tactile history uses offsets [−12,−6,−2,0], actions 0–23, and targets 1–24. The platform is IMETA-Y1 with bilateral DM-Tac W2 sensors, two RealSense D435i cameras and one USB camera. e-data
Updated components are tactile projections/queries, action–tactile fusion, and tactile heads; a complete freeze map is not given. Each task trains separately for 10,000 steps on eight NVIDIA H100 GPUs with AdamW, learning rate 10^{-4}, and global batch 64. Onset/release weights are 2/1, positive-slip weight 30, and slip-loss coefficient is zero for 3,000 steps then 0.01. e-contexte-training
During inference
Selection and Full use K=4 and progress/contact/deformation/slip weights (1,1,1,0.5). Full executes an eight-step prefix, checking causal tactile observations against aligned predictions. A task-specific KDE fitted on standardized successful-validation residuals scores deviation. Three consecutive threshold exceedances abort remaining prefix actions, trigger safe retreat and fresh observation, and regenerate candidates; at most two corrections are allowed per trial. e-selectione-verificatione-protocol
5.5 Implementation flow
- Condition touch on each candidate
FG-CLTP encodes tactile history into projected tokens. Tactile queries access tactile history and the candidate’s video–action context; visual/action queries cannot attend to tactile keys or values. Action at offset h−1 conditions the tactile target at h. Thus touch evaluates candidate consequences within the shared backbone. e-context
- Predict the physical hierarchy
Contact is predicted first. Its detached intermediate transition feature conditions a deformation-increment head; detached contact estimates and ungated future-deformation magnitude condition slip. Predicted contact gates both physical outputs. Training supervises the ungated increment and slip probability, while control consumes gated outputs. Shared trainable tactile features still couple the heads despite detached conditioning. e-hierarchy
- Rank valid candidates
Independent noise draws generate candidates; kinematic checks discard invalid chunks. One batched tactile evaluation supplies phase-inconsistent contact, excessive/imbalanced deformation, and peak-plus-mean slip costs. A visual-rollout task-progress estimate rewards advancement. All four components are normalized to [0,1] before weighting. e-selection
6. Experiments & results
HiTac-WAM attaches an explicit contact–deformation–slip forecast to each action candidate generated by a pretrained world action model. Touch affects candidate selection and execution monitoring through a directed tactile branch. The strongest controlled comparison improves robot success using tactile ranking at a fixed candidate budget; forecast accuracy and anomaly detection retain important evaluation gaps.
6.1 Read the original evidence
Table I. Both tactile candidate selection and subsequent verification improve observed robot success. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as a separate physical task, with 30 trials for every method–task pair and 360 trials overall. Success means an intact delivered chip stable for at least three seconds; erasing more than 80% of the target while maintaining effective contact; or complete USB insertion without sustained jamming and with LED activation. DreamZero executes one sampled candidate. Reactive Tactile uses current touch. Selection ranks four candidates with forecasts, and Full adds execution-time verification and corrective attempts. The Average column summarizes equal-sized task groups, so it should be interpreted alongside the individual columns. e-protocole-successe-ranking
What it supports. Selection reaches 61.1% on average, compared with 31.1% for DreamZero and 40.0% for Reactive Tactile. Full reaches 72.2%, including 90.0% on erasing but only 50.0% on USB insertion. The remaining USB failures show that improved contact handling does not solve this task reliably.
Where the evidence stops. DreamZero and Selection have different candidate budgets, so their difference does not isolate tactile scoring. Full additionally receives corrective opportunities. Consult the fixed-budget ranking comparison before attributing the whole gain to forecast quality.
Table II. The held-out forecast benchmark measures contact timing, deformation magnitude and slip ranking separately. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally through the three output families. Contact F1 measures state classification, while onset and release MAE use frames at 30 Hz. Deformation errors are millimeters, evaluated only on future frames with ground-truth contact. Top-20 Z MAE emphasizes the twenty grid points with largest ground-truth normal deformation; the norm ratio should approach one. Slip AUPRC should increase, and must be read against the low event prevalence of 1.39%. These evaluations use held-out recorded episodes after episode-level splitting. The mean row averages the three tasks without weighting, rather than pooling every frame into one metric. e-datae-metricse-predictione-ablatione-limitations
What it supports. Mean contact F1 is 0.921, with onset/release errors of 2.1/2.3 frames. Mean deformation L2 is 0.058 mm and slip AUPRC is 0.247. These establish useful forecasting on the recorded test distribution; they do not by themselves quantify the quality of forecasts attached to newly generated candidates.
Where the evidence stops. The 0.058 mm mean belongs to Table II. Table III’s independently trained hierarchy reports 0.056 mm and explicitly permits such differences. Neither table calibrates slip probability, and the paper separately acknowledges the generated-candidate evaluation gap.
Figure 6. The middle bars compare scoring rules at the same four-candidate generation budget. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Focus first on the two middle bars. Both ranking arms generate four candidates under matched seeds, task distribution and validity rules, but their robot executions are independent. Tactile-forecast ranking reuses the ninety Selection trials from Table I; task-progress ranking contributes an additional ninety trials. The outer bars provide context: DreamZero executes one candidate, and Full adds online verification to tactile ranking. Error bars are two-sided 95% Wilson intervals. This layout is therefore a progression of systems with distinct protocols, rather than four interchangeable measurements of one isolated factor. e-protocole-rankinge-success
What it supports. Tactile ranking reaches 61.1% versus 35.6% for task-progress ranking, a 25.5-percentage-point difference. Their reported Wilson intervals, [50.8,70.5] and [26.4,45.8], do not overlap. At a fixed generation budget, tactile forecasting supplies information useful for choosing actions beyond the visual progress score used here.
Where the evidence stops. The source labels this comparison descriptive. Matched candidate seeds do not make physical executions paired counterfactuals, and reusing Selection trials does not provide an independent replication. The graphic alone does not establish statistical significance for Full versus Selection.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Future tactile prediction on held-out episodes Three task-specific test sets, 20 complete episodes each; 24-step targets. Table II means are unweighted across tasks. | Mean: 0.921; 2.1/2.3 frames; 0.058 mm; 0.076 mm; 0.960; 0.247. Contact F1; onset/release MAE; 3D L2; Top-20 Z MAE; norm ratio; slip AUPRC | Task contact F1 spans 0.908–0.931. Slip prevalence/chance reference is 0.0139. Forecast quality is measured on held-out recorded sequences. Deformation metrics use ground-truth-contact frames; AUPRC measures ranking, not probability calibration or robot success. e-datae-metricse-prediction |
| Directed tactile hierarchy ablation Matched training budgets; separately trained ablation rows in Table III and cumulative conditioning in Figure 5. | Hierarchy: 0.056 mm and 0.247. 3D L2 and slip AUPRC | Deformation-only: 0.068 mm (17.6% relative error reduction); slip-only: 0.154 AUPRC (60.4% relative increase). Independent-head F1 is 0.896 versus 0.903 contact-only. Sharing alone does not consistently improve prediction. Table III’s 0.056 mm is distinct from Table II’s 0.058 mm; the caption explicitly permits differences between independently trained rows. e-ablatione-conditioning |
| Real-robot contact-rich manipulation Table I: 30 trials per method/task across chip grasping, blackboard erasing and USB insertion; 360 trials total. | Full: 76.7/90.0/50.0 by task; average 72.2. Selection: 70.0/73.3/40.0; average 61.1. Task success rate (%) | DreamZero: 33.3/50.0/10.0; average 31.1. Reactive Tactile: 40.0/60.0/20.0; average 40.0. These are executed robot outcomes. Selection gains 30.0 percentage points over single-candidate DreamZero but also uses more candidates; Full adds verification and corrective attempts. e-protocole-success |
| Fixed-budget tactile-forecast versus task-progress ranking K=4 in both arms, matched candidate seeds and validity rules; independent executions, 90 trials per arm. Tactile arm reuses Table I Selection trials. | Tactile ranking 61.1% [50.8,70.5]. Success rate; two-sided 95% Wilson interval | Task-progress ranking 35.6% [26.4,45.8], a 25.5-percentage-point difference. This descriptive comparison better isolates the ranking signal from candidate count. It is not an additional independent replication of Selection. e-protocole-ranking |
| Execution-time forecast verification Post-hoc recording inspection of 90 Full trials, without exhaustive anomaly annotation. | 18 triggers; 16 coincided with anomalies; 12 of those were followed by success; two false triggers had no observed outcome effect. Triggers, anomaly-associated triggers and subsequent completions | Full averages 72.2% versus Selection’s 61.1%. At most three of 21 untriggered failures involved an undetected anomaly. Useful descriptive recovery evidence, but not a controlled detector sensitivity estimate or proof that all 12 completions were caused by correction. e-monitor-resultse-success |
| Preservation of video and action predictions Comparison with the world action model without tactile prediction, Section V.B. | LPIPS is 1.4% higher; action MAE is 0.9% higher. Relative change in video LPIPS and normalized action-trajectory MAE | Reference is the WAM without the tactile prediction pathway; absolute metric values are not supplied in this comparison. The reported differences indicate modest degradation on these preservation metrics. They do not establish identical outputs, unchanged runtime, or a throughput bound. e-contexte-metricse-preservation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Corrupting contact worsens deformation prediction; adding hierarchical signals improves slip ranking. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) holds the checkpoint fixed and changes contact conditioning. The intervention replaces the normal transition feature with a learned contact embedding indexed by ground-truth, predicted, shuffled or zero contact. Smaller horizontal positions mean less error; the two colors distinguish overall 3D L2 from onset Top-20 normal error. Panel (b) adds conditioning cumulatively: its final Deformation row includes contact and deformation, rather than deformation alone. Larger AUPRC is better. The dashed line is prevalence, 0.0139. Error bars are episode-paired bootstrap 95% intervals in (a) and stratified episode-bootstrap 95% intervals in (b). e-hierarchye-conditioninge-ablatione-limitations
What it supports. Predicted-contact L2, 0.056 mm, is close to the ground-truth-conditioned 0.053 mm, while shuffled and zero conditioning worsen it to 0.071 and 0.078 mm. Slip AUPRC increases from 0.154 to 0.247 across cumulative conditioning. The model demonstrably uses the upstream information in these diagnostic settings.
Where the evidence stops. Reader interpretation: these interventions support useful conditioning, not universal physical causality. The figure does not isolate the benefit of stop-gradient against an otherwise identical differentiable hierarchy, and better held-out prediction need not guarantee better ranking under generated-action distribution shift.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors explicitly leave forecast quality on model-generated action chunks unevaluated separately and predicted slip risk uncalibrated for online detection. e-limitations
Experiments cover three tasks on fixed hardware with task-specific training/calibration. They do not establish cross-task or sensor transfer, long-horizon autonomy, or real-time throughput. Detector evidence relies on post-hoc inspection, and the fixed-budget ranking comparison is descriptive. e-datae-traininge-protocole-limitations
7.2 Questions for discussion
- Would the hierarchy still rank candidates well when their contact states differ substantially from recorded training trajectories?
- How much of Full’s improvement survives equalizing recovery attempts and execution time across monitoring baselines?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs synchronized bilateral grids, contact/slip labels, action alignment, episode-level splits, the pretrained WAM and FG-CLTP encoder, plus successful validation episodes for KDE calibration. Recover exact label thresholds, onset/release loss definitions, cost/progress implementations, KDE bandwidth/threshold values, and safe-retreat parameters: the supplied method gives their roles without a complete executable specification. e-contexte-hierarchye-losse-selectione-verificatione-datae-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the forecast identifies the better generated candidate
Reader-proposed check, not performed: reproduce K=4 generation with fixed task resets, seeds, validity rules and score weights. Compare normal tactile ranking with task-progress ranking and with a control that randomly permutes tactile forecasts among the same four candidates. Disable online corrections in all arms. Log each candidate’s forecast before selection; in a resettable subset, execute each valid candidate to measure its actual tactile trajectory and task outcome. Report candidate-wise forecast errors, ranking agreement with realized outcomes, and success with episode-level uncertainty. If correct pairing consistently beats shuffled pairing and task-progress scoring on newly generated actions, it supports the intended mechanism; no pairing advantage would challenge candidate-specific foresight despite good recorded-sequence metrics. e-contexte-selectione-protocole-rankinge-limitations
Check 2: Separate anomaly detection from extra recovery opportunities
Reader-proposed check, not performed: keep candidate selection, eight-step prefixes, safe-retreat policy and the two-correction maximum fixed. Compare the forecast-residual KDE monitor with a current-touch monitor calibrated to the same validation false-trigger rate. Include normal runs and controlled missed-contact, slip and lateral-jamming disturbances with fully annotated onset times; fix all thresholds before testing. Measure false triggers, missed anomalies, detection delay, completion rate and execution time under equal recovery budgets. A lower miss rate or faster detection at matched false-trigger rate, accompanied by higher completion, would support forecast-based verification. Equal detector performance with only extra recovery time explaining success would weaken that interpretation. e-motivatione-verificatione-protocole-monitor-resultse-limitations
8.3 Reading coverage
Visual audit: All eight pages of the supplied v1 PDF were rendered and visually inspected, including the title/author/version block, all seven figures, all three tables, Equations (1)–(6), training and hardware details on page 5, evaluation protocol on page 6, and limitations on page 8. All six final original crops were separately viewed; clipping in the conditioning labels and verification heading was corrected and the revised crops were viewed again. Crops retain graph legends, axes and table headers; table units/protocol notes and figure uncertainty definitions are explained in their reading guides from the inspected captions and text. No appendix is present. No external supplement or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF pp. 1–2: title, author block, abstract, I. Introduction
- PDF p. 2: II. Related Work, A–C
- PDF pp. 3–5: III. Method, A–D, Eqs. (1)–(6)
- PDF pp. 5–6: IV. Experiments, A–C
- PDF pp. 6–7: V. Results, A–D
- PDF p. 8: VI. Conclusion and References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the title and all six authors match the catalog after normalizing name order. The inspected title page states arXiv:2608.19574v1 [cs.RO], 20 Aug 2026; the catalog submittedDate is 2026-08-19. This report describes the supplied v1 artifact; the one-day date discrepancy is unresolved, and no other edition was supplied or compared.
- Acquisition omission preserved: text extraction does not reconstruct figure images; all eight retained PDF pages and Figures 1–7/Tables I–III were subsequently visually inspected.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- No code or external links were inspected, and no experiments were reproduced. No appendix is present in the supplied eight-page PDF.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block and arXiv margin stamp
Exact title and six authors: Chao Xue, Chaofan Zhang, Wenxuan Ma, Guocai Yao, Shaowei Cui, Shuo Wang. Affiliations are Institute of Automation, Chinese Academy of Sciences; ImprintX Robotics; Beijing Academy of Artificial Intelligence. Stamp identifies arXiv:2608.19574v1 [cs.RO], 20 Aug 2026.
Go to primary source ↓e-motivationPDF pp. 1–2, I. Introduction and Figure 1
Visually plausible candidates can differ in missed contact, pressure, slip or jamming. The proposed forecast accompanies each action through selection and later execution verification.
Go to primary source ↓e-contextPDF p. 3, III.A, Eqs. (1)–(2), Figure 2(a)
History includes RGB, robot state, bilateral touch and language; action chunks are H by 7. FG-CLTP features become tactile tokens. Only tactile queries access the other streams; video/action queries cannot read tactile keys. Listed trainable modules are tactile projections/queries, action–tactile fusion and tactile heads.
Go to primary source ↓e-hierarchyPDF p. 4, III.B, Eqs. (3a)–(3c); PDF p. 3, Figure 2(b)
Contact transition features condition deformation; detached contact and ungated future-deformation magnitude condition slip. Contact gates physical outputs. Deformation targets are future-minus-current grids; loss targets are ungated increments/probabilities. Contact interventions substitute learned embeddings indexed by the intervened value.
Go to primary source ↓e-lossPDF p. 4, III.B, Eq. (4)
Joint loss combines contact BCE and timing losses, contact-masked Huber deformation loss, and weighted slip BCE; detached inputs prevent direct slip-loss updates to upstream heads.
Go to primary source ↓e-selectionPDF p. 4, III.C, Eq. (5)
Stochastic candidate generation, kinematic filtering and batched tactile prediction precede ranking. Normalized costs encode phase-inconsistent contact, excessive/imbalanced deformation, and peak/mean slip risk, alongside visual task progress.
Go to primary source ↓e-verificationPDF pp. 4–5, III.D, Eq. (6); PDF p. 7, Figure 7
Aligned causal observations produce contact/slip absolute residuals and deformation Top-20 Z MAE. Task-standardized residuals are scored by a KDE fitted on successful validation episodes. Persistent deviations abort the remainder, retreat, reobserve and regenerate.
Go to primary source ↓e-dataPDF p. 5, IV.A and Figure 4
IMETA-Y1, bilateral DM-Tac W2, two RealSense D435i and one USB camera. Each task has 200 episodes split 160/20/20 before windows. At 30 Hz, tactile offsets are −12,−6,−2,0, action offsets 0–23, target offsets 1–24. Deformation labels reference the sensor’s no-contact grid.
Go to primary source ↓e-trainingPDF p. 5, IV.A, right-column training paragraph
Per-task 10,000-step training uses eight NVIDIA H100 GPUs, AdamW at 10^{-4}, batch 64; onset/release weights 2/1, slip-positive weight 30, and beta 0 for 3,000 steps then 0.01.
Go to primary source ↓e-metricsPDF p. 6, IV.B and V.A
Timing MAE uses 30-Hz frames; deformation metrics are restricted to ground-truth contact. Top-20 Z selects points by ground-truth absolute normal deformation; onset variant uses onset frames. Slip prevalence is 1.39%. LPIPS and normalized action MAE assess output preservation.
Go to primary source ↓e-protocolPDF p. 6, IV.C
Each method has 30 trials/task; chip success requires intact delivery and 3 s stability, erasing requires contact and >80% target removal, USB success requires full insertion without sustained jamming plus LED activation. Ranking uses four candidates and weights (1,1,1,0.5). Prefix E=8; three consecutive exceedances trigger correction; at most two corrections. Fixed-budget arms share seeds but execute independently. Detector annotations are post-hoc.
Go to primary source ↓e-successPDF p. 6, Table I, all task columns and Average
DreamZero 33.3/50.0/10.0/31.1; Reactive Tactile 40.0/60.0/20.0/40.0; Selection 70.0/73.3/40.0/61.1; Full 76.7/90.0/50.0/72.2 (%). Thirty trials per method/task, 360 total.
Go to primary source ↓e-predictionPDF p. 7, Table II, all rows; PDF p. 5, Figure 3
Table II mean F1/onset/release/3D L2/Top-20 Z/norm/AUPRC: 0.921/2.1/2.3/0.058/0.076/0.960/0.247. Task F1 is 0.924/0.908/0.931 and slip AUPRC 0.232/0.251/0.258. Figure 3 presents two representative 24-step forecasts.
Go to primary source ↓e-ablationPDF p. 7, Table III, caption and all rows; V.B
Contact-only F1 0.903; deformation-only L2 0.068; independent heads F1/L2 0.896/0.064; contact-conditioned 0.915/0.059; directed hierarchy 0.921/0.056. Hierarchy Top-20 Z 0.072 and norm 0.97. Deformation-only supervision uses contact frames; independently trained rows need not match Table II. Budgets are matched.
Go to primary source ↓e-conditioningPDF p. 6, Figure 5(a–b) and caption; PDF p. 7, V.B
GT/predicted/shuffled/zero contact yield L2 0.053/0.056/0.071/0.078 and onset Top-20 Z 0.068/0.072/0.096/0.103 mm. Slip-only/shared-state/contact/deformation cumulative conditioning yields AUPRC 0.154/0.188/0.218/0.247. Contact intervention uses a fixed checkpoint. Error bars are episode-paired bootstrap 95% CIs in (a), stratified episode-bootstrap 95% CIs in (b).
Go to primary source ↓e-rankingPDF p. 7, Figure 6/caption and V.C; PDF p. 6, IV.C
At K=4, tactile ranking reuses the 90 Selection trials and reaches 61.1%; extra independent task-progress trials reach 35.6%. Matched seeds, task distribution and validity rules; two-sided Wilson 95% intervals are [50.8,70.5] and [26.4,45.8]. DreamZero uses one candidate.
Go to primary source ↓e-monitor-resultsPDF p. 7, V.D and Figure 7
Post-hoc review of 90 Full trials found 18 replanning triggers, 16 associated with anomalies and 12 followed by success; two false triggers had no observed outcome effect. At most three of 21 untriggered failures involved an undetected anomaly. Figure 7 shows one real execution trace, with three consecutive threshold exceedances.
Go to primary source ↓e-limitationsPDF p. 8, VI. Conclusion, final paragraph
The authors state that forecast quality on model-generated chunks is not separately evaluated and slip risk is not calibrated for online detection; controlled anomalies, on-policy adaptation, calibrated uncertainty and learned long-horizon correction remain future work.
Go to primary source ↓e-preservationPDF p. 7, V.B, final sentences; PDF p. 8, VI. Conclusion
Relative to the WAM without tactile prediction, video LPIPS and action MAE are higher by 1.4% and 0.9%; no absolute values are supplied here. Current-grid/base-grid prior L2 and Top-20 Z are 0.079/0.104 and 0.073/0.099 mm.
Go to primary source ↓8.5 Primary sources
HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation ↗
PDF · 5,950 extracted words
Source fingerprint
a7fde63b1c8c1aee4afc3d5b537247209a28c7dc134eb4f6664189d4e3fd302a