WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation
1. Paper overview
In one sentence: A predictive visuotactile memory can improve dexterous control through policy conditioning, while its dependence on pretraining and limited severe-disturbance recovery constrain the generalization claim. e-probleme-rssme-policye-priore-stress
| At a glance | What to know |
|---|---|
| Research problem | Source description Sustained in-hand rotation requires maintaining contact while inferring partially observed geometry, pose, and slip risk. Object-specific finger gaits can fail after pose shifts or external forces. The authors seek deployable state estimation without object-ID annotations, on a 22-DoF five-finger hand. e-probleme-interface |
| Core mechanism | |
| A key reported result | Twenty-object real-robot z-axis rotation: 175/200. Rotation success count. Nine training objects plus eleven unseen objects; ten trials each, 200 total; half-spin success criterion. In-Hand Rotation: 53/200; other baselines: 33/200–45/200. Broad hardware rotation evidence; this aggregate combines seen/unseen objects and does not measure hardware perturbation recovery. e-stresse-protocol |
| Reading caution | Source description Tasks remain short-horizon. Oversized objects, offsets outside the recovery basin, drift, and jamming cause failures. Screwdriver translation/rotation is illustrated qualitatively without a quantitative tool-use benchmark. e-failurese-tool |
Core contributions
Figure 1. Predictive learning provides recurrent context to an independently optimized manipulation policy. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the blue panel as representation learning, the green panel as reuse across object distributions, and the orange panel as action generation. Proprioception, touch, depth, and the previous action feed the recurrent model; prediction heads supervise its state. The small state diagram distinguishes deterministic h from stochastic z. In the orange panel, current observations, the commanded axis, and WSM features converge on the actor–critic, whose relative action updates the joint target. Equations (5–9) provide the more precise implementation: the actor receives a projection of detached deterministic state, while the RSSM is optimized separately from PPO. The green panel summarizes intended capabilities rather than a separate planning algorithm. e-architecturee-encodere-rssme-policye-priore-interface
What it supports. The mechanism is recurrent perception for control. WSM supplies information about interaction history to the actor, and a low-level controller executes the actor’s relative joint commands. Reusing the predictive model can change what the policy knows without turning the world model into an action-search or imagined-rollout planner.
Where the evidence stops. The reward-to-WSM arrow is inconsistent with Sections 3.2–3.3 and A.1, which specify reward only as a training target. Follow the explicit update equations for deployable inputs; the faithful crop preserves this unresolved diagram discrepancy.
2. Motivation
2.1 The problem and the proposed response
Sustained in-hand rotation requires maintaining contact while inferring partially observed geometry, pose, and slip risk. Object-specific finger gaits can fail after pose shifts or external forces. The authors seek deployable state estimation without object-ID annotations, on a 22-DoF five-finger hand. e-probleme-interface
2.2 What this reading follows
Rotating an object inside a hand requires deciding which contacts can support the next movement from incomplete, noisy observations. WM-Craftnet learns a recurrent state from depth, touch, proprioception, and previous actions, then gives that state to a separate reinforcement-learning controller. Clean simulator depth and other prediction targets supervise the state during training. This reading follows the distinction between learning useful perception and planning inside a world model, then separates ordinary rotation, held-out-object transfer, and perturbation recovery. The experiments support useful control improvements, but the strongest hardware success count and the much lower simulated recovery rate answer different questions. e-probleme-rssme-policye-priore-stress
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned, so no existing quadrant can be endorsed. Architecturally, WSM predicts action-conditioned latent dynamics while a separate PPO actor produces actions through a stopped-gradient interface. This supports modular world-model-conditioned control, not a demonstrated single joint future/action predictor or inverse-dynamics controller. e-rssme-policy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Trace one executed action through two learning systems
Begin with what the robot can measure: noisy wrist depth, hand proprioception, binary contact, and its previous command. WSM takes compact single-step sensor vectors, whereas the actor receives short proprioceptive/contact stacks. The encoder forms the current observation embedding; the RSSM combines that embedding with action and latent history. Its deterministic state is projected and detached before entering the actor alongside the commanded axis and current observations. The actor then produces relative joint-target increments, followed by smoothing and hardware-limit clamping. New sensor measurements close the loop. During training, PPO supplies interaction data and receives privileged critic information, while replay chunks train WSM through prediction losses. The stopped-gradient boundary matters: useful predictive context influences decisions without the PPO objective differentiating through the world model or optimizing imagined trajectories. e-interfacee-encodere-rssme-policye-training
Figure 4. A clean reconstruction target shapes the latent state even though decoded depth is not the main actor’s input. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) supplies a third-person RGB view of the simulated hand and duck; it is scene context, not an additional actor modality. Panel (b) is ground-truth wrist depth. Compare the holes and artifacts in noisy input (c) with predicted depth (d), then compare (d) against (b) to judge preservation of the hand-object silhouette. The training setup explains this arrangement: corrupted depth enters the encoder while clean depth supervises reconstruction. The main WM-Craftnet actor consumes noisy depth and projected recurrent context. The separate IHR-plus-denoised-depth baseline is the experiment that directly feeds reconstructed depth into another controller. e-diagnosticse-encodere-interfacee-hardware-foure-variants
What it supports. The example visibly supports suppression of depth artifacts while retaining coarse hand-object geometry. Together with the clean-versus-noisy-target ablation, it motivates denoising as useful representation supervision. The relevant control benefit is carried through the learned state; the reconstruction picture itself is a diagnostic output.
Where the evidence stops. This is one simulated reconstruction example, not a quantitative depth-accuracy benchmark or a demonstrated future rollout. The supplied PDF’s raster depth panels remain coarse even at higher rendering resolution; no missing geometry or numerical error is inferred.
5.2 Distinguish memory from the supervision that makes memory useful
A recurrent feature can summarize history even without a predictive objective. The appendix tests that alternative with LSTM and GRU histories: their returns are 621.0 and 497.8, compared with full WSM’s 753.3. The main ablations then ask more specific questions about clean-depth targets, recurrent state, and per-step context; the auxiliary study changes value, pose, and shape supervision. Reader interpretation: the evidence favors a bundle of predictive representation choices over the reported alternatives, but does not isolate an abstract benefit of having a world model independently of capacity, data, or privileged targets. The stability tradeoff is especially instructive. Removing deterministic recurrence lowers return while improving OffAxis and AngVar. A reproduction should therefore track rotation progress and stability separately, rather than declaring every lower-return representation physically worse. e-historye-variantse-auxiliarye-rssme-protocol
5.3 Separate adapting a prior from testing an unseen object
The 49-object experiment begins with WSM pretrained in the nine-object z-axis setting. The transferred encoder, RSSM, decoder, and reward predictor continue learning from downstream rollouts, while a target controller is learned. Its reported 9.37 ± 0.13 radians per episode after 3000 epochs, against 3.28 without the prior, is therefore evidence about downstream learning. In contrast, Table 6 evaluates axis-specific policies on four held-out objects per axis without object-specific fine-tuning. A third protocol aggregates hardware rotation across twenty seen and unseen objects. None of these protocols should silently substitute for another. The separate severe-disturbance recovery experiment asks whether the policy can restore a controllable grasp; its low absolute success rate marks a boundary that broad ordinary-rotation success does not remove. e-priore-scalinge-stresse-failures
5.4 Training and inference
During training
PPO collects simulation rollouts; contiguous replay chunks update WSM after each PPO epoch through a separate gradient path. Reconstruction, reward prediction, dynamics/representation KL regularization, and pose/value/shape auxiliary heads supervise WSM. Pose and shape targets use simulator labels; shape is a basis-point-set-to-mesh displacement vector. e-rssme-policye-prior
Downstream transfer carries the encoder, RSSM, depth/proprioceptive decoder, and reward predictor into the 49-object task. These modules continue adapting; the actor–critic and task-specific heads learn the target distribution. The prior is therefore not frozen. e-prior
The shared baseline reward favors signed target-axis rotation and penalizes drift, off-axis motion, effort, large actions, and unstable contacts. Reset conditions include dropping, axis deviation, prolonged insufficient spin, and the horizon. e-policye-training
During inference
The main actor uses noisy sensor depth plus latent context. Neither clean depth, reconstructed depth, reward, nor object-ID labels is a deployment input. Simulator state supports critic/reward learning and auxiliary supervision. WSM does not optimize the policy through imagined rollouts. e-interfacee-rssme-policye-prior
5.5 Implementation flow
- Encode complementary sensors
MLPs encode proprioception and contact; a CNN encodes clipped, cropped depth. Temporally correlated dropout, Gaussian noise, and small image rotations corrupt the input, while the reconstruction target remains clean crop-only simulator depth. e-encoder
- Update predictive memory
An action-conditioned RSSM combines the previous stochastic and deterministic states with the current observation embedding. Its 512-dimensional deterministic state is detached and projected to 16 dimensions for policy conditioning. e-rssm
- Close the control loop
The actor combines sensor observations, commanded axis, and projected memory. Relative joint targets are smoothed and clamped before execution; new observations update WSM at the next control step. e-interfacee-policye-training
6. Experiments & results
WM-Craftnet learns a predictive visuotactile state that conditions a separate dexterous manipulation policy. Its main contribution is recurrent perception for executed control: noisy depth, touch, proprioception, and action history inform a latent state trained with clean-depth and other prediction targets. Hardware rotation improves substantially, but severe-disturbance recovery remains limited (e-rssm, e-policy, e-stress).
6.1 Read the original evidence
Table 1. Pretraining and sensor composition both matter to the z-axis comparison. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the object grid, which defines the nine-object rotation setting. The top row group compares reimplemented baselines and WM-Craftnet trained from scratch. The middle group masks tactile observations at test time; it does not represent independent training without touch. The bottom group compares pretrained WSM modality controls with the full representation. Return, episode length, and RotR favor larger values; OffAxis and AngVar favor smaller values. Entries are means with 95% confidence intervals over 128 evaluation episodes. Compare complete rows: a higher return can coexist with slower rotation or a worse stability measure, so the highlighted cells should not be read as one universal ranking. e-ze-protocole-prior
What it supports. Full WSM reaches return 753.3 ± 3.6 and RotR 1.293 ± 0.004. The scratch model reaches return 414.3 ± 15.7 but RotR 0.742 ± 0.012, below Touch Dexterity’s 1.018 ± 0.004. The full model’s strongest result therefore includes a substantial pretraining advantage.
Where the evidence stops. Test-time tactile masking probes reliance after training; it does not establish that tactile supervision was unnecessary. The prop+tac control has better OffAxis than full WSM, and episode confidence intervals do not measure variation across independent training runs.
Table 6. Generalization, recovery, and hardware rotation occupy separate columns and separate protocols. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three paired RotR/Return columns as separate zero-shot simulation evaluations, each using the four held-out objects shown for its axis. Those object sets are not the 49-object downstream adaptation set. Next, Sim. Recovery measures recovery from severe simulated perturbations, and Sim. Time averages elapsed time among recovered trials. Finally, Real Rot. SR counts standard hardware rotation successes across twenty objects with ten trials each. The latter mixes nine training objects and eleven unseen objects. Keeping these denominators and conditions separate prevents the large hardware success count from being mistaken for a hardware recovery statistic. e-stresse-protocole-scalinge-failures
What it supports. WM-Craftnet records 175/200 hardware rotation successes versus the strongest baseline’s 53/200. Severe simulated recovery is much lower: 14.1% ± 6.0%, versus 6.2% ± 4.2% for Touch Dexterity. Mean recovery time is 2.56 ± 0.77 seconds among successful trials. These results support improved robustness with a substantial remaining failure region.
Where the evidence stops. Conditional recovery time excludes unsuccessful recoveries, and the hardware total does not isolate unseen objects. Figure 6 contains selected successful sequences; it cannot establish a hardware recovery rate. The three paired held-out columns omit per-entry confidence intervals.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Nine-object z-axis rotation in simulation Reimplemented baselines share reward, randomization, initialization, and resets; 128 evaluation episodes; mean ± 95% confidence interval. | Full WSM: 753.3 ± 3.6; 1.293 ± 0.004. Return; rotation rate (RotR) | Touch Dexterity: 386.9 ± 3.1; 1.018 ± 0.004. WM-Craftnet from scratch: 414.3 ± 15.7; 0.742 ± 0.012. The pretrained model improves both metrics; the scratch model improves return but rotates slower than Touch Dexterity. Pretraining is a material comparison factor. e-protocole-z |
| Controlled WSM mechanism ablations Nine-object z-axis simulation; mean ± 95% confidence interval over 128 evaluation episodes. | Full WSM return 753.3 ± 3.6. Return; OffAxis and AngVar (lower is better) | Noisy-depth targets: 708.0 ± 3.5; no previous deterministic state: 705.4 ± 4.3; per-step encoder–decoder context: 667.6 ± 4.7. Removing recurrence improves OffAxis/AngVar to 1.125/1.185 from 1.225/1.324. Return gains do not imply uniformly better stability. e-protocole-variants |
| Axis-specific x/y rotation in simulation Separate WSMs/policies for four x-axis and nine y-axis objects; 128 evaluation episodes per method; mean ± 95% confidence interval. | x: 432.8 ± 2.1; 1.114 ± 0.001. y: 272.7 ± 13.5; 1.025 ± 0.026. Return; RotR | Best baseline return: x 295.1 ± 3.3 (In-Hand Rotation), y 177.8 ± 9.2 (Blind RL). Both tasks improve, but this is not one demonstrated unified multi-axis policy. e-protocole-axes |
| 49-object downstream policy learning Nine-object z-axis WSM pretraining, then adaptation and PPO on 49 new objects; result after 3000 epochs. | 9.37 ± 0.13 rad on average per object. Radians rotated per episode | No-prior baseline: 3.28 rad. This measures learned downstream transfer, not zero-shot performance on the 49 objects. Section 4.3 does not separately specify the aggregation underlying ±0.13. e-scalinge-prior |
| Four-object real-robot z-axis rotation Duck, cross block, corner block, and unseen double-notched block; ten 20-second trials per object; success requires more than half a spin. | Double-notched block: 4.32 rad; 8/10 successes. Unseen-object rotation amount (RR); success count (SR) | All four comparator rows, including IHR with WSM-denoised depth, report 0/10 on this object. The unseen-object result favors recurrent context; other Table 3 cells have an unresolved RR/SR consistency issue described below. e-hardware-foure-protocol |
| Severe simulated z-axis perturbation recovery Recovery to a controllable configuration; recovery time is conditional on successful trials; simulation mean ± 95% confidence interval. | 14.1% ± 6.0%; 2.56 ± 0.77 s. Recovery success; recovery time | Best baseline success: Touch Dexterity 6.2% ± 4.2%. Fastest baseline: depth-only RL 3.14 ± 0.79 s. The absolute success improvement is 7.9 percentage points. Most severe disturbances still defeat the method. e-stresse-protocol |
| Twenty-object real-robot z-axis rotation Nine training objects plus eleven unseen objects; ten trials each, 200 total; half-spin success criterion. | 175/200. Rotation success count | In-Hand Rotation: 53/200; other baselines: 33/200–45/200. Broad hardware rotation evidence; this aggregate combines seen/unseen objects and does not measure hardware perturbation recovery. e-stresse-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2, quantitative panel. Clean-depth targets and recurrent context improve return, with a stability tradeoff. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a targeted change to the predictive representation on the z-axis benchmark. Noisy-depth supervision changes the reconstruction target from clean depth to corrupted depth. The no-previous-state row removes the previous deterministic-state input. The encoder–decoder row replaces recurrent policy context with a per-step feature. The full row retains the proposed combination. The crop contains the original quantitative panel; the associated training curve remains on the inspected source page. Focus first on Return and RotR, then inspect the two lower-is-better stability columns. This order exposes why the authors’ performance claim should be stated in terms of particular metrics. e-variantse-rssme-encodere-protocol
What it supports. Full WSM has the highest return, 753.3 ± 3.6, versus 708.0 ± 3.5 with noisy targets and 667.6 ± 4.7 with per-step context. Removing recurrence gives 705.4 ± 4.3 return but improves OffAxis and AngVar to 1.125 and 1.185, respectively.
Where the evidence stops. The variants support task-performance benefits under the reported setup. They do not show that recurrence improves every stability measure, nor provide a full factorial test of interactions between clean-depth supervision, memory, auxiliary heads, and pretraining.
Table 10. Training-only prediction targets improve different aspects of the learned state. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the first row with each single-head row before reading the full combination. The base multimodal predictive model remains; the ablation varies the additional value, pose, and object-shape objectives. These heads supervise representation learning, with simulator labels supplying pose and shape targets. They are not new real-robot input sensors. Return makes the value-only row especially prominent, while AngVar reveals a different ranking for pose-only supervision. The full row is strongest on return and RotR, but not on every column. The table therefore supports differentiated effects of auxiliary targets rather than one interchangeable benefit from adding any prediction head. e-auxiliarye-rssme-priore-protocol
What it supports. Return rises from 688.1 ± 3.7 without auxiliary heads to 737.0 ± 4.9 with value alone and 753.3 ± 3.6 with all heads. Pose-only reaches AngVar 1.231 ± 0.011, lower than the full model’s 1.324 ± 0.027. The best objective combination depends on the metric.
Where the evidence stops. This does not independently identify the physical meaning of each latent coordinate. Auxiliary objectives also change the learning signal, so attributing all gains to action-conditioned dynamics would overlook the privileged supervision used during simulation training.
7. Analysis & limitations
7.1 What the evidence leaves open
Tasks remain short-horizon. Oversized objects, offsets outside the recovery basin, drift, and jamming cause failures. Screwdriver translation/rotation is illustrated qualitatively without a quantitative tool-use benchmark. e-failurese-tool
Figure 1 draws reward entering WSM, but Sections 3.2–3.3 and A.1 explicitly make it a training target. Table 8 labels object state reward/critic-only, while the method explicitly uses simulator pose/shape labels for auxiliary heads; its summary is incomplete. e-architecturee-encodere-rssme-interfacee-prior
Table 3 reports IHR with denoised depth at 2.43 rad and 10/10 successes on the corner block, despite the more-than-half-spin definition. The supplied text does not reconcile these quantities; no corrected values are inferred. e-hardware-foure-protocol
Episode confidence intervals do not establish training-seed robustness. t-SNE structure and one denoised frame support representation diagnostics, not calibrated physical-state inference or future prediction accuracy. e-protocole-diagnostics
The metrics paragraph names RotR, OffAxis, and AngVar but does not specify their units or normalization. This report retains those labels rather than assigning inferred units. e-protocol
7.2 Questions for discussion
- Would predictive losses retain their advantage with matched pretraining data, recurrent capacity, and training compute? (e-history, e-prior)
- How much transfer survives when the pretrained WSM is frozen rather than adapted? (e-prior, e-scaling)
8. Reproducibility audit
8.1 Requirements and known gaps
Reported settings include 1024 simulation environments, PPO horizon 16, WSM batch/chunk length 16/32 with ten updates per PPO epoch, and Adam learning rate 0.0001. Hardware is a Sharpa Wave hand with RealSense L515: control/camera/touch at 10 Hz, physics at 60 Hz. e-training
Reproduction needs joint-level system identification, object assets, clean-depth and privileged auxiliary targets, and the reported dynamics/sensor randomization. GPU resources, simulator/software versions, total compute, and training-seed counts are unspecified. Missing implementation settings include exact reset thresholds, target smoothing, action scale, Gaussian depth-noise magnitude, tactile threshold/latency ranges, and auxiliary-loss weights. Replay capacity is stated as 256 rows versus 10,000-transition warmup without clarifying row semantics. e-traininge-rssme-prior
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test clean-depth supervision under matched memory and data
Reader-proposed check, not performed: build a factorial comparison of clean versus noisy depth targets and recurrent versus per-step policy context. Match pretraining interactions, observation modalities, auxiliary objectives, actor architecture, and optimization budget; repeat across independent training seeds. Use identical held-out evaluation seeds and a preregistered depth-corruption sweep. Report return, RotR, OffAxis, AngVar, and clean-depth reconstruction error separately. The denoising explanation predicts a larger clean-target advantage as sensor corruption increases. If that interaction is absent, or return gains disappear under matched budgets, the proposed mechanism needs qualification even if reconstruction images look cleaner. e-encodere-variantse-rssme-historye-protocole-training
Check 2: Audit hardware rotation success separately from recovery
Reader-proposed check, not performed: repeat the twenty-object hardware protocol with separate seen/unseen totals and a baseline under matched resets. Save per-trial angular trajectories and independently scored videos so RR and the more-than-half-spin success indicator can be recomputed from one documented rule. This should resolve whether Table 3’s 2.43-rad/10-of-10 corner-block entry reflects an undocumented scoring convention or an error. Add a separately labeled perturbation condition with specified disturbance magnitudes and a preregistered recovery criterion; report all-trial recovery success alongside successful-trial recovery time. Broad ordinary-rotation gains that vanish under perturbation would bound the robustness claim rather than invalidate the rotation result. e-stresse-hardware-foure-protocole-traininge-failures
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version page, Figure 1 and method equations, all numerical result and ablation pages, and Appendix A–B implementation/diagnostic pages. All six final original crops were viewed: Figure 1, Table 1, Table 2 quantitative panel, Figure 4, Table 6, and Table 10. Figures 2–9 and Tables 1–10 were inspected on their source pages. These pages cover every retained method, numerical, evaluation, and reproduction claim, including evidence used in uncropped tables. Page 2 and reference pages 10–11 were read as text; they provide no otherwise-uncovered quantitative or method claim here. External videos, code, and supplements remain uninspected. Figure 1’s reward-input arrow and Table 8’s simplified privilege summary are qualified against the method text; Table 3’s RR/SR discrepancy remains unresolved.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 WM-Craftnet Framework; 3.1–3.5
- 4 Experiments and Evaluations; 4.1–4.5
- 5 Conclusions and Limitations
- References
- A Implementation Details; A.1–A.3
- B Additional Experiments; B.1–B.4
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The inspected title page identifies arXiv:2609.07002v1 [cs.RO], 7 September 2026. Its title and all six authors match the catalog. Only this supplied revision was reviewed; no other edition was compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF pages and final crops.
- Separate supplemental material availability has not been fully verified.
- The linked project website, supplementary videos, code, and external references were not opened. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/byline/affiliation and arXiv margin
Title and six-author byline match the catalog; Sharpa Robotics is the affiliation. Margin identifies arXiv:2609.07002v1 [cs.RO], 7 Sep 2026.
Go to primary source ↓e-problemPDF p. 1, Abstract and Section 1
Partial observations, changing contact, noisy depth, object variation, and perturbations motivate state estimation for robust in-hand manipulation.
Go to primary source ↓e-interfacePDF p. 3, Section 3.1, Eq. (1); p. 13, Table 8
Deployable actor inputs comprise stacked proprioception/contact, noisy depth, previous command, axis, and WSM context. The 22-DoF action is a relative joint target; clean/reconstructed depth is excluded from actor observations.
Go to primary source ↓e-architecturePDF p. 3, Figure 1 and caption; p. 4, Sections 3.2–3.3; p. 12, A.1
The overview separates predictive learning, prior reuse, and conditioned control. Its reward-to-WSM input arrow conflicts with the text specifying reward only as a prediction target.
Go to primary source ↓e-encoderPDF p. 4, Section 3.2, Eqs. (3–4)
MLP proprioception/contact and CNN depth branches encode single-step WSM inputs; the actor uses stacks. Noisy augmented depth is input; clean crop-only depth and reward are targets.
Go to primary source ↓e-rssmPDF p. 4, Section 3.3, Eqs. (5–7)
RSSM posterior update uses previous state/action, embedding, and episode-start flag. Detached 512-dimensional deterministic state projects to 16 dimensions. Multimodal/reward losses, KL terms, and pose/value/shape supervision train the representation.
Go to primary source ↓e-policyPDF p. 5, Section 3.5, Eqs. (8–9); p. 1, Abstract
A separate asymmetric PPO policy consumes observations, axis, and projected state; relative targets are smoothed/clamped. WSM updates after PPO epochs through a separate gradient path; no imagined policy rollouts are used.
Go to primary source ↓e-priorPDF p. 5, Section 3.4 continuation; p. 12, A.1
Nine-object pretraining initializes 49-object training. Transferred encoder, RSSM, depth/proprioceptive decoder, and reward predictor keep adapting; target actor–critic and task-specific heads are learned. Auxiliary objectives use simulator labels during pretraining and adaptation.
Go to primary source ↓e-protocolPDF p. 5, Section 4.1; p. 6, Metrics paragraph
Baselines are reimplemented with shared reward/randomization/initialization/resets. Simulation metrics use 128 episodes and mean ± 95% confidence intervals unless noted. Hardware RR is radians per 20 seconds; SR requires more than half a spin. Four annotators score videos independently and measurements are averaged.
Go to primary source ↓e-zPDF p. 6, Table 1, all rows; Section 4.2
Full WSM return/RotR are 753.3 ± 3.6/1.293 ± 0.004; scratch gives 414.3 ± 15.7/0.742 ± 0.012. Tactile masking and pretrained modality controls are separate row groups.
Go to primary source ↓e-variantsPDF p. 6, Table 2 quantitative panel and Controlled WSM variants paragraph
Full/noisy-target/no-recurrence/per-step-context returns are 753.3/708.0/705.4/667.6. No-recurrence OffAxis/AngVar are 1.125/1.185 versus full 1.225/1.324.
Go to primary source ↓e-hardware-fourPDF p. 7, Table 3 and Section 4.2 continuation; p. 6, Metrics
WM-Craftnet achieves 4.32 RR and 8/10 SR on the unseen block; other methods report 0/10. Denoised-depth IHR reports corner-block RR 2.43 with SR 10/10, unresolved against the stated half-spin criterion.
Go to primary source ↓e-diagnosticsPDF p. 7, Figures 3–4 and Section 4.3; p. 15, Figure 8 and B.2
Figure 4 juxtaposes RGB context, ground-truth depth, noisy input, and WSM reconstruction for a simulated duck rollout. t-SNE shows object-dependent and shared regions; labels color diagnostics rather than entering the policy.
Go to primary source ↓e-scalingPDF p. 7, Section 4.3, Reusable synesthetic prior; p. 15, B.2 and Figure 8
Downstream 49-object learning after 3000 epochs reports 9.37 ± 0.13 rad per episode versus 3.28 without the prior. The 49-object adaptation set is distinct from held-out zero-shot tests.
Go to primary source ↓e-axesPDF p. 8, Tables 4–5 and Section 4.4
Separate axis-specific WSMs are trained. On y-axis objects, full return/RotR are 272.7 ± 13.5/1.025 ± 0.026; x-axis gives 432.8 ± 2.1/1.114 ± 0.001.
Go to primary source ↓e-stressPDF pp. 8–9, Section 4.5; p. 9, Table 6 and Figure 6 caption
Four held-out objects per axis test zero-shot simulation transfer. Full x/y/z RotR are 1.002/0.715/0.842; returns 333.6/203.5/533.1. Severe simulated recovery is 14.1% ± 6.0%, with successful-trial time 2.56 ± 0.77 s. Separate hardware rotation is 175/200 across 20 objects, versus best baseline 53/200. Figure 6 selects successful sequences.
Go to primary source ↓e-failuresPDF p. 9, Section 5, Limitations
The authors limit claims to short-horizon tasks and acknowledge workspace, large-offset, drift, jamming, and severe-recovery failures.
Go to primary source ↓e-trainingPDF pp. 12–13, A.1–A.3, Eq. (10), Tables 7–8; p. 14, Figure 7
Appendix specifies hardware/control rates, PPO/RSSM settings, reward components, reset categories, calibration, and randomization. Table 7 lists 1024 environments, PPO horizon 16, WSM batch/chunk/updates 16/32/10, 256 replay rows, and 10,000 warmup transitions. It does not supply a full executable configuration, compute budget, or software versions.
Go to primary source ↓e-historyPDF p. 13, B.1; p. 14, Table 9
LSTM/GRU history baselines omit predictive losses. Their returns are 621.0 ± 4.3/497.8 ± 4.5 versus WSM 753.3 ± 3.6; full WSM also has higher RotR.
Go to primary source ↓e-auxiliaryPDF p. 14, Table 10 and B.1 discussion; p. 4, Section 3.3; p. 12, A.1
No auxiliary/value-only/pose-only/shape-only/full returns are 688.1/737.0/690.0/689.2/753.3. Pose-only gives AngVar 1.231 versus full 1.324; full yields the highest return and RotR.
Go to primary source ↓e-toolPDF p. 15, B.3–B.4 and Figure 9
The PDF illustrates goal-conditioned screwdriver translation and rotation; supplementary rollouts/failure cases are linked externally. No numerical tool-use evaluation is provided here.
Go to primary source ↓8.5 Primary sources
WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation ↗
PDF · 7,010 extracted words
Source fingerprint
333dcd69b5fc22ed3d8ed5d850191eb60eff1801f915af6c3362b7a23f03a8be