SANTS: A State-Adaptive Scheduler for World Action Models
1. Paper overview
In one sentence: SANTS spends video-denoising computation according to the current state, trading small potential action-quality gains against the latency of further refinement. e-identitye-interfacee-diagnostice-rewarde-robot
| At a glance | What to know |
|---|---|
| Research problem | Source description A visually refined future can be an expensive or misleading action condition. SANTS asks which intermediate video state supplies enough motion and contact information for each control decision. Its diagnostic varies video depth while fixing initial video/action noise, separating depth effects from sampling variation. e-interfacee-diagnostic |
| Core mechanism | Source description Two scheduler heads control terminal-state selection and continuous relative noise progression without changing the frozen video–action policy during scheduler training. e-interfacee-stope-progress |
| A key reported result | RoboTwin 2.0 manipulation: 94.4%; 523.7 ms. Three SANTS seeds: 94.1%, 94.5%, 94.6%; reported standard deviation 0.3 percentage points. Overall success; end-to-end policy inference latency. 50 tasks, 50 trials per task; comparable total adaptation budgets; A100 latency under the same pipeline. LingBot-VA: 92.2%, 2868.4 ms; Fast-WAM: 91.8%, 190 ms; pi0.5: 79.8%, 63 ms. Highest tabled success, with reported 81.7% latency reduction versus LingBot-VA full denoising; external policies remain faster. e-simulatione-tuning |
| Reading caution | Source description Generality is tested on RoboTwin 2.0 and two robot platforms with one backbone family. Joint backbone–scheduler optimization and adaptive budgets for action denoising, perception and communication remain untested. e-limitations |
Core contributions
- Source description
Two scheduler heads control terminal-state selection and continuous relative noise progression without changing the frozen video–action policy during scheduler training. e-interfacee-stope-progress
- Author claim
Scheduling and reward ablations support adaptive computation as the mechanism behind the reported tradeoff. e-schedulee-reward-ablation
Figure 2. A frozen video–action policy exposes intermediate features to two learned scheduling decisions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the token inputs through the joint self-attention and feed-forward blocks. The snowflake marks the frozen video–action Transformer, while the flame marks the scheduler trained afterward. Head A accumulates stopping evidence; Head B chooses the fraction of noise retained if denoising continues. Along the top trajectory, larger gaps represent skipped noise regions, and the scissors indicate early termination. Section 3.1 explains that the action branch conditions on the selected terminal video representation. The predicted action chunk is therefore downstream of the scheduling decision, while actual robot execution follows the separate platform interface. e-interfacee-stope-progresse-protocol
What it supports. The scheduler changes how the existing policy is queried rather than increasing its video/action policy capacity during post-training. Its two controls solve different problems: stopping selects the representation supplied to the action branch, while progression determines the route and computational cost of reaching that representation.
Where the evidence stops. The original diagram labels the threshold with local h_k, but Section 3.2 specifies cumulative F_k at deployment. This report follows the text and preserves the figure unchanged. The shared Transformer drawing also leaves the catalog’s Dual-system × IDM label insufficiently established.
2. Motivation
2.1 The problem and the proposed response
A visually refined future can be an expensive or misleading action condition. SANTS asks which intermediate video state supplies enough motion and contact information for each control decision. Its diagnostic varies video depth while fixing initial video/action noise, separating depth effects from sampling variation. e-interfacee-diagnostic
2.2 What this reading follows
A robot policy can use a predicted future without waiting for that future to become a fully denoised video. SANTS makes this choice explicit: a small scheduler reads intermediate video features, decides whether they are sufficient, and chooses the next noise level if refinement should continue. The backbone and action branch stay frozen during scheduler post-training. This reading follows the v3 paper from its offline depth diagnostic to controlled scheduling ablations and physical robot trials. The key distinction is between a future that looks refined and a representation that helps generate useful actions; the evidence also separates prediction error from executed task success. e-identitye-interfacee-diagnostice-rewarde-robot
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM and efficient inference are supported by inference-time future-video conditioning. The recorded Dual-system × IDM quadrant is less secure: Figure 2 places video/action tokens in a shared Transformer, and the text gives a future-conditioned action branch without a separate inverse-dynamics model or objective. A separate scheduler is not itself a second world/action system. Sequential conditioning alone does not establish the recorded architecture; no replacement quadrant is asserted. e-interfacee-progress
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First separate useful futures from polished futures
The depth diagnostic establishes the question that the scheduler must answer. The authors expose the action branch to intermediate video noise during target-domain tuning, freeze the policy, and then scan seven depths with paired initial noise. This makes the comparison about which intermediate representation conditions action prediction, rather than a contest between independently trained policies. Fine phases benefit more from refinement on average, yet individual trajectories frequently plateau or reverse. The paper therefore treats coarse versus fine labels as a diagnostic grouping, not as supervised stop labels. During actual inference, SANTS reads pooled video features and noise, and chooses its own depth. Figure 4 shows that those choices vary across control chunks, while explicitly leaving their per-decision optimality unproven. e-diagnostice-depth-tablee-interfacee-rollouts
Figure 3. Average gains from deeper denoising coexist with individual plateaus and reversals. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the 4% depth point, where each segment’s action error is normalized to one. Move right to follow the same segment as only video denoising depth changes; its initial video and action noise remain fixed. Gray lines reveal individual behavior, while orange and green curves summarize coarse and fine phases with standard-error bands. The diagnostic uses 500 manually annotated segments, with annotations made independently of depth-scan losses. Compare the stronger average decline in fine phases with the gray curves that flatten or rise. Table 1 on the following page quantifies these departures from monotonic improvement. e-diagnostice-depth-table
What it supports. Table 1 gives full-depth relative errors of 0.8943 for coarse phases and 0.6571 for fine phases. Full denoising is not the best scanned depth for 55.65% and 41.33% of those groups, respectively. The motivation for adaptation lies in this state-dependent variation, even though average curves improve.
Where the evidence stops. These are offline demonstration-action errors, not closed-loop success rates. Phase averages cannot identify the optimal depth for a particular state. The per-segment oracle in Table 1 uses hindsight and ignores computation; it is not a measured performance claim for SANTS.
5.2 Then separate stopping from movement along the noise axis
A stopping decision alone cannot choose how to traverse the remaining denoising trajectory. SANTS adds a progression head whose output is a fraction of the current noise level. A small fraction makes a large jump; a fraction near one preserves a finer update. During training, the Beta distribution permits exploration of different trajectories, and local stopping probabilities supply the path likelihood used by PPO. Deployment replaces sampled ratios with the predicted mode and stops when cumulative termination probability exceeds the threshold. This difference matters: accumulating evidence can terminate a path even when no individual local probability is large. The Figure 2 threshold label conflicts with the text, so a reproduction must explicitly document which rule it implements. The unchanged action branch then generates the chunk from the selected video condition. e-stope-progresse-ppo
5.3 Finally understand what the two reward anchors can teach
For each input, SANTS compares its generated action chunk with the demonstration and measures improvement relative to shallow denoising. A full-denoising anchor estimates how much error can be recovered on that input. The reward clips the normalized improvement, suppresses positive gains when the anchor gap is small or negative, and subtracts video-update cost. Sequence and temporal-difference errors provide complementary action signals; neither asks whether a video looks sharp. Reader analysis: this calibration helps prevent easy or low-information inputs from distorting the computation tradeoff, but it also creates a boundary. If both anchors are poor while an intermediate state is useful, the gate can suppress that useful state’s positive reward. Table S4 supports the aggregate reward design, but does not resolve this particular case or prove that every selected representation is best. e-errorse-normalizatione-reward-settingse-reward-ablation
5.4 Training and inference
During training
The Wan2.2-5B-based policy receives 30,000 RoboTwin adaptation updates, then 3,000 scheduler-only updates. Training uses multi-source simulated and real trajectories. The offline diagnostic specifically tunes the action branch with intermediate noise levels before freezing the policy. e-tuninge-diagnostic
Each sampled path generates actions alongside two no-gradient anchors at noise approximately 0.963 and 0. Sequence and temporal-difference errors cover position, quaternion orientation and gripper state. Anchor-normalized gains are clipped, difficulty-gated, weighted 0.6/0.4 and reduced by update cost. e-errorse-normalizatione-reward-settings
PPO uses AdamW at 10^{-4}, four epochs per path, clip 0.2, EMA baseline decay 0.95 and KL weight 0.02. Validation selects checkpoints using success, return and average updates; the video/action policy remains frozen. e-ppo
During inference
Deployment uses cumulative stopping and the Beta mode deterministically. On UR10, each generated chunk supplies 16 actions executed at 10 Hz before replanning; its approximately 1.6-second execution window differs from policy-query latency. e-stope-progresse-protocol
5.5 Implementation flow
- Expose the intermediate future
The frozen video–action Transformer predicts a flow field for latent x_k and supplies pooled features z_k from the same forward pass. The scheduler is a two-layer MLP with hidden width 512, bottleneck 256 and two heads. e-interfacee-scheduler
- Select the terminal representation
Nonnegative hazard increments accumulate along the path. Training samples local stopping probability h_k; deployment thresholds cumulative probability F_k. The default threshold is 0.85, start noise 1.00 and forced terminal noise 0.01. e-stope-scheduler
- Advance and generate actions
Continuation reduces noise by a relative ratio and applies the base first-order flow update. Stopping passes the current representation to the unchanged action-denoising process. A stopping decision still consumes a video forward pass but adds no state update. e-progress
6. Experiments & results
SANTS learns when to stop video denoising and how far to advance its noise level before a frozen action branch predicts a robot action chunk. Its reward measures demonstration-action agreement against shallow and full-denoising references, with a computation penalty. The v3 experiments report 94.4% RoboTwin 2.0 success at 523.7 ms and 73.1% mean real-robot success at 581.3 ms. These support a success–latency tradeoff within the evaluated tasks and backbone.
6.1 Read the original evidence
Table 2. The highest reported RoboTwin success comes with a substantial but selective latency advantage. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a success–latency pair rather than ranking only the bold percentages. The first three numerical columns separate easy, hard and overall RoboTwin performance; the last gives milliseconds on A100 under the same pipeline. Section 4.3 states 50 trials for each of 50 tasks and comparable total adaptation budgets. SANTS uses 30,000 backbone updates followed by 3,000 scheduler updates. The continuation on page 7 reports three SANTS seeds, so the overall mean has some seed context. Compare LingBot-VA for the full-denoising reference and the faster external policies for the remaining efficiency gap. e-simulatione-tuninge-schedule
What it supports. SANTS reports 94.4% overall success at 523.7 ms, compared with LingBot-VA’s 92.2% at 2868.4 ms: a 2.2-percentage-point success gain and the reported 81.7% latency reduction. Fast-WAM at 190 ms and pi0.5 at 63 ms are faster, so SANTS does not dominate every latency comparison.
Where the evidence stops. The paper gives SANTS seed results but no matching baseline uncertainty. Comparable adaptation budgets do not equal identical architectures or pretraining. Table 4’s Fixed-full row has different numbers and should not be silently substituted for LingBot-VA.
Table 3. Physical trials support the scheduler’s benefit while exposing task-specific exceptions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each platform block before comparing the mean row. The AgileX tasks cover cloth, bag, sock and charger manipulation; UR10 covers plate transfer, refrigerator placement and fruit sorting. Each method receives 50 independent trials per task, and the mean weights the seven tasks equally. Appendix A specifies shared data, camera inputs, action interfaces, reset distributions and termination rules, with no mid-trial human correction. The latency row averages policy inference across tasks. It is distinct from action execution: on UR10, a query produces 16 steps executed at 10 Hz before the next replan. e-robote-protocole-display-conflicts
What it supports. SANTS reaches 73.1% mean success versus Full WAM’s 65.7%, while latency falls from 2769.3 to 581.3 ms. Charger insertion improves from 38% to 58%. SANTS matches or exceeds Full WAM on every row, but pi0.5 wins fruit sorting at 92% versus 86% and has lower inference latency.
Where the evidence stops. Figure 1 labels the Full WAM mean 68.3%, conflicting with this table’s 65.7% and its task rows. This edition uses Table 3. No real-robot confidence intervals are supplied, so small per-task differences should not be treated as established statistical effects.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0 manipulation 50 tasks, 50 trials per task; comparable total adaptation budgets; A100 latency under the same pipeline. | 94.4%; 523.7 ms. Three SANTS seeds: 94.1%, 94.5%, 94.6%; reported standard deviation 0.3 percentage points. Overall success; end-to-end policy inference latency | LingBot-VA: 92.2%, 2868.4 ms; Fast-WAM: 91.8%, 190 ms; pi0.5: 79.8%, 63 ms. Highest tabled success, with reported 81.7% latency reduction versus LingBot-VA full denoising; external policies remain faster. e-simulatione-tuning |
| Seven real-robot manipulation tasks AgileX and UR10; 50 independent trials per task/method; shared 100 hours of data and adaptation budget; unweighted task mean. | 73.1%; 581.3 ms. Mean success; mean policy inference latency | Full WAM: 65.7%, 2769.3 ms; pi0.5: 54.6%, 63 ms. Charger insertion: SANTS 58% versus Full WAM 38%. Reported gains over Full WAM are 7.4 percentage points and 79.0% less latency. SANTS matches or exceeds Full WAM on every task; pi0.5 wins fruit sorting, 92% versus 86%. Table 3 governs these numbers. e-robote-protocole-display-conflicts |
| Offline denoising-depth diagnostic 500 manually phase-annotated segments; seven depths from 4% to 100%; fixed initial video/action noise. | Full-depth means: coarse 0.8943, fine 0.6571; scanned oracle: 0.8433, 0.6084. Action MSE relative to each segment’s 4%-depth baseline | Full depth is not best for 55.65% of coarse and 41.33% of fine segments. Intermediate states can lower offline error. The hindsight oracle ignores computation and is not SANTS performance. e-diagnostice-depth-table |
| Controlled scheduling ablation RoboTwin 2.0; frozen backbone, scheduler budget and evaluation protocol held fixed. | SANTS: 94.4%, 523.7 ms. Overall success; latency | Fixed-5-step: 87.2%, 553.9 ms; Fixed-full: 89.8%, 2769.3 ms; Jump-only: 91.6%, 1137.7 ms; Stop-only: 92.7%, 1329.3 ms. Both decisions together improve the measured tradeoff, especially against the roughly latency-matched five-step control. e-schedule |
| Path-reward ablation Controlled RoboTwin variants; only the reward definition changes. | Full reward: 94.4%, 523.7 ms. Overall success; latency | Raw MSE: 89.7%, 436.3 ms; no temporal-difference error: 93.6%, 576.4 ms; no difficulty gate: 90.6%, 489.6 ms; no update cost: 94.9%, 2197.8 ms. The computation penalty buys efficiency at a reported 0.5-point success cost versus the no-cost variant; ablation uncertainty is not supplied. e-reward-ablation |
| Real-robot scheduler transfer Seven-task mean; direct transfer versus 3,000 scheduler-only target-domain updates. | Target-tuned: 73.1%. Average success | Direct transfer: 70.6%. The main real-robot result includes target tuning. e-transfer |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Stopping and progression together outperform a fixed schedule with similar latency. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the Stop and Jump columns to identify which decision can respond to the current state. Fixed-5-step fixes both and provides the paper’s approximately latency-matched control. Fixed-full also fixes both but always reaches the fully denoised endpoint. Jump-only adapts progression without adaptive stopping; Stop-only adapts termination without the relative-progression component. The last row combines them. Section 4.4 holds the video–action backbone, scheduler training budget and evaluation protocol fixed. Read success and latency jointly, because enabling one component changes the resulting computation as well as the terminal representation used for actions. e-schedulee-simulation
What it supports. SANTS reaches 94.4% at 523.7 ms versus Fixed-5-step’s 87.2% at 553.9 ms, a 7.2-point gain at slightly lower latency. Both single-component variants improve over fixed schedules but remain slower and less successful than full SANTS. This supports complementary scheduling decisions in this controlled setting.
Where the evidence stops. This is one fixed five-step comparator, not an exhaustive search over fixed nonuniform schedules. Ablation error bars are absent. Fixed-full’s 89.8% and 2769.3 ms differ from Table 2’s full-denoising LingBot-VA values; the paper does not reconcile that difference.
Table S4. The full reward chooses an efficiency tradeoff rather than the largest success number. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the second column to see which reward component changes; the scheduler and frozen backbone otherwise remain the same. Raw action MSE replaces the anchor-normalized gain. Removing temporal-difference error omits adjacent-action trend information. Removing the difficulty gate permits positive gains without the same suppression when the full anchor barely improves on shallow denoising. Removing update cost eliminates the explicit charge for video-state updates. Appendix C defines these terms, and Table S3 lists the fixed reward configuration. Compare both output columns: some removals make inference cheaper by accepting lower action quality, whereas another spends much more computation. e-reward-ablatione-normalizatione-errorse-reward-settingse-progress
What it supports. The no-cost variant reaches 94.9% success but takes 2197.8 ms, versus 94.4% and 523.7 ms with the full reward. Thus the reported efficiency gain is purchased with a 0.5-point success difference. Raw MSE gives 89.7% at 436.3 ms, illustrating that cheaper termination alone is insufficient.
Where the evidence stops. No uncertainty accompanies these variants, so the small success difference cannot establish equivalence or superiority. The reward measures demonstration agreement and counts video-state updates; it is not a direct measurement of robot completion time or every forward pass incurred by scheduling.
7. Analysis & limitations
7.1 What the evidence leaves open
Generality is tested on RoboTwin 2.0 and two robot platforms with one backbone family. Joint backbone–scheduler optimization and adaptive budgets for action denoising, perception and communication remain untested. e-limitations
Figure 2 thresholds h_k, whereas Section 3.2 specifies F_k. Figure 1 gives Full WAM real mean 68.3%, versus Table 3’s 65.7%. Table 4’s Fixed-full also differs from Table 2’s LingBot-VA comparison; the source does not reconcile them, so they remain separate baselines. e-stope-display-conflictse-simulatione-schedule
Three SANTS simulation seeds are reported, but baseline/ablation error bars and real-robot confidence intervals are absent. Selected rollout depths demonstrate variation, not optimality; offline action error does not directly measure physical success. e-simulatione-robote-schedulee-reward-ablatione-rollouts
7.2 Questions for discussion
- Would stronger fixed nonuniform schedules match SANTS at equal forward-pass budgets?
- How much does deterministic stopping depart from the sampled PPO policy?
- Can reward anchors miss useful intermediate states when both endpoints are poor?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include the compatible pretrained policy, intermediate-feature access, nonuniform updates, demonstration actions and Appendix A’s camera/action interfaces. Scheduler training needs two anchor rollouts in addition to each sampled path. e-interfacee-protocole-ppo
Unspecified details include the full data mixture, split cardinalities, exact feature-pooling layer, full-reference update count, training GPU-hours and complete key-frame weighting implementation. Appendix A leaves numerical task timeouts and precise success tolerances unstated. e-tuninge-schedulere-reward-settingse-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the schedule must match the current state
Reader-proposed check; not performed. On held-out RoboTwin episodes, compare SANTS, the fixed five-step control, and replayed SANTS schedules assigned to other episodes independently of their observations. Preserve each replayed schedule’s stopping point and progression ratios, and match the aggregate video-forward-pass budget, including the stopping pass. Keep the frozen policy, initial noise seeds, reset distribution and latency hardware identical. Report paired task success, action error on aligned demonstration segments and measured latency. If shuffled schedules preserve the success advantage at matched computation, state matching is not necessary for that gain; a reliable loss under shuffling would strengthen the adaptive-state explanation. e-interfacee-progresse-diagnostice-schedule
Check 2: Probe intermediate optima hidden by poor reward anchors
Reader-proposed check; not performed. Repeat the seven-depth paired diagnostic on held-out segments and stratify inputs by whether full denoising improves on the shallow anchor. Within the non-improving group, identify segments whose intermediate depth has lower action error than both anchors. Compare the full reward with its no-difficulty-gate variant using identical scheduler initialization, frozen policy, training samples and update budgets. Log gated reward, selected depth, action errors and forward passes, then test matched-computation policies in closed loop. Frequent missed intermediate gains would expose the anchor calibration’s boundary; their absence, or recovery without extra computation, would weaken that concern. Select settings only on validation data. e-diagnostice-normalizatione-ppoe-reward-ablation
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version and Figure 1 on page 1; interface and stopping equations on page 3; architecture, progression and reward on page 4; depth diagnostic on page 5; Tables 1–2, Figure 4 and setup on page 6; Tables 3–4, evaluation and limitations on page 7; robot protocol on page 10; Figure S1 and Tables S1–S2 on page 11; reward/PPO equations and ablation controls on page 12; Tables S3–S4 on page 13. All six final original-PDF crops were individually viewed and retain relevant labels, axes, legends and table rows. Pages 2, 8 and 9 were read as text. Separate supplements and linked resources were not inspected. Source discrepancies in Figures 1–2 and the different full-denoising baseline rows are preserved in the explanations.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 10, 11, 12, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (p. 2)
- 2 Related Work (pp. 2–3)
- 3.1 Frozen Video–Action Policy Interface (p. 3)
- 3.2 Stopping Hazard and Relative Noise Progression (pp. 3–4)
- 3.3 Path-Level Reward and Scheduler Post-Training (pp. 4–5)
- 4.1 Offline Diagnostic and Scheduling Behavior (pp. 5–6)
- 4.2 Experimental Setup (p. 6)
- 4.3 Simulation and Real-Robot Results (pp. 6–7)
- 4.4 Ablation Study (p. 7)
- 5 Limitations and 6 Conclusion (p. 7)
- References (pp. 8–9)
- A Real-Robot Evaluation Protocol (p. 10)
- B Scheduler Implementation and Hyperparameters (pp. 10–11)
- C Reinforcement-Learning Post-Training and Reward Definition (pp. 11–13)
- D Reward-Function Ablations (pp. 12–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation above was addressed by inspecting PDF pages 1, 3–7 and 10–13 and six final crops. Pages 2, 8 and 9 were read as text.
- Identity: the exact observed title and all eight authors match the catalog. The inspected artifact is arXiv:2605.27947v3 [cs.RO], dated 26 August 2026. Earlier revisions and revision history were not supplied; their contents and initial submission date were not independently verified. The catalog date matches the displayed v3 date.
- The title page verifies Fudan University and NVIDIA; the catalog affiliation string is incomplete and includes a project-page fragment.
- No external project page, code, checkpoints, datasets or separate supplements were inspected; no experiments were reproduced. The preliminary report was not used as scientific evidence.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
Exact title: SANTS: A State-Adaptive Scheduler for World Action Models. Authors: Yirui Sun, Guangyu Zhuge, Keliang Liu, Jie Gu, Shiqin Dai, Xinyu Bing, Zhongxue Gan, Chunxu Tian. Shiqin Dai is affiliated with NVIDIA; the other seven authors with Fudan University. The margin states arXiv:2605.27947v3 [cs.RO], 26 Aug 2026.
Go to primary source ↓e-interfacePDF p. 3, Section 3.1; p. 4, Figure 2
A frozen video–action Transformer supplies pooled video-token hidden features and flow predictions. The action branch conditions on the selected terminal video representation. Figure 2 draws observation, language, timestep, video and action tokens with joint self-attention.
Go to primary source ↓e-stopPDF p. 3, Section 3.2, Eqs. (1)–(4); p. 4, Figure 2, Head A
The text defines nonnegative hazard increments, accumulated hazard, cumulative stopping F_k and local stopping h_k. It samples h_k in training and thresholds F_k in deployment; Figure 2 instead labels its threshold h_k >= eta.
Go to primary source ↓e-progressPDF pp. 3–4, Section 3.2, Eqs. (5)–(7) and final paragraph
A Beta-distributed retained-noise ratio advances noise multiplicatively; mode/concentration parameterization guarantees a unimodal Beta. Deployment uses the mode. A first-order flow update advances video state. A final stop consumes a forward pass but no state update.
Go to primary source ↓e-rewardPDF pp. 4–5, Section 3.3, Eq. (8)
The path reward compares the final action chunk with demonstrations, normalizes with shallow/full anchors and penalizes actual video updates; only the scheduler is optimized with PPO.
Go to primary source ↓e-diagnosticPDF p. 5, Section 4.1 and Figure 3
The frozen depth scan follows intermediate-noise backbone tuning, uses 500 manually phase-annotated segments, scans 4/16/32/48/64/80/100 percent depth and fixes both initial noises. Figure 3 plots group means, SEM and individual relative masked-action-MSE curves.
Go to primary source ↓e-depth-tablePDF p. 6, Table 1, all rows and Section 4.1 oracle discussion
Coarse/fine: full depth not best 55.65%/41.33%; adjacent increases 87.90%/77.49%; oracle mean 0.8433/0.6084; fixed-full mean 0.8943/0.6571. The oracle ignores computation.
Go to primary source ↓e-rolloutsPDF p. 6, Figure 4 and adjacent Section 4.1 discussion
Four task rollouts show different selected terminal depths across control chunks. The text explicitly says unselected depths were not evaluated, preventing a claim of per-decision optimality.
Go to primary source ↓e-tuningPDF p. 6, Sections 4.2–4.3; p. 7, Section 4.3 continuation
The backbone is Wan2.2-5B based. Simulated and real multi-source trajectories are described. RoboTwin adaptation uses 30,000 backbone updates then 3,000 scheduler updates; baselines receive the same total adaptation budget.
Go to primary source ↓e-simulationPDF p. 6, Table 2, all rows; p. 7, Section 4.3 simulation paragraph
SANTS easy/hard/overall success is 94.6%/94.2%/94.4%, latency 523.7 ms. LingBot-VA overall 92.2%, 2868.4 ms; Motus 87.8%, 1175 ms; Fast-WAM 91.8%, 190 ms; pi0.5 79.8%, 63 ms. There are 50 trials on each of 50 tasks; three SANTS seeds yield 94.1%, 94.5%, 94.6%, reported std 0.3%.
Go to primary source ↓e-robotPDF p. 7, Table 3, task rows, Mean SR and Latency; Section 4.3
Table 3 gives means 54.6%/65.7%/73.1% and latency 63/2769.3/581.3 ms for pi0.5/Full WAM/SANTS. SANTS task success is 62, 74, 78, 58, 80, 74, 86 percent; Full WAM gives 54, 68, 76, 38, 66, 72, 86 percent. pi0.5 gives 92% on fruit sorting. Each task has 50 trials and the mean is unweighted.
Go to primary source ↓e-display-conflictsPDF p. 1, Figure 1, Real-Robot Results/Mean SR; p. 7, Table 3 and Section 4.3
Figure 1 labels Full WAM mean 68.3%; Table 3 labels it 65.7%, consistent with its task rows. Section 4.3 reports an 18.6-point gain over pi0.5; subtracting rounded table means gives 18.5 points, while averaging the task rows before subtraction yields approximately 18.57 points.
Go to primary source ↓e-schedulePDF p. 7, Section 4.4 and Table 4, all rows
Fixed-5-step 87.2%/553.9 ms; Fixed-full 89.8%/2769.3 ms; Jump-only 91.6%/1137.7 ms; Stop-only 92.7%/1329.3 ms; SANTS 94.4%/523.7 ms. The backbone, scheduler budget and protocol are fixed. No uncertainty is tabled.
Go to primary source ↓e-limitationsPDF p. 7, Section 5
The authors limit evidence to RoboTwin 2.0 and two robot platforms, identify broader backbone/embodiment evaluation and joint optimization as future work, and note dependence on stopping/cost hyperparameters and remaining non-video costs.
Go to primary source ↓e-protocolPDF p. 10, Appendix A, hardware, control and trial protocol
Methods share 100 hours of real data and adaptation budget, cameras, action interface, resets and termination rules. AgileX has two wrist views plus a top-down view; UR10 has wrist and third-person D435i cameras. UR10 data are resampled to 15 Hz, while inference executes 16 actions at 10 Hz then replans. No mid-trial human correction is allowed.
Go to primary source ↓e-transferPDF pp. 10–11, Appendix B and Table S1
Seven-task average success is 70.6% for direct scheduler transfer and 73.1% after 3,000 target-domain scheduler-only updates.
Go to primary source ↓e-schedulerPDF pp. 10–11, Appendix B and Table S2
Scheduler input dimension is D_wam+1; MLP layers 2, hidden width 512, bottleneck 256, heads 2. Start noise 1.00, early-stop noise 0.963, threshold 0.85, forced terminal noise 0.01, full reference 0. Values are selected on held-out validation and fixed across tasks.
Go to primary source ↓e-errorsPDF p. 11, Appendix C, Action-Quality Error and Two-Anchor Quality Normalization
Physical action errors include position displacement, quaternion angular distance and gripper BCE after mapping to [0,1]. Temporal differences measure adjacent action trends; motion/gripper events receive extra weight, with gripper change threshold 0.25. Anchors use terminal noise approximately 0.963 and 0.
Go to primary source ↓e-normalizationPDF p. 12, Appendix C, Eqs. (S1)–(S5)
The error-gap floor stabilizes clipped normalized gains; a difficulty gate suppresses positive gains when full denoising barely helps. Negative gains are penalized. The return subtracts normalized cumulative video-update cost from weighted quality gain.
Go to primary source ↓e-ppoPDF p. 12, Appendix C, Path-Level PPO Post-Training and Training and Deployment Summary
One sampled path plus two no-gradient anchors supports path-level PPO with EMA decay 0.95. AdamW uses learning rate 10^{-4}, weight decay 0.01, four epochs, clip 0.2 and gradient norm 1.0. KL weight is 0.02, jump log-probability scale 0.5, with no entropy bonus. Validation selects checkpoints using success, return and update count.
Go to primary source ↓e-reward-settingsPDF p. 13, Table S3, quality, cost, normalization and PPO rows; p. 12, sensitivity paragraph
Sequence/difference quality weights 0.6/0.4; cost weight 0.25; cost schedule c0=0.2,c1=0,gamma=2; gap floor 0.001, difficulty kappa=0.02. Table S3 supplies key-frame constants but no complete weighting formula. Stopping and cost sensitivity is discussed qualitatively.
Go to primary source ↓e-reward-ablationPDF pp. 12–13, Appendix D and Table S4, all rows
Full reward 94.4%/523.7 ms; raw MSE 89.7%/436.3 ms; no temporal difference 93.6%/576.4 ms; no difficulty gate 90.6%/489.6 ms; no update cost 94.9%/2197.8 ms. Variants hold backbone, architecture, split, budget and protocol fixed.
Go to primary source ↓8.5 Primary sources
SANTS: A State-Adaptive Scheduler for World Action Models ↗
PDF · 7,328 extracted words
Source fingerprint
24b3db5cb9e638197bb8b199cebbf15fbafbca97f5b6269fe8967e2e01acc7cd