dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models
1. Paper overview
In one sentence: Scoring newly unmasked action tokens across denoising steps enables PPO refinement of a discrete VLA, while task-specific horizons trade computation against action quality. e-probleme-pathe-surrogatee-hybride-evaluatione-liberoe-robotwin
| At a glance | What to know |
|---|---|
| Research problem | Source description A final action chunk can arise through many intermediate masked sequences. Summing their probabilities is combinatorial, while using only the final denoising pass ignores how the action was generated. The paper seeks a usable PPO objective for this short internal generation chain, whose intermediate states receive no direct environmental reward. e-problem |
| Core mechanism | Source description Factorize the sampled denoising path, then optimize a token-only surrogate that excludes already decoded positions and explicit scheduler gradients. e-pathe-surrogate |
| A key reported result | RoboTwin 2.0 eight-task bimanual manipulation: 92.0; Handover Mic 96.9. Average success rate (%). Hybrid RL: highest online mean across eight tasks, 64 rollouts per task sampled from 1,000 training scenarios per task. SFT: four-step decoding, 100 trials per task. MM-ACT* average 61.4 and Handover Mic 49.0; absolute gains 30.6 and 47.9 percentage points. Improvement appears across all eight tasks, but remains training-distribution rollout evidence. WAM references use different training settings; the result does not establish superiority to them. e-robotwine-evaluatione-robotwin-confige-task-schedulee-comparability |
| Reading caution | Reader analysis Experiments establish simulated task completion, without physical-robot deployment or an RL held-out generalization result. Main tables lack uncertainty intervals; curve bands are labeled standard deviations without an identified independent-run basis. e-evaluatione-liberoe-robotwine-ablation |
Core contributions
- Source description
Factorize the sampled denoising path, then optimize a token-only surrogate that excludes already decoded positions and explicit scheduler gradients. e-pathe-surrogate
- Source description
Use the same objective for one-, two- and four-step decoding; select task-specific horizons from SFT performance to trade refinement against computation. e-hybride-task-schedule
Figure 2. Internal denoising supplies the policy score; completed actions supply environmental feedback. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the observation box: images, the instruction and robot state condition action generation. Follow the action rows from the fully striped x_K to completed x_0. The right-hand addition nodes collect transition log scores; the bottom approximation keeps only the newly unmasked positions defined by Equation (6). Follow the separate feedback route from outcome rewards and the value head into advantage A_t, then into PPO. The depicted robot poses belong to action generation, not separately rewarded environment transitions. Cross-check the gray tiles against the equations: the first tokens appearing after the all-mask row are newly decoded, despite the conflicting gray-color legend. e-pipelinee-probleme-pathe-surrogatee-algorithm
What it supports. The method makes internal generation history available to policy optimization while keeping environmental credit at the action-chunk level. The approximate sign matters: Equation (5) includes confidence-dependent scheduler probabilities, whereas the implemented score discards them and updates newly unmasked tokens. Algorithm 1 supplies the clipped PPO update and value fitting.
Where the evidence stops. The legend labels dark gray as already denoised, yet dark tiles first emerge from masked positions. Their M labels also follow displayed output rows. Use Equation (6) and Algorithm 1 to identify newly unmasked tokens, rather than inferring the loss mask from color alone.
2. Motivation
2.1 The problem and the proposed response
A final action chunk can arise through many intermediate masked sequences. Summing their probabilities is combinatorial, while using only the final denoising pass ignores how the action was generated. The paper seeks a usable PPO objective for this short internal generation chain, whose intermediate states receive no direct environmental reward. e-problem
2.2 What this reading follows
A discrete diffusion policy builds an action chunk through several partially masked sequences. The final robot action is observable, but its probability requires summing over many possible generation paths. dVLA-RL instead retains the path that actually occurred and constructs a PPO score from its newly generated tokens. This edition follows that information flow, the objective ablation, and the Hybrid strategy for allocating denoising steps across tasks. The reported simulation gains are substantial, but their interpretation depends on two details: the practical score omits scheduler probabilities, and the headline success rates are peaks from online training rollouts rather than a common held-out evaluation. e-probleme-pathe-surrogatee-hybride-evaluatione-liberoe-robotwin
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded VLA / One Model classification is supported by the unified MM-ACT action-token generator and its shared denoising network, not merely by joint training. A training value head is auxiliary; the reported control path does not couple a future-world predictor to a separate action model. Action-only SFT and direct action generation support predictionParadigm and quadrant remaining Not applicable. e-pipelinee-pathe-sfte-algorithm
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the robot decision from the denoising decisions
Consider one instruction-conditioned action chunk. The policy first receives visual observations and proprioception, then starts with every action token masked. Each forward pass sees a partially completed sequence and predicts token distributions; a confidence-based scheduler decides which positions to update. These internal decisions change what later passes condition on. However, the environment sees only the completed action chunk, so there is no separate reward for an intermediate masked sequence. dVLA-RL retains the internal path to construct its policy score but estimates credit from the environmental trajectory. The value head uses first-step hidden states, GAE supplies a chunk advantage, and that same advantage weights the chunk's denoising contributions. This distinction is why the method can use generation history without introducing a separate environment-level critic for every denoising step. e-probleme-pathe-algorithme-surrogate
5.2 Distinguish the exact factorization from the optimized score
Equation (4) factors a transition into two choices: selecting positions and generating their tokens. Equation (5) therefore includes both kinds of log-probability over the full path. The implemented update makes a further choice: it discards scheduler terms and scores only positions that change from a mask to a token. Algorithm 1 exponentiates the current-versus-old difference of this shielded score and applies PPO clipping. Reader interpretation: this is a practical trajectory-informed surrogate, not an exact likelihood ratio for every confidence-dependent choice in the generator. Figure 2's approximation sign is consistent with that reading. Figure 3 tests the value of retaining intermediate token scores, but it does not independently validate dropping scheduler probabilities or demonstrate the scheduler-calibration improvement claimed in Section 4.2. e-pathe-surrogatee-algorithme-pipelinee-ablation
5.3 Allocate refinement, then ask which budget is being saved
Hybrid begins with a task-level SFT comparison: choose the smallest denoising horizon that performs comparably to longer decoding. The resulting lookup spends one pass on three RoboTwin tasks, two on three tasks, and four on two tasks. During RL, all variants use the same scenario-pool protocol and number of rollouts per iteration. Figure 4 therefore supports improved success per training iteration, while Table 3 shows a separate computational tradeoff. Reader interpretation: a higher learning curve does not alone establish faster time to a target success rate. Hybrid costs less than always using four passes, yet more than always using two. The appendix curves also suggest that an average hides different task responses, so reproducing the lookup should retain both per-task success and measured elapsed computation. e-hybride-task-schedulee-robotwin-confige-efficiencye-task-curves
5.4 Training and inference
During training
SFT uses 500 demonstrations per LIBERO suite and 1,000 per RoboTwin task, with batches 128 and 96. Appendix B selects epoch 1 for LIBERO-Object and epoch 4 for other suites and RoboTwin; Appendix C's generic same-checkpoint wording does not erase this distinction. e-sfte-libero-config
RLinf supplies online PPO. Both benchmark configurations use bf16, global/micro batches 512/8, actor/value learning rates 3×10⁻⁵/2×10⁻³, PPO clip 0.2, discount 0.99 and GAE parameter 0.95. The source does not specify which backbone components, if any, remain frozen. e-evaluatione-libero-confige-robotwin-config
During inference
Run the assigned denoising chain, execute the completed action chunk, then condition subsequent control on environmental feedback. LIBERO uses two denoising steps; RoboTwin Hybrid uses the task lookup. These are action predictions, without a reported future-observation imagination or inverse-dynamics stage. e-probleme-algorithme-libero-confige-task-schedule
Decoding details remain ambiguous: Section 5.3 calls the path greedy, whereas Algorithm 1 samples tokens and scheduler choices. Do not infer a fully specified stochastic-versus-greedy deployment protocol. e-decoding-ambiguity
5.5 Implementation flow
- Initialize the action policy
MM-ACT* jointly learns tasks with action-only SFT. It changes the original MM-ACT recipe to chunk size 16 and omits its additional multimodal generation objectives. This initializer must be distinguished from the original MM-ACT baseline. e-sft
- Generate and retain an internal path
Condition on state s_t; begin with fully masked x_K. At each descending denoising step k, predict categorical token distributions and select update indices U_k through confidence-dependent Gumbel-TopK. Retain the path and newly unmasked subset M_k. Only the completed x_0 is executed. e-probleme-pathe-algorithm
- Assign environmental credit
Collect binary task-outcome rewards and estimate values from first-step hidden states. GAE supplies an advantage shared across the chunk's denoising transitions; there is no separately rewarded physical decision at each internal denoising step. e-evaluatione-algorithme-problem
- Update the surrogate policy
Sum log-probabilities over newly unmasked tokens across the chain. Algorithm 1 compares current and old sums in a clipped PPO ratio and fits value targets. Because scheduler probabilities depend on network confidence yet are omitted, the implemented score is a surrogate for the complete path probability. e-pathe-surrogatee-algorithm
- Assign refinement budgets
Hybrid selects the smallest horizon whose SFT success is comparable to longer decoding. It assigns one step to three tasks, two to three tasks and four to two tasks; the paper gives a fixed task lookup, not a learned per-observation stopping policy. e-hybride-task-schedule
6. Experiments & results
dVLA-RL applies online PPO to MM-ACT's discrete action denoising, scoring newly unmasked tokens across the sampled path. Its Hybrid variant assigns denoising budgets by task. Large reported simulation gains are peak online training-rollout results, and the practical likelihood drops scheduler probabilities (e-surrogate, e-hybrid, e-evaluation).
6.1 Read the original evidence
Table 1. Read the initializer row and the evaluation protocol before interpreting the LIBERO average. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across the bold dVLA-RL row to see success on the four suites, then use the rightmost column for the average. The closest architectural control is MM-ACT*, immediately above it. Original MM-ACT is a different row: Appendix B changes the action objective, chunk length and training configuration for the RL initializer. Figure 1's overview uses the original MM-ACT value, so its SFT bar must not replace the starred initializer in this comparison. The VLA and WAM group labels organize reference methods; they do not establish shared training data, checkpoint selection or evaluation conditions across every row. e-liberoe-evaluatione-sfte-libero-confige-overview-baseline
What it supports. dVLA-RL reports 99.8%, 100.0%, 99.6% and 99.2%, averaging 99.7%. MM-ACT* averages 88.1%, giving an 11.6-percentage-point difference between the printed averages. The original MM-ACT row averages 96.3%, and SimpleVLA-RL reports 99.1%; these are distinct reference comparisons, not interchangeable baselines.
Where the evidence stops. Section 5.1 selects best task-wise online training-rollout success for RL, with 512 episodes per task, while SFT uses separate trials. The table supplies no uncertainty intervals. Treat the ranking as reported performance under these protocols, not a controlled held-out leaderboard.
Table 2. Eight task columns reveal broad gains, alongside a substantial protocol boundary. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the average at the right, then compare MM-ACT* and dVLA-RL within each task column. This separates a general improvement from a result dominated by one task. Handover Mic and Lift Pot are useful examples of large gains, while Place A2B Left remains the lowest reported dVLA-RL task score. Read the gray WAM block as reference context: the paragraph below the original table explicitly says those methods use different RoboTwin training settings. Section 5.1 further specifies that the dVLA-RL row comes from the highest eight-task online mean, with the per-task entries recorded at that selected point. e-robotwine-evaluatione-comparabilitye-robotwin-config
What it supports. The reported mean rises from 61.4% to 92.0%, an absolute gain of 30.6 percentage points. Handover Mic rises from 49.0% to 96.9%. All eight task columns improve over MM-ACT*, supporting broad training-distribution improvement across the selected bimanual tasks rather than success on only one behavior.
Where the evidence stops. RL uses 64 rollouts per task from training scenario pools; SFT uses 100 trials per task at four denoising steps. These differ in evaluation and selection. The gray references are not controlled comparisons, and their higher averages preclude claiming universal WAM superiority.
Table 3. Hybrid saves computation relative to four-step decoding, while costing more than two-step decoding. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read vertically within a column before comparing rows across columns, because seconds, milliseconds and GFLOPs describe different quantities. The source caption defines inference costs for generating one action chunk. It calls the first column RL training wall-clock cost but does not specify the measured number of iterations or complete workload. The fixed-horizon rows show nearly proportional inference cost as denoising passes increase. Hybrid falls between fixed two-step and four-step inference, consistent with its task-specific mixture. Pair this table with Figure 4: that figure compares success by training iteration, while this table supplies a separate view of reported computation. e-efficiencye-hybride-task-schedule
What it supports. Hybrid reports 387.24 ms and 37289.12 GFLOPs per action chunk, below four-step's 728.92 ms and 70191.28 GFLOPs. Its training-time entry is 607.89 s versus 845.95 s. Fixed two-step is cheaper in all three columns, so Hybrid's efficiency benefit depends on the chosen comparator and desired success.
Where the evidence stops. The PDF does not identify accelerator models/counts, timing batch conditions or the training workload behind these measurements. Do not infer total training time, physical control frequency or time to a specified success threshold from this table.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO four-suite manipulation Two-step RL; best task-wise online training-rollout SR, with 512 episodes per task as stated in Section 5.1. SFT uses 50 trials per subtask, 2,000 total. | Spatial 99.8; Object 100.0; Goal 99.6; Long 99.2; average 99.7. Success rate (%) | MM-ACT* average 88.1; original MM-ACT 96.3; SimpleVLA-RL 99.1. The printed-average gain over the initializer is 11.6 percentage points. Peak rollout selection differs from fixed-checkpoint SFT evaluation; this is not a common held-out protocol. e-liberoe-evaluatione-libero-config |
| RoboTwin 2.0 eight-task bimanual manipulation Hybrid RL: highest online mean across eight tasks, 64 rollouts per task sampled from 1,000 training scenarios per task. SFT: four-step decoding, 100 trials per task. | 92.0; Handover Mic 96.9. Average success rate (%) | MM-ACT* average 61.4 and Handover Mic 49.0; absolute gains 30.6 and 47.9 percentage points. Improvement appears across all eight tasks, but remains training-distribution rollout evidence. WAM references use different training settings; the result does not establish superiority to them. e-robotwine-evaluatione-robotwin-confige-task-schedulee-comparability |
| Trajectory-level versus last-step objective LIBERO Spatial/Object, two/four denoising steps; shared backbone, rollout and reward. | Trajectory-level optimization retains higher late-training success in all four panels. EMA success-rate curves | Last-step-only curves plateau lower or degrade after initial improvement. A direct mechanism ablation supports retaining intermediate transitions. No exact endpoint values or independent-seed significance are inferred from the curves. e-ablation |
| RoboTwin denoising-horizon comparison Shared training protocol; 512 rollouts per optimization step. | Hybrid 0.920. Peak online average success rate | Fixed two-step 0.908; fixed one-step 0.885. Hybrid is ahead through much of training; task-level curves show uneven benefits. Figure 4 does not establish a comparison against a separate fixed four-step success curve. e-hybride-robotwin-confige-task-curves |
| Action-chunk generation efficiency Table 3's reported timing/FLOP configuration; hardware and training workload duration are unspecified. | Hybrid: 607.89 s; 387.24 ms; 37289.12 GFLOPs. Training seconds; inference milliseconds; inference GFLOPs | Four-step: 845.95 s; 728.92 ms; 70191.28 GFLOPs. Two-step: 452.06 s; 364.21 ms; 35095.64 GFLOPs. Hybrid costs less than fixed four-step but more than fixed two-step. These timings cannot establish total time to a target success rate. e-efficiency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Removing intermediate-transition scores degrades late-training behavior in the reported controlled ablation. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read rows as task suites and columns as denoising horizons. Pink retains transition scores across the path; blue uses only the last denoising pass. Compare the late-training portions within each panel, since the axes and training durations differ between panels. The caption identifies solid curves as exponentially smoothed success and shaded regions as plus or minus one standard deviation. The source says both variants share backbone, rollout setting and sparse reward, making this a closer test of the proposed objective than the cross-model tables. It tests whether intermediate transitions help; it does not compare alternative estimators for scheduler gradients. e-ablatione-pathe-surrogate
What it supports. The trajectory-level curves remain above the last-step proxy late in all four panels. Object is particularly revealing: both objectives improve initially, then the blue curves lose performance while the pink curves remain high. This supports retaining the generation history when optimizing these multi-step policies, within the tested LIBERO settings.
Where the evidence stops. The source does not identify the independent-run count or the aggregation basis behind the standard-deviation bands. No exact endpoint percentage or significance claim is inferred here. The result also does not prove that the token-only score equals the full path likelihood.
Figure 4. Hybrid improves the reported training curve by allocating refinement steps across tasks. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to identify blue one-step, purple two-step and green Hybrid curves. The stars denote reported peaks, whereas the lines denote EMA success; a peak need not sit on the smoothed line. The horizontal axis measures training steps, not wall-clock seconds. Appendix D holds rollout budgets constant across variants, so this plot is evidence about performance per training iteration. Table 7 supplies the Hybrid lookup: one step for Beat Block Hammer, Place Phone Stand and Place Empty Cup; two for Lift Pot, Move Can Pot and Place A2B Left; four for Handover Mic and Pick Dual Bottles. e-hybride-task-schedulee-robotwin-confige-task-curvese-efficiency
What it supports. Hybrid has the highest labeled peak, 0.920, compared with 0.908 for fixed two-step and 0.885 for fixed one-step, and stays ahead through much of training. Appendix task curves show that benefits are uneven, with conspicuous separation on Handover Mic and Pick Dual Bottles and closer curves on several easier tasks.
Where the evidence stops. The caption discusses four-step refinement, but the plotted fixed multi-step comparator is two-step; no separate fixed four-step curve is shown. Equal iteration budgets do not imply equal computation. The lookup is chosen from SFT performance, with no numerical definition of comparable success.
7. Analysis & limitations
7.1 What the evidence leaves open
Experiments establish simulated task completion, without physical-robot deployment or an RL held-out generalization result. Main tables lack uncertainty intervals; curve bands are labeled standard deviations without an identified independent-run basis. e-evaluatione-liberoe-robotwine-ablation
The final-step ablation does not isolate the benefit of dropping scheduler gradients or verify the claimed improvement in scheduler calibration. Figure 2 also has a gray-token legend/transition mismatch; use Eq. (6)'s explicit mask-to-token definition. e-surrogatee-ablatione-pipeline
7.2 Questions for discussion
- Would the trajectory-level advantage persist under equal-budget held-out evaluation rather than peak training rollouts?
- Does Hybrid's task lookup outperform other assignments with the same aggregate denoising budget?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs MM-ACT initialization and its inherited SFT recipe, RLinf, both simulators, demonstrations and scenario seeds. LIBERO uses seven-dimensional ee actions with horizons 256/320/320/512; RoboTwin uses 14-dimensional qpos actions and a 256-step limit. Both use chunk size 16. e-sfte-libero-confige-robotwin-config
Unresolved requirements include software revisions, accelerator models/counts, scheduler temperature and exact remasking implementation, the tolerance defining comparable SFT success, and Table 3's timing workload. Algorithm 1's reconstruction of x_k from x_0/U_k needs checking against its stored sampled paths. e-algorithme-decoding-ambiguitye-hybride-efficiencye-sfte-libero-confige-robotwin-config
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test the path objective with fixed evaluation and faithful replay
Reader-proposed check, not an executed experiment: initialize both objectives from the same MM-ACT* checkpoint on LIBERO-Object, using two- and four-step decoding as separate comparisons. Hold rollout seeds, PPO settings and total environment interactions fixed; vary only full-path versus last-step token scoring. Before updating, replay stored denoising states and masks and verify that their log scores equal the rollout scores, explicitly resolving Algorithm 1's reconstruction step and stochastic-versus-greedy wording. Use multiple training seeds and a fixed held-out evaluation set at predeclared checkpoints. Compare late-training success and variability, without choosing each run's best rollout. If the trajectory advantage disappears under this protocol, the published stability conclusion would not transfer to that more controlled evaluation. e-ablatione-algorithme-decoding-ambiguitye-sfte-libero-confige-evaluation
Check 2: Separate task-aware scheduling from simply spending more passes
Reader-proposed check, not an executed experiment: on the same eight RoboTwin tasks, compare Table 7's Hybrid lookup with fixed one-, two- and four-step policies and with shuffled task assignments preserving three one-step, three two-step and two four-step tasks. Use identical SFT weights, training seed pools and rollout budgets. Predeclare a numerical tolerance for selecting comparable SFT performance using validation seeds; reserve independent seeds for final evaluation. Measure per-task success, success versus interactions, elapsed training time and action-chunk latency on a documented machine. If matched-budget shuffled assignments equal Hybrid, task-aware allocation is not established; if Hybrid saves time only relative to fixed four-step, its efficiency claim must remain comparator-specific. e-hybride-task-schedulee-robotwin-confige-task-curvese-efficiencye-evaluation
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all method and evaluation pages, Figures 1–7, Tables 1–7 and Algorithm 1, including appendix configuration and task-schedule pages. All six final original crops were individually inspected. Figure 2's gray legend conflicts with its first unmasking transition; the explanation follows Eq. (6) and Algorithm 1. Figure 4 plots a fixed two-step comparator, not a separate fixed four-step curve. Figure 1's original MM-ACT baseline is distinguished from MM-ACT* in Table 1. Acknowledgment/reference pages 13–16 were read as text; they contain no retained method or numerical evidence requiring this visual pass. Separate supplements and external implementation details remain outside the supplied reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Section 1: Introduction
- Sections 2.1–2.3: Related work
- Sections 3.1–3.2: Discrete diffusion and PPO preliminaries
- Sections 4.1–4.2: Path probabilities and implicit unmasking optimization
- Sections 5.1–5.5: Setup, main results, objective ablation, denoising horizons and efficiency
- Section 6: Conclusion; acknowledgments; references
- Appendix A: Algorithm 1
- Appendix B.1–B.4: SFT implementation, checkpoint selection and evaluation
- Appendix C: LIBERO RL configuration and curves
- Appendix D and D.1: RoboTwin configuration, task schedules and curves
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Version scope: the inspected title page identifies arXiv:2606.23623v1, 22 June 2026. Title and all sixteen author identities agree with the supplied catalog; no different revision or edition was supplied or compared.
- All seven supplied text chunks were read individually, covering all 22 PDF pages. The extraction does not reconstruct figure images; this gap was addressed by inspecting the retained PDF's Figures 1–7 and Tables 1–7.
- Separate supplemental material availability has not been fully verified.
- Code, external references and implementation dependencies were not inspected. No experiments were reproduced. Reference-list and acknowledgment pages 13–16 were read as text, without a separate visual pass.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author/affiliation block and arXiv margin
The title matches the supplied observed title. The margin identifies arXiv:2606.23623v1 [cs.RO], 22 Jun 2026. Sixteen authors and five institutions are printed.
Go to primary source ↓e-problemPDF pp. 4–5, Sections 2.2 and 3.1–3.2, Eq. (3)
Only the fully decoded action interacts with the environment. Final-action likelihood sums over intermediate denoising paths; individual categorical predictions do not evaluate this marginal.
Go to primary source ↓e-pathPDF p. 6, Section 4.1, Eqs. (4)–(5)
Each transition includes a confidence-dependent Gumbel-TopK unmasking selection probability and token probabilities over selected indices U_k. The full path log-probability includes both terms.
Go to primary source ↓e-pipelinePDF p. 6, Figure 2 and caption; PDF p. 7, Section 4.2, Eq. (6)
The diagram connects observations, action denoising, sparse rewards, a value head and PPO. Its approximate likelihood sums newly generated tokens. Dark tokens first appearing after the fully masked row conflict with the legend's 'Already Denoised' label; Eq. (6) explicitly defines the mask-to-token subset M_k.
Go to primary source ↓e-surrogatePDF p. 7, Section 4.2, Eq. (6) and following paragraph
The practical objective drops the scheduler term and restricts gradients to M_k, the newly unmasked subset of U_k, weighted by a shared chunk-level advantage. Improved scheduler confidence/calibration is the authors' proposed indirect effect.
Go to primary source ↓e-algorithmPDF p. 17, Appendix A, Algorithm 1, lines 8–38
Rollout samples selected tokens, executes x_0, uses first-step hidden states for the value estimate and computes GAE. Updates accumulate shielded log-probabilities, apply PPO clipping and fit value targets. Line 27 reconstructs x_k from x_0 and U_k although stored x_k appears in line 18.
Go to primary source ↓e-sftPDF p. 8, Section 5.1, baseline paragraph; PDF p. 18, Appendix B.1–B.4
MM-ACT* uses action-only multi-task SFT, chunk size 16, 500 demonstrations per LIBERO suite and 1,000 per RoboTwin task. SFT batches are 128/96. Object uses epoch 1; the other LIBERO suites and RoboTwin use epoch 4. Remaining SFT hyperparameters are delegated to MM-ACT.
Go to primary source ↓e-evaluationPDF pp. 7–8, Section 5.1, benchmarks and online RL/evaluation protocol
Experiments are simulated single-arm LIBERO and bimanual RoboTwin 2.0. RLinf PPO uses binary outcome rewards. SFT uses 50 trials per LIBERO subtask, 2,000 total, and 100 trials per RoboTwin task at four denoising steps. Main RL results are peak online rollout success: LIBERO best task-wise values with 512 episodes; RoboTwin highest eight-task mean with 64 episodes per task.
Go to primary source ↓e-liberoPDF p. 9, Table 1, MM-ACT, MM-ACT*, dVLA-RL and SimpleVLA-RL rows
dVLA-RL reports Spatial/Object/Goal/Long success 99.8/100.0/99.6/99.2%, average 99.7%. MM-ACT* averages 88.1%, original MM-ACT 96.3%, and SimpleVLA-RL 99.1%.
Go to primary source ↓e-robotwinPDF pp. 8–9, Section 5.2 and Table 2, MM-ACT* and dVLA-RL rows
Across eight selected tasks the reported average rises from 61.4% to 92.0%, an absolute 30.6-point gain. dVLA-RL per-task values are 95.3, 96.9, 95.3, 89.1, 87.5, 79.7, 95.3 and 96.9% in table order. Handover Mic rises from 49.0% to 96.9%.
Go to primary source ↓e-comparabilityPDF p. 9, Table 2 WAM rows and paragraph immediately below
RoboTwin WAM references use different training settings, explicitly precluding strictly controlled comparisons. LingBot-VA and Fast-WAM average 97.4% and 96.6%, respectively.
Go to primary source ↓e-ablationPDF p. 10, Figure 3, caption and Section 5.3
Trajectory-level and last-step-only objectives share backbone, rollout setting and sparse reward on LIBERO Spatial/Object with two/four denoising steps. Trajectory-level curves retain higher late-training success. Curves are EMA and shaded bands are labeled ±1 standard deviation; independent-run count and aggregation basis are not identified.
Go to primary source ↓e-hybridPDF p. 11, Sections 5.4–5.5; PDF p. 12, Figure 4 and legend
Hybrid chooses the smallest task-specific denoising horizon with SFT success comparable to longer decoding. Figure 4 labels fixed 2-step, fixed 1-step and Hybrid peaks as 0.908, 0.885 and 0.920. It does not plot a separate fixed 4-step curve.
Go to primary source ↓e-efficiencyPDF p. 12, Table 3, caption and Section 5.5 continuation
Rows 1-step/2-step/4-step/Hybrid report training seconds 259.33/452.06/845.95/607.89; action-chunk inference milliseconds 182.47/364.21/728.92/387.24; GFLOPs 17547.82/35095.64/70191.28/37289.12. The duration unit for the training workload and hardware configuration are not specified here.
Go to primary source ↓e-libero-configPDF pp. 18–19, Appendix C and Tables 4–5
LIBERO uses two denoising steps, seven-dimensional ee actions, chunk size 16, 256×256 images, bf16, AdamW, global/micro batches 512/8, actor/value rates 3e-5/2e-3, gamma 0.99, GAE lambda 0.95 and PPO clip 0.2. Rollout horizons are 256/320/320/512 for Spatial/Object/Goal/Long. Appendix C's generic same-checkpoint statement needs to be read alongside B.3's epoch-specific selection.
Go to primary source ↓e-robotwin-configPDF pp. 20–21, Appendix D and Table 6
Each RoboTwin task has 1,000 training initialization seeds; 64 sampled seeds per task give 512 rollouts per optimization step across eight tasks. Variants share this protocol. Table 6 specifies 14-dimensional qpos actions, chunk 16, bf16, global/micro batches 512/8, actor/value rates 3e-5/2e-3, PPO clip 0.2, gamma 0.99, GAE lambda 0.95 and a 256-step episode limit.
Go to primary source ↓e-task-schedulePDF p. 21, Table 7
Hybrid assigns one step to Beat Block Hammer, Place Phone Stand and Place Empty Cup; two to Lift Pot, Move Can Pot and Place A2B Left; four to Handover Mic and Pick Dual Bottles.
Go to primary source ↓e-task-curvesPDF p. 20, Figure 5; PDF p. 22, Figures 6–7 and captions
Appendix curves cover all four LIBERO suites and eight RoboTwin tasks. In RoboTwin, Hybrid gains vary by task; Handover Mic and Pick Dual Bottles separate visibly, whereas several one-step-assigned tasks have similar curves.
Go to primary source ↓e-decoding-ambiguityPDF p. 10, Section 5.3 first paragraph; PDF p. 17, Algorithm 1 lines 12–14 and 27–30
Section 5.3 calls the path greedy, whereas Algorithm 1 samples scheduler choices and tokens. The relation between reconstructed update inputs and the stored sampled states is not fully resolved in the pseudocode.
Go to primary source ↓e-overview-baselinePDF p. 2, Figure 1(b); PDF p. 9, Table 1; PDF p. 18, Appendix B.2
The overview LIBERO bar labels SFT-only MM-ACT at 96.3%. Table 1 distinguishes this original MM-ACT from the modified MM-ACT* RL initializer at 88.1%; the two baselines must not be interchanged.
Go to primary source ↓8.5 Primary sources
dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models ↗
PDF · 10,107 extracted words
Source fingerprint
6adbb32de48e8da17c5a5c50fad57a47c5a2ba55c6abaae80fcb8e42f69830b8