RL Token: Bootstrapping Online RL with Vision-Language-Action Models
1. Paper overview
In one sentence: RLT turns a frozen VLA’s features and action proposals into a lightweight online refinement interface, improving precision-task execution while retaining human supervision and a strong behavioral anchor. e-probleme-tokene-policye-evaluatione-human
| At a glance | What to know |
|---|---|
| Research problem | Source description Precision manipulation can fail during final alignment or contact even when a VLA handles the preceding task. The paper seeks online improvement within limited robot interaction budgets, combining pretrained perception and behavior with the sample efficiency of small off-policy learners. e-problem |
| Core mechanism | Source description A learned RL token compresses VLA embeddings through autoregressive reconstruction; online RL freezes this interface and the VLA. e-token |
| A key reported result | Ethernet insertion — critical-phase episode length: 66 timesteps Median episode length, control timesteps; lower is faster. Figure 9 compares the final policy with the base VLA and expert teleoperated demonstrations on the insertion segment. Base VLA: 228; teleoperation: 146. The directly labeled medians support faster insertion than either comparator. They are not full-task timings or the mean-completion statistic discussed elsewhere. e-speede-evaluation |
| Reading caution | Reader analysis Training still needs human reward labels, corrections, and handoff decisions. Controlled trials start near the bottleneck; full-task testing covers only screw and zip tie. These results do not establish unattended improvement or broad transfer to unseen tasks. e-humane-evaluation |
Core contributions
- Source description
A learned RL token compresses VLA embeddings through autoregressive reconstruction; online RL freezes this interface and the VLA. e-token
- Source description
Reference-conditioned action chunks, a quadratic policy anchor, and reference dropout support lightweight refinement. Experiments cover screw installation, zip tie fastening, Ethernet insertion, and charger insertion. e-policye-implementatione-tasks
Figure 2. The RL interface is learned by reconstructing VLA embeddings through a compact readout. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the camera images and prompts at the bottom. The VLA has an action-expert branch on the right, producing reference actions, and an embedding branch rising through the middle. Follow those embeddings into the added encoder: the highlighted readout becomes the single RL token. The decoder receives this token and preceding target embeddings during reconstruction. Equations (1)–(2) specify the token position and autoregressive training more precisely than the schematic. The original diagram labels a general prompt interface, but footnote 1 says the experiments omit language embeddings from compression because each task has a fixed instruction. e-tokene-policye-implementation
What it supports. RLT creates a small state interface without asking online reinforcement learning to update the entire VLA. Representation learning happens before the online phase; the resulting frozen token supports small actor and critic networks. The separate action branch remains useful because the actor also receives a sampled VLA proposal.
Where the evidence stops. This is reconstruction of current VLA features, not prediction of future observations. Equation (2) stops gradients into VLA embeddings and supplies preceding targets to the decoder; the picture alone neither proves lossless compression nor specifies all gradient paths.
2. Motivation
2.1 The problem and the proposed response
Precision manipulation can fail during final alignment or contact even when a VLA handles the preceding task. The paper seeks online improvement within limited robot interaction budgets, combining pretrained perception and behavior with the sample efficiency of small off-policy learners. e-problem
2.2 What this reading follows
A robot can already know how to approach a socket yet waste time probing at the final contact. RLT concentrates learning on that bottleneck. It first adds a compact readout to a task-adapted VLA, then trains a small actor and critic using real interactions while the large model stays frozen. The actor sees both the readout and a proposed action chunk, making improvement a guided search around competent behavior. The figures explain that interface, separate reliability from speed, and reveal what the ablations actually isolate. The strongest timing evidence concerns critical insertion segments; broader full-task and autonomous-learning claims require additional evidence. e-probleme-tokene-policye-evaluatione-human
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Dual-system |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The VLA and separately trained online actor–critic support the recorded Dual-system VLA post-training classification. Reconstructing current internal embeddings is auxiliary training. The method predicts actions and return, without a future-observation world model or inverse-dynamics action extraction; the world/action prediction quadrant is therefore not applicable. e-tokene-policye-loop
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a state interface before asking reward to improve behavior
The starting VLA already contains visual and behavioral knowledge, but its many internal embeddings are an awkward input for rapid online actor–critic learning. RLT adds an encoder with a learned special token and trains a decoder to reconstruct the VLA embeddings. Equation (1) defines the special-position encoder output; Equation (2) makes it useful through an autoregressive reconstruction loss. The reconstruction targets and preceding decoder inputs are stopped-gradient VLA embeddings. Optional supervised VLA fine-tuning is a separate objective, and the experiments do task adaptation together with readout training. Once online RL begins, both the VLA and readout parameters are frozen. The critic therefore learns from a stable compact representation plus proprioception. This training objective reconstructs existing features; it does not introduce a predictive environment simulator. e-tokene-policye-budgete-implementation
5.2 Use a reference action as guidance, then learn when to depart
At a chunk boundary the VLA supplies a sampled proposal, and the RL actor receives its prefix along with the current state representation. This proposal carries information about a behavioral mode that a single Gaussian actor might otherwise miss. Equation (5) balances critic value against squared distance from that reference. The actor outputs a complete chunk, so refinement is not restricted to adding a hand-scaled residual. A training-time failure would be to copy the reference without using the learned value signal; RLT counters this by zeroing the reference input on half of training samples. The reference remains available at inference. Chunk-level temporal-difference targets use actual rewards and subsequent observations, and human corrections can replace both an executed chunk and its stored reference. e-policye-loope-implementatione-horizons
5.3 Ask which part of the task became better
Reader analysis: the key evaluation distinction is the distribution of states entering the learned segment. A controlled episode starts immediately before insertion or fastening, whereas a full task reaches that segment through earlier VLA actions. Consequently, good controlled success need not imply equally reliable complete tasks. Read Figures 4 and 5 together: throughput combines success with speed, and Ethernet already has little room to improve success. Figure 9 then isolates a concrete timing statistic with medians for the learned policy, base policy, and teleoperation. Finally, Figure 7 asks how quickly useful behavior appears rather than only how good the final checkpoint becomes. Its no-pass-through curve largely catches up, while its no-chunk comparison changes the encoder too. These distinctions explain why final success alone cannot validate every proposed mechanism. e-evaluatione-resultse-speede-ablation
5.4 Training and inference
During training
Per task, 1–10 hours of demonstrations support VLA fine-tuning and token training; Appendix B gives 2,000–10,000 gradient steps. Online RL then freezes both modules. The reported 400–1,000 training episodes yield approximately 15 minutes to five hours of robot data, excluding resets and other overhead. e-budgete-implementation
Actor and critic start from scratch. Appendix B specifies a two-layer MLP with hidden dimension 256 for zip tie, Ethernet, and charger, and a three-layer MLP with hidden dimension 512 for screw. It does not define the layer-count convention, so these are not established hidden-layer counts. Two critics supply the minimum target value. Reference input is zeroed on 50% of training samples; Gaussian noise has a small fixed, unspecified standard deviation. e-implementation
Rollouts and learning are asynchronous, with two critic updates per actor update and an update-to-data ratio of five. Overlapping chunks are stored every two control steps. Operators label success, may correct behavior, and choose the critical-phase handoff. e-loope-handoffe-implementation
During inference
At chunk boundaries, obtain fresh context and a VLA reference, then execute ten actor actions at 50 Hz before replanning. Reference dropout is disabled. The VLA handles easier phases. Section V describes an optional final supervised VLA fine-tune to predict handoff from human labels; switch accuracy is not quantified. e-horizonse-loope-handoffe-taskse-implementation
5.5 Implementation flow
- Expose the representation
The encoder appends a learned readout embedding to final-layer VLA tokens. Its output at that position becomes the 2048-dimensional RL token. The decoder reconstructs VLA embeddings autoregressively with stopped-gradient targets. Tasks use fixed instructions, so language embeddings are omitted from this compression step. e-token
- Refine a proposal
The frozen VLA predicts 50 actions. The actor uses the RL token, proprioception, and the first ten reference actions to propose a complete refined chunk; it does not predict an additive residual. The sampled reference supplies information about the VLA behavior mode. e-horizonse-policye-implementation
- Learn from executed transitions
VLA warmup, actor rollouts, and optional teleoperated corrections enter replay. An intervention replaces both the executed action and stored reference. A chunk-level temporal-difference target propagates sparse terminal success reward; the physical environment supplies the next observation. e-loope-policy
6. Experiments & results
RL Token (RLT) specializes a pretrained VLA through a compact representation and a small online actor–critic. A reconstruction-trained token supplies perceptual context, while sampled VLA actions guide useful refinements. Physical-robot experiments show faster critical phases and improved reliability, but rely on task demonstrations, human supervision, and carefully separated critical-phase and full-task evaluations.
The supplied paper contains no quantitative results table. Its results and ablations are reported in Figures 4–9, so this edition uses original plots rather than inventing a table. Six figures cover the architecture, outcomes, baseline comparison, ablation, and timing distribution. The source does not define its error bars or provide raw outcome counts. e-presentation
6.1 Read the original evidence
Figure 4. Throughput rises in both the controlled bottleneck setting and the two full-task evaluations. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the dotted box first: its screwdriver and zip tie plots measure complete tasks, including earlier stages handled by the VLA. The remaining four plots concern critical phases, starting from partially completed states. Green is the base policy and yellow is RLT. The vertical quantity is successful completions per ten minutes, so it combines reliability and execution speed. Each subplot has its own vertical scale; compare paired bars within a task before comparing across tasks. Figure 5 supplies the complementary success-rate view needed to distinguish increased reliability from faster successful execution. e-resultse-evaluatione-budgete-presentation
What it supports. RLT’s throughput advantage is visible across all four critical phases and both full-task comparisons. This supports improvement in practical completion efficiency within the evaluated settings. It does not imply that the same absolute rate or improvement factor applies to every task, since their initial states and durations differ.
Where the evidence stops. The full-task panels are only screw and zip tie. Whiskers are present but their statistical meaning is unspecified. Do not interpret this ten-minute throughput metric as the total wall-clock budget required to train the policy.
Figure 5. The hardest precision segments gain reliability; Ethernet offers more room for speed than success. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the same setting boundary as Figure 4: the boxed pair is full-task evaluation, while the other panels isolate critical phases. All vertical axes here show success rate, allowing direct comparison of reliability. The unboxed screwdriver panel is the comparison that matches the introduction’s reported improvement from 20% to 65%. It must not be read as the complete screw-installation success rate. Ethernet starts near the ceiling, making a throughput improvement more informative than its small success change. Section VI-A describes slightly randomized partial starts and 50 episodes per agent and task for controlled evaluation. e-resultse-evaluatione-handoffe-presentation
What it supports. The figure supports a task-dependent benefit: RLT substantially improves reliability on difficult screw and zip tie segments while maintaining strong Ethernet success. Full-task execution remains harder because earlier VLA behavior produces additional failures and a broader distribution of states at the handoff.
Where the evidence stops. The paper reports 65% but does not give raw counts or explain aggregation across the stated 50-episode evaluations. Its full-task gains are described as 40% and 60% without a percentage convention; those phrases should not be treated as exact relative gains.
Figure 6. Similar final success can conceal substantial differences in insertion throughput. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel for reliability and then track each method to the right panel for throughput. The dashed green line is the base VLA, not another trained baseline. DAgger learns from intervention data, HIL-SERL trains a small policy using standard visual features, PLD learns single-step residual actions, and DSRL steers VLA generation through latent noise. RLT instead conditions its chunk actor on a reference proposal. DAgger, DSRL, and RLT cluster near the top of the success axis, but their throughput bars separate. The paper evaluates these alternative methods on Ethernet rather than across the whole four-task suite. e-baselinese-baseline-confige-evaluatione-policy
What it supports. RLT’s advantage over the stronger baselines here concerns speed at high success, not simply whether insertion becomes possible. The paper attributes poor single-step performance to difficult sparse-reward credit assignment and describes latent-noise steering as more restrictive than direct action refinement; these are the authors’ explanations of this comparison.
Where the evidence stops. Appendix C changes HIL-SERL’s control frequency from its original 10 Hz to 50 Hz and omits its action-space bounding box. These implementation differences limit a general ranking of algorithms, even though the authors state matched RL data amounts.
Figure 9. The final insertion policy shifts the episode-length distribution toward faster completion. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is episode length in control timesteps; leftward means faster. The vertical axis is percentage, so the colored bars depict distributions rather than accumulated training reward. Read the labeled dashed medians directly: yellow RLT is 66, green teleoperation is 146, and red base policy is 228 timesteps. These are critical insertion episodes, not the complete sequence of grasping and transport. Read the histogram alongside the authors’ qualitative account of reduced probing and more decisive contact, while keeping that account separate from what the timing distribution alone can establish. e-speede-evaluatione-results
What it supports. The directly printed medians show the learned policy completing this insertion segment faster than both the base VLA and expert demonstrations. This is stronger evidence than a generic speedup phrase because the comparator and statistic are visible. The conclusion remains specific to this task segment and demonstrated evaluation distribution.
Where the evidence stops. Median episode length, mean completion steps, and throughput are different statistics. The plot does not by itself establish why behavior changed, document contact forces, or show that the same speed advantage holds across complete tasks or unseen objects.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Screw installation — critical-phase success Physical robot; slightly randomized starts immediately before the critical phase. Section VI-A specifies 50 evaluation episodes per agent and task. | 65% reported for RLT Success rate | 20% for the task-fine-tuned base VLA. The introduction states these values; Figure 5 places the comparison outside the full-task box. The reported difference is 45 percentage points. Raw counts and the meaning of plotted error bars are not supplied. e-resultse-evaluatione-presentation |
| Ethernet insertion — critical-phase episode length Figure 9 compares the final policy with the base VLA and expert teleoperated demonstrations on the insertion segment. | 66 timesteps Median episode length, control timesteps; lower is faster | Base VLA: 228; teleoperation: 146. The directly labeled medians support faster insertion than either comparator. They are not full-task timings or the mean-completion statistic discussed elsewhere. e-speede-evaluation |
| Ethernet insertion — comparison with other learning methods Controlled critical-phase protocol; authors state equal data amounts for RL methods. | RLT has the highest plotted throughput, with near-ceiling success. Success rate and successful completions per ten minutes | DAgger and DSRL have comparable high success but lower throughput; PLD and HIL-SERL perform worse in this implementation. The comparison is confined to Ethernet. Appendix C notes HIL-SERL runs at 50 Hz here versus its original 10 Hz and lacks its action-space bounding box. e-baselinese-baseline-confige-evaluation |
| Ethernet insertion — ablation learning speed Figure 7 evaluates successive checkpoints against minutes of collected critical-phase data. | RLT exceeds the base policy by five minutes of data; the caption reports about 40 minutes total experiment time. Throughput, successful completions per ten minutes | Removing reference pass-through slows early learning but eventually reaches similar throughput. Removing the BC regularizer gives the largest failure. No seed uncertainty is reported. The single-step variant also changes the encoder, so it does not isolate chunk length. e-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Reference actions accelerate early learning, while the regularizer anchors exploration. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is collected robot-data minutes; it excludes the reset and operational overhead discussed in the text. Track the yellow RLT curve against the dashed base-policy line, then compare the ablations by color. The red curve removes the reference action from actor input; purple replaces the RL token with ResNet-10; blue uses single-step actions and also changes the encoder; gray sets the anchoring coefficient to zero. Reading the entire trajectory matters: removing pass-through delays improvement but nearly catches up at the last checkpoint. Figure 8 on the same page provides a companion success-rate trajectory. e-ablatione-policye-budgete-presentation
What it supports. The full system exceeds the base policy after five minutes of critical-phase data, corresponding to about 40 minutes of total experiment time in the caption. Pass-through principally improves the learning path in this experiment rather than establishing an exclusive final-performance advantage. Removing the regularizer prevents useful learning in the plotted run.
Where the evidence stops. The single-step ablation confounds horizon and encoder. Also, the prose claims a 50% throughput loss without the token, while the final plotted values are about 150 versus 400. That numerical discrepancy remains unresolved; the direction of the effect is clear.
7. Analysis & limitations
7.1 What the evidence leaves open
Training still needs human reward labels, corrections, and handoff decisions. Controlled trials start near the bottleneck; full-task testing covers only screw and zip tie. These results do not establish unattended improvement or broad transfer to unseen tasks. e-humane-evaluation
The no-chunk ablation also substitutes ResNet-10, confounding horizon and representation. Section VI-C says removing the token reduces throughput by 50%, while Figure 7 endpoints are approximately 150 versus 400; the paper does not reconcile this difference. e-ablation
No results table, raw trial counts, seed count, or definition of plotted error bars is supplied. Section VI-C describes full-task gains as 40% and 60% without specifying relative change versus percentage points; this report does not convert those phrases into precise estimates. e-presentatione-results
7.2 Questions for discussion
- Does the RL token retain a benefit when encoder capacity, data, and action horizon are matched? [e-ablation]
- How much practical improvement survives automated rewards and handoffs under full-task starts? [e-human, e-handoff, e-evaluation]
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the specified VLA, task demonstrations, camera/proprioceptive interfaces, 50 Hz control, human labels, and staged critical-phase/full-task training. Missing settings include beta and gamma values, learning rates, exact warmup budget, noise magnitude, token-transformer depth, and training hardware. Appendix B also leaves the MLP layer-count convention unspecified. These are implementation gaps, not evidence of failed reproduction. e-tokene-policye-loope-budgete-implementatione-taskse-handoff
Proposed checks should separate representation from chunk length and test reference dropout under matched supervision. Record interaction and elapsed time. Algorithm 1 stores next states using t+1 although Equation (3) and Section III specify C-step backups; implementation must resolve that indexing explicitly. e-ablatione-policye-loope-horizonse-implementation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate chunk length from representation quality
Reader-proposed check, not performed: on controlled Ethernet insertion, compare three arms—RL token with C=10, ResNet-10 with C=10, and ResNet-10 with C=1. The first pair isolates representation under matched chunking; the second isolates chunking within the same encoder. Match demonstration data, reference-generation schedule, actor capacity where feasible, replay budget, reward labels, and intervention allowance. Repeat independent training runs and evaluate each checkpoint on the same reset distribution using the paper’s 50-episode protocol. Report success, throughput, failures during learning, and elapsed time. If the token arm has no repeatable advantage over matched ResNet chunking, the claimed representation benefit weakens even if chunking remains useful. e-ablatione-evaluatione-policye-implementation
Check 2: Test whether dropout prevents copying the VLA proposal
Reader-proposed check, not performed: retain the RL token, C=10, reference-conditioned actor, and identical anchoring strength, but compare the reported 50% reference dropout against zero dropout. Keep warmup, data and human supervision matched, and provide references at every inference call in both arms. Log action distance from the reference, success, throughput, and learning failures at matched interaction budgets; evaluate both with the same controlled starts. The proposed mechanism predicts that dropout permits useful departures from the VLA earlier without simply destabilizing behavior. If zero dropout achieves equally fast, stable improvement across repeated runs, the anti-copying rationale lacks support in that setting. Include total elapsed time to expose reset and supervision costs. e-policye-implementatione-ablatione-evaluation
8.3 Reading coverage
Visual audit: Visually inspected the title and author page, Figures 1–9, all method and evaluation pages, Algorithm 1, and Appendices A–C on pages 12–13. Checked Figure 2 against Eqs. (1)–(2), including reconstruction direction, preceding targets, and stop-gradient text. Appendix B on page 12 supports the stated MLP depths and hidden dimensions but leaves the layer-count convention unspecified; the report preserves this ambiguity. Figures 7–8 were compared with the ablation definitions; the token-ablation percentage discrepancy is disclosed. Every final crop was viewed at its helper-returned size with legends and axes intact. Pages 10–11 contain acknowledgments/references and were read as text, not visually inspected; external videos, code, model cards, and supplements were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- I. Introduction (pp. 1–2)
- II. Related Work (pp. 2–3)
- III. Preliminaries (p. 3)
- IV. Reinforcement Learning from RL Token, A–B (pp. 3–5)
- V. The Complete System, including Algorithm 1 (pp. 5–6)
- VI. Real-World Experiments, A–C (pp. 6–9)
- VII. Conclusion (p. 9)
- Acknowledgments and References (pp. 10–12)
- Appendix A. Contributions (p. 12)
- Appendix B. Additional experiment details (p. 12)
- Appendix C. Additional experimental details for the baselines (pp. 12–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting the original PDF figures and method pages. All six supplied text chunks were read in full, including references and appendices.
- The observed title and seven authors match the supplied identity; the title page lists Physical Intelligence. The PDF is arXiv:2604.23073v2, dated 30 April 2026, while the catalog submission date is 24 April 2026. Only v2 was supplied, so changes from v1 cannot be established and this report does not describe v1.
- No external project page, model card, code, videos, datasets, or separate supplement was inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, affiliation, and arXiv margin stamp
The observed title credits Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke at Physical Intelligence; the stamp identifies 2604.23073v2, 30 April 2026.
Go to primary source ↓e-problemPDF pp. 1–3, Sections I and IV opening
Precision-critical execution motivates a compact frozen-VLA interface for sample-efficient online actor–critic learning.
Go to primary source ↓e-horizonsPDF p. 3, Section III, VLA and RL preliminaries
The VLA predicts H=50 actions; RL uses C<H. The chunk-level value expression bootstraps from state t+C. Reward is a human-labeled terminal success/failure signal.
Go to primary source ↓e-tokenPDF p. 4, Figure 2, Section IV-A, Eqs. (1)–(2), footnote 1
The encoder extracts a 1×2048 readout. Autoregressive reconstruction uses stopped-gradient VLA embeddings. Language embeddings are omitted for fixed-instruction tasks; VLA and readout are frozen for online RL.
Go to primary source ↓e-policyPDF pp. 4–5, Section IV-B, Eqs. (3)–(5) and Reference action dropout
The state is the RL token plus proprioception. A Gaussian actor conditions on a VLA reference, maximizes chunk critic value with a quadratic anchor, and receives zero reference inputs on some training transitions.
Go to primary source ↓e-loopPDF pp. 5–6, Section V, Algorithm 1 and Warmup/Rollout/Subsampling/Update paragraphs
Replay includes warmup, autonomous, and human actions; corrections replace stored references. Chunks use stride two. Updates are asynchronous, with two critic updates per actor update and update-to-data ratio five. Algorithm lines 10–12 use t+1 notation despite chunk execution.
Go to primary source ↓e-handoffPDF p. 6, Section V, Targeted improvement of critical phases
Operators select handoff and terminal outcome. An optional final supervised VLA fine-tune can learn handoff prediction from intervention labels.
Go to primary source ↓e-tasksPDF pp. 6–7, Figure 3 and Section VI-A, task descriptions and Experiment Details
Four tasks involve screw, zip tie, Ethernet, and charger manipulation. Experiments use two wrist views, one base view, proprioception, 50 Hz control, and 14-dimensional actions forming a 140-dimensional RL chunk.
Go to primary source ↓e-budgetPDF p. 7, Section VI-B; p. 12, Appendix B
Task demonstrations span 1–10 hours; online runs span 400–1,000 episodes and 15 minutes to five hours of data excluding overhead. Initial adaptation uses 2,000–10,000 gradient steps; harder-task policies are reported after about five hours of data.
Go to primary source ↓e-evaluationPDF p. 7, Section VI-A, Critical-phase evaluation and Full-task evaluation; Section VI-B
Controlled evaluation uses slightly randomized partially completed states and 50 episodes per agent/task. Full-task evaluations start at home and cover screw and zip tie. Metrics are success and successful completions per ten minutes.
Go to primary source ↓e-resultsPDF p. 2, Section I, first results paragraph; p. 8, Figures 4–5 and Section VI-C Q1
The introduction reports screw success rising from 20% to 65%; Figure 5 places that comparison in the critical-phase setting. Figures 4–5 show throughput/success across tasks. Q1 labels full-task gains 40% and 60%, without defining the percentage convention.
Go to primary source ↓e-baselinesPDF pp. 7–8, Section VI-B baseline definitions; p. 8, Figure 6 and Section VI-C Q2
The Ethernet comparison shows higher RLT throughput than DAgger, HIL-SERL, PLD, and DSRL. DAgger and DSRL reach similarly high success. Authors state matched data amounts for RL methods.
Go to primary source ↓e-ablationPDF pp. 7–9, Section VI-B ablation definitions and Section VI-C Q3; p. 9, Figures 7–8 and captions
No-chunk also replaces the token with ResNet-10. Removing pass-through slows learning; removing regularization fails. Figure 7 reports five minutes of data versus about 40 minutes total experiment time. Q3 claims 50% token-ablation throughput loss, whereas final plotted values are around 150 versus 400.
Go to primary source ↓e-speedPDF p. 9, Figure 9 median labels, caption, and Section VI-C Q4
Ethernet insertion medians are RLT 66, teleoperation 146, base policy 228 timesteps. Authors describe less probing and emergent pressing/wiggling; the caption confines the distribution to critical insertion.
Go to primary source ↓e-humanPDF p. 9, Section VII, limitations paragraph
Human reward labels, intervention corrections, and switching remain required during training; reward/progress models and a fully autonomous pipeline are future directions.
Go to primary source ↓e-implementationPDF p. 12, Appendix B, all paragraphs
Appendix B specifies a two-layer MLP with hidden dimension 256 for zip tie, Ethernet, and charger, and a three-layer MLP with hidden dimension 512 for screw. It does not say whether output layers count toward these depths. Two critics use the minimum target. Reference dropout is 50% during training and absent at inference. C=10, stride two, fixed small Gaussian noise, and staged harder-task training are specified. Exact optimizer, noise magnitude, and hardware settings are absent.
Go to primary source ↓e-baseline-configPDF pp. 12–13, Appendix C
PLD critic pretraining uses 50 base-policy rollouts and Cal-QL. DSRL repeats a 1×32 latent 50 times. HIL-SERL starts with 20 demonstration episodes, runs at 50 Hz instead of its original 10 Hz, and lacks an exploration bounding box. DAgger uses demonstrations plus the same intervention data.
Go to primary source ↓e-presentationPDF pp. 8–9, Figures 4–9 and captions; pp. 12–13, Appendices B–C
Results are plotted rather than tabulated. Bar plots contain uncertainty whiskers but no definition is supplied; raw outcome counts and training seed count are not reported. Appendices add implementation descriptions rather than results tables.
Go to primary source ↓8.5 Primary sources
RL Token: Bootstrapping Online RL with Vision-Language-Action Models ↗
PDF · 9,560 extracted words
Source fingerprint
a64c94a365332756ee45c5762a630bae9bfa13fdba80bd626744883032ac4c8b