π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
1. Paper overview
In one sentence: Tractable stochastic denoising enables online RL for flow-based robot policies, improving familiar-task execution while leaving new-task generalization limited. e01e02e03e04e05e06e16e18
| At a glance | What to know |
|---|---|
| Research problem | Source description A flow VLA already maps images and instructions into continuous action chunks, but its final-action likelihood is difficult to evaluate for policy gradients. Deterministic denoising also limits exploration. The paper seeks to improve demonstration-initialized policies through rewarded interaction while retaining the pretrained flow action generator. e02e03 |
| Core mechanism | |
| A key reported result | LIBERO few-shot manipulation: Flow-Noise: 98.3 average; 94.0 Long. Success rate (%). π0.5; 40 SFT trajectories across four suites; 500 evaluation states per suite. Few-shot SFT: 77.1 average, 43.9 Long; full-data SFT: 96.9 average. Long improves by 50.1 percentage points. Demonstration efficiency does not imply equal total interaction cost. e09 |
| Reading caution |
Core contributions
Figure 2. Exploration is inserted into the action denoising process. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the observation box, then follow the arrow through the vision-language model and its KV cache into the action expert. The snowflake and flame mark the frozen VLM and trainable expert during the main RL stage. Across the top, the blue circles are successive action candidates, beginning at Gaussian noise and ending at the action. The lower-right box distinguishes learned diagonal noise in Flow-Noise from the fixed schedule and drift correction in Flow-SDE. Its formulas agree with Equations (4) and (9). The state list in this schematic should be read alongside the model-specific qualification below. e03e04e05e07e08
What it supports. The trainable stochastic object is the action-generation trajectory. Gaussian transitions provide a tractable density for RL updates while preserving the pretrained VLM as a conditioning module. Flow-Noise learns the noise scale; Flow-SDE fixes its schedule and modifies the mean as well as the variance.
Where the evidence stops. The schematic includes proprioception, and Figure 4 feeds state into π0.5’s VLM; Appendix B.2 explicitly says the experimental π0.5 configuration omits state. This unresolved source difference prevents treating the drawing as an exact input specification for every run. A separate formula concern is unresolved: this figure repeats Equation (9), but pp. 3–5 use inconsistent time/score conventions. The printed expressions do not establish marginal preservation under the stated noise-to-action time direction. This issue is distinct from finite-step sampling error; implementation behavior has not been checked.
2. Motivation
2.1 The problem and the proposed response
A flow VLA already maps images and instructions into continuous action chunks, but its final-action likelihood is difficult to evaluate for policy gradients. Deterministic denoising also limits exploration. The paper seeks to improve demonstration-initialized policies through rewarded interaction while retaining the pretrained flow action generator. e02e03
2.2 What this reading follows
A pretrained flow VLA can turn an instruction and camera images into a smooth action sequence, yet that generator is awkward to optimize with policy gradients: its final-action probability is difficult to calculate. πRL changes the training process by inserting Gaussian transitions whose probabilities are accessible. The two alternatives differ in how they inject noise and organize denoising decisions. This reading follows that mechanism into the benchmark evidence, emphasizing the distinction between stochastic exploration and ODE evaluation. The supplied January 2026 revision reports large simulation gains, a useful efficiency ablation, and a much narrower physical transfer demonstration; its protocol inconsistencies remain explicit. e01e02e03e04e05e06e16e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded VLA post-training and flow-matching categories fit the architecture. πRL predicts actions and learns from environment rewards; it does not introduce a learned future-world predictor, inverse-dynamics action extractor or joint future/action model. The WAM architecture, prediction-paradigm and quadrant axes are therefore not applicable, rather than unverified. e03e04e06e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Make the action generator expose a usable probability
The starting model already has a useful conditional action distribution from flow matching. Its VLM supplies image and language features, and its action expert transforms Gaussian noise into an action chunk. πRL keeps that information flow but changes the rollout distribution during RL. In Flow-Noise, the original Euler update becomes the mean of a Gaussian transition, with a learned noise network setting diagonal covariance. The product of these conditional densities is a probability for the complete denoising trajectory. This distinction matters: the paper optimizes a surrogate based on that trajectory, rather than claiming to have directly evaluated the final action’s marginal density. Once fine-tuning ends, the extra noise network is discarded and the evaluation sampler returns to ODE denoising. e03e04e17
Figure 3. The two formulations expose different units of policy optimization. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper panel as an environment transition with a complete denoising trajectory attached. Its product of Gaussian probabilities is the trajectory density in Equation (5). In the lower panel, each green observation is paired with a blue noisy action candidate to form an inner state. The repeated observation means the robot has not yet advanced in the environment. The solid environment arrow appears only after the final denoised action; it produces a new observation and reward, then resets the next action candidate to Gaussian noise. This ordering matches the branches at flow time below one and equal to one in Equations (10)–(12). e04e06e07
What it supports. Flow-SDE turns denoising itself into a policy decision process, so the likelihood needed by PPO can be computed at a Gaussian transition. The hybrid variant described in the text then selects one stochastic transition and wraps the remaining ODE steps to shorten the effective training horizon.
Where the evidence stops. The inner states are computational action candidates, not simulated future observations. The full two-layer drawing also does not display the single-stochastic-step hybrid implementation; Section 4.2.3 is needed to understand that efficiency modification.
5.2 2. Decide which denoising transitions PPO should optimize
Flow-SDE separates two clocks. During flow time, the observation remains fixed while the noisy action evolves; during environment time, the completed action is executed and the robot receives new feedback. The reward belongs to this outer transition. The full two-layer MDP makes Gaussian denoising decisions available to PPO, but extends the effective optimization chain. Hybrid sampling selects a single stochastic denoising time for each environment step and lets a wrapper perform the other ODE updates. PPO then combines the resulting likelihood ratio with a GAE advantage computed from chunk rewards and a shared critic. The critic’s feature source is model-specific: π0 uses the action expert with trajectory averaging, whereas π0.5 uses the VLM. The source’s conflicting π0.5 state-input descriptions still require implementation clarification. e05e06e07e08
5.3 3. Read the gains as action refinement under particular distributions
The most defensible interpretation is that online interaction improves execution within a learned repertoire. LIBERO supplies a concrete example: the π0.5 few-shot Long score improves from 43.9% to 94.0%, although the RL policy also receives additional simulator experience. ManiSkill’s environmental perturbations provide evidence that those gains are not confined to the exact training appearance. MetaWorld ML45 sets a harder boundary: changing the task objective produces an oscillating held-out curve rather than consistent improvement. This is a reader interpretation of the reported evaluations, not proof of which internal representation changed. The separate 40% physical transfer result adds an execution demonstration, but its unreported trial count prevents a precision assessment. Reproduction should therefore test task success, exploration stability and computational cost separately. e09e11e13e14e18
5.4 Training and inference
During training
SFT tunes the full 3.3B model on expert demonstrations. Main RL runs freeze the VLM and tune the 300M action expert, plus the required critic and, for Flow-Noise, noise network. LIBERO π0.5 uses 40 SFT trajectories across its suites; online interaction supplies additional experience. e04e07e08e09
Rewards depend on the benchmark: LIBERO and MetaWorld use binary task success, CALVIN rewards completed subtasks, and ManiSkill adds attachment reward to placement success. These settings differ in reward density and training data. e09e11e12e13
During inference
Evaluation uses ODE denoising without injected exploration noise; Flow-Noise discards its noise network. Initial Gaussian noise remains part of the described generator, so “deterministic” denotes the denoising updates. Prediction horizon H and executed replan horizon H′ can differ; fresh observations then close the control loop. e03e04e17e22
5.5 Implementation flow
- Condition a flow action expert
The VLM extracts visual-language features; its KV cache conditions the action expert. Starting from Gaussian action noise, the expert predicts velocities toward an action sequence. The latent trajectory is an evolving action candidate, not a predicted future image or environment state. e03e04
- Flow-Noise: score a stochastic trajectory
A learned noise network supplies diagonal covariance around each Euler update. Multiplying Gaussian transition densities gives an exact joint denoising-trajectory density used for the policy-gradient surrogate; this is not an exact marginal likelihood of the final executed action. e04
- Flow-SDE: expose denoising decisions
A fixed schedule adds diffusion and corrects the drift. The inner state contains the current observation and noisy action; observation stays fixed until execution in the outer environment produces feedback. Hybrid sampling makes only one randomly chosen denoising step stochastic per environment step. e05e06
- Estimate values and update PPO
Chunk rewards sum constituent rewards; GAE supplies advantages for clipped PPO updates. π0’s critic reads the action expert and averages values over denoising; π0.5’s critic reads the VLM. Figure 4 includes π0.5 state input, whereas Appendix B.2 says the experiments omit it. e07e08
6. Experiments & results
πRL makes flow-based robot action policies trainable with online PPO by assigning tractable Gaussian probabilities to denoising transitions. Flow-Noise learns exploration noise; Flow-SDE combines a fixed noise schedule with a nested, optionally shortened MDP. Simulation gains are large, but transfer is stronger for environmental variations than unseen task objectives (e04–e06, e09–e14).
6.1 Read the original evidence
Table 3. Few-shot initialization plus RL can exceed the reported full-data SFT baseline. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the gray full-dataset block from the two few-shot-plus-RL blocks. Within each lower block, compare the SFT row with Flow-SDE and Flow-Noise for the same base model. The four suite columns report success percentages, and Avg. aggregates them. The final column subtracts the corresponding few-shot SFT average, so its entries are percentage-point gains. For π0.5, look especially at Long: the weak 43.9 baseline rises to 94.0 under Flow-Noise. Appendix C.1 specifies 40 demonstrations across the four suites and evaluation on 500 initial states per suite; “one-shot” does not mean one trajectory for the entire benchmark. e09
What it supports. π0.5 Flow-Noise reaches 98.3% average success, compared with 77.1% for its few-shot starting policy and 96.9% for full-data SFT. Its Long improvement is 50.1 percentage points. This is strong evidence that simulator interaction can refine a demonstration-limited action policy on these tasks.
Where the evidence stops. The RL policies receive additional online interaction, so the comparison does not hold total experience or compute fixed. The table reports no seed-level uncertainty, and these LIBERO results do not establish physical deployment success.
Figure 5. Environmental robustness and new-task transfer behave differently. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat these panels as three separate protocols. The CALVIN bars concern the source’s ABC-to-D OOD experiment. The ManiSkill radar moves through instruction, image, texture, confounder and position variations; farther from the center means higher success. Use Table 7 for exact variant scores because this figure labels the RL curve generically. In MetaWorld, the blue line is performance on the training distribution, while the red line evaluates held-out tasks under ML45. Those task objectives differ from the 45 training tasks. The scales also differ: CALVIN displays percentages, while the other panels use success fractions. e11e12e13e14
What it supports. The CALVIN bars report a 17.8-percentage-point increase, and the ManiSkill radar shows broadly improved robustness. The MetaWorld held-out curve oscillates while training success rises. Together these results support transfer to environmental changes more clearly than transfer to entirely new manipulation objectives.
Where the evidence stops. Appendix D.2 defines ABC→D but inserts D→D into its results sentence; the protocol conflict remains unresolved. The 79.1% figure must remain separate from Table 5’s 87.0% main result. Generic RL labels here do not identify every sampler.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO few-shot manipulation π0.5; 40 SFT trajectories across four suites; 500 evaluation states per suite. | Flow-Noise: 98.3 average; 94.0 Long. Success rate (%) | Few-shot SFT: 77.1 average, 43.9 Long; full-data SFT: 96.9 average. Long improves by 50.1 percentage points. Demonstration efficiency does not imply equal total interaction cost. e09 |
| ManiSkill pick-and-place robustness 4,352 training combinations; OOD visual-language, confounder and execution variations. | π0.5 Flow-SDE: 90.9 ID, 49.3 OOD; Flow-Noise: 89.7 ID, 53.4 OOD. Success rate (%) | SFT: 40.1 ID, 26.4 OOD. Both improve robustness; Flow-SDE leads ID while Flow-Noise leads aggregate OOD. e11 |
| CALVIN sequential execution Main Scene-D evaluation, 1,000 episodes; separate Figure 5 evaluation labeled ABC→D. | Main π0.5 Flow-SDE: 87.0%, 4.717 subtasks; separately reported OOD RL: 79.1%. Five-subtask success (%) and completed subtasks | SFT: 61.3%, 3.838 subtasks. Keep the main and OOD results separate: D.2 contradicts its ABC→D setup with a D→D phrase. e12e14 |
| MetaWorld skill breadth and held-out tasks MT50: SFT on 2,500 trajectories; ML45 separately trains 45 tasks and tests five unseen tasks. | MT50 π0 Flow-Noise: 85.8; π0.5 Flow-SDE: 70.7. Success rate (%) | SFT: 50.8 and 43.8 respectively. ML45’s oscillating OOD curve shows no sustained transfer of the ID gains to novel objectives. e13 |
| MDP formulation efficiency π0 Flow-SDE, LIBERO-Goal, fixed noise across formulations. | Hybrid two-layer: 428.6. Update time (seconds) | Two-layer: 814.2; one-layer: 821.4. Similar final success; roughly 1.9× faster updates, not a measured end-to-end speedup. e16 |
| Flow-SDE noise sensitivity π0, LIBERO-Spatial, evaluation after 100 training steps. | Noise 0.2/0.5/0.8: 73.1/94.5/98.1. Deterministic evaluation success (%) | Corresponding SFT Eval: 65.2 for all three; stochastic SFT Train falls as noise rises. Preserving initial rollout quality alone does not select the best RL noise setting. e17 |
| SIMPLER single-task policies Four simulated WidowX tasks; 144 SFT demonstrations per task; Flow-Noise. | π0: 86.7; π0.5: 79.1. Mean success rate (%) | SFT: 67.2 and 59.2. These are simulation outcomes, separate from physical transfer. e18 |
| Real2Sim2Real pick-and-place transfer Franka Panda; 20 planned SFT trajectories, 100 simulator RL iterations, zero-shot physical deployment. | 40%. Reported physical success rate (%) | SFT baseline reportedly fails. A limited deployment demonstration: the source gives no trial count or uncertainty estimate. e18 |
| GR00T N1.5 LIBERO adaptation Frozen VLM, trainable action expert with dropout removed; Flow-SDE PPO using π0 configurations. | 89.9. Mean success rate (%) | SFT: 52.5. Supports applicability to another action architecture on LIBERO; wider transfer remains untested here. e20 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Shortening the effective denoising chain reduces update cost. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Match the cyan two-layer curve, blue hybrid curve and magenta one-layer curve across the success and explained-variance panels. The one-layer formulation improves fastest early in training, but the final success levels are similar. Then compare the update-time bars: the hybrid bar is substantially lower even though its final task performance is comparable. Section 5.3.2 holds the noise-injection strategy fixed for this comparison, separating MDP organization from learned-versus-fixed noise. The one-layer formulation still recomputes complete denoising trajectories for likelihood estimation, which explains why its shorter conceptual environment chain does not automatically reduce computation. e06e16
What it supports. The hybrid formulation reports 428.6 seconds per update versus 814.2 for the full two-layer formulation and 821.4 for one-layer optimization. The first comparison is approximately 1.9× faster. The result supports selective stochastic denoising as a practical efficiency mechanism on this benchmark.
Where the evidence stops. These are update times for π0 on LIBERO-Goal, not an end-to-end training or inference speed measurement. The figure gives no timing variability, and similar final curves do not establish statistical equivalence.
Table 2. Good stochastic rollouts and good RL learning require different checks. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column group as a separate sweep, rather than a full factorial experiment. Train means stochastic rollout performance; Eval means ODE-based evaluation, not a dataset train/test split. The SFT rows describe the initial checkpoint and the RL rows describe performance after 100 training steps. Under noise level, the initial Eval score stays at 65.2 while Train deteriorates as noise rises. Under denoising steps, one-step sampling has a large initial Train–Eval gap. Under action chunk, longer execution postpones observation feedback. These distinctions connect the table to the clipping, denoising and explained-variance curves on the same page. e05e17e21
What it supports. Noise 0.8 yields 98.1% RL Eval versus 73.1% at noise 0.2, despite worse initial stochastic performance. More denoising is not uniformly better either: the two-step setting reaches 97.0% RL Eval and the eight-step setting 86.7%. Initialization quality alone is an incomplete tuning criterion.
Where the evidence stops. These are π0 LIBERO-Spatial results after a fixed number of training steps. They do not identify a universal optimum or equal-compute winner. The source’s marginal-preservation motivation must be distinguished from finite-step sampling behavior.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge low online-RL sample efficiency, ODE–SDE discretization loss and limited semantic generalization. The physical result does not establish reliable deployment across tasks or environments. e18e21
Source inconsistencies matter: π0.5 state input differs between Figure 4 and Appendix B.2; CALVIN D.2 mixes ABC→D and D→D wording. Its ManiSkill discussion assigns 102.3% relative OOD gain to Flow-SDE although that calculation matches Flow-Noise’s table values. Absolute table values are retained here. e07e08e11e14
Reported headline tables lack seed-level uncertainty. VLM LoRA-II changes learning rate and update count together, so matching the frozen baseline does not isolate representation adaptation. The main results establish executed simulated control, not future-world prediction accuracy. e09e11e12e13e19
7.2 Questions for discussion
- Does the hybrid sampler retain its advantage when compared by total environment interactions and wall-clock time?
- Can broader task distributions improve held-out task transfer without destabilizing the pretrained VLM?
8. Reproducibility audit
8.1 Requirements and known gaps
The documented setup uses eight H100 80GB GPUs. A focused π0 LIBERO-Goal run needs its few-shot checkpoint, binary reward, 64 environments, batch 2048, four update epochs, actor/critic learning rates 5e−6/1e−4, H=50, H′=5, four denoising steps and noise 0.5. Preserve the distinction between prediction and execution horizons. e08e09e22
Resolve the state-input and CALVIN protocol contradictions before claiming exact reproduction. Specify endpoint handling for the printed SDE schedule, software versions and seed protocol; these are not fully pinned down by the paper. Proposed checks in the illustrated edition compare MDP efficiency and noise stability under controlled evaluation. e05e07e08e14e16e17e22
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does hybrid sampling reduce total time to comparable success?
Reader-proposed check: start one-layer, full two-layer and hybrid two-layer runs from the same π0 LIBERO-Goal SFT checkpoint, using fixed noise, identical critic, prediction/replan horizons and the Table 11 configuration. Use several independent seeds and the same 500 evaluation states. Measure update time, rollout time, environment transitions and total time to a prespecified success threshold. Figure 7’s efficiency interpretation is strengthened if hybrid preserves final success while reducing total time; it is weakened if the update saving disappears through slower learning or rollout overhead. e06e09e16e22
Check 2: Separate exploration magnitude from low-noise optimization instability
Reader-proposed check: repeat the π0 LIBERO-Spatial noise sweep at 0.2, 0.5 and 0.8 with checkpoint, denoising steps, action horizon and interaction budget fixed. Include the paper’s common-learning-rate comparison and a separate learning-rate sweep for each noise level. Measure initial stochastic/ODE success, PPO clipping fraction, gradient norms and final ODE success across seeds. If suitably reduced learning rates rescue low-noise performance, optimization instability explains part of the original ranking; if low noise still limits final success after stabilization, restricted exploration remains a plausible explanation. e17e19e22
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, Figures 1–17, Tables 1–13, method equations, evaluation protocols, physical-transfer description and all appendix configuration pages. All six final original crops were viewed at their saved resolution; the two narrow method figures were rendered at 400 DPI and the other crops at 200 DPI. Figure 2 formulas were checked against Equations (4) and (9), and Figure 3 transition/reward ordering against Equations (10)–(12). The π0.5 input and CALVIN protocol discrepancies are disclosed in the guides. All eight supplied text chunks, including reference pages 11–12, were read; those reference-only pages were not visually inspected. Separate supplements and linked code remain outside the supplied review.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract (p. 1)
- 1. Introduction; 2. Related Work (pp. 2–3)
- 3. Preliminary; 4. Methodology, all subsections (pp. 3–6)
- 5. Experimental Results, all subsections; 6. Conclusion (pp. 6–10)
- References (pp. 10–12)
- Appendices A–B: organization and experimental setup (p. 13)
- Appendix C.1–C.4: all ID benchmarks (pp. 13–16)
- Appendix D.1–D.4: all OOD evaluations (pp. 16–18)
- Appendix E.1–E.2: SIMPLER and Real2Sim2Real (pp. 17–18)
- Appendices F–G: algorithms, VLM ablation and training insights (pp. 18–20)
- Appendix H: GR00T N1.5 (pp. 20–21)
- Appendix I: limitations (pp. 21–22)
- Appendix J: all hyperparameter tables (pp. 22–24)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Version scope: the supplied PDF is arXiv:2510.25889v3 (29 January 2026), with a January 30, 2026 preprint line. Earlier editions were not supplied or compared. The catalog’s π_RL title is a typographic variant; its author list omits Bingwen Wei and Jiakai Zhou, both present in this PDF (e01).
- The acquisition notes state that text extraction does not reconstruct figure images; original PDF pages and all final crops were visually inspected for this edition.
- Separate supplemental material availability has not been fully verified.
- Code, linked resources and external datasets were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block, arXiv margin and preprint date
The title is πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. The margin identifies arXiv:2510.25889v3, 29 January 2026; the preprint line says January 30, 2026. Sixteen authors are listed, including Bingwen Wei and Jiakai Zhou.
Go to primary source ↓e02PDF pp. 2–3, Sections 1 and 4 opening
The authors motivate online RL by demonstration costs and the difficulty of tractable action likelihoods and exploration in flow-based action generation; they introduce Flow-Noise and Flow-SDE.
Go to primary source ↓e03PDF p. 3, Sections 3.1–3.2, Eqs. (1)–(3) and inference paragraph
Observations condition a VLM and action expert. The expert learns a conditional velocity field with flow matching; the interpolation is A_t^τ = τ A_t + (1−τ)ε, with target velocity A_t−ε. Gaussian initial noise is integrated into an action sequence.
Go to primary source ↓e04PDF p. 4, Figure 2, Sections 4.1.1–4.1.2, Eqs. (4)–(5)
Flow-Noise uses the Euler mean and learned diagonal Gaussian covariance. The noise network is jointly trained with the velocity and then discarded. Policy optimization uses the tractable joint density of the denoising trajectory.
Go to primary source ↓e05PDF pp. 4–5, Section 4.2.1, Figure 2 and Eqs. (6)–(9)
Flow-SDE is motivated by a marginal-preserving continuous-time conversion. The printed Gaussian mean includes the drift correction and covariance σ_τ²δI, with σ_τ = a sqrt(τ/(1−τ)). This section does not specify an implementation convention for the endpoints.
Go to primary source ↓e06PDF pp. 4–5, Figure 3, Sections 4.2.2–4.2.3, Eqs. (10)–(12)
The inner MDP keeps the observation fixed while denoising; only completion and environment interaction produce an environment reward. Hybrid sampling chooses one stochastic denoising time per environment step, with the remaining updates handled by an ODE wrapper.
Go to primary source ↓e07PDF pp. 5–6, Sections 4.3.1–4.3.2, Eqs. (13)–(16), Figure 4
PPO uses chunk-level rewards and GAE. The shared critic attaches to the action expert for π0 and to the VLM for π0.5. The expert value is averaged over the denoising trajectory. This description and Figure 4 feed state to the π0.5 VLM.
Go to primary source ↓e08PDF p. 13, Appendix B.2, Implementation Details
SFT tunes the full 3.3B model; RL freezes the VLM and tunes the 300M action expert. Experiments use eight NVIDIA H100 80GB GPUs and serial, co-located environment, rollout, and actor execution in RLinf. Unlike Section 4.3.2, the experimental configuration says π0.5 omits state information.
Go to primary source ↓e09PDF pp. 13–14, Appendix C.1 and Table 3, all LIBERO rows
π0 uses 58 demonstrations for Spatial/Object/Goal and 208 for Long; π0.5 uses 40 across all suites. Each suite is evaluated on 500 initial states. π0.5 Flow-Noise reaches 98.3% average and 94.0% Long, versus 77.1% and 43.9% few-shot SFT; full-data π0.5 SFT is 96.9% average. The prose’s single-trajectory shorthand must be read with the 40-trajectory setup.
Go to primary source ↓e10PDF p. 7, Table 1, benchmark and average columns; p. 15, Table 5
Across LIBERO, ManiSkill, MetaWorld and CALVIN, average success increases from 51.1 to 80.3 for π0 Flow-Noise and from 55.6 to 86.6 for π0.5 Flow-SDE. These are heterogeneous benchmark averages; the CALVIN entry is Len-5 success.
Go to primary source ↓e11PDF pp. 14–16, Appendix C.2, Table 4 and Table 7
ManiSkill uses 4,352 object/receptacle/scene combinations, 16,384 motion-planned SFT episodes, and 15 appended completion frames. Rewards are 1.0 for placement and 0.1 for attachment. π0.5 SFT/Flow-SDE/Flow-Noise score 40.1/90.9/89.7 ID and 26.4/49.3/53.4 OOD. Table 7 separates visual-language, confounder and execution variations.
Go to primary source ↓e12PDF p. 15, Appendix C.3 and Table 5, π0.5 rows; p. 7, Table 1
CALVIN SFT uses ABC play data. The main experiment evaluates 1,000 five-subtask episodes in Scene D. π0.5 Flow-SDE reports Len-5 success 87.0% and average 4.717 completed subtasks, versus 61.3% and 3.838 SFT. This is the main result grouped as ID in Table 1.
Go to primary source ↓e13PDF pp. 15–17, Table 6, Appendix C.4 and D.3; p. 7, Figure 5(c)
MetaWorld MT50 SFT uses 2,500 trajectories across 50 tasks and sparse success rewards. π0 Flow-Noise reaches 85.8% versus 50.8% SFT; π0.5 Flow-SDE reaches 70.7% versus 43.8%. ML45 trains on 45 tasks and tests five held-out tasks; its OOD curve oscillates without sustained improvement.
Go to primary source ↓e14PDF p. 7, Figure 5 and Section 5.2.2; pp. 16–18, Appendix D.1–D.4; p. 14, Table 4
The authors distinguish improved robustness to environmental changes from weak transfer to novel tasks. Figure 5(a) reports CALVIN 61.3% to 79.1%; D.2 defines ABC→D but its results sentence also says D→D. D.1 attributes 102.3% relative OOD gain to Flow-SDE, although Table 4’s 53.4 versus 26.4 values yielding that gain belong to Flow-Noise.
Go to primary source ↓e15PDF pp. 7–8, Section 5.3.1 and Figure 6
On π0 Flow-SDE LIBERO-Long, a four-layer critic outperforms a one-layer critic; the VLM placement gives slightly better performance and value diagnostics despite lacking proprioception. The authors retain the expert critic for π0 to include state.
Go to primary source ↓e16PDF p. 8, Section 5.3.2 and Figure 7(a)–(c)
With fixed noise on π0 LIBERO-Goal, one-layer, two-layer and hybrid two-layer formulations reach similar final success. Reported update times are 821.4, 814.2 and 428.6 seconds respectively. Full-trajectory likelihood recomputation prevents the one-layer formulation from being faster.
Go to primary source ↓e17PDF pp. 8–10, Sections 5.3.3 and 5.4, Table 2 and Figures 8–11
The injection comparison sets learnable-noise entropy bonus to zero. Table 2 reports π0 Flow-SDE LIBERO-Spatial after 100 training steps, separating stochastic Train from deterministic Eval. Noise levels 0.2/0.5/0.8 yield RL Eval 73.1/94.5/98.1%; K=1 gives SFT Train 9.4% but Eval 63.8%; K=2 and K=8 yield RL Eval 97.0% and 86.7%.
Go to primary source ↓e18PDF pp. 17–18, Table 8, Figure 12 and Appendix E.1–E.2
SIMPLER uses 144 SFT demonstrations per task and Flow-Noise: π0 average rises 67.2%→86.7%, π0.5 59.2%→79.1%. A separate real transfer study uses a Franka Panda, RealSense D435, ManiSkill dynamics, Gaussian Splatting rendering and manual camera calibration; 20 planned demonstrations plus 100 RL iterations yield reported 40% physical success, with no trial count given.
Go to primary source ↓e19PDF p. 18, Table 9, π0.5 PPO and GRPO rows; pp. 19–20, Appendix F.2–G, Figures 13–16
π0.5 Flow-SDE averages 97.9% with PPO versus 91.5% GRPO on LIBERO. VLM LoRA-II changes both learning rate and update count and tracks the frozen baseline. Cosine scheduling limits KL growth in LIBERO-Long; ManiSkill episode lengths approach the expert range, with partial resets offered as an explanation.
Go to primary source ↓e20PDF pp. 20–21, Appendix H.1–H.2, Figure 17 and Table 10
GR00T N1.5 combines Eagle 2.5 with an action denoising transformer. Its expert is tuned with the VLM frozen and dropout replaced by identity layers; a critic reads the action head. Flow-SDE PPO raises LIBERO average success from 52.5% to 89.9% using π0 configurations.
Go to primary source ↓e21PDF p. 10, Section 6 limitation; pp. 21–22, Appendix I
The authors cite online RL sample inefficiency, reliance on sim-to-real deployment, ODE–SDE numerical performance loss, a simplistic single-stochastic-step hybrid sampler and limited semantic generalization. Improving sampler precision did not automatically improve RL in their reported exploration.
Go to primary source ↓e22PDF pp. 23–24, Appendix J, Tables 11–13
Tables specify per-benchmark learning rates, rollout counts, prediction/replan horizons, denoising steps, noise bounds and entropy bonuses. For π0 LIBERO-Goal: 64 environments, batch 2048, four update epochs, actor/critic rates 5e−6/1e−4, H=50, H′=5, four denoising steps and Flow-SDE noise 0.5. No exact software package versions or random-seed protocol accompany these tables.
Go to primary source ↓8.5 Primary sources
πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models ↗
PDF · 12,789 extracted words
Source fingerprint
484e9f8be118be4f7adfd18470abc330e29b1b4cf457d316dc85a63e65299d36