LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
1. Paper overview
In one sentence: LIBERO-Plus exposes how simulation policies fail under controlled shifts, while augmentation improves scores without establishing robust instruction use or initial-state control. e01e02e04e05e06e08e10e11e15
| At a glance | What to know |
|---|---|
| Research problem | Source description Near-saturated scores on fixed LIBERO scenes can hide reliance on familiar viewpoints, object positions and task cues. The study tests executed manipulation under controlled shifts, then separates apparent invariance from useful modality dependence. Its conclusions concern the evaluated simulation tasks and checkpoints, not general robotic competence. e02 |
| Core mechanism | |
| A key reported result | Post-training on the final LIBERO-Plus benchmark: 79.5, as printed in Tables 2 and 10. Reported Total task success rate (%). Filtered 10,030-instance test benchmark; mixed OFT fine-tuning on the separate augmentation set. OFT_m: 67.9 (+11.6 percentage points); OFT: 69.6. Camera success is 92.8 versus π0-Fast 65.1 and OFT_m 55.6. The gain is broad but incomplete: robot success is 30.3 versus UniVLA's 46.2. Section 6.2's 79.6 and 'next best' 37.2-point camera comparison conflict with the tables; 37.2 is the gain over OFT_m. e10e12e15 |
| Reading caution | Reader analysis Filtering and difficulty labels depend on evaluated models, making difficulty partially endogenous to their success. Figure 6 misprints the layout count; Figure 7 conflicts with Table 7 on three totals. Easy-task filtering, surviving L1 cases and the inconsistent language-component naming need clarification. e10e11 |
Core contributions
Figure 6. The benchmark's structure organizes interventions, rather than neural-network modules. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the inner ring as seven ways to change a LIBERO evaluation episode. Move outward to see the concrete interventions within each family: camera distance and pose, scene and surface textures, initial arm pose, lighting properties, image corruptions, language variants, and distractors or target displacement. The areas represent dataset composition, not robustness scores. Appendix A supplies the operational definitions; Appendix C explains how generated instances are filtered and balanced. The paper calls this an architecture figure, but it is a benchmark composition diagram. Its language slices include Original and Paraphrasing, so they do not map one-to-one onto Appendix A.6's three rewrite categories. e10e11e16
What it supports. LIBERO-Plus makes the source of a distribution shift explicit, enabling per-dimension evaluation instead of a single aggregate score. Table 7 reports 10,030 instances across the seven families. Difficulty is assigned separately from the success of four reference policies, so the rings themselves say nothing about which interventions are hardest.
Where the evidence stops. Preserve the printed error: Layout says 1599, while its outer counts 829 + 696 and Table 7 give 1525. Figure 7 also disagrees with Table 7 on camera, robot and noise totals. Use Table 7 for the reported benchmark composition.
2. Motivation
2.1 The problem and the proposed response
Near-saturated scores on fixed LIBERO scenes can hide reliance on familiar viewpoints, object positions and task cues. The study tests executed manipulation under controlled shifts, then separates apparent invariance from useful modality dependence. Its conclusions concern the evaluated simulation tasks and checkpoints, not general robotic competence. e02
2.2 What this reading follows
A manipulation policy can score well because its evaluation scene supplies familiar cues. LIBERO-Plus tests that possibility by changing the scene, camera, robot initialization, language and image quality, then examining which failures follow. The most revealing experiments remove camera streams or instructions and change the requested target while preserving the scene. A separate paired-perturbation study asks whether isolated robustness predicts behavior under combined shifts. Finally, additional OpenVLA-OFT training raises performance on a filtered benchmark. This reading follows the v3 PDF's controls and results while retaining its conflicting counts, labels and summary numbers, which limit how precisely the experiment can be reconstructed. e01e02e04e05e06e08e10e11e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator category and robotics robustness subcategories fit the contribution. Architecture, prediction paradigm and quadrant are not applicable to this benchmark. Evaluating WorldVLA does not make LIBERO-Plus a One Model system or a joint future/action predictor. e10e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate invariance from dependence on the intended input
Reader analysis: begin with two questions that require different interventions. An instruction rewrite asks whether a policy is stable under changed wording. An empty instruction asks whether the policy needed that wording at all. Figure 3 shows why both are necessary: OFT can keep strong Object-suite performance with no instruction, yet fail when the requested target changes. The target-replacement test then asks for behavioral sensitivity to meaning while preserving the visual scene. Appendix E's example of selecting alphabet soup after being asked for butter makes the mismatch concrete. Still, replacement changes the requested task, and failures can combine language grounding with unfamiliar object manipulation. A convincing causal diagnosis therefore needs balanced exposure to both targets, not only examples where one target was familiar. e05e06
5.2 Understand what the compositionality gap conditions on
Reader analysis: keep the four conditions—neither perturbation, each alone, and both—separate before calculating anything. The paper first measures the success rate in each condition. It then normalizes those rates into a distribution over perturbation settings among successful trials. Equation (8) compares the joint mass in that distribution with the product of its marginals. This explains why Figure 4's lower triangle differs from Appendix F's raw joint success table. A negative gap is evidence against this specific success-conditioned independence reference. It is not a direct measurement of representational entanglement or a guarantee that every pair is statistically significant. Table 9 is essential context: twelve pairs pass its nominal threshold, three do not, and the paper does not describe a multiplicity correction. e07e08e09
5.3 Follow the augmentation pipeline into a continuous-action policy
The training experiment reuses the benchmark's perturbation machinery but generates separate variations. Original LIBERO state-action pairs are executed in modified environments; unsuccessful replays are discarded and no-op actions are removed. The retained demonstrations span six augmentation families. Robot-initialization augmentation is absent, and the layout family contains added distractors rather than target-pose changes because reliable automatic trajectories were unavailable. Distractors and textures avoid test overlap; camera angles differ from the test set by five degrees. Mixed fine-tuning then updates an existing OFT policy using the reported eight-A100 configuration. Appendix B describes OFT's parallel continuous-action MLP and L1 training objective. The result measures executed task success, with strong camera gains but low robot-initialization performance; it does not establish that the policy acquired a predictive world model. e12e13e14e15
5.4 Training and inference
During training
Replay original state-action pairs in modified environments, retain successful trajectories and remove no-ops. Six augmentation types yield 22,400 collected trajectories and over 20,000 after filtering. Robot-state augmentation is absent; layout augmentation adds distractors but excludes target-pose changes because automatic trajectories are unreliable. e12
Mixed fine-tuning starts from official OFT weights: eight A100 GPUs, 100,000 steps, learning rate 5×10−4, effective batch 16, AdamW, weight decay 0.1 and cosine scheduling with warmup. The source describes OFT's continuous MLP action head, parallel prediction and L1 objective; the post-training section does not explicitly specify frozen modules. e13e14
During inference
A policy receives the perturbed observation and available instruction and predicts actions for execution in LIBERO. OFT predicts continuous actions in parallel. The benchmark observes task completion; it adds no learned future-video planner or inverse-dynamics action extractor. Camera masking and empty instructions are evaluation interventions, not training objectives. e02e04e05e14e10
5.5 Implementation flow
- Generate controlled shifts
Alter layout, background, lighting, camera, initial joints, instructions or image quality. Target-pose perturbations preserve semantic relations needed to identify the task object. This is a benchmark intervention pipeline; it does not learn a new predictive state representation. e02e16e10
- Filter and stratify
Expand 40 LIBERO tasks into 14,000 candidates; remove commonly solved instances and balance subdimensions to retain 10,030. Four reference policies define L1–L5 by how many succeed, from all four to none. These levels depend on the reference models; they are not calibrated physical perturbation magnitudes. e10e11
- Intervene on modalities
Mask the third-person image or every image; separately replace language with an empty input. Goal replacement changes the requested object and success goal in the same scene. These controls distinguish robustness to paraphrases from the ability to choose an action using the instruction. e04e05e06
- Measure paired interference
Evaluate OpenVLA-OFT over six non-language dimensions with 2,000 trials per condition. Compare observed joint success-conditioned probabilities with products of their conditional marginals. Keep these experiments separate from both the initial single-factor table and the filtered benchmark. e07e08e09
6. Experiments & results
LIBERO-Plus turns high simulation success into a diagnostic question: does a policy still execute the intended task when appearance, geometry, initialization or language changes? Its 10,030-instance benchmark exposes uneven robustness and weak instruction dependence in several settings. Additional OpenVLA-OFT training improves the final benchmark score to 79.5%, while robot-initialization robustness remains poor. Evidence: e02, e05, e10, e15.
LIBERO-Plus contributes a benchmark and an augmentation study, so its supplied architecture visual is the benchmark composition chart rather than a new neural-network diagram. The quantitative results and failure rollouts concern simulated LIBERO execution; no physical-deployment experiment is supplied. e02e10e17
6.1 Read the original evidence
Table 1. A strong original score can conceal large losses under camera and initial-state changes. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. For each model, first read the unshaded row as success percentages. Then read the shaded row beneath it as the absolute decrease from that model's Original column; the downward arrows denote percentage-point losses. Follow OpenVLA-OFT across Original, Camera and Robot before comparing it with the third-view-only OFT_w variant. The suffix w identifies a policy without wrist-camera input, while m identifies mixed training across all four suites. These rows belong to the initial vulnerability study. The later filtered benchmark has different values in Table 2, so the two tables should not be treated as interchangeable measurements. e03e14e05e06
What it supports. OpenVLA-OFT falls from 97.1% to 59.7% under camera changes and 37.2% under changed robot initialization. Its third-view-only variant reaches just 16.8% under camera changes. The large variation across columns shows why original success alone is a weak indicator of reliability under changed conditions.
Where the evidence stops. OFT_w was separately trained, so this comparison changes more than an inference-time camera mask. The table has no confidence intervals. Language robustness here also does not establish instruction following; the separate removal and replacement controls test that interpretation.
Table 2. Additional augmentation training improves overall success, with a persistent initial-state weakness. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the blue Ours row and compare it with Openvla-OFT_m immediately above. The green arrows give absolute improvements over that mixed-training baseline, not over the strongest competitor in each column. Next compare against all rows: π0-Fast is the strongest previous camera result, while UniVLA is stronger than Ours on robot initialization. The table evaluates the filtered LIBERO-Plus benchmark after training on automatically replayed successful trajectories in modified scenes. It is distinct from Table 1's initial perturbation evaluation. Tables 2 and 10 agree on the Ours total of 79.5, which is the value retained here. e15e12e13e10
What it supports. Post-training raises the reported total from 67.9% to 79.5%, an 11.6-point improvement over OFT_m. Camera success reaches 92.8%; its gain over π0-Fast's 65.1% is 27.7 points, calculated from the table. Robot initialization remains weak at 30.3%, below UniVLA's 46.2%.
Where the evidence stops. Section 6.2 says 79.6% and misidentifies the 37.2-point camera gain as over the next best model; it is over OFT_m. Boldface also incorrectly suggests Ours leads on Robot. No matched-budget training ablation isolates which augmentation causes the gains.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Initial single-dimension robustness Section 2 evaluation on perturbed LIBERO episodes; Table 1, before the final benchmark filtering. | OpenVLA-OFT: original 97.1; camera 59.7; robot initialization 37.2. Task success rate (%) | The reported absolute drops are 37.4 and 59.9 percentage points. OFT_w has camera success 16.8%. High original success hides severe geometric sensitivity. OFT_w was separately trained, so the cross-model comparison alone does not isolate wrist-camera causality. e03e14 |
| Camera-input masking Section 3 extreme ablation; third-view-only masking preserves the wrist view. | Third-view masked: OFT 43.6, π0 43.0, π0-Fast 67.3. All views masked: 0.0, 2.0, 0.0. Task success rate (%) | Original rates in this figure are 97.1, 94.2 and 85.5. Useful visual information remains in the wrist stream. This demonstrates dependence on available views, without proving illumination-invariant representations. e04 |
| Instruction removal and target replacement Figure 3 across four LIBERO suites and the Object-Var replacement panel; the suite naming conflict is retained. | OFT's Object-suite performance remains high with empty instructions; changed-target performance collapses. Exact bar values are not printed. Task success rate, shown as bars | The same figure shows substantial OFT losses on Spatial and Goal, contrary to the prose's claim that only Long degrades significantly. Instruction use is task-dependent. Goal replacement changes task demand as well as wording, so failure does not uniquely identify a language-understanding defect. e05e06 |
| Pairwise camera and robot perturbations OpenVLA-OFT, 2,000 trials per condition in Section 5 and Appendix F. | Camera only 57.30; robot only 39.10; joint 19.05; p=0.00931. Task success rate (%) and reported chi-square p-value | All heatmap gaps are negative, but only 12 of 15 Table 9 p-values are below the unadjusted 0.05 threshold. Several interactions are supported under this protocol; universal significance and a particular representation-level cause are not established. e08e09 |
| Post-training on the final LIBERO-Plus benchmark Filtered 10,030-instance test benchmark; mixed OFT fine-tuning on the separate augmentation set. | 79.5, as printed in Tables 2 and 10. Reported Total task success rate (%) | OFT_m: 67.9 (+11.6 percentage points); OFT: 69.6. Camera success is 92.8 versus π0-Fast 65.1 and OFT_m 55.6. The gain is broad but incomplete: robot success is 30.3 versus UniVLA's 46.2. Section 6.2's 79.6 and 'next best' 37.2-point camera comparison conflict with the tables; 37.2 is the gain over OFT_m. e10e12e15 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 2. Masking cameras reveals which visual stream still supports execution. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four horizontal labels as distinct evaluation conditions, not as a calibrated severity axis. Origin supplies normal images. Light applies illumination changes. In 3rd Black, the third-person image is replaced by black pixels while the wrist image remains available. All Black removes the images from both views. Trace each model's color across these interventions, using the printed point labels for success rates. The decisive comparison is between 3rd Black and All Black: it asks whether the remaining wrist stream contains useful information. The source attributes this benefit to close-range geometric and contact cues; that explanation goes beyond the masking measurement itself. e04e03
What it supports. With only the wrist view remaining, OpenVLA-OFT, π0 and π0-Fast retain 43.6%, 43.0% and 67.3% success. Removing both views reduces those values to 0.0%, 2.0% and 0.0%. The experiment supports a meaningful contribution from visual input even when ordinary lighting changes have a smaller effect.
Where the evidence stops. The shaded bands are undefined, so they cannot be read as confidence intervals. Light values here differ from Table 1. Masking does not isolate illumination invariance, and black frames are a different intervention from physically changing a camera's viewpoint.
Figure 3. Instruction removal and target substitution expose different forms of task dependence. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to follow a model, then compare light and dark bars within each suite. Panel (a) preserves the task goal but removes the instruction. Panel (b) changes both the linguistic target and the success goal while keeping the scene. For OpenVLA-OFT, the Object bars nearly coincide in panel (a), yet its dark Object-Var bar in panel (b) nearly disappears. This is compatible with scene cues supporting familiar Object tasks without an instruction. Also inspect Spatial and Goal: their large light-to-dark differences matter because they contradict the prose's claim that significant OFT degradation appears only on Long. e05e06
What it supports. The figure supports task-dependent use of language. Successful execution without a prompt in one suite is insufficient evidence that the model can select a new target from language. Appendix E's rollouts strengthen the behavioral diagnosis by showing original-target actions after instructions request another object.
Where the evidence stops. Exact bar values are not printed. Section 4.2 calls the replacement suite layout; Appendix E says object and the panel says Object-Var. Changed-target failures also involve changed task demand, and the figure contradicts the blanket claim that these models ignore language.
Figure 4. Paired perturbations depart from the paper's success-conditioned independence reference. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a pair of dimensions and compare its two cells across the diagonal. The upper triangle gives the product of the two perturbation marginals conditioned on successful execution. The lower triangle gives their observed joint probability, also conditioned on success, followed by the gap in parentheses. For Background and Camera, compare 15.19 above with 13.57 below and the printed −1.62 gap. These displayed quantities are on a percentage scale after normalization across the four experimental conditions. They are not raw task-success rates: Appendix F separately reports 37.70% success when background and camera are jointly perturbed. e07e08e09
What it supports. Every displayed gap is negative for this OpenVLA-OFT study, indicating that successful cases under joint shifts are less frequent than the chosen conditional-independence reference predicts. Background–camera is a particularly clear example, with Table 9 reporting χ²=26.1 and p=3.33×10−7.
Where the evidence stops. The statistic conditions on success and assumes equally weighted conditions in its rate normalization. Three of fifteen pairs miss p<0.05, and no multiple-testing correction is reported. The heatmap alone does not prove a specific cause inside the model's representations.
7. Analysis & limitations
7.1 What the evidence leaves open
Filtering and difficulty labels depend on evaluated models, making difficulty partially endogenous to their success. Figure 6 misprints the layout count; Figure 7 conflicts with Table 7 on three totals. Easy-task filtering, surviving L1 cases and the inconsistent language-component naming need clarification. e10e11
Comparing different architectures and training histories does not isolate a general causal advantage. Selected failure rollouts support concrete error examples, not their prevalence. There is no physical-robot validation of these robustness conclusions in the supplied experiments. e02e14e17
The paired statistic conditions on success and uses equally weighted rates. Table 9 reports no multiplicity adjustment; three pairs miss its nominal threshold. Negative gaps do not directly reveal the internal representation mechanism proposed by the authors. e07e08e09
Tables 2 and 10 contain several 0.1-point disagreements, and Table 2's bold robot entry is not the maximum. Figure 2's lighting values differ from Table 1. The report retains each experiment's own numbers without combining them or inventing uncertainty. e03e04e15
7.2 Questions for discussion
- Would instruction dependence persist if the same scene supported several equally trained target choices? (e05, e06)
- Would the post-training gains survive a larger camera separation and a new set of difficulty-defining policies? (e10, e12)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires exact task definitions, filtering membership, camera inputs, checkpoint variants and evaluation settings. Appendix C leaves 'large majority' filtering unspecified. Table 6's OFT_m address names a spatial checkpoint despite its mixed-training description, so the exact weight identity needs confirmation. e10e14
Preserve split separation for distractors, textures, camera angles and noise; retain success filtering and RLDS formatting. Exact retained trajectory count, no-op threshold, warmup duration, seeds, software versions and complete post-training module configuration remain unspecified in this recipe. These gaps prevent claiming exact reproducibility from the PDF alone. e12e13e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Balance target exposure before testing language dependence
Reader-proposed experiment, not performed here: choose scenes containing two valid target objects and provide equal training demonstrations for each target. Hold scene, initialization and rollout horizon fixed while testing original wording, a paraphrase, an empty instruction and a switched-target instruction. Score both requested-goal success and selection of the formerly preferred object. Compare against the same policy trained with the original target imbalance. Persistent original-target selection after balanced exposure would support a grounding failure; recovery specifically under balanced exposure would weaken the claim that language is intrinsically ignored. e05e06e12
Check 2: Recheck pairwise gaps with matched trials and post-training
Reader-proposed experiment, not performed here: evaluate official OFT and the augmented model on identical camera–background and camera–robot task seeds, using 2,000 trials in each of the four perturbation conditions and recording the unperturbed cell explicitly. Compute raw success rates and Equation (8)'s conditional gap separately, bootstrap by task seed, and adjust significance across the tested pairs. Control camera separation and simulator settings across policies. If post-training raises single-factor success but leaves the paired gap negative and unchanged, it improves coverage without resolving the measured interaction; a reproducible gap reduction would support stronger compositional robustness. e07e08e09e12e15
8.3 Reading coverage
Visual audit: All eight text chunks were read individually, including the complete references and Appendices A–H. The listed pages were rendered and visually inspected; they cover every retained method, numerical, training and evaluation claim. All Figures 1–19 and Tables 1–10 were visually read, including the complete Table 10 continuation. Six final original crops were separately viewed, retaining labels, legends, axes and table comparison rows. Figure 6 is a composition chart with no computational arrows. Figure 3's shading, Figure 4's triangle direction and Equations (1)–(8), and both tables' improvement arrows were checked against captions and text. Conflicting counts, suite names, claims and table values remain disclosed. Page 26's damaged text labels were checked in the PDF; no meaning was inferred from replacement characters. Reference-only pages 11–12 were read as text but not rendered. Linked resources and separate supplements were outside the visual pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, byline and abstract
- Sections 1–2: motivation, perturbations, models and initial evaluation
- Sections 3–4: visual and language controls
- Section 5: compositionality definition and experiments
- Section 6: benchmark and post-training
- Sections 7–8 and References
- Appendix A: perturbation specifications
- Appendix B: evaluated model descriptions
- Appendix C: filtering, composition and difficulty
- Appendix D: training data and configuration
- Appendix E: goal replacement
- Appendix F: pairwise results and significance tests
- Appendix G: failure cases
- Appendix H: detailed results
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Some PDF font mappings yielded unpaired Unicode surrogates; replacement characters are recorded by page in extractionNotes and require PDF visual verification.
- Version scope: this reading verifies the supplied 32-page arXiv:2510.13626v3 PDF dated 26 December 2025, not the original October submission. Its exact title and all 13 authors match the catalog. The manifest identifies the previous HTML selection as the same revision; earlier versions were not compared.
- Text extraction does not reconstruct figure images; the retained PDF was visually inspected for every figure and table, including equation layouts.
- Separate supplemental material availability has not been fully verified.
- Some PDF font mappings yielded unpaired Unicode surrogates on page 26. The affected Figure 13 labels were visually inspected; replacement characters were not interpreted as mathematics.
- Code, checkpoints, linked pages and datasets were not opened. No experiments were reproduced. References were read as supplied bibliographic text, without checking the cited works.
- Acquisition provenance identifies a recovered, unchanged cached PDF; the earlier HTTP-request time was not independently retained. The supplied accessedAt value is preserved.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, byline, affiliations and arXiv margin
The observed title and all 13 authors match the catalog. The PDF identifies arXiv:2510.13626v3, 26 December 2025, and lists Fudan University, Tongji University, Shanghai Innovation Institute and National University of Singapore.
Go to primary source ↓e02PDF pp. 1–2, Introduction and Sections 2.1–2.2
The study evaluates executed manipulation tasks in LIBERO under seven perturbation dimensions: object layout, camera viewpoints, robot initial states, language, lighting, background and sensor noise. Ten model configurations span multiple existing policy families.
Go to primary source ↓e03PDF p. 3, Table 1, OpenVLA-OFT, OpenVLA-OFT_w and π0 rows
OpenVLA-OFT scores 97.1% originally, 59.7% with camera perturbations, 37.2% with robot-state perturbations and 81.5% with language perturbations. Its third-view-only variant scores 95.3%, 16.8%, 43.7% and 73.2%, respectively. π0 falls from 94.2% to 6.6% under robot-state perturbations. Downward-arrow rows denote absolute drops.
Go to primary source ↓e04PDF pp. 4–5, Section 3, Figures 1–2 and Finding 6
Figure 1 separates confounding objects from target displacement. Figure 2 reports third-view masking success of 43.6%, 43.0% and 67.3% for OpenVLA-OFT, π0 and π0-Fast; masking all views gives 0.0%, 2.0% and 0.0%. Its Light values are 86.5%, 79.3% and 73.1%, differing from Table 1. The shaded bands are not defined in the caption or accompanying discussion.
Go to primary source ↓e05PDF p. 5, Figure 3(a–b) and Section 4.1
Light bars retain instructions; dark bars remove them in panel (a) and replace goals and instructions in panel (b). OpenVLA-OFT retains strong Object-suite performance without language, but visibly loses performance on Spatial and especially Goal as well as Long. This conflicts with Section 4.1's statement that significant degradation is observed only on Long. Exact bar values and uncertainty are not printed.
Go to primary source ↓e06PDF p. 6, Section 4.2; pp. 20–22, Appendix E and Figure 10(a–e)
Goal replacement changes both the instruction's target and the success goal while retaining the scene. Section 4.2 calls the suite layout, whereas Appendix E calls it object and Figure 3 labels Object-Var. Figure 10(a) shows alphabet soup being selected after the requested target changes to butter. Appendix E also refers to Figure 5(b), whereas the quantitative target-replacement panel is Figure 3(b).
Go to primary source ↓e07PDF pp. 6–7, Section 5.1, Equations (1)–(8)
D_i and D_j indicate perturbation presence and Y indicates task success. Conditional success rates s are normalized across the four perturbation combinations to define p given success. The compositionality gap is Cov(D_i,D_j | Y=1), equal to the joint conditional probability minus the product of the two conditional marginals.
Go to primary source ↓e08PDF p. 7, Section 5.2 and Figure 4
OpenVLA-OFT is evaluated with 2,000 trials for single and pairwise perturbations across six non-language dimensions. The heatmap's upper triangle shows products of success-conditioned marginals; the lower triangle shows success-conditioned joint probabilities and parenthesized gaps. Camera–background reads 15.19 above and 13.57 below, with −1.62 printed as the gap. Every printed gap is negative.
Go to primary source ↓e09PDF pp. 23–24, Appendix F, Tables 8–9 and chi-square formulation
Table 8 reports camera-only 57.30%, robot-only 39.10%, camera–robot 19.05%, background-only 85.75% and background–camera 37.70%. Table 9 reports background–camera χ²=26.1, p=3.33e−07 and camera–robot p=9.31e−03. Twelve of fifteen listed p-values are below 0.05; layout–light, background–light and light–robot are not. No multiple-comparison adjustment is described, and Table 8 does not explicitly list the unperturbed cell.
Go to primary source ↓e10PDF p. 8, Section 6.1 and Figure 6; p. 18, Appendix C.1–C.3 and Table 7
The benchmark starts with 40 LIBERO evaluation tasks, generates 14,000 candidates and retains 10,030 test-only instances after model-based filtering and balancing. Table 7 totals are camera 1,599, robot 1,550, language 1,537, light 1,142, background 1,076, noise 1,601 and layout 1,525. Difficulty is defined by success counts among OpenVLA-OFT, π0, π0-fast and UniVLA: four successes give L1, decreasing to none at L5.
Go to primary source ↓e11PDF p. 8, Figures 5–6; pp. 18–20, Appendix C.1–C.3, Table 7, Figures 7–8; pp. 13–15, Appendix A
Figure 6 prints Layout 1599 although its outer counts 829 and 696 sum to 1525, agreeing with Table 7. Figure 7 totals camera 1758, robot 1671 and noise 1716 differ from Table 7. Figure 6 includes four language slices, including Original, whereas Appendix A.6 enumerates three rewrite categories. Figures 5 and 8 describe increasing intensity, while C.3 defines levels by model success. The relationship between easy-task filtering and retained L1 tasks is not fully specified.
Go to primary source ↓e12PDF pp. 19–20, Appendix D.1–D.3
The training set covers six variant types, omitting robot-state augmentation. Object augmentation uses distractors but excludes target-pose changes because automatic trajectories are unreliable. Original LIBERO state-action pairs are executed in modified environments; only successes remain and no-ops are filtered. Collection yields 22,400 trajectories, of which over 20,000 remain. Distractors and textures avoid test overlap; camera angles differ from test angles by 5 degrees, and noise parameters differ.
Go to primary source ↓e13PDF p. 20, Appendix D.4–D.5
Mixed fine-tuning starts from official OpenVLA-OFT weights and uses eight A100 GPUs, learning rate 5×10−4, 100,000 steps, batch size two per GPU (effective 16), AdamW with weight decay 0.1, cosine scheduling with warmup, and RLDS trajectories. Warmup length, random seeds and frozen-module choices are not specified here.
Go to primary source ↓e14PDF pp. 15–16, Appendix B.2, base architecture and OpenVLA-OFT variants; p. 17, Appendix B.5
The paper describes OpenVLA's fused visual features, projection and language backbone; OFT uses parallel continuous-action prediction through an MLP, an L1 regression objective and FiLM. OFT_w removes wrist input and is separately trained for 150K steps on eight A100 GPUs; OFT_m uses mixed training across four suites. WorldVLA's joint action/world modeling is a description of an evaluated baseline, not a proposed LIBERO-Plus architecture.
Go to primary source ↓e15PDF pp. 8–9, Section 6.2 and Table 2; pp. 31–32, Table 10
Table 2 and Table 10 report Ours total 79.5%, versus OFT_m 67.9% and OFT 69.6%. Camera is 92.8% versus OFT_m 55.6% and π0-Fast 65.1%; robot is 30.3% versus UniVLA 46.2%. Section 6.2 instead says 79.6% and calls the 37.2-point camera gain a gain over the next best model, though it is over OFT_m. Table 2 bolds Ours robot 30.3 despite higher alternatives. Minor Table 2/Table 10 discrepancies include OFT_w robot 38.7/38.6, π0 layout 68.9/68.8 and RIPT-VLA language 77.6/77.5 and light 88.4/88.3.
Go to primary source ↓e16PDF pp. 13–15, Appendix A.1–A.7, Tables 4–5
Layout changes add distractors or perturb target pose while preserving required semantic relations. Background and lighting changes edit scene definitions. Camera changes vary distance, spherical position and orientation; initialization changes joint positions. Language has distraction, commonsense and reasoning rewrites. Noise types are motion, Gaussian, zoom and glass blur plus fog; Table 5 supplies parameters but not a complete implementation specification.
Go to primary source ↓e17PDF pp. 24, 28–30, Appendix G and Figures 17–19
Qualitative simulation rollouts illustrate localization, instruction-target, shadow, image-corruption, initial-state and distractor failures. Figure 19 includes a distractor collision and a failure following target displacement. These are selected examples without a population-frequency estimate.
Go to primary source ↓e18PDF pp. 25–27, Figures 11–16, especially Figure 13 labels
Original rendered scenes show background, camera, joint-state, lighting, noise and distractor changes. Figure 13's visually legible delta-qpos labels replace the unreadable symbols in text extraction. Figure 12's caption incorrectly calls its camera-transformation examples background perturbations; the parameter labels and images depict viewpoint changes.
Go to primary source ↓8.5 Primary sources
LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models ↗
PDF · 12,433 extracted words
Source fingerprint
661de08e0ed1bb4bb4774177cac9fafa94f06b5385fa2f4107a1927b9a4aada7