RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
1. Paper overview
In one sentence: RT-2 turns language generation into direct robot control, gaining semantic generalization while remaining limited by demonstrated motions and costly inference. e01-identitye02-overviewe04-actionse09-generalization-resultse18-dynamics
| At a glance | What to know |
|---|---|
| Research problem | Source description Web-trained vision-language models understand images and instructions, while robots need grounded end-effector commands. RT-2 asks whether the same learned model can connect these capabilities directly, using demonstrations to associate its output vocabulary with physical actions. e02-overviewe04-actions |
| Core mechanism | Source description A shared language/action token interface adapts existing PaLI-X and PaLM-E backbones without introducing action-only model layers. Co-fine-tuning retains web tasks while learning robotic behavior. e03-sharinge05-tokens |
| A key reported result | Generalization to unseen objects, backgrounds and environments: PaLI-X-55B: 62%; PaLM-E-12B: 62%. Task success, reported unseen average (%). Physical mobile manipulator; easy/hard conditions across three shift categories; over 280 tasks, mostly picking/placing. All baselines share robot data. RT-1: 32%; MOO: 35%; R3M: 12%; VC-1: 10%. Table 4 supports substantial transfer: 30 percentage points above RT-1. Table 6 instead prints 63 for the same PaLI-X component scores; this discrepancy is unresolved. PaLM-E pretraining included related robot images without evaluated low-level actions. e07-data-baselinese08-generalization-protocole09-generalization-resultse12-ablatione21-pretraining-caveat |
| Reading caution | Source description Semantic recognition does not guarantee physical competence: pushing a marker or banana can fail after correct object approach. The source also reports weak part-specific grasps, tool use, dexterity and extended reasoning. e18-dynamicse14-limitations |
Core contributions
- Source description
A shared language/action token interface adapts existing PaLI-X and PaLM-E backbones without introducing action-only model layers. Co-fine-tuning retains web tasks while learning robotic behavior. e03-sharinge05-tokens
- Author claim
The authors demonstrate semantic transfer through physical-robot generalization and novel-instruction evaluations, plus a separate simulation benchmark. Their plan-before-action extension provides qualitative evidence of combining language planning and low-level control. e09-generalization-resultse11-emergent-resultse19-language-tablee13-planning
Figure 1. A shared token interface connects web supervision to robot commands. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the diagram from left to right. Web question-answer examples and robot demonstrations enter co-fine-tuning; their answers share a token-based format. In the central block, image input passes through the ViT and joins the task prompt in the language model. Follow the purple output path down to the token sequence, then right through De-Tokenize to the robot action. The gray deployment arrow leads to physical examples. The diagram establishes an information path, while Section 3.2 supplies the executable details: action-token vocabulary restriction, fixed field ordering and discretized values. Closed-loop control is stated in the caption; the drawing does not spell out every feedback operation. e02-overviewe03-sharinge04-actionse05-tokens
What it supports. The policy generates commands through the pretrained model’s output interface, allowing web and robot supervision to share model weights. This supports the One Model VLA classification. It does not depict a learned future-state simulator or an inverse-dynamics stage between a predicted future and the executed action.
Where the evidence stops. Figure 1 shows six displacement tokens, omitting termination and gripper extension. Section 3.2 specifies eight fields but prints a seven-token example. Use the field specification; the source leaves these illustrative omissions unresolved.
2. Motivation
2.1 The problem and the proposed response
Web-trained vision-language models understand images and instructions, while robots need grounded end-effector commands. RT-2 asks whether the same learned model can connect these capabilities directly, using demonstrations to associate its output vocabulary with physical actions. e02-overviewe04-actions
2.2 What this reading follows
RT-2 asks whether the knowledge inside a web-trained vision-language model can influence the robot’s next physical command. Its answer is to encode actions as ordinary output tokens and train them alongside language tasks. The resulting policy can apply familiar motions to unfamiliar objects, symbols and instructions. Reading this paper requires separating that semantic transfer from mastery of new dynamics: selecting the right object can still end in a failed push. This edition follows the supplied arXiv v1, preserves its reporting inconsistencies, and connects the architecture to physical-robot results, a separate simulation benchmark and revealing failure cases. e01-identitye02-overviewe04-actionse09-generalization-resultse18-dynamics
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Autoregressive VLA and One Model labels are supported by shared language/action weights and direct autoregressive action generation. A ViT plus language backbone remains one policy architecture here. RT-2 does not predict future world states or extract actions through inverse dynamics, so the world/action prediction quadrant remains not applicable; Language-Table auxiliary targets do not change that assessment. e03-sharinge04-actionse20-language-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 An action string becomes a physical command
Begin with the output interface. A conventional vision-language model answers an image-and-text request with text; RT-2 trains the same type of model to answer a robot request with a serialized action. The robot target includes termination, three translational displacements, three rotational displacements and gripper extension. Quantization assigns continuous values to bins, and each bin is associated with an existing vocabulary token. This makes demonstration learning compatible with next-token prediction. At deployment, a vocabulary constraint allows action tokens, and de-tokenization converts the generated sequence into commands. The robot executes them and provides another observation for closed-loop control. The paper’s fixed field specification is the reliable reference: Figure 1 simplifies the action to six components, while the numeric example in Section 3.2 has only seven tokens. e02-overviewe04-actionse05-tokense16-training
5.2 Why keep web examples during robot fine-tuning?
The training mixture is the second mechanism to follow. Pretraining provides visual and linguistic concepts, while demonstrations connect observations and instructions to executable movements. Co-fine-tuning continues both kinds of supervision, using approximately equal robot and web proportions for PaLI-X and a larger robot share for PaLM-E. The authors interpret the retained web stream as protection against forgetting pretrained concepts. Table 6 supports an average benefit, but its details matter: the gain at 5B is small, and the background columns actually worsen. Reader interpretation: retaining semantic supervision is a plausible explanation, not a directly measured internal mechanism. A careful comparison should match robot-example exposure as well as starting checkpoint, and should report each shift category instead of relying only on the final average. e05-tokense15-mixturese12-ablatione16-training
5.3 Separate a sensible plan from a successful interaction
The novel-instruction evaluations ask the robot to reuse familiar manipulation skills in semantically unfamiliar settings. Their symbol, reasoning and person-recognition scores measure completed robot tasks, so they go beyond a textual answer alone. Still, the failure sequences show that understanding the requested object does not ensure successful contact dynamics. The plan-augmented PaLM-E variant adds another distinction: it produces a natural-language plan before action tokens after a few hundred extra training steps. Its examples include selecting a rock in response to a hammer-related request and choosing a drink for sleepiness. Reader interpretation: these examples illustrate useful semantic selection, but they do not demonstrate a new hammering skill or establish a controlled quantitative advantage for plan generation. Judge semantic choice, action execution and newly acquired motor behavior separately. e10-emergent-protocole11-emergent-resultse13-planninge14-limitationse18-dynamics
5.4 Training and inference
During training
Next-token prediction acts as behavior cloning on demonstrations. Robot data comes from 13 robots over 17 months; it occupies about 50% of the PaLI-X mixture and 66% of PaLM-E. Web tasks include VQA and captioning; PaLI-X co-fine-tuning excludes Episodic WebLI. e07-data-baselinese15-mixturese16-training
PaLI-X-55B uses learning rate 0.001, batch 2048 and 80K steps; 5B uses the same rate/batch for 270K steps. PaLM-E-12B uses 0.0004, batch 512 and 1M steps. The paper does not give a module-by-module freeze policy for RT-2. e16-training
Language-Table adds instruction, arm-position, frame-separation and success prediction to action prediction. Its PaLI-3B-labelled model uses learning rate 0.001, batch 128 and 300K steps. These auxiliary tasks are training targets; inference-time planning with them is not described. e20-language-traininge19-language-tablee16-training
During inference
A multi-TPU cloud service is queried over the network. Reported control rates are 1–3 Hz for PaLI-X-55B and around 5 Hz for 5B. The Language-Table policy also runs around 5 Hz, emitting two discretized Cartesian increments. e06-servinge19-language-table
The chain-of-thought variant receives a few hundred further gradient steps on Plan-then-Action examples. It can describe a selected action before outputting control tokens; the examples do not establish a quantitative benefit over the ordinary policy. e13-planning
5.5 Implementation flow
- Encode the scene and request
Image embeddings and the task prompt enter the pretrained multimodal backbone. PaLI-X uses an encoder-decoder language model; PaLM-E uses a decoder-only model. Visual/text representations support action generation without a separate learned action head. e03-sharinge05-tokens
- Make actions a vocabulary
Discretize each continuous action dimension into 256 uniform bins. PaLI-X uses existing integer tokens; PaLM-E overwrites its 256 least frequent tokens. The target is a space-separated sequence in a fixed field order, trained like a textual answer. e04-actionse05-tokens
- Decode, execute and observe again
On robot prompts, restrict decoding to valid action tokens, de-tokenize them into commands, execute through the robot control interface and repeat with visual feedback. Prediction supplies commands; physical movement is performed by the robot. e02-overviewe05-tokense06-serving
6. Experiments & results
RT-2 makes a pretrained vision-language model a robot policy by encoding low-level actions as text tokens and co-fine-tuning on demonstrations and web tasks. Its clearest gain is transferring visual and semantic knowledge to unfamiliar objects and instructions. Physical execution improves, but unfamiliar dynamics and new motor skills remain major boundaries.
6.1 Read the original evidence
Table 4. Transfer gains are much larger under distribution shift than on seen tasks. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with Seen Tasks, where RT-1 and both RT-2 variants are close. Then move through the six easy/hard columns before consulting Unseen Average. Each pair changes a different part of the evaluation scene: objects, backgrounds or environments. Section 4.1 and Table 2 explain that these conditions have different instructions and difficulty, with novel objects also appearing in hard-background tests. They are not interchangeable replications of one task. Compare methods within a column, then read the rightmost aggregate as the paper reports it. All baselines use the same robot data, but their pretrained components and architectures differ. e07-data-baselinese08-generalization-protocole09-generalization-resultse12-ablatione21-pretraining-caveat
What it supports. Both RT-2 variants report 62% unseen-average success, compared with 32% for RT-1 and 35% for MOO. Seen-task scores are 91% and 93%, around RT-1’s 92%. The contrast supports generalization as the main empirical gain, while leaving considerable room for failure in unfamiliar scenes.
Where the evidence stops. The superscript’s p. 24 footnote says PaLM-E pretraining included related robot images for high-level VQA, without evaluated low-level actions. Table 6 prints 63 for the repeated PaLI-X row where this table prints 62. No uncertainty is supplied.
Table 5. Semantic transfer is substantial but uneven across reasoning categories. Original paper, p. 24 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the grouped headers to avoid mixing subtask scores with averages. Symbol Understanding has three symbol sets; Reasoning includes Math, Logos, Nutrition and Color/Multilingual; Person Recognition includes Celebrities and CelebA. Each group has its own Average, followed by the overall Average at the far right. Table 3 gives the actual instructions, and the illustrated scenarios show object movement toward symbols or images. The evaluation runs the four models consecutively under matched conditions, with five trials per instruction. These are robot-task outcomes, rather than standalone visual question-answering scores, and the model names distinguish different pretrained backbones and sizes. e10-emergent-protocole11-emergent-resultse14-limitations
What it supports. PaLI-X-55B reaches 60% overall versus RT-1’s 17%, with particularly strong symbol understanding at 82%. Its reasoning average is 46%, and its math score is only 25%. PaLM-E scores 35% on math despite a lower 40% overall average, showing that the ranking depends on the semantic category.
Where the evidence stops. Five trials per instruction and no reported uncertainty limit precision. The compared backbones differ in pretraining and size, so category differences do not isolate either factor. Novel semantic instructions also do not imply newly learned motor skills.
Table 1. A separate simulation benchmark tests the action-token recipe on planar control. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This compact table belongs to the Language-Table experiment, not the mobile-manipulator evaluation. Read the rows as the reported simulation comparison and retain each plus/minus value exactly as printed. The surrounding method describes a two-coordinate action string for planar end-effector increments and an inference rate around 5 Hz. Appendix B adds that this variant is co-fine-tuned on several auxiliary prediction tasks, including instruction and success prediction. The adjacent Figure 5 shows separate qualitative real-world behavior using the same checkpoint. Neither those images nor the table turn simulation performance into a measured real-world success rate. e19-language-tablee20-language-traininge16-training
What it supports. The RT-2-PaLI-3B-labelled model has the highest reported central value, 90 ± 10, ahead of LAVA’s 77 ± 4. This extends the empirical comparison to another robot setting and action format, but combines VLM pretraining, a different model and auxiliary training tasks.
Where the evidence stops. The paper does not explicitly define table units, the ± statistic or replication count. Appendix D also leaves the PaLI-3B label inconsistent with its stated component sizes. Preserve the printed label and values without inventing a resolution.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Generalization to unseen objects, backgrounds and environments Physical mobile manipulator; easy/hard conditions across three shift categories; over 280 tasks, mostly picking/placing. All baselines share robot data. | PaLI-X-55B: 62%; PaLM-E-12B: 62%. Task success, reported unseen average (%) | RT-1: 32%; MOO: 35%; R3M: 12%; VC-1: 10%. Table 4 supports substantial transfer: 30 percentage points above RT-1. Table 6 instead prints 63 for the same PaLI-X component scores; this discrepancy is unresolved. PaLM-E pretraining included related robot images without evaluated low-level actions. e07-data-baselinese08-generalization-protocole09-generalization-resultse12-ablatione21-pretraining-caveat |
| Seen-task manipulation Over 200 seen instructions, with varying object placement, time of day and robot position. | PaLI-X-55B: 91%; PaLM-E-12B: 93%. Task success (%) | RT-1: 92%; MOO: 75%. Similar success to RT-1; the principal advantage is under distribution shift. Table 4 supplies no uncertainty. e08-generalization-protocole09-generalization-results |
| Emergent semantic instruction following Matched consecutive physical trials; five runs per instruction across symbol understanding, reasoning and person recognition. | PaLI-X-55B: 60%; PaLM-E-12B: 40%. Reported average task success (%) | RT-1: 17%; VC-1: 11%. PaLI-X exceeds three times RT-1’s average. Its 82% symbol score exceeds its 46% reasoning score; math remains 25%. These tasks test semantic deployment of existing motions. e10-emergent-protocole11-emergent-resultse14-limitations |
| Model-size and co-fine-tuning ablation PaLI-X on unseen evaluations; seen tasks excluded; no 55B-from-scratch evaluation. | 5B: scratch 9%, fine-tuning 42%, co-fine-tuning 44%; 55B: fine-tuning 52%, co-fine-tuning 63%. Reported average task success (%) | Co-fine-tuning gains 2 percentage points at 5B and 11 at 55B. Average improvements are not uniform: 5B background scores deteriorate. Different training durations across sizes and absent uncertainty limit causal conclusions about scaling. e12-ablatione16-training |
| Simulated Language-Table instruction following Separate Language-Table simulation benchmark using the PaLI-3B-labelled variant and auxiliary prediction tasks. | 90 ± 10. Reported benchmark performance (table units and ± statistic not explicitly defined) | LAVA: 77 ± 4; RT-1: 74 ± 13; BC-Zero: 72 ± 3. Highest reported central value, but the uncertainty type and replication count are unspecified. This number does not quantify the separate qualitative real-world rollouts. e19-language-tablee20-language-training |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 6. Co-fine-tuning improves averages, with a much smaller gain at 5B. Original paper, p. 24 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this table in two passes. First hold size at 5B and compare the three training rows; this distinguishes initialization from the choice to retain web data. Next compare fine-tuning against co-fine-tuning at 55B. The scratch row is absent at 55B because the authors skipped it after the poor 5B result. Inspect the six condition columns as well as Average: co-fine-tuning helps some conditions while hurting others at 5B. Section 4.3 attributes the average benefit to retaining pretrained concepts, but the table measures task success rather than forgetting itself. Appendix E also reports different co-fine-tuning durations for the two sizes. e12-ablatione16-training
What it supports. The 5B averages are 9% from scratch, 42% with robot-only fine-tuning and 44% with co-fine-tuning. At 55B, fine-tuning reaches 52% and co-fine-tuning 63%. Yet 5B background scores fall from 79/50 to 67/29, so an average gain is not a universal improvement.
Where the evidence stops. Table 4 reports 62 for the same 55B co-fine-tuning component scores. No uncertainty resolves the discrepancy or the small 5B gain. The size comparison also changes training duration, limiting a pure parameter-count interpretation.
Figure 9. Correct semantic selection can still produce unsuccessful contact dynamics. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each strip from left to right as a separate physical rollout. The left instruction concerns a red marker and a game controller; the right concerns a banana and an apple. Track the pusher’s initial approach, subsequent contact and object motion, rather than treating the object names as evidence of task completion. Appendix G states that the model moves toward the correct object but fails to control dynamics unlike those of its training blocks. The marker rolls off the table, while the banana is contacted far from its center of mass. This visual diagnoses a failure after the semantic target has been identified. e18-dynamicse14-limitations
What it supports. These examples locate a concrete boundary of web knowledge transfer: recognizing what to manipulate does not ensure that the robot can manipulate it successfully. They motivate separate measurements of semantic selection, contact behavior and final completion, rather than interpreting an appropriate first movement as task success.
Where the evidence stops. The strips are qualitative examples, with no failure frequency or controlled dynamics intervention. The proposed benefit of more diverse interaction data is an author hypothesis; this figure does not demonstrate that such training fixes the failures.
7. Analysis & limitations
7.1 What the evidence leaves open
Semantic recognition does not guarantee physical competence: pushing a marker or banana can fail after correct object approach. The source also reports weak part-specific grasps, tool use, dexterity and extended reasoning. e18-dynamicse14-limitations
Large models incur high serving cost and constrain control frequency. Limited access to suitable pretrained VLMs also restricts replication. e06-servinge14-limitations
Unresolved reporting inconsistencies include eight action fields versus a seven-token example; seven named skills followed by eight entries; PaLI-3B’s label versus Appendix D’s component sizes; and 62 versus 63 for the repeated generalization row. None is silently corrected. e04-actionse15-mixturese20-language-traininge12-ablation
7.2 Questions for discussion
- How much of co-fine-tuning’s benefit survives when robot-example exposure and inference latency are matched?
- Can a policy preserve correct semantic selection while learning unfamiliar contact dynamics from a small added demonstration set?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs pretrained checkpoints, original web mixtures, instruction-labelled demonstrations, action-bin ranges and a compatible control/serving stack. Detailed mixtures, schedules and regularization are delegated to predecessor papers; this PDF does not supply a full implementation, exact RT-2 freeze configuration, TPU count/model or software versions. e04-actionse06-servinge15-mixturese16-training
Preserve Table 2’s six shift conditions and Table 3’s semantic groups. The paper gives 1–5 trials per unseen instruction and five per emergent instruction, but leaves exact unseen-task trial allocation and success-scoring implementation insufficiently specified for an exact numerical replication. e17-repeatse08-generalization-protocole10-emergent-protocol
Reader-proposed checks: compare robot-only fine-tuning with co-fine-tuning under matched robot exposure, and separately score object selection versus completed pushing on familiar blocks and unfamiliar dynamics. These isolate semantic retention and motor generalization. e12-ablatione15-mixturese18-dynamics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does co-fine-tuning help when robot exposure is held fixed?
Reader-proposed experiment: start matched PaLI-X-5B runs from the same pretrained checkpoint and compare robot-only fine-tuning with the reported roughly 50% robot co-fine-tuning mixture. Use one comparison with equal optimizer steps and another with equal numbers of robot examples, recording the extra compute in the latter. Hold action quantization, deployment rate and paired initial scenes fixed. Evaluate every Table 2 shift category and a held-out web-task set, with multiple training seeds and confidence intervals. If the small average gain disappears after exposure matching, the original ablation is insufficient to attribute it to semantic retention; consistent web-task retention plus robot gains would strengthen that interpretation. e12-ablatione15-mixturese16-traininge17-repeatse08-generalization-protocol
Check 2: Can correct object selection survive a change in contact dynamics?
Reader-proposed experiment: compare familiar training-style blocks with the marker and banana scenarios motivating Figure 9. Keep instructions, target locations, initial distances and control frequency fixed across paired trials. Score first-object selection, successful contact and final task completion separately, and record rolling or rotation failures. Then add a small dynamics-specific demonstration set while holding web supervision fixed. The hypothesis is that semantic selection can remain accurate while completion drops on unfamiliar dynamics; improved completion after added demonstrations without a comparable selection change would support a motor-data bottleneck. If selection fails first, these trials do not isolate dynamics. e18-dynamicse14-limitationse19-language-tablee20-language-training
8.3 Reading coverage
Visual audit: Inspected the title/author/version page, all ten numbered figures, all six tables, and every page supporting retained method, training, evaluation, numerical or reproduction details. Figure 1’s token arrows and de-tokenization direction agree with its caption; its six-component illustration omits termination/gripper fields specified in Section 3.2. The seven-token numeric example and Tables 4/6 average discrepancy are disclosed. Table 4’s superscript footnote appears on the following page, p. 24: that page was inspected and its qualification is preserved in the visual caution. All six final crops were viewed; Table 5 was recropped to retain every row. Pages 12–18 were read as text only. Separate supplements, external code and videos were not inspected; no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 19, 20, 21, 22, 23, 24, 25, 26. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Vision-Language-Action Models (3.1–3.3)
- 4. Experiments (4.1–4.4)
- 5. Limitations
- 6. Conclusions
- Acknowledgments
- References
- A. Contributions
- B. Datasets
- C. Baselines
- D. VLMs for RT-2
- E. Training Details
- F. Evaluation Details (F.1–F.2)
- G. Example Failure Cases
- H. Quantitative Experimental Results (H.1–H.3)
- I. Additional Chain-Of-Thought Reasoning Results
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected PDF is arXiv:2307.15818v1 (margin: 28 Jul 2023; header: 2023-8-1), not a verified copy of the catalogued CoRL proceedings edition. The exact title and author identities agree. The PDF orders authors alphabetically; the catalog uses a different order. Its shorter name forms are Krzysztof Choromanski, Avinava Dubey, Montse Gonzalez Arenas, Nikhil Joshi, Michael Ryoo, Pannag Sanketi and Huong Tran. Metadata preserves the inspected forms. Edition equivalence was not established.
- All seven supplied text chunks were read, covering all 26 pages. Text extraction does not reconstruct figure images; the retained PDF was used to inspect every numbered figure and table. PDF pages 12–18 were read as text, without page-image inspection.
- Separate supplemental material availability has not been fully verified. No separate supplements were supplied.
- Code, project-site videos, predecessor papers, training datasets and model checkpoints were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01-identityPDF p. 1, title, author block, date header and arXiv margin
The exact catalog title appears with 54 authors, alphabetically ordered from Anthony Brohan to Brianna Zitkovich, and Google DeepMind affiliation. The margin reads arXiv:2307.15818v1, 28 Jul 2023; the header date is 2023-8-1.
Go to primary source ↓e02-overviewPDF p. 2, Figure 1 and Introduction; p. 3, Introduction
Robot observations and instructions enter a vision-language model; output tokens are de-tokenized into actions for closed-loop control. Training combines web tasks and robot demonstrations without new model parameters. Semantic transfer reuses motions learned from robot data.
Go to primary source ↓e03-sharingPDF p. 4, Related Work, final pretraining paragraph; pp. 20–21, Appendix D
The unified language/action output shares model weights without action-only layers. PaLI-X uses image tokens with an encoder-decoder language backbone; PaLM-E uses a decoder-only language backbone and projected visual inputs. Output generation is autoregressive.
Go to primary source ↓e04-actionsPDF pp. 5–6, Section 3.2, action definition and unnumbered target string; p. 2, Figure 1
The action contains termination, three positional displacements, three rotational displacements and gripper extension. Continuous dimensions use 256 uniform bins. Figure 1 depicts only translation/rotation; the p. 6 numeric example contains seven integers despite the eight-field specification.
Go to primary source ↓e05-tokensPDF p. 6, Section 3.2, tokenization, Co-Fine-Tuning and Output Constraint
PaLI-X maps bins to existing integer tokens; PaLM-E repurposes its 256 least frequent tokens. Images and task descriptions form VQA-style prompts. Robot-action decoding restricts the output vocabulary; ordinary vision-language tasks retain the full vocabulary.
Go to primary source ↓e06-servingPDF p. 6, Section 3.3; p. 11, Section 5
A network-queried multi-TPU cloud service enables control at 1–3 Hz for PaLI-X-55B and around 5 Hz for 5B. High computational cost and higher-frequency control remain limitations; TPU count/model and latency distributions are not specified here.
Go to primary source ↓e07-data-baselinesPDF p. 7, Section 4, opening paragraphs and Baselines; p. 20, Appendix C
Approximately 6,000 evaluation trajectories use mostly a 7DoF mobile manipulator. Demonstrations were collected with 13 robots over 17 months. Baselines share robotic data: RT-1, VC-1, R3M and MOO. VC-1 and R3M encoders are unfrozen during baseline training.
Go to primary source ↓e08-generalization-protocolPDF pp. 7–8, Section 4.1 and Figures 3–4; p. 26, Table 2
Seen evaluation covers over 200 tasks with realistic layout variability. Over 280 generalization tasks cover objects, backgrounds and environments, each with easy/hard conditions; hard backgrounds also introduce novel objects. New environments include a kitchen sink and an office desk.
Go to primary source ↓e09-generalization-resultsPDF p. 23, Appendix H.1, Table 4, Seen Tasks and Unseen Average columns; p. 8, Figure 4
Table 4 reports seen/unseen-average success percentages: RT-1 92/32, MOO 75/35, PaLI-X-55B 91/62 and PaLM-E-12B 93/62. R3M and VC-1 unseen averages are 12 and 10. No uncertainty accompanies Table 4.
Go to primary source ↓e10-emergent-protocolPDF p. 9, Section 4.2; p. 21, Appendix F.2; p. 22, Figure 8 and Table 3
Four models are evaluated consecutively under the same conditions. Emergent tasks test symbols, reasoning and person recognition, using five runs per instruction. Reasoning includes math, logos, nutrition and color/multilingual instructions.
Go to primary source ↓e11-emergent-resultsPDF p. 24, Appendix H.2, Table 5; p. 10, Figure 6a
Overall success is 60% for PaLI-X-55B, 40% for PaLM-E-12B, 17% for RT-1 and 11% for VC-1. PaLI-X category averages are 82% symbols, 46% reasoning and 53% person recognition. Math is 25% for PaLI-X versus 35% for PaLM-E. No uncertainty is given.
Go to primary source ↓e12-ablationPDF p. 10, Section 4.3 and Figure 6b; p. 24, Appendix H.3, Table 6; p. 23, Table 4
Table 6 averages are 9/42/44 for 5B scratch/fine-tuning/co-fine-tuning and 52/63 for 55B fine-tuning/co-fine-tuning. The 55B scratch experiment was skipped. At 5B, background scores drop from 79/50 to 67/29 with co-fine-tuning. The 55B co-fine-tuning row repeats Table 4 component scores but reports average 63 instead of 62.
Go to primary source ↓e13-planningPDF p. 10, Section 4.4; p. 11, Figure 7; pp. 24–25, Appendix I and Figure 10
A PaLM-E variant receives a few hundred additional gradient steps on examples with a natural-language Plan followed by action tokens. Qualitative rollouts include selecting a rock as a potentially useful hammer and choosing a drink for sleepiness. No controlled quantitative chain-of-thought comparison is supplied.
Go to primary source ↓e14-limitationsPDF p. 11, Section 5; p. 23, Appendix G
Web pretraining does not confer new physical motions. Weaknesses include part-specific grasps, novel motions/tool use, dexterity and extended reasoning. Large models impose inference cost, and access to suitable VLMs is limited.
Go to primary source ↓e15-mixturesPDF pp. 19–20, Appendix B
WebLI is described as roughly 10B image-text pairs in 109 languages, filtered by cross-modal similarity to the top 10%, yielding 1B examples. RT-2-PaLI-X excludes Episodic WebLI during co-fine-tuning and uses about 50% robot data; PaLM-E uses about 66%. Detailed mixtures are referred to predecessor papers. The text says seven robot skills but enumerates eight.
Go to primary source ↓e16-trainingPDF p. 21, Appendix E; p. 7, Section 4, training paragraph
Training uses next-token prediction, corresponding to behavior cloning on robot examples. PaLI-X-55B: learning rate 0.001, batch 2048, 80K steps; 5B: same rate/batch, 270K steps. PaLM-E-12B: 0.0004, batch 512, 1M steps. Language-Table PaLI-3B: 0.001, batch 128, 300K steps. Other schedules and regularization are referred to prior papers; a module-level RT-2 freeze policy is not given.
Go to primary source ↓e17-repeatsPDF p. 21, Appendix F.2; p. 26, Table 2; p. 22, Table 3
Unseen-category instructions were run 1–5 times depending on evaluation-set size. Emergent instructions were each run five times. Tables list wording, but exact per-instruction allocations for unseen-category trials are not supplied.
Go to primary source ↓e18-dynamicsPDF p. 23, Appendix G, first paragraph and Figure 9
Real-world Language-Table failures show correct object approach followed by unsuccessful pushing: a marker rolls off the table; banana contact is far from its center of mass. The authors connect these failures to object dynamics unlike the training blocks; these are qualitative examples.
Go to primary source ↓e19-language-tablePDF pp. 8–9, Section 4.1, Language-Table paragraph, Figure 5 and Table 1
The simulated Language-Table table reports RT-2-PaLI-3B 90 ± 10, LAVA 77 ± 4, RT-1 74 ± 13 and BC-Zero 72 ± 3. Units, uncertainty statistic and replication count are not explicitly defined. Two action coordinates range from -10 to +10; inference runs at about 5 Hz. Figure 5 separately depicts qualitative real-world behavior.
Go to primary source ↓e20-language-trainingPDF p. 20, Appendix B, Language-Table paragraph; p. 21, Appendices D–E
Language-Table training includes action prediction from two consecutive frames and an instruction, plus instruction, arm-position, frame-separation and success prediction. Appendix D labels the model PaLI-3B while describing a 2B visual encoder and UL2-3B backbone; the component count and model label are not reconciled.
Go to primary source ↓e21-pretraining-caveatPDF p. 23, Table 4, PaLM-E superscript 1; p. 24, footnote 1
PaLM-E pretraining included robot images for high-level VQA planning that can resemble generalization scenarios. Those examples did not include the low-level actions evaluated here.
Go to primary source ↓8.5 Primary sources
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control ↗
PDF · 12,762 extracted words
Source fingerprint
0a62dc36bbbfdea232ad45a0ab75c82e9cf535b55333ff22c02d5da1897ccac3