PaLM-E: An Embodied Multimodal Language Model
1. Paper overview
In one sentence: PaLM-E learns to turn continuous observations into language-conditioned plans, gaining from broad multimodal training while relying on separate robot controllers and facing a tradeoff between adaptation and language retention. e-identitye-groundinge-tokenizatione-controle-ablatione-qualitativee-retention
| At a glance | What to know |
|---|---|
| Research problem | Source description Text-only planners cannot directly inspect the geometry that makes a robot plan feasible. PaLM-E asks whether continuous observations can ground a pretrained language model while retaining useful language and vision capabilities, and whether abundant nonrobot data can compensate for scarce embodied supervision. e-identitye-groundinge-transfer |
| Core mechanism | Source description The model inserts continuous observation embeddings at flexible positions within a language sequence and learns textual completions through the existing decoder. e-tokenization |
| A key reported result | Low-data TAMP grasping and stacking: 94.9 with finetuning and full mixture. Mean planning success (%). PaLM-E-12B; 1% TAMP data, 320 examples per planning task; mean of p1 grasping and p2 stacking. 48.6 finetuned/single robot; 74.3 frozen/full mixture; 31.8 frozen/single robot; 42.9 without pretraining. Both mixture and LLM adaptation matter in this setting; the figure gives no uncertainty intervals. e-ablatione-tamp-tasks |
| Reading caution | Reader analysis Physical mobile manipulation and novel-object tabletop examples are qualitative demonstrations. They do not establish population success rates or systematic robustness to disturbances. Figure 2's reasoning examples likewise do not provide a measured generalization benchmark. e-qualitativee-groundinge-mobile-loop |
Core contributions
- Source description
The model inserts continuous observation embeddings at flexible positions within a language sequence and learns textual completions through the existing decoder. e-tokenization
- Source description
The authors compare global image features, object-centric encodings and explicit entity references, alongside frozen versus finetuned language models. e-encoderse-traininge-osrte-ood
- Author claim
Experiments connect broad multimodal training to data-efficient robot planning, while larger models retain more language performance and perform strongly on knowledge-based VQA. e-ablatione-language-tablee-vqae-retention
Figure 1. Continuous observations enter the language model as embeddings; generated text then connects to control. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the center rather than the surrounding task examples. Orange arrows carry text into the prefix; the green encoder and blue ViT produce continuous observation vectors at other positions in that sequence. Equation (3) specifies this interface: each prefix vector comes from either the word embedder or an observation encoder. Follow the generated answer downward, then follow the purple arrow left into Control. This direction matters: PaLM produces a textual instruction, and the controller consumes it. The surrounding panels show possible uses of that shared interface, including mobile manipulation, block planning, captioning and language-only questions. e-groundinge-tokenizatione-controle-encoderse-training
What it supports. The architectural contribution is a common embedding interface through which a language decoder can condition on sensor information. The same output format supports both answers and high-level plans. The diagram is consistent with Section 3's separation between language generation and the low-level policies that execute robot skills.
Where the evidence stops. The green question-mark block denotes an unspecified observation encoder in this overview; it should not be read as a particular learned dynamics module. The diagram also does not distinguish frozen from finetuned training variants.
2. Motivation
2.1 The problem and the proposed response
Text-only planners cannot directly inspect the geometry that makes a robot plan feasible. PaLM-E asks whether continuous observations can ground a pretrained language model while retaining useful language and vision capabilities, and whether abundant nonrobot data can compensate for scarce embodied supervision. e-identitye-groundinge-transfer
2.2 What this reading follows
A robot asked to retrieve a blocked object needs more than the names of things in the scene: it needs an instruction grounded in their arrangement. PaLM-E places sensor-derived vectors directly beside word embeddings and trains a language model to complete the resulting sequence. Its output can answer a question or name a robot skill. This reading follows the connection from observation embeddings to executed instructions, then separates the evidence for transfer, embodied perception and retained language ability. The supplied source is the March 2023 arXiv v1; its simulated scores and selected physical demonstrations establish different kinds of evidence. e-identitye-groundinge-tokenizatione-controle-ablatione-qualitativee-retention
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/backbone and planning classification is supported. PaLM-E unifies perception-conditioned text generation, but execution remains in separate controllers. Its target is text, not joint future-state/action prediction or inverse dynamics. Thus 'Not applicable' for the world-action architecture, prediction paradigm and quadrant is appropriate; the paper's 'single model' wording alone does not establish a One Model world-action architecture. e-tokenizatione-controle-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make a sensor observation part of the sentence
The key interface lies before the language decoder. A normal word token is mapped by the word embedder into a vector; a continuous observation is mapped by an encoder into several vectors of the same language-space dimension. PaLM-E interleaves these vectors, allowing a question to surround one or more observations. The decoder then predicts an ordinary textual continuation, and training loss applies to the non-prefix text. This means the encoder must make its output useful for the language task rather than deliver a manually written scene description. Freezing the LLM still permits training the input interface through it. The output remains text: answering what is visible completes a VQA task, while naming a skill starts a separate execution process. e-tokenizatione-traininge-osrte-control
5.2 Give the plan stable ways to refer to objects
The representation study addresses a difficulty hidden by a generic image-to-text diagram: a plan must refer to the correct physical object. Color can identify objects in some scenes, but it is insufficient when similar objects coexist. PaLM-E therefore experiments with prompts that associate object labels with their continuous embeddings, and assumes the controller understands the same labels. OSRT supplies object slots learned through novel-view synthesis, whereas the object-centric ViT alternative uses supplied instance masks. Table 1 favors OSRT in the limited-data setting, and Table 7 distinguishes larger scenes from a separate shift to color-based references. Reader interpretation: successful token binding is part of the planning interface, so object-count generalization and language-reference generalization should be tested separately. e-encoderse-low-datae-oode-tamp-tasks
5.3 Locate the boundary between a better answer and a better robot
PaLM-E is trained with text supervision, but its evaluations reach different distances into the control loop. TAMP tests the generated plan against geometric constraints. Language-Table Tasks 2 and 3 run a low-level controller in simulation, while Task 1 measures validation accuracy. Mobile affordance and failure detection evaluate image-and-skill questions; the kitchen demonstrations then illustrate full physical sequences qualitatively. Reader analysis: these layers should remain separate when assessing transfer. A better F1 score may help a robot, but it does not measure accumulated execution errors, and a successful illustrated recovery does not estimate its probability. The strongest next experiment would hold the controller fixed and test whether updated observations improve recovery over matched stale-observation prompts. e-transfere-tamp-taskse-language-protocole-mobile-definitionse-mobile-scorese-mobile-loope-qualitativee-control
5.4 Training and inference
During training
Training minimizes cross-entropy averaged over non-prefix text tokens. PaLM-E-12B combines PaLM-8B with ViT-4B; 84B combines 62B with 22B; 562B combines 540B with 22B. Experiments either update the components jointly or freeze the LLM. The OSRT experiments also freeze its scene representation and train only the projector. e-traininge-osrt
The full mixture samples 8.9% embodied data: mobile manipulation 3.1%, Language-Table 4.2% and TAMP 1.6%. WebLI accounts for 52.4%. These are sampling proportions, not unique-example counts. Language-Table task-specific finetuning adds 9,000 steps; mobile planning uses 2,912 sequences from prior robot runs. e-mixturee-language-protocole-mobile-loop
During inference
Mobile planning conditions on the human instruction, previous steps and current image, generates the next step, maps it to a low-level policy, and continues until 'terminate'. RT-1 supplies the mobile manipulation controller. Affordance and failure questions are separately evaluated as visual question answering tasks. e-mobile-loope-mobile-definitions
Real Language-Table demonstrations use 1 Hz language subgoals and 5 Hz low-level actions. Appendix B.2 describes data collection at 10 Hz for four-second commands and simulated evaluation with four-second execution between prompts. These timing descriptions are preserved as different reported contexts; their implementation relationship is not resolved. e-real-tablee-language-protocol
5.5 Implementation flow
- Embed observations into the language sequence
An observation encoder produces multiple vectors in the language embedding space. These replace special placeholders among embedded words. The existing positional encoding and decoder process the resulting multimodal prefix; observation positions are determined by the surrounding text. e-tokenization
- Choose a scene representation
State vectors use an MLP; ViT features use a learned affine projection. Object-centric ViT variants use ground-truth masks. OSRT instead learns object slots through in-domain novel-view synthesis and maps each slot to several language-space vectors. Explicit object labels let generated plans refer to otherwise ambiguous entities. e-encoders
- Connect language to execution
PaLM-E sequences an available skill vocabulary through text. It infers usable skills from training and prompts, without an additional output filter. A separate policy or planner executes each decision, and subsequent observations permit replanning. e-control
6. Experiments & results
PaLM-E turns sensor observations into vectors interleaved with language embeddings, then generates answers or high-level robot instructions. Its central empirical finding is that broad multimodal training can improve robot-task learning with little task-specific data. Separate controllers execute its instructions. Results support a reusable embodied language backbone, with important differences between simulation scores, perception diagnostics and qualitative physical demonstrations.
6.1 Read the original evidence
Tables 2 and 3. Demonstration count and model configuration must be read together with each task's evaluation protocol. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the task prompts on the right to interpret the three groups of numbers on the left. Task 1 finds a specified block and moves it to its color partner; Task 2 sorts colors into corners; Task 3 groups blocks from only one side. Read the checkmarks before comparing scores: full-mixture training, pretrained components, LLM freezing and additional task finetuning are separate flags. Appendix B.2 supplies a crucial distinction absent from the table heading: Task 1 uses validation accuracy, whereas Tasks 2 and 3 use simulated rollout success. Demonstration columns also differ, with Task 3 extending to 80 demonstrations. e-language-tablee-language-protocol
What it supports. For Task 2 with ten demonstrations, the task-finetuned full-mixture 12B model reaches 31.3%, compared with 2.5% for the scratch single-robot row. Scaling the unfrozen generalist from 12B to 84B improves the maximum-demo Task 1 and Task 3 scores, but Task 2 falls from 57.5% to 53.8%.
Where the evidence stops. The scratch-versus-full-mixture comparison changes pretraining and data composition simultaneously. Tasks 2 and 3 use 80 evaluation rollouts each, but the source gives no uncertainty intervals here. Do not average Task 1 accuracy with rollout success.
Figure 5. Selected physical executions illustrate the role of updated observations and transferable language subgoals. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper sequence from left to right. The instruction remains to bring rice chips from the drawer, but the observed scene changes after a person knocks the chips back inside. The next shown instruction repeats extraction from the drawer before delivery proceeds. This is consistent with the mobile prompt in Section 6.4, which includes previous steps and the current image before generating the next skill. The lower panels show tabletop examples labeled one-shot grouping and zero-shot movement toward a turtle. Their arrows mark the illustrated object motions, while the green borders identify the authors' selected successful endpoints. e-qualitativee-mobile-loope-real-tablee-control
What it supports. PaLM-E can supply useful textual subgoals to two different physical robot systems in the examples shown. The kitchen sequence demonstrates a plausible need to revisit a prior step after an intervention, and the tabletop examples show the intended distinction between task-specific experience and generalization to a new object.
Where the evidence stops. These are selected demonstrations, with no aggregate success rate or matched no-feedback control. The figures cannot isolate whether recovery comes from fresh visual reasoning, task history or controller robustness, and the linked videos were not inspected.
Table 8. Larger unfrozen PaLM-E models retain more of their inherited language performance in these evaluations. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each adjacent PaLM/PaLM-E column pair as a before-and-after comparison at a particular scale. The PaLM-E headers explicitly say unfrozen. The rightmost column divides tasks into natural language generation and understanding, so inspect the separate averages near the bottom rather than combining all benchmarks. For NLG, the small pair changes from 32.4 to 4.1, the middle pair from 47.8 to 18.4, and the large pair from 53.8 to 51.7. The bottom rows give relative percentage changes, not percentage-point differences. The caption states that frozen-LLM variants retain the language performance of their corresponding PaLM models. e-retentione-traininge-discussion
What it supports. The observed forgetting pattern is strongly scale-dependent: the table reports NLG losses of 87.3%, 61.6% and 3.8% across increasing model sizes. The largest model still loses performance on some individual tasks, so a small average change does not mean every language capability is preserved.
Where the evidence stops. Figure 6 and Section 6.6 report 3.9% for the largest NLG drop, while this original table prints 3.8%; the discrepancy is preserved. These benchmark pairs show a trend, not a controlled explanation of why scaling reduces forgetting.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Low-data TAMP grasping and stacking PaLM-E-12B; 1% TAMP data, 320 examples per planning task; mean of p1 grasping and p2 stacking. | 94.9 with finetuning and full mixture. Mean planning success (%) | 48.6 finetuned/single robot; 74.3 frozen/full mixture; 31.8 frozen/single robot; 42.9 without pretraining. Both mixture and LLM adaptation matter in this setting; the figure gives no uncertainty intervals. e-ablatione-tamp-tasks |
| TAMP input representations with limited supervision Table 1, three to five objects, 320 examples each for p1/p2; pretrained LLM frozen. | OSRT: 82.5 / 76.2. Planning success, p1 / p2 (%) | ViT-4B full mixture: 74.1 / 74.6; single robot: 30.6 / 32.9; OSRT without VQA: 71.9 / 75.1. Object-centric geometry is effective here, but representation and pretraining resources differ across rows. e-low-datae-tamp-tasks |
| Language-Table color sorting in simulation Task 2; 10 demonstrations; 12B full mixture with task finetuning; Appendix B.2 evaluates 80 rollouts per task. | 31.3. Rollout success (%) | Single-robot model trained from scratch and task-finetuned: 2.5. With 40 demos the respective values are 58.8 and 2.5. This comparison changes pretraining and data mixture together. Table 2's Task 1 uses validation accuracy, not rollout success. e-language-tablee-language-protocol |
| Mobile affordance prediction and failure detection PaLM-E-12B, full mixture; separate image-and-skill classification evaluations. | Unfrozen: affordance 0.91, failure detection 0.77; frozen: 0.87 and 0.91. F1 score | PaLI zero-shot: 0.62 and 0.73; QT-OPT affordance: 0.63; CLIP-FT-hindsight failure detection: 0.89. Freezing has task-dependent effects. These F1 scores are not long-horizon robot success rates. e-mobile-definitionse-mobile-scores |
| Knowledge-based visual question answering OK-VQA validation; PaLM-E-562B generalist checkpoint without task-specific finetuning. | 66.1. Reported VQA score | Task-finetuned PaLI: 64.5; PaLM-E-12B generalist: 55.5. A strong generalist result within the paper's comparison, not evidence of superior performance on every vision-language benchmark. e-vqa |
| Language retention after multimodal finetuning Table 8, one-shot language evaluation; unfrozen PaLM-E compared with its inherited PaLM. | 540B to 562B: 53.8 to 51.7; Table 8 reports −3.8%. Average NLG benchmark score and relative change | 8B to 12B: 32.4 to 4.1, −87.3%; 62B to 84B: 47.8 to 18.4, −61.6%. Larger models retain more performance here. Figure 6 and main text give 3.9% rather than Table 8's 3.8%; the source discrepancy is unresolved. e-retention |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. TAMP planning improves with both broad data mixing and language-model finetuning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the bars as the mean success of two TAMP tasks, grasping and stacking, with only 320 examples for each task. First compare the two finetuned rows: changing from single-robot training to the full mixture moves the score from 48.6% to 94.9%. Next make the same mixture comparison with a frozen LLM, from 31.8% to 74.3%. Finally compare frozen and finetuned full-mixture models. The small bent arrow ties the 'without pretraining' bar to the single-robot setting; it is not a separate full-mixture ablation. All five bars refer to the PaLM-E-12B study described by the caption. e-ablatione-tamp-taskse-mixturee-transfer
What it supports. Broad training helps in both freezing regimes, and allowing LLM adaptation raises full-mixture success further. This supports the paper's transfer argument more directly than the overview figure, which bundles pretraining and mixture changes. The improvement concerns feasibility-sensitive simulated planning under the specified low-data task setting.
Where the evidence stops. The full mixture changes several data sources together, so these bars do not identify whether web data or other robot embodiments cause the gain. No uncertainty bars are shown, and these are not physical robot execution rates.
Table 4. Freezing the language model affects affordance prediction and failure detection differently. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two numeric columns answer different questions. Affordance prediction asks whether a named skill is possible now; failure detection asks whether that skill succeeded. Both are scored by F1, not by completion of an entire robot task. Focus on the bottom two rows: both use pretrained components and the full mixture, while the freezing flag changes. Read upward to compare the frozen full mixture with frozen single-robot training. The dash symbols mark absent evaluations, not zero performance. The upper baselines also differ in their information and training setup; the paper explicitly says the hindsight CLIP method has additional information. e-mobile-definitionse-mobile-scorese-mobile-loop
What it supports. With a frozen LLM, broad mixing improves affordance F1 from 0.78 to 0.87 while failure-detection F1 remains 0.91. Unfreezing raises affordance F1 to 0.91 but lowers failure-detection F1 to 0.77. Appendix Table 9 shows a corresponding failure-detection precision drop from 0.89 to 0.66.
Where the evidence stops. The strongest affordance and failure-detection numbers come from different freezing configurations. They must not be presented as one checkpoint's paired performance. These diagnostics alone do not establish the reliability of the closed-loop mobile planner.
7. Analysis & limitations
7.1 What the evidence leaves open
Physical mobile manipulation and novel-object tabletop examples are qualitative demonstrations. They do not establish population success rates or systematic robustness to disturbances. Figure 2's reasoning examples likewise do not provide a measured generalization benchmark. e-qualitativee-groundinge-mobile-loop
A single multimodal model does not eliminate dependence on competent low-level skills. Unconstrained textual outputs may name unavailable skills; the paper specifies no extra mechanism to filter them. It does not evaluate an explicit learned future-state rollout used for control. e-controle-training
The study lacks a dedicated limitations section. Figure 3 combines pretraining and mixture changes; Figure 4 is more informative but does not separate other-robot transfer from web-data transfer. Freezing preserves language capability yet can weaken embodied performance. e-transfere-ablatione-language-tablee-discussion
7.2 Questions for discussion
- Which mixture component supplies the planning gain when pretraining and compute are fixed?
- How much demonstrated disturbance recovery depends on fresh visual input versus low-level controller robustness?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the corresponding pretrained PaLM and vision encoders, the mixture in Table 6, domain expert supervision and compatible low-level policies. TAMP data generation and control refer to Driess et al.; Language-Table and mobile control refer to prior systems. e-traininge-mixturee-tamp-taskse-language-protocole-mobile-loop
The supplied training recipe does not specify a complete optimizer schedule, batch size, total multimodal training budget, hardware allocation or decoding configuration. The appendix supplies task protocols and sampling ratios but does not close these implementation gaps. e-traininge-osrte-mixturee-language-protocol
Proposed checks should separate mixture composition from pretraining at fixed task data, and test current-image replanning against stale-image control under matched disturbances. Record planning validity separately from controller failures, with repeated trials and uncertainty estimates. e-ablatione-controle-qualitativee-language-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate web transfer from other-robot transfer in low-data TAMP
Reader-proposed experiment, not performed: start all runs from the same pretrained 12B initialization and retain the same 320 examples per planning task. Keep the target-task sampling schedule, optimizer, total updates, freezing choice and evaluation scenes fixed. Allocate matched auxiliary batches to in-domain VQA repetition, other-robot data, general vision-language data, or both auxiliary domains. Evaluate p1 and p2 separately over repeated seeds, reporting uncertainty and plan-validity failures. A gain only when both domains are present would support complementary transfer; no gain over matched in-domain repetition would weaken the claim that diversity itself explains Figure 4. Repeat the comparison with a frozen LLM to test its interaction with adaptation. e-ablatione-low-datae-mixturee-traininge-tamp-tasks
Check 2: Measure whether fresh observations cause disturbance recovery
Reader-proposed experiment, not performed: use the same trained Language-Table model, low-level controller, prompts and initial scenes in paired simulated runs. At a fixed stage of color sorting, displace a block by a standardized amount. Compare current-image replanning with a stale pre-disturbance image while preserving the same step history and replanning interval; include undisturbed controls for both conditions. Follow Appendix B.2's four-second command protocol and report task completion, recovery steps, invalid subgoals and controller failures separately. A selective advantage for fresh images after intervention would support observation-driven recovery. Similar recovery with stale images would leave the qualitative demonstrations compatible with controller robustness or history-based behavior. e-controle-language-tablee-language-protocole-qualitativee-real-table
8.3 Reading coverage
Visual audit: Read all seven supplied chunks, covering the complete 18-page v1 PDF. Visually inspected the title and author block, Figures 1–8, Tables 1–10, method equations, training text, task definitions and relevant appendix protocols on the listed pages. Inspected every final crop: Figure 1, Figure 4, Tables 2–3, Table 4, Figure 5 and Table 8. Architecture arrow directions agree with Equation (3) and Section 3; the Figure 4 no-pretraining branch agrees with its single-robot context. Table 2 metrics were checked against Appendix B.2. Figure 6's 3.9% and Table 8's 3.8% NLG discrepancy, and the distinct real-robot versus appendix timing descriptions, remain explicit. Reference-only pages 11–13 were read as text. No external videos, code, supplements or later edition were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. PaLM-E: An Embodied Multimodal Language Model
- 4. Input & Scene Representations for Different Sensor Modalities
- 5. Training Recipes
- 6. Experiments
- 6.1. Robot Environments / Tasks
- 6.2. TAMP Environment
- 6.3. Language-Table Environment
- 6.4. Mobile Manipulation Environment
- 6.5. Performance on General Visual-Language Tasks
- 6.6. Performance on General Language Tasks
- 7. Summary of Experiments & Discussion
- 8. Conclusion
- Acknowledgements
- References
- A. Data Mixture
- B. Environment Details
- B.1. Task and Motion Planning (TAMP)
- B.2. Interactive Language Table
- C. Natural Language Generation and Understanding Results
- D. Additional Data for Affordance and Success Detection
- E. Image Attribution
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Reviewed all seven supplied text chunks covering all 18 PDF pages, including appendices and references. Visually inspected PDF pages 1–10 and 14–18; reference-only pages 11–13 were read as text.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the retained PDF pages and six final original crops.
- Separate supplemental material availability has not been fully verified.
- The supplied artifact is arXiv:2303.03378v1, dated 6 March 2023. Its title and all 22 authors match the catalog. The catalog cites ICML 2023; the venue edition and later revisions were not supplied or compared, so this report describes v1 only.
- No external project pages, videos, code, model weights or datasets were inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author and affiliation block, arXiv margin, Abstract
Title and 22 authors match the catalog; affiliations are Robotics at Google, TU Berlin and Google Research. Margin identifies arXiv:2303.03378v1, 6 March 2023. The abstract frames continuous grounding and joint multimodal training.
Go to primary source ↓e-groundingPDF pp. 1–2, Introduction; Figure 1 and Figure 2 captions
The introduction motivates grounding in continuous observations and reports generalist embodied capabilities. Figure 1 depicts interleaved observation/text tokens feeding PaLM and text feeding control. Figure 2 offers qualitative multimodal reasoning examples.
Go to primary source ↓e-tokenizationPDF p. 4, Section 3, token embedding space and multimodal sentences, Equations (1)–(3)
Continuous observations map into sequences in the language embedding space and interleave dynamically with word embeddings. The decoder produces text autoregressively using its existing positional encodings.
Go to primary source ↓e-controlPDF p. 4, Section 3, 'Embodying the output: PaLM-E in a robot control loop'
Separate low-level policies execute generated skills. The model infers the skill vocabulary from data and prompts without an additional output constraint/filter. New observations support replanning.
Go to primary source ↓e-encodersPDF p. 5, Section 4, state vectors, ViT, object-centric representations, OSRT and entity referrals
MLPs embed states, an affine transform projects ViT features, masked ViT uses ground-truth instances, and OSRT discovers object slots through novel-view synthesis. Object labels connect input embeddings to references in output plans.
Go to primary source ↓e-trainingPDF p. 5, Section 5, training dataset, loss and model sizes
Cross-entropy averages over non-prefix textual targets. PaLM-E model sizes sum PaLM and ViT parameter counts. The recipe considers updating all components versus freezing the LLM.
Go to primary source ↓e-osrtPDF p. 6, continuation of Section 5, model freezing and co-training
In OSRT experiments the scene representation and LLM are frozen and the small projector is trained. Full-mixture embodied sampling is 8.9%; encoders can be trained through a frozen LLM.
Go to primary source ↓e-transferPDF p. 6, Figure 3, Section 6 and Section 6.1
The overview combines PaLM/ViT pretraining with broad data mixing, contrasting with in-domain training from scratch across three robot domains. Two domains have physical robot executions.
Go to primary source ↓e-ablationPDF p. 7, Figure 4, bars and caption; Section 6.2 low-data setup
TAMP mean p1/p2 success is 94.9% finetuned/full mixture, 48.6% finetuned/single robot, 42.9% without pretraining, 74.3% frozen/full mixture and 31.8% frozen/single robot. The 1% setting retains 320 training examples per planning task.
Go to primary source ↓e-low-dataPDF p. 8, Table 1, ViT-4B and OSRT rows, p1/p2 columns and caption; PDF p. 7, Section 6.2
With the pretrained LLM frozen, OSRT achieves 82.5/76.2, OSRT without VQA 71.9/75.1, ViT-4B full mixture 74.1/74.6 and single-robot ViT-4B 30.6/32.9. Ground-truth object-centric inputs and unavailable q1 evaluation are identified in the caption.
Go to primary source ↓e-tamp-tasksPDF pp. 14–15, Appendix B.1, Figure 8 and task definitions
Training scenes contain three to five blocks. Tasks p1 and p2 are grasping and stacking; object-centric prompts can reference labeled object tokens. Expert plans and low-level policies use the cited Driess et al. method.
Go to primary source ↓e-oodPDF p. 16, Table 7, object-count groups and OOD-task caption; PDF p. 7, Section 6.2
Table 7 distinguishes training-range object counts, larger scenes and a color-reference OOD task replacing object tokens. Section 6.2 describes the 96,000-scene TAMP training setting and compares pretraining and entity referrals.
Go to primary source ↓e-language-tablePDF p. 9, Tables 2–3, model configuration flags, demonstration columns and task prompts
For color sorting (Task 2), the 12B full-mixture task-finetuned model scores 31.3/58.8/58.8 with 10/20/40 demos; the scratch single-robot row scores 2.5/6.3/2.5. At maximum listed demos, unfrozen non-task-finetuned 12B scores 80.0/57.5/50.0 across tasks, versus 90.0/53.8/64.4 for 84B. Task 1 identifies and moves a block; Task 3 groups blocks from one side.
Go to primary source ↓e-language-protocolPDF p. 16, Appendix B.2, Data collection and Train and Evaluation
Data collection uses four-second commands with 40 low-level steps at 10 Hz. Task finetuning adds 9,000 steps. Simulated Tasks 2/3 use automated reward and 80 rollouts per task, executing four seconds per prompt. Task 1 uses a test set and reports validation accuracy.
Go to primary source ↓e-real-tablePDF p. 7, Section 6.3, 'Real Robot Results and Few-Shot Generalization'; PDF p. 14, Figure 7
The real-robot description and Figure 7 diagram give 1 Hz PaLM-E replanning and 5 Hz low-level actions. The text describes finetuning on 100 tasks with one example each and qualitative generalization and disturbance recovery.
Go to primary source ↓e-mobile-loopPDF p. 7, Section 6.4 opening; PDF p. 8, 'Real robot results: Long-horizon planning'
RT-1 supplies low-level mobile manipulation policies. Planning prompts include instruction, history and image, produce next steps until terminate, and are trained from 2,912 prior run sequences. Kitchen evaluation is explicitly qualitative.
Go to primary source ↓e-mobile-definitionsPDF p. 8, Section 6.4, Affordance prediction and Failure detection
Affordance asks whether a skill is possible in the observed scene; failure detection asks whether a skill succeeded. These are formulated as image-conditioned questions. The hindsight baseline has additional information.
Go to primary source ↓e-mobile-scoresPDF p. 9, Table 4; PDF p. 18, Tables 9–10, full-mixture and baseline rows
Table 4 reports failure/affordance F1: full-mixture frozen 0.91/0.87, unfrozen 0.77/0.91; frozen single robot 0.91/0.78. PaLI is 0.73/0.62, hindsight CLIP failure F1 0.89, QT-OPT affordance F1 0.63. Full-mixture failure precision/recall changes from 0.89/0.93 frozen to 0.66/0.91 unfrozen.
Go to primary source ↓e-qualitativePDF p. 8, Figure 5; PDF p. 14, Figure 7 and captions
Original image sequences show a kitchen robot recovering after chips are knocked back into a drawer and tabletop one-shot/zero-shot examples, including an unseen toy turtle. They show selected physical executions, not aggregate success estimates.
Go to primary source ↓e-vqaPDF p. 9, Table 5, OK-VQA val column and generalist/task-specific row groups
PaLM-E-562B generalist achieves 66.1, PaLM-E-12B generalist 55.5 and task-finetuned PaLI 64.5 on OK-VQA. Generalist rows share a checkpoint across evaluations; task-specific rows do not. Table 5 does not establish superiority on every benchmark.
Go to primary source ↓e-retentionPDF p. 17, Appendix C, Table 8, headers, Avg NLG and relative deltas; PDF p. 9, Figure 6 and Section 6.6
One-shot NLG averages for PaLM/PaLM-E are 32.4/4.1, 47.8/18.4 and 53.8/51.7. Table 8 prints relative changes −87.3%, −61.6%, −3.8%; Figure 6 and text print 3.9% for the largest pair. Table 8 states frozen-LLM variants preserve underlying language performance.
Go to primary source ↓e-mixturePDF p. 15, Appendix A, Table 6, sampling proportions
WebLI is 52.4%, mobile robot data 3.1%, Language-Table 4.2%, TAMP 1.6%; the table also lists VQ2A, VQG, CC3M, object-aware data, OK-VQA, VQAv2, COCO and Wikipedia text. It specifies relative sampling, not a complete training schedule.
Go to primary source ↓e-discussionPDF pp. 9–10, Section 7 and Section 8
The discussion highlights transfer, data efficiency and language retention. Freezing can struggle on robot tasks; larger unfrozen models lose less language performance. No dedicated limitations section is present before the conclusion.
Go to primary source ↓8.5 Primary sources
PaLM-E: An Embodied Multimodal Language Model ↗
PDF · 12,104 extracted words
Source fingerprint
535cc3c7fe1ae7428b9ac7d6a3738a0529975aadc372b2229ea0cb2ce724576a