Vision-Language Foundation Models as Effective Robot Imitators
1. Paper overview
In one sentence: RoboFlamingo turns single-step vision-language features into a recurrent manipulation policy, gaining CALVIN performance while risking loss of the backbone's general vision-language abilities. e-architecturee-policye-benchmarke-maine-languagee-retention
| At a glance | What to know |
|---|---|
| Research problem | Source description Static image–text pre-training supplies useful representations but does not directly provide temporal memory or continuous control. The paper asks whether an accessible VLM can become a robot imitator using language-labeled demonstrations, without web-data co-training in the default recipe. e-probleme-architecturee-training |
| Core mechanism | Source description A modular policy separates single-step multimodal fusion from temporal action prediction, combining OpenFlamingo features with an LSTM and MLP action head. e-architecturee-policy |
| A key reported result | CALVIN five-instruction chains, ABCD → D: 4.09 tasks; 0.660 success fraction Average completed chain length; success through all five tasks. M-3B-IFT trained on language-annotated A–D data; D evaluation, 1,000 chains; best checkpoint. HULC with full data: 3.06 and 0.383; HULC with language data: 2.90 and 0.335; reimplemented RT-1 with language data: 2.45 and 0.227. Strong simulated imitation. D is represented during training; the full-data HULC comparison does not match the training-data budget. e-benchmarke-main |
| Reading caution |
Core contributions
- Source description
A modular policy separates single-step multimodal fusion from temporal action prediction, combining OpenFlamingo features with an LSTM and MLP action head. e-architecturee-policy
- Source description
Experiments compare control formulations, pre-training, adaptation, backbone size, instruction tuning, and open-loop deployment; appendices expose the tradeoff between manipulation and retained foundation-model skills. e-ablatione-backbonese-lowdatae-openloope-cotraine-retention
Figure 2. Current perception and temporal control meet at a pooled feature vector. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the two small camera images at the lower left. The ViT produces tokens for the third-person and gripper views; the resampler compresses them before they enter the feature fusion decoder. Read the language branch upward from the example instruction. Section 4.2 clarifies the attention direction: language tokens provide queries, while visual tokens provide keys and values. Pooling then reduces the fused token sequence to the pink feature passed into the LSTM. Its curved arrow denotes recurrence across decision steps. Snowflakes identify frozen components in the schematic, while the resampler, cross-attention module, and policy head are the trainable components specified in Section 4.4. e-architecturee-fusione-policye-traininge-language
What it supports. The architecture provides a concrete route from image–language pre-training to robot commands without making the language model itself retain the observation sequence. The pooled feature describes the current multimodal input; the LSTM hidden state supplies temporal context. This separation supports the paper's perception/control decomposition.
Where the evidence stops. The tokenizer's frozen marker and Section 4.4's blanket freezing statement do not fully explain Table 1's separate freeze-emb intervention. The report preserves that ambiguity. Stacked action symbols also do not establish a learned world model.
2. Motivation
2.1 The problem and the proposed response
Static image–text pre-training supplies useful representations but does not directly provide temporal memory or continuous control. The paper asks whether an accessible VLM can become a robot imitator using language-labeled demonstrations, without web-data co-training in the default recipe. e-probleme-architecturee-training
2.2 What this reading follows
Imagine an instruction to pick up a block and place it in a drawer. Understanding the words and locating the objects are only part of the job: the controller must also remember what has happened and issue suitable motor commands. RoboFlamingo assigns these responsibilities to different modules. OpenFlamingo fuses the current camera views with the instruction, and a temporal policy head translates those features into actions. The experiments test this division of labor in the CALVIN simulator. Read the strong manipulation scores alongside the harder paraphrase tests and the appendix's evidence of lost captioning and question-answering ability after robot-only fine-tuning. e-architecturee-policye-benchmarke-maine-languagee-retention
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Dual-system |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
VLA and Dual-system fit a multimodal perception backbone feeding a distinct temporal policy head. This functional perception/control decomposition does not establish a symbolic high-level planner. There is no future-world prediction or inverse-dynamics action extraction, supporting the recorded Not applicable prediction paradigm and quadrant. e-architecturee-policye-heads
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Ground the instruction once per observation, carry memory between observations
For the block-and-drawer instruction, the backbone must represent the current scene in relation to the language goal. Section 4.2 provides the mechanism: compressed visual tokens become keys and values for cross-attention queried by language features. The controller then pools the fused tokens and updates its LSTM state. My interpretation is that this gives the two modules complementary jobs: current grounding can reuse image–text pre-training, while temporal integration is learned in a head designed for sequences. Equation (5) makes the distinction explicit because the current pooled feature and previous hidden state both enter the recurrent update. Figure 11 shows why the MLP-with-history alternative is different: it moves the burden of sequence integration into the VLM. Figure 3 supports the tested decomposition, without isolating one universal reason for its advantage. e-fusione-policye-headse-ablation
Figure 11. An MLP with history and an LSTM use history at different points. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the upper Current observation path. Its VLM produces a pooled feature for one decision step. The right-hand alternatives show what happens downstream: LSTM cells pass recurrent state from the previous step to the current step; the GPT head processes a feature sequence; the history-free MLP uses only its current feature. Now trace the lower History observation sequence path. Here the images are combined before multimodal fusion, so its MLP receives a representation whose history must already have been integrated by the VLM. Appendix C.1 confirms this distinction even though the schematic does not draw every connection from the upper pooled feature to each alternative. e-headse-policye-ablatione-openloop
What it supports. This figure explains why simply saying that both models have history is insufficient. The recurrent and transformer heads perform temporal integration after per-step perception; MLP w hist asks the VLM to perform that integration. Figure 3 favors the former arrangements in the tested setup.
Where the evidence stops. Stacked action icons do not specify an execution horizon or feedback scheduler. The schematic also omits detailed head dimensions and history-window settings, so it guides implementation structure without fully specifying a reproducible controller.
5.2 Adapt the multimodal bridge to motor targets
Training supervises actions directly. Equation (6) applies mean squared error to demonstrated relative poses and weighted binary cross-entropy to gripper status. Thus the feature extractor must provide information useful to continuous control, even though its pre-training concerned language and images. The stated tuning recipe updates the resampler, gated cross-attention, and policy head while leaving other parameters frozen. Figure 3(b) tests both sides of this choice: removing multimodal bridge pre-training hurts, and retaining the VLM unchanged while training only the head also hurts. Appendix B.3 adds a different comparison, where full-model tuning performs much worse than selective tuning under the tested recipe. These findings motivate adaptation with a restricted trainable interface. The separate freeze-emb experiment still leaves the exact default treatment of language embeddings ambiguous. e-traininge-fusione-ablatione-fullfinetunee-language
5.3 Read transfer, wording robustness, and retained skills as separate tests
The top block of Table 1 evaluates imitation with environment D represented in training. The middle block withholds D and asks about transfer to a different visual context. The bottom block changes instruction wording, so its decline cannot be described as the same generalization result. Appendix B.1 then asks another question: can the adapted model still caption images and answer visual questions? Table 6 shows that strong robot-only performance can coexist with severe losses on those tasks. Co-training recovers some of that ability but does not improve the enriched-instruction result in Table 5. My interpretation is that these evaluations should remain distinct axes of model quality. Better multimodal benchmark retention is not by itself evidence that the robot follows unfamiliar paraphrases more reliably, and none of these simulated tests establishes physical deployment. e-maine-transfere-languagee-retentione-cotraine-real
5.4 Training and inference
During training
Default tuning uses language-annotated CALVIN demonstrations. Section 4.4 specifies training the resampler, gated cross-attention, and policy head while freezing other parameters; the separate freeze-emb experiment leaves an embedding-state qualification unresolved. e-traininge-maine-language
Pose prediction uses mean squared error; gripper classification uses weighted binary cross-entropy. Web-data co-training is an additional experiment, mixing equal numbers of vision-language and robot batches per epoch. e-traininge-cotrain
During inference
The VLM supplies current features and the recurrent controller supplies memory. Neither imagined future observations nor an inverse-dynamics model are required to produce commands. e-architecturee-policy
Stacked actions reduce expensive VLM calls. Direct open-loop use loses success; retraining on jump-step demonstrations alleviates, but does not remove, the plotted gap to closed-loop control. e-openloop
5.5 Implementation flow
- Compress the current visual observation
A frozen ViT encodes both camera views. A trainable perceiver resampler uses learned queries to compress the visual-token sequence. The default backbone processes one decision step at a time. e-architecturee-fusion
- Fuse vision with the instruction
Language tokens query visual keys and values through gated cross-attention interleaved with frozen language-model layers. A learnable tanh gate controls the visual contribution. The output is a multimodal feature sequence, rather than generated action text. e-fusion
- Place memory in the controller
Max-pool across feature tokens, update the LSTM hidden state, and use an MLP to predict pose and gripper outputs. GPT-style and MLP heads are alternatives; the default LSTM carries history outside the VLM. e-policye-heads
- Execute and obtain feedback
Execute the next predicted command and process fresh images. CALVIN supplies another instruction only after the current goal succeeds; the model does not generate the evaluation task chain. e-openloope-benchmark
6. Experiments & results
RoboFlamingo adapts OpenFlamingo into a language-conditioned manipulation policy: a vision-language backbone interprets each camera observation, while a recurrent head remembers history and predicts robot commands. CALVIN results support this decomposition, but robot-only fine-tuning sharply reduces general vision-language performance. The evidence concerns simulated action execution, with no learned future-world predictor or real-robot deployment.
6.1 Read the original evidence
Table 1. Training coverage and instruction wording change the meaning of a CALVIN score. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each horizontal block as a separate evaluation setting. The top block includes environment D in training; the middle withholds D; the bottom changes the instructions used at evaluation. Full includes play data without language pairs, whereas Lang restricts training to language-annotated data. According to the original caption, gray rows are results from models retrained by the authors. Columns one through five report success in completing a prefix of the instruction chain, not independent or conditional success on each numbered task. Avg Len summarizes how far a chain proceeds. All rows use selected best checkpoints, so comparisons should retain both the training-data label and evaluation setting. e-maine-transfere-languagee-benchmark
What it supports. RoboFlamingo reaches 4.09 tasks on ABCD → D and 2.48 on ABC → D. Its enriched-instruction average is 1.85, improving to 2.12 with freeze-emb. These are complementary findings: strong simulated imitation and visual transfer coexist with a sizeable sensitivity to instruction wording.
Where the evidence stops. The HULC Full rows use a different data budget from RoboFlamingo Lang. No uncertainty accompanies these selected checkpoints, and the small 1.85 versus 1.82 enriched-instruction difference does not establish statistical significance. No physical-robot conclusion follows.
Table 3. Relative data efficiency improves before reliable long chains emerge. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The caption defines a much smaller training regime than the main result: ten percent of the language-annotated subset, which the text calls 0.1 percent of the full dataset. All variants receive the same number of training epochs. Read the rows using the model definitions in Table 2: M refers to MPT, G to GPT-NeoX, and IFT denotes instruction tuning. First compare M-3B and M-9B, then examine the IFT pairs. Keep the first-task and fifth-task columns distinct; a better average can come largely from completing the first few goals, without yielding dependable five-task chains. e-lowdatae-backbonese-compute
What it supports. M-9B's average length of 0.83 exceeds M-3B's 0.05 and M-3B-IFT's 0.13. Yet M-9B completes all five tasks with a reported fraction of only 0.003. The result supports a relative advantage under limited demonstrations, while showing that the absolute level of long-horizon competence remains low.
Where the evidence stops. Equal epochs are not equal compute, and the backbone comparison also involves pre-training and architecture choices. Table 2's near-competitive smaller models under the larger training regime should not be collapsed into a universal model-size scaling rule.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN five-instruction chains, ABCD → D M-3B-IFT trained on language-annotated A–D data; D evaluation, 1,000 chains; best checkpoint. | 4.09 tasks; 0.660 success fraction Average completed chain length; success through all five tasks | HULC with full data: 3.06 and 0.383; HULC with language data: 2.90 and 0.335; reimplemented RT-1 with language data: 2.45 and 0.227. Strong simulated imitation. D is represented during training; the full-data HULC comparison does not match the training-data budget. e-benchmarke-main |
| CALVIN visual transfer, ABC → D Language-annotated A–C training; unseen environment D; same chain evaluation and best-checkpoint convention. | 2.48 tasks; 0.235 Average completed chain length; five-task success fraction | RT-1 (language data): 0.90 and 0.013; HULC (full data): 0.67 and 0.011. Supports transfer across these simulated visual contexts, not arbitrary new skills or physical environments. e-benchmarke-transfer |
| CALVIN enriched-instruction evaluation ABCD training, D testing with randomly sampled GPT-4 alternatives; 50 alternatives generated per task. | 1.85; freeze-emb variant 2.12 Average completed chain length | HULC 1.82; RT-1 0.86. Standard-instruction RoboFlamingo scores 4.09. Paraphrases cause substantial decline. Freezing embeddings helps; no uncertainty estimates establish the small default advantage over HULC. e-languagee-main |
| CALVIN with 10% of language-annotated training data ABCD → D; 0.1% of full data according to the paper; equal training epochs across variants. | M-9B 0.83; M-3B 0.05; M-3B-IFT 0.13 Average completed chain length | G-4B 0.48; G-4B-IFT 0.55. Larger backbones help here, but every average remains below one completed task. Equal epochs do not mean equal compute. e-lowdatae-backbones |
| Vision-language retention after manipulation fine-tuning Table 6 four-shot evaluation on COCO and VQAv2; checkpoints selected as in Table 5. | Robot-only fine-tune: 0.005 CIDEr and 3.87 accuracy; co-training: 0.426 and 38.73. COCO CIDEr in Table 6 units; VQAv2 accuracy as reported | OpenFlamingo 3B: 0.822 and 43.86. Co-training scores 3.76 on ABCD → D versus 4.09 for robot-only tuning. Co-training recovers some VL ability at a manipulation cost. Preserve Table 6's CIDEr scale; do not merge it numerically with Table 2. e-retentione-cotrain |
| Full-model versus selective fine-tuning MPT-3B-IFT, ABCD language data → D; Appendix B.3. | Selective tuning: 4.09; full-model tuning: 0.50 Average completed chain length | Approximately 1B versus 3B trainable parameters. The tested full-model recipe deteriorates; this does not prove all fully tuned recipes must fail. e-fullfinetune |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3(a–c). Memory placement, multimodal adaptation, and feedback each affect control. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first five groups in each panel against the left success-rate axis; the Avg. Len. group after the dashed divider uses the right axis. Panel (a) compares the MLP variants with GPT and LSTM heads. Panel (b) isolates removing VL adaptation or resampler/cross-attention pre-training; No VL Pretrain does not remove every pre-trained component. Its Full label means the complete RoboFlamingo recipe, not the full-parameter tuning tested in Table 8. Panel (c) contrasts direct open-loop execution, open-loop execution after retraining, and closed-loop control. Follow the legends carefully: matching pastel colors across different panels does not imply matching models. e-ablatione-openloope-headse-fullfinetune
What it supports. The qualitative ordering supports three practical choices: retain history in a temporal head, adapt the multimodal bridge using demonstrations, and preserve feedback when possible. Jump-step retraining makes open-loop execution more effective than applying it directly, but the closed-loop bars remain higher. The figure supports these orderings without requiring guessed bar values.
Where the evidence stops. Changing the history formulation changes more than a single memory switch. The static-image pre-training explanation is an author hypothesis. The open-loop comparison does not state its exact action horizon here or establish a measured latency–success curve.
Table 6. Better robot imitation can accompany severe loss of general vision-language skills. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare models within one shot setting before comparing settings. Each of the three 3B model groups has a zero-shot row followed by a four-shot row. The COCO columns assess caption generation; the final column assesses visual question answering. The original caption ties fine-tuned and co-trained checkpoints to the selection used in Table 5, rather than independent optimization of every VL metric. Preserve CIDEr exactly on this table's reported scale, which differs from Table 2's presentation. The final Original Flamingo row is an 80B model in a fine-tuned setting and should not be treated as the same-model control. e-retentione-cotraine-backbones
What it supports. In the four-shot rows, robot-only tuning gives 0.005 CIDEr and 3.87 VQA accuracy; co-training yields 0.426 and 38.73, below OpenFlamingo's 0.822 and 43.86. Table 5 adds the tradeoff: co-training lowers the ABCD → D manipulation average from 4.09 to 3.76. Retention is therefore a separate evaluation target.
Where the evidence stops. Recovery on captioning and VQA does not guarantee robustness to robot paraphrases: Table 5 reports enriched-instruction averages of 1.73 for co-training and 1.85 for robot-only tuning. These tasks test different behavior.
7. Analysis & limitations
7.1 What the evidence leaves open
There is no real-robot deployment, attributed to lacking real-robot data. RT-2 is not experimentally compared because its code, data, and weights were unavailable to the authors. e-reale-main
Figure 3 favors LSTM/GPT memory over both MLP formulations and supports VL pre-training and adaptation. The explanation that static-image pre-training hinders temporal fusion is the authors' hypothesis; these variants also change where and how history is represented. e-ablatione-heads
Main tables select best checkpoints without uncertainty. Section 5.1 counts dataset units as steps, while Appendix A.1 calls the corresponding quantities trajectories. The exact trajectory count therefore remains unresolved. e-maine-benchmark
Figure 2 marks the tokenizer frozen and Section 4.4 says all other parameters are frozen, yet Table 1 separately tests freezing the decoder embedding layer. The default embedding update state is not unambiguously reconciled. e-architecturee-traininge-language
7.2 Questions for discussion
- Does temporal memory still help when competing heads receive matched history and training budgets?
- Can instruction robustness improve without sacrificing manipulation success or general vision-language ability?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components are the relevant OpenFlamingo checkpoint, two-view CALVIN demonstrations and chain evaluator, resampler/cross-attention adapters, and temporal action head. Match data splits and checkpoint selection before comparing scores. e-fusione-headse-benchmarke-main
Appendix A.3 reports one server with eight NVIDIA Tesla A100 GPUs, batch size six per GPU. MPT-3B takes 13 hours per epoch and peaks at epoch three; MPT-9B takes 26 hours and peaks at epoch four. This does not establish one-GPU training. e-compute
Optimizer, learning-rate schedule, numerical gripper-loss weight, precise history window, and detailed controller dimensions are unspecified. Resolve those and embedding freezing before calling an implementation faithful; the abstract's promised code release does not settle them. e-traininge-policye-headse-languagee-identity
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does recurrent history help after controlling the feature interface?
Reader-proposed experiment, not performed: use the same M-3B-IFT checkpoint, resampler/cross-attention tuning policy, CALVIN language subset, action targets, and training schedule for an LSTM head, a current-feature MLP, and a GPT head. Match head capacity as closely as practical and report residual parameter and compute differences. Give the temporal heads the same observation history and evaluate all models on identical held-out chains with a predeclared checkpoint rule and multiple seeds. Add an LSTM control that resets its state every step. Report prefix success, average chain length, and paired uncertainty. If resetting history leaves performance unchanged, the proposed benefit of remembered observations is weakened; a consistent drop under matched inputs supports it. Log the actual embedding update state because the source leaves it ambiguous. e-headse-ablatione-policye-traininge-maine-language
Check 2: Can retained VL skills and instruction robustness improve together?
Reader-proposed experiment, not performed: cross robot-only versus COCO/VQA co-training with frozen versus trainable decoder embeddings, holding the robot-example budget, backbone, and action head fixed. Log the additional VL updates and compute instead of treating co-training as equal cost. Evaluate the same chains with original instructions and human-checked meaning-preserving alternatives; use paired initial states and multiple seeds. At the same preselected checkpoints, measure CALVIN average length, the paired wording-induced drop, and COCO/VQAv2 performance under an identical shot protocol. A recovery of VL scores without a smaller wording-induced drop would weaken the idea that generic skill retention resolves robot-language robustness. This comparison would also clarify whether embedding freezing helps independently of web-data co-training. e-languagee-cotraine-retentione-maine-training
8.3 Reading coverage
Visual audit: The title, authors, affiliations, arXiv version and Preprint label were verified on page 1. Pages 2–9 were inspected for the problem, architecture, equations, losses, benchmark protocol, main tables, ablations, deployment and real-robot limitation. Pages 13–19 were inspected for environment/wording examples, compute, co-training, retention, representation comparisons, full-model tuning, training curves, per-task results, rollouts and policy-head details. All Figures 1–11 and Tables 1–9 were visually inspected; six selected original crops were then individually viewed at their final dimensions. Claims about Figure 2 and Figure 11 were cross-checked against Sections 4.1–4.4 and C.1; the embedding-freezing ambiguity is disclosed. References and acknowledgements on pages 10–12 were read as text. No separate supplement, code inspection, or experimental reproduction is included.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–3: Introduction, Related Work, Background
- Sections 4.1–4.4: control formulation, backbone, policy head, objective
- Sections 5.1–5.5: benchmark, imitation, generalization, ablations, deployment
- Section 6: Conclusion and Future Work; Acknowledgements; References
- Appendix A.1–A.3: CALVIN, enriched instructions, computing resources
- Appendix B.1–B.7: co-training, representation baselines, full-model tuning, training curves, qualitative examples, per-task results, rollouts
- Appendix C.1: policy-head formulations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Recovered-edition limitation: arXiv v3 preprint, with Preprint on page 1. Catalog cites ICLR 2024; this recovered preprint is not asserted to be identical to the blocked ICLR publication PDF. Title and all 12 authors match.
- The complete supplied 19-page text was read in six individual chunks. Original PDF pages 1–9 and 13–19 were rendered and visually inspected, resolving the extraction-only figure limitation; reference/acknowledgement pages 10–12 were read as text.
- No separate supplements were supplied. Code and model artifacts were not inspected, and no experiments were reproduced.
- Identity verified on PDF page 1: the title and all 12 authors match the catalog after normalizing typography. The observed version is arXiv:2311.01378v3, 5 February 2024, marked Preprint. Equivalence to the catalog's ICLR 2024 publication edition is not established.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block, arXiv margin and Abstract
The title and all 12 catalog authors match after normalizing typography. The page says Preprint and arXiv:2311.01378v3, 5 Feb 2024. Affiliations are ByteDance Research, Tsinghua University, Shanghai Jiao Tong University, and National University of Singapore. The abstract promises future public code/models.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1; Figure 1
The authors motivate adapting static vision-language models to continuous manipulation through an explicit policy head and robot demonstrations.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and Section 4.1
Two images and a language goal feed the Flamingo backbone; fused representations feed a recurrent policy. The schematic marks ViT, language self-attention, and tokenizer with the Frozen symbol.
Go to primary source ↓e-fusionPDF p. 5, Sections 4.2.1–4.2.2, Eqs. (2)–(4)
ViT tokens are compressed using learned resampler queries. Language queries attend to visual keys/values; gated cross-attention is trainable and copied language-model layers are frozen.
Go to primary source ↓e-policyPDF pp. 5–6, Section 4.3, Eq. (5)
Token-wise max pooling precedes the LSTM. Its hidden state feeds an MLP for relative end-effector pose and gripper predictions.
Go to primary source ↓e-trainingPDF p. 6, Section 4.4, Eq. (6) and following paragraph
Pose MSE and weighted gripper BCE supervise imitation. Specified trainable modules are the resampler, gated cross-attention, and policy head; all other parameters are described as frozen.
Go to primary source ↓e-benchmarkPDF p. 6, Section 5.1; PDF p. 13, Appendix A.1 and Figure 4
CALVIN has 34 tasks, four environments, and 1,000 up-to-five-instruction chains; the next goal follows success. Only 1% of play data is language annotated. The main text says steps where the appendix says trajectories for corresponding dataset counts.
Go to primary source ↓e-mainPDF p. 7, Table 1 top block and caption, Sections 5.1–5.2
Best-checkpoint ABCD → D averages/five-task fractions are 4.09/0.66 for RoboFlamingo, 3.06/0.383 for full-data HULC, 2.90/0.335 for language-data HULC, and 2.45/0.227 for reimplemented RT-1. The backbone is M-3B-IFT; RT-2 is not compared. Gray rows mark author-retrained models.
Go to primary source ↓e-transferPDF p. 7, Table 1 middle block and Section 5.3
ABC → D averages/five-task fractions are 2.48/0.235 for RoboFlamingo, 0.90/0.013 for RT-1, and 0.67/0.011 for full-data HULC.
Go to primary source ↓e-languagePDF p. 7, Table 1 bottom block and Section 5.3; p. 8, paragraph preceding Section 5.4; pp. 13–14, Appendix A.2 and Table 4
Fifty GPT-4 alternatives per task support enriched evaluation. Average lengths are 1.85 for default RoboFlamingo, 2.12 for freeze-emb, 1.82 for HULC, and 0.86 for RT-1. The text describes an added embedding-freezing intervention.
Go to primary source ↓e-backbonesPDF p. 8, Table 2 and caption
Backbones differ in LLM family, parameter count, and instruction tuning. Best averages within five epochs include M-3B 3.94, M-3B-IFT 4.09, and M-9B 3.97; trainable size is reported as 1B. Mean averages refer to the last three epochs.
Go to primary source ↓e-ablationPDF p. 8, Figure 3(a,b) and Sections 5.4.1–5.4.2; p. 9, continuation of Section 5.4.2
LSTM and GPT outperform MLP history formulations. No VL Pretrain removes pre-trained resampler/cross-attention weights, not all pre-trained encoders; No VL Finetune trains only the policy head. Both underperform the full recipe.
Go to primary source ↓e-lowdataPDF p. 9, Table 3 and Section 5.4.3
With 10% of language-labeled data and matched epochs, averages are 0.05, 0.13, 0.48, 0.55, and 0.83 for M-3B, M-3B-IFT, G-4B, G-4B-IFT, and M-9B respectively. M-9B's five-task success is 0.003.
Go to primary source ↓e-openloopPDF p. 8, Figure 3(c); p. 9, Section 5.5
Predicting stacked actions reduces VLM invocation frequency. Open-loop execution deteriorates; jump-step retraining improves it but remains below closed-loop in the plotted results.
Go to primary source ↓e-realPDF p. 9, Section 6, final paragraph
The authors explicitly state that they do not deploy on real-world robots because of missing real-robot data.
Go to primary source ↓e-computePDF p. 14, Appendix A.3
The server has eight NVIDIA Tesla A100 GPUs, batch size six per GPU. MPT-3B and MPT-9B require 13 and 26 hours per epoch, with best epochs three and four.
Go to primary source ↓e-cotrainPDF p. 14, Appendix B.1 and Figures 5–6; p. 15, Table 5
Co-training includes equal VL and robot batches per epoch. Best-checkpoint averages are 3.76 versus 4.09 on ABCD → D, 2.47 versus 2.48 on ABC → D, and 1.73 versus 1.85 with enriched instructions.
Go to primary source ↓e-retentionPDF p. 15, Table 6, four-shot 3B rows and caption
Four-shot COCO CIDEr/VQA accuracy are 0.005/3.87 after robot-only tuning, 0.822/43.86 for OpenFlamingo, and 0.426/38.73 after co-training; epochs follow Table 5. Original 80B Flamingo is a different model/setting.
Go to primary source ↓e-fullfinetunePDF p. 15, Appendix B.3; p. 16, Table 8
Full-model MPT-3B-IFT tuning has 3B trainable parameters and yields 0.50 average chain length versus 4.09 for selective 1B-parameter tuning.
Go to primary source ↓e-headsPDF p. 19, Appendix C.1 and Figure 11
LSTM and GPT receive per-step VLM features and integrate history in the head. MLP w/o hist uses current features only; MLP w hist feeds a history image sequence through the VLM before its MLP.
Go to primary source ↓8.5 Primary sources
VISION -L ANGUAGE FOUNDATION MODELS AS EFFEC - TIVE ROBOT IMITATORS ↗
PDF · 9,754 extracted words
Source fingerprint
ff81c416a352674cbb38c6e8f150c59e50efd1dffe9c0ae06c5fb8e7254e8f10