Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
1. Paper overview
In one sentence: Seer makes predicted visual latents available to an inverse-dynamics policy inside one transformer, improving task adaptation while leaving other-embodiment transfer much less reliable. e01e03e05e08e09e12e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations contain visual, temporal, state, and action information, yet action-only cloning or separately trained visual predictors may exploit only part of it. Seer asks whether jointly learning future visual expectations and the actions leading toward them improves downstream manipulation after limited fine-tuning. e02e03 |
| Core mechanism | |
| A key reported result | CALVIN ABC-D instruction chains: Seer: 3.98 / 64.3; Seer-Large: 4.28 / 74.0. Average completed chain length; five-task completion (%). Unlabeled then labeled A/B/C training; held-out environment D; 1,000 five-instruction sequences, top-three checkpoints. Standard scratch: 3.64 / 53.3; Large scratch: 3.83 / 60.3; CLOVER: 3.53 / 45.4. The 0.75-task gain over CLOVER belongs to Seer-Large. Standard Seer gains 0.34 over its scratch counterpart. e09 |
| Reading caution |
Core contributions
- Source description
Pre-training uses future robot states as goals when language annotations are absent, while disabling attention to previous image/state tokens to reduce reliance on arbitrary play behavior. e05
Figure 2. A predicted visual representation becomes an input to action prediction within the shared transformer. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the current image at the left and the language and robot-state tokens above the encoder. The red [FRS] token reads those inputs to represent a forecasted state; the purple [INV] token can attend to both the inputs and [FRS]. The curved arrows indicate which tokens are attended to, so a purple arrow pointing toward red means that the action readout accesses foresight. Follow red latents downward toward image reconstruction with added mask tokens, and purple latents toward action decoding. The caption and Equation (3) establish that inverse dynamics conditions on the predicted latent; the decoded RGB image need not be re-encoded for that connection. e03e04e05
What it supports. The mechanism supports One Model × IDM: a shared transformer explicitly routes a predicted future representation into the action readout. Image reconstruction provides supervision, while inverse dynamics turns the associated latent into intermediary actions. This is stronger architectural evidence than the mere presence of two training losses.
Where the evidence stops. The central 'Vision–Action' two-way symbol denotes synergy; it does not specify bidirectional token attention. Section 3.3 explicitly permits [INV] to attend to [FRS]. This overview also omits the pre-training history mask and future-state goal substitution described in Section 3.4.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations contain visual, temporal, state, and action information, yet action-only cloning or separately trained visual predictors may exploit only part of it. Seer asks whether jointly learning future visual expectations and the actions leading toward them improves downstream manipulation after limited fine-tuning. e02e03
2.2 What this reading follows
A robot can benefit from anticipating what its actions should achieve, but a useful forecast must influence its controller. Seer implements that connection with two readout tokens: one represents a future visual state, and the other predicts the actions leading toward it. Both are trained together on robot demonstrations before downstream adaptation. This reading follows the latent connection, the distinction between pre-training and deployment, and the evidence from simulation and physical tasks. It also preserves the supplied v1 paper's reporting inconsistencies, including incompatible printed averages, rather than treating every headline improvement as an independently verified aggregate. e01e03e05e08e09e12e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | IDM |
| Quadrant | Q2 · One Model × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded One Model × IDM classification is supported by architecture, not joint losses alone: a shared transformer lets [INV] consume the predicted [FRS] latent during inference. Separate readout decoders are components of this integrated policy. The action target is the interval leading to forecasted observations, matching predictive inverse dynamics rather than independent future/action prediction. e03e05e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the future useful before decoding it
Seer's visual prediction is most useful to control before it becomes an RGB image. Observed camera images first become embeddings, which are compressed and combined with language and robot-state tokens. The shared transformer then produces [FRS] latents associated with future images and [INV] latents associated with actions. Because [INV] can attend to [FRS], the controller receives the representation that is being trained to explain the future observation. The image decoder reconstructs patches from that representation; the action decoder predicts the interval leading toward that future. Equations (1)–(5) tie these outputs to visual and action supervision. This explains why retaining an image-prediction loss alone is not the whole PIDM design: the architecture also supplies the forecast representation to action prediction at inference. e03e04e05e06
5.2 Use unlabeled play without requiring future observations at deployment
A language-conditioned future predictor needs a goal, but robot play datasets can lack language annotations and contain exploratory behavior. Section 3.4 handles that pre-training setting by letting [FRS] attend to a future robot-state token instead of a language token. The specified state index is t+n+1, whereas the image target is t+n; the report preserves that indexing. The same section prevents [FRS] and [INV] from attending to previous image/state tokens during pre-training. At deployment, complete language instructions and observed history become available to the readouts, and the predicted visual latent conditions the action output. A future state is therefore a training-time goal source in the described unlabeled setting, not a sensor measurement that must somehow be obtained from the future at runtime. e03e05e09
5.3 Separate a useful result from a complete explanation
Reader analysis: the cleanest internal evidence comes from the matched Seer comparisons, particularly the CALVIN objective ablations. They show gains from adding visual prediction and further gains from predictive inverse dynamics, but they do not identify whether attention routing, shared optimization, or their interaction causes the improvement. External-baseline gaps add practical context while introducing differences in pre-training, architecture, and evaluation provenance. The physical results establish successful executions rather than merely plausible forecasts, yet their printed four-task average needs arithmetic clarification. Finally, Appendix A.6.4 shows that moving from DROID to non-Franka OXE pre-training weakens transfer and can hurt individual tasks. Together these observations support a useful learned coupling between vision and action while keeping its causal mechanism and transfer range open to sharper tests. e11e12e17e19
5.4 Training and inference
During training
Both stages optimize pixel reconstruction and action supervision. Pre-training without language substitutes the state at t+n+1 as the goal, although the predicted image is at t+n. Section 3.4 also masks previous image/state tokens during pre-training; inference restores history and language conditioning. e03e05
The visual/text encoders contribute 251M frozen parameters; standard Seer has 65M trainable parameters. LIBERO uses LIBERO-90 pre-training; CALVIN uses unlabeled play from A/B/C before labeled fine-tuning; real-world experiments use DROID before task demonstrations. e05e08e09e12
During inference
Given language, current observations and history, Seer computes foresight and action latents together. The paper permits executing the first predicted action or temporally ensembling overlapping chunks, but does not identify the choice and weights for every experiment. Real-world settings use history seven and a three-step foresight/action horizon. e04e05e16
The feedback loop is renewed observation followed by another prediction. The method does not describe reward-based trajectory search; decoded image quality alone is not its robot-success criterion. e03e05e12
5.5 Implementation flow
- Encode observations
Frozen MAE ViT-B and CLIP ViT-B/32 encode images and language. A Perceiver Resampler compresses visual tokens; an MLP encodes robot state. The standard trainable transformer has 24 layers, width 384, and 12 heads. e05e06e07
- Predict a visual latent
Each timestep receives [FRS] readout tokens. These attend to the permitted goal and observation tokens; a ViT decoder combines foresight latents with mask tokens to reconstruct future image patches. e03e05e06
- Condition inverse dynamics
[INV] attends to input tokens and [FRS], then an MLP decodes intermediary actions. Conditioning uses the predicted latent, not a ground-truth future image. This shared attention pathway makes visual prediction relevant to action inference. e03e05
6. Experiments & results
Seer learns manipulation by forecasting a future visual latent and conditioning an inverse-dynamics action chunk on that latent inside one transformer. Joint robot-data pre-training improves subsequent task learning, but the strongest CALVIN result belongs to Seer-Large, and transfer from other embodiments is much weaker than DROID transfer. The evidence below separates predictive representations, executed-task metrics, and unresolved reporting details.
6.1 Read the original evidence
Table 2. Keep model size and pre-training separate when interpreting the strongest CALVIN result. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the numbered columns as the percentage of evaluation sequences completing at least that many consecutive instructions. The last column gives average completed length, with larger values better. The protocol uses 1,000 five-instruction sequences in environment D after training on A, B, and C, and reports top-three checkpoint averages. Compare Seer with Seer (scratch) to examine pre-training at the standard model size. Repeat that comparison within the two Seer-Large rows before comparing sizes. The bold 4.28 belongs to the pre-trained Large model; the standard model's corresponding result is 3.98. CLOVER provides the strongest baseline average listed in this table. e09e19
What it supports. Standard Seer rises from 3.64 to 3.98 average length with pre-training; Seer-Large rises from 3.83 to 4.28. The Large model completes all five tasks in 74.0% of sequences, versus 45.4% for CLOVER. These are manipulation outcomes in a held-out simulated environment, rather than image-reconstruction scores.
Where the evidence stops. Top-checkpoint averaging is not a confidence interval, and the table provides no uncertainty estimates. Appendix A.3 imports several baseline scores from prior papers, so the comparison does not establish a controlled advantage at matched data, model size, and compute.
Figure 3. The reported benefits persist across downstream data budgets and the tested model sizes. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the left plot for LIBERO-LONG success rate and the middle plot for CALVIN average task length; their vertical axes measure different quantities. Red denotes pre-training, blue denotes training from scratch, and the green dashed line marks a previous-best reference. At each downstream data fraction, compare red with blue rather than comparing heights across benchmarks. The right panel changes trainable model size on CALVIN. Its vertical axis begins at 3.0, so the bar heights visually magnify differences. The figure demonstrates discrete evaluated configurations; it does not provide a fitted scaling-law equation or measurements beyond those configurations. e10e05
What it supports. The pre-trained curves remain above their scratch counterparts across the displayed data fractions. At the smallest downstream budget, the difference is especially pronounced. The size comparison likewise favors pre-training at each tested size, supporting the practical value of adapting a robot-pre-trained model when task data are scarce.
Where the evidence stops. Section 4.3's 187% and 150% gains at 10% data are relative author-reported changes, not percentage-point gains. Figure 3 labels the largest model 316M trainable parameters, while Section 3.4 says 315M. The source does not reconcile this label difference or supply plotted uncertainty.
Table 4. Physical-task gains are visible, but the printed Seer averages do not match the displayed columns. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each cell pairs success rate, in percent, with Score, which counts completed task stages. The setup uses a Franka Research 3 with a Robotiq gripper and wrist/base cameras, and the stated evaluation repeats each of 15 initial-state cases three times. Compare Seer against its scratch row within each task before reading the average. All four tasks use 100 demonstrations per task. Appendix task definitions and detailed case tables explain why a Score such as 26 for bowl flipping is an aggregate, despite the task's maximum score of two per execution. It is not an additional success percentage. e12e15e16e18
What it supports. Every Seer task column improves over the corresponding scratch column. The table prints averages of 78.4% success and 39.5 Score, compared with scratch 60.0% and 32.8. However, the displayed Seer columns average approximately 76.7% and 37.8. The direction of improvement remains supported; the headline aggregate requires clarification.
Where the evidence stops. The faithful crop preserves the inconsistency. Comparisons also differ in camera access and pre-training, while OpenVLA's size is stated as 3B here and 7B in Appendix A.6.3. No uncertainty is reported, and the four-task average excludes the two additional contact/precision tasks.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO-LONG manipulation LIBERO-90 pre-training; ten downstream tasks; 20 rollouts per task, averaging top-three checkpoints. | 87.7, as printed in Table 1. Average success rate (%) | Scratch 78.7; MPI 77.3; OpenVLA 54.0. Reported advantage over MPI is 10.4 percentage points. The printed mean is internally inconsistent with the displayed task cells; see limitations. e08 |
| CALVIN ABC-D instruction chains Unlabeled then labeled A/B/C training; held-out environment D; 1,000 five-instruction sequences, top-three checkpoints. | Seer: 3.98 / 64.3; Seer-Large: 4.28 / 74.0. Average completed chain length; five-task completion (%) | Standard scratch: 3.64 / 53.3; Large scratch: 3.83 / 60.3; CLOVER: 3.53 / 45.4. The 0.75-task gain over CLOVER belongs to Seer-Large. Standard Seer gains 0.34 over its scratch counterpart. e09 |
| CALVIN objective ablations Table 3(a): fine-tuning variants; Table 3(b): pre-training variants with full downstream objectives. | Fine-tuning: 3.31 → 3.41 → 3.64. Pre-training: 3.64 → 3.73 → 3.98. Average completed chain length | Fine-tuning compares ordinary BC, BC plus foresight, and predictive IDM; pre-training compares none, visual-only, and joint. The results favor action use of foresight, but do not separately isolate attention routing from optimization effects. e11 |
| Four real-world generalization-centric tasks Franka Research 3, two cameras, DROID pre-training, 100 demonstrations per task; 15 cases × three executions per case. | 78.4 / 39.5, as printed in Table 4. Average success rate (%) / reported aggregate Score | Scratch 60.0 / 32.8; MVP 55.0 / 29.8; MPI 48.4 / 29.3; OpenVLA 16.7 / 11.0. Subtracting printed success averages gives 18.4 percentage points over scratch, but Seer's displayed task cells do not reproduce its printed averages. Score counts completed stages, not a percentage or per-rollout mean. e12e18 |
| Press Button and Insertion Two additional physical tasks, 100 demonstrations each, same real-world evaluation protocol. | 60.0 on each task. Success rate (%) | Scratch 40.0 on each; MVP 46.7 on Press Button and 26.7 on Insertion. These tasks extend contact/precision coverage; they are excluded from Table 4's four-task average. e12e15 |
| Transfer from non-Franka OXE data OXE mixture excludes Franka subsets and filters subsets with 'odd action labels'; downstream fine-tuning on six physical tasks. | OXE 56.7; DROID 71.1. Six-task average success rate (%) | Scratch 53.3; OXE Press Button falls to 33.3 from scratch 40.0. Other-embodiment pre-training gives only a small average gain, unlike DROID. This is transfer followed by adaptation, not zero-shot cross-robot control. e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3(a–b). Predicting images and using their latents for actions contribute distinct improvements in the reported ablations. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left and right panels as different experiments. On the left, Section 4.5 identifies the crossed-out first row as ordinary behavioral cloning, not a model trained without any action supervision. Adding future-image prediction gives the middle row; integrating predictive inverse dynamics gives the last row. On the right, all variants use the full downstream objectives, while pre-training changes from none to visual-only to joint visual/action learning. The numbered columns retain CALVIN's consecutive-task interpretation. Compare the last column within each panel: moving across panels without accounting for the changed training stage would conflate different interventions and misread the repeated 3.64 result. e11e05
What it supports. The fine-tuning progression is 3.31 to 3.41 to 3.64 average completed tasks. The pre-training progression is 3.64 to 3.73 to 3.98. Within this design, combining visual foresight with inverse dynamics outperforms adding image prediction alone, both when adapting the policy and when acquiring its pre-training initialization.
Where the evidence stops. The objective labels do not expose every implementation change. This table does not separately test removing the [FRS]-to-[INV] information route, stopping gradients through that route, or replacing predicted foresight with ground truth. A specific causal explanation still needs those controls.
Table A-IV. The appendix narrows the claim: robot-data pre-training transfers unevenly across embodiments. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the OXE row as an additional experiment with subsequent downstream fine-tuning. The authors remove Franka-containing subsets from an Octo-style mixture and filter subsets with unsuitable action labels before pre-training. Compare OXE with scratch first, then compare both with DROID. This table includes Press Button and Insertion in addition to the four main physical tasks, so its average must not be compared directly with Table 4's four-task average. Each cell again pairs success rate with accumulated stage Score. Inspect individual columns as well as the average: transfer can help one task while lowering another task's success rate. e17e14e12
What it supports. OXE increases the reported six-task average success from 53.3% to 56.7%, while DROID reaches 71.1%. OXE decreases Stack Cups from 46.7% to 40.0% and Press Button from 40.0% to 33.3%. Broad robot-data diversity therefore does not reproduce the stronger same-embodiment gains in this evaluated setting.
Where the evidence stops. The authors propose missing wrist images and embodiment/controller gaps as explanations, but do not isolate them experimentally. Section 6 calls for cross-robot evaluation even though this appendix reports a limited transfer study. These results do not establish zero-shot control of a new robot.
7. Analysis & limitations
7.1 What the evidence leaves open
Section 6 acknowledges only six physical tasks and asks for cross-robot evaluation. Appendix A.6.4 already contains a limited cross-embodiment transfer experiment; its mixed results qualify, rather than erase, that limitation. e14e17
Robustness gains concern four specific disturbances: extra bowls, background change, new swept objects, and added lighting. No broad distributional guarantee follows. The comparative tables omit uncertainty; several CALVIN baselines are imported from prior papers. e13e09e19
Reporting discrepancies remain: Table 1's displayed Seer task rates average about 87.2%, versus printed 87.7%; Table 4's four Seer columns average about 76.7% SR and 37.8 Score, versus printed 78.4/39.5. Section 3.4 gives Seer-Large 315M trainable parameters whereas Figure 3 labels 316M; real-world OpenVLA is called 3B on p. 9 but 7B in A.6.3. These are not silently reconciled. e08e05e10e12e16
Table 4 and the appendix specify 100 demonstrations per task, but A.6.3 describes 400 downstream demonstrations despite the six-task study. Tables A-V–A-VIII give one entry per case rather than separately exposing all repeated executions; full replication and uncertainty estimation require clarification. e12e15e16e18
7.2 Questions for discussion
- Would removing only foresight-to-action attention preserve the gain from auxiliary image prediction?
- How much of the DROID–OXE transfer gap comes from missing wrist views versus action-controller mismatch?
8. Reproducibility audit
8.1 Requirements and known gaps
Table A-I specifies AdamW, cosine decay, learning rates 1e-4/1e-3 for pre-training/fine-tuning, simulation pre-training batch 640 and fine-tuning batch 512. Histories are seven for LIBERO/real and ten for CALVIN; action chunks are three. Appendix A.1 reports eight 4090 GPUs, with CALVIN pre-training/fine-tuning taking about 40/24 hours. e06e07
Recover the exact attention mask, resampler latent count, action normalization, deployment chunk selection, dataset filtering, and checkpoint-selection protocol before claiming equivalence. The appendix names checkpoints but leaves several implementation choices or conflicts unresolved. The paper's code-availability statement does not establish inspected or runnable artifacts here. e01e04e06e16e17
Reader-proposed checks: compare matched action models with and without [INV] access to [FRS], and independently control camera availability versus pre-training embodiment. The illustrated edition specifies controls and falsifiable outcomes; neither check has been run. e05e11e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate foresight information from shared optimization
Reader-proposed, not run: use standard Seer on CALVIN ABC-D with identical data, initialization, action chunks, losses, training budget, and pre-specified checkpoint selection. Compare the original mask with a variant blocking only [INV] access to [FRS], retaining both supervised heads. Add a stop-gradient variant that preserves foresight values while preventing action-loss gradients through that connection. Evaluate the same instruction chains across multiple seeds and report average length plus uncertainty. If blocking access leaves performance unchanged, the necessity of inference-time foresight is weakened; if stopping gradients alone harms performance, joint optimization contributes beyond merely providing a future representation. e03e04e05e07e09e11
Check 2: Disentangle camera coverage from embodiment transfer
Reader-proposed, not run: reproduce the six-task scratch, DROID, and non-Franka OXE comparison with fixed downstream demonstrations and matched pre-training budgets. First remove wrist images from DROID pre-training while keeping its embodiment and action representation fixed. Separately compare matched subsets or preprocessing conditions that hold camera availability fixed while changing embodiment/controller compatibility. Keep downstream sensing and checkpoint selection constant and report each task, not only the average. A large loss from wrist removal would support the visual-coverage hypothesis; a residual OXE gap under matched sensing would motivate investigation of controller or embodiment mismatch. Preserve exact subset/filter manifests so the intervention is reproducible. e12e15e16e17
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, all eight original figures, Tables 1–4 and A-I–A-VIII, equations, methods, evaluation protocols, limitations, and supporting appendix configuration pages. All six final crops were separately viewed and checked for readable labels, intact bounds, and faithful content. Figure 2 attention arrows were cross-checked against its caption, Equation (3), and Section 3.3; the decorative two-way synergy marker was not interpreted as bidirectional attention. Pages 11–13 are references, read in the complete text pass but not rendered for scientific claim verification. Separate supplements, code, and external resources remain outside this reading. Table 1 and Table 4 printed-average inconsistencies, parameter-count discrepancies, and the main-text/appendix cross-embodiment difference are explicitly retained.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract (p. 1)
- Sections 1–2: Introduction and Related Work (pp. 2–3)
- Sections 3.1–3.4: formulation, PIDM, architecture, implementation (pp. 3–5)
- Sections 4.1–4.5: simulation, data efficiency, scaling, ablations (pp. 5–8)
- Sections 5.1–5.3: real-world setup, results, robustness (pp. 8–10)
- Section 6 and acknowledgments (p. 10)
- References (pp. 11–13)
- Appendix A.1–A.5: implementation, architecture, baselines, simulation protocols (pp. 14–17)
- Appendix A.6.1–A.6.5: task definitions, implementation, cross-embodiment experiment, detailed records (pp. 17–20)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction's figure limitation was addressed by inspecting original PDF pages 1–10 and 14–20 and all six final crops; reference pages 11–13 were read as text.
- Identity/version: the title and all seven authors match the catalog, allowing for capitalization and the line-break hyphen in 'Scalable'. The supplied PDF explicitly identifies arXiv:2412.15109v1, 19 December 2024. The catalog says ICLR 2025; no separate venue edition or later revision was supplied or compared.
- Code, linked project resources, and external datasets were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, arXiv margin, abstract
The title, seven authors and six affiliations are visible; the artifact is arXiv:2412.15109v1 dated 19 Dec 2024. The abstract supplies a code link and claims availability.
Go to primary source ↓e02PDF pp. 2–3, Sections 1–2 and 3.1
The motivation contrasts action-focused cloning with visual pre-training and introduces large robot-data pre-training followed by limited downstream adaptation.
Go to primary source ↓e03PDF pp. 3–4, Sections 3.1–3.2, Eqs. (1)–(3), Figure 2
Goal-conditioned visual prediction uses image/state history; inverse dynamics consumes the predicted visual latent to predict the intervening action sequence.
Go to primary source ↓e04PDF p. 4, Eqs. (2), (4), (5), final paragraph of Section 3.2
Pixel MSE, Smooth-L1 arm loss and BCE gripper loss combine with alpha 0.5 and lambda 0.01; inference may select the first action or temporally ensemble.
Go to primary source ↓e05PDF p. 5, Sections 3.3–3.4; PDF p. 4, Figure 2 caption
[INV] attends to [FRS] and inputs; [FRS] attends to inputs. Pre-training without language uses state t+n+1 as goal and masks previous image/state tokens. Encoders remain frozen; standard/Large trainable counts are 65M/315M.
Go to primary source ↓e06PDF p. 14, Appendix A.1, Vision through Decoder and final compute paragraph
MAE ViT-B, CLIP ViT-B/32, Perceiver resampling, state encoding, two-view foresight and three action readouts are described. The arm/gripper decoders and eight-4090 simulation training times are specified.
Go to primary source ↓e07PDF p. 15, Tables A-I–A-II and Figure A-1; p. 16, Appendix A.2
Training schedules, batches, epochs, history/chunk lengths and transformer dimensions are reported; the standard backbone has 24 layers, hidden size 384 and 12 heads.
Go to primary source ↓e08PDF p. 6, Table 1 and Sections 4.1–4.2; p. 16, Appendix A.4
LIBERO-90 pre-trains ten-task LIBERO-LONG evaluation. Top-three checkpoint averages use 20 rollouts/task. Printed averages are Seer 87.7, scratch 78.7, MPI 77.3 and OpenVLA 54.0. Seer's ten displayed task cells are 91.7, 90.0, 98.3, 100, 91.7, 93.3, 85.0, 88.3, 61.7 and 71.7.
Go to primary source ↓e09PDF p. 6, Section 4.1 Metrics; p. 7, Table 2 and Section 4.2; pp. 16–17, Appendix A.5
CALVIN evaluates 1,000 five-task sequences in D after A/B/C training. Table 2 reports standard Seer 3.98 and 64.3% five-task completion; Large 4.28 and 74.0%; scratch variants 3.64/53.3% and 3.83/60.3%; CLOVER 3.53/45.4%.
Go to primary source ↓e10PDF p. 7, Figure 3 and Section 4.3; p. 8, Section 4.4
Plots compare downstream data fractions 10/20/40/70/100% and trainable sizes labeled 65M/107M/316M. Pre-training improves plotted performance. Section 4.3 reports 187%/150% relative gains at 10% data.
Go to primary source ↓e11PDF p. 8, Table 3(a–b), Section 4.5
Fine-tuning rows give 3.31/3.41/3.64; the text identifies ordinary BC even when both displayed objectives are crossed out. Pre-training rows give 3.64/3.73/3.98 with full fine-tuning objectives.
Go to primary source ↓e12PDF pp. 8–9, Sections 5.1–5.2, Figure 4, Table 4
The setup uses Franka Research 3, Robotiq-2f-85, two D435i cameras and 15 Hz demonstrations, 100 per task. Evaluation is 15 cases × three repeats; Table 4 covers four tasks and reports Seer 78.4/39.5 versus scratch 60.0/32.8. Page 9 calls real-world OpenVLA 3B and specifies only an eye-on-base camera.
Go to primary source ↓e13PDF pp. 9–10, Section 5.3 and Figure 5
Each physical task receives one disturbance type. Pre-trained success rates are 60.0/33.3/60.0/66.7 for extra bowls/background/novel objects/lighting, compared with scratch 33.3/6.67/46.7/46.7.
Go to primary source ↓e14PDF p. 10, Section 6
The stated limitations are six downstream tasks, the need for more contact/precision tasks, and the need to test across robots.
Go to primary source ↓e15PDF pp. 18–19, Appendix A.6.2, Figure A-3 and Table A-III
Press Button requires sufficient downward button travel; Insertion requires collision-free placement. With 100 demos/task, Seer reaches 60.0% on both versus scratch 40.0%; MVP scores 46.7% and 26.7%.
Go to primary source ↓e16PDF p. 19, Appendix A.6.3
Real-world history is seven and foresight/action horizon three; the vision encoder uses bfloat16. The text states 76K successful DROID trajectories and 400 downstream demos, checkpoints 9 and 17, and full fine-tuning of 7B OpenVLA.
Go to primary source ↓e17PDF p. 19, Appendix A.6.4 and Table A-IV
Non-Franka OXE subsets are filtered for action labels before pre-training and downstream adaptation. Six-task mean SR is scratch 53.3, OXE 56.7, DROID 71.1. OXE lowers Stack Cups and Press Button performance. Missing wrist views and embodiment/controller gaps are author hypotheses.
Go to primary source ↓e18PDF pp. 17–18, Appendix A.6.1; p. 19, A.6.5; p. 20, Tables A-V–A-VIII
Physical task scores count completed stages, with maximum per-trial scores 2/4/3/3 for bowl/cups/pick-place-close/wipe. Detailed tables list 15 case entries per method; Seer's bowl case scores sum to 26, matching Table 4 rather than a per-trial mean.
Go to primary source ↓e19PDF p. 16, Appendix A.3
Roboflamingo, Susie, GR-1 and 3D Diffusor Actor scores are taken from their respective papers; MTACT/OpenVLA are rerun by the authors; MVP/MPI replace the policy's vision encoder.
Go to primary source ↓8.5 Primary sources
PREDICTIVE INVERSE DYNAMICS MODELS ARE SCAL - ABLE LEARNERS FOR ROBOTIC MANIPULATION ↗
PDF · 11,851 extracted words
Source fingerprint
6ef743b757386ce294152934f696d498ce5092e85828cb32b981b2914086e7ef