GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
1. Paper overview
In one sentence: A one-step visual future makes separate video and action pretraining connectable, but useful co-training requires selecting futures compatible with the demonstrated action. architecturevalidity-gapkasorobot-ablationscalinggrounding-results
| At a glance | What to know |
|---|---|
| Research problem | Source description A future can look plausible yet depict a different grasp, timing, or target from the recorded demonstration. Supervising that future with the recorded action creates the paper's validity gap. GE-Act also addresses a data-access problem: instruction–video pairs and instruction-free action trajectories can pretrain different components, but their connection must handle generated inputs while preserving useful pretrained behavior. validity-gapsvp-conditioning |
| Core mechanism | Source description CoAE supplies a compact, decodable control representation; the differentiable one-step SVP provides an explicit completed-future interface to a separately pretrained IDM. coaemeanflowarchitecture |
| A key reported result | Real-robot zero-shot OOD atomic manipulation: At 300/1,200/5,000/30,000 co-training hours: G1-OP 17.1/22.6/27.3/44.1; G2-90D 13.4/21.0/23.5/31.1. Suite success rate (%). 100 tasks, 20 groups, both embodiments; 10 trials per task/scale; held-out visual conditions and objects; no evaluation-task adaptation. 300→30,000 hours: +27.0 and +17.7 percentage points; 19/20 and 18/20 groups improve. Shared pretrained initialization supports a practical scaling comparison, but data and compute are not independently controlled. real-protocolscaling |
| Reading caution | Source description The authors leave large-scale egocentric-video effects untested and position the system as low-level control without deliberative planning, memory, or task decomposition. Selected instruction-switch rollouts are illustrative, not quantitative robustness estimates. limitationsinstruction-conflicts |
Core contributions
- Source description
CoAE supplies a compact, decodable control representation; the differentiable one-step SVP provides an explicit completed-future interface to a separately pretrained IDM. coaemeanflowarchitecture
- Source description
KASO selectively applies generated-future action supervision while retaining both original objectives. The study evaluates practical data scaling and fine-grained grounding through executed real-robot trials. kasoreal-protocolgrounding-results
Figure 2. A completed visual future connects the planner to the action model. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at observation encoding in the lower-left panel: CoAE compresses the left, head, and right views. Above it, the head image and instruction enter the VLM; the yellow text-token path carries image-contextualized language into the SVP. The blue arrow marked one step traverses noise time 1 to endpoint 0 and outputs dense and sparse predicted visual states. Follow those states into the separate IDM, whose own denoising remains iterative. The caption and Eq. (15) additionally specify current visual latents as IDM inputs, although the diagram foregrounds the predicted-state branch. The dense/sparse distinction carries through to action prediction. architecturesvp-conditioningmeanflowidmtraining-recipe
What it supports. The visual future is an inference-time input to control. Appendix B specifies one SVP pass followed by five IDM Euler steps, with the first 30 dense actions executed before another prediction. Sparse actions supply longer-horizon training targets; they are not the executed controller output.
Where the evidence stops. The diagram is an information-flow overview, not a latency measurement. It does not establish that one visual-generation pass meets a real-time deadline, and joint training does not make the SVP and IDM a shared backbone.
2. Motivation
2.1 The problem and the proposed response
A future can look plausible yet depict a different grasp, timing, or target from the recorded demonstration. Supervising that future with the recorded action creates the paper's validity gap. GE-Act also addresses a data-access problem: instruction–video pairs and instruction-free action trajectories can pretrain different components, but their connection must handle generated inputs while preserving useful pretrained behavior. validity-gapsvp-conditioning
2.2 What this reading follows
Imagine a demonstration that grasps an object from the left, while a generated future shows a successful right-side grasp. Both futures may be plausible, yet the left-grasp action is a poor label for the right-grasp video. GE-Act 2.0 builds its training method around this mismatch. A compact encoder feeds a visual planner, a separate inverse dynamics model reads the predicted transition, and KASO selects generated futures using that action model's response. The figures below connect this mechanism to controlled pickup experiments and broader data scaling. Read the execution and language results separately: contacting the requested object is easier than completing the manipulation. architecturevalidity-gapkasorobot-ablationscalinggrounding-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned, so there is no existing quadrant judgment to confirm. Architecture evidence supports a world-action model with separate visual-generation and inverse-dynamics modules: completed futures condition action prediction, and joint optimization does not merge their backbones. This is a two-model inverse-dynamics mechanism, not joint future/action generation in one shared model; sparse action supervision is auxiliary rather than a separate controller. architecturesvp-conditioningidm
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why the future interface needs both compression and grounding
The SVP must predict enough visual change for an action model to infer motion, while keeping future generation tractable. CoAE makes that interface small: each camera frame becomes 24 tokens, but reconstruction and three frozen feature teachers encourage preservation of complementary information. Language takes a different route. A frozen VLM processes the current head image with the instruction, and a learned gate combines its text-span states across layers. The generator therefore receives scene-grounded language without passing all VLM image tokens into every cross-attention layer. Conditional MeanFlow then predicts dense near-term and sparse farther-future latents in one pass. The IDM reads those latents without receiving language directly. This separation explains why the planner can pretrain on action-free video and the IDM can pretrain on instruction-free robot trajectories. coaesvp-conditioningmeanflowarchitecture
Figure 7. Select without gradients; replay the selected future to train both modules. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow Search & Select from independent generation noises to the candidate pool. The recorded future z enters the same active IDM probe and defines the reference response. All candidates share the probe's action noise and time, so differences in dense-token velocity reflect changed future conditioning. The checked response illustrates the lowest-energy choice; the implementation uses four candidates and keeps one. In Replay, the stored generation noise reconstructs exactly that future with gradients enabled. Fresh action noise and flow time are then sampled for training. Finally, the dashed arrows in Joint Update run backward through the IDM to the SVP; they are gradient flow, not an additional deployment stage. validity-gapkasoidmtraining-recipe
What it supports. KASO addresses incompatible supervision even when a predicted future looks plausible. It compares action-model responses rather than pixels, then preserves both recorded-data objectives alongside the selected-future action loss. The reference is what the current IDM reads from recorded video, so the method's notion of compatibility inherits that model's knowledge.
Where the evidence stops. The replay panel labels time as discrete U{1,…,T}; Eq. (15) and Appendix B.4 use continuous flow time, with logit-normal IDM sampling. This notation discrepancy is unresolved; the explanation follows the equations and recipe. Candidate selection is training-only.
5.2 Why plausible video is not sufficient action supervision
Consider two valid approaches to the same object. A recorded future and its action share the actual approach taken; a separately sampled future need not. The paper's validity-gap argument says that improving the future marginal does not repair this lost pairing. Retaining video and recorded-action objectives helps preserve modules, but the generated-future action branch can still teach the wrong correspondence. KASO uses a high-noise action probe so the IDM cannot mostly infer the answer from a nearly clean action input. It then asks which candidate produces an IDM response closest to the recorded future. The reader's interpretation is that this supplies a learned correspondence test, not a physical-validity oracle. Its limitations include poor IDM judgments and a candidate pool that fails to cover the demonstrated behavior; these are natural targets for controlled reproduction checks. validity-gapkasotraining-recipe
5.3 How to connect the ablation to the scaling claim
Table 2 asks a local mechanism question: given the same pretrained components and alignment mixture, does compatibility selection improve executed pickup? The equal follow scores and different pickup scores make that comparison informative. The large real-robot suite asks a broader question: how does the pretrained policy's repertoire change when co-training uses progressively larger nested datasets? These checkpoints receive no task-specific adaptation, yet their data and training exposure are not independently controlled. The reader should therefore treat the suite as evidence of practical scaling, while reserving causal claims about selection for the matched ablation. Simulation supplies another comparison layer after benchmark-specific in-distribution fine-tuning, rather than corroborating untouched real-world zero-shot deployment. Finally, the toy's seed-0 illustration explains a possible failure mechanism; its deterministic IDM and seed spread limit extrapolation to robot action diversity or later reinforcement learning. robot-ablationreal-protocolscalingrobotwingeniesimliberotoy
5.4 Training and inference
During training
The SVP flow generator and IDM start from random initialization; this does not mean all components lack pretrained knowledge. CoAE transfers DC-AE weights, its teachers are pretrained, and the VLM stays frozen. SVP and IDM have 2.51B and 0.56B parameters; the frozen VLM is reported separately as 2.72B. coaearchitecturetraining-recipe
SVP pretraining uses 39,000 instruction–video hours, including 3,000 ego/human-video hours. IDM pretraining uses 32,000 action-labeled hours, including 2,000 rollout/failure hours. The 30,000-hour co-training mixture contains 61.7% G1-OP and 1.7% G2-90D. These are overlapping stage mixtures, not additive unique-data totals. data-mixtures
MeanFlow uses mean-velocity and auxiliary instantaneous-velocity heads, masked future-coordinate losses, and a stopped-gradient Jacobian–vector correction. Co-training retains the SVP, recorded-future IDM, and selected-future action losses with coefficients 1, 0.1, and 1. Three micro-step gradients are accumulated and divided by three; the generator/IDM train while CoAE/VLM remain frozen. meanflowkasotraining-recipe
During inference
Encode three views, ground the instruction, sample future latents in one SVP pass, and integrate the IDM action flow with five Euler steps without guidance. Execute the first 30 dense actions at 30 Hz before predicting again. Sparse actions are training auxiliaries. Thus one-step visual generation does not make the entire policy a single network evaluation or establish measured real-time latency. idmtraining-recipe
5.5 Implementation flow
- Compress observations for control
Framewise CoAE maps each 256×384 image to a 4×6 grid of 512-channel latents. Reconstruction combines pixel, perceptual, and adversarial losses; alignment heads match frozen SigLIP 2, V-JEPA 2.1, and DINOv3 features. Compatible DC-AE weights are inherited, with new parameters initialized randomly. coae
- Ground the instruction and predict a future
Frozen Qwen3.5-2B sees the head image and instruction. A learned layer gate fuses image-contextualized text states for every DiT block. Conditional MeanFlow predicts four dense and two sparse future frames per view. The latter extend toward the step-caption segment's end; they are not six equally spaced control-rate frames. svp-conditioningmeanflowtraining-recipe
- Convert visual change into actions
The separate IDM attends to current and predicted visual tokens together with proprioception and an embodiment embedding. It predicts 52 dense actions plus two sparse actions. Language reaches this module through generated visual futures, allowing its standalone pretraining to omit instructions. architecturesvp-conditioningidmtraining-recipe
- Select compatible supervision
For each training context, KASO generates four candidates without gradients. The active IDM scores their dense-token velocity disagreement with its response to the recorded future, using shared action noise at probe time 0.95. The lowest-energy candidate is regenerated from its retained noise with gradients, then trained using fresh action noise and time. kasotraining-recipe
6. Experiments & results
GE-Act 2.0 connects a compact visual representation, a one-step future generator, and a separate inverse dynamics model. Its central training problem is pairing generated futures with compatible recorded actions. KASO selects futures using the current action model before updating both modules. Real-robot experiments show broader manipulation capability as co-training data grows, with persistent weaknesses in fine manipulation and ordinal language; simulation comparisons use a separate adaptation protocol.
6.1 Read the original evidence
Figure 10. Scaling broadens the repertoire, but improvement varies sharply by skill. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each panel averages the tasks in one skill group. Follow solid circles for G1-OP and dashed squares for G2-90D across 0.3, 1.2, 5, and 30 thousand co-training hours on a logarithmic horizontal axis. Compare within a panel before comparing embodiments: their grippers and training shares differ. Wipe rises strongly, Straighten emerges late, and Zip remains flat at zero. The legend calls the embodiment shares proportions of the pretraining corpus; the caption and Figure 8 specify co-training instead. That distinction matters because G1-OP is 47.4% of SVP pretraining but 61.7% of co-training. data-mixturesreal-protocolscaling
What it supports. The curves support broad but incomplete scaling gains. Section 5.2.1 reports improvement in 19 of 20 G1-OP groups and 18 of 20 G2-90D groups. The corresponding suite success rises from 17.1% to 44.1% and from 13.4% to 31.1%; those exact aggregate values are in the accompanying text, not plotted as suite curves here.
Where the evidence stops. All scales share pretrained starting components, but runs stop after one epoch or the compute limit. This is not a matched-compute causal experiment. G2-90D's own data also increases, so its improvement alone does not isolate cross-embodiment transfer.
Figure 14. Attribute grounding is strong; comparative and ordinal references expose weaknesses. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally by descriptor. Purple bars extend left on a reversed logarithmic axis and compare lexical occurrence in GE instructions with RefCOCOg expressions; they are not success rates. The middle and right panels use ordinary percentage axes for Pick and Place. Dark bars measure reaching only the requested referent, while pale bars require completion. Position covers direct left/right and near/far relations; Order combines middle and ordinal references. The final G1-OP checkpoint receives 59 instructions, each tested five times without adaptation. Appendix C.3 supplies the prompt-level results and the matching rules, including overlapping categories and the exception that size frequencies include simulation instructions. grounding-resultsdescriptor-frequency
What it supports. Object, color, position, and shape reach at least 90% Follow Score for both primitives. Size drops to 82.5% for Pick and 65.7% for Place; Order falls to 13.3% and 26.7%. Across all 295 rollouts, 83.1% correct following becomes 72.9% complete-task success, exposing failures after the referent has been reached.
Where the evidence stops. Frequency is observational and measured by lexical rules, not experimentally assigned. Descriptor categories overlap and differ in perceptual difficulty. Shape succeeds despite low frequency, so the plot cannot establish that increasing rare words alone will repair relational grounding.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-robot zero-shot OOD atomic manipulation 100 tasks, 20 groups, both embodiments; 10 trials per task/scale; held-out visual conditions and objects; no evaluation-task adaptation. | At 300/1,200/5,000/30,000 co-training hours: G1-OP 17.1/22.6/27.3/44.1; G2-90D 13.4/21.0/23.5/31.1. Suite success rate (%) | 300→30,000 hours: +27.0 and +17.7 percentage points; 19/20 and 18/20 groups improve. Shared pretrained initialization supports a practical scaling comparison, but data and compute are not independently controlled. real-protocolscaling |
| Controlled KASO picking ablation G2-90D; shared full-scale pretrained components, 300-hour alignment mixture including 30 embodiment hours; 40 four-object trials and 25 block trials per variant. | KASO: four-object follow 95, pick macro average 37.5; single-block pick 40. Follow score and completed-pick success (%) | E2E+PT: 95, 22.5, 12; E2E: 87.5, 27.5, 12, respectively. Selection improves pickup beyond an unchanged follow score versus E2E+PT; these reduced-stage results are not final full-model performance. robot-ablation |
| Frozen representation probes Action probe: 400 training/100 held-out shelf samples; caption probe: 5,000 GenieSim-Instruction episodes with VLM conditioning. | CoAE: 0.01673; 2.05. Action MAE; caption-match error (%) | DC-AE: 0.02573; 2.71. DINOv3 has lower action MAE, 0.01273. CoAE offers a compact compromise, not best action accuracy. Different channel widths and pretraining prevent isolating teacher alignment alone. probescoae |
| Fine-grained real-robot instruction grounding Final G1-OP checkpoint, no adaptation; 59 Pick/Place instructions × five trials. | 83.1 / 72.9. Overall Follow Score / full-task success (%) | Object, color, direct position, and shape follow scores are ≥90% for both primitives; ordinal/middle references score only 13.3% Pick and 26.7% Place. Correct contact and completed manipulation are distinct; strong attribute grounding does not establish general relational reasoning. grounding-results |
| Skill coverage association 30,000-hour G1-OP checkpoint; 19 groups excluding Close; training hours summed across embodiments. | 0.80; 0.85. Pearson r; Spearman rho | Fitted slope: 1.94 logit success units per tenfold training-hours increase. Coverage is associated with success, but the cross-skill comparison does not isolate causation. skill-coverage |
| RoboTwin Clean-to-Random Easy/Clean SFT followed by OOD combined-Hard simulation evaluation. | 60.52. Task-averaged success (%) | pi0.5: 47.90 (+12.62 percentage points for GE-Act). Strong compounded-shift performance after adaptation; lighting remains a weaker column. robotwin |
| GenieSim-Instruction SFT on released training demonstrations; ten tasks under prescribed OOD scene configurations. | 0.770. Average normalized task score | ACoT-VLA 0.757; pi0.5 0.746. Highest reported average, with leadership on four tasks rather than uniform superiority. geniesim |
| LIBERO-Plus Standard LIBERO adaptation; 10,030 perturbed task instances, weighted by axis instance counts. | 80.4. Overall success (%) | pi0.5 84.4; StarVLA 74.1. Uneven robustness: background 60.5% and robot-state 50.7% remain weak. libero |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Compression preserves useful information, with different winners for action and caption probes. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two rightmost columns as separate tests, both lower-is-better. Action MAE comes from the same two-layer inverse-dynamics probe trained on 400 shelf-task samples and evaluated on 100 held-out samples. Caption matching instead uses a VLM-conditioned classifier over 5,000 GenieSim-Instruction episodes. The spatial column gives downsampling along each image axis: CoAE's 64×64 reduction yields 24 tokens per 256×384 frame, while the 16×16 teachers yield 384. The channel column matters too: CoAE and DC-AE have the same spatial compression but different latent widths. Pixel decoder indicates whether the representation retains a reconstruction route. coaeprobes
What it supports. CoAE records action MAE 0.01673 and caption error 2.05%, compared with DC-AE's 0.02573 and 2.71%. DINOv3 recovers actions more accurately at 0.01273. CoAE therefore offers a compact, decodable compromise and the strongest caption score here, rather than uniformly dominating every representation.
Where the evidence stops. This is a representation probe, not a robot-success ablation of each teacher. Channel widths and pretraining differ, caption matching retains VLM conditioning, and its held-out split is not specified. The results cannot isolate multi-teacher alignment alone.
Table 2. Equal target following can conceal different pickup reliability. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) first: Follow Score means contacting the commanded object without contacting a distractor, even if lifting fails. Panel (b) requires the completed pickup. Every entry is a percentage, and the four-object macro average weights sponge, apple, mouse, and remote equally. The small-block column belongs to a separate single-object protocol, so it has no follow-score counterpart. All methods begin with the same 39,000-hour SVP and 32,000-hour IDM checkpoints, then use the same 300-hour alignment mixture, including 30 G2-90D hours. Compare E2E+PT with KASO to examine selective pairing while retaining both original training objectives. robot-ablation
What it supports. E2E+PT and KASO both follow the correct target in 95% of four-object trials. Pickup macro success nevertheless rises from 22.5% to 37.5%, a 15-percentage-point gain. Single-block success rises from 12% to 40%. These observations support improved physical execution beyond the measured target-contact score.
Where the evidence stops. The four-object experiment has ten trials per target and the block experiment has 25; no uncertainty intervals are given. These are reduced alignment-stage ablations atop full-scale pretraining, not the final model's performance or training from only 300 total hours.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors leave large-scale egocentric-video effects untested and position the system as low-level control without deliberative planning, memory, or task decomposition. Selected instruction-switch rollouts are illustrative, not quantitative robustness estimates. limitationsinstruction-conflicts
Ten trials per atomic task and the small ablation lack reported uncertainty intervals. Real-robot scaling has no matched cross-model baseline, and increasing own-embodiment data alongside other data prevents cleanly attributing G2-90D gains to transfer alone. real-protocolscalingdata-mixturesrobot-ablation
The toy demonstrates a possible interface failure with a deterministic action regressor and deliberately amplified off-manifold sensitivity. It uses a multi-step generator and eight candidates, unlike the full system, and its displayed seed is not a seed-averaged benchmark or evidence of improved downstream reinforcement learning. toytraining-recipe
Descriptor frequency is lexical and overlapping, with different object-residual definitions across corpora and a simulation exception for size. Its relationship to grounding is observational, so rare-instruction frequency alone cannot explain all failures. descriptor-frequencygrounding-results
7.2 Questions for discussion
- Does compatibility selection remain reliable when the pretrained IDM misreads a physically valid future?
- Would G2-90D gains persist if its own data were held fixed while only other embodiments grew?
8. Reproducibility audit
8.1 Requirements and known gaps
A reconstruction requires the stage mixtures, aligned camera/action streams, per-source action statistics, per-channel CoAE statistics, and trained component checkpoints. Appendix B supplies bf16/ZeRO-2/AdamW and learning-rate recipes, but not GPU count/model, exact training duration, end-to-end latency, or fully specified temporal jitter/extension probabilities; complete CoAE optimization details also remain absent. coaedata-mixturestraining-recipe
Proposed checks should first isolate compatibility selection from retained losses under matched initialization and updates, then test whether the high-noise IDM score detects behavioral mismatches. Keep evaluation single-sample, retain both recorded-data losses, and measure physical pickup separately from target contact. kasotraining-reciperobot-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does action compatibility outperform random selection under matched updates?
Reader-proposed, not executed: initialize from identical SVP/IDM checkpoints and use the same 300-hour G2-90D ablation mixture, sample order, optimizer steps, and retained-loss weights. Generate four futures in every arm. Compare lowest-energy selection with selecting one uniformly at random and with E2E+PT averaging all four candidates. Keep selected-noise replay and single-sample deployment identical. Repeat training seeds and evaluate the same four targets and single block with more matched-reset trials. Report confidence intervals for contact-only Follow Score and completed-pick success separately. A selective-pairing explanation predicts better pickup for minimum-energy selection than random selection; equal performance would weaken that explanation. kasotraining-reciperobot-ablation
Check 2: Does the high-noise probe actually detect behavioral mismatch?
Reader-proposed, not executed: retain candidate pools and paired recorded futures for contexts with multiple valid targets or grasp approaches. Label whether each candidate matches the demonstrated behavior using an independent review, blinded to KASO scores; keep task-irrelevant appearance changes as controls. Compare dense-token energy rankings at the reported probe time 0.95 with a lower-noise probe, using shared action noise within each pool. Measure ranking agreement with those labels, selected-future compatibility, and subsequent pickup under otherwise matched training. Also verify that replay reproduces the scored candidate. The mechanism predicts that high-noise scoring better rejects wrong behavior while tolerating irrelevant appearance changes; failure would reveal limits of the active IDM as judge. validity-gapkasotraining-reciperobot-ablation
8.3 Reading coverage
Visual audit: Visually inspected the title/byline and contributor credits, all Figures 1–15, Tables 1–10, Algorithm 1, and all supporting method, training, evaluation, and appendix pages listed here. Every final original crop was separately viewed at its returned dimensions. Figure 7's discrete replay-time notation and Figure 10's pretraining/co-training legend ambiguity are preserved and discussed. Bibliography-only pages 26–29 were read as text. The supplied PDF contains the reviewed evidence; external videos, code, datasets, and separate supplements were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1 Introduction; 2 Related Work (2.1–2.5)
- 3 GE-Act 2.0: system, CoAE, SVP, conditional MeanFlow, IDM, validity gap, KASO, Algorithm 1, and controlled ablation (3.1–3.4.5)
- 4 Training Data and Mixture Composition
- 5 Evaluation: zero-shot protocol, data/skill scaling, grounding/conflicts, and simulation (5.1–5.4)
- 6 Conclusion; 7 Limitations and Outlook; 8 Contributions and Acknowledgments; References
- Appendix A.1–A.3: all simulation protocols and comparisons
- Appendix B.1–B.7: architectures, sampling, training, deployment, and toy controls
- Appendix C.1–C.3: full taxonomy, both-embodiment task results, instruction prompts/results, and descriptor matching
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity verified as arXiv:2609.05588v1 [cs.RO], 4 September 2026. The exact title matches the catalog. The title-page byline is AgiBot Research Team; the catalog also appends individuals credited by role in Section 8, rather than listed as a separate title-page byline. Metadata preserves the collective byline. No later revision or alternative edition was supplied or reviewed.
- The supplied text's figure-image omission was addressed by inspecting the retained PDF: all 13 text chunks were read; PDF pages 1–25 and 30–39 were visually inspected, including every figure and table. Bibliography-only pages 26–29 were read as text.
- Separate supplemental material availability has not been fully verified. External project material, code, datasets, and videos were not inspected; no experiments were reproduced.
- No institutional affiliations are explicitly listed in the inspected identity/credit blocks; affiliations are omitted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title/byline and arXiv margin; p. 25, Section 8 contributor credits
The exact title matches the supplied catalog. The PDF identifies arXiv:2609.05588v1 [cs.RO], 4 September 2026, and the collective byline AgiBot Research Team. Catalog-listed individuals appear under contributor, advisor, leader, or supervisor roles on p. 25; no institutional affiliation block is provided.
Go to primary source ↓architecturePDF p. 3, Section 2.1; pp. 4–5, Section 3.1 and Figure 2; p. 10, Section 3.4.1
A decoupled SVP predicts completed future visual latents; a separate IDM consumes current/future latents and proprioception to predict actions. Dense actions are executed; sparse actions are auxiliary training targets. This differs from GE-Act 1.0's parallel action branch.
Go to primary source ↓coaePDF pp. 5–6, Sections 3.2–3.2.1, Eqs. (1)–(2), Figure 3
CoAE compresses 256×384 frames to 24 tokens with 512 channels using 64-fold spatial downsampling per axis. Compatible DC-AE weights are transferred and new parameters randomly initialized. Reconstruction and alignment to frozen SigLIP 2, V-JEPA 2.1, and DINOv3 supervise training.
Go to primary source ↓probesPDF pp. 6–7, Section 3.2.2, Table 1 and Figure 4
The action probe trains on 400 shelf-task samples and tests on 100 held-out samples with frozen encoders. CoAE/DC-AE/DINOv3/V-JEPA 2.1/SigLIP action MAEs are 0.01673/0.02573/0.01273/0.01487/0.03138. A VLM-conditioned caption probe uses 5,000 GenieSim-Instruction episodes; corresponding errors are 2.05/2.71/2.75/4.15/3.28%.
Go to primary source ↓svp-conditioningPDF pp. 8–9, Sections 3.3.1–3.3.3, Eqs. (3)–(5); p. 32, Table 6 and Appendix B.2
A frozen VLM contextualizes instruction tokens using the current head image. A learned gate fuses text-span states across layers for DiT cross-attention; the IDM receives no direct language. Dense and sparse visual targets capture near-term control and longer-horizon intent.
Go to primary source ↓meanflowPDF pp. 9–10, Section 3.3.4, Eqs. (6)–(14); p. 32, Appendix B.3
Conditional MeanFlow holds observation coordinates fixed, masks future-coordinate supervision and the derivative tangent, and uses mean/instantaneous velocity heads with a stopped-gradient JVP correction. Eq. (14) generates futures in one pass from noise at r=0, t=1.
Go to primary source ↓idmPDF p. 10, Section 3.4.1, Eq. (15); p. 32, Table 6 and Appendices B.2/B.4; p. 33, Appendix B.6
IDM uses action flow matching with recorded compatible futures during pretraining. Actions are standardized absolute joint positions in zero-padded 32-dimensional vectors. Deployment uses five Euler steps and executes the first 30 of 52 dense actions before replanning.
Go to primary source ↓validity-gapPDF pp. 10–12, Sections 3.4.2–3.4.3, Eqs. (16)–(18), Figures 5–6
Retaining video and recorded-future action losses protects pretrained objectives but does not couple an independently sampled future to the recorded action's outcome mode. Even a correct future marginal can create incompatible imitation pairs.
Go to primary source ↓kasoPDF pp. 12–14, Section 3.4.4, Figure 7, Eqs. (19)–(21), Algorithm 1
KASO compares active-IDM velocities under generated and recorded futures at a shared high-noise action input, using dense-token squared disagreement. Lowest-energy candidates are replayed from identical generation noises with gradients; fresh action noise/time is used for the selected flow-matching loss. Both pretraining losses remain.
Go to primary source ↓training-recipePDF pp. 32–33, Appendices B.1–B.6, Table 6
SVP/IDM have 2.51B/0.56B parameters; frozen Qwen3.5-2B has 2.72B. Defaults include bf16, ZeRO stage 2, AdamW, four dense/two sparse future frames, 30-Hz alignment, and a 52-action horizon. KASO uses N=4, k=1, probe time 0.95, loss coefficients (SVP,IDM,E2E)=(1,0.1,1), three accumulated micro-steps, and frozen CoAE/VLM. GPU models/counts and wall-clock training/inference timings are not supplied.
Go to primary source ↓data-mixturesPDF pp. 15–16, Section 4 and Figure 8
SVP/IDM/co-training mixtures contain 39,000/32,000/30,000 hours. SVP includes 3,000 hours of ego/human video; IDM includes 2,000 hours of rollout/failure data. Co-training shares are 61.7% G1-OP and 1.7% G2-90D; G1-OP is 47.4% of SVP pretraining. The stages reuse source categories and hours.
Go to primary source ↓robot-ablationPDF pp. 14–15, Section 3.4.5 and Table 2(a)–(b); p. 33, Appendix B.5, controlled ablation
All arms share full-scale pretrained components and a 300-hour alignment mixture including 30 G2-90D hours. Four-object evaluation uses 10 trials per target; single-block evaluation uses 25. E2E+PT/KASO follow scores are both 95%, four-object pick macro averages are 22.5/37.5%, and block success is 12/40%; E2E obtains 87.5%, 27.5%, and 12% respectively.
Go to primary source ↓real-protocolPDF p. 16, Section 5.1; p. 18, Section 5.2; pp. 33–35, Appendices B.6/C.1 and Table 7
One last checkpoint per data scale is evaluated without per-task fine-tuning, demonstrations, or checkpoint selection. The suite has 100 tasks in 20 groups, 10 trials per task/embodiment/scale, matched reset configurations, and held-out scenes, backgrounds, lighting, and physical object instances.
Go to primary source ↓scalingPDF pp. 18–19, Section 5.2.1 and Figure 10; pp. 36–37, Table 8
Nested co-training pools of 300/1,200/5,000/30,000 hours yield G1-OP suite success of 17.1/22.6/27.3/44.1% and G2-90D success of 13.4/21.0/23.5/31.1%. Improvements span 19/20 and 18/20 groups. Runs stop after an epoch or the compute limit; this is not matched-compute data scaling. Zip remains zero throughout both embodiments.
Go to primary source ↓skill-coveragePDF pp. 19–21, Section 5.2.2 and Figure 12
Across 19 skill groups, excluding Close, log training hours and logit G1-OP success have Pearson r=0.80 and Spearman rho=0.85; fitted slope is 1.94 logit units per decade. This is a cross-skill coverage association, not a randomized data-allocation intervention.
Go to primary source ↓grounding-resultsPDF pp. 21–22, Section 5.3.1 and Figures 13–14; pp. 37–39, Appendix C.3 and Table 9
The final G1-OP checkpoint is evaluated on 33 Pick and 26 Place instructions, five trials each. Overall Follow Score/SR are 83.1/72.9%. Object, color, direct position, and shape follow scores are at least 90% for both primitives; size is 82.5/65.7% and order is 13.3/26.7% for Pick/Place.
Go to primary source ↓descriptor-frequencyPDF p. 22, Figure 14 and discussion; p. 39, Appendix C.3, Descriptor frequencies and Table 10
Frequency estimates use lexical matching over 95,010 RefCOCOg expressions and approximately 3.7 million GE Pick/Place segments. Non-object categories overlap; object identity has corpus-specific residual definitions. Simulation is normally excluded but included for size. GE size/shape/order shares are 0.96/0.67/0.13%; association does not establish causation.
Go to primary source ↓instruction-conflictsPDF pp. 22–23, Section 5.3.2 and Figure 15
Selected rollouts show target switching, arm switching, and placing a cup in a shoebox despite a conventional association. The caption explicitly calls these illustrative and outside the quantitative evaluation.
Go to primary source ↓robotwinPDF p. 24, Section 5.4; p. 30, Appendix A.1 and Table 3
After Easy/Clean SFT, GE-Act reaches 60.52% on RoboTwin Hard versus pi0.5 at 47.90%; it leads five of six columns but trails pi0.5 on lighting (65.92 versus 69.20%).
Go to primary source ↓geniesimPDF pp. 30–31, Appendix A.2 and Table 4
After SFT on released training demonstrations, OOD GenieSim-Instruction average normalized score is 0.770 for GE-Act, 0.757 for ACoT-VLA, and 0.746 for pi0.5. GE-Act leads four of ten tasks rather than every category.
Go to primary source ↓liberoPDF p. 31, Appendix A.3 and Table 5
After standard LIBERO adaptation, LIBERO-Plus overall success across 10,030 instances is 80.4% for GE-Act, 84.4% for pi0.5, and 74.1% for StarVLA. Aggregation weights axes by instance counts. GE-Act has 60.5% background and 50.7% robot-state success.
Go to primary source ↓toyPDF pp. 11–12, Section 3.4.3 and Figure 6; pp. 33–34, Appendix B.7
The four-mode toy uses a deterministic IDM proxy, a 48-step generator, eight training candidates, and one evaluation sample. The displayed seed-0 behavior is an off-manifold sensitivity stress test; Appendix B.7 reports substantial variation across seeds 0–4 and rejects a seed-averaged benchmark interpretation.
Go to primary source ↓limitationsPDF p. 24, Section 7, Limitations and Outlook
The authors leave large-scale egocentric-video effects for future study and characterize the policy as low-level instruction-conditioned control without explicit deliberation, task decomposition, memory, or high-level planning.
Go to primary source ↓8.5 Primary sources
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation ↗
PDF · 23,064 extracted words
Source fingerprint
4e6e98d3a1c30739e968afff7be63898de8bec0442c15a2ab63706d6498c0248