Act2Goal: From World Model To General Goal-conditioned Policy
1. Paper overview
In one sentence: Act2Goal uses multi-scale visual-transition features to guide a separate action expert, improving goal-conditioned manipulation while leaving the exact temporal schedule and some adaptation details unresolved. e02e03e04e12e14
| At a glance | What to know |
|---|---|
| Research problem | Source description A final image specifies object configurations precisely, but direct goal-conditioned action prediction can lose track of progress over long tasks. Act2Goal introduces intermediate visual-transition features to connect local execution with the distant goal. e02 |
| Core mechanism | |
| A key reported result | MSTH ablation on whiteboard word writing: With MSTH: ID 0.95 / 0.90 / 0.90; OOD 0.93 / 0.90 / 0.88. Task execution success rate (fraction). ID/OOD words: short ≤3 letters, medium 4–6, long ≥7; real robot; appendix specifies 40 rollouts per experiment. Without MSTH: ID 0.95 / 0.35 / 0.10; OOD 0.60 / 0.20 / 0.00, ordered short/medium/long. The large long-word gains support multi-scale guidance. OOD short words also improve substantially; benefits are not confined to long words. Budget matching is unspecified. e12e09 |
| Reading caution |
Core contributions
- Source description
Hindsight goal relabeling turns self-collected transitions, including unsuccessful attempts, into training targets for optional LoRA adaptation without task rewards. e06
Figure 3. Layer-wise visual features guide a distinct action-generating network. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the two inputs across the top. Multi-view current and goal frames enter the video encoder on the left; robot state enters the action branch on the right. Follow the central cross-attention arrow from the Video DiT toward the Action DiT: this is the architectural connection that lets transition features condition motor prediction. Both stacks contain 28 blocks, while their widths and parameter counts differ. The bottom outputs use the same MSTH organization. The dashed return on the action side is consistent with iterative action refinement in Eq. (4); it should not be read as an environment-feedback sensor connection. e03e05e14
What it supports. The diagram supports a dual-system architecture despite end-to-end optimization. The action expert receives internal world-model features and proprioception, rather than only a final goal embedding. Stage 2 updates both modules through action loss, giving the predictive representation a direct control-training signal. The appendix specifies that deployment generates actions only.
Where the evidence stops. The figure also displays latent-frame outputs, and Section III.A describes visual denoising. Appendix A limits deployment to action generation; the precise deployment schedule for obtaining the conditioning features is not fully described. Decoded video is not established as a required inference output.
2. Motivation
2.1 The problem and the proposed response
A final image specifies object configurations precisely, but direct goal-conditioned action prediction can lose track of progress over long tasks. Act2Goal introduces intermediate visual-transition features to connect local execution with the distant goal. e02
2.2 What this reading follows
A goal photograph says where a robot should finish, but offers little direct guidance about the actions in between. Act2Goal learns that intermediate structure with a visual world model, then passes its features into an action generator. Its central design separates dense near-term control from sparse predictions farther toward the goal. The most informative evidence is the writing ablation: performance on long words changes sharply when this temporal structure is removed. Read the architecture and generated-frame examples together, then compare executed-task results and online learning curves. The appendix adds an essential qualification: deployment generates actions only, so the visual predictions should not be mistaken for a mandatory decoded-video planning loop. e02e03e04e12e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system label is supported by distinct Video and Action DiTs with directed feature cross-attention. The visual-planning/IDM category fits a separate action expert conditioned on imagined transitions, with a caveat: the paper defines feature-conditioned action flow matching rather than a standalone state-pair inverse-dynamics model. Joint training does not make this a single shared predictor. e03e04e05
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a target image into a control-conditioned representation
The world model is useful here because it supplies an intermediate description of how the current scene could approach the goal. Current and target frames are encoded without language conditioning, and world-model layers provide the action expert with transition features. The action expert additionally receives proprioception, so visual intent and robot state meet inside the motor predictor. Stage 1 trains visual and action flow matching together; Stage 2 removes visual loss while retaining gradient flow through both networks. Reader interpretation: this lets a predictive representation specialize for actions without requiring every useful internal feature to remain a faithful video-rendering feature. Appendix A strengthens that distinction by stating that inference generates only actions. The paper's visual-generation capability and its deployed action-generation pathway should therefore be described separately. e03e05e14
Figure 8. Generated frames illustrate the intended separation between local detail and distant progress. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each example horizontally, using the current observation at the far left and the goal image at the far right as endpoints. Three selected proximal frames precede three selected distal frames. The second row is an enlargement of the writing example above it, not a fourth independent trajectory. Its red boxes and connectors identify the writing region. In the two lower examples, red arrows highlight moving objects. The caption states that the bottom example's distal frames cover a longer horizon than those in the preceding example. These are selected visual predictions, not a complete frame-by-frame record of physical execution. e13e04e12
What it supports. The examples make the intended temporal roles visible: proximal frames depict nearby changes, while distal frames move the scene farther toward its target configuration. They support a qualitative account of the learned representation. Whether these predictions improve control is addressed separately by executed-task rates and the MSTH ablation.
Where the evidence stops. The frames carry no numeric timestamps. They cannot resolve Eq. (5)'s conflict: the text claims increasing intervals, but the printed logarithm has decreasing gaps before flooring. Nor do selected images quantify prediction accuracy, contact feasibility or robot success.
5.2 Separate what guides a plan from what the robot executes
MSTH gives local and distant predictions different jobs. Within the proximal horizon P, visual samples use stride r, while actions remain dense. Distal visual states and distal actions share M selected indices extending toward total horizon K. The appendix instantiates this as 54 proximal actions with 50 executed, plus nine distal actions that remain guidance. Reader interpretation: distant predictions can preserve information about the intended endpoint while a shorter executed segment allows subsequent observations to correct the trajectory. Table III supports the combined design, particularly on long words, but does not isolate that interpretation. A further reproduction obstacle is Eq. (5): its printed logarithmic schedule has decreasing unfloored gaps, despite prose claiming increasing gaps. Both formulations must remain visible until the intended schedule is clarified. e04e12e14
5.3 Understand what hindsight learning learns from a failure
Suppose the robot attempts a target image but ends an executed step elsewhere. The online procedure retains its starting observation, proprioception, action and achieved observation, then replaces the requested goal with that achieved observation. The transition now provides an example of reaching a state the robot actually visited. This removes the need for a task-reward label during updating. Only LoRA parameters are trained, and the replay buffer is cleared afterward. Reader interpretation: this can improve control around newly visited states, but does not guarantee progress toward the original goal. Figure 6's failed-only decline illustrates why held-out original-goal evaluation matters. Algorithm 1's direct squared action error also differs from the prose's Stage-2 flow-matching prescription. Reward-free updates still coexist with manual physical resets and later human success scoring. e06e10e09e16
5.4 Training and inference
During training
Stage 1 jointly optimizes visual and action flow matching, weighting action loss by 0.1. Stage 2 uses action flow matching alone, with gradients through both modules rather than a frozen world model. Training combines AgiBot World with a small proprietary dataset. e05e15
Online adaptation records each executed transition and replaces its original goal with the achieved observation. Once the buffer fills, only LoRA parameters update; base weights stay frozen. The buffer is cleared afterward. The prose specifies Stage-2-style updates, although Algorithm 1 prints direct squared action error. e06
During inference
The appendix specifies 54 proximal predictions, of which 50 execute, plus nine unexecuted distal actions. It states that only actions are generated at inference, so decoded video should not be assumed to be a deployment prerequisite despite the generic visual-flow equations. New observations support subsequent closed-loop inference steps. e03e04e06e14
5.5 Implementation flow
- Encode visual endpoints
A VAE encodes current and goal frames. The goal is concatenated with the current observation along the hidden-state sequence; Genie Envisioner's language conditioning is removed. The 1.6B world model and 160M action expert each have 28 DiT blocks. e03
- Condition actions on predicted transitions
World-model layer features enter the corresponding action blocks through cross-attention. Proprioception and noisy actions provide the remaining action-generation inputs. This is feature-conditioned motor prediction, rather than a search over actions scored by a simulator. e03
- Separate temporal roles
Proximal visual states are subsampled with stride r, but proximal actions cover every timestep. Distal visual states and actions share selected future indices. Their intended role is long-term guidance; they are not executed. e04
6. Experiments & results
Act2Goal converts current and target images into world-model features that guide a separate action expert. Dense near-term control and sparse distal predictions support long-horizon manipulation, with optional hindsight-based LoRA adaptation. Real-robot results and a writing ablation are strong within the tested setups, while sampling, timing and implementation ambiguities constrain reproduction.
6.1 Read the original evidence
Table I. Simulation gains are broad, with a clear Hard Move Can exception. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare rows within each Easy or Hard block, keeping task columns fixed. The entries are task-execution success fractions, and bold type identifies the best reported value in that setting. The appendix specifies 90 automatically scored rollouts per experiment. Goals come from successful trajectories under the paper's fixed-seed construction. DP-GC uses current and goal SigLIP features; pi0.5-GC combines current and goal images with a fixed language condition. Move across the Hard block carefully: the first bold entry belongs to pi0.5-GC, whereas the remaining three belong to Act2Goal. These are task-level comparisons, not a single averaged generalization score. e07e09
What it supports. Act2Goal leads all four Easy tasks and three of four Hard tasks. Its Hard Pick Bottles rate is 0.43 versus 0.06 for pi0.5-GC. Hard Move Can reverses the ranking: Act2Goal reaches 0.13 and pi0.5-GC 0.42. The original table therefore supports a more qualified conclusion than the nearby sweeping superiority language.
Where the evidence stops. Four selected tasks and one goal-construction protocol provide limited coverage of generalization. The paper gives no confidence intervals or repeated-training-seed statistics, and it does not establish that baseline pretraining resources are matched.
Table II. Offline gains persist under the paper's task-specific distribution shifts. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top block as in-domain performance and the lower block as out-of-domain performance. The three columns represent different changes at test time: unseen word combinations, changed dessert/plate/background appearance, and insertion transfer from a metal workpiece to a bottle and cup holder. All entries precede online autonomous improvement. Appendix C specifies 40 manually scored physical rollouts per experiment. Compare Act2Goal with the baseline directly above it only after checking the same task and block. The task families themselves have data in offline training, although no additional task-specific supervised fine-tuning is performed for these tests. e08e09e17
What it supports. Act2Goal reports ID rates of 0.93, 0.75 and 0.45, and OOD rates of 0.90, 0.48 and 0.30 for writing, plating and insertion. It leads all listed baselines. The OOD drop is relatively small for writing but substantial for plating and insertion, showing that generalization strength depends on the task.
Where the evidence stops. The rates are rounded point estimates without uncertainty bars. Physical evaluation uses a manually pre-grasped, taped marker and silicone desserts. Those setup choices matter when interpreting autonomous manipulation and transfer to natural objects.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0 goal-conditioned manipulation Four Easy/Hard tasks; fixed-seed goal-image construction; 90 automatically scored rollouts per experiment. | Act2Goal Easy: 0.62 / 0.80 / 0.64 / 0.52; Hard: 0.13 / 0.43 / 0.13 / 0.15. Task execution success rate (fraction) | pi0.5-GC Easy: 0.54 / 0.13 / 0.16 / 0.30; Hard: 0.42 / 0.06 / 0.04 / 0.06. Order: Move Can, Pick Bottles, Place Cup, Place Shoe. Leads all Easy tasks and three Hard tasks, but loses Hard Move Can. This exception limits the prose's sweeping superiority claim. e07e09 |
| Offline real-world goal-conditioned manipulation Genie-01; writing / dessert plating / insertion; ID and OOD; offline training only, without task-specific SFT; 40 manually scored rollouts per experiment. | Act2Goal ID: 0.93 / 0.75 / 0.45; OOD: 0.90 / 0.48 / 0.30. Task execution success rate (fraction) | pi0.5-GC ID: 0.23 / 0.18 / 0.00; OOD: 0.20 / 0.05 / 0.00. OOD tests vary word combinations, visual appearance or insertion objects. Task-family training data are present; these are not wholly unseen-task evaluations. e08e09 |
| MSTH ablation on whiteboard word writing ID/OOD words: short ≤3 letters, medium 4–6, long ≥7; real robot; appendix specifies 40 rollouts per experiment. | With MSTH: ID 0.95 / 0.90 / 0.90; OOD 0.93 / 0.90 / 0.88. Task execution success rate (fraction) | Without MSTH: ID 0.95 / 0.35 / 0.10; OOD 0.60 / 0.20 / 0.00, ordered short/medium/long. The large long-word gains support multi-scale guidance. OOD short words also improve substantially; benefits are not confined to long words. Budget matching is unspecified. e12e09 |
| Online adaptation on OOD Plug-In Operation Real robot with hindsight-based LoRA updates; checkpoints evaluated after training; appendix's real-world protocol uses 40 rollouts per experiment. | 0.30 before adaptation to 0.90 afterward. Reported task execution success rate (fraction) | Same policy before online adaptation; gain is 60 percentage points. A source-reported endpoint improvement, without a detailed plug-in learning curve or task-specific timing in the PDF. It must not inherit drawing-example timings. e11e09 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table III. The writing ablation connects multi-scale structure to executed long-horizon success. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each ID or OOD block, compare the paired rows before moving across word lengths. Short means three letters or fewer, medium means four to six, and long means seven or more. The paper describes the without-MSTH alternative as fixed-horizon action chunking. Start with ID short words: both variants reach 0.95. Moving right exposes a large separation on longer words. Then inspect OOD short words, where the variants already differ substantially. This avoids repeating the body text's broad suggestion that short-word performance is always comparable. These numbers assess complete robot writing attempts, not visual quality of generated word images. e12e09
What it supports. For long words, MSTH raises reported ID success from 0.10 to 0.90 and OOD success from 0.00 to 0.88. OOD short-word success also rises from 0.60 to 0.93. The pattern supports the usefulness of the overall multi-scale mechanism, with particularly large gains as the required sequence length grows.
Where the evidence stops. The ablation does not document a matched token, horizon or compute budget, and no separate test isolates distal visual features from distal action prediction. It supports MSTH as a combined design, not a unique causal explanation for every gain.
Figure 6. All-rollout adaptation finishes highest; failed-only gains are not sustained uniformly. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two panels as different experiments. On the left, A through D refer to the four hard-mode configurations illustrated in Figure 5, corresponding to Move Can Pot, Pick Dual Bottles, Place Empty Cup and Place Shoe. The axis displays Iter 0, 100 and 200 with two round labels. On the right, rollout selection is the variable: square, triangle and circle markers denote successful-only, failed-only and all-rollout data. This panel extends to Iter 300 and three rounds. Follow the triangles to the final point rather than stopping at their earlier improvement. The appendix says checkpoints were evaluated after adaptation finished. e10e06e09
What it supports. All-rollout training has the highest final plotted success. Failed-only data can initially improve performance, consistent with learning from achieved states even when intended goals are missed. Its final decline shows that this benefit is not monotonic. The source's broad description of improvement should therefore be read together with the full curve.
Where the evidence stops. The caption attributes three rounds and convergence to the left panel, but only two update rounds are drawn there. No uncertainty bands or selection-strategy sample counts are supplied, so the figure alone cannot distinguish relabeling benefits from differences in data coverage.
7. Analysis & limitations
7.1 What the evidence leaves open
Reported rates lack uncertainty estimates. The experiments cover four simulation tasks and three physical task families; they do not establish general robustness across robot platforms. Generated-video examples are qualitative and do not measure dynamics accuracy. e07e08e09e13
MSTH's printed logarithmic equation conflicts with its increasing-gap description. Figure 6's left plot shows two update rounds although its caption discusses three; failed-only training drops at the final right-panel point. Drawing timing also conflicts: the body says 15 minutes, Figure 7 shows 10/20/25-minute labels, and the appendix describes five-minute rounds. e04e10e11e16
Reward-free updating still relies on physical setup support: manual restoration is allowed, the marker is pre-grasped and taped, and desserts are silicone toys. These conditions narrow what autonomous deployment and object generalization mean here. e06e17
7.2 Questions for discussion
- Would a token- and compute-matched uniform distal schedule recover the writing gains?
- Does relabeling improve held-out original-goal success beyond adapting to newly visited states?
8. Reproducibility audit
8.1 Requirements and known gaps
Full retraining requires the pretrained world model and both training datasets. Reported compute is seven days on 16 A800 GPUs for Stage 1 and 48 hours on 16 A800s for Stage 2. Proprietary-data composition, optimizer/batch settings and software versions remain unspecified. e15
The deployment reference is Genie-01 plus RTX 4090, LoRA rank 64, buffer size 20 and 10 epochs per round. Reported latency is 200 ms for 50 executable actions; this does not specify action frequency. LoRA insertion sites, learning rate, flow-step count and exact horizon-selection implementation need clarification. e03e04e16
Reader-proposed checks should match data and compute while varying distal guidance, then compare hindsight relabeling against original-goal updates on identical rollouts. Evaluation should use held-out goals, repeated runs and explicit reset/time accounting. e04e06e10e12e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate temporal guidance under matched prediction budgets
Reader-proposed experiment, not performed: train writing variants with the same demonstrations, initialization, proximal execution length, prediction-token count and optimization budget. Compare MSTH with uniform distal sampling and a proximal-only control that preserves the total token budget. Include both the printed Eq. (5) schedule and a separately labeled increasing-gap schedule to expose the source ambiguity. Evaluate ID/OOD words in all three length bins with repeated training runs. The temporal-anchoring interpretation predicts a larger benefit on long words that survives budget matching; disappearance of that advantage would weaken the mechanism claim. Log the actual sampled indices and distinguish visual-feature from distal-action ablations. e04e12e14e15
Check 2: Separate hindsight relabeling from exposure to new states
Reader-proposed experiment, not performed: use the same collected transitions and equal gradient budgets for achieved-goal relabeling, original-goal updates and a frozen-policy control. Add matched-count successful-only and failed-only subsets to distinguish selection from data volume. Keep LoRA rank, buffer size and update epochs fixed to the reported deployment settings; document which online loss resolves the Algorithm 1/prose mismatch. Score held-out original goals after each saved checkpoint, report uncertainty across repeated runs, and account separately for rollout, updating and manual reset time. If relabeling helps only recycled achieved goals, or gains vanish after sample matching, the claimed adaptation mechanism would need a narrower interpretation. e06e09e10e11e16
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version page; all Figures 1–8; Tables I–III; Section III equations and Algorithm 1; experiment definitions and online-result text; and Appendix A–C hardware, training, execution and evaluation details. Six final original crops were individually inspected with readable labels, table headers and legends. Page 2's malformed extracted label is visually pi0.5-GC. The printed Eq. (5), Figure 6 round labels and Figure 7 timing labels were checked against adjacent prose; discrepancies are retained in the report. Pages 9–10 contain references and were read in the complete text pass, but not rendered. Separate project-page material, supplements and code remain outside the supplied reading scope.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 11. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, byline, affiliation, version and abstract
- PDF pp. 1–3: I. Introduction and II.A–C. Related Works
- PDF pp. 3–5: III.A–D. architecture, MSTH, offline training and online improvement
- PDF pp. 5–8: IV.A–C. generalization, adaptation and MSTH experiments
- PDF p. 8: V. Conclusion
- PDF pp. 9–10: References
- PDF p. 11: Appendix A–C and Acknowledgments
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Some PDF font mappings yielded unpaired Unicode surrogates; replacement characters are recorded by page in extractionNotes and require PDF visual verification.
- Identity/version scope: the inspected title and all eight authors match the catalog; this is arXiv:2512.23541v1 dated 29 December 2025. The manifest identifies the previous HTML artifact as the same revision, but that artifact was not supplied for independent comparison; no later edition was consulted.
- The supplied text spans all 11 pages and all five chunks were read. Original PDF pages 1–8 and 11 were visually inspected, resolving the page-2 replacement character as the pi0.5-GC label. Reference-only pages 9–10 were read as text.
- Separate supplements, project-page details, code, datasets and checkpoints were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, byline, affiliation and arXiv strip
Title: Act2Goal: From World Model To General Goal-conditioned Policy. Authors in order: Pengfei Zhou, Liliang Chen, Shengcong Chen, Di Chen, Wenzhi Zhao, Rongjun Jin, Guanghui Ren, Jianlan Luo. All are affiliated with Agibot Research. Zhou and Liliang Chen share equal contribution; Luo is corresponding author. The strip identifies arXiv:2512.23541v1, 29 Dec 2025.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Section I and Figure 2
The paper motivates visual goals and intermediate visual dynamics for long-horizon manipulation, combining a goal-conditioned world model, Multi-Scale Temporal Hashing and optional hindsight-based adaptation.
Go to primary source ↓e03PDF p. 4, Figure 3 and Section III.A, Eqs. (1)–(4)
Current and goal observations are VAE-encoded; language conditioning is removed. A 1.6B Video DiT passes layer-wise features through cross-attention to a 160M Action DiT; both have 28 blocks. The action flow model conditions on world features and proprioception. The equations describe iterative latent and action refinement.
Go to primary source ↓e04PDF p. 4, Section III.B, Eq. (5) and following paragraphs
MSTH uses proximal visual stride r, dense proximal actions, and distal visual/action indices d_m = P + floor((K-P) log(m+1)/log(M+1)). K is total imagined length, P proximal horizon and M distal samples. The prose claims increasing temporal gaps. Only proximal actions execute.
Go to primary source ↓e05PDF pp. 4–5, Section III.C, Eqs. (6)–(8)
Stage 1 jointly trains visual and action flow matching with action-loss weight 0.1. Stage 2 uses only action flow matching and propagates gradients through the entire model, including the world model.
Go to primary source ↓e06PDF p. 5, Algorithm 1 and Section III.D
Transitions store observation, proprioception, action and resulting observation. Achieved end-of-step observations replace intended goals regardless of success. Buffer-triggered LoRA updates freeze base parameters and then clear the buffer. The prose specifies Stage-2-style training; Algorithm 1 instead prints squared policy-to-action error. Manual restoration or modification of the physical setup can be required.
Go to primary source ↓e07PDF pp. 5–6, Section IV.A; p. 6, Table I, all rows
RoboTwin 2.0 uses goal images from successful trajectories under a fixed environment seed. Act2Goal Easy rates are 0.62/0.80/0.64/0.52 versus pi0.5-GC 0.54/0.13/0.16/0.30; Hard rates are 0.13/0.43/0.13/0.15 versus 0.42/0.06/0.04/0.06, ordered Move Can, Pick Bottles, Place Cup, Place Shoe. DP-GC and HyperGoalNet score zero throughout Hard mode. DP-GC uses current/goal SigLIP features; pi0.5-GC uses goal/current images and fixed language conditioning.
Go to primary source ↓e08PDF pp. 6–7, Section IV.A; p. 6, Figure 4 and Table II
Real tasks are writing, dessert plating and insertion. Their task data appear in offline training, without task-specific SFT. ID/OOD writing uses seen/unseen word combinations; training includes 200 words. OOD plating changes visual appearance; OOD insertion transfers from a metal workpiece to a drink bottle and cup holder. Act2Goal ID rates are 0.93/0.75/0.45, OOD 0.90/0.48/0.30; pi0.5-GC gives 0.23/0.18/0.00 and 0.20/0.05/0.00. Table II uses offline imitation only.
Go to primary source ↓e09PDF p. 11, Appendix C, Testing Metric
Real rates use 40 manually labeled rollouts per experiment; simulation uses 90 automatically scored rollouts. Online checkpoints are saved after each round and evaluated individually after adaptation concludes. No confidence intervals or repeated-training-seed statistics are supplied.
Go to primary source ↓e10PDF p. 7, Section IV.B and Figures 5–6
Four hard-mode scenarios support online adaptation. Figure 6 left displays Iter 0, 100 and 200 with two labeled rounds, while the caption/body discuss approximately three rounds and convergence. The right panel reaches Iter 300: all-rollout training finishes highest; failed-only training improves initially but falls sharply at the last point. The body reports a maximum gain up to eightfold.
Go to primary source ↓e11PDF p. 7, Section IV.B, real-world paragraph and Figure 7
The body reports OOD Plug-In success increasing from 0.30 to 0.90, referring elsewhere for details. Drawing examples qualitatively improve. The body says 15 minutes, whereas Figure 7 labels later drawing snapshots 10, 20 and 25 minutes.
Go to primary source ↓e12PDF p. 8, Table III and Section IV.C
Writing success without/with MSTH is ID short 0.95/0.95, medium 0.35/0.90, long 0.10/0.90; OOD short 0.60/0.93, medium 0.20/0.90, long 0.00/0.88. Short means at most 3 letters, medium 4–6, long at least 7. Without MSTH uses fixed-horizon action chunking. Detailed matching of horizon, token and compute budgets is not supplied.
Go to primary source ↓e13PDF pp. 7–8, Section IV.C; p. 8, Figure 8 and caption
Three generated head-view clips show current and goal images with three selected proximal and three selected distal frames. Row two enlarges row one's red boxes; red arrows in lower rows mark object motion. The bottom example's distal frames represent longer horizons than the preceding example. No quantitative video quality or dynamics metric is provided.
Go to primary source ↓e14PDF p. 11, Appendix A, Model
Four latent frames, two proximal and two distal, decode into nine proximal and nine distal visual frames. The action expert outputs 54 proximal actions, executing 50, and nine distal actions for guidance only. At inference only actions are generated.
Go to primary source ↓e15PDF p. 11, Appendix A, Training
Training uses AgiBot World and a small proprietary dataset. Stage 1 fine-tunes pretrained 1.6B Genie Envisioner for 7 times 24 hours on 16 A800 GPUs; Stage 2 takes 48 hours on 16 A800 GPUs. Dataset mixture counts, optimizer settings and software versions are not specified.
Go to primary source ↓e16PDF p. 11, Appendix A, Deployment and Online Learning
Deployment uses AgiBot Genie-01 and NVIDIA RTX 4090. LoRA rank is 64; buffer size is 20; each round uses 10 epochs and reportedly takes five minutes including rollout, backpropagation and resetting. Inference latency is 200 ms for 50 executable actions.
Go to primary source ↓e17PDF p. 11, Appendix B, Real World Task Setup
The marker is manually placed in the gripper and taped to reduce slipping. Dessert objects are silicone toys. Bearing insertion uses a bearing over 2 kg, approximately 1 cm base diameter and approximately 1.5 cm hole diameter.
Go to primary source ↓8.5 Primary sources
Act2Goal: From World Model To General Goal-conditioned Policy ↗
PDF · 7,007 extracted words
Source fingerprint
8b27463858672122c331db3b8be6ad8ecaea0b8a6d033ba5e4d28ae429d22308