EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
1. Paper overview
In one sentence: A fixed scene anchor, refreshed generated memory and metric action attention make egocentric synthesis more useful for robot training, while still requiring supplied trajectories and a separate control model. problemarchitectureoapma3dablationrobot-resultsdownstream
| At a glance | What to know |
|---|---|
| Research problem | Source description Synthetic manipulation data must preserve the relationship between images and supplied actions while changing appearance. The authors diagnose uncontrolled camera/gripper motion and drifting scene identity in baseline examples; these failures can corrupt the supervision obtained from otherwise plausible videos. problem |
| Core mechanism | Source description OAPM separates persistent first-frame geometry from replaceable recent scene state, allowing generated history to update conditioning without overwriting the anchor. oapm |
| A key reported result | OOD real-robot manipulation after synthetic augmentation: 400 real + 400 synthetic: single-arm 84%, dual-arm 70%. Task success rate (%) ↑. Tianji M6; independently fine-tuned LingBot-VA; four tasks per arm configuration, 25 trials each, held-out appearances/layouts. 400 real: 77%/53%; 400 synthetic: 76%/56%. ID real→mixed: 84%→88% single-arm and 72%→76% dual-arm. Mixed-data OOD gains are 7 and 17 percentage points. OOD cube stacking improves 56%→72% and bimanual pick-and-place 56%→80%. The larger mixed dataset leaves data-volume effects unresolved. robot-resultsrobot-protocoldownstream |
| Reading caution | Reader analysis The paper has no dedicated limitations section. Evaluation leaves total generation-benchmark size and uncertainty unspecified. The 50-clip hand metric cannot establish gripper accuracy; Kimi scoring lacks a stated frame count, judge-repeat protocol or human calibration. The unexplained Mask2IV asterisk in Table 1 is retained without interpretation. generationmetrics |
Core contributions
- Source description
OAPM separates persistent first-frame geometry from replaceable recent scene state, allowing generated history to update conditioning without overwriting the anchor. oapm
- Source description
A3D-RoPE adds metric skeleton geometry to video–skeleton cross-attention, retaining the pretrained self-attention positional path. a3d
- Source description
Real-robot augmentation experiments connect improved synthesis to downstream generalization across held-out appearances and layouts. robot-protocolrobot-results
Figure 3. Two geometry pathways condition a video generator whose output can become training data. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left: text specifies the task, video latents carry appearance, skeleton latents carry action control, and VGGT-Ω extracts the initial scene anchor. Follow the upper row through causal DiT chunks and the frame-appending strip. The lower-right inset distinguishes the fixed anchor from the recent slot; these feed scene tokens into gated cross-attention. The lower-left inset assigns x, y and z rotary phases to query/key features. Read that inset with Eqs. (7)–(9) and (15): queries come from video, keys/values from skeleton tokens, and the metric adapter writes only to skeleton-supported patches. The pretrained self-attention positional path is preserved. architectureoapma3dalgorithmdownstream
What it supports. The design gives persistent scene identity and current action geometry different conditioning routes. Algorithm 1 confirms that memory refresh follows block commitment and replaces only the recent snapshot. The arrows therefore describe feedback within video synthesis; a separate downstream model must learn and execute robot actions.
Where the evidence stops. The figure is schematic: it does not specify adapter-layer indices, a numerical refresh stride or which pretrained parameters are trainable. The VAE is explicitly frozen, but a complete freezing specification is absent.
2. Motivation
2.1 The problem and the proposed response
Synthetic manipulation data must preserve the relationship between images and supplied actions while changing appearance. The authors diagnose uncontrolled camera/gripper motion and drifting scene identity in baseline examples; these failures can corrupt the supervision obtained from otherwise plausible videos. problem
2.2 What this reading follows
A generated manipulation video becomes useful training data only if its pixels remain aligned with the action labels. EgoGenesis addresses that requirement through two complementary mechanisms: OAPM remembers both the initial scene and its recent generated state, while A3D-RoPE expresses skeleton control in camera-aware metric coordinates. This reading follows those information paths, checks their controlled ablations, and then examines the separate real-robot experiment. The reported OOD gains are substantial, especially for dual-arm tasks, but the comparison also changes dataset size. Read image fidelity, reconstructed geometry and executed task success as three related kinds of evidence with different limits. problemarchitectureoapma3dablationrobot-resultsdownstream
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Benchmarks & simulators. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work classification fits a neural simulator and data-generation method with explicit 3D scene representation. The generator takes actions as conditions and predicts video; a separately initialized downstream model learns control. This supports leaving the joint future/action and inverse-dynamics quadrants not applicable. Two conditioning adapters do not establish a One Model policy architecture. architecturealgorithmdownstream
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep an anchor without freezing the scene in its initial state
Imagine removing a lid while the camera moves. The first image is a useful reference for the cups and room, but it becomes an outdated description of where the lid should be. OAPM preserves that image-derived anchor while making the recent slot replaceable. Before generating a block, VGGT-Ω encodes both slots into memory tokens; selected transformer layers consult them through gated cross-attention. Only after the block is denoised and committed can its latest decoded frame refresh recent memory. Thus the next block can see the changed interaction state without replacing the initial reference. Reader interpretation: this division reduces a stability-versus-update tension, but refresh can also reintroduce generated mistakes. The qualitative memory ablation and the controlled table test complementary parts of that reasoning. oapmalgorithmmemory-visualablation
5.2 Turn a skeleton image into coordinates that attention can use
A rendered skeleton identifies image regions but does not by itself tell attention the metric depth of the moving manipulator. Appendix A constructs coordinates by projecting patch centers onto valid skeleton edges, interpolating perspective-correct depth, and blending edge support. Camera unprojection then expresses each supported patch in the reference-camera frame. A3D-RoPE uses those x, y and z coordinates to rotate channel pairs in video queries and skeleton keys before their cross-attention interaction. The same coordinate construction supplies the selected query and key entries; values carry skeleton features. Unsupported background patches do not receive this adapter update. Reader interpretation: the architecture gives an explicit geometric route for trajectory conditioning, but its reliability depends on the quality and coordinate consistency of the supplied camera and skeleton estimates. coordinatesa3darchitecture
5.3 Trace generated supervision into a different robot model
The robot experiment tests whether resimulated demonstrations improve a downstream learner. EgoGenesis takes a demonstration's scene anchor, camera trajectory, skeleton or end-effector track and prompt, then changes visual conditions while preserving its action-label space. LingBot-VA is independently initialized from its own released checkpoint for each training-data condition; generator weights do not become policy weights. The downstream learner receives synchronized head and wrist observations, and the bimanual experiment uses joint and gripper action channels. The paper fixes its training and inference settings while comparing real, synthetic and mixed trajectories. Reader interpretation: the resulting OOD success gains are stronger practical evidence than visual plausibility alone, but the larger mixed dataset prevents attributing the entire gain specifically to synthetic diversity. A matched-budget control is the natural next test. robot-protocoldownstreamrobot-results
5.4 Training and inference
During training
Generator training uses 210K clips: 100K each from EgoDex and AgiBot, plus 4K RoboTwin, 5K Real-world Ego and 1K DexJoCo. Clips are standardized to 81 frames at 16 FPS and 832×480, with disjoint clips/trajectories across train and test. The source calls the mixture source-balanced but does not give sampling weights. training
Two 6,000-step stages run on eight A100 GPUs: supervised fine-tuning with preprocessed memory, then autoregressive training with generated recent frames. Appendix B describes two teacher-forced clean context blocks, random target windows and one denoised target. The exact pairing of generated recent memory with clean cached context is unspecified. training
During inference
Initialize the anchor from the first frame and leave recent memory and cache empty. Sample each block from noise under current conditions, integrate flow, commit the denoised latent to the causal cache, and refresh recent memory from the latest decoded generated frame at the configured stride. This feedback is generated history, not a new physical observation. algorithmflow
5.5 Implementation flow
- Build scene conditioning
The frozen VAE maps video to latents, and the first video frame is pinned during generation. OAPM uses pretrained VGGT-Ω reconstruction features, coordinates and confidence from an immutable first-frame anchor and a replace-only recent snapshot. Selected DiT layers read the concatenated slots through gated cross-attention. architectureoapm
- Attach metric action coordinates
Skeleton edges supply perspective-correct patch depths with Gaussian support. Unprojection and camera transforms express supported patches in the anchor-camera frame. Video queries and skeleton keys are split into x/y/z channel groups and rotated; values remain the action features. Only valid skeleton-supported positions receive this adapter update. coordinatesa3d
- Resimulate demonstrations
The generator changes scene or appearance conditions while retaining the action-label space. The resulting trajectories fine-tune LingBot-VA, initialized independently from its own checkpoint. The generator itself neither infers actions from a desired future nor executes a robot command. robot-protocoldownstreamalgorithm
6. Experiments & results
EgoGenesis resimulates manipulation demonstrations with an autoregressive video generator conditioned on camera and action geometry. A permanent scene anchor and refreshed recent memory stabilize the scene; metric rotary attention aligns skeleton control. Synthetic videos improve a separately trained robot model, but video plausibility and executed success remain distinct evidence streams (architecture, oapm, a3d, robot-results, downstream).
6.1 Read the original evidence
Table 1. Generation quality improves across several metrics, with different evidential meanings. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the direction arrows before comparing rows: PSNR, SSIM, physical faithfulness and temporal consistency increase with the reported quality, whereas LPIPS and keypoint error decrease. The highlighted EgoGenesis row leads six columns. Subject consistency is the exception: EGOSIM scores 0.9101 versus 0.8923. These columns are not interchangeable measures of control. Appendix F defines Kpt.Err as image-diagonal-normalized 2D hand error on a 50-clip EgoDex subset, and physical faithfulness as a normalized Kimi K2.7 judgment. Subject and background consistency use temporal DINO and CLIP feature similarity, respectively. Blue bold and green underlined cells mark first and second place. generationmetrics
What it supports. EgoGenesis reports LPIPS 0.2399 and Kpt.Err 0.0501, compared with EGOSIM's 0.2750 and 0.0811. Together with PSNR 21.8609, these values support improved appearance and hand-trajectory alignment on the tested data. The subject-consistency exception prevents interpreting the table as uniform dominance across every reported dimension.
Where the evidence stops. The full benchmark size and uncertainty are unspecified. The hand-only keypoint subset cannot establish gripper alignment, and a judge score cannot certify physical correctness. Table 1's Mask2IV asterisk is not explained in the supplied text.
Table 6. Synthetic augmentation improves measured robot success, with task-level variation. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) covers single-arm skills and panel (b) dual-arm skills. Within each panel, compare real-only, mixed and synthetic-only rows separately for ID and OOD; all entries are success percentages. Each task has 25 trials, so one success changes a task score by four percentage points, while Avg. SR averages four tasks. The source holds LingBot-VA architecture, optimization and inference settings fixed across data conditions. OOD changes object appearance and initial or goal layout on the same Tianji M6 setup. Read the per-task columns before the averages: augmentation has a larger reported effect on OOD cube stacking than on already-high pointing success. robot-resultsrobot-protocoldownstream
What it supports. The mixed condition raises OOD averages from 77% to 84% for single-arm tasks and from 53% to 70% for dual-arm tasks. Cube stacking rises from 56% to 72%, while bimanual pick-and-place rises from 56% to 80%. Synthetic-only scores of 76% and 56% show that mixing is the strongest tested condition.
Where the evidence stops. The mixed condition contains 800 trajectories, whereas each unmixed condition contains 400. Without an 800-real control and repeated training runs, the table does not fully separate synthetic diversity, data volume and training variability.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Held-out action-conditioned video generation Mixed held-out robot/gripper/human-hand trajectories; common scene/action conditioning, 81-frame clips. Kpt.Err uses only 50 EgoDex clips. | 21.8609; 0.8509; 0.2399; 0.0501; 0.8278; 0.8923/0.9546. PSNR ↑; SSIM ↑; LPIPS ↓; Kpt.Err ↓; Phys.Faith ↑; subject/background consistency ↑ | EGOSIM-14B: LPIPS 0.2750, Kpt.Err 0.0811, subject consistency 0.9101. Best in six of seven reported metrics; second in subject consistency. The keypoint score is normalized 2D error and physical faithfulness is a model-judge rating. generationmetricstraining |
| Controlled scene-memory and action-encoding ablations Wan2.2-5B-Control backbone; training and complementary component held fixed within each block. | Full model: PSNR 21.8609, LPIPS 0.2399, Kpt.Err 0.0501. PSNR ↑; LPIPS ↓; Kpt.Err ↓ | Anchor-only: 20.4135/0.2533/0.0532. With OAPM fixed, RoPE and PRoPE yield Kpt.Err 0.07719 and 0.06135. Both additions help the reported generation metrics. This is not a complete factorial test of interactions or a downstream robot ablation. ablation |
| Geometric drift over an 80-frame rollout OAPM and training fixed; VGGT-Ω estimates on 15 non-anchor frames. | At frame 80: depth/camera error reduced 79.30%/78.26% versus RoPE; 66.46%/49.04% versus PRoPE. Accumulated depth and camera error ↓ | RoPE and camera-aware PRoPE positional alternatives. Depth uses clip-scale-aligned AbsRel in action regions; camera error uses first-camera-relative Plücker coordinates. These reconstructed-geometry proxies do not establish measured physical accuracy. drift |
| OOD real-robot manipulation after synthetic augmentation Tianji M6; independently fine-tuned LingBot-VA; four tasks per arm configuration, 25 trials each, held-out appearances/layouts. | 400 real + 400 synthetic: single-arm 84%, dual-arm 70%. Task success rate (%) ↑ | 400 real: 77%/53%; 400 synthetic: 76%/56%. ID real→mixed: 84%→88% single-arm and 72%→76% dual-arm. Mixed-data OOD gains are 7 and 17 percentage points. OOD cube stacking improves 56%→72% and bimanual pick-and-place 56%→80%. The larger mixed dataset leaves data-volume effects unresolved. robot-resultsrobot-protocoldownstream |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Within-block controls separate recent-memory benefits from positional-encoding benefits. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the italic section labels as experimental controls. In the scene-memory block, A3D-RoPE stays fixed while the recent refresh is added to the first-frame anchor. In the positional-encoding block, OAPM stays fixed while RoPE, PRoPE and A3D-RoPE are compared. The repeated full-model row is the same configuration, not another independent replication. The first row supplies an autoregressive Wan2.2-5B-Control reference. Use the keypoint column alongside fidelity: PRoPE is already close to the full model in PSNR, but its Kpt.Err remains higher. The full appendix table exposes this distinction more clearly than the compact three-metric table on page 6. ablationmetrics
What it supports. Adding recent refresh raises PSNR from 20.4135 to 21.8609 and lowers LPIPS from 0.2533 to 0.2399. With OAPM fixed, Kpt.Err falls from 0.07719 with RoPE and 0.06135 with PRoPE to 0.0501 with A3D-RoPE. The reported advantages cover both visual fidelity and the measured hand-alignment proxy.
Where the evidence stops. These blocks do not form a complete factorial study and contain no uncertainty estimates. They support component benefits under the fixed complementary configuration, but do not isolate their interaction or measure component-specific robot success.
Figure 5. Metric action encoding reduces accumulated error over the tested rollout horizon. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow video frame along each horizontal axis and accumulated error upward. Purple circles denote RoPE, green squares PRoPE, and blue stars A3D-RoPE. The left panel measures depth error; the right panel measures camera error. The curved red arrows annotate reductions relative to the two baselines at frame 80, not reductions in robot failure rate. The accompanying text specifies clip-scale-aligned depth AbsRel within action-keypoint regions and per-pixel camera-ray error in first-camera-relative Plücker coordinates. Both are computed using VGGT-Ω estimates on 15 sampled non-anchor frames. OAPM and the training setup remain fixed for this comparison. driftoapm
What it supports. A3D-RoPE has the lowest curve throughout the plotted frames. At frame 80, reported depth/camera reductions are 79.30%/78.26% against RoPE and 66.46%/49.04% against PRoPE. This diagnostic links the positional adapter to lower reconstructed geometric drift, extending the evidence beyond averaged image-quality scores.
Where the evidence stops. The vertical axes show accumulated estimator-based errors; they are not calibrated physical distances. Shared use of VGGT-Ω in conditioning and evaluation motivates an independent geometry check, and this plot alone does not establish behavior beyond 80 frames.
Figure 7. The memory ablation makes stale state and disappearing content visually distinct. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right, then compare corresponding columns. Row (a) uses both slots: the small anchor inset records the starting scene, and the later recent-memory inset reflects a generated update. Row (b) retains only the initial anchor; the red boxes draw attention to an outdated lid state persisting later in the sequence. Row (c), labeled without OAPM, marks scene content that disappears or drifts. The labels and red boxes belong to the original figure. Connect the comparison to Eq. (6): only the recent snapshot is replaced, so state changes need not erase the reference scene. memory-visualoapmablation
What it supports. This example illustrates why a persistent anchor alone may be insufficient: stable identity must coexist with changing object state. The full-memory row visually supports that intended division of labor, while Table 4 supplies the aggregate fidelity and alignment measurements that a single displayed sequence cannot provide.
Where the evidence stops. These are selected frames from a qualitative example. They cannot establish how frequently each failure occurs, prove physical contact correctness, or show that a particular refresh interval is optimal across tasks.
7. Analysis & limitations
7.1 What the evidence leaves open
The paper has no dedicated limitations section. Evaluation leaves total generation-benchmark size and uncertainty unspecified. The 50-clip hand metric cannot establish gripper accuracy; Kimi scoring lacks a stated frame count, judge-repeat protocol or human calibration. The unexplained Mask2IV asterisk in Table 1 is retained without interpretation. generationmetrics
OOD robot tests keep hardware, camera mounting and workspace fixed. No 800-real control, multiple-seed variability or confidence intervals are reported. Cross-embodiment synthesis is qualitative; it does not establish execution transfer to a new robot. robot-protocolrobot-resultsqualitative
Generated recent memory can propagate errors despite a fixed anchor. Figure 5 also evaluates geometry using the VGGT-Ω family employed in conditioning; agreement with that estimator is weaker evidence than independent 3D measurements. oapmdrift
7.2 Questions for discussion
- How sensitive are the two geometry mechanisms to camera/depth calibration errors (coordinates, oapm)?
- Do synthetic-data gains survive equal-size real-data and ordinary-augmentation controls (robot-results)?
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuilding the generator requires camera/pose conversion, the pretrained Wan prior, frozen VAE and VGGT-Ω features. The source omits generator optimizer/batch settings, exact block size, numerical refresh stride, adapter-layer placement, full trainable-module specification and exact checkpoint/software versions. architecturetrainingalgorithmcoordinates
The downstream recipe is more concrete: three 256×256 camera views, 16 active bimanual joint/gripper channels, AdamW at 10^-5, batch 32 and 20,000 steps, approximately 96 hours on eight H100 GPUs. Inference uses video/action guidance 5/1 and denoising steps 5/10. These are reported requirements, not reproduced measurements. downstream
Proposed checks: sweep recent-memory refresh under controlled geometry noise, and compare synthetic augmentation with equal-budget real and simple-augmentation controls. Measure geometric alignment and task success separately; detailed controls are specified in the illustrated edition. oapmablationrobot-results
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether recent refresh remains helpful with imperfect geometry
Reader-proposed experiment, not performed here: keep the backbone, A3D-RoPE, training clips and paired sampling seeds fixed. Compare anchor-only memory with recent refresh after every one, two and four committed blocks; these are proposed strides because the paper does not specify its numeric choice. Cross those settings with clean versus deliberately perturbed camera/depth inputs of recorded magnitude. Evaluate hand-keypoint error on the same fixed subset, image quality, and independent depth/camera measurements where available, in addition to the source's VGGT-Ω diagnostic. The mechanism predicts better evolving-state fidelity with refresh; disappearance of that benefit under small geometry perturbations would expose a robustness boundary. Inspect whether errors accumulate specifically after refresh events. oapmcoordinatesalgorithmablationdriftmetrics
Check 2: Separate synthetic diversity from training-set size
Reader-proposed experiment, not performed here: retain the Tianji M6 tasks, held-out appearances/layouts and fixed LingBot-VA recipe. Compare 400 real trajectories, 400 real plus 400 EgoGenesis trajectories, 400 real plus 400 ordinary visual augmentations, and 800 distinct real trajectories if collection is feasible. Keep gradient-update counts equal, record effective sample exposure, and repeat training with multiple seeds. Use matched evaluation configurations, task-specific terminal criteria and at least the source's 25 trials per task; report success counts and uncertainty rather than only averages. A mixed-data advantage over ordinary augmentation would support value from resimulation; comparison with 800 real would quantify its data-efficiency tradeoff. Track cube stacking and bimanual pick-and-place separately to avoid hiding task-specific effects. robot-protocolrobot-resultsdownstream
8.3 Reading coverage
Visual audit: Visually inspected the title/version and affiliation block, Figures 1–14, Tables 1–7, Algorithm 1, the metric definitions, judge prompt, and all supporting method/training/evaluation pages. All six final original crops were separately opened and inspected. Figure 3's memory paths and metric Q/K rotations were checked against Eqs. (4)–(9), (10)–(18) and Algorithm 1; no claim-relevant discrepancy was identified. Figure 5 legends and annotated reductions agree with the accompanying text. PDF pages 9–10 contain references and were read as text. No external videos, separate supplements, code or experiments were inspected. All six text chunks, including Appendix F, were read.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Preliminaries
- 4. Method
- 5. Experiments
- 6. Conclusion
- References
- Appendix A. Technical Details for OAPM and A3D-RoPE
- Appendix B. Training Corpus and Autoregressive Procedure
- Appendix C. Complete Component Ablation
- Appendix D. Detailed Downstream Real-Robot Results
- Appendix E. Additional Qualitative Comparisons
- Appendix F. Evaluation Metrics and Kimi K2.7 Prompt
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting PDF pages 1–8 and 11–19, including all figures and tables. Reference-only pages 9–10 were read as text.
- All six supplied text chunks were read individually, covering the complete 19-page PDF and its integrated supplement. No separate supplements, code, project website or external checkpoints were inspected; no experiments were reproduced.
- Identity/version: the title and twelve authors match the catalog after name-order normalization. The inspected title page identifies arXiv:2607.28243v1, 30 July 2026; the integrated supplement agrees. No other revision or edition was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author block, affiliations and arXiv margin; p. 11, supplementary title block
The title matches the catalog; all twelve authors agree in order. The PDF identifies arXiv:2607.28243v1, 30 July 2026. Its integrated supplement repeats the title and authors.
Go to primary source ↓problemPDF p. 2, Introduction and Figure 2
The authors motivate action-label-preserving synthesis by camera/gripper trajectory deviations and scene/object drift in example baseline generations.
Go to primary source ↓architecturePDF p. 3, Figure 3 and Section 3; p. 4, Model Overview
A chunkwise DiT combines noisy video, text/appearance context, scene-memory conditioning and supplied skeleton/end-effector geometry. The VAE encoder and decoder are frozen.
Go to primary source ↓flowPDF p. 4, Section 3, Eqs. (1)–(3)
Linear noise interpolation trains the DiT to predict noise minus clean latent; noise-to-data integration precedes committing the denoised block to the causal cache.
Go to primary source ↓oapmPDF p. 4, Online Anchored Projective Memory, Eqs. (4)–(6); p. 11, Appendix A, Eqs. (10)–(12)
VGGT-Ω encodes anchor and recent slots for gated cross-attention. Refresh replaces only the recent slot using the latest decoded causally visible generated frame.
Go to primary source ↓a3dPDF pp. 4–5, Action-3D Rotary Position Embedding, Eqs. (7)–(9); p. 11, Appendix A, Eqs. (13)–(15)
Video queries and skeleton keys receive axis-wise metric rotary phases only at skeleton-supported patches. The WAN self-attention positional path stays unchanged; the adapter writes back only at supported indices.
Go to primary source ↓coordinatesPDF p. 12, Appendix A, Eqs. (16)–(18)
Patch centers are projected onto valid skeleton edges; perspective-correct depths are blended with Gaussian support, then rays are unprojected into the reference camera frame. The adaptive radius uses r0 = 0.10.
Go to primary source ↓trainingPDF pp. 5–6, Experimental Setup; p. 12, Appendix B, Training Data and Autoregressive Training and Inference
The 210K corpus has 100K EgoDex, 100K AgiBot, 4K RoboTwin, 5K Real-world Ego and 1K DexJoCo clips. Clips use 81 frames, 16 FPS and 832×480. Two 6K-step stages use eight A100 GPUs. Appendix B specifies two teacher-forced context blocks and one target block.
Go to primary source ↓algorithmPDF p. 12, Appendix B, Eqs. (19)–(20); p. 13, Algorithm 1, lines 1–26
The recent slot and cache start empty. Each block is sampled, committed, and optionally used for memory refresh when the block index is divisible by the configured stride; the anchor remains fixed.
Go to primary source ↓generationPDF p. 5, Table 1, all method rows; p. 6, Quantitative Results
EgoGenesis reports PSNR 21.8609, SSIM 0.8509, LPIPS 0.2399, Kpt.Err 0.0501, Phys.Faith 0.8278, subject consistency 0.8923 and background consistency 0.9546. It leads six columns; EGOSIM leads subject consistency at 0.9101.
Go to primary source ↓metricsPDF p. 6, Experimental Setup; p. 16, Appendix F, aggregation; p. 18, unnumbered metric-definition table; p. 19, judge prompt and Eq. (22)
Kpt.Err is image-diagonal-normalized 2D hand error on 50 EgoDex clips. Phys.Faith divides a Kimi K2.7 integer score by five. DINO and CLIP feature similarities measure subject/background consistency. Samples are averaged after temporal aggregation.
Go to primary source ↓ablationPDF p. 6, Table 2 and Ablation Study; p. 14, Appendix C, Table 4, scene-memory and positional-encoding blocks
With A3D-RoPE fixed, recent refresh improves PSNR 20.4135→21.8609 and Kpt.Err 0.0532→0.0501. With OAPM fixed, RoPE/PRoPE/A3D-RoPE Kpt.Err is 0.07719/0.06135/0.0501.
Go to primary source ↓driftPDF p. 6, Figure 5 and Geometric drift; p. 7, continuation above Spatial influence
With OAPM and training fixed, frame-80 depth/camera error reductions are 79.30%/78.26% versus RoPE and 66.46%/49.04% versus PRoPE. Metrics use VGGT-Ω estimates on 15 sampled non-anchor frames.
Go to primary source ↓memory-visualPDF p. 7, Figures 6–7 and accompanying ablation paragraphs
Figure 6 shows changes in cross-attention output concentrated near manipulators. Figure 7 contrasts full memory, anchor-only memory and no OAPM; red boxes highlight stale or disappearing scene content.
Go to primary source ↓robot-protocolPDF p. 7, Downstream WAM Generalization; p. 13, Appendix D, platform and evaluation protocol; p. 15, Table 5
Resimulated demonstrations preserve action labels under edited appearances/scenes. Tianji M6 uses four tasks per arm configuration and 25 trials per task; OOD holds out appearances and initial/goal layouts. Stage-progress percentages are not success rates.
Go to primary source ↓robot-resultsPDF p. 7, Table 3; p. 16, Table 6, ID/OOD rows in panels (a) and (b)
Real-only versus mixed OOD success is 77%→84% single-arm and 53%→70% dual-arm. Synthetic-only gives 76% and 56%. OOD cube stacking is 56%→72%; bimanual pick-and-place is 56%→80%.
Go to primary source ↓downstreamPDF pp. 13–14, Appendix D, Downstream WAM Training and Inference Details, Eq. (21); pp. 15 and 17, optimization/inference details and Table 7
LingBot-VA is independently initialized from its own checkpoint for each data setting. Three 256×256 camera views feed a canonical 30-channel action model; bimanual experiments activate 16 joint/gripper channels. Training uses 20,000 steps, AdamW, batch 32, and eight H100 GPUs for approximately 96 hours.
Go to primary source ↓qualitativePDF p. 5, Figure 4; p. 8, Figure 9; pp. 14–16, Figures 11–12 and Appendix E; pp. 18–19, Figures 13–14
The paper supplies sampled generation and robot execution sequences, stage-progress curves, and a qualitative unseen-scene hand/gripper transfer example. The gripper skeleton is derived from index-finger and thumb trajectories.
Go to primary source ↓8.5 Primary sources
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE ↗
PDF · 9,445 extracted words
Source fingerprint
246ae42ae14b2cb289e2c1d7474cd97c6eee7ef3ad45cf42a46624ec1062e0dc