RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfer
1. Paper overview
In one sentence: RoboTransfer turns recorded geometry into appearance-diverse multi-view training videos, improving a separate imitation policy while leaving physical fidelity and evaluation uncertainty unresolved. e01e03e05e10e11e13e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Demonstrations are expensive, and visual changes can defeat imitation policies. Framewise augmentation can introduce temporal inconsistency, while independently generated camera views may disagree. The paper targets controlled changes to background and object appearance that preserve a demonstrated scene's geometry across moving wrist cameras and an overhead camera. e02e03e13 |
| Core mechanism | |
| A key reported result | Spoon pick-and-place under Diff-All: Real + Obj&Bg Aug: 46.7%; 2.98 Success rate (%) and Stage Score. Physical Cobot Magic; novel objects and environment; 100 task demonstrations; selected 50/50 real/synthetic mixture. Real only: 13.3%; 1.56. Domain Random Aug: 11.1%; 1.58. Real + Obj Aug: 22.2%; 2.04. The success increase is 33.4 percentage points. Scores track four task phases; trial counts and uncertainty are unreported. e13e16e17 |
| Reading caution |
Core contributions
Figure 2. Spatial conditions and object appearance enter the video generator through different routes. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the two upper-left VAE routes into the first Cat node: geometry and background features join the video representation before the attention blocks. The lower route encodes individual object images with CLIP and supplies their concatenated features to cross-attention. Equation (5) on page 4 explains the top-center representation: synchronized views are placed side by side along width before VAE encoding. Thus ordinary spatial processing can exchange information across camera regions. The decoder arrow ends in generated RGB video. This schematic omits diffusion scheduling; Section 7.1 on page 10 specifies 30 EDM denoising steps at inference. e03e08e14e15
What it supports. The architectural economy is to reuse a pretrained video model's spatial reasoning for multiple views, while giving geometry, background and objects different conditioning paths. Page 4 and Section 7.2 on page 10 support this reading; the paper does not add a dedicated cross-view module or an action-prediction head to this generator.
Where the evidence stops. Shared attention and geometry conditioning encourage consistency; the arrows are not a proof of guaranteed 3D agreement. Section 9 on page 11 describes a separately trained ACT controller, so this diagram does not establish unified future-and-action prediction.
2. Motivation
2.1 The problem and the proposed response
Demonstrations are expensive, and visual changes can defeat imitation policies. Framewise augmentation can introduce temporal inconsistency, while independently generated camera views may disagree. The paper targets controlled changes to background and object appearance that preserve a demonstrated scene's geometry across moving wrist cameras and an overhead camera. e02e03e13
2.2 What this reading follows
A manipulation demonstration contains both a useful motion and an incidental visual setting. RoboTransfer tries to reuse the former while changing the latter. Its diffusion model receives synchronized geometry sequences, a background reference and separate object references, then generates camera views together. The resulting videos augment the observations used to train ACT; they are not action plans executed by the generator. This reading follows the supplied January 2026 arXiv v2 from condition construction to physical policy evaluation. The most useful evidence is the combination of component ablations and real-robot results, provided the visual metrics, task success rates and unresolved protocol details remain distinct. e01e03e05e10e11e13e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Datasets. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The recorded taxonomy is entirely unassigned, so there is no positive quadrant claim to confirm. Architecturally, this is offline video augmentation followed by a separate ACT controller: neither joint future/action prediction nor inverse dynamics is demonstrated. A single diffusion backbone for multiple views does not make generator and policy a One Model system; the described method sits outside that action-prediction distinction. e03e08e13e14e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Keep the demonstrated trajectory, change its appearance
Begin with a recorded manipulation and its synchronized camera streams. RoboTransfer constructs per-frame geometry rather than asking a text prompt to determine what the robot should do next. Metric depth specifies scene distances, normals add local orientation cues, and width concatenation places the views in one encoded representation. Background references enter through spatial features, whereas object images become global CLIP features supplied through cross-attention. The source's division of labor is therefore between scene structure and appearance. Reader interpretation: preserving demonstrated geometry makes the method attractive for augmenting observations associated with an existing behavior, but gives no evidence that it discovers a new successful behavior. The generated video's plausibility and the downstream policy's ability to execute remain separate questions, answered by different experiments. e03e05e06e13e16
Figure 3. The preprocessing pipeline builds compatible geometry and appearance conditions from demonstrations. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward from the same demonstration at the top. On the left, predicted depth and sensor depth meet at Alignment, while monocular normals follow their own route. Section 8.2 on page 10 identifies dense Video Depth Anything predictions, sensor metric anchors and LOTUS normals; RGB-only data use MoGe. On the right, sampled keyframes feed object descriptions and segmentation. Masks support background inpainting and object-wise CLIP features. Sections 8.3 on pages 10–11 distinguish initial/final tabletop references from visibility-based selection in complex scenes. These steps explain how the generator obtains controls that the architecture figure assumes already exist. e05e06e07e10e11e18
What it supports. Reference preparation is part of the mechanism. Removing objects from the background reduces conflicting appearance cues, while individual object embeddings let their appearance vary separately. The sensor contributes metric scale without forcing the generator to use its noisy depth image directly. These are proposed preprocessing advantages, tested indirectly by the following ablations.
Where the evidence stops. Figure 13 on page 12 exposes an implementation ambiguity: Algorithm 1 fits metric depth but filters errors using unaligned predicted depth. Its two iterations and 80th-percentile rule are explicit; replacing that residual with a post-fit residual would be a reader-proposed change.
5.2 2. Test compatibility before treating realism as control
The appearance pipeline is designed to avoid giving the generator incompatible instructions. A background image that still contains the original objects can compete with a request to change those objects; inpainting removes that cue, while object-wise features separate their appearance conditions. Table 3 tests these choices directly. Table 2 asks a different question: whether cleaner, metric-aligned depth and normals improve consistency. Read both tables camera by camera, because a condition can help moving wrist views while failing to lead a static head-view metric. Reader interpretation: the two ablations support complementary controls, but neither alone proves physical correctness. The depth metrics are scale-invariant, correspondences depend on an image matcher, and object CLIP similarity is unavailable in the difficult wrist views. These boundaries matter before linking visual quality to action learning. e03e05e09e10e11
5.3 3. Follow the handoff from generated observations to ACT
The handoff to robotics occurs during policy training. The paper collects separate task demonstrations, generates new appearances from their geometry, and introduces synthetic data after real-data ACT pretraining. During deployment, ACT sees three RGB frames and the current state, predicts a two-second chunk, and executes all 100 actions before refreshing observations. RoboTransfer is not described as running inside that feedback loop. Table 4 therefore evaluates whether augmented training improves an imitation policy's response to visual shifts. It reports substantial gains, but Figure 10 shows that additional synthetic data is not uniformly helpful. Reader interpretation: a convincing reproduction should hold the policy schedule and data exposure fixed while varying augmentation, then repeat physical evaluations. Without those controls and trial uncertainty, the observed improvement is useful evidence but its reliability and cause remain incompletely established. e13e14e15e16e17
5.4 Training and inference
During training
SVD fine-tuning uses a denoising reconstruction objective aggregated over noise levels. Approximately 24k training and 1.6k evaluation clips contain 30 frames at 10 Hz. Each view is resized to 640×384. AdamW uses learning rate 3×10^{-5}, global batch 8 and 70k steps. Frozen-module choices are not specified. e04e08
The dedicated corpus covers twelve tasks, described as 1,000 samples per task across ten object configurations; AgiBot-World adds diversity. Policy experiments exclude the generator's task dataset and collect 100 separate expert demonstrations per task. Synthetic policy observations use demonstration geometry and held-out appearance references. e06e07e13
ACT is pretrained on real data for 100k steps with batch 512 and learning rate 10^{-4}, then fine-tuned for 50k steps at 10^{-5}. The policy appendix reports eight NVIDIA H20 GPUs, approximately 24 hours pretraining and 12 hours fine-tuning; these are policy-training costs, not a stated generator-training cost. e14
During inference
Offline video synthesis uses 30 EDM denoising steps with classifier-free guidance. Geometry trajectories are supplied from demonstrations or simulation; the generator neither selects new actions nor performs online planning. The source permits a clean captured background reference at inference. e03e08e12e19
ACT receives one RGB image per camera plus current state, then predicts 100 actions. All are executed at 50 Hz before fresh observations, giving a two-second open-loop chunk between feedback updates. Reported policy inference latency is 10 ms; this is not video-generation latency. e14e15
5.5 Implementation flow
- Construct geometry
For RGB-D recordings, Video Depth Anything supplies dense relative depth, aligned to sensor depth through scale-and-shift fitting with outlier filtering. LOTUS estimates normals. For RGB-only data, the appendix uses MoGe for metric depth and normals. These are preprocessing conditions, not dynamics learned from candidate actions. e05e06e18
- Separate appearance references
Keyframes feed a VLM descriptor, Grounding DINO and SAM2. Masking and inpainting remove objects from background references. Individual object patches become CLIP embeddings. Standard tabletop references use initial/final frames; complex scenes select maximal object visibility and minimal object presence, respectively. e05e07
- Generate views jointly
Concatenate synchronized videos along width before VAE encoding. VAE-encoded geometry and background conditions join noisy video latents through channel concatenation. Object embeddings are concatenated and supplied through cross-attention. Joint spatial modeling allows information exchange across camera regions; it does not impose a hard geometric constraint. e03e08
6. Experiments & results
RoboTransfer augments robot demonstrations by changing their visual appearance while conditioning video diffusion on the demonstrated geometry. Jointly encoded camera views, metric depth, normals and separate background/object references support multi-view synthesis. A separately trained ACT policy benefits from the augmented observations on two physical manipulation tasks. This is evidence for offline data augmentation, with remaining uncertainty about statistical reliability and physical fidelity.
6.1 Read the original evidence
Table 4. Appearance-diverse demonstrations improve a separately trained ACT policy on both physical tasks. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task's Diff-Obj and Diff-All columns separately: they test novel objects and joint object/environment changes. SR measures full success; Score measures progress through the four spoon phases or three towel phases described below the table. Compare Real only with Real + Obj Aug, then with Real + Obj&Bg Aug to assess the added background diversity. Section 5.1 on page 8 specifies the Cobot Magic setup and 100 demonstrations per task. Section 9 on page 11 clarifies that ACT is a separate policy, first pretrained on real data and then fine-tuned; Table 4 uses the selected 50/50 real/synthetic mixture. e13e14e15e16e17e20
What it supports. For spoon Diff-All, joint augmentation raises success from 13.3% to 46.7% and Stage Score from 1.56 to 2.98. The success gain is 33.4 percentage points. Towel Diff-All also improves, from 12% to 28%, while Diff-Obj reaches 50.0%. These are executed robot-policy outcomes, unlike the preceding synthesis metrics.
Where the evidence stops. Trial counts, seeds and uncertainty are absent, and equal total training exposure across baselines is not fully specified. Table 4 also disagrees with Figure 1 on towel Diff-All real-only/object-augmentation scores: 1.08/1.24 here versus 1.04/1.20 there.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Spoon pick-and-place under Diff-All Physical Cobot Magic; novel objects and environment; 100 task demonstrations; selected 50/50 real/synthetic mixture. | Real + Obj&Bg Aug: 46.7%; 2.98 Success rate (%) and Stage Score | Real only: 13.3%; 1.56. Domain Random Aug: 11.1%; 1.58. Real + Obj Aug: 22.2%; 2.04. The success increase is 33.4 percentage points. Scores track four task phases; trial counts and uncertainty are unreported. e13e16e17 |
| Towel folding under object and combined shifts Physical three-stage task; Diff-Obj changes objects, Diff-All changes objects and environment; selected 50/50 mixture. | Real + Obj&Bg Aug: Diff-Obj 50.0%, 2.60; Diff-All 28%, 1.92. Success rate (%) and Stage Score | Real only: 16.7%, 1.81 and 12%, 1.08, respectively. Both settings improve, but success remains limited under combined shifts. Table 4 supplies these values; Figure 1 differs on some towel scores. e16e17e20 |
| Geometry conditioning ablation Synthesis evaluation corpus: approximately 1.6k clips; Table 2 compares conditions within each camera. | Metric predicted depth + normals, left: 0.047, 202.03, 107.43; right: 0.058, 75.67, 220.12. Depth RMSE, matched pixels and FVD | Raw sensor depth, left: 0.074, 142.90, 218.51; right: 0.090, 40.70, 396.33. Wrist-view metrics improve, but head-view predicted depth alone has lower RMSE (0.132 versus 0.133) and FVD (95.89 versus 101.17). The caption's universal-best claim overstates its table. e08e09e10 |
| Appearance conditioning ablation Same synthesis evaluation; Table 3 crosses background inpainting and object splitting. | Both enabled: 0.720 / 0.858 / 101.17 Head-view background/object CLIP similarity and FVD | Neither enabled: 0.712 / 0.847 / 108.65. Joint conditioning improves these proxy metrics. Wrist object similarity is omitted because tracking is unreliable; CLIP similarity does not establish manipulation success. e08e09e11 |
| Synthetic-data mixture for spoon Diff-All Figure 10 varies synthetic proportion on the spoon task under combined shifts. | 50% synthetic: 46.7%; 100% synthetic: 40.0% Success rate (%) | 0% synthetic: 13.3%. The tested midpoint performs best. This does not establish a universal optimum or explain how the 100% condition relates to the real-pretraining recipe. e14e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Geometry conditioning improves the wrist views, with clear head-view exceptions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare rows within a camera block before moving between views. D.S. is sensor depth, D.P. is predicted depth, Metric D.P. adds scale alignment, and +N. adds normals. Lower arrows apply to depth errors, normal angular errors and FVD; higher matched-pixel counts indicate more correspondences with the head view. Section 4.1 on page 6 defines these metrics and omits head matched-pixel values. Section 7.1 on page 10 describes approximately 1.6k evaluation clips. The depth metrics are scale-invariant, so their improvement alone does not establish absolute metric calibration, even though the conditioning pipeline uses metric anchors. e08e09e10e11
What it supports. For the left camera, replacing raw depth with metric predicted depth plus normals changes RMSE from 0.074 to 0.047, matched pixels from 142.90 to 202.03 and FVD from 218.51 to 107.43. The right-view trend agrees. The head camera differs: predicted depth alone has FVD 95.89, compared with 101.17 for the full condition set.
Where the evidence stops. The table's caption and page 6 prose claim best performance across all metrics, but the head rows contradict that wording. Also, right-camera full-model RMSE is 0.058 here and 0.059 in Table 3. No uncertainty estimates resolve these differences.
Table 3. Clean backgrounds and object-wise features provide complementary appearance control. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two checkmark columns form a two-factor comparison. Within each camera block, the rows enable neither change, background inpainting alone, object splitting alone, or both. Sections 4.1 on pages 6–7 explain that the unsplit object condition uses a global image feature, whereas splitting encodes objects individually. BG. Sim. compares generated frames with a background reference using CLIP; Obj. Sim. compares segmented objects with their reference. Do not read the wrist-view dashes as zero similarity: page 6 says motion and occlusion make object tracking unreliable there. The table still reports geometry and video-quality proxies for those views. e05e07e09e11
What it supports. The joint condition has the strongest reported head-view appearance similarities: background rises from 0.712 to 0.720 and object similarity from 0.847 to 0.858, with FVD falling from 108.65 to 101.17. This supports combining the two preprocessing choices, while keeping the magnitude and proxy nature of the gains visible.
Where the evidence stops. Individual changes are not uniformly beneficial: left-view inpainting alone raises FVD from 117.25 to 119.65. CLIP similarity cannot establish precise shape, contact fidelity or successful action execution; wrist object fidelity also remains unmeasured by this metric.
Figure 10. The best tested mixture balances synthetic visual diversity with real observations. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the blue left axis for success rate and the orange right axis for Stage Score; the lines' vertical positions are not directly comparable. The horizontal axis is synthetic-data proportion. The paper's caption identifies zero synthetic as the real-only baseline, and Section 5.2 limits this experiment to spoon pick-and-place under Diff-All. The red arrow marks the tested 50/50 mixture, which the authors then select for Table 4. Read exact endpoint success rates from the accompanying prose rather than estimating intermediate values from pixels. The curve has no error bars and does not by itself explain the training schedule behind each point. e14e16e17
What it supports. Reported success rises from 13.3% at zero synthetic to 46.7% at the midpoint; the all-synthetic point remains above baseline at 40.0%. Increasing synthetic proportion beyond the midpoint therefore does not monotonically improve this task. The experiment supports a useful mixture in this setup, rather than a general optimal ratio.
Where the evidence stops. Page 11 specifies real-data pretraining before synthetic fine-tuning, but the source does not reconcile that recipe with the 'exclusively synthetic' wording here. The authors' contact/material-fidelity explanation is a hypothesis, not an isolated experimental cause.
7. Analysis & limitations
7.1 What the evidence leaves open
Only two physical tasks are tested. Sim-to-real evidence is qualitative video synthesis, while policy augmentation uses real-demonstration geometry. External-generator comparisons are qualitative rather than matched quantitative policy benchmarks. e12e13e16
The authors suggest excessive synthetic data loses contact/material fidelity; the ratio experiment does not isolate that cause. Geometry errors, image correspondences and CLIP similarity measure visual proxies rather than calibrated physical dynamics. e09e17
Reporting inconsistencies remain: Table 2's right full-model RMSE is 0.058 versus Table 3's 0.059; Figure 1 shows towel Diff-All scores 1.04/1.20 for real-only/object augmentation, versus Table 4's 1.08/1.24. These differences are unresolved. e10e11e20
7.2 Questions for discussion
- Would the mixture advantage persist with matched update counts, explicit pretraining controls and repeated trials?
- How much does correcting the alignment residual change wrist-view geometry and policy success?
- Can visual correspondence metrics predict failures during contact-rich towel folding?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs SVD, the geometry/segmentation/inpainting pipeline, synchronized demonstrations and ACT. Policy RGB is sampled at 10 Hz and resized to 640×360; state is downsampled to 50 Hz. Exact action-label alignment, generator train/evaluation trajectory separation, guidance strength and evaluation trial counts require clarification. The source promises code release without establishing availability. e07e08e13e16e19
Algorithm 1 fits aligned depth but computes rejection errors from unaligned D_pred minus D_sensor, with two iterations and an 80th-percentile cutoff. Its epsilon is unspecified. This differs from filtering post-fit residuals and should be tested as written before assuming a correction. e18
Equations (1)–(2) leave u(sigma) undefined and use x_0 both in the clean-latent description and inside VAE encoding. The paper names ACT without giving its complete loss. These gaps prevent claiming an exact implementation from this text alone. e04e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check whether the alignment filter removes noise or scale mismatch
Reader-proposed check, not performed: start from known metric depth, create relative predictions with a controlled scale/shift, and inject sensor holes and outliers. Compare Algorithm 1 exactly as printed against an explicitly modified version that computes rejection errors after alignment, plus a no-rejection least-squares control. Keep the initial valid mask, two iterations and percentile threshold fixed. Measure recovered scale/shift and depth error on held-out valid pixels; then compare generated wrist-view RMSE and matched pixels with identical references and diffusion seeds. If the printed rule rejects good pixels merely because of scale mismatch, its purported robustness requires qualification. Report degenerate fits separately rather than silently repairing them. e06e09e10e18
Check 2: Separate augmentation gains from policy training exposure
Reader-proposed check, not performed: reproduce spoon Diff-All with the same 100 demonstrations and a fixed ACT checkpoint, then compare real-only continued training, domain randomization, object augmentation and joint object/background augmentation for identical update counts and observation budgets. Test multiple mixture ratios under that common schedule. Add a separately labeled synthetic-from-initialization arm to distinguish it from synthetic fine-tuning after real pretraining. Keep appearance references and evaluation scenes explicitly separated, repeat across seeds and physical trials, and report success counts, confidence intervals and Stage Scores. If the joint-augmentation advantage disappears under matched exposure, or the midpoint fails to outperform neighboring mixtures reliably, the proposed generalization mechanism or claimed optimum needs narrowing. e13e14e15e16e17
8.3 Reading coverage
Visual audit: Actually inspected the title/authors/version on PDF page 1; Table 1 and introduction on page 2; architecture and diffusion equations on page 3; construction diagram and Eq. (5) on page 4; Tables 2–3 on page 5; metrics and qualitative panels on pages 6–8; Table 4 and Figure 10 on page 9; generator/data details on page 10; platform, policy training, H20 hardware and deployment on page 11; Algorithms 1–2, descriptor template and discussion on page 12; reference-selection discussion on page 13. Checked architecture arrows against Section 3.2 and Eq. (5), and alignment operations against Section 8.2 and Figure 13. Every final crop was viewed and retains complete headers, axes and legends without body prose. Table 2's universal-best wording, the cross-table RMSE discrepancy, Figure 1 versus Table 4 scores and the alignment residual ambiguity are preserved. Pages 14–16 are bibliography-only and were read as text. Static PDF panels do not establish full-video inspection; external media, code and separately hosted supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Section 1 Introduction
- Sections 2.1–2.2 Related Work
- Sections 3.1–3.3 Methods, diffusion preliminaries, framework and dataset construction
- Sections 4.1–4.4 Synthesis quantitative analysis, qualitative results and comparisons
- Sections 5.1–5.2 Real Robot Experiments; Section 6 Conclusion
- Supplementary Sections 7.1–7.2 RoboTransfer Implementation Details
- Supplementary Sections 8.1–8.3 Dataset Construction Details
- Supplementary Sections 9.1–9.3 Robot Policy Implementation, Training and Deployment
- Supplementary Sections 10.1–10.3 Discussion; References [1]–[65]
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The inspected title page identifies arXiv:2505.23171v2 [cs.CV], 6 January 2026. Title and all ten authors match the catalog. The catalog's 29 May 2025 submission date describes an earlier point in the record; this report reviews the supplied v2, not an inspected v1 or a comparison of revisions.
- All six supplied text chunks were read without truncation, covering all 16 PDF pages. PDF pages 1–13 and all final crops were visually inspected; bibliography-only pages 14–16 were read as text.
- The acquisition omission that text extraction does not reconstruct figure images was addressed by inspecting the retained PDF; no external project videos were inspected.
- Separate supplemental material availability has not been fully verified. The Supplementary Material embedded in this PDF was read in full; no additional supplement was supplied.
- Code was not inspected and experiments were not reproduced. Linked project resources and cited papers were not independently accessed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/byline, affiliations and left-margin arXiv stamp
Exact catalog title and ten authors are present. The stamp is arXiv:2505.23171v2 [cs.CV], 6 Jan 2026; affiliations are Horizon Robotics, GigaAI and CASIA.
Go to primary source ↓e02PDF p. 2, Section 1 and Table 1
Motivates costly demonstrations, sim-to-real gaps, temporal/multi-view consistency and control over background and object appearance; Table 1 is an author capability comparison.
Go to primary source ↓e03PDF p. 3, Figure 2; p. 4, Section 3.2 and Eq. (5)
Views concatenate along width before VAE encoding; geometry/background latents concatenate with noisy inputs, while object CLIP embeddings enter cross-attention. Output is video for downstream training.
Go to primary source ↓e04PDF p. 3, Section 3.1, Eqs. (1)–(4)
Describes conditional squared-error denoising and noise-weighted aggregation. The displayed loss contains u(sigma) without an accompanying definition; x_0 is described as a clean latent yet appears within the encoder in Eq. (1).
Go to primary source ↓e05PDF p. 4, Figure 3; p. 5, Section 3.3
Demonstrations supply depth/normal conditions and keyframes for VLM-assisted object segmentation, background inpainting and object CLIP features.
Go to primary source ↓e06PDF p. 10 (supplement p. 1), Sections 8.1–8.2
Twelve tasks, ten object configurations and 1,000 samples per task are described. Three RGB-D views support Video Depth Anything alignment to sensor depth and LOTUS normals; RGB-only data use MoGe. AgiBot-World adds diversity.
Go to primary source ↓e07PDF pp. 10–11 (supplement pp. 1–2), Section 8.3; p. 13 (supplement p. 4), Section 10.3
Initial/final tabletop frames or visibility-based complex-scene selection supply references. Grounding DINO and SAM2 produce masks; inpainting cleans backgrounds; 224×224 object patches feed CLIP. A clean background may instead be captured for inference.
Go to primary source ↓e08PDF p. 10 (supplement p. 1), Sections 7.1–7.2
Approximately 24k training and 1.6k evaluation clips at 10 Hz/30 frames; SVD fine-tuning at 640×384 per view, AdamW 3×10^-5, batch 8, 70k steps; 30 EDM steps and classifier-free guidance. Object embeddings concatenate; width concatenation avoids a dedicated cross-view module.
Go to primary source ↓e09PDF p. 6, Section 4.1, Evaluation Metrics and Geometry Consistency Analysis
Matched pixels compare wrist/head view pairs; depth metrics are scale-invariant; normal errors are angular. CLIP similarity assesses references, with wrist object similarity excluded for tracking problems. Framewise scores are averaged; FVD assesses generated versus real video. Head matched-pixel entries are omitted.
Go to primary source ↓e10PDF p. 5, Table 2, all camera blocks; p. 6, geometry-analysis prose
Left/right full-model RMSE, matched pixels and FVD are 0.047/202.03/107.43 and 0.058/75.67/220.12. Raw depth gives 0.074/142.90/218.51 and 0.090/40.70/396.33. Head predicted-depth-only RMSE/FVD 0.132/95.89 outperform full-model 0.133/101.17 despite the universal-best caption.
Go to primary source ↓e11PDF p. 5, Table 3; pp. 6–7, Appearance Consistency Analysis
The two binary conditions are background inpainting and object splitting. Head baseline background/object similarity and FVD are 0.712/0.847/108.65; joint conditions give 0.720/0.858/101.17. Full-model right RMSE is printed as 0.059.
Go to primary source ↓e12PDF pp. 6–8, Figures 4–9; p. 7, Sections 4.2–4.4
Static panels illustrate background/object changes, simulated-to-realistic appearance transfer and diverse scenes. Comparisons with Cosmos-Transfer and RoboEngine are qualitative; the figures are not physical policy success tests.
Go to primary source ↓e13PDF p. 8, Section 5.1; p. 11 (supplement p. 2), Section 9.1
Cobot Magic has two PIPER arms and three D435i cameras. ACT uses 100 ALOHA demonstrations per task, excluding the generator's task dataset. Images become 640×360 at 10 Hz; state is downsampled to 50 Hz. Synthetic observations use real geometry and held-out appearance references.
Go to primary source ↓e14PDF p. 11 (supplement p. 2), Section 9.2
ACT predicts 100 states over two seconds. Real pretraining: 100k steps, batch 512, learning rate 10^-4; fine-tuning: 50k steps, 10^-5. The policy recipe reports eight NVIDIA H20 GPUs and approximately 24/12 hours.
Go to primary source ↓e15PDF p. 11 (supplement p. 2), Section 9.3
Each policy decision consumes three RGB frames and current state. All 100 actions execute at 50 Hz before new observations. Reported inference latency is 10 ms.
Go to primary source ↓e16PDF p. 9, Table 4 and Section 5.2 task/effectiveness paragraphs
Spoon Diff-All real-only/domain-random/object/joint augmentation gives success 13.3/11.1/22.2/46.7% and scores 1.56/1.58/2.04/2.98. Towel joint augmentation gives 50.0%/2.60 (Diff-Obj), 28%/1.92 (Diff-All), versus 16.7%/1.81 and 12%/1.08. Four spoon and three towel phases define progression; no trial counts or uncertainty appear.
Go to primary source ↓e17PDF p. 9, Figure 10 and Section 5.2, Effect of Synthetic Data Proportions
Spoon Diff-All peaks at the tested 50/50 mixture, with 46.7% success versus 13.3% at zero synthetic and 40.0% at all synthetic. The authors adopt 50/50 for Table 4 and suggest physical-fidelity loss at high synthetic proportions.
Go to primary source ↓e18PDF p. 12 (supplement p. 3), Figure 13, Algorithms 1–2; p. 10, Section 8.2
Algorithm 1 initializes positive-depth masks, performs two scale/shift fits and filters at the 80th percentile. Line 5 forms aligned depth; line 7 nevertheless uses unaligned predicted-minus-sensor depth. Algorithm 2 supplies least-squares fitting. Epsilon and degenerate-fit handling are unspecified.
Go to primary source ↓e19PDF pp. 12–13 (supplement pp. 3–4), Sections 10.1–10.3
Discussion positions appearance references as style conditions anchored by geometry, outlines RGB-D/RGB-only and simulator condition acquisition, promises code release and permits clean captured inference backgrounds.
Go to primary source ↓e20PDF p. 1, Figure 1, towel Diff-All score bars; p. 9, Table 4, Towel Folding / Diff-All / Score
Figure 1's real-only/object-augmentation towel Diff-All bars read 1.04/1.20, whereas Table 4 reports 1.08/1.24. This report preserves Table 4 values and discloses the disagreement.
Go to primary source ↓8.5 Primary sources
RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfer ↗
PDF · 8,963 extracted words
Source fingerprint
f2dac9480cae0cd703a96d901fdf81723da08e5eedf2fa4dfa36c3c91a945043