Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction
1. Paper overview
In one sentence: CogRobot uses predicted optical flow to guide bimanual video plans, improving reported video quality and trained-task execution while retaining a separate action policy for each task. e02-framinge04-flowe05-videoe06-controllere08-simulatione09-reale11-ablation-rdte12-ablation-robominde13-limit
| At a glance | What to know |
|---|---|
| Research problem | Source description Coordinating two arms expands the action space while bimanual demonstrations remain scarce. The authors ask whether pretrained video generation can provide useful visual plans without first unifying heterogeneous robot actions. Their central difficulty is grounding broad language instructions in precise, feasible movements from the current image. e02-framinge03-failures |
| Core mechanism | Source description Two separately adapted CogVideoX models turn language and an initial observation into flow and then RGB futures, with a separate diffusion controller translating visual goals into actions. e02-framinge05-videoe06-controller |
| A key reported result | Pull Box: CogRobot: 0.75. Task success rate (fraction). Physical Realman dual-arm system; 100 teleoperation demonstrations for this task; average over 20 trials. DP: 0.05; an absolute difference of 0.70, or 70 percentage points. Large observed gain on coordinated rope manipulation; no uncertainty interval or held-out-task transfer protocol is reported. e09-reale15-tasks |
| Reading caution | Reader analysis The authors explicitly acknowledge that every task requires its own action-extraction policy. Four simulation tasks and two trained physical tasks do not establish a generalist low-level controller. e08-simulatione09-reale13-limit |
Core contributions
- Source description
Two separately adapted CogVideoX models turn language and an initial observation into flow and then RGB futures, with a separate diffusion controller translating visual goals into actions. e02-framinge05-videoe06-controller
- Reader analysis
The authors attribute improved motion grounding and reduced data requirements to flow guidance. The paper demonstrates fixed-data video-quality improvements, but supplies no data-scaling curve quantifying the claimed reduction. e02-framinge11-ablation-rdte12-ablation-robomind
- Source description
A Vision Pro teleoperation setup supplies demonstrations for two Realman arms, enabling evaluation on coordinated bag lifting and rope-mediated box pulling. e14-teleoperatione15-tasks
Figure 1. Predict motion first, then use it to guide the visual plan. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the platform at left: an exterior camera observes both grippers. On the right, each stage receives a noised sequence together with the current image, and the instruction branches into both CogVideoX blocks. Follow the blue dashed arrow from future optical flow down into the second model; this is the additional motion condition. The lower output is a future bimanual video. Section 3 then adds a further operation outside this diagram: a separately trained goal-reaching policy converts a desired video frame into actions. The two numbered stages therefore describe visual prediction, not the whole robot-control pipeline. e02-framinge04-flowe05-videoe06-controllere18-training
What it supports. The key architectural separation is between predicting where visible motion should occur and rendering the resulting scene. Both visual models receive language and the current observation. A third learned component supplies joint actions, so the diagram does not support interpreting CogRobot as one network that jointly emits future states and actions.
Where the evidence stops. The future flow is a prediction, not a measurement of an already completed movement. The diagram shows the intended inference chain; Appendix C.3 instead supplies reference flow during RGB-model training, creating an upstream-error question for deployment.
2. Motivation
2.1 The problem and the proposed response
Coordinating two arms expands the action space while bimanual demonstrations remain scarce. The authors ask whether pretrained video generation can provide useful visual plans without first unifying heterogeneous robot actions. Their central difficulty is grounding broad language instructions in precise, feasible movements from the current image. e02-framinge03-failures
2.2 What this reading follows
A video model can depict a robot completing an instruction without supplying a usable movement plan. CogRobot addresses this gap by predicting an RGB-encoded motion field before predicting the scene's appearance. A separate goal-reaching controller then turns a predicted image into joint actions. The useful distinction is between three questions: whether flow improves video prediction, whether predicted goals help a controller, and whether the resulting system generalizes. The paper supplies video comparisons, four simulation tasks and two physical tasks, with encouraging but uneven results. Its clearest physical gain is on rope-mediated box pulling; its task-specific action policies remain a significant boundary. e02-framinge04-flowe05-videoe06-controllere08-simulatione09-reale11-ablation-rdte12-ablation-robominde13-limit
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is unassigned, so there is no populated category to endorse or reject. Architecture evidence supports a modular visual planner plus a separate inverse-dynamics-like goal-reaching action model. Future video and action are not jointly predicted by one network; flow and video prediction are used at inference, beyond auxiliary training supervision. e02-framinge05-videoe06-controllere09-reale16-architecture
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make motion compatible with a pretrained video representation
CogRobot starts from a practical representation problem. Raw optical flow has two displacement channels, whereas the pretrained video encoder expects RGB. Section 3.1 converts each displacement into direction and magnitude, then maps direction to color and magnitude to saturation. This creates a flow video that the existing VAE can encode without training a new flow-specific VAE. The flow fields are anchored to the initial frame, so every future field describes displacement from the same visual reference. Appendix C.2 makes the construction concrete: sixteen future fields are estimated for a seventeen-frame clip and a white first frame is prepended. The first diffusion transformer learns to predict this sequence from the initial observation and instruction; the second receives its encoded motion guidance. e04-flowe05-videoe17-preprocessing
Figure 7. The RGB denoiser receives flow as a conditioning latent. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three inputs from top to bottom: task text, bimanual video and flow video. The two video branches use the same pretrained 3D VAE according to the caption. The circled plus marks additive noise on the vision branch; the circled C denotes channel concatenation. Trace the flow branch into the joint latent without passing through that noise marker. Equation (3) formalizes the video/flow input as a noisy RGB latent concatenated with a flow latent, while Appendix C.1 also describes text concatenation. The encoder snowflakes and transformer flames visually distinguish the pretrained encoding blocks from the adapted diffusion backbone. e04-flowe05-videoe16-architecturee18-training
What it supports. The loss asks the transformer to predict noise added to the RGB latent while using flow as guidance. This supplies motion information without putting low-level robot actions into the video predictor. The frozen VAE reuses a video representation for both ordinary RGB clips and color-encoded displacement fields.
Where the evidence stops. Figure 7 omits the explicit initial-observation branch stated in Section 3.2. Its text branch is more detailed than Eq. (3), which writes language as separate conditioning. VAE freezing is explicit in C.1; text-encoder update settings are not separately specified there.
5.2 Follow the change from supervised flow to generated flow
The two visual predictors are not trained as one end-to-end action model. Appendix C.3 trains the flow predictor against flow estimated from demonstration videos, while the RGB predictor receives reference flow as its condition. Deployment follows a different path: the first predictor must supply that condition from language and the current image. Reader interpretation: even a strong RGB decoder can fail when the upstream motion proposal is wrong, and reference-flow training alone does not quantify this sensitivity. The independent diffusion policy then adds another learned mapping from desired appearance to action. This separation makes fault isolation possible, but it also means that video reconstruction scores cannot reveal which component limits physical success. Table 3's unspecified evaluation-flow provenance is consequently a material reproduction question. e02-framinge05-videoe06-controllere09-reale11-ablation-rdte18-training
5.3 Separate useful visual planning from generalist robot control
The clearest supported control result compares CogRobot with single-view DP on the two physical tasks. A visual goal is especially helpful in the reported Pull Box experiment, where the arms must coordinate rope handling before moving the box. Yet the action policy is still trained separately for each task, and the simulation comparison shows that richer sensory baselines can perform substantially better on some tasks. Reader interpretation: the evidence supports visual planning as a useful addition to these trained controllers, while leaving open whether one shared controller could exploit the same plans across unseen tasks. Repeated goal generation provides observational feedback, but the paper does not report its timing, goal-selection rule or action execution horizon. Those details are necessary to distinguish planning quality from the effectiveness of the feedback schedule. e06-controllere08-simulatione09-reale13-limite15-taskse16-architecture
5.4 Training and inference
During training
Both video models first adapt to RDT and the AgileX bimanual subset of RoboMIND, then to RoboTwin or Realman. Flow targets are estimated from reference videos; the RGB model is trained with reference flow as input. Each downstream task supplies 100 demonstrations, while its action policy is trained separately. e07-setupe17-preprocessinge18-training
Clips contain 17 frames at 256×256. Sixteen initial-to-future flow maps receive a white first frame. Table 5 reports training clips: RDT 157,912; RoboMIND 198,789; RoboTwin 106,771; Realman 12,866. RDT reserves 5% of episodes; RoboMIND uses its official validation split. Low-motion filtering is described without a numerical threshold. e17-preprocessing
The reported recipe uses AdamW, learning rate 2×10⁻⁵, 100 warm-up steps, four H100 80GB GPUs, and batch size 128 per GPU. Initial flow/video adaptation lasts 20k/30k steps; both receive 15k downstream steps on RoboTwin or 5k on Realman. e18-training
During inference
The current observation and instruction generate flow, then future RGB goals; the task-specific policy converts a visual goal into executable actions. Deployment repeatedly generates fresh goals from current observations. The paper does not specify goal-frame selection, execution chunk length, replanning rate, or measured inference latency. e02-framinge06-controllere09-reale16-architecture
Reference flow used during training is replaced by generated flow in the intended deployment pipeline. This exposes the RGB predictor to upstream errors. Optical flow is an image-motion representation, not an explicit joint-limit or contact-force constraint. e04-flowe05-videoe18-training
5.5 Implementation flow
- Represent motion relative to the initial image
FlowFormer++ estimates displacement from frame 0 to each future frame, rather than between consecutive frames. Directions become colors and magnitudes determine saturation, producing three-channel flow videos compatible with the pretrained video VAE. e04-flowe17-preprocessing
- Generate flow before appearance
An image-conditioned CogVideoX-Fun-V1.1-2b-InP variant predicts the flow video from the current observation and instruction. Its VAE stays frozen while its diffusion transformer learns a noise-prediction loss. e04-flowe16-architecture
- Condition RGB denoising on flow
The second model encodes flow and RGB video with the same frozen 3D VAE. It concatenates the flow latent with the noisy RGB latent and conditions on language and the initial observation. Appendix C.1 additionally describes concatenating text embeddings; Figure 7 omits an explicit initial-observation branch. e05-videoe16-architecture
- Learn actions from visual endpoints
A separate goal-conditioned Diffusion Policy samples a goal step uniformly during training and denoises the corresponding action sequence. Two ResNet-18 encoders represent observation history and goal image. Section 3.3 uses images o0/on, but Eq. (4) switches to s0/sn without clarifying their mapping. e06-controllere16-architecture
6. Experiments & results
CogRobot adapts video diffusion to bimanual control through three learned components: instruction-conditioned flow prediction, flow-conditioned RGB prediction, and a task-specific goal-reaching action policy. Flow offers an intermediate description of motion without requiring action labels for the video models. The strongest physical result is Pull Box success of 0.75 versus 0.05 for DP, but the evidence does not establish a universal action policy or broad unseen-task transfer (e02-framing, e05-video, e06-controller, e09-real, e13-limit).
6.1 Read the original evidence
Table 1. Single-view planning improves on DP, with task-dependent rankings against richer inputs. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task block independently, keeping the input type beside the method name. CogRobot and DP both use single-view RGB, whereas RDT uses multiple views and DP3 uses point clouds, optionally with color. The dashed horizontal line separates the single-view rows. Compare CogRobot to DP first: the displayed means are 100 versus 82, 36 versus 2, 15 versus 3, and 55 versus 51 across the four tasks. Then look upward to the other baselines: the best method changes with the task. Section 5.2 specifies ten random seeds and ten evaluation runs per seed. e07-setupe08-simulatione19-baselines
What it supports. CogRobot exceeds DP on all four tasks and reaches the highest listed success on Put Apple Cabinet. Its gains do not imply dominance over richer observations: colored DP3 reaches 86 on Block Handover and 68.7 on Pick Apple Messy, while uncolored DP3 reaches 89 on Container Place.
Where the evidence stops. The table mixes uncertainty magnitudes such as 0.11 and 15.1 without explaining a shared scale. Preserve the original entries; do not infer significance from them. Sensors, camera choices, pretraining and multitask versus single-task training also differ across methods.
Table 2. The largest physical gain is on coordinated rope manipulation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each pair of rows compares a single-view DP baseline with CogRobot on one trained task. Values are success fractions, unlike Table 1's 0–100 display scale. Section 5.3 reports one hundred demonstrations per task and twenty evaluation trials. Lift Bag requires both arms to hold and lift the bag; Pull Box requires grasping rope ends, positioning the rope behind the box and pulling jointly. Compare rows within a task before comparing tasks: the two tasks have different manipulation requirements and were recorded from different camera viewpoints. CogRobot repeatedly generates visual goals from the current observation during deployment. e09-reale13-limite15-tasks
What it supports. Pull Box improves from 0.05 to 0.75, an absolute gain of 70 percentage points. Lift Bag improves from 0.50 to 0.70, a gain of 20 percentage points. These are measured physical success results for the two evaluated tasks, distinct from the paper's video-quality measurements.
Where the evidence stops. Twenty trials per task provide limited evidence, and no uncertainty intervals are reported. Each task has its own trained action policy. Different viewpoints across different tasks do not establish transfer to a new camera viewpoint within the same task.
Figure 4. Selected predicted and executed sequences show the intended visual-goal relationship. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose the task using the instruction box at the top. Read the upper blue dashed sequence left to right as the predicted video, then inspect the lower yellow sequence as real-world deployment. Compare the visible bag height, gripper arrangement, rope placement and box position across each task's two rows. The source pairs these examples to illustrate that predicted visual trajectories can guide actual movements. Section 3.3 supplies the missing link: the goal-conditioned policy learns actions between an observation and a desired image. The frame strips themselves do not show the controller's action sequence or the frequency of replanning. e06-controllere09-reale10-execution
What it supports. The displayed examples are consistent with the intended hierarchy: a visual model proposes intermediate appearances and a separate controller attempts to realize them. The paper identifies the lower rows as physical execution. Aggregate success must be read from Table 2, rather than inferred from the visual similarity of these selected sequences.
Where the evidence stops. These are selected stills, with no timestamps, trajectory-error metric, contact measurements or failure distribution. The paper's description of physically executable trajectories is an author interpretation of these examples, not a guarantee that every generated frame sequence is feasible.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin bimanual manipulation (four tasks) 100 demonstrations per task; 10 random seeds with 10 evaluation runs per seed. CogRobot and DP use single-view RGB; other baselines have different sensory inputs. | Put Apple Cabinet: 100.0 ± 0.0; Block Handover: 36.0 ± 0.11; Pick Apple Messy: 15.0 ± 0.07; Container Place: 55.0 ± 0.07. Mean success rate on the table's 0–100 display scale ± reported standard deviation | DP, in the same task order: 82.0 ± 0.09; 2.0 ± 0.04; 3.0 ± 0.07; 51.0 ± 0.14. Colored DP3 reaches 86.0 ± 15.1 on Block Handover and 68.7 ± 6.8 on Pick Apple Messy; uncolored DP3 reaches 89.0 ± 7.5 on Container Place. CogRobot improves over DP throughout, but leads all listed methods only on Put Apple Cabinet. Uncertainty scales appear inconsistent across rows; values are preserved literally, without rescaling or significance claims. e07-setupe08-simulatione19-baselines |
| Pull Box Physical Realman dual-arm system; 100 teleoperation demonstrations for this task; average over 20 trials. | CogRobot: 0.75. Task success rate (fraction) | DP: 0.05; an absolute difference of 0.70, or 70 percentage points. Large observed gain on coordinated rope manipulation; no uncertainty interval or held-out-task transfer protocol is reported. e09-reale15-tasks |
| Lift Bag Physical Realman dual-arm system; 100 teleoperation demonstrations for this task; average over 20 trials. | CogRobot: 0.70. Task success rate (fraction) | DP: 0.50; an absolute difference of 0.20, or 20 percentage points. Supports improved success on this trained transport task. Different camera viewpoints across the two tasks do not isolate within-task viewpoint generalization. e09-reale15-tasks |
| Bimanual video prediction on RDT and RoboMIND validation clips RDT: 1,757 validation clips; RoboMIND: 5,346. Direct 2B/5B fine-tuning and the flow-guided variant use the same stated data, resolution, and clip length. | CogVideoX-2B-Flow: RDT 22.663 / 0.836 / 0.097 / 760; RoboMIND 21.977 / 0.864 / 0.085 / 576, in metric order. PSNR and SSIM (higher better); LPIPS and FVD (lower better) | CogVideoX-5B-SFT: RDT 20.502 / 0.806 / 0.132 / 1064; RoboMIND 19.418 / 0.837 / 0.123 / 740. The 2B-SFT rows also score worse on all four metrics. The flow-guided system improves reported video quality. This is not an executed-action ablation; Table 3 does not explicitly identify whether its evaluation conditioning flow is predicted or reference flow. e11-ablation-rdte12-ablation-robominde17-preprocessing |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Flow guidance improves every reported video metric on both validation sets. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one dataset block at a time and follow the arrows in the headers: larger PSNR and SSIM values are preferred, while smaller LPIPS and FVD values are preferred. The top two rows are direct fine-tuning baselines with different backbone sizes. The bold final row uses flow guidance. Section 5.4 states that the variants share training data, image resolution and clip length. The evaluation uses 1,757 RDT clips and 5,346 RoboMIND clips. These are comparisons against reference videos; none of the four columns measures whether a robot successfully executed a predicted plan. e11-ablation-rdte12-ablation-robominde16-architecturee17-preprocessinge18-training
What it supports. On RDT, the flow-guided row improves PSNR from the 5B baseline's 20.502 to 22.663 and lowers FVD from 1064 to 760. On RoboMIND, corresponding values change from 19.418 to 21.977 and from 740 to 576. SSIM and LPIPS improve as well, supporting the fixed-data video-quality claim.
Where the evidence stops. Table 3 does not explicitly identify whether evaluation uses generated or reference conditioning flow. It also does not match the full two-model system's compute to a direct predictor. These rows cannot establish an isolated flow contribution to executed-task success.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors explicitly acknowledge that every task requires its own action-extraction policy. Four simulation tasks and two trained physical tasks do not establish a generalist low-level controller. e08-simulatione09-reale13-limit
Two video models add capacity and inference stages relative to one direct predictor. The video ablation does not equalize total compute or isolate flow's contribution to physical success. Attention maps and selected frame sequences offer qualitative support, not measured contact accuracy. e16-architecturee11-ablation-rdte12-ablation-robominde20-attentione21-qualitative
Sensor and training protocols differ across simulation baselines. Camera descriptions also need reconciliation: Section 5.1 names D435 for RDT and L515 for other single-view methods, whereas Appendix A names a RealSense 435i for real data collection. e07-setupe14-teleoperatione19-baselines
7.2 Questions for discussion
- Does Table 3's improvement survive explicitly using generated flow with matched total inference compute?
- How much physical failure arises in visual planning versus the task-specific goal-reaching controller?
- Would a shared action controller retain the gains on held-out bimanual tasks?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the named VideoX-Fun checkpoints, FlowFormer++ preprocessing, task demonstrations, and RoboTwin-derived goal-conditioned policy. Table 5 specifies training/validation start strides 4/16 and frame interval 2 for RDT/RoboMIND; downstream training uses stride 1 and interval 4. Model and software commit hashes are not supplied. e16-architecturee17-preprocessinge19-baselines
Resolve the missing low-flow threshold, nmax, action-generation settings and deployment schedule before comparing runs. Appendix A's 60 Hz rate concerns teleoperation data collection; it is not a reported CogRobot inference rate. Exact coordinate transforms and the 'slight' joint-change acceptance threshold remain unspecified. e06-controllere09-reale14-teleoperatione17-preprocessing
Proposed checks: compare predicted, reference and shuffled flow using a fixed RGB predictor; separately compare DP with true-future, generated-future and current-image goals under identical action training. Measure both video error and task success to locate the bottleneck. e05-videoe06-controllere09-reale11-ablation-rdte18-training
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether useful motion survives the predicted-flow bottleneck
Reader-proposed check, not performed: freeze the trained flow-to-video model and evaluate identical held-out RDT/RoboMIND clips using reference flow, generated flow and temporally shuffled reference flow. Keep initial image, instruction, diffusion seeds and denoising budget fixed; preserve flow magnitudes in the shuffled control. Report all four Table 3 metrics and explicitly label each flow source. A large reference-to-generated drop would locate an upstream prediction bottleneck. Similar scores for correctly ordered and shuffled flow would weaken the claim that the RGB predictor uses coherent temporal motion guidance. This also resolves the original table's unspecified evaluation-flow provenance. e04-flowe05-videoe11-ablation-rdte12-ablation-robominde17-preprocessinge18-training
Check 2: Locate the visual-planner versus action-controller failure
Reader-proposed check, not performed: on RoboTwin Put Apple Cabinet and Block Handover, train the same goal-conditioned controller on the same 100 demonstrations per task. Evaluate paired initial resets with goals from the flow-guided predictor, the direct-SFT predictor and held-out reference rollouts at matched goal horizons; include ordinary DP as a baseline. Fix action execution chunks and replanning intervals, and report their actual values. Use the paper's ten-seed, ten-run protocol and record task success plus goal-image error. A large reference-goal advantage would implicate planning; poor success even with reachable reference goals would implicate action grounding. Better video scores without better execution would weaken the assumed connection between visual fidelity and control. e06-controllere07-setupe08-simulatione11-ablation-rdte16-architecture
8.3 Reading coverage
Visual audit: The title/byline/version on p. 1 and all method, training, evaluation and reproduction evidence pages were rendered and visually inspected. This includes Figure 1 on p. 3, diagnostic Figures 2–3 on pp. 4–5, equations on pp. 5–6, experimental protocols on p. 7, Figure 4 and Tables 1–2 on p. 8, Figure 5 and Table 3 on p. 9, teleoperation Figure 6 and Eq. (5) on p. 14, tasks and controller details on p. 15, architecture Figure 7 and the full training recipe on p. 16, dataset Table 5 and baseline details on p. 17, and qualitative Figures 8–9 on pp. 18–19. All six final crops were separately inspected. Figure 1's flow-to-video arrow and Figure 7's noise/concatenation markers were checked against Sections 3.1–3.3 and Appendix C.1; the omitted initial-image branch and differing text-conditioning notation are disclosed. Reference pages 10–13 were read as text but not rendered. External videos, code, datasets and separate supplements were outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Preliminaries
- 3 Methods
- 3.1 Text-to-Flow Generation
- 3.2 Flow-to-Video Prediction
- 3.3 Diffusion Policy from Videos
- 4 Related Works
- 5 Experiments
- 5.1 Experiment setup
- 5.2 Main Results in Simulation Setup
- 5.3 Real-World Experiments
- 5.4 Visualization and Ablation
- 6 Conclusion
- Acknowledgments
- References
- A Details of Data Collection
- B Details of Real-world Experiments
- C Implementation Details
- C.1 Architecture details
- C.2 Datasets
- C.3 Training details
- C.4 Details of baselines
- D More Visualization on Robotwin
- E More Visualization on Realman
Outside the original text pass
- Identity/version: the inspected title page identifies arXiv:2505.24156v1 [cs.CV], 30 May 2025, and 'Preprint. Under review.' Its exact title and all seven authors match the catalog. No revision or edition discrepancy is observed; no other version was supplied or compared (e01-identity).
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation above was addressed by inspecting original PDF pages, all nine figures, all five tables, and every final crop. All six supplied text chunks were read in full, including the references and Appendices A–E.
- Separate supplemental material availability has not been fully verified.
- No separate supplement, external video, linked implementation, checkpoint, or dataset was inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01-identityPDF p. 1, title, byline, institutional superscripts, arXiv margin and preprint footer
The exact title matches the catalog. The byline lists Chenyou Fan, Fangzheng Yan, Chenjia Bai, Jiepeng Wang, Chi Zhang, Zhen Wang and Xuelong Li. The margin identifies v1, 30 May 2025; the footer says preprint under review. Three institutions are printed.
Go to primary source ↓e02-framingPDF p. 2, Introduction and Section 2; p. 3, Figure 1 and opening of Section 3
The work introduces CogRobot's two-stage flow/video predictor and separate action extraction. Observations are external RGB images, actions are combined joint positions, and the physical platform has two 7-DoF Realman arms. Figure 1 sends stage-one flow into stage-two video prediction.
Go to primary source ↓e03-failuresPDF pp. 3–4, Section 3.1 and Figure 2(a–e)
The authors illustrate hallucinated hands/arms, incorrect subtasks and inaccurate movements in direct video generation, motivating optical-flow guidance.
Go to primary source ↓e04-flowPDF p. 4, Section 3.1, flow-sequence definition; p. 5, Eq. (1), Eq. (2) and surrounding text
Flow is anchored to the initial observation. Displacement direction and magnitude become RGB direction color and saturation. A frozen VAE encodes flow videos, and the first transformer is trained by noise prediction conditioned on observation and language.
Go to primary source ↓e05-videoPDF p. 6, Section 3.2 and Eq. (3)
Video generation conditions on initial observation, language and flow. Encoded flow is concatenated with noisy video latents; the diffusion loss predicts the added video noise.
Go to primary source ↓e06-controllerPDF p. 6, Section 3.3 and Eq. (4)
A separate Diffusion Policy learns action sequences from initial and desired observations with uniformly sampled goal step n between 1 and nmax. The displayed loss uses s0/sn despite image notation in the preceding prose; no numerical nmax is given here.
Go to primary source ↓e07-setupPDF p. 7, Sections 5.1–5.2, architecture, baselines and evaluation paragraphs
The source specifies 100 demonstrations per task, multitask video/RDT training versus task-specific controller/other-baseline training, different camera choices, and evaluation over 10 seeds with 10 runs per seed.
Go to primary source ↓e08-simulationPDF p. 8, Table 1, all four task blocks and input-type labels
CogRobot/DP means are 100/82 for Put Apple Cabinet, 36/2 for Block Handover, 15/3 for Pick Apple Messy and 55/51 for Container Place. DP3 and RDT use other input modalities; several outperform CogRobot. The printed uncertainties mix values such as 0.11 and 15.1 without clarifying a common scale.
Go to primary source ↓e09-realPDF p. 8, Section 5.3 and Table 2, Lift Bag and Pull Box columns
Each task has 100 human demonstrations and 20 evaluation trials. Success fractions are Lift Bag 0.70 versus DP 0.50 and Pull Box 0.75 versus DP 0.05. Goals are regenerated from current observations; the two tasks are recorded from different viewpoints. No uncertainty interval or numerical replanning schedule is supplied.
Go to primary source ↓e10-executionPDF p. 8, Figure 4 and caption; Section 5.3, final paragraph
Blue dashed rows are predicted videos and yellow rows are actual real-world execution for Lift Bag and Pull Box. The authors describe close correspondence; the figure provides selected frames without numerical trajectory error.
Go to primary source ↓e11-ablation-rdtPDF p. 9, Section 5.4 and Table 3, RDT rows
On 1,757 RDT validation samples, 2B-SFT scores PSNR 19.677, SSIM 0.784, LPIPS 0.151, FVD 1222; 5B-SFT scores 20.502, 0.806, 0.132, 1064; 2B-Flow scores 22.663, 0.836, 0.097, 760. Data, resolution and length are described as matched; evaluation flow provenance is not explicitly stated.
Go to primary source ↓e12-ablation-robomindPDF p. 9, Section 5.4 and Table 3, RoboMIND rows
On 5,346 official validation samples, 2B-SFT scores PSNR 18.928, SSIM 0.821, LPIPS 0.136, FVD 848; 5B-SFT scores 19.418, 0.837, 0.123, 740; 2B-Flow scores 21.977, 0.864, 0.085, 576.
Go to primary source ↓e13-limitPDF p. 9, Section 6, final sentence
The explicit limitation is that each task needs a separate policy to extract actions from videos.
Go to primary source ↓e14-teleoperationPDF p. 14, Appendix A, Figure 6 and Eq. (5); p. 15, Appendix A continuation
Vision Pro wrist poses undergo coordinate-frame and pose-alignment transformations; normalized finger distance controls grippers. Appendix A names a RealSense 435i camera and 60 Hz data-collection control, accepting valid IK solutions with only slight joint changes. Numerical transforms and that acceptance threshold are not given.
Go to primary source ↓e15-tasksPDF p. 15, Appendix B and Table 4
Pull Box involves grasping rope ends, positioning the rope behind the box and pulling jointly; Lift Bag requires grasping both handles and lifting together.
Go to primary source ↓e16-architecturePDF p. 15, Appendix C.1; p. 16, Figure 7 and caption
The actual starting checkpoints are community VideoX-Fun CogVideoX-Fun-V1.1-2b-InP and the corresponding 5b-InP ablation model. VAEs are frozen; the flow transformer is fine-tuned. The video model integrates text, noisy RGB and flow latents. Figure 7 marks noise on the vision branch and channel concatenation on conditioning branches. The controller concatenates goal and observation-history ResNet-18 features.
Go to primary source ↓e17-preprocessingPDF p. 16, Appendix C.2; p. 17, Table 5, all rows
The source uses RDT and RoboMIND's AgileX subset; reserves 5% of RDT episodes and uses official RoboMIND validation. Clips are 256×256 with 17 frames; FlowFormer++ estimates frame-0-to-1…16 flow, with a white frame prepended. Low-motion filtering lacks a numerical threshold. Table 5 supplies sample counts, start strides and down-sampling intervals.
Go to primary source ↓e18-trainingPDF p. 16, Appendix C.3, both paragraphs
Flow and video models are trained separately with reference flow targets/conditioning, first on mixed RDT/RoboMIND and then downstream. The recipe gives AdamW, 2×10⁻⁵ learning rate, 100 warm-up steps, 20k/30k initial flow/video steps, 15k RoboTwin or 5k Realman steps, four H100 80GB GPUs, and 128 samples per GPU.
Go to primary source ↓e19-baselinesPDF p. 17, Appendix C.4, DP, DP3, RDT and real-world paragraphs
DP and DP3 are adapted to dual-arm action dimensions using RoboTwin defaults. DP3 uses point clouds; RDT uses three RGB views. Real experiments compare DP with the same stated hyperparameters as simulation. Exact software commits and a full numerical hyperparameter listing are absent.
Go to primary source ↓e20-attentionPDF p. 5, Figure 3 and Section 3.2; p. 6, opening paragraph
Selected word-conditioned attention maps compare CogVideoX-5B-SFT with CogRobot. The authors interpret the latter as better localizing motion and relevant objects; no quantitative grounding metric accompanies these maps.
Go to primary source ↓e21-qualitativePDF p. 9, Figure 5; p. 18, Appendix D, Figure 8; p. 19, Appendix E, Figure 9
Qualitative panels compare reference video, reference RGB-encoded flow, generated flow and video predicted using generated flow in RoboTwin and Realman. The appendix captions identify black, blue, yellow and brown row borders; these are selected image sequences, not aggregate execution statistics.
Go to primary source ↓8.5 Primary sources
Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction ↗
PDF · 9,905 extracted words
Source fingerprint
81503a797d18a1906fce165195d996e588ced9d1cf419e7fa72b05c0bccd9694