Action Images: End-to-End Policy Learning via Multiview Video Generation
1. Paper overview
In one sentence: Encoding robot pose and gripper state as multiview images lets one video backbone generate actions, while geometric decoding and open-loop execution remain critical limits. e01e02e04e05e08e09e10e11e15e16
| At a glance | What to know |
|---|---|
| Research problem | Author claim How can video generation yield transferable executable actions? The paper argues that separate policy modules and actions without pixel grounding may fail to inherit visual generalization. It represents end-effector pose and gripper state as camera-aligned images so observation and action prediction share a backbone. e01e02e05 |
| Core mechanism | |
| A key reported result | Zero-shot real-world manipulation: Place Cup 40; Pick Unseen Toy 20; Pick Tissue 15; Close Drawer 45; Close Box 10. Task success rate (%). One-trial open-loop xArm; arm, objects and environments unseen; instruction forms resemble training. MolmoAct: 10, 5, 5, 5, 0; π0.5: 5, 0, 0, 0, 0; other listed baselines: all zero. Best listed on each task, but modest absolute success. Trial counts and uncertainty are absent. e08e09 |
| Reading caution | Reader analysis Figure 4 appears to expose an initial blue action slice during joint generation, whereas Section 3.3 specifies only the first observation frame as unmasked. The exact initial-action conditioning is unresolved. e05 |
Core contributions
Figure 2. Pose becomes spatial structure that a video backbone can predict. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the robot frame on the left and follow each colored connection into its channel. Red marks end-effector position. Green marks a point displaced along the gripper-plane normal, and blue marks the up point; their relative 3D locations encode orientation. These are semantic points derived from the action, rather than arbitrary tracked object features. Equations (1)–(3) define their geometry and projection. The blue sheet has a second role that the heatmap rings alone cannot show: Equation (5) retains responses above 0.25 and writes the binary gripper signal, multiplied by 0.25, into the remaining background. e02e03
What it supports. The image representation carries pose through locations and gripper state through background intensity. This makes the action target spatially aligned with observations and compatible with the same video encoder. It also makes camera calibration and subsequent geometric reconstruction part of the policy’s information path.
Where the evidence stops. Equation (7) selects blue values strictly below 0.25, excluding the exact background value encoded for an open gripper. The resulting empty selection for an ideal open-state image is unresolved; the diagram does not supply a threshold correction.
2. Motivation
2.1 The problem and the proposed response
How can video generation yield transferable executable actions? The paper argues that separate policy modules and actions without pixel grounding may fail to inherit visual generalization. It represents end-effector pose and gripper state as camera-aligned images so observation and action prediction share a backbone. e01e02e05
2.2 What this reading follows
Action Images asks whether the visual knowledge in a video generator can become a robot policy through a better action representation. Its answer is to draw actions as camera-aligned heatmaps, predict them alongside future observations, and reconstruct controls from agreement between views. Read the method in three stages: encoding, joint generation and geometric decoding. Then separate the evidence for executed task success from the evidence for video fidelity and motion labeling. The zero-shot results favor this representation across several held-out settings, while the optional action-head comparison exposes a substantial decoding gap. The supplied v2 also leaves conflicting backbone specifications and several implementation details unresolved. e01e02e04e05e08e09e10e11e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
One shared backbone jointly predicts future observation and action-image latents; deterministic geometry recovers controls. This supports One Model × Joint prediction. Video-to-action labeling is an alternate conditioning mode, not the principal two-stage inverse-dynamics policy. The optional supervised MLP qualifies any blanket claim that every evaluated variant lacks an action head. e04e05e06e10
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a pose into something the video model can see
Begin with the action vector rather than the generated movie. Its position fixes one 3D point; two short offsets derived from orientation fix a local frame around the gripper. Camera projection converts those points into three 2D targets per view. Gaussian rendering spreads each target over nearby pixels, giving the video backbone a spatial prediction problem. The blue channel also carries the binary gripper signal in its background. The supplement sets the point distance to 0.1 and Gaussian width to 0.05 relative to image resolution, while leaving the distance unit unstated. Keep the representation claim separate from a guarantee of invertibility: the printed gripper encoding and decoding thresholds disagree at the exact open-state value, so a faithful implementation must investigate that boundary. e02e03e15
Figure 3. A second view converts an image-space prediction into a 3D action point. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the numbered stages rather than treating the curved trajectory as a directly generated 3D output. First compute a weighted heatmap centroid in the main view, as Equation (8) specifies. The blue ray then contains candidate points at different depths. The green dashed lines show where candidates land in the side-view image. Although the diagram labels this step “Back Projection,” Equation (9) explicitly projects 3D candidates into the side view and selects the strongest heatmap response. Repeat the procedure for position, normal and up points. Their reconstructed offsets provide orientation; decoding the blue background supplies the remaining gripper component. e03e04
What it supports. The backbone generates action images, while calibrated geometry produces continuous controls. Multiple views therefore do more than provide extra observations: they help resolve depth ambiguity in the output representation. The authors’ low-loss decoding argument is conditional on accurate heatmaps and adequate ray and image resolution.
Where the evidence stops. No numerical near/far bounds, ray sample count or calibration-noise sweep is supplied. Nor does the source fully specify how inconsistent reconstructed direction vectors are corrected into a valid rotation. The schematic alone cannot establish execution precision.
5.2 Learn one conditional generator, then choose the visible blocks
Training places observation and action images in a common VAE latent space, packs the streams temporally, and changes which tokens the model must recover. For policy use, the text specifies an initial observation with future observation and action tokens hidden, although the illustrated first-action slice leaves the exact mask ambiguous. Given actions instead, the model predicts observations; given a video, it predicts action images. Video-only data remain useful when camera/action alignment is inadequate, which is how BridgeV2 enters training after camera estimation. The main objective penalizes masked velocity prediction. The supplementary task mixture heavily favors joint generation, allocating 85% to it. This explains the shared training setup without treating every dataset as if it supplied equivalent action supervision. e05e06e07e15
Figure 4. Task switching changes which latent blocks are observed and predicted. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend first: yellow denotes video, blue action, speckled blocks noise, green text and pink cameras. The two view rows pass through the same backbone. Batch 2 exposes action blocks while predicting video; Batch 3 exposes video while predicting actions. The prose also describes video-only training, omitted from this graphic. In Batch 1, a first blue action slice appears visible alongside the initial yellow observation slice, whereas the text says only the first observation frame is unmasked. That mask detail remains unresolved. For the loss, follow Equation (11): the squared error applies to masked latent velocities, not directly to displayed RGB pixels. e05e06e15e16
What it supports. This architecture supports joint prediction because observation and action-image latents share the generative computation. Alternate masks let the same learned model act as an action-conditioned predictor or a video labeler. These alternate capabilities do not imply that the main policy first predicts video and then invokes a separate learned inverse-dynamics model.
Where the evidence stops. The graphic names Wan 2.2, but supplementary training names Wan2.1-I2V-14B-480P, and Table 7 lists a 5B model. The shared-backbone mechanism is clear; the checkpoint and variant responsible for each reported experiment are not reconciled.
5.3 Separate reconstruction accuracy from policy generalization
After generation, the main-view centroid constrains a point to a ray, and the side view chooses a depth. A small localization or calibration error can therefore affect the recovered control even when the future RGB frames look plausible; this is a reader inference from the reconstruction procedure. Evaluate three distinct outputs: visual fidelity, trajectory accuracy and executed task completion. Table 4 measures the first two, while Tables 2 and 3 report success after action decoding under open-loop execution. The optional head’s gain suggests a useful decoding bottleneck, but does not identify its cause because the head adds supervision and capacity. The source’s explicit closed-loop limitation matters here: no online correction is demonstrated to rescue an initially inaccurate action sequence. e04e08e09e10e11e14
5.4 Training and inference
During training
Masks hide both future streams, only video, only action, or enable video-only training. Masked flow matching fine-tunes the pretrained backbone. Frozen modules and full noisy-latent/time conditioning are not explicitly specified. e05e06
Table 1 lists DROID 80k trajectories/2 views, RLBench 180k/4 views and BridgeV2 30k/1–4 views. DROID calibration is filtered; RLBench receives Robot-Colosseum backgrounds. BridgeV2 lacks camera labels and usable action-camera alignment, so VGGT estimates cameras and it supplies video-only supervision. e07
Supplementary settings: Bridge/RLBench/DROID ratios 0.2/0.5/0.3; 85% joint generation and 5% each other mode; 41 frames per view/modality, 164 total; 32 A100 GPUs, DeepSpeed ZeRO, bfloat16, gradient checkpointing, per-device batch 1; 100,000 steps, learning rate 5×10⁻⁷, 1,000 warmup steps and clipping norm 1.0. e15
During inference
Given instruction, cameras and initial views, sample both future streams, decode actions and execute without online replanning. The supplement specifies guidance scale 10, 50 denoising steps and four-GPU Unified Sequence Parallelism; VGGT supplies missing calibration. e08e15
The optional in-domain MLP consumes output video latents, cameras, decoded actions and observations to regress continuous actions. Its gains belong to a distinct supervised decoder variant. e10
5.5 Implementation flow
- Render control as image structure
For action a_t = [p_t, θ_t, g_t], retain position and offset two points along rotated canonical axes. Project through each camera. Red encodes position, green normal and blue up; low-response blue background stores binary gripper openness. The supplement specifies point offset 0.1, without an explicit unit, and Gaussian width 0.05 relative to image resolution. e02e03e15
- Pack both video streams
A 3D-VAE encodes observation and action streams, concatenated temporally within each view. Weights are shared across views. Text and camera conditions enter the backbone; Plücker features are added between spatial and subsequent 3D attention. e05
- Recover controls geometrically
A weighted main-view heatmap centroid anchors a camera ray. Sample candidate depths, project them into the side view and select the strongest heatmap response. Repeat for all points. Position follows directly; normalized point differences and their cross product determine orientation. Gripper decoding averages low-response blue pixels, subject to the printed threshold inconsistency below. e03e04
6. Experiments & results
Action Images makes robot control a multiview video-generation target: a shared backbone predicts future observations and RGB action heatmaps, which geometry converts into 7-DoF controls. Reported transfer is open-loop; an optional learned head improves in-domain success, and conflicting implementation specifications limit exact reproduction.
6.1 Read the original evidence
Table 2. Zero-shot gains depend on which aspect of the task is held out. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the RLBench and Real column groups separate. For RLBench, the specific tasks are removed from that dataset’s training split, but the arm and environment are seen and equivalent tasks may occur in other datasets. For Real, xArm, objects and environments are unseen. Both use the Section 4.1 one-trial open-loop setting with similar instruction forms to training. Compare each column vertically: Ours reaches 60 on reach target and 45 on real Close Drawer, but its 15 on close laptop is below π0.5’s 20. The baselines include author-adapted camera-aware or multiview interfaces, not simply unchanged published checkpoints. e08e09e10
What it supports. All five real tasks favor Action Images: its displayed success rates are 40, 20, 15, 45 and 10. MolmoAct reaches 10, 5, 5, 5 and 0 on those same tasks. This is evidence of improved transfer under the specified open-loop protocol, with substantial failure rates still remaining.
Where the evidence stops. The paper omits evaluation trial counts and uncertainty. Zero-shot RLBench is not task exclusion across the entire training mixture. Close drawer also appears in the in-domain table without explaining how those evaluation splits differ.
Table 4. Better generated videos and better 3D trajectories are related but distinct findings. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the direction arrows and dataset scope before comparing values. PSNR and SSIM increase with better scores; FVD, LPIPS and both action errors decrease. The video columns cover in-domain RLBench, Bridge and DROID, whereas action errors use RLBench only. The dagger identifies Cosmos-Predict2.5-14B as a zero-shot model, and its action columns are absent. Action Images reports 23.48 PSNR, 78.62 SSIM, 143.74 FVD and 0.209 LPIPS. In the final columns it reports 1.61 and 12.2; preserve the printed 3DErr ×10³ scaling because the source does not define physical error units. e11
What it supports. The method leads all displayed video-quality columns and has the lowest displayed 3D error. The exception matters: TesserAct-RGB has lower 2D error, 1.55 versus 1.61. Thus the table supports a particular combination of visual and 3D accuracy, not a uniform advantage on every prediction metric.
Where the evidence stops. Prediction quality is not executed task success. Evaluation sample counts, uncertainty and the detailed TesserAct-RGB configuration are not supplied. The action columns and video columns also use different dataset scopes, so their gains cannot be interpreted as paired improvements on every example.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot real-world manipulation One-trial open-loop xArm; arm, objects and environments unseen; instruction forms resemble training. | Place Cup 40; Pick Unseen Toy 20; Pick Tissue 15; Close Drawer 45; Close Box 10. Task success rate (%) | MolmoAct: 10, 5, 5, 5, 0; π0.5: 5, 0, 0, 0, 0; other listed baselines: all zero. Best listed on each task, but modest absolute success. Trial counts and uncertainty are absent. e08e09 |
| Zero-shot RLBench manipulation Tasks removed from RLBench training; arm/environment seen; equivalent tasks may occur in other datasets; open-loop. | Pick cup 30; reach target 60; close drawer 50; close laptop 15. Task success rate (%) | π0.5: 0, 5, 35, 20; MolmoAct: 20, 5, 10, 0. Leads three tasks but loses close laptop to π0.5; this is not whole-mixture task exclusion. e08e09 |
| In-domain RLBench and optional action head Nine listed tasks, same open-loop policy protocol. | 20.6 without the head; 36.7 with the head. Average task success rate (%) | TesserAct 20.6; Cosmos-Policy 20.0; MV-Diffusion Policy 17.8. Head gain: 16.1 percentage points, reader-calculated. Decoder capacity and supervision change together; this does not isolate pixel grounding. e08e10 |
| Video-and-action joint generation Video: in-domain RLBench, Bridge and DROID; action: RLBench only; sample counts unspecified. | 23.48; 78.62; 143.74; 0.209; 1.61; 12.2. PSNR↑; SSIM (%)↑; FVD↓; LPIPS↓; 2DErr↓; 3DErr ×10³↓ | TesserAct-RGB: 20.31; 60.19; 147.83; 0.372; 1.55; 14.2. Video metrics and 3D error favor Action Images; 2D error favors TesserAct-RGB. Physical action-error units are unspecified. Prediction fidelity is separate from task success. e11 |
| Action-conditioned video generation Given actions; evaluation dataset/split and sample count unspecified in Section 4.2. | 31.35; 67.16; 115.02; 21.78. PSNR↑; SSIM (%)↑; LVD↓ (table label); LPIPS (%)↓ | Tora: 19.76; 52.43; 187.41; 39.62. All displayed columns favor the method. Prose says FVD whereas Table 5 prints LVD; unresolved. e12 |
| Video-to-action labeling Input-video motion inference; evaluation dataset/split and sample count unspecified. | 5.785; 64.92; 46.71. Trajectory error↓; Jaccard@4↑; average Jaccard↑ | CoTracker: 12.91; 46.15; 31.20; TAPIR: 14.80; 40.26; 29.77. Tracking accuracy supports motion labeling; it does not establish human-to-robot execution transfer or full 7-DoF labeling accuracy. e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The learned action head reveals room for improvement in continuous control. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the last two rows together before comparing against other models. Ours uses the reconstruction-based decoder; w/ action head adds a supervised MLP over video latents, camera parameters, decoded actions and observations. The average rises from 20.6 to 36.7. Several columns illustrate the change: close box rises from 55 to 80 and open bottle from 5 to 40, while open door rises from 0 to 15. The base average equals TesserAct’s 20.6; the stronger average belongs to the additional-head variant. Section 4.1 retains the demanding open-loop setting for this in-domain comparison. e08e10
What it supports. The reader-calculated average gain is 16.1 percentage points. It shows that the generated representation can support stronger supervised action decoding. It also prevents attributing the strongest in-domain score to geometric decoding alone, or claiming that the base model dominates all listed in-domain policies.
Where the evidence stops. This comparison adds both decoder capacity and supervision, so it does not isolate multiview geometry or pixel grounding. A zero-shot evaluation of the same optional head is not reported, and average success remains far from universal completion.
7. Analysis & limitations
7.1 What the evidence leaves open
Figure 4 appears to expose an initial blue action slice during joint generation, whereas Section 3.3 specifies only the first observation frame as unmasked. The exact initial-action conditioning is unresolved. e05
Closed-loop control is future work. No isolated single-view or calibration-noise ablation establishes the causal contribution of multiview grounding. e14e04e10
Equation (5) writes 0.25 when binary g_t=1, but Equation (7) averages only values strictly below 0.25. An exact encoded open-state image therefore gives an empty selection. A working threshold/tolerance is unresolved; the equations are not silently repaired. e03
Close drawer appears in both zero-shot and in-domain RLBench tables without split reconciliation. Success trial counts, uncertainty, exact evaluation samples and action-error units remain unspecified. e09e10e11e12
Section 4.3 swaps Figure 5/6 references: captions identify Figure 5 as xArm execution and Figure 6 as FR3M generation. FR3M images do not establish physical success. e13
7.2 Questions for discussion
- Would calibrated ground-truth action images round-trip under the printed gripper threshold? (e03, e04)
- Does the learned head improve held-out embodiment transfer with generation held fixed? (e09, e10)
8. Reproducibility audit
8.1 Requirements and known gaps
Resolve checkpoint identity: main text/Figure 4 say Wan 2.2, supplementary training says Wan2.1-I2V-14B-480P, and Table 7 labels Ours 5B. Four-GPU inference prose differs from one/eight-H100 table configurations. The source does not map these variants to results. e05e15e16
Table 7 reports 49.1 s on one H100/50 steps, 11.8 s on eight H100s/50 steps and 2.3 s with caching on eight H100s/16 steps, for 164 packed 512×512 frames. The claimed 71 FPS is packed-frame throughput, not demonstrated control frequency; caching and step reduction are confounded. e15e16
Missing prerequisites include ray near/far bounds and sample count, orientation handling for inconsistent points, DROID filtering thresholds, exact splits and software versions. No quantitative sweep supports claimed insensitivity to point offsets/Gaussian width. Proposed checks below test decoding fidelity and the action-head gain. e04e07e10e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Round-trip the action representation before training a policy
Reader-proposed check, not performed: render known 7-DoF poses with two exact cameras, point offset 0.1 and Gaussian width 0.05 as specified. Decode the resulting ideal images for both binary gripper states. Compare the literal Equation (7) strict threshold with an explicitly labeled inclusive/tolerance variant; record empty selections, gripper error, position error and orientation error. Then vary only ray spacing or camera perturbation while holding images and all other settings fixed. The exact open-state encoding should expose the threshold problem; absent that issue, finer ray sampling should reduce discretization error under accurate calibration. Failure of that trend would challenge the stated decoding explanation. This check requires documenting source-omitted ray bounds and conventions rather than claiming they were provided. e02e03e04e15
Check 2: Hold generated sequences fixed when testing the action head
Reader-proposed check, not performed: resolve the checkpoint discrepancy, freeze one common generator and save identical predictions for every evaluation episode. Compare geometric decoding with the supervised MLP using matched in-domain training data, camera calibration and open-loop execution. Also evaluate both on the same held-out RLBench tasks and unseen-arm real-world protocol, with explicit splits, equal trial counts and reported uncertainty. Record per-task success and pose errors, keeping video metrics unchanged by construction. An in-domain gain with no held-out gain would support decoder fitting without establishing improved transfer; a persistent held-out gain would show that the optional head can preserve the representation’s transfer advantage. Verify the duplicated close-drawer task naming before treating the splits as distinct. e04e05e08e09e10e15
8.3 Reading coverage
Visual audit: Visually inspected the title/author/revision page; all original Figures 1–10 and Tables 1–7; method Equations (1)–(11); evaluation protocols; limitations; and supplementary training, inference and qualitative pages. Every final crop was separately viewed at its final 300-DPI extraction. Figure 4’s apparent visible first action slice differs from the prose mask description; backbone specifications also differ. Figure 3’s projection is interpreted using Equation (9). Figures 5/6 are identified by their captions because the prose swaps their references. Bibliography pages 20–25 and introductory/related-work pages 2–4 were read in text, without a separate page-image inspection. No external videos, code or separate supplements were inspected.
PDF pages inspected for this edition: 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author/affiliation block and arXiv:2604.06168v2, 15 April 2026 (p. 1)
- Abstract and Section 1 Introduction (pp. 1–3)
- Section 2 Related Work, including 2.1–2.3 (pp. 3–4)
- Section 3 Method, including 3.1–3.3 (pp. 4–10)
- Section 4 Experiments, including 4.1–4.3 (pp. 10–13)
- Section 5 Conclusion and Limitations; Section 6 Acknowledgement (pp. 13–15)
- Supplementary Material: 1 Implementation Details; 2 More Zero-shot Qualitative Results (pp. 16–19)
- Bibliography (pp. 20–25); all five supplied text chunks read individually
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The retained PDF resolves the extraction-only visual omission: Figures 1–10 and Tables 1–7 were visually inspected. Separate external supplements were not supplied or inspected.
- Identity/version note: the title and all ten authors match the catalog. The inspected PDF is v2 dated 15 April 2026; the catalog records submission on 7 April 2026. No earlier edition or revision history was supplied, so changes from v1 cannot be established.
- Code and linked project resources were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block, revision stamp, Figure 1 and abstract
Title and all ten authors agree with the catalog. The artifact is arXiv:2604.06168v2, 15 April 2026. Affiliations: UMass Amherst, NVIDIA, Harvard University and Genesis AI. The abstract motivates pixel-grounded action images.
Go to primary source ↓e02PDF pp. 5–6, Section 3.1, Figure 2 and Equations (1)–(6)
Position, normal and up points represent pose; camera projections become red, green and blue Gaussian channels aligned with observations.
Go to primary source ↓e03PDF p. 6 Equation (5) and p. 7 Equation (7), blue-channel gripper encoding/decoding
Encoding keeps Gaussian responses above 0.25 and otherwise writes 0.25 times binary gripper state. Decoding selects blue values strictly below 0.25 and divides their mean by 0.25.
Go to primary source ↓e04PDF pp. 7–8, Section 3.2, Figure 3, Equations (8)–(9), pose recovery and discussion
The main heatmap centroid anchors ray candidates; side-view scores select depth. Three points determine pose. Under accurate heatmaps, the authors attribute remaining error to image resolution and ray sampling.
Go to primary source ↓e05PDF pp. 8–9, Section 3.3, Equation (10), Figure 4 and mask/camera-conditioning paragraphs
The main text identifies Wan 2.2, VAE-encoded video/action temporal packing, shared weights across views, four conditioning modes and Plücker camera-feature injection. Figure 4 illustrates three mask patterns.
Go to primary source ↓e06PDF pp. 9–10, Optimization objective and Equation (11)
Masked flow matching uses target velocity v = epsilon − X and squared error over masked tokens, with text and camera conditions.
Go to primary source ↓e07PDF p. 10, Table 1 and Training datasets paragraph
DROID 80k, RLBench 180k and BridgeV2 30k trajectories. The prose specifies calibration filtering, background augmentation and video-only BridgeV2 supervision with VGGT-estimated cameras.
Go to primary source ↓e08PDF p. 10, Section 4.1 opening evaluation protocol
Text and initial multiview observations condition joint generation; action images are decoded. Unless otherwise specified, Section 4.1 uses one-trial open-loop evaluation without online replanning.
Go to primary source ↓e09PDF p. 11, Table 2 (all task columns) and Zero-shot policy results
The table reports simulation and real-world success. Prose defines differing zero-shot scopes and describes camera-aware π0.5, multiview MolmoAct trajectories, and Wan-based TesserAct/Cosmos-Policy reproductions.
Go to primary source ↓e10PDF pp. 11–12, RLBench in-domain results and Table 3 (Ours, w/ action head and baseline rows)
An MLP consumes video latents, cameras, decoded actions and observations. Average success is 20.6 without it and 36.7 with it, versus TesserAct 20.6 and Cosmos-Policy 20.0.
Go to primary source ↓e11PDF p. 12, Table 4 (headers and all model rows), Joint generation quality paragraph
Videos use in-domain RLBench, Bridge and DROID; actions use RLBench. Ours reports 23.48/78.62/143.74/0.209 and action errors 1.61/12.2 under displayed scaling. TesserAct-RGB has 2D error 1.55.
Go to primary source ↓e12PDF pp. 12–13, Section 4.2 and Tables 5–6
Action-conditioned generation compares with Tora; Table 5 labels LVD where prose says FVD and displays LPIPS as percent. Table 6 compares motion tracking with TAPIR/CoTracker; the evaluation dataset/split is unspecified.
Go to primary source ↓e13PDF pp. 13–15, Section 4.3 and Figures 5–6 with captions
Prose references xArm as Figure 6 and FR3M as Figure 5, opposite the captions. Figure 5 includes Real Exec and Veo3.1 rows; Figure 6 shows generated mouse-picking imagery and LTX-2 comparisons.
Go to primary source ↓e14PDF p. 14, Limitations paragraph
The authors state closed-loop policy development remains incomplete and propose distillation and closed-loop integration.
Go to primary source ↓e15PDF p. 16, Supplementary Material Section 1: Training Details, Inference Details and Action Images Details
Names Wan2.1-I2V-14B-480P; provides data/task mixtures, 41/164 frames, hardware and optimization settings, four-GPU inference, guidance 10, 50 steps, point offset 0.1 and Gaussian width 0.05; claims up to 71 FPS.
Go to primary source ↓e16PDF p. 17, Table 7, Ours / + Parallelism / + Caching rows and configuration headers
Labels Ours 5B, with one/eight H100 GPUs; configurations take 49.1, 11.8 and 2.3 seconds for 164 frames at 512×512. Caching also reduces denoising steps from 50 to 16.
Go to primary source ↓8.5 Primary sources
Action Images: End-to-End Policy Learning via Multiview Video Generation ↗
PDF · 8,317 extracted words
Source fingerprint
f0b6d8aad562ddd7ba69e803c218e5d146ecfcee7fbcef4d833c05da40d6abaf