Learning Universal Policies via Text-Guided Video Generation
1. Paper overview
In one sentence: UniPi shares planning through text-conditioned videos, but translating those plans into reliable behavior still depends on a separate action model and executable image trajectories. e02e04e06e10e12e14e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Different environments have incompatible state/action spaces, while reward design is task dependent. UPDP proposes images and text as shared interfaces, allowing a planner to learn from videos without requiring action labels for every training example. Environment-specific control remains necessary, and learning now depends on suitable video/task descriptions. e02e06 |
| Core mechanism | |
| A key reported result | Combinatorial manipulation: novel instructions: Novel Place 60.1 ± 3.9; Novel Relation 46.1 ± 3.0. Task completion accuracy (reported table scale). Scripted simulated block coloring and placement; 30% of language instructions held out, randomized configurations, joint-space execution. Best listed baseline per column: Image + TT 13.2 ± 4.1; Diffuser 9.6 ± 1.7. Strong gains on withheld prompt combinations within this simulator. The source does not define the ± statistic or evaluation episode count. e09e10 |
| Reading caution | Author claim The authors report that generating photorealistic videos can take a minute and identify hallucination risks under partial observability. Their stated 16× distillation speedup is initial experimentation, not a fully specified control-latency benchmark. e18 |
Core contributions
Figure 2. A video plan passes through a separate control decoder. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left with the observed scene and text. The Tiling arrow repeats the observed image as context; it does not advance the environment. Follow the downward arrows through Video Diffusion to the sparse frame strip, then through Temporal Super Resolution to the denser strip. Noise enters both generative stages from the left. Only after this visual plan exists does the path move right into Inverse Dynamics and upward to Robot Actions. Sections 3.1–3.2 confirm this direction of information flow. The planner's output is an image trajectory, while the controller provides the environment-specific interpretation needed to execute it. e02e04e05e06e09
What it supports. The architecture makes the division of labor visible: observation-conditioned video synthesis handles the intended visual evolution, temporal refinement supplies intermediate behavior, and an independently trained module translates frames into controls. The shared representation therefore belongs primarily to planning; it does not make robot action interfaces disappear.
Where the evidence stops. The action box uses schematic displacement/rotation labels. Appendix A.4 specifies the implemented controls as six joint values plus contact. Treat the drawing as an information-flow diagram, not a complete control-interface specification.
2. Motivation
2.1 The problem and the proposed response
Different environments have incompatible state/action spaces, while reward design is task dependent. UPDP proposes images and text as shared interfaces, allowing a planner to learn from videos without requiring action labels for every training example. Environment-specific control remains necessary, and learning now depends on suitable video/task descriptions. e02e06
2.2 What this reading follows
Imagine specifying a manipulation task by describing its goal, then asking a model to show the robot completing it. UniPi makes that imagined sequence the plan: video diffusion predicts future frames, temporal super-resolution fills in motion, and inverse dynamics translates the result into controls. The useful reading question is where evidence crosses from depicting behavior to executing it. This edition follows that boundary through the architecture, simulated manipulation results, component ablations and real-scene pretraining study. The latter evaluates generated videos, including a learned final-frame success judgment; its improvements do not establish physical-robot execution success. e02e04e06e10e12e14e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded snapshot is entirely unassigned, so it supplies no positive classification to confirm. Architecture evidence suggests a video planner plus separate inverse-dynamics control module, with future-video prediction used during inference. This is not one shared model jointly predicting future video and actions; the authors' joint world-model/planning language does not remove the separate controller. e02e04e06e21
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why put a video between the instruction and the controller?
UPDP begins with a representation problem: different environments expose different states and actions, but many can be observed as images and described with language. The paper therefore defines a distribution over future image sequences conditioned on an initial observation and task. A separate policy maps the resulting trajectory into the relevant action space. Reader interpretation: this moves the shared interface to intended visual behavior, allowing planner training to use videos whose action labels are absent. It also relocates a hard problem rather than solving it automatically: a generated trajectory must remain consistent with the scene and realizable by the action decoder. UniPi is consequently best understood as task-conditioned video planning plus action adaptation, not as an action-conditioned simulator that predicts consequences for arbitrary candidate controls. e02e04e06e09
5.2 Condition the scene, refine time, then execute
The method distinguishes three jobs that are easy to collapse into one description. First-frame conditioning teaches the model to begin from a specified scene. Tiling provides that same observation alongside each noisy frame, maintaining a persistent reference while the future is denoised. Temporal super-resolution then refines a sparse video into a denser plan. Appendix A.1 makes this concrete in simulation with ten-frame and twenty-frame stages, using frame spacing eight and four respectively. Finally, the separately trained inverse model regresses controls from images. Training and inference should stay separate in this explanation: the initial-image context is learned during training, classifier-free guidance steers sampling at inference, and the reported experiments execute the resulting action sequence open-loop. Table 2 evaluates the stacked video-planning mechanisms on seen instructions. e03e04e05e06e07e08e12
5.3 Read the transfer evidence at the level actually measured
The experiments answer three related questions with different measurements. Table 1 withholds language combinations and checks completed simulated tasks after action execution. Table 3 changes the manipulation task settings while retaining the simulated joint-control interface. Table 4 instead studies whether internet pretraining improves generated robot videos after Bridge finetuning. Its final-frame success classifier is a surrogate, even though the column is simply labeled Success. Reader interpretation: the first two studies provide evidence about control in their simulator, while the third provides evidence about the visual planner's transfer. Figures 8–9 and Appendix B add qualitative examples of new commands and altered backgrounds, but do not connect those generated trajectories to successful physical execution. That missing connection motivates checking plan feasibility and decoder behavior separately. e09e10e13e14e15e16e17e19
5.4 Training and inference
During training
Video U-Net learns conditional denoising using temporal convolutions. Each video model is trained for 2M steps, batch 2048, learning rate 1e-4 and 10k warmup; A.1 reports 256 TPU-v4 chips. T5-XXL supplies language embeddings. Simulation uses 1.7B-parameter stages at 10×48×64 and 20×48×64, with frame spacing eight and four. e03e07
The combinatorial planner uses 200k scripted videos, with a 70%/30% instruction split and randomized object locations. Inverse dynamics uses 20k action-annotated videos there and annotations across 200k videos in multitask training. Its mean-squared-error regression uses Adam, learning rate 1e-4, clipping at 1, 2M steps and 10k warmup. e08e09e13
Real-video training uses internet pretraining on 14M video-text pairs, 60M image-text pairs and LAION-400M, followed by Bridge finetuning: 7.2k video-text pairs split 80%/20%, with task IDs as text. A.1 lists successive real-video models from 16×40×24 to 32×320×192. e07e15
During inference
Sample the conditional video with classifier-free guidance, perform temporal super-resolution, then regress the action sequence. Although the formulation permits replanning after each executed action, the reported control experiments use open-loop action sequences for efficiency. e03e04e06
5.5 Implementation flow
- Separate planning from control
The UPDP generator models a distribution over H future images conditioned on the observation and task. The action policy is deferred until a video trajectory exists; it is not a joint video/action diffusion output. e02e04e06
- Anchor the denoising process
Train with the initial image as explicit context. Tile that image through time and concatenate it channel-wise to noisy frames. This gives every denoising position a reference to the initial environment; merely fixing the first frame during sampling performed poorly in the authors' account. e04e07
- Refine the temporal plan
Generate sparsely sampled frames, then interpolate behavior through temporal super-resolution. An optional prior h(τ) can impose additional sampling constraints; Figure 7 illustrates intermediate-image guidance selecting a particular block. e05
- Decode executable controls
An independent convolutional inverse model regresses seven simulated controls. These mean six joint values and a contact command in the implemented environment. Predicted joint states are pursued by a PyBullet controller, with simplified nearest-block attachment when contact is active. e06e08e09
6. Experiments & results
UniPi turns a language instruction and current image into a video plan, refines its timing, then uses a separately trained inverse model to produce robot controls. Simulated manipulation results support compositional and multitask transfer. Internet pretraining improves generated real-scene plans, but its reported success is a classifier judgment on imagined final frames, not measured physical execution.
6.1 Read the original evidence
Table 1. Withheld combinations remain solvable in the simulated coloring-and-placement task. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two header levels together. Seen and Novel describe whether an instruction was included in training; Place and Relation describe the requested manipulation relation. The underlying tasks require coloring an initially white block in a bowl before placing it, and scene configurations are randomized. Compare models within a column, since each column represents a different evaluation subset. State + Transformer BC observes robot state, the image baselines use visual observations, and all receive T5 language features. The Image + TT baseline predicts a sequence of controls, while Diffuser predicts controls through diffusion. The numbers are task completion accuracy entries, with the original uncertainty notation retained. e08e09e10e07
What it supports. UniPi's Novel Place entry is 60.1 ± 3.9, compared with 13.2 ± 4.1 for the strongest listed baseline in that column. Novel Relation is 46.1 ± 3.0 versus Diffuser's 9.6 ± 1.7. These comparisons support compositional transfer within the evaluated manipulation environment.
Where the evidence stops. Novel instructions do not imply new robot embodiments or realistic contact physics. The paper does not define the ± statistic or evaluation episode count, and transformer model capacity differs substantially from the video planner.
Figure 4. The generated plan and its simulated execution are related but distinct observations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left panel gives the initial scene and the requested relation between a particular cyan block and an orange block. Read the upper blue strip from left to right as the synthesized plan, then read the lower pink strip as frames from executing inferred actions. Compare where the arm and manipulated object move relative to the bowls and placement region. The paired layout is useful because it exposes deviations that a generated final image alone would conceal. However, the caption only claims rough alignment: the panels provide neither a timestamp mapping nor a numerical frame-by-frame trajectory error. e06e09e11
What it supports. This figure supplies a qualitative bridge between video synthesis and action execution in simulation. The lower strip is an executed trajectory, making its evidential role different from the paper's real-scene generation montages. It illustrates the intended correspondence while leaving the magnitude and frequency of execution deviations unmeasured.
Where the evidence stops. Section 3.2 uses open-loop plans in the experiments. Appendix A.4 also simplifies grasping by attaching the nearest block when contact is active. This example cannot establish robustness to physical grasp failures or disturbances.
Table 3. Transfer covers three held-out task settings using the same simulated control interface. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The three columns summarize the test environments enumerated in Appendix A.5: putting blocks in a bowl, grouping unseen Google objects, and packing box pairs with unseen colors. Read vertically to compare methods on the same task. Read horizontally only to understand how performance varies across those task settings; the paper does not define an aggregate score here. Training uses a collection of scripted manipulation demonstrations, while testing changes the task settings. The baseline rows follow the earlier experiment's families. The original CLIPort policy is not a row because this experiment uses predicted robot joint controls instead of its pick-and-place primitives. e09e13e14e20
What it supports. The highest UniPi entry is Pack Object at 75.5 ± 3.1, against the best listed baseline's 21.7 ± 3.5. Place Bowl and Pack Pair also favor UniPi, at 51.6 ± 3.6 and 45.7 ± 3.7. The table supports transfer across these held-out simulated tasks.
Where the evidence stops. The training inventory needs clarification: the second sentence of the bottom Setup paragraph on p. 7 specifies “10 separate tasks,” whereas A.5 on p. 15 enumerates eleven names. These two locations, not Table 3, establish the discrepancy. The shared simulated controller also limits conclusions about transfer across robot morphologies.
Table 4. Internet pretraining improves measured video quality and predicted final-frame success. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare No Pretrain and Pretrain within each metric column, following the arrows: CLIP and Success increase, while FID and FVD decrease. The header specifies a 24×40 model, so these entries should not be attributed to the largest high-resolution generation stage. Section 4.3 uses an 80%/20% split of Bridge video-text pairs and describes generation metrics computed on 32 samples. Success is obtained by feeding the last generated frame to a learned classifier, rather than executing the depicted robot motion. The preceding page's prose says higher FID/FVD, but the visible table values and downward arrows support lower values. e07e15e16
What it supports. FID changes from 17.75 ± 0.56 to 14.54 ± 0.57, while FVD changes from 288.02 ± 10.45 to 264.66 ± 13.64. Classifier success rises from 72.6% to 77.1%, a calculated 4.5-percentage-point increase. These are improvements in generated-video evaluation under the reported protocol.
Where the evidence stops. The classifier inspects only the imagined final frame. Its calibration, success-rate uncertainty and connection to executable motion are unspecified. A higher classifier score cannot be read as a measured improvement in real-robot task completion.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Combinatorial manipulation: novel instructions Scripted simulated block coloring and placement; 30% of language instructions held out, randomized configurations, joint-space execution. | Novel Place 60.1 ± 3.9; Novel Relation 46.1 ± 3.0. Task completion accuracy (reported table scale) | Best listed baseline per column: Image + TT 13.2 ± 4.1; Diffuser 9.6 ± 1.7. Strong gains on withheld prompt combinations within this simulator. The source does not define the ± statistic or evaluation episode count. e09e10 |
| Seen-task planner ablation Table 2, seen Place and Relation tasks, progressively enabling three components. | Full model: Place 59.1 ± 2.5; Relation 53.2 ± 2.0. Task completion accuracy (reported table scale) | Frame condition + consistency without hierarchy: 53.2 ± 3.0 and 39.4 ± 2.8; no components: 13.2 ± 3.2 and 12.4 ± 2.4. The hierarchy adds performance conditional on both conditioning mechanisms. Four nested configurations do not isolate all interactions. e12 |
| Transfer to three held-out manipulation tasks Multitask simulated joint-space control trained on 200k videos; Table 3 test tasks. | Place Bowl 51.6 ± 3.6; Pack Object 75.5 ± 3.1; Pack Pair 45.7 ± 3.7. Task completion accuracy (reported table scale) | Strongest listed baselines: Diffuser 14.8 ± 2.9; State + Transformer BC 21.7 ± 3.5; Diffuser 10.5 ± 2.4. All three columns favor UniPi. These are modified CLIPort environments, not a direct evaluation against the original CLIPort primitive-action policy. e13e14 |
| Bridge video generation after internet pretraining Bridge held-out split; Table 4 model resolution 24×40. Generation metrics are described as computed on 32 samples. | Pretrained: 24.54 ± 0.03; 14.54 ± 0.57; 264.66 ± 13.64; 77.1%. CLIP score ↑; FID ↓; FVD ↓; generated-final-frame classifier success ↑ | Scratch: 24.43 ± 0.04; 17.75 ± 0.56; 288.02 ± 10.45; 72.6%. Pretraining improves the reported generation measures. The 4.5-percentage-point classifier-success increase is not a physical-robot success improvement. Follow the table's lower-is-better FID/FVD arrows; p. 8 incorrectly says higher. e15e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. The largest conditioning gain appears first; temporal refinement adds further Relation accuracy. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three Yes/No columns as a sequence of added mechanisms. Frame Condition means explicitly conditioning on the initial observation during training. Frame Consistency adds the observed image as context across generated time positions; when disabled, non-start frames receive zeroed image context. Temporal Hierarchy adds coarse-to-fine video generation through temporal super-resolution. The Place and Relation columns evaluate seen instructions, so these results should not be relabeled as novel-instruction ablations. Compare neighboring rows to see what adding a component does in the presence of the earlier components, rather than treating each row as a fully independent factor test. e04e05e12
What it supports. Initial-frame conditioning raises the reported Place/Relation entries from 13.2/12.4 to 52.4/34.7. With conditioning and consistency already enabled, temporal hierarchy raises Relation from 39.4 ± 2.8 to 53.2 ± 2.0. The evidence supports the complete stack, with a particularly visible final-stage gain on relational manipulation.
Where the evidence stops. Only four nested configurations are reported, rather than all combinations of three factors. Component interactions, matched sampling costs and statistical significance are not established by this table; the meaning of ± is unspecified.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report that generating photorealistic videos can take a minute and identify hallucination risks under partial observability. Their stated 16× distillation speedup is initial experimentation, not a fully specified control-latency benchmark. e18
The simulator attaches the nearest block when contact is activated. Success under this mechanism does not establish realistic grasping robustness. Figure 4 shows rough plan/execution alignment without a trajectory-error metric. e09e11
Real-scene figures and Appendix B show generated examples, not measured physical rollouts. The final-frame classifier can miss infeasible intermediate motion; its calibration is unspecified. Transformer comparisons also differ greatly in model capacity from UniPi's billion-parameter video stages. e07e08e16e17e19
7.2 Questions for discussion
- Does temporal super-resolution improve executable motion after controlling decoder capacity and sampling cost?
- How often does a successful-looking generated final frame correspond to a physically executable trajectory?
8. Reproducibility audit
8.1 Requirements and known gaps
Retain A.1's U-Net configuration: three residual blocks, 512 base channels, multipliers [1,2,4], attention resolutions [6,12,24], head dimension 64, conditioning dimension 1024 and log-SNR range [-20,20]. Match frame spacing and the two-stage simulation plan before comparing accuracy. e07
Reconcile the Table 3 training inventory before reproduction: p. 7, Section 4.2 Setup, explicitly says “10 separate tasks,” while A.5 on p. 15 enumerates eleven training names. The source does not identify a name to exclude. The three test names are explicit. Preserve the contact threshold above 0.5 and two-second controller timeout. e09e13e20
Missing details include exact inverse-model temporal input packing, diffusion sampling steps and guidance strength, language-encoder freezing status, seed/error-bar definitions and the success-classifier training/calibration protocol. The paper's architecture and optimizer settings alone do not close these gaps. e03e07e08e10e12e14e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate temporal refinement from added sampling budget
Reader-proposed check, not performed: on the same held-out simulated scenes and seen Relation prompts, compare the full hierarchy with a single-stage planner producing the same number of final frames. Keep training videos, T5 features, action decoder and execution protocol fixed; include a control that spends an equivalent denoising budget without temporal super-resolution. Report task accuracy, generated-to-executed object-position error and measured sampling time across repeated runs, with explicitly defined uncertainty. A persistent Relation gain under comparable output length and sampling cost would support temporal refinement as the mechanism. A gain that vanishes under the compute control would weaken that interpretation of Table 2. e05e06e07e11e12
Check 2: Measure the gap between an imagined success and an executable plan
Reader-proposed check, not performed: use the existing simulated tasks to compare an independently trained final-frame success classifier with ground-truth simulator task completion. From matched initial states, decode both generated plans and successful scripted video trajectories using the same inverse model and controller. Tabulate classifier false positives, executed success and object-trajectory deviations separately for both inputs, including cases with visually successful endings but infeasible intermediate motion. If scripted videos execute reliably while generated plans do not, the planner is implicated; failure on both implicates the decoder or execution interface. This tests the surrogate concept behind Table 4 without claiming to reproduce its real-world success percentage. e06e09e11e16
8.3 Reading coverage
Visual audit: All five source chunks were read. Original PDF pages 1–10 and 14–17 were rendered and visually inspected: p. 1 verifies title, authors, affiliations and v3 date; pp. 2–5 establish UPDP, guidance, architecture and execution; pp. 6–9 supply all four result/ablation tables and Figures 3–9; p. 10 supplies limitations; pp. 14–15 supply architecture, hardware, optimizer, action interface and task-list details; pp. 16–17 supply all Appendix B examples. Figures 1–12 and Tables 1–4 were visually read. Every final crop was separately viewed with complete graphic/table labels retained. Figure 2's arrows agree with the described planner-to-decoder flow; its action labels are schematic. Table 4's downward FID/FVD arrows conflict with p. 8's higher wording, explicitly disclosed above. Bibliography-only pages 11–13 were read in text but not rendered. External videos and separate supplements remain uninspected. The training-task count was checked again on p. 7 at both 200 and 400 DPI: in the second sentence of the bottom Setup paragraph, immediately after “a set of,” the printed number is 10. Page 15 supplies the eleven-name training enumeration, itemized in e20. This discrepancy is between two locations in the supplied revision; it is not evidence of a change between paper editions.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Problem Formulation (2.1–2.3)
- 3 Decision Making with Videos (3.1–3.2)
- 4 Experimental Evaluation (4.1–4.3)
- 5 Related Work
- 6 Conclusion and Limitations
- References
- A Architecture, Training, and Evaluation Details (A.1–A.5)
- B Additional Results (B.1–B.3)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read. Original PDF figures, tables and supporting method pages were visually inspected; this resolves the extraction-only image gap within the retained PDF.
- The reviewed edition is arXiv:2302.00111v3, 20 November 2023, labeled NeurIPS 2023. Its title and eight authors match the catalog. The catalog submission date, 31 January 2023, is not this revision date; earlier versions were not supplied or compared.
- External project videos, code and separately hosted supplements were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, arXiv margin and conference footer
The title and all eight authors match the catalog. The retained source is arXiv:2302.00111v3, 20 November 2023, labeled NeurIPS 2023. Affiliations are MIT, Google DeepMind, UC Berkeley, Georgia Tech and University of Alberta; Du and Yang share first authorship.
Go to primary source ↓e02PDF pp. 2–3, Sections 1 and 2.2, UPDP definitions
UPDP uses images, textual tasks, a finite horizon and a conditional video generator. It separates video planning from a trajectory-conditioned, environment-dependent action policy; learning requires video/task data.
Go to primary source ↓e03PDF pp. 3–4, Section 2.3, unnumbered diffusion and classifier-free guidance equations
The forward distribution is Gaussian with scheduled signal and noise scales. Sampling uses a conditional denoiser combined with an unconditional denoiser, with guidance strength omega.
Go to primary source ↓e04PDF pp. 4–5, Figure 2 and Section 3.1, Conditional Video Synthesis and Trajectory Consistency through Tiling
The initial image is an explicit training condition and is concatenated to noisy frames throughout denoising. Figure 2 routes tiled context through video diffusion, temporal super-resolution and a separate inverse-dynamics action module.
Go to primary source ↓e05PDF p. 5, Section 3.1, Hierarchical Planning and Flexible Behavioral Modulation; p. 7, Figure 7 and Adaptability
Sparse video plans are temporally super-resolved. An additional prior h(tau), such as an image classifier or an intermediate-image constraint, can guide sampling; Figure 7 illustrates selecting which block to move.
Go to primary source ↓e06PDF p. 5, Section 3.2, Inverse Dynamics and Action Execution
The inverse model is trained independently of the planner. Generated frames are translated into actions. Both open-loop execution and replanning after each action are described, but the experiments use open-loop control for efficiency.
Go to primary source ↓e07PDF p. 14, Appendix A.1, complete architecture and training specification
Video U-Net uses three residual blocks, 512 base channels, multipliers [1,2,4], attention resolutions [6,12,24], head dimension 64, conditioning dimension 1024, temporal convolutions and log-SNR range [-20,20]. Training is 2M steps, batch 2048, learning rate 1e-4, 10k warmup, on 256 TPU-v4 chips. T5-XXL is reported as 4.6B parameters. Simulation stages use 10x48x64 and 20x48x64 videos, skipping every eight and four frames respectively, with 1.7B parameters each. Real-video model sizes are 16x40x24 (1.7B), 32x40x24 (1.7B), 32x80x48 (1.4B) and 32x320x192 (1.2B).
Go to primary source ↓e08PDF p. 14, Appendices A.2–A.3; p. 15, continuation of State-Based Diffusion
The inverse model predicts seven controls using mean squared error, convolution/residual layers, pixel mean pooling and a (128,7) MLP; Adam uses learning rate 1e-4, gradient clipping at 1, 2M steps and 10k warmup. A.2 says image observation without precisely specifying temporal input packing. Transformer BC follows a 10M architecture; the state-based diffusion baseline reuses a similar U-Net while replicating controls spatially.
Go to primary source ↓e09PDF pp. 5–6, Section 4.1, Setup, Baselines and Metrics; p. 15, Appendix A.4
Combinatorial tasks require coloring white blocks in bowls and placing them as instructed. Instructions are split 70%/30%, positions randomized, and 200k scripted videos train the planner; 20k action-annotated videos train inverse dynamics. All compared policies use T5 language features. Controls are six joint values plus contact, thresholded above 0.5; contact attaches the nearest block, and PyBullet joint control has a two-second timeout.
Go to primary source ↓e10PDF p. 6, Table 1, all rows and Seen/Novel Place/Relation columns
UniPi reports Seen Place 59.1 ± 2.5, Seen Relation 53.2 ± 2.0, Novel Place 60.1 ± 3.9, Novel Relation 46.1 ± 3.0. The strongest listed novel baselines are Image + TT at 13.2 ± 4.1 for Place and Diffuser at 9.6 ± 1.7 for Relation. The table does not define the ± statistic.
Go to primary source ↓e11PDF p. 6, Figure 4 and caption
The upper strip is synthesized frames and the lower strip executed-action frames for a simulated block manipulation prompt. The caption describes rough alignment; it supplies no numerical trajectory-error metric.
Go to primary source ↓e12PDF p. 7, Table 2 and Section 4.1, Ablations
On seen Place/Relation, successive frame condition, tiling consistency and temporal hierarchy yield 13.2/12.4, 52.4/34.7, 53.2/39.4 and 59.1/53.2, with the table's accompanying ± values. Without consistency, non-start frames receive zeroed image context. Only four nested configurations are shown.
Go to primary source ↓e13PDF p. 7, Section 4.2 Setup; p. 8, Baselines; p. 15, Appendix A.5
Section 4.2 reports 200k scripted videos and three test tasks. A.5 names the test environments put-block-in-bowl-seen-colors, packing-unseen-google-objects-group and packing-boxes-pairs-unseen-colors. Inverse dynamics uses action annotations across 200k videos. Controls use the same PyBullet setup as A.4; p. 8 explains why the original CLIPort primitive-action policy is not directly compared.
Go to primary source ↓e14PDF p. 8, Table 3, all rows and Place Bowl/Pack Object/Pack Pair columns
UniPi obtains 51.6 ± 3.6, 75.5 ± 3.1 and 45.7 ± 3.7. The strongest listed baselines per column are Diffuser 14.8 ± 2.9, State + Transformer BC 21.7 ± 3.5 and Diffuser 10.5 ± 2.4. The table does not define the ± statistic.
Go to primary source ↓e15PDF p. 8, Section 4.3, Setup and Video Synthesis
Internet pretraining uses 14M video-text pairs, 60M image-text pairs and LAION-400M. Bridge supplies 7.2k video-text pairs, task IDs as text and an 80%/20% train/test split. The pretrained model is finetuned on the training split. The paper describes generation metrics as averaged across frames and computed on 32 samples.
Go to primary source ↓e16PDF pp. 8–9, Section 4.3 Video Synthesis; p. 9, Table 4 and success-classifier paragraph
Table 4 at 24x40 reports scratch/pretrained CLIP 24.43 ± 0.04/24.54 ± 0.03, FID 17.75 ± 0.56/14.54 ± 0.57, FVD 288.02 ± 10.45/264.66 ± 13.64, and success 72.6%/77.1%. Success is predicted from the last generated frame by a learned classifier. Down arrows identify lower FID/FVD as better, conflicting with the prose on p. 8 saying higher.
Go to primary source ↓e17PDF p. 8, Figure 6; p. 9, Figures 8–9 and Generalization
These are generated real-scene video frames. Figure 8 contrasts scratch and pretrained generations for novel commands; Figure 9 illustrates blacked-out background regions and inserted objects. They provide qualitative examples, without executed real-robot task measurements.
Go to primary source ↓e18PDF p. 10, Section 6, Limitations
The authors report that photorealistic generation can take a minute, with a 16x speedup in initial distillation experimentation. They identify generally fully observed evaluation environments and possible hallucinated objects or motion under partial observability.
Go to primary source ↓e19PDF pp. 16–17, Appendices B.1–B.3, Figures 10–12 and captions
Appendix B adds synthesized frame sequences for novel combinatorial prompts, new multitask settings and real-world video plans. These pages contain qualitative examples, not additional quantitative execution or ablation tables.
Go to primary source ↓e20PDF p. 7, Section 4.2, bottom Setup paragraph, second sentence; p. 15, Appendix A.5, training-task enumeration before the test-data sentence
The p. 7 sentence visibly specifies “a set of 10 separate tasks.” The A.5 training list contains eleven distinct names, counted in printed order: (1) put-block-in-bowl-unseen-colors; (2) packing-unseen-google-objects-seq; (3) assembling-kits-seq-unseen-colors; (4) stack-block-pyramid-seq-seen-colors; (5) tower-of-hanoi-seq-seen-colors; (6) assembling-kits-seq-seen-colors; (7) tower-of-hanoi-seq-unseen-colors; (8) stack-block-pyramid-seq-unseen-colors; (9) packing-seen-google-objects-seq; (10) packing-boxes-pairs-seen-colors; (11) packing-seen-google-objects-group. Thus the stated count and the enumerated inventory disagree in the supplied revision; the source does not explain which, if any, listed environment was excluded.
Go to primary source ↓e21PDF p. 9, Section 5, Learning Generative Models of the World, final sentence; p. 3, Section 2.2; p. 5, Section 3.2
The authors characterize text-conditioned video policies as jointly learning a world model and conducting hierarchical planning. This characterization concerns video planning; Sections 2.2 and 3.2 separately define and train the action adaptation module.
Go to primary source ↓8.5 Primary sources
Learning Universal Policies via Text-Guided Video Generation ↗
PDF · 8,417 extracted words
Source fingerprint
5ded462cfcad30a6317610a4f991d3c1eca7826cbe208dbd9929c3837529708e