RoboDreamer: Learning Compositional World Models for Robot Imagination
1. Paper overview
In one sentence: Composing phrase-conditioned video predictions improves alignment with unfamiliar robot instructions, while execution still depends on a separate controller and incomplete spatial information. E02E03E04E10E11E12E13
| At a glance | What to know |
|---|---|
| Research problem | Source description A plausible manipulation video can move the wrong object or satisfy the wrong spatial relation. The paper targets unfamiliar combinations of familiar actions and relations, using language structure to factor video generation. Its abstract names RT-X, but the documented video experiment uses RT-1; this report follows the explicit setup. E02E09 |
| Core mechanism | |
| A key reported result | Language-only video alignment on unseen RT-1 instructions: RoboDreamer 81.3 unseen and 90.1 seen. Reported human-evaluation score, higher is better; table gives no explicit percentage unit. Randomly held-out instructions; binary human judgments. Appendix A reports about 128 total samples, more than 20 prompts and at least three raters per sample. Unseen: AVDC 46.9, HiP 50.1, without parsing 68.8. Seen: 63.1, 70.3, 85.5 respectively. The parsing comparison improves unseen scores by 12.5 reported-score points (reader subtraction), versus 4.6 seen. These are judgments of generated videos, with no uncertainty reported. E09E10 |
| Reading caution | Source description Authors report restricted single-camera use, poor generalization to many tested real-world images, and difficulty with moving cameras. These constrain the deployment scope despite favorable in-dataset video results. E14 |
Core contributions
Figure 3. Phrase conditions constrain a shared video-generation process. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the left: the instruction separates into two verb phrases and one prepositional relation. On the right, arrows carry the same noisy video into separate conditioned predictions; the fourth branch receives a goal image. The predictions merge at the composition block before the output video. These branches describe simultaneous constraints on a video, not successive robot actions or one generated clip per phrase. The graphic labels all components m_i and its noisy input x_t; Sections 3.2–3.3 use l_i for language, m_i for visual conditions, and τ_t for noisy video. Read the equations for the precise composition weights. E03E04E05E06E07
What it supports. The mechanism exposes reusable conditions such as an action and a drawer relation. Equation (1) combines their conditional video densities, and Equation (4) trains with subsets so a component can contribute without every original phrase. A goal-image branch adds a more concrete spatial constraint to the same generation process.
Where the evidence stops. The composition box hides both repeated denoising and downstream action extraction. No action head is shown here. Section 2.2 supplies a separate inverse-dynamics controller; diagrammatic composition alone does not establish executable motion.
2. Motivation
2.1 The problem and the proposed response
A plausible manipulation video can move the wrong object or satisfy the wrong spatial relation. The paper targets unfamiliar combinations of familiar actions and relations, using language structure to factor video generation. Its abstract names RT-X, but the documented video experiment uses RT-1; this report follows the explicit setup. E02E09
2.2 What this reading follows
Consider an instruction that names both an object to move and another object it should end up near. A video generator can produce convincing arm motion while getting that relation wrong. RoboDreamer splits the instruction into action and relation phrases, then combines their diffusion predictions while generating one future video. Goal images or sketches can add spatial detail. This reading follows that information flow through the original architecture, the parsing comparison, and the evaluation tables. The key distinction is between people judging a generated plan and a controller successfully executing it: the paper measures both, but in different experiments with different inputs. E02E03E04E10E11E12E13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is unassigned, so no recorded quadrant can be confirmed or contradicted. Architecture evidence supports a video world model followed by a separate inverse-dynamics action model. It predicts future observations and then extracts actions, rather than jointly predicting video and action in one model. Compositional denoising does not make the entire control pipeline a One Model architecture. E03E04E16
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn an unfamiliar sentence into reusable constraints
The concrete example in Figure 3 asks the model to pick an orange from a bottom drawer and place it on a counter. Parsing exposes the action phrases and the relation phrase instead of encoding only the complete sentence as a single condition. Each fragment conditions a video noise prediction, and their composition guides one evolving sample. Equation (1) expresses this as a product of conditional video densities, each raised to 1/N. It therefore describes multiple constraints on the same trajectory, rather than concatenating independently imagined subtasks. The intended generalization occurs when a new instruction recombines familiar fragments. Figure 4 supplies selected qualitative comparisons and Table 1 supplies human ratings; neither turns the paper’s idealized claim of perfect component-based generalization into an unconditional guarantee. E04E05E10E19
5.2 Teach fragments to work both alone and together
A model trained only on the average prediction for a full sentence could fit the composed training target without making each individual phrase useful. Section 3.2 addresses that weakness by first describing single-component denoising and then a random-subset objective. Equation (4) averages predictions from M selected components and compares the result with the known Gaussian noise added to the training video. This connects learning on complete instructions to inference with recombined fragments. Visual conditions extend the distribution product through Equation (5). Implementation details remain consequential: Algorithm 1 prints an unsquared norm and omits the subset draw, while the text gives squared objectives. Equation (6) uses equal 1/(2M) coefficients for two modality sums without fully explaining unequal subset sizes. Those gaps should remain explicit in a reproduction. E05E06E07
5.3 Follow the plan across the boundary into action
Reader interpretation: the decisive control question is whether an improved imagined trajectory is also easier for the action model to realize. RoboDreamer’s video model does not itself output motor commands. Section 2.2 assigns that role to inverse dynamics; Appendix A adds current state to the adjacent-frame inputs and specifies a ResNet18-plus-MLP predictor. Periodically generating a fresh plan can correct drift, but the paper leaves the interval unspecified. This makes the evidence hierarchy important. Human scores judge generated task behavior, multimodal metrics assess video outputs, and Table 3 measures executed RLBench outcomes. The latter supports the whole pipeline in simulation, while its low stacking and shoe-task scores expose remaining difficulty. Separating plan quality from controller realization would clarify which component limits success. E03E09E11E12E13E16E17
5.4 Training and inference
During training
Full-composition denoising alone need not teach useful individual factors. Equation (3) trains a single component; Equation (4) unifies these cases by sampling subsets. Algorithm 1 omits this subset draw and prints an unsquared norm, unlike the squared losses in the equations. E06
Video training uses about 70k RT-1 demonstrations, roughly 500 tasks and sampling every five frames. Appendix A reports four U-Net blocks, 128 base channels, multipliers [1,2,4,8], batch size 256, learning rate 5e-5 and about 100 V100 GPUs; training duration is absent. E09E15
Inverse dynamics uses ResNet18 plus an MLP, taking adjacent frames and current state. Adam training uses learning rate 1e-4 for 10K steps. This action model is distinct from the video denoising objective. E16E06
During inference
Starting from Gaussian noise, Algorithm 2 combines an unconditional prediction with a guidance-weighted sum of conditioned-minus-unconditional predictions and repeatedly denoises. Equation (5) permits variable numbers of language and visual conditions. The algorithm’s guidance sum has no explicit component-count normalization; the numerical guidance setting is unspecified. E07
A sampled video is a plan, not an executed trajectory. Actions are inferred separately and new videos are periodically generated from updated observations. The paper does not specify the replanning interval. E03
5.5 Implementation flow
- Parse the instruction
A pretrained constituency parser and rules separate verb phrases from prepositional relations. For example, placing a bottle and specifying its destination become distinct conditions. Figure 3 shows these conditions acting on one noisy video, rather than assigning separate time segments to phrases. E04
- Encode and condition
Frozen T5-XXL encodes text; a pretrained Stable Diffusion image encoder handles images and sketches. PerceiverSampler outputs enter U-Net cross-attention. The source explicitly freezes T5, but does not equally clarify every image-module freezing choice. E08
- Generate and execute
The initial frame is concatenated to noisy frames to encourage background consistency. A cascaded model generates eight frames at 64×64 and upsamples to 128×128 then 256×256. The controller maps adjacent frames and current state to actions; periodic replanning addresses accumulated execution error. E03E08E16
6. Experiments & results
RoboDreamer composes phrase-conditioned video diffusion predictions to imagine robot plans for unfamiliar instruction combinations. Optional goal images or sketches sharpen spatial specifications; a separate inverse-dynamics model converts imagined frames into actions. The strongest language-only evidence concerns human-rated video alignment, while executed success is measured separately in RLBench simulation (E03, E10–E13).
6.1 Read the original evidence
Figure 5. Visual goals illustrate the spatial information added to language. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each row from the instruction and small visual condition through the arrow to the generated sequence. The upper three rows show goal-image conditions for opening a drawer, picking an apple and knocking over a bottle. The lower three use sketches for picking a can, closing a drawer and moving an apple near a can. Compare the visible drawer configuration or object relation at the end with the supplied condition. Section 4.1 says goal images are final frames and sketches are produced with ControlNet annotators. The figure is therefore an illustration of generation with explicit target information, rather than an observation-only robot test. E11E18E17
What it supports. These selected examples show how a visual condition can specify a configuration more concretely than a phrase alone. Their role is explanatory: the quantitative claim that these inputs improve alignment comes from the separate human, FVD and IMO comparisons. The figure itself supplies no frequency estimate for failures or successes.
Where the evidence stops. Static strips cannot establish continuous dynamics, collision-free motion or physical execution. The selected rows contain different tasks, so comparing upper and lower rows cannot isolate the effect of image versus sketch conditioning.
Table 3. Executed success improves on average, with substantial task variation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at Average, then inspect the six task columns to see what that summary conceals. RoboDreamer reaches 49.3 against UniPi’s 41.0 and Hiveformer’s 44.2. Its high lamp-off and close-box entries contrast with 18.5 for stacking blocks and 10.5 for taking shoes. Hiveformer wins lamp on and lift block, so the full model is not best everywhere. Section 4.2 describes RLBench macro-steps, front-camera RGB and language without goal images for RoboDreamer. It describes Hiveformer as using multiple views and observation/action history. The six columns delimit the reported evaluation, regardless of the larger task collection mentioned in the setup. E03E12E13
What it supports. The table is the paper’s direct evidence that the video-planning pipeline can support executed manipulation in simulation. The average exceeds UniPi by 8.3 reported-rate points, calculated from the table. Low scores on several tasks make the result a partial capability improvement rather than uniformly reliable manipulation.
Where the evidence stops. Trial counts, uncertainty and a detailed evaluation split are not reported. The baselines do not have identical stated observation interfaces. The adjacent prose’s 15% long-task statement should not replace the separate 18.5 and 10.5 entries.
Table 4. Additional target information improves three reported video metrics. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the row suffixes as t for language, s for sketch and i for goal image. Higher Human and IMO values are favored; lower FVD is favored. This appendix table repeats Table 2’s RoboDreamer human and FVD values and adds IMO. Appendix B.3 describes IMO as target-object bounding-box IoU using pretrained GroundingDino detections, so it measures a restricted spatial signal rather than complete task dynamics. Text-plus-image reaches 95.8 Human, 444.3 FVD and 78.1 IMO. Keep the video evaluation separate from the RLBench table: goal images were deliberately excluded from that robot-planning experiment. E11E12E17
What it supports. Text-plus-sketch improves the language-only row on all three measures, and text-plus-image improves them further. The agreement is useful evidence that richer goals improve the evaluated outputs, but it does not identify composition as the sole cause: the variants also receive more information about the desired final arrangement.
Where the evidence stops. Neither Table 2 nor Table 4 explicitly labels the seen/unseen split. Detector thresholds, box matching and temporal aggregation for IMO are unspecified; metric scaling and uncertainty are also not fully documented. These scores cannot certify physical feasibility.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Language-only video alignment on unseen RT-1 instructions Randomly held-out instructions; binary human judgments. Appendix A reports about 128 total samples, more than 20 prompts and at least three raters per sample. | RoboDreamer 81.3 unseen and 90.1 seen. Reported human-evaluation score, higher is better; table gives no explicit percentage unit | Unseen: AVDC 46.9, HiP 50.1, without parsing 68.8. Seen: 63.1, 70.3, 85.5 respectively. The parsing comparison improves unseen scores by 12.5 reported-score points (reader subtraction), versus 4.6 seen. These are judgments of generated videos, with no uncertainty reported. E09E10 |
| Video generation with additional visual goals RT-1 multimodal comparison; goal images use final frames and sketches use ControlNet annotators. Table 2 does not explicitly label the test split. | Text+image: 95.8 / 444.3 / 78.1. Text+sketch: 94.7 / 454.7 / 72.5. Human score ↑; FVD ↓; appendix IMO ↑ | Text only: 81.3 / 487.8 / 63.5. Table 2 AVDC: human 46.9, FVD 517.1. Visual goals improve all three reported measures, while providing extra target information. IMO uses detected target boxes; its matching protocol is underspecified. E11E17 |
| Executed manipulation across six RLBench tasks Simulation with macro-steps, front-camera RGB and no goal images for RoboDreamer; baseline information differs. | Average 49.3; lamp off 96.3, lamp on 51.9, stack blocks 18.5, lift block 22.2, take shoes 10.5, close box 96.3. Reported task success rate and Average | Average: Image-BC 31.6, Hiveformer 44.2, UniPi 41.0. Hiveformer leads lamp on (53.2) and lift block (28.2). The 8.3-point advantage over UniPi is reader subtraction. This is six-task simulation evidence, not all 74 tasks mentioned in the setup or physical deployment. Use table entries rather than the prose’s ambiguous 15% long-task summary. E12E13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. The parsing advantage is larger for unseen instructions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each column before comparing across columns. Seen and Unseen refer to language-instruction availability, and every method receives language alone in this comparison. The row labeled RoboDreamer w/o removes text parsing; it is the closest comparison for the proposed decomposition. The full model rises from 68.8 to 81.3 on unseen instructions and from 85.5 to 90.1 on seen instructions. Section 4.1 defines binary human judgments of task completion in generated videos. Appendix A reports about 128 samples across more than 20 prompts, with at least three raters per sample. The table does not print a percentage unit or explain aggregation. E06E09E10
What it supports. Subtracting the ablated row gives a 12.5-point unseen improvement and a 4.6-point seen improvement on the reported score scale. This pattern supports the utility of parsing for unfamiliar instructions. The full model also exceeds the AVDC and HiP rows in both columns, but those comparisons change more of the modeling system.
Where the evidence stops. The table does not isolate subset training, parsing accuracy, or component-count effects. Per-condition sample counts and uncertainty are absent. A human judgment that a video appears executable is not a measured execution success.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors report restricted single-camera use, poor generalization to many tested real-world images, and difficulty with moving cameras. These constrain the deployment scope despite favorable in-dataset video results. E14
The assertion of perfect generalization from in-distribution fragments exceeds what the experiments establish. A learned product can still mishandle interactions or incompatible constraints; held-out instruction performance does not certify arbitrary compositions. E05E10
Exact held-out prompts, per-condition counts, rater aggregation, uncertainty and RLBench trial counts are absent. Multimodal gains change available goal information, and baseline observation access is not uniform. E09E11E12E13
7.2 Questions for discussion
- Does parsing still improve unseen combinations when component counts and guidance strength are matched?
- How often do human-approved plans fail under a fixed inverse-dynamics controller?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the parser/rules, pretrained encoders, cascaded video model, RT-1 preprocessing and a separately trained inverse-dynamics controller. Missing video training duration, diffusion schedule, guidance weight and subset distribution prevent a fully specified recipe. Equation (6) also leaves unequal modality-subset allocation unclear. E04E06E07E08E09E15E16
Before executing plans, resolve the action-coordinate convention: Appendix A describes an eight-dimensional action space plus an additional gripper state, without defining the coordinates. Fix replanning cadence, trial counts and camera access; for IMO, specify box matching, detector thresholds and aggregation. E03E12E17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate parsing from subset training and guidance strength
Reader-proposed check, not a completed experiment: construct a held-out instruction split in which every primitive phrase appears during training but selected combinations do not. Compare a whole-instruction model, parsed full-set training, and parsed random-subset training using the same video backbone, data, update budget and frame resolution. Record component counts and evaluate both the printed summed guidance and a prespecified count-normalized control. Use blinded task-alignment judgments with equal per-prompt counts and uncertainty estimates. The compositional account predicts a specific held-out-combination benefit for parsing and useful component training. If the gain disappears after matching guidance strength, attributing it solely to learned compositional structure would be weakened. E04E05E06E07E08E09E10
Check 2: Measure where good video plans fail during execution
Reader-proposed check, not a completed experiment: evaluate RoboDreamer and a language-only video baseline from identical initial states on the six reported RLBench tasks, using one fixed inverse-dynamics model, front-camera observations and no goal images. Log blinded video-plan judgments separately from environment success. Include demonstration-frame plans as a controller diagnostic, and compare two explicitly fixed replanning intervals with all other settings held constant. Report per-task plan approval, execution success and failures conditional on approved plans. If RoboDreamer improves judged plans without improving execution, the controller or plan realizability limits the benefit. If demonstration-frame plans also fail, controller calibration deserves attention before retraining the world model. E03E12E13E16
8.3 Reading coverage
Visual audit: Inspected the title/byline/affiliations and v1 marker on page 1; all seven original figures; all four result tables; the composition equations and Algorithms 1–2; evaluation protocols; limitations; and Appendix A/B implementation and metric details on pages 11–12. Figure 3’s arrows and branch labels were checked against Sections 3.1–3.3, Equations (1)–(6) and the algorithms. All five final original crops were opened and checked for labels, table boundaries and readability. Bibliography pages 9–10 were read in the complete text chunks but not rendered. External linked videos, separate supplements and code remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 11, 12. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Background
- 2.1. Planning with Text-Conditioned Video Generation
- 2.2. Executing Videos Plans
- 3. RoboDreamer
- 3.1. Text Parser
- 3.2. Compositional Generation
- 3.3. Multi-modal Composition
- 4. Experiments
- 4.1. Evaluation on Video Generation
- 4.2. Evaluation on Robotic Planning
- 5. Related Work
- 6. Conclusion
- References
- A. Experimental Details
- A.1. Video Diffusion
- A.2. Other Details
- B. Additional Results
- B.1. Visualization on RLBench
- B.2. More Results on Video Generation
- B.3. IMO Metrics
- Impact Statement
- Limitations within Section 6
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks were read, including references and Appendices A–B. Figure extraction limitations were addressed by inspecting original PDF pages and final crops.
- The inspected artifact is arXiv:2404.12377v1, 18 April 2024; its title and all six authors match the catalog. No later revision or alternative edition was supplied or compared (E01).
- Code, external project videos and separate supplements were not inspected; experiments were not reproduced. Bibliography pages 9–10 were read as text, not rendered.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
E01PDF p. 1, title, byline, affiliation footnote and arXiv margin
The title and six authors match the catalog. The artifact identifies arXiv:2404.12377v1 [cs.RO], 18 April 2024. Six numbered institutions appear in the footnote.
Go to primary source ↓E02PDF pp. 1–2, Abstract, Introduction and Figure 2
Factorization targets unfamiliar combinations of actions and object relations and accepts language plus goal images or sketches. The abstract names RT-X; the experimental setup later specifies RT-1.
Go to primary source ↓E03PDF p. 3, Sections 2.1–2.2
An initial image and task description condition a future video. A separate inverse-dynamics policy converts adjacent generated frames into actions; periodic video regeneration provides feedback.
Go to primary source ↓E04PDF pp. 3–4, Section 3.1 and Figure 3
A pretrained constituency parser plus rules extracts verb and prepositional phrases. Figure 3 routes the same noisy video through differently conditioned noise predictions and composes them; its component labels use m rather than the text-only equations’ l.
Go to primary source ↓E05PDF p. 4, Section 3.2, Equations (1)–(2)
The conditional video distribution is a product of component densities raised to 1/N, up to normalization; training averages component noise predictions. Perfect compositional generalization is asserted conditional on in-distribution components.
Go to primary source ↓E06PDF p. 5, Section 3.2, Equations (3)–(4) and Algorithm 1
Individual-component denoising motivates random-subset training. Equation (4) averages M selected components and uses a squared norm. Algorithm 1 displays all language components and an unsquared norm, without the subset draw.
Go to primary source ↓E07PDF p. 5, Algorithm 2, Section 3.3 and Equations (5)–(6)
Inference starts with Gaussian noise and adds a sum of component-conditioned guidance differences to the unconditional prediction. Equation (5) weights N language and K multimodal factors by 1/(N+K). Equation (6) uses 1/(2M) on both modality sums without fully specifying subset allocation.
Go to primary source ↓E08PDF pp. 6–7, Implementation Details; p. 11, Appendix A.1 items 1–4, 6
The cascaded video U-Net uses spatial-temporal convolutions and temporal attention. Initial-frame concatenation anchors noisy frames. Frozen T5-XXL and a pretrained Stable Diffusion image encoder feed PerceiverSampler and cross-attention; output progresses from eight 64×64 frames to 128×128 and 256×256.
Go to primary source ↓E09PDF p. 6, Section 4.1, Experimental Setup, Baselines and Metrics; p. 11, Appendix A.2 items 1, 4
RT-1 supplies about 70k demonstrations and 500 tasks; unseen instructions are randomly selected. Frames are sampled every five frames. Human judgments are binary; at least three raters judge each sample. About 128 total samples span more than 20 prompts; per-condition counts and aggregation are not specified.
Go to primary source ↓E10PDF p. 5, Table 1, all rows, Seen and Unseen columns; p. 6, Baselines
Seen/unseen scores are AVDC 63.1/46.9, HiP 70.3/50.1, RoboDreamer without parsing 85.5/68.8, and full RoboDreamer 90.1/81.3. This comparison supplies language only.
Go to primary source ↓E11PDF p. 7, Multi-modal Generation and Table 2, all rows
Goal images are final frames and sketches come from ControlNet annotators. Human/FVD values are AVDC 46.9/517.1, text 81.3/487.8, text+sketch 94.7/454.7 and text+image 95.8/444.3. Table 2 does not explicitly label its seen/unseen split.
Go to primary source ↓E12PDF p. 7, Section 4.2, Experimental Setup and Baselines; p. 11, Appendix A.2 item 3
RLBench evaluation uses macro-steps, front-camera RGB and no goal images for RoboDreamer. Hiveformer is described with multiple views and history. Appendix A.2 names a Franka Panda arm and gripper, seven DoF, and an eight-dimensional action space plus an additional gripper state, without defining coordinates.
Go to primary source ↓E13PDF p. 8, Table 3, six task columns and Average; Section 4.2 continuation
Average success rates are 31.6 Image-BC, 44.2 Hiveformer, 41.0 UniPi and 49.3 RoboDreamer. RoboDreamer’s six entries are 96.3, 51.9, 18.5, 22.2, 10.5 and 96.3. Hiveformer leads lamp on and lift block. The prose’s 15% long-task summary is not an individual table entry.
Go to primary source ↓E14PDF p. 8, Conclusion, Limitations paragraph
Authors identify single-camera restrictions, poor generalization to many tested real-world images, and limitations with moving cameras.
Go to primary source ↓E15PDF p. 11, Appendix A.1 items 2–6
Four U-Net ResNet blocks use base channel 128 and multipliers [1,2,4,8]; temporal attention is placed at the last encoder and first decoder blocks. Training uses batch size 256, learning rate 5e-5 and about 100 V100 GPUs. Training duration and video-model step count are not given.
Go to primary source ↓E16PDF p. 11, Appendix A.2 item 2
Inverse dynamics takes two adjacent frames and current state, using ResNet18 followed by an MLP. It is trained with Adam at 1e-4 for 10K steps.
Go to primary source ↓E17PDF p. 11, Appendix B.3; p. 12, Table 4, all rows
IMO uses GroundingDino target-object bounding boxes and is described as box IoU. Text/text+sketch/text+image scores are 63.5/72.5/78.1. Human and FVD entries repeat Table 2. Matching, aggregation and detector thresholds are not supplied.
Go to primary source ↓E18PDF p. 6, Figure 5 and caption; p. 7, Multi-modal Generation
Six selected video strips pair text and visual goals with generated plans. Upper examples use goal images; lower examples use sketches. These are generated-video illustrations, not physical-robot execution measurements.
Go to primary source ↓E19PDF p. 4, Figure 4 and caption
Three task pairs compare AVDC and RoboDreamer generated videos for unfamiliar language combinations, illustrating the claimed improvement in target-object relations.
Go to primary source ↓8.5 Primary sources
RoboDreamer: Learning Compositional World Models for Robot Imagination ↗
PDF · 6,628 extracted words
Source fingerprint
7faf7287c5920d0c16b1f55849a214d97b8355af621632176178860d07b237e0