DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models
1. Paper overview
In one sentence: DreamPlan moves expensive video-based action verification into offline preference training, improving a deformable-manipulation planner's reported scores while allowing direct action inference at deployment. e-probleme-worlde-preferencee-evaluatione-taske-efficiency
| At a glance | What to know |
|---|---|
| Research problem | Source description A semantically sensible grasp-and-place decision can deform cloth, rope or a soft toy in the wrong way. The authors seek physical grounding for image-goal planning without repeatedly updating a large VLM through costly robot interaction or constructing a high-fidelity deformable simulator. Their solution still requires initial physical exploration; only subsequent planner adaptation is performed entirely offline. e-probleme-datae-preference |
| Core mechanism | Source description Rendered robot-motion videos condition a pretrained video diffusion model through a residual control branch, while object-focused prediction reduces irrelevant scene reconstruction. e-world |
| A key reported result | Real-world deformable manipulation across three tasks: DreamPlan: rope 0.60, cloth 0.35, toy arm 0.85; overall 0.60. Average ordinal task score, higher is better. Rope straightening, cloth folding and toy arm repositioning; 10 randomized-initial-state trials per task; one executed action primitive per trial. Scores are 0 for failure, 0.5 for meaningful progress and 1 for complete success. A held-out split is not specified. Zero-shot Qwen3-VL-8B: 0.20, 0.10, 0.70; overall 0.33. Qwen3-VL-32B has the highest baseline overall score, 0.35; GPT-4o has 0.28 and Qwen3-VL-4B has 0.22. The within-backbone gains are 0.40, 0.25 and 0.15 score units. Figure 5 labels equivalent values as success percentages, but the protocol includes partial credit; binary completion rates cannot be recovered. No uncertainty estimates are reported. e-evaluatione-taske-score-plot |
| Reading caution | Reader analysis Three task types and ten trials each provide narrow evidence. No explicit held-out-object protocol, confidence intervals or multi-primitive completion evaluation is reported. The paper has no dedicated limitations section. e-evaluatione-taske-repro-gaps |
Core contributions
- Source description
Rendered robot-motion videos condition a pretrained video diffusion model through a residual control branch, while object-focused prediction reduces irrelevant scene reconstruction. e-world
- Source description
Best-of-K outcome ranking produces reusable preference pairs, separating expensive video generation from planner optimization and eventual action inference. e-preferencee-efficiency
- Reader analysis
The authors interpret improved robot scores as internalization of physical dynamics. The experiments demonstrate improved decisions on the tested tasks; they do not directly measure an internal physics representation. e-taske-interpretation
Figure 2. Physical exploration trains the predictor; predicted comparisons then train the planner. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: text, observation and target enter the VLM, whose sampled actions produce physical rollout data. Follow those trajectories into the upper-right world-model learning stage. Rendered arm motion supplies structured action information, while the observed trajectory supplies the learning target; this panel depicts training, not a need to provide a future video at deployment. Along the bottom, candidate actions enter the trained world model and produce alternative predicted outcomes. The green plus and red minus mark relative goal preference. Their path returns through ORPO Update to the VLM. Section III-C specifies that these comparisons are prepared before optimization, so the loop does not require regenerating videos at every update. e-architecturee-worlde-preferencee-efficiency
What it supports. The diagram supports a separation between action proposal, dynamics prediction and preference learning. Physical data are needed initially, but the adaptation signal thereafter comes from imagined outcomes. At deployment, the paper removes the prediction-and-ranking path and uses the adapted planner directly, which explains the intended decision-time saving.
Where the evidence stops. The boxes labeled object-only videos show dark scene backgrounds, whereas Section III-B specifies white-background training targets. Preserve that discrepancy. The figure also abstracts away the frozen backbone/residual-branch distinction in Equation (2); it does not establish joint training of all modules.
2. Motivation
2.1 The problem and the proposed response
A semantically sensible grasp-and-place decision can deform cloth, rope or a soft toy in the wrong way. The authors seek physical grounding for image-goal planning without repeatedly updating a large VLM through costly robot interaction or constructing a high-fidelity deformable simulator. Their solution still requires initial physical exploration; only subsequent planner adaptation is performed entirely offline. e-probleme-datae-preference
2.2 What this reading follows
Choosing where to grasp a rope is a spatial decision; predicting how the rope will deform is a dynamics problem. DreamPlan connects these capabilities through a learned video verifier. A zero-shot VLM first supplies exploratory robot actions, an action-conditioned diffusion model learns their consequences, and predicted outcomes provide preferences for adapting the planner. The resulting planner can act without generating videos at deployment. Read the evidence in three layers: how actions enter the world model, whether its predictions support useful decisions, and what physical robot trials establish. The headline scores include partial credit, and the small evaluation leaves generalization and verifier calibration open. e-probleme-worlde-preferencee-evaluatione-taske-efficiency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system is supported by separate planner and video-model modules. The world model predicts action-conditioned futures and supplies offline preference supervision; it neither jointly emits executable actions with futures nor extracts actions by inverse dynamics. Policy post-training and Other mechanisms therefore fit the recorded outside-quadrants classification. Removing the world model at inference does not make the training architecture a unified One Model. e-architecturee-worlde-preferencee-efficiency
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make the action concrete before predicting its consequences
The planner's job begins with two images, not a measured deformable-object state. DreamPlan marks candidate points on the current and goal objects, then asks the VLM to choose a grasp and placement. Those point sets are sampled independently; selecting a goal point does not imply a known physical correspondence between the images. The execution pipeline adds an arm choice, unprojects pixels into 3D and runs a fixed grasp–transport–release primitive. This distinction matters because a predicted keypoint pair is only a command specification. Actual motion and its resulting deformation come from the robot and object. For training the world model, that command becomes a rendered robot-arm trajectory. The rendering supplies a spatial description of intended motion that the diffusion model can use to condition its prediction of object appearance. e-plannere-executione-world
Figure 3. The proposed action is the reference for judging each generated deformation. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from Input and Action into the three model groups. The red circle and arrow indicate the location and direction to which the generated motion should respond. Within each model group, scan the frames from left to right before comparing the final shapes across groups. The shown CogVideoX sequences change relatively little in several rows, while some Wan sequences introduce hands or other scene content. DreamPlan's examples display object motion corresponding to the proposed manipulation. Section IV-B.1 is essential context: the baselines receive text describing the action, whereas DreamPlan receives rendered robot trajectories and has been adapted to the target interactions. e-generatione-world
What it supports. These selected examples make the authors' action-consistency argument visible: a model intended to compare candidate manipulations must respond to the action, not merely produce a plausible-looking continuation. The figure motivates using structured motion conditioning, but it supplies qualitative examples rather than a measured action-ranking accuracy.
Where the evidence stops. Conditioning format and task-specific adaptation differ across methods, so this is not an isolated ControlNet ablation. Generated frames are predictions; no matched real-outcome column or quantitative verifier-calibration statistic appears in this figure.
5.2 Transfer the verifier's preference into the planner
After learning the video model from exploration, DreamPlan generates alternative futures for sampled actions at a collected observation–goal input. GPT-4o compares those rollout images with the target and picks the preferred action; the others supply negative comparisons. The expensive futures are prepared outside the planner-update loop. Equation (5) expresses the preference objective through a log-sigmoid of the preferred action's log probability minus the negative action's log probability. Read this as the paper's displayed objective to increase, following its prose; additional implementation terms are not supplied. Once adaptation is complete, the planner can select an action directly. Reader interpretation: the dynamics model's contribution is mediated through ranking decisions, so a video need not be perfect everywhere to be useful, but mistakes that reverse the ranking can train the wrong behavior. e-preferencee-efficiencye-world
5.3 Follow the evidence from image reconstruction to physical decisions
DreamPlan's experiments support different parts of a causal argument. The qualitative video comparison asks whether generated motion responds plausibly to an action. The object-only ablation asks whether a focused prediction target improves final-frame reconstruction. Table I then tests physical actions selected by the adapted planner, and Table III compares the decision cost with explicit verification. These are complementary results, but they do not automatically validate every link between them. Reader interpretation: higher PSNR does not show that the correct action wins a comparison, and improved robot scores do not identify whether motion conditioning, visual judging or the preference update contributed most. The evaluation is also one primitive per trial with partial credit. A careful reproduction should retain those exact metrics while adding separate ranking accuracy and binary completion measurements to test the missing links. e-generatione-ablatione-taske-evaluatione-efficiency
5.4 Training and inference
During training
The method section describes several hundred action sequences collected in approximately four hours; the collection section instead reports 2,056 trajectories. The paper does not explain whether these are different units or subsets, so neither count should silently replace the other. e-data
Adaptation follows world-model fitting and offline preference construction. No separate value function is required. The supplied description does not specify training K, rollout length, optimization hyperparameters or preference-pair counts. e-preferencee-repro-gaps
During inference
DreamPlan directly predicts an action without sampling video futures at deployment. The explicit-verification comparator instead samples N actions from the zero-shot planner, predicts each outcome and uses GPT-4o to choose. Both ultimately rely on the robot primitive for execution. Evaluation tests one primitive per trial, despite the method's general sequential notation. e-efficiencye-executione-evaluation
5.5 Implementation flow
- Ground the action vocabulary
SAM2 propagates the current-object mask after one manual bounding-box prompt; the goal mask is annotated offline. Farthest Point Sampling supplies candidate points in each mask. The VLM selects source and goal points without explicit correspondence matching between images, avoiding direct continuous-coordinate regression. e-plannere-execution
- Execute exploration through a primitive
Selected pixels are unprojected to 3D waypoints. The chosen arm moves from home to grasp, closes its gripper, transports, releases and returns home, using joint-space interpolation. Reobserving and repeating this cycle collects trajectories; the video predictor itself does not execute robot actions. e-execution
- Learn action-conditioned deformation
CogVideoX-5B image-to-video supplies the frozen diffusion backbone specified in Equation (2). A ControlNet-style residual branch receives rendered arm trajectories. Training predicts noise with squared error and targets cropped object-only videos on a white background. Figure 2 sketches the stages but depicts dark backgrounds in its object-output examples. e-worlde-architecture
- Turn predictions into preferences
For each collected observation–goal input, sample K action candidates and generate their outcomes. GPT-4o compares rollout images with the goal to select the preferred action; other candidates become negatives. Qwen3-VL-8B is fine-tuned on these comparisons using the paper's ORPO formulation, without repeated world-model queries during updates. e-preference
6. Experiments & results
DreamPlan adapts a Qwen3-VL-8B manipulation planner using preferences derived from an action-conditioned video world model. Exploratory robot interactions train the dynamics predictor; predicted outcomes then support offline action ranking and planner fine-tuning. Deployment uses the adapted planner directly. On three physical deformable-object tasks, the reported mean score rises from 0.33 to 0.60 relative to the same zero-shot backbone, under a single-action evaluation that awards partial credit. The results support task-specific adaptation, while leaving verifier calibration and broader generalization unresolved.
6.1 Read the original evidence
Table I. The adapted 8B planner improves every task column relative to its zero-shot starting point. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First compare the Qwen3-VL-8B row with the final DreamPlan row: this holds the named planner backbone fixed while adding the proposed adaptation pipeline. Read the task columns separately before the average: rope rises from 0.20 to 0.60, cloth from 0.10 to 0.35, and toy from 0.70 to 0.85. Next compare with Qwen3-VL-32B, the strongest zero-shot baseline by overall score, at 0.35. Section IV-A.3 defines the entries: ten randomized trials per task, one action primitive in each trial, and scores of zero, one-half or one. The final column therefore summarizes partial-credit task performance, not simply completed-trial frequency. e-taske-evaluatione-score-plot
What it supports. DreamPlan reports 0.60 overall, compared with 0.33 for its own zero-shot backbone and 0.35 for the best overall baseline. The largest within-backbone gain is on rope, while cloth remains the lowest-scoring task after adaptation. This establishes improvement within the tested physical task setup.
Where the evidence stops. Figure 5 calls these values success percentages, but the stated protocol allows partial credit. Treat the gains as score differences. No confidence intervals, binary completion counts or explicit held-out-object split accompany the table.
Table III. Direct inference avoids the decision-time cost of generating and judging multiple videos. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first two numerical rows belong to Explicit Verification, with N specifying how many zero-shot action candidates are considered. Section IV-D explains that each candidate receives a world-model rollout and GPT-4o evaluation. Compare these rows with DreamPlan, which directly predicts its action after offline adaptation. Read the arrows carefully: average score should rise, while seconds and TFLOPs should fall. Increasing N from four to eight raises the displayed score from 0.48 to 0.50 while increasing both costs. DreamPlan's row combines a higher score, 0.60, with 1.12 seconds and 15.14 TFLOPs in the reported comparison. e-efficiencye-preference
What it supports. The table supports the paper's central deployment argument. Explicit verification reports 926.32 seconds for four candidates and 2605.56 seconds for eight, versus 1.12 seconds for DreamPlan. Its benefit is therefore not merely avoiding a modest sampling overhead: the reported video-based selection path dominates decision latency.
Where the evidence stops. The timing boundary and TFLOP accounting procedure are not detailed. Offline training and preference-generation costs are not amortized here, and decision latency is not complete robot execution time. This table is not an equal-training-budget comparison.
Figure 6. Selected physical executions show how different proposed motions change deformable objects. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task occupies a pair of rows. In the leftmost column, inspect the initial scene with colored action vectors and the target below it. Then follow the green-bordered fine-tuned execution across time and compare its endpoint with that target. Repeat for the pink-bordered zero-shot execution beneath it. The toy example concerns an arm's placement; the rope example concerns straightening its shape; the cloth example concerns a fold. The panels depict robot execution rather than the white-background generated videos in Figure 3. Their temporal frames show phases of a manipulation primitive, which should not be mistaken for a benchmark of several successive planning decisions. e-robot-examplese-evaluatione-taske-executione-generation
What it supports. The selected examples connect the planner's discrete choices to physical deformation: different grasp-and-place motions can move the same object toward or away from the desired configuration. They make the improvement in Table I easier to interpret while preserving the distinction between a predicted outcome and an executed one.
Where the evidence stops. The caption presents fine-tuned successes and typical zero-shot failures, not an exhaustive trial set. Use Table I for aggregate performance and the stated scoring protocol for its meaning; these examples cannot establish failure frequency or unseen-object generalization.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-world deformable manipulation across three tasks Rope straightening, cloth folding and toy arm repositioning; 10 randomized-initial-state trials per task; one executed action primitive per trial. Scores are 0 for failure, 0.5 for meaningful progress and 1 for complete success. A held-out split is not specified. | DreamPlan: rope 0.60, cloth 0.35, toy arm 0.85; overall 0.60. Average ordinal task score, higher is better | Zero-shot Qwen3-VL-8B: 0.20, 0.10, 0.70; overall 0.33. Qwen3-VL-32B has the highest baseline overall score, 0.35; GPT-4o has 0.28 and Qwen3-VL-4B has 0.22. The within-backbone gains are 0.40, 0.25 and 0.15 score units. Figure 5 labels equivalent values as success percentages, but the protocol includes partial credit; binary completion rates cannot be recovered. No uncertainty estimates are reported. e-evaluatione-taske-score-plot |
| Object-only world-model prediction ablation Two action-conditioned models trained under stated identical settings, using the same rendered trajectories; evaluation on the cropped object region of the final rollout frame. Test-set size and split are unspecified. | Object-only prediction: 26.25. PSNR, higher is better; table does not print a unit | Full-scene prediction: 24.70; arithmetic difference 1.55. The result supports improved final-frame reconstruction under this protocol. It does not establish better preference ranking or higher downstream robot scores for this particular design change. e-ablation |
| Direct planning versus explicit world-model verification Table III compares DreamPlan to zero-shot candidate sampling followed by world-model rollout and GPT-4o selection, with N=4 or N=8. Detailed timing boundaries are not specified. | DreamPlan: 0.60; 1.12 s; 15.14 TFLOPs. Average task score; decision time in seconds; reported computational overhead in TFLOPs | N=4: 0.48; 926.32 s; 7.82 × 10^4 TFLOPs. N=8: 0.50; 2605.56 s; 1.56 × 10^5 TFLOPs. Reported decision-time cost strongly favors transferring verification into training. These values do not include a disclosed amortization of offline fitting and preference generation, nor establish total physical execution time. e-efficiency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table II. Restricting the prediction target improves the reported object-region reconstruction score. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two column headings describe what each model learns to generate, not two different evaluation regions. Section IV-B.2 says both models use the same rendered robot trajectories and otherwise identical settings. One predicts the full scene, including background and robot arms; the other predicts a cropped object video on white. Evaluation is then performed on the cropped object region in the final rollout frame for both models. Read the upward arrow beside PSNR as the preferred direction. The comparison asks whether excluding irrelevant scene content during training helps reconstruct the region that matters for the manipulation outcome. e-ablatione-world
What it supports. Object-only prediction reaches 26.25 compared with 24.70 for full-scene prediction, a difference of 1.55 in reported PSNR. Because evaluation focuses on the same object region, the result supports the proposed choice of prediction target under this reconstruction protocol, rather than simply rewarding an easier background region.
Where the evidence stops. The table provides no sample count, split, variance or printed unit. Final-frame PSNR does not directly measure trajectory physics, preference correctness or robot success. A downstream planner ablation for this target change is not reported.
7. Analysis & limitations
7.1 What the evidence leaves open
Three task types and ten trials each provide narrow evidence. No explicit held-out-object protocol, confidence intervals or multi-primitive completion evaluation is reported. The paper has no dedicated limitations section. e-evaluatione-taske-repro-gaps
Figure 3 compares rendered-motion conditioning and task-specific adaptation against text-conditioned video baselines. This changes multiple factors; its selected plausible examples do not isolate the conditioning mechanism or calibrate the verifier's action ranking. e-generation
GPT-4o preferences inherit errors in predicted rollouts and visual goal judgment. The supplied evaluation gives no direct predicted-versus-real ranking agreement or judge-reliability measurement. e-preferencee-generatione-ablatione-task
7.2 Questions for discussion
- Does the object-only model's PSNR advantage improve action-ranking agreement on real outcomes?
- How much of the planner improvement survives when preference labels are shuffled or generated without a world model?
- Would the same gains hold for unseen objects and sequences requiring several corrective actions?
8. Reproducibility audit
8.1 Requirements and known gaps
The reported setup uses two opposing Franka FR3 arms with Franka hands, a RealSense D435i recording 960 × 540 RGB at 30 Hz, Cartesian end-effector pose control at 30 Hz and a workstation with a 32 GB RTX 5090. Reproduction also needs mask initialization, camera-to-robot mapping, motion rendering and the fixed primitive. e-hardwaree-executione-world
Resolve the dataset-count discrepancy and obtain the actual split, candidate count, video horizon, diffusion settings, optimizer schedule, planner update configuration and GPT-4o judging prompt/version. The local-workstation statement does not explain how GPT-4o execution was provided or accounted for. Proposed checks should record these choices explicitly. e-datae-preferencee-hardwaree-repro-gapse-efficiency
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does object-only prediction improve real-outcome action ranking?
Reader-proposed check, not performed: train full-scene and object-only models on the same declared split and optimization budget, then give both an identical frozen set of candidate actions on held-out initial states. Hold the GPT-4o judge prompt/version and sampling settings fixed. Execute the candidate primitives under repeatable resets and blind-score real outcomes using the paper's 0/0.5/1 scale. Compare pairwise ranking agreement, selected-action score and final-frame PSNR. Handle score ties explicitly. If object-only prediction raises PSNR but not agreement or selected-action performance, the proposed reconstruction-to-verification link would fail this test. Report reset variability rather than treating nominally repeated deformable states as identical. e-ablatione-preferencee-evaluatione-execution
Check 2: Separate informative preferences from the effect of fine-tuning
Reader-proposed check, not performed: start identical Qwen3-VL-8B copies and hold inputs, candidate actions, training steps and update settings fixed. Compare DreamPlan's predicted-outcome labels with labels shuffled within each candidate pool and with GPT-4o labels produced from action/goal information without generated futures. Retain an untouched zero-shot control. Use matched held-out robot trials, reporting both the original partial-credit score and binary completion, with uncertainty across trials and seeds. Record offline generation and optimization cost separately from decision latency. A specific world-model benefit requires its labels to outperform the controls; comparable improvements would weaken the claim that predicted physical consequences drive the gain. Training K and judging details must first be resolved because the paper omits them. e-preferencee-taske-evaluatione-efficiencye-repro-gaps
8.3 Reading coverage
Visual audit: All nine pages of the supplied arXiv v1 PDF were rendered and visually inspected, covering the title/author/version block; Figures 1–6; Tables I–III; Equations (1)–(5); method, hardware, collection and evaluation text; conclusion and references. All four text chunks were read individually in full. All six final original crops were separately viewed, including higher-resolution renders for the narrow ablation and efficiency tables. Arrow directions and plus/minus preferences in Figure 2 were checked against its caption and Section III-C; the dark-background examples versus white-background training description are disclosed. Figure 5's success-percentage labeling was checked against the partial-credit protocol. No supplied appendix is present, separate supplement availability remains unverified, and no linked material or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, author credits, version stamp, abstract and Figure 1
- PDF pp. 1–2: I. Introduction
- PDF pp. 2–3: II. Related Work, A–B
- PDF pp. 3–5: III. Method, A–C, Equations (1)–(5), Figure 2
- PDF pp. 5–8: IV. Experiments, A–D, Figures 3–6 and Tables I–III
- PDF p. 8: V. Conclusion and VI. Acknowledgement
- PDF pp. 8–9: References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all nine supplied PDF pages and all six final crops. No separate supplemental material was supplied; no appendix appears in this PDF.
- The inspected title and all six authors match the catalog. The title-page stamp identifies arXiv:2603.16860v1 [cs.RO], 17 March 2026. Only this supplied version was reviewed; no revision or edition comparison was possible.
- Code, project-page content and external references were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin stamp
The exact DreamPlan title and six authors match the catalog. Credits name USC Physical Superintelligence Lab and Toyota Research Institute. The stamp reads arXiv:2603.16860v1 [cs.RO], 17 Mar 2026.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and I. Introduction
The motivation concerns physically ineffective zero-shot VLM plans for deformable manipulation, expensive physical RL and initial exploratory data followed by offline world-model-assisted adaptation.
Go to primary source ↓e-plannerPDF p. 3, III. Method and III-A, Equation (1)
The paper defines observation/goal-conditioned planning and an action-conditioned future-observation model. Keypoints are independently detected, not explicitly matched, and the planner selects source and goal keypoints.
Go to primary source ↓e-worldPDF pp. 3–4, III-B, Equations (2)–(4)
Rendered robot trajectories condition a residual branch added to a frozen pretrained noise predictor. Training uses squared noise-prediction error and white-background object-only videos; CogVideoX-5B image-to-video is the backbone.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and caption; III-B below the figure
The three stages are zero-shot collection, world-model learning and preference-guided ORPO updates to the VLM. Plus/minus branches indicate preferred/disfavored predictions. The diagram's object-output examples show dark scene backgrounds although III-B specifies white-background training videos.
Go to primary source ↓e-dataPDF p. 4, III-B final paragraph; p. 6, IV-A.2 end of Pipeline Formulation
III-B reports several hundred action sequences from approximately four hours of interaction; IV-A.2 reports 2,056 trajectories for world-model training without reconciling the counts.
Go to primary source ↓e-preferencePDF pp. 4–5, III-C, Best-of-K construction, Equation (5) and implementation paragraph
Candidate rollouts yield a positive action and negatives, used without more video generation during updates. Qwen3-VL-8B is the planner and GPT-4o compares predicted images with the goal. Equation (5) prints a log-sigmoid of positive-minus-negative log probabilities; the text describes increasing preferred-action likelihood and requires no separate value function.
Go to primary source ↓e-hardwarePDF p. 5, Figure 4 and IV-A.1 Hardware Setup
Two Franka FR3 arms and Franka hands, RealSense D435i, 960 × 540 RGB at 30 Hz, Cartesian pose control at 30 Hz and a local workstation with a 32 GB NVIDIA RTX 5090 are specified.
Go to primary source ↓e-executionPDF p. 6, IV-A.2, Action primitive and Pipeline Formulation
Goal masks are annotated offline; SAM2 propagates a current mask from one bounding-box prompt. Farthest Point Sampling supplies keypoints; the VLM chooses an arm and grasp/target points, unprojected to 3D for a fixed grasp–transport–release sequence and return home.
Go to primary source ↓e-evaluationPDF p. 6, IV-A.3 Evaluation Protocol
Each of three tasks has ten randomized-initial-state trials. Each trial executes one action primitive and receives 0, 0.5 or 1 for failure, meaningful progress or complete success; average score is the primary metric.
Go to primary source ↓e-taskPDF p. 6, Table I, all method rows and task/average columns
DreamPlan scores 0.60/0.35/0.85 and 0.60 overall. Qwen3-VL-8B scores 0.20/0.10/0.70 and 0.33 overall; Qwen3-VL-32B averages 0.35, GPT-4o 0.28 and Qwen3-VL-4B 0.22. No uncertainty columns are given.
Go to primary source ↓e-score-plotPDF p. 6, Figure 5, axis, bars and annotations; IV-A.3
Figure 5 labels the bars Success Rate (%) and marks +40%, +25% and +15% for rope, cloth and toy. The corresponding Table I values arise from the partial-credit average-score protocol, not a separately defined binary success metric.
Go to primary source ↓e-generationPDF p. 5, Figure 3; p. 6, IV-B.1 Qualitative Comparison
The figure compares selected toy, cloth and rope rollouts. All start from cropped object images, but CogVideoX and Wan receive text descriptions whereas DreamPlan receives rendered robot trajectories. The authors describe DreamPlan predictions as action-consistent and physically plausible.
Go to primary source ↓e-ablationPDF p. 6, Table II; p. 7, IV-B.2 Ablation on Object-Only Video Prediction
Full-scene and object-only action-conditioned models use stated identical settings and the same rendered trajectories. PSNR on the final-frame cropped object region is 24.70 versus 26.25; a sample count, test split and downstream ablation score are not provided.
Go to primary source ↓e-robot-examplesPDF p. 7, Figure 6 and caption
Initial and target scenes accompany toy, rope and cloth execution sequences. Green-bordered fine-tuned rows show selected successes; pink-bordered zero-shot rows show typical failures, with colored arrows indicating proposed actions.
Go to primary source ↓e-interpretationPDF pp. 7–8, IV-C World-Model RL Enhances VLM Planning
The authors interpret improved scores and execution examples as the planner internalizing physical dynamics through world-model-derived preference pairs.
Go to primary source ↓e-efficiencyPDF p. 7, Table III, all rows; p. 8, IV-D Efficient Inference without World Model Sampling
DreamPlan reports average score/time/TFLOPs of 0.60/1.12/15.14; explicit verification N=4 reports 0.48/926.32/7.82 × 10^4 and N=8 reports 0.50/2605.56/1.56 × 10^5. IV-D contrasts direct planner inference with zero-shot proposals, video rollouts and GPT-4o selection. Timing boundaries and offline amortization are not detailed.
Go to primary source ↓e-repro-gapsPDF pp. 3–8, III-A–C and IV-A–D; p. 8, V–VI; p. 9, end of References
The supplied method and experiment sections give named model components, hardware and the action/scoring protocol but omit training K, rollout length, pair counts, optimizer schedule, detailed split and judge prompt/version. The nine-page paper ends in references without an appendix or dedicated limitations section.
Go to primary source ↓8.5 Primary sources
DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models ↗
PDF · 5,918 extracted words
Source fingerprint
7b19c6a4c2b785434e74630607229eef55bd46ca2237c3b972e4c047e637c094