RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulation
1. Paper overview
In one sentence: RoboHorizon couples staged reward generation and key-horizon visual learning to an imagined-control policy, improving simulated task success while leaving reward correctness and several training details unresolved. setuprewardsrepresentationworld-modelpolicytable-longablationimplementationreward-code
| At a glance | What to know |
|---|---|
| Research problem | Source description Sequential manipulation couples perception failures with sparse feedback: opening a box must precede removing its shoes. The authors propose Recognize–Sense–Plan–Act (RSPA) to expose sub-task structure through reward generation and representation learning before learning control. identity-problemsetup |
| Core mechanism | Source description RSPA adds a language-to-plan-to-reward pipeline to visual model-based RL; generated reward code is integrated into the task environment. architecturerewards |
| A key reported result | Nine long-horizon tasks across RLBench and FurnitureBench: 45.47% Mean task success rate (%). Six RLBench variation0 tasks and three FurnitureBench assembly tasks with low-randomness initialization; simulation, front-camera control; Table 1 averages five seeds. MV-MWM: 16.23%. The paper reports a 29.23-point improvement; subtracting rounded means gives 29.24, while averaging the listed per-task differences gives 29.23. The aggregate mixes benchmarks. For take shoes out of box, success is 36.5% versus 3.5%; the task-specific criterion is both shoes on the table. This demonstrates simulated execution, not physical deployment. table-longevaluationimplementationtask-definitions |
| Reading caution | Reader analysis The main text specifies 50 demonstrations per camera view for every task; A.2 specifies 50 for short tasks and 100 for long tasks in both benchmarks. The actual long-task training budget remains unresolved. evaluationimplementation |
Core contributions
- Source description
RSPA adds a language-to-plan-to-reward pipeline to visual model-based RL; generated reward code is integrated into the task environment. architecturerewards
- Source description
KMV-MAE learns multi-view representations over variable-length key-horizons, while a separate RSSM supports actor–critic learning in imagination. representationworld-modelpolicy
Figure 2. Reward structure and visual structure feed a separate world model and control policy. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the upper left: the plan descriptor feeds the reward generator, whose output modifies the environment. Follow the demonstrations into the upper-right Sense block, where keyframe discovery precedes convolutional features, masking and ViT reconstruction. The large downward arrows carry these training ingredients to Plan. In the lower-left timeline, observed representations z_t support state inference s_t, while action arrows connect successive states; Section 3.4 supplies the precise posterior and dynamics definitions. The lower-right imagination tree belongs to policy learning. Section 3.5 describes differentiating imagined returns to train an actor, so the tree alone does not establish an online search controller. architecturerepresentationworld-modelpolicy
What it supports. The method's integration occurs through rewards, representations and latent rollouts. The world model and actor have distinct roles and parameters, and autoencoder representations are explicitly frozen for world-model learning. The diagram therefore supports the catalog's modular Dual-system interpretation, while the detailed objectives explain how the parts communicate.
Where the evidence stops. The pictured masking stacks retain different views; Section 3.3 describes tube masking within a key-horizon. The schematic does not resolve the exact cross-frame mask correspondence. Its arrows should be read with the text, rather than as a complete training schedule.
2. Motivation
2.1 The problem and the proposed response
Sequential manipulation couples perception failures with sparse feedback: opening a box must precede removing its shoes. The authors propose Recognize–Sense–Plan–Act (RSPA) to expose sub-task structure through reward generation and representation learning before learning control. identity-problemsetup
2.2 What this reading follows
Removing two shoes from a box requires more than recognizing the objects: the lid must open, each shoe must be grasped and placed, and progress must survive several dependent actions. RoboHorizon addresses that sequence through two forms of training structure. An LLM writes rewards for intermediate progress, while multi-view demonstrations are divided into intervals between discovered keyframes. A separate latent world model then supports policy learning through imagined outcomes. The figures below connect those ingredients to the reported success rates and ablations. They also expose the limits of the evidence: simulator evaluation, confounded component comparisons, and a printed reward expression whose direction deserves checking. setuprewardsrepresentationworld-modelpolicytable-longablationimplementationreward-code
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Separate representation, dynamics and actor modules support Dual-system and Other mechanisms: action selection is actor–critic learning through imagined latent dynamics, rather than joint future/action generation or inverse dynamics. Thus Outside quadrants is supported. WM-RL and recurrent long-horizon modeling fit; the subcategory wording needs qualification because this paper uses multi-view RGB without explicit 3D geometry and does not demonstrate post-training a pretrained action policy. representationworld-modelpolicyevaluation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Locate the supervision before attributing intelligence
The task language does not directly become gripper motion. First, the Recognize stage combines language with a template describing objects and success conditions. Its plan feeds a coding stage that creates the reward function used by the simulator. Only then do the learned representation, dynamics and actor benefit from that signal. This distinction explains why RGB-only control does not mean the entire training pipeline is free of state information: the printed reward queries object positions, grasp conditions and success sensors, while keyframe discovery uses joint velocities and gripper states. As a reader's interpretation, much of the task structure is supplied through these interfaces. The reward-code discrepancy matters because the policy maximizes the generated signal, so a plausible verbal plan cannot by itself establish correct reinforcement. rewardsreward-examplereward-coderepresentationevaluation
Figure 7. The Recognize stage turns a task-specific plan into environment reward code. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward through the central chain, and then trace the horizontal arrows from the right. The setting template names graspable objects and conditions such as an opened lid; the plan descriptor expands these into ordered motions. The reward generator receives this plan together with API guidance, an example and rules distinguishing stepwise progress from stage completion. Its code flows into the environment, where learning receives dense feedback. This makes the LLM's role concrete: it prepares a training reward function. The objects, sensors and conditions in the example are simulator interfaces, even though the eventual control policy consumes RGB images. rewardsreward-examplereward-codeevaluation
What it supports. The method uses explicit task knowledge to reduce reward sparsity. The example assigns larger rewards to an opened box and completed shoe placement, while distance-based terms provide intermediate feedback. This also reveals a dependency: generated programs must agree with both the environment API and the intended direction of progress.
Where the evidence stops. The displayed grasped-shoe branch adds 1 − exp(−shoe1_to_success_dist), increasing with distance. Page 12 repeats it with a 'moving towards target' label. This apparent sign-direction problem is retained faithfully; the PDF does not establish whether the experiments used corrected code.
5.2 2. Connect masked perception to latent imagination
The Sense stage first divides each demonstration into intervals between discovered keyframes. It trains the autoencoder to recover observations across views and time while predicting rewards, so the representation carries both visual and task-related information. The Plan stage then uses frozen representations as inputs and reconstruction targets. Its posterior sees the current representation; the dynamics prior must predict the next latent state from preceding state and action alone. That asymmetry is what permits imagination without future observations. The actor and critic learn from these imagined sequences, with the horizon set to the duration of a key-horizon and behavior cloning anchoring the actor to expert actions. My interpretation is that segmentation affects both representation learning and policy-training rollout length, making a single 'key-horizon benefit' difficult to isolate without further controls. representationworld-modelpolicy
Figure 3. A key-horizon contains the intervening observations, not only its endpoint keyframes. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a column vertically to see one discovered moment from front, left, right and wrist cameras. Read a row horizontally to follow the demonstration across keyframes k = 1 through k = 11. The inset enlarges the interval between the final pair from the front view, showing why the training unit contains motion between endpoints. Section 3.3 defines each key-horizon using the observations from one keyframe time through the next. The paper's discovery criterion uses near-zero joint velocities and unchanged gripper states; the images themselves show the selected moments, but cannot verify that thresholding rule. keyframe-figurerepresentationpolicycameras-baselinesimplementation
What it supports. KMV-MAE reconstructs multi-view observations over these intervals and also predicts rewards. The intended benefit is a representation sensitive to interactions within task phases, rather than a collection of isolated endpoint images. Later, the policy-learning description also ties imagination length to the duration of each key-horizon.
Where the evidence stops. The figure illustrates one demonstration rather than measuring discovery accuracy. A numerical velocity threshold and the batching of variable-length intervals are not specified. The four-view masking description also leaves its adaptation to two-view FurnitureBench unresolved.
5.3 3. Read the comparisons as a sequence of narrower questions
Table 1 establishes the performance of complete configurations: RoboHorizon's staged rewards and key-horizon learning outperform the listed alternatives in the reported simulator settings. Figure 5 then tests a narrower reward substitution, adding stepwise LLM rewards to baselines while leaving out staged rewards. Its remaining gap does not isolate the value of staging because representations still differ. Figure 6 moves closer to component attribution, but its w/o key condition replaces the representation-learning method and does not disentangle segmentation from every downstream consequence. The strongest warranted conclusion is therefore configuration-level improvement with evidence for complementary components. Claims about unseen task variants or physical execution would require additional evaluation: RLBench uses variation0, FurnitureBench uses low-randomness initialization, and the real-robot cabinet sequence is explicitly an illustration. table-shorttable-longstepwise-comparisonablationevaluationimplementationfurniture-tasksconclusion
5.4 Training and inference
During training
Expert demonstrations seed replay. RLBench representation training uses front, left, right and wrist views; FurnitureBench uses front and wrist. The control agent uses front images, with no depth or proprioception as policy input. Reward code nevertheless queries simulator state. evaluationcameras-baselinesreward-code
The appendix describes eight parallel simulators, an eight-layer ViT encoder, six-layer decoder and 256-dimensional embeddings. Equation (3) collects representation, world-model and control losses, but the world-model description explicitly freezes its autoencoder inputs; it does not establish end-to-end joint optimization. implementationworld-modelpolicy
During inference
The described control path encodes the current front image, updates the latent state using the previous state/action, and supplies that state to the actor. New observations provide feedback. The LLM prepares rewards rather than issuing online motor commands; imagination is explicitly used to train the actor, and deployment-time tree search is not specified. architectureworld-modelpolicycameras-baselines
5.5 Implementation flow
- Recognize the stages
A pretrained LLM expands language into ordered motion descriptions; a second LLM role converts them into reward code using environment APIs, examples and rules. Rewards combine stepwise feedback with larger stage rewards. The authors report GPT-4o as the only tested model reliably handling every task, without a quantitative model comparison. rewardsreward-example
- Sense key-horizons
Keyframes are described as near-zero joint velocities with unchanged gripper states. Adjacent keyframes delimit variable-duration observation sequences. KMV-MAE masks convolutional features: three of four views completely and 95% of the remaining view, with tube masking through the interval. A ViT reconstructs pixels and predicts rewards. keyframe-figurerepresentation
- Learn latent dynamics
The RSSM consumes frozen autoencoder representations. Its posterior combines the preceding state and action with the current representation; its prior predicts without that observation. Decoders reconstruct representations and predict reward, allowing imagined rewards without rendering future images. world-model
- Learn actions in imagination
A stochastic actor and deterministic critic use imagined trajectories whose horizon H matches each key-horizon's length. The critic regresses stop-gradient lambda-return targets; actor gradients pass through the world model, with entropy regularization and auxiliary behavior cloning from expert actions. policy
6. Experiments & results
RoboHorizon trains a visual manipulation policy using LLM-generated staged rewards, representations learned from intervals between demonstration keyframes, and a separate latent dynamics model. The policy learns through imagined rollouts and expert actions, then controls from a front camera. The reported simulation gains are substantial, but conflicting data counts and an apparent reward-code error complicate reproduction.
6.1 Read the original evidence
Table 1. RoboHorizon leads the reported simulation comparisons under the paper's task-specific setup. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the colored headers before comparing averages: the long average combines six RLBench tasks with three furniture assembly tasks, whereas the short average uses four RLBench tasks. Compare RoboHorizon with MV-MWM row by row within the same task column. All entries are success percentages; the original caption states averaging over five seeds. Pages 5 and 13 define the surrounding protocol: front-camera control, RLBench variation0 and low-randomness FurnitureBench simulation. The green annotations are intended as gains over MV-MWM. They should be interpreted as percentage-point differences, not relative percentage improvements over a baseline success probability. table-shorttable-longheadline-discrepancyevaluationcameras-baselinesimplementationtask-definitions
What it supports. Short-task success is 75.58% versus 50.23%, a 25.35-point difference. Long-task success is 45.47% versus 16.23%; the paper's 29.23-point gain is consistent with calculating before rounding the averages. For shoe removal specifically, the reported rates are 36.5% and 3.5%. These are task-completion measurements, not video-quality scores.
Where the evidence stops. The abstract's 23.35% short-task gain conflicts with this table. No uncertainty intervals accompany the means. The main text and appendix disagree on long-task demonstration counts, and reward design changes alongside representation design in these baseline comparisons.
Figure 5. Stepwise LLM rewards help the baselines, but leave a gap to the complete system. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each baseline color together with its lighter extension. The dark portion corresponds to the original baseline, and the lighter addition shows the improvement from LLM-generated stepwise rewards; the annotations mark that increment. Gray bars show the full RoboHorizon system. Use the upper panel for short tasks and the lower panel for long tasks, noting that their vertical scales differ. Section 4.1 explains the intervention: these baselines receive stepwise rewards without the staged reward structure of RoboHorizon. Consequently, this figure asks whether adding that limited reward signal closes the gap, rather than matching every component across methods. stepwise-comparisontable-longcameras-baselines
What it supports. The baseline extensions indicate improved success, yet the complete system remains higher in every displayed task. For example, the MV-MWM shoe-removal bar gains 6.1 percentage points from stepwise LLM rewards, while the full model still reaches 36.5%. The source therefore supports the narrower finding that this reward substitution alone is insufficient.
Where the evidence stops. The remaining gap cannot be assigned solely to staged rewards: representation learning and other method choices also differ. The plot provides no uncertainty bars, and its increment annotations are not separate final success rates.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Four short-horizon RLBench tasks Simulation; variation0; front-camera control; Table 1 averages five seeds. The four tasks are phone placement, umbrella removal, rubbish disposal and wine stacking. | 75.58% Mean task success rate (%) | MV-MWM: 50.23%; a 25.35 percentage-point gain. Table 1 and the introduction support 25.35 points; the abstract instead says 23.35%. These are absolute success-rate differences, not relative percentage improvements. Table 1 supplies no uncertainty intervals. table-shortheadline-discrepancyevaluationimplementation |
| Nine long-horizon tasks across RLBench and FurnitureBench Six RLBench variation0 tasks and three FurnitureBench assembly tasks with low-randomness initialization; simulation, front-camera control; Table 1 averages five seeds. | 45.47% Mean task success rate (%) | MV-MWM: 16.23%. The paper reports a 29.23-point improvement; subtracting rounded means gives 29.24, while averaging the listed per-task differences gives 29.23. The aggregate mixes benchmarks. For take shoes out of box, success is 36.5% versus 3.5%; the task-specific criterion is both shoes on the table. This demonstrates simulated execution, not physical deployment. table-longevaluationimplementationtask-definitions |
| FurnitureBench assembly tasks Cabinet, lamp and round table in the simulator, low-randomness initialization, front-camera control; five-seed Table 1 protocol. | Cabinet 41.0%; lamp 58.5%; round table 61.3%. Task success rate (%) | MV-MWM: 26.6%, 43.5% and 46.7%, respectively. Success means assembling the specified furniture components. Figure 10 includes a real-robot cabinet illustration, but A.2 defines these experiments as simulation and Section 5 leaves real-world application to future work. table-longevaluationimplementationfurniture-tasksconclusion |
| Component ablation across nine long-horizon tasks Figure 6 compares the full model, manual rewards (w/o LLM), and an MV-MWM-based replacement for key-horizon representation learning (w/o key). | The full model has the highest bar in each of the nine task groups; all three conditions are shown for every task. Task success rate, qualitative bar comparison | The w/o key model exceeds w/o LLM on the two shoe tasks, books and drawer placement; w/o LLM exceeds w/o key on empty container, slide cabinet & place cups and the three furniture assemblies. Both components contribute in these configurations. No printed per-bar values or uncertainty are supplied, so these comparisons are qualitative. Replacing the key-horizon module is broader than isolating temporal segmentation alone; object density is the authors' explanation of the pattern, not an independently controlled variable. ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. Reward generation and key-horizon learning contribute differently across the displayed tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: orange removes LLM-generated rewards and uses manual reward structures; blue removes key-horizon representation learning and uses MV-MWM for world-model construction. Gray is the complete method. Each task has all three bars. Compare the reduced models to gray, then compare orange with blue across tasks. Blue is higher on shoe removal, shoe insertion, books and drawer placement. Orange is higher on emptying the container, slide-cabinet placement and the three furniture assemblies. The authors relate this pattern to dispersed versus crowded objects. That explanation is a hypothesis about these task groups; scene density was not independently manipulated. ablation
What it supports. The complete method has the highest bar in all nine task groups. The changing ordering of the reduced models supports complementary contributions across these configurations. It motivates a more controlled segmentation experiment: the current w/o key replacement changes the representation-learning method more broadly than just moving temporal boundaries.
Where the evidence stops. The bars have no printed values or uncertainty intervals, so the comparison supports qualitative rankings rather than exact ablation scores or statistical significance. The authors' object-density explanation is not isolated by a controlled change to scene density.
7. Analysis & limitations
7.1 What the evidence leaves open
The main text specifies 50 demonstrations per camera view for every task; A.2 specifies 50 for short tasks and 100 for long tasks in both benchmarks. The actual long-task training budget remains unresolved. evaluationimplementation
The printed grasped-shoe reward adds 1 − exp(−shoe-to-target distance), which increases with distance despite its 'moving towards target' label. This is an apparent shaping-direction error in the example, not proof of which code generated the results. reward-codereward-example
Evaluation is restricted to selected tasks, RLBench variation0 and low-randomness FurnitureBench. No held-out task/variation protocol or physical success evaluation is reported. Manual-reward baseline comparisons also change reward design, limiting architectural attribution. evaluationcameras-baselinesimplementationconclusion
7.2 Questions for discussion
- How much of the gain survives a reward-matched, duration-matched segmentation comparison?
- Can the staged reward code remain correct under new objects, task variants and observation errors?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs synchronized demonstrations, expert actions, keyframe metadata, reward API bindings and the MV-MWM implementation used as the base. A.2 doubles the Panda's maximum demonstration velocity. Exact software versions, interaction budgets, evaluation episode counts, keyframe velocity thresholds and the two-view masking adaptation are not specified. representationevaluationimplementation
A.2 reports one RTX 4090 with 24 GB VRAM for RLBench, and 12 hours for 'MV-RoboWM' versus 16 hours for MV-MWM. The unexplained 'MV-RoboWM' name prevents unqualified attribution of that timing to RoboHorizon. FurnitureBench hardware and a complete hyperparameter table are absent. implementation
Proposed checks: audit the printed reward against distance-controlled states, then compare keyframe-aligned and duration-matched random segments while fixing rewards, demonstrations and imagination horizons. These checks would separate reward correctness from the temporal-representation mechanism. reward-coderepresentationpolicyablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check reward direction before optimizing a policy
Reader-proposed, not executed: construct controlled simulator states for take shoes out of box with the box opened, one shoe grasped but not placed, and all other conditions fixed. Sweep the shoe-to-success-sensor distance and evaluate the printed reward separately from task completion. Compare it with a disclosed alternative that decreases with distance while retaining the same stage bonuses and API conditions. The printed branch should increase as distance increases; failure to observe this would indicate a mismatch between the tested function and the PDF example. Then train paired policies with identical data, seeds and interaction budgets, measuring both stage completion and final success. A changed learning outcome would identify reward implementation as a material reproduction variable. reward-codereward-exampleevaluationtask-definitions
Check 2: Separate keyframe alignment from imagination length
Reader-proposed, not executed: compare keyframe-aligned intervals against randomly positioned intervals matched for duration, using the same demonstrations, reward code, view/tube masks, architecture and optimization budget. First hold imagination lengths identical between conditions to isolate temporal alignment in representation learning. Then cross this comparison with a fixed imagination horizon to test whether the policy-training schedule explains part of the gain. Use one shoe task and empty container, where Figure 6 gives different reduced-model orderings, and report repeated-seed final success plus sub-task completion. If matched random intervals perform equally well, the claim that discovered boundaries themselves drive improvement weakens. Resolve the 50-versus-100 demonstration discrepancy before treating either setting as a reproduction. representationpolicyablationevaluationimplementationtask-definitions
8.3 Reading coverage
Visual audit: Inspected the title, author affiliation and v3 margin; Figures 1–10; Table 1; method equations and masking description; both appendix prompt stages and printed reward code; implementation, data, hardware and task-definition pages. All six final crops were inspected against their source pages. Figure 6 retains all 27 bars, including the orange put-books bar and the blue put-item and slide-cabinet bars, with the complete axes, task labels and legend. Its task rankings agree with the adjacent ablation discussion. Figure 7 retains its right-side prompt labels. The Figure 2 masks remain schematic relative to the tube-masking description. Figure 7 and page 12 repeat the distance-shaping discrepancy. References on pages 8–9 were read in the complete text but are outside the visual pass. No separate supplement, external implementation or reproduced experiment is included.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 Method: 3.1–3.5
- 4 Experiments and 4.1 Performance Comparison
- 5 Conclusion and Discussion
- References
- A Technical Appendix: A.1–A.4, including both reward-generation stages and all task descriptions
Outside the original text pass
- Identity: PDF p. 1 identifies arXiv:2501.06605v3 [cs.RO], 24 January 2025. The observed title and all four authors match the catalog. The catalog submission date is 11 January 2025; this review concerns the supplied v3, and does not establish earlier-version equivalence.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction-only visual omission was addressed by inspecting PDF pages 1–7 and 10–16, including every figure and table, and all six final crops.
- Separate supplemental material availability has not been fully verified.
- No separate supplements were supplied. External code was not inspected and experiments were not reproduced; printed appendix code was read as source text only.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identity-problemPDF p. 1, title block, arXiv margin, Abstract and Introduction
Title and authors match the catalog; the artifact is v3, dated 24 January 2025. The introduction motivates sparse-feedback and multi-stage perception problems.
Go to primary source ↓setupPDF p. 3, Section 3.1
The task is a POMDP with sequentially dependent sub-tasks, RGB observations, language task descriptions and multi-view demonstrations.
Go to primary source ↓architecturePDF p. 2, Figure 2 and Introduction
The overview separates reward generation, key-horizon representation learning, world-model learning and imagined policy learning.
Go to primary source ↓rewardsPDF p. 4, Section 3.2; PDF p. 10, A.1
Two LLM roles translate task language into plans and reward code; stage and stepwise rewards are distinguished. GPT-4o reliability is an author observation.
Go to primary source ↓keyframe-figurePDF p. 3, Figure 3 and caption
Four viewpoint rows show eleven discovered keyframes; an inset expands the interval between the last two front-view keyframes.
Go to primary source ↓representationPDF p. 4, Section 3.3, Eq. (1) and KMV-MAE loss
Keyframes use near-zero joint velocities and unchanged gripper states. Adjacent intervals train masked multi-view reconstruction and reward prediction; three views and 95% of the fourth are masked.
Go to primary source ↓world-modelPDF pp. 4–5, Section 3.4, Eq. (2) and world-model loss
Frozen autoencoder features condition an RSSM posterior and serve as reconstruction targets; action-conditioned dynamics predict without current observations.
Go to primary source ↓policyPDF p. 5, Section 3.5 and Eq. (3)
Actor–critic training uses variable key-horizon imagination lengths, lambda returns, stop-gradient critic targets, entropy regularization and expert-action behavior cloning.
Go to primary source ↓evaluationPDF p. 5, Section 4, Environmental Setup and Multi-view Camera Setup
The paper specifies RGB-only control, replay demonstrations, relative gripper-position changes, 50 demonstrations per task/view, and low-randomness FurnitureBench.
Go to primary source ↓cameras-baselinesPDF p. 6, Multi-view Camera Setup continuation and Baselines
Control uses front images; representation learning uses four RLBench views or two FurnitureBench views. Baselines differ in representations and use manual rewards in Table 1.
Go to primary source ↓table-shortPDF p. 6, Table 1, RoboHorizon and MV-MWM rows, four short-task columns and Short Avg.; caption
Five-seed means are 75.58 and 50.23. RoboHorizon short-task entries are 78.4, 75.2, 74.8 and 73.9 percent.
Go to primary source ↓table-longPDF p. 6, Table 1, RoboHorizon and MV-MWM rows, long-task columns and Long Avg.; caption
Five-seed long averages are 45.47 and 16.23. Shoes: 36.5 versus 3.5; furniture cabinet/lamp/table: 41.0/58.5/61.3 versus 26.6/43.5/46.7.
Go to primary source ↓headline-discrepancyPDF p. 1, Abstract; p. 2, Introduction; p. 6, Table 1 and Section 4.1
The abstract states 23.35% short-task improvement; Introduction and Table 1 state 25.35%. The table's long improvement annotation is 29.23%.
Go to primary source ↓stepwise-comparisonPDF p. 6, Section 4.1, LLM-generated stepwise-reward comparison; p. 7, Figure 5
Baselines gain stepwise LLM rewards without staged rewards. Bars remain below RoboHorizon; annotations show absolute additions to success rate.
Go to primary source ↓ablationPDF p. 7, Figure 6 and Ablation Study of RoboHorizon
Removing LLM rewards uses manual rewards; removing key-horizon learning uses MV-MWM for world-model construction. All three bars are present in each of nine task groups, with the full model highest. W/o key exceeds w/o LLM on shoes, books and drawer placement; the ordering reverses on empty container, slide-cabinet placement and furniture assembly.
Go to primary source ↓conclusionPDF p. 7, Section 5
Human feedback for reward generation and real-world manipulation with sim-to-real transfer are future work.
Go to primary source ↓reward-examplePDF p. 10, Figure 7 and A.1; p. 11, Stage 1 prompt and plan
Templates and reward-interface examples feed separate planning and reward-generation roles; the shoes example includes box opening, two placements and empty-gripper checks.
Go to primary source ↓reward-codePDF p. 12, Data Flow for Stage 2, Generated Rewards, both shoe-grasped branches; compare p. 10, Figure 7
The printed code queries simulator conditions/positions and adds 1 − exp(−distance) when a shoe is grasped. It gives 5 for an opened box and 4 for each placed shoe.
Go to primary source ↓implementationPDF p. 13, A.2 and A.3 opening paragraph
Simulated 7-DoF Panda experiments; 50 short-task and 100 long-task demonstrations/view; doubled demonstration velocity; eight simulators; ViT 8/6 layers and 256 embedding; one 24-GB RTX 4090; unexplained MV-RoboWM timing label; variation0 only.
Go to primary source ↓task-definitionsPDF pp. 14–15, Figure 9 and A.3, Take Shoes Out Of Box and other long-task entries
Shoes success is both shoes on the table. Figure 9 illustrates the task sequences; slide-cabinet has four pictured sub-tasks, whereas its text lists task horizon five.
Go to primary source ↓furniture-tasksPDF p. 16, Figure 10 caption and A.4
Lamp, round table and cabinet assembly have listed horizons 7, 8 and 11. The caption identifies cabinet frames as a real-robot illustration, not a reported policy evaluation.
Go to primary source ↓8.5 Primary sources
RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulation ↗
PDF · 10,251 extracted words
Source fingerprint
0df76f014f7b531fec74fc45e7d1737d880e804be66d252d3d3ffdfe20437200