PAPER REPORTENAll readings ↗

From World Models to World Action Models: A Concise Tutorial for Robotics

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Xiaoxiong Zhang; Xiong Zeng; Wei Zhang

Source: 2607.00836 ↗ · Project page ↗ · Catalog record

Reading: 45 / 558 · 4 original figures & tables · ~18 min ·

1. Paper overview

In one sentence: The tutorial separates prediction targets from action-generation interfaces, revealing why future modeling can support a robot policy without guaranteeing useful control. e-identitye-rolee-predictore-taxonomy-figuree-auxiliarye-scope

At a glanceWhat to know
Research problem
Source description

World-model terminology spans video generators, latent predictors and physics simulators, although these systems predict different quantities and serve different roles. The tutorial starts from a task-relevant world containing the robot, relevant objects and ambient environment. A policy should move that world toward a goal while respecting constraints. Driving and table cleaning illustrate these definitions; they are not benchmark tasks evaluated here. e-worlde-scope

Core mechanism
Source description

The paper separates a predictor's functional role from a policy's role. A WM consumes candidate actions and predicts consequences; a WAM generates actions using future-oriented modeling. Modular decision–planning–control, fast–slow and end-to-end policies remain alternative implementations of the same policy function. e-role

Reading caution
Source description

This is deliberately a concise tutorial rather than an exhaustive survey. Its WAM discussion emphasizes recent video-based robotic policies; classical model-based control and reinforcement learning enter only to clarify that connection. No success rates, datasets/splits, baseline comparisons or ablations are reported. e-scopee-role

Core contributions

  • Source description

    The paper separates a predictor's functional role from a policy's role. A WM consumes candidate actions and predicts consequences; a WAM generates actions using future-oriented modeling. Modular decision–planning–control, fast–slow and end-to-end policies remain alternative implementations of the same policy function. e-role

  • Source description

    Two complementary maps organize prediction: observation-space models vary in spatial explicitness and action abstraction; state-space models use latent states, point tracks, neural-symbolic predicates or physical variables. e-observatione-state

  • Source description

    Four video-based policy paradigms expose different interfaces between prediction and action: explicit imagined futures, predictive features, joint generation and auxiliary training. Their tradeoffs concern grounding, interpretability, data requirements and inference cost. e-imaginee-featuree-jointe-auxiliarye-closing

Figure 4. Prediction and action generation occupy different functional roles. Original paper, p. 2 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with the upper block. The current observation, o_t, and a candidate action, a_t, enter the world model; the output is a predicted observation or state. Follow the lower block next: action has moved to the output side, and the future-observation output appears in parentheses. This is a schematic distinction between predicting consequences and producing a policy action. Section 1.2 and Figure 3 place the policy in a feedback loop with the robot and environment. Section 1.3 adds that the predictive capability needs a decision, planning or control mechanism to complete a physical task. e-rolee-worlde-auxiliarye-taxonomy-figure

What it supports. The key distinction is which side of the model contains the action. Supplying an action to forecast its consequences solves a prediction problem. Generating an action that moves toward a goal solves a policy problem. The tutorial uses this role distinction across several possible internal architectures.

Where the evidence stops. The single WAM box does not establish a single neural network or a deployed controller. Section 3 later expands it into several architectures, including an auxiliary branch removable at inference; that case complicates the earlier inference-time definition.

2. Motivation

2.1 The problem and the proposed response

Source description

World-model terminology spans video generators, latent predictors and physics simulators, although these systems predict different quantities and serve different roles. The tutorial starts from a task-relevant world containing the robot, relevant objects and ambient environment. A policy should move that world toward a goal while respecting constraints. Driving and table cleaning illustrate these definitions; they are not benchmark tasks evaluated here. e-worlde-scope

2.2 What this reading follows

A system that predicts the next image still needs a way to choose what the robot should do. This tutorial makes that distinction concrete, then maps the representations and interfaces that connect prediction to action. Read the diagrams in two passes: first ask what future the model represents, then ask how that representation reaches an executable action. The reviewed source is arXiv v7, dated 26 August 2026. Its evidence is conceptual rather than experimental: it provides useful architectural comparisons, but no measured ranking of methods. A key reading issue is its inclusion of training-only future prediction under an earlier definition framed around inference-time coupling. e-identitye-rolee-predictore-taxonomy-figuree-auxiliarye-scope

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryFoundational work
ArchitectureNot applicable
Prediction paradigmNot applicable
QuadrantNot applicable

This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded Foundational work / Surveys & technical resources classification fits this tutorial. Architecture, prediction paradigm and quadrant are appropriately Not applicable to the resource itself: it compares several architectures instead of introducing one. Figure 8 distinguishes separate inverse-dynamics modules from a single joint generator and training-only prediction; none licenses assigning the entire tutorial to One Model or a joint-prediction quadrant. e-scopee-imaginee-jointe-auxiliarye-taxonomy-figure

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • For the reviewed WM formulations: current observation o_t and candidate action a_t, whose representation depends on the intended use.
  • For the reviewed video-based policies: current observation o_t and language instruction l.
  • Predicted observation or structured state, optionally over a future trajectory.
  • Robot action sequence; explicit future observations, latent predictive features or training-only future supervision depend on the paradigm.

4.2 Equations and their role

yt+1pθ(ot,at)y_{t+1}\sim p_\theta(\cdot\mid o_t,a_t)
The parameterized predictor p_theta conditions on current observation o_t and action a_t. Its target y is either the predicted observation hat{o}_{t+1} or state x_{t+1}. The same prediction role thus permits observation-space or state-space implementations. e-predictor
o^t+1:t+Hpθ(ot,l),at:t+H1qϕ(ot,o^t+1:t+H).\begin{aligned}\hat{o}_{t+1:t+H}&\sim p_\theta(\cdot\mid o_t,l),\\a_{t:t+H-1}&\sim q_\phi(\cdot\mid o_t,\hat{o}_{t+1:t+H}).\end{aligned}
The video model p_theta generates a future-observation sequence from observation o_t and language instruction l. The inverse-dynamics or goal-conditioned distribution q_phi maps that future and the current observation to the action sequence. H indexes the displayed prediction horizon; the tutorial assigns no numerical horizon. e-imaginee-taxonomy-figure
ft=H(uθ(ot,l))f_t=\mathcal{H}\bigl(u_\theta(o_t,l)\bigr)
The video backbone u_theta supplies intermediate representations, the extraction operation calligraphic H selects features, and f_t is the resulting spatiotemporal representation. This is only the complete first relation in the printed display; its second relation lacks a left-hand variable. e-featuree-equation-gap
(o^t+1:t+H,at:t+H1)pθ(ot,l)(\hat{o}_{t+1:t+H},a_{t:t+H-1})\sim p_\theta(\cdot\mid o_t,l)
The joint formulation puts predicted observations and corresponding robot actions in one conditional generative distribution. It describes their modeled coupling, without proving that sampled actions physically realize the generated scene. e-joint

5. Method in detail

5.1 Begin with the task before choosing the prediction space

Reader analysis

The tutorial's table-cleaning example makes the abstraction concrete: the relevant world includes the robot, dishes, rag, table and household environment, while the desired state includes acceptable object placement. Its policy framework connects intent and current observation to an action, then returns observations from the evolving world. The reader's implication is that selecting RGB, point tracks or physical states should follow the information needed for that task. A representation can discard visual detail, yet still needs to preserve what makes the next action appropriate. Figures 6 and 7 organize those representation choices; they do not specify the controller that completes the loop. Keeping the task and the controller visible prevents a prediction interface from being treated as a complete robot solution. e-worlde-rolee-observatione-state

Figure 6. The predicted observation and the conditioning action define separate design choices. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read vertically for observation structure: the source orders RGB, multi-view RGB, RGB-D and point clouds by increasing spatial explicitness. Read horizontally for the abstraction of action conditioning: robot commands, interface controls, learned latent actions and language. The center block preserves the common observation-prediction role across these choices. Model names are the tutorial authors' placements. DreamDojo appears separately for pretraining and finetuning, illustrating that a configuration or training stage can occupy a different position. Sections 2.1's examples explain why camera controls support interactive visual environments, while robot commands can support action-conditioned simulation and language can guide visual planning. e-observatione-predictor

What it supports. The map makes representation and usability separate questions. Adding explicit geometry changes what is predicted; changing the action interface changes how a user or robot can condition the prediction. A system can therefore support rich visual interaction without providing the low-level commands needed to execute a robot task.

Where the evidence stops. These axes have qualitative categories, not calibrated scales or performance scores. Named placements were read from this tutorial, without independently reviewing the cited systems. Greater spatial explicitness alone does not demonstrate more accurate dynamics or better executed control.

5.2 Trace the representation that reaches the action module

Source description

Imagine-then-execute begins with a language-conditioned visual future and then asks inverse dynamics or a goal-conditioned policy to produce actions realizing it. The split lets the visual generator and action-grounding model learn from different data sources, but predicted-subgoal errors can contaminate the action sequence. Feature conditioning changes the intermediate object: a video backbone supplies spatiotemporal features without decoding a full future sequence. The source motivates this choice through the iterative cost of diffusion sampling. Joint generation removes the separate interface and represents future observations and actions together. It still needs labeled robot data to adapt the action output, and its visual and action objectives can be misaligned. These are distinct information flows, so the tutorial's unified vocabulary should not erase their different training requirements. e-imaginee-featuree-joint

Figure 7. State prediction chooses which structure to preserve from observations. Original paper, p. 5 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Begin with the left block: observation and action lead to a predicted state, x_{t+1}. The branches on the right describe what that state can mean. Latent-state models use compact features, with separate pretrained and learned groupings. Point-track models preserve motion at selected scene locations. Neural-symbolic models use perceptually grounded predicates and skill transitions. Physical-state models represent variables used by mechanics. The accompanying text on pages 5–6 explains that physical pipelines can reconstruct a scene and align simulator parameters with real trajectories before rollout. Follow the branches as alternative abstractions, rather than successive stages of one system. e-statee-physicale-predictor

What it supports. The map broadens world modeling beyond video synthesis. A future can be represented as compact features, moving points, logical facts or physical variables. The tutorial relates these choices to different balances of compactness, semantic structure and physical interpretability; it does not establish a universally best state representation.

Where the evidence stops. Each abstraction carries assumptions. Point selection can restrict scalability, predicates require grounded skill structure, and physical models need scene and dynamics alignment. This figure provides categories and examples, with no numerical comparison of their information loss or control quality.

5.3 Separate future supervision from future use at deployment

Reader analysis

The auxiliary branch is the tutorial's most revealing boundary case. During training, a prediction head encourages the policy representation to encode future scene evolution alongside the action objective. During inference, the source explicitly allows that head to disappear. The authors therefore caution that accurate visual prediction need not improve control. Reader analysis: this family demonstrates why training signals and deployed computation should be recorded separately when classifying a policy. It also exposes the tension with Section 1.3's inference-time definition of WAM. A broad category of future-supervised policies could include this branch, while a narrower category would demand an identifiable predictive computation during action selection. The tutorial does not resolve that choice with experiments; the proposed checks below make the conceptual and empirical questions testable. e-rolee-auxiliarye-taxonomy-figuree-scope

Figure 8. Four interfaces connect future-oriented modeling to action generation. Original paper, p. 7 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Compare the upper branches first. Imagine-then-execute passes predicted observations to inverse dynamics; the feature-conditioned branch passes f_t through the dotted feature-extraction arrow instead. Both also provide the current observation to the action module. The lower-left branch produces video and actions jointly. The lower-right branch encloses its future output in the dotted training-only notation explained by the legend. Section 3 clarifies that its auxiliary prediction branch can be removed at inference. Across the panels, l is language instruction, o_t is current observation, and the indexed sequences show future observations and actions over the displayed horizon. The colors group paradigms and their representative method names. e-imaginee-featuree-jointe-auxiliarye-taxonomy-figuree-rolee-scope

What it supports. The diagrams identify where to inspect the connection between a future and an action. It may be a decoded visual plan, an internal feature interface, a joint output distribution or a training objective shaping the policy. This separation helps prevent auxiliary supervision from being mistaken for explicit planning during execution.

Where the evidence stops. Section 1.3's inference-time definition and this figure's training-only branch are not fully reconciled. The paper offers qualitative efficiency and consistency arguments, without latency measurements, control-success comparisons or ablations that rank these four families.

5.4 Training and inference

During training

Source description

Imagine-then-execute permits separate data sources: visual data for the generator and action-labeled robot trajectories for grounding. Latent-action pretraining instead infers compact transition variables from unlabeled videos through reconstruction with an information bottleneck. The tutorial supplies neither dataset sizes nor a fixed training schedule. e-imaginee-observatione-scope

Source description

Joint modeling commonly starts with a pretrained video backbone, changes its architecture or output space to produce actions, and adapts it on labeled robot trajectories. Auxiliary prediction adds future-video supervision to action supervision. No universal loss coefficient, optimizer, frozen-module specification or compute budget is provided. e-jointe-auxiliarye-scope

During inference

Source description

Explicit visual planning first generates a future sequence, then grounds it in actions, potentially through end-effector poses, object poses or optical flow. The authors warn that subgoal errors propagate into action errors. Feature conditioning avoids complete video decoding, but its latent interface makes meaningful future use harder to verify. e-imaginee-feature

Source description

Joint modeling samples corresponding actions and visual futures from one generative model. Auxiliary prediction can retain only the action head at inference. The source gives no shared controller frequency, replanning horizon, candidate-selection rule or deployment protocol, so its diagrams should not be read as executable algorithms. e-jointe-auxiliarye-scope

5.5 Implementation flow

  1. Specify the world and the decision loop

    Select entities, state variables, objectives and constraints from the task. Figure 3 routes intent and current observation through a policy to a robot action, with observations feeding back from the world. A predictor needs a planning, decision or control mechanism to influence execution; a predicted scene alone does not accomplish the task. e-worlde-role

  2. Choose the observation/action interface

    RGB supplies abundant data but implicit geometry; multi-view RGB adds geometric constraints, RGB-D adds depth, and point clouds expose 3D structure. Concrete robot commands support control and simulation. Interface controls generate interactive views; latent actions learned from action-free videos support pretraining; language supports high-level visual planning. Figure 6 separates these axes rather than ranking model quality. e-observation

  3. Choose a structured prediction target

    Latent states compress observations using learned or pretrained encoders. Point tracks retain selected motion while adding a point-selection prior. Neural-symbolic predicates ground Boolean facts in perception and describe skills through preconditions and effects. Physical-state models use quantities such as pose, velocity, contact and friction; scene reconstruction and trajectory-based simulator alignment precede physics rollouts. e-statee-physical

  4. Locate the future-to-action connection

    Imagine-then-execute passes generated subgoals to inverse dynamics or a goal-conditioned policy. Feature-conditioned methods pass internal video representations without decoding complete futures. Joint methods generate video and actions together. Auxiliary methods supervise future prediction during policy training and can discard the prediction branch during deployment. These are family-level descriptions, not one proposed architecture. e-imaginee-featuree-jointe-auxiliary

6. Experiments & results

This tutorial separates predicting a robot's world from choosing actions within it. It organizes world models by prediction space, then compares four ways of connecting visual futures to policies. Its contribution is a conceptual design map, with qualitative tradeoffs and a notable training-versus-inference definition tension; it introduces no evaluated robot system.

Source and visual limitations
Reader analysis

The supplied v7 is a nonempirical tutorial: its eight figures explain definitions and taxonomies, and Section 3 proceeds directly into references. It contains no quantitative results table, experimental comparison, ablation or diagnostic measurement. Accordingly, this edition uses four original mechanism/design-space figures and leaves the base results array empty. The diagrams support conceptual reading, not measured claims about robotics performance. Separate supplemental material availability remains unverified. e-scopee-closinge-taxonomy-figure

6.1 Read the original evidence

No results visual is included in this edition. The source-specific visual limitations are recorded above.

6.2 Results and evaluation conditions

No quantitative results are included in this reading.

6.3 Ablations and diagnostic examples

No ablation or diagnostic visual is included in this edition. The source-specific visual limitations are recorded above.

7. Analysis & limitations

7.1 What the evidence leaves open

Source description

This is deliberately a concise tutorial rather than an exhaustive survey. Its WAM discussion emphasizes recent video-based robotic policies; classical model-based control and reinforcement learning enter only to clarify that connection. No success rates, datasets/splits, baseline comparisons or ablations are reported. e-scopee-role

Reader analysis

There is an unresolved definition tension: Section 1.3 requires action generation to couple with observation prediction during inference, but Section 3 includes a branch removable before inference. A broad future-supervised-policy category accommodates that branch; a strict inference-time definition does not automatically do so. e-rolee-auxiliarye-taxonomy-figure

Source description

The authors identify error propagation in explicit plans, opaque future information in latent features, scarce action labels and potentially misaligned objectives in joint models, and uncertain control benefit from auxiliary visual accuracy. These are qualitative concerns, not measured comparative effects in this paper. e-imaginee-featuree-jointe-auxiliary

Reader analysis

The feature-conditioned equation is incomplete in the rendered PDF itself. Its prose and Figure 8 explain the interface, but supplying the absent action variable as if printed would conceal a source omission. e-equation-gape-taxonomy-figure

7.2 Questions for discussion

  1. Should training-only future supervision qualify as a WAM under the inference-time definition in Section 1.3?
  2. What matched intervention would distinguish causal use of predictive features from task-correlated visual cues?
  3. Which task constraints are lost when a world is reduced to selected points, predicates or latent features?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Reproducing a particular system requires selecting a cited implementation and establishing data, action encoding, prediction horizon, loss weights and control timing that this tutorial does not specify. The described prerequisites differ: labeled robot trajectories ground actions, whereas physical-state pipelines additionally require scene reconstruction and simulator parameter alignment. e-imaginee-jointe-physicale-scope

Reader analysis

Proposed conceptual check: classify Figure 8's four branches twice, using strict inference-time prediction and broader training-time prediction criteria. Record which modules survive deployment. Use explicit and joint generation as positive controls; the auxiliary branch should expose the definitional disagreement. e-rolee-auxiliarye-taxonomy-figure

Reader analysis

Proposed empirical check: match policy backbone, robot data and training budget while comparing action-only learning, auxiliary future supervision and shuffled-future supervision. Remove auxiliary branches at evaluation and measure held-out task success and latency. Better video prediction without better execution would challenge the proposed control benefit. e-auxiliary

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Audit the four branches under two explicit membership rules

Reader-proposed consistency check, not a completed experiment: construct a row for each Figure 8 paradigm and record its training objective, future representation, action interface and modules retained at inference. Apply two rules separately: prediction coupled during inference, and prediction used during either training or inference. Use imagine-then-execute and joint generation as controls with explicit future outputs. Record feature conditioning as a latent interface, and evaluate auxiliary prediction with its future branch removed. The falsifiable outcome is whether the same four memberships can be justified under both rules without adding inference components absent from the source. A disagreement localized to auxiliary prediction would expose the tutorial's definition boundary, rather than justify reclassifying the entire resource. e-rolee-imaginee-featuree-jointe-auxiliarye-taxonomy-figure

Check 2: Test whether auxiliary future learning improves executed control

Reader-proposed experiment, not a reported result: use one robot dataset and policy backbone to compare action-only training, action plus true future-prediction supervision, and action plus a shuffled-future control. Shuffle future targets within instruction groups to preserve coarse task information while breaking correspondence to the current scene. Keep action labels, data splits, action-head capacity, update count and hyperparameter-selection budget matched; record any extra training compute. Sweep the auxiliary weight on validation data only. At deployment remove prediction branches and evaluate identical held-out tasks and scene changes over multiple seeds. Report task success with uncertainty, action latency and prediction error separately. A video-error improvement without an execution improvement would fail to support the claimed representation benefit for this setting. e-auxiliarye-scope

8.3 Reading coverage

Visual audit: Visually inspected the title, authors, affiliation footnote and v7 stamp on page 1; Figures 1–2 on page 1, Figures 3–5 on page 2, Figure 6 on page 4, Figure 7 on page 5 and Figure 8 with its legend on page 7. Pages 3 and 6 supplied representation, training and equation details; page 8 supplied the closing comparison and transition to references. All four final original-PDF crops were separately viewed and retain needed labels and legends while excluding captions and body prose. The incomplete feature equation is visible on page 6. Reference-only pages 9–10 were read in full as text, without visual inspection. No experiment-table or ablation pages exist in this tutorial; separate supplements remain unverified.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • PDF p. 1: title, authors, affiliations, revision stamp and Abstract
  • PDF pp. 1–2: Section 1, Foundations: World, Task, Policy, and Models; Sections 1.1–1.2
  • PDF pp. 2–3: Section 1.3, World Models and World Action Models
  • PDF pp. 3–4: Section 2 and Section 2.1, Observation-space World Models
  • PDF pp. 4–6: Section 2.2, State-space World Models
  • PDF pp. 6–8: Section 3, World Action Models, including all four paradigms and closing discussion
  • PDF pp. 8–10: References

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.
  • The main-PDF image limitation was resolved by visually inspecting pages 1–8, all eight figures, and the four final crops. Reference-only pages 9–10 were read as text.
  • The inspected title page identifies arXiv:2607.00836v7 [cs.RO], 26 August 2026. Title and all three authors match the supplied catalog; the catalog date matches this revision stamp. Earlier revisions and their differences were not supplied or compared; this report describes v7 only.
  • The PDF contains no appendix, empirical results tables, ablation experiments, or reproducible implementation recipe. The manifest's apparent headings beginning 'M. Particleformer' and 'A. Unified world models' are reference continuations, not additional sections.
  • The printed feature-conditioned display on page 6 omits the variable before its second sampling relation. The complete first relation is transcribed; the missing variable is not silently supplied.
  • Referenced papers, linked code and project pages were not inspected. No experiments were run.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title block, affiliation footnote and arXiv margin stampInspect

Title is From World Models to World Action Models: A Concise Tutorial for Robotics. Authors are Xiaoxiong Zhang, Xiong Zeng and Wei Zhang. All list Southern University of Science and Technology; Wei Zhang also lists LimX Dynamics. Stamp: arXiv:2607.00836v7 [cs.RO], 26 Aug 2026.

Go to primary source ↓
e-worldPDF pp. 1–2, Sections 1.1–1.2 and Figures 1–3Inspect

The world comprises task-relevant robot and environment entities. Task objectives and constraints determine relevant states; driving and table cleaning are illustrative examples. The policy receives intent and current observations and outputs actions.

Go to primary source ↓
e-rolePDF pp. 2–3, Section 1.3, Figure 4; PDF p. 2, Section 1.2 and Figure 3Inspect

WM predicts observations or states under candidate actions. WAM is defined through inference-time coupling of action generation and observation prediction, without requiring a separate WM. The taxonomy emphasizes video-based robotic policies and distinguishes functional role from implementation style.

Go to primary source ↓
e-predictorPDF p. 3, Section 2 opening display and Section 2.1; PDF p. 4, Section 2.2 opening displayInspect

The one-step target y follows a parameterized conditional prediction distribution given o_t and a_t. Observation-space models target predicted observations; state-space models target x. Trajectories may replace one-step targets.

Go to primary source ↓
e-observationPDF pp. 3–4, Section 2.1, representation paragraphs and Figure 6Inspect

Observation choices range from RGB to multi-view RGB, RGB-D and point clouds. Action choices range from robot commands to interface controls, latent actions and language. Latent actions use reconstruction with an information bottleneck on unlabeled video. The map separately labels DreamDojo pretraining and finetuning.

Go to primary source ↓
e-statePDF pp. 4–5, Section 2.2, Figure 7 and latent-state, point-track and neural-symbolic paragraphsInspect

State choices include latent representations learned in-domain or extracted with pretrained encoders, point tracks, grounded Boolean predicates and physical variables. Point selection adds structural assumptions; predicates represent high-level skills through preconditions and effects.

Go to primary source ↓
e-physicalPDF pp. 5–6, Section 2.2, Physical state model paragraphInspect

Physical models represent poses, velocities, contacts, masses and friction. A typical pipeline reconstructs a scene, aligns dynamics and rendering against real interaction trajectories, then predicts with physics transition equations.

Go to primary source ↓
e-imaginePDF p. 6, Section 3, Imagine-then-execute paragraph and its two displaysInspect

A video predictor generates visual subgoals from observation and instruction; a separate inverse-dynamics or goal-conditioned model generates actions. Separate visual and labeled robot data are possible. Intermediate pose or flow estimates can ground actions; subgoal errors may propagate.

Go to primary source ↓
e-featurePDF p. 6, Section 3, Video-feature-conditioned action prediction paragraphInspect

Intermediate video-backbone features condition an action module without full future-video decoding. The source defines u_theta, calligraphic H and f_t. Reduced sampling cost is a qualitative benefit; interpreting whether features contain useful futures remains difficult.

Go to primary source ↓
e-equation-gapPDF p. 6, right column, displayed equation under Video-feature-conditioned action predictionInspect

The rendered display contains a complete f_t = calligraphic H(u_theta(o_t,l)) relation followed by a sampling relation with q_phi but no printed left-hand variable.

Go to primary source ↓
e-jointPDF pp. 6–7, Section 3, Joint video-action modeling display and paragraphInspect

One generative model represents observations and action sequences jointly. A pretrained video backbone can be modified and adapted with labeled robot data. Shared representation may improve consistency, while data scarcity and differing objectives remain challenges.

Go to primary source ↓
e-auxiliaryPDF p. 7, Section 3, Auxiliary video prediction for policy learning paragraphInspect

Future prediction supplies additional training supervision to the policy representation. The branch can be removed at inference, leaving the action head. Accurate video prediction need not improve control, and balancing the losses is nontrivial.

Go to primary source ↓
e-taxonomy-figurePDF p. 7, Figure 8, all four branches and legendInspect

Figure 8 depicts explicit video followed by inverse dynamics, extracted features followed by an action module, joint video/action outputs, and auxiliary prediction. A dotted arrow marks feature extraction; a dotted outline marks training-only use. Inputs are observation o_t and language instruction l.

Go to primary source ↓
e-closingPDF pp. 7–8, Section 3, concluding paragraphsInspect

The closing comparison weighs modularity and interpretability against grounding errors, feature opacity, joint optimization and implicit representation transfer. It identifies future explicitness and coupling tightness as central design tradeoffs.

Go to primary source ↓
e-scopePDF p. 1, Abstract; PDF pp. 1–8, Sections 1–3 and Figures 1–8; PDF pp. 8–10, ReferencesInspect

The work calls itself a concise tutorial rather than an exhaustive survey. Its body consists of definitions, prediction-space design maps and four policy paradigms, then ends with references. It provides no original experimental protocol, quantitative results table, ablation, appendix or model-specific training recipe.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.