Hierarchical Latent Action Model
1. Paper overview
In one sentence: HiLAM learns variable-duration skills from pretrained motion latents to improve hierarchical policy adaptation, while retaining dependence on a pretrained extractor and action-labeled fine-tuning. e-formulatione-boundariese-policye-finetunee-efficiencye-qualitativee-frame-diagnostic
| At a glance | What to know |
|---|---|
| Research problem | Source description Frame-pair latent action models capture short motions but may miss temporally extended behavior. Fixed-duration skill windows can assign different representations to the same behavior performed at different speeds. HiLAM asks whether motion-derived latent sequences can supply adaptive skill boundaries and useful control supervision without action labels during pretraining. Task instructions still condition the downstream policy; actionless skill discovery does not imply a language-free control system. e-probleme-policy |
| Core mechanism | Source description A two-stage H-Net compresses pretrained latent actions into variable-length skill segments, avoiding a predefined skill inventory. Training predicts subsequent latent actions and checks their compatibility with future images. e-boundariese-objectivee-implementation |
| A key reported result | LIBERO-Long with 10% fine-tuning demonstrations: HiLAM: 45% Task success rate (%). Simulated LIBERO-Long; default human-video pretraining, followed by 10% of provided expert demonstrations and official evaluation rollouts. The suite contains ten tasks with fifty demonstrations each before subsampling. BAKU: 23% under the same reported demonstration fraction. A 22-percentage-point absolute gain. The source does not report seed counts, evaluation rollout counts or uncertainty. e-implementatione-evaluatione-efficiency |
| Reading caution | Author claim The authors identify simulated evaluation as a limitation and call for real-world validation. They also leave end-to-end learning of the pretrained motion extractor and hierarchical model to future work. Language-assisted skill discovery remains prospective even though policy learning already uses instructions. e-limitationse-policy |
Core contributions
- Source description
A two-stage H-Net compresses pretrained latent actions into variable-length skill segments, avoiding a predefined skill inventory. Training predicts subsequent latent actions and checks their compatibility with future images. e-boundariese-objectivee-implementation
- Source description
Hierarchical policy pretraining separates persistent skill targets from low-level latent-action targets, followed by action-labeled adaptation of only the low-level policy. e-unfolde-policye-finetune
- Reader analysis
The LIBERO gains and coherent example segments support useful temporal representations, but the experiments do not isolate adaptive chunking against a matched fixed-window alternative. e-efficiencye-ablatione-qualitative
Figure 2. Turn sparse skill representations into per-timestep supervision, then learn two linked policies. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) from its bottom row upward: latent actions z^l become the shorter stage sequence z^s, then each stage token is repeated across its segment to form z^h. Equation (5) supplies the segment-index lookup behind this repetition. In panel (b), the instruction l and observation o_t feed the high-level policy, whose predicted skill follows the downward arrow into the low-level policy. The observation also feeds that low-level policy directly. Dotted comparisons link predictions to extracted targets; they are supervision links, not instructions to substitute the target skill for the predicted conditioning input. e-unfolde-policye-formulatione-finetune
What it supports. Repeated skill labels provide a temporally stable target while low-level action labels retain finer variation. After pretraining, the high-level policy is frozen and the low-level output is adapted to real actions. The plotted latent-action output therefore describes pretraining, rather than the final action interface.
Where the evidence stops. Repeated training targets do not establish a fixed execution horizon. The formulation predicts a skill at each decision step; it does not specify a separate skill-termination controller or give detailed timing for asynchronous high- and low-level updates.
2. Motivation
2.1 The problem and the proposed response
Frame-pair latent action models capture short motions but may miss temporally extended behavior. Fixed-duration skill windows can assign different representations to the same behavior performed at different speeds. HiLAM asks whether motion-derived latent sequences can supply adaptive skill boundaries and useful control supervision without action labels during pretraining. Task instructions still condition the downstream policy; actionless skill discovery does not imply a language-free control system. e-probleme-policy
2.2 What this reading follows
A video can show a robot approach a bowl, grasp it and set it down, yet provide no action labels or skill boundaries. HiLAM first represents local changes with an existing latent action model, then learns a temporal hierarchy over those motion tokens. The resulting skills become supervision for a high-level policy, while a low-level policy learns finer actions and is adapted with robot demonstrations. Read the policy diagram alongside the training walkthrough: representation learning and action execution use different components. The strongest quantitative evidence concerns simulated LIBERO control; segmentation examples and generated frames provide useful but qualitative checks on what the representations contain. e-formulatione-boundariese-policye-finetunee-efficiencye-qualitativee-frame-diagnostic
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work / latent-action-pretraining classification is supported. An IDM extracts motion latents; H-Net predicts latent sequences; separate hierarchical policies learn control. The FDM provides auxiliary reconstruction and qualitative diagnostics, with no described inference-time world-model planning. Thus this source does not establish a unified One Model that jointly emits future observations and executable actions; retaining the catalog’s not-applicable architecture, paradigm and quadrant is appropriate. e-architecturee-policye-finetunee-frame-diagnostic
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Compress changes in motion into candidate skills
Begin with a video rather than a robot action trajectory. The pretrained IDM converts each pair of frames separated by k into a latent motion token. HiLAM then learns over that token sequence. At each stage, its encoder produces features, and consecutive normalized query/key features determine a dissimilarity score. Equation (1) starts a new segment when that score reaches 0.5, with the first token forced to start one. Equation (2) selects features at those segment starts for the shorter sequence. The main network processes the compressed representation before decoding restores a finer sequence. Stacking the mechanism yields higher-level skills. The key distinction is between variable-duration segmentation and a predefined list of semantic skills: the model learns boundaries from features without receiving names such as grasp or place. e-formulatione-boundariese-objective
5.2 2. Require predicted motion latents to remain decodable
Temporal compression alone does not ensure that the model’s latent predictions retain the motion information supplied by the IDM. HiLAM therefore predicts the next low-level latent and compares it directly with its target through an element-wise L1 loss. A second route sends the predicted latent through the pretrained FDM with the appropriate current frame, then compares the predicted future image with the observed future. For the next latent at t+1, that pair of images is indexed t+1 and t+k+1. Figure 1 uses generic t for its image branch; the next-token example shifts these indices together. A ratio regularizer also discourages degenerate boundary patterns. Together these objectives train a hierarchy whose output remains compatible with the original motion representation. Figure 5 visualizes that compatibility, while leaving the individual contribution of reconstruction supervision unmeasured. e-objectivee-architecturee-frame-diagnostice-ablation
5.3 3. Transfer the hierarchy into control and test what changed
The learned hierarchy first supplies targets, rather than directly executing a discovered video segment. Equation (5) repeats each skill representation over its segment, allowing an observation-conditioned high-level policy to learn a target at every timestep. The low-level policy receives the predicted skill and learns the corresponding latent action. Fine-tuning then freezes the high-level policy and adapts the low-level policy to real action labels. Reader interpretation: this creates a useful division between persistent behavioral context and rapidly varying control. Table 1 supports that interpretation most clearly for human pretraining, where stage-two skills with stage-zero actions reach 0.94 versus 0.91 for the flat human-pretrained baseline. However, neither that comparison nor the low-data result isolates dynamic boundaries from all other effects of pretraining and hierarchy. e-unfolde-policye-finetunee-ablatione-efficiency
5.4 Training and inference
During training
HiLAM combines element-wise L1 next-latent prediction, weighted frame reconstruction through a pretrained FDM, and a weighted H-Net ratio regularizer that discourages degenerate boundaries and controls average chunk length. For a predicted latent at t+1, reconstruction uses I_{t+1} to predict I_{t+k+1}; the reconstruction norm and loss weights are not specified. e-objective
The dataset description names Something-Something V2, DROID and BridgeV2 for representation learning. Policy pretraining specifically compares Something-Something V2 human videos with BridgeV2 robot videos, discarding available action annotations. Both policies use BAKU architectures with a T5 language encoder. The high-level policy matches extracted skills; the low-level policy conditions on the predicted skill and matches extracted latent actions. e-policye-implementation
Fine-tuning freezes the high-level policy and trains the low-level policy with expert actions. The implementation states 100k gradient steps for both policy pretraining and fine-tuning unless otherwise noted; it does not separately specify the complete HiLAM representation-training schedule. e-finetunee-implementation
During inference
At each decision step, the high-level policy predicts a skill from the current observation and instruction. The adapted low-level policy receives the observation and predicted skill and outputs a real action. The described execution path contains no FDM rollout, candidate-action search or video-based planning loop. e-formulatione-policye-finetune
For the separate frame-prediction diagnostic, a latent predicted from preceding latent history is supplied to the pretrained FDM with the current image. This tests retained motion information and is not itself executed robot control. e-frame-diagnostic
5.5 Implementation flow
- Extract local motion
UniSkill’s pretrained inverse dynamics model maps frames I_t and I_{t+k} to a low-level latent action z_t^l. A T-frame video initially supplies T−k such actions; these form the stage-zero sequence. Figure 1 marks the pretrained IDM and FDM with snowflakes. e-formulatione-architecturee-implementation
- Select segment starts
At stage s, an encoder maps each token to a feature h_t^s. A dissimilarity score from consecutive normalized query/key features triggers a segment-start indicator at the threshold in Equation (1). Chunking selects encoder features at these boundaries, rather than explicitly averaging each segment. e-boundaries
- Predict through the hierarchy
A main network processes the shorter sequence, and decoding with the same boundary pattern restores the finer resolution. Stacked stages produce higher-level skills while the complete stack predicts the next stage-zero latent action. The implementation uses stage-two representations as skills. e-boundariese-objectivee-implementation
- Unfold skills into supervision
The cumulative sum of unfolded segment-start indicators assigns each timestep a segment ID. Repeating that segment’s representation provides aligned skill targets. This repetition constructs training labels; it does not specify a policy execution duration. e-unfolde-policy
6. Experiments & results
HiLAM turns observation-only videos into variable-duration latent skills, then uses those skills to pretrain a hierarchical robot policy. Its central evidence is improved simulated LIBERO control, especially with limited demonstrations; qualitative segmentation and frame predictions provide narrower support for the learned representation (e-boundaries, e-policy, e-efficiency, e-qualitative, e-frame-diagnostic).
The four selected visuals cover hierarchical policy learning, benchmark results, the quantitative ablation and future-frame prediction. Figure 1 has unreadable processing-block/panel labels and Figure 4 has unreadable semantic legend text in the supplied PDF rendering, including higher-DPI inspection; neither is used as a standalone crop. Their supported architecture and segmentation findings remain in the report, grounded in readable equations, captions and Section 4.2.3. The source supplies no aggregate boundary-accuracy metric, quantitative frame-prediction metric or learned-boundary versus fixed-window ablation. e-architecturee-boundariese-qualitativee-frame-diagnostice-ablation
6.1 Read the original evidence
Figure 3. Video-pretrained HiLAM improves LIBERO-Long success with limited demonstrations. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) compares BAKU in gray with HiLAM in blue across Spatial, Object, Goal and Long. Notice the broken vertical axis: the upper portion starts near 0.7, so bar-length ratios cannot be read as success-rate ratios. Panel (b) uses the full zero-to-one success scale and varies the fraction of expert demonstrations used for fine-tuning. The accompanying paragraph on page 7 supplies exact anchors: 23% versus 45% success at 10% demonstrations, and HiLAM at 84% with half the demonstrations and 94% with all demonstrations. Those anchors are more precise than estimating unlabelled points. e-evaluatione-efficiencye-ablatione-implementatione-limitations
What it supports. At the 10% demonstration setting, HiLAM improves success by 22 percentage points. Its half-data result is described as comparable to full-data BAKU. This supports an advantage in supervised adaptation efficiency after video pretraining, under the reported simulated benchmark protocol; it does not measure total training cost.
Where the evidence stops. The figure reports no error bars or seed counts. Table 1 contains separately pretrained BAKU variants, so their success rates should not be silently substituted for this plot’s baseline. Real-world execution is not evaluated here.
Figure 5. Predicted latent actions remain usable by the pretrained image dynamics model in selected examples. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row left to right: current observation, ground-truth future observation, then the FDM’s predicted future across the dashed separator. The conditioning latent is itself predicted from earlier latent history, rather than extracted from the displayed target future pair. Compare the gripper or manipulated object between the current and future columns, not just the largely unchanged background. Section 4.2.4 uses these examples to argue that next-latent prediction retains motion information. Section 3.2 explains the supporting training objective: future-image reconstruction supplements direct latent prediction, so the image diagnostic probes a property the training explicitly encourages. e-frame-diagnostice-objectivee-evaluation
What it supports. These rows show that some predicted latents can be decoded into visually consistent future frames using the pretrained FDM. That is useful evidence about the latent interface. It does not establish quantitative dynamics accuracy, long rollout stability, or the success of actions executed by the hierarchical policy.
Where the evidence stops. No reconstruction metric, sample-selection protocol or controlled comparison without the image loss is reported. The displayed robot scenes are qualitative visual predictions; the downstream control results are the separate simulated LIBERO evaluations.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO-Long with 10% fine-tuning demonstrations Simulated LIBERO-Long; default human-video pretraining, followed by 10% of provided expert demonstrations and official evaluation rollouts. The suite contains ten tasks with fifty demonstrations each before subsampling. | HiLAM: 45% Task success rate (%) | BAKU: 23% under the same reported demonstration fraction. A 22-percentage-point absolute gain. The source does not report seed counts, evaluation rollout counts or uncertainty. e-implementatione-evaluatione-efficiency |
| LIBERO-Long demonstration scaling Same reported benchmark protocol, varying the fraction of expert demonstrations used for fine-tuning. | HiLAM: 84% at 50% demonstrations; 94% at 100%. Task success rate (%) | The authors describe 84% as comparable to BAKU using 100% demonstrations; its exact full-data value is not tabulated for Figure 3. Supports data efficiency in this setup. Do not replace the Figure 3 baseline with a differently pretrained BAKU row from Table 1. e-evaluatione-efficiencye-ablation |
| LIBERO-Long pretraining and hierarchy ablation Table 1; default fine-tuning protocol, varying policy pretraining data and unfolded skill/action stages. | Human-pretrained HiLAM with stage-2 skills/stage-0 actions: 0.94; robot counterpart: 0.90. Task success rate (fraction) | Human-pretrained flat BAKU with stage-0 actions: 0.91; HiLAM without pretraining: 0.67. Human HiLAM alternatives stage-1/stage-0 and stage-2/stage-1 both score 0.89. Robot stage-1/stage-0 ties stage-2/stage-0 at 0.90. The best human setting supports separating coarse skills from fine actions. The robot tie qualifies the prose claim that deeper skills are best; these comparisons do not isolate adaptive segmentation itself. e-implementatione-ablation |
| Skill boundary discovery in an example robot trajectory Figure 4 and Section 4.2.3 show one segmented robot demonstration; this is a qualitative diagnostic, not a control evaluation. | Section 4.2.3 identifies segment 2 as approaching the bowl, 3 as picking it up, and 4 as moving to the target and placing it down. The timeline associates these with sampled times 2/7/12, 14/17, and 21/24/30, respectively. Qualitative correspondence between segments and behavior; no boundary-accuracy metric | No annotated-boundary comparison, fixed-window control or aggregate consistency measure is reported. The example supports plausible variable-duration segmentation. Figure 4 displays IDs starting at 0, whereas Equations (1) and (5) imply an initial ID of 1; the display offset is unexplained. Semantic meanings are verified from the prose because the graphical legend text is unreadable. e-qualitativee-boundariese-unfold |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Coarse skills and fine actions work best in the human-pretrained hierarchy. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the flat BAKU rows from the hierarchical HiLAM rows. Within each group, compare the pretraining dataset before comparing latent stages. A bar over z denotes a representation unfolded to the original temporal resolution; superscripts identify stages rather than powers. The strongest human HiLAM row uses stage-two skills and stage-zero actions. Its two human alternatives each score 0.89. The upper BAKU block is also pretrained, so the 0.91 human/stage-zero row is a more targeted hierarchy comparison than the unpretrained HiLAM row at 0.67. Read the Robot rows separately: stage-one and stage-two skills tie. e-ablatione-implementatione-unfold
What it supports. Human-pretrained HiLAM reaches 0.94 against 0.91 for the human-pretrained flat stage-zero baseline. For robot pretraining, stage-one/stage-zero and stage-two/stage-zero both reach 0.90. Thus the prose preference for deeper skills has a clear human-data example but no strict stage-two improvement in that robot comparison.
Where the evidence stops. Changing human versus robot data changes the source distribution, and the table does not establish matched dataset sizes. No uncertainty is given. The ablations vary pretraining and conditioning stages, without directly comparing learned boundaries against fixed chunks.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify simulated evaluation as a limitation and call for real-world validation. They also leave end-to-end learning of the pretrained motion extractor and hierarchical model to future work. Language-assisted skill discovery remains prospective even though policy learning already uses instructions. e-limitationse-policy
Figures 4–5 show selected qualitative examples, without segmentation accuracy or quantitative frame-prediction metrics. They support plausibility, not reliable semantic boundaries across a dataset. Neither Table 1 nor Figure 3 reports uncertainty or a controlled dynamic-versus-fixed chunking comparison. e-qualitativee-frame-diagnostice-ablatione-efficiency
7.2 Questions for discussion
- Would matched fixed-length chunks preserve the LIBERO gain when pretraining data and policy capacity are held constant?
- How stable are the discovered boundaries when the same physical behavior is performed at different speeds?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include UniSkill IDM/FDM, a two-stage H-Net, BAKU-based high/low policies, T5, the named video datasets and LIBERO demonstrations. Human and robot policy pretraining should be kept separate when comparing Table 1 rows. e-implementatione-ablation
Unspecified details include frame interval and sampling, mapping T−k action latents onto T-frame skill labels, model sizes, exact dataset mixtures, loss weights, ratio settings, optimizer, learning rate, batch size, hardware, software versions and seeds. The supplied paper has no implementation appendix. These choices must be documented before a claimed reproduction. e-formulatione-objectivee-unfolde-implementatione-evaluation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Proposed check: learned boundaries versus matched fixed windows
Keep the UniSkill extractor, video subset, hierarchy depth, policy capacity, training budget and LIBERO fine-tuning subset fixed. Compare learned boundaries with fixed windows matched to the learned model’s mean stage-wise chunk lengths. On held-out videos, repeat the comparison after resampling the same demonstrations at different speeds, mapping predicted boundaries back to original time. Measure boundary consistency and LIBERO-Long success with 10% demonstrations across multiple seeds. A repeatable control gain plus greater speed stability would support the proposed adaptive-segmentation mechanism; parity would weaken that explanation even if both variants outperform a flat baseline. This is a proposed experiment, not a reproduced result. e-probleme-boundariese-implementatione-efficiencye-ablatione-qualitative
Check 2: Proposed check: does image reconstruction improve transferable latents?
Train otherwise matched HiLAM variants with the reconstruction weight enabled and set to zero, keeping the latent objective, ratio regularization and pretrained IDM/FDM fixed. Document the chosen weights and optimizer because the paper does not provide them. Evaluate held-out next-latent L1 error, frame reconstruction through the same FDM, and LIBERO-Long success after identical policy training. Include true IDM latents as a reference and shuffled predicted latents as a negative control for the frame test. If image supervision improves only visual appearance without improving action prediction or control, its proposed transfer benefit remains unsupported. These comparisons are reader-proposed and have not been run. e-objectivee-implementatione-policye-finetunee-frame-diagnostice-evaluation
8.3 Reading coverage
Visual audit: All eleven PDF pages were rendered and visually inspected: title, version and affiliations (p. 1); motivation and related work (pp. 1–3); all method equations and training details (pp. 3–6); evaluation and ablations (pp. 6–7); qualitative diagnostics (pp. 8–9); limitations (p. 9); and references (pp. 9–11). Figures 1–5 and Table 1 were inspected in page context. The four final crops are Figure 2, Figure 3, Table 1 and Figure 5, and each final image was individually inspected. These retain the visible policy labels and arrow directions, plot axes and legend, every table row/header, and all frame-prediction columns. Figures 1 and 4 were additionally inspected at higher DPI; their unreadable labels remain disclosed in coverage and visualLimitations. The report preserves the Figure 4 result and unexplained segment-index offset, and the walkthrough distinguishes Figure 1’s generic image indices from the shifted next-token training example. All supporting pages remain included for review. No appendix is present; separate supplements, code and external references remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work, including 2.1–2.2 (pp. 2–3)
- 3 Method, including 3.1.1–3.1.2, 3.2 and 3.3; Equations (1)–(5) (pp. 3–6)
- 4 Experiments, including 4.1 and 4.2.1–4.2.4 (pp. 6–9)
- 5 Conclusion and Limitations (p. 9)
- References (pp. 9–11)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: title and all three authors match the catalog. The inspected artifact is arXiv:2603.05815v1 [cs.RO], dated 6 March 2026, and says Published as a workshop paper at ICLR 2026 (e-identity). The specific World Models workshop designation and correspondence to the catalog OpenReview submission were not independently verified; no other edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting all eleven PDF pages and all five figures plus Table 1.
- Separate supplemental material availability has not been fully verified. No appendix is present in the supplied eleven-page PDF.
- Some processing-block and panel labels in Figure 1 and semantic legend text in Figure 4 remain unreadable in the supplied PDF rendering, including higher-DPI inspection. Both figures were inspected in page context, but were not selected as standalone visuals. Their supported mechanism and qualitative findings are retained using readable captions, equations and accompanying prose; no labels were reconstructed.
- Code and external references were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block, workshop header and arXiv margin stamp
Title: Hierarchical Latent Action Model. Authors: Hanjung Kim, Lerrel Pinto, Seon Joo Kim. Hanjung Kim is affiliated with Yonsei University and New York University; Lerrel Pinto with New York University; Seon Joo Kim with Yonsei University. The margin identifies arXiv:2603.05815v1 [cs.RO], 6 March 2026; the header identifies an ICLR 2026 workshop paper.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1
The paper motivates temporally extended skills from actionless videos, noting that short-horizon latent actions and fixed-duration windows can miss behavioral structure or vary with execution speed.
Go to primary source ↓e-formulationPDF p. 3, Section 3.1.1
A T-frame video yields T−k frame-pair latent actions, compressed into fewer high-level skills. The high-level policy conditions on observation and language; the low-level policy conditions on observation and skill to generate executable actions.
Go to primary source ↓e-boundariesPDF p. 3, Section 3.1.2, Equations (1)–(3); PDF p. 4, Section 3.2
Boundary probability is one at the first token and otherwise half of one minus normalized consecutive query/key similarity. A score at least 0.5 starts a segment. Chunking selects boundary encoder features, processes the shorter sequence, and expands through decoding using the same boundaries; stages can be stacked.
Go to primary source ↓e-architecturePDF p. 4, Figure 1 and caption; PDF p. 3, Equations (2)–(3)
Figure 1 shows bottom-to-top sequence compression and expansion, predicted/target latent comparison, image prediction conditioned on a latent, and frame-pair IDM extraction. Snowflakes mark the pretrained IDM and FDM. Some interior block/panel labels are not readable in the rendering; the caption and method text establish their roles.
Go to primary source ↓e-objectivePDF pp. 4–5, Section 3.2, Equation (4) and Training objective paragraphs
The objective sums element-wise L1 next-latent loss, weighted reconstruction loss through a pretrained FDM, and weighted H-Net ratio regularization. At the t+1 indexing used in the text, I_{t+1} and the predicted next latent produce I-hat_{t+k+1}. The reconstruction norm, coefficient values and ratio configuration are not specified here.
Go to primary source ↓e-unfoldPDF p. 5, Figure 2(a), Latent Skill Extraction and Equation (5)
The cumulative sum of unfolded segment-start indicators selects each timestep’s skill representation, repeating stage-level tokens to original video resolution. Equation (5) uses k_t^s as a segment ID.
Go to primary source ↓e-policyPDF p. 5, Figure 2(b) and Section 3.3 Pretraining
The high-level policy uses observation and instruction to predict an extracted skill target. The low-level policy uses observation and the predicted skill to predict an extracted latent-action target. The diagram shows predicted skill flowing to the low-level policy and dotted comparisons with training targets.
Go to primary source ↓e-finetunePDF p. 6, Section 3.3 Fine-tuning continuation
The high-level policy is frozen; target-domain action labels fine-tune the low-level policy to produce real actions conditioned on observation and predicted skill.
Go to primary source ↓e-implementationPDF p. 6, Section 4.1 Datasets and Implementation details; PDF p. 7, Section 4.2.2
The source names Something-Something V2, DROID and BridgeV2; specifies two-stage H-Net, UniSkill IDM/FDM, stage-2 skills, BAKU-based policies and T5. Policy pretraining compares human Something-Something V2 and robot BridgeV2 videos with action annotations discarded. Policy pretraining and fine-tuning use 100k gradient steps unless otherwise stated. Human pretraining is the default selected after ablations.
Go to primary source ↓e-evaluationPDF p. 6, Section 4.1 Benchmark
LIBERO-Spatial, Object, Goal and Long each contain ten tasks with fifty demonstration trajectories per task. Policies fine-tune on expert demonstrations and are evaluated through official rollouts. Rollout counts, seed counts and uncertainty are not supplied.
Go to primary source ↓e-efficiencyPDF p. 6, Figure 3(a–b); PDF p. 7, Section 4.2.1
Figure 3 shows higher HiLAM success across all four suites and a broken vertical axis in panel (a). The text reports 10%-data success of 45% for HiLAM and 23% for BAKU, 84% for HiLAM with 50% data, and 94% with full data. It calls the 50%-data HiLAM result comparable to full-data BAKU without tabulating that baseline value.
Go to primary source ↓e-ablationPDF p. 7, Table 1, all rows and Success rate column; Section 4.2.2
Flat BAKU: robot/stage-0 0.87, robot/stage-2 0.81, human/stage-0 0.91, human/stage-2 0.87. HiLAM without pretraining: 0.67. HiLAM robot skill/action stages 1/0, 2/0, 2/1: 0.90, 0.90, 0.87; human counterparts: 0.89, 0.94, 0.89. The robot 1/0 and 2/0 tie qualifies the prose preference for stage-2 skills. No fixed-chunk control or uncertainty is reported.
Go to primary source ↓e-qualitativePDF p. 8, Figure 4, caption and Section 4.2.3; PDF p. 3, Equation (1); PDF p. 5, Equation (5)
The timeline displays segment IDs 0–4. Sampled times 2, 7 and 12 lie in segment 2; 14 and 17 in segment 3; 21, 24 and 30 in segment 4. Section 4.2.3 identifies segment 2 as moving toward the bowl, 3 as picking it up and 4 as moving to the target and placing it down. Legend swatches are visible but their text is unreadable in the rendering; semantic descriptions come from the prose. No boundary-accuracy metric is provided. Figure 4 begins IDs at 0, whereas Equation (5), combined with the first-token boundary in Equation (1), begins at 1; the display-index offset is unexplained.
Go to primary source ↓e-frame-diagnosticPDF p. 8, Figure 5 and Section 4.2.4; PDF p. 9, continuation of Section 4.2.4
Figure 5 places current image I_t, target future I_{t+k}, and predicted future I-hat_{t+k} in three columns. The FDM conditions on the current image and a predicted latent inferred from preceding latent history. The authors interpret the examples as retaining motion information; no numerical image-quality or rollout metric is reported.
Go to primary source ↓e-limitationsPDF p. 9, Section 5 Conclusion and Limitations
The authors identify primarily simulated experiments and the use of a pretrained IDM as limitations, proposing real-world validation and end-to-end training. They discuss adding language to motion-based skill discovery as future work.
Go to primary source ↓8.5 Primary sources
Hierarchical Latent Action Model ↗
PDF · 5,453 extracted words
Source fingerprint
3c53d71e46cf7b0f4670847805ee887f79c94526ba36efbd3349326cad203b3a