Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
1. Paper overview
In one sentence: Human corrections make real-world RL exploration useful, while reward-based updates turn those experiences into fast, reliable task-specific manipulation. e02e03e04e07e09e10e19
| At a glance | What to know |
|---|---|
| Research problem | Source description Can image-based RL learn precise contact, coordinated bimanual motion and dynamic manipulation directly on real robots within practical training time? Exploration, sparse success feedback, optimization stability and safe contact all constrain this goal. The paper addresses their interaction through a complete training system rather than proposing a new dynamics model. e01e02e03e07 |
| Core mechanism | |
| A key reported result | RAM Insertion: 100%; 4.8 s Success rate; cycle time. Physical pre-grasped RAM; 100 evaluation trials with randomized resets; 1.5 hours of online training. Success excludes engaging the locking mechanism. Table 1(a): HG-DAgger-based BC 29%; 8.3 s. A 71-percentage-point observed success gain under this insertion protocol; no confidence interval or repeated-training variability is supplied. e08e09e10 |
| Reading caution | Source description Each skill still requires task-specific training. The authors did not test extensive randomization or generalization in unstructured environments, and scaling to substantially longer horizons remains open. e19 |
Core contributions
- Author claim
The authors argue that reward-driven exploration produces reliable recovery behaviors and faster execution than imitation. Ablations support the importance of corrections for the tested contact tasks; diagnostic plots offer an interpretation of reliability rather than a convergence proof. e09e13e16e17
Figure 2. Human control changes the executed action and the distribution of replayed experience. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the left-hand actor loop. The chosen action is either the policy output or a human replacement; it reaches the robot controller before a new observation returns. Follow orange intervention transitions into both replay buffers and blue policy transitions into the RL buffer alone. The learner draws equal portions from the buffers, updates the continuous actor and critic with RLPD, and returns policy parameters along the left-pointing arrow. The optional grasp critic uses a separate DQN update. The upper-right inset explains the inputs: camera features and encoded proprioception feed an MLP, rather than a future-image generator. e02e03e04e05e07
What it supports. The same correction has an immediate control effect and a continuing training effect. It can rescue a rollout, then be sampled again through either buffer. This data routing is consistent with Section 3.4 and explains how human guidance enters reward-based learning without imposing a behavior-cloning loss on the actor.
Where the evidence stops. The 10 Hz label describes policy/environment updates. Appendix D specifies a faster 1,000 Hz robot controller. The figure does not establish frozen visual weights or prove that either buffer-routing choice is independently necessary.
2. Motivation
2.1 The problem and the proposed response
Can image-based RL learn precise contact, coordinated bimanual motion and dynamic manipulation directly on real robots within practical training time? Exploration, sparse success feedback, optimization stability and safe contact all constrain this goal. The paper addresses their interaction through a complete training system rather than proposing a new dynamics model. e01e02e03e07
2.2 What this reading follows
A robot inserting a RAM card must recover from small alignment errors without damaging the connector. HIL-SERL makes this a practical reinforcement-learning problem by combining visual features, compliant control, demonstrations and occasional human takeovers. Corrections become transitions that a value-learning algorithm can reuse, helping the policy discover successful behavior beyond the recorded motions. The paper tests this approach on contact, bimanual and dynamic tasks, reporting high observed success and shorter cycle times. Read the results together with their intervention budgets, task-specific resets and exceptions: the evidence concerns trained physical skills in controlled setups, with broader generalization still open. e02e03e04e07e09e10e19
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The architecture supports the recorded outside-quadrant placement: model-free actor–critic learning predicts actions and returns, without a learned future-state generator, joint future/action head or inverse-dynamics action extraction. The term predictive policy describes learned behavior. Policy foundations and training optimization fit; distillation is proposed future use, not an evaluated component. e02e03e05e18e19
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a corrective action can teach more than a demonstrated motion
Consider the RAM policy reaching a poor alignment that its initial demonstrations rarely visited. A human can briefly replace the command and move the robot toward a recoverable state. Section 3.4 puts that executed transition in both buffers, while surrounding autonomous transitions remain in the RL buffer. Equal buffer sampling makes correction experience repeatedly available to the learner. The critic then fits discounted returns, and the actor favors actions with higher learned value. My interpretation is that demonstrations and corrections help establish useful routes through the state space, while trial-and-error learning improves behavior around those routes. Figure 7's RAM visitation funnel is consistent with that account, but it is descriptive evidence. It does not prove that corrections alone cause reliability or that the same routes transfer to a different workcell. e02e04e10e17
Figure 3. Success labeling, demonstration collection and corrective training are distinct stages. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two preparation blocks on the left before the online loop. Success and failure images train a binary reward classifier; demonstrations initialize the demo buffer. In the central block, the policy acts while teleoperation can override it, and the classifier labels the resulting experience. On the right, training time moves upward, so the lowest episode bars represent earlier learning. Blue denotes policy transitions, orange human control, and green or red denotes successful or failed outcomes. The shrinking orange portions sketch the intended reduction in intervention. These bars illustrate the procedure and should not be read as measured episode counts or durations. e04e06e20e23e15
What it supports. Human input defines both what counts as completion and how exploration can recover from difficulty. The training strategy seeks a policy that eventually succeeds with little intervention, while preserving human transitions for further updates. Figure 5 provides the measured learning curves that this schematic only anticipates.
Where the evidence stops. This is the general workflow. Jenga uses human reward annotation at episode end and no real-time corrections. Section 3.5 also warns that persistently using long, sparse interventions to achieve success can destabilize value learning.
5.2 The policy's action is only the first part of physical control
The visual policy does not directly prescribe every motor torque. Camera embeddings are combined with task-dependent proprioception, and the policy produces either a Cartesian twist or selected wrench components. For contact tasks, Appendix D expresses proprioceptive pose relative to the episode's initial end-effector frame and interprets actions in the current end-effector frame. Coordinate transformations then connect these commands to the robot interface. A 1,000 Hz impedance controller follows the policy's 10 Hz setpoints and limits reference error to constrain contact forces. Gripper decisions, when required, come from the separate discrete critic and are concatenated with motion. For dynamic manipulation, a feedforward wrench controller provides the corresponding fast execution layer. These choices explain why reproducing the neural loss alone would leave crucial parts of the manipulation system unspecified. e03e05e07
5.3 Separate reliable task execution from a claim about prediction
The strongest evidence is the physical endpoint evaluation, read with its setup and reset protocol. RAM's result concerns pre-grasped insertion without engaging the lock; whole-IKEA success concerns chained skills with retries. The paper also describes reactive contact recovery and calls the consistent dynamic motions predictive behavior. This terminology should be read alongside the architecture: HIL-SERL contains an action policy and return estimators, without an explicit future-state model. My interpretation is that the behavior analysis explains how one model-free learning framework can acquire different control strategies. Figure 8's action standard deviations characterize policy outputs, not forecast accuracy. Similarly, Figure 7's Q variance measures sensitivity to perturbed actions rather than calibrated uncertainty. Neither diagnostic extends the controlled evaluation into evidence of unrestricted environmental generalization. e02e03e10e12e17e18e19
5.4 Training and inference
During training
First train a task reward classifier from teleoperated success/failure images, typically about 200 positive and 1,000 negative examples. Appendix B specifies ResNet-10 plus a two-layer MLP, cross-entropy, Adam at 0.0003 and 100 iterations. Additional hard negatives address wrong or partial insertions. e06e23
Initialize replay with roughly 20–30 demonstrations, then train online with human or scripted resets and selective corrections. The authors warn that persistent long, sparse interventions ending in success can overestimate values. Grasp tasks also penalize unnecessary gripper actions. Jenga instead uses 30 demonstrations without online corrections and human terminal reward annotation. e04e05e20
During inference
The trained actor maps current observations to motion; the grasp critic, when present, selects discrete actions by argmax. Policies supply 10 Hz setpoints to a 1,000 Hz controller. Physical transitions provide feedback; there is no imagined rollout or explicit learned transition model in this pipeline. e02e05e07
5.5 Implementation flow
- Encode the current observation
A shared ImageNet-pretrained ResNet-10 processes the camera images. Image embeddings are concatenated with encoded proprioception and passed to MLP networks. The source describes pretrained initialization but does not establish whether the policy's visual backbone remains frozen. e03e06
- Replace actions when a human intervenes
The actor normally executes the RL action; a SpaceMouse intervention replaces it. Intervention transitions enter both demo and RL buffers, whereas autonomous transitions enter only the RL buffer. The learner samples equally from the two buffers and asynchronously returns updated parameters. e02e03e04
- Learn motion and grasp decisions
RLPD trains a continuous Gaussian actor and Q-function. A separate DQN-style grasp critic chooses among three discrete actions for one gripper, or nine joint combinations for two. Motion and gripper outputs are concatenated for execution; they are not inverse-dynamics predictions. e02e05
- Execute through the robot controller
Proprioceptive pose is relative to the episode's initial end-effector frame, while actions use the current end-effector frame. The controller maps these to robot commands. Contact tasks use impedance control with reference limiting; dynamic tasks use feedforward wrench control. e03e07
6. Experiments & results
HIL-SERL combines visual model-free reinforcement learning, demonstrations, corrective teleoperation and task-specific robot control. Its central mechanism uses human recoveries as replayable transitions, allowing reward-based learning to improve beyond imitation. The paper reports perfect observed success across its evaluation tasks, usually after 1–2.5 hours of online training, with a six-hour timing-belt exception. These are physical task results under controlled setups, not demonstrations of a predictive world model or generalist deployment.
6.1 Read the original evidence
Table 1(a). Observed reliability improves across the suite, but training costs and evaluation units vary. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use each row as its own experiment. The training-time column belongs to HIL-SERL; the next two columns compare success, and the final two compare cycle time. The BC heading includes HG-DAgger-based imitation for most rows, as the retained footnote explains. Jenga and object flipping instead use flat BC with different demonstration counts. Whole-IKEA entries are fractions from ten complete trials, not percentages from a hundred trials. Read the timing-belt row before generalizing the training-time headline: it took six hours. The percentage increases in parentheses are relative improvements, while the cycle-time factors compare completion durations. e08e09e10e11e12
What it supports. For RAM insertion, HIL-SERL reports 100% success and 4.8 seconds, compared with 29% and 8.3 seconds for the imitation baseline. Timing-belt assembly reaches 100% after six hours. The aggregate row reports 100% versus 49.7% success and 5.4 versus 9.6 seconds, supporting substantial gains within the evaluated setups.
Where the evidence stops. The results have no displayed uncertainty or repeated-training variation. Whole-IKEA uses scripted transitions and allows two attempts per subpolicy. Missing Jenga and whole-IKEA cycle times are not zeros, and perfect observed success is not a deployment guarantee.
Figure 5. Read training success together with the amount of human assistance. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow a single task down its column: success is at the top, intervention fraction in the middle, and completion time at the bottom. Solid pink is HIL-SERL; dashed teal is HG-DAgger. Each curve is a running average over twenty episodes, with elapsed training minutes on the horizontal axis. Intervention fraction means human-controlled timesteps divided by all timesteps in that episode. Dashboard illustrates why the middle row matters: the imitation system can record high assisted success while continuing to need interventions. HIL-SERL's success rises as its intervention curve approaches zero, and its cycle time also falls. Autonomous endpoint evaluation is reported separately in Table 1. e08e09e15
What it supports. The joint trend supports increasing autonomy during HIL-SERL training: success improves while human control shrinks and execution becomes faster. A high training success curve alone would not establish this. Reading all three rows explains why apparently successful assisted rollouts can coexist with a weaker autonomous policy.
Where the evidence stops. These smoothed curves have no uncertainty bands. The caption says HG-DAgger cycle time shows no improvement, but its dashboard trace visibly decreases. The plot supports a more qualified comparison than that blanket caption statement.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RAM Insertion Physical pre-grasped RAM; 100 evaluation trials with randomized resets; 1.5 hours of online training. Success excludes engaging the locking mechanism. | 100%; 4.8 s Success rate; cycle time | Table 1(a): HG-DAgger-based BC 29%; 8.3 s. A 71-percentage-point observed success gain under this insertion protocol; no confidence interval or repeated-training variability is supplied. e08e09e10 |
| Timing Belt Assembly Two arms thread a belt onto two pulleys and tighten the tensioner; 100 evaluation trials; 6 hours training. Appendix: human resets and 2 cm x/y randomization. | 100%; 7.2 s Success rate; cycle time | HG-DAgger-based BC 2%; 9.1 s. Strong performance on flexible-object coordination, with a training cost outside the abstract's 1–2.5-hour headline. e08e09e11 |
| IKEA - Whole Assembly Three trained subpolicies connected by scripted transitions; randomized grasp translations; at most two attempts per subpolicy; 10 complete trials. | 10/10 Successful assemblies | BC baseline 1/10. A composed system result with retries, not a single learned long-horizon policy or a 100-trial estimate. e09e12 |
| Human-correction ablation RAM insertion, dashboard assembly and object flipping; Table 1(b), nominal 100-trial evaluation. No-intervention condition uses 200 demonstrations versus typically 20–30 plus corrections. | Full HIL-SERL: 100%, 100%, 100%. Without interventions: 48%, 0%, 100%. Without demonstrations or interventions: 0%, 0%, 0%. Success rate | DP: 27%, disputed dashboard value, 56%; IBRL: 75%, 0%, 95%; Residual RL: 0%, 0%, 97%; DAPG: 8%, 18%, 72%. Corrections help the tested contact tasks, but are not required for flipping in this comparison. Human-data budgets differ. Dashboard DP is 18% in the table and 28% in prose; it remains unresolved. e08e13e14 |
| Reported aggregate across manipulation tasks Table 1(a)'s task-level summary mixes interactive imitation and flat BC for dynamic tasks; IKEA whole assembly has a separate ten-trial protocol. | 100%; 5.4 s Reported average success rate; reported average cycle time | 49.7%; 9.6 s for the paper's BC summary. The authors report +101% relative success and 1.8× faster execution. These are reported aggregate values, not a pooled estimate over identical protocols; missing cycle times remain missing. e08e09 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1(b). The benefit of corrections depends on the manipulation task. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the three rightmost conditions. Removing both demonstrations and interventions yields zero success on RAM, dashboard and flipping. Supplying demonstrations without interventions raises RAM to 48% and flipping to 100%, while dashboard stays at zero. Full HIL-SERL reaches 100% on all three. Read the remaining methods as contextual comparisons, with the footnote's distinct data budgets kept visible: several baselines use 200 demonstrations, whereas the full system usually starts with 20–30 and receives corrections. Section 4.5 also specifies 200 demonstrations for the no-intervention comparison. Values are success percentages under the paper's nominal hundred-trial evaluation protocol. e08e13e14
What it supports. The dashboard and RAM rows make a strong case for corrective interaction in the tested contact settings. Flipping is the discriminating counterexample: removing interventions does not lower its reported success. The table therefore supports a task-dependent explanation, not a universal requirement for human corrections in every learned skill.
Where the evidence stops. Dashboard Diffusion Policy is 18% here but 28% in Section 4.5; the source conflict remains unresolved. The correction comparison also changes the source and amount of human data, so it does not isolate a matched human-time intervention effect.
7. Analysis & limitations
7.1 What the evidence leaves open
Each skill still requires task-specific training. The authors did not test extensive randomization or generalization in unstructured environments, and scaling to substantially longer horizons remains open. e19
Perfect observed evaluation success does not establish zero deployment risk. Figure 6's perturbation recoveries are qualitative; training curves include assisted episodes and do not replace autonomous evaluation. e08e15e16
The RAM visitation funnel and action-perturbed Q variance illustrate local value sensitivity, not calibrated uncertainty. Low Jenga action standard deviation is compatible with consistent motion but does not demonstrate an explicit predictive model. Figure 8's caption says variance while its axes and Section 5.2 say standard deviation. e17e18
Reporting inconsistencies remain: dashboard DP is 18% versus 28%; object-flipping prose describes randomized placement while Table 12 says no randomization; handover prose names opposite destination baskets. These details should be resolved before copying those protocols. e14e21e12
7.2 Questions for discussion
- Would a matched human-time budget preserve the correction advantage on RAM and dashboard assembly?
- How well does classifier success agree with independently judged physical completion under new disturbances?
- Can the learned skills tolerate broader object and camera variation without retraining?
8. Reproducibility audit
8.1 Requirements and known gaps
A focused RAM setup needs the two wrist views, the defined insertion criterion, safe compliant control and a reward detector. Table 2 gives 20 demos, 32,000 RL transitions, a 100-step horizon, discount 0.97, a 64-unit proprio encoder, a 256 × 256 policy MLP, random crops and Adam at 0.0003. Section 4.3 reports one RTX 4090. e08e10e07
The PDF does not fully specify batch size, update-to-data ratio, target-update settings, controller gains/reference limits, classifier split construction, or policy-backbone freezing. Its tables are useful but insufficient for exact implementation. DP's task-dependent observation/prediction/execution horizons should also be preserved when reproducing that comparison. e02e03e10e23e24e07
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does copying interventions into both buffers improve learning?
Reader-proposed check, not performed: reproduce the RAM setup and compare the published dual-buffer insertion rule with a condition that stores intervention transitions only in the RL buffer. Keep the initial 20 demonstrations, reward classifier, controller, 32,000-transition budget and equal buffer sampling fixed. Use a predefined intervention policy and record actual human-control seconds, since diverging policies can need different amounts of help. Repeat training and evaluate each final policy on 100 unassisted trials, reporting success, cycle time and intervention cost separately. If dual-buffer storage accelerates autonomous recovery at comparable human effort, it supports the proposed replay mechanism; indistinguishable learning would weaken the claim that duplication itself matters. e02e04e08e10
Check 2: Does learned reward agree with physical RAM completion?
Reader-proposed check, not performed: construct a held-out collection of wrong-slot, partial and one-sided insertions like Figure 25, plus valid insertions under the paper's non-locking success definition. Split by rollout rather than neighboring image frames, and have independent reviewers label physical completion. Compare the trained classifier's false-positive and false-negative rates on these cases with its nominal validation accuracy, then independently score 100 autonomous policy trials. Hold the camera crop and control configuration fixed. Frequent classifier-positive physical failures, or a large gap between classifier-based and independently judged policy success, would reveal a reward-validity problem despite high headline accuracy. e06e10e23e08
8.3 Reading coverage
Visual audit: The title/author page, architecture and training diagrams, Table 1 with both protocol footnotes, learning curves, qualitative perturbations, state/Q diagnostics and policy-output plots were visually inspected. Appendix inspection covers the cited task settings, reward examples, DP horizons, coordinate/control definitions and additional training curves. All five final original crops were inspected for complete labels and readability. Figure 2's action replacement and buffer routing agree with Sections 3.2–3.4; Figure 3's upward training-time arrow is preserved. Table 1(b)'s dashboard DP conflict, Figure 5's overgeneralized cycle-time caption and Figure 8's variance/standard-deviation mismatch are disclosed. Appendix text was read completely; uncited setup photographs were not all visually inspected. Videos, code and any separate supplements were outside the supplied visual material.
PDF pages inspected for this edition: 1, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 32, 33, 41, 45, 47, 49, 50, 51, 52, 53, 54. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1. Introduction
- 2. Related Work
- 3. Human-in-the-Loop Reinforcement Learning System, Sections 3.1–3.5
- 4. Experiment Results, Sections 4.1–4.5
- 5. Result Analysis, Sections 5.1–5.2
- 6. Discussion; Acknowledgments; References
- Supplementary A. Task Setup and Policy Training Details, A.1–A.11
- Supplementary B. Reward Classifier Training Details
- Supplementary C. Diffusion Policy Training Details
- Supplementary D. Robot Controller and Proprioceptive Information Representation, D.1–D.2
- Supplementary E. Policy Training Plots
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Reviewed all 12 supplied text chunks spanning all 54 PDF pages. Visual inspection covers the pages declared in the illustrated edition, not every appendix photograph.
- Identity and edition scope: the exact title and four authors match the catalog. The supplied title page identifies arXiv:2410.21845v3, 20 March 2025, while retaining an October 2024 masthead. The catalog cites Science Robotics (2025); the publisher edition was not supplied, and equivalence or changes relative to it and earlier preprints remain unverified.
- The acquisition's text-only figure omission was addressed by inspecting the retained PDF and final crops. Separate supplemental material availability has not been fully verified.
- Project videos, external references and code were not opened. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, arXiv margin stamp, Abstract and Section 1
Exact title; Jianlan Luo, Charles Xu, Jeffrey Wu and Sergey Levine; UC Berkeley EECS; v3 dated 20 March 2025 with October 2024 masthead. Introduces HIL-SERL and its practical real-world learning problem.
Go to primary source ↓e02PDF p. 5, Section 3.1, Eqs. (1)–(2), and Section 3.2
RLPD critic and entropy-regularized Gaussian actor; equal sampling from demo and RL buffers; asynchronous actor/learner operation.
Go to primary source ↓e03PDF p. 6, Figure 2 and Section 3.3, Pretrained Vision Backbones / Downstream Robotic System
ImageNet-pretrained ResNet-10 image embeddings are combined with proprioception. Figure routes replay to RLPD and optional DQN updates; no future-state prediction head is shown.
Go to primary source ↓e04PDF pp. 7–9, Section 3.4, Figure 3 and Section 3.5
Teleoperation overrides policy actions; interventions enter both buffers, policy transitions only RL buffer. Training starts with demonstrations and uses selective corrections; persistent long sparse success-producing interventions can overestimate values.
Go to primary source ↓e05PDF p. 7, Gripper Control and Eq. (3); p. 9, Section 4.1
Separate discrete grasp critic uses three actions per gripper, nine combinations for two; its argmax is concatenated with motion actions. Grasp tasks penalize unnecessary gripper operation.
Go to primary source ↓e06PDF pp. 8–9, Section 3.5
Camera selection, task cropping and 128 × 128 resizing; typical reward data of 200 positive and 1,000 negative samples; initial 20–30 demonstrations and scripted or human resets.
Go to primary source ↓e07PDF pp. 52–53, Appendix D.1–D.2, Eqs. (5)–(7) and controller discussion
Initial-frame proprioception, current-frame actions, coordinate mapping, 10 Hz setpoints and 1,000 Hz impedance/feedforward controllers; reference-error limiting is described without numerical gain/limit settings.
Go to primary source ↓e08PDF pp. 13–14, Section 4.3 and Table 1 footnotes
Typically 100 evaluation trials with randomized resets; whole IKEA uses ten. Training time includes scripted motion, rollouts, intended stops and computation on one RTX 4090. HG-DAgger human episode/intervention matching; flat BC uses 50 Jenga and 200 flipping demos.
Go to primary source ↓e09PDF p. 13, Table 1(a), named task rows, Average row, and footnote
RAM 100%/4.8 s versus 29%/8.3 s; belt 100%/7.2 s versus 2%/9.1 s with six-hour training; IKEA whole 10/10 versus 1/10; reported aggregate 100%/5.4 s versus 49.7%/9.6 s. Jenga and whole-IKEA cycle times are absent.
Go to primary source ↓e10PDF p. 11, RAM Insertion; pp. 32–33, Figures 9–10 and Table 2
RAM is pre-grasped and success excludes locking. Two wrist views, pose/velocity/force inputs; 20 demos; 32,000 transitions; 10 Hz; 100 steps; 4 cm x/y and 6-degree rz randomization; specified encoder, MLP, optimizer and discount settings.
Go to primary source ↓e11PDF p. 12, Timing Belt Assembly; p. 45, Table 10
Success requires threading both pulleys and tightening tensioner. Table lists 12D twists, 20 demos, 108,000 transitions, human resets and 2 cm x/y randomization.
Go to primary source ↓e12PDF p. 12, IKEA The Whole Assembly and Object Handover
IKEA chains three trained policies with scripted transitions, 1 cm translation randomization and two attempts per subpolicy across ten trials. Handover describes delivery to the left basket but calls the success destination the right basket.
Go to primary source ↓e13PDF p. 13, Table 1(b), all rows; pp. 16–17, Section 4.5
Full/no-intervention/no-data success triples are 100/100/100, 48/0/100 and 0/0/0. No-intervention comparison increases offline demonstrations to 200. DP, BC, IBRL, Residual RL and DAPG budgets and scores are reported.
Go to primary source ↓e14PDF p. 13, Table 1(b), Dashboard Assembly / DP; p. 17, Section 4.5, Diffusion Policy paragraph
The same revision reports dashboard DP as 18 in the table and 28% in prose; RAM 27% and flipping 56% agree.
Go to primary source ↓e15PDF pp. 14–15, Figure 5 and Section 4.3; p. 54, Figure 26
Curves are 20-episode running averages. Intervention rate is intervened timesteps divided by episode timesteps. Assisted success can overstate policy-only success. Discounting is offered as an explanation for faster completion.
Go to primary source ↓e16PDF pp. 15–16, Section 4.4 and Figure 6(A–H)
Qualitative perturbations include moving a motherboard, deforming a belt and opening grippers; the authors describe adaptive retries and regrasping.
Go to primary source ↓e17PDF pp. 18–19, Figure 7, Section 5.1 and Eq. (4)
RAM Y/Z visitation plots motivate a funnel interpretation. Q variance is measured over 100 action perturbations drawn uniformly from [-0.2, 0.2] for normalized actions.
Go to primary source ↓e18PDF pp. 19–20, Section 5.2 and Figure 8(E–H)
Three successful trajectories illustrate RAM versus Jenga action means and standard deviations. Figure caption calls these variance plots, while axes and prose identify standard deviation. Interpretation concerns reactive versus predictive behavior.
Go to primary source ↓e19PDF p. 21, Section 6
Model-free RL; each task trains separately; longer horizons and unstructured-environment generalization remain untested. Data generation for later distillation and value pretraining are future directions.
Go to primary source ↓e20PDF p. 12, Jenga Whipping; p. 47, Table 11
Jenga uses 30 offline demonstrations without real-time corrections, human end-of-episode reward labels, human resets and no randomization; episodes run to their 20-step maximum.
Go to primary source ↓e21PDF p. 13, Object Flipping; p. 49, Table 12
Prose describes randomized object placement; the settings table lists no randomization. Table specifies 20 demos, wrist/side images, proprioception and feedforward wrench actions.
Go to primary source ↓e22PDF p. 41, Figure 16 and Table 8
Dashboard settings include wrist and side images, 20 demos, human resets, 2 cm x/y randomization, 36,000 transitions, a 200-step limit and separate motion/grasp networks.
Go to primary source ↓e23PDF p. 50, Appendix B and Figure 25
Reward classifier: pretrained ResNet-10 plus two-layer MLP, cross-entropy, Adam at 0.0003, 100 iterations. RAM negatives include wrong-slot, partial and one-sided insertion.
Go to primary source ↓e24PDF p. 51, Appendix C, Table 13
DP uses 200 demos per task. Observation/prediction/action chunk sizes are 1/8/2 for RAM, 1/8/4 for dashboard and 1/1/1 for flipping.
Go to primary source ↓8.5 Primary sources
Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning ↗
PDF · 20,467 extracted words
Source fingerprint
547a4d5d9440a3f70a773798813db1c8a612f006a6994398bbee2a566daab526