Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
1. Paper overview
In one sentence: A recurrent policy trained across randomized simulated physics transfers puck pushing to a real robot, but the evidence remains tied to state-based sensing and a small physical evaluation. e01e03e05e06e08e09e12e13e14e15
| At a glance | What to know |
|---|---|
| Research problem | Source description A policy can exploit simulator-specific contact or actuator behavior and fail on hardware. Collecting the reinforcement-learning experience directly on a robot is costly and complicates exploration. The paper asks whether a deliberately variable simulator can instead teach a policy to handle unknown physical dynamics. Puck pushing stresses this idea because contact, friction, sensing, and controller timing jointly determine whether a commanded motion moves the object toward its goal. e02e03e05 |
| Core mechanism | |
| A key reported result | Sim-to-real puck pushing: LSTM: simulation 0.91 ± 0.03; real 0.89 ± 0.06. Episode-end success: puck within 0.07 m of target. Simulation-only training; 100 randomized-dynamics simulation trials and 28 physical LSTM trials. Simulation/real horizons are 100/200 control steps. Starts and goals are randomized within 0.3×0.3 m. FF no Rand: 0.51 ± 0.05 / 0.0 ± 0.0 (10 real trials); FF: 0.83 ± 0.04 / 0.67 ± 0.14 (12); FF + Hist: 0.87 ± 0.03 / 0.70 ± 0.10 (20). Values are simulation / real. The LSTM has the highest reported physical success. Similar simulation/real rates use different horizons, and small unequal physical samples limit comparison. The ± statistic is undefined. e11e13e14 |
| Reading caution |
Core contributions
Figure 4. The policy learns from history; the critic has extra information during training. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper diagram from left to right. The goal g and current state s_t enter the feedforward path; s_t also joins previous action a_(t−1) in the lower path. The loop at layer 2 marks recurrent memory, identified in Section IV.F as an LSTM. These paths merge before the action output a_t. In the lower diagram, the queried action a_t and simulator parameters μ join the critic's feedforward inputs, ending in Q. The 128 labels denote hidden-layer widths. The arrow routing agrees with Sections IV.D–F: goals bypass recurrence, and dynamics parameters are available to the critic, not the actor. e05e08e09e13
What it supports. The architecture provides a route for adapting actions to observed response history without requiring a physical-parameter estimate at deployment. The critic can distinguish different simulated dynamics while teaching the actor. This explains how privileged training information can help a policy whose deployed inputs contain only observable interaction information.
Where the evidence stops. The diagram establishes information access, not successful system identification. Neither the recurrent loop nor the critic's parameter input proves that actor memory recovers mass or friction; no direct memory-identification probe is reported.
2. Motivation
2.1 The problem and the proposed response
A policy can exploit simulator-specific contact or actuator behavior and fail on hardware. Collecting the reinforcement-learning experience directly on a robot is costly and complicates exploration. The paper asks whether a deliberately variable simulator can instead teach a policy to handle unknown physical dynamics. Puck pushing stresses this idea because contact, friction, sensing, and controller timing jointly determine whether a commanded motion moves the object toward its goal. e02e03e05
2.2 What this reading follows
The useful starting point is the command that reaches the robot: a set of relative joint-angle targets. Everything learned here serves that control decision. Peng and colleagues vary simulated masses, friction, actuator properties, timing, and observations, then train a recurrent policy to push a puck toward a requested location. A separate critic sees simulator parameters during training; the deployed policy must rely on state and interaction history. The six visuals below connect that information flow to the randomization ranges, measured calibration mismatch, learning curves, physical transfer table, and removal experiments. Together they show a successful transfer recipe while leaving its internal adaptation mechanism only indirectly tested. e01e03e05e06e08e09e12e13e14e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/action-policy classification and Not applicable architecture, prediction paradigm, and quadrant are supported. This is an actor–critic transfer method using an external physics simulator. Its recurrent state may summarize dynamics, but it does not learn a future-observation predictor or jointly predict futures and actions. Having actor and critic networks does not place it in a world-action-model architecture quadrant. Distillation is not demonstrated. e03e08e09e17
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep the task goal out of the dynamics memory
Imagine the puck moves less than expected after an action. Current state alone describes where it is now; pairing observations with past actions also describes how the system responded. The recurrent branch is designed to accumulate that second kind of information. Figure 4 routes current state and the previous action through an embedding and LSTM, while the goal enters a parallel feedforward branch. The authors justify this separation by arguing that the goal carries no information about dynamics. Current state enters both paths so action selection need not recover every immediate detail from memory. The merged representation then produces joint targets. This is a learned control representation: the policy does not first announce a mass or friction estimate that a separate controller subsequently consumes. e05e08e09
Table I. Randomization spans actuator response, contact physics, and control timing. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First distinguish relative multipliers from absolute quantities. Link mass, joint damping, and controller gains multiply simulator defaults, whereas puck mass and table height have physical units. The last row lists λ in inverse seconds: it is the rate of an exponential delay distribution, not a sampled action duration. Section V defines each duration as 0.04 seconds plus a fresh exponential draw. Physical parameters and λ are sampled per episode, while duration and observation noise vary each step. The surrounding text also specifies logarithmic sampling for mass, damping, friction, and gains. Observation noise is described separately and has no row in this table. e06e07e11
What it supports. The simulator family is broader than a collection of puck masses. Link mass ranges from 0.25 to 4 times default and joint damping from 0.2 to 20 times default, while timing variability changes how long each action acts. These choices expose the policy to several causes of real-world response mismatch during training.
Where the evidence stops. Range endpoints alone do not reproduce the distribution: sampling rules, nominal model values, and noise scaling also matter. Several default values are not enumerated in the paper, and this table does not establish an optimal range.
5.2 Make failed pushes useful without changing their physics
Sparse reward gives the agent zero when the target is satisfied and minus one otherwise. An episode that misses its requested target can therefore contain little positive guidance. HER reuses that recorded episode with a goal achieved by the trajectory and recomputes its rewards. The physical states and actions stay the same; their task interpretation changes. Algorithm 1 combines this relabeling with off-policy recurrent actor–critic updates and retains the episode's sampled dynamics parameters. The critic can use those parameters to evaluate actions under the actual simulated conditions that produced the experience. The actor receives no such privileged input. The reported training uses HER with probability 0.8, so this is part of the learning procedure, not a goal substitution performed when scoring the real robot. e04e08e10e11
5.3 Separate successful transfer from an identified mechanism
My interpretation is that the strongest evidence concerns the complete transfer recipe. Table II measures executed robot behavior and favors the recurrent policy; Table III shows that removing timing variation or observation noise damages that recipe. Figure 5 gives an independent reason to expect actuator response mismatch. Together these observations support training for uncertain response dynamics. They do not directly show which physical properties the LSTM encodes, or whether better temporal filtering explains part of its advantage. There is also a protocol boundary: real trials allow 200 control steps, whereas simulated episodes use 100, and physical counts differ by model. A convincing follow-up would preserve the physical success definition while controlling horizon, training seed, evaluation cases, and recurrent-memory access. e08e11e12e14e15
5.4 Training and inference
During training
MuJoCo uses a 0.002 s physics step, nominally 20 physics steps per action, and 100 control steps per simulated episode. Observation noise is zero-mean Gaussian with standard deviation 5% of each feature's running standard deviation; action exploration adds Gaussian noise with standard deviation 0.01 rad. e06e07e11
Actor and critic use ADAM at 5×10^-4, batches of 128 episodes, and HER probability 0.8. Each policy receives approximately 8000 update iterations and about 100 million samples. The reported eight hours on 100 cores refers to simulation, not a complete measured end-to-end training runtime. Target networks are used but omitted from the pseudocode. e08e10
During inference
On hardware, measured state and the previous action update policy memory; the policy emits relative joint-angle targets, the position controller executes them, and new observations close the loop. PhaseSpace tracks the puck. No real-world policy fine-tuning or online gradient update is described: runtime adaptation refers to recurrent-state evolution, and the privileged critic is absent. e02e05e08e09e11
5.5 Implementation flow
- Sample a family of environments
At each episode start, choose a goal, random puck position, and dynamics parameters. The paper reports 95 randomized parameters spanning robot links, joints, puck, table, controller, timing, and observations. Most physical parameters remain fixed within the episode; action duration and observation noise vary every step. Mass, damping, friction, and gains use logarithmic sampling. e05e06e07
- Build memory from interactions
Current state and previous action pass through a 128-unit embedding and 128-unit LSTM. A parallel feedforward path receives current state and goal. The goal bypasses recurrence because the authors argue that it carries no information about dynamics. Both paths merge before two further 128-unit layers and a scaled tanh action output. e08e09
- Train with a better-informed critic
The separate recurrent critic receives the queried action and true simulator parameters in its feedforward path, producing a linear value output. These parameters help training feedback without requiring their measurement on the robot. Actor memory is learned through the control objective rather than a supervised mass or friction target. e08e09
- Relabel unsuccessful experience
Store complete episodes with their goals, rewards, and dynamics. HER substitutes an achieved goal and recomputes rewards, allowing unsuccessful original attempts to provide useful learning signals. RDPG supplies off-policy actor and critic updates with recurrent memories computed along the replayed trajectory. e04e08e10
6. Experiments & results
Dynamics randomization trains a recurrent pushing policy across varied simulated physics, then deploys it directly on a Fetch arm. Its memory can use interaction history, while a training-only critic receives privileged simulator parameters. Reported real success is 0.89 ± 0.06 over 28 trials. This is evidence for physical policy transfer under the tested setup, with different simulation and real evaluation horizons; it is not a learned future-prediction model.
6.1 Read the original evidence
Figure 6. Memory changes learning under randomized test dynamics. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as optimizer steps and the vertical axis as simulation success. The blue curve is LSTM, orange is memoryless FF, green is FF trained without randomization, and red is FF with eight previously observed states and actions. Crucially, the green policy is still evaluated under randomized dynamics, so it is not showing its training-environment score. Section V.A and the caption state that four policies with different random seeds are trained for each architecture and evaluations use 100 simulated episodes. Use the curves to compare learning speed and final ordering, then turn to Table II for separately tabulated endpoint values. e11e13e14
What it supports. The LSTM rises faster and reaches the highest success in this simulation comparison. Explicit finite history substantially improves the feedforward alternative, supporting the practical value of temporal information. Randomized training also helps the memoryless network withstand varied test dynamics, although its learning curve remains below the recurrent policy.
Where the evidence stops. These are simulated evaluations, not physical deployments. The paper does not define the shaded-band statistic, and this architectural comparison does not isolate recurrence from all differences in capacity or optimization.
Table II. The recurrent policy has the highest reported real-robot success. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each row before comparing methods: the first success column is simulation, the second is real deployment, and the final column counts physical trials. The caption specifies 100 randomized simulation trials. Sections V and V.A add essential protocol details outside the crop: success means ending within 0.07 m of the target; simulation episodes use 100 control steps and real episodes 200. Starts and targets are randomized within a 0.3 by 0.3 m region. FF + Hist receives eight past states/actions, whereas FF no Rand has no randomized training. All listed policies learn from simulation, with real data used here for evaluation. e02e11e13e14
What it supports. LSTM reports 0.89 ± 0.06 real success over 28 trials, compared with 0.70 ± 0.10 for FF + Hist over 20 and 0.67 ± 0.14 for FF over 12. FF no Rand reports zero over 10. The result supports useful direct transfer in this physical pushing setup.
Where the evidence stops. The ± statistic is undefined, physical counts are small and unequal, and seed aggregation is unclear. Similar simulation and real rates are not a matched-horizon comparison because hardware receives twice as many control steps.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Sim-to-real puck pushing Simulation-only training; 100 randomized-dynamics simulation trials and 28 physical LSTM trials. Simulation/real horizons are 100/200 control steps. Starts and goals are randomized within 0.3×0.3 m. | LSTM: simulation 0.91 ± 0.03; real 0.89 ± 0.06. Episode-end success: puck within 0.07 m of target | FF no Rand: 0.51 ± 0.05 / 0.0 ± 0.0 (10 real trials); FF: 0.83 ± 0.04 / 0.67 ± 0.14 (12); FF + Hist: 0.87 ± 0.03 / 0.70 ± 0.10 (20). Values are simulation / real. The LSTM has the highest reported physical success. Similar simulation/real rates use different horizons, and small unequal physical samples limit comparison. The ± statistic is undefined. e11e13e14 |
| Randomization component ablation on the real robot Retrained LSTM policies with individual randomization components removed; physical pushing evaluation. | All: 0.89 ± 0.06 (28); fixed action timestep: 0.29 ± 0.11 (17); no observation noise: 0.25 ± 0.12 (12). Episode-end success rate; physical trial count | Fixed link mass: 0.64 ± 0.10 (22); fixed puck friction: 0.48 ± 0.10 (27). Timing variation and sensor noise have the largest observed removals' costs. These separate removals do not measure interactions or establish significance; uncertainty is undefined. e11e15 |
| Pushing with altered puck contact dynamics Physical LSTM deployment with a packet of chips attached under the puck. | 0.91 ± 0.04 with attachment. Reported success rate | 0.89 ± 0.06 without attachment. Supports tolerance to this particular contact change. The attachment trial count is absent, uncertainty is undefined, and the numerical increase does not establish improved performance. e11e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. The same commanded trajectory produces different simulated and physical joint responses. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue target, orange simulated pose, and green real pose within each panel. The horizontal axis is timestep and the pose axis is in radians. Joint 0, joint 3, and joint 6 correspond to shoulder, elbow, and wrist examples. Compare the timing and shape of their oscillations rather than assuming identical tracking from an identical command. The panels use different vertical ranges, so apparent curve separation should not be compared directly as a common error magnitude. Section V explains that the authors executed the same target trajectory in simulation and on hardware specifically to expose their calibration mismatch. e06e07e12
What it supports. The real and simulated response traces visibly differ, with mismatch varying across joints. This gives concrete context for randomizing actuator-related quantities and action timing. The transfer experiment therefore operates in a setting with demonstrated response discrepancies, rather than relying solely on a verbal assertion that simulation and hardware differ.
Where the evidence stops. This is a tracking diagnostic, not a success-rate experiment or a comparison of learned policies. It does not isolate whether mass, gain, damping, or latency explains each discrepancy, and no aggregate calibration-error metric is reported.
Table III. Removing timing variation or observation noise produces the largest observed drops. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row, all, as the reference policy trained with the full randomization setup. Each subsequent row names a training change: hold action duration fixed, remove observation noise, fix robot link masses, or fix puck friction. These are separately trained LSTM policies evaluated on the real robot, rather than switches applied to one policy during evaluation. Read the trial column alongside success because the counts differ substantially. Section IV.C explains the distinction between episode-level physical parameters and per-step timing/noise; that distinction helps interpret why the first two removals test different disturbances from link mass or contact friction. e06e07e11e15
What it supports. Full randomization reports 0.89 ± 0.06 success. Fixing action timestep gives 0.29 ± 0.11, and removing observation noise gives 0.25 ± 0.12; fixed link mass and fixed puck friction give 0.64 ± 0.10 and 0.48 ± 0.10. The pattern motivates careful modeling of sensing and controller timing alongside contact physics.
Where the evidence stops. The separate removals do not measure interactions among randomizations. Unequal trial counts and an undefined ± statistic preclude a precise significance claim; additional controls would be needed to attribute the drops to a particular adaptation mechanism.
7. Analysis & limitations
7.1 What the evidence leaves open
Evidence covers one arm and one state-based pushing task. Puck tracking uses external motion capture; image-based control and a broader task repertoire are future work. e05e11e17
History-dependent control is consistent with implicit system identification, but the paper does not probe memory for physical parameters or compare intact versus reset memory at deployment. Architecture comparisons therefore leave adaptation versus general recurrent robustness partly unresolved. e08e09e13e14
No reported success uncertainty is defined, and physical trial counts differ across policies and ablations. The 200-step real horizon versus 100 simulated steps prevents treating similar success rates as a matched-budget reality-gap measurement. e11e14e15
7.2 Questions for discussion
- Does recurrent memory identify changing dynamics, or mainly stabilize control under noisy observations?
- Would the ranking persist with matched horizons, equal trial counts, and repeated training seeds?
- How dependent is transfer on the randomization ranges and motion-capture state quality?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the Fetch position-control interface, MuJoCo model and nominal parameters underlying Table I's multipliers, a puck/table setup, and PhaseSpace-equivalent state measurements. The text states a 52D state but does not fully enumerate its encoding. Exact software versions and default controller gains are not supplied. e05e06e09e11
The paper specifies network widths, sampling ranges, optimizer settings, and approximate data cost, but not the discount value, replay capacity, target-network update coefficient, exact HER goal-sampling strategy used in experiments, or physical policy selection across seeds. A reproduction should expose these choices and report per-seed counts and defined intervals under matched horizons. e04e08e10e11e13e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does useful history change control under fixed unknown dynamics?
Reader-proposed check, not performed here: train several LSTM seeds with the reported randomization recipe, then evaluate each frozen policy on identical held-out simulated puck masses, frictions, and action-timing rates. Compare intact recurrent memory against memory reset before every action, retaining the same current-state and previous-action inputs. Match initial states, goals, noise streams, and horizon, and report per-seed endpoint success and distance. If memory helps adapt to episode-specific response, intact memory should improve performance as interaction evidence accumulates. Little or no difference would weaken that interpretation. Resetting memory can itself create an unfamiliar internal-state distribution, so include the trained FF + Hist baseline as a complementary control rather than treating this intervention alone as decisive. e05e06e07e08e09e11e13e14
Check 2: Disentangle timing variation, observation noise, and evaluation budget
Reader-proposed check, not performed here: train a factorial comparison with action-timing variation on/off and observation noise on/off, leaving all other randomizations and optimizer settings fixed. Use repeated training seeds and equal numbers of matched physical start/goal cases. Score the same runs at 100 and 200 control steps, reporting success, final distance, trial counts, and explicitly defined uncertainty intervals. Compare each single removal with the combined removal to test interaction, and compare both horizons to test whether extra execution time changes the ranking. If the large penalties persist across seeds and horizons, the timing/noise conclusion strengthens; a disappearing or reversed penalty would narrow it to the original evaluation conditions. e06e07e10e11e14e15
8.3 Reading coverage
Visual audit: All eight PDF pages were rendered and visually inspected, including the title/authors/version on p. 1, objective and HER on p. 3, task and privileged-critic formulation on p. 4, Algorithm 1, architecture text and Table I on p. 5, architecture, calibration, learning curves and compute/protocol details on p. 6, Tables II–III and robustness/conclusions on p. 7, and references on p. 8. Figures 1–7 and Tables I–III were inspected. Each of the six final crops was separately viewed; they retain complete diagram inputs/outputs, axes, legends or table headers as applicable. Figure 4's arrows and recurrence were cross-checked against Sections IV.D–F, with no claim-relevant conflict found. Table captions were read on the full pages; their evaluation details are carried into the guides. No appendix is present. Separate supplements and the linked video were not supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction (p. 1)
- II. Related Work, A–C (pp. 1–2)
- III. Background, A. Policy Gradient Methods and B. Hindsight Experience Replay (pp. 2–3)
- IV. Method, A–F, including Algorithm 1 (pp. 3–5)
- V. Experiments, A. Comparison of Architectures, B. Ablation and C. Robustness (pp. 5–7)
- VI. Conclusions; VII. Acknowledgement; References (pp. 7–8)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by visually inspecting all eight supplied PDF pages and all final crops.
- Separate supplemental material availability has not been fully verified.
- The linked supplemental video was not supplied or viewed. Code was not inspected and experiments were not reproduced.
- Version scope: the verified title and all four authors match the catalog. This report reads arXiv:1710.06537v3, dated 3 March 2018. The catalog lists ICRA 2018; the publisher edition and earlier arXiv revisions were not supplied or compared, so edition differences beyond the observed version remain unverified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin stamp
The title matches the catalog. Authors are Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. The artifact is arXiv:1710.06537v3 [cs.RO], dated 3 March 2018. Affiliations are OpenAI and UC Berkeley, Department of Electrical Engineering and Computer Science.
Go to primary source ↓e02PDF p. 1, Abstract and Section I; p. 2, Sections II.B–C
The paper motivates simulator-only policy training by sample cost, exploration concerns, and simulation-to-reality mismatch. It proposes dynamics randomization and recurrent policies for physical puck pushing with sparse rewards.
Go to primary source ↓e03PDF p. 3, Section IV, displayed dynamics-randomization objective
A physics simulator approximates real dynamics. Parameters mu are sampled from a distribution, and the policy maximizes expected trajectory reward averaged over those dynamics.
Go to primary source ↓e04PDF p. 3, Section III.B, binary reward and HER explanation; p. 5, Algorithm 1, lines 4–24
Reward is zero when the goal is satisfied and minus one otherwise. HER relabels goals and recomputes rewards for stored trajectories; RDPG updates recurrent actor and critic using replayed episodes.
Go to primary source ↓e05PDF pp. 3–4, Section IV.A; p. 4, Section IV.B and Figure 2
The task uses a 7-DOF Fetch arm, random puck start and goal, and a default initial arm pose. The stated 52D state contains arm joint positions/velocities, gripper position, and puck pose and velocities. The 7D action is a relative target-joint-angle offset for a position controller.
Go to primary source ↓e06PDF p. 4, Section IV.C; p. 5, Table I and Section V; p. 6, Section V continuation below Figure 4
The paper reports 95 randomized parameters. Table I gives link-mass multipliers 0.25–4, joint-damping multipliers 0.2–20, puck mass 0.1–0.4 kg, friction 0.1–5, puck damping 0.01–0.2 Ns/m, table height 0.73–0.77 m, controller-gain multipliers 0.5–2, and action-timestep rate 125–1000 inverse seconds. Mass, damping, friction and gains are sampled logarithmically. Noise standard deviation is 5% of each feature's running standard deviation.
Go to primary source ↓e07PDF p. 5, Section V, action-timestep formula; p. 6, first continuation of Section V
Each action duration is 0.04 seconds plus an exponential draw with rate lambda; lambda is fixed within an episode while duration varies each step. Observation noise is resampled each step. Gaussian exploration noise has standard deviation 0.01 radians.
Go to primary source ↓e08PDF p. 4, Sections IV.D–E
The recurrent policy summarizes past states and actions in internal memory. It does not explicitly output physical parameters. The omniscient recurrent critic receives the simulator parameters during training; those parameters are not policy inputs. Target networks are used but omitted from Algorithm 1 for brevity.
Go to primary source ↓e09PDF p. 5, Section IV.F; p. 6, Figure 4 and caption
Actor and critic have recurrent and feedforward branches. Current state and previous action feed a 128-unit embedding and 128-unit LSTM; current state also enters the feedforward path with the goal. The critic adds the queried action and dynamics parameters to that path. Concatenated features pass through two 128-unit layers; actor output is scaled tanh and critic output linear.
Go to primary source ↓e10PDF p. 5, Algorithm 1; p. 6, Section V, optimizer paragraph
ADAM uses step size 5×10^-4 for actor and critic, batches of 128 episodes of 100 steps, and HER probability 0.8. Each policy uses approximately 8000 update iterations and about 100 million samples; generating the simulation data takes approximately eight hours on a 100-core cluster.
Go to primary source ↓e11PDF p. 5, Section V, simulation timing; p. 6, Section V and V.A; p. 7, Section V.A and Table II caption
MuJoCo uses a 0.002-second physics step and nominally 20 physics steps per control step; simulation episodes have 100 control steps. Physical evaluation uses 200 steps, a roughly 0.2 kg puck of radius 0.065 m, PhaseSpace tracking, and success within 0.07 m at episode end. Starts and goals lie within a 0.3×0.3 m region. Simulation evaluation uses 100 randomized-dynamics trials; real counts differ by model.
Go to primary source ↓e12PDF p. 6, Figure 5 and Section V, calibration paragraph
The same target joint trajectories are executed by the real and simulated robots. Shoulder, elbow, and wrist responses differ, with visible joint-dependent tracking mismatch. The authors report little calibration.
Go to primary source ↓e13PDF p. 6, Section V.A and Figure 6
Four random-seed policies are trained per architecture. The simulation learning curves compare LSTM, memoryless FF, FF trained without randomization but tested with it, and FF with eight past states/actions. LSTM learns faster and reaches higher success in the plotted comparison; the shading statistic is not specified.
Go to primary source ↓e14PDF p. 7, Table II, all rows and columns; Figure 7; Section V.A
LSTM success is 0.91 ± 0.03 in simulation and 0.89 ± 0.06 on the robot (28 real trials). FF no Rand: 0.51 ± 0.05 / 0.0 ± 0.0 (10); FF: 0.83 ± 0.04 / 0.67 ± 0.14 (12); FF + Hist: 0.87 ± 0.03 / 0.70 ± 0.10 (20). The paper does not define the ± statistic or real-policy seed aggregation.
Go to primary source ↓e15PDF p. 7, Section V.B and Table III, all rows
Physical LSTM success with all randomizations is 0.89 ± 0.06 (28 trials). Fixed action timestep: 0.29 ± 0.11 (17); no observation noise: 0.25 ± 0.12 (12); fixed link mass: 0.64 ± 0.10 (22); fixed puck friction: 0.48 ± 0.10 (27). The authors identify timing and noise as especially consequential.
Go to primary source ↓e16PDF p. 5, Figure 3 and caption; p. 7, Section V.C
A packet of chips attached under the puck changes contact dynamics and, according to the authors, reduces friction. LSTM success is 0.91 ± 0.04 versus 0.89 ± 0.06 without the attachment. The modified-condition trial count and uncertainty definition are not stated. Fine-positioning behaviors are described qualitatively.
Go to primary source ↓e17PDF p. 7, Section VI; pp. 4–6, Sections IV–V
The demonstrated task is state-based puck pushing on a Fetch arm. The conclusion proposes broader tasks and vision as future work. The supplied paper has no appendix; the remaining material is acknowledgements and references on pp. 7–8.
Go to primary source ↓8.5 Primary sources
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization ↗
PDF · 6,942 extracted words
Source fingerprint
a15cb24630a11218865027024ae2353f80f46d40371b42dccb2b24e9d00d62d8