Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning
1. Paper overview
In one sentence: Success-terminated constant rewards can improve final reaching behavior, while reliable learning still depends on how often exploration encounters the goal. motivationformulationssimulation-trainingsimulation-finaltimeout-diagnosticrobot-learning
| At a glance | What to know |
|---|---|
| Research problem | Source description Dense rewards can encourage progress while also encoding unintended preferences. The paper asks whether simply charging time until success better captures reaching quickly and remaining at the target, and when such sparse feedback supports learning from scratch. Goal recognition and a workable exploration/reset setup remain necessary. motivationformulationshit-heuristic |
| Core mechanism | Source description A controlled SAC comparison separates learning speed from final behavior and evaluates policies under each competing reward specification. simulation-trainingsimulation-finalcross-evaluation |
| A key reported result | Reacher-Easy final behavior: Minimum-time: 18.50; 4491.82. Mean steps to goal ↓; steps on goal ↑. 30 policies per formulation; reported evaluation over 500 episodes of 5000 steps. Guiding: 56.88; 4173.55. Contact: 49.64; 4033.12. Final behavior favors minimum-time despite slower training. Figure 5 error bars are standard errors over 30 runs. simulation-finalsimulation-training |
| Reading caution | Reader analysis Reward and termination change together, including Reacher’s near-zero-velocity requirement. The experiments therefore support these task specifications jointly, not a causal claim that reward sparsity alone improves control. formulationscross-evaluation |
Core contributions
- Source description
A controlled SAC comparison separates learning speed from final behavior and evaluates policies under each competing reward specification. simulation-trainingsimulation-finalcross-evaluation
- Author claim
Initial goal-hit frequency is proposed as an early indicator of learnability, with timeout tuning tested in simulation and used to configure four robot tasks. timeout-diagnostichit-heuristicrobot-learning
Table 1. Termination is part of the task specification being compared. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column vertically: the reward rule and the stopping rule form one specification. In the guiding column, being inside the target gives +1; otherwise the penalty is fingertip-to-goal distance. Contact rewards retain only the inside/outside distinction. Both continue for 1000 steps, encouraging occupancy after arrival. The minimum-time column charges −1 at every step and ends on reaching the target with near-zero velocity. The paragraph below the table explains that a timeout can reset the arm without ending this conceptual episode. Reset costs remain part of its accumulated return. formulationsreset-accounting
What it supports. The useful signal is how soon penalties stop, not a changing reward magnitude. This explains why constant rewards can support reaching. It also shows why success recognition and termination design carry substantive task knowledge even when the scalar reward is simple.
Where the evidence stops. This comparison changes reward, horizon and success conditions together. It cannot isolate reward density as the cause of better behavior. The paper does not give a numerical tolerance for “near-zero” velocity.
2. Motivation
2.1 The problem and the proposed response
Dense rewards can encourage progress while also encoding unintended preferences. The paper asks whether simply charging time until success better captures reaching quickly and remaining at the target, and when such sparse feedback supports learning from scratch. Goal recognition and a workable exploration/reset setup remain necessary. motivationformulationshit-heuristic
2.2 What this reading follows
A reward that never changes can still distinguish good behavior: terminating sooner accumulates fewer penalties. This paper tests that idea with SAC on simulated reaching and four physical robot platforms. Its strongest lesson is to separate attractive training curves from the behavior of the finished policy. Guiding rewards learn quickly, but minimum-time policies can reach sooner and remain at the goal longer. A second lesson concerns task setup: reset timeouts alter the chance of discovering success. Read the results as evidence about particular reward-and-termination specifications, with a useful exploration heuristic, rather than a general guarantee for sparse-reward learning. motivationformulationssimulation-trainingsimulation-finaltimeout-diagnosticrobot-learning
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The broad foundational/action-policy placement and “Not applicable” world-action architecture, prediction paradigm and quadrant are supported. This is model-free actor–critic control with reward and timeout design; a shared encoder does not make it a joint world/action predictor. The training category applies through optimization and task setup; no distillation method is demonstrated. formulationstimeout-diagnosticarchitectureconclusion
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Let success end the stream of penalties
The minimum-time specification puts information into termination. A policy that reaches the target stops accumulating −1 rewards sooner. In simulated Reacher, the agent must also arrive with near-zero velocity, which matters for interpreting its improved ability to remain at the target. Timeouts have a different role: they reposition the arm to support exploration but retain the target and do not finish the conceptual episode. The source’s worked example counts three failed 100-step attempts, three 20-step reset costs and a final 25-step success as an episode of length 385 with return −385. This bookkeeping connects performance to wasted attempts and reset time. The stated objective and plotted returns are undiscounted; Appendix A.1 nevertheless uses a 0.99 discount in SAC training. Those are distinct design choices. formulationsreset-accountinghyperparameters
Figure 11. A shared visual representation supports action selection and value learning. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrows from the four yellow convolution blocks toward the upper policy branch and the lower Q branch. Appendix A.2 specifies 32 channels per convolution, 3×3 kernels, and strides 2,2,2,1. Spatial softmax converts visual features into soft coordinates; the teal proprioception vector is concatenated before the magenta 512-unit layers. The policy uses a squashed Gaussian. Forward arrows do not specify backpropagation: the continuation on page 14 says only critic loss trains the shared encoder. Random cropping augments training minibatches. Thus the learning signal for perception differs from the actor’s own gradient path. architectureconclusion
What it supports. The system learns a representation for model-free control. Its branches produce policy outputs and value estimates, not future images or simulated trajectories. Sharing a CNN therefore supplies no evidence for a unified world-and-action prediction model, and no planning rollout is described.
Where the evidence stops. The downward branch leaves before the drawn spatial-softmax marker, while A.2 describes spatial-softmax features before proprioception concatenation. The lower branch’s exact placement is not resolved by the schematic. The caption also calls the red input “previous action” without explaining temporal alignment.
5.2 Use the first goal encounters to choose the setup
A constant reward does not remove the exploration problem: the agent still needs to encounter success to experience its terminating consequence. The paper therefore measures initial goal hits before training and changes reset timeouts to alter their frequency. Its approximately ten-hits-per-20K-steps rule is a practical screening heuristic derived from the simulation studies. With the reported batch size and replay capacity, the authors estimate that goal transitions would appear in about one of every eight minibatch updates. Reader interpretation: this provides an inexpensive reason to try a setup, not a sufficient condition for reliable learning in every environment. Figure 8 connects timeout choices to later learning, while the robot experiments extend the idea to physical platforms with much more limited timeout sweeps. timeout-diagnostichit-heuristicrobot-learning
5.3 Trace learning signals through the controller, then judge behavior
The visual policy is learned through SAC rather than a predictive world model. Appendix A.2 combines convolutional features, spatial softmax and proprioception; actor and critic use the representation, but only critic loss updates the shared encoder. During interaction, the policy’s action is executed and the robot supplies new observations. Reader interpretation: the relevant evidence is therefore achieved behavior, not the visual plausibility of predicted futures. The paper evaluates arrival and occupancy after training, then also tests simulated policies under competing reward formulations. Minimum-time policies can perform well even on guiding-reward evaluations, although the methods were trained with different termination rules. On the physical robots, the gains in occupancy are more pronounced than the small arrival-time differences. The separate curriculum demonstration does not establish a replicated curriculum advantage. architecturerobot-taskssimulation-finalcross-evaluationrobot-finalcurriculum
5.4 Training and inference
During training
SAC uses replay capacity 100K, batches of 256, and actor/critic/entropy step sizes of 3e-4. One update epoch occurs every two steps. Warm-up is 1000 steps, except 20000 in the timeout study. Training discount is 0.99 although the stated minimum-time objective and plotted returns are undiscounted. hyperparametersreset-accountingmotivation
Random image cropping augments minibatches. Only critic loss updates the shared encoder; actor gradients stop there. Robot experiments use asynchronous SAC with ReLoD. No pretrained perception module or future-state prediction objective is described. architecturehit-heuristicrobot-learning
During inference
The policy maps current observations to a squashed-Gaussian action, which is executed and followed by fresh observations and success checking. Vector receives wheel velocities every 100 ms; Franka receives joint velocities at 25 Hz. These are physical control experiments, not generated-video evaluations. formulationsarchitecturerobot-tasks
Success is task-specific image coverage: Create at least 20%, UR5 above 1.5%, Vector roughly 25% with centering, and Franka above 12%. These visual conditions do not establish general manipulation or physical docking success. robot-tasks
5.5 Implementation flow
- Specify success and reward
Guiding Reacher rewards give +1 inside the target and negative Euclidean distance outside; contact rewards give +1/0. Both use 1000-step episodes. Minimum-time rewards stay at −1 and terminate at the goal with near-zero velocity. Thus the comparison changes termination as well as reward density. formulations
- Count resets within the episode
Timeouts reposition the arm while retaining the goal. The conceptual episode ends only at success; reset time incurs an additional cost. This prevents the reported undiscounted return from ignoring repeated failures. reset-accounting
- Screen exploration before learning
Count initial Gaussian-policy goal hits over 20K steps at candidate timeouts. The authors use approximately ten hits as a practical screening threshold, then run asynchronous SAC. This is a setup-dependent heuristic, not a guarantee. timeout-diagnostichit-heuristic
- Encode observations and learn control
For visual tasks, four 32-channel convolutions use 3×3 kernels and strides 2,2,2,1. Spatial-softmax features join proprioception before two 512-unit layers. Actor and critic share an encoder; the caption labels a critic-side input “previous action,” without elaborating its temporal alignment. architecture
6. Experiments & results
This empirical SAC study finds that constant −1 rewards with goal termination can produce better final reaching behavior despite slower learning. Initial goal encounters help select useful reset timeouts. Its physical-robot evidence concerns learned pixel-based control, with no learned world model or imagined rollout (formulations; simulation-final; timeout-diagnostic; architecture; conclusion).
6.1 Read the original evidence
Figure 5. Final behavior favors minimum-time policies despite their slower learning curves. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panels (a) and (b) as arrival time: lower bars are better. Panels (c) and (d) measure time within the goal region: higher bars are better. Use the method names below each bar because colors change between Easy and Hard. The paper evaluates 30 policies per formulation and reports 500 evaluation episodes of 5000 steps. Error bars are standard errors across 30 independent runs. The axes for occupancy start above zero, so compare the printed values rather than visual bar-height ratios. Figure 3 on the same page supplies the separate evidence that minimum-time training progresses more slowly. simulation-finalsimulation-trainingformulations
What it supports. On Easy, minimum-time reaches in 18.50 steps versus 56.88 for guiding rewards and 49.64 for contact rewards. On Hard, the corresponding values are 42.97, 70.15 and 72.37. Minimum-time also has the highest occupancy in both tasks, supporting a distinction between learning speed and finished-policy quality.
Where the evidence stops. The prose’s “over 500” higher accrued rewards does not match the displayed steps-on-goal differences; retain the printed bar values. These results concern the full task specifications, including the near-zero-velocity termination condition, and do not prove optimality.
Table 2. Initial-hit screening is practical, but robot time accounting needs care. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the first three columns: the timeout is paired with a target-hit estimate measured over five independent 20K-step runs of the initial Gaussian policy. Franka has three tested limits; the other robots have one selected setting each. Then read training steps and control frequency before interpreting experience time. The caption is retained because it explicitly excludes environment resets, battery charging and other overhead. The table is quantitative evidence about setup and initial exploration, not a final-policy success-rate table. The associated learning curves are Figure 9, whose thin lines show separate training runs. robot-setuprobot-learningmotivationhit-heuristic
What it supports. Franka reports 12.6 ± 1.0, 13.0 ± 2.08 and 7.8 ± 0.87 hits at limits 75, 150 and 750. Create, UR5 and Vector report 18.8 ± 3.92, 14.4 ± 2.5 and 11 ± 1.1. These entries show how the authors operationalize initial-hit screening.
Where the evidence stops. The ± statistic is undefined. Vector’s approximately 4.444 experience hours exceed the abstract’s two-to-three-hour wording. Franka’s 60K steps, 25 Hz and approximately 1.111 hours do not reconcile by simple step/rate conversion; the accounting is unexplained.
Figure 10. The physical robots show larger occupancy gains than arrival-time gains. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panels (a) and (b) measure steps to goal, where lower is better; panels (c) and (d) measure steps on goal, where higher is better. Orange denotes guiding rewards and dark blue minimum-time. UR5 is evaluated over 2500 episodes of 500 steps; Franka over 1000 episodes of 1000 steps. Those are evaluation horizons from Section 6, not the 150-step guiding-training episodes described in Appendix A.3. Keep each robot’s comparison separate because occupancy depends on horizon and task. Read the numeric labels for the magnitude of the effect rather than inferring statistical confidence from the bars. robot-finalrobot-guiding-rewardrobot-tasks
What it supports. UR5 minimum-time occupancy is 211.08 steps versus 92.54 for guiding rewards, while arrival improves from 37.96 to 35.42. Franka occupancy rises from 178.64 to 237.12, with arrival improving from 37.43 to 34.73. The descriptive evidence is strongest for remaining on target.
Where the evidence stops. The figure’s error-bar statistic is not defined. No significance test is supplied here, so the small arrival differences should not be described as statistically established. Success follows visual target criteria; it does not demonstrate grasping, charging or general manipulation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Reacher-Easy final behavior 30 policies per formulation; reported evaluation over 500 episodes of 5000 steps. | Minimum-time: 18.50; 4491.82. Mean steps to goal ↓; steps on goal ↑ | Guiding: 56.88; 4173.55. Contact: 49.64; 4033.12. Final behavior favors minimum-time despite slower training. Figure 5 error bars are standard errors over 30 runs. simulation-finalsimulation-training |
| Reacher-Hard final behavior Same simulation protocol as Reacher-Easy. | Minimum-time: 42.97; 4289.21. Mean steps to goal ↓; steps on goal ↑ | Guiding: 70.15; 4006.95. Contact: 72.37; 3824.78. Arrival and target occupancy improve. Cross-evaluation also favors minimum-time on the Hard guiding-reward task; this is empirical, not an optimality proof. simulation-finalcross-evaluation |
| Ball-in-Cup initial exploration Initial Gaussian policy; 20K interaction steps per timeout setting. | 47.20 at timeout 10. Reported target-hit count ↑ | 0.57 at timeout 5000. Longer uninterrupted exploration can yield fewer successes. Figure 8 links timeout choices to learning, without establishing universal predictive accuracy. timeout-diagnostic |
| UR5-VisualReacher final behavior 2500 evaluation episodes of 500 steps; physical robot. | Minimum-time: 35.42; 211.08. Mean steps to goal ↓; steps on goal ↑ | Guiding: 37.96; 92.54. The stronger descriptive difference is occupancy. Figure 10 does not define its error-bar statistic; significance is not established. robot-final |
| Franka-VisualReacher final behavior 1000 evaluation episodes of 1000 steps; physical robot. | Minimum-time: 34.73; 237.12. Mean steps to goal ↓; steps on goal ↑ | Guiding: 37.43; 178.64. Reaching is marginally faster and occupancy higher. Different horizons preclude directly comparing raw occupancy with UR5. robot-final |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Reset frequency changes how often an untrained policy discovers success. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis lists timeout choices, not equal increments of elapsed time; each bar counts successes within a fixed 20K-step interaction budget. The policy is the initial Gaussian policy described in Section 5. Read each task separately: Reacher-Easy has its highest displayed count at timeout 2, Reacher-Hard at 25, and Ball-in-Cup at 10. Figure 8 below this crop then tests learning under multiple timeouts on three nonvisual tasks and visual Ball-in-Cup. Its curves average 30 runs with 95% confidence bands. The pair of figures motivates checking exploration before committing to a long training run. timeout-diagnostichit-heuristicreset-accounting
What it supports. Ball-in-Cup drops from 47.20 initial hits at timeout 10 to 0.57 at 5000. Giving an exploratory policy more uninterrupted time can therefore reduce useful discoveries within the same interaction budget. The best timeout differs across tasks, supporting task-specific tuning.
Where the evidence stops. Hits are a diagnostic, not a calibrated probability of learning success. The ten-hit heuristic is chosen from these studies. A fixed interaction-step budget also does not by itself establish equal wall-clock cost when resets take time.
7. Analysis & limitations
7.1 What the evidence leaves open
Reward and termination change together, including Reacher’s near-zero-velocity requirement. The experiments therefore support these task specifications jointly, not a causal claim that reward sparsity alone improves control. formulationscross-evaluation
The ten-hit rule was chosen after simulation analysis. Robot timeout coverage is sparse: three settings for Franka, one for each other platform. No calibrated out-of-distribution success predictor is established. hit-heuristicrobot-learning
The abstract’s two-to-three-hour claim is not a universal training-budget bound: Table 2 lists about 4.444 experience hours for Vector, excluding resets and charging. It also lists Franka at 60K steps, 25 Hz and about 1.111 hours without explaining their accounting. motivationrobot-setup
The 200K-step curriculum is a separate single demonstration, not a multi-run ablation. Reported results do not test broad target, scene or task generalization. curriculumrobot-tasksrobot-final
7.2 Questions for discussion
- Would minimum-time rewards retain their advantage under matched termination and success conditions?
- How well would the initial-hit rule predict learning on preregistered tasks with different reset costs and observation noise?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruct success detectors, reset trajectories and costs, observation preprocessing, SAC settings, and asynchronous control timing. The robot table gives experience budgets rather than complete wall-clock cost; Create’s onboard hardware is specified, but a complete compute/software manifest is absent. robot-tasksrobot-setuphyperparametersarchitecture
Resolve source gaps before matching baselines: the numerical near-zero-velocity tolerance is not stated; Figure 10 and Table 2 omit uncertainty definitions; Appendix A.3’s mask-times-weight equation omits explicit scalar aggregation. Do not silently supply these choices as reported facts. formulationsrobot-finalrobot-setuprobot-guiding-reward
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Factor reward density and termination apart
Reader-proposed check, not performed here: run a factorial Reacher experiment crossing the three reward rules with fixed-length versus goal-terminated episodes. Match target distributions, observation/action interfaces, near-zero-velocity success tolerance, reset accounting, SAC settings and interaction budgets; disclose any tolerance chosen to resolve the paper’s omission. Use the original 30-run scale and evaluate every policy with the same arrival and occupancy protocol. If the minimum-time reward retains an advantage under matched termination, that supports a reward-specific effect. If the advantage follows goal termination across reward rules, it points instead to the stopping condition. formulationsreset-accountingsimulation-trainingsimulation-finalhyperparameters
Check 2: Test initial-hit prediction on unseen timeout settings
Reader-proposed check, not performed here: choose held-out task/timeout combinations before observing learning curves. Estimate initial Gaussian-policy hits using five independent 20K-step runs, then train independent policies with the reported SAC settings and Section 5 warm-up. Predefine a final-return criterion and measure how accurately the ten-hit rule predicts it, recording both interaction steps and reset-inclusive wall-clock cost. Include configurations on either side of the threshold. Frequent failures above it, or reliable learning below it, would challenge its portability; consistent prediction across held-out configurations would strengthen the heuristic. timeout-diagnostichit-heuristicrobot-setuphyperparametersreset-accounting
8.3 Reading coverage
Visual audit: The title and credits on page 1, task photographs on page 2, all Figures 2–11 and Tables 1–3 on pages 4–9 and 13, and Appendix A.2/A.3 details on page 14 were visually inspected. Six final original crops were separately viewed and checked for complete labels and legibility. Page 6 supplies cross-evaluation and initial-policy protocol; page 7 supplies the uncropped learning diagnostic and heuristic; page 8 supplies robot criteria and learning curves; page 13 supplies training settings; page 14 supplies gradient routing and the dense baseline. Architecture branch placement was cross-checked against A.2 and its unresolved schematic detail is disclosed. Table 2’s retained caption is necessary for its estimation protocol and time exclusions. Pages 3 and 10–12 were read in the complete text chunks but were not rendered in this visual pass; they contain related work, acknowledgments and references. No external videos, code or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 9, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 How to Specify Goal-Reaching Tasks in RL?
- 4 Which Formulation Achieves the Desired Reaching Behavior?
- 5 Does Choice of Time Limit Impact Success in Minimum-Time Tasks?
- 6 Learning Vision-Based Minimum-Time Tasks on Real Robots
- 7 Conclusions
- Acknowledgments
- References
- A.1 Choice of Hyper-parameters
- A.2 Neural Network Architecture
- A.3 Guiding Reward Specification for the UR5 and Franka Reaching Tasks
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks were read, covering PDF pages 1–14. Original figures and tables were subsequently inspected on PDF pages 1, 2, 4–9, 13 and 14, resolving the extraction-only image limitation for this paper.
- The inspected edition is marked RLJ | RLC 2024. No numbered revision or revision history is supplied; no other edition was compared. The six authors agree with the catalog, except that the title page prints Martin Jagersand without the catalog’s umlaut. Metadata preserves the inspected spelling.
- Linked repositories, demonstration videos and external references were not opened. Code was not inspected and experiments were not reproduced. No separate supplement was supplied.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title and author block; running header
The title matches the catalog. Credits read Gautham Vasan, Yan Wang, Fahim Shahriar, James Bergstra, Martin Jagersand, and A. Rupam Mahmood. The header identifies RLJ | RLC 2024; no numbered revision is printed. The affiliations include Amii, University of Alberta, Canada CIFAR AI Chair, and Ocado Technology.
Go to primary source ↓motivationPDF pp. 1–2, Abstract and Section 1
The authors contrast engineered guiding rewards with constant negative rewards and goal termination. They claim pixel-policy learning on four robot platforms within two to three hours and emphasize initial goal encounters.
Go to primary source ↓formulationsPDF p. 4, Section 3, Table 1 and Background
Reacher observations contain fingertip position/velocity and the target-relative vector; actions are two joint torques. Guiding rewards give 1 inside the target and negative distance otherwise; contact rewards give 1 inside and 0 outside. Both have 1000-step episodes. Minimum-time rewards are −1 with termination at the target at near-zero velocity.
Go to primary source ↓reset-accountingPDF p. 4, Section 3, Minimum-Time formulation
A timeout resets the arm but retains the goal; the conceptual episode continues until success. A reset costing K steps incurs −K. Three 100-step timeouts with 20-step reset costs followed by 25 steps to success yield return −385 and episode length 385.
Go to primary source ↓simulation-trainingPDF pp. 4–5, Section 4 and Figure 3
SAC uses the same hyperparameters across the three Reacher formulations. Guiding rewards learn faster; minimum-time learning is slower and more variable. Figure 3 averages 30 independent runs with 95% confidence bands, through 500K steps.
Go to primary source ↓simulation-finalPDF p. 5, Section 4, evaluation paragraph and Figure 5(a–d)
Evaluation uses 30 learned policies per formulation and 500 episodes of 5000 steps. Steps to goal, guiding/minimum-time/contact: Easy 56.88/18.5/49.64; Hard 70.15/42.97/72.37. Steps on goal: Easy 4173.55/4491.82/4033.12; Hard 4006.95/4289.21/3824.78. Error bars are standard errors over 30 runs. The prose describes approximately 3X and 2X faster reaching and over 500 higher accrued rewards.
Go to primary source ↓cross-evaluationPDF pp. 5–6, Section 4, cross-formulation evaluation and Figure 6(a–f)
Policies are assessed across reward formulations every 10K training steps. Minimum-time policies are comparable on the Easy guiding-reward task and exceed guiding-trained policies on Hard. Guiding/contact policies struggle with the near-zero-velocity requirement of minimum-time evaluation. Figure 6 reports 30-run means with 95% confidence bands.
Go to primary source ↓timeout-diagnosticPDF pp. 6–7, Section 5, Figures 7–8
Initial Gaussian N(0,1) policies are evaluated for target hits over 20K steps under varying time limits. Figure 7 reports Ball-in-Cup hits of 47.20 at limit 10 versus 0.57 at 5000; Reacher-Easy 57.20 at limit 2; Reacher-Hard 12.03 at limit 25. Figure 8 tests SAC on three nonvisual tasks and visual Ball-in-Cup, with 30-run means and 95% confidence bands. The authors associate hit frequency with learning and propose tuning timeouts as solution parameters.
Go to primary source ↓hit-heuristicPDF p. 7, Section 6, opening two paragraphs
The authors choose approximately 10 initial target hits per 20K steps as a practical sufficiency heuristic, with batch size 256 and replay capacity 100K. They estimate about one in eight minibatch updates would contain a goal transition and use asynchronous SAC after selecting a timeout.
Go to primary source ↓robot-tasksPDF p. 8, Section 6, four task descriptions; PDF p. 2, Figure 1
Create reaches a green note occupying at least 20% of the camera image and uses a Depstech 4K camera and Jetson Nano 4GB. UR5 terminates above 1.5% red-target image coverage, retaining the target until success. Vector sends wheel velocities every 100 ms and terminates with its charger symbol centered at roughly 25% image coverage. Franka sends joint velocities at 25 Hz and terminates above 12% bean-bag coverage.
Go to primary source ↓robot-setupPDF p. 8, Table 2, all rows and caption
Initial target-hit estimates use five independent 20K-step Gaussian-policy runs. Franka limits 75/150/750 give 12.6±1.0/13.0±2.08/7.8±0.87 hits. Create/UR5/Vector report 18.8±3.92/14.4±2.5/11±1.1. Training steps are 60K/100K/100K/160K for Franka/Create/UR5/Vector; frequencies are 25/about 22.22/25/10 Hz. Listed robot experience hours are about 1.111/1.25/1.111/4.444. These exclude resets and charging. The caption does not define the ± statistic.
Go to primary source ↓robot-learningPDF pp. 8–9, Section 6, Figure 9 and surrounding paragraphs
Five runs are conducted for each robot task. Three time limits are tested for Franka and only one is selected heuristically for each other robot because collection is costly. Figure 9 shows independent runs and averages; the poorly performing initial random policy is excluded. ReLoD is described as essential for effective real-time learning.
Go to primary source ↓robot-finalPDF p. 9, Section 6, Figure 10(a–d) and final-behavior evaluation paragraph
UR5 evaluation comprises 2500 episodes of 500 steps; Franka comprises 1000 episodes of 1000 steps. Guiding/minimum-time steps to goal are UR5 37.96/35.42 and Franka 37.43/34.73. Steps on goal are UR5 92.54/211.08 and Franka 178.64/237.12. The caption does not define its error-bar statistic.
Go to primary source ↓curriculumPDF p. 9, Section 6, Using a curriculum to train a robust policy
A single demonstration involving UR5 and Franka uses a hand-crafted curriculum over 200K steps, with success thresholds progressively tightened. It is presented via a linked video, separately from the main learning curves.
Go to primary source ↓hyperparametersPDF p. 13, Appendix A.1, Table 3 and following paragraph
Replay capacity 100K; actor, critic, and entropy-coefficient step sizes 3e-4; batch size 256; discount 0.99; one update epoch every two steps; actor and critic hidden sizes [512,512]; warm-up 1000, changed to 20000 in Section 5; Adam betas [0.9,0.999]; initial temperature 0.1; ReLU.
Go to primary source ↓architecturePDF pp. 13–14, Appendix A.2, Figure 11, policy equation and continuation
The CNN has four 32-channel 3×3 convolutions with strides 2,2,2,1, spatial softmax, proprioception concatenation, and two 512-unit MLP layers. The policy is a tanh-squashed Gaussian. Figure 11 shows actor and Q branches and calls the red input the previous action. Random image crops augment minibatches. Only critic loss trains the shared encoder; actor gradients do not propagate into it.
Go to primary source ↓robot-guiding-rewardPDF p. 14, Appendix A.3, reward equation and following paragraphs
The guiding reward uses a binary red-pixel mask M_t, an h×w center-weight matrix W, and scaling c/(hw), with c=800 and rewards bounded between 0 and 4. W decreases from 1 in the center to 0 at the edges. The printed Hadamard product has no explicit scalar-reduction operator. Episodes span 150 steps of 40 ms; SenseAct repeats commands five times at 8 ms intervals.
Go to primary source ↓conclusionPDF p. 9, Section 7
The paper describes its system as model-free RL for physical robot pixel control, reports nearly 200 total robot-use hours across experiments, and advocates tuning time limits and reconsidering constant negative rewards.
Go to primary source ↓8.5 Primary sources
Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning ↗
PDF · 6,728 extracted words
Source fingerprint
5085c905afb84fabb4f1eb7b7feb0e60cabb3726b47ac84dd2db3235c476e457