Contrastive Learning as Goal-Conditioned Reinforcement Learning
1. Paper overview
In one sentence: A contrastive future-state discriminator can train a goal-reaching actor, but its value interpretation depends on how trajectories and goals are sampled. e02e04e05e06e08e10e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Can an RL objective itself learn useful representations without auxiliary perception losses or image augmentation? The paper targets goal-conditioned control from action-labeled trajectories. Goals specify desired states; a probability-based reward supports the analysis, while the practical learner receives no ground-truth reward function. e01e03e10 |
| Core mechanism | |
| A key reported result | Offline D4RL AntMaze large-play-v2: 2 critics: 41.6 ± 6.0; 5 critics: 48.6 ± 4.4. Reported AntMaze performance; score normalization and table ± convention are unspecified.. Static D4RL data; NCE+BC, two versus five critics. Baseline values are quoted. GCBC 32.4; IQL 39.6; TD3+BC 0.2. Five critics exceed IQL by 9.0 table points. A substantial reported gain, but IQL/TD3+BC use v0 and no goal relabeling; this does not isolate avoiding TD. e14e14-large |
| Reading caution | Reader analysis Approximate policy improvement assumes tabular states/actions, a Bayes-optimal critic and trajectory filtering, with tolerance 2 gamma epsilon/(1−gamma). Main experiments omit filtering. Its reported performance penalty and the complete proof cannot be checked because Appendix Figure 13 and Appendix B are absent. e08 |
Core contributions
Figure 1. Use actions and temporal outcomes to train the representation that will guide control. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the two trajectories. The green state and action come from trajectory 1, and its later green observation supplies the positive future state. The red observation from trajectory 2 illustrates a random negative. On the left, phi encodes the state together with the action; on the right, psi encodes either candidate future state. Both arrows meet at the similarity comparison. Equation (6) makes that comparison a binary classification objective using an inner product. The caption’s word “closest” therefore means similarity under the learned score, not a specified Euclidean distance. The separate actor is described in Section 4.3 rather than drawn here. e02e04e05e06
What it supports. The action input makes the score useful for choosing behavior: different actions at the same state can receive different scores for the same goal. The encoders are trained from observed outcomes, and the actor is then optimized to raise the commanded-goal score. The figure depicts the training signal, not a generated video or a planned rollout.
Where the evidence stops. The caption compresses the value connection. Lemma 4.1 relates the exponentiated optimal score, not the raw inner product, to a goal-averaged policy value. It does not identify the critic with the value of every commanded-goal policy.
2. Motivation
2.1 The problem and the proposed response
Can an RL objective itself learn useful representations without auxiliary perception losses or image augmentation? The paper targets goal-conditioned control from action-labeled trajectories. Goals specify desired states; a probability-based reward supports the analysis, while the practical learner receives no ground-truth reward function. e01e03e10
2.2 What this reading follows
Suppose a robot has collected trajectories showing what happened after its actions. This paper asks whether distinguishing those outcomes from unrelated observations already supplies the information needed for control. Its critic compares an encoded state-action pair with an encoded goal, and its actor learns to increase that comparison score. The useful theoretical bridge is discounted future occupancy: at the contrastive optimum, exponentiating the score recovers a value function up to a goal-dependent factor. The figures below follow that bridge into simulated online and offline experiments. They also expose important boundaries: goal averaging, restrictive convergence assumptions, incomplete appendix access and differences between the offline comparison protocols. e02e04e05e06e08e10e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational theory/action-policy classification is supported. An occupancy-related critic and separate actor neither jointly generate futures/actions nor extract actions through inverse dynamics. The world-action-model architecture, prediction-paradigm and quadrant fields are appropriately Not applicable: an outside-quadrant foundation, not an unverified quadrant assignment. e05e06e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn trajectory time into a value-learning label
Begin with the outcome rather than a hand-designed distance reward. Section 3 defines reaching a goal through transition probability or density, and Proposition 1 relates the resulting discounted value to discounted state occupancy. Sampling a future observation with a geometric time offset therefore supplies a training example with the relevant temporal weighting. The contrastive problem asks whether that observation is a future of this particular state-action pair or a sample from the marginal outcome distribution. The state-action encoder matters because control requires distinguishing actions available from the same observation. Algorithm 1 implements the comparison efficiently by scoring all state-action embeddings against all goal embeddings in a batch, assigning positive labels to matched pairs. No image reconstruction target is required by this loss. e03e04e05e07
5.2 Use the score for actions without overstating the theorem
At the optimal binary classifier, the critic score is the logarithm of conditional future occupancy divided by marginal goal density. Exponentiating recovers the value relationship in Lemma 4.1. For a fixed goal, the marginal density does not depend on the chosen action, so Equation (7) trains the actor to increase the score without estimating that normalizer. The learned actor maps an observation and commanded goal to an action distribution; the algorithm alternates its updates with critic learning and new experience. The qualification is consequential: replay combines trajectories commanded toward different goals, so the critic describes a goal-averaged policy. The approximate-improvement guarantee adds tabular states/actions, an optimal critic and a trajectory filter. The practical experiments omit that filter, and the supplied PDF omits the full proof. e04e05e06e07e08
5.3 Interpret offline success as a controlled-learning question
The offline adaptation changes more than the data source. It adds goal-conditioned behavioral cloning, whose weight trades off staying close to recorded actions against increasing the critic score, and it takes a minimum across multiple critics. Table 1 then shows substantial large-maze gains while preserving clear exceptions. My interpretation is that these results motivate testing how the pieces interact, rather than concluding that temporal-difference learning is unnecessary for every hard control task. The table compares quoted baselines, different goal-relabeling choices and two ensemble sizes; increasing critic count also changes capacity and aggregation. A useful next experiment would hold dataset version, parameter budget, training updates and goal sampling fixed. That would distinguish an ensemble benefit from additional capacity or a favorable comparison protocol. e14e14-largee14-all
5.4 Training and inference
During training
Basic NCE needs no TD targets, target Q networks, multiple critics, augmentation or auxiliary representation losses. Image tasks add actor entropy. Although theoretically on-policy, the implementation reuses transitions for multiple updates as in off-policy methods. e06e07
Offline training adds lambda-weighted goal-conditioned behavioral cloning and uses the minimum of two or five critics for actor updates. Lambda equal to one recovers GCBC; the selected lambda is not supplied. e14
During inference
The actor receives the current observation and desired goal and outputs actions, trained to increase their critic score. The environment supplies subsequent observations. Algorithm 1 samples actions, but the supplied body does not specify a separate deterministic-versus-stochastic evaluation convention. e06e07
5.5 Implementation flow
- Construct outcome supervision
Sample a replay state-action pair. Its positive comes from discounted future occupancy, conceptually using a geometric time offset. A negative comes from the marginal future-state distribution of random replay pairs. Actions explicitly condition the representation of future outcomes. e04e05
- Fit a factorized critic
Encode the pair with phi and the goal with psi; score their inner product. Algorithm 1 forms all pairwise batch scores with positive diagonal labels and negative off-diagonal labels. It reuses goal embeddings. These are learned representations, not reconstructed future frames. e02e05e07
- Improve the actor and collect feedback
Train the neural policy on replay states and random goals to maximize critic scores through reparameterized sampled actions. Alternate critic fitting, actor updates and new experience. No learned rollout planner or inverse-dynamics decoder is specified. e06e07
6. Experiments & results
Contrastive RL turns discrimination between actual future states and random states into a goal-reaching critic. State-action and goal encoders produce an inner-product score; a separate actor learns actions that increase it. The exponentiated optimal score is proportional to a goal-averaged policy value. Simulated online and offline results support this approach, with restrictive proof assumptions and unavailable appendices limiting verification.
The supplied 18-page conference PDF ends with references and a checklist; the cited appendices are absent. Main-text design comparisons and the camera diagnostic can be illustrated, but the filtering ablation in Appendix Figure 13, detailed variant derivations and implementation configurations cannot be visually verified. Section 5.4 also references a rollout panel absent from its supplied Figure 6. e08e12e13e16
6.1 Read the original evidence
Figure 2. Contrastive outcome learning is competitive across goals, with task-dependent exceptions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the modality labels below the two blocks: state-based observations are on the left and image-based observations on the right. Within each block, match colors using its local legend. Each panel plots success rate against environment interactions, but the powers of ten and total budgets differ across tasks, so equal horizontal distance does not mean equal training cost. Section 5.1 specifies means and standard deviations over five random seeds. The model-based baseline fits the discounted future-state distribution explicitly; it is not the same architecture as the contrastive critic. Compare the difficult Sawyer tasks with ant_umaze to see where the relative ordering changes. e09e10
What it supports. NCE learns on the challenging image Sawyer tasks where the listed baselines show little sustained progress. The state-based ant_umaze result is a clear counterexample to universal dominance: the model-based baseline performs best. The paper associates that exception with the goal being only an XY location, but the plot alone does not isolate dimensionality as the cause.
Where the evidence stops. These are simulated tasks with differing interaction budgets. Section 5.1 reports image Sawyer push/bin below 50%, so relative improvement should not be read as reliable task completion. Exact final means are not tabulated.
Table 1. Offline gains combine contrastive value learning, behavioral cloning and critic ensembles. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a row horizontally before comparing maze sizes. The central “Contrastive RL + BC” columns correspond to two and five learned critics; Section 5.5 takes their minimum for the actor update and adds a goal-conditioned cloning term. The “no TD” heading groups several distinct methods, so it is not a matched architectural control. In large-play-v2, the five-critic result is 48.6 ±4.4 against IQL’s 39.6; in large-diverse-v2 it is 54.1 ±5.5 against 47.5. Keep the bottom footnote attached: the TD3+BC and IQL values were reported on v0 tasks. The supplied text does not explicitly define table normalization or its ± convention. e14e14-largee14-all
What it supports. The five-critic large-maze gains over IQL are 9.0 and 6.6 table points, computed from the displayed values. A contrastive configuration leads five of the six rows, but IQL wins umaze-v2. Increasing the ensemble helps four rows and lowers the central value in two, so the table supports a useful tendency rather than universal improvement.
Where the evidence stops. Baseline results are quoted, baseline uncertainty is absent, and goal relabeling differs. More critics also change capacity and the minimum aggregation. The table cannot isolate a causal benefit of ensemble size or of avoiding TD.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Offline D4RL AntMaze large-play-v2 Static D4RL data; NCE+BC, two versus five critics. Baseline values are quoted. | 2 critics: 41.6 ± 6.0; 5 critics: 48.6 ± 4.4. Reported AntMaze performance; score normalization and table ± convention are unspecified. | GCBC 32.4; IQL 39.6; TD3+BC 0.2. Five critics exceed IQL by 9.0 table points. A substantial reported gain, but IQL/TD3+BC use v0 and no goal relabeling; this does not isolate avoiding TD. e14e14-large |
| Offline D4RL AntMaze large-diverse-v2 Same offline adaptation and table protocol. | 2 critics: 49.3 ± 6.3; 5 critics: 54.1 ± 5.5. Reported AntMaze performance; retain source scale and unspecified ± convention. | GCBC 36.9; IQL 47.5. Five critics exceed IQL by 6.6 table points. The source rounds the two large-task improvements to 7–9 absolute percent. e14e14-large |
| Offline D4RL AntMaze umaze-v2 Same offline table; easiest listed maze. | 2 critics: 81.9 ± 1.7; 5 critics: 79.8 ± 1.4. Reported AntMaze performance. | IQL 87.5; GCBC 65.4. IQL wins. More critics improve four of six rows; umaze and medium-diverse decrease. e14-all |
| Online state- and image-based goal reaching Manipulation/navigation; five-seed mean and standard deviation, task-specific interaction budgets. | NCE leads most comparisons; Section 5.1 reports image sawyer_push/bin below 50%. Success rate. | TD3+HER, GCBC and future-density modeling; the model-based method wins state ant_umaze. These are simulated control results. NCE makes progress on challenging image tasks where the listed baselines show little sustained learning. e09e10 |
| Image representation-learning comparison Four manipulation tasks, TD3+HER plus DrQ, autoencoder or CURL; five seeds. | NCE leads final performance on all four plots; endpoints are not tabulated. Success-rate curves. | Autoencoding helps fetch_reach and DrQ helps sawyer_push. Supports the complete algorithm here; does not exclude further benefits from augmenting NCE. e11 |
| Contrastive estimator and TD variants State/image tasks; NCE, CPC, C-learning and NCE+C-learning, five seeds. | The combination often ranks among the strongest; none dominates every curve. Success-rate curves. | CPC changes the estimator; C-learning changes TD use and policy conditioning. A family comparison, not a controlled attribution to a single design choice. e09e12 |
| Moving-camera sawyer_push Camera follows the hand; the wall initially occludes the puck. | Text reports approximately 35% success; return approaches 0.35. Graph: return; Section 5.4: success rate. | Section 5.4 quotes 75% fixed-camera success, unlike the below-50% image result in Section 5.1; protocol differences are unexplained. Learning in one partially observed setup, with unresolved metric/protocol differences and an absent referenced rollout panel. e13e10 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Changing the control objective outperforms the tested representation add-ons. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the pale blue NCE curve across fetch reach, sawyer push, fetch push and sawyer bin. Then compare it with the dark TD3+HER baseline and the three marked add-ons in the legend. DrQ supplies image augmentation, AE adds reconstruction, and CURL adds augmentation-based contrastive representation learning to the actor-critic baseline. Thus the contrastive label describes two different uses: NCE supplies the value-learning objective, while CURL supplies a representation objective alongside RL. The axes are success rate and environment steps, with task-specific scales. As specified in Section 5.2, the shaded variation summarizes five seeds rather than uncertainty over new task families. e11
What it supports. Each add-on has a context in which it can help: the autoencoder improves fetch reach and DrQ improves sawyer push. Nevertheless, NCE ends above these baselines in all four panels while using no augmentation or auxiliary representation loss. This supports the paper’s proposed organization of learning on the tested image-control suite.
Where the evidence stops. The comparison changes the complete learning algorithm, not just representation quality. It does not establish that augmentation or auxiliary losses would fail to improve NCE itself; the authors explicitly leave that possibility open.
Figure 5. The contrastive interpretation produces a family of useful control algorithms. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legends separately for the state and image blocks, since the NCE colors differ between them. NCE is the simple binary contrastive method; CPC replaces its estimator with an infoNCE-based alternative; C-learning uses temporal-difference learning; and the crossed pale line combines NCE with C-learning. Read both learning speed and late-training behavior rather than selecting a transient peak. The plots use the five-seed mean and standard deviation described in Section 5.3. Section 4.6 also matters: C-learning’s occupancy target retains commanded-goal conditioning, whereas NCE averages experience across goals. Therefore the curves compare related algorithms whose statistical targets are not identical. e09e12e08
What it supports. The combination often performs strongly, particularly on image fetch push, while individual tasks favor other variants. The plot supports expanding C-learning into a broader contrastive family and offers a simplicity/performance tradeoff. It does not show that either the binary estimator or TD learning is uniformly superior across tasks and observation modalities.
Where the evidence stops. This is not a one-variable ablation. TD use, estimator and policy-conditioning target differ, and detailed variant derivations are in missing Appendices C/D. The separate trajectory-filtering ablation referenced in Section 4.5 is also unavailable.
Figure 6. A moving camera still permits learning, with unresolved reporting details. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this curve together with Section 5.4: the camera follows the Sawyer hand at a fixed distance, and a wall initially hides the puck. The horizontal axis reaches roughly five million environment steps. The vertical label is “return,” although the surrounding text calls the outcome a success rate and reports about 35%. The checklist says figure bands show mean and standard deviation over five seeds. This is the complete plotted diagnostic available in Figure 6. Although Section 5.4 refers to a left-hand example rollout, the supplied PDF contains only this curve; no unseen rollout content is inferred here. e13e10e16
What it supports. The reported agent makes progress despite this specific source of partial observability, and the curve stabilizes near the value described in the text. This supports the feasibility of learning with the moving-camera setup. It does not demonstrate that the representation recovers hidden state, remembers occluded objects, or transfers across arbitrary camera motions.
Where the evidence stops. Section 5.4 quotes 75% with a fixed camera, while Section 5.1 gives below 50% for image sawyer_push. The PDF does not reconcile those protocols. The return/success terminology and missing rollout panel should also remain explicit.
7. Analysis & limitations
7.1 What the evidence leaves open
Approximate policy improvement assumes tabular states/actions, a Bayes-optimal critic and trajectory filtering, with tolerance 2 gamma epsilon/(1−gamma). Main experiments omit filtering. Its reported performance penalty and the complete proof cannot be checked because Appendix Figure 13 and Appendix B are absent. e08
Application beyond goal-conditioned RL remains open. Occupancy similarity does not establish general world-model prediction or long-horizon planning capability. e15e05
Offline comparisons mix relabeling, ensembles and quoted protocols. The v0/v2 equivalence claim has no matched rerun here; baseline uncertainty and the table’s ± definition are missing. e14e14-all
7.2 Questions for discussion
- How much practical policy improvement survives the gap between goal-averaged occupancy and the commanded-goal policy?
- Would a fixed-capacity ensemble retain the offline advantage under identical dataset versions and relabeling?
- Which camera protocol explains the differing static-image sawyer_push results?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful run needs absent Appendix E architectures, hyperparameters, preprocessing, success conventions and offline lambda. The body supplies loss code and JAX/Acme throughput on one TPUv2: 1100 batches/s for states, 105 for images, versus 28 for its DrQ. Throughput does not specify total compute or software versions. e07e16
Proposed checks should fix goal sampling, update budgets and evaluation: remove action conditioning to test the critic mechanism, then separate critic count from total capacity on an identical offline dataset version. e05e07e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check the occupancy identity and the role of actions
Reader-proposed, not run: in a small tabular goal-reaching environment, collect a fixed dataset under known goal-conditioned policies. Train NCE with geometric future sampling and compare exp(f) times the empirical marginal goal probability against independently estimated discounted occupancy. Compare action rankings with a control trained on shuffled action labels, keeping state-goal pairs, sample count and fitting budget fixed. Also contrast occupancy averaged over commanded goals with occupancy conditioned on one goal. The stated identity predicts agreement with the averaged quantity as data and fitting improve. Persistent ranking disagreement would flag a sampling/implementation error or a failed assumption; shuffled actions retaining equally accurate rankings would weaken the claimed need for action information in that test. e04e05e06e08
Check 2: Separate offline ensemble count from capacity and protocol
Reader-proposed, not run: use one fixed AntMaze large-play-v2 dataset and the same goal relabeling and evaluation definition throughout. Compare two and five critics first with identical per-critic architectures, then with approximately matched total parameters; add a two-critic capacity control and keep actor updates and training samples fixed. Because the supplied text omits lambda, preregister and report the same BC-weight grid for every condition rather than claiming to recover the paper’s setting. Evaluate multiple seeds with a declared uncertainty statistic. If the five-critic advantage survives the fixed-capacity comparison, it supports an ensemble/aggregation effect; if widening two critics closes it, capacity is a plausible explanation. e14e14-largee14-alle07e16
8.3 Reading coverage
Visual audit: Actually rendered and viewed the title page, all method/equation/algorithm pages, all main experiment pages and the checklist. Inspected Figures 1–6 and Table 1, then viewed all six final original crops. Figure 1 arrows and similarity wording were checked against Equations (6)–(7) and Algorithm 1. Table headers, all values and the version footnote are retained. Figure 6’s absent rollout panel and metric/protocol differences are disclosed. Pages 11–17 contain references read as text; no cited method or numerical claim depends on an unviewed reference page. Referenced appendices were not supplied, and no code or experiments were inspected or run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 18. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1 Introduction (pp. 1–2)
- 2 Related Work (pp. 2–3)
- 3 Preliminaries (pp. 3–4)
- 4 Contrastive Learning as an RL Algorithm, including 4.1–4.6 and Algorithm 1 (pp. 4–7)
- 5 Experiments, including 5.1–5.5 (pp. 7–10)
- 6 Conclusion and acknowledgements (p. 10)
- References (pp. 11–17)
- Checklist (p. 18)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All seven supplied text chunks were read individually. The complete supplied 18-page conference PDF body was reviewed; its referenced Appendices A–G are absent. Appendix B proofs, C/D variant derivations, E implementation details, F/G additional results and failures, and the filtering ablation in Figure 13 were unavailable.
- The extraction limitation was addressed by visually inspecting pages 1–10 and 18, all main figures and Table 1, and all six final crops. Reference pages 11–17 were read as text.
- Identity/version note: the title exactly matches the catalog. The title page credits Benjamin Eysenbach; Tianjun Zhang; Sergey Levine; Ruslan Salakhutdinov. The catalog spells the final author Russ R. Salakhutdinov; the other three credits agree. The inspected artifact identifies NeurIPS 2022 but supplies no revision number/date or revision chain. This report describes this conference PDF only; no edition comparison was available.
- Code, linked videos, external baseline sources and separate supplements were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, venue footer, Abstract and Section 1
Exact title and credits: Benjamin Eysenbach; Tianjun Zhang; Sergey Levine; Ruslan Salakhutdinov. Affiliations: CMU; Google Research; UC Berkeley. Footer: NeurIPS 2022. Introduction frames representation learning as the RL algorithm itself.
Go to primary source ↓e02PDF p. 2, Figure 1 and caption; p. 5, Section 4.4
State-action and goal encoders feed a similarity comparison; the positive comes from the same trajectory and the illustrated negative from another. The caption says closest; Section 4.4 specifies an inner product.
Go to primary source ↓e03PDF p. 3, Section 3, Equations (1)–(3) and footnote 4
Defines goal reward as transition probability/density, discounted return/value and occupancy. Initial-state reward adds initial goal probability. The reward supports analysis rather than being estimated.
Go to primary source ↓e04PDF p. 4, Section 3 continuation and Section 4.1, Proposition 1 / Equation (5)
Geometric sampling gives discounted occupancy. Combining commanded goals defines an averaged policy. Goal-conditioned Q equals corresponding discounted goal occupancy.
Go to primary source ↓e05PDF pp. 4–5, Section 4.2, Equation (6), Lemma 4.1 and following paragraphs
Replay state-action pairs are contrasted with conditional future states and marginal negatives. The optimal log density-ratio score exponentiates to goal-averaged-policy Q divided by marginal goal density.
Go to primary source ↓e06PDF p. 5, Sections 4.3–4.4, Equation (7)
The neural actor takes state and goal, samples actions and maximizes critic scores through reparameterization gradients. Image tasks add entropy. Training alternates critic, actor and collection.
Go to primary source ↓e07PDF p. 6, Algorithm 1, Section 4.4 continuation and final summary paragraph
Batch dot products use identity-matrix binary labels; actor maximizes paired logits. NCE is theoretically on-policy but reuses transitions. It needs no target network, multiple Q values or auxiliary image machinery. JAX/Acme and single-TPUv2 throughput are stated; architecture/hyperparameters are deferred to Appendix E.
Go to primary source ↓e08PDF p. 6, Section 4.5, trajectory exclusion rule and Lemma 4.2
Assumes tabular states/actions, Bayes-optimal critic and epsilon filtering of trajectory likelihood ratios; approximate improvement allows loss 2 gamma epsilon/(1−gamma). Main experiments omit filtering. Appendix B and Figure 13 are referenced but absent.
Go to primary source ↓e09PDF p. 7, Sections 4.6 and 5.1, Figure 3 and footnote 8
C-learning uses TD and commanded-goal policy occupancy, unlike NCE averaging. Baselines: TD3+HER, GCBC, normalized future-density modeling. Figure 3 shows simulated task environments.
Go to primary source ↓e10PDF p. 7, Figure 2; p. 8, Section 5.1 Tasks and results paragraphs
State/image manipulation, image point navigation and 111-dimensional ant locomotion with XY goals are evaluated. Five-seed means/standard deviations are plotted. NCE leads most tasks, model-based wins state ant_umaze, and image Sawyer push/bin are described as below 50%.
Go to primary source ↓e11PDF p. 8, Figure 4 and Section 5.2; p. 9, Section 5.2 continuation
Four image-task curves compare NCE with TD3+HER plus DrQ, AE or CURL. NCE leads final performance without augmentation/auxiliary objectives; authors do not rule out further representation-learning benefits.
Go to primary source ↓e12PDF p. 9, Figure 5 and Section 5.3
NCE, CPC, C-learning and their combination are compared with five-seed mean/standard deviation curves. The combination often ranks strongly, with task-dependent exceptions. Appendices C/D derivations are absent.
Go to primary source ↓e13PDF p. 9, Section 5.4 and Figure 6
Camera follows the hand; the puck is initially occluded. Text reports about 35% success and 75% with fixed camera. The graph is labeled return. The referenced left rollout panel is absent from the supplied figure.
Go to primary source ↓e14PDF p. 10, Section 5.5, offline actor objective and Table 1 footnote
Offline NCE adds lambda-weighted goal-conditioned BC and uses the minimum of two or five critics. Quoted TD3+BC/IQL results use v0 while table tasks use v2; footnote claims negligible TD impact. Normalization, lambda value and table ± definition are not specified.
Go to primary source ↓e14-largePDF p. 10, Table 1, large-play-v2 and large-diverse-v2 rows, GCBC / Contrastive RL+BC / IQL columns
Large-play: GCBC 32.4, two critics 41.6 ±6.0, five 48.6 ±4.4, IQL 39.6, TD3+BC 0.2. Large-diverse: GCBC 36.9, two 49.3 ±6.3, five 54.1 ±5.5, IQL 47.5. Five-critic minus IQL differences are 9.0 and 6.6 table points.
Go to primary source ↓e14-allPDF p. 10, Table 1, all rows; particularly umaze-v2 and medium-diverse-v2
Umaze: two critics 81.9 ±1.7, five 79.8 ±1.4, IQL 87.5, GCBC 65.4. Medium-diverse decreases from 72.5 ±2.8 to 71.5 ±1.3. Five critics improve four rows. A contrastive configuration exceeds all baselines on five rows. Baseline uncertainty is absent.
Go to primary source ↓e15PDF p. 10, Section 6 Conclusion
Application to arbitrary RL objectives and stronger contrastive algorithms remain open; authors limit this work to goal-conditioned RL.
Go to primary source ↓e16PDF p. 18, Checklist items 2(b) and 3(a)–(d)
Proofs are deferred to Appendix B and training details to E. Code is promised upon acceptance. Figure error bars are five-seed mean/standard deviation. Compute answer points to TPUv2 throughput.
Go to primary source ↓8.5 Primary sources
Contrastive Learning as Goal-Conditioned Reinforcement Learning ↗
PDF · 10,892 extracted words
Source fingerprint
5e01faa569d2d6f9acd41b1b4260486d05f85ae60bd4581ccff23038246c0008