Deriving Success Guided Sampling

2.1The goal-conditioned MDP

A Markov decision process (MDP) is the formal game RL plays. Goal-conditioned means the policy is also told what to achieve, so one policy can serve many goals.

One episode: draw \((s_0,g)\sim\rho\). At every step \(t\) the policy picks \(a_t\sim\pi(\cdot\mid s_t,g)\), physics produces \(s_{t+1}\sim P(\cdot\mid s_t,a_t)\), and the robot receives \(r(s_t,a_t,g)\). The objective is the expected discounted return:

Read it from the inside out. Inside the brackets: the rewards of one episode, each multiplied by \(\gamma^t\) so later rewards count less. Outside: the average over many episodes, where the randomness comes from which start and goal \(\rho\) draws and from the policy's random actions.

2.2Task configurations

The paper's unit of choice is a task configuration:

Before training, \(N\) configurations are drawn once and frozen. SGS only ever chooses among these \(N\).

2.3Policy gradients, GAE and PPO

All RL training in the paper uses PPO; the RGB students are distilled with DAgger (§7.3). Four formulas carry the whole algorithm.

Batch bookkeeping. One PPO iteration collects \(E\) environments \(\times\ n\) steps, splits the batch into \(M\) minibatches, and runs \(K\) epochs over them: \(M\times K\) gradient steps per iteration.

2.4Why a 0% or 100% configuration teaches nothing

This is the paper's whole argument in one derivation.

• At \(p=0\) or \(p=1\) the advantage is zero on every episode, so that configuration adds exactly zero task gradient.
• The signal is largest at \(p=\tfrac12\), where \(\mathbb{E}|A|=\Delta R/2\).
• \(p(1-p)\) is the variance of a coin that lands heads with probability \(p\). A configuration is useful exactly when its outcome is uncertain.

2.5The Beta distribution, as a weighting curve

The Beta distribution lives on \([0,1]\), the same range as a success rate:

\(B(\alpha,\beta)\) only makes the area equal 1. SGS never needs it, because the softmax in §2.6 normalizes anyway.

Width, for \(t^\star=0.5\), from the Beta variance \(\alpha\beta/[(\alpha+\beta)^2(\alpha+\beta+1)]\) with \(\alpha=\beta=1+\kappa/2\):

\(\kappa=0\) gives \(1/\sqrt{12}\approx0.289\), the standard deviation of a uniform variable on \([0,1]\), as it should. \(\kappa=1\) gives 0.250, \(\kappa=5\) gives 0.177, \(\kappa=10\) gives 0.139.

2.6Softmax with a temperature

A softmax turns scores \(\ell_i\) (logits) into probabilities. The temperature \(T\) flattens it:

• In the interior, \(\kappa\) and \(T\) are one knob. Only \(\kappa/T\) matters. This is why the authors sweep \(\kappa\) and leave \(T\) at 2; the paper says \(T\) is "tied to" \(\kappa\).
• At the floor, \(T\) matters separately. Floored configurations get \((2\epsilon)^{1/T}\), so the peak-to-floor ratio is \(\big((w_{\text{peak}}+\epsilon)/2\epsilon\big)^{1/T}\). A larger \(T\) flattens the curve and lifts the floor.
• The implementation uses \(T_{\text{eff}}=\max(T,1)\). The softmax can only flatten the raw kernel, never sharpen it.

2.7Estimating a success rate from \(H\) outcomes

Each configuration keeps its last \(H\) outcomes \(o_{i,1},\dots,o_{i,H}\in\{0,1\}\) and estimates

The policy keeps changing, so the window mixes old and new behavior. A long window means less noise but more lag. That trade-off matches the paper's sweep: \(H=25\) scores 0.39, \(H=100\) scores 0.49, \(H=400\) scores 0.47 (two seeds each, §6.1).

3.1The loop

before training
  draw N configurations τ_1 … τ_N from the task distribution     (frozen for the run)
  give each configuration a ring buffer of its last H outcomes

whenever an environment finishes an episode on configuration i
  push 1 (success) or 0 (no success) into buffer i
  p_i ← mean of buffer i
  w_i ← (p_i + ε)^(κ·t*) · (1 − p_i + ε)^(κ·(1 − t*))
  ℓ_i ← log( max(w_i, ε) + ε )
  this environment's next configuration:  j ~ softmax( ℓ / max(T, 1) )

What SGS does not use: no value function, no learned model, no extra evaluation rollouts, no difficulty ordering. Its only signal is the success bit each training episode already produces.

3.2Equation 1: the kernel

The paper's Eq. 1, with the target written \(t^\star\):

The implementation (Appendix B.2) adds a floor before the log and a minimum temperature:

The floor guarantees every configuration a minimum weight. That keeps impossible and mastered configurations in rotation, so their estimates get refreshed when the policy changes.

3.3Equation 2: from weights to a draw

At the end of every episode, that environment's next configuration is drawn independently from

With \(T=2\) this is a square root of the floored weight, renormalized. A configuration's probability depends on its own success rate and, through the denominator, on everyone else's.

3.4The numbers the paper uses

Relative sampling rate per configuration, with the peak set to 1:

3.5Explorer: shape the kernel yourself

The curve is \(q(p)\) from Eq. 1 and 2, sampled at the 101 values an \(H=100\) window can take. The dashed line is \(p(1-p)\) and the dotted line is uniform sampling, each scaled to peak at 1.

3.6The two training decisions around the sampler

4.1Locomotion (one reward, all 13 terrains)

\(\odot\) is the elementwise product, \(\lVert x\rVert_1=\sum_j|x_j|\), \(\mathbf{1}[\cdot]\) is 1 when the condition holds and 0 otherwise, superscript \((b)\) means "in the body frame". \(F_{\min}\) is not given.

4.2Manipulation (one reward, every assembly task)

5.1Tasks and configuration spaces

Locomotion. ANYmal-D with system-identified actuator models. Thirteen terrain types: slopes, pits, gaps and stairs from earlier work; radiating beams, balance beams and stepping stones; harder procedurally generated terrains (navigation mazes, contoured terrain, floating islands joined by narrow stepping stones or jump boxes at different heights); and an inverted slope, which Fig. 3 omits for space. One policy is trained from scratch on all of them, where much prior work trains per-terrain specialists and distills.

Manipulation. NIST Assembly Task Board tasks, one policy per task.

Franka needs 20/8 = 2.5 times fewer physics steps per policy step, which the paper credits for part of its faster training.

Reset buffer. Each manipulation task pre-collects 32,768 collision-checked configurations, balanced across three reset families from OmniReset (about 10,900 each):

5.2The baselines

5.3Protocol

5.4Scaling results (Fig. 5)

Mean success over three seeds. Exact values are from the text and Table 1; values marked ≈ I read from the point coordinates in the paper's SVG plot files (about ±0.01).

UR5e rod-in-hole, Fig. 5c (an easier task): at 32K, SGS succeeds while uniform and PLR score zero; at 256K all methods reach about 98%. The plot image is broken in arXiv's HTML build, so I have only the values the text states.

5.5Against SFL (Table 1)

Locomotion success, mean ± 95% CI over three seeds.

• 4K: the intervals do not overlap (SGS 0.437 to 0.475; SFL best 0.274 to 0.304). A clear win.
• 32K: SGS spans 0.381 to 0.607 and SFL best spans 0.389 to 0.453. They overlap, so this comparison cannot separate the two.
• SFL decays. Its final score at 32K (0.051) is far below its best (0.421). The paper reports the decline but does not explain it.
• "Best" is the maximum over checkpoints, which flatters SFL. Comparing SGS's final score against it is conservative toward SGS.

5.6What SGS samples over time (Fig. 7)

On a floating-islands terrain, the figure shows where sampling mass sits early, midway and late in training. Early, it is spread uniformly. Midway, it concentrates on jumps between islands at the same height. Late, it moves to configurations that involve the top of the tall central pillar.

5.7Six UR5e tasks (Table 6)

Success (%) from Reaching resets under full gravity. No seeds or intervals are given.

• On the five insertion tasks the three samplers are within about 2.6 points of each other, except PLR on the peg. The paper calls SGS comparable to uniform; without intervals, gaps this small cannot be ranked.
• The nut is the only large gap: 90 versus 49 and 60. The authors' hypothesis is that less forgiving contact and a longer horizon make focused practice matter more.
• UR5e nut (90%) beats Franka nut (70%). The authors credit mostly the control interface: OSC lets the policy command Cartesian motion directly, while joint-position control makes it coordinate seven joints itself. Embodiment, gripper and policy rate also differ.

6.1Hyperparameter sensitivity (Table 3)

Locomotion, 32K environments, one knob changed at a time, two seeds per setting. Nominal settings in bold.

• Noise yardstick. SGS at 32K over three seeds is 0.494 ± 0.113 (Table 1). Most differences in this table (0.05 or less) sit well inside that. The drops that stand out are \(H=25\) (.39), \(n_{\text{patch}}=10\) (.38) and \(t^\star=.80\) (.42); the \(H\) and \(t^\star\) rows have two seeds, and the patch rows' seed count is not stated.
• Kernel shape barely matters. A Normal-shaped kernel scores .45 against the Beta's .46. The paper does not give the Normal kernel's exact form.
• Manipulation sweep. \(\kappa\in\{1,5,10\}\) by \(t^\star\in\{.33,.5,.66\}\), \(T=2\), 32K environments; \(\kappa=1,\ t^\star=0.5\) won "by a small margin". Its numbers are not reported.

6.2Which resets matter (Table 4)

UR5e rod-in-hole retrained on two of the three reset families, three seeds each. Success (%) on each family:

• Without Near-Goal, nothing is learned. If no configuration ever succeeds, every \(p_i\) stays at 0, every configuration gets the same weight, and SGS degenerates to uniform sampling over hopeless starts.
• Without Stable Grasp, one of three seeds collapsed and two stayed high, which drags the means to 63.8 and 81.3.
• Without Reaching, the other two families are unaffected. No Reaching score is reported for that run.

6.3The gravity curriculum (Table 5)

The main manipulation runs, including the Franka scaling study, also used a gravity curriculum: training starts at zero gravity, and gravity steps toward its normal value whenever success exceeds 80% at the current level. Afterwards, one run at 1M environments without it:

7.1Hardware and the board

• UR5e with a Robotiq 2F-85, OmniReset's relative Cartesian operational-space control, one third-person camera and one wrist camera (OmniReset used three cameras).
• The NIST board is polished acrylic. Facing it, the wrist camera saw the robot's own reflection, an image it never saw in training. The board's low friction also broke strategies that push objects against it to reorient them. The fix was covering the board with tape.
• The board is fixed to the table with command strips. Simulation treats the receptacles as static, so a board that shifts on contact causes failures.

7.2Gripper pad friction: the "glue grasp" trap

• OmniReset gave the pads \(\mu=100\) with the max combine mode. When two surfaces touch, a simulator needs one coefficient for the pair; max takes the larger of the two. Training was easy, and the authors found this a major source of sim-to-real gap.
• Lowering friction removed the fake grasps but stopped learning altogether. A rod-in-hole sweep from scratch at \(\mu\in\{0.5,1,1.5,2\}\) (Fig. 9, measuring grasp-in-air success) trained stably only at \(\mu=2\). Every run uses \(\mu=2\).

7.3Teacher, student, and DAgger

The critic only exists during training, so it may see information the deployed robot never will. That is called an asymmetric actor–critic.

Why a point-cloud teacher. A teacher that reads exact object poses can act on information no camera provides. Point clouds are geometry, closer to what a student can infer from images, so the teacher's behavior is easier to imitate.

• Why hold out environments. In online DAgger, the student was just updated on the teacher's label for almost the same state one step earlier, so training success is inflated. The paper reserves 6.67% of environments (one in 15) that never contribute gradients. On the nut, held-out success lags training success early and catches up later (Fig. 10).
• Distillation takes about two days on four L40S GPUs with 480 environments each: 1,920 environments, so the held-out share is 128 (32 per GPU), assuming the split is per GPU or global.
• Visual randomization and image augmentation follow OmniReset.

7.4Symmetry-aware resets and rewards

Resets randomize each object over its equivalent orientations: discrete symmetries are sampled uniformly, and continuous ones uniformly over \([0^\circ,360^\circ)\). The success test accepts any equivalent goal orientation.

7.5Hardware results (Table 2)

Simulation numbers come from Reaching resets. First-try counts are back-computed from the percentages, and each comes out a whole number.

• The gap grows with difficulty: 2 points on the gear, 32 on the rod, 52 on the nut (student sim minus real).
• Retries do much of the work. First-try successes are 6 of 18 successes on the nut and 14 of 30 on the rod, but 38 of 47 on the gear.
• Throughput. Successes divided by total time, resets included. The runs lasted about 83, 42 and 15 minutes: 18/83 = 0.22, 30/42 = 0.71, 47/15 = 3.13, close to the reported 0.23, 0.71 and 3.08; the nut's 0.23 implies about 78 minutes rather than 83.
• The simulation teachers are the SGS policies of Table 6 (90.04, 95.70, 96.09 match exactly).
• Metrics follow OmniReset, with one change: re-grasps before the first insertion attempt are not counted as retries. Objects start at uniformly random positions on the board.

7.6Transfer lessons

The paper's three limitations

1. Success estimates live on a discrete set of configurations. Continuous or much larger task spaces would need a learned or hierarchical score.
2. SGS samples from a distribution; it does not build one. Coverage comes from reset generators like OmniReset, and choosing which configurations to offer is open.
3. A sim-to-real gap remains.

Inconsistencies I found

How much the numbers can carry

• Three seeds for the main results, two for the sensitivity table, one for the gravity ablation, and no intervals at all for Table 6.
• Real-robot results rest on about 50 trials each, so their 95% intervals span 14 to 26 points (§7.5).
• SFL is also reported at its best checkpoint, and it ran with its own paper's defaults while SGS had a small grid search.

Not in the paper, so not reproducible from it alone

PPO settings (\(\gamma\), \(\lambda\), \(\epsilon_{\text{clip}}\), epochs, rollout length, minibatch count, learning rate), episode lengths, PLR's \(\beta\) and staleness weight, how an empty buffer is scored, the Normal kernel's form, the gravity step size, the locomotion success tolerance, \(F_{\min}\), and the simulator.