1The game
What does the robot see, what does it do, and what does it get back?
In simulation the arm acts ten times a second. Each of those ticks it gets a snapshot of the world and has to answer with a move. Then the simulator tells it how good that moment was. RL papers call a tick a time step; section 7 sorts out the other things the code calls a step.
| It sees | a 16 × 16 colour picture from the camera on its wrist, and 12 numbers: its 6 joint angles and the 6 joint targets (where each joint has been told to go) |
|---|---|
| It does | 6 numbers between −1 and 1, one per joint: how far to move that joint's target this tick. All the way to 1 means 0.1 radian, about 6°; the gripper gets twice that |
| It gets | a reward: one number that scores the moment, at most 1 per tick |
The 12 numbers: where each joint is, and where it's going
Six of the numbers are joint angles: where each joint is right now, as measured. In simulation each reading gets 5° of random noise, because a real servo's reading is never exact. The other six are joint targets: where each joint is being pulled to. Between the two sits the motor, acting like a spring: the bigger the gap, the harder it pulls.
Nobody hands the robot its targets; it builds them itself. When an episode starts, each target is set to where its joint already is. Then every tick, the actor's 6 numbers become small changes, added to the targets:
So a target is the starting pose plus the running total of every move the policy has made, like a cursor moved with the arrow keys while the arm chases the cursor. Try it on one joint:
In our simulated run the actor told the gripper to close, over and over:
| tick | actor's gripper a | target | angle, ±5° noise | gap |
|---|
The target falls by about 11° a tick (0.2 × a radians). By tick 8 it has passed −10°, where the jaws are fully closed, so the joint stops while the target keeps going: by tick 12 the gap is , and the motor is squeezing as hard as that gap says. Squint's simulator never clips targets, so they can wind up far past a joint's limit; on our real arm the goal sent to each servo is capped at 8° ahead of where the joint is (5° for the gripper). Every move on the real arm is also scaled to 15% first, so the target walks ahead in steps of 0.86° while the joint catches up:
| step | actor's pan a | sent, 0.15 × a | target | measured |
|---|
That is why the policy gets both. The target is the exact sum of its own outputs, its record of what it has commanded. The angle is where the arm actually got to, measured and lagging. And the gap between them is the policy's only hint about motion, since it sees no speeds and a single picture, and about pushing against something.
Our running example is ReachCube: bring the gripper to the cube, close it, hold still. Its reward adds up how close the gripper is (0 to 1), how closed it is (0 to 1), and 1 more once it's at the cube. Success, meaning at the cube, closed and still, is worth 4; touching the table costs 3. Everything is divided by 4, so one tick is worth at most 1.
In the simulated run on this page the rewards went 0.25, 0.29, 0.33, 0.37, 0.42, 0.70, 0.73, and from tick 7 on, 1.00 every tick: success, held. An episode lasts 50 ticks, five simulated seconds, then the scene resets.
Four words carry the rest of the page:
- observation (or state): what it sees
- action: what it does
- reward: the score for one tick
- policy: its habit, the rule that turns what it sees into what it does
Nobody shows the robot what to do. The only teacher is the reward, and learning means changing the policy so that more reward comes in. That is reinforcement learning.
Does the robot see everything it needs?
Not quite. In one tick it gets a 16 × 16 picture and the 12 numbers, and no memory of earlier ticks. It can't see the cube once the cube leaves the camera's view, its own speed, its exact joint angles (those 5° of noise), the friction and sizes the simulator randomizes each episode, or how many of the 50 ticks are left.
Squint treats what the robot sees as if it were everything, the usual shortcut in visual RL; the paper itself calls the problem partially observable (section III). It mostly works: the 12 numbers carry nearly all of the arm's own state, and the camera sees the cube at the moments that decide the task. It fails when the cube leaves the view, because nothing remembers it. Our arm hit exactly that: in one ReachCube run the cube first came into view at step 34 and the gripper ended off it; in one PlaceCube run, with the cube out of view, the gripper closed on the bin wall.
The mathsThe game, written as a Markov decision process
Formally the game is a Markov decision process: states \(\mathcal S\), actions \(\mathcal A\), a transition rule \(p(s'\mid s,a)\) that the simulator applies and the robot never sees, a reward \(r(s,a)\), and a discount \(\gamma\). A policy is a probability distribution over moves for each state, \(\pi(a\mid s)\), and the goal is the policy with the largest expected discounted return:
\[ \pi^\star = \arg\max_\pi\; \mathbb E_\pi\Big[\sum_{t\ge 0}\gamma^t\, r(s_t,a_t)\Big]. \]Ours: \(s\) is the 16 × 16 × 3 picture and the 12 joint numbers, \(a\in[-1,1]^6\), \(-1\le r\le 1\), \(\gamma = 0.9\). Strictly the robot gets an observation, not the full state (the camera misses things, the joint readings are noisy), so the problem is partially observable. Squint, following DrQ, encodes the picture and treats it with the 12 numbers as the state (Squint, section III).
"Markov" means the next state depends only on the current state and move, not on the history. That is why the 6 joint targets are in the state: each move is added to the previous target, so with joint angles alone two identical-looking moments could lead to different next moments.
The targets integrate the moves. With \(\Delta = 0.1\) rad (0.2 for the gripper) and \(q_0\) the angles at reset,
\[ q^{\text{target}}_{t} \;=\; q_0 + \Delta\sum_{kWhat partial observability changes. In a POMDP the policy gets an observation drawn from the hidden state, \(o_t\sim\mathcal O(\cdot\mid s_t)\). If two states produce the same observation, a critic \(Q(o,a)\) can only learn their average, weighted by how often the current policy visits each, so its value depends on the policy and the MDP guarantees of sections 4 and 6 no longer apply. The standard fixes add memory: stacking the last few frames (DQN stacks 4; Mnih et al., 2015) or a recurrent network (DRQN; Hausknecht and Stone, 2015). Squint does neither: each tick's input is one picture and the 12 numbers.
The codethe reward, the state and the move
reaching_reward = 1 - torch.tanh(5 * tcp_to_goal_dist) reward = reaching_reward reward += (gripper_max - qpos_gripper) / (gripper_max - gripper_min) reward += info["is_reached"].float()
reward[info["success"]] = 4 reward -= 3 * info["robot_touching_table"].float() reward -= 1 * info["robot_touching_item"].float() return self.compute_dense_reward(...) / 4
qpos = qpos + torch.randn_like(qpos) * robot_qpos_noise_std obs = dict(noisy_qpos=qpos) obs.update(controller=controller_state) # the 6 targets
self._target_qpos[self.scene._reset_mask] = self.qpos[self.scene._reset_mask]
self._target_qpos = self._target_qpos + action
scaled_action = np.clip(action * args.action_scale, -1, 1) # action_scale = 0.15
envs/reach.py L374–400 · envs/place.py L413–423 (5° noise, L26) · ManiSkill pd_joint_pos.py L72 (reset) and L86–87, scale 0.1 rad (0.2 gripper) from envs/robot/so101.py L96–109 · 100 Hz physics, 10 Hz policy (envs/base_random_env.py L155) · 50 ticks per episode (reach.py L405) · deploy.py L539 (×0.15 on the real arm) · no frame stacking anywhere in train_squint.py: each tick's input is one picture
Check yourselfWhy are the 6 joint targets part of the state, not just the 6 angles?
Each move is added to the previous target, not to the measured angle. Two moments with the same angles but different targets lead to different next moments, so without the targets the state would not be Markov.
Check yourselfStarting at 60°, the actor gives the gripper −0.5 three ticks in a row. Where is the gripper's target now?
Each tick moves it by 0.2 × −0.5 = −0.1 rad, about −5.7°, so the target is about 60 − 17.2 = 42.8°, wherever the jaws actually are.
2Keeping score
How do you judge a move whose payoff comes later?
Closing the gripper early earns little right away but makes success possible a few ticks later. So a move should be judged by everything that follows it, not by the next reward alone. Add up all the rewards from now on: that total is called the return.
One adjustment: rewards further in the future count a little less. Each tick further away is worth 0.9 times the one before. That factor is the discount, written γ (gamma). It keeps the total finite, and it makes sooner better than later.
With γ = 0.9 the largest possible return is 1 + 0.9 + 0.81 + … = 10: full reward every tick, forever. Keep that 10 in mind. Every value on this page lives on that scale, and a robot holding a success sits near it.
The mathsthe return, and why it can't exceed 10
The return from tick \(t\) is \(G_t = \sum_{k\ge0}\gamma^k\, r_{t+k}\). If every reward satisfies \(|r|\le r_{\max}\), the geometric series bounds it:
\[ |G_t| \;\le\; \sum_{k\ge 0}\gamma^k\, r_{\max} \;=\; \frac{r_{\max}}{1-\gamma}. \]ReachCube's reward lies between −1 and 1 (the worst case is touching the table and the cube at once, \(-4/4\)), so with \(\gamma = 0.9\) every return lies between −10 and 10. A reward \(k\) ticks ahead is weighted \(\gamma^k\): 0.35 at \(k = 10\), 0.04 at \(k = 30\). The number \(1/(1-\gamma) = 10\) also reads as how many ticks the robot effectively looks ahead.
The codeone number
gamma: float = 0.9
train_squint.py L146, lower than the 0.99 common in other benchmarks; it suits 50-tick tasks.
Check yourselfA reward of 1 arrives 20 ticks from now. What is it worth to the robot today?
0.920 ≈ 0.12.
3The critic
How can the robot know a return before it has happened?
It learns to forecast it. The critic is a neural network that answers one question: if I'm here and make this move, then keep playing the way I usually do, what return should I expect? Its answer is called Q, for quality. (The same forecast without choosing the first move is called V.)
Think of the evaluation bar on a chess engine. It reads the position and says who's ahead long before the game ends. The critic is the robot's evaluation bar, measured in reward.
Hover or tap the chart to read a tick.
The critic sees what the robot sees, plus the move being judged. Inside, the picture goes through a small camera network (the encoder: two convolution layers) and is squeezed to 50 numbers; the 12 joint numbers are spread out to 256; the 6-number move is attached; three more layers turn all of that into the forecast.
The mathsvalue functions and the Bellman equation
For a policy \(\pi\), the value of a state, and of a state with a first move, are expected returns:
\[ V^\pi(s) = \mathbb E_\pi\big[G_t \mid s_t = s\big],\qquad Q^\pi(s,a) = \mathbb E_\pi\big[G_t \mid s_t = s,\ a_t = a\big],\qquad V^\pi(s) = \mathbb E_{a\sim\pi}\big[Q^\pi(s,a)\big]. \]Split the first reward off the return and the infinite sum becomes recursive:
\[ \begin{aligned} Q^\pi(s,a) &= \mathbb E\big[r_t + \gamma\,(r_{t+1} + \gamma\, r_{t+2} + \dots)\ \big|\ s_t = s,\ a_t = a\big]\\ &= r(s,a) + \gamma\,\mathbb E_{s'\sim p(\cdot\mid s,a)}\Big[\mathbb E_\pi\big[G_{t+1}\mid s_{t+1} = s'\big]\Big]\\ &= r(s,a) + \gamma\,\mathbb E_{s'\sim p}\big[V^\pi(s')\big]. \end{aligned} \]That is the Bellman equation (Bellman, 1957): one real step, plus the value of where it lands. Sections 4 to 9 are all ways of solving it from samples, without knowing \(p\).
Why \(Q\) and not only \(V\): choosing between moves with \(V\) alone needs \(p\), to know where each move lands. \(Q\) has the consequences folded in, so moves can be compared without a model of the physics.
SAC's critic forecasts the soft \(Q\), which also counts future randomness (section 6). In our run the randomness terms add up to at tick 0, which is why the chart's "what happened" line includes them.
The codethe critic's wiring
nn.Conv2d(3, 32, 4, stride=2), nn.ReLU(), nn.Conv2d(32, 64, 4, stride=1), nn.ReLU(), nn.Flatten()
self.rgb_proj = nn.Sequential(nn.Linear(n_obs, 50), nn.LayerNorm(50), nn.Tanh()) self.state_proj = nn.Sequential(nn.Linear(n_state, 256), nn.LayerNorm(256), nn.ReLU())
proj = self.proj(rgb_features, state) x = torch.cat([proj, actions], dim=-1) return torch.vmap(self._vmap_q, (0, None))(self.q_params, x)
train_squint.py L249–253 (encoder, 34,400 weights), L268–280 (projection), L392–396 (forward), L377–384 (each head: three layers of 512 with layer normalization). Critic total, both heads: 1,536,096 weights.
Check yourselfTo compare two moves without a model of the physics, do you need V or Q?
Q. It already includes what each move leads to; V would need the transition rule to know where each move lands.
4How the critic learns
How do you train a forecaster of the future?
The obvious way is to wait. Play the episode out, add up what actually happened, and correct the forecast toward that. It works (it's called Monte Carlo), but it's slow, because you wait for the end, and noisy, because one episode's luck decides the answer (Finn, slide 19).
The trick reinforcement learning uses instead: the return from now equals the reward you get now, plus 0.9 times the return from the next tick. After one tick you already know the first part. For the second part, ask the critic itself about the next tick. That gives a target that contains one more real reward than the forecast did (Finn, slide 18):
Then nudge the forecast a little toward the target. This is called bootstrapping, or temporal-difference learning: it learns from the difference between two forecasts made one tick apart. Here is one real moment from our run, tick 3, mid-reach:
| the critic's forecast for tick 3, for the move the arm made | |
| the reward that move earned | |
| the slow copy's forecast for tick 4, for a fresh move the actor draws there (section 8.1) | |
| the randomness term for that fresh move (section 6) | |
| target = − + 0.9 × | |
| the gap, target minus forecast |
The forecast was a little low, so this update raises it, but only by a small part of the gap: each update averages over 512 moments and takes a small step. The gap is small because this critic is already trained. Early in training the forecasts are near zero and the gaps are large.
Each target holds one real reward. Repeat this over millions of moments and the real rewards seep backward through the chain of forecasts, from the ticks that earn them to the moves that set them up.
Here is that seeping, step by step, on six moments in a row where only the last one pays:
Why does repeating this settle, instead of wandering? Because each backup pulls any two guesses toward each other by the factor 0.9. Two very different starting guesses end up in the same place:
Notice the catch, though: the target is built from the critic's own guess. If that guess is wrong, so is the target. Section 8 is about exactly this.
The mathsthe Bellman operator, and why repeating it converges
Turn the Bellman equation into an operator that takes any guess \(Q\) and returns a new one (soft form, section 6):
\[ (\mathcal T^\pi Q)(s,a) = r(s,a) + \gamma\,\mathbb E_{s'\sim p,\ a'\sim\pi}\big[Q(s',a') - \alpha\log\pi(a'\mid s')\big]. \]Claim: it brings any two guesses closer by the factor \(\gamma\), in the worst-case distance \(\|Q_1 - Q_2\|_\infty = \max_{s,a}|Q_1(s,a) - Q_2(s,a)|\).
- Subtract the two outputs. The reward and the randomness term don't depend on the guess, so they cancel: \[(\mathcal T^\pi Q_1 - \mathcal T^\pi Q_2)(s,a) = \gamma\,\mathbb E\big[Q_1(s',a') - Q_2(s',a')\big].\]
- An average is never further from zero than the largest thing averaged: \(\big|\mathbb E[X]\big| \le \mathbb E|X| \le \max|X|\).
- So \(\big|(\mathcal T^\pi Q_1 - \mathcal T^\pi Q_2)(s,a)\big| \le \gamma\,\|Q_1 - Q_2\|_\infty\) at every \((s,a)\), hence \[\|\mathcal T^\pi Q_1 - \mathcal T^\pi Q_2\|_\infty \le \gamma\,\|Q_1 - Q_2\|_\infty. \quad ∎\]
A map that shrinks every distance by a fixed factor has exactly one fixed point, and repeating it from any start reaches that point with the error shrinking like \(\gamma^k\) (Banach's fixed-point theorem). The fixed point is \(Q^\pi\), because \(Q^\pi\) satisfies the equation. The second widget above is this proof, run: the gap is \(20\times 0.9^k\).
Temporal-difference learning is the sampled, approximate version. One moment \((s,a,r,s')\) and one fresh move \(\tilde a'\sim\pi(\cdot\mid s')\) give a one-sample target
\[ y = r + \gamma\big(\bar Q(s',\tilde a') - \alpha\log\pi(\tilde a'\mid s')\big), \]and the critic's weights \(\theta\) take a gradient step on \(L(\theta) = \tfrac12\,\mathbb E\big[(Q_\theta(s,a) - y)^2\big]\), with \(y\) held fixed:
\[ \theta \leftarrow \theta + \eta\;\mathbb E\big[(y - Q_\theta(s,a))\,\nabla_\theta Q_\theta(s,a)\big]. \]The gap \(y - Q_\theta\), the TD error, sets the sign and size of the nudge (+0.063 in our example); the learning rate \(\eta = 3\times10^{-4}\), with Adam, keeps each step small. Squint swaps the squared error for a cross-entropy on histograms (section 9.1); the logic is the same.
The worked numbers, exactly. The randomness term is −0.9 × α × log π(ã′ | s′) = −0.9 × × . It is negative here because the actor was fairly sure of that move (log π above 0); for an unsure move it would add a little. The fresh move is a random draw, so another draw would give a slightly different target; every update draws again.
Which networks: the target uses the slow copy, as in training. The tick-3 forecast should come from the live critic, but the checkpoint saves only the slow copy (train_squint.py L878), so both forecasts here come from it; by the end of training the live critic is a few dozen updates ahead of it (section 8.1). These are head 1's numbers. Head 2: forecast , slow copy , target , gap .
In Squint the forecast is a histogram (section 9.1), so the update pulls the critic's histogram for (s, a) toward a target histogram with cross-entropy, and the average follows. For head 1 here the cross-entropy is , against a floor of (the target's own entropy), so is all that learning can still remove at this moment.
Monte Carlo targets (the actual return \(G_t\)) are unbiased but carry the variance of a whole episode; TD targets have low variance but inherit the critic's errors through \(\bar Q\) (Finn, slides 19–20). Squint uses the 1-step TD target. And the proof above assumes exact expectations and a table that can hold any \(Q\); a neural network trained on samples gets no such guarantee (section 10).
The codethe target, line by line
next_state_actions, next_state_log_pi, _ = actor.get_action(next_obs, next_state)
entropy_bonus = alpha * next_state_log_pi.flatten() rewards_with_entropy = rewards - bootstrap.flatten() * discount * entropy_bonus
target_z = rewards.unsqueeze(1) + bootstrap.unsqueeze(1) * discount * self.q_support
with torch.no_grad(): # everything above is computed without gradients
q_log_probs = F.log_softmax(q_outputs, dim=-1) q_losses = -torch.sum(target_distributions * q_log_probs, dim=-1).mean(dim=-1)
train_squint.py L762–787; bootstrap is always 1 in Squint (section 9.2). Critic learning rate: q_lr = 3e-4 with Adam (L132, L726).
Check yourselfForecast 8.0, reward 0.5, next-tick forecast 8.0; ignore the randomness term. Which way does the update push?
The target is 0.5 + 0.9 × 8.0 = 7.7, so the gap is −0.3 and the forecast is pushed down, by a small fraction of 0.3.
5The actor
With a forecaster in hand, how does the robot pick a move?
If there were ten possible moves, you would forecast all ten and take the best. But a move is six numbers, each anywhere between −1 and 1: infinitely many choices. So a second network, the actor, proposes the move, and learns from the critic how to improve it.
The critic is a smooth function, so besides a forecast it can give a slope: move this joint a little this way and the forecast goes up by this much. The actor follows those slopes uphill. It's like hiking in fog: you can't see the summit, but you can feel which way the ground rises.
Two details make this work. First, the actor's move is built as center + spread × noise (section 6). Because the move is a smooth function of the center and the spread, the critic's slope passes straight through to them; this is called the reparameterization trick (Finn, slide 34). Second, the actor practises on situations from memory (section 7) but never copies the moves stored there. For each situation it proposes a fresh move and asks the critic to score that (Finn, slide 32).
The mathstwo ways to get a gradient through a random move
The actor's weights \(\phi\) should minimize
\[ J_\pi(\phi) = \mathbb E_{s\sim\mathcal D}\Big[\mathbb E_{a\sim\pi_\phi(\cdot\mid s)}\big[\alpha\log\pi_\phi(a\mid s) - Q(s,a)\big]\Big]. \]The trouble: the distribution being averaged over depends on \(\phi\), so the gradient can't simply move inside. There are two ways out.
Score function. Differentiate under the integral and use \(\nabla\pi = \pi\,\nabla\log\pi\):
\[ \nabla_\phi\,\mathbb E_{a\sim\pi_\phi}\big[f(a)\big] = \mathbb E_{a\sim\pi_\phi}\big[f(a)\,\nabla_\phi\log\pi_\phi(a)\big]. \]Unbiased, but it uses only the value \(f(a)\), never its slope, so it infers direction from correlation: high variance (Finn, slides 32–33).
Reparameterization. Write the move as a smooth function of fixed noise, \(a = f_\phi(\varepsilon;s) = \tanh\big(\mu_\phi(s) + \sigma_\phi(s)\odot\varepsilon\big)\), \(\varepsilon\sim\mathcal N(0,I)\). Now the noise doesn't depend on \(\phi\), the gradient moves inside, and the chain rule applies:
\[ \nabla_\phi\,\mathbb E_\varepsilon\big[g(f_\phi(\varepsilon;s))\big] = \mathbb E_\varepsilon\Big[\nabla_a g(a)\big|_{a=f_\phi(\varepsilon;s)}\ \nabla_\phi f_\phi(\varepsilon;s)\Big]. \]This uses \(\nabla_a Q\), the critic's slope: the actor learns which way to move, not only whether a move was good. SAC chooses it for that reason, a lower-variance estimator (SAC 2018a, section 4.2, Eqs. 11–12; Finn, slide 34). In Squint, \(Q\) is the average of the two heads, and the critic's weights are frozen during this step.
The codethe actor, line by line
mean, log_std = self.forward(rgb, state, get_log_std=True) normal = torch.distributions.Normal(mean, log_std.exp()) x_t = normal.rsample() # slopes flow through mean and std
y_t = torch.tanh(x_t)
log_prob = normal.log_prob(x_t) log_prob -= torch.log(self.action_scale * (1 - y_t.pow(2)) + 1e-6) log_prob = log_prob.sum(1, keepdim=True)
log_std = torch.tanh(log_std) log_std = LOG_STD_MIN + 0.5 * (LOG_STD_MAX - LOG_STD_MIN) * (log_std + 1)
q_values = critic.get_q_values(obs, state, pi, detach_critic=True) critic_value = q_values.mean(dim=0) # average of the two heads actor_loss = (alpha * log_pi - critic_value).mean()
train_squint.py L306–336 (actor), L828–834 (loss). The figure's slope is this gradient at the actor's deterministic move. The + 1e-6 guards the logarithm when tanh saturates; it makes the density integrate to 1 only approximately (checked: 0.9999).
Check yourselfThe actor trains on situations from memory. Whose moves is it scored on?
Fresh moves it draws itself, from its current bell curves. The moves stored in memory are only used by the critic.
6Random on purpose
Why would a learner act randomly, and how random should it be?
An actor that always plays its current best guess never finds out about better moves it hasn't tried. So the actor doesn't output one move. For each joint it outputs a center and a spread, a bell curve, and the move is a random draw from it.
The draw can land anywhere, but moves must stay between −1 and 1, so Squint bends the bell with a squashing function, tanh. A center near 0 stays bell-shaped; a center far out piles the draws up against the edge.
How much randomness is right? SAC answers by paying the actor for it. The actor's goal becomes reward plus α times the entropy, a measure of how spread out its choices are. α, the temperature, is the exchange rate between reward and randomness.
What does the best random policy look like? Take three moves whose forecasts are 2, 1 and 0, and slide α:
α isn't set by hand. A thermostat adjusts it toward a target amount of randomness: entropy −6, one unit per joint, roughly a spread of 0.09 per joint near the middle. If the actor is more random than that, α shrinks; if it's less, α grows. In our run α started at 1 and settled at .
The actor gets to choose where to spend that randomness. By tick 30, holding at the cube, the arm joints' spread was , but the gripper's was . At the hold the forecast barely changes with the gripper move, so randomness there costs nothing (our reading of the trace).
When the trained policy runs on the real arm, the randomness is switched off. It plays the center of each bell, squashed: tanh(center).
The mathsentropy, the best random policy, the squash and the thermostat
Entropy is average surprise, \(\mathcal H(\pi(\cdot\mid s)) = \mathbb E_{a\sim\pi}[-\log\pi(a\mid s)]\). For a Gaussian with spread \(\sigma\):
\[ \mathcal H = -\mathbb E\big[\log p(x)\big] = \tfrac12\log(2\pi\sigma^2) + \frac{\mathbb E[(x-\mu)^2]}{2\sigma^2} = \tfrac12\log\big(2\pi e\,\sigma^2\big). \]It can be negative: \(\sigma = 0.1\) gives \(-0.88\). Setting \(\mathcal H = -1\) gives \(\sigma = e^{-1}/\sqrt{2\pi e}\approx 0.089\), the per-joint spread behind the target of \(-6\) (before the squash; entropies of independent joints add).
SAC maximizes reward plus entropy (SAC 2018a, Eq. 1):
\[ J(\pi) = \sum_t \mathbb E_{(s_t,a_t)\sim\rho_\pi}\Big[r(s_t,a_t) + \alpha\,\mathcal H\big(\pi(\cdot\mid s_t)\big)\Big], \]so values count future randomness: \(V(s) = \mathbb E_{a\sim\pi}[Q(s,a) - \alpha\log\pi(a\mid s)]\) and \(Q(s,a) = r(s,a) + \gamma\,\mathbb E_{s'}[V(s')]\).
The best policy at one state. Maximize \(\sum_a\pi(a)Q(a) - \alpha\sum_a\pi(a)\log\pi(a)\) subject to \(\sum_a\pi(a) = 1\). With a Lagrange multiplier \(\lambda\), set the derivative with respect to each \(\pi(a)\) to zero:
\[ Q(a) - \alpha\log\pi(a) - \alpha - \lambda = 0 \quad\Longrightarrow\quad \pi^\star(a) = \frac{e^{Q(a)/\alpha}}{Z},\qquad Z = \sum_{a'}e^{Q(a')/\alpha}. \]Substituting back, the soft value is \(V = \alpha\log Z\), a smoothed maximum that tends to \(\max_a Q(a)\) as \(\alpha\to 0\). That is the "soft" in Soft Actor-Critic, and the widget above computes exactly these quantities.
The actor's loss is this target in disguise. For any policy \(\pi\),
\[ \mathrm{KL}\Big(\pi\ \Big\|\ \tfrac{1}{Z}e^{Q/\alpha}\Big) = \mathbb E_{a\sim\pi}\Big[\log\pi(a) - \tfrac{Q(a)}{\alpha}\Big] + \log Z, \]so minimizing \(\mathbb E[\alpha\log\pi - Q]\) minimizes the KL to the best policy; \(\log Z\) doesn't depend on the actor and drops out. SAC proves that this projection never makes the policy worse (soft policy improvement, SAC 2018a, Lemma 2) and that alternating it with evaluation converges to the best policy in the chosen family (Theorem 1), for finite action sets and exact updates.
The squash. With \(u\sim\mathcal N(\mu,\sigma^2)\) and \(a=\tanh u\), probability is conserved, \(\pi(a)\,|da| = p(u)\,|du|\), and \(\tfrac{d}{du}\tanh u = 1-\tanh^2u\), so per joint
\[ \log\pi(a\mid s) = \log\mathcal N(u;\mu,\sigma^2) - \log\big(1-\tanh^2 u\big), \]summed over the 6 joints. Leave it out and nothing crashes; the entropy is just wrong, and so is everything α does.
The thermostat. SAC 2018b recasts the objective as maximizing reward subject to an average entropy of at least \(\bar{\mathcal H}\); the multiplier on that constraint is \(\alpha\), learned by minimizing
\[ J(\alpha) = \mathbb E_{a\sim\pi}\big[-\alpha\log\pi(a\mid s) - \alpha\bar{\mathcal H}\big] = \alpha\,\big(\hat{\mathcal H} - \bar{\mathcal H}\big), \]where \(\hat{\mathcal H}\) is the current entropy. The slope is \(\hat{\mathcal H}-\bar{\mathcal H}\): too little randomness makes it negative, and gradient descent raises \(\alpha\); too much lowers it. Squint learns \(\log\alpha\), which keeps \(\alpha\) positive and has the same sign rule. \(\bar{\mathcal H} = -\dim\mathcal A = -6\) is a convention, not a derived value.
The codethe entropy, the squash and the thermostat
target_entropy = -torch.prod(torch.Tensor(envs.unwrapped.single_action_space.shape)).item()
log_alpha = torch.zeros(1, requires_grad=True)
_, log_pi, _ = actor.get_action(obs, state) alpha_loss = (-log_alpha.exp() * (log_pi + target_entropy)).mean()
action = torch.tanh(mean) * self.action_scale + self.action_bias
train_squint.py L667–670, L805–815, L320–323. The Squint paper's Algorithm 1 prints this update with the sign flipped; the code is right.
Check yourselfThe actor has become more certain than the target entropy. What does α do next?
It rises: the slope Ĥ − H̄ is negative, so gradient descent increases α, paying the actor more for randomness until its entropy is back near −6.
7Memory and the loop
Where does the learning data come from, what exactly is stored, and how often is it used?
Squint runs 1024 simulated arms at once on one GPU. Every tick, each arm adds one moment to a shared memory, which keeps the last 1,000,000 of them. It's called the replay buffer.
One moment, up close
Each entry in memory is one tick of one arm, written (s, a, r, s′): what the arm saw, what it did, what that earned, and what it saw next. In simulation one tick unfolds like this:
| s | the 16 × 16 picture, after colour jitter, and the 12 joint numbers at the start of the tick |
|---|---|
| a | the 6 numbers the actor chose, between −1 and 1; the controller turns them into target changes of at most 0.1 rad |
| r | one number, at most 1: the reward for the state the tick ended in |
| s′ | the picture and 12 numbers at the end of the tick, which are also the next tick's s |
| done | always "no" in Squint, because a time limit isn't an ending (section 9.2) |
One moment takes about 1.7 KB, so a million fit in 1.7 GB of GPU memory. With 128 × 128 pictures, the same million would need about 98 GB.
The tick is where the action frequency comes in. In simulation the policy acts 10 times a second, so s and s′ are 0.1 s apart, and in that time a joint can move at most 0.1 rad, about 6°. The discount counts in ticks too: "10 ticks ahead" means one second. The policy never sees a clock, only pictures and numbers, so it has learned how much the world changes between s and s′ at this pace. On our real arm it acts 15 times a second and every move is scaled to 15%, so a tick moves a joint at most 0.86° instead of 5.7°: under a quarter of the training speed.
Four things called a step
| Name | What it is | How long, or how many |
|---|---|---|
| tick, or time step | one decision of one arm: see, act, get a reward | 0.1 s in simulation; 1/15 s on our real arm |
| physics step | one update of the simulated physics, inside a tick | 10 per tick, 0.01 s each; the policy never sees them |
| iteration | all 1024 arms take one tick, then the learner makes its updates | 1024 new moments; the code's global_step counts moments, so it jumps by 1024 |
| update, or gradient step | one round of learning from 512 random moments | 256 per iteration |
For scale: checkpoint A trained for 196 iterations, which is 200,704 moments, 50,176 updates, and 19.6 simulated seconds per arm, about four episodes each. Its memory never filled. Checkpoint P, for PlaceCube, ran 1,465 iterations: 1.5 million moments and 375,040 updates, so its memory filled and the oldest half-million moments were dropped.
How often each moment is used
Each iteration's 256 updates draw 256 × 512 = 131,072 moments at random, against 1024 new ones. So on average every moment is studied about 128 times.
Why keep old moments? A moment is a fact about the world: from here, this move led there and paid this much. It stays true after the actor changes, so the critic can keep learning from it. That's what "off-policy" means (Finn, slide 31). Drawing at random also breaks up runs of near-identical neighbouring ticks, which would otherwise pull the networks back and forth.
The paper found that manipulation likes this many updates per new moment (0.25), more than humanoid walking needs (under 0.06) (Squint, section IV).
The mathswhy old data is still valid, and how often it's used
The critic's target for a stored moment needs only \((s,a,r,s')\) from memory and a fresh move from the current actor:
\[ y = r + \gamma\big(\bar Q(s',\tilde a') - \alpha\log\pi(\tilde a'\mid s')\big),\qquad \tilde a'\sim\pi(\cdot\mid s'). \]Everything that depends on the current policy is recomputed; everything stored is a fact about the world. Because \(Q(s,a)\) conditions on the stored move \(a\), no correction for "the old actor chose it" is needed. That is what lets SAC learn off-policy (Finn, slides 31–32). The losses are averages over memory, \(\mathbb E_{(s,a,r,s')\sim\mathcal D}[\cdot]\), each estimated from 512 random draws.
Use per moment: each iteration adds \(N = 1024\) moments and makes \(U = 256\) updates of \(B = 512\) draws, so on average a moment is drawn \(UB/N = 128\) times. A full memory of \(M = 10^6\) holds about \(M/N\approx 977\) iterations, about 98 simulated seconds per arm.
The codewhat gets stored, and the loop
next_obs, rewards, terminations, truncations, infos = envs.step(actions)
real_next_obs[...][need_final_obs] = infos["final_observation"][...][need_final_obs]
transition = TensorDict(observations=obs, next_observations=real_next_obs,
actions=actions, rewards=rewards, dones=dones)
rb.extend(transition)
for grad_step in range(args.num_updates):
data = rb.sample(args.batch_size)
out_main = update_main(data) # critic, encoder, alpha
if grad_step % args.policy_frequency == 0:
out_main.update(update_actor(data, encoded_rgb))
torch._foreach_lerp_(critic_target_params, critic_online_params, args.tau)
global_step += args.num_envs
train_squint.py L885–960. Inside envs.step, ManiSkill runs sim_freq // control_freq = 10 physics steps, then scores the new state. Episodes end together because early terminations are ignored (ignore_terminations=True, L632). The memory lives on the GPU (LazyTensorStorage, L741); the paper found 1M moments beat 100K by 7% in final success.
Check yourselfCheckpoint A's memory never filled. Why?
It trained for 196 iterations, 200,704 moments in all, fewer than the 1,000,000 the memory holds.
8Why learning wobbles, and three steadiers
What goes wrong when two networks teach each other?
There are two feedback loops in what you've seen so far. The critic learns from its own forecasts (section 4). The actor chases the critic's slopes (section 5). Either loop can turn a small error into a big one. Three standard fixes each guard one weak spot.
8.1 A slow copy for the targets
The problem: the target is reward + 0.9 × the critic's forecast for the next tick. Every update changes the critic, so it also moves the target. It's like aiming at a target taped to your own arm: each correction moves it. Errors can chase themselves around and grow.
The fix: keep a slow copy of the critic and use it only to compute targets. The copy is never trained. After each update it moves 1% of the way toward the live critic (this rate is called τ, tau). The goalpost still moves, but slowly and smoothly.
In Squint, τ = 0.01, so a change in the critic takes 69 updates to reach halfway into the targets. The copy covers the critic's own layers; the camera network isn't copied, and targets use the live one. One more detail that matters for us: the checkpoint saves the slow copy, so the critic we used to score the real arm is the slow copy.
The mathsthe slow copy is a moving average
After every update, \(\bar\theta \leftarrow \bar\theta + \tau(\theta - \bar\theta) = (1-\tau)\,\bar\theta + \tau\,\theta\). Unrolled over \(k\) updates:
\[ \bar\theta_k = (1-\tau)^k\,\bar\theta_0 + \tau\sum_{j=0}^{k-1}(1-\tau)^{k-1-j}\,\theta_j. \]A live weight from \(n\) updates ago carries weight \(\tau(1-\tau)^n\): an exponential moving average with a memory of about \(1/\tau = 100\) updates. Its weight halves after \(n_{1/2} = \ln\tfrac12/\ln(1-\tau)\), 69 updates at \(\tau = 0.01\).
There is no general proof that a slow copy makes deep TD learning converge: function approximation, bootstrapping and off-policy data together can diverge with or without it (the "deadly triad", Sutton and Barto, 2018, section 11.3). It is an empirical stabilizer, introduced with DQN (Mnih et al., 2015). SAC's own ablation shows both failure modes: a fast-moving target (large τ) can cause instability, a slow one makes training slower (SAC 2018a, section 5.2, Fig. 3c).
The codeone line, after every update
torch._foreach_lerp_(critic_target_params, critic_online_params, args.tau)
'critic': critic_target.state_dict(),
next_obs = encoder(data["next_observations"]["rgb"]) # under no_grad
train_squint.py L936–938 (τ at L148), L878, L763.
8.2 Two critics against the optimist's curse
The problem: the actor goes wherever the critic forecasts high. Forecasts have errors, and the actor is drawn precisely to the ones that are accidentally too high. Those inflated values then feed into targets, and the inflation spreads.
You can see the bias without any robot. Give several moves the same true value, 0. Add noise to each forecast. Pick the move that looks best. Its forecast is above 0 on average, and more so with more moves to choose from.
The standard fix, from TD3 and then SAC: train two critics and build targets from the smaller of their two forecasts, which cancels much of the optimism.
Squint keeps two critic heads but does something milder. Each head learns from its own slow copy, and the actor follows the average of the two. The paper found the average slightly better than the minimum (Squint, section IV, Fig. 4).
The mathswhy the best-looking move is overestimated, exactly
For any random variables, \(\max_i X_i \ge X_j\) for each \(j\), so \(\mathbb E[\max_i X_i] \ge \max_j \mathbb E[X_j]\): the expected maximum of noisy forecasts is at least the maximum of the true values.
Exactly, for two moves of true value 0 with unit noise, \(X,Y\sim\mathcal N(0,1)\) independent: \(\max(X,Y) = \tfrac{X+Y}{2} + \tfrac{|X-Y|}{2}\), and \(X-Y\sim\mathcal N(0,2)\) has \(\mathbb E|X-Y| = \sqrt2\cdot\sqrt{2/\pi}\), so
\[ \mathbb E\big[\max(X,Y)\big] = \frac{1}{\sqrt\pi}\approx 0.564,\qquad \mathbb E\big[\min(X,Y)\big] = -\frac{1}{\sqrt\pi}. \]For \(N\) moves the bias is \(\sigma e_N\), with \(e_N\) the expected maximum of \(N\) standard normals (1.539 for \(N = 10\)). Averaging two independent forecasts halves the noise variance, so the bias shrinks by \(1/\sqrt2\) but stays positive; taking the smaller of two pushes it down. The widget's 1.52, 1.08 and 0.67 at \(N = 10\) match these.
Squint's two heads, written out: head \(i\) bootstraps from its own slow copy \(\bar Z_i\),
\[ y_i = r + \gamma\Big(\mathbb E\big[\bar Z_i(s',\tilde a')\big] - \alpha\log\pi(\tilde a'\mid s')\Big), \]and the actor follows \(\tfrac12(Q_1+Q_2)\). Standard SAC uses \(\min_i\bar Q_i\) in one shared target instead (SAC 2018a, section 4.2, after van Hasselt, 2010, and TD3). Squint's choice rests on its own ablation: the average was slightly better on its tasks (section IV, Fig. 4).
The codetwo heads, two targets, one average
logits = self.forward(rgb_features, state, actions) # [2 heads, batch, 101] next_dists = F.softmax(logits, dim=-1)
q_losses = -torch.sum(target_distributions * q_log_probs, dim=-1).mean(dim=-1) critic_loss = q_losses.sum()
critic_value = q_values.mean(dim=0) # "Mean (No CDQ)"
train_squint.py L433–454 (per-head projection), L787–790, L831–832. The paper's Algorithm 1 (line 13) writes one target averaged over both slow copies; the code gives each head its own.
8.3 Let the critic settle before the actor listens
The problem: the actor learns from the critic's slopes. When the critic is wrong, the actor learns wrong moves; the wrong moves change the data, and the changed data confuses the critic further.
The fix, also from TD3: update the actor less often than the critic. Squint updates the actor on every 4th update (policy_frequency = 4, train_squint.py L136, L932). The paper doesn't test this choice on its own.
The mathswhat delaying the actor buys
The actor's step follows \(\nabla_a Q_\theta\). If the critic's forecast carries an error \(\epsilon(s,a)\), the actor follows \(\nabla_a(Q + \epsilon)\) instead, and learns the error. TD3 argues that letting the critic take several TD steps between actor steps reduces the error the actor sees, and shows it helps empirically (Fujimoto et al., 2018, section 5.2). It is a heuristic, not a theorem; Squint uses one actor step per 4 critic steps.
Check yourselfWith τ = 0.01, after how many updates is a change in the critic halfway into the targets?
About 69: ln(0.5) / ln(0.99) = 68.97.
9What Squint adds
Everything so far is SAC. What does Squint change, and why?
The paper names its ingredients: many simulated arms in parallel, a histogram critic, squinting, layer normalization, more updates per moment, and a fast implementation (Squint, section IV). Parallel arms and updates per moment were section 7. Here are the rest.
9.1 A histogram instead of one number
Instead of saying "I expect 8.3", Squint's critic gives a chance for every possible return, in 101 slots from −20 to 20, each 0.4 apart. The forecast is the histogram's average.
The target is built just as in section 4, but for the whole shape: take the slow copy's histogram for the next tick, shrink it toward zero by 0.9, slide it by the reward, and split each bar between the two nearest slots. The critic is trained like a classifier, with cross-entropy, to match that shape.
That is tick 3 of our run again, now as shapes. At the real reward the target averages 8.33 and the critic's current forecast 8.27, so training pulls the bottom shape toward the middle one. The re-binning in this widget is the training code's, ported and checked against it.
The paper found this critic makes training much faster in wall-clock time, even though each update costs more (Squint, section IV). A shape also says how sure the critic is. Here is checkpoint P's critic on three frames from a real PlaceCube run on our arm:
The mathsthe histogram critic: target, re-binning, loss, and what is proven
The return from \((s,a)\) is a random variable \(Z(s,a)\), and the Bellman equation holds for its whole distribution (Bellemare et al., 2017), here in SAC's soft form:
\[ Z(s,a)\ \overset{D}{=}\ r + \gamma\big(Z(s',a') - \alpha\log\pi(a'\mid s')\big),\qquad s'\sim p,\ a'\sim\pi, \]where \(\overset{D}{=}\) means "has the same distribution". Taking expectations gives back the soft Bellman equation for \(Q = \mathbb E[Z]\).
The critic represents \(Z\) on fixed slots \(z_j = -20 + 0.4j\), \(j = 0,\dots,100\), with probabilities \(p_j\). The target moves slot \(j\) to \(\hat z_j = \mathrm{clip}(r_{\text{soft}} + \gamma z_j,\,-20,\,20)\), with \(r_{\text{soft}} = r - \gamma\alpha\log\pi(\tilde a'\mid s')\). That point usually falls between two slots \(z_\ell\le\hat z_j\le z_u\), and its mass is split in proportion to closeness:
\[ \text{to } z_\ell:\ \ p_j\,\frac{z_u - \hat z_j}{\Delta z},\qquad \text{to } z_u:\ \ p_j\,\frac{\hat z_j - z_\ell}{\Delta z},\qquad \Delta z = 0.4. \]The split keeps the average. The two pieces have mean \(\big(z_\ell(z_u-\hat z_j) + z_u(\hat z_j - z_\ell)\big)/\Delta z = \hat z_j\), so when nothing is clipped,
\[ \sum_k z_k\,\hat p_k \;=\; \sum_j p_j\,\hat z_j \;=\; r_{\text{soft}} + \gamma\,\mathbb E\big[\bar Z(s',\tilde a')\big]: \]the target histogram's average is exactly the scalar soft target (8.3294 in our worked example). Clipping at ±20 would break this; our returns stay within ±10 for ReachCube (within ±11.2 for PlaceCube), far inside.
The loss splits in two.
\[ -\sum_k\hat p_k\log p^\theta_k \;=\; \underbrace{-\sum_k\hat p_k\log\hat p_k}_{\mathcal H(\hat p),\ \text{fixed}} \;+\; \underbrace{\sum_k\hat p_k\log\frac{\hat p_k}{p^\theta_k}}_{\mathrm{KL}(\hat p\,\|\,p^\theta)}, \]so minimizing it over \(\theta\) minimizes the KL from the target histogram to the critic's (1.335 = 1.319 + 0.016 at tick 3).
What is proven. For a fixed policy, the distributional Bellman update is a \(\gamma\)-contraction in the maximal Wasserstein metric, though not in total variation (Bellemare et al., 2017, Lemma 3). Re-binning onto fixed slots breaks the Wasserstein contraction in general, but the re-binned update is a \(\sqrt\gamma\)-contraction in the maximal Cramér distance, with a unique fixed point (Rowland et al., 2018, Lemma 2 and Proposition 2). Both results assume exact updates without sampling. That the histogram critic trains faster is an empirical finding (Squint, section IV), not a theorem.
The codethe re-binning, line by line
target_z = rewards.unsqueeze(1) + bootstrap.unsqueeze(1) * discount * self.q_support target_z = target_z.clamp(self.v_min, self.v_max)
b = (target_z - self.v_min) / delta_z lower, upper = torch.floor(b).long(), torch.ceil(b).long()
proj.index_add_(0, lower_indices, (next_dists * (upper - b)).view(-1)) proj.index_add_(0, upper_indices, (next_dists * (b - lower)).view(-1))
probs = F.softmax(logits, dim=-1) return torch.sum(probs * self.q_support, dim=-1)
train_squint.py L415–454, L412–413 (trimmed for reading). When b is a whole number, lines 429–431 shift an index so the mass still lands on its slot; in the one case b = 1 exactly, that shift counts the mass twice, which floating-point targets essentially never hit.
Check yourselfWhy does re-binning keep the histogram's average?
Each moved point's mass is split between its two neighbouring slots in proportion to closeness, which keeps that point's mean; adding up all the points keeps the overall mean, unless something is clipped at ±20.
9.2 Small things that keep numbers sane
Every layer in all three networks is followed by layer normalization, which rescales that layer's numbers to a standard size. With each moment studied over a hundred times, nothing then drifts to extremes. The paper found it speeds training up (section IV).
Episodes stop at 50 ticks because a clock runs out, not because the world ends. So every target keeps its "0.9 × next forecast" part, even at the last tick: a timer isn't treated as an ending (bootstrap_at_done = "always", train_squint.py L897–899).
The mathslayer normalization, and bootstrapping through a time limit
Layer normalization rescales each sample's feature vector, \(\mathrm{LN}(x) = g\odot\frac{x-\mu(x)}{\sqrt{\sigma^2(x)+\epsilon}} + b\), with \(\mu(x)\) and \(\sigma^2(x)\) the mean and variance over that vector's entries and \(g, b\) learned (Ba et al., 2016). Whatever the scale of the weights feeding it, its output has a controlled size. That it speeds training is an empirical finding (Squint, section IV).
Time limits. If the clock isn't part of the state, ending the target at the time-out would make the same state worth different amounts depending on an unseen clock. When the limit is only a training convenience, the right target bootstraps from the state where the episode was cut, \(y = r + \gamma V(s'_{\text{last}})\) (Pardo et al., 2018). Squint does exactly this, with the real last picture as \(s'\).
The codenever marking the end
dones = torch.zeros_like(terminations, dtype=torch.bool) # bootstrap_at_done == 'always'
nn.Linear(hidden_dim, hidden_dim), nn.LayerNorm(hidden_dim), nn.ReLU(),
train_squint.py L897–899, L381–383 (critic), L291–293 (actor), L273–276 (projections).
9.3 Squinting
The policy sees 16 × 16 pictures, which makes the networks, the memory and every update small and fast. The paper found smaller pictures learn just as much per moment but train faster per minute (section IV).
The name is about how the small picture is made. Squint renders at 128 × 128 and averages each 8 × 8 block into one pixel, rather than rendering at 16 × 16 directly. Averaging smooths the picture and keeps the scene's shapes. The paper found it trains better, with real-world success similar to higher resolutions. The pictures in section 1 are exactly this.
Only the critic trains the camera network. The actor uses its output, but the actor's updates don't change it (following DrQ; the paper found little difference on its tasks). During training the colours are also jittered at random, brightness, contrast and saturation by up to 30% and hue by 5%, so that the policy cares less about exactly how the scene looks:
The mathssquinting is an average
Area downsampling from 128 to 16 averages each 8 × 8 block:
\[ x^{16}_{i,j,c} = \frac{1}{64}\sum_{u=0}^{7}\sum_{v=0}^{7} x^{128}_{8i+u,\ 8j+v,\ c}. \]Every part of the scene contributes to the pixel it falls in, so thin or small things still register as a weaker colour instead of vanishing between sample points. The paper's reasoning is the same: averaging gives natural anti-aliasing and keeps the scene's structure (section IV). We checked that the policy's 16 × 16 input is exactly this block mean.
The codethe camera path
rgb = F.interpolate(rgb, size=(16, 16), mode='area')
critic_optimizer = optim.Adam(list(critic.parameters()) + list(encoder.parameters()), lr=args.q_lr) actor_optimizer = optim.Adam(list(actor.parameters()), lr=args.policy_lr) encoded_rgb = obs.detach() # what the actor receives
self.jitter = torchvision.transforms.ColorJitter(0.3, 0.3, 0.3, 0.05)
utils.py (DownsampleObsWrapper, ColorJitterWrapper); train_squint.py L500, L726–729, L821.
Check yourselfWhy render at 128 and average down, instead of rendering at 16?
Averaging lets everything in each 8 × 8 block contribute to its pixel, so small or thin things still show up faintly instead of disappearing between sample points: natural anti-aliasing.
9.4 Why minutes, not hours
All 1024 arms run on one GPU; the memory sits on the GPU too, because the pictures are tiny; and the update code is compiled (PyTorch compile and CUDA graphs) and runs in 16-bit floats, which the paper reports as more than 5× faster together. Given 15 minutes of training, Squint averaged 96.1% success in simulation across its 8 tasks and 73 of 80 trials on a real SO-101 (Squint, Tables I and II).
10The whole picture
Now all the pieces at once. What happens in one of the 256 updates, and how much of it is proven?
One update, start to finish
- Draw 512 random moments from memory: situation s, move a, reward r, next situation s′. L925
- Run all the pictures through the camera network. L763, L779
- The actor proposes a fresh next move a′ at s′, and reports how likely it was, log π(a′ | s′). L765
- The slow copy gives each head's histogram for (s′, a′). Shrink by 0.9, slide by r − 0.9·α·log π(a′ | s′), re-bin: that's the target. L771–776
- The critic and camera network take one step to bring their histogram for (s, a) closer to the target. L783–800
- The α thermostat takes one step toward entropy −6. L805–815
- On every 4th update, the actor proposes a fresh move at s and takes one step toward a higher average forecast plus randomness. L828–837
- The slow copy moves 1% toward the critic. L938
After 256 of these, all 1024 arms take their next tick, and it starts again.
The mathsthe whole algorithm in four lines
with \(r_{\text{soft}} = r - \gamma\alpha\log\pi(\tilde a'\mid s')\), \(\tilde a'\sim\pi(\cdot\mid s')\), \(\Phi\) the re-binning of section 9.1, \(\gamma = 0.9\), 512 moments per update and 256 updates per iteration.
Where each part came from
None of Squint's parts is new on its own. Each one answered a failure that someone hit first:
- 1957Bellman: a value can be written as one step plus the value of the next (section 3)
- 1988Sutton: temporal-difference learning, training a forecast toward the next forecast (section 4)
- 1989Watkins: Q-learning, learning the value of each move from experience (section 3)
- 1992Lin: experience replay, studying stored moments again and again (section 7)
- 2015Mnih et al., DQN: replay and a slowly updated target network, at scale (sections 7, 8.1)
- 2016Ba, Kiros and Hinton: layer normalization (section 9.2)
- 2017Bellemare, Dabney and Munos, C51: forecasting a histogram of returns (section 9.1)
- 2018Fujimoto, van Hoof and Meger, TD3: two critics against overestimation, and a delayed actor (sections 8.2, 8.3)
- 2018Haarnoja et al., SAC: an actor-critic that is paid for randomness, with a self-tuning temperature (sections 5, 6)
- 2018Pardo et al.: bootstrapping through training time limits (section 9.2)
- 2020Kostrikov, Yarats and Fergus, DrQ: SAC from pixels, with the critic training the camera network (section 9.3)
- 2026Almuzairee and Christensen, Squint: all of it at once, tuned to train a real arm's policy in minutes on one GPU
What the proofs cover, and what they don't
Proven, for finite sets of moves and exact updates (no sampling, a table that can hold any value): the soft Bellman update converges to a unique answer (section 4); \(e^{Q/\alpha}/Z\) is the best random policy at a state (section 6); projecting the actor onto it never makes the policy worse, and alternating evaluation with projection converges to the best policy in the chosen family (SAC 2018a, Lemma 2 and Theorem 1); and the histogram critic's update converges too, in the right distance (section 9.1).
Not covered by any of those proofs: continuous moves (a narrow bell curve's entropy is unbounded below, and the proofs need it bounded), neural networks (nothing guarantees the fitted critic reaches the fixed point), one gradient step at a time instead of each step run to the end, sampled averages, and every stabilizer in section 8 and section 9.2. Those are justified by experiments: SAC's, TD3's and Squint's ablations. The policy family matters too: one bell curve per joint cannot hold two different good ways to grasp, so SAC finds the best single bell.
Check yourselfIn one update, which networks change, and which only get read?
The critic and the camera network change every update; α changes every update; the actor changes on every 4th; the slow copy changes only by drifting 1% toward the critic, and is otherwise only read, to build targets.
11Common confusions
- "The critic predicts the reward." It forecasts the return, all future reward, discounted. One tick's reward is at most 1; a return can reach 10.
- "The actor learns from the moves in memory." Only the situations come from memory. The moves it's scored on are fresh from the current actor.
- "The slow copy is a third network that learns." It's never trained. It's a lagging average of the critic.
- "Two critics means twice the learning." They're there to temper the critic's mistakes. Squint averages them; standard SAC takes the smaller.
- "Randomness is only for the beginning." SAC keeps a set level of randomness all through training. It's switched off only when the robot is deployed.
- "A step is one thing." In this code it can mean one tick of one arm, one tick of all 1024 arms (which
global_stepcounts as 1024), or one learning update. Section 7 has all four. - "Once trained, the critic is useless." Deployment leaves it behind, but it still knows how a run is going. We use it to grade real runs on our arm.
12The knobs
If you want to make Squint better, what can you turn?
| Knob | Squint | What it changes | Tested in the paper? |
|---|---|---|---|
| discount γ | 0.9 | how far ahead the critic looks (about 10 ticks) | no |
| slow-copy rate τ | 0.01 | how fast targets move (halfway in 69 updates) | no |
| actor every N updates | 4 | how settled the critic is before the actor listens | no |
| arms × updates per tick | 1024 × 256 | data versus learning; 0.25 updates per new moment | yes: more updates helped manipulation |
| batch | 512 | moments per update | yes: as good as larger, faster per update |
| memory | 1,000,000 | how old the data can be | yes: +7% over 100K |
| critic type | 101-slot histogram, −20 to 20 | shape of the target, bounded values | yes: much faster in wall time |
| two heads | average | how optimistic the actor's guide is | yes: slightly better than minimum |
| target entropy | −6 | how random the actor stays | no (SAC's default: minus the number of joints) |
| picture size | 16 × 16, from 128 | speed versus detail | yes: same per moment, faster per minute |
| layer normalization | every layer | numeric stability | yes: faster training |
| who trains the encoder | critic only | what shapes the robot's vision | yes: negligible difference |
| learning rates | 3 × 10⁻⁴ | step size for critic, actor and α | no |
Values from train_squint.py L90–156 at 7145d47; findings from the paper's section IV and Fig. 4.
13Notation across sources
A large part of why SAC feels confusing across sources is that the letters swap. On Finn's slides θ is the policy and α a learning rate; in the SAC papers θ is the Q-function and α the temperature.
| Meaning | Finn | SAC 2018a | SAC 2018b | Squint paper | Code |
|---|---|---|---|---|---|
| policy (actor) | θ | φ | φ | φ | actor |
| Q-function (critic) | φ | θ | θ | θ | critic |
| state value V | φ | ψ | none | none | none |
| slow copy | none | ψ̄ (of V) | θ̄ | θ̄ | critic_target |
| camera network | none | none | none | ψ | encoder |
| learning rate | α | λ | λ | not named | q_lr, policy_lr |
| temperature | none | reward scale | α | α | alpha |
| slow-copy rate | none | τ | τ | ρ = 1 − τ | tau |
14How this page was checked
Why should you trust any of this?
Every number from our runs is recomputed from the recorded trace and the checkpoints each time the page is built, and the build refuses to finish if the ported re-binning or the real-arm values disagree with the training code's output. Every derived number is checked by a script against the training code, a simulation or a numerical integral; this is its last run:
| § | Check | Computed | Expected |
|---|
Every paper claim names the section, figure, lemma or equation it comes from, read in the paper itself; claims we could not find in a source were removed. Code is shown trimmed for reading (shorter names, no device arguments), and the line numbers point at the full lines in our Squint fork at 7145d47.
What the checking caught, so far: our own earlier notes said SAC "still learns with τ = 1, a hard copy every step"; the SAC paper's hard-copy variant copies every 1,000 steps, and its ablation shows a fast-moving target causing instability (corrected here, section 8.1). An estimate of the randomness terms' size, about 0.05, was for a different checkpoint; the trace gives −0.21 for this one (section 3). And the Squint paper's Algorithm 1 prints the temperature update with the sign flipped; the code is right (section 6).