ACT policy, shape by shape

Three cameras see the task from the wrist, top and front. Six numbers say where the robot is. ACT turns that synchronized observation into a coherent chunk of future joint commands—then looks again before the world gets away from it.

That last part matters. Action Chunking with Transformers is often summarized as “predict many actions at once,” which makes it sound open-loop. The useful version is a closed-loop visual policy: plan a short trajectory, execute one action or a small prefix, observe the changed scene, and replan.

This is the model from the shapes outward. We use B for batch size, T for chunk length, and six commanded joint values. Our visual input contains exactly three synchronized RGB views: wrist, top and front.

01Why change the prediction unit

A single-step behaviour-cloning policy predicts only the next command. The robot executes it, observes again, and repeats. The loop is simple, but every tiny error changes the next observation. Those errors can accumulate into jitter or carry the policy into states its demonstrations never contained.

ACT changes the unit from one command to a local trajectory. The transformer predicts T future controls together, so reaching, closing, and lifting can form one consistent motion instead of a row of unrelated guesses.

02The whole model in seven shapes

The easiest way to understand the architecture is to keep a ledger. Our image tensor has a camera axis of exactly three, ordered wrist, top and front. Its next axis is also three, but that one means RGB channels. The state and every action have six values. Everything else exists to move information into the transformer's hidden width d and back out again.

The first 3 is the camera count; the second is the RGB channel count. d is the policy width; dz is the smaller latent-style width.

Tensor	Shape	What it means
camera images	(B, 3, 3, H, W)	wrist, top and front RGB views
current state	(B, 6)	the six current joint values
demonstrated actions	(B, T, 6)	the future training target
posterior μ, log σ²	(B, dz) each	parameters of the training style distribution
sampled style z	(B, dz)	how this demonstrated chunk is performed
learned action queries	(T, d)	one query for each future position
predicted chunk	(B, T, 6)	T continuous six-value commands

03Training has a path that deployment never sees

ACT is trained as a conditional variational autoencoder. Its upper path sees the answer: the real future action chunk from the demonstration. A transformer encoder reads [CLS], the current state, and those action tokens. The [CLS] output becomes a mean and log-variance, then reparameterization samples a style z.

The lower path is the policy. The wrist, top and front images become spatial feature tokens with 2D positions. State and style become context tokens. Then T learned queries cross-attend to that observation memory in parallel, one query for each future action position.

04What each block is responsible for

The visual backbone preserves where useful objects and robot parts appear; flattening too early would throw away the spatial structure the decoder needs. A small projection turns the six-value state into one token in the same hidden width. The latent projection does the same for z.

The transformer policy receives those memory tokens and the T learned queries. Each query retrieves the observation context it needs through cross-attention and represents one future position in the chunk. The action head is deliberately plain: it maps each decoder output to six continuous commands.

That separation is useful when debugging. If none of the three views contains the object, a better decoder cannot invent it. If outputs have the wrong shape, the action head or dataset window is wrong. If the motion is locally inconsistent, inspect the queries, positions, and chunk construction before changing the cameras.

05How ACT trains

Every training sample is anchored at a timestep t. It contains the synchronized wrist, top and front images, current state, and the next T demonstrated actions. Near the end of an episode there may be fewer than T real actions, so the remainder is padded and a mask marks which rows are valid.

The CVAE encoder infers μ and log σ² from the state and real action chunk. Reparameterization produces z = μ + exp(0.5 log σ²) ε. The policy uses that z to reconstruct the same chunk.

L1 reconstruction teaches the commands. KL regularization keeps the posterior close enough to the simple prior used at deployment. β controls the trade-off. Too much pressure toward the prior can make the decoder ignore z, a failure called posterior collapse.

The useful diagnostic is behavioural, not just a KL number: reconstruct once with the encoded posterior z, again with z = 0, and again with z shuffled across the batch. If the result barely changes, the decoder is probably not using the style latent.

06Inference stays closed-loop

At deployment there is no future demonstration, so the CVAE encoder cannot run. The live loop is short: read the synchronized wrist, top and front images plus the state; set z = 0 unless intentionally sampling diverse behaviour; predict T actions; execute one action or a short prefix; observe again.

Repeated calls create overlapping predictions for the same future timestep. Temporal ensembling combines them, with larger weight on newer chunks. The policy keeps frequent visual feedback while abrupt command changes are smoothed away.

newer-chunk preference

0.35 · gentle

07What ACT adds beyond chunked behaviour cloning

Single-step behaviour cloning is the smallest baseline: one observation, one action, maximum feedback, but the highest exposure to jitter and compounding error. Plain chunked behaviour cloning already tests the strongest part of the idea by predicting T actions at once.

ACT adds interaction. Learned queries retrieve different evidence from the observation memory for different future positions, while the optional CVAE latent can represent variation in how demonstrations accomplish the task. Diffusion Policy also models an action trajectory and is especially strong when several futures are plausible, but iterative denoising asks for more inference compute.

Policy	Predicts	Strength	Limitation
single-step BC	1 action	simple and fast	jitter and compounding error
chunked BC	T actions	tests chunking directly	no query-to-context interaction or latent style
ACT	T actions	coherent chunks and observation attention	larger model; CVAE may collapse
Diffusion Policy	trajectory	strong multimodal modeling	iterative denoising costs more

08The implementation checklist

• DataSplit by episode, preserve temporal order, normalize state and actions with training statistics, and return both the action chunk and its padding mask.
• VisionKeep the wrist, top and front camera order fixed. Use the same resolution and normalization for every view during training and rollout, and preserve spatial features for the transformer.
• CVAEUse bidirectional self-attention over [CLS], state, and action tokens. Mask padded action tokens.
• PolicyUse T learned queries, observation memory, positional embeddings, and an action head that returns (B, T, 6).
• LossLog reconstruction, raw KL, weighted KL, and total loss separately. A single total does not reveal which term moved.
• EvaluationJudge closed-loop rollouts. Success and behaviour matter more than comparing raw loss scales across different policies.

The central idea is smaller than the architecture diagram. ACT changes the basic prediction unit from one motor command to a short trajectory. Chunking improves local consistency; repeated three-camera observations keep the robot responsive; the CVAE can explain variation in how the demonstrations get there.

For our setup, the first invariant worth checking is the camera axis: images.shape[1] == 3, ordered wrist, top and front. From there the path should close cleanly from (B, 3, 3, H, W) and (B, 6) to (B, T, 6).