SmolVLA, explained from the ground up

A red cube sits on the table in front of the arm, you type “pick up the red cube,” and from that moment six motors need an angle from somebody, thirty times a second, until the cube is in the gripper. Pixels and a sentence in, joint angles out.

That used to mean writing it out. Find the cube, estimate its pose, plan a path, hand it to a controller, then rewrite the chain when someone leaves a spoon on the table. A vision-language-action model throws the chain out. One network reads the frames and the sentence and emits joint angles, and what a person used to specify is learned from recordings.

That works because the seeing half does not start from scratch. Trained on internet images and captions, it knows what “red” and “cube” mean before a robot is involved, so only the half that turns understanding into motion has to be taught. It is also why these models are big, and the SmolVLA paper calls them too costly to train and deploy while recordings from cheap arms pile up unused.

So SmolVLA sets a budget instead of a capability target. One GPU to train on, a consumer GPU or a CPU to run on, community data only, everything released. It lands at 450 million parameters and 87.3 on the LIBERO benchmark, where π₀ needs 3.3 billion and 10,000 hours of robot pretraining to reach 86.0. Every choice inside it is about what to leave out.

One convention. Anything marked measured is ours, from forward hooks on the released code. Everything else is the paper's, and where they disagree we say so.

01Almost none of it is new

Two stacks. The left is a vision-language model carrying what it learned from the internet, and it is frozen, so none of that knowledge moves. The right is a smaller action expert that only speaks joint angles, and it is the half that learns.

Counted layer by layer as the released config builds it, the model is 450,046,176 parameters, of which 99,880,992 train, and 47.3 million of the frozen half is an lm_head for turning hidden states back into words, on the GPU for nothing because the forward pass never calls it. Since the expert is a copy of the frozen architecture at smaller widths, the better question is how many weights are structurally new, with no counterpart in what was borrowed, and the answer is 1,635,152 across five small linear layers, 0.36% of the thing.

02From a 4:3 camera to a square

Nothing runs until the frame is square, because the vision tower takes 512×512 and only that, while our cameras give 640 by 480. The obvious move is to squash it, and nobody does, because the frozen tower was never shown a squashed picture, so it would arrive as geometry the tower has never seen, a mug as an ellipse, a cube as a slab, with nothing downstream retrained to undo it. Better to waste pixels and keep the shapes honest. That reasoning is ours.

The code takes both shrink factors, 640/512 = 1.25 and 480/512 = 0.9375, and uses the larger for both axes. Fit-inside rather than fill, so nothing is cropped and the frame lands at 512 by 384.

That leaves 128 rows short of square, and they all go on top rather than split between the edges, holding plain zeros that img * 2.0 - 1.0 carries to −1, so they are black rather than a sentinel to be learned around. Measured. The processor the backbone ships with pads bottom and right, so the frozen tower reads padding on an edge its pretraining never used.

A quarter of every frame is a constant, and nothing downstream throws it away. Those 128 rows are 8 of the 32 patch rows, so 16 of that camera's 64 tokens are blank, and across three cameras that is 48 of the 192 image tokens paying full attention cost while carrying nothing. Measured.

03One frame becomes 1024 patches

A transformer reads one flat list, so the frame, the sentence and the arm's pose all become entries in it. One entry is a token.

The square frame is cut into 16×16 patches, and 512 / 16 = 32, so there are 32 × 32 = 1024 of them. One convolution turns a patch into numbers, 16×16 kernel and stride 16, and because those are equal the windows never overlap, which makes it the same as flattening each patch and multiplying by a matrix.

768 filters, each holding 3 channels × 16 × 16 = 768 weights, so a patch arrives as 768 raw values and leaves as 768 filter responses. Across the whole frame that is 3 × 512 × 512 = 786,432 numbers in and 32 × 32 × 768 = 786,432 out, the same total to the digit. It is not a summary of the picture but the whole picture rewritten into learned features, and every reduction happens after this.

That convolution is the only part of this model that thinks in pictures. From here to the motors, everything is a list.

Then a learned position table is added on, one row per patch, shape (1024, 768). Added, element by element, the position row landing on the patch row, so a configuration flag is frozen into the shape of a weight and the 512×512 above cannot change. Hold onto the adding, because the expert faces the same problem later and solves it the other way, at more cost than anything else new.

04Twelve layers where every patch sees every patch

Those 1024 tokens go through the vision tower proper, a plain ViT twelve layers deep, and its attention is bidirectional. No causal mask, no ordering, so patch 900 reads patch 3 as freely as patch 3 reads patch 900. The right call for a picture, which has no reading direction.

By the twelfth layer a token describes not its own 16×16 square but that square in the context of the whole frame.

Width 768 splits into 12 heads of 64, so Q, K and V each come out as (B, 12, 1024, 64), and the same shapes return in the expert at other numbers. Inside one head, Q times K transposed is a 1024 × 1024 table, one number per ordered pair of patches. Divide by √64 = 8, softmax each row, multiply by V, and each patch leaves with 64 numbers blended from the whole frame. That is 1,048,576 numbers per head per layer, so twelve layers of twelve heads is about 151 million attention scores per frame, none of them stored.

05A thousand patches become sixty-four

SmolVLM-2 was trained with image tiling, several crops of a frame alongside the whole frame, which helps a describer see small things. SmolVLA switches it off, so the 64 tokens below are that camera's whole budget rather than 64 per crop. Detail traded for speed.

Carrying 1024 tokens per camera is not affordable. Every 4×4 block of patches is stacked end to end into one vector of 16 × 768 = 12,288 features, and a single Linear(12288 → 960) with no bias decides what survives, leaving 64 tokens per camera, each covering a 64×64 region. That matrix is 11,796,480 weights, seven times everything structurally new put together.

This is not pooling. Nothing is averaged before the projection sees it, so spatial detail folds into the channels and a learned matrix picks from it, and a thin red edge that an average over sixteen patches would wash flat can survive.

Count them. Three cameras give 192 image tokens, the arm gives 1 and the sentence up to 48, putting the prefix at 241 and the full list at 291 once the 50 actions being written are counted.

Both are ceilings, because the sentence is padded only to the longest in the batch, and “pick up the red cube” is 6 tokens, giving a 199-token prefix. We padded to the 48-token cap for every measurement below. Measured.

06Who is allowed to look at whom

All 291 tokens sit in one sequence, but they cannot all see each other, and that rule shapes more of the architecture than anything else.

Every token gets a block number. Images and language share block 0, the pose gets 1, and each action gets its own, 2 through 51. A token may attend to another only if that other token's block number is at most its own.

Two things fall out, both coming back later. The prefix never looks at the actions, so its keys and values do not depend on them, which makes it legal to compute the prefix once and reuse it across all ten denoising passes. And the actions are strictly causal among themselves, so action 30 sees action 29 and not the reverse, which Table 7 prices at 74.5 against 67.5 for looking both ways, both from a self-attention-only variant.

07Two towers, sixteen floors, twenty-four attentions

The two stacks walk forward together, and what passes between them depends on the floor. This is the part described wrongly most often, including by the paper.

On even floors there is no separate expert attention. Each tower projects its own tokens into queries, keys and values, the two sets are concatenated on the token axis, and one attention runs over the whole thing, one matrix and one softmax, then the rows split and each tower keeps its half. In training we hooked it as a (291, 291) map. Measured.

The paper calls those the floors where “action tokens attend to each other”, which leaves out the half that matters, because in the same call the actions also attend over all 241 scene tokens. Nothing fetches the scene here, it is already in the matrix.

On odd floors they come apart. The frozen tower attends over its own 241 tokens and the expert runs a separate 50×241 attention against the frozen side. Hooking it showed the expert does not re-read the frozen hidden states, it takes the keys and values the frozen tower already computed and pushes each through its own Linear(320 → 320), so it cannot change what the frozen model saw, only how it reads it.

An action token arrives 720 wide and q_proj takes it up to 960, the frozen tower's 15 heads of 64. The keys are the frozen tower's, 5 key-value heads read by 3 query heads each, which is grouped-query attention, fifteen people asking questions of five filing cabinets. Q times K transposed gives a 50 × 241 table with no column for any action token, so actions cannot see each other here, and o_proj drops the answer back to 720. Measured.

What it saves is commonly overstated. Expanding 5 key-value heads back out to 15 costs what 15 real ones would, so the saving is a third off the attention projections but an eighth off a whole layer. The cache is what matters, 14.8 MB down to 4.9 MB in bfloat16. Measured.

Eight even floors are one call each, eight odd floors are two, because the frozen tower still does its own 241×241 pass first, so a training step is eight at (291, 291), eight at (241, 241) and eight at (50, 241), which is 24. Measured by wrapping the attention function. Sixteen floors rather than the language model's 32, because Table 8 prices the top half at 1.8 points.

Alternating beats either alone, 85.5 against 79.0 for cross-attention only and 74.5 for self-attention only, because they do different jobs. Cross-attention is the expert's cheap look at the scene, self-attention is how action 30 learns what action 29 committed to, and without it you hold 50 independent guesses rather than one motion.

08Why it starts from noise

Put a mug on the table between the gripper and the cube. Left around it works and right works, and if you train a model to predict the one correct trajectory under a loss that averages, what it learns from seeing both is the average of the two, straight through the mug. It learned exactly what was asked, and what was asked was wrong.

So the expert never predicts the answer directly. Training mixes the true actions with noise, stands at a random point on the line between them, and asks only which direction the answer lies in.

Because the thing predicted is noise − actions, the action term never drops out. Hold that against a plain denoiser trained to predict noise alone, where at the noisy end the input very nearly is the noise, so predicting it collapses into copying the input and the loss stops measuring anything. Here the error floors at the action-prediction error instead.

And the sampling tilts towards that end. Beta(1.5, 1.0) has density proportional to √t and a mean of 0.6, with 28.5% of draws in the top fifth, so most of the training budget goes where the loss still says something.

09The clock the expert reads

The expert cannot point anywhere useful without knowing how far along it is, because the right direction from a nearly-clean chunk is not the right direction from pure noise. That number is the clock, and everything below is the code's time, 1 at pure noise down to 0 at the answer, the paper's τ upside down.

One number is not much to condition on, so it is widened into 720 values, 360 sines and 360 cosines, periods running 0.004 up to 4.000, each 1.0194 times the one before.

It is sharper than you would guess. Measured, the clock at 0.50 and 0.51 has cosine similarity 0.5169, falling to 0.2842 by 0.55, so neighbouring steps do not look alike. Its length is √360 at every value, so it changes direction only, never magnitude.

In section 03 the vision tower told 1024 patch tokens where they sat by adding a position row onto each patch row, free. Telling 50 action tokens what time it is goes the other way. The 720-number clock is concatenated, not added.

action_in_proj takes the noisy chunk up to (B, 50, 720), the clock arrives as (B, 720) stretched across all fifty positions, and concatenating makes (B, 50, 1440), which two layers with a SiLU between them bring back to 720. Every token then carries the same clock and its own content side by side rather than mixed.

Adding would have been free. Concatenating costs Linear(1440 → 720) and Linear(720 → 720), 1,556,640 weights, 95.2% of everything structurally new in SmolVLA. Measured.

What the money buys is separability. Side by side, the MLP can learn any interaction between a token's place in the chunk and how noisy the chunk is. Adding folds both into the same 720 directions, where a late step in a clean chunk gets hard to tell from an early step in a noisy one. The tower gets away with adding because a patch's position never changes.

Coming back out is cheaper. action_out_proj drops each token to 32 numbers, of which only the first 6 mean anything on a six-joint SO-101. The other 26 are the cross-embodiment slot, computed on every token of every step and dropped. Measured.

10Ten steps, and why ten is exact

At generation time the expert starts from pure noise and takes ten equal Euler steps to the actions.

This is what the straight path buys. Ten and two steps land in the same place, so the step count neither buys nor costs accuracy, and any error in a real run is the model pointing the wrong way, not the integrator. A curved path would cut the corner by an amount that depends on the count.

11Where the speed actually comes from

Inference is two passes that look nothing like each other.

The prefix is 241 of the 291 tokens, and after the sixteen floors it is kept as 16 layers of two tensors of shape (241, 5, 64), 4.9 MB in bfloat16. Everything after reads it instead of recomputing, so the ten sampling passes each move 50 tokens rather than 291, and the chunk costs 16 + 10 × 16 = 176 attention calls against 24 in training. Measured.

12How long can it act blind

Predicting 50 actions is cheap. Executing all 50 before looking again is not, because the policy sees one frame with no history and no hidden state carried over, so a frame from 49 steps ago is all it has to go on. Drag the slider.

Steps before looking again

1 step · 80.3

13So stop making it wait

If acting blind is expensive and thinking is slow, the compromise is to act, stop, think, act again, and the stopping is the waste. So SmolVLA separates the robot from the model. The arm drains a queue of actions and ships the current frame to a server when the queue runs low, then keeps moving while the server thinks, which puts the thinking underneath the motion rather than in the gaps.

14What it was trained on

481 community datasets from Hugging Face, 22,900 episodes, 10.6 million frames. The paper's own limitations put OpenVLA at around a million trajectories against this 23k, so it is roughly forty times less data.

The data was messy the way community data always is. Task descriptions were missing or wrong, so a vision-language model rewrote them, and camera names disagreed, so viewpoints had to be mapped onto a shared scheme. Neither is a research contribution and both had to happen first.

15Read the tables, not the captions

Three things in the paper do not line up with themselves, and checking beats trusting any summary, including this one.

Table 9 tests four expert widths and its caption says larger capacities do better, while its own rows say ×0.50 is smaller and scores 2.8 points higher than the ×0.75 that shipped. Table 11 says feeding the pose token to the frozen tower beats feeding it to the expert, with no qualification, when its own self-attention rows go the other way, 53.3 against 74.8.

The plainest one is not an ablation. Table 1 lists 22.9K episodes and 10.6M frames, then its own caption says the set stands “at ~10M episodes”, taking the frame count and putting the wrong noun on it.

The shipped choices might still be right, because these ablations are all trained from scratch on LIBERO with no robotics pretraining, a different setting from the released model. But anyone who reads the captions and skips the tables gets a tidier story than the numbers support.

The two ablations whose captions contradict their own rows. LIBERO average, trained from scratch with the backbone frozen. Shipped choice in red. Tables 9 and 11.

Option	Avg
Expert width
× 1.00	82.3
× 0.75	77.5
× 0.50	80.3
× 0.25	73.8
Pose token goes to
frozen tower, cross-attn	80.3
action expert, cross-attn	73.3
frozen tower, self-attn	53.3
action expert, self-attn	74.8

16The whole thing on one page

Everything above, in the order it happens.

Taking it apart mostly taught us how little of it is new. Flow matching and the action expert come from π₀, the frozen backbone is SmolVLM-2 off the shelf, and action chunking goes back to ACT in 2023. What SmolVLA adds is decisions about what to leave out, and they cover most of the distance from 3.3 billion parameters to 450 million.