SO-101 zero to hero GitHub ENTR

Inside SmolVLA

SmolVLA is 450 M parameters: a small VLM that reads the scene, and a 100 M action expert that writes motion. This page follows one training step and one control step through it, with the real numbers from this project.

top 640×480 ─┐                                   ┌──────────────────────────┐
wrist 640×480├─► SigLIP + pixel shuffle ─► 128 ─►│ SmolVLM2, first 16 layers │──► keys/values (cached)
"put the part from pocket 3…" ─► tokens ─► 48 ──►│        (frozen)           │          │
6 joint angles ─► pad to 32 ─► linear ─► 1 ─────►└──────────────────────────┘          ▼
                                                 ┌──────────────────────────┐   cross-attention
noise (50 × 32) ─► + time embedding ─► 50 ──────►│  action expert, 16 layers │◄─────────┘
                                                 │        (trained)          │──► 50 actions × 6 joints
                                                 └──────────────────────────┘

What goes in

Input Processing Tokens
Each camera frame Resized to fit 512 × 512 (ours become 512 × 384, padded), scaled to [−1, 1], SigLIP, pixel shuffle 64
Sentence A newline is appended, tokenised, padded to 48 48
Joint angles Normalised with the dataset mean and std, zero-padded from 6 to 32, one linear layer 1

So the VLM reads 177 tokens per decision. The pretrained model calls its cameras camera1..3; --rename_map renames top and wrist onto the first two, and the third is simply left out.

The padding to 32 is why one checkpoint works on arms with different joint counts.

Two transformers, side by side

During one decision the VLM runs once. Its output is cached and the expert reuses it for all ten denoising steps. That cache is what makes a VLA fast enough.

Flow matching: how noise becomes a motion

The expert does not output actions directly. It learns a velocity field: a rule that says, for any half-noisy action chunk, which direction leads to a clean one.

Training, for one sample (a random frame, plus the 50 actions that followed it):

A  = the 50 real actions                    (50 × 6, normalised; zero-padded to 50 × 32)
ε  = random noise of the same shape
t  = a random number in (0, 1], biased toward 1 (the noisy end)

x_t = t·ε + (1 − t)·A          # mix: t = 1 is pure noise, t = 0 is the real chunk
u   = ε − A                    # the direction from data to noise
loss = mean( (expert(x_t, t, scene) − u)² )

The network sees a blurred version of what you did and must say which way the blur points. Steps past the end of an episode are masked out of the loss.

Inference runs the rule backwards from pure noise, in ten Euler steps:

x = ε                           # start: noise
for t in 1.0, 0.9, … 0.1:
    x = x − 0.1 · expert(x, t, scene)
return x                        # 50 actions

Why not just regress the actions? Because of the "several right answers" problem in how imitation learning works: regression averages two valid grasps into an invalid one. Starting from a random ε, flow matching ends at one of the demonstrated behaviours, a different one for a different ε.

From 50 actions to a moving arm

Chunk 50 actions = 1.67 s at 30 fps
Default (--inference.type=sync) Play all 50, then stop and think. The arm pauses at every chunk boundary when the model is slow
Real-time chunking (rtc, used by scripts/rollout.sh) A background thread starts the next chunk when 30 actions are left

RTC has one subtlety. The new chunk is computed from an observation that is already old by the time it arrives. So while the new chunk is being denoised, its first actions are steered toward what is left of the old chunk: fully for the steps that will already have run by the time it arrives, fading to nothing by step 10. Those stale steps are then dropped. The result is continuous motion and a new plan every 20 actions (0.67 s), or less often if the model is slower than that.

What fine-tuning changes

lerobot-train --policy.path=lerobot/smolvla_base keeps the vision encoder and the VLM frozen and trains about 100 M parameters: the action expert, the projections in and out of it, and the one layer that embeds the joint state.

Setting Value Meaning
Optimiser AdamW, lr 1e-4 Warm-up, then cosine decay to 2.5e-6
Batch × steps 64 × 20 000 1.28 M samples: each of our ~36 000 frames is seen about 35 times
Normalisation mean/std from meta/stats.json Your dataset's statistics replace the pretrained ones
Augmentation off Lighting robustness has to come from your data

The model was pretrained on 481 community datasets (22.9 K episodes, mostly SO-100 arms), with cameras mapped to top / wrist / side. That is why your top and wrist views are a good match, and why 10 episodes per sentence can be enough.

How it compares

Policy Size Reads language Action head Needs
ACT ~52 M in LeRobot (80 M in the paper) no (in LeRobot) CVAE transformer, chunks of 100, L1 loss Any GPU, even a laptop
SmolVLA 450 M yes Flow matching, ~100 M expert One consumer GPU
π0 / π0.5 3–4 B yes Flow matching, ~300 M expert 24–40 GB
GR00T N1.7 3 B yes Flow-matching diffusion transformer under a slower VLM Large GPU, Jetson to run
Cosmos 3 Edge 4.6 B yes Actions denoised by the world model itself 21 GB with LoRA, LeRobot 0.7

On real SO-100 tasks with 50 demonstrations each, the SmolVLA paper reports 78 % average success, against 62 % for π0 and 48 % for ACT, and 52 % for SmolVLA without its pretraining. Our task has 12 sentences that differ in one word, which is harder than three separate tasks: expect the pocket number to be the first thing that fails.

Sources: LeRobot 0.6.1 source (policies/smolvla/modeling_smolvla.py, smolvlm_with_expert.py, policies/rtc/) · SmolVLA paper · Flow matching · π0 · Real-time chunking · Vision Language Models, ch. 11, p. 504–514