SO-101 zero to hero GitHub ENTR

Measuring a policy

A robot demo that works once proves almost nothing. This page is the small amount of statistics that keeps you from fooling yourself.

Loss is not success

Training loss measures how well the model reproduces recorded motions on recorded states. Success depends on states the policy creates itself (compounding error). Two checkpoints with the same loss can score 30 % and 80 % on the arm. There is no shortcut: you count outcomes on the real robot.

How sure is "18 out of 24"?

Every trial is a coin flip with an unknown bias. The Wilson interval gives the range of true success rates that fit what you saw (95 %):

centre = (p + z²/2n) / (1 + z²/n)
half   = z · sqrt( p(1−p)/n + z²/4n² ) / (1 + z²/n)        with p = successes/n, z = 1.96
Result 95 % interval
18 / 24 (75 %) 55 % – 88 %
20 / 24 (83 %) 64 % – 93 %
6 / 24 (25 %) 12 % – 45 %
45 / 60 (75 %) 63 % – 84 %

18 and 20 out of 24 overlap almost completely: that is not an improvement you can claim. 6 versus 18 is. To separate 75 % from 85 % you need a few hundred trials, so do not try: use 24 trials to tell broken from working, and use failure types to decide what to do next.

Fix the test before you run it

Decide all of this first and keep it identical between policy versions:

If you choose the trials after seeing the policy move, you will unconsciously pick the ones it can do.

Two questions, two numbers

The task has two separable skills. Score them separately:

Question Number A low value means
Did it go to the commanded pocket and bin? followed the command The sentence is not being used: a data balance problem
Did it complete the pick and drop? success A manipulation problem: grasp consistency, camera view

A policy that always empties pocket 1 into the OK bin perfectly has high manipulation skill and scores 1 in 12 on command following. Knowing which of the two failed tells you which block to record again.

Log failures by type

trial  pocket  bin     result   failure
1      3       ok      ✓
2      3       reject  ✗        wrong bin
3      5       ok      ✗        missed grasp

After 24 trials the failure column is your recording plan. This is the same idea the book describes for large systems: judge the end state ("is the part in the bin?"), not each step, and feed the outcome back into training. π*0.6 does it at scale by labelling every attempt as better or worse than expected and conditioning the policy on that label.

Automating the judge

The top camera already knows how to read the tray and could read the bins. A check like "after the episode, pocket 3 is empty and the REJECT bin has one more part" is a verifier you can run on every attempt, which is what turns record → train → evaluate into a loop that runs without you watching. That is the flywheel this repo is named after, and it is not built yet.

Sources: Wilson (1927) score interval · SmolVLA paper (evaluation protocol) · Vision Language Models, ch. 11 p. 474–475 and 509–512