SO-101 zero to hero GitHub ENTR

5 · Data quality

Imitation learning copies what is in the data: your hesitations, your shortcuts, and every accident of the scene. A policy that fails is nearly always showing you something about the dataset. These rules are ordered by how often breaking them ruins a model.

1. Be consistent in how, varied in what

Keep the same Vary on purpose
Start pose and end pose of the arm Which part is in the pocket
How you grasp each part type (rim by the edge, pad across the short side…) The part's rotation in the pocket
The path: up, across, down, release, back Which other pockets are filled
Speed: smooth, about 10 s per episode Light (record a few blocks at another time of day)
Camera positions, bin positions, tray position How full the bins are

One way of doing the task, shown many times, is learnable from 10 episodes. Two ways shown 5 times each is not.

2. The sentence must be the only clue to the target

The policy takes any shortcut the data allows. If white parts always went right, it learns "white → right" and ignores the words. So in every block, put sound and defective parts, white and orange ones, into the commanded bin. The only thing that predicts the pocket and the bin is then the sentence.

For the same reason, record every pocket with both bins, and the same number of episodes per sentence. scripts/status.sh shows the balance.

3. No dead time, no hands

4. Only keep clean successes (at first)

A clean episode: one grasp, no bump against the tray, part lands in the bin, arm returns. Anything else: ← and redo. Recovery behaviour (re-grasping after a slip) is valuable, but add it later, deliberately, from the failures the trained policy actually makes (step 8).

5. You must be able to do it from the cameras

The policy sees two 640 × 480 images, nothing else. If the part is hidden behind the gripper in both views at the moment of grasping, there is nothing to learn from. Check in setup, before the first episode.

6. Do not touch the scene between record and run

Same camera angle, same bin positions, same calibration files. If a camera was bumped, re-aim it using a frame from the dataset as reference before recording more.

How much

Episodes per sentence Expect
5 Moves toward the right area, often misses the grasp
10 (this plan, 120 total) A working baseline; the floor LeRobot recommends per location
20–30 Clearly more reliable; add where the evaluation shows failures, not everywhere

More data only helps where the policy is weak. Train on 120, evaluate, then record more for the sentences that fail.

Before each block