SO-101 zero to hero GitHub ENTR

What to try next

Each of these is one experiment with a clear question. Do them after the SmolVLA baseline has a score: without a baseline you cannot tell whether anything helped.

A bigger policy: Cosmos 3 Edge

nvidia/Cosmos3-Edge is a 4 B-parameter world model: it generates video, text and actions with one network, so it is not a drop-in VLA like SmolVLA. LeRobot (0.7 and later) can fine-tune it as a policy:

lerobot-train --policy.type=cosmos3_edge --policy.pretrained_path=lerobot/Cosmos3-Edge \
  --policy.load_action_modules=false --dataset.repo_id=fport/so101-qc ...

Sim to real: Cosmos 3 Nano Sim-Transfer

nvidia/Cosmos3-Nano-Sim-Transfer turns simulator output (depth, segmentation or edges) plus a text prompt into photoreal video. This repo has a MuJoCo copy of the cell and a scripted expert that solves it (uv run flywheel-sim batch).

Grounding: LocateAnything

nvidia/LocateAnything-3B returns boxes or points (0–1000 coordinates) for a text prompt such as "gear with a missing tooth". Demo.

Reinforcement learning

Imitation stops at the quality of your demonstrations. RL lets the arm improve past them from its own attempts and a success signal. LeRobot ships HIL-SERL (human-in-the-loop RL on the real arm). The inspection camera could give the reward: "is the part in the commanded bin?". Start this only when imitation plateaus.

A newer LeRobot

This repo pins LeRobot 0.6.1 (the arm is calibrated and tested on it). 0.7 adds Cosmos 3 policies and remote inference. The dataset format (v3.0) is the same, so you can record on 0.6.1 and train on 0.7.