Imitation Learning on a 6DoF Robot Arm: Peg Transfer

Overview

I compared three imitation learning policies, ACT, Diffusion Policy, and SmolVLA, on a peg transfer task with a low-cost robot arm. I collected the demonstrations, trained each policy, and ran 25 trials per policy to see where each one works and where it breaks. The three finished the task at similar rates, but they failed at different stages.

At a glance…

• 3 policies compared: ACT, Diffusion Policy, SmolVLA

• 200 teleoperated demonstrations from two cameras

• 25 trials per policy across 5 start positions

• 28% to 32% end-to-end success for the final checkpoints

An Independent Project to Learn Robot Learning Hands-On

• Why: most of my robotics work had been classical control and perception. I wanted hands-on experience collecting demonstrations, training policies, and evaluating them honestly.

• Why peg transfer: it is a standard exercise in Fundamentals of Laparoscopic Surgery (FLS), used to train hand-eye coordination, so it connects directly to surgical robotics.

• Inspired by, not a replica: I kept the pick, move, and place, and used one arm with a parallel-jaw gripper, one peg per trial, five fixed start positions, and no handoff.

• The holes are wide and self-centering, which suits imitation learning. Tight-tolerance placement is saved for a later reinforcement learning phase.

Overhead view of the setup: the SO-101 arm and the pegboard.

Demonstrations Come From Teleoperating the Arm

• I move a leader arm by hand and a follower arm copies it, while the cameras and joint positions are recorded as one demonstration.

• I tested teleoperation first, before the pegboard or any learned policy.

First teleoperation check, before the pegboard or any policy: the follower arm copies the leader arm.

ACT Went First, and My Starting Setup Failed

• Why ACT: it is a standard first baseline for imitation learning on low-cost arms, and it predicts a short sequence of actions at a time.

• First attempt: 50 demonstrations from one overhead camera, with the arm executing about 3 seconds of actions (100 steps) between observations.

• Result: in rollout, the arm stalled and retreated in a loop.

• My diagnosis: with long chunks, the arm acted for about 3 seconds without looking, so small errors compounded.

I Rebuilt the Setup, and ACT Grasped Rarely but Placed Well

What changed:

• 200 demonstrations instead of 50, spread across the five start positions, with recovery moves mixed in

• A wrist camera added to the overhead view

• Shorter chunks: 40 actions per prediction instead of 100

• The pegboard stuck down after it slid during recording, and a corrupted episode boundary in the dataset repaired

Training and rollout:

• 100,000 steps on a cloud GPU

• My laptop has no GPU, so rollouts ran on CPU at about 1.2 seconds per prediction. I executed 32 actions per prediction to cover the delay.

Result: ACT grasped the peg in 9 of 25 trials (36%) and placed it in 8 of those 9 (89%).

ACT, success: a clean grasp and placement.

ACT, failure: the arm overshoots left of the peg and cannot recover. This happened in columns 1 and 2, which had 0 of 10 successes.

Diffusion Policy Grasped Often but Placed Poorly

• Why next: ACT's problem was finding a grasp. Diffusion Policy builds each motion by denoising, so it can represent several valid ways to move, which I hoped would help it find a grasp.

• Same 200 demonstrations and 100,000 steps, with more than five times ACT's parameters (about 278 million).

• Inference was slow on my CPU laptop, about 10 seconds per prediction at 50 denoising steps. An early 15-step test succeeded in 2 of 15 trials, so I used 50 steps for the formal run.

• Result: it grasped in 20 of 25 trials (80%) and placed only 7 of those 20 (35%).

Diffusion Policy, success.

Diffusion Policy, failure.

SmolVLA Tested Whether Pretraining Helps

• Why next: ACT and Diffusion Policy learn from my 200 demonstrations alone. SmolVLA starts from a pretrained vision-language model, so I wanted to see whether that helps with limited data.

• Only the action expert was trained: about 100 million of 450 million parameters, with the vision encoder frozen.

• I trained for 20,000 steps, one fifth of the other two, to fit my compute budget, and evaluated at 10,000 steps and again at 20,000.

• At 20,000 steps: grasped 16 of 25 (64%), placed 7 of those 16 (44%). At 10,000: 80% grasped, 20% placed.

• It never succeeded in columns 4 or 5, at either checkpoint.

SmolVLA, success.

SmolVLA, failure: successful grasp, but inaccurate placement of the peg near the hole.

Similar Success Rates, Different Failure Stages

• ACT: grasped 9 of 25 (36%), placed 8 of 9 (89%), end to end 8 of 25 (32%)

• Diffusion Policy: grasped 20 of 25 (80%), placed 7 of 20 (35%), end to end 7 of 25 (28%)

• SmolVLA at 20,000 steps: grasped 16 of 25 (64%), placed 7 of 16 (44%), end to end 7 of 25 (28%)

• SmolVLA at 10,000 steps: grasped 20 of 25 (80%), placed 4 of 20 (20%), end to end 4 of 25 (16%)

Difficulty by position was not consistent across policies. Column 3 was one of ACT's best and one of Diffusion Policy's worst.

My Explanation:

• ACT commits to a chunk of actions, so a good first move got finished cleanly, but a bad one often could not recover. It closed its gripper 239 times across 25 trials, against about 70 for Diffusion Policy.

• Diffusion Policy samples a new trajectory each time, which may have helped it find grasps but made its endpoints scatter, with misses to the left, to the right, and overcorrected.

• SmolVLA traded grasping for placing between checkpoints, and it is the least-trained policy, so I read its results as undertrained, not as a verdict on pretraining.

Next: Reinforcement Learning

• The failures point to what to fix: finding a grasp and recovering for ACT, and precise placement for Diffusion Policy and SmolVLA.

• Next I will try reinforcement learning with human-in-the-loop corrections (HIL-SERL) on the same arm, with tighter placement tolerances. Not started yet.

Three Things I Learned

1. Where a policy fails matters as much as how often. Similar totals hid opposite failure stages, and the stage tells me what to fix next.

2. Change one variable at a time. I changed five things between my first and second ACT runs, so I can't say which one fixed the stall.

3. Inference speed is part of the design. My CPU laptop took about 1.2 seconds per ACT prediction and about 10 seconds per Diffusion Policy prediction, which shaped how each could run.

Data: View the dataset on Hugging Face.

It contains 201 recorded episodes. Built with LeRobot and the SO-101 arm from The RobotStudio and Hugging Face. Policy implementations are from LeRobot.