Engineering

Robotics

Computer vision

Cup Tracking for a Four-Robot Trail Mix Line

A ROS perception module that reports where each cup is, in each robot’s own coordinate frame, for a four-arm Franka Panda assembly line.

Role

Module owner, cup tracking

Team

Team 4, four people

Timeline

March to May 2026

Context

MEAM 5200 Introduction to Robotics, University of Pennsylvania

Tools

ROS, Python, apriltag_ros, TF, Franka Panda arms

Code

GitHub repository

20

teams building one system

4

Franka Panda arms

8

robot teams used the cup poses

10 Hz

pose publishing rate

25

cups, 3 tags each

Twenty teams built a four-arm robotic trail mix line for the ROBO MS thesis reception at Penn. I owned cup tracking: a ROS module that detects AprilTags on each cup with a wrist camera, combines them with the arm’s forward kinematics, and publishes the cup’s location in the robot’s base frame at 10 Hz. Eight robot teams used those poses to handle cups, and the full line ran at the May 8 demo.

01 Context

Every robot in the line needed to know where the cups were

The Trail Mix Bar was a class-wide project in MEAM 5200. Twenty four-person teams each built one module of a single system: four Franka Panda arms arranged in two assembly lines. A service robot takes a cup from a stack and adds cereal, passes it through a turntable to a toppings robot that adds nuts and candy, then mixes and serves it. The robots work in separate spaces and never hand cups directly to each other.

Course staff provided a basic AprilTag detector that reports tag poses in the camera frame. Nothing turned those detections into a position a robot could plan against. That gap was my module.

S1 1.27:1

Layout of the Trail Mix Bar from the course project brief. Cups start in a stack beside each service robot.

Top-down layout of the Trail Mix Bar: two assembly lines, each with a service robot, a turntable, and a toppings robot, with cereal, candy, and nut dispensers and a service counter at left.

02 Scope

The module reports one thing, and reports it in a form every robot can use

My team and I started with a written compact that fixed what the module would own and what it would not. It owns detecting cups and publishing their pose in the robot base frame. It does not own dispenser or turntable tracking, planning, scheduling, or camera hardware.

The output is a single message, CupDetection, published on /cup_state at 10 Hz. It carries a pose and a tracking status, and the status switches to not_visible when no detection arrives for 0.5 seconds. Early drafts also carried a cup ID, an order ID, a location, and the assembly state. I cut them. The scheduler is the source of truth for what is in a cup and where it is in the build, and a camera can only report what it currently sees.

Consumers hold on to the last good pose, so a brief occlusion does not stall a robot.

How a detection becomes a cup pose

Wrist camera

Image stream from the arm

AprilTag detection

Tag pose in the camera frame

Tag pose array

Detections collected per robot

cupTracker

Forward kinematics and camera offset

/cup_state

Pose and status in the base frame, 10 Hz

03 Tags

Where the tags go decided how the whole vision pipeline works

The cups are small and clear, and the camera is mounted on the robot’s wrist, so the arm has to be in the right place before anything can be seen. That made tag placement the most important design decision in the module.

I weighed sweeping the camera to find a single tag against fixing several tags to each cup. I chose three tags per cup: two identical tags on the sides and one unique tag on the bottom. A top-down camera pose reads the bottom tag from above, including on the top cup of a stack. An angled side pose reads the side tags when a cup sits under a dispenser. The second side tag gives a backup when a cup comes back from the turntable rotated. The tags are taped on their edges only, to avoid glare on the clear plastic.

S3A 1.1:1

Each cup carries two side tags and one bottom tag. Arrows mark tag orientation.

A clear plastic cup with three AprilTags, two on the sides and one on the bottom, with arrows marking each tag’s orientation.

I started with tag25h9 and moved to tag36h11, the 6x6 family, after Team 3 reported false positives. The tag IDs were split so detections never overlapped: Team 3 used 0 to 25 for the dispensers and turntable, and the cups use 26 and up. Tags were reliable within about 20 centimeters, so both camera poses keep the camera close to the cup.

Two camera poses cover every cup.

Robot pose

Camera view

Side pose: reads the side tags

S3B 1:1

A Franka Panda arm with a wrist camera angled at a clear cup under a black food dispenser.

S3C 1:1

Wrist camera view of a cup in a black tray with an AprilTag visible on its side.

Top-down pose: reads the bottom tag

S3D 1:1

A Franka Panda arm with a wrist camera pointing down at a stack of clear cups on a wooden bench.

S3E 1:1

Top-down camera view of the inside of a cup with an AprilTag on its bottom and detection axes drawn on it.

04 Pose

A tag pose in the camera frame becomes a cup pose in the robot’s base frame

The detector says where a tag is relative to the camera. A robot plans in its own base frame. I closed the gap by chaining three transforms: the camera-to-tag pose from the detector, the hand-to-camera transform from the TF tree with a measured fallback, and the hand pose in the base frame from forward kinematics on the live joint state. Each tag sits at a fixed place on its cup, so the tag’s pose locates the cup.

The node waits for the first joint state before publishing, so it never reports a pose computed from zeroed joints. When the tag disappears, the status changes to not_visible and the last pose is held.

T_base_tag = T_base_ee(q) · T_ee_camera · T_cam_tag

Base-to-hand pose from forward kinematics, hand-to-camera transform, camera-to-tag detection.

05 Teams

Eight teams built on the module, so the interface mattered as much as the vision

Every robot team that handled a cup depended on this module: service robot teams for picking cups, turntable exchange, cereal dispensing, mixing, and serving, and toppings robot teams for turntable exchange, candy, and nuts. I worked with them on three things. First, the message: they needed a pose and a status, and nothing about assembly state. Second, the two camera poses that put each robot’s camera where it could see a cup. Third, the tag conventions on the cups.

With Team 3, who tracked the dispensers and the turntable, I agreed the tag family and ID ranges so our detections never collided. I shared the tracking poses, the message definition, and the tag conventions with every consuming team.

Who used it

Service robot:

Pick cups, Turntable exchange, Cereal dispense, Mix, Serve.

Toppings robot:

Turntable exchange, Candy dispense, Nuts dispense.

06 Demo

The same module ran on all four robots and through the full demo

One codebase ran on every arm, with a launch argument selecting each robot’s joint-state topic. Integration turned up three bugs. The joint-state topic resolved in the wrong namespace, so forward kinematics ran with every joint at zero and the cup appeared to move with the camera. A topic remap pointed at a dead link. And robots shared one tag pose topic, so detections crossed between arms. I fixed each with per-robot parameters and remaps, and added the joint-state guard described above. Because each bug had a distinct symptom, I debugged the transform chain link by link.

On May 8, the full line ran at the ROBO MS thesis reception, and the downstream teams used the cup poses from /cup_state to move cups through the pipeline.

Full demo, portrait recording.

07 Takeaways

Three things I took from it

Tag choices set the pipeline.

Tag family, count, and placement fixed where the camera had to go and what the detector could handle. I would settle all three together, with the neighboring team, before writing detection code.

A narrow interface lets other teams move on their own.

A pose and a status were enough for eight teams to build on without knowing how detection worked. Keeping assembly state out of the perception message kept the module small and left the scheduler in charge of what only it knows.

Own the calibration checkpoint.

A depth offset of roughly 1.5 to 2 centimeters appeared on three of the four arms, most likely from a hand-to-camera transform carried over from earlier work instead of measured per robot. Next time I would calibrate each robot’s camera transform and compare forward kinematics against the robot model before integration.

View the code on GitHub

Team 4: Sidhant Srivastava, Caroline Chen, Pavitpaul Makkar, Aiden Lee. Built on the course-provided AprilTag detector, with Team 3 (World Tracking) on tag conventions. MEAM 5200 Introduction to Robotics, University of Pennsylvania, Spring 2026.

By Pavitpaul Makkar

By Pavitpaul Makkar

By Pavitpaul Makkar