seb/saltylab-firmware
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(perception): MediaPipe hand tracking — Leap Motion pivot (Issue #342) #345

Merged
sl-jetson merged 1 commits from sl-perception/issue-342-hand-tracking into main 2026-03-03 13:19:29 -05:00
Conversation 0 Commits 1 Files Changed 12 +1181
sl-perception commented 2026-03-03 12:47:46 -05:00
Collaborator
Copy Link

Part 1 — Audit

Zero Leap Motion / UltraLeap references found in any saltybot_* package (searched all .py, .cpp, .hpp, .h, .launch.py, .yaml, .xml, .txt, .json files).

Existing gesture_node.py in saltybot_social already uses MediaPipe (import mediapipe as mp with try/except) — no cleanup or migration required.

Part 2 — New packages

saltybot_hand_tracking_msgs (ament_cmake)

  • HandLandmarks.msg — per-hand result: 21 landmarks as float32[63] (x,y,z × 21), handedness, gesture label, pointing direction, wrist position
  • HandLandmarksArray.msg — array wrapper

saltybot_hand_tracking (ament_python)

_hand_gestures.py — pure-Python gesture classifier (no ROS2/MediaPipe deps)

Gesture Input pose Robot command
stop Open palm (4+ fingers extended) Pause/stop
point Index up, others curled Direction command (8-compass)
disarm Fist (all fingers + thumb curled) Emergency-off
confirm Thumbs-up Confirm action
follow_me Peace/victory (index+middle up) Follow mode
greeting Lateral wrist oscillation Greeting response

WaveDetector: sliding-window, min_reversals=2, min_amplitude=0.08, history=24 frames.

hand_tracking_node.py — ROS2 node

  • Subscribes: /camera/color/image_raw (BEST_EFFORT QoS)
  • Publishes: /saltybot/hands (HandLandmarksArray), /saltybot/hand_gesture (String)
  • MediaPipe model_complexity=0 (lite) for 20+ FPS target on Orin Nano Super
  • Background init thread (non-blocking startup)
  • Per-hand WaveDetector instances keyed by hand index
  • All tunable params declared as ROS2 parameters

Test plan

  • Landmark dataclass: fields, default z, frozen
  • HandGestureResult NamedTuple fields
  • Low-level helpers: _finger_up, _count_fingers_up, _four_fingers_curled, _thumb_curled, _thumb_extended_up
  • _point_direction: up, right, left, upper_right
  • WaveDetector: too few samples, oscillation trigger, small amplitude, reset
  • classify_hand: all 6 gestures, priority ordering, flat-hand no-crash, wrist position, confidence bounds, wave beats static, <21 landmarks → none
  • 35 tests, 35 passing

Closes #342

🤖 Generated with Claude Code

## Part 1 — Audit **Zero Leap Motion / UltraLeap references found** in any `saltybot_*` package (searched all `.py`, `.cpp`, `.hpp`, `.h`, `.launch.py`, `.yaml`, `.xml`, `.txt`, `.json` files). Existing `gesture_node.py` in `saltybot_social` **already uses MediaPipe** (`import mediapipe as mp` with try/except) — no cleanup or migration required. ## Part 2 — New packages ### `saltybot_hand_tracking_msgs` (ament_cmake) - `HandLandmarks.msg` — per-hand result: 21 landmarks as `float32[63]` (x,y,z × 21), handedness, gesture label, pointing direction, wrist position - `HandLandmarksArray.msg` — array wrapper ### `saltybot_hand_tracking` (ament_python) **`_hand_gestures.py`** — pure-Python gesture classifier (no ROS2/MediaPipe deps) | Gesture | Input pose | Robot command | |---------|-----------|---------------| | `stop` | Open palm (4+ fingers extended) | Pause/stop | | `point` | Index up, others curled | Direction command (8-compass) | | `disarm` | Fist (all fingers + thumb curled) | Emergency-off | | `confirm` | Thumbs-up | Confirm action | | `follow_me` | Peace/victory (index+middle up) | Follow mode | | `greeting` | Lateral wrist oscillation | Greeting response | `WaveDetector`: sliding-window, `min_reversals=2`, `min_amplitude=0.08`, `history=24` frames. **`hand_tracking_node.py`** — ROS2 node - Subscribes: `/camera/color/image_raw` (BEST_EFFORT QoS) - Publishes: `/saltybot/hands` (`HandLandmarksArray`), `/saltybot/hand_gesture` (`String`) - MediaPipe `model_complexity=0` (lite) for 20+ FPS target on Orin Nano Super - Background init thread (non-blocking startup) - Per-hand `WaveDetector` instances keyed by hand index - All tunable params declared as ROS2 parameters ## Test plan - [x] `Landmark` dataclass: fields, default z, frozen - [x] `HandGestureResult` NamedTuple fields - [x] Low-level helpers: `_finger_up`, `_count_fingers_up`, `_four_fingers_curled`, `_thumb_curled`, `_thumb_extended_up` - [x] `_point_direction`: up, right, left, upper_right - [x] `WaveDetector`: too few samples, oscillation trigger, small amplitude, reset - [x] `classify_hand`: all 6 gestures, priority ordering, flat-hand no-crash, wrist position, confidence bounds, wave beats static, <21 landmarks → none - **35 tests, 35 passing** Closes #342 🤖 Generated with [Claude Code](https://claude.com/claude-code)
sl-perception added 1 commit 2026-03-03 12:47:46 -05:00 
PART 1 AUDIT: Zero Leap Motion / UltraLeap references found in any
saltybot_* package. Existing gesture_node.py (saltybot_social) already
uses MediaPipe — no cleanup required.

PART 2 NEW PACKAGES:

saltybot_hand_tracking_msgs (ament_cmake)
  - HandLandmarks.msg   — 21 landmarks (float32[63]), handedness,
                          gesture label + direction, wrist position
  - HandLandmarksArray.msg

saltybot_hand_tracking (ament_python)
  - _hand_gestures.py   — pure-Python gesture classifier (no ROS2/MP deps)
                          Vocabulary: stop (open palm) → pause/stop,
                          point (index up) → direction command + 8-compass,
                          disarm (fist) → emergency-off,
                          confirm (thumbs-up) → confirm action,
                          follow_me (peace sign) → follow mode,
                          greeting (wrist oscillation) → greeting response
                          WaveDetector: sliding-window lateral wrist tracking
  - hand_tracking_node.py — ROS2 node
                          sub: /camera/color/image_raw (BEST_EFFORT)
                          pub: /saltybot/hands (HandLandmarksArray)
                               /saltybot/hand_gesture (std_msgs/String)
                          MediaPipe model_complexity=0 (lite) for 20+ FPS
                          on Orin Nano Super; background MP init thread;
                          per-hand WaveDetector instances
  - test/test_hand_gestures.py — 35 tests, 35 passing
    Covers: Landmark, HandGestureResult, WaveDetector, all 6 gesture
    classifiers, priority ordering, direction vectors, confidence bounds

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sl-jetson merged commit 3fce9bf577 into main 2026-03-03 13:19:29 -05:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
shaytac sl-controls sl-firmware sl-jetson sl-mechanical sl-perception sl-uwb sl-webui seb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#345
No description provided.
Delete Branch "sl-perception/issue-342-hand-tracking"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?