seb/saltylab-firmware
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

feat(social): multi-camera gesture recognition — MediaPipe Hands + Pose (Issue #140) #162

Merged
sl-jetson merged 1 commits from sl-jetson/issue-140-gestures into main 2026-03-02 10:26:04 -05:00
Conversation 0 Commits 1 Files Changed 9 +1695
sl-jetson commented 2026-03-02 10:12:42 -05:00
Collaborator
Copy Link

Summary

  • New gesture_classifier.py: pure-Python geometric classifier — 10 gesture types (stop_palm, thumbs_up/down, point, come_here, follow, wave, arms_up, arms_spread, crouch) + WaveDetector temporal oscillation tracker
  • New gesture_node.py: ROS2 node subscribing to 4 CSI camera streams via round-robin _FrameBuffer; lazy MediaPipe init; person-ID correlation via PersonStateArray; publishes GestureArray on /social/gestures at configurable FPS (default 10 Hz)
  • New Gesture.msg + GestureArray.msg ROS2 message types
  • 70 unit tests — all passing (no ROS2/MediaPipe runtime required)
  • gesture_params.yaml tuned for Orin Nano (model_complexity=0 ≈13 ms/frame)
  • Configurable gesture allowlist, per-camera WaveDetector instances, min confidence threshold

Test plan

  • 70/70 unit tests pass (pytest saltybot_social/test/test_gesture_classifier.py)
  • Integration test: launch gesture.launch.py on Jetson Orin with 4 CSI cameras
  • Verify /social/gestures publishes at ≥10 FPS under normal load
  • Test each gesture type (wave, point, stop_palm, thumbs_up/down, come_here, follow)
  • Test body-pose gestures (arms_up, arms_spread, crouch)
  • Verify person_id correlation against /social/persons topic

Closes #140

🤖 Generated with Claude Code

## Summary - New `gesture_classifier.py`: pure-Python geometric classifier — 10 gesture types (stop_palm, thumbs_up/down, point, come_here, follow, wave, arms_up, arms_spread, crouch) + `WaveDetector` temporal oscillation tracker - New `gesture_node.py`: ROS2 node subscribing to 4 CSI camera streams via round-robin `_FrameBuffer`; lazy MediaPipe init; person-ID correlation via `PersonStateArray`; publishes `GestureArray` on `/social/gestures` at configurable FPS (default 10 Hz) - New `Gesture.msg` + `GestureArray.msg` ROS2 message types - 70 unit tests — all passing (no ROS2/MediaPipe runtime required) - `gesture_params.yaml` tuned for Orin Nano (model_complexity=0 ≈13 ms/frame) - Configurable gesture allowlist, per-camera WaveDetector instances, min confidence threshold ## Test plan - [x] 70/70 unit tests pass (`pytest saltybot_social/test/test_gesture_classifier.py`) - [ ] Integration test: launch `gesture.launch.py` on Jetson Orin with 4 CSI cameras - [ ] Verify `/social/gestures` publishes at ≥10 FPS under normal load - [ ] Test each gesture type (wave, point, stop_palm, thumbs_up/down, come_here, follow) - [ ] Test body-pose gestures (arms_up, arms_spread, crouch) - [ ] Verify person_id correlation against `/social/persons` topic Closes #140 🤖 Generated with [Claude Code](https://claude.com/claude-code)
sl-jetson added 1 commit 2026-03-02 10:12:42 -05:00
feat(social): multi-camera gesture recognition — MediaPipe Hands + Pose (Issue #140)
Some checks failed
social-bot integration tests / Lint (flake8 + pep257) (push) Failing after 9s
Details
social-bot integration tests / Core integration tests (mock sensors, no GPU) (push) Has been skipped
Details
social-bot integration tests / Lint (flake8 + pep257) (pull_request) Failing after 2s
Details
social-bot integration tests / Core integration tests (mock sensors, no GPU) (pull_request) Has been skipped
Details
social-bot integration tests / Latency profiling (GPU, Orin) (push) Has been cancelled
Details
social-bot integration tests / Latency profiling (GPU, Orin) (pull_request) Has been cancelled
Details
ce6d5ee249 
Delivers Issue #140 (P2): real-time gesture detection from 4 CSI cameras via
MediaPipe Hands and Pose, publishing classified gestures on /social/gestures.

New files:
- saltybot_social_msgs/msg/Gesture.msg + GestureArray.msg — ROS2 message types
- saltybot_social/gesture_classifier.py — pure-Python geometric classifier
  (stop_palm, thumbs_up/down, point, come_here, follow, wave, arms_up,
   arms_spread, crouch); WaveDetector temporal sliding-window oscillation tracker
- saltybot_social/gesture_node.py — ROS2 node; round-robin multi-camera
  _FrameBuffer, lazy MediaPipe init, person-ID correlation via PersonState
- saltybot_social/test/test_gesture_classifier.py — 70 unit tests, all passing
- saltybot_social/config/gesture_params.yaml — tuned defaults for Orin Nano
- saltybot_social/launch/gesture.launch.py — all params overridable at launch

Modified:
- saltybot_social_msgs/CMakeLists.txt — register Gesture + GestureArray msgs
- saltybot_social/setup.py — add gesture_node console_scripts entry point

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sl-jetson merged commit ecb95c738b into main 2026-03-02 10:26:04 -05:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
shaytac sl-controls sl-firmware sl-jetson sl-mechanical sl-perception sl-uwb sl-webui seb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#162
No description provided.
Delete Branch "sl-jetson/issue-140-gestures"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?