feat(social): hand gesture pointing direction node (Issue #221) #226

sl-perception · 2026-03-02T11:57:12-05:00

sl-perception commented

2026-03-02 11:57:12 -05:00

Summary

saltybot_social_msgs: Add PointingTarget.msg (3-D ray origin/direction/target, range, person_id, confidence, coarse_direction, is_active); register in CMakeLists.txt
saltybot_social/_pointing_ray.py (pure Python, no rclpy): unproject() pixel+depth→3-D; sample_depth() median with outlier rejection; compute_pointing_ray() — reprojects INDEX_MCP (knuckle) and INDEX_TIP into 3-D using D435i depth, forms unit direction vector, handles single-side missing depth gracefully, falls back to image-plane direction when 3-D points coincide
saltybot_social/pointing_node.py: subscribes /social/gestures + ApproximateTimeSynchronizer on D435i colour+depth; re-runs MediaPipe Hands only when a fresh point gesture is cached; picks hand closest to gesture anchor; publishes /saltybot/pointing_target (PointingTarget) at 5 Hz via timer
18/18 unit tests pass

Closes #221

Test plan

python3 -m pytest test/test_pointing.py -v — 18 passed
Point at an object in front of D435i, verify /saltybot/pointing_target publishes with is_active=true
Verify target XYZ matches the pointed-at surface
Verify is_active goes false after gesture timeout

🤖 Generated with Claude Code

## Summary - **`saltybot_social_msgs`**: Add `PointingTarget.msg` (3-D ray origin/direction/target, range, person_id, confidence, coarse_direction, is_active); register in CMakeLists.txt - **`saltybot_social/_pointing_ray.py`** (pure Python, no rclpy): `unproject()` pixel+depth→3-D; `sample_depth()` median with outlier rejection; `compute_pointing_ray()` — reprojects INDEX_MCP (knuckle) and INDEX_TIP into 3-D using D435i depth, forms unit direction vector, handles single-side missing depth gracefully, falls back to image-plane direction when 3-D points coincide - **`saltybot_social/pointing_node.py`**: subscribes `/social/gestures` + ApproximateTimeSynchronizer on D435i colour+depth; re-runs MediaPipe Hands only when a fresh `point` gesture is cached; picks hand closest to gesture anchor; publishes `/saltybot/pointing_target` (PointingTarget) at 5 Hz via timer - 18/18 unit tests pass Closes #221 ## Test plan - [ ] `python3 -m pytest test/test_pointing.py -v` — 18 passed - [ ] Point at an object in front of D435i, verify `/saltybot/pointing_target` publishes with `is_active=true` - [ ] Verify `target` XYZ matches the pointed-at surface - [ ] Verify `is_active` goes false after gesture timeout 🤖 Generated with [Claude Code](https://claude.com/claude-code)

sl-perception added 1 commit 2026-03-02 11:57:13 -05:00

feat(social): hand gesture pointing direction node (Issue #221 )

social-bot integration tests / Lint (flake8 + pep257) (pull_request) Failing after 2s

Details

social-bot integration tests / Core integration tests (mock sensors, no GPU) (pull_request) Has been skipped

Details

social-bot integration tests / Latency profiling (GPU, Orin) (pull_request) Has been cancelled

Details

bd4a9f094d

saltybot_social_msgs:
- Add PointingTarget.msg: origin (INDEX_MCP), direction (unit vec), target,
  range_m, person_id, confidence, coarse_direction, is_active
- Register in CMakeLists.txt

saltybot_social:
- _pointing_ray.py (pure Python, no rclpy): unproject(), sample_depth()
  (median with outlier rejection), compute_pointing_ray() — reprojects
  INDEX_MCP and INDEX_TIP into 3-D using D435i depth; falls back to image-
  plane direction when both depths are equal; gracefully handles one-sided
  missing depth
- pointing_node.py: subscribes /social/gestures + synced D435i colour+depth;
  re-runs MediaPipe Hands when a 'point' gesture is cached (within
  gesture_timeout_s); picks closest hand to gesture anchor; publishes
  PointingTarget on /saltybot/pointing_target at 5 Hz
- setup.py: adds pointing_node entry point
- 18/18 unit tests pass

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sl-jetson merged commit 213d7fe13d into main

2026-03-02 11:57:27 -05:00

sl-jetson referenced this issue from a commit

2026-03-02 11:57:28 -05:00

Merge pull request 'feat(social): hand gesture pointing direction node (Issue #221)' (#226) from sl-perception/issue-221-pointing into main

seb referenced this pull request

2026-03-03 12:41:01 -05:00

[P1] Leap Motion SDK has no ARM64 support — pivot hand tracking to MediaPipe #342

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#226