seb/saltylab-firmware
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

[P1] Leap Motion SDK has no ARM64 support — pivot hand tracking to MediaPipe #342

New Issue
Closed
opened 2026-03-03 12:41:01 -05:00 by seb · 0 comments
seb commented 2026-03-03 12:41:01 -05:00
Owner
Copy Link

Problem

The Ultraleap/Leap Motion SDK (both V2 and Gemini V5+) is x86_64 only. No ARM64 binaries exist, no cross-compile path, and leapd is proprietary with no source.

The Orin Nano runs aarch64 Linux — Leap Motion hardware is a dead end on this platform.

Current State

  • Leap Motion Controller (f182:0003) is plugged into the Orin USB
  • Device is detected by the kernel but no SDK can talk to it
  • MediaPipe Hands (0.10.18) is already installed and working on the Orin
  • Hand tracking test confirmed: RealSense color stream → MediaPipe → 21 landmarks per hand, gesture classification

Required Changes

1. Replace any Leap Motion references in the codebase

  • Check all saltybot_* packages for Leap Motion imports or dependencies
  • The merged PR #226 (Issue #221 — hand gesture pointing direction) may reference Leap — needs audit

2. Create saltybot_hand_tracking ROS2 package

  • Subscribe to RealSense color topic (/camera/color/image_raw)
  • Run MediaPipe Hands inference
  • Publish:
    • /saltybot/hands — custom msg with landmarks, handedness, gesture
    • /saltybot/hand_gesture — string topic with classified gesture (Fist, Open, Point, Peace, Thumbs Up, etc.)
  • Target: 20+ FPS on Orin Nano Super

3. Gesture vocabulary for robot control

  • Open hand → stop/pause
  • Point → go in pointed direction
  • Fist → disarm
  • Thumbs up → acknowledge/confirm
  • Peace → follow me mode
  • Wave (detect via wrist oscillation) → greeting response

4. Integration with existing nodes

  • saltybot_follower should accept hand gesture overrides
  • saltybot_social personality should react to gestures
  • Web UI should show hand skeleton + gesture label (already prototyped in viz dashboard)

MediaPipe Setup (already done on Orin)

pip3 install mediapipe opencv-python-headless
# Working: 640x480 @30fps via RealSense, gesture detection confirmed

Decision

Leap Motion hardware can stay plugged in as a raw IR stereo camera (accessible via V4L2/OpenCV as two IR streams) but all hand tracking must go through MediaPipe.

/cc @max

## Problem The Ultraleap/Leap Motion SDK (both V2 and Gemini V5+) is **x86_64 only**. No ARM64 binaries exist, no cross-compile path, and `leapd` is proprietary with no source. The Orin Nano runs aarch64 Linux — Leap Motion hardware is a dead end on this platform. ## Current State - Leap Motion Controller (f182:0003) is plugged into the Orin USB - Device is detected by the kernel but **no SDK can talk to it** - MediaPipe Hands (0.10.18) is already installed and working on the Orin - Hand tracking test confirmed: RealSense color stream → MediaPipe → 21 landmarks per hand, gesture classification ## Required Changes ### 1. Replace any Leap Motion references in the codebase - Check all `saltybot_*` packages for Leap Motion imports or dependencies - The merged PR #226 (Issue #221 — hand gesture pointing direction) may reference Leap — needs audit ### 2. Create `saltybot_hand_tracking` ROS2 package - Subscribe to RealSense color topic (`/camera/color/image_raw`) - Run MediaPipe Hands inference - Publish: - `/saltybot/hands` — custom msg with landmarks, handedness, gesture - `/saltybot/hand_gesture` — string topic with classified gesture (Fist, Open, Point, Peace, Thumbs Up, etc.) - Target: 20+ FPS on Orin Nano Super ### 3. Gesture vocabulary for robot control - **Open hand** → stop/pause - **Point** → go in pointed direction - **Fist** → disarm - **Thumbs up** → acknowledge/confirm - **Peace** → follow me mode - **Wave** (detect via wrist oscillation) → greeting response ### 4. Integration with existing nodes - `saltybot_follower` should accept hand gesture overrides - `saltybot_social` personality should react to gestures - Web UI should show hand skeleton + gesture label (already prototyped in viz dashboard) ## MediaPipe Setup (already done on Orin) ```bash pip3 install mediapipe opencv-python-headless # Working: 640x480 @30fps via RealSense, gesture detection confirmed ``` ## Decision Leap Motion hardware can stay plugged in as a raw IR stereo camera (accessible via V4L2/OpenCV as two IR streams) but **all hand tracking must go through MediaPipe**. /cc @max
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
shaytac sl-controls sl-firmware sl-jetson sl-mechanical sl-perception sl-uwb sl-webui seb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#342
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?