feat(webui): hand tracking skeleton visualization (Issue #344) #346

Merged
sl-jetson merged 1 commits from sl-webui/issue-344-hand-viz into main 2026-03-03 13:19:47 -05:00

1 Commits

Author SHA1 Message Date
347449ed95 feat(webui): hand pose tracking and gesture visualization (Issue #344)
Features:
- Subscribes to /saltybot/hands (21 landmarks per hand - MediaPipe format)
- Subscribes to /saltybot/hand_gesture (String gesture label)
- Canvas-based hand skeleton rendering with bone connections
- Support for dual hand tracking (left and right)
- Handedness indicators with color coding
  * Left hand: green
  * Right hand: yellow
- Real-time gesture display with confidence indicator
- Per-landmark confidence visualization
- Bone connections between all 21 joints

Hand Skeleton Features:
- 21 MediaPipe landmarks per hand
  * Wrist (1)
  * Thumb (4)
  * Index finger (4)
  * Middle finger (4)
  * Ring finger (4)
  * Pinky finger (4)
- 20 bone connections between joints
- Confidence-based rendering (only show high-confidence points)
- Scaling and normalization for viewport
- Joint type indicators (tips with ring outline)
- Glow effects around landmarks

Gesture Recognition:
- Real-time gesture label display
- Confidence percentage (0-100%)
- Color-coded confidence:
  * Green: >80% (high confidence)
  * Yellow: 50-80% (medium confidence)
  * Blue: <50% (detecting)

Hand Status Display:
- Live detection status for both hands
- Visual indicators (✓ detected / ◯ not detected)
- Dual-hand canvas rendering
- Gesture info panel with confidence bar

Integration:
- Added to SOCIAL tab group as "Hands" tab
- Positioned after "Faces" tab
- Uses subscribe hook for real-time updates
- Dark theme with color-coded hands
- Canvas-based rendering for smooth visualization

Build: 125 modules, no errors
Main bundle: 270.08 KB

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-03-03 12:43:19 -05:00