feat(perception): geometric face emotion classifier (Issue #359) #361

sl-perception · 2026-03-03T14:40:06-05:00

sl-perception commented

2026-03-03 14:40:06 -05:00

Closes #359

Summary

5 emotion classes: neutral / happy / surprised / angry / sad
Pure geometric rules on MediaPipe Face Mesh landmarks — no ML model at runtime
5 normalised features: mouth_open, smile (lip-corner elevation), brow_raise, brow_furl, eye_open — all relative to face height
MediaPipe FaceMesh initialised lazily in background thread (same pattern as hand tracking)
Publishes FaceEmotionArray to /saltybot/face_emotions at ≤15 fps

Classification rules (priority order)

Emotion	Trigger
surprised	brow_raise > 0.12 and eye_open > 0.07 and mouth_open > 0.07
happy	smile > 0.025 (corners above lip midpoint)
angry	brow_furl > 0.02 and smile < 0.01
sad	smile < −0.025 and brow_furl < 0.015
neutral	default

Test plan

python3 -m pytest test/test_face_emotion.py -v — 48/48 passed
Synthetic landmark fixtures verify each emotion class
Priority (surprised > happy tested)
Threshold crossing tests for happy/sad
Confidence capped at 1.0, no crash on degenerate geometry

🤖 Generated with Claude Code

Closes #359 ## Summary - 5 emotion classes: **neutral / happy / surprised / angry / sad** - Pure geometric rules on MediaPipe Face Mesh landmarks — no ML model at runtime - 5 normalised features: `mouth_open`, `smile` (lip-corner elevation), `brow_raise`, `brow_furl`, `eye_open` — all relative to face height - MediaPipe FaceMesh initialised lazily in background thread (same pattern as hand tracking) - Publishes `FaceEmotionArray` to `/saltybot/face_emotions` at ≤15 fps ## Classification rules (priority order) | Emotion | Trigger | |---------|--------| | surprised | brow_raise > 0.12 **and** eye_open > 0.07 **and** mouth_open > 0.07 | | happy | smile > 0.025 (corners above lip midpoint) | | angry | brow_furl > 0.02 **and** smile < 0.01 | | sad | smile < −0.025 **and** brow_furl < 0.015 | | neutral | default | ## Test plan - [x] `python3 -m pytest test/test_face_emotion.py -v` — 48/48 passed - [x] Synthetic landmark fixtures verify each emotion class - [x] Priority (surprised > happy tested) - [x] Threshold crossing tests for happy/sad - [x] Confidence capped at 1.0, no crash on degenerate geometry 🤖 Generated with [Claude Code](https://claude.com/claude-code)

sl-perception added 1 commit 2026-03-03 14:40:07 -05:00

feat(perception): geometric face emotion classifier (Issue #359 ) 672120bb50

Classifies facial expressions into neutral/happy/surprised/angry/sad
using geometric rules over MediaPipe Face Mesh landmarks — no ML model
required at runtime.

Rules
-----
  surprised: brow_raise > 0.12 AND eye_open > 0.07 AND mouth_open > 0.07
  happy:     smile > 0.025  (lip corners above lip midpoint)
  angry:     brow_furl > 0.02 AND smile < 0.01
  sad:       smile < -0.025 AND brow_furl < 0.015
  neutral:   default

Changes
-------
- saltybot_scene_msgs/msg/FaceEmotion.msg       — per-face emotion + features
- saltybot_scene_msgs/msg/FaceEmotionArray.msg
- saltybot_scene_msgs/CMakeLists.txt            — register new msgs
- _face_emotion.py   — pure-Python: FaceLandmarks, compute_features,
                        classify_emotion, detect_emotion, from_mediapipe
- face_emotion_node.py  — subscribes /camera/color/image_raw,
                           publishes /saltybot/face_emotions (≤15 fps)
- test/test_face_emotion.py  — 48 tests, all passing
- setup.py  — add face_emotion entry point

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sl-jetson merged commit bcf848109b into main

2026-03-03 15:07:17 -05:00

sl-jetson referenced this issue from a commit

2026-03-03 15:07:19 -05:00

Merge pull request 'feat(perception): geometric face emotion classifier (Issue #359)' (#361) from sl-perception/issue-359-face-emotion into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#361