seb/saltylab-firmware
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(social): facial expression recognition — TRT FP16 emotion CNN (Issue #161) #180

Merged
sl-jetson merged 1 commits from sl-jetson/issue-161-emotion into main 2026-03-02 10:46:22 -05:00
Conversation 0 Commits 1 Files Changed 10 +1461
sl-jetson commented 2026-03-02 10:41:15 -05:00
Collaborator
Copy Link

Issue #161 — Emotion Detection: Facial Expression Recognition

What

7-class real-time facial expression classifier piggybacking on the SCRFD face detection pipeline.

Changes

New message types (saltybot_social_msgs):

  • Expression.msg — per-face emotion with person_id, emotion, confidence, scores[7], is_opt_out, source
  • ExpressionArray.msg — array wrapper with header

New nodes (saltybot_social):

  • emotion_node.py — subscribes /social/faces/detections, runs TRT FP16 crop inference, publishes /social/faces/expressions (ExpressionArray) and /social/emotion/context (JSON for LLM)
  • emotion_classifier.py — TensorRT FP16 engine wrapper + 5-point landmark geometry fallback + EMA per-person smoothing + opt-out registry

LLM integration (conversation_node.py):

  • Subscribes /social/emotion/context; injects emotion hint (e.g. [The person seems sad right now.]) into prompt when speaker shows non-neutral affect

Config / launch:

  • config/emotion_params.yaml — tuned defaults, includes TRT build command
  • launch/emotion.launch.py — all params overridable from CLI

Design

  • < 5 ms additional latency — no extra camera polling, TRT FP16 @ 48×48 on Orin ~2–3 ms/crop
  • Graceful degradation — landmark_fallback=true when TRT engine unavailable
  • Privacy — per-person opt-out; opted-out faces published with is_opt_out=True, emotion=""
  • Smoothing — configurable EMA alpha (default 0.30) with age expiry (16 frames)

Tests

67/67 passing (test_emotion_classifier.py): classifier, PersonEmotionTracker, opt-out, landmark heuristic, edge cases

Closes #161

## Issue #161 — Emotion Detection: Facial Expression Recognition ### What 7-class real-time facial expression classifier piggybacking on the SCRFD face detection pipeline. ### Changes **New message types** (`saltybot_social_msgs`): - `Expression.msg` — per-face emotion with `person_id`, `emotion`, `confidence`, `scores[7]`, `is_opt_out`, `source` - `ExpressionArray.msg` — array wrapper with header **New nodes** (`saltybot_social`): - `emotion_node.py` — subscribes `/social/faces/detections`, runs TRT FP16 crop inference, publishes `/social/faces/expressions` (ExpressionArray) and `/social/emotion/context` (JSON for LLM) - `emotion_classifier.py` — TensorRT FP16 engine wrapper + 5-point landmark geometry fallback + EMA per-person smoothing + opt-out registry **LLM integration** (`conversation_node.py`): - Subscribes `/social/emotion/context`; injects emotion hint (e.g. `[The person seems sad right now.]`) into prompt when speaker shows non-neutral affect **Config / launch**: - `config/emotion_params.yaml` — tuned defaults, includes TRT build command - `launch/emotion.launch.py` — all params overridable from CLI ### Design - **< 5 ms** additional latency — no extra camera polling, TRT FP16 @ 48×48 on Orin ~2–3 ms/crop - **Graceful degradation** — `landmark_fallback=true` when TRT engine unavailable - **Privacy** — per-person opt-out; opted-out faces published with `is_opt_out=True, emotion=""` - **Smoothing** — configurable EMA alpha (default 0.30) with age expiry (16 frames) ### Tests 67/67 passing (`test_emotion_classifier.py`): classifier, PersonEmotionTracker, opt-out, landmark heuristic, edge cases Closes #161
sl-jetson added 1 commit 2026-03-02 10:41:15 -05:00
feat(social): facial expression recognition — TRT FP16 emotion CNN (Issue #161)
Some checks failed
social-bot integration tests / Lint (flake8 + pep257) (push) Failing after 2s
Details
social-bot integration tests / Core integration tests (mock sensors, no GPU) (push) Has been skipped
Details
social-bot integration tests / Lint (flake8 + pep257) (pull_request) Failing after 2s
Details
social-bot integration tests / Core integration tests (mock sensors, no GPU) (pull_request) Has been skipped
Details
social-bot integration tests / Latency profiling (GPU, Orin) (push) Has been cancelled
Details
social-bot integration tests / Latency profiling (GPU, Orin) (pull_request) Has been cancelled
Details
50971c0946 
- Add Expression.msg / ExpressionArray.msg ROS2 message definitions
- Add emotion_classifier.py: 7-class CNN (happy/sad/angry/surprised/fearful/disgusted/neutral)
  via TensorRT FP16 with landmark-geometry fallback; EMA per-person smoothing; opt-out registry
- Add emotion_node.py: subscribes /social/faces/detections, runs TRT crop inference (<5ms),
  publishes /social/faces/expressions and /social/emotion/context JSON for LLM
- Wire emotion context into conversation_node.py: emotion hint injected into LLM prompt
  when speaker shows non-neutral affect; subscribes /social/emotion/context
- Add emotion_params.yaml config and emotion.launch.py launch file
- Add 67-test suite (test_emotion_classifier.py): classifier, tracker, opt-out, heuristic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sl-jetson merged commit 9ca0e0844c into main 2026-03-02 10:46:22 -05:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
shaytac sl-controls sl-firmware sl-jetson sl-mechanical sl-perception sl-uwb sl-webui seb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#180
No description provided.
Delete Branch "sl-jetson/issue-161-emotion"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?