14 Commits

Author	SHA1	Message	Date
sl-perception	24340dea9b	feat(social): add 68-point Kalman landmark smoother (Issue #227) ... Adds per-person constant-velocity Kalman filter that smooths raw 68-point facial landmarks and republishes on /social/faces/landmarks_smooth at input rate. New FaceLandmarks / FaceLandmarksArray messages added to saltybot_social_msgs. 21/21 pure-Python tests pass (no ROS2 required). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 12:05:24 -05:00
sl-perception	bd4a9f094d	feat(social): hand gesture pointing direction node (Issue #221) ... saltybot_social_msgs: - Add PointingTarget.msg: origin (INDEX_MCP), direction (unit vec), target, range_m, person_id, confidence, coarse_direction, is_active - Register in CMakeLists.txt saltybot_social: - _pointing_ray.py (pure Python, no rclpy): unproject(), sample_depth() (median with outlier rejection), compute_pointing_ray() — reprojects INDEX_MCP and INDEX_TIP into 3-D using D435i depth; falls back to image- plane direction when both depths are equal; gracefully handles one-sided missing depth - pointing_node.py: subscribes /social/gestures + synced D435i colour+depth; re-runs MediaPipe Hands when a 'point' gesture is cached (within gesture_timeout_s); picks closest hand to gesture anchor; publishes PointingTarget on /saltybot/pointing_target at 5 Hz - setup.py: adds pointing_node entry point - 18/18 unit tests pass Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 11:56:54 -05:00
sl-jetson	e5236f781b	feat(social): robot mesh comms — peer announce + person handoff (Issue #171) ... New MeshPeer.msg (1 Hz DDS heartbeat: robot_id, social_state, active persons, greeted names) and MeshHandoff.msg (person context transfer on STATE_LEAVING). mesh_comms_node subscribes to person_states and orchestrator/state, publishes announce heartbeat, triggers handoff on LEAVING, tracks peers with timeout cleanup, and propagates mesh-wide greeting deduplication via /social/mesh/greeted. 73/73 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 11:14:00 -05:00
sl-jetson	90c8b427fc	feat(social): multi-language support — Whisper LID + per-lang Piper TTS (Issue #167) ... - Add SpeechTranscript.language (BCP-47), ConversationResponse.language fields - speech_pipeline_node: whisper_language param (""=auto-detect via Whisper LID); detected language published in every transcript - conversation_node: track per-speaker language; inject "[Please respond in X.]" hint for non-English speakers; propagate language to ConversationResponse. _LANG_NAMES: 24 BCP-47 codes -> English names. Also adds Issue #161 emotion context plumbing (co-located in same branch for clean merge) - tts_node: voice_map_json param (JSON BCP-47->ONNX path); lazy voice loading per language; playback queue now carries (text, lang) tuples for voice routing - speech_params.yaml, tts_params.yaml: new language params with docs - 47/47 tests pass (test_multilang.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:57:34 -05:00
sl-jetson	50971c0946	feat(social): facial expression recognition — TRT FP16 emotion CNN (Issue #161) ... - Add Expression.msg / ExpressionArray.msg ROS2 message definitions - Add emotion_classifier.py: 7-class CNN (happy/sad/angry/surprised/fearful/disgusted/neutral) via TensorRT FP16 with landmark-geometry fallback; EMA per-person smoothing; opt-out registry - Add emotion_node.py: subscribes /social/faces/detections, runs TRT crop inference (<5ms), publishes /social/faces/expressions and /social/emotion/context JSON for LLM - Wire emotion context into conversation_node.py: emotion hint injected into LLM prompt when speaker shows non-neutral affect; subscribes /social/emotion/context - Add emotion_params.yaml config and emotion.launch.py launch file - Add 67-test suite (test_emotion_classifier.py): classifier, tracker, opt-out, heuristic Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:40:54 -05:00
sl-jetson	ce6d5ee249	feat(social): multi-camera gesture recognition — MediaPipe Hands + Pose (Issue #140) ... Delivers Issue #140 (P2): real-time gesture detection from 4 CSI cameras via MediaPipe Hands and Pose, publishing classified gestures on /social/gestures. New files: - saltybot_social_msgs/msg/Gesture.msg + GestureArray.msg — ROS2 message types - saltybot_social/gesture_classifier.py — pure-Python geometric classifier (stop_palm, thumbs_up/down, point, come_here, follow, wave, arms_up, arms_spread, crouch); WaveDetector temporal sliding-window oscillation tracker - saltybot_social/gesture_node.py — ROS2 node; round-robin multi-camera _FrameBuffer, lazy MediaPipe init, person-ID correlation via PersonState - saltybot_social/test/test_gesture_classifier.py — 70 unit tests, all passing - saltybot_social/config/gesture_params.yaml — tuned defaults for Orin Nano - saltybot_social/launch/gesture.launch.py — all params overridable at launch Modified: - saltybot_social_msgs/CMakeLists.txt — register Gesture + GestureArray msgs - saltybot_social/setup.py — add gesture_node console_scripts entry point Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:10:54 -05:00
sl-jetson	fabb988e72	feat(social): voice command NLU — 30+ intents, confirmation flow (Issue #137) ... ## New files - saltybot_social_msgs/msg/VoiceCommand.msg intent + entities[] + confidence + confirmation_token + requires_confirmation - saltybot_social/voice_command_parser.py Pure-Python regex NLU, zero ROS2/ML deps, < 1 ms/call 30+ named intents across nav.*, social.*, system.*, config.*, confirm.* Entity extraction: location, name, mode, level, route name Dangerous-command flag: system.shutdown, system.restart, social.forget_me - saltybot_social/voice_command_node.py Subscribes /social/speech/transcript, publishes /social/voice_command Confirmation flow with UUID token + 10 s timeout Below-threshold → intent=fallback → LLM conversation engine - saltybot_social/test/test_voice_command_parser.py 191 unit tests (all pass), no ROS2 runtime required - saltybot_social/config/voice_command_params.yaml - saltybot_social/launch/voice_command.launch.py ## Intent taxonomy nav: go_to, go_home, follow_me, stop, wait, come_here, patrol, set_mode (shadow/lead/side/orbit/loose/tight), teach_route, stop_teaching, replay_route social: remember_me, forget_me [CONFIRM], whats_my_name, tell_joke system: battery_status, map_status, shutdown [CONFIRM], restart [CONFIRM], volume_up, volume_down, volume_set config: personality, sass_level, follow_mode ## Updated - saltybot_social_msgs/CMakeLists.txt: register VoiceCommand.msg - saltybot_social/setup.py: add voice_command_node entry point Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 09:38:17 -05:00
sl-jetson	5043578934	feat(social): speech pipeline + LLM conversation + TTS + orchestrator (#81 #83 #85 #89) ... Issue #81 — Speech pipeline: - speech_pipeline_node.py: OpenWakeWord "hey_salty" → Silero VAD → faster-whisper STT (Orin GPU, <500ms wake-to-transcript) → ECAPA-TDNN speaker diarization - speech_utils.py: pcm16↔float32, EnergyVad, UtteranceSegmenter (pre-roll, max- duration), cosine speaker identification — all pure Python, no ROS2/GPU needed - Publishes /social/speech/transcript (SpeechTranscript) + /social/speech/vad_state Issue #83 — Conversation engine: - conversation_node.py: llama-cpp-python GGUF (Phi-3-mini Q4_K_M, 20 GPU layers), streaming token output, per-person sliding-window context (4K tokens), summary compression, SOUL.md system prompt, group mode - llm_context.py: PersonContext, ContextStore (JSON persistence), build_llama_prompt (ChatML format), context compression via LLM summarization - Publishes /social/conversation/response (ConversationResponse, partial + final) Issue #85 — Streaming TTS: - tts_node.py: Piper ONNX streaming synthesis, sentence-by-sentence first-chunk streaming (<200ms to first audio), sounddevice USB speaker playback, volume control - tts_utils.py: split_sentences, pcm16_to_wav_bytes, chunk_pcm, apply_volume, strip_ssml Issue #89 — Pipeline orchestrator: - orchestrator_node.py: IDLE→LISTENING→THINKING→SPEAKING state machine, GPU memory watchdog (throttle at <2GB free), rolling latency stats (p50/p95 per stage), VAD watchdog (alert if speech pipeline hangs), /social/orchestrator/state JSON pub - social_bot.launch.py: brings up all 4 nodes with TimerAction delays New messages: SpeechTranscript.msg, VadState.msg, ConversationResponse.msg Config YAMLs: speech_params, conversation_params, tts_params, orchestrator_params Tests: 58 tests (28 speech_utils + 30 llm_context/tts_utils), all passing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 08:23:19 -05:00
sl-controls	fa0162fadc	feat(social): multi-modal tracking fusion — UWB+camera Kalman filter (Issue #92) ... New packages: saltybot_social_msgs — FusedTarget.msg custom message saltybot_social_tracking — 4-state Kalman fusion node saltybot_social_tracking/tracking_fusion_node.py Subscribes to /uwb/target (PoseStamped, ~10 Hz) and /person/target (PoseStamped, ~30 Hz) and publishes /social/tracking/fused_target (FusedTarget) at 20 Hz. Source arbitration: • "fused" — both UWB and camera are fresh; confidence-weighted blend • "uwb" — UWB fresh, camera stale • "camera" — camera fresh, UWB stale • "predicted" — all sources stale; KF coasts for up to predict_timeout (3 s) Kalman filter (kalman_tracker.py): State [x, y, vx, vy] with discrete Wiener acceleration noise model (process_noise=3.0 m/s²) sized for EUC speeds (20-30 km/h, ≈5.5-8.3 m/s). Separate UWB (0.20 m) and camera (0.12 m) measurement noise. Velocity estimate converges after ~3 s of 10 Hz UWB measurements. Confidence model (source_arbiter.py): Per-source confidence = quality × max(0, 1 - age/timeout). Composite confidence accounts for KF positional uncertainty and is capped at 0.4 during dead-reckoning ("predicted") mode. Tests: 58/58 pass (no ROS2 runtime required). Note: saltybot_social_msgs here adds FusedTarget.msg; PR #98 (Issue #84) adds PersonalityState.msg + QueryMood.srv to the same package. The maintainer should squash-merge #98 first and rebase this branch on top of it before merging to avoid the package.xml conflict. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-01 23:59:10 -05:00
sl-controls	44771751e2	feat(social): personality system — SOUL.md persona, mood engine, relationship DB (Issue #84) ... New packages: - saltybot_social_msgs: PersonalityState.msg + QueryMood.srv custom interfaces - saltybot_social_personality: full personality node Features: - SOUL.md YAML/Markdown persona file: name, humor_level (0-10), sass_level (0-10), base_mood, per-tier greeting templates, mood prefix strings - Hot-reload: SoulWatcher polls SOUL.md every reload_interval seconds, applies changes live without restarting the node - Per-person relationship memory in SQLite: score, interaction_count, first/last_seen, learned preferences (JSON), full interaction log - Mood engine (pure functions): happy | curious | annoyed | playful driven by relationship score, interaction count, recent event window (120s) - Greeting personalisation: stranger | regular | favorite tiers keyed on interaction count thresholds from SOUL.md - Publishes /social/personality/state (PersonalityState) at publish_rate Hz - /social/personality/query_mood (QueryMood) service for on-demand mood query - Full ROS2 dynamic reconfigure: soul_file, db_path, reload_interval, publish_rate - 52 unit tests, no ROS2 runtime required ROS2 interfaces: Sub: /social/person_detected (std_msgs/String JSON) Pub: /social/personality/state (saltybot_social_msgs/PersonalityState) Srv: /social/personality/query_mood (saltybot_social_msgs/QueryMood)	2026-03-01 23:56:05 -05:00
sl-jetson	d6a6965af6	Merge pull request 'feat(social): person enrollment system #87' (#95) from sl-perception/social-enrollment into main	2026-03-01 23:55:16 -05:00
sl-firmware	5143e5bfac	feat(social): Issue #86 — physical expression + motor attention ... ESP32-C3 NeoPixel sketch (esp32/social_expression/social_expression.ino): - Adafruit NeoPixel + ArduinoJson, serial JSON protocol 115200 8N1 - Mood→colour: happy=green, curious=blue, annoyed=red, playful=rainbow - Idle breathing animation (sine-modulated warm white) - Auto-falls to idle after IDLE_TIMEOUT_MS (3 s) with no command ROS2 saltybot_social_msgs (new package): - Mood.msg — {mood, intensity} - Person.msg — {track_id, bearing_rad, distance_m, confidence, is_speaking, source} - PersonArray.msg — {persons[], active_id} ROS2 saltybot_social (new package): - expression_node: subscribes /social/mood → JSON serial to ESP32-C3 reconnects on port error; sends idle frame after idle_timeout_s - attention_node: subscribes /social/persons → /cmd_vel rotation-only proportional control with dead zone; prefers active speaker, falls back to highest-confidence person; lost-target idle after 2 s - launch/social.launch.py — combined launch - config YAML for both nodes with documented parameters - test/test_attention.py — 15 pytest-only unit tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-01 23:35:59 -05:00
sl-perception	5c4f18e46c	feat(social): person enrollment system — SQLite gallery + voice trigger (Issue #87) ... - saltybot_social_msgs: 6 msg + 5 srv definitions for social interaction - saltybot_social_enrollment: enrollment_node + enrollment_cli - PersonDB: thread-safe SQLite-backed gallery (embeddings, voice samples) - Voice-triggered enrollment via "remember me my name is X" phrase - CLI: enroll/list/delete/rename via ros2 run - Services: /social/enroll, /social/persons/list|delete|update - Gallery sync from /social/faces/embeddings topic	2026-03-01 23:32:26 -05:00
sl-perception	84790412d6	feat(social): multi-modal person state tracker (Issue #82)	2026-03-01 23:08:22 -05:00