16 Commits

Author	SHA1	Message	Date
sl-jetson	b8a14e2bfc	feat(social): personal space respector node (Issue #310) ... Subscribes /social/faces/detected + /social/person_states; backs robot away (cmd_vel linear.x = -backup_speed) when closest person <= 0.8 m, latches /saltybot/too_close Bool; hysteresis band prevents chatter. 92/92 tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 21:42:33 -05:00
sl-jetson	797ed711b9	feat(social): conversation topic memory — Issue #299 ... Adds topic_memory_node to saltybot_social: - Subscribes /social/conversation_text (String, JSON {person_id, text}) - Extracts keywords via stop-word filtered pipeline: lowercase -> strip punctuation -> filter length/alpha -> remove stop words -> deduplicate -> cap at max_keywords_per_msg - Maintains per-person rolling topic history (PersonTopicMemory): ordered list, dedup with recency promotion, capped at max_topics_per_person, evicts oldest when full - Prunes persons idle for > prune_after_s (default 30 min; 0=disabled) - Publishes JSON on /saltybot/conversation_topics on each utterance: {person_id, recent_topics (most-recent first), new_topics, ts} - Enables recall like 'last time you mentioned X' - 71/71 tests passing Closes #299	2026-03-02 21:11:14 -05:00
sl-jetson	e8b787a987	feat(social): speech volume auto-adjuster — Issue #289 ... Adds volume_adjust_node to saltybot_social: - Subscribes /social/speech/energy (Float32 linear RMS [0..1] from vad_node) - Maps energy to TTS volume via: 1. Normalise into [noise_floor, noise_ceil] 2. Power-curve shaping (curve_gamma; <1 concave, 1 linear, >1 convex) 3. Lerp to [min_volume, max_volume] 4. Exponential moving-average smoothing (smoothing_alpha) - Publishes Float32 on /saltybot/tts_volume at publish_rate Hz (default 5 Hz) - Holds last value when energy is stale (> stale_timeout_s) - All params exposed: min/max_volume, noise_floor/ceil, curve_gamma, smoothing_alpha, publish_rate, stale_timeout_s - 74/74 tests passing Closes #289	2026-03-02 20:47:53 -05:00
sl-jetson	c3d36e9943	feat(social): face-tracking head servo controller — Issue #279 ... Adds face_track_servo_node to saltybot_social: - Subscribes /social/faces/detected (FaceDetectionArray) - Picks closest face by largest bbox area (proximity proxy) - Computes pan/tilt error from bbox centre vs image centre using configurable FOV (fov_h_deg=60°, fov_v_deg=45°) - Independent PID controllers for pan and tilt (velocity/incremental output with anti-windup); servo position integrates velocity*dt - Clamps commands to ±pan_limit_deg / ±tilt_limit_deg - Returns to centre at return_rate_deg_s when face lost >lost_timeout_s - Dead zone suppresses jitter for small errors - Publishes Float32 on /saltybot/head_pan and /saltybot/head_tilt - 81/81 tests passing Closes #279	2026-03-02 17:38:02 -05:00
sl-jetson	c7dd07f9ed	feat(social): proximity-based greeting trigger — Issue #270 ... Adds greeting_trigger_node to saltybot_social: - Subscribes to /social/faces/detected (FaceDetectionArray) for face arrivals - Subscribes to /social/person_states (PersonStateArray) to cache face_id→distance - Fires greeting when face_id is within proximity_m (default 2m) and not in per-face_id cooldown window (default 300s) - Publishes JSON on /saltybot/greeting_trigger: {face_id, person_name, distance_m, ts} - unknown_distance param controls assumed distance for faces with no PersonState yet - Thread-safe distance cache and greeted map - 50/50 tests passing Closes #270	2026-03-02 17:26:40 -05:00
sl-jetson	3cd9faeed9	feat(social): ambient sound classifier via mel-spectrogram — Issue #252 ... Adds ambient_sound_node to saltybot_social: - Accumulates 1 s of PCM-16 audio from /social/speech/audio_raw - Extracts mel-spectrogram feature vector (energy_db, zcr, mel_centroid, mel_flatness, low_ratio, high_ratio) using pure numpy (no torch/onnx) - Priority-cascade classifier: silence → music → speech → crowd → outdoor → alarm - Publishes label as std_msgs/String on /saltybot/ambient_sound on each buffer fill - All 11 thresholds exposed as ROS parameters (yaml + launch file) - numpy-free energy-only fallback for edge environments - 77/77 tests passing Closes #252	2026-03-02 13:22:38 -05:00
sl-jetson	4919dc0bc6	feat(social): energy+ZCR voice activity detection node (Issue #242) ... Add vad_node to saltybot_social: subscribes to /social/speech/audio_raw (UInt8MultiArray PCM-16), computes RMS energy (dBFS) and zero-crossing rate per chunk, applies onset/offset hysteresis (VadStateMachine), and publishes /social/speech/is_speaking (Bool) and /social/speech/energy (Float32 linear RMS). All thresholds configurable via ROS params: rms_threshold_db=-35.0, zcr_min=0.01, zcr_max=0.40, onset_frames=2, offset_frames=8, audio_topic. 69/69 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 12:26:01 -05:00
sl-perception	bd4a9f094d	feat(social): hand gesture pointing direction node (Issue #221) ... saltybot_social_msgs: - Add PointingTarget.msg: origin (INDEX_MCP), direction (unit vec), target, range_m, person_id, confidence, coarse_direction, is_active - Register in CMakeLists.txt saltybot_social: - _pointing_ray.py (pure Python, no rclpy): unproject(), sample_depth() (median with outlier rejection), compute_pointing_ray() — reprojects INDEX_MCP and INDEX_TIP into 3-D using D435i depth; falls back to image- plane direction when both depths are equal; gracefully handles one-sided missing depth - pointing_node.py: subscribes /social/gestures + synced D435i colour+depth; re-runs MediaPipe Hands when a 'point' gesture is cached (within gesture_timeout_s); picks closest hand to gesture anchor; publishes PointingTarget on /saltybot/pointing_target at 5 Hz - setup.py: adds pointing_node entry point - 18/18 unit tests pass Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 11:56:54 -05:00
sl-jetson	e5236f781b	feat(social): robot mesh comms — peer announce + person handoff (Issue #171) ... New MeshPeer.msg (1 Hz DDS heartbeat: robot_id, social_state, active persons, greeted names) and MeshHandoff.msg (person context transfer on STATE_LEAVING). mesh_comms_node subscribes to person_states and orchestrator/state, publishes announce heartbeat, triggers handoff on LEAVING, tracks peers with timeout cleanup, and propagates mesh-wide greeting deduplication via /social/mesh/greeted. 73/73 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 11:14:00 -05:00
sl-jetson	90c8b427fc	feat(social): multi-language support — Whisper LID + per-lang Piper TTS (Issue #167) ... - Add SpeechTranscript.language (BCP-47), ConversationResponse.language fields - speech_pipeline_node: whisper_language param (""=auto-detect via Whisper LID); detected language published in every transcript - conversation_node: track per-speaker language; inject "[Please respond in X.]" hint for non-English speakers; propagate language to ConversationResponse. _LANG_NAMES: 24 BCP-47 codes -> English names. Also adds Issue #161 emotion context plumbing (co-located in same branch for clean merge) - tts_node: voice_map_json param (JSON BCP-47->ONNX path); lazy voice loading per language; playback queue now carries (text, lang) tuples for voice routing - speech_params.yaml, tts_params.yaml: new language params with docs - 47/47 tests pass (test_multilang.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:57:34 -05:00
sl-jetson	50971c0946	feat(social): facial expression recognition — TRT FP16 emotion CNN (Issue #161) ... - Add Expression.msg / ExpressionArray.msg ROS2 message definitions - Add emotion_classifier.py: 7-class CNN (happy/sad/angry/surprised/fearful/disgusted/neutral) via TensorRT FP16 with landmark-geometry fallback; EMA per-person smoothing; opt-out registry - Add emotion_node.py: subscribes /social/faces/detections, runs TRT crop inference (<5ms), publishes /social/faces/expressions and /social/emotion/context JSON for LLM - Wire emotion context into conversation_node.py: emotion hint injected into LLM prompt when speaker shows non-neutral affect; subscribes /social/emotion/context - Add emotion_params.yaml config and emotion.launch.py launch file - Add 67-test suite (test_emotion_classifier.py): classifier, tracker, opt-out, heuristic Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:40:54 -05:00
sl-jetson	ce6d5ee249	feat(social): multi-camera gesture recognition — MediaPipe Hands + Pose (Issue #140) ... Delivers Issue #140 (P2): real-time gesture detection from 4 CSI cameras via MediaPipe Hands and Pose, publishing classified gestures on /social/gestures. New files: - saltybot_social_msgs/msg/Gesture.msg + GestureArray.msg — ROS2 message types - saltybot_social/gesture_classifier.py — pure-Python geometric classifier (stop_palm, thumbs_up/down, point, come_here, follow, wave, arms_up, arms_spread, crouch); WaveDetector temporal sliding-window oscillation tracker - saltybot_social/gesture_node.py — ROS2 node; round-robin multi-camera _FrameBuffer, lazy MediaPipe init, person-ID correlation via PersonState - saltybot_social/test/test_gesture_classifier.py — 70 unit tests, all passing - saltybot_social/config/gesture_params.yaml — tuned defaults for Orin Nano - saltybot_social/launch/gesture.launch.py — all params overridable at launch Modified: - saltybot_social_msgs/CMakeLists.txt — register Gesture + GestureArray msgs - saltybot_social/setup.py — add gesture_node console_scripts entry point Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 10:10:54 -05:00
sl-jetson	fabb988e72	feat(social): voice command NLU — 30+ intents, confirmation flow (Issue #137) ... ## New files - saltybot_social_msgs/msg/VoiceCommand.msg intent + entities[] + confidence + confirmation_token + requires_confirmation - saltybot_social/voice_command_parser.py Pure-Python regex NLU, zero ROS2/ML deps, < 1 ms/call 30+ named intents across nav.*, social.*, system.*, config.*, confirm.* Entity extraction: location, name, mode, level, route name Dangerous-command flag: system.shutdown, system.restart, social.forget_me - saltybot_social/voice_command_node.py Subscribes /social/speech/transcript, publishes /social/voice_command Confirmation flow with UUID token + 10 s timeout Below-threshold → intent=fallback → LLM conversation engine - saltybot_social/test/test_voice_command_parser.py 191 unit tests (all pass), no ROS2 runtime required - saltybot_social/config/voice_command_params.yaml - saltybot_social/launch/voice_command.launch.py ## Intent taxonomy nav: go_to, go_home, follow_me, stop, wait, come_here, patrol, set_mode (shadow/lead/side/orbit/loose/tight), teach_route, stop_teaching, replay_route social: remember_me, forget_me [CONFIRM], whats_my_name, tell_joke system: battery_status, map_status, shutdown [CONFIRM], restart [CONFIRM], volume_up, volume_down, volume_set config: personality, sass_level, follow_mode ## Updated - saltybot_social_msgs/CMakeLists.txt: register VoiceCommand.msg - saltybot_social/setup.py: add voice_command_node entry point Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 09:38:17 -05:00
sl-jetson	5043578934	feat(social): speech pipeline + LLM conversation + TTS + orchestrator (#81 #83 #85 #89) ... Issue #81 — Speech pipeline: - speech_pipeline_node.py: OpenWakeWord "hey_salty" → Silero VAD → faster-whisper STT (Orin GPU, <500ms wake-to-transcript) → ECAPA-TDNN speaker diarization - speech_utils.py: pcm16↔float32, EnergyVad, UtteranceSegmenter (pre-roll, max- duration), cosine speaker identification — all pure Python, no ROS2/GPU needed - Publishes /social/speech/transcript (SpeechTranscript) + /social/speech/vad_state Issue #83 — Conversation engine: - conversation_node.py: llama-cpp-python GGUF (Phi-3-mini Q4_K_M, 20 GPU layers), streaming token output, per-person sliding-window context (4K tokens), summary compression, SOUL.md system prompt, group mode - llm_context.py: PersonContext, ContextStore (JSON persistence), build_llama_prompt (ChatML format), context compression via LLM summarization - Publishes /social/conversation/response (ConversationResponse, partial + final) Issue #85 — Streaming TTS: - tts_node.py: Piper ONNX streaming synthesis, sentence-by-sentence first-chunk streaming (<200ms to first audio), sounddevice USB speaker playback, volume control - tts_utils.py: split_sentences, pcm16_to_wav_bytes, chunk_pcm, apply_volume, strip_ssml Issue #89 — Pipeline orchestrator: - orchestrator_node.py: IDLE→LISTENING→THINKING→SPEAKING state machine, GPU memory watchdog (throttle at <2GB free), rolling latency stats (p50/p95 per stage), VAD watchdog (alert if speech pipeline hangs), /social/orchestrator/state JSON pub - social_bot.launch.py: brings up all 4 nodes with TimerAction delays New messages: SpeechTranscript.msg, VadState.msg, ConversationResponse.msg Config YAMLs: speech_params, conversation_params, tts_params, orchestrator_params Tests: 58 tests (28 speech_utils + 30 llm_context/tts_utils), all passing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-02 08:23:19 -05:00
sl-firmware	5143e5bfac	feat(social): Issue #86 — physical expression + motor attention ... ESP32-C3 NeoPixel sketch (esp32/social_expression/social_expression.ino): - Adafruit NeoPixel + ArduinoJson, serial JSON protocol 115200 8N1 - Mood→colour: happy=green, curious=blue, annoyed=red, playful=rainbow - Idle breathing animation (sine-modulated warm white) - Auto-falls to idle after IDLE_TIMEOUT_MS (3 s) with no command ROS2 saltybot_social_msgs (new package): - Mood.msg — {mood, intensity} - Person.msg — {track_id, bearing_rad, distance_m, confidence, is_speaking, source} - PersonArray.msg — {persons[], active_id} ROS2 saltybot_social (new package): - expression_node: subscribes /social/mood → JSON serial to ESP32-C3 reconnects on port error; sends idle frame after idle_timeout_s - attention_node: subscribes /social/persons → /cmd_vel rotation-only proportional control with dead zone; prefers active speaker, falls back to highest-confidence person; lost-target idle after 2 s - launch/social.launch.py — combined launch - config YAML for both nodes with documented parameters - test/test_attention.py — 15 pytest-only unit tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-01 23:35:59 -05:00
sl-perception	84790412d6	feat(social): multi-modal person state tracker (Issue #82)	2026-03-01 23:08:22 -05:00