seb/saltylab-firmware
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

feat: person detection + tracking (YOLOv8n TensorRT) #54

Merged
seb merged 1 commits from sl-jetson/person-detection into main 2026-02-28 23:25:22 -05:00
Conversation 0 Commits 1 Files Changed 15 +1383
sl-jetson commented 2026-02-28 23:21:49 -05:00
Collaborator
Copy Link

Summary

New package saltybot_perception — person detection and tracking for person-following mode.

  • YOLOv8n inference via TensorRT FP16 engine on Orin Nano Super (67 TOPS); ONNX Runtime fallback for dev/CI
  • Synchronized color + depth frames via ApproximateTimeSynchronizer (50ms slop)
  • 3D position from RealSense aligned depth: median 7×7px window + back-projection using camera intrinsics
  • TF transform to base_link (fallback to camera_color_optical_frame)
  • SimplePersonTracker: IoU re-association, picks closest person (depth), holds 2s through occlusion

Published Topics

Topic Type Content
/person/detections vision_msgs/Detection2DArray All detected persons (bbox + confidence)
/person/target geometry_msgs/PoseStamped Tracked person 3D position in base_link
/person/debug_image sensor_msgs/Image Annotated RGB (optional, off by default)

Config defaults (person_detection_params.yaml)

  • confidence_threshold: 0.40
  • min_depth / max_depth: 0.5 m – 5.0 m
  • track_hold_duration: 2.0 s
  • target_frame: base_link

Files

  • person_detector_node.py — main node (TRT/ONNX backends, preprocessing, postprocessing, depth lookup, TF)
  • tracker.py — SimplePersonTracker (IoU association, hold, track ID)
  • detection_utils.py — pure helpers: nms(), letterbox(), get_depth_at(), pixel_to_3d()
  • scripts/build_trt_engine.py — ONNX → TensorRT FP16 engine builder
  • launch/person_detection.launch.py, config/person_detection_params.yaml

Test plan

  • 26 unit tests pass: pytest jetson/ros2_ws/src/saltybot_perception/test/ -v
  • ros2 launch saltybot_perception person_detection.launch.py starts without errors (model not found is OK — logs warning)
  • With RealSense running: /person/detections publishes when person in frame
  • /person/target PoseStamped frame = base_link
  • Tracker holds last position for 2s when person walks out of frame
  • build_trt_engine.py --onnx yolov8n.onnx --engine yolov8n.engine --verify succeeds on Orin

🤖 Generated with Claude Code

## Summary New package `saltybot_perception` — person detection and tracking for person-following mode. - **YOLOv8n inference** via TensorRT FP16 engine on Orin Nano Super (67 TOPS); ONNX Runtime fallback for dev/CI - **Synchronized** color + depth frames via `ApproximateTimeSynchronizer` (50ms slop) - **3D position** from RealSense aligned depth: median 7×7px window + back-projection using camera intrinsics - **TF transform** to `base_link` (fallback to `camera_color_optical_frame`) - **SimplePersonTracker**: IoU re-association, picks closest person (depth), holds 2s through occlusion ## Published Topics | Topic | Type | Content | |-------|------|---------| | `/person/detections` | `vision_msgs/Detection2DArray` | All detected persons (bbox + confidence) | | `/person/target` | `geometry_msgs/PoseStamped` | Tracked person 3D position in `base_link` | | `/person/debug_image` | `sensor_msgs/Image` | Annotated RGB (optional, off by default) | ## Config defaults (`person_detection_params.yaml`) - `confidence_threshold`: 0.40 - `min_depth` / `max_depth`: 0.5 m – 5.0 m - `track_hold_duration`: 2.0 s - `target_frame`: `base_link` ## Files - `person_detector_node.py` — main node (TRT/ONNX backends, preprocessing, postprocessing, depth lookup, TF) - `tracker.py` — `SimplePersonTracker` (IoU association, hold, track ID) - `detection_utils.py` — pure helpers: `nms()`, `letterbox()`, `get_depth_at()`, `pixel_to_3d()` - `scripts/build_trt_engine.py` — ONNX → TensorRT FP16 engine builder - `launch/person_detection.launch.py`, `config/person_detection_params.yaml` ## Test plan - [ ] 26 unit tests pass: `pytest jetson/ros2_ws/src/saltybot_perception/test/ -v` - [ ] `ros2 launch saltybot_perception person_detection.launch.py` starts without errors (model not found is OK — logs warning) - [ ] With RealSense running: `/person/detections` publishes when person in frame - [ ] `/person/target` PoseStamped frame = `base_link` - [ ] Tracker holds last position for 2s when person walks out of frame - [ ] `build_trt_engine.py --onnx yolov8n.onnx --engine yolov8n.engine --verify` succeeds on Orin 🤖 Generated with [Claude Code](https://claude.com/claude-code)
sl-jetson added 1 commit 2026-02-28 23:21:49 -05:00 
New package: saltybot_perception

person_detector_node.py:
- Subscribes /camera/color/image_raw + /camera/depth/image_rect_raw
  (ApproximateTimeSynchronizer, slop=50ms)
- Subscribes /camera/color/camera_info for intrinsics
- YOLOv8n inference via TensorRT FP16 engine (Orin Nano 67 TOPS)
  Falls back to ONNX Runtime when engine not found (dev/CI)
- Letterbox preprocessing (640x640), YOLOv8n post-process + NMS
- Median-window depth lookup at bbox centre (7x7 px)
- Back-projects 2D pixel + depth to 3D point in camera frame
- tf2 transform to base_link (fallback: camera_color_optical_frame)
- Publishes:
    /person/detections  vision_msgs/Detection2DArray  all persons
    /person/target      geometry_msgs/PoseStamped     tracked person 3D
    /person/debug_image sensor_msgs/Image              (optional)

tracker.py — SimplePersonTracker:
- Single-target IoU-based tracker
- Picks closest valid person (smallest depth) on first lock
- Re-associates across frames using IoU threshold
- Holds last known position for configurable duration (default 2s)
- Monotonically increasing track IDs

detection_utils.py — pure helpers (no ROS2 deps, testable standalone):
- nms(), letterbox(), remap_bbox(), get_depth_at(), pixel_to_3d()

scripts/build_trt_engine.py:
- Converts ONNX to TensorRT FP16 engine using TRT Python API
- Prints trtexec CLI alternative
- Includes YOLOv8n download instructions

config/person_detection_params.yaml:
- confidence_threshold: 0.40, min_depth: 0.5m, max_depth: 5.0m
- track_hold_duration: 2.0s, target_frame: base_link

launch/person_detection.launch.py:
- engine_path, onnx_path, publish_debug_image, target_frame overridable

Tests: 26/26 passing (test_tracker.py + test_postprocess.py)
- IoU computation, NMS suppression, tracker state machine,
  depth filtering, hold duration, re-association, track ID

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
seb merged commit 9014b6738d into main 2026-02-28 23:25:22 -05:00
seb deleted branch sl-jetson/person-detection 2026-02-28 23:25:22 -05:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
shaytac sl-controls sl-firmware sl-jetson sl-mechanical sl-perception sl-uwb sl-webui seb
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#54
No description provided.
Delete Branch "sl-jetson/person-detection"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?