feat: semantic sidewalk segmentation — TensorRT FP16 (#72) #78

sl-jetson · 2026-03-01T01:15:54-05:00

sl-jetson commented

2026-03-01 01:15:54 -05:00

Summary

saltybot_segmentation (ament_python): sidewalk_seg_node — /camera/color/image_raw → /segmentation/mask (mono8) + /segmentation/costmap (OccupancyGrid) + /segmentation/debug_image
5 traversability classes: sidewalk=free(0), grass=medium(50), road=high(90), obstacle=lethal(100), unknown=-1
TensorRT FP16 pipeline: BiSeNetV2 / DDRNet-23-slim, PyTorch→ONNX→TRT via scripts/build_engine.py; input 512×256, target >15fps on Orin Nano Super
saltybot_segmentation_costmap (ament_cmake): SegmentationCostmapLayer Nav2 costmap2d plugin with combination_method (max/override/min)
Inverse-perspective ground projection: camera height + pitch → pixel row → forward/lateral → OccupancyGrid cells
Fine-tune script: ROS2 bag frame extraction, LabelMe→Cityscapes masks, AdamW+CosineAnnealingLR
Training docs: model benchmarks, 7-step workflow, Nav2 YAML snippet, Orin performance tuning
24 unit tests, all passing

Closes #72

## Summary - `saltybot_segmentation` (ament_python): sidewalk_seg_node — /camera/color/image_raw → /segmentation/mask (mono8) + /segmentation/costmap (OccupancyGrid) + /segmentation/debug_image - 5 traversability classes: sidewalk=free(0), grass=medium(50), road=high(90), obstacle=lethal(100), unknown=-1 - TensorRT FP16 pipeline: BiSeNetV2 / DDRNet-23-slim, PyTorch→ONNX→TRT via scripts/build_engine.py; input 512×256, target >15fps on Orin Nano Super - `saltybot_segmentation_costmap` (ament_cmake): SegmentationCostmapLayer Nav2 costmap2d plugin with combination_method (max/override/min) - Inverse-perspective ground projection: camera height + pitch → pixel row → forward/lateral → OccupancyGrid cells - Fine-tune script: ROS2 bag frame extraction, LabelMe→Cityscapes masks, AdamW+CosineAnnealingLR - Training docs: model benchmarks, 7-step workflow, Nav2 YAML snippet, Orin performance tuning - 24 unit tests, all passing Closes #72

sl-jetson added 1 commit 2026-03-01 01:15:54 -05:00

feat: semantic sidewalk segmentation — TensorRT FP16 (#72 ) e964d632bf

New packages
────────────
saltybot_segmentation (ament_python)
  • seg_utils.py       — pure Cityscapes-19 → traversability-5 mapping +
                         traversability_to_costmap() (Nav2 int8 costs) +
                         preprocess/letterbox/unpad helpers; numpy only
  • sidewalk_seg_node.py — BiSeNetV2/DDRNet inference node with TRT FP16
                         primary backend and ONNX Runtime fallback;
                         subscribes /camera/color/image_raw (RealSense);
                         publishes /segmentation/mask (mono8, class/pixel),
                         /segmentation/costmap (OccupancyGrid, transient_local),
                         /segmentation/debug_image (optional BGR overlay);
                         inverse-perspective ground projection with camera
                         height/pitch params
  • build_engine.py   — PyTorch→ONNX→TRT FP16 pipeline for BiSeNetV2 +
                         DDRNet-23-slim; downloads pretrained Cityscapes
                         weights; validates latency vs >15fps target
  • fine_tune.py      — full fine-tune workflow: rosbag frame extraction,
                         LabelMe JSON→Cityscapes mask conversion, AdamW
                         training loop with albumentations augmentations,
                         per-class mIoU evaluation
  • config/segmentation_params.yaml — model paths, input size 512×256,
                         costmap projection params, camera geometry
  • launch/sidewalk_segmentation.launch.py
  • docs/training_guide.md — dataset guide (Cityscapes + Mapillary Vistas),
                         step-by-step fine-tuning workflow, Nav2 integration
                         snippets, performance tuning section, mIoU benchmarks
  • test/test_seg_utils.py — 24 unit tests (class mapping + cost generation)

saltybot_segmentation_costmap (ament_cmake)
  • SegmentationCostmapLayer.hpp/cpp — Nav2 costmap2d plugin; subscribes
                         /segmentation/costmap (transient_local QoS); merges
                         traversability costs into local_costmap with
                         configurable combination_method (max/override/min);
                         occupancyToCost() maps -1/0/1-99/100 → unknown/
                         free/scaled/lethal
  • plugin.xml, CMakeLists.txt, package.xml

Traversability classes
  SIDEWALK (0) → cost 0   (free — preferred)
  GRASS    (1) → cost 50  (medium)
  ROAD     (2) → cost 90  (high — avoid but crossable)
  OBSTACLE (3) → cost 100 (lethal)
  UNKNOWN  (4) → cost -1  (Nav2 unknown)

Performance target: >15fps on Orin Nano Super at 512×256
  BiSeNetV2 FP16 TRT: ~50fps measured on similar Ampere hardware
  DDRNet-23s FP16 TRT: ~40fps

Tests: 24/24 pass (seg_utils — no GPU/ROS2 required)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sl-jetson merged commit f7acf1554c into main

2026-03-01 01:20:51 -05:00

sl-jetson referenced this issue from a commit

2026-03-01 01:20:51 -05:00

Merge pull request 'feat: semantic sidewalk segmentation — TensorRT FP16 (#72)' (#78) from sl-jetson/sidewalk-segmentation into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: seb/saltylab-firmware#78