saltylab-firmware/jetson/ros2_ws/src/saltybot_segmentation/docs/training_guide.md

Sidewalk Segmentation — Training & Deployment Guide

Overview

SaltyBot uses BiSeNetV2 (default) or DDRNet-23-slim for real-time semantic segmentation, pre-trained on Cityscapes and optionally fine-tuned on site-specific data.

The model runs at >15fps on Orin Nano Super via TensorRT FP16 at 512×256 input, publishing per-pixel traversability costs to Nav2.

---

Traversability Classes

Class	ID	OccupancyGrid	Description
Sidewalk	0	0 (free)	Preferred navigation surface
Grass/vegetation	1	50 (medium)	Traversable but non-preferred
Road	2	90 (high cost)	Avoid but can cross
Obstacle	3	100 (lethal)	Person, car, building, fence
Unknown	4	-1 (unknown)	Sky, unlabelled

---

Quick Start — Pretrained Cityscapes Model

No training required for standard sidewalks in Western cities.

# 1. Build the TRT engine (once, on Orin):
python3 /opt/ros/humble/share/saltybot_segmentation/scripts/build_engine.py

# 2. Launch:
ros2 launch saltybot_segmentation sidewalk_segmentation.launch.py

# 3. Verify:
ros2 topic hz /segmentation/mask        # expect ~15Hz
ros2 topic hz /segmentation/costmap     # expect ~15Hz
ros2 run rviz2 rviz2                    # add OccupancyGrid, topic=/segmentation/costmap

---

Model Benchmarks — Cityscapes Validation Set

Model	mIoU	Sidewalk IoU	Road IoU	FPS (Orin FP16)	Engine size
BiSeNetV2	72.6%	82.1%	97.8%	~50 fps	~11 MB
DDRNet-23-slim	79.5%	84.3%	98.1%	~40 fps	~18 MB

Both exceed the >15fps target with headroom for additional ROS2 overhead.

BiSeNetV2 is the default — faster, smaller, sufficient for traversability. Use DDRNet if mIoU matters more than latency (e.g., complex European city centres).

---

Fine-Tuning on Custom Site Data

When is fine-tuning needed?

The Cityscapes-trained model works well for:

• European-style city sidewalks, curbs, roads
• Standard pedestrian infrastructure

Fine-tuning improves performance for:

• Gravel/dirt paths (not in Cityscapes)
• Unusual kerb styles or non-standard pavement markings
• Indoor+outdoor transitions (garage exits, building entrances)
• Non-Western road infrastructure

Step 1 — Collect data (walk the route)

Record a ROS2 bag while walking the intended robot route:

# On Orin, record the front camera for 5–10 minutes of the target environment:
ros2 bag record /camera/color/image_raw -o sidewalk_route_2024-01

# Transfer to workstation for labelling:
scp -r jetson:/home/ubuntu/sidewalk_route_2024-01 ~/datasets/

Step 2 — Extract frames

python3 fine_tune.py \
    --extract-frames ~/datasets/sidewalk_route_2024-01/ \
    --output-dir ~/datasets/sidewalk_raw/ \
    --every-n 5       # 1 frame per 5 = ~6fps from 30fps bag

This extracts ~200–400 frames from a 5-minute bag. You need to label 50–100 frames minimum.

Step 3 — Label with LabelMe

pip install labelme
labelme ~/datasets/sidewalk_raw/

Class names to use (must be exact):

What you see	LabelMe label
Footpath/pavement	sidewalk
Road/tarmac	road
Grass/lawn/verge	vegetation
Gravel path	terrain
Person	person
Car/vehicle	car
Building/wall	building
Fence/gate	fence

Labelling tips:

• Use polygon tool for precise boundaries
• Focus on the ground plane (lower 60% of image) — that's what the costmap uses
• 50 well-labelled frames beats 200 rushed ones
• Vary conditions: sunny, overcast, morning, evening

Step 4 — Convert labels to masks

python3 fine_tune.py \
    --convert-labels ~/datasets/sidewalk_raw/ \
    --output-dir ~/datasets/sidewalk_masks/

Output: <frame>_mask.png per frame — 8-bit PNG with Cityscapes class IDs. Unlabelled pixels are set to 255 (ignored during training).

Step 5 — Fine-tune

# On Orin (or workstation with CUDA):
python3 fine_tune.py --train \
    --images  ~/datasets/sidewalk_raw/ \
    --labels  ~/datasets/sidewalk_masks/ \
    --weights /mnt/nvme/saltybot/models/bisenetv2_cityscapes.pth \
    --output  /mnt/nvme/saltybot/models/bisenetv2_custom.pth \
    --epochs  20 \
    --lr      1e-4

Expected training time:

• 50 images, 20 epochs, Orin: ~15 minutes
• 100 images, 20 epochs, Orin: ~25 minutes

Step 6 — Evaluate mIoU

python3 fine_tune.py --eval \
    --images ~/datasets/sidewalk_raw/ \
    --labels ~/datasets/sidewalk_masks/ \
    --weights /mnt/nvme/saltybot/models/bisenetv2_custom.pth

Target mIoU on custom classes: >70% on sidewalk/road/obstacle.

Step 7 — Build new TRT engine

python3 build_engine.py \
    --weights /mnt/nvme/saltybot/models/bisenetv2_custom.pth \
    --engine  /mnt/nvme/saltybot/models/bisenetv2_custom_512x256.engine

Update segmentation_params.yaml:

engine_path: /mnt/nvme/saltybot/models/bisenetv2_custom_512x256.engine

---

Nav2 Integration — SegmentationCostmapLayer

Add to nav2_params.yaml:

local_costmap:
  local_costmap:
    ros__parameters:
      plugins:
        - "voxel_layer"
        - "inflation_layer"
        - "segmentation_layer"   # ← add this

      segmentation_layer:
        plugin: "saltybot_segmentation_costmap::SegmentationCostmapLayer"
        enabled: true
        topic: /segmentation/costmap
        combination_method: max     # max | override | min

combination_method:

• max (default) — keeps the worst-case cost between existing costmap and segmentation. Most conservative; prevents Nav2 from overriding obstacle detections.
• override — segmentation completely replaces existing cell costs. Use when you trust the camera more than other sensors.
• min — keeps the most permissive cost. Use for exploratory navigation in open environments.

---

Performance Tuning

Too slow (<15fps)

1. Reduce process_every_n (e.g., process_every_n: 2 → effective 15fps from 30fps camera)
2. Reduce input resolution — edit build_engine.py to use 384×192 (2.2× faster)
3. Ensure nvpmodel is in MAXN mode: sudo nvpmodel -m 0 && sudo jetson_clocks
4. Check GPU is not throttled: jtop → GPU frequency should be 1024MHz

False road detections (sidewalk near road)

• Lower camera_pitch_deg to look further ahead
• Enable unknown_as_obstacle: true to be more cautious
• Fine-tune with site-specific data (Step 5)

Costmap looks noisy

• Increase costmap_resolution (0.1m cells reduce noise)
• Reduce costmap_range_m to 3.0m (projection less accurate at far range)
• Apply temporal smoothing in Nav2 inflation layer

---

Datasets

Cityscapes (primary pre-training)

• 2975 training / 500 validation finely annotated images
• 19 semantic classes (roads, sidewalks, people, vehicles, etc.)
• License: Cityscapes Terms of Use (non-commercial research)
• Download: https://www.cityscapes-dataset.com/
• Required for training from scratch; BiSeNetV2 pretrained checkpoint (~25MB) available at: https://github.com/CoinCheung/BiSeNet/releases

Mapillary Vistas (supplementary)

• 18,000 training images from diverse global street scenes
• 124 semantic classes (broader coverage than Cityscapes)
• Includes dirt paths, gravel, unusual sidewalk types
• License: CC BY-SA 4.0
• Download: https://www.mapillary.com/dataset/vistas
• Useful for mapping to our traversability classes in non-European environments

Custom saltybot data

• Collected per-deployment via fine_tune.py --extract-frames
• 50–100 labelled frames typical for 80%+ mIoU on specific routes
• Store in /mnt/nvme/saltybot/training_data/

---

File Locations on Orin

/mnt/nvme/saltybot/
├── models/
│   ├── bisenetv2_cityscapes.pth              ← downloaded pretrained weights
│   ├── bisenetv2_cityscapes_512x256.onnx     ← exported ONNX (FP32)
│   ├── bisenetv2_cityscapes_512x256.engine   ← TRT FP16 engine (built on Orin)
│   ├── bisenetv2_custom.pth                  ← fine-tuned weights (after step 5)
│   └── bisenetv2_custom_512x256.engine       ← TRT FP16 engine (after step 7)
├── training_data/
│   ├── raw/                                  ← extracted JPEG frames
│   └── labels/                              ← LabelMe JSON + converted PNG masks
└── rosbags/
    └── sidewalk_route_*/                     ← recorded ROS2 bags for labelling