saltylab-firmware/jetson/ros2_ws/src/saltybot_diagnostics/README.md

SaltyBot Diagnostic Self-Test System

Comprehensive hardware diagnostics and health monitoring for SaltyBot. Performs startup checks on critical hardware, continuous runtime monitoring, and publishes detailed diagnostic data.

Features

Startup Hardware Checks

Validates hardware availability and connectivity at boot:

• RPLIDAR: Serial port detection, rotation verification
• RealSense D435i: USB enumeration, stream availability
• VESC Motor Controller: UART connection, firmware status
• Jabra Microphone: USB audio device detection
• STM32 Bridge: Serial port verification, watchdog status
• Servo Controller: I2C bus communication
• WiFi: Network interface status
• GPS Module: Serial port and fix detection
• Disk Space: Storage availability checking
• System RAM: Memory availability

Runtime Monitoring

Continuous health checks during operation:

• Sensor FPS: RealSense and RPLIDAR frame rates
• Motor Stall: Encoder latency detection
• Temperature: Orin GPU (>80°C warn, >85°C error), VESC (>60°C warn, >70°C error)
• Network Latency: Ping time monitoring
• System Resources: CPU, RAM usage trends

Notifications & Feedback

• TTS Announcements: Boot result via text-to-speech
• Face Animations: Boot success/error display
• Diagnostic Publishing: /saltybot/diagnostics (DiagnosticArray)
• JSON Logging: Detailed logs to /home/seb/saltybot-data/diagnostics/

Topics

Published

• /saltybot/diagnostics (diagnostic_msgs/DiagnosticArray): System health status
• /saltybot/tts_say (std_msgs/String): Boot result announcement
• /saltybot/face/boot_animation (std_msgs/String): Boot animation trigger

Subscribed

• Implicitly monitors system topics for FPS/latency data

Configuration

Edit config/diagnostic_checks.yaml:

startup_checks:
  enabled: true
  checks: [rplidar, realsense, vesc, jabra_microphone, ...]

runtime_monitoring:
  enabled: true
  frequency_hz: 1
  temperatures:
    jetson_gpu: {warn_c: 80, error_c: 85, critical_c: 90}
    vesc_motor: {warn_c: 60, error_c: 70, critical_c: 80}

logging:
  directory: /home/seb/saltybot-data/diagnostics
  retention_days: 30

Launch

# Default launch with startup checks + runtime monitoring
ros2 launch saltybot_diagnostics diagnostics.launch.py

# Startup checks only
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  enable_runtime_monitoring:=false

# Custom config
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  config_file:=/path/to/custom_checks.yaml

Diagnostic Array Format

Published to /saltybot/diagnostics:

diagnostic_msgs/DiagnosticArray:
  header:
    stamp: <timestamp>
  status:
    - name: "saltybot/rplidar"
      level: 0  # OK=0, WARN=1, ERROR=2, STALE=3
      message: "RPLIDAR detected on /dev/ttyUSB0"
      values:
        - key: "port"
          value: "/dev/ttyUSB0"

    - name: "saltybot/realsense"
      level: 2  # ERROR
      message: "RealSense not found on expected USB bus"
      values: []

    - name: "saltybot/gpu_temp"
      level: 1  # WARN
      message: "Runtime check"
      values:
        - key: "temperature_c"
          value: "82.5"
        - key: "threshold_warn"
          value: "80"

JSON Diagnostics Log Format

Files saved to /home/seb/saltybot-data/diagnostics/diagnostics_YYYYMMDD_HHMMSS.json:

{
  "timestamp": "2025-03-05T10:00:00.123456",
  "check_type": "startup_checks",
  "hardware_checks": {
    "rplidar": {
      "status": "OK",
      "message": "RPLIDAR detected on /dev/ttyUSB0",
      "details": {"port": "/dev/ttyUSB0"}
    },
    "realsense": {
      "status": "ERROR",
      "message": "RealSense not found",
      "details": {}
    }
  },
  "runtime_metrics": {
    "gpu_temp": {
      "status": "OK",
      "temperature_c": 65.0,
      "threshold_warn": 80
    },
    "network_latency": {
      "status": "WARN",
      "latency_ms": 150
    }
  }
}

TTS Announcements

Boot result messages published to /saltybot/tts_say:

• Success: "Boot complete. All systems online."
• With errors: "Boot complete with errors. RPLIDAR, RealSense offline."

Status Levels

• OK (0): System healthy, no action needed
• WARN (1): Minor issues, monitor closely
• ERROR (2): Critical failure, may affect operation
• STALE (3): No data, check unavailable

Logs and Data

Diagnostic logs stored in /home/seb/saltybot-data/diagnostics/:

• Auto-rotated every 100MB or 30 days
• JSON format for easy parsing
• Full boot record + runtime metrics

Integration with Full Stack

Add to full_stack.launch.py at t=0s:

IncludeLaunchDescription(
    PythonLaunchDescriptionSource([...diagnostics.launch.py]),
    launch_arguments={'enable_startup_check': 'true'}.items(),
)

Debugging

Check current diagnostics:

ros2 topic echo /saltybot/diagnostics

View latest diagnostic log:

tail -f /home/seb/saltybot-data/diagnostics/diagnostics_*.json | jq

Simulate diagnostic errors (for testing):

# Monitor what would be logged
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  enable_startup_check:=true enable_runtime_monitoring:=true

Hardware Requirements

• Jetson Orin (for temperature monitoring)
• Linux with psutil (for system resources)
• Standard ROS2 diagnostic_msgs package