# SaltyBot Diagnostic Self-Test System

Comprehensive hardware diagnostics and health monitoring for SaltyBot. Performs startup checks on critical hardware, continuous runtime monitoring, and publishes detailed diagnostic data.

## Features

### Startup Hardware Checks
Validates hardware availability and connectivity at boot:
- **RPLIDAR**: Serial port detection, rotation verification
- **RealSense D435i**: USB enumeration, stream availability
- **VESC Motor Controller**: UART connection, firmware status
- **Jabra Microphone**: USB audio device detection
- **STM32 Bridge**: Serial port verification, watchdog status
- **Servo Controller**: I2C bus communication
- **WiFi**: Network interface status
- **GPS Module**: Serial port and fix detection
- **Disk Space**: Storage availability checking
- **System RAM**: Memory availability

### Runtime Monitoring
Continuous health checks during operation:
- **Sensor FPS**: RealSense and RPLIDAR frame rates
- **Motor Stall**: Encoder latency detection
- **Temperature**: Orin GPU (>80°C warn, >85°C error), VESC (>60°C warn, >70°C error)
- **Network Latency**: Ping time monitoring
- **System Resources**: CPU, RAM usage trends

### Notifications & Feedback
- **TTS Announcements**: Boot result via text-to-speech
- **Face Animations**: Boot success/error display
- **Diagnostic Publishing**: `/saltybot/diagnostics` (DiagnosticArray)
- **JSON Logging**: Detailed logs to `/home/seb/saltybot-data/diagnostics/`

## Topics

### Published
- `/saltybot/diagnostics` (diagnostic_msgs/DiagnosticArray): System health status
- `/saltybot/tts_say` (std_msgs/String): Boot result announcement
- `/saltybot/face/boot_animation` (std_msgs/String): Boot animation trigger

### Subscribed
- Implicitly monitors system topics for FPS/latency data

## Configuration

Edit `config/diagnostic_checks.yaml`:

```yaml
startup_checks:
  enabled: true
  checks: [rplidar, realsense, vesc, jabra_microphone, ...]

runtime_monitoring:
  enabled: true
  frequency_hz: 1
  temperatures:
    jetson_gpu: {warn_c: 80, error_c: 85, critical_c: 90}
    vesc_motor: {warn_c: 60, error_c: 70, critical_c: 80}

logging:
  directory: /home/seb/saltybot-data/diagnostics
  retention_days: 30
```

## Launch

```bash
# Default launch with startup checks + runtime monitoring
ros2 launch saltybot_diagnostics diagnostics.launch.py

# Startup checks only
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  enable_runtime_monitoring:=false

# Custom config
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  config_file:=/path/to/custom_checks.yaml
```

## Diagnostic Array Format

Published to `/saltybot/diagnostics`:

```python
diagnostic_msgs/DiagnosticArray:
  header:
    stamp: <timestamp>
  status:
    - name: "saltybot/rplidar"
      level: 0  # OK=0, WARN=1, ERROR=2, STALE=3
      message: "RPLIDAR detected on /dev/ttyUSB0"
      values:
        - key: "port"
          value: "/dev/ttyUSB0"

    - name: "saltybot/realsense"
      level: 2  # ERROR
      message: "RealSense not found on expected USB bus"
      values: []

    - name: "saltybot/gpu_temp"
      level: 1  # WARN
      message: "Runtime check"
      values:
        - key: "temperature_c"
          value: "82.5"
        - key: "threshold_warn"
          value: "80"
```

## JSON Diagnostics Log Format

Files saved to `/home/seb/saltybot-data/diagnostics/diagnostics_YYYYMMDD_HHMMSS.json`:

```json
{
  "timestamp": "2025-03-05T10:00:00.123456",
  "check_type": "startup_checks",
  "hardware_checks": {
    "rplidar": {
      "status": "OK",
      "message": "RPLIDAR detected on /dev/ttyUSB0",
      "details": {"port": "/dev/ttyUSB0"}
    },
    "realsense": {
      "status": "ERROR",
      "message": "RealSense not found",
      "details": {}
    }
  },
  "runtime_metrics": {
    "gpu_temp": {
      "status": "OK",
      "temperature_c": 65.0,
      "threshold_warn": 80
    },
    "network_latency": {
      "status": "WARN",
      "latency_ms": 150
    }
  }
}
```

## TTS Announcements

Boot result messages published to `/saltybot/tts_say`:
- Success: "Boot complete. All systems online."
- With errors: "Boot complete with errors. RPLIDAR, RealSense offline."

## Status Levels

- **OK** (0): System healthy, no action needed
- **WARN** (1): Minor issues, monitor closely
- **ERROR** (2): Critical failure, may affect operation
- **STALE** (3): No data, check unavailable

## Logs and Data

Diagnostic logs stored in `/home/seb/saltybot-data/diagnostics/`:
- Auto-rotated every 100MB or 30 days
- JSON format for easy parsing
- Full boot record + runtime metrics

## Integration with Full Stack

Add to `full_stack.launch.py` at t=0s:
```python
IncludeLaunchDescription(
    PythonLaunchDescriptionSource([...diagnostics.launch.py]),
    launch_arguments={'enable_startup_check': 'true'}.items(),
)
```

## Debugging

Check current diagnostics:
```bash
ros2 topic echo /saltybot/diagnostics
```

View latest diagnostic log:
```bash
tail -f /home/seb/saltybot-data/diagnostics/diagnostics_*.json | jq
```

Simulate diagnostic errors (for testing):
```bash
# Monitor what would be logged
ros2 launch saltybot_diagnostics diagnostics.launch.py \
  enable_startup_check:=true enable_runtime_monitoring:=true
```

## Hardware Requirements

- Jetson Orin (for temperature monitoring)
- Linux with psutil (for system resources)
- Standard ROS2 diagnostic_msgs package