saltylab-firmware/jetson/docs/headscale-vpn-setup.md

# Headscale VPN Auto-Connect Setup — Jetson Orin

This document describes the auto-connect VPN setup for the Jetson Orin using Tailscale client connecting to the Headscale server at `tailscale.vayrette.com:8180`.

## Overview

**Device Name**: `saltylab-orin`
**Headscale Server**: `https://tailscale.vayrette.com:8180`
**Primary Features**:
- Auto-connect on system boot
- Persistent auth key for unattended login
- SSH + HTTP over Tailscale (tailnet)
- WiFi resilience and fallback
- systemd auto-restart on failure

## Architecture

### Components

1. **Tailscale Client** (`/usr/sbin/tailscaled`)
   - VPN daemon running on Jetson
   - Manages WireGuard tunnels
   - Connects to Headscale coordination server

2. **systemd Service** (`tailscale-vpn.service`)
   - Auto-starts on boot
   - Restarts on failure
   - Manages lifecycle of tailscaled daemon
   - Logs to journald

3. **Auth Key Manager** (`headscale-auth-helper.sh`)
   - Generates and validates auth keys
   - Stores keys securely at `/opt/saltybot/tailscale-auth.key`
   - Manages revocation

4. **Setup Script** (`setup-tailscale.sh`)
   - One-time installation of Tailscale package
   - Configures IP forwarding
   - Sets up persistent state directories

## Installation

### 1. Run Jetson Setup

```bash
sudo bash jetson/scripts/setup-jetson.sh
```

This configures the base system (Docker, NVMe, power mode, etc.).

### 2. Install Tailscale

```bash
sudo bash jetson/scripts/setup-tailscale.sh
```

This:
- Installs Tailscale from official Ubuntu repository
- Creates state/cache directories at `/var/lib/tailscale`
- Enables IP forwarding
- Creates environment config at `/etc/tailscale/tailscale-env`

### 3. Generate and Store Auth Key

```bash
sudo bash jetson/scripts/headscale-auth-helper.sh generate
```

This prompts you to enter a pre-authorized key (preauthkey) from the Headscale server.

**To get a preauthkey from Headscale**:

```bash
# On Headscale server
sudo headscale preauthkeys create --expiration 2160h --user default
# Copy the generated key (tskey_...)
```

Then paste into the helper script when prompted.

The key is stored at: `/opt/saltybot/tailscale-auth.key`

### 4. Install systemd Services

```bash
sudo bash jetson/systemd/install_systemd.sh
```

This:
- Deploys repo to `/opt/saltybot/jetson`
- Installs `tailscale-vpn.service` to `/etc/systemd/system/`
- Enables service for auto-start

### 5. Start the VPN Service

```bash
sudo systemctl start tailscale-vpn
```

Check status:

```bash
sudo systemctl status tailscale-vpn
sudo journalctl -fu tailscale-vpn
sudo tailscale status
```

## Usage

### Check VPN Status

```bash
sudo tailscale status
```

Example output:
```
  saltylab-orin        100.x.x.x    juno    linux   -
    100.x.x.x    is local IP address
     100.x.x.x  is remote IP address
```

### Access via SSH over Tailnet

From any machine on the tailnet:

```bash
ssh <username>@saltylab-orin.tail12345.ts.net
```

Or use the IP directly:

```bash
ssh <username>@100.x.x.x
```

### Access via HTTP

If running a web service (e.g., ROS2 visualization):

```
http://saltylab-orin.tail12345.ts.net:8080
# or
http://100.x.x.x:8080
```

### View Logs

```bash
# Real-time logs
sudo journalctl -fu tailscale-vpn

# Last 50 lines
sudo journalctl -n 50 -u tailscale-vpn

# Since last boot
sudo journalctl -b -u tailscale-vpn
```

## WiFi Resilience

The systemd service is configured with aggressive restart policies to handle WiFi drops:

```ini
Restart=always
RestartSec=5s
StartLimitInterval=60s
StartLimitBurst=10
```

This means:
- Automatically restarts after WiFi drops
- Waits 5 seconds between restart attempts
- Allows up to 10 restarts per 60-second window

The daemon will continuously attempt to reconnect when the primary network is unavailable.

## Persistent Storage

### Auth Key

**Location**: `/opt/saltybot/tailscale-auth.key`
**Permissions**: `600` (readable only by root)
**Ownership**: root:root

If the key file is missing on boot:
1. The systemd service will not auto-authenticate
2. Manual login is required: `sudo tailscale login --login-server=https://tailscale.vayrette.com:8180`
3. Re-run `headscale-auth-helper.sh generate` to store the key

### State Directory

**Location**: `/var/lib/tailscale/`
**Contents**:
- Machine state and private key
- WireGuard configuration
- Connection state

These are persisted so the device retains its identity and tailnet IP across reboots.

## Troubleshooting

### Service Won't Start

```bash
sudo systemctl status tailscale-vpn
sudo journalctl -u tailscale-vpn -n 30
```

Check for:
- Missing auth key: `ls -la /opt/saltybot/tailscale-auth.key`
- Tailscale package: `which tailscale`
- Permissions: `ls -la /var/lib/tailscale`

### Can't Connect to Headscale Server

```bash
sudo tailscale debug derp
curl -v https://tailscale.vayrette.com:8180
```

Check:
- Network connectivity: `ping 8.8.8.8`
- DNS: `nslookup tailscale.vayrette.com`
- Firewall: Port 443 (HTTPS) must be open

### Auth Key Expired

If the preauthkey expires:

```bash
# Get new key from Headscale server
# Then update:
sudo bash jetson/scripts/headscale-auth-helper.sh revoke
sudo bash jetson/scripts/headscale-auth-helper.sh generate
sudo systemctl restart tailscale-vpn
```

### Can't SSH Over Tailnet

```bash
# Verify device is in tailnet:
sudo tailscale status

# Check tailnet connectivity from another machine:
ping <device-ip>

# SSH with verbose output:
ssh -vv <username>@saltylab-orin.tail12345.ts.net
```

### Memory/Resource Issues

Monitor memory with Tailscale:

```bash
ps aux | grep tailscaled
sudo systemctl show -p MemoryUsage tailscale-vpn
```

Tailscale typically uses 30-80 MB of RAM depending on network size.

## Integration with Other Services

### ROS2 Services

Expose ROS2 services over the tailnet by ensuring your nodes bind to:
- `0.0.0.0` for accepting all interfaces
- Or explicitly bind to the Tailscale IP (`100.x.x.x`)

### Docker Services

If running Docker services that need tailnet access:
```dockerfile
# In docker-compose.yml
services:
  app:
    network_mode: host  # Access host's Tailscale interface
```

## Security Considerations

1. **Auth Key**: Stored in plaintext at `/opt/saltybot/tailscale-auth.key`
   - Use file permissions (`600`) to restrict access
   - Consider encryption if sensitive environment

2. **State Directory**: `/var/lib/tailscale/` contains private keys
   - Restricted permissions (700)
   - Only readable by root

3. **SSH Over Tailnet**: No ACL restrictions by default
   - Configure Headscale ACL rules if needed: `headscale acl`

4. **Key Rotation**: Rotate preauthkeys periodically
   - Headscale supports key expiration (set to 2160h = 90 days default)

## Maintenance

### Update Tailscale

```bash
sudo apt-get update
sudo apt-get install --only-upgrade tailscale
sudo systemctl restart tailscale-vpn
```

### Backup

Backup the auth key:

```bash
sudo cp /opt/saltybot/tailscale-auth.key ~/tailscale-auth.key.backup
```

### Monitor

Set up log rotation to prevent journal bloat:

```bash
# journald auto-rotates, but you can check:
journalctl --disk-usage
```

## MQTT Reporting

The Jetson reports VPN status to the MQTT broker:

```
saltylab/jetson/vpn/status -> online|offline|connecting
saltylab/jetson/vpn/ip -> 100.x.x.x
saltylab/jetson/vpn/hostname -> saltylab-orin.tail12345.ts.net
```

This is reported by the cellular/MQTT bridge node on boot and after reconnection.

## References

- [Headscale Documentation](https://headscale.net/)
- [Tailscale Documentation](https://tailscale.com/kb)
- [Tailscale CLI Reference](https://tailscale.com/kb/1080/cli)