saltylab-firmware/ui/social-bot/docs/SALTY_FACE_WEB_APP.md

# SaltyFace Web App UI

**Issue #370**: Animated robot expression UI as lightweight web app.

Runs in Chromium fullscreen kiosk mode (via Issue #374 Cage compositor) on MageDok 7" display.

## Architecture

```
┌────────────────────────────────────────────┐
│ MageDok 7" IPS Touchscreen (1024×600)       │
├────────────────────────────────────────────┤
│ Chromium Browser (Kiosk Mode)               │
├────────────────────────────────────────────┤
│ SaltyFace Web App (HTML5 Canvas)            │
│ ├─ Canvas-based face rendering             │
│ ├─ Touch overlay (status display)           │
│ ├─ 8 emotional states                       │
│ └─ ROS2 WebSocket bridge integration        │
├────────────────────────────────────────────┤
│ Node.js HTTP Server (localhost:3000)        │
│ ├─ Serves public/index.html                 │
│ ├─ Serves public/salty-face.js              │
│ └─ CORS headers for ROS bridge              │
├────────────────────────────────────────────┤
│ ROS2 Workloads                              │
│ ├─ /saltybot/state                          │
│ ├─ /saltybot/battery                        │
│ ├─ /saltybot/target_track                   │
│ ├─ /saltybot/obstacles                      │
│ ├─ /social/speech/is_speaking               │
│ └─ /social/speech/is_listening              │
└────────────────────────────────────────────┘
```

## Features

### 8 Emotional States

| State | Trigger | Display | Color |
|-------|---------|---------|-------|
| **Happy** | IDLE, TRACKING | Normal eyes, smile | Green (#10b981) |
| **Alert** | Obstacles < 0.5m | Wide eyes, tense | Red (#ef4444) |
| **Confused** | Target lost, SEARCHING | Wandering eyes, blink | Amber (#f59e0b) |
| **Sleeping** | Idle timeout | Closed eyes | Gray (#6b7280) |
| **Excited** | Target acquired | Bouncing eyes | Green (#22c55e) |
| **Emergency** | E-stop activated | Wide eyes, flashing | Red (#dc2626) |
| **Listening** | Microphone active | Focused eyes, upward | Cyan (#0ea5e9) |
| **Talking** | TTS output | Animated mouth | Cyan (#06b6d4) |

### UI Elements

- **Canvas Face**: 1024×600 animated face on 50% of screen
- **Status Overlay** (tap-toggleable):
  - Battery %
  - Robot state (IDLE, TRACKING, SEARCHING, EMERGENCY)
  - Distance to target
  - Speed (m/s)
  - System health %
- **Connection Status**: ROS bridge WebSocket status indicator
- **Debug Stats**: Current emotion, talking state, audio level

### Animation Performance

- **Frame Rate**: 60fps (Wayland native, Cage compositor)
- **Rendering**: Canvas 2D (GPU accelerated via WebGL fallback)
- **Target**: Orin Nano GPU (8-core NVIDIA Ampere GPU)
- **Memory**: ~80MB (Node.js server + browser tab)

## File Structure

```
ui/social-bot/
├── public/
│   ├── index.html          # Main page (1024×600)
│   └── salty-face.js       # Canvas rendering + ROS integration
├── server.js               # Node.js HTTP server (localhost:3000)
└── docs/
    └── SALTY_FACE_WEB_APP.md   # This file
```

## Installation & Setup

### 1. Install Node.js

```bash
# If not installed
sudo apt install -y nodejs npm
```

### 2. Copy Web App Files

```bash
# Copy to /opt/saltybot/app
sudo mkdir -p /opt/saltybot/app
sudo cp -r ui/social-bot/public /opt/saltybot/app/
sudo cp ui/social-bot/server.js /opt/saltybot/app/
sudo chmod +x /opt/saltybot/app/server.js
```

### 3. Install Systemd Service

```bash
# Copy service file
sudo cp systemd/salty-face-server.service /etc/systemd/system/

# Reload and enable
sudo systemctl daemon-reload
sudo systemctl enable salty-face-server.service
```

### 4. Start Service

```bash
# Manual start for testing
sudo systemctl start salty-face-server.service

# Check logs
sudo journalctl -u salty-face-server.service -f
```

### 5. Verify Web App

```bash
# From any machine on network:
# Open browser: http://<orin-ip>:3000

# Or test locally:
curl http://localhost:3000
# Should return index.html
```

## ROS2 Integration

### WebSocket Bridge

The web app connects to ROS2 via WebSocket using ROSLIB:

```javascript
const ros = new ROSLIB.Ros({
    url: 'ws://localhost:9090'  // rosbridge_server
});
```

### Topic Subscriptions

| Topic | Type | Purpose |
|-------|------|---------|
| `/saltybot/state` | `std_msgs/String` | Emotion trigger (EMERGENCY, TRACKING, SEARCHING) |
| `/saltybot/battery` | `std_msgs/Float32` | Battery % display |
| `/saltybot/target_track` | `geometry_msgs/Pose` | Target acquired → EXCITED |
| `/saltybot/obstacles` | `sensor_msgs/LaserScan` | Obstacle distance → ALERT |
| `/social/speech/is_speaking` | `std_msgs/Bool` | TTS output → TALKING emotion |
| `/social/speech/is_listening` | `std_msgs/Bool` | Microphone → LISTENING emotion |
| `/saltybot/audio_level` | `std_msgs/Float32` | Audio level display |

### Message Format

Most topics use simple JSON payloads:

```json
{
  "state": "TRACKING",
  "hasTarget": true,
  "obstacles": 0
}
```

## Browser Compatibility

- **Target**: Chromium 90+ (standard on Ubuntu 20.04+)
- **Features Used**:
  - Canvas 2D rendering
  - WebSocket (ROSLIB)
  - Touch events (MageDok HID input)
  - requestAnimationFrame (animation loop)

## Performance Tuning

### Reduce CPU/GPU Load

1. Lower animation frame rate:
   ```javascript
   // In salty-face.js, reduce frameCount checks
   if (state.frameCount % 2 === 0) {
       // Only render every 2 frames → 30fps
   }
   ```

2. Simplify eye rendering:
   ```javascript
   // Remove highlight reflection
   // Remove eye wander effect (confused state)
   ```

3. Disable unnecessary subscriptions:
   ```javascript
   // Comment out /saltybot/obstacles subscription
   // Comment out /social/speech subscriptions
   ```

### Monitor Resource Usage

```bash
# Watch CPU/GPU load during animation
watch -n 0.5 'top -bn1 | grep -E "PID|node|chromium"'

# Check GPU memory
tegrastats

# Check Node.js memory
ps aux | grep node
```

## Troubleshooting

### Web App Won't Load

**Issue**: Browser shows "Cannot GET /"

**Solutions**:
1. Verify server is running:
   ```bash
   sudo systemctl status salty-face-server.service
   sudo journalctl -u salty-face-server.service -n 20
   ```

2. Check port 3000 is listening:
   ```bash
   sudo netstat -tlnp | grep 3000
   ```

3. Verify public/ directory exists:
   ```bash
   ls -la /opt/saltybot/app/public/
   ```

### ROS Bridge Connection Fails

**Issue**: "WebSocket connection failed" in browser console

**Solutions**:
1. Verify rosbridge_server is running:
   ```bash
   ps aux | grep rosbridge
   ros2 run rosbridge_server rosbridge_websocket
   ```

2. Check ROS2 domain ID matches:
   ```bash
   echo $ROS_DOMAIN_ID
   # Should be same on robot and web app
   ```

3. Test WebSocket connectivity:
   ```bash
   # From browser console:
   new WebSocket('ws://localhost:9090')
   # Should show "WebSocket is open"
   ```

### Touch Input Not Working

**Issue**: MageDok touchscreen doesn't respond

**Solutions**:
1. Verify touch device exists:
   ```bash
   ls -la /dev/magedok-touch
   ```

2. Check udev rule is applied:
   ```bash
   sudo udevadm control --reload
   sudo udevadm trigger
   ```

3. Test touch input:
   ```bash
   sudo cat /dev/input/event* | xxd
   # Touch screen and watch for input data
   ```

### High Memory Usage

**Issue**: Node.js server consuming >256MB

**Solutions**:
1. Check for memory leaks in ROS topic subscriptions:
   ```javascript
   // Ensure topics are properly unsubscribed
   // Limit history of emotion changes
   ```

2. Monitor real-time memory:
   ```bash
   watch -n 1 'ps aux | grep node'
   ```

3. Adjust Node.js heap:
   ```bash
   # In salty-face-server.service:
   # NODE_OPTIONS=--max-old-space-size=128  # Reduce from 256MB
   ```

## Performance Benchmarks

### Boot Time
- Node.js server: 2-3 seconds
- Web app loads: <1 second
- Total to interactive: ~3-4 seconds

### Memory Usage
- Node.js server: ~30MB
- Chromium tab: ~50MB
- Total: ~80MB (vs 450MB for GNOME desktop)

### Frame Rate
- Canvas rendering: 60fps (Wayland native)
- Mouth animation: ~10fps (100ms per frame)
- Eye blinking: Instant (state change)

## Related Issues

- **Issue #374**: Cage + Chromium kiosk (display environment)
- **Issue #369**: MageDok display setup (hardware config)
- **Issue #371**: Accessibility mode (keyboard/voice input enhancements)

## Development

### Local Testing

```bash
# Start Node.js server
npm install  # Install dependencies if needed
node server.js --port 3000

# Open in browser
open http://localhost:3000
```

### Modify Emotion Config

Edit `public/salty-face.js`:

```javascript
const EMOTION_CONFIG = {
    [EMOTIONS.HAPPY]: {
        eyeScale: 1.0,  // Eye size
        pupilPos: { x: 0, y: 0 },  // Pupil offset
        blinkRate: 3000,  // Blink interval (ms)
        color: '#10b981',  // Eye color (CSS)
    },
    // ... other emotions
};
```

### Add New Topics

Edit `public/salty-face.js`, in `subscribeToRosTopics()`:

```javascript
const newTopic = new ROSLIB.Topic({
    ros,
    name: '/your/topic',
    messageType: 'std_msgs/Float32',
});
newTopic.subscribe((msg) => {
    state.newValue = msg.data;
});
```

## References

- [Canvas API](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API)
- [ROSLIB.js](http://wiki.ros.org/roslibjs)
- [Chromium Kiosk Mode](https://chromium.googlesource.com/chromium/src/+/main/docs/kiosk_mode.md)
- [Wayland Protocol](https://wayland.freedesktop.org/)