saltylab-firmware/ui/social-bot/docs/SALTY_FACE_WEB_APP.md

SaltyFace Web App UI

Issue #370: Animated robot expression UI as lightweight web app.

Runs in Chromium fullscreen kiosk mode (via Issue #374 Cage compositor) on MageDok 7" display.

Architecture

┌────────────────────────────────────────────┐
│ MageDok 7" IPS Touchscreen (1024×600)       │
├────────────────────────────────────────────┤
│ Chromium Browser (Kiosk Mode)               │
├────────────────────────────────────────────┤
│ SaltyFace Web App (HTML5 Canvas)            │
│ ├─ Canvas-based face rendering             │
│ ├─ Touch overlay (status display)           │
│ ├─ 8 emotional states                       │
│ └─ ROS2 WebSocket bridge integration        │
├────────────────────────────────────────────┤
│ Node.js HTTP Server (localhost:3000)        │
│ ├─ Serves public/index.html                 │
│ ├─ Serves public/salty-face.js              │
│ └─ CORS headers for ROS bridge              │
├────────────────────────────────────────────┤
│ ROS2 Workloads                              │
│ ├─ /saltybot/state                          │
│ ├─ /saltybot/battery                        │
│ ├─ /saltybot/target_track                   │
│ ├─ /saltybot/obstacles                      │
│ ├─ /social/speech/is_speaking               │
│ └─ /social/speech/is_listening              │
└────────────────────────────────────────────┘

Features

8 Emotional States

State	Trigger	Display	Color
Happy	IDLE, TRACKING	Normal eyes, smile	Green (#10b981)
Alert	Obstacles < 0.5m	Wide eyes, tense	Red (#ef4444)
Confused	Target lost, SEARCHING	Wandering eyes, blink	Amber (#f59e0b)
Sleeping	Idle timeout	Closed eyes	Gray (#6b7280)
Excited	Target acquired	Bouncing eyes	Green (#22c55e)
Emergency	E-stop activated	Wide eyes, flashing	Red (#dc2626)
Listening	Microphone active	Focused eyes, upward	Cyan (#0ea5e9)
Talking	TTS output	Animated mouth	Cyan (#06b6d4)

UI Elements

• Canvas Face: 1024×600 animated face on 50% of screen
• Status Overlay (tap-toggleable):
  • Battery %
  • Robot state (IDLE, TRACKING, SEARCHING, EMERGENCY)
  • Distance to target
  • Speed (m/s)
  • System health %
• Connection Status: ROS bridge WebSocket status indicator
• Debug Stats: Current emotion, talking state, audio level

Animation Performance

• Frame Rate: 60fps (Wayland native, Cage compositor)
• Rendering: Canvas 2D (GPU accelerated via WebGL fallback)
• Target: Orin Nano GPU (8-core NVIDIA Ampere GPU)
• Memory: ~80MB (Node.js server + browser tab)

File Structure

ui/social-bot/
├── public/
│   ├── index.html          # Main page (1024×600)
│   └── salty-face.js       # Canvas rendering + ROS integration
├── server.js               # Node.js HTTP server (localhost:3000)
└── docs/
    └── SALTY_FACE_WEB_APP.md   # This file

Installation & Setup

1. Install Node.js

# If not installed
sudo apt install -y nodejs npm

2. Copy Web App Files

# Copy to /opt/saltybot/app
sudo mkdir -p /opt/saltybot/app
sudo cp -r ui/social-bot/public /opt/saltybot/app/
sudo cp ui/social-bot/server.js /opt/saltybot/app/
sudo chmod +x /opt/saltybot/app/server.js

3. Install Systemd Service

# Copy service file
sudo cp systemd/salty-face-server.service /etc/systemd/system/

# Reload and enable
sudo systemctl daemon-reload
sudo systemctl enable salty-face-server.service

4. Start Service

# Manual start for testing
sudo systemctl start salty-face-server.service

# Check logs
sudo journalctl -u salty-face-server.service -f

5. Verify Web App

# From any machine on network:
# Open browser: http://<orin-ip>:3000

# Or test locally:
curl http://localhost:3000
# Should return index.html

ROS2 Integration

WebSocket Bridge

The web app connects to ROS2 via WebSocket using ROSLIB:

const ros = new ROSLIB.Ros({
    url: 'ws://localhost:9090'  // rosbridge_server
});

Topic Subscriptions

Topic	Type	Purpose
/saltybot/state	std_msgs/String	Emotion trigger (EMERGENCY, TRACKING, SEARCHING)
/saltybot/battery	std_msgs/Float32	Battery % display
/saltybot/target_track	geometry_msgs/Pose	Target acquired → EXCITED
/saltybot/obstacles	sensor_msgs/LaserScan	Obstacle distance → ALERT
/social/speech/is_speaking	std_msgs/Bool	TTS output → TALKING emotion
/social/speech/is_listening	std_msgs/Bool	Microphone → LISTENING emotion
/saltybot/audio_level	std_msgs/Float32	Audio level display

Message Format

Most topics use simple JSON payloads:

{
  "state": "TRACKING",
  "hasTarget": true,
  "obstacles": 0
}

Browser Compatibility

• Target: Chromium 90+ (standard on Ubuntu 20.04+)
• Features Used:
  • Canvas 2D rendering
  • WebSocket (ROSLIB)
  • Touch events (MageDok HID input)
  • requestAnimationFrame (animation loop)

Performance Tuning

Reduce CPU/GPU Load

1. Lower animation frame rate:

   // In salty-face.js, reduce frameCount checks
   if (state.frameCount % 2 === 0) {
       // Only render every 2 frames → 30fps
   }
   

2. Simplify eye rendering:

   // Remove highlight reflection
   // Remove eye wander effect (confused state)
   

3. Disable unnecessary subscriptions:

   // Comment out /saltybot/obstacles subscription
   // Comment out /social/speech subscriptions
   

Monitor Resource Usage

# Watch CPU/GPU load during animation
watch -n 0.5 'top -bn1 | grep -E "PID|node|chromium"'

# Check GPU memory
tegrastats

# Check Node.js memory
ps aux | grep node

Troubleshooting

Web App Won't Load

Issue: Browser shows "Cannot GET /"

Solutions:

1. Verify server is running:

   sudo systemctl status salty-face-server.service
   sudo journalctl -u salty-face-server.service -n 20
   

2. Check port 3000 is listening:

   sudo netstat -tlnp | grep 3000
   

3. Verify public/ directory exists:

   ls -la /opt/saltybot/app/public/
   

ROS Bridge Connection Fails

Issue: "WebSocket connection failed" in browser console

Solutions:

1. Verify rosbridge_server is running:

   ps aux | grep rosbridge
   ros2 run rosbridge_server rosbridge_websocket
   

2. Check ROS2 domain ID matches:

   echo $ROS_DOMAIN_ID
   # Should be same on robot and web app
   

3. Test WebSocket connectivity:

   # From browser console:
   new WebSocket('ws://localhost:9090')
   # Should show "WebSocket is open"
   

Touch Input Not Working

Issue: MageDok touchscreen doesn't respond

Solutions:

1. Verify touch device exists:

   ls -la /dev/magedok-touch
   

2. Check udev rule is applied:

   sudo udevadm control --reload
   sudo udevadm trigger
   

3. Test touch input:

   sudo cat /dev/input/event* | xxd
   # Touch screen and watch for input data
   

High Memory Usage

Issue: Node.js server consuming >256MB

Solutions:

1. Check for memory leaks in ROS topic subscriptions:

   // Ensure topics are properly unsubscribed
   // Limit history of emotion changes
   

2. Monitor real-time memory:

   watch -n 1 'ps aux | grep node'
   

3. Adjust Node.js heap:

   # In salty-face-server.service:
   # NODE_OPTIONS=--max-old-space-size=128  # Reduce from 256MB
   

Performance Benchmarks

Boot Time

• Node.js server: 2-3 seconds
• Web app loads: <1 second
• Total to interactive: ~3-4 seconds

Memory Usage

• Node.js server: ~30MB
• Chromium tab: ~50MB
• Total: ~80MB (vs 450MB for GNOME desktop)

Frame Rate

• Canvas rendering: 60fps (Wayland native)
• Mouth animation: ~10fps (100ms per frame)
• Eye blinking: Instant (state change)

Related Issues

• Issue #374: Cage + Chromium kiosk (display environment)
• Issue #369: MageDok display setup (hardware config)
• Issue #371: Accessibility mode (keyboard/voice input enhancements)

Development

Local Testing

# Start Node.js server
npm install  # Install dependencies if needed
node server.js --port 3000

# Open in browser
open http://localhost:3000

Modify Emotion Config

Edit public/salty-face.js:

const EMOTION_CONFIG = {
    [EMOTIONS.HAPPY]: {
        eyeScale: 1.0,  // Eye size
        pupilPos: { x: 0, y: 0 },  // Pupil offset
        blinkRate: 3000,  // Blink interval (ms)
        color: '#10b981',  // Eye color (CSS)
    },
    // ... other emotions
};

Add New Topics

Edit public/salty-face.js, in subscribeToRosTopics():

const newTopic = new ROSLIB.Topic({
    ros,
    name: '/your/topic',
    messageType: 'std_msgs/Float32',
});
newTopic.subscribe((msg) => {
    state.newValue = msg.data;
});

References

• Canvas API
• ROSLIB.js
• Chromium Kiosk Mode
• Wayland Protocol