Add "Vision System"

seb 2026-02-28 20:49:21 -05:00
parent f77882f814
commit 617520fe04

73
Vision-System.md Normal file

@ -0,0 +1,73 @@
# Vision System
## Camera Array — 360° Surround Vision
### 4× IMX219 160° Wide Angle (CSI)
- **Sensor:** Sony IMX219, 8MP
- **Resolution:** 3280×2464 (stills), 1080p @ 30fps, 720p @ 60fps (video)
- **FOV:** 160° wide angle — 4 cameras = full 360° with overlap
- **Interface:** MIPI CSI-2 (15-pin ribbon cable)
- **Focus:** Manual focus, 2.96mm lens
- **Arriving:** March 1, 2026
### Mounting Layout
```
FRONT
┌────────┐
│ CAM 0 │ → Forward (0°)
┌────┤ 160° ├────┐
│ └────────┘ │
│CAM 3 CAM 1│
│ Left Right│
│(270°) (90°) │
│ ┌────────┐ │
└────┤ CAM 2 ├────┘
│ 160° │
└────────┘
REAR (180°)
```
Each camera covers 160° — with 4 cameras at 90° intervals, there's ~70° overlap between adjacent cameras. This overlap is useful for:
- Cross-camera object tracking (hand off between views)
- Panoramic stitching
- Stereo depth estimation from adjacent pairs
### RealSense D435i (USB3)
- **Purpose:** Primary depth sensing + RGB for SLAM/navigation
- **Mount:** Forward-facing, mid-height on stem
- **Resolution:** 1280×720 depth @ 30fps + 1080p RGB
- **Depth range:** 0.2m - 10m
- **IMU:** Built-in accelerometer + gyroscope
### RPLIDAR A1M8 (USB)
- **Purpose:** 360° 2D laser scan for SLAM/nav2
- **Mount:** Top of sensor tower (under helmet)
- **Range:** 12m, 360°, ~5.5Hz scan rate
## CSI Adapter Required
The Orin Nano Dev Kit carrier exposes 2 CSI connectors. To use all 4 cameras simultaneously, need a **quad CSI multiplexer**:
- ArduCam Multi-Camera Adapter for Jetson (~$40 USD)
- Or Arducam Camarray HAT (hardware sync option)
- **TODO:** Confirm Orin Nano compatibility before ordering
## Processing Pipeline (on Orin Nano @ 67 TOPS)
```
4× IMX219 (CSI) ──→ Jetson ISP (hardware) ──→ TensorRT inference
├── Person detection (all 4 streams)
├── Object classification
└── Optical flow / motion
RealSense (USB3) ──→ depth + RGB ──→ SLAM (slam_toolbox)
──→ obstacle avoidance (Nav2 costmap)
RPLIDAR (USB) ──→ 2D scan ──→ SLAM (primary scan source)
```
## Total Sensor Budget
| Sensor | Interface | Power | Weight |
|--------|-----------|-------|--------|
| 4× IMX219 | CSI (0W USB) | ~1W total | ~20g total |
| RealSense D435i | USB3 | ~1.5W | 72g |
| RPLIDAR A1 | USB | ~2W | 170g |
| BME280 | I2C (on FC) | ~0.01W | 2g |
| **Total** | | **~4.5W** | **~264g** |