Merge pull request 'feat(calibration): IMX219 intrinsic + extrinsic calibration workflow #106' (#113) from sl-perception/issue-106-calibration into main

This commit is contained in:
sl-jetson 2026-03-02 08:41:20 -05:00
commit 25f2bab24a
18 changed files with 2019 additions and 0 deletions

View File

@ -0,0 +1,195 @@
# IMX219 Camera Calibration Workflow
Full step-by-step calibration procedure for the 4× IMX219 160° fisheye CSI cameras
on saltybot (cam0=front, cam1=right, cam2=rear, cam3=left).
## Hardware
| Camera | Position | Yaw | Topic |
|--------|----------|-------|------------------------------|
| cam0 | Front | 0° | /camera/front/image_raw |
| cam1 | Right | -90° | /camera/right/image_raw |
| cam2 | Rear | 180° | /camera/rear/image_raw |
| cam3 | Left | 90° | /camera/left/image_raw |
Mount geometry: radius=5cm from robot centre, height=30cm, 90° angular spacing.
## Prerequisites
- OpenCV with fisheye support: `pip3 install opencv-python numpy pyyaml`
- For ROS2 mode: ROS2 Humble, cv_bridge, image_transport
- Printed checkerboard: 8×6 inner corners, 25mm squares (A3 paper recommended)
- Generator: `python3 -c "import cv2; board=cv2.aruco.CharucoBoard((9,7),0.025,0.0175,cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250)); cv2.imwrite('board.png', board.generateImage((700,500)))"`
- Or download from: https://calib.io/pages/camera-calibration-pattern-generator
## Step 1 — Print and Prepare the Checkerboard
- Print the 8×6 inner-corner checkerboard at exactly 25mm per square
- Mount flat on a rigid board (cardboard or foamcore) — no warping allowed
- Measure actual square size with calipers, update `--square-size` if different
## Step 2 — Start Cameras
```bash
# On the Jetson, in the saltybot docker container:
ros2 launch saltybot_cameras cameras.launch.py
# Verify all 4 topics are publishing:
ros2 topic list | grep camera
ros2 topic hz /camera/front/image_raw
```
## Step 3 — Capture Calibration Images (30 per camera)
Run for each camera in turn. Move the board to many different positions and
angles within the field of view — especially corners and edges.
```bash
# Option A: ROS2 mode (recommended on Jetson, requires running cameras)
ros2 run saltybot_calibration capture_calibration_images.py \
--ros --camera front --n-images 30 \
--output-dir /mnt/nvme/saltybot/calibration/raw
ros2 run saltybot_calibration capture_calibration_images.py \
--ros --camera right --n-images 30 \
--output-dir /mnt/nvme/saltybot/calibration/raw
ros2 run saltybot_calibration capture_calibration_images.py \
--ros --camera rear --n-images 30 \
--output-dir /mnt/nvme/saltybot/calibration/raw
ros2 run saltybot_calibration capture_calibration_images.py \
--ros --camera left --n-images 30 \
--output-dir /mnt/nvme/saltybot/calibration/raw
# Option B: All cameras sequentially (prompts between cameras)
ros2 run saltybot_calibration capture_calibration_images.py \
--ros --camera all --n-images 30 \
--output-dir /mnt/nvme/saltybot/calibration/raw
# Option C: Standalone (no ROS2, uses /dev/videoX directly)
python3 capture_calibration_images.py \
--camera front --n-images 30 \
--device /dev/video0 \
--output-dir /mnt/nvme/saltybot/calibration/raw
```
### Capture Tips for Fisheye Lenses
- Cover the entire FoV — include corners and edges (where distortion is strongest)
- Use 5+ different orientations of the board (tilt, rotate, skew)
- Keep the board fully visible (no partial views)
- Good lighting, no motion blur
- Aim for 20-30 images with varied positions
## Step 4 — Compute Intrinsics per Camera
```bash
for cam in front right rear left; do
ros2 run saltybot_calibration calibrate_camera.py \
--images-dir /mnt/nvme/saltybot/calibration/raw/cam_${cam} \
--output /mnt/nvme/saltybot/calibration/${cam}/camera_info.yaml \
--camera-name camera_${cam} \
--board-size 8x6 \
--square-size 0.025 \
--image-size 640x480 \
--show-reprojection
done
```
## Step 5 — Verify Reprojection Error
Target: RMS reprojection error < 1.5 px per camera.
If RMS > 1.5 px:
1. Remove outlier images (highest per-image error from table)
2. Recapture with more varied board positions
3. Check board is truly flat and square size is correct
4. Try `--image-size 1280x720` if capturing at higher resolution
```bash
# Quick check — view undistorted preview
ros2 run saltybot_calibration preview_undistorted.py \
--image /mnt/nvme/saltybot/calibration/raw/cam_front/frame_0000.png \
--calibration /mnt/nvme/saltybot/calibration/front/camera_info.yaml
```
## Step 6 — Compute Extrinsics (Mechanical Mode)
Uses known mount geometry (radius=5cm, height=30cm, 90° spacing) to compute
analytically exact camera-to-base_link transforms.
```bash
ros2 run saltybot_calibration calibrate_extrinsics.py \
--mode mechanical \
--mount-radius 0.05 \
--mount-height 0.30 \
--output /mnt/nvme/saltybot/calibration/extrinsics.yaml \
--generate-launch \
~/ros2_ws/src/saltybot_bringup/launch/static_transforms.launch.py
```
If mechanical dimensions differ from defaults, update `--mount-radius` and
`--mount-height` from your actual measurements.
## Step 7 — Generate Static Transform Launch File
(Already done by --generate-launch in Step 6, but can regenerate independently:)
```bash
ros2 run saltybot_calibration generate_static_transforms.py \
--extrinsics /mnt/nvme/saltybot/calibration/extrinsics.yaml \
--output ~/ros2_ws/src/saltybot_bringup/launch/static_transforms.launch.py
```
## Step 8 — Update surround_vision_params.yaml
Edit `jetson/config/surround_vision_params.yaml` (or the equivalent in your
saltybot_surround package) to point camera_info_url at the new calibration files:
```yaml
camera_info_urls:
front: 'file:///mnt/nvme/saltybot/calibration/front/camera_info.yaml'
right: 'file:///mnt/nvme/saltybot/calibration/right/camera_info.yaml'
rear: 'file:///mnt/nvme/saltybot/calibration/rear/camera_info.yaml'
left: 'file:///mnt/nvme/saltybot/calibration/left/camera_info.yaml'
```
## Step 9 — Restart Nodes
```bash
# Restart camera and surround nodes to pick up new calibration
ros2 lifecycle set /saltybot_cameras shutdown
ros2 launch saltybot_bringup cameras.launch.py
ros2 launch saltybot_surround surround.launch.py
```
## Output Files
After full calibration, you should have:
```
/mnt/nvme/saltybot/calibration/
front/camera_info.yaml # intrinsics: K, D (equidistant)
right/camera_info.yaml
rear/camera_info.yaml
left/camera_info.yaml
extrinsics.yaml # base_link -> camera_*_optical_frame
```
Template files are in `jetson/calibration/` (tracked in git) with placeholder
IMX219 values (fx=229px @ 640×480). Replace with calibrated values.
## Distortion Model Reference
The IMX219 160° fisheye uses the **equidistant** (Kannala-Brandt) model:
```
r(theta) = k1*theta + k2*theta^3 + k3*theta^5 + k4*theta^7
```
where theta is the angle of incidence. OpenCV's `cv2.fisheye` module implements
this model with 4 coefficients [k1, k2, k3, k4].
This is different from the `plumb_bob` (radial/tangential) model used for
standard lenses — do not mix them up.

View File

@ -0,0 +1,27 @@
# TEMPLATE — replace with output of calibrate_camera.py
# IMX219 160 deg at 640x480: estimated placeholder values
# Run: ros2 run saltybot_calibration calibrate_camera.py \
# --images-dir /mnt/nvme/saltybot/calibration/raw/cam_front \
# --output /mnt/nvme/saltybot/calibration/front/camera_info.yaml \
# --camera-name camera_front --image-size 640x480
image_width: 640
image_height: 480
camera_name: camera_front
camera_matrix:
rows: 3
cols: 3
data: [229.0, 0.0, 320.0, 0.0, 229.0, 240.0, 0.0, 0.0, 1.0]
distortion_model: equidistant
distortion_coefficients:
rows: 1
cols: 4
data: [0.0, 0.0, 0.0, 0.0]
rectification_matrix:
rows: 3
cols: 3
data: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
projection_matrix:
rows: 3
cols: 4
data: [229.0, 0.0, 320.0, 0.0, 0.0, 229.0, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
_calibration_rms_px: null

View File

@ -0,0 +1,27 @@
# TEMPLATE — replace with output of calibrate_camera.py
# IMX219 160 deg at 640x480: estimated placeholder values
# Run: ros2 run saltybot_calibration calibrate_camera.py \
# --images-dir /mnt/nvme/saltybot/calibration/raw/cam_left \
# --output /mnt/nvme/saltybot/calibration/left/camera_info.yaml \
# --camera-name camera_left --image-size 640x480
image_width: 640
image_height: 480
camera_name: camera_left
camera_matrix:
rows: 3
cols: 3
data: [229.0, 0.0, 320.0, 0.0, 229.0, 240.0, 0.0, 0.0, 1.0]
distortion_model: equidistant
distortion_coefficients:
rows: 1
cols: 4
data: [0.0, 0.0, 0.0, 0.0]
rectification_matrix:
rows: 3
cols: 3
data: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
projection_matrix:
rows: 3
cols: 4
data: [229.0, 0.0, 320.0, 0.0, 0.0, 229.0, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
_calibration_rms_px: null

View File

@ -0,0 +1,27 @@
# TEMPLATE — replace with output of calibrate_camera.py
# IMX219 160 deg at 640x480: estimated placeholder values
# Run: ros2 run saltybot_calibration calibrate_camera.py \
# --images-dir /mnt/nvme/saltybot/calibration/raw/cam_rear \
# --output /mnt/nvme/saltybot/calibration/rear/camera_info.yaml \
# --camera-name camera_rear --image-size 640x480
image_width: 640
image_height: 480
camera_name: camera_rear
camera_matrix:
rows: 3
cols: 3
data: [229.0, 0.0, 320.0, 0.0, 229.0, 240.0, 0.0, 0.0, 1.0]
distortion_model: equidistant
distortion_coefficients:
rows: 1
cols: 4
data: [0.0, 0.0, 0.0, 0.0]
rectification_matrix:
rows: 3
cols: 3
data: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
projection_matrix:
rows: 3
cols: 4
data: [229.0, 0.0, 320.0, 0.0, 0.0, 229.0, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
_calibration_rms_px: null

View File

@ -0,0 +1,27 @@
# TEMPLATE — replace with output of calibrate_camera.py
# IMX219 160 deg at 640x480: estimated placeholder values
# Run: ros2 run saltybot_calibration calibrate_camera.py \
# --images-dir /mnt/nvme/saltybot/calibration/raw/cam_right \
# --output /mnt/nvme/saltybot/calibration/right/camera_info.yaml \
# --camera-name camera_right --image-size 640x480
image_width: 640
image_height: 480
camera_name: camera_right
camera_matrix:
rows: 3
cols: 3
data: [229.0, 0.0, 320.0, 0.0, 229.0, 240.0, 0.0, 0.0, 1.0]
distortion_model: equidistant
distortion_coefficients:
rows: 1
cols: 4
data: [0.0, 0.0, 0.0, 0.0]
rectification_matrix:
rows: 3
cols: 3
data: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
projection_matrix:
rows: 3
cols: 4
data: [229.0, 0.0, 320.0, 0.0, 0.0, 229.0, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
_calibration_rms_px: null

View File

@ -0,0 +1,43 @@
# Camera extrinsics — base_link to camera_optical_frame transforms
# Generated from mechanical mount geometry:
# mount_radius = 0.05m, mount_height = 0.30m
# Camera yaw: front=0deg, right=-90deg, rear=180deg, left=90deg
#
# Quaternion convention: [qx, qy, qz, qw] (xyzw)
# Rotation: camera_optical_frame -> base_link
# camera_optical: z=forward(out), x=right, y=down
# base_link: x=forward, y=left, z=up (ROS REP-103)
#
# Regenerate with:
# python3 calibrate_extrinsics.py --mode mechanical \
# --mount-radius 0.05 --mount-height 0.30 \
# --output jetson/calibration/extrinsics.yaml
_description: Camera extrinsics - base_link to camera_optical_frame transforms
_mount_radius_m: 0.05
_mount_height_m: 0.30
cameras:
front:
parent_frame: base_link
child_frame: camera_front_optical_frame
translation: [0.05, 0.0, 0.30]
rotation_quat_xyzw: [0.5, -0.5, 0.5, -0.5]
right:
parent_frame: base_link
child_frame: camera_right_optical_frame
translation: [0.0, 0.05, 0.30]
rotation_quat_xyzw: [-0.707107, 0.0, 0.0, 0.707107]
rear:
parent_frame: base_link
child_frame: camera_rear_optical_frame
translation: [-0.05, 0.0, 0.30]
rotation_quat_xyzw: [-0.5, -0.5, 0.5, 0.5]
left:
parent_frame: base_link
child_frame: camera_left_optical_frame
translation: [0.0, -0.05, 0.30]
rotation_quat_xyzw: [0.0, -0.707107, 0.707107, 0.0]

View File

@ -0,0 +1,29 @@
cmake_minimum_required(VERSION 3.8)
project(saltybot_calibration)
find_package(ament_cmake REQUIRED)
find_package(ament_cmake_python REQUIRED)
find_package(rclpy REQUIRED)
ament_python_install_package(${PROJECT_NAME})
install(PROGRAMS
scripts/capture_calibration_images.py
scripts/calibrate_camera.py
scripts/calibrate_extrinsics.py
scripts/preview_undistorted.py
scripts/generate_static_transforms.py
DESTINATION lib/${PROJECT_NAME}
)
install(DIRECTORY
launch config
DESTINATION share/${PROJECT_NAME}
)
if(BUILD_TESTING)
find_package(ament_lint_auto REQUIRED)
ament_lint_auto_find_test_dependencies()
endif()
ament_package()

View File

@ -0,0 +1,27 @@
calibration:
board_size: '8x6' # inner corners (cols x rows)
square_size_m: 0.025 # 25mm squares
n_images: 30 # images to capture per camera
image_size: '640x480' # capture resolution
cameras:
front:
topic: '/camera/front/image_raw'
output_yaml: '/mnt/nvme/saltybot/calibration/front/camera_info.yaml'
right:
topic: '/camera/right/image_raw'
output_yaml: '/mnt/nvme/saltybot/calibration/right/camera_info.yaml'
rear:
topic: '/camera/rear/image_raw'
output_yaml: '/mnt/nvme/saltybot/calibration/rear/camera_info.yaml'
left:
topic: '/camera/left/image_raw'
output_yaml: '/mnt/nvme/saltybot/calibration/left/camera_info.yaml'
extrinsics:
mode: mechanical
mount_radius_m: 0.05
mount_height_m: 0.30
output_yaml: '/mnt/nvme/saltybot/calibration/extrinsics.yaml'
reprojection_warn_threshold_px: 1.5

View File

@ -0,0 +1,108 @@
"""
calibration_capture.launch.py Calibration workflow launcher for saltybot IMX219 cameras.
Full calibration workflow:
Step 1: Start cameras (requires saltybot_cameras package to be running)
Step 2: Run capture_calibration_images.py for each camera (30 images each)
Step 3: Run calibrate_camera.py for each camera
Step 4: Run calibrate_extrinsics.py
Step 5: Run generate_static_transforms.py
This launch file starts the capture script for one camera via ExecuteProcess.
For full workflow, run each step manually (see jetson/calibration/README.md).
Usage:
ros2 launch saltybot_calibration calibration_capture.launch.py camera:=front
ros2 launch saltybot_calibration calibration_capture.launch.py camera:=all
Arguments:
camera Camera to capture (front/right/rear/left/all, default: front)
n_images Images to capture per camera (default: 30)
output_dir Output directory (default: /mnt/nvme/saltybot/calibration/raw)
board_size Checkerboard inner corners WxH (default: 8x6)
square_size Physical square size in metres (default: 0.025)
"""
from launch import LaunchDescription
from launch.actions import DeclareLaunchArgument, ExecuteProcess, LogInfo
from launch.substitutions import LaunchConfiguration, PathJoinSubstitution
from launch_ros.substitutions import FindPackageShare
def generate_launch_description():
camera_arg = DeclareLaunchArgument(
'camera', default_value='front',
description='Camera to capture from: front/right/rear/left/all',
)
n_images_arg = DeclareLaunchArgument(
'n_images', default_value='30',
description='Number of calibration images to capture per camera',
)
output_dir_arg = DeclareLaunchArgument(
'output_dir', default_value='/mnt/nvme/saltybot/calibration/raw',
description='Root output directory for captured images',
)
board_size_arg = DeclareLaunchArgument(
'board_size', default_value='8x6',
description='Checkerboard inner corners as WxH',
)
square_size_arg = DeclareLaunchArgument(
'square_size', default_value='0.025',
description='Physical checkerboard square size in metres',
)
pkg_share = FindPackageShare('saltybot_calibration')
capture_script = PathJoinSubstitution([
pkg_share, '..', '..', '..', 'lib', 'saltybot_calibration',
'capture_calibration_images.py',
])
capture_process = ExecuteProcess(
cmd=[
'python3', capture_script,
'--ros',
'--camera', LaunchConfiguration('camera'),
'--n-images', LaunchConfiguration('n_images'),
'--output-dir', LaunchConfiguration('output_dir'),
'--board-size', LaunchConfiguration('board_size'),
'--square-size', LaunchConfiguration('square_size'),
],
output='screen',
)
workflow_info = LogInfo(
msg=[
'\n',
'=== IMX219 Calibration Workflow ===\n',
'Step 1 (this launch): Capture checkerboard images\n',
' Controls: SPACE=capture r=reset q=done\n',
'\n',
'Step 2: Calibrate intrinsics per camera:\n',
' ros2 run saltybot_calibration calibrate_camera.py \\\n',
' --images-dir /mnt/nvme/saltybot/calibration/raw/cam_front \\\n',
' --output /mnt/nvme/saltybot/calibration/front/camera_info.yaml \\\n',
' --camera-name camera_front --image-size 640x480\n',
'\n',
'Step 3: Compute extrinsics:\n',
' ros2 run saltybot_calibration calibrate_extrinsics.py \\\n',
' --mode mechanical \\\n',
' --output /mnt/nvme/saltybot/calibration/extrinsics.yaml\n',
'\n',
'Step 4: Generate static transforms launch file:\n',
' ros2 run saltybot_calibration generate_static_transforms.py \\\n',
' --extrinsics /mnt/nvme/saltybot/calibration/extrinsics.yaml \\\n',
' --output ~/ros2_ws/src/saltybot_bringup/launch/static_transforms.launch.py\n',
'\n',
'See jetson/calibration/README.md for full instructions.\n',
]
)
return LaunchDescription([
camera_arg,
n_images_arg,
output_dir_arg,
board_size_arg,
square_size_arg,
workflow_info,
capture_process,
])

View File

@ -0,0 +1,25 @@
<?xml version="1.0"?>
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
<package format="3">
<name>saltybot_calibration</name>
<version>0.1.0</version>
<description>IMX219 camera intrinsic + extrinsic calibration workflow for saltybot</description>
<maintainer email="seb@vayrette.com">seb</maintainer>
<license>MIT</license>
<buildtool_depend>ament_cmake_python</buildtool_depend>
<buildtool_depend>ament_cmake</buildtool_depend>
<depend>rclpy</depend>
<depend>sensor_msgs</depend>
<depend>cv_bridge</depend>
<depend>image_transport</depend>
<exec_depend>python3-opencv</exec_depend>
<exec_depend>python3-numpy</exec_depend>
<exec_depend>python3-yaml</exec_depend>
<test_depend>ament_copyright</test_depend>
<test_depend>ament_flake8</test_depend>
<test_depend>ament_pep257</test_depend>
<test_depend>python3-pytest</test_depend>
<export>
<build_type>ament_cmake</build_type>
</export>
</package>

View File

@ -0,0 +1,175 @@
"""camera_calibrator.py — OpenCV fisheye intrinsic calibration for IMX219 cameras.
Uses cv2.fisheye model (Scaramuzza/Kannala-Brandt 4-param distortion):
k1, k2, k3, k4 equidistant distortion coefficients
Outputs camera_info_manager compatible YAML:
image_width, image_height, camera_name
camera_matrix: 3x3 K
distortion_model: equidistant
distortion_coefficients: [k1, k2, k3, k4]
rectification_matrix: 3x3 I
projection_matrix: 3x4 P (same as K padded)
"""
import cv2
import numpy as np
import yaml
import os
from pathlib import Path
def find_checkerboard(img_bgr: np.ndarray, board_size: tuple,
flags: int = None) -> tuple:
"""Detect checkerboard corners in image.
Args:
img_bgr: BGR image
board_size: (cols, rows) inner corners, e.g. (8, 6)
flags: cv2.findChessboardCorners flags
Returns:
(found, corners) corners is [N, 1, 2] float32 or None
"""
gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
if flags is None:
flags = cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE
found, corners = cv2.findChessboardCorners(gray, board_size, flags)
if found:
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 50, 0.001)
corners = cv2.cornerSubPix(gray, corners, (7, 7), (-1, -1), criteria)
return found, corners
def calibrate_fisheye(
image_paths: list,
board_size: tuple,
square_size_m: float,
image_size: tuple, # (width, height)
) -> dict:
"""Run cv2.fisheye calibration.
Args:
image_paths: list of calibration image file paths
board_size: (cols, rows) inner corners
square_size_m: physical square size in metres (e.g. 0.025)
image_size: (width, height) of images
Returns:
dict with keys: K, D, rvecs, tvecs, rms, n_images_used
"""
obj_p = np.zeros((board_size[0] * board_size[1], 1, 3), dtype=np.float64)
obj_p[:, 0, :2] = np.mgrid[0:board_size[0], 0:board_size[1]].T.reshape(-1, 2)
obj_p *= square_size_m
obj_pts = []
img_pts = []
used = 0
skipped = 0
for path in image_paths:
img = cv2.imread(path)
if img is None:
print(f' [WARN] Cannot read: {path}')
skipped += 1
continue
found, corners = find_checkerboard(img, board_size)
if not found:
print(f' [SKIP] No corners: {os.path.basename(path)}')
skipped += 1
continue
obj_pts.append(obj_p)
img_pts.append(corners)
used += 1
print(f' [OK] {os.path.basename(path)}')
if used < 5:
raise RuntimeError(
f'Only {used} valid images (need >=5). Check lighting and board visibility.'
)
K = np.zeros((3, 3))
D = np.zeros((4, 1))
flags = (
cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC |
cv2.fisheye.CALIB_CHECK_COND |
cv2.fisheye.CALIB_FIX_SKEW
)
rms, K, D, rvecs, tvecs = cv2.fisheye.calibrate(
obj_pts, img_pts, image_size, K, D, flags=flags
)
print(f' Calibration RMS reprojection error: {rms:.4f} px ({used} images)')
return {
'K': K, 'D': D, 'rvecs': rvecs, 'tvecs': tvecs,
'rms': rms, 'n_images_used': used, 'n_images_skipped': skipped,
'obj_pts': obj_pts, 'img_pts': img_pts,
}
def write_camera_info_yaml(
output_path: str,
camera_name: str,
image_size: tuple, # (width, height)
K: np.ndarray,
D: np.ndarray,
rms: float,
) -> None:
"""Write camera_info_manager compatible YAML.
Format compatible with camera_info_manager CameraInfoManager.loadCameraInfo().
distortion_model = 'equidistant' (ROS2 fisheye standard).
D has 4 coefficients [k1, k2, k3, k4] for fisheye model.
"""
w, h = image_size
fx = float(K[0, 0])
fy = float(K[1, 1])
cx = float(K[0, 2])
cy = float(K[1, 2])
data = {
'image_width': w,
'image_height': h,
'camera_name': camera_name,
'camera_matrix': {
'rows': 3, 'cols': 3,
'data': [fx, 0.0, cx, 0.0, fy, cy, 0.0, 0.0, 1.0],
},
'distortion_model': 'equidistant',
'distortion_coefficients': {
'rows': 1, 'cols': 4,
'data': [float(D[0, 0]), float(D[1, 0]),
float(D[2, 0]), float(D[3, 0])],
},
'rectification_matrix': {
'rows': 3, 'cols': 3,
'data': [1.0, 0.0, 0.0,
0.0, 1.0, 0.0,
0.0, 0.0, 1.0],
},
'projection_matrix': {
'rows': 3, 'cols': 4,
'data': [fx, 0.0, cx, 0.0,
0.0, fy, cy, 0.0,
0.0, 0.0, 1.0, 0.0],
},
# Metadata (not part of camera_info_manager spec, but informational)
'_calibration_rms_px': round(rms, 4),
}
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
yaml.dump(data, f, default_flow_style=False, sort_keys=False)
print(f' Written: {output_path}')
def compute_reprojection_errors(
obj_pts: list, img_pts: list, rvecs: list, tvecs: list,
K: np.ndarray, D: np.ndarray,
) -> list:
"""Compute per-image reprojection errors for fisheye model."""
errors = []
for i, (op, ip) in enumerate(zip(obj_pts, img_pts)):
projected, _ = cv2.fisheye.projectPoints(op, rvecs[i], tvecs[i], K, D)
err = cv2.norm(ip, projected, cv2.NORM_L2) / np.sqrt(len(ip))
errors.append(err)
return errors

View File

@ -0,0 +1,139 @@
"""extrinsic_calibrator.py — Extrinsic calibration between 4 IMX219 cameras.
Method: overlapping stereo calibration using shared checkerboard views.
Each adjacent pair (cam0-cam1, cam1-cam2, cam2-cam3, cam3-cam0) is calibrated
via cv2.fisheye.stereoCalibrate to find R, T between them.
Then each camera's pose relative to base_link is computed by chaining transforms,
anchoring on the known mechanical mount geometry (4 cameras at 90° intervals,
radius 0.05m from centre, all at height 0.30m).
Output: extrinsics.yaml with camera-to-base transforms as translation + quaternion.
Also outputs static_transforms.launch.py.
"""
import cv2
import numpy as np
import yaml
import math
from pathlib import Path
def rotation_matrix_to_quaternion(R: np.ndarray) -> tuple:
"""Convert 3x3 rotation matrix to (qx, qy, qz, qw)."""
trace = R[0, 0] + R[1, 1] + R[2, 2]
if trace > 0:
s = 0.5 / math.sqrt(trace + 1.0)
w = 0.25 / s
x = (R[2, 1] - R[1, 2]) * s
y = (R[0, 2] - R[2, 0]) * s
z = (R[1, 0] - R[0, 1]) * s
elif R[0, 0] > R[1, 1] and R[0, 0] > R[2, 2]:
s = 2.0 * math.sqrt(1.0 + R[0, 0] - R[1, 1] - R[2, 2])
w = (R[2, 1] - R[1, 2]) / s
x = 0.25 * s
y = (R[0, 1] + R[1, 0]) / s
z = (R[0, 2] + R[2, 0]) / s
elif R[1, 1] > R[2, 2]:
s = 2.0 * math.sqrt(1.0 + R[1, 1] - R[0, 0] - R[2, 2])
w = (R[0, 2] - R[2, 0]) / s
x = (R[0, 1] + R[1, 0]) / s
y = 0.25 * s
z = (R[1, 2] + R[2, 1]) / s
else:
s = 2.0 * math.sqrt(1.0 + R[2, 2] - R[0, 0] - R[1, 1])
w = (R[1, 0] - R[0, 1]) / s
x = (R[0, 2] + R[2, 0]) / s
y = (R[1, 2] + R[2, 1]) / s
z = 0.25 * s
return (x, y, z, w)
def mechanical_extrinsics(
mount_radius_m: float = 0.05,
mount_height_m: float = 0.30,
cam_yaw_degs: list = None,
) -> dict:
"""
Compute camera-to-base_link transforms from known mount geometry.
Camera mount: ring of radius mount_radius_m at height mount_height_m.
Cameras face outward (yaw angle determines facing direction).
Camera optical axis is +Z (into scene), x-right, y-down.
base_link: x=forward, y=left, z=up (ROS convention).
Transform = cam_optical_frame -> base_link
Returns dict: {cam_name: {translation: [x,y,z], rotation: [qx,qy,qz,qw]}}
"""
if cam_yaw_degs is None:
cam_yaw_degs = [0.0, -90.0, 180.0, 90.0] # front, right, rear, left
cam_names = ['front', 'right', 'rear', 'left']
result = {}
for name, yaw_deg in zip(cam_names, cam_yaw_degs):
yaw_rad = math.radians(yaw_deg)
# Camera optical centre position relative to base_link
tx = mount_radius_m * math.cos(yaw_rad) # forward component
ty = -mount_radius_m * math.sin(yaw_rad) # left component (ROS y=left)
tz = mount_height_m # up
# Rotation: camera optical frame (+Z out, +X right, +Y down)
# -> base_link frame (+X forward, +Y left, +Z up)
# Camera Z (optical) = robot X rotated by yaw
# This is the transform from base_link to camera_optical_frame:
# Rz(yaw) * Rx(90 deg) [tilt camera to face out + coordinate swap]
# Rotation from camera_optical to base_link (inverse):
# camera_optical: z=forward, x=right, y=down
# base_link: x=forward, y=left, z=up
# R_cam_to_base_unrotated = [[0,-1,0],[0,0,-1],[1,0,0]] (optical->ROS)
# Then rotate by -yaw around z to get directional orientation
cos_y = math.cos(yaw_rad)
sin_y = math.sin(yaw_rad)
# Full rotation: R_z(-yaw) * R_cam_optical_to_base_unrotated
# R_cam_optical_to_base_unrotated maps: x_cam->-y_base, y_cam->-z_base, z_cam->x_base
# After yaw rotation:
R = np.array([
[cos_y, sin_y, 0],
[-sin_y, cos_y, 0],
[0, 0, 1],
]) @ np.array([
[0, 0, 1],
[-1, 0, 0],
[0, -1, 0],
])
qx, qy, qz, qw = rotation_matrix_to_quaternion(R)
result[name] = {
'translation': [round(tx, 4), round(ty, 4), round(tz, 4)],
'rotation_quat': [round(qx, 6), round(qy, 6),
round(qz, 6), round(qw, 6)],
}
return result
def write_extrinsics_yaml(output_path: str, extrinsics: dict,
mount_radius_m: float, mount_height_m: float) -> None:
"""Write extrinsics.yaml."""
data = {
'_description': 'Camera extrinsics — base_link to camera_optical_frame transforms',
'_mount_radius_m': mount_radius_m,
'_mount_height_m': mount_height_m,
'cameras': {},
}
for name, ex in extrinsics.items():
data['cameras'][name] = {
'parent_frame': 'base_link',
'child_frame': f'camera_{name}_optical_frame',
'translation': ex['translation'],
'rotation_quat_xyzw': ex['rotation_quat'],
}
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
yaml.dump(data, f, default_flow_style=False, sort_keys=False)
print(f'Written: {output_path}')

View File

@ -0,0 +1,215 @@
#!/usr/bin/env python3
"""
calibrate_camera.py Run OpenCV fisheye calibration from captured images.
Usage:
python3 calibrate_camera.py \\
--images-dir /path/to/cal/cam_front \\
--output /path/to/calibration/front/camera_info.yaml \\
--camera-name camera_front \\
--board-size 8x6 \\
--square-size 0.025 \\
--image-size 640x480 \\
[--show-reprojection]
"""
import argparse
import os
import sys
from pathlib import Path
import cv2
import numpy as np
def parse_size(s: str) -> tuple:
"""Parse 'WxH' string to (width, height) tuple of ints."""
parts = s.lower().split('x')
if len(parts) != 2:
raise argparse.ArgumentTypeError(f'Size must be WxH, got: {s}')
return (int(parts[0]), int(parts[1]))
def main():
parser = argparse.ArgumentParser(
description='Compute IMX219 fisheye camera intrinsics from checkerboard images.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument(
'--images-dir', required=True,
help='Directory containing captured calibration images (*.png, *.jpg).',
)
parser.add_argument(
'--output', required=True,
help='Output path for camera_info.yaml.',
)
parser.add_argument(
'--camera-name', required=True,
help='Camera name for YAML header (e.g. camera_front).',
)
parser.add_argument(
'--board-size', type=str, default='8x6',
help='Checkerboard inner corners as WxH (default: 8x6).',
)
parser.add_argument(
'--square-size', type=float, default=0.025,
help='Physical square size in metres (default: 0.025 = 25mm).',
)
parser.add_argument(
'--image-size', type=str, default='640x480',
help='Image dimensions as WxH (default: 640x480).',
)
parser.add_argument(
'--show-reprojection', action='store_true',
help='Display original vs reprojected corners for each image.',
)
args = parser.parse_args()
# Parse board and image sizes
bw, bh = parse_size(args.board_size)
board_size = (bw, bh)
image_size = parse_size(args.image_size)
# Collect image paths
images_dir = Path(args.images_dir)
if not images_dir.exists():
print(f'[ERROR] Images directory not found: {images_dir}')
sys.exit(1)
image_paths = sorted(
[str(p) for p in images_dir.glob('*.png')] +
[str(p) for p in images_dir.glob('*.jpg')] +
[str(p) for p in images_dir.glob('*.jpeg')]
)
if not image_paths:
print(f'[ERROR] No images found in: {images_dir}')
sys.exit(1)
print(f'Board size: {board_size[0]}x{board_size[1]} inner corners')
print(f'Square size: {args.square_size * 1000:.0f}mm')
print(f'Image size: {image_size[0]}x{image_size[1]}')
print(f'Images found: {len(image_paths)}')
print(f'Output: {args.output}')
print()
# Run calibration using camera_calibrator module
# Add package to path for standalone use
script_dir = Path(__file__).resolve().parent
pkg_dir = script_dir.parent
if str(pkg_dir) not in sys.path:
sys.path.insert(0, str(pkg_dir))
from saltybot_calibration.camera_calibrator import (
calibrate_fisheye,
write_camera_info_yaml,
compute_reprojection_errors,
)
print('Processing images...')
result = calibrate_fisheye(image_paths, board_size, args.square_size, image_size)
K = result['K']
D = result['D']
rms = result['rms']
rvecs = result['rvecs']
tvecs = result['tvecs']
obj_pts = result['obj_pts']
img_pts = result['img_pts']
n_used = result['n_images_used']
n_skipped = result['n_images_skipped']
# Per-image reprojection errors
errors = compute_reprojection_errors(obj_pts, img_pts, rvecs, tvecs, K, D)
# Summary table
print()
print(f'{"Image":<40} {"RMS error (px)":>14}')
print('-' * 57)
for path, err in zip(
[p for p in image_paths if Path(p).name not in
{Path(p).name for p in image_paths if not Path(p).exists()}],
errors
):
flag = ' <<' if err > 2.0 else ''
print(f' {Path(path).name:<38} {err:>14.4f}{flag}')
print('-' * 57)
print(f' {"Overall RMS":<38} {rms:>14.4f}')
print(f' {"Images used":<38} {n_used:>14}')
print(f' {"Images skipped":<38} {n_skipped:>14}')
print()
# Warn on high reprojection error
if rms > 1.5:
print(f'[WARN] RMS error {rms:.4f} px > 1.5 px threshold.')
print(' Suggestions:')
print(' - Remove images with corners at extreme edges (strong fisheye distortion)')
print(' - Ensure board is flat and rigidly printed (not bent)')
print(' - Increase n_images and vary board orientation more')
print(' - Check that --image-size matches actual capture resolution')
else:
print(f'[OK] RMS error {rms:.4f} px — good calibration.')
# Intrinsics summary
print()
print('Intrinsics:')
print(f' fx={K[0,0]:.2f} fy={K[1,1]:.2f} cx={K[0,2]:.2f} cy={K[1,2]:.2f}')
print(f' D=[{D[0,0]:.6f}, {D[1,0]:.6f}, {D[2,0]:.6f}, {D[3,0]:.6f}]')
print()
# Write YAML
write_camera_info_yaml(
output_path=args.output,
camera_name=args.camera_name,
image_size=image_size,
K=K,
D=D,
rms=rms,
)
# Show reprojection visualisation if requested
if args.show_reprojection:
print('Showing reprojection errors. Press any key to advance, q to quit.')
for i, (path, err) in enumerate(zip(
[p for p in image_paths],
errors + [0.0] * (len(image_paths) - len(errors))
)):
img = cv2.imread(path)
if img is None:
continue
# Project points back
projected, _ = cv2.fisheye.projectPoints(
obj_pts[i] if i < len(obj_pts) else None,
rvecs[i] if i < len(rvecs) else None,
tvecs[i] if i < len(tvecs) else None,
K, D
) if i < len(obj_pts) else (None, None)
vis = img.copy()
if projected is not None and i < len(img_pts):
# Draw original corners (green)
for pt in img_pts[i]:
cv2.circle(vis, tuple(pt[0].astype(int)), 5, (0, 255, 0), 2)
# Draw reprojected corners (red)
for pt in projected:
cv2.circle(vis, tuple(pt[0].astype(int)), 3, (0, 0, 255), -1)
cv2.putText(vis, f'{Path(path).name} RMS={err:.3f}px', (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 0), 2)
cv2.putText(vis, 'Green=detected Red=reprojected', (10, 60),
cv2.FONT_HERSHEY_SIMPLEX, 0.6, (200, 200, 200), 1)
cv2.imshow('Reprojection', vis)
key = cv2.waitKey(0) & 0xFF
if key == ord('q'):
break
cv2.destroyAllWindows()
print('Calibration complete.')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""
calibrate_extrinsics.py Compute camera-to-base_link extrinsic transforms.
Two modes:
1. --mode mechanical: use known mount geometry (radius, height, yaw angles)
-> produces analytically exact transforms from CAD measurements
2. --mode stereo: use cv2.fisheye.stereoCalibrate on paired checkerboard images
-> requires shared-view images from adjacent camera pairs
Usage (mechanical mode, recommended first pass):
python3 calibrate_extrinsics.py \\
--mode mechanical \\
--mount-radius 0.05 \\
--mount-height 0.30 \\
--output /path/to/calibration/extrinsics.yaml \\
--generate-launch /path/to/jetson/launch/static_transforms.launch.py
"""
import argparse
import math
import sys
from pathlib import Path
def print_extrinsics_table(extrinsics: dict) -> None:
"""Print formatted extrinsics table to stdout."""
print()
print(f'{"Camera":<8} {"tx":>8} {"ty":>8} {"tz":>8} '
f'{"qx":>10} {"qy":>10} {"qz":>10} {"qw":>10}')
print('-' * 86)
for name, ex in extrinsics.items():
tx, ty, tz = ex['translation']
qx, qy, qz, qw = ex['rotation_quat']
print(f' {name:<6} {tx:>8.4f} {ty:>8.4f} {tz:>8.4f} '
f'{qx:>10.6f} {qy:>10.6f} {qz:>10.6f} {qw:>10.6f}')
print()
def generate_static_transforms_launch(extrinsics: dict, output_path: str) -> None:
"""Generate static_transforms.launch.py from extrinsics dict."""
# Add package to path
script_dir = Path(__file__).resolve().parent
pkg_dir = script_dir.parent
if str(pkg_dir) not in sys.path:
sys.path.insert(0, str(pkg_dir))
from saltybot_calibration.extrinsic_calibrator import write_extrinsics_yaml
lines = [
'"""',
'static_transforms.launch.py — Auto-generated by calibrate_extrinsics.py',
'DO NOT EDIT manually — regenerate via:',
' python3 calibrate_extrinsics.py --mode mechanical --generate-launch <this_file>',
'"""',
'',
'from launch import LaunchDescription',
'from launch_ros.actions import Node',
'',
'',
'def generate_launch_description():',
' return LaunchDescription([',
]
for name, ex in extrinsics.items():
tx, ty, tz = ex['translation']
qx, qy, qz, qw = ex['rotation_quat']
child_frame = f'camera_{name}_optical_frame'
lines += [
f' # camera_{name}: base_link -> {child_frame}',
f' Node(',
f' package=\'tf2_ros\',',
f' executable=\'static_transform_publisher\',',
f' name=\'tf_base_to_camera_{name}\',',
f' arguments=[',
f' \'{tx}\', \'{ty}\', \'{tz}\',',
f' \'{qx}\', \'{qy}\', \'{qz}\', \'{qw}\',',
f' \'base_link\', \'{child_frame}\',',
f' ],',
f' ),',
]
lines += [
' ])',
'',
]
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
f.write('\n'.join(lines))
print(f'Generated launch file: {output_path}')
def run_mechanical_mode(args) -> None:
"""Run mechanical extrinsics calculation."""
script_dir = Path(__file__).resolve().parent
pkg_dir = script_dir.parent
if str(pkg_dir) not in sys.path:
sys.path.insert(0, str(pkg_dir))
from saltybot_calibration.extrinsic_calibrator import (
mechanical_extrinsics,
write_extrinsics_yaml,
)
print(f'Mode: mechanical')
print(f'Mount radius: {args.mount_radius * 100:.1f}cm')
print(f'Mount height: {args.mount_height * 100:.1f}cm')
print(f'Camera yaw angles: front=0 deg, right=-90 deg, rear=180 deg, left=90 deg')
print()
extrinsics = mechanical_extrinsics(
mount_radius_m=args.mount_radius,
mount_height_m=args.mount_height,
)
print_extrinsics_table(extrinsics)
write_extrinsics_yaml(
output_path=args.output,
extrinsics=extrinsics,
mount_radius_m=args.mount_radius,
mount_height_m=args.mount_height,
)
if args.generate_launch:
generate_static_transforms_launch(extrinsics, args.generate_launch)
def run_stereo_mode(args) -> None:
"""Stereo extrinsics calibration — stub for future implementation.
TODO: Implement cv2.fisheye.stereoCalibrate for adjacent camera pairs.
This mode requires:
- Shared-view checkerboard images from each adjacent camera pair:
{images_dir}/pair_front_right/*.png (visible to both cam0 and cam1)
{images_dir}/pair_right_rear/*.png
{images_dir}/pair_rear_left/*.png
{images_dir}/pair_left_front/*.png
- Per-camera intrinsics (camera_info.yaml) for each camera
- Each pair: cv2.fisheye.stereoCalibrate -> R, T between cameras
- Chain transforms: cam0->cam1->cam2->cam3, anchor to base_link via
mechanical position of cam0
Recommended workflow:
1. First run --mode mechanical to get approximate transforms
2. Mount robot in place, place checkerboard in overlap zones
3. Capture simultaneous images from adjacent pairs (requires hardware sync
or approximate temporal sync)
4. Run --mode stereo to refine R, T
5. Run generate_static_transforms.py to update launch file
For now, use --mode mechanical which gives correct transforms from CAD
dimensions (typically accurate to ~5mm / ~1 deg for a rigid mount).
"""
print('[ERROR] Stereo mode is not yet implemented.')
print()
print('Use --mode mechanical for now (accurate from CAD measurements).')
print()
print('Future implementation plan:')
print(' 1. Capture simultaneous checkerboard images from adjacent camera pairs')
print(' in their overlapping FoV regions (approx 70 deg overlap per pair)')
print(' 2. For each pair: cv2.fisheye.stereoCalibrate with per-camera intrinsics')
print(' 3. Chain transforms cam0->cam1->cam2->cam3, anchor to base_link')
print(' 4. Refine against mechanical prior using least-squares optimisation')
print()
print('Required directory structure for stereo mode:')
print(' {images_dir}/pair_front_right/ — images visible to both front+right')
print(' {images_dir}/pair_right_rear/')
print(' {images_dir}/pair_rear_left/')
print(' {images_dir}/pair_left_front/')
sys.exit(1)
def main():
parser = argparse.ArgumentParser(
description='Compute camera-to-base_link extrinsic transforms for 4x IMX219 cameras.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument(
'--mode', choices=['mechanical', 'stereo'], default='mechanical',
help='Calibration mode: mechanical (from CAD) or stereo (from images). '
'Default: mechanical.',
)
parser.add_argument(
'--mount-radius', type=float, default=0.05,
help='Camera mount ring radius in metres (default: 0.05).',
)
parser.add_argument(
'--mount-height', type=float, default=0.30,
help='Camera mount height above base_link in metres (default: 0.30).',
)
parser.add_argument(
'--output', required=True,
help='Output path for extrinsics.yaml.',
)
parser.add_argument(
'--generate-launch', type=str, default=None,
metavar='LAUNCH_FILE',
help='Also generate static_transforms.launch.py at this path.',
)
# Stereo mode arguments (for future use)
parser.add_argument(
'--images-dir', type=str, default=None,
help='[stereo mode] Directory containing paired calibration images.',
)
parser.add_argument(
'--intrinsics-dir', type=str, default=None,
help='[stereo mode] Directory containing per-camera camera_info.yaml files.',
)
args = parser.parse_args()
print('=== IMX219 Extrinsic Calibration ===')
print()
if args.mode == 'mechanical':
run_mechanical_mode(args)
elif args.mode == 'stereo':
run_stereo_mode(args)
print()
print('Extrinsics calibration complete.')
print(f'Output: {args.output}')
if args.generate_launch:
print(f'Launch: {args.generate_launch}')
print()
print('Next steps:')
print(' 1. Update saltybot_surround/config/surround_vision_params.yaml')
print(' with camera_info_url paths pointing to calibrated YAMLs')
print(' 2. Run generate_static_transforms.py (or use --generate-launch above)')
print(' 3. Restart saltybot_cameras and saltybot_surround nodes')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,316 @@
#!/usr/bin/env python3
"""
capture_calibration_images.py Capture checkerboard calibration images from IMX219 cameras.
Usage (standalone, no ROS2 required uses OpenCV VideoCapture for CSI):
python3 capture_calibration_images.py --camera front --n-images 30 --output-dir /tmp/cal
Usage (with ROS2 subscribes to image topic):
python3 capture_calibration_images.py --ros --camera front --n-images 30
Controls:
SPACE capture current frame (if checkerboard detected)
q quit / done
r reset captured images for this camera
Outputs: output_dir/cam_{name}/frame_{NNNN}.png
"""
import argparse
import os
import sys
from pathlib import Path
import cv2
import numpy as np
CAMERA_TOPICS = {
'front': '/camera/front/image_raw',
'right': '/camera/right/image_raw',
'rear': '/camera/rear/image_raw',
'left': '/camera/left/image_raw',
}
ALL_CAMERAS = ['front', 'right', 'rear', 'left']
def parse_board_size(s: str) -> tuple:
"""Parse 'WxH' string to (cols, rows) tuple of ints."""
parts = s.lower().split('x')
if len(parts) != 2:
raise argparse.ArgumentTypeError(f'board-size must be WxH, got: {s}')
return (int(parts[0]), int(parts[1]))
def find_checkerboard(img_bgr: np.ndarray, board_size: tuple):
"""Detect checkerboard corners. Returns (found, corners, gray)."""
gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
flags = cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE
found, corners = cv2.findChessboardCorners(gray, board_size, flags)
if found:
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
corners = cv2.cornerSubPix(gray, corners, (7, 7), (-1, -1), criteria)
return found, corners, gray
def draw_overlay(img: np.ndarray, board_size: tuple, found: bool,
corners, captured: int, n_images: int) -> np.ndarray:
"""Draw corner detections and status overlay onto a copy of img."""
display = img.copy()
if found and corners is not None:
cv2.drawChessboardCorners(display, board_size, corners, found)
# Status bar background
cv2.rectangle(display, (0, 0), (display.shape[1], 36), (0, 0, 0), -1)
status_color = (0, 255, 0) if found else (0, 80, 220)
status_text = 'BOARD DETECTED' if found else 'searching for board...'
cv2.putText(display, status_text, (10, 24),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, status_color, 2)
count_text = f'Captured: {captured}/{n_images} [SPACE=capture r=reset q=quit]'
cv2.putText(display, count_text, (display.shape[1] // 2 - 10, 24),
cv2.FONT_HERSHEY_SIMPLEX, 0.55, (200, 200, 200), 1)
if captured >= n_images:
cv2.rectangle(display, (0, 0), (display.shape[1], display.shape[0]),
(0, 200, 0), 8)
cv2.putText(display, 'DONE! Press q to finish.', (60, display.shape[0] // 2),
cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 255, 0), 3)
return display
def capture_loop_opencv(cam_name: str, device: str, board_size: tuple,
n_images: int, output_dir: Path) -> int:
"""Capture calibration images using cv2.VideoCapture (non-ROS mode).
Returns number of images captured.
"""
out_dir = output_dir / f'cam_{cam_name}'
out_dir.mkdir(parents=True, exist_ok=True)
cap = cv2.VideoCapture(device)
if not cap.isOpened():
print(f'[ERROR] Cannot open device: {device}')
return 0
# Try to set resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
captured = 0
frame_idx = 0
window_title = f'Calibration Capture — {cam_name} | SPACE=capture q=quit r=reset'
cv2.namedWindow(window_title, cv2.WINDOW_NORMAL)
print(f'\n[{cam_name.upper()}] Capturing {n_images} images. Output: {out_dir}')
print(' Point the checkerboard at the camera, press SPACE when board is detected.')
while captured < n_images:
ret, frame = cap.read()
if not ret:
print('[WARN] Failed to read frame')
continue
found, corners, _ = find_checkerboard(frame, board_size)
display = draw_overlay(frame, board_size, found, corners, captured, n_images)
cv2.imshow(window_title, display)
key = cv2.waitKey(1) & 0xFF
if key == ord('q'):
print(f' Quit requested. Captured {captured}/{n_images} images.')
break
elif key == ord('r'):
# Remove all saved images and reset counter
for f in out_dir.glob('frame_*.png'):
f.unlink()
captured = 0
frame_idx = 0
print(f' [RESET] Cleared captured images for {cam_name}.')
elif key == ord(' '):
if found:
save_path = out_dir / f'frame_{frame_idx:04d}.png'
cv2.imwrite(str(save_path), frame)
captured += 1
frame_idx += 1
print(f' Saved ({captured}/{n_images}): {save_path.name}')
# Brief flash to confirm capture
flash = frame.copy()
cv2.rectangle(flash, (0, 0), (flash.shape[1], flash.shape[0]),
(0, 255, 0), 12)
cv2.imshow(window_title, flash)
cv2.waitKey(120)
else:
print(' [SKIP] No checkerboard detected — move board into view.')
cap.release()
cv2.destroyWindow(window_title)
print(f' Done: {captured} images saved to {out_dir}')
return captured
def capture_loop_ros(cam_name: str, board_size: tuple,
n_images: int, output_dir: Path) -> int:
"""Capture calibration images via ROS2 topic subscription.
Returns number of images captured.
"""
try:
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
except ImportError as e:
print(f'[ERROR] ROS2 not available: {e}')
print(' Run without --ros for standalone OpenCV mode.')
return 0
out_dir = output_dir / f'cam_{cam_name}'
out_dir.mkdir(parents=True, exist_ok=True)
topic = CAMERA_TOPICS[cam_name]
print(f'\n[{cam_name.upper()}] Subscribing to: {topic}')
print(f' Capturing {n_images} images. Output: {out_dir}')
print(' Press SPACE in the preview window when the board is detected.')
rclpy.init()
node = Node('calibration_capture')
bridge = CvBridge()
captured = [0]
frame_idx = [0]
latest_frame = [None]
def image_callback(msg):
try:
latest_frame[0] = bridge.imgmsg_to_cv2(msg, 'bgr8')
except Exception as e:
node.get_logger().warn(f'cv_bridge error: {e}')
sub = node.create_subscription(Image, topic, image_callback, 1) # noqa: F841
window_title = f'Calibration Capture — {cam_name} (ROS2)'
cv2.namedWindow(window_title, cv2.WINDOW_NORMAL)
import threading
spin_thread = threading.Thread(target=rclpy.spin, args=(node,), daemon=True)
spin_thread.start()
try:
while captured[0] < n_images:
frame = latest_frame[0]
if frame is None:
cv2.waitKey(50)
continue
found, corners, _ = find_checkerboard(frame, board_size)
display = draw_overlay(frame, board_size, found, corners,
captured[0], n_images)
cv2.imshow(window_title, display)
key = cv2.waitKey(1) & 0xFF
if key == ord('q'):
print(f' Quit. Captured {captured[0]}/{n_images}.')
break
elif key == ord('r'):
for f in out_dir.glob('frame_*.png'):
f.unlink()
captured[0] = 0
frame_idx[0] = 0
print(f' [RESET] Cleared images for {cam_name}.')
elif key == ord(' '):
if found:
save_path = out_dir / f'frame_{frame_idx[0]:04d}.png'
cv2.imwrite(str(save_path), frame)
captured[0] += 1
frame_idx[0] += 1
print(f' Saved ({captured[0]}/{n_images}): {save_path.name}')
else:
print(' [SKIP] No board detected.')
finally:
cv2.destroyWindow(window_title)
node.destroy_node()
rclpy.shutdown()
print(f' Done: {captured[0]} images saved to {out_dir}')
return captured[0]
def main():
parser = argparse.ArgumentParser(
description='Capture checkerboard images for IMX219 fisheye calibration.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument(
'--camera', choices=['front', 'right', 'rear', 'left', 'all'],
default='front',
help='Which camera to capture from (default: front). Use "all" to iterate all 4.',
)
parser.add_argument(
'--n-images', type=int, default=30,
help='Number of images to capture per camera (default: 30).',
)
parser.add_argument(
'--output-dir', type=str, default='/tmp/saltybot_cal',
help='Root output directory. Images saved to {output_dir}/cam_{name}/ (default: /tmp/saltybot_cal).',
)
parser.add_argument(
'--board-size', type=str, default='8x6',
help='Checkerboard inner corners as WxH (default: 8x6).',
)
parser.add_argument(
'--square-size', type=float, default=0.025,
help='Physical square size in metres (default: 0.025 = 25mm).',
)
parser.add_argument(
'--ros', action='store_true',
help='Use ROS2 topic subscription instead of direct VideoCapture.',
)
parser.add_argument(
'--device', type=str, default='/dev/video0',
help='Video device (non-ROS mode only, default: /dev/video0).',
)
args = parser.parse_args()
board_size = parse_board_size(args.board_size)
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
cameras = ALL_CAMERAS if args.camera == 'all' else [args.camera]
print(f'Checkerboard: {board_size[0]}x{board_size[1]} inner corners, '
f'{args.square_size * 1000:.0f}mm squares')
print(f'Target: {args.n_images} images per camera')
print(f'Output: {output_dir}')
print(f'Mode: {"ROS2 topics" if args.ros else f"VideoCapture ({args.device})"}')
for i, cam_name in enumerate(cameras):
if len(cameras) > 1:
print(f'\n--- Camera {i + 1}/{len(cameras)}: {cam_name.upper()} ---')
if i > 0:
input(' Position the board in view of this camera, then press ENTER...')
if args.ros:
n_captured = capture_loop_ros(cam_name, board_size, args.n_images, output_dir)
else:
n_captured = capture_loop_opencv(cam_name, args.device, board_size,
args.n_images, output_dir)
if n_captured < 5:
print(f'[WARN] Only {n_captured} images captured for {cam_name}'
f'calibration may fail (need >= 5, recommend >= 20).')
print('\nCapture complete. Run calibrate_camera.py for each camera to compute intrinsics.')
print('Example:')
for cam in cameras:
print(f' python3 calibrate_camera.py \\')
print(f' --images-dir {output_dir}/cam_{cam} \\')
print(f' --output /mnt/nvme/saltybot/calibration/{cam}/camera_info.yaml \\')
print(f' --camera-name camera_{cam} --board-size {args.board_size} \\')
print(f' --square-size {args.square_size} --image-size 640x480')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""
generate_static_transforms.py Generate static_transforms.launch.py from extrinsics.yaml.
Usage:
python3 generate_static_transforms.py \\
--extrinsics /path/to/calibration/extrinsics.yaml \\
--output /path/to/jetson/ros2_ws/src/saltybot_bringup/launch/static_transforms.launch.py
"""
import argparse
import sys
from pathlib import Path
import yaml
def load_extrinsics(yaml_path: str) -> dict:
"""Load and parse extrinsics.yaml.
Returns the 'cameras' sub-dict with per-camera transforms.
"""
with open(yaml_path, 'r') as f:
data = yaml.safe_load(f)
if 'cameras' not in data:
raise ValueError(f'extrinsics.yaml missing "cameras" key: {yaml_path}')
return data['cameras'], data
def generate_launch_content(cameras: dict) -> str:
"""Generate static_transforms.launch.py content from cameras dict."""
lines = [
'"""',
'static_transforms.launch.py — Auto-generated by generate_static_transforms.py',
'DO NOT EDIT manually — regenerate with:',
' python3 generate_static_transforms.py \\',
' --extrinsics /path/to/calibration/extrinsics.yaml \\',
' --output <this_file>',
'"""',
'',
'from launch import LaunchDescription',
'from launch_ros.actions import Node',
'',
'',
'def generate_launch_description():',
' return LaunchDescription([',
]
for cam_name, cam_data in cameras.items():
parent = cam_data.get('parent_frame', 'base_link')
child = cam_data.get('child_frame', f'camera_{cam_name}_optical_frame')
tx, ty, tz = cam_data['translation']
qx, qy, qz, qw = cam_data['rotation_quat_xyzw']
lines += [
f' # {cam_name}: {parent} -> {child}',
f' Node(',
f' package=\'tf2_ros\',',
f' executable=\'static_transform_publisher\',',
f' name=\'tf_base_to_camera_{cam_name}\',',
f' arguments=[',
f' \'{tx}\', \'{ty}\', \'{tz}\',',
f' \'{qx}\', \'{qy}\', \'{qz}\', \'{qw}\',',
f' \'{parent}\', \'{child}\',',
f' ],',
f' ),',
]
# Also add RealSense D435i frame at origin (identity) — placeholder
# (actual D435i extrinsics from its own calibration)
lines += [
' # RealSense D435i: base_link -> camera_color_optical_frame',
' # NOTE: Replace with actual D435i extrinsics from stereo calibration',
' Node(',
' package=\'tf2_ros\',',
' executable=\'static_transform_publisher\',',
' name=\'tf_base_to_realsense\',',
' arguments=[',
' \'0.0\', \'0.0\', \'0.25\',',
' \'0.0\', \'0.0\', \'0.0\', \'1.0\',',
' \'base_link\', \'camera_color_optical_frame\',',
' ],',
' ),',
' ])',
'',
]
return '\n'.join(lines)
def main():
parser = argparse.ArgumentParser(
description='Generate static_transforms.launch.py from extrinsics.yaml.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument(
'--extrinsics', required=True,
help='Path to extrinsics.yaml (produced by calibrate_extrinsics.py).',
)
parser.add_argument(
'--output', required=True,
help='Output path for static_transforms.launch.py.',
)
args = parser.parse_args()
extrinsics_path = args.extrinsics
if not Path(extrinsics_path).exists():
print(f'[ERROR] extrinsics.yaml not found: {extrinsics_path}')
sys.exit(1)
print(f'Loading extrinsics: {extrinsics_path}')
cameras, full_data = load_extrinsics(extrinsics_path)
print(f'Found {len(cameras)} camera(s): {", ".join(cameras.keys())}')
content = generate_launch_content(cameras)
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
with open(output_path, 'w') as f:
f.write(content)
print(f'Written: {output_path}')
print()
print('Transforms:')
print(f' {"Camera":<8} {"Parent":<12} {"Child frame":<36} {"Translation"}')
print(' ' + '-' * 80)
for cam_name, cam_data in cameras.items():
parent = cam_data.get('parent_frame', 'base_link')
child = cam_data.get('child_frame', f'camera_{cam_name}_optical_frame')
tx, ty, tz = cam_data['translation']
print(f' {cam_name:<8} {parent:<12} {child:<36} '
f'[{tx:.3f}, {ty:.3f}, {tz:.3f}]')
print()
print('Next step: include static_transforms.launch.py in your main launch file:')
print(' from launch.actions import IncludeLaunchDescription')
print(' from launch.launch_description_sources import PythonLaunchDescriptionSource')
if __name__ == '__main__':
main()

View File

@ -0,0 +1,259 @@
#!/usr/bin/env python3
"""
preview_undistorted.py Preview undistorted IMX219 image using calibration YAML.
Usage (ROS2 mode):
python3 preview_undistorted.py \\
--camera front \\
--calibration /path/to/calibration/front/camera_info.yaml
Usage (image file mode):
python3 preview_undistorted.py \\
--image /path/to/image.png \\
--calibration /path/to/calibration/front/camera_info.yaml
"""
import argparse
import sys
from pathlib import Path
import cv2
import numpy as np
import yaml
CAMERA_TOPICS = {
'front': '/camera/front/image_raw',
'right': '/camera/right/image_raw',
'rear': '/camera/rear/image_raw',
'left': '/camera/left/image_raw',
}
def load_camera_info(yaml_path: str) -> dict:
"""Load and parse camera_info YAML.
Returns dict with keys: K, D, image_size, camera_name, distortion_model
"""
with open(yaml_path, 'r') as f:
data = yaml.safe_load(f)
w = data['image_width']
h = data['image_height']
K_data = data['camera_matrix']['data']
K = np.array(K_data, dtype=np.float64).reshape(3, 3)
D_data = data['distortion_coefficients']['data']
D = np.array(D_data, dtype=np.float64).reshape(-1, 1)
model = data.get('distortion_model', 'equidistant')
name = data.get('camera_name', 'unknown')
return {
'K': K,
'D': D,
'image_size': (w, h),
'camera_name': name,
'distortion_model': model,
}
def build_undistort_maps(K: np.ndarray, D: np.ndarray,
image_size: tuple, balance: float = 0.0) -> tuple:
"""Compute fisheye undistortion maps.
Args:
K: 3x3 camera matrix
D: (4,1) fisheye distortion coefficients
image_size: (width, height)
balance: 0.0 = no black borders (crop), 1.0 = keep all pixels (black borders)
Returns:
(map1, map2) suitable for cv2.remap
"""
new_K = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(
K, D, image_size, np.eye(3), balance=balance
)
map1, map2 = cv2.fisheye.initUndistortRectifyMap(
K, D, np.eye(3), new_K, image_size, cv2.CV_16SC2
)
return map1, map2, new_K
def undistort_image(img: np.ndarray, map1: np.ndarray, map2: np.ndarray) -> np.ndarray:
"""Apply precomputed undistortion maps to image."""
return cv2.remap(img, map1, map2, interpolation=cv2.INTER_LINEAR,
borderMode=cv2.BORDER_CONSTANT)
def show_side_by_side(original: np.ndarray, undistorted: np.ndarray,
title: str = '', rms_info: str = '') -> np.ndarray:
"""Create side-by-side comparison image."""
# Resize to same height if needed
h1, w1 = original.shape[:2]
h2, w2 = undistorted.shape[:2]
if h1 != h2:
undistorted = cv2.resize(undistorted, (w2 * h1 // h2, h1))
divider = np.zeros((h1, 4, 3), dtype=np.uint8)
divider[:] = (80, 80, 80)
combined = np.hstack([original, divider, undistorted])
# Labels
cv2.putText(combined, 'ORIGINAL (fisheye)', (10, 28),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 220, 0), 2)
cv2.putText(combined, 'UNDISTORTED', (w1 + 14, 28),
cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 220, 255), 2)
if title:
cv2.putText(combined, title, (10, combined.shape[0] - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (200, 200, 200), 1)
if rms_info:
cv2.putText(combined, rms_info, (w1 + 14, combined.shape[0] - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (200, 200, 200), 1)
return combined
def preview_image_file(image_path: str, cal_info: dict, balance: float) -> None:
"""Show undistorted preview of a single image file."""
img = cv2.imread(image_path)
if img is None:
print(f'[ERROR] Cannot read image: {image_path}')
sys.exit(1)
K = cal_info['K']
D = cal_info['D']
image_size = cal_info['image_size']
name = cal_info['camera_name']
print(f'Camera: {name}')
print(f'Image size: {image_size[0]}x{image_size[1]}')
print(f'fx={K[0,0]:.2f} fy={K[1,1]:.2f} cx={K[0,2]:.2f} cy={K[1,2]:.2f}')
print(f'D={D.flatten().tolist()}')
# Resize input if needed
ih, iw = img.shape[:2]
if (iw, ih) != image_size:
print(f'[WARN] Image size {iw}x{ih} != calibration size {image_size}. Resizing.')
img = cv2.resize(img, image_size)
map1, map2, new_K = build_undistort_maps(K, D, image_size, balance=balance)
undist = undistort_image(img, map1, map2)
print(f'New K after undistort (balance={balance}):')
print(f' fx={new_K[0,0]:.2f} fy={new_K[1,1]:.2f} cx={new_K[0,2]:.2f} cy={new_K[1,2]:.2f}')
combined = show_side_by_side(img, undist, title=str(Path(image_path).name))
cv2.imshow('Undistorted Preview (q=quit)', combined)
print('Press any key to close...')
cv2.waitKey(0)
cv2.destroyAllWindows()
def preview_ros_live(cam_name: str, cal_info: dict, balance: float) -> None:
"""Subscribe to ROS2 camera topic and show live undistorted preview."""
try:
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
except ImportError as e:
print(f'[ERROR] ROS2 not available: {e}')
sys.exit(1)
K = cal_info['K']
D = cal_info['D']
image_size = cal_info['image_size']
topic = CAMERA_TOPICS[cam_name]
print(f'Subscribing to: {topic}')
print(f'Image size: {image_size[0]}x{image_size[1]}')
print('Press q in preview window to quit.')
map1, map2, new_K = build_undistort_maps(K, D, image_size, balance=balance)
rclpy.init()
node = Node('preview_undistorted')
bridge = CvBridge()
latest = [None]
def cb(msg):
try:
latest[0] = bridge.imgmsg_to_cv2(msg, 'bgr8')
except Exception as e:
node.get_logger().warn(f'cv_bridge: {e}')
node.create_subscription(Image, topic, cb, 1)
import threading
threading.Thread(target=rclpy.spin, args=(node,), daemon=True).start()
window = f'Undistorted — {cam_name}'
cv2.namedWindow(window, cv2.WINDOW_NORMAL)
try:
while True:
frame = latest[0]
if frame is not None:
fh, fw = frame.shape[:2]
if (fw, fh) != image_size:
frame = cv2.resize(frame, image_size)
undist = undistort_image(frame, map1, map2)
combined = show_side_by_side(frame, undist)
cv2.imshow(window, combined)
key = cv2.waitKey(1) & 0xFF
if key == ord('q'):
break
finally:
cv2.destroyAllWindows()
node.destroy_node()
rclpy.shutdown()
def main():
parser = argparse.ArgumentParser(
description='Preview undistorted IMX219 fisheye image from calibration YAML.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
mode_group = parser.add_mutually_exclusive_group(required=True)
mode_group.add_argument(
'--camera', choices=['front', 'right', 'rear', 'left'],
help='Camera name for ROS2 live preview mode.',
)
mode_group.add_argument(
'--image', type=str,
help='Path to a single image file for static preview.',
)
parser.add_argument(
'--calibration', required=True,
help='Path to camera_info.yaml produced by calibrate_camera.py.',
)
parser.add_argument(
'--balance', type=float, default=0.0,
help='Undistort balance: 0.0=crop to valid pixels, 1.0=keep all (black borders). '
'Default: 0.0.',
)
args = parser.parse_args()
cal_path = args.calibration
if not Path(cal_path).exists():
print(f'[ERROR] Calibration file not found: {cal_path}')
sys.exit(1)
print(f'Loading calibration: {cal_path}')
cal_info = load_camera_info(cal_path)
if cal_info['distortion_model'] not in ('equidistant', 'fisheye'):
print(f'[WARN] Distortion model "{cal_info["distortion_model"]}"'
f'expected "equidistant". Results may be incorrect.')
if args.image:
preview_image_file(args.image, cal_info, args.balance)
else:
preview_ros_live(args.camera, cal_info, args.balance)
if __name__ == '__main__':
main()