feat(docker): per-profile s6 supervision + container-restart reconciliation

Phase 4 of the s6-overlay supervision plan. Activates the Phase 3
S6ServiceManager by hooking it into the profile lifecycle and the
`hermes gateway start/stop/restart` dispatcher, and adds a cont-
init.d-time reconciliation pass that survives `docker restart`.

Task 4.0 — container-boot reconciliation:
  /run/service/ is tmpfs, so every `docker restart` wipes every
  per-profile gateway slot. /etc/cont-init.d/02-reconcile-profiles
  invokes hermes_cli.container_boot.reconcile_profile_gateways() on
  every boot, which walks $HERMES_HOME/profiles/<name>/, reads each
  gateway_state.json, recreates the s6 service slot, and auto-starts
  only those whose last state was 'running'. Other states
  (stopped, starting, startup_failed, missing) register the slot
  in the down state — avoiding crash-loops across restarts for a
  gateway that was broken last boot. Per-profile outcome is recorded
  to $HERMES_HOME/logs/container-boot.log.

  Implementation: hermes_cli/container_boot.py + 12 unit tests.
  Profile-marker is SOUL.md, not config.yaml, because `hermes profile
  create` only seeds SOUL.md by default (config.yaml comes from
  `hermes setup`).

Task 4.1 / 4.2 — profile create/delete hooks:
  hermes_cli/profiles.py::create_profile now calls
  _maybe_register_gateway_service(<canon>) at the end, which routes
  through ServiceManager.register_profile_gateway when running on s6
  and no-ops on host backends. delete_profile mirrors with
  _maybe_unregister_gateway_service. _allocate_gateway_port produces
  a deterministic SHA-256-derived port in [9200, 9800).

Task 4.3 — gateway dispatch + remove rejection arms:
  _dispatch_via_service_manager_if_s6(action) intercepts
  start/stop/restart at the top of each subcommand and routes them
  through S6ServiceManager.{start,stop,restart}. The pre-Phase-4
  `elif is_container():` rejection arms are kept as fallback for
  pre-s6 containers / unsupported runtimes, but only ever fire when
  detect_service_manager() != 's6'. install/uninstall under s6
  print informational guidance pointing users at profile create/delete.

  Removed the two xfail(strict=True) markers from
  tests/docker/test_profile_gateway.py — both tests now pass strictly.

Task 4.4 — status reporting:
  get_gateway_runtime_snapshot() reports
  Manager: 's6 (container supervisor)' inside an s6 container instead
  of 'docker (foreground)'.

Plan-vs-reality drift fixed in this commit:
  - Plan's S6ServiceManager._render_run_script used
    `gateway start --foreground --port {port}` — invented args; the
    real CLI is `gateway run`. Switched accordingly. port arg
    retained for API parity but now documented as 'currently ignored'.
  - Plan's reconciler keyed on config.yaml; switched to SOUL.md
    (config.yaml is created by hermes setup, not by hermes profile
    create, so the original gate caught nothing).
  - The plan's _dispatch helper used _profile_arg() which returns
    '--profile <name>' (i.e. with the flag prefix). Switched to
    _profile_suffix() which returns the bare name.
  - Architecture B's docker exec doesn't get /command on PATH or
    the venv on PATH; Dockerfile's runtime PATH now includes
    /opt/hermes/.venv/bin so 'docker exec <c> hermes ...' works
    without sourcing the venv.
  - stage2-hook now chowns $HERMES_HOME/profiles to hermes on every
    boot, not just on the UID-remap path. Without this, files created
    by docker-exec-as-root accumulate and the next reconciler run
    fails with PermissionError reading SOUL.md.

Test harness:
  19 passed, 0 xfailed (the two pre-Phase-4 xfail targets flip to
  passing). 78 unit tests across service_manager + container_boot +
  profiles_s6_hooks + gateway_s6_dispatch. Hadolint + shellcheck
  pass cleanly.

Refs: docs/plans/2026-05-07-s6-overlay-dynamic-subagent-gateways.md
This commit is contained in:
Ben
2026-05-24 18:05:33 -07:00
committed by teknium1
parent 0abf661f71
commit 2afefc501c
13 changed files with 1217 additions and 39 deletions
+235
View File
@@ -0,0 +1,235 @@
"""Tests for hermes_cli.container_boot — the cont-init.d-time
reconciliation that recreates per-profile gateway s6 service slots
from the persistent profiles directory.
These tests run against a fake $HERMES_HOME under tmp_path; no real
s6 supervision tree is required. The in-container integration test
covering end-to-end "docker restart" survival lives in
tests/docker/test_container_restart.py.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from hermes_cli.container_boot import (
ReconcileAction,
reconcile_profile_gateways,
)
# ---------------------------------------------------------------------------
# Fixtures + helpers
# ---------------------------------------------------------------------------
def _make_profile(
hermes_home: Path,
name: str,
*,
state: str | None,
with_pid: bool = False,
config: bool = True,
) -> Path:
"""Create a fake profile directory under hermes_home/profiles/<name>/."""
p = hermes_home / "profiles" / name
p.mkdir(parents=True)
if config:
# SOUL.md is what the reconciler keys on — it's always seeded by
# `hermes profile create`. See container_boot._render_run_script.
(p / "SOUL.md").write_text("# fake profile\n")
if state is not None:
(p / "gateway_state.json").write_text(json.dumps({
"gateway_state": state, "timestamp": 1234567890,
}))
if with_pid:
(p / "gateway.pid").write_text(json.dumps(
{"pid": 99999, "host": "old-container"},
))
(p / "processes.json").write_text("[]")
return p
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
def test_running_profile_is_registered_and_autostarted(tmp_path: Path) -> None:
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "coder", state="running")
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions == [ReconcileAction(
profile="coder", prior_state="running", action="started",
)]
svc = scandir / "gateway-coder"
assert (svc / "run").exists()
assert (svc / "run").stat().st_mode & 0o111 # executable
assert (svc / "type").read_text().strip() == "longrun"
# Auto-start means no down-marker.
assert not (svc / "down").exists()
def test_stopped_profile_is_registered_but_not_started(tmp_path: Path) -> None:
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "writer", state="stopped")
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions == [ReconcileAction(
profile="writer", prior_state="stopped", action="registered",
)]
# down marker tells s6-svscan to NOT start the service.
assert (scandir / "gateway-writer" / "down").exists()
def test_startup_failed_does_not_autostart(tmp_path: Path) -> None:
"""Avoid crash-loop on restart when the gateway was failing to boot."""
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "broken", state="startup_failed")
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions[0].action == "registered"
assert (scandir / "gateway-broken" / "down").exists()
def test_starting_state_does_not_autostart(tmp_path: Path) -> None:
"""`starting` means the gateway died mid-boot last time; treat as
failed, not as a candidate for auto-restart."""
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "unlucky", state="starting")
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions[0].action == "registered"
def test_stale_runtime_files_are_removed(tmp_path: Path) -> None:
scandir = tmp_path / "run-service"; scandir.mkdir()
profile = _make_profile(tmp_path, "coder", state="running", with_pid=True)
assert (profile / "gateway.pid").exists()
assert (profile / "processes.json").exists()
reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert not (profile / "gateway.pid").exists()
assert not (profile / "processes.json").exists()
def test_profile_without_state_file_is_registered_but_not_started(
tmp_path: Path,
) -> None:
"""A freshly-created profile that's never been started: register
its slot but don't auto-start."""
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "fresh", state=None)
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions == [ReconcileAction(
profile="fresh", prior_state=None, action="registered",
)]
assert (scandir / "gateway-fresh" / "down").exists()
def test_directory_without_marker_file_is_skipped(tmp_path: Path) -> None:
"""A stray dir under profiles/ that isn't actually a profile (no
SOUL.md — the marker the reconciler keys on) should be skipped."""
scandir = tmp_path / "run-service"; scandir.mkdir()
# Create a profile dir but without SOUL.md
(tmp_path / "profiles" / "stray").mkdir(parents=True)
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions == []
assert not (scandir / "gateway-stray").exists()
def test_corrupt_state_file_treated_as_no_prior_state(tmp_path: Path) -> None:
"""If gateway_state.json is malformed JSON, don't blow up the whole
reconciliation — register the slot in the down state."""
scandir = tmp_path / "run-service"; scandir.mkdir()
profile = _make_profile(tmp_path, "junk", state="running")
(profile / "gateway_state.json").write_text("{ not valid json")
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions[0].action == "registered" # not "started"
assert (scandir / "gateway-junk" / "down").exists()
def test_reconcile_log_is_written(tmp_path: Path) -> None:
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "a", state="running")
_make_profile(tmp_path, "b", state="stopped")
reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
log = (tmp_path / "logs" / "container-boot.log").read_text()
assert "profile=a" in log
assert "action=started" in log
assert "profile=b" in log
assert "action=registered" in log
def test_dry_run_makes_no_filesystem_changes(tmp_path: Path) -> None:
scandir = tmp_path / "run-service"; scandir.mkdir()
profile = _make_profile(tmp_path, "coder", state="running", with_pid=True)
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=True,
)
# The action list is still produced...
assert actions == [ReconcileAction(
profile="coder", prior_state="running", action="started",
)]
# ...but nothing on disk was touched.
assert (profile / "gateway.pid").exists() # not removed under dry_run
assert not (scandir / "gateway-coder").exists()
assert not (tmp_path / "logs" / "container-boot.log").exists()
def test_missing_profiles_root_returns_empty(tmp_path: Path) -> None:
"""When $HERMES_HOME/profiles doesn't exist (fresh install), the
reconciliation should return an empty list without raising."""
scandir = tmp_path / "run-service"; scandir.mkdir()
actions = reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
assert actions == []
def test_invalid_profile_name_in_directory_raises(tmp_path: Path) -> None:
"""A profile dir whose name doesn't match validate_profile_name's
rules (uppercase, etc.) must surface as a hard error rather than
silently produce an invalid s6 service dir."""
scandir = tmp_path / "run-service"; scandir.mkdir()
_make_profile(tmp_path, "BadName", state="running")
with pytest.raises(ValueError):
reconcile_profile_gateways(
hermes_home=tmp_path, scandir=scandir, dry_run=False,
)
@@ -0,0 +1,117 @@
"""Tests for the Phase 4 s6 dispatch helper in hermes_cli.gateway.
`_dispatch_via_service_manager_if_s6` decides whether a
`hermes gateway start/stop/restart` invocation should be routed to
the in-container S6ServiceManager instead of falling through to the
host systemd/launchd/windows code path.
"""
from __future__ import annotations
from typing import Any
import pytest
class _CallRecorder:
"""Minimal stand-in for S6ServiceManager."""
kind = "s6"
def __init__(self) -> None:
self.calls: list[tuple[str, str]] = []
def start(self, name: str) -> None:
self.calls.append(("start", name))
def stop(self, name: str) -> None:
self.calls.append(("stop", name))
def restart(self, name: str) -> None:
self.calls.append(("restart", name))
def test_dispatch_returns_false_on_host(monkeypatch: pytest.MonkeyPatch) -> None:
"""When the environment isn't s6 (host run), the helper must
return False and not invoke a manager — callers continue with
their existing systemd/launchd/windows path."""
from hermes_cli import gateway as gw
monkeypatch.setattr(
"hermes_cli.service_manager.detect_service_manager", lambda: "systemd",
)
# Should not even attempt to construct a manager.
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager",
lambda: pytest.fail("manager should not be constructed on host"),
)
assert gw._dispatch_via_service_manager_if_s6("start", profile="x") is False
def test_dispatch_returns_true_and_calls_start_on_s6(
monkeypatch: pytest.MonkeyPatch,
) -> None:
from hermes_cli import gateway as gw
rec = _CallRecorder()
monkeypatch.setattr(
"hermes_cli.service_manager.detect_service_manager", lambda: "s6",
)
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: rec,
)
assert gw._dispatch_via_service_manager_if_s6("start", profile="coder") is True
assert rec.calls == [("start", "gateway-coder")]
@pytest.mark.parametrize("action,expected", [
("start", "start"),
("stop", "stop"),
("restart", "restart"),
])
def test_dispatch_translates_action_to_manager_method(
monkeypatch: pytest.MonkeyPatch, action: str, expected: str,
) -> None:
from hermes_cli import gateway as gw
rec = _CallRecorder()
monkeypatch.setattr(
"hermes_cli.service_manager.detect_service_manager", lambda: "s6",
)
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: rec,
)
assert gw._dispatch_via_service_manager_if_s6(action, profile="x") is True
assert rec.calls == [(expected, "gateway-x")]
def test_dispatch_unknown_action_returns_false(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An unrecognized action (e.g. 'install') must not silently
succeed — return False so the host code path handles it."""
from hermes_cli import gateway as gw
rec = _CallRecorder()
monkeypatch.setattr(
"hermes_cli.service_manager.detect_service_manager", lambda: "s6",
)
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: rec,
)
assert gw._dispatch_via_service_manager_if_s6("install", profile="x") is False
assert rec.calls == []
def test_dispatch_defaults_profile_to_default(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When profile is None, the helper resolves it via _profile_arg().
With no profile context set anywhere, that resolves to "default"."""
from hermes_cli import gateway as gw
rec = _CallRecorder()
monkeypatch.setattr(
"hermes_cli.service_manager.detect_service_manager", lambda: "s6",
)
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: rec,
)
monkeypatch.setattr(
"hermes_cli.gateway._profile_suffix", lambda: "",
)
assert gw._dispatch_via_service_manager_if_s6("start") is True
assert rec.calls == [("start", "gateway-default")]
+190
View File
@@ -0,0 +1,190 @@
"""Tests for the Phase 4 s6 hooks in hermes_cli.profiles.
Specifically: _allocate_gateway_port, _maybe_register_gateway_service,
_maybe_unregister_gateway_service. The integration with
create_profile and delete_profile is covered indirectly by the
existing TestCreateProfile and TestDeleteProfile classes in
tests/hermes_cli/test_profiles.py; here we only exercise the new
helper surface that doesn't touch the filesystem.
"""
from __future__ import annotations
from typing import Any
import pytest
from hermes_cli.profiles import (
_allocate_gateway_port,
_maybe_register_gateway_service,
_maybe_unregister_gateway_service,
)
# ---------------------------------------------------------------------------
# _allocate_gateway_port
# ---------------------------------------------------------------------------
def test_allocate_gateway_port_is_deterministic() -> None:
"""Same profile name → same port across calls. This matters because
a profile's gateway must come back up on the same port across
container restarts."""
a = _allocate_gateway_port("coder")
b = _allocate_gateway_port("coder")
assert a == b
def test_allocate_gateway_port_in_advertised_range() -> None:
"""[9200, 9800) — the window the helper's docstring promises."""
for name in ("a", "b", "coder", "assistant", "very-long-profile-name-here"):
port = _allocate_gateway_port(name)
assert 9200 <= port < 9800, f"{name} got {port}"
def test_allocate_gateway_port_distributes_across_range() -> None:
"""Sanity check: ports for ~100 random-ish names should land in
enough distinct buckets that the distribution is plausibly uniform.
Catches accidental hash truncation that would collapse the range."""
ports = {_allocate_gateway_port(f"profile-{i}") for i in range(100)}
# 100 inputs mapped into 600 slots — expect at least ~60 distinct.
assert len(ports) >= 60, f"Only {len(ports)} distinct ports across 100 names"
# ---------------------------------------------------------------------------
# _maybe_register_gateway_service / _maybe_unregister_gateway_service
# ---------------------------------------------------------------------------
class _HostManager:
"""Mimics a host backend that doesn't support runtime registration."""
kind = "systemd"
def supports_runtime_registration(self) -> bool:
return False
def register_profile_gateway(self, *args: Any, **kwargs: Any) -> None:
raise AssertionError("host backend register_profile_gateway should not be called")
def unregister_profile_gateway(self, *args: Any, **kwargs: Any) -> None:
raise AssertionError("host backend unregister_profile_gateway should not be called")
class _S6Manager:
"""Mimics S6ServiceManager just enough for the hooks."""
kind = "s6"
def __init__(self) -> None:
self.registered: list[tuple[str, int]] = []
self.unregistered: list[str] = []
self.raise_on_register: Exception | None = None
self.raise_on_unregister: Exception | None = None
def supports_runtime_registration(self) -> bool:
return True
def register_profile_gateway(
self, profile: str, *, port: int,
extra_env: dict[str, str] | None = None,
) -> None:
if self.raise_on_register is not None:
raise self.raise_on_register
self.registered.append((profile, port))
def unregister_profile_gateway(self, profile: str) -> None:
if self.raise_on_unregister is not None:
raise self.raise_on_unregister
self.unregistered.append(profile)
def test_register_noop_on_host(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager",
lambda: _HostManager(),
)
# Should NOT raise the AssertionError from _HostManager.register
_maybe_register_gateway_service("hostprof")
def test_register_calls_through_on_s6(monkeypatch: pytest.MonkeyPatch) -> None:
mgr = _S6Manager()
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: mgr,
)
_maybe_register_gateway_service("coder")
assert len(mgr.registered) == 1
profile, port = mgr.registered[0]
assert profile == "coder"
assert 9200 <= port < 9800
def test_register_swallows_duplicate_value_error(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A pre-existing s6 registration (from container-boot reconcile)
is a benign condition — register must not propagate ValueError."""
mgr = _S6Manager()
mgr.raise_on_register = ValueError("already registered")
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: mgr,
)
# Should NOT raise
_maybe_register_gateway_service("coder")
def test_register_swallows_arbitrary_error(
monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str],
) -> None:
"""Even an unexpected exception from the manager must not bring
down `hermes profile create` — print and continue."""
mgr = _S6Manager()
mgr.raise_on_register = RuntimeError("svscanctl exploded")
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: mgr,
)
_maybe_register_gateway_service("coder")
captured = capsys.readouterr()
assert "Could not register" in captured.out
def test_register_swallows_no_backend_runtime_error(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When `get_service_manager()` raises RuntimeError (no backend
detected), the hook must silently no-op."""
def _no_backend() -> None:
raise RuntimeError("no supported service manager detected")
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", _no_backend,
)
# Should NOT raise
_maybe_register_gateway_service("anywhere")
def test_unregister_noop_on_host(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager",
lambda: _HostManager(),
)
_maybe_unregister_gateway_service("hostprof")
def test_unregister_calls_through_on_s6(monkeypatch: pytest.MonkeyPatch) -> None:
mgr = _S6Manager()
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: mgr,
)
_maybe_unregister_gateway_service("coder")
assert mgr.unregistered == ["coder"]
def test_unregister_swallows_errors(
monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str],
) -> None:
mgr = _S6Manager()
mgr.raise_on_unregister = RuntimeError("svc gone weird")
monkeypatch.setattr(
"hermes_cli.service_manager.get_service_manager", lambda: mgr,
)
_maybe_unregister_gateway_service("coder")
captured = capsys.readouterr()
assert "Could not unregister" in captured.out
+1 -2
View File
@@ -332,8 +332,7 @@ def test_s6_register_creates_service_dir_and_triggers_scan(
assert run_path.is_file()
assert run_path.stat().st_mode & 0o111 # executable
run_text = run_path.read_text()
assert "hermes -p coder gateway start" in run_text
assert "--port 9150" in run_text
assert "hermes -p coder gateway run" in run_text
assert "s6-setuidgid hermes" in run_text
log_run = svc_dir / "log" / "run"