feat(docker): auto-redirect gateway run to supervised mode inside s6 image

Pre-s6, `docker run nousresearch/hermes-agent gateway run` was the
standard invocation: gateway ran as the container's main process,
tini reaped zombies, container exit code matched gateway exit code,
no supervision. With s6-overlay as PID 1, the same invocation now
auto-upgrades to supervised semantics — auto-restart on crash,
dashboard supervised alongside (when HERMES_DASHBOARD=1 is set),
multiple profile gateways under the same /init.

Users get the new behavior with zero changes to their docker run
command. A loud one-line breadcrumb on stderr explains the upgrade
and points at the opt-out for users who genuinely want pre-s6
foreground semantics.

How it works:

  1. `_gateway_command_inner` (the `gateway run` handler) checks if
     we're inside a container with s6 as PID 1.
  2. If yes, dispatches `start` to the s6 service manager (registers
     and starts gateway-default), then `exec sleep infinity` to keep
     the CMD process alive without binding container lifetime to
     gateway PID lifetime. The supervised gateway can flap freely;
     `docker stop` still tears everything down via /init stage 3.
  3. If no, falls through to the existing foreground code path
     unchanged. Host runs of `hermes gateway run` are unaffected.

Three gates make the redirect inert outside the intended scope:

  * `detect_service_manager() != "s6"` — host/non-s6-container runs.
  * `HERMES_S6_SUPERVISED_CHILD=1` env var (recursion guard) —
    exported by `S6ServiceManager._render_run_script` for the
    s6-supervised invocation itself. Without this guard, the
    supervised `gateway run --replace` would re-enter the redirect
    and recurse (run → start → run → start → ...) infinitely.
  * `--no-supervise` CLI flag OR `HERMES_GATEWAY_NO_SUPERVISE=1` env
    var — explicit user opt-out for CI smoke tests, debugging the
    foreground startup path, or any case wanting "CMD exit =
    container exit" semantics. Strict truthiness (1/true/yes,
    case-insensitive); typos like `=0` do NOT silently opt out.

Tests:

  * Unit tests in tests/hermes_cli/test_gateway_s6_dispatch.py
    cover all five paths (host no-op, supervised fire, sentinel
    recursion guard, CLI flag, env var truthy + falsy). The two
    load-bearing gates (sentinel + opt-out) were mutation-tested
    by removing each gate in isolation and confirming the dedicated
    test fails with the expected error.
  * Docker harness tests in tests/docker/test_gateway_run_supervised.py
    cover the round trips end-to-end against a built image: redirect
    fires (sleep-infinity heartbeat + supervised gateway-default
    slot + breadcrumb), --no-supervise opt-out (foreground gateway,
    no want-up on the slot), HERMES_GATEWAY_NO_SUPERVISE env var
    works identically, recursion is impossible (≤1 supervised
    python gateway-run + exactly 1 sleep-infinity parented to the
    CMD wrapper), and HERMES_DASHBOARD=1 produces both supervised
    gateway and supervised dashboard.

Docs:

  * Added a `:::tip Gateway runs supervised` admonition near the
    main docker.md example explaining the upgrade and pointing at
    the opt-out. Pre-s6 (tini-based) images still run gateway run
    as the foreground main process, so the note is scoped to the
    s6 image only.

Trade-off documented in the helper docstring: container exit code
under the redirect is sleep's exit code (always 0 on SIGTERM), not
the gateway's. That was an explicit design call — the supervised
gateway is allowed to flap without taking the container with it,
which is what "supervision" means. CI users who want exit-code
forwarding can pass --no-supervise.
This commit is contained in:
Ben Barclay
2026-05-28 12:42:13 +10:00
committed by Ben Barclay
parent 36c99af37a
commit 0927fb5584
7 changed files with 626 additions and 0 deletions
+72
View File
@@ -5150,11 +5150,83 @@ def gateway_command(args):
sys.exit(1)
def _maybe_redirect_run_to_s6_supervision(args) -> bool:
"""Inside an s6 container, redirect bare ``gateway run`` to the
supervised path.
Background. Before the s6 image landed, ``docker run <image> gateway
run`` was the standard way to start a containerized gateway: the
gateway was the container's main process, tini reaped zombies, and
container exit code == gateway exit code. With s6-overlay as PID 1,
we'd much rather have the gateway run as a supervised s6 longrun
(auto-restart on crash, dashboard supervised alongside, multiple
profile gateways under the same /init). This redirect upgrades the
old invocation transparently — the user gets the new behavior
without changing their docker run command.
Three gates make this a no-op outside the intended scope:
1. ``_dispatch_via_service_manager_if_s6`` returns False unless
we're in a container with s6 as PID 1. Host runs of
``hermes gateway run`` are unaffected.
2. ``HERMES_S6_SUPERVISED_CHILD`` is exported by
``S6ServiceManager._render_run_script`` for the supervised
process itself — i.e. when s6-supervise execs ``hermes gateway
run --replace`` as a longrun, this guard short-circuits the
redirect so the supervised gateway actually runs in
foreground (otherwise we'd recurse: run → start → run → start
→ ...).
3. ``--no-supervise`` (or ``HERMES_GATEWAY_NO_SUPERVISE=1``) opts
out for users who genuinely want pre-s6 semantics — CI smoke
tests, debugging the foreground startup path, etc.
Returns True iff dispatched (caller should ``return``).
"""
no_supervise = getattr(args, "no_supervise", False) or \
os.environ.get("HERMES_GATEWAY_NO_SUPERVISE", "").lower() in ("1", "true", "yes")
if no_supervise:
return False
if os.environ.get("HERMES_S6_SUPERVISED_CHILD"):
# We ARE the supervised child s6-supervise is running. Fall
# through to the foreground code path so the gateway actually
# starts.
return False
if not _dispatch_via_service_manager_if_s6("start"):
return False
# Loud breadcrumb: explain the upgrade and how to opt out. Print to
# stderr so it doesn't pollute stdout-parsing scripts. The
# supervised gateway's own logs are routed by s6-log to both
# `docker logs` and ${HERMES_HOME}/logs/gateways/<profile>/current,
# so the user sees a clear sequence: this banner first, then the
# gateway's own stdout/stderr from the supervisor.
print(
"→ gateway is now running under s6 supervision (auto-restart on crash,\n"
" dashboard supervised alongside if HERMES_DASHBOARD is set).\n"
" This is the recommended setup for the s6 container image — the\n"
" gateway will keep running even if it crashes.\n"
" Use `--no-supervise` (or HERMES_GATEWAY_NO_SUPERVISE=1) to opt out\n"
" and get the pre-s6 foreground behavior instead.",
file=sys.stderr,
flush=True,
)
# Block until the container is signalled. The supervised gateway's
# lifetime is independent of this process — s6-supervise restarts
# it on crash, and we don't want the container to exit when the
# gateway flaps. `sleep infinity` matches the static main-hermes
# service's pattern (see docker/s6-rc.d/main-hermes/run): the CMD
# process is a no-op heartbeat that keeps /init alive until
# `docker stop` sends SIGTERM, at which point /init runs stage 3
# shutdown (which tears down the supervised gateway cleanly).
os.execvp("sleep", ["sleep", "infinity"])
def _gateway_command_inner(args):
subcmd = getattr(args, 'gateway_command', None)
# Default to run if no subcommand
if subcmd is None or subcmd == "run":
if _maybe_redirect_run_to_s6_supervision(args):
return # unreachable; execvp doesn't return
verbose = getattr(args, 'verbose', 0)
quiet = getattr(args, 'quiet', False)
replace = getattr(args, 'replace', False)
+13
View File
@@ -11326,6 +11326,19 @@ def main():
action="store_true",
help="Replace any existing gateway instance (useful for systemd)",
)
gateway_run.add_argument(
"--no-supervise",
action="store_true",
help=(
"Inside the s6-overlay Docker image, normally `gateway run` is "
"automatically redirected to the supervised s6 service (so the "
"gateway gets auto-restart on crash, plus a supervised dashboard "
"if HERMES_DASHBOARD is set). Pass --no-supervise to opt out and "
"get the historical pre-s6 foreground behavior: the gateway is "
"the container's main process and the container exits with the "
"gateway's exit code. No effect outside an s6 container."
),
)
_add_accept_hooks_flag(gateway_run)
_add_accept_hooks_flag(gateway_parser)
+8
View File
@@ -602,6 +602,14 @@ class S6ServiceManager:
]
for k, v in sorted(extra_env.items()):
lines.append(f"export {k}={shlex.quote(v)}")
# Sentinel for the supervised-child path. Prevents recursive
# redirect when the supervised gateway re-enters
# `_gateway_command_inner` with subcmd == "run" — without it the
# supervisor would dispatch `gateway start` which would re-exec
# `gateway run --replace` which would re-dispatch `gateway
# start`, etc. See `_gateway_command_inner` for the matching
# guard.
lines.append("export HERMES_S6_SUPERVISED_CHILD=1")
if profile == "default":
lines.append("exec s6-setuidgid hermes hermes gateway run")
else: