feat(docker): auto-redirect gateway run to supervised mode inside s6 image
Pre-s6, `docker run nousresearch/hermes-agent gateway run` was the
standard invocation: gateway ran as the container's main process,
tini reaped zombies, container exit code matched gateway exit code,
no supervision. With s6-overlay as PID 1, the same invocation now
auto-upgrades to supervised semantics — auto-restart on crash,
dashboard supervised alongside (when HERMES_DASHBOARD=1 is set),
multiple profile gateways under the same /init.
Users get the new behavior with zero changes to their docker run
command. A loud one-line breadcrumb on stderr explains the upgrade
and points at the opt-out for users who genuinely want pre-s6
foreground semantics.
How it works:
1. `_gateway_command_inner` (the `gateway run` handler) checks if
we're inside a container with s6 as PID 1.
2. If yes, dispatches `start` to the s6 service manager (registers
and starts gateway-default), then `exec sleep infinity` to keep
the CMD process alive without binding container lifetime to
gateway PID lifetime. The supervised gateway can flap freely;
`docker stop` still tears everything down via /init stage 3.
3. If no, falls through to the existing foreground code path
unchanged. Host runs of `hermes gateway run` are unaffected.
Three gates make the redirect inert outside the intended scope:
* `detect_service_manager() != "s6"` — host/non-s6-container runs.
* `HERMES_S6_SUPERVISED_CHILD=1` env var (recursion guard) —
exported by `S6ServiceManager._render_run_script` for the
s6-supervised invocation itself. Without this guard, the
supervised `gateway run --replace` would re-enter the redirect
and recurse (run → start → run → start → ...) infinitely.
* `--no-supervise` CLI flag OR `HERMES_GATEWAY_NO_SUPERVISE=1` env
var — explicit user opt-out for CI smoke tests, debugging the
foreground startup path, or any case wanting "CMD exit =
container exit" semantics. Strict truthiness (1/true/yes,
case-insensitive); typos like `=0` do NOT silently opt out.
Tests:
* Unit tests in tests/hermes_cli/test_gateway_s6_dispatch.py
cover all five paths (host no-op, supervised fire, sentinel
recursion guard, CLI flag, env var truthy + falsy). The two
load-bearing gates (sentinel + opt-out) were mutation-tested
by removing each gate in isolation and confirming the dedicated
test fails with the expected error.
* Docker harness tests in tests/docker/test_gateway_run_supervised.py
cover the round trips end-to-end against a built image: redirect
fires (sleep-infinity heartbeat + supervised gateway-default
slot + breadcrumb), --no-supervise opt-out (foreground gateway,
no want-up on the slot), HERMES_GATEWAY_NO_SUPERVISE env var
works identically, recursion is impossible (≤1 supervised
python gateway-run + exactly 1 sleep-infinity parented to the
CMD wrapper), and HERMES_DASHBOARD=1 produces both supervised
gateway and supervised dashboard.
Docs:
* Added a `:::tip Gateway runs supervised` admonition near the
main docker.md example explaining the upgrade and pointing at
the opt-out. Pre-s6 (tini-based) images still run gateway run
as the foreground main process, so the note is scoped to the
s6 image only.
Trade-off documented in the helper docstring: container exit code
under the redirect is sleep's exit code (always 0 on SIGTERM), not
the gateway's. That was an explicit design call — the supervised
gateway is allowed to flap without taking the container with it,
which is what "supervision" means. CI users who want exit-code
forwarding can pass --no-supervise.
This commit is contained in:
@@ -5150,11 +5150,83 @@ def gateway_command(args):
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _maybe_redirect_run_to_s6_supervision(args) -> bool:
|
||||
"""Inside an s6 container, redirect bare ``gateway run`` to the
|
||||
supervised path.
|
||||
|
||||
Background. Before the s6 image landed, ``docker run <image> gateway
|
||||
run`` was the standard way to start a containerized gateway: the
|
||||
gateway was the container's main process, tini reaped zombies, and
|
||||
container exit code == gateway exit code. With s6-overlay as PID 1,
|
||||
we'd much rather have the gateway run as a supervised s6 longrun
|
||||
(auto-restart on crash, dashboard supervised alongside, multiple
|
||||
profile gateways under the same /init). This redirect upgrades the
|
||||
old invocation transparently — the user gets the new behavior
|
||||
without changing their docker run command.
|
||||
|
||||
Three gates make this a no-op outside the intended scope:
|
||||
|
||||
1. ``_dispatch_via_service_manager_if_s6`` returns False unless
|
||||
we're in a container with s6 as PID 1. Host runs of
|
||||
``hermes gateway run`` are unaffected.
|
||||
2. ``HERMES_S6_SUPERVISED_CHILD`` is exported by
|
||||
``S6ServiceManager._render_run_script`` for the supervised
|
||||
process itself — i.e. when s6-supervise execs ``hermes gateway
|
||||
run --replace`` as a longrun, this guard short-circuits the
|
||||
redirect so the supervised gateway actually runs in
|
||||
foreground (otherwise we'd recurse: run → start → run → start
|
||||
→ ...).
|
||||
3. ``--no-supervise`` (or ``HERMES_GATEWAY_NO_SUPERVISE=1``) opts
|
||||
out for users who genuinely want pre-s6 semantics — CI smoke
|
||||
tests, debugging the foreground startup path, etc.
|
||||
|
||||
Returns True iff dispatched (caller should ``return``).
|
||||
"""
|
||||
no_supervise = getattr(args, "no_supervise", False) or \
|
||||
os.environ.get("HERMES_GATEWAY_NO_SUPERVISE", "").lower() in ("1", "true", "yes")
|
||||
if no_supervise:
|
||||
return False
|
||||
if os.environ.get("HERMES_S6_SUPERVISED_CHILD"):
|
||||
# We ARE the supervised child s6-supervise is running. Fall
|
||||
# through to the foreground code path so the gateway actually
|
||||
# starts.
|
||||
return False
|
||||
if not _dispatch_via_service_manager_if_s6("start"):
|
||||
return False
|
||||
# Loud breadcrumb: explain the upgrade and how to opt out. Print to
|
||||
# stderr so it doesn't pollute stdout-parsing scripts. The
|
||||
# supervised gateway's own logs are routed by s6-log to both
|
||||
# `docker logs` and ${HERMES_HOME}/logs/gateways/<profile>/current,
|
||||
# so the user sees a clear sequence: this banner first, then the
|
||||
# gateway's own stdout/stderr from the supervisor.
|
||||
print(
|
||||
"→ gateway is now running under s6 supervision (auto-restart on crash,\n"
|
||||
" dashboard supervised alongside if HERMES_DASHBOARD is set).\n"
|
||||
" This is the recommended setup for the s6 container image — the\n"
|
||||
" gateway will keep running even if it crashes.\n"
|
||||
" Use `--no-supervise` (or HERMES_GATEWAY_NO_SUPERVISE=1) to opt out\n"
|
||||
" and get the pre-s6 foreground behavior instead.",
|
||||
file=sys.stderr,
|
||||
flush=True,
|
||||
)
|
||||
# Block until the container is signalled. The supervised gateway's
|
||||
# lifetime is independent of this process — s6-supervise restarts
|
||||
# it on crash, and we don't want the container to exit when the
|
||||
# gateway flaps. `sleep infinity` matches the static main-hermes
|
||||
# service's pattern (see docker/s6-rc.d/main-hermes/run): the CMD
|
||||
# process is a no-op heartbeat that keeps /init alive until
|
||||
# `docker stop` sends SIGTERM, at which point /init runs stage 3
|
||||
# shutdown (which tears down the supervised gateway cleanly).
|
||||
os.execvp("sleep", ["sleep", "infinity"])
|
||||
|
||||
|
||||
def _gateway_command_inner(args):
|
||||
subcmd = getattr(args, 'gateway_command', None)
|
||||
|
||||
# Default to run if no subcommand
|
||||
if subcmd is None or subcmd == "run":
|
||||
if _maybe_redirect_run_to_s6_supervision(args):
|
||||
return # unreachable; execvp doesn't return
|
||||
verbose = getattr(args, 'verbose', 0)
|
||||
quiet = getattr(args, 'quiet', False)
|
||||
replace = getattr(args, 'replace', False)
|
||||
|
||||
@@ -11326,6 +11326,19 @@ def main():
|
||||
action="store_true",
|
||||
help="Replace any existing gateway instance (useful for systemd)",
|
||||
)
|
||||
gateway_run.add_argument(
|
||||
"--no-supervise",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Inside the s6-overlay Docker image, normally `gateway run` is "
|
||||
"automatically redirected to the supervised s6 service (so the "
|
||||
"gateway gets auto-restart on crash, plus a supervised dashboard "
|
||||
"if HERMES_DASHBOARD is set). Pass --no-supervise to opt out and "
|
||||
"get the historical pre-s6 foreground behavior: the gateway is "
|
||||
"the container's main process and the container exits with the "
|
||||
"gateway's exit code. No effect outside an s6 container."
|
||||
),
|
||||
)
|
||||
_add_accept_hooks_flag(gateway_run)
|
||||
_add_accept_hooks_flag(gateway_parser)
|
||||
|
||||
|
||||
@@ -602,6 +602,14 @@ class S6ServiceManager:
|
||||
]
|
||||
for k, v in sorted(extra_env.items()):
|
||||
lines.append(f"export {k}={shlex.quote(v)}")
|
||||
# Sentinel for the supervised-child path. Prevents recursive
|
||||
# redirect when the supervised gateway re-enters
|
||||
# `_gateway_command_inner` with subcmd == "run" — without it the
|
||||
# supervisor would dispatch `gateway start` which would re-exec
|
||||
# `gateway run --replace` which would re-dispatch `gateway
|
||||
# start`, etc. See `_gateway_command_inner` for the matching
|
||||
# guard.
|
||||
lines.append("export HERMES_S6_SUPERVISED_CHILD=1")
|
||||
if profile == "default":
|
||||
lines.append("exec s6-setuidgid hermes hermes gateway run")
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user