* fix(gateway): auto-start after container restart via planned-stop marker
On Docker (s6-overlay), the gateway runs as a dynamically-registered s6
service. When the container stops/restarts/upgrades, s6 sends the gateway
a plain SIGTERM. The shutdown path (_stop_impl) ended with an
unconditional _update_runtime_status("stopped"), persisting
gateway_state=stopped to the volume. container_boot.py reads that on the
next boot and only auto-starts gateways whose last state was "running"
(_AUTOSTART_STATES) — so after a routine `docker compose up
--force-recreate` the gateway stays down and messaging channels silently
go dark, with no error surfaced (issue #42675).
The codebase already distinguishes intentional stops from unexpected
signals via the planned-stop marker (write_planned_stop_marker /
consume_planned_stop_marker_for_self): `hermes gateway stop`,
systemd/launchd ExecStop, and Ctrl+C write a marker before signalling,
so the handler classifies them as planned. An unmarked SIGTERM
(container/s6 restart, OOM, bare kill) is signal-initiated.
This wires that existing classification through to the state persist,
rather than adding unreliable signal-source inference:
- run.py: GatewayRunner._signal_initiated_shutdown, set in
shutdown_signal_handler's unmarked-signal branch. In _stop_impl, a
signal-initiated (non-restart) teardown now persists "running" instead
of "stopped" — preserving the operator's run-intent and overwriting the
mid-shutdown "draining" marker so _AUTOSTART_STATES matches on reboot.
Operator stops and restarts persist "stopped" as before.
- service_manager.py: S6ServiceManager.stop() now writes the planned-stop
marker for the supervised PID (read from s6-svstat) before `s6-svc -d`,
so an in-container `hermes gateway stop` is correctly classified as
intentional (parity with the systemd/launchd/host stop paths, which
already mark). Best-effort: a marker-write failure falls back to the
safe signal-initiated path.
Tests: shutdown persist-decision table (signal→running, operator→stopped,
restart→stopped), s6 stop marker write + svstat PID parse + failure
tolerance. The signal→running and s6-marker tests fail without the
respective source change. Verified end-to-end against a container built
from this branch: an unmarked SIGTERM to the live gateway leaves
gateway_state=running (shutdown-context log confirms signal path);
existing real container-restart suite still green.
* docs(docker): clarify gateway autostart distinguishes operator-stop from container-kill
The per-profile-supervision section described the autostart-across-restart
contract as "running gateways come back, stopped stay stopped" without
spelling out what records 'stopped'. That contract was the source of
#42675 confusion: users expected a restart to bring the gateway back and
it didn't. With the write-side fix, only an explicit `hermes gateway stop`
records 'stopped'; container/s6 restart SIGTERMs (incl. image upgrades and
unexpected exits) leave the state 'running' so the gateway auto-starts.
Make that distinction explicit in both the multi-profile and
per-profile-supervision sections.
* test(docker): real-restart autostart E2E for #42675
Adds test_live_gateway_autostarts_after_real_restart_without_manual_state_stamp:
a live s6-supervised gateway is killed by an actual `docker restart`
SIGTERM (no manual gateway_state stamp, no planned-stop marker) and must
auto-start on the next boot. Exercises the WRITE side of the fix that the
existing stamp-based tests bypass.
Verified to FAIL against an origin/main image (reconciler logs
prior_state=stopped action=registered — the #42675 bug) and PASS against
the fixed image (prior_state=running action=started).
This commit is contained in:
Ben Barclay
+47
-1
@@ -1953,6 +1953,16 @@ class GatewayRunner(GatewayAuthorizationMixin, GatewayKanbanWatchersMixin, Gatew
|
||||
self._exit_code: Optional[int] = None
|
||||
self._draining = False
|
||||
self._restart_requested = False
|
||||
# Set by shutdown_signal_handler when a SIGTERM/SIGINT arrived
|
||||
# WITHOUT a planned-stop / takeover marker — i.e. an unexpected
|
||||
# external signal (container/s6 SIGTERM on `docker restart` or
|
||||
# image upgrade, OOM-killer, bare `kill`). Distinct from an
|
||||
# operator-requested stop, which writes a marker first. Used by
|
||||
# _stop_impl to decide whether to persist gateway_state=stopped
|
||||
# (see issue #42675): an unexpected signal must NOT persist
|
||||
# "stopped", or container_boot refuses to auto-start the gateway
|
||||
# on the next boot.
|
||||
self._signal_initiated_shutdown = False
|
||||
self._restart_task_started = False
|
||||
self._restart_detached = False
|
||||
self._restart_via_service = False
|
||||
@@ -5952,7 +5962,36 @@ class GatewayRunner(GatewayAuthorizationMixin, GatewayKanbanWatchersMixin, Gatew
|
||||
self._exit_reason = self._exit_reason or "Gateway restart requested"
|
||||
|
||||
self._draining = False
|
||||
self._update_runtime_status("stopped", self._exit_reason)
|
||||
# Persist the terminal gateway_state. The default is "stopped",
|
||||
# but when this teardown was triggered by an UNEXPECTED external
|
||||
# signal (container/s6 SIGTERM on `docker restart` or image
|
||||
# upgrade, OOM-killer, bare `kill`) we instead persist "running"
|
||||
# to preserve the operator's run-intent across the restart.
|
||||
#
|
||||
# On Docker (s6-overlay), container_boot.py reads gateway_state
|
||||
# on the next boot and only auto-starts gateways whose last
|
||||
# state was "running" (_AUTOSTART_STATES). Persisting "stopped"
|
||||
# — or leaving the mid-shutdown "draining" marker in place — for
|
||||
# a routine `docker compose up --force-recreate` permanently
|
||||
# suppresses auto-start, so the messaging channels silently stay
|
||||
# dark until the operator manually restarts (issue #42675).
|
||||
#
|
||||
# An operator-initiated stop (`hermes gateway stop`,
|
||||
# systemd/launchd ExecStop, the s6 stop path, Ctrl+C) writes a
|
||||
# planned-stop marker BEFORE signalling, so it is classified as
|
||||
# a planned stop (not signal-initiated) and correctly persists
|
||||
# "stopped" — respecting the explicit intent. A restart also
|
||||
# persists "stopped" here; the restarting process brings the
|
||||
# gateway back up itself.
|
||||
if getattr(self, "_signal_initiated_shutdown", False) and not self._restart_requested:
|
||||
logger.info(
|
||||
"Gateway stopped by an unexpected signal — persisting "
|
||||
"gateway_state=running so container_boot auto-starts on "
|
||||
"the next boot (issue #42675)"
|
||||
)
|
||||
self._update_runtime_status("running", self._exit_reason)
|
||||
else:
|
||||
self._update_runtime_status("stopped", self._exit_reason)
|
||||
logger.info("Gateway stopped (total teardown %.2fs)", _phase_elapsed())
|
||||
|
||||
self._stop_task = asyncio.create_task(_stop_impl())
|
||||
@@ -15711,6 +15750,13 @@ async def start_gateway(config: Optional[GatewayConfig] = None, replace: bool =
|
||||
)
|
||||
else:
|
||||
_signal_initiated_shutdown = True
|
||||
# Mirror onto the runner so _stop_impl can suppress the
|
||||
# gateway_state=stopped persist for unexpected signals
|
||||
# (container/s6 SIGTERM on restart, OOM, bare kill) — see
|
||||
# issue #42675. Operator-initiated stops set a planned-stop
|
||||
# marker first, land in the `planned_stop` branch above, and
|
||||
# leave this flag False so they DO persist "stopped".
|
||||
runner._signal_initiated_shutdown = True
|
||||
logger.info(
|
||||
"Received %s — initiating shutdown",
|
||||
_shutdown_ctx["signal"] if _shutdown_ctx else "SIGTERM/SIGINT",
|
||||
|
||||
Reference in New Issue
Block a user