fix(gateway): fall back to in-process heartbeat when s6 sleep is missing (#36208) (#37120)

Inside an s6 container, `gateway run` redirects to the supervised
gateway and then keeps the CMD process alive as a no-op heartbeat so
/init doesn't start stage-3 shutdown. That heartbeat is
`os.execvp("sleep", ["sleep", "infinity"])`, which does a PATH lookup
for the `sleep` binary. When PATH was empty/truncated/clobbered at that
point — e.g. after user customizations rewrote PATH, or on a minimal
image without `sleep` on PATH — the exec raised FileNotFoundError,
killing the CMD process and causing /init to tear down every service:
the container failed to start (issue #36208, a regression in the s6
image from 2026.5.28).

Wrap the exec in try/except OSError: on success it still replaces the
process with the cheap `sleep` heartbeat (no resident Python
interpreter, and the existing process-tree/recursion contract is
preserved); on failure it falls back to `_block_until_terminated()` —
a SIGTERM handler (clean 128+signum exit on `docker stop`) plus a
signal.pause() loop, which needs no external binary and so can't fail
on PATH state. A threading.Event().wait() fallback covers platforms
without signal.pause().

Keeping execvp as the primary path (rather than replacing it outright)
preserves the `sleep infinity` heartbeat that the docker integration
tests assert (test_gateway_run_supervised.py) and avoids leaving a
full Python interpreter resident for the container's lifetime.

Verified end-to-end on a built image: with execvp forced to fail,
_block_until_terminated() blocks cleanly instead of raising
FileNotFoundError; normal boot still runs the cheap `sleep infinity`
heartbeat; the 6 test_gateway_run_supervised.py integration tests pass.

Salvages the two community fixes for this issue — the fallback design
from #36221 (@Pluviobyte) and the signal.pause() heartbeat from #36267
(@karmeleon) — and adds regression tests for both the normal and
sleep-missing paths.

Co-authored-by: Pluviobyte <Pluviobyte@users.noreply.github.com>
Co-authored-by: karmeleon <karmeleon@users.noreply.github.com>

Closes #36208.
This commit is contained in:
Ben Barclay
2026-06-02 11:59:27 +10:00
committed by GitHub
co-authored by Pluviobyte karmeleon
parent abe0e19c0a
commit eee32cdd52
2 changed files with 168 additions and 34 deletions
+52 -7
View File
@@ -5877,15 +5877,60 @@ def _maybe_redirect_run_to_s6_supervision(args) -> bool:
file=sys.stderr,
flush=True,
)
# Block until the container is signalled. The supervised gateway's
# lifetime is independent of this process — s6-supervise restarts
# it on crash, and we don't want the container to exit when the
# gateway flaps. `sleep infinity` matches the static main-hermes
# service's pattern (see docker/s6-rc.d/main-hermes/run): the CMD
# process is a no-op heartbeat that keeps /init alive until
# Keep the CMD process alive as a no-op heartbeat. The supervised
# gateway's lifetime is independent of this process — s6-supervise
# restarts it on crash, and we don't want the container to exit when
# the gateway flaps. The CMD process keeps /init alive until
# `docker stop` sends SIGTERM, at which point /init runs stage 3
# shutdown (which tears down the supervised gateway cleanly).
os.execvp("sleep", ["sleep", "infinity"])
#
# Prefer `sleep infinity` (matches the static main-hermes service's
# pattern in docker/s6-rc.d/main-hermes/run, and frees the Python
# interpreter — the heartbeat is a tiny `sleep` process, not a
# resident interpreter). But `os.execvp` does a PATH lookup for the
# `sleep` binary and historically crashed the whole container with
# FileNotFoundError when PATH was empty/truncated/clobbered at this
# point — e.g. after user customizations rewrote PATH, or on minimal
# images without `sleep` on PATH (issue #36208). Fall back to an
# in-process block (no external binary, can't fail on PATH) so the
# container keeps running instead of dying during boot.
try:
os.execvp("sleep", ["sleep", "infinity"])
except OSError:
# execvp only returns by raising; on success it replaces this
# process. ENOENT (no `sleep` on PATH) and any other exec error
# land here.
print(
"→ `sleep` is unavailable; keeping the s6 CMD process alive "
"in-process until the container is stopped.",
file=sys.stderr,
flush=True,
)
_block_until_terminated()
return True # unreachable on the execvp success path
def _block_until_terminated() -> None:
"""Keep the s6 CMD process alive until the container is stopped.
Fallback heartbeat for when ``os.execvp("sleep", ...)`` can't run
(``sleep`` missing from PATH — issue #36208). Installs a SIGTERM
handler that exits with the conventional 128+signum code so
``docker stop`` produces a clean, expected exit, then blocks on
``signal.pause()``. Falls back to ``threading.Event().wait()`` on
platforms without ``signal.pause()`` (e.g. Windows) — although this
path only runs inside the s6 Linux container image, the fallback
keeps the helper safe to import and unit-test anywhere.
"""
signal.signal(signal.SIGTERM, lambda signum, _frame: sys.exit(128 + signum))
pause = getattr(signal, "pause", None)
if pause is not None:
while True:
pause()
else: # pragma: no cover - non-Unix fallback, not exercised in the s6 image
import threading
threading.Event().wait()
def _gateway_command_inner(args):