fix(gateway,windows): reliability — supervisor task, JOB breakaway, status --deep
Three coordinated fixes for the Windows gateway reliability story:
1. CREATE_BREAKAWAY_FROM_JOB on every detached spawn
The 'hermes update' triggered from the Electron Desktop GUI ran inside
Electron's job object. Without breakaway, the post-update gateway
watcher spawned by update — already DETACHED_PROCESS — was still
reaped when Electron's job tore down, so the gateway never came back
after a GUI-initiated update. Adds CREATE_BREAKAWAY_FROM_JOB (0x01000000)
to:
- hermes_cli/_subprocess_compat.py::windows_detach_flags() — used by
every helper that calls windows_detach_popen_kwargs(), including
launch_detached_profile_gateway_restart()
- The watcher subprocess's own respawn snippet in
hermes_cli/gateway.py (inlined flags so the watcher's child
respawn also breaks away)
_spawn_detached() in gateway_windows.py already had the flag; this
change brings the rest of the codebase to parity.
2. Per-minute supervisor Scheduled Task — Windows equivalent of
systemd Restart=always
Introduces hermes_cli/gateway_supervisor.py and registers it as a
second Scheduled Task ('Hermes_Gateway_Supervisor', SC MINUTE /MO 1,
LIMITED rights) alongside the existing ONLOGON task. Every minute,
the supervisor uses the same gateway.status.get_running_pid() probe
as 'hermes gateway status' and, if no gateway is alive, calls
gateway_windows._spawn_detached() (which now includes BREAKAWAY) to
bring one back.
Covers every crash mode, not just 'machine rebooted': taskkill,
OOM, GUI update SIGTERM, parent job teardown. Cheap — one pythonw
startup per minute when down, one PID-existence check per minute
when up.
Wired into both the schtasks-success and Startup-folder-fallback
install paths via _install_supervisor_best_effort(), and removed in
uninstall(). Best-effort: a failing supervisor install logs a
warning but doesn't roll back the primary install.
3. 'hermes gateway status --deep' shows per-probe PASS/FAIL
Replaces the existing terse '--deep' output (which only printed
paths) with an actual diagnostic table:
[1] PID file present
[2] Lock file held by a live process
[3] get_running_pid() result
[4] _pid_exists(pid) — OS-level liveness
[5] gateway_state.json (state + age)
[6] Last lifecycle event from gateway-exit-diag.log
When the high-level summary disagrees with reality, the user can
see exactly which signal is lying.
Test-leak fix
-------------
tests/hermes_cli/test_gateway_wsl.py::TestGatewayCommandWSLMessages
monkey-patched is_linux/is_wsl/supports_systemd_services to simulate
WSL but did NOT stub is_windows(). On a Windows host, the dispatcher
in _gateway_command_inner takes the is_windows() branch BEFORE the
WSL guidance branch, so the test invoked gateway_windows.install()
for real. install() writes to %APPDATA%\...\Startup\Hermes_Gateway.cmd
— the REAL user Startup folder, never sandboxed by tmp_path — pointing
at the test's pytest-of-<user>/pytest-<N>/.../gateway-service/ wrapper.
When pytest tore down the tmp_path, every subsequent Windows login
flashed a cmd.exe window that failed to find the missing target.
Stubs is_windows=False on all four affected tests:
test_install_wsl_no_systemd
test_start_wsl_no_systemd
test_status_wsl_running_manual
test_status_wsl_not_running
Defense-in-depth: _build_startup_launcher() now prefixes the launcher
with 'if not exist <target> exit /b 0', so any future stale Startup
entry silently no-ops instead of flashing a console window.
Status enhancements
-------------------
- status() now reports supervisor task presence alongside the existing
schtasks/Startup info, and nudges the user to reinstall if the
supervisor isn't registered.
- Deep mode dumps both the supervisor task name + script path.
This commit is contained in:
@@ -97,6 +97,16 @@ def resolve_node_command(name: str, argv: Sequence[str]) -> list[str]:
|
||||
_CREATE_NEW_PROCESS_GROUP = 0x00000200
|
||||
_DETACHED_PROCESS = 0x00000008
|
||||
_CREATE_NO_WINDOW = 0x08000000
|
||||
# Escape any Win32 job object the parent process belongs to. Without this,
|
||||
# a detached child still inherits its parent's job object membership, and
|
||||
# when that parent (Electron, Tauri, Windows Terminal, the Desktop GUI's
|
||||
# bootstrap-installer) dies, the OS tears down the whole job — taking the
|
||||
# "detached" child with it. Critical for the post-update gateway watcher:
|
||||
# Electron spawns the Tauri updater inside its own job, the updater spawns
|
||||
# the watcher subprocess; without BREAKAWAY the watcher dies the instant
|
||||
# Electron exits, so the gateway never gets respawned after a `hermes
|
||||
# update` triggered from the GUI. See fix/windows-gateway-reliability.
|
||||
_CREATE_BREAKAWAY_FROM_JOB = 0x01000000
|
||||
|
||||
|
||||
def windows_detach_flags() -> int:
|
||||
@@ -116,10 +126,30 @@ def windows_detach_flags() -> int:
|
||||
- ``CREATE_NO_WINDOW`` — suppress the brief cmd flash that would
|
||||
otherwise appear when launching a console app. Redundant with
|
||||
DETACHED_PROCESS but explicit for clarity.
|
||||
- ``CREATE_BREAKAWAY_FROM_JOB`` — escape any job object the parent is
|
||||
in. Electron (Desktop app) and Tauri (bootstrap installer) wrap
|
||||
their children in job objects; without breakaway, those children
|
||||
die when the parent process exits even if they were spawned with
|
||||
DETACHED_PROCESS. This was the missing flag that made the
|
||||
post-update gateway respawn watcher silently die alongside the
|
||||
Tauri updater after the Electron Desktop's update flow finished.
|
||||
|
||||
If a process is in a job that disallows breakaway (rare —
|
||||
JOB_OBJECT_LIMIT_BREAKAWAY_OK isn't set), CreateProcess returns
|
||||
ERROR_ACCESS_DENIED. Python surfaces that as ``PermissionError``
|
||||
on the ``subprocess.Popen`` call. Callers in this codebase already
|
||||
wrap detached spawns in ``try/except OSError`` and fall back to a
|
||||
cmd.exe wrapper, so the breakaway-denied case degrades gracefully
|
||||
rather than crashing.
|
||||
"""
|
||||
if not IS_WINDOWS:
|
||||
return 0
|
||||
return _CREATE_NEW_PROCESS_GROUP | _DETACHED_PROCESS | _CREATE_NO_WINDOW
|
||||
return (
|
||||
_CREATE_NEW_PROCESS_GROUP
|
||||
| _DETACHED_PROCESS
|
||||
| _CREATE_NO_WINDOW
|
||||
| _CREATE_BREAKAWAY_FROM_JOB
|
||||
)
|
||||
|
||||
|
||||
def windows_hide_flags() -> int:
|
||||
|
||||
Reference in New Issue
Block a user