fix(kanban): don't permanently block tasks that hit a provider rate limit (#38223)

A kanban worker that exhausted its retries purely on a provider rate
limit / quota wall (e.g. opencode-go's 5-hour window) exited with code 1.
The dispatcher counted that as a crash, and with DEFAULT_FAILURE_LIMIT=2
two quota-wall hits permanently blocked the card. Fanning out many
workers against one shared quota made this routine.

Now a rate-limited worker exits with EX_TEMPFAIL (75); the dispatcher
classifies that as a 'rate_limited' exit, releases the task back to
'ready' WITHOUT incrementing consecutive_failures (the breaker can't trip
on a transient throttle), and the respawn guard defers the next attempt
on a cooldown (default 5min, HERMES_KANBAN_RATE_LIMIT_COOLDOWN_SECONDS)
until the quota window clears. Genuine crashes still count and trip the
breaker as before. The 120s Retry-After cap is unchanged — no worker
parks for hours holding a slot.

- conversation_loop.py: surface failure_reason in the exhaustion return
- cli.py: kanban worker picks exit 75 on rate_limit/billing failure
- kanban_db.py: rate_limited exit kind, no-count requeue, cooldown guard
This commit is contained in:
Teknium
2026-06-03 06:19:32 -07:00
committed by GitHub
parent 60b6352fe5
commit 4c544b633d
4 changed files with 408 additions and 16 deletions
+27 -3
View File
@@ -15807,9 +15807,33 @@ def main(
# Session ID goes to stderr so piped stdout is clean.
print(f"\nsession_id: {cli.session_id}", file=sys.stderr)
# Ensure proper exit code for automation wrappers
sys.exit(1 if isinstance(result, dict) and result.get("failed") else 0)
# Ensure proper exit code for automation wrappers.
#
# Kanban workers get a special case: when the run failed
# purely because the provider rate-limited / exhausted
# quota (not because the task itself is broken), exit with
# the EX_TEMPFAIL sentinel instead of the generic 1. The
# dispatcher's reap classifier maps that code to a
# ``rate_limited`` exit and releases the task back to
# ``ready`` WITHOUT incrementing the failure counter, so a
# 5-hour quota window can't trip the circuit breaker and
# permanently block the card. Non-kanban runs keep the
# plain 0/1 contract automation wrappers expect.
_exit_code = 0
if isinstance(result, dict) and result.get("failed"):
_exit_code = 1
if os.environ.get("HERMES_KANBAN_TASK") and result.get(
"failure_reason"
) in ("rate_limit", "billing"):
try:
from hermes_cli.kanban_db import (
KANBAN_RATE_LIMIT_EXIT_CODE as _RL_CODE,
)
_exit_code = _RL_CODE
except Exception:
_exit_code = 1
sys.exit(_exit_code)
# Exit with error code if credentials or agent init fails
sys.exit(1)