fix(kanban): don't permanently block tasks that hit a provider rate limit (#38223)
A kanban worker that exhausted its retries purely on a provider rate limit / quota wall (e.g. opencode-go's 5-hour window) exited with code 1. The dispatcher counted that as a crash, and with DEFAULT_FAILURE_LIMIT=2 two quota-wall hits permanently blocked the card. Fanning out many workers against one shared quota made this routine. Now a rate-limited worker exits with EX_TEMPFAIL (75); the dispatcher classifies that as a 'rate_limited' exit, releases the task back to 'ready' WITHOUT incrementing consecutive_failures (the breaker can't trip on a transient throttle), and the respawn guard defers the next attempt on a cooldown (default 5min, HERMES_KANBAN_RATE_LIMIT_COOLDOWN_SECONDS) until the quota window clears. Genuine crashes still count and trip the breaker as before. The 120s Retry-After cap is unchanged — no worker parks for hours holding a slot. - conversation_loop.py: surface failure_reason in the exhaustion return - cli.py: kanban worker picks exit 75 on rate_limit/billing failure - kanban_db.py: rate_limited exit kind, no-count requeue, cooldown guard
This commit is contained in:
@@ -15807,9 +15807,33 @@ def main(
|
||||
|
||||
# Session ID goes to stderr so piped stdout is clean.
|
||||
print(f"\nsession_id: {cli.session_id}", file=sys.stderr)
|
||||
|
||||
# Ensure proper exit code for automation wrappers
|
||||
sys.exit(1 if isinstance(result, dict) and result.get("failed") else 0)
|
||||
|
||||
# Ensure proper exit code for automation wrappers.
|
||||
#
|
||||
# Kanban workers get a special case: when the run failed
|
||||
# purely because the provider rate-limited / exhausted
|
||||
# quota (not because the task itself is broken), exit with
|
||||
# the EX_TEMPFAIL sentinel instead of the generic 1. The
|
||||
# dispatcher's reap classifier maps that code to a
|
||||
# ``rate_limited`` exit and releases the task back to
|
||||
# ``ready`` WITHOUT incrementing the failure counter, so a
|
||||
# 5-hour quota window can't trip the circuit breaker and
|
||||
# permanently block the card. Non-kanban runs keep the
|
||||
# plain 0/1 contract automation wrappers expect.
|
||||
_exit_code = 0
|
||||
if isinstance(result, dict) and result.get("failed"):
|
||||
_exit_code = 1
|
||||
if os.environ.get("HERMES_KANBAN_TASK") and result.get(
|
||||
"failure_reason"
|
||||
) in ("rate_limit", "billing"):
|
||||
try:
|
||||
from hermes_cli.kanban_db import (
|
||||
KANBAN_RATE_LIMIT_EXIT_CODE as _RL_CODE,
|
||||
)
|
||||
_exit_code = _RL_CODE
|
||||
except Exception:
|
||||
_exit_code = 1
|
||||
sys.exit(_exit_code)
|
||||
|
||||
# Exit with error code if credentials or agent init fails
|
||||
sys.exit(1)
|
||||
|
||||
Reference in New Issue
Block a user