fix(kanban): align recompute_ready guard with breaker's configured failure_limit

Follow-up to the budget-exhaustion recovery fix. recompute_ready's
new circuit-breaker guard resolved its effective limit from per-task
max_retries -> DEFAULT_FAILURE_LIMIT, skipping the dispatcher's
configured kanban.failure_limit. _record_task_failure resolves
max_retries -> failure_limit(config) -> DEFAULT, so the two disagreed
whenever an operator set kanban.failure_limit != 2:

- config > 2: a task could get stuck at DEFAULT(2) before reaching its
  allowed retry count.
- config < 2: a task the breaker already blocked could be auto-recovered
  back to ready, defeating the stricter limit.

Thread the dispatcher's failure_limit through dispatch_once into
recompute_ready so the guard and the breaker share one resolution order.
Updated test_circuit_breaker_block_still_auto_promotes (it asserted a
failures=5 block auto-recovers and resets the counter — that's the
pre-#35072 behavior the loop fix removes); it now exercises a below-limit
transient block, with the at-limit case covered in test_kanban_db.py.
Added two tests for the config-tier and per-task override resolution.
This commit is contained in:
teknium1
2026-05-30 01:40:57 -07:00
committed by Teknium
parent 6ab71d3bb4
commit 8e5a6854c3
3 changed files with 108 additions and 18 deletions
+20 -8
View File
@@ -2591,7 +2591,9 @@ def _has_sticky_block(conn: sqlite3.Connection, task_id: str) -> bool:
return bool(row) and row["kind"] == "blocked"
def recompute_ready(conn: sqlite3.Connection) -> int:
def recompute_ready(
conn: sqlite3.Connection, failure_limit: int = None,
) -> int:
"""Promote ``todo`` tasks to ``ready`` when all parents are ``done`` or ``archived``.
Returns the number of tasks promoted. Safe to call inside or outside
@@ -2606,12 +2608,22 @@ def recompute_ready(conn: sqlite3.Connection) -> int:
``kanban_unblock`` (#28712).
2. The task's ``consecutive_failures`` has reached the effective
failure limit (per-task ``max_retries`` or
``DEFAULT_FAILURE_LIMIT``). This prevents infinite retry
loops when a task repeatedly exhausts its iteration budget:
without this guard the counter would reset on every recovery
cycle and the circuit breaker could never trip.
failure limit. This prevents infinite retry loops when a task
repeatedly exhausts its iteration budget: without this guard the
counter would reset on every recovery cycle and the circuit
breaker could never trip (#35072).
The effective failure limit resolves in the same order as the
circuit breaker in ``_record_task_failure`` so the two never
disagree about when a task is permanently blocked:
1. per-task ``max_retries`` if set
2. caller-supplied ``failure_limit`` (the dispatcher passes the
``kanban.failure_limit`` config value through ``dispatch_once``)
3. ``DEFAULT_FAILURE_LIMIT``
"""
if failure_limit is None:
failure_limit = DEFAULT_FAILURE_LIMIT
promoted = 0
with write_txn(conn):
todo_rows = conn.execute(
@@ -2647,7 +2659,7 @@ def recompute_ready(conn: sqlite3.Connection) -> int:
task_limit = row["max_retries"]
effective_limit = (
int(task_limit) if task_limit is not None
else DEFAULT_FAILURE_LIMIT
else int(failure_limit)
)
if failures >= effective_limit:
continue
@@ -5577,7 +5589,7 @@ def dispatch_once(
if _crash_auto_blocked:
result.auto_blocked.extend(_crash_auto_blocked)
result.timed_out = enforce_max_runtime(conn)
result.promoted = recompute_ready(conn)
result.promoted = recompute_ready(conn, failure_limit=failure_limit)
# Count tasks already running so max_spawn enforces concurrency rather
# than a per-tick spawn budget. See the docstring above for the full