fix(kanban): align recompute_ready guard with breaker's configured failure_limit
Follow-up to the budget-exhaustion recovery fix. recompute_ready's new circuit-breaker guard resolved its effective limit from per-task max_retries -> DEFAULT_FAILURE_LIMIT, skipping the dispatcher's configured kanban.failure_limit. _record_task_failure resolves max_retries -> failure_limit(config) -> DEFAULT, so the two disagreed whenever an operator set kanban.failure_limit != 2: - config > 2: a task could get stuck at DEFAULT(2) before reaching its allowed retry count. - config < 2: a task the breaker already blocked could be auto-recovered back to ready, defeating the stricter limit. Thread the dispatcher's failure_limit through dispatch_once into recompute_ready so the guard and the breaker share one resolution order. Updated test_circuit_breaker_block_still_auto_promotes (it asserted a failures=5 block auto-recovers and resets the counter — that's the pre-#35072 behavior the loop fix removes); it now exercises a below-limit transient block, with the at-limit case covered in test_kanban_db.py. Added two tests for the config-tier and per-task override resolution.
This commit is contained in:
+20
-8
@@ -2591,7 +2591,9 @@ def _has_sticky_block(conn: sqlite3.Connection, task_id: str) -> bool:
|
||||
return bool(row) and row["kind"] == "blocked"
|
||||
|
||||
|
||||
def recompute_ready(conn: sqlite3.Connection) -> int:
|
||||
def recompute_ready(
|
||||
conn: sqlite3.Connection, failure_limit: int = None,
|
||||
) -> int:
|
||||
"""Promote ``todo`` tasks to ``ready`` when all parents are ``done`` or ``archived``.
|
||||
|
||||
Returns the number of tasks promoted. Safe to call inside or outside
|
||||
@@ -2606,12 +2608,22 @@ def recompute_ready(conn: sqlite3.Connection) -> int:
|
||||
``kanban_unblock`` (#28712).
|
||||
|
||||
2. The task's ``consecutive_failures`` has reached the effective
|
||||
failure limit (per-task ``max_retries`` or
|
||||
``DEFAULT_FAILURE_LIMIT``). This prevents infinite retry
|
||||
loops when a task repeatedly exhausts its iteration budget:
|
||||
without this guard the counter would reset on every recovery
|
||||
cycle and the circuit breaker could never trip.
|
||||
failure limit. This prevents infinite retry loops when a task
|
||||
repeatedly exhausts its iteration budget: without this guard the
|
||||
counter would reset on every recovery cycle and the circuit
|
||||
breaker could never trip (#35072).
|
||||
|
||||
The effective failure limit resolves in the same order as the
|
||||
circuit breaker in ``_record_task_failure`` so the two never
|
||||
disagree about when a task is permanently blocked:
|
||||
|
||||
1. per-task ``max_retries`` if set
|
||||
2. caller-supplied ``failure_limit`` (the dispatcher passes the
|
||||
``kanban.failure_limit`` config value through ``dispatch_once``)
|
||||
3. ``DEFAULT_FAILURE_LIMIT``
|
||||
"""
|
||||
if failure_limit is None:
|
||||
failure_limit = DEFAULT_FAILURE_LIMIT
|
||||
promoted = 0
|
||||
with write_txn(conn):
|
||||
todo_rows = conn.execute(
|
||||
@@ -2647,7 +2659,7 @@ def recompute_ready(conn: sqlite3.Connection) -> int:
|
||||
task_limit = row["max_retries"]
|
||||
effective_limit = (
|
||||
int(task_limit) if task_limit is not None
|
||||
else DEFAULT_FAILURE_LIMIT
|
||||
else int(failure_limit)
|
||||
)
|
||||
if failures >= effective_limit:
|
||||
continue
|
||||
@@ -5577,7 +5589,7 @@ def dispatch_once(
|
||||
if _crash_auto_blocked:
|
||||
result.auto_blocked.extend(_crash_auto_blocked)
|
||||
result.timed_out = enforce_max_runtime(conn)
|
||||
result.promoted = recompute_ready(conn)
|
||||
result.promoted = recompute_ready(conn, failure_limit=failure_limit)
|
||||
|
||||
# Count tasks already running so max_spawn enforces concurrency rather
|
||||
# than a per-tick spawn budget. See the docstring above for the full
|
||||
|
||||
Reference in New Issue
Block a user