fix(kanban): bridge worker runtime activity to board heartbeat (#31752)
The dispatcher watchdog (release_stale_claims) reads tasks.last_heartbeat_at to decide whether to reclaim a running task. The agent maintains its own in-process `_last_activity_ts` for every chunk/tool result, but those liveness ticks never reach the board unless the model explicitly calls the `kanban_heartbeat` tool — so a worker actively executing a long run without tool-level heartbeats can be reclaimed mid-flight as 'stale', returning the task to ready and orphaning the in-flight worker's progress. Fix: in `_touch_activity` (the canonical 'we just did work' hook in run_agent.py), call a new `heartbeat_current_worker_from_env` helper in `tools/kanban_tools.py` that: - No-ops outside dispatcher-spawned worker context (no HERMES_KANBAN_TASK). - Rate-limited to one DB write per 60s (runtime activity ticks too often to faithfully mirror; we just need the watchdog to see liveness). - Best-effort: never raises. heartbeat_claim + heartbeat_worker calls are individually try/except'd; any DB error logs at debug and returns. - Uses worker env identity: HERMES_KANBAN_TASK + HERMES_KANBAN_RUN_ID + HERMES_KANBAN_CLAIM_LOCK (all pinned by the dispatcher at spawn time). - No durable note on auto-heartbeats — that's reserved for the explicit `kanban_heartbeat` tool which carries a model-supplied note. The explicit `kanban_heartbeat` tool stays available unchanged for workers that want to attach a note or pre-emptively extend a claim across a known-long single tool call. Co-authored-by: faisfamilytravel <223516181+faisfamilytravel@users.noreply.github.com>
This commit is contained in:
committed by
Teknium
co-authored by
faisfamilytravel
parent
40217aa194
commit
bc31ee5cf8
+18
-1
@@ -2144,9 +2144,26 @@ class AIAgent:
|
||||
return apply_pending_steer_to_tool_results(self, messages, num_tool_msgs)
|
||||
|
||||
def _touch_activity(self, desc: str) -> None:
|
||||
"""Update the last-activity timestamp and description (thread-safe)."""
|
||||
"""Update the last-activity timestamp and description (thread-safe).
|
||||
|
||||
Also bridges to the kanban board's heartbeat fields when this
|
||||
process is a dispatcher-spawned worker (HERMES_KANBAN_TASK set),
|
||||
so the dispatcher watchdog doesn't reclaim an actively-running
|
||||
worker as stale (#31752). Bridge is rate-limited (60s) and
|
||||
best-effort — it never raises into the agent loop.
|
||||
"""
|
||||
self._last_activity_ts = time.time()
|
||||
self._last_activity_desc = desc
|
||||
if os.environ.get("HERMES_KANBAN_TASK"):
|
||||
try:
|
||||
from tools.kanban_tools import heartbeat_current_worker_from_env
|
||||
heartbeat_current_worker_from_env()
|
||||
except Exception:
|
||||
# Never let the bridge break the agent loop. The function
|
||||
# already swallows exceptions internally; this outer guard
|
||||
# covers import-time failures (kanban_tools unavailable,
|
||||
# etc.) on niche deployment surfaces.
|
||||
pass
|
||||
|
||||
def _capture_rate_limits(self, http_response: Any) -> None:
|
||||
"""Parse x-ratelimit-* headers from an HTTP response and cache the state.
|
||||
|
||||
Reference in New Issue
Block a user