fix(memory): run end-of-turn sync off the turn thread (#41945)
A misconfigured/slow external memory provider could hold the agent in the 'running' state for minutes after the final response was delivered. MemoryManager.sync_all / queue_prefetch_all looped provider.sync_turn / queue_prefetch INLINE on the turn-completion path; a provider making a blocking network/daemon call (a broken Hindsight daemon was observed blocking ~298s before failing) blocked run_conversation from returning. Because every interface (CLI, TUI, gateway) marks the agent 'running' until run_conversation returns, the agent stayed busy for the full block and any follow-up message triggered an aggressive interrupt that dropped the message. Dispatch provider sync/prefetch to a lazily-created single-worker background executor. sync_all / queue_prefetch_all return immediately; work completes (or fails, logged) in the background. A single worker serializes writes so turn N lands before turn N+1. flush_pending() provides a barrier for session boundaries and deterministic tests. shutdown_all() drains the executor with a bounded timeout so a wedged provider can never hang teardown. Builtin-only / no-provider sessions spawn no executor (zero new threads in the common case).
This commit is contained in:
@@ -229,6 +229,7 @@ class TestMemoryManager:
|
||||
mgr.add_provider(p2)
|
||||
|
||||
mgr.queue_prefetch_all("next turn")
|
||||
mgr.flush_pending(timeout=5)
|
||||
assert p1.queued_prefetches == ["next turn"]
|
||||
assert p2.queued_prefetches == ["next turn"]
|
||||
|
||||
@@ -240,6 +241,7 @@ class TestMemoryManager:
|
||||
mgr.add_provider(p2)
|
||||
|
||||
mgr.sync_all("user msg", "assistant msg")
|
||||
mgr.flush_pending(timeout=5)
|
||||
assert p1.synced_turns == [("user msg", "assistant msg")]
|
||||
assert p2.synced_turns == [("user msg", "assistant msg")]
|
||||
|
||||
@@ -253,7 +255,7 @@ class TestMemoryManager:
|
||||
]
|
||||
|
||||
mgr.sync_all("user msg", "assistant msg", session_id="sess-1", messages=messages)
|
||||
|
||||
mgr.flush_pending(timeout=5)
|
||||
assert p.synced_turns == [("user msg", "assistant msg", "sess-1", messages)]
|
||||
|
||||
def test_sync_all_omits_messages_for_legacy_provider(self):
|
||||
@@ -262,7 +264,7 @@ class TestMemoryManager:
|
||||
mgr.add_provider(p)
|
||||
|
||||
mgr.sync_all("user msg", "assistant msg", messages=[{"role": "tool"}])
|
||||
|
||||
mgr.flush_pending(timeout=5)
|
||||
assert p.synced_turns == [("user msg", "assistant msg")]
|
||||
|
||||
def test_sync_failure_doesnt_block_others(self):
|
||||
@@ -275,6 +277,7 @@ class TestMemoryManager:
|
||||
mgr.add_provider(p2)
|
||||
|
||||
mgr.sync_all("user", "assistant")
|
||||
mgr.flush_pending(timeout=5)
|
||||
# p1 failed but p2 still synced
|
||||
assert p2.synced_turns == [("user", "assistant")]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user