Fearvox
4b06c98fe4
fix(gateway): close ResponseStore + dispose unowned adapter on reconnect failure
...
Three separate code paths in the gateway's platform reconnect loop
leaked file descriptors every retry, exhausting the default 2560-fd
ulimit in ~12 hours of continuous failure and turning the gateway
into a zombie that raises OSError: [Errno 24] on every open() (#37011 ).
Root cause:
* APIServerAdapter.__init__ opens a ResponseStore SQLite connection
that holds 2 fds (db file + WAL sidecar).
* APIServerAdapter.disconnect() previously only stopped the aiohttp
web server — the ResponseStore connection was never closed.
* The reconnect watcher in _platform_reconnect_watcher constructs a
fresh adapter on every retry attempt. When the connect call fails
(3 paths: non-retryable error, retryable error, exception during
connect) the adapter is dropped without ever being installed on
self.adapters, so nothing else calls its disconnect(). Result: the
2 ResponseStore fds stay open until GC sweeps the unreachable
object, which Python's cyclic GC does not do promptly for
asyncio-bound native handles.
2 fds × 1 retry × (3600s / 300s backoff cap) ≈ 12 fds/hour.
2560 fds / 12 fds/hr ≈ 12h to ulimit exhaustion.
Fix:
* APIServerAdapter.disconnect() now also calls
self._response_store.close() (with a try/except so a SQLite
close failure doesn't abort the aiohttp teardown).
* New module-level helper _dispose_unused_adapter(adapter) in
gateway/run.py that calls adapter.disconnect() and swallows
any exception (so half-constructed adapters whose __init__
crashed don't kill the watcher loop).
* _platform_reconnect_watcher calls _dispose_unused_adapter() in
all three failure paths: non-retryable, retryable, and the
except Exception arm. adapter = None is initialized
before the try so the except arm can see the partial
construction.
Tests:
* New file tests/gateway/test_platform_reconnect_fd_leak.py with
7 regression tests covering all three failure paths, the
_dispose_unused_adapter helper (None + raising-disconnect cases),
and the APIServerAdapter ResponseStore close behavior (success +
close-exception cases). The _CountingAdapter fixture tracks
disconnect() invocations and an _open_fds counter that is
decremented on dispose, so the assertion is the literal
observable behavior of the leak.
Refs:
- Closes #37011 (the original fd-leak report)
- Supersedes #37018 , #37110 , #37238 , #37260 , #37394 (7 competing
open PRs all addressing the same root cause from different angles;
none of them rebased cleanly against current main, and none
covered all three failure paths in one fix with regression tests
for both the watcher and the platform-level close behavior)
2026-06-02 17:27:44 -07:00
..
2026-05-20 13:00:57 -07:00
2026-04-30 01:19:34 -07:00
2026-05-28 22:26:25 -07:00
2026-04-30 20:30:31 -07:00
2026-05-31 21:05:53 -07:00
2026-05-28 22:26:25 -07:00
2026-05-24 01:31:45 -07:00
2026-05-29 22:29:48 -07:00
2026-05-28 22:26:25 -07:00
2026-05-17 02:29:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-20 04:16:13 -07:00
2026-04-12 18:03:16 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-06-02 16:55:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-24 01:31:45 -07:00
2026-06-01 18:52:05 -07:00
2026-05-18 21:38:05 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-19 01:48:42 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-23 19:38:33 -05:00
2026-05-18 10:19:40 -07:00
2026-04-24 02:55:43 -07:00
2026-04-24 02:55:43 -07:00
2026-05-25 01:44:46 -07:00
2026-05-28 22:26:25 -07:00
2026-06-01 19:51:21 -07:00
2026-05-24 01:31:45 -07:00
2026-06-02 00:33:50 -07:00
2026-04-22 11:59:39 -07:00
2026-05-29 19:06:05 -07:00
2026-05-25 14:54:02 -07:00
2026-05-09 11:04:46 -07:00
2026-05-28 22:26:25 -07:00
2026-04-24 03:04:42 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 04:52:02 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-27 03:35:33 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-22 14:21:41 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 04:52:02 -07:00
2026-05-17 02:29:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-06-01 20:18:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-30 07:28:06 -07:00
2026-05-30 18:58:46 -07:00
2026-05-30 07:38:03 -07:00
2026-05-28 22:26:25 -07:00
2026-05-23 02:40:33 -07:00
2026-05-24 06:54:16 -07:00
2026-05-28 22:26:25 -07:00
2026-04-30 20:30:31 -07:00
2026-04-17 19:04:11 -07:00
2026-05-28 22:26:25 -07:00
2026-05-07 05:09:39 -07:00
2026-05-31 00:50:19 -07:00
2026-05-28 22:26:25 -07:00
2026-05-27 02:41:24 -07:00
2026-05-28 22:26:25 -07:00
2026-05-31 21:05:53 -07:00
2026-05-07 06:31:08 -07:00
2026-05-07 17:33:09 -07:00
2026-05-11 07:06:27 -07:00
2026-05-24 17:58:52 -07:00
2026-05-03 08:47:49 -07:00
2026-04-17 01:05:09 -07:00
2026-05-28 22:26:25 -07:00
2026-04-15 17:54:16 -07:00
2026-05-28 22:26:25 -07:00
2026-05-24 01:31:45 -07:00
2026-05-28 22:26:25 -07:00
2026-06-01 20:30:24 -07:00
2026-05-28 22:26:25 -07:00
2026-04-27 19:09:32 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-24 15:03:27 -07:00
2026-05-29 03:58:45 -07:00
2026-04-27 21:22:44 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-24 18:05:33 -07:00
2026-05-26 14:28:51 -07:00
2026-05-28 22:26:25 -07:00
2026-06-01 00:00:26 -07:00
2026-05-16 12:55:23 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-25 15:22:23 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-30 20:26:35 -07:00
2026-05-01 13:33:06 -07:00
2026-05-29 13:17:46 -07:00
2026-05-28 22:26:25 -07:00
2026-04-30 05:00:25 -07:00
2026-04-18 19:32:26 -07:00
2026-04-19 03:03:57 -07:00
2026-06-02 05:52:54 -07:00
2026-05-29 17:36:58 +00:00
2026-06-01 00:00:26 -07:00
2026-05-24 04:25:20 -07:00
2026-05-28 22:26:25 -07:00
2026-06-02 17:27:44 -07:00
2026-05-30 07:33:34 -07:00
2026-05-28 22:26:25 -07:00
2026-04-29 21:56:51 -07:00
2026-05-07 05:04:37 -07:00
2026-04-24 03:02:03 -07:00
2026-05-28 22:26:25 -07:00
2026-05-31 17:46:56 -05:00
2026-05-28 22:26:25 -07:00
2026-04-30 20:40:46 -07:00
2026-04-29 21:07:47 -07:00
2026-05-22 14:21:41 -07:00
2026-05-28 22:26:25 -07:00
2026-04-21 13:33:02 -07:00
2026-05-24 01:31:45 -07:00
2026-05-31 21:05:53 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-25 01:33:32 -07:00
2026-05-20 13:00:57 -07:00
2026-05-27 05:21:53 -07:00
2026-05-28 22:26:25 -07:00
2026-05-27 05:21:53 -07:00
2026-05-30 07:38:03 -07:00
2026-05-15 14:32:14 -07:00
2026-05-28 22:26:25 -07:00
2026-04-26 07:15:23 -07:00
2026-05-23 02:27:08 -07:00
2026-05-07 05:49:16 -07:00
2026-04-28 06:50:04 -07:00
2026-05-08 18:50:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:25:15 -07:00
2026-05-29 12:27:49 -07:00
2026-05-17 02:29:41 -07:00
2026-05-09 14:18:20 +03:00
2026-05-28 22:26:25 -07:00
2026-04-16 02:26:14 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-26 18:49:48 -07:00
2026-05-23 02:25:53 -07:00
2026-04-26 19:01:50 -07:00
2026-05-27 02:41:24 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-20 13:00:57 -07:00
2026-04-12 13:05:56 -07:00
2026-04-21 00:54:46 -07:00
2026-05-28 22:26:25 -07:00
2026-05-10 09:01:51 -07:00
2026-04-27 06:41:16 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-31 17:46:56 -05:00
2026-06-01 16:36:43 -07:00
2026-05-28 22:26:25 -07:00
2026-04-26 18:25:41 -07:00
2026-05-07 06:54:29 -07:00
2026-05-31 17:46:56 -05:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-29 19:14:37 -07:00
2026-05-29 17:36:58 +00:00
2026-04-18 04:17:18 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-31 09:29:03 -07:00
2026-05-18 21:51:39 -07:00
2026-06-01 17:31:32 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 03:15:19 -07:00
2026-06-02 00:33:50 -07:00
2026-05-18 22:59:40 -07:00
2026-05-28 22:26:25 -07:00
2026-05-25 16:23:24 +00:00
2026-05-08 11:18:14 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:08:08 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:51:35 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:01:31 -07:00
2026-05-28 22:26:25 -07:00
2026-05-25 23:16:00 -07:00
2026-05-18 22:34:48 -07:00
2026-06-01 20:18:41 -07:00
2026-05-18 22:59:40 -07:00
2026-04-20 00:10:22 -07:00
2026-05-30 01:41:33 -07:00
2026-05-02 01:55:04 -07:00
2026-05-18 22:14:45 -07:00
2026-05-27 02:41:24 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-12 17:02:29 -07:00
2026-05-18 21:52:39 -07:00
2026-05-09 11:10:36 -07:00
2026-06-02 00:33:50 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:28:21 -07:00
2026-05-23 02:42:10 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-31 21:05:53 -07:00
2026-05-28 23:25:52 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-06-01 17:31:32 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-18 22:43:14 -07:00
2026-05-02 02:00:09 -07:00
2026-06-01 02:04:14 -07:00
2026-04-22 16:23:21 -07:00
2026-05-31 21:05:53 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-27 12:37:33 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-12 18:05:41 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-04-19 22:45:08 -07:00
2026-05-24 01:30:47 -07:00
2026-06-01 19:20:36 -07:00
2026-06-01 17:31:40 -07:00
2026-05-15 14:32:14 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-28 22:26:25 -07:00
2026-05-30 07:33:15 -07:00
2026-05-28 22:26:25 -07:00
2026-04-13 16:32:04 -07:00