perf(/model): prewarm picker provider-models cache in background (#39847)

* fix: respect disabled auto-compaction on context overflow

Port from anomalyco/opencode#30749.

When compression.enabled is false, NO automatic compaction trigger may
fire. The proactive token-threshold paths (preflight + post-response
should_compress gate) already honoured the setting, but the three
provider-overflow recovery paths in the agent loop — long-context-tier
429, 413 payload-too-large, and context-overflow — called
_compress_context() unconditionally, silently compressing and rotating
the session against the user's explicit choice.

Add a single guard at the top of the overflow-recovery dispatch: when
compression is disabled and the error is one of those three overflow
classes, surface a terminal error (compaction_disabled: True) telling the
user to /compress manually, /new, switch to a larger-context model, or
reduce attachments. Manual /compress (force=True) is unaffected — it never
enters this loop.

Tests: new TestOverflowWithCompactionDisabled (413 + 400 overflow don't
compress when disabled; control case still compresses when enabled).
Existing overflow-recovery tests updated to enable compaction explicitly
(they verify the recovery fires); fixture defaults flipped to True to
match production (compression.enabled defaults to True).

* perf(/model): prewarm picker provider-models cache in background

The no-args /model picker calls list_authenticated_providers(), which
fetches each authenticated provider's live /v1/models list serially. On a
cold or stale (>1h TTL) cache that blocks ~1.5s on the user's critical path
the first time /model is opened in a session.

Warm that exact path off-thread during the idle window right after the CLI
banner is shown: a once-per-process daemon thread runs
list_authenticated_providers() to populate provider_models_cache.json for
every authed provider. By the time the user types /model, the picker hits
the warm disk cache (~136ms vs ~1500ms).

Process-level Event guard (mirrors run_agent's _openrouter_prewarm_done)
ensures at most one thread per process; fully exception-isolated so an
offline/no-creds provider can never affect the session.
This commit is contained in:
Teknium
2026-06-05 06:55:09 -07:00
committed by GitHub
parent ca1fb32c26
commit 9ca11b35d5
3 changed files with 126 additions and 0 deletions
+56
View File
@@ -1117,6 +1117,62 @@ def switch_model(
# Authenticated providers listing (for /model no-args display)
# ---------------------------------------------------------------------------
# Process-level guard so the picker prewarm thread is spawned at most once per
# process — mirrors run_agent's _openrouter_prewarm_done. Without a guard a
# long-lived process (or repeated triggers) would leak one OS thread per call.
import threading as _threading # noqa: E402
_picker_prewarm_done = _threading.Event()
def prewarm_picker_cache_async() -> Optional["_threading.Thread"]:
"""Warm the provider-models disk cache in a background daemon thread.
The no-args ``/model`` picker calls ``list_authenticated_providers()``,
which fetches each authenticated provider's live ``/v1/models`` list on a
cold/stale cache. Those fetches are independent HTTP round-trips but run
serially, so the first ``/model`` open in a session (or any open after the
1h cache TTL expires) blocks ~1-2s on the user's critical path.
This pre-warms that exact path off-thread during idle session time: it
runs ``list_authenticated_providers()`` once, which populates
``provider_models_cache.json`` for every authed provider. By the time the
user types ``/model``, the picker hits the warm disk cache and renders in
~100ms.
Fire-and-forget. Process-level Event guard ensures it runs at most once.
Fully exception-isolated — a slow or offline provider can never affect the
session. Returns the spawned thread (for tests) or None if already warmed.
"""
if _picker_prewarm_done.is_set():
return None
_picker_prewarm_done.set()
def _warm() -> None:
try:
from hermes_cli.inventory import load_picker_context
ctx = load_picker_context()
# Calling this is what populates cached_provider_model_ids() ->
# provider_models_cache.json for each authed provider. We discard
# the result; the side effect (warm disk cache) is the point.
list_authenticated_providers(
current_provider=ctx.current_provider,
current_base_url=ctx.current_base_url,
current_model=ctx.current_model,
user_providers=ctx.user_providers,
custom_providers=ctx.custom_providers,
max_models=50,
)
except Exception:
# Best-effort warmup — never surface errors into the session.
logger.debug("picker cache prewarm failed", exc_info=True)
t = _threading.Thread(target=_warm, daemon=True, name="picker-cache-prewarm")
t.start()
return t
def list_authenticated_providers(
current_provider: str = "",
current_base_url: str = "",