fix(model_metadata): consult DEFAULT_CONTEXT_LENGTHS before 256K fallback on custom endpoints

Problem: get_model_context_length() had an early return at the end of the
custom-endpoint probe branch (step 3) that returned DEFAULT_FALLBACK_CONTEXT
(256K) without ever consulting the hardcoded DEFAULT_CONTEXT_LENGTHS catalog
(step 8). Models served through a custom/proxied gateway (e.g. corporate
Anthropic proxy) that didn't expose Ollama or local-server endpoints would
hit this path and get capped at 256K, even when the model name clearly
matched a known entry in the catalog (e.g. claude-opus-4-8 → 1M).

Changes:
- agent/model_metadata.py: Before returning DEFAULT_FALLBACK_CONTEXT at the
  end of the custom-endpoint branch, consult DEFAULT_CONTEXT_LENGTHS using
  the same longest-key-first fuzzy matching as step 8. Only fall through
  to 256K if no catalog entry matches.
- tests/agent/test_model_metadata.py: Updated existing test and added new
  test covering the custom-endpoint → catalog fallback behavior.

Fixes #38865
This commit is contained in:
islam666
2026-06-07 21:50:57 -07:00
committed by Teknium
parent f1d3afb151
commit 2e61de0638
2 changed files with 89 additions and 3 deletions
+20
View File
@@ -1684,6 +1684,26 @@ def get_model_context_length(
"in config.yaml to override.",
model, base_url, f"{DEFAULT_FALLBACK_CONTEXT:,}",
)
# 3b. Before falling back to the hard 256K default, consult the
# hardcoded catalog as a last resort. A proxied/custom Anthropic
# gateway (e.g. corporate proxy) fails the Ollama/local probes
# above, but the model name may still match an entry in
# DEFAULT_CONTEXT_LENGTHS (e.g. "claude-opus-4-8" → 1M).
# Without this, the early return here short-circuits the catalog
# lookup at step 8 and silently caps context at 256K.
model_lower = model.lower()
for default_model, length in sorted(
DEFAULT_CONTEXT_LENGTHS.items(),
key=lambda x: len(x[0]),
reverse=True,
):
if default_model in model_lower:
logger.info(
"Using hardcoded context length %s for model %r "
"(custom endpoint, catalog match on %r)",
f"{length:,}", model, default_model,
)
return length
return DEFAULT_FALLBACK_CONTEXT
# 4. Anthropic /v1/models API (only for regular API keys, not OAuth)