Original commit 13c3d4b4e by kchantharuan touched __init__ and
_apply_client_headers_for_base_url in pre-refactor run_agent.py. Re-applied to:
- __init__: agent/agent_init.py (3 hunks — NVIDIA branch + _custom_headers
fallback in routed-client and fallback-client paths)
- _apply_client_headers_for_base_url: still in run_agent.py (1 hunk)
build_nvidia_nim_headers was already present in agent/auxiliary_client.py
from the prior merge — no additional port needed.
Co-authored-by: kchantharuan <kchantharuan@nvidia.com>
Original commit b62c99797 by Jaaneek targeted six locations in
pre-refactor run_agent.py. Re-applied to the extracted post-PR locations:
- api_mode dispatch → agent/agent_init.py
- is_xai_responses build_api_kwargs → agent/chat_completion_helpers.py
- codex_auth_retry block + 401 hint → agent/conversation_loop.py
- _try_refresh_codex_client_credentials body → run_agent.py (kept)
The non-run_agent.py portions of the commit (auxiliary_client, codex
transport, hermes_cli/auth, tools/xai_http, tests, docs) merged cleanly
from main via the prior merge commit.
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
previously only checked provider ID and
base URL. When kimi-k2.6 is served via ollama-cloud (or any third-party
provider), provider is not 'kimi-coding' and base URL is not
api.kimi.com — so reasoning_content pad was never injected. This caused
HTTP 400 from Ollama Cloud's Go backend: 'invalid message content type:
map[string]interface {}'.
Fix: add model-name detection ('kimi' in model.lower()) so any route
serving a kimi model gets the required reasoning_content echo-back.
Refs the 400/401 Telegram errors where kimi-k2.6 via ollama-cloud
consistently failed after tool-call turns.
(cherry picked from commit 9a9f8a6d99)
Four fixes from PR #27248 review:
1. **__init__ forwarder is now keyword-forwarded** (daimon-nous review).
Previously the run_agent.AIAgent.__init__ wrapper forwarded all 64
params positionally to agent.agent_init.init_agent, so adding a
65th param on main would require three lockstep edits (signature,
init_agent signature, forwarder call) or silently shift every value.
Keyword forwarding makes this trivially safe — adding a param now
only needs the two signatures and one extra keyword line.
2. **Drop dead _ra() in agent/codex_runtime.py** (daimon-nous + Copilot).
The lazy run_agent reference was defined but never called inside
this module — the codex paths use agent.* accessors only.
3. **Drop unused imports in agent/codex_runtime.py** (Copilot):
contextvars, threading, time, uuid, Optional. Carried over from
run_agent.py during the original extraction.
4. **Tighten three source-introspection test guards** (Copilot):
- test_memory_nudge_counter_hydration.py — was scanning the
concatenated source of run_agent.py + agent/conversation_loop.py
and matching self.X or agent.X form. Now asserts the
hydration block lives in agent/conversation_loop.py specifically
with the agent.X form — the body never moves back, so if it
ever drifts a future re-introduction fails the guard.
- test_run_agent.py::TestMemoryNudgeCounterPersistence — anchor on
agent.iteration_budget = IterationBudget exactly (was just
iteration_budget = IterationBudget) so an unrelated identifier
ending in iteration_budget can't match.
- test_run_agent.py::TestMemoryProviderTurnStart — assert the
agent._user_turn_count form directly (the extracted body uses
agent.X, not self.X — accepting either was a transitional fudge).
- test_jsondecodeerror_retryable.py — scan agent/conversation_loop.py
only, not the concatenation.
Not addressed in this commit:
* Pre-existing bugs in agent/tool_executor.py (heartbeat index
mismatch when calls are blocked, _current_tool clobber in result
loop, blocked-counted-as-completed in spinner summary, dead
result_preview computation). These were preserved byte-for-byte from
the original _execute_tool_calls_concurrent — worth a separate
follow-up PR with proper tests.
* _OpenAIProxy.__instancecheck__ concern — pre-existing, not flagged
by any of the original test patches (nothing actually does
isinstance(x, OpenAI) against the proxy instance).
* agent_init.py:949 mem_config potential NameError — pre-existing;
only triggers if _agent_cfg.get('memory', {}) itself raises, which
it can't with a stock dict.
tests/run_agent/ + tests/agent/: 4313 passed, 1 pre-existing
test_auxiliary_client failure (unchanged).
run_agent.py: 3821 -> 3937 lines (+116 from the keyword-forwarded
init call's verbosity). Final: 16083 -> 3937 (-12146, 75% reduction).
previously only checked provider ID and
base URL. When kimi-k2.6 is served via ollama-cloud (or any third-party
provider), provider is not 'kimi-coding' and base URL is not
api.kimi.com — so reasoning_content pad was never injected. This caused
HTTP 400 from Ollama Cloud's Go backend: 'invalid message content type:
map[string]interface {}'.
Fix: add model-name detection ('kimi' in model.lower()) so any route
serving a kimi model gets the required reasoning_content echo-back.
Refs the 400/401 Telegram errors where kimi-k2.6 via ollama-cloud
consistently failed after tool-call turns.
Pass skip_memory=True to the AIAgent constructor used by
_spawn_background_review() so the review fork's __init__ no longer
rebuilds a _memory_manager wired to honcho / mem0 / supermemory /
etc. under the parent's session_id.
Before this change, the review fork ingested its harness prompt
(the 'Review the conversation above and update the skill library...'
text) into the user's real memory namespace via three sites in
run_conversation():
- on_turn_start(turn_count, prompt) cadence + turn-message
- prefetch_all(prompt) recall query
- sync_all(prompt, review_output, ...) harness + review output
recorded as a
(user, assistant) pair
Built-in MEMORY.md / USER.md state is still rebound from the parent
right after construction, so memory(action='add') writes from the
review continue to land on disk; only the external-plugin side
effects are removed.
Reported by @Utku.
The same root cause as the auxiliary compression fix (commit 7becb19):
get_model_context_length() is called without custom_providers, so per-model
context_length overrides are silently skipped. The fallback activation path
(_try_activate_fallback) had the same missing parameter.
When the agent switches to a fallback provider, the fallback model would use
the models.dev value (e.g. 204800 for NVIDIA NIM minimax-m2.7) instead of
the user-configured one in custom_providers (e.g. 196608) — a subtle
discrepancy that could cause the fallback model to run with an incorrect
context window, leading to truncated messages or failed API requests when
the model does not support the detected length.
Fix: pass self._custom_providers to get_model_context_length() so the
fallback path sees the same per-model overrides as the main model path.
The largest method left on AIAgent (60+ parameters, the entire startup
sequence — credential resolution, provider auto-detection, context
engine bootstrap, memory store hydration, plugin lifecycle hooks)
moves into agent/agent_init.py.
AIAgent.__init__ is now a thin wrapper that calls
agent.agent_init.init_agent(self, ...) with the original full
parameter list preserved.
Module-level run_agent names referenced in the body (_openrouter_prewarm_done,
_qwen_portal_headers, _routermint_headers, _hermes_home, OpenAI,
get_tool_definitions, check_toolset_requirements) are resolved through
_ra() so test patches on those names keep working. agent_init's logger
warnings are routed via _ra().logger so tests patching run_agent.logger
capture them (TestStringKSuffixContextLengthWarns,
TestCustomProvidersInvalidContextLengthWarns).
Live E2E reconfirmed on three model paths (openai/gpt-5.4,
anthropic/claude-sonnet-4.6, moonshotai/kimi-k2-thinking).
tests/run_agent/ + tests/agent/: 4313 passed (same pre-existing
test_auxiliary_client failure).
run_agent.py: 5944 -> 4564 lines (-1380).
Total reduction since baseline: 16083 -> 4564 (-11519, 72%).
The 3,877-line run_conversation body — the agent loop itself — moves out
of run_agent.py into a dedicated module. AIAgent.run_conversation is
now a thin forwarder that delegates to agent.conversation_loop.run_conversation
with the AIAgent instance as the first argument.
This is the largest single extraction in the run_agent.py refactor.
The body keeps all 163 self.X references intact (rewritten as agent.X),
all nested closures, all retry/backoff/compression machinery. Symbols
that tests or callers patch on run_agent (_set_interrupt,
handle_function_call, AIAgent class attrs) are resolved through _ra()
inside the extracted module so the patch surface is preserved.
Five tests doing inspect.getsource(AIAgent.run_conversation) updated to
scan agent.conversation_loop.run_conversation. Two source-introspection
tests (TestMemoryNudgeCounterPersistence, TestMemoryProviderTurnStart)
updated to accept either self.X (legacy) or agent.X (extracted
form) in the matched assertions.
Live E2E verified on three model paths:
* openai/gpt-5.4 (OpenAI chat completions via OpenRouter)
* anthropic/claude-sonnet-4.6 (Anthropic Messages via OpenRouter)
* moonshotai/kimi-k2-thinking (reasoning model, reasoning_content path)
Plus read_file tool execution, terminal tool, web_search.
tests/run_agent/ + tests/agent/: 4313 passed, 1 pre-existing failure
(test_auxiliary_client::test_custom_endpoint... — same as on main).
run_agent.py: 9800 -> 5944 lines (-3856).
Total reduction since baseline: 16083 -> 5944 (-10139, 63%).
The three big review-prompt strings (_MEMORY_REVIEW_PROMPT,
_SKILL_REVIEW_PROMPT, _COMBINED_REVIEW_PROMPT — 183 lines combined) move
out of the AIAgent class body and into agent/background_review.py where
they're consumed.
AIAgent re-exposes them as class attributes via 'from ... import' inside
the class body — Python binds those names into the class namespace so
existing AIAgent._MEMORY_REVIEW_PROMPT references keep working.
spawn_background_review_thread also falls back to the module-level
constants if an agent doesn't have the attribute (preserves the test
pattern of mocking these on the agent).
tests/run_agent/ + tests/agent/: 4313 passed (same pre-existing
test_auxiliary_client failure).
run_agent.py: 9986 -> 9800 lines (-186).
Move _interruptible_streaming_api_call out of run_agent.py — the biggest
single method in the file. Body lives next to interruptible_api_call
in agent/chat_completion_helpers.py so streaming + non-streaming code
share one home.
Nested closures (_call_chat_completions, _call_anthropic, the codex
stream branch) all come along with the body and still capture the
parent function's locals as expected.
AIAgent keeps a thin forwarder method. is_local_endpoint added to
the import block (used by the stream stale-timeout disable logic).
One source-introspection test in TestAnthropicInterruptHandler is
updated to scan agent.chat_completion_helpers.interruptible_streaming_api_call
instead of AIAgent._interruptible_streaming_api_call.
tests/run_agent/ + tests/agent/: 4312 passed (same pre-existing
test_auxiliary_client failure).
run_agent.py: 12277 -> 11385 lines (-892).
Move the two big tool-dispatch methods out of run_agent.py:
* execute_tool_calls_concurrent — 408-line concurrent path (interrupt
pre-flight, guardrail+plugin block, callback fan-out, ContextVar-
preserving ThreadPoolExecutor, periodic heartbeats for the gateway
inactivity monitor, per-tool result handling with subdir hints +
guardrail observations + checkpoint, /steer drain)
* execute_tool_calls_sequential — 441-line sequential path (the
original behavior used for single-tool batches and interactive
tools)
Both take the parent AIAgent as their first argument; AIAgent keeps
thin forwarders so call sites unchanged. handle_function_call is
routed through _ra() so tests that patch run_agent.handle_function_call
keep working. _set_interrupt likewise.
The AST guard in test_tool_executor_contextvar_propagation.py is
updated to scan both run_agent.py AND agent/tool_executor.py so it
still catches the executor.submit(_run_tool, ...) regression
regardless of which file the body lives in.
tests/run_agent/ + tests/agent/: 4313 passed (same pre-existing
test_auxiliary_client failure as before).
run_agent.py: 14309 -> 13461 lines (-848).
Move the background-review subsystem (the self-improvement loop — see the
README) out of run_agent.py into a dedicated module.
* summarize_background_review_actions — was the @staticmethod that builds
the user-facing action summary
* spawn_background_review_thread — builds the thread target + prompt;
the actual review loop body (forked AIAgent, runtime inheritance,
tool whitelist, suppression, teardown) lives in _run_review_in_thread
* build_memory_write_metadata — provenance for external memory mirrors
AIAgent keeps thin wrappers for backward compatibility AND because tests
patch run_agent.threading.Thread to assert lifecycle behavior — the
threading.Thread construction stays in AIAgent._spawn_background_review,
the inner work moves out.
tests/run_agent/ + tests/agent/: 4313 passed, 1 pre-existing failure
(test_auxiliary_client.py::test_custom_endpoint... — confirmed failing
on main before this change). 3 skipped.
run_agent.py: 15272 -> 14972 lines (-300).
Three small extractions into focused modules:
* agent/process_bootstrap.py — \_OpenAIProxy (lazy openai.OpenAI import),
\_SafeWriter (broken-pipe-resistant stdio wrapper), \_install_safe_stdio,
\_get_proxy_from_env, \_get_proxy_for_base_url. All process / IO bootstrap.
* agent/iteration_budget.py — IterationBudget class (thread-safe consume/
refund counter shared by parent agent and subagents).
run_agent re-exports every name so existing test patches like
patch('run_agent.OpenAI', ...) and 'from run_agent import IterationBudget'
keep working unchanged. Verified the patch-rebinding contract for OpenAI
explicitly.
tests/run_agent/ + tests/agent/test_gemini_fast_fallback.py:
1347 passed, 3 skipped.
run_agent.py: 15427 -> 15261 lines (-166).
Pull the 10 pure sanitization/repair helpers (\_sanitize_surrogates,
\_sanitize_structure_surrogates, \_sanitize_messages_surrogates,
\_escape_invalid_chars_in_json_strings, \_repair_tool_call_arguments,
\_strip_non_ascii, \_sanitize_messages_non_ascii, \_sanitize_tools_non_ascii,
\_strip_images_from_messages, \_sanitize_structure_non_ascii) and the
\_SURROGATE_RE constant out of run_agent.py into a new module.
These are stateless byte-walking helpers with no AIAgent dependency.
Backward compatibility: run_agent re-exports every name via a single
import block, so existing 'from run_agent import _sanitize_surrogates'
imports in tests and cli.py keep working unchanged. Same pattern the
file already uses for _summarize_user_message_for_log (codex_responses_adapter).
run_agent.py: 16077 -> 15682 lines (-395).
In long-lived interactive sessions, _try_activate_fallback() advances
_fallback_index before attempting client resolution. When resolution
fails (provider not configured, etc.) the function returns False without
ever setting _fallback_activated=True. _restore_primary_runtime() then
skips its reset block entirely (guarded by `if not _fallback_activated`),
leaving _fallback_index >= len(_fallback_chain) for all subsequent turns.
The eager-fallback guard at the top of the retry loop checks
`_fallback_index < len(_fallback_chain)`, so the condition fails silently
and no fallback is ever attempted again for that session.
Cron jobs spawn a fresh AIAgent per run and never hit this path, which is
why the same fallback chain works reliably for cron but not interactive.
Fix: reset _fallback_index=0 in the `not _fallback_activated` early-return
branch so every new turn starts with the full chain available.
Fixes#20465
xAI's Responses stream emits 'type=error' as the FIRST SSE frame when an
OAuth account is unsubscribed/exhausted or rejects the encrypted-reasoning
replay introduced in the May 2026 SuperGrok rollout. The SDK helper
raises RuntimeError(Expected to have received response.created before
error), which the caller correctly routes to
_run_codex_create_stream_fallback. The fallback then opens a new stream
that emits the same 'error' frame — but the fallback loop only handled
{response.completed, response.incomplete, response.failed} and silently
continue'd past 'error' events. Result: the loop fell off the end of
the stream and raised the useless 'fallback did not emit a terminal
response' RuntimeError, which the classifier marked retryable=True and
looped 3x before failing with no clue what went wrong.
Now: 'error' frames raise a synthesized _StreamErrorEvent with an OpenAI
SDK-shaped .body so _summarize_api_error, _extract_api_error_context,
_is_entitlement_failure, and classify_api_error all see the real
provider message. Users on unsubscribed accounts now see 'do not have
an active Grok subscription' once, not three RuntimeErrors.
Verified end-to-end: classifier returns reason=auth retryable=False;
entitlement detector matches even with status_code=None; summarizer
returns the full xAI message.
Tests: 4 new in TestCodexFallbackErrorEvent covering xAI subscription
message, dict-shaped events, summarizer integration, and the empty-stream
case (must still raise the original RuntimeError so 'truncated mid-flight'
stays distinguishable from 'provider rejected the call').
xAI announced on 2026-05-16 (https://x.ai/news/grok-hermes) that X Premium
subscriptions now work in Hermes Agent. The hint we shipped in PR #26644
asserted the opposite ("X Premium+ does NOT include xAI API access — only
standalone SuperGrok subscribers can use this provider"), which would now
misdirect Premium+ users who hit any other 403 (no Grok sub at all, wrong
tier, exhausted quota) into thinking they need to switch subscriptions
when their sub is in fact valid.
Remove _decorate_xai_entitlement_error and its two call sites in
_summarize_api_error. xAI's own body text already says "Manage subscriptions
at https://grok.com/?_s=usage" — surface that verbatim and let xAI's wording
do the diagnosis.
The _is_entitlement_failure guard (which prevents credential-pool refresh
loops on entitlement 403s) and the reasoning-replay gating for xai-oauth
are unrelated and untouched.
Update tests to assert the body still surfaces verbatim and that no
Hermes-side editorializing is appended.
Follow-up improvements on top of @konsisumer's cherry-picked fix for #10648:
1. Deprecation patterns required BOTH a product fingerprint ('gh-copilot') and
a deprecation marker. The previous list included 'copilot-cli' and bare
'deprecation', which would false-positive on stderr from the NEW
@github/copilot CLI — whose repo is literally github.com/github/copilot-cli
and which legitimately surfaces those substrings in its own messages.
2. Replace the deprecation hint. The user in #10648 installed
'gh extension install github/gh-copilot' (the deprecated extension)
thinking that's what ACP mode uses, when ACP actually spawns the new
'copilot' binary from '@github/copilot'. The hint now points users at the
correct install command ('npm install -g @github/copilot') with the new
CLI's repo URL, and demotes provider-switching to a fallback alternative.
3. Change _URL_TO_PROVIDER value for models.inference.ai.azure.com from the
'github-models' alias to the canonical 'copilot' provider id, matching the
convention used by every other entry in the table.
4. Sharpen the 413 hint message. The free tier's ~8K cap is below the
system-prompt floor, so this endpoint is fundamentally incompatible with
an agentic loop — not a 'use a different URL' problem.
Tests:
- New parametrized false-positive coverage for the new CLI's stderr shape.
- Updated assertion to require canonical 'copilot' provider mapping.
- All 14 deprecation/URL tests pass.
Address two blocking issues when using GitHub Copilot integrations:
1. ACP mode: detect the gh-copilot CLI deprecation error from stderr
and surface an actionable message with alternatives instead of
hanging or showing a cryptic error.
2. GitHub Models (Azure) 413: recognize models.inference.ai.azure.com
as a known GitHub Models URL, and print a targeted hint explaining
the hard 8K token limit that makes this endpoint incompatible with
Hermes' system prompt size.
Port from openai/codex#17667: MCP servers can now opt-in to parallel
tool execution by setting supports_parallel_tool_calls: true in their
config. This allows tools from the same server to run concurrently
within a single tool-call batch, matching the behavior already available
for built-in tools like web_search and read_file.
Previously all MCP tools were forced sequential because they weren't in
the _PARALLEL_SAFE_TOOLS set. Now _should_parallelize_tool_batch checks
is_mcp_tool_parallel_safe() which looks up the server's config flag.
Config example:
mcp_servers:
docs:
command: "docs-server"
supports_parallel_tool_calls: true
Changes:
- tools/mcp_tool.py: Track parallel-safe servers in _parallel_safe_servers
set, populated during register_mcp_servers(). Add is_mcp_tool_parallel_safe()
public API.
- run_agent.py: Add _is_mcp_tool_parallel_safe() lazy-import wrapper. Update
_should_parallelize_tool_batch() to check MCP tools against server config.
- 11 new tests covering the feature end-to-end.
- Updated MCP docs and config reference.
The #1 confusing cause of the xAI 403 (per Teknium): X Premium+
subscribers see Grok inside the X app and assume API access is
included. It is NOT — only standalone SuperGrok subscribers can use
xai-oauth with Hermes today. Without calling this out, every Premium+
user hits the 403 with no idea why.
PR #26666's neutral 4-cause list was correct but buried the most
common cause. Lead with the Premium+ gotcha, then list the other
possibilities (no subscription, wrong tier, exhausted quota) as
fallbacks. Same neutral framing — does not accuse anyone of being
unsubscribed.
PR #26644 confidently told users "xAI OAuth account lacks SuperGrok /
X Premium entitlement" on any 403 from xAI's permission-denied surface.
But that body is returned for at least four distinct causes that
Hermes cannot distinguish from the wire:
* Account has no Grok subscription at all
* Account has SuperGrok but the tier doesn't include the requested
model (e.g. grok-4.3 needs SuperGrok Heavy)
* Monthly quota for the subscribed tier is exhausted
* SuperGrok is active but the API access add-on isn't enabled
Don Piedro pushed back that he IS subscribed yet still hit this.
Picking the worst-case interpretation ("you're not subscribed")
reads as wrong and insulting to subscribers, and points them at a
fix they already did.
New wording lists all 4 possibilities and points at
https://grok.com/?_s=usage where the user can check which applies.
The detection logic and credential-pool short-circuit (PR #26664)
are unchanged — only the user-facing wording is rephrased.
Don Piedro's 18-minute hang on grok-4.3 traced to two issues PR #26644
didn't cover:
- _recover_with_credential_pool classifies 403 as FailoverReason.auth
and calls pool.try_refresh_current(). For xAI OAuth on an
unsubscribed account, refresh succeeds (mints a new token from the
same account) but the next API call 403s with the same entitlement
error. Result: infinite refresh → retry → 403 loop until Ctrl+C
(1133s in Don's log). New _is_entitlement_failure(error_context,
status_code) detects the subscription-shape body ("do not have an
active Grok subscription" / "out of available resources" + grok /
"does not have permission" + grok) and short-circuits recovery so
_summarize_api_error surfaces PR #26644's friendly hint.
- grok-4.3 resolved to 256k via the grok-4 catch-all in
DEFAULT_CONTEXT_LENGTHS. Per docs.x.ai/developers/models/grok-4.3
the model ships with 1M context. Add explicit grok-4.3 entry
before the grok-4 fallback (longest-first substring matching
ensures grok-4.3 and grok-4.3-latest both land on the new value).
Tests: 8 new (23 total in test_codex_xai_oauth_recovery.py).
E2E verified Don's 100-iteration loop bails out with 0 refresh calls
while genuine auth failures still refresh once and recover.
The cherry-picked PR #15251 from @tw2818 correctly identified the
DeepSeek 400 root cause but placed the fix in the legacy fallback path
of `build_kwargs`, which DeepSeek never reaches — DeepSeek has a
registered ProviderProfile and goes through `_build_kwargs_from_profile`
instead. The legacy-path block was therefore dead code.
This commit pivots the fix to where it actually fires:
- New `DeepSeekProfile` in `plugins/model-providers/deepseek/__init__.py`
overrides `build_api_kwargs_extras` to emit DeepSeek's expected wire
format (mirrors `KimiProfile`):
{"reasoning_effort": "<low|medium|high|max>",
"extra_body": {"thinking": {"type": "enabled" | "disabled"}}}
- Model gating: only `deepseek-v4-*` and `deepseek-reasoner` emit
thinking control. `deepseek-chat` (V3) is untouched — current behavior.
- Effort mapping: low/medium/high passthrough, xhigh/max → max, unset →
omitted (DeepSeek server applies its own default).
- Revert the legacy-path additions from PR #15251 — they were dead code,
and the `_copy_reasoning_content_for_api` strip block specifically
would have nullified the existing reasoning_content padding machinery
(`_needs_deepseek_tool_reasoning` → space-pad on replay) that the
active provider already relies on for replay correctness.
- Unit tests pin the wire-shape contract and the model gating rules
(26 tests, all passing). Existing transport + provider profile suites
(321 tests) continue to pass.
- AUTHOR_MAP: map twebefy@gmail.com → tw2818 for release notes credit.
Closes#15700, #17212, #17825.
Co-authored-by: tw2818 <twebefy@gmail.com>
DeepSeek's thinking mode requires both:
- extra_body.thinking.type: "enabled" to activate thinking mode
- top-level reasoning_effort: "max" or "high" to control depth
Previously, the ChatCompletionsTransport only handled Kimi's thinking
mode — DeepSeek was left unmapped, so reasoning_effort config was
silently dropped.
This patch:
1. Adds is_deepseek: bool to the Params dataclass, detected by
base_url matching api.deepseek.com
2. Maps Hermes effort levels (xhigh/max → "max", low/medium/high →
themselves) to the top-level reasoning_effort parameter
3. Sets extra_body.thinking.type alongside the effort
4. Strips reasoning_content from assistant messages sent back to
DeepSeek, preventing 400 errors when thinking was enabled
Three fixes for the May 2026 xAI OAuth (SuperGrok / X Premium) rollout
failures:
- _run_codex_stream: when openai SDK raises RuntimeError("Expected to
have received `response.created` before `<type>`"), retry once then
fall back to responses.create(stream=True) — same path used for
missing-response.completed postlude. Fallback surfaces the real
provider error with body+status_code intact. Also fixes#8133
(response.in_progress prelude on custom relays) and #14634
(codex.rate_limits prelude on codex-lb).
- _summarize_api_error: when error body matches xAI's entitlement
shape, append a one-line hint pointing to https://grok.com and
/model. Once-only, applies to both auxiliary warnings and
main-loop error surfacing.
- _chat_messages_to_responses_input: new is_xai_responses kwarg
drops replayed codex_reasoning_items (encrypted_content) before
they reach xAI. Also drops reasoning.encrypted_content from the
xAI include array. Native Codex behavior unchanged. Grok still
reasons natively each turn; coherence rides on visible message
text alone.
Closes#8133, #14634.
Adds a new authentication provider that lets SuperGrok subscribers sign
in to Hermes with their xAI account via the standard OAuth 2.0 PKCE
loopback flow, instead of pasting a raw API key from console.x.ai.
Highlights
----------
* OAuth 2.0 PKCE loopback login against accounts.x.ai with discovery,
state/nonce, and a strict CORS-origin allowlist on the callback.
* Authorize URL carries `plan=generic` (required for non-allowlisted
loopback clients) and `referrer=hermes-agent` for best-effort
attribution in xAI's OAuth server logs.
* Token storage in `auth.json` with file-locked atomic writes; JWT
`exp`-based expiry detection with skew; refresh-token rotation
synced both ways between the singleton store and the credential
pool so multi-process / multi-profile setups don't tear each other's
refresh tokens.
* Reactive 401 retry: on a 401 from the xAI Responses API, the agent
refreshes the token, swaps it back into `self.api_key`, and retries
the call once. Guarded against silent account swaps when the active
key was sourced from a different (manual) pool entry.
* Auxiliary tasks (curator, vision, embeddings, etc.) route through a
dedicated xAI Responses-mode auxiliary client instead of falling back
to OpenRouter billing.
* Direct HTTP tools (`tools/xai_http.py`, transcription, TTS, image-gen
plugin) resolve credentials through a unified runtime → singleton →
env-var fallback chain so xai-oauth users get them for free.
* `hermes auth add xai-oauth` and `hermes auth remove xai-oauth N` are
wired through the standard auth-commands surface; remove cleans up
the singleton loopback_pkce entry so it doesn't silently reinstate.
* `hermes model` provider picker shows
"xAI Grok OAuth (SuperGrok Subscription)" and the model-flow falls
back to pool credentials when the singleton is missing.
Hardening
---------
* Discovery and refresh responses validate the returned
`token_endpoint` host against the same `*.x.ai` allowlist as the
authorization endpoint, blocking MITM persistence of a hostile
endpoint.
* Discovery / refresh / token-exchange `response.json()` calls are
wrapped to raise typed `AuthError` on malformed bodies (captive
portals, proxy error pages) instead of leaking JSONDecodeError
tracebacks.
* `prompt_cache_key` is routed through `extra_body` on the codex
transport (sending it as a top-level kwarg trips xAI's SDK with a
TypeError).
* Credential-pool sync-back preserves `active_provider` so refreshing
an OAuth entry doesn't silently flip the active provider out from
under the running agent.
Testing
-------
* New `tests/hermes_cli/test_auth_xai_oauth_provider.py` (~63 tests)
covers JWT expiry, OAuth URL params (plan + referrer), CORS origins,
redirect URI validation, singleton↔pool sync, concurrency races,
refresh error paths, runtime resolution, and malformed-JSON guards.
* Extended `test_credential_pool.py`, `test_codex_transport.py`, and
`test_run_agent_codex_responses.py` cover the pool sync-back,
`extra_body` routing, and 401 reactive refresh paths.
* 165 tests passing on this branch via `scripts/run_tests.sh`.
* fix(langfuse): reject placeholder credentials with one-shot warning
When operators leave HERMES_LANGFUSE_PUBLIC_KEY / HERMES_LANGFUSE_SECRET_KEY
at a template value like 'placeholder', 'test-key', or 'your-langfuse-key',
the Langfuse SDK silently accepts the credentials at construction time and
drops every trace at flush time. No warning, no error — just an empty
Langfuse dashboard the operator only notices hours later.
Add prefix-based validation in _get_langfuse() against the documented
'pk-lf-' / 'sk-lf-' prefixes that Langfuse always issues server-side.
Anything else fires a single warning naming the offending env var(s)
with a log-safe value preview (full string for short placeholders so the
operator knows which template they left in place; truncated for long
values so a real secret pasted into the wrong field never hits the log),
then short-circuits via the existing _INIT_FAILED cache so the warning
fires once per process, not once per hook invocation.
The check sits after the 'Langfuse is None' SDK-installed guard so hosts
without the optional langfuse SDK don't see misleading 'set real keys'
hints when the actionable fix is 'pip install langfuse'. Missing
credentials remains the documented opt-out path and stays silent — no
log noise for unconfigured installs.
Fixes#22763Fixes#23823
* fix(langfuse): use actual API request messages for generation input
on_pre_llm_request previously used the messages kwarg alone, which
could be None when Hermes passes the payload via request_messages,
conversation_history, or user_message instead. Add _coerce_request_messages
to pick the first available list across all variants, falling back to a
synthetic user message. Generations now show the real outbound payload
rather than an empty input.
* fix(langfuse): record tool call outputs in traces
Tool observations showed input (arguments) but output was always
undefined. Root cause: when tool_call_id is empty, pre_tool_call stored
observations under a unique time-based key that post_tool_call could
never reconstruct, so every tool span was closed without output by the
_finish_trace sweep.
Fix pre/post matching by routing empty-tool_call_id tools through a
per-name FIFO queue (pending_tools_by_name) instead of the time-based
key. Tools with a tool_call_id continue to use the id-keyed dict.
Also:
- Preserve OpenAI-style nested function shape in serialized tool calls
so Langfuse renders name/arguments correctly
- Keep name + tool_call_id on role:tool messages for proper pairing
- Backfill tool results onto the matching turn_tool_calls entry so the
generation's tool-call record carries the result alongside arguments
- Coerce request messages from whichever field the runtime provides
(request_messages, messages, conversation_history, user_message)
* fix(langfuse): salvage-review polish — drop dead is_first_turn, shallow-copy request_messages, real threaded FIFO test
Self-review of the combined #22345 + #23831 salvage surfaced three issues
worth fixing in the same PR rather than as follow-ups:
1. Drop is_first_turn from the pre_api_request hook. The boolean expression
`not bool(conversation_history)` was wrong: conversation_history is
reassigned to None mid-run after compression (5 sites in run_agent.py),
so the value flips False -> True mid-conversation on every post-compression
API call. The langfuse plugin never consumed it, so the kwarg was both
misleading AND dead.
2. Replace copy.deepcopy(request_messages) with shallow list() copy. The
pre_api_request hook contract discards return values (invoke_hook never
writes back to api_kwargs), and the langfuse plugin's _serialize_messages
already builds its own snapshot dicts via _safe_value. A deepcopy on every
API call would walk every tool result and base64 image — significant
overhead for no real isolation benefit. Shallow copy of the outer list
protects against later mutations of api_messages without paying for the
inner-dict walk.
3. Rename test_empty_tool_call_id_concurrent_fifo_order ->
test_empty_tool_call_id_observations_are_fifo_within_tool_name and add a
real test_threaded_post_calls_preserve_fifo_under_lock that spawns 8
threads behind a barrier to actually exercise _STATE_LOCK on the
pending_tools_by_name queue. The original test was sequential and only
validated Python list semantics; this one validates the lock discipline.
4. Fix stale 'Cleared by reset_cache_for_tests()' comment on _INIT_FAILED —
that function does not exist. Tests reload the module via sys.modules.pop
+ importlib.import_module instead.
Tests: 37 langfuse plugin tests pass, 658 plugin tests overall pass.
---------
Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>
Co-authored-by: Brian Conklin <brian@dralth.com>
Replace O(n²) string concatenation of truncated_response_prefix in the
length-continuation retry loop with a list + ''.join(). Functionally
equivalent: same partial response on early return, same prepend on
final assembly. The legacy retry path is capped at 3 iterations, so
the practical wall-clock win is small, but the new idiom matches the
rest of the codebase and removes a needless repeated allocation.
Salvaged from PR #2717 (the run_conversation portion only — trajectory
refactor dropped because it silently rewrote </tool_response> to </think>).
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
Mirrors openclaw beta.8's app-server resilience fixes so a stuck codex
subprocess can't burn the full turn deadline and so users get a
`codex login` pointer instead of raw RPC errors when their token expires.
- TurnResult.should_retire signals the caller to drop+respawn codex.
- Deadline-hit path and dead-subprocess detection set should_retire so
the next turn doesn't ride a CPU-spinning or auth-broken process.
- Post-tool watchdog (post_tool_quiet_timeout=90s): if a tool item
completes and codex goes silent past the threshold without further
output or turn/completed, fast-fail instead of waiting the full 600s.
Resets on any non-tool activity so normal think-after-tool flows are
not affected.
- <turn_aborted> and <turn_aborted/> in agent text are treated as
terminal — some codex builds tear down a turn that way without
emitting turn/completed.
- _classify_oauth_failure() inspects RPC error message + stderr tail
for invalid_grant / token refresh / 401 / etc. and rewrites
user-facing errors to 'run codex login'. Conservative: generic
failures still surface verbatim. Fires at turn/start failure,
turn/completed failure, and dead-subprocess paths.
- thread/start cross-fill: tolerate thread.id, thread.sessionId,
top-level sessionId/threadId so future codex schema drift doesn't
KeyError us at handshake.
- run_agent.py: when run_turn returns should_retire=True OR raises,
close + null self._codex_session so the next turn respawns.
Tests: +30 cases across session + integration suites.
tests/agent/transports/test_codex_app_server_session.py 50/50 pass
tests/run_agent/test_codex_app_server_integration.py 27/27 pass
Broader codex scope (transports + cli runtime/migration) 376/376 pass
Background review fork redirected stdout/stderr around run_conversation()
so its iteration messages stay silent. But the memory-provider teardown
(shutdown_memory_provider() and review_agent.close()) fired in the outer
finally block AFTER the redirect_stdout context exited — so provider
teardown prints (Honcho disconnect, Hindsight sync, etc.) leaked into
the parent terminal at end of every turn.
Moves the teardown inside the redirect_stdout scope on the success path
(and nulls review_agent so the finally safety-net skips double-shutdown).
The finally block is rewritten as an exception-path safety net that
re-opens a devnull redirect, since the original 'with' context has
already exited by the time finally runs.
Salvage of #25342 by @ayushere (manually re-applied + merged conflict
with current main's set_thread_tool_whitelist wiring).
