seb/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(agent): surface model refusals instead of retrying them as errors

Browse Source 
A Claude refusal (HTTP 200, stop_reason="refusal", empty content) was
laundered into a generic retry loop and surfaced as a misleading
"rate limited / invalid response" or "no content after retries" error,
burning paid attempts reproducing a deterministic refusal.

This hit two distinct paths:

- Direct Anthropic (anthropic_messages): validate_response rejected the
  empty-content refusal *before* normalize_response mapped refusal ->
  content_filter, so it fell into the invalid-response retry loop.
- Nous Portal / OpenAI-compatible (chat_completions): the portal surfaces
  a Claude refusal via message.refusal with empty content, which sailed
  past validation and died in the empty-response retry loop.

Fix (one unified content_filter dispatch for all backends):
- AnthropicTransport.validate_response: accept empty content when
  stop_reason == "refusal" so it flows to normalize_response.
- ChatCompletionsTransport.normalize_response: promote message.refusal to
  content + a content_filter finish reason.
- conversation_loop: handle finish_reason == "content_filter" - fire the
  api_request_error hook (content_policy_blocked), try a configured
  fallback once, else return a clear terminal refusal message. Never retry
  a deterministic refusal.

Supersedes #43084, which fixed only the direct-Anthropic path and could
not reach the chat_completions/portal path.

Tests: transport-level (validate_response refusal, message.refusal
promotion) + end-to-end loop (refusal surfaced, exactly one API call).

(cherry picked from commit 01f546f92c)
This commit is contained in:
SHL0MS
2026-06-14 12:10:08 +05:30
committed by kshitijk4poor
parent 4b5ba112ad
commit bb46bf8ce4
6 changed files with 276 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/conversation_loop.py
+110
View File
@@ -1348,6 +1348,116 @@ def run_conversation(
)
finish_reason = "length"
# ── Content-policy refusal (HTTP 200) ──────────────────
# The model — or the provider's safety system — returned a
# *successful* response whose stop/finish reason is a refusal:
# Anthropic ``stop_reason="refusal"`` → ``content_filter``;
# OpenAI / portal ``finish_reason="content_filter"`` or a
# populated ``message.refusal`` (mapped in the chat_completions
# transport); Bedrock ``guardrail_intervened``. The content is
# typically empty, so without this branch the response falls
# through to the empty-response / invalid-response retry loops
# and is mis-surfaced as "rate limited" / "no content after
# retries" — burning paid attempts reproducing a deterministic
# refusal. Surface it clearly and stop. Mirrors the
# exception-based ``content_policy_blocked`` recovery: try a
# configured fallback once, otherwise return the refusal.
if finish_reason == "content_filter":
_refusal_transport = agent._get_transport()
if agent.api_mode == "anthropic_messages":
_refusal_result = _refusal_transport.normalize_response(
response, strip_tool_prefix=agent._is_anthropic_oauth
)
else:
_refusal_result = _refusal_transport.normalize_response(response)
_refusal_text = (getattr(_refusal_result, "content", None) or "").strip()
# Some refusals carry the explanation only in the reasoning
# channel; fall back to it so the user sees *something*.
if not _refusal_text:
_refusal_text = (agent._extract_reasoning(_refusal_result) or "").strip()
agent._invoke_api_request_error_hook(
task_id=effective_task_id,
turn_id=turn_id,
api_request_id=api_request_id,
api_call_count=api_call_count,
api_start_time=api_start_time,
api_kwargs=api_kwargs,
error_type="ContentPolicyBlocked",
error_message=_refusal_text or "model declined to respond (content_filter)",
status_code=None,
retry_count=retry_count,
max_retries=max_retries,
retryable=False,
reason="content_policy_blocked",
)
if thinking_spinner:
thinking_spinner.stop("")
thinking_spinner = None
if agent.thinking_callback:
agent.thinking_callback("")
# Deterministic for the unchanged prompt — never retry.
# Try a configured fallback once (a different model may not
# refuse); otherwise surface the refusal terminally.
if agent._has_pending_fallback():
agent._buffer_status(
"⚠️ Model declined to respond (safety refusal) — trying fallback..."
)
if agent._try_activate_fallback():
retry_count = 0
compression_attempts = 0
_retry.primary_recovery_attempted = False
continue
agent._flush_status_buffer()
_refusal_log = (
_refusal_text[:500] + "..."
if len(_refusal_text) > 500
else _refusal_text
)
logger.warning(
"%sModel declined to respond (finish_reason=content_filter). "
"model=%s provider=%s refusal=%s",
agent.log_prefix, agent.model, agent.provider,
_refusal_log or "(no text)",
)
agent._emit_status(
"⚠️ The model declined to respond to this request (safety refusal)."
)
if _refusal_text:
_refusal_response = (
"⚠️ The model declined to respond to this request "
"(safety refusal — not a Hermes/gateway failure).\n\n"
f"Model's explanation: {_refusal_text}\n\n"
"Try rephrasing the request, narrowing the context, or "
"adding a fallback provider with `hermes fallback add`."
)
else:
_refusal_response = (
"⚠️ The model declined to respond to this request "
"(safety refusal — not a Hermes/gateway failure). The "
"model returned no explanation.\n\n"
"Try rephrasing the request, narrowing the context, or "
"adding a fallback provider with `hermes fallback add`."
)
agent._cleanup_task_resources(effective_task_id)
agent._persist_session(messages, conversation_history)
return {
"final_response": _refusal_response,
"messages": messages,
"api_calls": api_call_count,
"completed": False,
"failed": True,
"error": (
"content_policy_blocked: "
+ (_refusal_text or "model declined (content_filter)")
),
}
if finish_reason == "length":
if getattr(response, "id", "") == PARTIAL_STREAM_STUB_ID:
agent._vprint(