fix(agent): surface model refusals instead of retrying them as errors
A Claude refusal (HTTP 200, stop_reason="refusal", empty content) was
laundered into a generic retry loop and surfaced as a misleading
"rate limited / invalid response" or "no content after retries" error,
burning paid attempts reproducing a deterministic refusal.
This hit two distinct paths:
- Direct Anthropic (anthropic_messages): validate_response rejected the
empty-content refusal *before* normalize_response mapped refusal ->
content_filter, so it fell into the invalid-response retry loop.
- Nous Portal / OpenAI-compatible (chat_completions): the portal surfaces
a Claude refusal via message.refusal with empty content, which sailed
past validation and died in the empty-response retry loop.
Fix (one unified content_filter dispatch for all backends):
- AnthropicTransport.validate_response: accept empty content when
stop_reason == "refusal" so it flows to normalize_response.
- ChatCompletionsTransport.normalize_response: promote message.refusal to
content + a content_filter finish reason.
- conversation_loop: handle finish_reason == "content_filter" - fire the
api_request_error hook (content_policy_blocked), try a configured
fallback once, else return a clear terminal refusal message. Never retry
a deterministic refusal.
Supersedes #43084, which fixed only the direct-Anthropic path and could
not reach the chat_completions/portal path.
Tests: transport-level (validate_response refusal, message.refusal
promotion) + end-to-end loop (refusal surfaced, exactly one API call).
(cherry picked from commit 01f546f92c)
This commit is contained in:
@@ -4660,6 +4660,47 @@ class TestRetryExhaustion:
|
||||
assert "error" in result
|
||||
assert "Invalid API response" in result["error"]
|
||||
|
||||
def test_content_filter_refusal_surfaced_not_retried(self, agent):
|
||||
"""A model refusal must be surfaced immediately, NOT laundered into
|
||||
the empty-response retry loop and reported as "rate limited" / "no
|
||||
content after retries".
|
||||
|
||||
Regression: running a Claude refusal through an OpenAI-compatible
|
||||
portal (Nous Portal fronting Anthropic) returns ``message.refusal``
|
||||
with empty content. The transport now promotes that to a
|
||||
``content_filter`` finish reason and the loop surfaces it as a terminal
|
||||
``content_policy_blocked`` result instead of retrying a deterministic
|
||||
refusal three times.
|
||||
"""
|
||||
self._setup_agent(agent)
|
||||
refusal_resp = SimpleNamespace(
|
||||
choices=[SimpleNamespace(
|
||||
message=SimpleNamespace(
|
||||
content=None, tool_calls=None, reasoning=None,
|
||||
reasoning_content=None, refusal="I won't help with that.",
|
||||
),
|
||||
finish_reason="stop",
|
||||
)],
|
||||
model="test/model",
|
||||
usage=None,
|
||||
id="resp_1",
|
||||
)
|
||||
agent.client.chat.completions.create.return_value = refusal_resp
|
||||
with (
|
||||
patch.object(agent, "_persist_session"),
|
||||
patch.object(agent, "_save_trajectory"),
|
||||
patch.object(agent, "_cleanup_task_resources"),
|
||||
):
|
||||
result = agent.run_conversation("please do something disallowed")
|
||||
assert result.get("completed") is False
|
||||
assert result.get("failed") is True
|
||||
assert "content_policy_blocked" in result.get("error", "")
|
||||
# The model's refusal text is surfaced to the user, not swallowed.
|
||||
assert "I won't help with that." in (result.get("final_response") or "")
|
||||
# Crucial regression guard: a deterministic refusal is NOT retried —
|
||||
# exactly one API call, no empty-response retry loop.
|
||||
assert agent.client.chat.completions.create.call_count == 1
|
||||
|
||||
def test_api_error_returns_gracefully_after_retries(self, agent):
|
||||
"""Exhausted retries on API errors must return error result, not crash."""
|
||||
self._setup_agent(agent)
|
||||
|
||||
Reference in New Issue
Block a user