fix(streaming): route mid-tool-call partial-stream-stub through length continuation (#31998) (#32012)
* fix(streaming): route mid-tool-call partial-stream-stub through length continuation (#31998) When a stream stalls mid-tool-call (e.g. a large write_file), the partial-stream-stub recovery used finish_reason='stop' which caused the conversation loop to treat the turn as complete, returning only the warning text. When users said 'continue', the model retried the same large tool call, hit the same stale timeout, and looped indefinitely. Changes: - chat_completion_helpers.py: change _stub_finish_reason from 'stop' to 'length' for mid-tool-call partials. The stub still has tool_calls=None so no tool auto-executes — the model gets a fresh API call through the existing length-continuation machinery (bounded to 3 retries). Also attach _dropped_tool_names to the stub for downstream use. - conversation_loop.py: add a third continuation prompt branch for partial-stream-stubs with dropped tool calls. Instead of the generic 'continue where you left off' (which would retry the same large call), tell the model to break the output into smaller tool calls (~8K tokens each) to avoid stream timeouts. - test_partial_stream_finish_reason.py: update existing test from finish_reason='stop' to 'length', add _dropped_tool_names assertion, add new test_dropped_tool_call_uses_chunking_prompt for the 3-way prompt branching. Safety: tool_calls=None is preserved on the stub, so the conversation loop enters the text-continuation branch (line 1513), NOT the tool-call execution branch (line 3246). No tool auto-executes. The model simply gets another API call with targeted guidance. * refactor: extract constants and continuation prompt helper - Move magic strings to hermes_constants.py (PARTIAL_STREAM_STUB_ID, FINISH_REASON_LENGTH) - Extract _get_continuation_prompt() in conversation_loop.py — DRYs the 3-way prompt branching and lets tests import the real function - Trim verbose inline comments in chat_completion_helpers.py - Tests import constants + helper instead of duplicating logic --------- Co-authored-by: alt-glitch <balyan.sid@gmail.com>
This commit is contained in:
daimon-nous[bot]
co-authored by
alt-glitch
alt-glitch
parent
46d8b5dadf
commit
ac5359a3f3
@@ -34,6 +34,7 @@ from typing import Any, Dict, List, Optional, Tuple
|
||||
from urllib.parse import urlparse, parse_qs, urlunparse
|
||||
|
||||
from hermes_cli.timeouts import get_provider_request_timeout, get_provider_stale_timeout
|
||||
from hermes_constants import PARTIAL_STREAM_STUB_ID, FINISH_REASON_LENGTH
|
||||
from agent.error_classifier import classify_api_error, FailoverReason
|
||||
from agent.model_metadata import is_local_endpoint
|
||||
from agent.message_sanitization import (
|
||||
@@ -2172,37 +2173,15 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
|
||||
if deltas_were_sent["yes"]:
|
||||
# Streaming failed AFTER some tokens were already delivered to
|
||||
# the platform. Re-raising would let the outer retry loop make
|
||||
# a new API call, creating a duplicate message. Return a
|
||||
# partial response stub instead and let the outer loop decide:
|
||||
#
|
||||
# - text-only partials → finish_reason="length" so the
|
||||
# conversation loop persists the partial assistant content
|
||||
# and asks the model to continue from where the stream
|
||||
# died (issue #30963: partial stop misclassified as a
|
||||
# clean completion was exiting the loop with budget
|
||||
# remaining and an unfinished goal).
|
||||
#
|
||||
# - partial mid-tool-call → finish_reason="stop" stays.
|
||||
# The user-visible warning we append says "Ask me to
|
||||
# retry if you want to continue", so the agent should
|
||||
# hand control back rather than auto-retry a tool call
|
||||
# that may have side-effects.
|
||||
#
|
||||
# Recover whatever content was already streamed to the user.
|
||||
# _current_streamed_assistant_text accumulates text fired
|
||||
# through _fire_stream_delta, so it has exactly what the
|
||||
# user saw before the connection died.
|
||||
# Return a partial response stub with finish_reason="length"
|
||||
# so the conversation loop's continuation machinery fires.
|
||||
# tool_calls=None prevents auto-execution of incomplete calls.
|
||||
_partial_text = (
|
||||
getattr(agent, "_current_streamed_assistant_text", "") or ""
|
||||
).strip() or None
|
||||
|
||||
# If the stream died while the model was emitting a tool call,
|
||||
# the stub below will silently set `tool_calls=None` and the
|
||||
# agent loop will treat the turn as complete — the attempted
|
||||
# action is lost with no user-facing signal. Append a
|
||||
# human-visible warning to the stub content so (a) the user
|
||||
# knows something failed, and (b) the next turn's model sees
|
||||
# in conversation history what was attempted and can retry.
|
||||
# Append a user-visible warning if tool calls were dropped so
|
||||
# the user and model both know what was attempted.
|
||||
_partial_names = list(result.get("partial_tool_names") or [])
|
||||
if _partial_names:
|
||||
_name_str = ", ".join(_partial_names[:3])
|
||||
@@ -2214,8 +2193,7 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
|
||||
f"Ask me to retry if you want to continue."
|
||||
)
|
||||
_partial_text = (_partial_text or "") + _warn
|
||||
# Also fire as a streaming delta so the user sees it now
|
||||
# instead of only in the persisted transcript.
|
||||
# Fire as streaming delta so the user sees it immediately.
|
||||
try:
|
||||
agent._fire_stream_delta(_warn)
|
||||
except Exception:
|
||||
@@ -2225,7 +2203,7 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
|
||||
"of text; surfaced warning to user: %s",
|
||||
_partial_names, len(_partial_text or ""), result["error"],
|
||||
)
|
||||
_stub_finish_reason = "stop"
|
||||
_stub_finish_reason = FINISH_REASON_LENGTH
|
||||
else:
|
||||
logger.warning(
|
||||
"Partial stream delivered before error; returning "
|
||||
@@ -2235,18 +2213,19 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
|
||||
len(_partial_text or ""),
|
||||
result["error"],
|
||||
)
|
||||
_stub_finish_reason = "length"
|
||||
_stub_finish_reason = FINISH_REASON_LENGTH
|
||||
_stub_msg = SimpleNamespace(
|
||||
role="assistant", content=_partial_text, tool_calls=None,
|
||||
reasoning_content=None,
|
||||
)
|
||||
return SimpleNamespace(
|
||||
id="partial-stream-stub",
|
||||
id=PARTIAL_STREAM_STUB_ID,
|
||||
model=getattr(agent, "model", "unknown"),
|
||||
choices=[SimpleNamespace(
|
||||
index=0, message=_stub_msg, finish_reason=_stub_finish_reason,
|
||||
)],
|
||||
usage=None,
|
||||
_dropped_tool_names=_partial_names or None,
|
||||
)
|
||||
raise result["error"]
|
||||
return result["response"]
|
||||
|
||||
Reference in New Issue
Block a user