fix(patch): widen new_string \t/\r unescape to all match strategies (#33733)

Extends @liuhao1024's escape-normalized fix so the patch tool also
recovers when old_string carries a real tab byte and matches via the
`exact` strategy — which is the headline reproduction in the issue and
the most common case in practice (LLMs frequently get old_string right
because they re-read the file, but still serialize new_string's tabs as
two-character `\t`).

Instead of gating on the match strategy, decide per-sequence by looking
at the *matched region of the file*: only convert `\t` -> tab and
`\r` -> CR when the file region we're replacing actually contains the
corresponding control byte. That mirrors the region-based heuristic in
`_detect_escape_drift` and keeps legitimate writes of the literal
two-character string `"\t"` (e.g. patching `sep = "\t"` in Python
source) untouched — those files have a backslash+t in the matched
region, not a real tab, so new_string passes through verbatim. `\n` is
still excluded because newlines serialize correctly through JSON and
unescaping would corrupt source escape sequences far more often than
help.

E2E verified against the live `patch` tool: tab-indented file + literal
`\t` in new_string under both `exact` (Variant 1) and `escape_normalized`
(Variant 2) strategies now produces real tab bytes; a Python source line
containing `sep = "\t"` (legitimate literal backslash-t) survives a
patch unchanged.

Tests updated to cover both strategies and the legitimate-literal case,
and to assert that `\n` is intentionally preserved.

Refs #33733
This commit is contained in:
teknium1
2026-05-28 03:27:20 -07:00
committed by Teknium
parent e9f3f2b34a
commit 78be458608
2 changed files with 130 additions and 51 deletions
+52 -16
View File
@@ -113,14 +113,27 @@ def fuzzy_find_and_replace(content: str, old_string: str, new_string: str,
# old_string/new_string — e.g. LLM used 2-space indent but the
# file is 4-space. Shift new_string by the indentation delta so
# the replacement matches the file's actual indent pattern.
effective_new = new_string
if strategy_name == "escape_normalized":
# The escape_normalized strategy matched because old_string
# contained literal \t/\n/\r that were unescaped to match
# real control characters in the file. Apply the same
# unescaping to new_string so we don't write literal
# backslash sequences where the file has real tabs/newlines.
effective_new = _unescape_common_sequences(new_string)
# LLMs frequently serialize tabs / carriage returns in JSON
# tool-call arguments as the two-character sequences ``\t`` and
# ``\r`` (backslash + letter) instead of the real control bytes.
# If we write new_string verbatim, the file ends up with literal
# backslash sequences where the surrounding code uses real tabs.
#
# Strategy: only unescape when the matched region of the file
# *actually contains* the corresponding real control character.
# That mirrors the region-based heuristic in
# ``_detect_escape_drift`` and keeps legitimate writes of the
# literal two-character string ``"\t"`` (e.g. patching Python
# source that contains a tab string literal in source text)
# untouched — those files have a backslash+t in the matched
# region, not a real tab, so we leave new_string alone.
#
# ``\n`` is intentionally excluded: newlines serialize correctly
# through JSON, and rewriting backslash-n would mangle escape
# sequences in source code constants far more often than help.
effective_new = _maybe_unescape_new_string(
new_string, content, matches,
)
new_content = _apply_replacements(
content, matches, effective_new,
old_string=old_string if strategy_name != "exact" else None,
@@ -255,17 +268,40 @@ def _reindent_replacement(file_region: str, old_string: str, new_string: str) ->
return "\n".join(out_lines)
def _unescape_common_sequences(s: str) -> str:
"""Unescape common C-style escape sequences that LLMs produce literally.
def _maybe_unescape_new_string(new_string: str,
content: str,
matches: List[Tuple[int, int]]) -> str:
"""Conditionally unescape ``\\t``/``\\r`` in new_string.
When the model sends ``\\t`` (two characters: backslash + t) instead of a
real tab byte (0x09), the patch tool would write the literal characters.
This helper converts common escape sequences to their actual byte values.
LLMs frequently send the two-character sequences ``\\t`` (backslash + t)
and ``\\r`` (backslash + r) inside JSON tool-call arguments where they
meant a real tab or carriage-return byte. Writing the string verbatim
corrupts tab-indented files with literal backslash-letter pairs.
Only call this when the matching strategy confirmed that the file already
contains real control characters (i.e. ``escape_normalized`` matched).
The unescape is only applied per-sequence when the *matched region of
the file* actually contains the corresponding control character — that
is, we only convert ``\\t`` -> tab when the file region we're replacing
contains a real tab byte. Files that legitimately contain the literal
two-character string ``"\\t"`` (e.g. a Python source line that defines
``sep = "\\t"``) get a backslash+t in the matched region instead of a
tab, so we leave new_string alone.
``\\n`` is intentionally excluded: newlines serialize correctly through
JSON and rewriting backslash-n would corrupt escape sequences in
string literals far more often than it would help.
"""
return s.replace('\\t', '\t').replace('\\n', '\n').replace('\\r', '\r')
# Cheap pre-check — bail out unless new_string actually contains one of
# the suspect sequences. Keeps the common case free.
if "\\t" not in new_string and "\\r" not in new_string:
return new_string
matched_regions = "".join(content[start:end] for start, end in matches)
out = new_string
if "\\t" in out and "\t" in matched_regions:
out = out.replace("\\t", "\t")
if "\\r" in out and "\r" in matched_regions:
out = out.replace("\\r", "\r")
return out
def _apply_replacements(content: str, matches: List[Tuple[int, int]],