fix(tui): stop persisting full tool output in trail lines (silent OOM death)
A heavy --tui session (browser snapshots, large tool outputs) silently OOM-killed the Node parent within minutes — closing the gateway child's stdin, which the user saw only as a bare "gateway exited" / stdin EOF. CLI was immune. Root cause: each completed tool's verbose trail line embedded up to 16KB of result_text, persisted in transcript Msg.tools[] for the whole session and rendered EXPANDED by default, so an Ink render-node tree was built for every one of up to 800 messages at once. That tree blew past Node's heap at a few hundred MB — far below the 2.5GB memory-monitor exit threshold, so the death was never even attributed. - text.ts: persisted verbose tool-trail blocks now cap to a small preview (VERBOSE_TRAIL_MAX_CHARS=800/12 lines), not the 16KB live-render budget. Retained trail strings drop ~17x (12.2MB -> 0.7MB at 800 msgs); the live streaming tail still uses the larger LIVE_RENDER budget. - tui_gateway/server.py: lower the gateway-side verbose text cap to match (1KB/16 lines) so we stop shipping output the TUI no longer renders. - memoryMonitor.ts: derive critical/high thresholds from the real V8 heap ceiling (~88%/70%) instead of the hardcoded 2.5GB that killed the process at 31% of an 8GB ceiling; add a one-shot onWarn early-warning on fast sub-threshold heap growth so the next such death is diagnosable, not silent. - entry.tsx: wire onWarn to a crash-log breadcrumb + stderr line. Full tool output is unchanged in the agent context and SQLite session — this is display/transport only, no behavior or context change. Fixes #34095. Related #27282. Tests: ui-tui text + new memoryMonitor suites (33 pass), python verbose-cap guard (5 pass); full ui-tui suite shows no new failures vs pristine main. E2E repro confirms the retention drop.
This commit is contained in:
+11
-1
@@ -75,7 +75,17 @@ const stopMemoryMonitor = startMemoryMonitor({
|
||||
process.stderr.write('hermes-tui: exiting to avoid OOM; restart to recover\n')
|
||||
process.exit(137)
|
||||
},
|
||||
onHigh: (snap, dump) => process.stderr.write(dumpNotice(snap, dump))
|
||||
onHigh: (snap, dump) => process.stderr.write(dumpNotice(snap, dump)),
|
||||
// Sub-threshold abnormal heap growth (#34095). The TUI used to die silently
|
||||
// here — Node OOMs from a render-tree blowup well below the exit threshold,
|
||||
// so the only trace was a bare gateway `stdin EOF`. Persist a breadcrumb +
|
||||
// stderr line so the next such death is attributable instead of silent.
|
||||
onWarn: snap => {
|
||||
recordParentLifecycle(`memory-warning fast heap growth heap=${formatBytes(snap.heapUsed)} rss=${formatBytes(snap.rss)}`)
|
||||
process.stderr.write(
|
||||
`hermes-tui: heap climbing fast (${formatBytes(snap.heapUsed)}) — a large tool output or long session may be straining memory\n`
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
if (process.env.HERMES_HEAPDUMP_ON_START === '1') {
|
||||
|
||||
Reference in New Issue
Block a user