perf(desktop): floor assistant-text flush gap to 33ms for predictable batching

`scheduleDeltaFlush` previously coalesced via `requestAnimationFrame`
only. The "at most one flush per frame" guarantee that gives you is fine
for fast streams (>~80 tok/sec) where multiple tokens arrive within a
single frame, but breaks down at typical LLM token rates (30-80 tok/sec)
where each token arrives slower than the rAF cadence and triggers its
own React commit + Streamdown markdown re-parse.

Track `lastFlushAt` and require at least 33 ms between two flushes.
React 18+ auto-batching probabilistically already collapsed some of
these, but the floor makes it deterministic.

A/B on the 34 MB session, 300 tokens at 50 tok/sec (markdown chunks):

| | avgFps | p99 frame | LTs / 5 s | max LT |
|---|---|---|---|---|
| no floor (current rAF) | 54.0 | 38 ms | 2.0 | 145 ms |
| 33 ms floor (this PR) | 54.3 | 41 ms | 1.7 | 110 ms |

`inter-mutation` p50 also tightens from 22-28 ms to a clean 33 ms,
which is the expected signature of a deterministic floor. Doesn't fully
solve the user's perceived hitches — Streamdown's per-Block parse cost
when the last block grows past ~2 k chars is still the elephant — but
it consistently shaves the worst-case longtask and makes the streaming
cadence visibly steadier.

Also threads a matching `flushMinMs` option through the synthetic
stream driver in `perf-probe.tsx` + `scripts/measure-synthetic-stream.mjs`
so the harness can A/B both regimes without spending LLM credits.

See `scripts/profile-typing-lag.md` for the full investigation.
This commit is contained in:
Brooklyn Nicholson
2026-05-21 20:08:49 -05:00
parent ea510a7c02
commit 7003df708c
4 changed files with 151 additions and 30 deletions
@@ -19,6 +19,10 @@ import { writeFileSync } from 'node:fs'
const CDP_HTTP = 'http://127.0.0.1:9222'
const TOKENS = Number(process.env.TOKENS || 300)
const INTERVAL_MS = Number(process.env.INTERVAL_MS || 16)
// Upstream flush throttle to apply in the synthetic driver. Mirrors what the
// real gateway path does in `use-message-stream.scheduleDeltaFlush`. 0
// disables (worst-case, every token = one React commit).
const FLUSH_MIN_MS = Number(process.env.FLUSH_MIN_MS || 0)
const CHUNK = process.env.CHUNK || 'lorem ipsum '
const TYPE_WHILE_STREAMING = process.env.TYPE_WHILE_STREAMING === '1'
const LABEL = process.env.LABEL || 'baseline'
@@ -144,7 +148,7 @@ async function main() {
// Drive a synthetic stream.
const streamStart = Date.now()
await cdp.eval(`window.__PERF_DRIVE__.stream({ chunk: ${JSON.stringify(CHUNK)}, intervalMs: ${INTERVAL_MS}, totalTokens: ${TOKENS} })`)
await cdp.eval(`window.__PERF_DRIVE__.stream({ chunk: ${JSON.stringify(CHUNK)}, intervalMs: ${INTERVAL_MS}, totalTokens: ${TOKENS}, flushMinMs: ${FLUSH_MIN_MS} })`)
// After the first paint, arm MO on the new message.
await new Promise((r) => setTimeout(r, 200))
@@ -269,7 +273,7 @@ async function main() {
const result = {
label: LABEL,
timestamp: new Date().toISOString(),
config: { TOKENS, INTERVAL_MS, CHUNK, TYPE_WHILE_STREAMING },
config: { TOKENS, INTERVAL_MS, CHUNK, TYPE_WHILE_STREAMING, FLUSH_MIN_MS },
streamWallMs: Date.now() - streamStart,
frames: {
total: streamFrames.length,
+48 -5
View File
@@ -246,12 +246,55 @@ field-by-field. Verified via temporary render counter:
on its own — Streamdown dwarfs it — but eliminates a class of forced
layouts and removes a steady CPU floor.
### Not fixed: Streamdown markdown re-parse
### Also landed: `MarkdownText` plugins memo + upstream flush floor
This is the dominant cost and the cause of the user's perceived hitches.
The renderer re-parses the entire message buffer on every stream update.
At ~3-5 k chars, each parse costs ~30 ms; when several pile into one
frame the result is a 75-125 ms longtask = the "5 fps moment".
Two smaller follow-ups in the same investigation:
1. **`MarkdownText` `plugins` object useMemo'd.** The inline
`plugins={{ math: mathPlugin, ...(isStreaming ? {} : { code }) }}`
was constructing a new object on every render, which churns
`<Streamdown>`'s outer memo and forces its internal `rehypePlugins` /
`remarkPlugins` arrays to rebuild. CPU profile after the change shows
`parser` self-time dropping out of the top 10, `compile` cut roughly
in half, and `bn$1` / `m$1` (micromark internals) dropping off the
top entries.
2. **`use-message-stream.scheduleDeltaFlush` got a real minimum floor.**
Previously the rAF-only path effectively meant "at most one flush per
frame," but at typical LLM token rates of 30-80 tok/sec each token
arrives slower than rAF cadence and gets its own React commit. With
`STREAM_DELTA_FLUSH_MS = 33` (two frames) and a `lastFlushAt`-tracked
floor, slower streams now coalesce ~2 tokens per commit, halving
markdown re-parses. React's auto-batching already covers part of this
probabilistically; the floor makes the batching deterministic so the
max-longtask number tightens up.
A/B on the 34 MB session, 300 tokens at 50 tok/sec, markdown chunks
(3 trials each):
| | avgFps | p99 frame | LTs/5s | max LT | mutations |
|---|---|---|---|---|---|
| no throttle | 54.0 | 38 ms | 2.0 | 145 ms | varies (2-112) |
| 33 ms throttle | 54.3 | 41 ms | 1.7 | 110 ms | ~135 |
Modest. `inter-mutation` p50 tightens from 22-28 ms to a clean 33 ms,
which is what you'd expect from a deterministic floor.
### Not fixed: Streamdown markdown re-parse (the elephant)
This is still the dominant cost and the cause of the user's perceived
"5 fps moment" hitches. The renderer re-parses the *changed* block on
every commit, and as the last block grows the per-commit parse cost grows
linearly. With the throttle and React's batching there's still typically
1-2 longtasks per 5 s window on a big-session real-LLM stream, each
75-125 ms — and worst-case bursts up to 380-420 ms when a long
paragraph parses with many micromark backtracks.
The synthetic harness now mirrors the real upstream pipeline via the
`flushMinMs` option in `__PERF_DRIVE__.stream({ flushMinMs: 33 })`, so
future Streamdown experiments can A/B without LLM credit cost. The
synthetic numbers tracked the one real-LLM run we caught within noise,
so it's a reliable proxy.
Possible approaches (none implemented here):