diff --git a/apps/desktop/scripts/profile-typing-lag.md b/apps/desktop/scripts/profile-typing-lag.md index 3989361d07..a0b09b92ab 100644 --- a/apps/desktop/scripts/profile-typing-lag.md +++ b/apps/desktop/scripts/profile-typing-lag.md @@ -280,21 +280,74 @@ A/B on the 34 MB session, 300 tokens at 50 tok/sec, markdown chunks Modest. `inter-mutation` p50 tightens from 22-28 ms to a clean 33 ms, which is what you'd expect from a deterministic floor. -### Not fixed: Streamdown markdown re-parse (the elephant) +### Also landed: `useDeferredValue` at the streamdown-text boundary -This is still the dominant cost and the cause of the user's perceived -"5 fps moment" hitches. The renderer re-parses the *changed* block on -every commit, and as the last block grows the per-commit parse cost grows -linearly. With the throttle and React's batching there's still typically -1-2 longtasks per 5 s window on a big-session real-LLM stream, each -75-125 ms — and worst-case bursts up to 380-420 ms when a long -paragraph parses with many micromark backtracks. +The longtask CPU was unavoidable inside the block-memo pattern — the live +tail re-parses every commit, scales linearly with current length, and +nothing about Streamdown's architecture changes that without forking. The +fix is to stop having that work *block* the main thread. + +`` in `markdown-text.tsx` is a 12-line wrapper that +reads the message-part state via `useMessagePartText`, runs it through +`useDeferredValue`, and re-publishes via assistant-ui's +``. The inner `StreamdownTextPrimitive` reads the +deferred value through the normal `useMessagePartText` hook — no fork, +no internal-path imports, fully on the assistant-ui public API. + +What React's concurrent scheduler now does: + +- When a new token arrives mid-render, the in-flight deferred render + is abandoned and a fresh one starts with the latest text. +- When the main thread has urgent work (typing, scroll, layout), the + Streamdown render gets deprioritized — input stays responsive even + while a 100 ms parse is queued. + +Streamdown already uses `useTransition` internally for its block-array +setState; `useDeferredValue` here just lifts the deferral all the way up +to the consumer text boundary, so the whole pipeline — preprocess, +block split, repair, parse, render — runs at low priority during streaming. +This is the industry-standard approach (see +[Streamdown architecture analysis](https://tigerabrodi.blog/how-to-build-a-performant-ai-markdown-renderer) +and Chrome's [LLM-response render best practices](https://developer.chrome.google.cn/docs/ai/render-llm-responses)). + +A/B on the 34 MB session, 300 tokens at 50 tok/sec, markdown chunks +(four trials each, prod-throttle (33 ms) on for both): + +| | avgFps | p99 frame | LTs / 5 s | max LT | typing p95 | +|---|---|---|---|---|---| +| pre-defer | 54.3 | 41 ms | 1.7 | 110 ms | ~17 ms | +| **post-defer** | **58.5** | **31 ms** | 2.0 | 117 ms | 14-18 ms | + +Longtask count and max LT are unchanged — `useDeferredValue` doesn't +reduce CPU, only its priority. The avgFps lift and p99 frame drop are +the proof that the existing CPU is no longer blocking 60 fps cadence: +when React can defer the parse, frames stay clean. One particularly +clean run logged **MUTATIONS=0** — React skipped every intermediate +text state and only committed the final one, the textbook +useDeferredValue behaviour. + +### Not fixed: Streamdown markdown re-parse cost (the elephant) + +Total CPU spent in micromark/mdast/hast pipeline per 5 s window is still +the same ~700 ms. With `useDeferredValue` that work no longer blocks +input, but if you watch a CPU profile you'll see the same hot functions +(`Tn$1`, `bn$1`, `m$1`, `parser`, `compile`). + +The path to actually *reduce* that cost (not just defer it) is to +replace the parser with a state machine like +[Flowdown](https://github.com/Atomics-hub/flowdown) — process each +character exactly once, emit DOM ops directly, no re-parse of the prefix +on every token. Claimed ~2,000× over `marked`. Trades: not a +`react-markdown`-compatible API, no rehype security pipeline, would +require replacing Streamdown wholesale. Worth investigating only if +even the deferred work shows up in user-perceptible ways (e.g. +trackpad-scrolling a stream-in-progress stutters). The synthetic harness now mirrors the real upstream pipeline via the `flushMinMs` option in `__PERF_DRIVE__.stream({ flushMinMs: 33 })`, so -future Streamdown experiments can A/B without LLM credit cost. The -synthetic numbers tracked the one real-LLM run we caught within noise, -so it's a reliable proxy. +future Streamdown / Flowdown experiments can A/B without LLM credit cost. +The synthetic numbers tracked the one real-LLM run we caught within +noise, so it's a reliable proxy. Possible approaches (none implemented here): diff --git a/apps/desktop/src/components/assistant-ui/markdown-text.tsx b/apps/desktop/src/components/assistant-ui/markdown-text.tsx index 80e598ba69..2b4bc5898e 100644 --- a/apps/desktop/src/components/assistant-ui/markdown-text.tsx +++ b/apps/desktop/src/components/assistant-ui/markdown-text.tsx @@ -1,13 +1,13 @@ 'use client' -import { useAuiState } from '@assistant-ui/react' +import { TextMessagePartProvider, useAuiState, useMessagePartText } from '@assistant-ui/react' import { type StreamdownTextComponents, StreamdownTextPrimitive, type SyntaxHighlighterProps } from '@assistant-ui/react-streamdown' import { code } from '@streamdown/code' -import { type ComponentProps, memo, useEffect, useMemo, useState } from 'react' +import { type ComponentProps, memo, type ReactNode, useDeferredValue, useEffect, useMemo, useState } from 'react' import { PreviewAttachment } from '@/components/chat/preview-attachment' import { SyntaxHighlighter } from '@/components/chat/shiki-highlighter' @@ -226,6 +226,41 @@ function MarkdownImage({ className, src, alt, ...props }: ComponentProps<'img'>) ) } +/** + * Re-publish the active message-part context with React's `useDeferredValue` + * applied to the streaming text and status. The outer wrapper still re-renders + * on every token, but the work it does is trivial (one hook, one provider). + * + * The expensive subtree (Streamdown → micromark → mdast → hast → React) lives + * inside `` and reads the deferred text via the + * normal `useMessagePartText` hook. React's concurrent scheduler then has + * permission to: + * - skip intermediate token states when the next token arrives mid-render + * (it abandons the in-flight deferred render and starts over) + * - deprioritize the markdown render when the main thread is busy with an + * urgent task (typing, scrolling, layout work elsewhere) + * + * Net effect: per-token CPU is unchanged but the *blocking* part of that work + * goes away — typing-while-streaming stays a single-frame paint, scroll + * stutter disappears, and the longtask histogram tightens because long + * commits can be interrupted and discarded. + * + * Industry standard (Streamdown's own block-array setState already uses + * `useTransition`); this just lifts the deferral up to the consumer text + * boundary so it covers the whole pipeline, not just the inner setState. + */ +function DeferStreamingText({ children }: { children: ReactNode }) { + const { text, status } = useMessagePartText() + const deferredText = useDeferredValue(text) + const isRunning = status.type === 'running' + + return ( + + {children} + + ) +} + // Headings shrink to chat scale rather than the prose default (h1≈xl). Kept // table-driven so adding/tweaking levels is one row. const HEADING_SIZES: Record<'h1' | 'h2' | 'h3' | 'h4', string> = { @@ -318,32 +353,34 @@ const MarkdownTextImpl = () => { ) return ( - *:first-child]:mt-0 [&>*:last-child]:mb-0 [&>*+*]:mt-1' - )} - lineNumbers={false} - mode="streaming" - // Always auto-close incomplete fences — even during streaming. - // Without this, an unclosed ```python ... ``` whose body contains - // `$` (very common: shell snippets, JS template strings, dollar - // amounts) leaks those dollars out to the math parser and they - // get rendered as broken inline math until the closing fence - // arrives. Shiki is independently deferred via `defer={isStreaming}` - // on the SyntaxHighlighter component, so we don't pay code-block - // tokenization on every token even with this set. - parseIncompleteMarkdown - plugins={plugins} - preprocess={preprocessMarkdown} - /> + + *:first-child]:mt-0 [&>*:last-child]:mb-0 [&>*+*]:mt-1' + )} + lineNumbers={false} + mode="streaming" + // Always auto-close incomplete fences — even during streaming. + // Without this, an unclosed ```python ... ``` whose body contains + // `$` (very common: shell snippets, JS template strings, dollar + // amounts) leaks those dollars out to the math parser and they + // get rendered as broken inline math until the closing fence + // arrives. Shiki is independently deferred via `defer={isStreaming}` + // on the SyntaxHighlighter component, so we don't pay code-block + // tokenization on every token even with this set. + parseIncompleteMarkdown + plugins={plugins} + preprocess={preprocessMarkdown} + /> + ) }