fix(web): make streaming code fences incremental
This commit is contained in:
07akioni
parent
713ae6c29b
commit
1dd3e60f50
20 changed files with 649 additions and 106 deletions
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.md
|
||||
2026-08-20-web-streaming-fence-highlight.md: ccc961da1febba3611e3e558087b586c6f1f474b
|
||||
2026-08-20-web-streaming-fence-highlight.zh.md: e5d29545bf659ed572d052f1212d68d344d49b5a
|
||||
2026-08-20-web-streaming-fence-highlight.md: aa44a8c7ea91ff4913d92449d62a6ebf9406e922
|
||||
2026-08-20-web-streaming-fence-highlight.zh.md: 88bae36928ac8d00960e529addeb4f30240e3579
|
||||
|
|
|
|||
|
|
@ -10,17 +10,18 @@ While a reply streamed, `MarkdownText` stripped the fence language before `CodeB
|
|||
|
||||
## Decision
|
||||
|
||||
Streaming fences highlight incrementally through grammar-state resumption; the settled arm is unchanged.
|
||||
Streaming fences parse, tokenize, and reconcile from retained frontiers; the settled output is unchanged.
|
||||
|
||||
- **`StreamingHighlightSession`** (`packages/client/ui-primitives/src/markdown/highlight.ts`) exploits that TextMate tokenization is line-based and forward-only: a line's tokens depend only on its own text and the grammar state entering it, so appended text never changes a completed line's tokens. The session caches completed lines' spans plus shiki's `GrammarState` after them (`getLastGrammarState`), and each update tokenizes newly completed text via `codeToTokensBase(…, { grammarState })` plus the still-growing last line. Per-chunk cost excludes the completed prefix; the result is token-identical to a from-scratch tokenization. Non-append input and a resolved-grammar change reset the cache and re-tokenize fully. Each run carries the style shiki's HTML arm would assign it — the css-variables color plus the markup font-style bits the theme lets through (bold/italic/underline; markdown fences carry them); whitespace-only runs fold into their following token as shiki's default `mergeWhitespaces` does (its underlined/struck-whitespace exemption cannot occur under this theme, whose only underline rule styles inline-link scopes that tokenize spaced text as one run); and a CRLF cut never leaks its `\r` into the last completed line, matching shiki's own line splitting — so the streaming spans and the settled `codeToHtml` swap render one identical span tree.
|
||||
- **`CodeBlock`** gains a `streaming` prop: it renders the session's spans as a `pre.shiki.css-variables` React tree with the same attributes shiki's HTML emits, holds the session and per-line elements in refs, and reuses a retained line's element identity so React leaves that line's DOM untouched. Unknown or absent languages keep the identical-geometry plain arm; a lazy grammar renders plain until it registers, then the existing `useSyncExternalStore` load signal re-renders into highlight — one plain→highlighted transition, no flicker back.
|
||||
- **`render.tsx`** passes `lang` and `context.streaming` to fences. Wrong-grammar transients are structurally impossible: a fence whose info string is still mid-chunk (`` ```py `` completing to `` ```python ``) has no content yet — content only exists after the info line's newline, which finalizes the language — and the empty-value fence keeps the stock `<pre>`. The streaming CodeBlock instance survives every chunk because streaming render keys are source offsets. `` ```math `` fences and TeX stay literal until the settled pass; the language banner shows the fence language during streaming.
|
||||
- **`IncrementalMarkdownParser`** (`packages/client/ui-primitives/src/markdown/incremental.ts`) recognizes a parser-confirmed final unclosed top-level fence after the ordinary tail parse. Completed content remains in the retained `code` node; only the last completed line and current partial line pass through the caller's GFM grammar, preserving its newline, indentation, CRLF, and value semantics without re-parsing the fence prefix. A closing delimiter, non-append input, nested/container fence, or ambiguous reconstruction returns to the ordinary tail parse.
|
||||
- **`StreamingHighlightSession`** (`packages/client/ui-primitives/src/markdown/highlight.ts`) exploits that TextMate tokenization is line-based and forward-only: a line's tokens depend only on its own text and the grammar state entering it, so appended text never changes a completed line's tokens. The session caches completed lines' spans plus shiki's `GrammarState` after them (`getLastGrammarState`); `updateFrame` publishes only newly completed lines plus the still-growing last line, while the compatibility `update` method materializes the complete result. Per-chunk tokenization excludes the completed prefix; the result is token-identical to a from-scratch tokenization. Non-append input and a resolved-grammar change reset the cache and re-tokenize fully. Each run carries the style shiki's HTML arm would assign it — the css-variables color plus the markup font-style bits the theme lets through (bold/italic/underline; markdown fences carry them); whitespace-only runs fold into their following token as shiki's default `mergeWhitespaces` does (its underlined/struck-whitespace exemption cannot occur under this theme, whose only underline rule styles inline-link scopes that tokenize spaced text as one run); and a CRLF cut never leaks its `\r` into the last completed line, matching shiki's own line splitting.
|
||||
- **`CodeBlock`** renders the delta frames as a `pre.shiki.css-variables` React tree with the same attributes and token spans shiki's HTML emits. Completed lines seal into fixed-size React fragments; later updates reuse those fragment elements and reconcile only a bounded pending group plus the mutable tail. Unknown or absent languages keep the identical-geometry plain arm; a lazy grammar renders plain until it registers, then the existing `useSyncExternalStore` load signal re-renders into highlight — one plain→highlighted transition, no flicker back. Group size is an internal reconciliation unit, not a deployment policy or content limit.
|
||||
- **`render.tsx` and `MarkdownText`** pass `lang` and `context.streaming` to fences and key both streaming and settled top-level blocks by source offset. Wrong-grammar transients are structurally impossible: a fence whose info string is still mid-chunk (`` ```py `` completing to `` ```python ``) has no content yet — content only exists after the info line's newline, which finalizes the language — and the empty-value fence keeps the stock `<pre>`. `` ```math `` fences and TeX stay literal until the settled pass; the language banner shows the fence language during streaming.
|
||||
|
||||
The settle swap re-renders through `highlightToHtml`: same tokens, same span tree, so the swap is visually invisible and never touches the code content.
|
||||
The final full-document parse still resolves document-wide references and math. When that parse produces the same fence code and language, the source-offset key preserves its `CodeBlock` instance and the component reuses the complete streamed React tree; cold settled fences continue through `highlightToHtml`.
|
||||
|
||||
## Testing
|
||||
|
||||
Package tests cover incremental/from-scratch equivalence across multiline grammar state, blank lines, CRLF, and markup styles; cache identity and reset/lazy paths; streaming/settled token-tree parity; DOM retention; and plain or math fallbacks. The assembled Web browser snapshot boots the real Web composition, streams a TypeScript fence through the Host and SSE path, pauses the deterministic LLM adapter while the reply is still active, and snapshots Chromium's Shiki token tree before verifying that settlement preserves it. The `tests/fixtures/markdown-dom/*.streaming.txt` fixtures pin the intentional streaming divergence from their react-markdown origin: the Shiki span tree and visible language banner replace the plain arm.
|
||||
Package tests bound the grammar input accumulated across 800 open-fence lines, compare each incremental result with a full parse, and cover indented delimiters, CRLF split across chunks, closure fallback, and non-append reset. Highlighter tests cover incremental/from-scratch equivalence across multiline grammar state, blank lines, CRLF, and markup styles; delta identity and reset/lazy paths; fixed-group DOM retention; streamed-to-settled DOM identity; and plain or math fallbacks. The assembled Web browser snapshot boots the real Web composition, streams a TypeScript fence through the Host and SSE path, pauses the deterministic LLM adapter while the reply is still active, and snapshots Chromium's Shiki token tree before verifying that settlement preserves it. The `tests/fixtures/markdown-dom/*.streaming.txt` fixtures pin the intentional streaming divergence from their react-markdown origin: the Shiki span tree and visible language banner replace the plain arm.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
|
@ -28,10 +29,12 @@ Package tests cover incremental/from-scratch equivalence across multiline gramma
|
|||
|
||||
**Highlight only frozen (closed, settled-position) fences during streaming.** Bounded cost, but an unclosed fence pins the incremental parser's tail, so the actively growing fence — the one on screen — would stay plain until the reply finishes, failing the issue's "识别语言后即可增量高亮".
|
||||
|
||||
**Keep only a fixed window of highlighted lines and turn the older prefix into plain text.** This bounds live token DOM and can reduce layout further, but changes already rendered content, complicates selection across the window boundary, and makes a tunable presentation policy part of `CodeBlock`. Retained parser, tokenizer, and React frontiers remove the avoidable repeated work without discarding colors; the complete token DOM remains an explicit limitation rather than a hidden semantic change.
|
||||
|
||||
**Move highlighting to a worker or async pass.** Rejected when shiki was adopted ([synchronous highlighting note](../process/2026-07-26-web-syntax-highlighting-shiki.md)); an async swap also reintroduces the plain→colored→plain flicker class this change must avoid.
|
||||
|
||||
**Build the settled HTML string incrementally and keep `dangerouslySetInnerHTML`.** Exact settled parity for free, but React replaces the whole `innerHTML` per chunk, so the browser re-parses and rebuilds every line's DOM each time — O(fence) DOM churn that forfeits the token-level win the session provides.
|
||||
|
||||
## Consequences
|
||||
|
||||
Streaming code is readable as it arrives: tokens color as soon as the language is known, completed lines never re-tokenize or re-render, and the finalize swap is invisible for fences. The package owns a small mirror of shiki's HTML-arm conventions — the `pre` attributes and the whitespace fold — pinned by the arm-parity test, so a shiki upgrade that changes either fails loud there instead of drifting the two arms apart. The streaming DOM-parity fixtures pin Shiki span trees as an intentional divergence from their react-markdown origin. The still-growing last line re-tokenizes per chunk (bounded by one line), and a pathological single-line fence still degrades to full re-tokenization per chunk — the same degradation class the incremental block parser accepts for a single giant block.
|
||||
Streaming code is readable as it arrives: tokens color as soon as the language is known; completed top-level fence content neither re-parses nor re-tokenizes; sealed React groups keep their elements and DOM; and settlement preserves the highlighted tree. The package owns a small mirror of shiki's HTML-arm conventions — the `pre` attributes and the whitespace fold — pinned by the arm-parity test, so a shiki upgrade that changes either fails loud there instead of drifting the two arms apart. The streaming DOM-parity fixtures pin Shiki span trees as an intentional divergence from their react-markdown origin. The retained DOM still grows with final token count, so browser style and layout work is not length-independent. Nested/container fences use the ordinary tail parser, and the still-growing last line re-tokenizes per chunk; a pathological single-line fence therefore remains the worst case.
|
||||
|
|
|
|||
|
|
@ -10,17 +10,18 @@ Status: implemented
|
|||
|
||||
## Decision
|
||||
|
||||
流式围栏通过 grammar state 续接实现增量高亮；定稿臂保持不变。
|
||||
流式围栏从保留的解析、tokenize 与 reconcile 前沿继续推进；定稿输出保持不变。
|
||||
|
||||
- **`StreamingHighlightSession`**（`packages/client/ui-primitives/src/markdown/highlight.ts`）利用 TextMate tokenize 按行、且只向前推进的性质：一行的 token 只取决于该行文本与进入该行时的 grammar state，因此追加的文本永远不会改变已完成行的 token。会话缓存已完成行的 span 以及其后的 shiki `GrammarState`（`getLastGrammarState`），每次更新通过 `codeToTokensBase(…, { grammarState })` tokenize 新完成的文本，外加仍在增长的最后一行。每分片成本不包含已完成的前缀；结果与从头 tokenize 逐 token 一致。非追加输入与解析后语法变化会重置缓存并完整重新 tokenize。每个 run 携带 shiki HTML 臂会赋予它的样式——css-variables 颜色加上主题放行的 markup 字体位（bold/italic/underline；markdown 围栏会携带它们）；纯空白 run 并入其后的 token，与 shiki 默认的 `mergeWhitespaces` 一致（其对带下划线/删除线空白的豁免在该主题下不可能出现：主题唯一的 underline 规则作用于 inline-link scope，其含空格文本整体成一个 run）；CRLF 切割点的 `\r` 绝不进入最后一个已完成行，与 shiki 自身的行切分一致——因此流式 span 与定稿 `codeToHtml` 换入的 span 树完全一致。
|
||||
- **`CodeBlock`** 新增 `streaming` prop：把会话的 span 渲染为带有 shiki HTML 同款属性的 `pre.shiki.css-variables` React 树，用 ref 持有会话与逐行元素，并复用保留行的元素标识，让 React 完全不触碰该行的 DOM。未知或缺失语言保持几何一致的纯文本臂；懒加载语法在注册前渲染纯文本，注册后由既有的 `useSyncExternalStore` 加载信号触发重渲染进入高亮——只有一次纯文本→高亮的转换，不会闪回。
|
||||
- **`render.tsx`** 向围栏传递 `lang` 与 `context.streaming`。错误语法的瞬时着色在结构上不可能出现：info string 尚在分片中途的围栏（`` ```py `` 补全为 `` ```python ``）还没有内容——内容只在 info 行的换行之后才存在，而该换行恰恰定格了语言——空值围栏保持原生 `<pre>`。流式渲染 key 是源偏移，围栏的 CodeBlock 实例因此跨分片存活。`` ```math `` 围栏与 TeX 在定稿前保持字面量；语言横幅在流式期间显示围栏语言。
|
||||
- **`IncrementalMarkdownParser`**（`packages/client/ui-primitives/src/markdown/incremental.ts`）会在普通尾部解析后识别经 parser 确认、位于末尾且未闭合的顶层 fence。已完成内容保留在既有 `code` node 中；只有最后一个已完成行与当前未完成行再次进入调用方的 GFM grammar，因此无需重新解析 fence 前缀，也能保留其换行、缩进、CRLF 与 value 语义。出现闭合分隔符、非追加输入、嵌套／容器内 fence 或无法明确重建时，会回到普通尾部解析。
|
||||
- **`StreamingHighlightSession`**（`packages/client/ui-primitives/src/markdown/highlight.ts`）利用 TextMate tokenize 按行、且只向前推进的性质：一行的 token 只取决于该行文本与进入该行时的 grammar state，因此追加的文本永远不会改变已完成行的 token。会话缓存已完成行的 span 以及其后的 shiki `GrammarState`（`getLastGrammarState`）；`updateFrame` 只发布新完成行与仍在增长的最后一行，兼容方法 `update` 则物化完整结果。每分片 tokenize 成本不包含已完成的前缀；结果与从头 tokenize 逐 token 一致。非追加输入与解析后语法变化会重置缓存并完整重新 tokenize。每个 run 携带 shiki HTML 臂会赋予它的样式——css-variables 颜色加上主题放行的 markup 字体位（bold/italic/underline；markdown 围栏会携带它们）；纯空白 run 并入其后的 token，与 shiki 默认的 `mergeWhitespaces` 一致（其对带下划线/删除线空白的豁免在该主题下不可能出现：主题唯一的 underline 规则作用于 inline-link scope，其含空格文本整体成一个 run）；CRLF 切割点的 `\r` 绝不进入最后一个已完成行，与 shiki 自身的行切分一致。
|
||||
- **`CodeBlock`** 把增量 frame 渲染为带有 shiki HTML 同款属性与 token span 的 `pre.shiki.css-variables` React 树。已完成行会封入固定大小的 React fragment；后续更新复用这些 fragment element，只 reconcile 一个有界的待完成分组与可变尾部。未知或缺失语言保持几何一致的纯文本臂；懒加载语法在注册前渲染纯文本，注册后由既有的 `useSyncExternalStore` 加载信号触发重渲染进入高亮——只有一次纯文本→高亮的转换，不会闪回。分组大小只是内部 reconcile 单元，不是部署策略或内容上限。
|
||||
- **`render.tsx` 与 `MarkdownText`** 向围栏传递 `lang` 与 `context.streaming`，并让流式和定稿的顶层 block 都按源偏移设置 key。错误语法的瞬时着色在结构上不可能出现：info string 尚在分片中途的围栏（`` ```py `` 补全为 `` ```python ``）还没有内容——内容只在 info 行的换行之后才存在，而该换行恰恰定格了语言——空值围栏保持原生 `<pre>`。`` ```math `` 围栏与 TeX 在定稿前保持字面量；语言横幅在流式期间显示围栏语言。
|
||||
|
||||
定稿切换经 `highlightToHtml` 重渲染：token 相同、span 树相同，切换在视觉上不可见，也绝不触碰代码内容。
|
||||
最终的全量文档解析仍会解决跨文档引用与数学语法。当该解析产生相同的 fence 代码与语言时，源偏移 key 会保留其 `CodeBlock` 实例，组件则复用完整的流式 React 树；冷启动的定稿 fence 继续使用 `highlightToHtml`。
|
||||
|
||||
## Testing
|
||||
|
||||
包测试覆盖跨多行 grammar state、空行、CRLF 与 markup 样式的增量／从头等价性，缓存标识与重置／懒加载路径，流式／定稿 token 树一致性，DOM 保留，以及纯文本和 math 回退。组装后的 Web 浏览器快照会启动真实 Web 组合，让 TypeScript 围栏经过 Host 与 SSE 路径流式传输，在回复仍活跃时暂停确定性 LLM 适配器并对 Chromium 中的 Shiki token 树做快照，然后验证定稿保留该 token 树。`tests/fixtures/markdown-dom/*.streaming.txt` fixture 锁定相对 react-markdown 来源的一项有意分叉：Shiki span 树与可见语言横幅取代纯文本臂。
|
||||
包测试会约束 800 行未闭合 fence 的累计 grammar 输入量、逐次比较增量结果与全量解析，并覆盖缩进分隔符、跨分片 CRLF、闭合回退与非追加重置。高亮测试覆盖跨多行 grammar state、空行、CRLF 与 markup 样式的增量／从头等价性，delta 标识与重置／懒加载路径，固定分组的 DOM 保留，从流式到定稿的 DOM 标识，以及纯文本和 math 回退。组装后的 Web 浏览器快照会启动真实 Web 组合，让 TypeScript 围栏经过 Host 与 SSE 路径流式传输，在回复仍活跃时暂停确定性 LLM 适配器并对 Chromium 中的 Shiki token 树做快照，然后验证定稿保留该 token 树。`tests/fixtures/markdown-dom/*.streaming.txt` fixture 锁定相对 react-markdown 来源的一项有意分叉：Shiki span 树与可见语言横幅取代纯文本臂。
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
|
@ -28,10 +29,12 @@ Status: implemented
|
|||
|
||||
**流式期间只高亮已冻结（闭合且位置定格）的围栏。** 成本有界，但未闭合围栏会钉住增量解析器的尾部，于是正在增长的围栏——屏幕上的那个——要等回复结束才高亮，不满足 issue 的"识别语言后即可增量高亮"。
|
||||
|
||||
**只保留固定窗口内的高亮行，并把更早的前缀转成纯文本。** 这能限制流式 token DOM 并进一步降低布局成本，但会改变已经渲染的内容、让跨窗口边界的选择更复杂，还会把可调的展示策略塞进 `CodeBlock`。保留解析、tokenize 与 React 前沿可以在不丢颜色的情况下消除可避免的重复工作；完整 token DOM 被明确记录为限制，而不是隐藏的语义变化。
|
||||
|
||||
**把高亮移到 worker 或异步流程。** 采纳 shiki 时已否决（[同步高亮笔记](../process/2026-07-26-web-syntax-highlighting-shiki.zh.md)）；异步换入还会重新引入本变更必须避免的纯文本→彩色→纯文本闪烁类问题。
|
||||
|
||||
**增量拼接定稿 HTML 字符串并继续使用 `dangerouslySetInnerHTML`。** 白得定稿一致性，但 React 每个分片都会整体替换 `innerHTML`，浏览器每次重新解析并重建所有行的 DOM——O(围栏) 的 DOM 翻搅，抵消了会话在 token 层的收益。
|
||||
|
||||
## Consequences
|
||||
|
||||
流式代码随到达即可读：语言一经识别 token 即着色，已完成行绝不重新 tokenize 或重渲染，定稿切换对围栏而言不可见。该包持有一小份 shiki HTML 臂约定的镜像——`pre` 属性与空白折叠——由双臂一致性测试锁定，shiki 升级若改变任一处会在该测试处响亮失败，而不是让两臂悄然漂移。流式 DOM 一致性 fixture 锁定 Shiki span 树，这是相对其 react-markdown 来源的一项有意分叉。仍在增长的最后一行每分片重新 tokenize（以一行为界）；病态的单行超长围栏仍退化为每分片完整重新 tokenize——与增量块解析器对单个巨型块接受的是同一退化类。
|
||||
流式代码随到达即可读：语言一经识别 token 即着色；顶层 fence 的已完成内容不再重新解析或 tokenize；封存的 React 分组保留其 element 与 DOM；定稿也会保留高亮树。该包持有一小份 shiki HTML 臂约定的镜像——`pre` 属性与空白折叠——由双臂一致性测试锁定，shiki 升级若改变任一处会在该测试处响亮失败，而不是让两臂悄然漂移。流式 DOM 一致性 fixture 锁定 Shiki span 树，这是相对其 react-markdown 来源的一项有意分叉。保留的 DOM 仍随最终 token 数增长，因此浏览器 style 与 layout 工作量并非与长度无关。嵌套／容器内 fence 使用普通尾部解析器，仍在增长的最后一行则会每分片重新 tokenize；病态的单行超长 fence 因而仍是最坏情况。
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-25-loaded-turn-chat-navigation.md
|
||||
2026-08-25-loaded-turn-chat-navigation.md: 5d9d93b07f7a8c527bf7376bf111c6a709afa4d1
|
||||
2026-08-25-loaded-turn-chat-navigation.zh.md: 21dba024710b306848fee9bc4fe1f16913c475b3
|
||||
2026-08-25-loaded-turn-chat-navigation.md: b01e64d23fbc87f61c0de5b32cdf293c8d0afa94
|
||||
2026-08-25-loaded-turn-chat-navigation.zh.md: 85397c4a33cbc4ff1a08d253845f0112fe9b051b
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ The rail renders the complete loaded Turn set with a 10px natural interval and n
|
|||
|
||||
The rail sits against the scrollport's right edge and centers on the band the sticky composer leaves visible. That band is the scrollport's own height minus the seat's, so ConversationRoot publishes `--dsh-conversation-viewport-height` beside the `--dsh-composer-height` it already measures on the same element, and the rail centers on their difference instead of a viewport height that ignores the Session header.
|
||||
|
||||
The active mark follows a reading line near the top of the shared Chat scrollport. A scroll frame resolves the owning Turn with one hit test at that line, falling back to a single row scan where layout cannot answer, so cost does not grow with the number of marks. Flow-height changes that move rows across the line without a scroll event resync through the existing column observer. Scroll updates are coalesced with `requestAnimationFrame`; reaching the bottom selects the final loaded Turn. Activating a mark computes the target node's position in the existing scroll coordinate system, moves that same scrollport, and records the resulting Chat scroll-restoration anchor.
|
||||
The active mark follows a reading line near the top of the shared Chat scrollport. A pinned frame selects the final loaded Turn from scroll distance before reading any row geometry; streaming and other observed height changes can therefore follow the floor without a hit test or scan. Away from the floor, a scroll frame resolves the owning Turn with one hit test at the reading line, falling back to a single row scan where layout cannot answer, so cost does not grow with the number of marks. Flow-height changes that move rows across the line without a scroll event resync through the existing column observer. Scroll updates are coalesced with `requestAnimationFrame`. Activating a mark computes the target node's position in the existing scroll coordinate system, moves that same scrollport, and records the resulting Chat scroll-restoration anchor.
|
||||
|
||||
Every Turn remains an accessible button even when dense marks visually overlap. The rail maps pointer height to the nearest loaded Turn, while keyboard focus and activation operate the individual buttons. Hover and focus show a compact prompt-and-response preview, the active mark is longer and darker, the rail is hidden when the Chat container is at most 900px wide, and reduced-motion preferences disable redistribution and mark-entry animation.
|
||||
|
||||
|
|
@ -42,4 +42,4 @@ Desktop-width Chat views can jump among all currently loaded Turns and inspect a
|
|||
|
||||
## Testing
|
||||
|
||||
Builder tests pin the accumulated projection, the bounded preview, and preview freshness under an in-place chunk update. Component tests pin the published items, accessible previews, scroll-coordinate jumps, DOM identity, and percentage redistribution after prepend. The long-interaction Chromium scenario pins the real paginated boundary, prompt completion after `Load earlier`, stable-mark movement, keyboard activation, active-state update, and the narrow-container hide. The multi-Turn recorded Web snapshot includes the navigation landmark and buttons.
|
||||
Builder tests pin the accumulated projection, the bounded preview, and preview freshness under an in-place chunk update. Component tests pin the published items, accessible previews, scroll-coordinate jumps, DOM identity, percentage redistribution after prepend, and a pinned `ResizeObserver` update that rejects every row-geometry read. The long-interaction Chromium scenario pins the real paginated boundary, prompt completion after `Load earlier`, stable-mark movement, keyboard activation, active-state update, and the narrow-container hide. The multi-Turn recorded Web snapshot includes the navigation landmark and buttons.
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ Chat snapshot 构建层为当前已加载且含可见 transcript node 的每个
|
|||
|
||||
轨道紧贴滚动视口右缘，并在粘性输入区之外的可见区间内垂直居中。该区间等于滚动视口自身高度减去输入区高度，因此 ConversationRoot 在同一元素上除已有的 `--dsh-composer-height` 外再发布 `--dsh-conversation-viewport-height`，轨道按两者之差居中，而不是按忽略 Session 头部的视口高度居中。
|
||||
|
||||
活跃刻度跟随共享 Chat 滚动区顶部附近的阅读线。每个滚动帧用一次命中测试解析该行所属 Turn，布局无法作答时退化为一次行扫描，成本不随刻度数量增长。图片加载、工具卡展开等不产生滚动事件的高度变化，通过既有的 column observer 重新同步。滚动更新由 `requestAnimationFrame` 合并；到达底部时选择最后一个已加载 Turn。激活刻度会在现有滚动坐标系中计算目标 node 的位置，移动同一个滚动区，并记录由此产生的 Chat 滚动恢复锚点。
|
||||
活跃刻度跟随共享 Chat 滚动区顶部附近的阅读线。跟随底部的 frame 会先按滚动距离选中最后一个已加载 Turn，不读取任何行几何；流式输出及其他被 observer 捕获的高度变化因此无需命中测试或扫描即可追随底部。离开底部后，每个滚动 frame 用一次命中测试解析阅读线所属 Turn，布局无法作答时退化为一次行扫描，成本不随刻度数量增长。不产生滚动事件却让行跨过阅读线的高度变化通过既有的 column observer 重新同步。滚动更新由 `requestAnimationFrame` 合并。激活刻度会在现有滚动坐标系中计算目标 node 的位置，移动同一个滚动区，并记录由此产生的 Chat 滚动恢复锚点。
|
||||
|
||||
即使密集刻度在视觉上重叠，每个 Turn 仍是可访问的按钮。轨道把指针高度映射到最近的已加载 Turn，键盘聚焦和激活则作用于各个按钮。悬停或聚焦显示紧凑的问题与回复预览，活跃刻度更长、更深；Chat 容器宽度不超过 900px 时隐藏轨道，用户偏好减少动态效果时关闭重排和刻度入场动画。
|
||||
|
||||
|
|
@ -42,4 +42,4 @@ Chat snapshot 构建层为当前已加载且含可见 transcript node 的每个
|
|||
|
||||
## 测试
|
||||
|
||||
构建层测试固定累积投影、预览截断，以及原地 chunk 更新后的预览新鲜度。组件测试固定已发布条目、可访问预览、滚动坐标跳转、DOM 身份以及前插后的百分比重排。长交互 Chromium 场景固定真实分页边界、`加载更早` 后补齐问题、稳定刻度移动、键盘激活、活跃状态更新与窄容器隐藏。多 Turn 的 Web 录制快照包含导航 landmark 和按钮。
|
||||
构建层测试固定累积投影、预览截断，以及原地 chunk 更新后的预览新鲜度。组件测试固定已发布条目、可访问预览、滚动坐标跳转、DOM 身份、前插后的百分比重排，以及拒绝任何行几何读取的底部跟随 `ResizeObserver` 更新。长交互 Chromium 场景固定真实分页边界、`加载更早` 后补齐问题、稳定刻度移动、键盘激活、活跃状态更新与窄容器隐藏。多 Turn 的 Web 录制快照包含导航 landmark 和按钮。
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/client/ui-chat/README.md
|
||||
README.md: 9fe462316e6ab6d664a43b9fc481e07b01eda093
|
||||
README.zh.md: bad8a9143cd0307cdc5edb3e66bb9984cd5ca2be
|
||||
README.md: 1b24e6bb6825826a97a22a32f43d0210a2f4c6ad
|
||||
README.zh.md: 86596bc73ffd1b41867af6559739ab5426adb7b1
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ The browser Chat target for Conversation assembly. It registers Chat event defin
|
|||
- [System prompt row](#system-prompt-row)
|
||||
- [Turn token usage](#turn-token-usage)
|
||||
- [Turn Process Folding](#turn-process-folding)
|
||||
- [Scroll ownership](#scroll-ownership)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
|
@ -42,6 +43,13 @@ Settings → General exposes a persisted `Normal` / `Compact` conversation-displ
|
|||
|
||||
-----
|
||||
|
||||
<a id="scroll-ownership"></a>
|
||||
## Scroll ownership
|
||||
|
||||
Chat restores semantic anchors across history prepend and renderer remounts. While the reader is pinned to the floor, `ResizeObserver` follows the new floor and selects the latest loaded Turn without reading row geometry. Once the reader moves away, flow-height changes preserve the top position and the reading-line geometry selects the active Turn ([loaded-Turn navigation](../../../.agents/notes/implemented/feature/2026-08-25-loaded-turn-chat-navigation.md)).
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ Conversation 组装的浏览器 Chat target。本包注册 Chat event definition
|
|||
- [系统提示词行](#system-prompt-row)
|
||||
- [轮次 token 用量](#turn-token-usage)
|
||||
- [轮次过程折叠](#turn-process-folding)
|
||||
- [滚动归属](#scroll-ownership)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与暂缓事项](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
|
@ -42,6 +43,13 @@ Chat 会为每个非空的初始或恢复请求、显式消息序列起点或真
|
|||
|
||||
-----
|
||||
|
||||
<a id="scroll-ownership"></a>
|
||||
## 滚动归属
|
||||
|
||||
Chat 会在历史前插与 renderer 重新挂载时恢复语义锚点。读者跟随底部时，`ResizeObserver` 追随新的底部，并且无需读取行几何就选中最后一个已加载 Turn；读者离开底部后，高度变化会保持顶部位置，再由阅读线几何选择活跃 Turn（[已加载 Turn 导航](../../../.agents/notes/implemented/feature/2026-08-25-loaded-turn-chat-navigation.zh.md)）。
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
|
|
|
|||
|
|
@ -321,6 +321,11 @@ export function ChatView({
|
|||
return
|
||||
}
|
||||
const el = scrollerOf(local)
|
||||
if (el.scrollHeight - el.scrollTop - el.clientHeight <= FOLLOW_THRESHOLD + 1) {
|
||||
const latest = turnNavigationItems.at(-1)?.turn ?? first.turn
|
||||
setActiveTurn(current => current === latest ? current : latest)
|
||||
return
|
||||
}
|
||||
const readingLine = el.getBoundingClientRect().top + Math.min(96, el.clientHeight * 0.2)
|
||||
const reading = turnAtLine(local, readingLine)
|
||||
// No row reaches the line yet: the flow head still owns the mark. Otherwise
|
||||
|
|
@ -333,9 +338,6 @@ export function ChatView({
|
|||
next = item.turn
|
||||
}
|
||||
}
|
||||
if (el.scrollHeight - el.scrollTop - el.clientHeight <= FOLLOW_THRESHOLD + 1) {
|
||||
next = turnNavigationItems.at(-1)?.turn ?? next
|
||||
}
|
||||
setActiveTurn(current => current === next ? current : next)
|
||||
}, [turnNavigationItems])
|
||||
|
||||
|
|
|
|||
|
|
@ -2179,6 +2179,58 @@ describe('ChatView', () => {
|
|||
expect(observe).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('pinned dynamic-height updates select the latest Turn without reading row geometry', () => {
|
||||
let notify: (() => void) | undefined
|
||||
let nextFrame = 0
|
||||
const frames = new Map<number, FrameRequestCallback>()
|
||||
vi.stubGlobal('requestAnimationFrame', (callback: FrameRequestCallback) => {
|
||||
nextFrame += 1
|
||||
frames.set(nextFrame, callback)
|
||||
return nextFrame
|
||||
})
|
||||
vi.stubGlobal('cancelAnimationFrame', (id: number) => { frames.delete(id) })
|
||||
class ResizeObserverStub {
|
||||
constructor(callback: ResizeObserverCallback) {
|
||||
notify = () => { callback([], this as unknown as ResizeObserver) }
|
||||
}
|
||||
|
||||
observe = vi.fn()
|
||||
disconnect = vi.fn()
|
||||
}
|
||||
vi.stubGlobal('ResizeObserver', ResizeObserverStub)
|
||||
const rect = vi.spyOn(HTMLElement.prototype, 'getBoundingClientRect')
|
||||
.mockReturnValue({ top: 0, bottom: 40 } as DOMRect)
|
||||
const h = makeHarness({
|
||||
nodes: [
|
||||
userInTurn(1, 'first', 1), assistant(2, 'first answer', 1),
|
||||
userInTurn(4, 'second', 2), assistant(5, 'second answer', 2),
|
||||
],
|
||||
turnEnds: new Map([[1, 3], [2, 6]]),
|
||||
})
|
||||
const view = render(<h.ChatView {...h.props} />)
|
||||
const scroller = view.container.querySelector('[class*="scroll"]') as HTMLDivElement
|
||||
const metrics = installScrollMetrics(scroller, 1_000, 300)
|
||||
scroller.scrollTop = 700
|
||||
act(() => {
|
||||
const pending = [...frames.values()]
|
||||
frames.clear()
|
||||
for (const callback of pending) callback(0)
|
||||
})
|
||||
rect.mockClear()
|
||||
|
||||
metrics.setHeight(1_200)
|
||||
act(() => { notify?.() })
|
||||
act(() => {
|
||||
const pending = [...frames.values()]
|
||||
frames.clear()
|
||||
for (const callback of pending) callback(0)
|
||||
})
|
||||
|
||||
expect(scroller.scrollTop).toBe(900)
|
||||
expect(view.getByRole('button', { name: '跳转到第 2 轮' }).getAttribute('aria-current')).toBe('true')
|
||||
expect(rect).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('entering the at-bottom threshold does not snap the remaining scroll distance', () => {
|
||||
const h = makeHarness({ nodes: [user(1, 'q'), assistant(2, 'a')] })
|
||||
const view = render(<h.ChatView {...h.props} />)
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/client/ui-primitives/README.md
|
||||
README.md: 42c1110e8735dd2191c8b7a2e4dcc1f2b9b938bc
|
||||
README.zh.md: 9f3c06cacebb7a596204dbe1c8e00a549ce1fcae
|
||||
README.md: 525c2c01ec3f3aa3f6146beb3b3b7a6209998a85
|
||||
README.zh.md: b2758c5de1f63d6e211dc462c2386bb9e59e4826
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ Compose feature UI from these atoms whenever the web client needs a standard con
|
|||
|
||||
### Rendering agent output
|
||||
|
||||
`MarkdownText` renders untrusted GFM and TeX math, blocks unsafe links and images, and can turn resolved file mentions into explicit controls. While a reply streams, it freezes completed blocks and highlights a growing fence from saved Shiki grammar state; the final render uses the same span tree ([incremental renderer](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.md), [streaming fence highlighting](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.md)). `TerminalBlock`, `ReadBlock`, `DiffBlock`, `SearchBlock`, and `WebBlock` render the matching tool-result intent with copy controls, overflow handling, and ANSI processing where applicable. `JsonTree` and `JsonBlock` inspect JSON values read-only, while `MessageText` remains the literal-text primitive for user-authored content.
|
||||
`MarkdownText` renders untrusted GFM and TeX math, blocks unsafe links and images, and can turn resolved file mentions into explicit controls. While a reply streams, it freezes completed blocks, advances a top-level open fence by completed lines, and highlights that fence from saved Shiki grammar state. Completed token lines enter fixed-size React groups, so later chunks reconcile only the growing group; an unchanged fence retains that DOM when the final full parse resolves cross-document syntax ([incremental renderer](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.md), [streaming fence highlighting](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.md)). `TerminalBlock`, `ReadBlock`, `DiffBlock`, `SearchBlock`, and `WebBlock` render the matching tool-result intent with copy controls, overflow handling, and ANSI processing where applicable. `JsonTree` and `JsonBlock` inspect JSON values read-only, while `MessageText` remains the literal-text primitive for user-authored content.
|
||||
|
||||
### Localizing copy
|
||||
|
||||
|
|
@ -63,7 +63,7 @@ The package is one separation: presentational React atoms with zero Cordis and z
|
|||
|
||||
### Streaming markdown
|
||||
|
||||
While a reply streams, `MarkdownText` parses incrementally: all but the trailing two blocks freeze as cached React elements and only the source tail re-parses per chunk, so per-chunk work tracks the tail instead of the whole reply. A growing fenced block tokenizes completed text from saved Shiki grammar state plus the unfinished last line; completed lines retain their DOM, and the settled render uses the same span tree. The settled full parse at finalize also resolves references that crossed the freeze boundary ([incremental renderer](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.md), [streaming fence highlighting](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.md)).
|
||||
While a reply streams, `MarkdownText` parses incrementally: all but the trailing two blocks freeze as cached React elements and only the source tail re-parses per chunk, so per-chunk work tracks the tail instead of the whole reply. A final unclosed top-level fence keeps its parsed code node and sends only the last completed line plus the current partial line through the same GFM grammar; a closing fence or ambiguous parse returns to the ordinary tail path. Highlighting likewise resumes from saved Shiki grammar state and publishes only newly completed lines plus the mutable tail. `CodeBlock` seals completed lines into fixed-size React groups, reuses earlier groups, and retains the whole highlighted tree across settlement when code and language are unchanged. The settled full parse still resolves references that crossed the freeze boundary ([incremental renderer](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.md), [streaming fence highlighting](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.md)).
|
||||
|
||||
### Geometry and overflow
|
||||
|
||||
|
|
@ -103,6 +103,7 @@ None; this package neither assembles nor sends a provider request.
|
|||
These limits define how the atoms behave at the edges; they are current package constraints, not a component roadmap.
|
||||
|
||||
- **Streaming defers cross-boundary reference resolution** — a reference-style link or footnote whose definition sits on the other side of the incremental freeze boundary renders as literal text while the reply streams; the settled full parse at finalize resolves it.
|
||||
- **A long highlighted fence retains its complete token DOM** — streaming avoids re-parsing, re-tokenizing, and reconciling the completed prefix, but it does not discard old colors or virtualize token spans. Final DOM cardinality therefore still follows the fence's token count; nested/container fences and a pathological single long line remain on the general tail path.
|
||||
- **Glyph-level icons are redrawn approximations** — the fish logo and the sparkle mark come from font glyphs whose vector geometry is not exportable from the local design data; hand-authored recreations stand in until an exact export path exists.
|
||||
- **`Pill` and `Input` have no design source** — both atoms are self-defined; the sidebar search field and view-tab strip that resemble them are consumer-owned compositions, not these atoms.
|
||||
- **No `Active` `StateDot` variant** — the supported states are done, warning, ongoing, and error.
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ kind: "package-library"
|
|||
|
||||
### 渲染 agent 输出
|
||||
|
||||
`MarkdownText` 渲染不可信的 GFM 与 TeX 公式、阻止不安全的链接与图片，并可把已解析的文件提及转换为显式控件。回复流式输出时，它冻结已完成的块，并从保存的 Shiki grammar state 为不断增长的 fence 增量高亮；最终渲染使用相同的 span 树（[增量渲染器](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.zh.md)、[流式 fence 高亮](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.zh.md)）。`TerminalBlock`、`ReadBlock`、`DiffBlock`、`SearchBlock` 与 `WebBlock` 把对应的工具结果意图渲染为带复制控件、溢出处理及适用时 ANSI 处理的卡片。`JsonTree` 与 `JsonBlock` 以只读方式检查 JSON 值；`MessageText` 仍是用户创作内容的字面文本原语。
|
||||
`MarkdownText` 渲染不可信的 GFM 与 TeX 公式、阻止不安全的链接与图片，并可把已解析的文件提及转换为显式控件。回复流式输出时，它冻结已完成的块、按已完成行推进顶层未闭合 fence，并从保存的 Shiki grammar state 为该 fence 增量高亮。已完成的 token 行进入固定大小的 React 分组，后续分片只 reconcile 正在增长的分组；最终全量解析解决跨文档语法时，未变化的 fence 会保留该 DOM（[增量渲染器](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.zh.md)、[流式 fence 高亮](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.zh.md)）。`TerminalBlock`、`ReadBlock`、`DiffBlock`、`SearchBlock` 与 `WebBlock` 把对应的工具结果意图渲染为带复制控件、溢出处理及适用时 ANSI 处理的卡片。`JsonTree` 与 `JsonBlock` 以只读方式检查 JSON 值；`MessageText` 仍是用户创作内容的字面文本原语。
|
||||
|
||||
### 本地化文案
|
||||
|
||||
|
|
@ -63,7 +63,7 @@ kind: "package-library"
|
|||
|
||||
### 流式 markdown
|
||||
|
||||
回复流式输出期间，`MarkdownText` 增量解析：除末尾两个块外全部冻结为缓存的 React 元素，每个分片只重新解析其后的源文本尾部，因此每分片的工作量跟随尾部而非整个回复。不断增长的 fenced block 会从已保存的 Shiki grammar state 加上尚未完成的最后一行继续分词；已完成行保留其 DOM，定稿渲染则使用相同的 span 树。定稿时的全量解析还会解析跨过冻结边界的引用（[增量渲染器](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.zh.md)、[流式 fence 高亮](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.zh.md)）。
|
||||
回复流式输出期间，`MarkdownText` 增量解析：除末尾两个块外全部冻结为缓存的 React 元素，每个分片只重新解析其后的源文本尾部，因此每分片的工作量跟随尾部而非整个回复。末尾的顶层未闭合 fence 会保留已解析的 code node，只把最后一个已完成行与当前未完成行交给同一套 GFM grammar；闭合 fence 或有歧义的解析会回到普通尾部路径。高亮同样从保存的 Shiki grammar state 续接，并只发布新完成行与可变尾部。`CodeBlock` 把已完成行封入固定大小的 React 分组、复用更早的分组，并在代码与语言未变化时跨定稿保留整棵高亮树。定稿时的全量解析仍会解析跨过冻结边界的引用（[增量渲染器](../../../.agents/notes/implemented/architecture/2026-08-06-web-markdown-incremental-ast-renderer.zh.md)、[流式 fence 高亮](../../../.agents/notes/implemented/feature/2026-08-20-web-streaming-fence-highlight.zh.md)）。
|
||||
|
||||
### 几何与溢出
|
||||
|
||||
|
|
@ -103,6 +103,7 @@ kind: "package-library"
|
|||
这些限制说明原子组件在边缘情况下的行为；它们是当前包约束，不是组件路线图。
|
||||
|
||||
- **流式期间跨边界引用解析被推迟**：定义落在增量冻结边界另一侧的引用式链接或脚注，在回复流式输出期间渲染为字面文本；定稿时的全量解析会将其解析。
|
||||
- **长高亮 fence 会保留完整 token DOM**：流式路径避免重新解析、重新 tokenize 和 reconcile 已完成前缀，但不会丢弃旧颜色或虚拟化 token span。因此最终 DOM 数量仍随 fence 的 token 数增长；嵌套／容器内 fence 与病态的单个超长行仍走通用尾部路径。
|
||||
- **字形级图标是重新绘制的近似版本**：鱼形标志与闪光标记来自字体字形，而本地设计数据无法导出其矢量几何；在获得精确导出路径前，使用手工重建版本代替。
|
||||
- **`Pill` 与 `Input` 没有设计来源**：两个原子组件均自行定义；与其相似的侧边栏搜索字段和视图标签条由消费方组合，不是这些原子组件。
|
||||
- **`StateDot` 没有 `Active` 变体**：支持的状态为 done、warning、ongoing 和 error。
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ import { writeClipboard } from '../clipboard.ts'
|
|||
import {
|
||||
StreamingHighlightSession, grammarLoadCount, highlightToHtml, subscribeGrammarLoaded,
|
||||
} from './highlight.ts'
|
||||
import type { HighlightSpan } from './highlight.ts'
|
||||
import type { HighlightSpan, StreamingHighlightFrame } from './highlight.ts'
|
||||
import css from './CodeBlock.module.css'
|
||||
|
||||
export interface CodeBlockProps {
|
||||
|
|
@ -16,9 +16,10 @@ export interface CodeBlockProps {
|
|||
/**
|
||||
* The code is still growing (a streaming markdown fence): highlight through
|
||||
* a per-instance {@link StreamingHighlightSession}, which re-tokenizes only
|
||||
* appended text and keeps completed lines' elements (and DOM) untouched.
|
||||
* The caller must keep the component instance stable across growth (a
|
||||
* stream-stable React key); settled callers omit this and get shiki's HTML.
|
||||
* appended text and keeps completed line groups (and DOM) untouched. The
|
||||
* caller must keep the component instance stable across growth (a
|
||||
* stream-stable React key); an unchanged streamed fence also retains that
|
||||
* tree when it settles. Cold settled callers get shiki's HTML.
|
||||
*/
|
||||
streaming?: boolean | undefined
|
||||
/** Extra class merged onto the wrapper (callers position; this component draws). */
|
||||
|
|
@ -41,49 +42,92 @@ const SHIKI_PRE_PROPS = {
|
|||
tabIndex: 0,
|
||||
} as const
|
||||
|
||||
/** Completed-line group size; React reconciles groups while the DOM remains line-for-line identical. */
|
||||
const STREAMING_LINE_GROUP_SIZE = 32
|
||||
|
||||
function renderLine(line: readonly HighlightSpan[], index: number): ReactNode {
|
||||
return (
|
||||
<Fragment key={index}>
|
||||
{index > 0 && '\n'}
|
||||
<span className="line">
|
||||
{line.map((span, spanIndex) => <span key={spanIndex} style={span.style}>{span.text}</span>)}
|
||||
</span>
|
||||
</Fragment>
|
||||
)
|
||||
}
|
||||
|
||||
export function CodeBlock({ code, lang, streaming, className, copyLabel, copiedLabel }: CodeBlockProps) {
|
||||
const trimmed = code.endsWith('\n') ? code.slice(0, -1) : code
|
||||
// Re-render when a lazy grammar finishes loading, so a fence that showed plain
|
||||
// text while its language's grammar imported picks up highlighting. The
|
||||
// snapshot value is opaque; only its change across renders drives the memo.
|
||||
const loaded = useSyncExternalStore(subscribeGrammarLoaded, grammarLoadCount, grammarLoadCount)
|
||||
const html = useMemo(
|
||||
() => (streaming === true ? undefined : highlightToHtml(trimmed, lang)),
|
||||
[streaming, trimmed, lang, loaded],
|
||||
)
|
||||
// Streaming state lives in refs mutated inside the memo (the MarkdownText
|
||||
// streaming-cache pattern): the session's caches carry across chunks only
|
||||
// because the owner keys this instance stably while the fence grows.
|
||||
const sessionRef = useRef<StreamingHighlightSession | null>(null)
|
||||
const lineCacheRef = useRef<{ lines: readonly HighlightSpan[][]; elements: ReactNode[] } | null>(null)
|
||||
const lineCacheRef = useRef<{
|
||||
code: string
|
||||
lang: string | undefined
|
||||
generation: number
|
||||
frame: StreamingHighlightFrame
|
||||
groups: ReactNode[]
|
||||
pending: ReactNode[]
|
||||
nextLine: number
|
||||
body: ReactNode
|
||||
} | null>(null)
|
||||
const settledRef = useRef(false)
|
||||
const streamedBody = useMemo(() => {
|
||||
if (streaming !== true) {
|
||||
const previous = lineCacheRef.current
|
||||
if (previous !== null && previous.code === trimmed && previous.lang === lang) {
|
||||
settledRef.current = true
|
||||
return previous.body
|
||||
}
|
||||
sessionRef.current = null
|
||||
lineCacheRef.current = null
|
||||
settledRef.current = true
|
||||
return undefined
|
||||
}
|
||||
if (settledRef.current) {
|
||||
sessionRef.current = null
|
||||
lineCacheRef.current = null
|
||||
settledRef.current = false
|
||||
}
|
||||
sessionRef.current ??= new StreamingHighlightSession()
|
||||
const lines = sessionRef.current.update(trimmed, lang)
|
||||
if (lines === undefined) {
|
||||
const frame = sessionRef.current.updateFrame(trimmed, lang)
|
||||
if (frame === undefined) {
|
||||
lineCacheRef.current = null
|
||||
return undefined
|
||||
}
|
||||
// A retained line keeps its span-array identity across chunks, so its
|
||||
// cached element is reused and React leaves that line's DOM untouched.
|
||||
const previous = lineCacheRef.current
|
||||
const elements = lines.map((line, index) => previous !== null && previous.lines[index] === line
|
||||
? previous.elements[index]
|
||||
: (
|
||||
<Fragment key={index}>
|
||||
{index > 0 && '\n'}
|
||||
<span className="line">
|
||||
{line.map((span, spanIndex) => <span key={spanIndex} style={span.style}>{span.text}</span>)}
|
||||
</span>
|
||||
</Fragment>
|
||||
))
|
||||
lineCacheRef.current = { lines, elements }
|
||||
return <pre {...SHIKI_PRE_PROPS}><code>{elements}</code></pre>
|
||||
if (previous?.frame === frame && previous.code === trimmed && previous.lang === lang) {
|
||||
return previous.body
|
||||
}
|
||||
const sameGeneration = previous?.generation === frame.generation
|
||||
const groups = sameGeneration ? [...previous.groups] : []
|
||||
let pending = sameGeneration ? [...previous.pending] : []
|
||||
let nextLine = sameGeneration ? previous.nextLine : 0
|
||||
for (const line of frame.appended) {
|
||||
pending.push(renderLine(line, nextLine))
|
||||
nextLine += 1
|
||||
if (pending.length !== STREAMING_LINE_GROUP_SIZE) continue
|
||||
const start = nextLine - pending.length
|
||||
groups.push(<Fragment key={start}>{pending}</Fragment>)
|
||||
pending = []
|
||||
}
|
||||
const tail = frame.tail.map((line, index) => renderLine(line, nextLine + index))
|
||||
const tailGroup = <Fragment key={nextLine - pending.length}>{pending}{tail}</Fragment>
|
||||
const body = <pre {...SHIKI_PRE_PROPS}><code>{groups}{tailGroup}</code></pre>
|
||||
lineCacheRef.current = {
|
||||
code: trimmed, lang, generation: frame.generation, frame, groups, pending, nextLine, body,
|
||||
}
|
||||
return body
|
||||
}, [streaming, trimmed, lang, loaded])
|
||||
const html = useMemo(
|
||||
() => (streaming !== true && streamedBody === undefined ? highlightToHtml(trimmed, lang) : undefined),
|
||||
[streaming, streamedBody, trimmed, lang, loaded],
|
||||
)
|
||||
const rootRef = useRef<HTMLDivElement>(null)
|
||||
const [copied, setCopied] = useState(false)
|
||||
|
||||
|
|
|
|||
|
|
@ -43,7 +43,11 @@ function renderSettled(
|
|||
footnoteCounts: new Map(),
|
||||
}
|
||||
const blocks = wrapBlockChildren(
|
||||
renderBlocks(root.children.map((node, index) => ({ node, key: index })), context),
|
||||
renderBlocks(root.children.map((node, index) => ({
|
||||
node,
|
||||
/* v8 ignore next -- parseFull uses parseGfm, which stamps every top-level node. */
|
||||
key: node.position?.start.offset ?? -(index + 1),
|
||||
})), context),
|
||||
false,
|
||||
)
|
||||
const section = renderFootnoteSection(context)
|
||||
|
|
|
|||
|
|
@ -333,10 +333,11 @@ function lineSpans(line: ThemedToken[]): HighlightSpan[] {
|
|||
* tokenization is line-based and forward-only — a line's tokens depend only on
|
||||
* its own text and the grammar state entering it — so appended text never
|
||||
* changes a completed line's tokens. The session caches the spans of every
|
||||
* completed line together with the grammar state after them; each
|
||||
* {@link update} tokenizes newly completed text from that state, plus the
|
||||
* still-growing last line. Per-call cost therefore excludes the completed
|
||||
* prefix, and the result equals a from-scratch tokenization of the same code.
|
||||
* completed line together with the grammar state after them;
|
||||
* {@link updateFrame} reports only newly completed lines plus the still-growing
|
||||
* last line, while {@link update} materializes the complete compatibility
|
||||
* result. Per-call tokenization cost therefore excludes the completed prefix,
|
||||
* and the result equals a from-scratch tokenization of the same code.
|
||||
* Non-append input and a change of resolved grammar reset the cache and
|
||||
* re-tokenize fully, so any input stays correct.
|
||||
*/
|
||||
|
|
@ -352,12 +353,16 @@ export class StreamingHighlightSession {
|
|||
private lastCode: string | undefined
|
||||
private lastLang: string | undefined
|
||||
private lastResult: HighlightSpan[][] | undefined
|
||||
private generation = 0
|
||||
private lastFrame: StreamingHighlightFrame | undefined
|
||||
|
||||
private reset(resolved: string | undefined): void {
|
||||
this.resolved = resolved
|
||||
this.prefix = ''
|
||||
this.spans = []
|
||||
this.state = undefined
|
||||
this.generation += 1
|
||||
this.lastFrame = undefined
|
||||
}
|
||||
|
||||
/** Tokenize `text` with `resolved`, resuming from the cached grammar state when one exists. */
|
||||
|
|
@ -369,6 +374,43 @@ export class StreamingHighlightSession {
|
|||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Tokenize one update as a delta for a retained renderer.
|
||||
* @param code - the fence text accumulated so far.
|
||||
* @param lang - the language hint.
|
||||
* @returns Newly completed lines plus the current tail, or `undefined` for the plain arm.
|
||||
*/
|
||||
updateFrame(code: string, lang: string | undefined): StreamingHighlightFrame | undefined {
|
||||
if (code === this.lastCode && lang === this.lastLang && this.lastFrame !== undefined) {
|
||||
return this.lastFrame
|
||||
}
|
||||
this.lastCode = code
|
||||
this.lastLang = lang
|
||||
this.lastResult = undefined
|
||||
const resolved = lang === undefined ? undefined : LANG_ALIASES.get(lang.toLowerCase())
|
||||
if (resolved === undefined || !ensureGrammar(resolved)) {
|
||||
this.reset(undefined)
|
||||
return undefined
|
||||
}
|
||||
if (resolved !== this.resolved || !code.startsWith(this.prefix)) this.reset(resolved)
|
||||
const firstNewLine = this.spans.length
|
||||
const rest = code.slice(this.prefix.length)
|
||||
const lastNewline = rest.lastIndexOf('\n')
|
||||
if (lastNewline >= 0) {
|
||||
const grownEnd = rest[lastNewline - 1] === '\r' ? lastNewline - 1 : lastNewline
|
||||
const tokens = this.tokenize(resolved, rest.slice(0, grownEnd))
|
||||
for (const line of tokens) this.spans.push(lineSpans(line))
|
||||
this.state = highlighter().getLastGrammarState(tokens)
|
||||
this.prefix = code.slice(0, this.prefix.length + lastNewline + 1)
|
||||
}
|
||||
this.lastFrame = {
|
||||
generation: this.generation,
|
||||
appended: this.spans.slice(firstNewLine),
|
||||
tail: this.tokenize(resolved, rest.slice(lastNewline + 1)).map(lineSpans),
|
||||
}
|
||||
return this.lastFrame
|
||||
}
|
||||
|
||||
/**
|
||||
* Tokenize the fence's current text into per-line highlighted runs;
|
||||
* `undefined` means the caller renders its plain fallback. Idempotent per
|
||||
|
|
@ -385,39 +427,23 @@ export class StreamingHighlightSession {
|
|||
if (code === this.lastCode && lang === this.lastLang && this.lastResult !== undefined) {
|
||||
return this.lastResult
|
||||
}
|
||||
this.lastCode = code
|
||||
this.lastLang = lang
|
||||
const resolved = lang === undefined ? undefined : LANG_ALIASES.get(lang.toLowerCase())
|
||||
if (resolved === undefined || !ensureGrammar(resolved)) {
|
||||
this.reset(undefined)
|
||||
this.lastResult = undefined
|
||||
return undefined
|
||||
}
|
||||
if (resolved !== this.resolved || !code.startsWith(this.prefix)) this.reset(resolved)
|
||||
const rest = code.slice(this.prefix.length)
|
||||
const lastNewline = rest.lastIndexOf('\n')
|
||||
// Everything before the last newline is newly completed lines: tokenize
|
||||
// them once from the cached state and retain their spans. What follows is
|
||||
// the still-growing line, re-tokenized per call but never retained.
|
||||
if (lastNewline >= 0) {
|
||||
// Tokenize what shiki's own line splitting would see: splitLines strips
|
||||
// the \r of a \r\n terminator (interior pairs are shiki's to split), so
|
||||
// a CRLF cut must not leak its \r into the last completed line — a bash
|
||||
// continuation's grammar state, for example, differs with it.
|
||||
const grownEnd = rest[lastNewline - 1] === '\r' ? lastNewline - 1 : lastNewline
|
||||
const tokens = this.tokenize(resolved, rest.slice(0, grownEnd))
|
||||
// Per-line push, not one spread call: a reconnect can deliver the whole
|
||||
// accumulated fence as one update, and spreading tens of thousands of
|
||||
// lines into arguments can exceed the engine's argument limit.
|
||||
for (const line of tokens) this.spans.push(lineSpans(line))
|
||||
this.state = highlighter().getLastGrammarState(tokens)
|
||||
this.prefix = code.slice(0, this.prefix.length + lastNewline + 1)
|
||||
}
|
||||
this.lastResult = [...this.spans, ...this.tokenize(resolved, rest.slice(lastNewline + 1)).map(lineSpans)]
|
||||
const frame = this.updateFrame(code, lang)
|
||||
if (frame === undefined) return undefined
|
||||
this.lastResult = [...this.spans, ...frame.tail]
|
||||
return this.lastResult
|
||||
}
|
||||
}
|
||||
|
||||
/** One retained-renderer update from {@link StreamingHighlightSession.updateFrame}. */
|
||||
export interface StreamingHighlightFrame {
|
||||
/** Changes whenever prior completed lines must be discarded. */
|
||||
readonly generation: number
|
||||
/** Completed lines added since the preceding frame in this generation. */
|
||||
readonly appended: readonly HighlightSpan[][]
|
||||
/** The still-growing final line or lines, replaced by the next frame. */
|
||||
readonly tail: readonly HighlightSpan[][]
|
||||
}
|
||||
|
||||
/**
|
||||
* Tokenize `code` into per-line highlighted runs when `lang` maps to a
|
||||
* registered grammar; `undefined` means the caller renders its plain fallback.
|
||||
|
|
|
|||
|
|
@ -4,19 +4,23 @@
|
|||
* Re-parsing the whole accumulated document on every streaming chunk is
|
||||
* quadratic in the final reply length. CommonMark block parsing is line-based
|
||||
* and appended text can only reshape the parse frontier — the last top-level
|
||||
* block (a paragraph becoming a setext heading or a table, a list continuing
|
||||
* after a blank line, an unclosed fence swallowing lines) — so earlier blocks
|
||||
* are final. This parser therefore freezes all but the trailing
|
||||
* {@link UNSTABLE_TAIL_BLOCKS} blocks and re-parses only the source tail
|
||||
* behind them: each source region is parsed O(1) times over the stream
|
||||
* instead of once per chunk.
|
||||
* block (a paragraph becoming a setext heading or a table, or a list
|
||||
* continuing after a blank line) — so earlier blocks are final. This parser
|
||||
* therefore freezes all but the trailing {@link UNSTABLE_TAIL_BLOCKS} blocks
|
||||
* and re-parses only the source tail behind them. A final unclosed top-level
|
||||
* fence cannot freeze as a block, so its completed content lines use a second
|
||||
* frontier: only the last completed line and current partial line return
|
||||
* through the caller's grammar. Each source region is therefore parsed a
|
||||
* bounded number of times over the stream instead of once per chunk.
|
||||
*
|
||||
* The freeze boundary comes from the parser's own `position` offsets, never
|
||||
* from custom source scanning. The cut sits at the *end offset* of the last
|
||||
* frozen block (not the next block's start): a following block's start offset
|
||||
* excludes up to three spaces of insignificant leading indentation, which is
|
||||
* harmless to drop, but cutting at the previous end also keeps the
|
||||
* inter-block blank lines in the tail so the sliced source stays verbatim.
|
||||
* The block freeze boundary comes from the parser's own `position` offsets.
|
||||
* The cut sits at the *end offset* of the last frozen block (not the next
|
||||
* block's start): a following block's start offset excludes up to three spaces
|
||||
* of insignificant leading indentation, which is harmless to drop, but
|
||||
* cutting at the previous end also keeps the inter-block blank lines in the
|
||||
* tail so the sliced source stays verbatim. Fence scanning only recognizes a
|
||||
* parser-confirmed code node and closing delimiter; ambiguous input returns to
|
||||
* the normal tail parse.
|
||||
*
|
||||
* Known deviation, shared with any prefix-freeze scheme: micromark resolves
|
||||
* reference-style links and footnotes document-wide at parse time, so a
|
||||
|
|
@ -24,7 +28,7 @@
|
|||
* renders literally until the settled full parse self-heals it.
|
||||
*/
|
||||
|
||||
import type { Root, RootContent } from 'mdast'
|
||||
import type { Code, Root, RootContent } from 'mdast'
|
||||
|
||||
/**
|
||||
* Trailing blocks kept unstable. Appended text reshapes at most the last
|
||||
|
|
@ -68,6 +72,103 @@ function blockKey(node: RootContent, base: number, index: number): number {
|
|||
return offset === undefined ? -(index + 1) : base + offset
|
||||
}
|
||||
|
||||
interface OpenFenceState {
|
||||
readonly marker: '`' | '~'
|
||||
readonly markerLength: number
|
||||
readonly syntheticPrefix: string
|
||||
readonly codeIndex: number
|
||||
readonly frozen: readonly PositionedBlock[]
|
||||
readonly tail: readonly PositionedBlock[]
|
||||
readonly pendingStart: number
|
||||
readonly valuePrefix: string
|
||||
readonly end: { readonly line: number; readonly column: number; readonly offset: number }
|
||||
readonly endedWithCarriageReturn: boolean
|
||||
}
|
||||
|
||||
/** Return the first line terminator at or after `start`, including a CRLF pair. */
|
||||
function lineTerminatorEnd(text: string, start: number): number | undefined {
|
||||
for (let index = start; index < text.length; index += 1) {
|
||||
const char = text[index]
|
||||
if (char === '\n') return index + 1
|
||||
if (char === '\r') return text[index + 1] === '\n' ? index + 2 : index + 1
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Source prefix before the last completed line. Keeping that line beside the
|
||||
* current partial line lets the grammar retain its trailing-newline semantics.
|
||||
*/
|
||||
function committableLinePrefixLength(text: string): number {
|
||||
let previousEnd = 0
|
||||
let end = 0
|
||||
for (let index = 0; index < text.length; index += 1) {
|
||||
const char = text[index]
|
||||
if (char === '\n') {
|
||||
previousEnd = end
|
||||
end = index + 1
|
||||
continue
|
||||
}
|
||||
if (char !== '\r' || index + 1 >= text.length) continue
|
||||
if (text[index + 1] === '\n') index += 1
|
||||
previousEnd = end
|
||||
end = index + 1
|
||||
}
|
||||
return previousEnd
|
||||
}
|
||||
|
||||
/** Exact source terminator ending a non-empty committable prefix. */
|
||||
function trailingLineTerminator(text: string): '\n' | '\r' | '\r\n' {
|
||||
return text.endsWith('\r\n') ? '\r\n' : text.endsWith('\r') ? '\r' : '\n'
|
||||
}
|
||||
|
||||
/** Whether `text` contains a CommonMark closing fence on one of its logical lines. */
|
||||
function containsClosingFence(text: string, marker: '`' | '~', markerLength: number): boolean {
|
||||
let start = 0
|
||||
while (start <= text.length) {
|
||||
let end = start
|
||||
while (end < text.length && text[end] !== '\n' && text[end] !== '\r') end += 1
|
||||
const line = text.slice(start, end)
|
||||
let indent = 0
|
||||
while (indent < 3 && line[indent] === ' ') indent += 1
|
||||
let run = indent
|
||||
while (line[run] === marker) run += 1
|
||||
if (run - indent >= markerLength && /^[ \t]*$/.test(line.slice(run))) return true
|
||||
if (end === text.length) return false
|
||||
start = text[end] === '\r' && text[end + 1] === '\n' ? end + 2 : end + 1
|
||||
}
|
||||
/* v8 ignore next -- each loop iteration returns at EOF or advances past a line terminator. */
|
||||
return false
|
||||
}
|
||||
|
||||
/** Advance an mdast point across one append while treating a split CRLF as one line ending. */
|
||||
function advancePoint(
|
||||
point: OpenFenceState['end'],
|
||||
appended: string,
|
||||
precededByCarriageReturn: boolean,
|
||||
): OpenFenceState['end'] {
|
||||
let line = point.line
|
||||
let column = point.column
|
||||
let afterCarriageReturn = precededByCarriageReturn
|
||||
for (const char of appended) {
|
||||
if (char === '\n') {
|
||||
if (!afterCarriageReturn) line += 1
|
||||
column = 1
|
||||
afterCarriageReturn = false
|
||||
continue
|
||||
}
|
||||
if (char === '\r') {
|
||||
line += 1
|
||||
column = 1
|
||||
afterCarriageReturn = true
|
||||
continue
|
||||
}
|
||||
column += 1
|
||||
afterCarriageReturn = false
|
||||
}
|
||||
return { line, column, offset: point.offset + appended.length }
|
||||
}
|
||||
|
||||
/**
|
||||
* Append-only incremental parser over a caller-supplied grammar. One instance
|
||||
* accumulates one streaming document; non-append input resets it.
|
||||
|
|
@ -78,10 +179,127 @@ export class IncrementalMarkdownParser {
|
|||
private frozen: PositionedBlock[] = []
|
||||
private generation = 0
|
||||
private cached: IncrementalBlocks | null = null
|
||||
private openFence: OpenFenceState | null = null
|
||||
|
||||
/** @param parse - Grammar shared with whatever renders the blocks, so boundaries agree. */
|
||||
constructor(private readonly parse: (text: string) => Root) {}
|
||||
|
||||
/** Parse one unclosed-fence content slice through the caller's grammar. */
|
||||
private fenceValue(state: Pick<OpenFenceState, 'syntheticPrefix'>, text: string): string | undefined {
|
||||
const root = this.parse(`${state.syntheticPrefix}${text}`)
|
||||
if (root.children.length !== 1) return undefined
|
||||
const node = root.children[0] as RootContent
|
||||
return node.type === 'code' ? node.value : undefined
|
||||
}
|
||||
|
||||
/** Recognize the parsed tail's final unclosed fence and prepare its incremental content frontier. */
|
||||
private openFenceState(
|
||||
text: string,
|
||||
base: number,
|
||||
tail: readonly PositionedBlock[],
|
||||
frozen: readonly PositionedBlock[],
|
||||
): OpenFenceState | null {
|
||||
const codeIndex = tail.length - 1
|
||||
const block = tail[codeIndex]
|
||||
if (block?.node.type !== 'code') return null
|
||||
const node = block.node
|
||||
const startOffset = node.position?.start.offset
|
||||
const end = node.position?.end
|
||||
if (startOffset === undefined || end?.offset === undefined) return null
|
||||
/* v8 ignore next -- the caller's parse slice ends at text.length, so its final node ends there. */
|
||||
if (base + end.offset !== text.length) return null
|
||||
const source = text.slice(base)
|
||||
const previousLf = source.lastIndexOf('\n', startOffset - 1)
|
||||
const previousCr = source.lastIndexOf('\r', startOffset - 1)
|
||||
const lineStart = Math.max(previousLf, previousCr) + 1
|
||||
const terminatorEnd = lineTerminatorEnd(source, startOffset)
|
||||
/* v8 ignore next -- a parser-confirmed fenced code node requires its opening line terminator. */
|
||||
if (terminatorEnd === undefined) return null
|
||||
if (terminatorEnd === source.length && source.endsWith('\r')) return null
|
||||
const openingLine = source.slice(lineStart, terminatorEnd).replace(/[\r\n]+$/, '')
|
||||
const opening = /^( {0,3})(`{3,}|~{3,})/.exec(openingLine)
|
||||
if (opening === null) return null
|
||||
const indent = opening[1] as string
|
||||
const run = opening[2] as string
|
||||
/* v8 ignore next -- mdast positions a fenced code node at the matched delimiter after indentation. */
|
||||
if (lineStart + indent.length !== startOffset) return null
|
||||
const marker = run[0] as '`' | '~'
|
||||
const contentStart = base + terminatorEnd
|
||||
const content = text.slice(contentStart)
|
||||
if (containsClosingFence(content, marker, run.length)) return null
|
||||
const syntheticPrefix = `${indent}${run}\n`
|
||||
const stableLength = committableLinePrefixLength(content)
|
||||
const stableValue = stableLength === 0
|
||||
? ''
|
||||
: this.fenceValue({ syntheticPrefix }, content.slice(0, stableLength))
|
||||
if (stableValue === undefined) return null
|
||||
const pendingStart = contentStart + stableLength
|
||||
const stableSource = content.slice(0, stableLength)
|
||||
const valuePrefix = stableLength === 0 ? '' : `${stableValue}${trailingLineTerminator(stableSource)}`
|
||||
const pendingValue = this.fenceValue({ syntheticPrefix }, text.slice(pendingStart))
|
||||
if (pendingValue === undefined || `${valuePrefix}${pendingValue}` !== node.value) return null
|
||||
return {
|
||||
marker,
|
||||
markerLength: run.length,
|
||||
syntheticPrefix,
|
||||
codeIndex,
|
||||
frozen,
|
||||
tail,
|
||||
pendingStart,
|
||||
valuePrefix,
|
||||
end: { line: end.line, column: end.column, offset: end.offset },
|
||||
endedWithCarriageReturn: text.endsWith('\r'),
|
||||
}
|
||||
}
|
||||
|
||||
/** Extend a recognized unclosed fence without parsing its completed content prefix again. */
|
||||
private updateOpenFence(
|
||||
state: OpenFenceState,
|
||||
text: string,
|
||||
previousText: string,
|
||||
): IncrementalBlocks | undefined {
|
||||
const pending = text.slice(state.pendingStart)
|
||||
if (containsClosingFence(pending, state.marker, state.markerLength)) return undefined
|
||||
const pendingValue = this.fenceValue(state, pending)
|
||||
if (pendingValue === undefined) return undefined
|
||||
const stableLength = committableLinePrefixLength(pending)
|
||||
const stableValue = stableLength === 0
|
||||
? ''
|
||||
: this.fenceValue(state, pending.slice(0, stableLength))
|
||||
if (stableValue === undefined) return undefined
|
||||
// OpenFenceState is private and is installed only from this exact retained
|
||||
// code entry; updates replace that entry with another positioned Code.
|
||||
const block = state.tail[state.codeIndex] as PositionedBlock
|
||||
const previousNode = block.node as Code & { position: NonNullable<Code['position']> }
|
||||
const end = advancePoint(
|
||||
state.end,
|
||||
text.slice(previousText.length),
|
||||
state.endedWithCarriageReturn,
|
||||
)
|
||||
const node: Code = {
|
||||
...previousNode,
|
||||
value: `${state.valuePrefix}${pendingValue}`,
|
||||
position: { start: previousNode.position.start, end },
|
||||
}
|
||||
const tail = state.tail.map((entry, index) => index === state.codeIndex ? { ...entry, node } : entry)
|
||||
const cached = {
|
||||
frozen: state.frozen,
|
||||
tail,
|
||||
generation: this.generation,
|
||||
}
|
||||
this.openFence = {
|
||||
...state,
|
||||
tail,
|
||||
pendingStart: state.pendingStart + stableLength,
|
||||
valuePrefix: stableLength === 0
|
||||
? state.valuePrefix
|
||||
: `${state.valuePrefix}${stableValue}${trailingLineTerminator(pending.slice(0, stableLength))}`,
|
||||
end,
|
||||
endedWithCarriageReturn: text.endsWith('\r'),
|
||||
}
|
||||
return cached
|
||||
}
|
||||
|
||||
/**
|
||||
* Fold the current accumulated text and return the frozen/tail split.
|
||||
* Idempotent for identical input (the previous result is returned as-is),
|
||||
|
|
@ -101,8 +319,19 @@ export class IncrementalMarkdownParser {
|
|||
this.prevText = ''
|
||||
this.tailStart = 0
|
||||
this.frozen = []
|
||||
this.openFence = null
|
||||
this.generation += 1
|
||||
}
|
||||
const previousText = this.prevText
|
||||
if (previousText !== '' && this.openFence !== null) {
|
||||
const incremental = this.updateOpenFence(this.openFence, text, previousText)
|
||||
if (incremental !== undefined) {
|
||||
this.prevText = text
|
||||
this.cached = incremental
|
||||
return incremental
|
||||
}
|
||||
this.openFence = null
|
||||
}
|
||||
this.prevText = text
|
||||
const base = this.tailStart
|
||||
const blocks = this.parse(text.slice(base)).children
|
||||
|
|
@ -125,6 +354,7 @@ export class IncrementalMarkdownParser {
|
|||
key: blockKey(node, base, index),
|
||||
}))
|
||||
this.cached = { frozen: [...this.frozen], tail, generation: this.generation }
|
||||
this.openFence = this.openFenceState(text, base, tail, this.cached.frozen)
|
||||
return this.cached
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -117,6 +117,16 @@ describe('incremental streaming rendering', () => {
|
|||
live.unmount()
|
||||
settled.unmount()
|
||||
})
|
||||
|
||||
it('keeps a highlighted fence mounted across the final full-document parse', () => {
|
||||
const doc = 'before.\n\n```ts\nconst answer = 42\n```\n\nafter.'
|
||||
const live = render(<MarkdownText text={doc} streaming />)
|
||||
const line = live.container.querySelector('pre.shiki .line')
|
||||
expect(line).not.toBeNull()
|
||||
live.rerender(<MarkdownText text={doc} />)
|
||||
expect(live.container.querySelector('pre.shiki .line')).toBe(line)
|
||||
live.unmount()
|
||||
})
|
||||
})
|
||||
|
||||
describe('incremental parsing is actually in effect', () => {
|
||||
|
|
@ -145,6 +155,25 @@ describe('incremental parsing is actually in effect', () => {
|
|||
expect(totalParsed).toBeLessThan(text.length * 5)
|
||||
})
|
||||
|
||||
it('parses an open fence through bounded grammar slices as completed lines accumulate', () => {
|
||||
const calls: string[] = []
|
||||
const recording = (text: string): Root => {
|
||||
calls.push(text)
|
||||
return parseGfm(text)
|
||||
}
|
||||
const parser = new IncrementalMarkdownParser(recording)
|
||||
let text = '```ts\n'
|
||||
let result = parser.update(text)
|
||||
for (let index = 0; index < 800; index += 1) {
|
||||
text += `const value${String(index)} = ${String(index)}\n`
|
||||
result = parser.update(text)
|
||||
}
|
||||
const parsed = calls.reduce((sum, call) => sum + call.length, 0)
|
||||
expect(Math.max(...calls.slice(10).map(call => call.length))).toBeLessThan(80)
|
||||
expect(parsed).toBeLessThan(text.length * 4)
|
||||
expect(result.tail.at(-1)?.node).toEqual(parseGfm(text).children[0])
|
||||
})
|
||||
|
||||
it('shows the documented streaming fingerprint: a definition frozen earlier no longer resolves a new reference, and settling heals it', () => {
|
||||
const doc = [
|
||||
'[ref]: https://example.com/target',
|
||||
|
|
@ -197,6 +226,98 @@ describe('freeze dynamics around frontier-sensitive constructs', () => {
|
|||
expect(frozenCode?.type === 'code' && frozenCode.value).toContain('looks like a list')
|
||||
})
|
||||
|
||||
it('keeps indented CRLF fence nodes equal to a fresh parse, then falls back when the fence closes', () => {
|
||||
const parser = new IncrementalMarkdownParser(parseGfm)
|
||||
const opening = 'p1.\n\np2.\n\np3.\n\n ```ts\r\n'
|
||||
const suffix = ' const a = 1\r\n const b = 2\r\n ```\r\nafter'
|
||||
let text = ''
|
||||
for (const char of `${opening}${suffix}`) {
|
||||
text += char
|
||||
const result = parser.update(text)
|
||||
const actual = [...result.frozen, ...result.tail].at(-1)
|
||||
const expected = parseGfm(text).children.at(-1)
|
||||
expect(actual?.key).toBe(expected?.position?.start.offset)
|
||||
expect(actual?.node.type).toBe(expected?.type)
|
||||
if (actual?.node.type === 'code' && expected?.type === 'code') {
|
||||
expect({ lang: actual.node.lang, meta: actual.node.meta, value: actual.node.value })
|
||||
.toEqual({ lang: expected.lang, meta: expected.meta, value: expected.value })
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('preserves lone-CR fence lines and ignores indented code as a fence frontier', () => {
|
||||
const parser = new IncrementalMarkdownParser(parseGfm)
|
||||
let text = '```ts\rfirst\r'
|
||||
parser.update(text)
|
||||
text += 'second\rthird'
|
||||
const result = parser.update(text)
|
||||
expect(result.tail.at(-1)?.node).toEqual(parseGfm(text).children.at(-1))
|
||||
|
||||
const indented = ' alpha\n beta\n'
|
||||
const indentedResult = new IncrementalMarkdownParser(parseGfm).update(indented)
|
||||
expect(indentedResult.tail.at(-1)?.node).toEqual(parseGfm(indented).children.at(-1))
|
||||
})
|
||||
|
||||
it('falls back to the full grammar tail when a custom grammar rejects fence slices', () => {
|
||||
type Corruption = 'many' | 'paragraph' | 'mismatch'
|
||||
const custom = (corruption: Corruption): ((text: string) => Root) => (text) => {
|
||||
if (!text.startsWith('```\n')) return parseGfm(text)
|
||||
if (corruption === 'many') return parseGfm('one\n\ntwo')
|
||||
if (corruption === 'paragraph') return parseGfm('one')
|
||||
const root = parseGfm(text)
|
||||
const node = root.children[0]
|
||||
if (node?.type === 'code') node.value += 'mismatch'
|
||||
return root
|
||||
}
|
||||
const cases = [
|
||||
{ corruption: 'many' as const, text: '```ts\nfirst' },
|
||||
{ corruption: 'paragraph' as const, text: '```ts\nfirst' },
|
||||
{ corruption: 'many' as const, text: '```ts\nfirst\nsecond\nthird' },
|
||||
{ corruption: 'mismatch' as const, text: '```ts\nfirst' },
|
||||
]
|
||||
for (const { corruption, text } of cases) {
|
||||
const result = new IncrementalMarkdownParser(custom(corruption)).update(text)
|
||||
expect(result.tail.at(-1)?.node).toEqual(parseGfm(text).children.at(-1))
|
||||
}
|
||||
|
||||
const positionless = new IncrementalMarkdownParser((text) => {
|
||||
const root = parseGfm(text)
|
||||
for (const node of root.children) delete node.position
|
||||
return root
|
||||
}).update('```ts\nfirst')
|
||||
expect(positionless.tail.at(-1)?.node.type).toBe('code')
|
||||
})
|
||||
|
||||
it('abandons an installed fence frontier when later custom-grammar slices fail', () => {
|
||||
let syntheticCall = 0
|
||||
let reject: 'none' | 'first' | 'second' = 'none'
|
||||
const custom = (text: string): Root => {
|
||||
if (!text.startsWith('```\n')) return parseGfm(text)
|
||||
syntheticCall += 1
|
||||
if (reject === 'first' && syntheticCall === 1) return parseGfm('one\n\ntwo')
|
||||
if (reject === 'second' && syntheticCall === 2) return parseGfm('one\n\ntwo')
|
||||
return parseGfm(text)
|
||||
}
|
||||
|
||||
const pendingParser = new IncrementalMarkdownParser(custom)
|
||||
let text = '```ts\nfirst\nsecond'
|
||||
pendingParser.update(text)
|
||||
syntheticCall = 0
|
||||
reject = 'first'
|
||||
text += ' tail'
|
||||
expect(pendingParser.update(text).tail.at(-1)?.node).toEqual(parseGfm(text).children.at(-1))
|
||||
|
||||
reject = 'none'
|
||||
syntheticCall = 0
|
||||
const stableParser = new IncrementalMarkdownParser(custom)
|
||||
text = '```ts\nfirst\nsecond'
|
||||
stableParser.update(text)
|
||||
syntheticCall = 0
|
||||
reject = 'second'
|
||||
text += '\nthird\nfourth'
|
||||
expect(stableParser.update(text).tail.at(-1)?.node).toEqual(parseGfm(text).children.at(-1))
|
||||
})
|
||||
|
||||
it('a list can keep extending across blank lines until it freezes whole', () => {
|
||||
const parser = new IncrementalMarkdownParser(parseGfm)
|
||||
let text = 'intro.\n\nsecond.\n\nthird.\n\n- item a\n- item b\n'
|
||||
|
|
|
|||
|
|
@ -67,6 +67,19 @@ describe('StreamingHighlightSession', () => {
|
|||
expect(second?.[1]).not.toBe(first?.[1])
|
||||
})
|
||||
|
||||
it('reports only newly completed lines to a retained renderer', () => {
|
||||
const session = new StreamingHighlightSession()
|
||||
const first = session.updateFrame('const a = 1\nlet', 'ts')
|
||||
const second = session.updateFrame('const a = 1\nlet b = 2\n// tail', 'ts')
|
||||
expect(first?.appended).toHaveLength(1)
|
||||
expect(first?.tail).toHaveLength(1)
|
||||
expect(second?.appended).toHaveLength(1)
|
||||
expect(second?.appended[0]?.map(span => span.text).join('')).toBe('let b = 2')
|
||||
expect(second?.tail[0]?.map(span => span.text).join('')).toBe('// tail')
|
||||
expect(second?.generation).toBe(first?.generation)
|
||||
expect(session.updateFrame('const a = 1\nlet b = 2\n// tail', 'ts')).toBe(second)
|
||||
})
|
||||
|
||||
it('is idempotent per input: repeated calls return the identical result array', () => {
|
||||
const session = new StreamingHighlightSession()
|
||||
const result = session.update('const a = 1', 'ts')
|
||||
|
|
@ -211,19 +224,46 @@ describe('CodeBlock streaming arm', () => {
|
|||
expect(view.container.querySelector('pre.shiki')?.textContent).toBe('const a = 1\nlet partial = 2\n// tail')
|
||||
})
|
||||
|
||||
it('keeps completed line groups mounted while later groups grow', () => {
|
||||
const code = (count: number) => Array.from({ length: count }, (_, index) => `const v${String(index)} = ${String(index)}`).join('\n')
|
||||
const view = render(<CodeBlock code={code(40)} lang="ts" streaming {...LABELS} />)
|
||||
const firstLine = view.container.querySelector('pre.shiki .line')
|
||||
const thirtySecond = view.container.querySelectorAll('pre.shiki .line')[31]
|
||||
view.rerender(<CodeBlock code={code(80)} lang="ts" streaming {...LABELS} />)
|
||||
const lines = view.container.querySelectorAll('pre.shiki .line')
|
||||
expect(lines).toHaveLength(80)
|
||||
expect(lines[0]).toBe(firstLine)
|
||||
expect(lines[31]).toBe(thirtySecond)
|
||||
})
|
||||
|
||||
it('reuses an unchanged frame when an unrelated lazy grammar finishes loading', async () => {
|
||||
const view = render(<CodeBlock code={'const stable = 1\n'} lang="ts" streaming {...LABELS} />)
|
||||
const line = view.container.querySelector('pre.shiki .line')
|
||||
expect(line).not.toBeNull()
|
||||
const loader = new StreamingHighlightSession()
|
||||
expect(loader.update('puts 1', 'ruby')).toBeUndefined()
|
||||
await vi.waitFor(() => { expect(loader.update('puts 1', 'ruby')).toBeDefined() }, { timeout: 5_000 })
|
||||
expect(view.container.querySelector('pre.shiki .line')).toBe(line)
|
||||
})
|
||||
|
||||
it('streaming with an unknown language stays on the identical plain arm', () => {
|
||||
const view = render(<CodeBlock code={'IDENTIFICATION DIVISION.\n'} lang="cobol" streaming {...LABELS} />)
|
||||
expect(view.container.querySelector('pre.shiki')).toBeNull()
|
||||
expect(view.getByText('IDENTIFICATION DIVISION.')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('the settle swap (streaming to settled) preserves the code content', () => {
|
||||
it('the settle transition preserves the highlighted DOM when the code is unchanged', () => {
|
||||
const code = 'const answer = 42\n'
|
||||
const view = render(<CodeBlock code={code} lang="ts" streaming {...LABELS} />)
|
||||
const streamedLine = view.container.querySelector('pre.shiki .line')
|
||||
const streamedText = view.container.querySelector('pre.shiki')?.textContent
|
||||
view.rerender(<CodeBlock code={code} lang="ts" {...LABELS} />)
|
||||
const settledText = view.container.querySelector('pre.shiki')?.textContent
|
||||
expect(streamedText).toBe('const answer = 42')
|
||||
expect(settledText).toBe(streamedText)
|
||||
expect(view.container.querySelector('pre.shiki .line')).toBe(streamedLine)
|
||||
view.rerender(<CodeBlock code={code} lang="ts" streaming {...LABELS} />)
|
||||
expect(view.container.querySelector('pre.shiki')?.textContent).toBe(streamedText)
|
||||
expect(view.container.querySelector('pre.shiki .line')).toBe(streamedLine)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue