refactor(test): reserve snapshots for session recordings
This commit is contained in:
Tianyi Cui
parent
d4e81b6af7
commit
1cfe0f9942
50 changed files with 273 additions and 222 deletions
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-02-bilingual-docs-and-pairing-gate.md
|
||||
2026-07-02-bilingual-docs-and-pairing-gate.md: 8fbda5e8987d7c575e3ef96f1724b04dd7070621
|
||||
2026-07-02-bilingual-docs-and-pairing-gate.zh.md: b022abf61be733368263ff3de4176e43832b09a2
|
||||
2026-07-02-bilingual-docs-and-pairing-gate.md: f08b5b55b67007cb8c735525afffa5983b9b5863
|
||||
2026-07-02-bilingual-docs-and-pairing-gate.zh.md: f857f79f952b67adf50b1a18fb6daf85a5e3a404
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ This repo's documentation corpus is read by people and agents inside and outside
|
|||
|
||||
## Verification
|
||||
|
||||
The verification contract covers each boundary independently. `verify-translation-pairing` pins pair completeness, hashes, switchers, and structure, while its discovery tests pin the named root policy documents and automatic README coverage; [`project-doc-site.spec.ts`](../../../../scripts/project-doc-site.spec.ts) pins locale-specific source selection for published pairs; [`cordis-config-files.spec.ts`](../../../../scripts/cordis-config-files.spec.ts) pins discovery of Loader YAML and exclusion of translation records; and the [translation-prompt runnable snapshot](../../../../scripts/translation-prompt.snapshot.ts) pins the rendered system message, five reviewed example pairs, source request, and consumed response. Together these checks make pair drift, publication drift, configuration misclassification, and model-visible prompt drift review-visible.
|
||||
The verification contract covers each boundary independently. `verify-translation-pairing` pins pair completeness, hashes, switchers, and structure, while its discovery tests pin the named root policy documents and automatic README coverage; [`project-doc-site.spec.ts`](../../../../scripts/project-doc-site.spec.ts) pins locale-specific source selection for published pairs; [`cordis-config-files.spec.ts`](../../../../scripts/cordis-config-files.spec.ts) pins discovery of Loader YAML and exclusion of translation records; and the [translation-prompt runnable golden](../../../../scripts/translation-prompt.golden.spec.ts) pins the rendered system message, five reviewed example pairs, source request, and consumed response. Together these checks make pair drift, publication drift, configuration misclassification, and model-visible prompt drift review-visible.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ Status: implemented
|
|||
|
||||
## 验证
|
||||
|
||||
验证约定分别覆盖每个边界。`verify-translation-pairing` 固定配对完整性、hash、语言切换行和结构，其发现测试则固定具名的根目录政策文档与自动 README 覆盖；[`project-doc-site.spec.ts`](../../../../scripts/project-doc-site.spec.ts) 固定已发布配对按 locale 选择对应源文件；[`cordis-config-files.spec.ts`](../../../../scripts/cordis-config-files.spec.ts) 固定 Loader YAML 的发现以及翻译记录的排除；[翻译提示词可运行快照](../../../../scripts/translation-prompt.snapshot.ts)则固定渲染后的系统消息、五对经评审的示例、源请求和所消费的响应。这些检查共同使配对漂移、发布漂移、配置误分类和模型可见提示词漂移都可在评审中看见。
|
||||
验证约定分别覆盖每个边界。`verify-translation-pairing` 固定配对完整性、hash、语言切换行和结构，其发现测试则固定具名的根目录政策文档与自动 README 覆盖；[`project-doc-site.spec.ts`](../../../../scripts/project-doc-site.spec.ts) 固定已发布配对按 locale 选择对应源文件；[`cordis-config-files.spec.ts`](../../../../scripts/cordis-config-files.spec.ts) 固定 Loader YAML 的发现以及翻译记录的排除；[翻译提示词可运行 golden](../../../../scripts/translation-prompt.golden.spec.ts)则固定渲染后的系统消息、五对经评审的示例、源请求和所消费的响应。这些检查共同使配对漂移、发布漂移、配置误分类和模型可见提示词漂移都可在评审中看见。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-08-24-session-log-snapshot-corpus.md
|
||||
2026-08-24-session-log-snapshot-corpus.md: 061bb72f0382448d29aec0cd6c38795ab1e82f87
|
||||
2026-08-24-session-log-snapshot-corpus.zh.md: 2b321ebb90fa30a51ab131ba5a8a64134d51492a
|
||||
2026-08-24-session-log-snapshot-corpus.md: 8706a0c431435319bd89ba5013853ebac60e4c50
|
||||
2026-08-24-session-log-snapshot-corpus.zh.md: fc64595267dd89b69a3097d552b858e3793eff46
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ Workspace inputs remain scenario-local. A mutating scenario compares a complete
|
|||
- Every top-level scenario owns or references session JSONL; non-session goldens remain owner-local.
|
||||
- Committed session fixtures are redaction fixed points, contain no system-prompt or tool-schema bulk, and retain exactly one pin per header class.
|
||||
- Mutating scenarios verify their final workspace externally.
|
||||
- Owner-local process goldens use `*.golden.e2e.ts` and a separate built-output gate.
|
||||
- Source and built launch modes, browser replay, SDK projections, packaged Python runtime cases, documentation gates, and repository hygiene pass.
|
||||
|
||||
## Consequences
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ Workspace 输入继续归各场景本地所有。变更文件的场景比较完
|
|||
- 每个顶层场景都拥有或引用会话 JSONL；非会话 golden 继续归所有者本地所有。
|
||||
- 提交的会话 fixture 是脱敏固定点，不含 system prompt 或工具 schema 正文，并为每个 header 类保留且仅保留一个 pin。
|
||||
- 变更内容的场景从外部验证最终 workspace。
|
||||
- 所属位置的进程 golden 使用 `*.golden.e2e.ts`，并由单独的构建产物门禁运行。
|
||||
- 源码和构建启动模式、浏览器回放、SDK 投影、打包 Python 运行时场景、文档门禁和仓库卫生检查通过。
|
||||
|
||||
## Consequences
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ There is no universal local baseline beyond the hooks. Every behavior change nee
|
|||
- **Package or script behavior:** run the owning Vitest file or focused test name. Add adjacent package tests when a shared contract changes; leave repository-wide coverage to CI unless the change is genuinely cross-cutting or the user requests it.
|
||||
- **Documentation, Agent Notes, catalogs, or doc-linked comments:** run `pnpm run doc-sync`; run full lint when the documentation workflow requires it.
|
||||
- **Model-, editor-, CLI-, or terminal-visible output:** run the focused keyless snapshot or real runnable-example scenario that owns the output.
|
||||
- **Expected-output placement:** a test whose recorded `session.jsonl` is replay input and expected persisted output belongs under top-level `snapshots/`, with `snapshot.yml` naming its shipped `dsh` profile and composition/header pin. ARIA, geometry, generator, CLI, and unit expectations without that session round trip stay beside their owning test under `tests/goldens/`; do not place them in `snapshots/` or give them a `*.snapshot.ts` owner.
|
||||
- **Expected-output placement:** a test whose recorded `session.jsonl` is replay input and expected persisted output belongs under top-level `snapshots/`, with `snapshot.yml` naming its shipped `dsh` profile and composition/header pin. ARIA, geometry, generator, CLI, and unit expectations without that session round trip stay beside their owning test under `tests/goldens/`; do not place them in `snapshots/` or give them a `*.snapshot.ts` owner. Use the owning `test:golden`, `test:web`, or `test` lane.
|
||||
- **Package manifests, public exports, build configuration, worker/bin entries, or built runtime paths:** run `pnpm run build`, the relevant hygiene checks, and the owning built-artifact smoke.
|
||||
- **Real provider or agent behavior:** run the relevant `pnpm run test:e2e` target when credentials are available; never print secrets.
|
||||
|
||||
|
|
|
|||
|
|
@ -67,9 +67,10 @@ Package groups: [packages/README.md](packages/README.md).
|
|||
```sh
|
||||
pnpm install # pnpm workspaces, node ^22.19 || >=24
|
||||
pnpm run clean # remove build outputs and safe residue from deleted packages
|
||||
pnpm run test # vitest unit tests
|
||||
pnpm run test # unit tests
|
||||
pnpm run test:coverage # CI coverage gate: per-file 100% on packages/*/*/src
|
||||
pnpm run test:e2e # real-API tests; self-skip without DEEPSEEK_API_KEY
|
||||
pnpm run test:golden # owner-local process goldens
|
||||
pnpm run test:snapshot # keyless recorded-session replay through shipped profiles; filter: -t <name>
|
||||
pnpm run test:snapshot:record # re-record expected outputs (needs key)
|
||||
pnpm run typecheck
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/testing.md
|
||||
testing.md: d6d6cdb4b2cdab45b599a3a131f1790fcfcca118
|
||||
testing.zh.md: b9f4830be38e41f095eb18aa13087a904ef02023
|
||||
testing.md: 8792d928798cabae0a4d2f12ff779248cf44d8ee
|
||||
testing.zh.md: ea7d8a32f26d98656b977a8efeda698ff7382986
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
|
|||
- **Unit** (`pnpm run test`): vitest over package and example specs under their `tests/**` directories plus repository script specs under `scripts/**/*.spec.ts`; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent tests for contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
|
||||
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. Per-file 100% on `packages/shell/pwsh-local/src` needs a real `pwsh`: without one its executor suites self-skip and `vitest.config.ts` exempts the file so pwsh-less hosts stay green, while CI runners ship pwsh and enforce the full bar.
|
||||
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
||||
- **Snapshot** (`pnpm run test:snapshot`): a top-level scenario's recorded `session.jsonl` supplies user input and model replay, then serves as the expected persisted result. Process scenarios start through `dsh`: headless owns one-shot behavior, the SDK owns persistent control, ACP owns automation-protocol behavior, and Web scenarios retain browser/ARIA evidence beside the same canonical session. `snapshot.yml` declares the shipped profile, composition/header class, recording policy, exceptional replay or input metadata, and workspace facts. Typed identity tokens preserve relationships across parent and child logs; prompts and tool schemas live only in the small set of sidecars named by header pins. A mutating scenario independently compares the complete `workspace.expected/` tree, which record and refresh never rewrite. Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every session, sidecar, protocol, UI, and workspace diff.
|
||||
- **Owner-local golden** (`pnpm run test:golden`): keyless assembled CLI/process expectations without a recorded-session round trip. Drivers use `*.golden.e2e.ts` beside `tests/goldens/`; CI runs built exports. Package/script goldens use `test`, while browser goldens use `test:web`.
|
||||
- **Snapshot** (`pnpm run test:snapshot`): a top-level scenario's recorded `session.jsonl` supplies user input and model replay, then serves as the expected persisted result. Process scenarios start through `dsh`: headless owns one-shot behavior, the SDK owns persistent control, ACP owns automation-protocol behavior, and Web retains browser/ARIA evidence beside the same session. `snapshot.yml` declares the profile, composition/header class, recording policy, exceptional replay or input metadata, and workspace facts. Typed tokens preserve parent/child identity relationships; only header pins own prompt/schema sidecars. A mutating scenario independently compares the complete `workspace.expected/` tree, which record and refresh never rewrite. Use `test:snapshot:record` when a model transcript changes and `test:snapshot:refresh` when replay input remains valid; review every resulting diff.
|
||||
- **Web browser snapshot** (`pnpm run test:web`; required Linux PR gate): Chromium compares session-driven output under `snapshots/web/` and UI-only output under `apps/web/tests/goldens/`. CI forces read-only `DSH_SNAPSHOT=replay`, never writing expected outputs; record/refresh stay local and every diff is reviewed ([web e2e lane](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md), [CI gate decision](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md)). `test:web` [builds first](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.md) for plugin CSS.
|
||||
|
||||
Session fixtures keep headers and payloads but omit body sequence/time envelopes. Replay synthesizes them; runtime persistence is unchanged. Fixtures use canonical packed rows; [the migrator](../scripts/migrate-packed-session-fixtures.ts) rewrites old layouts.
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@
|
|||
- **单元测试**（`pnpm run test`）：vitest 运行包和示例各自的 `tests/**` 目录下的测试，以及匹配 `scripts/**/*.spec.ts` 的仓库脚本测试；测试文件与其所覆盖的代码区域放在一起。每个注册表都有一个 HMR（热模块替换）安全测试（对向该注册表贡献内容的 fiber 执行 dispose（资源释放），并断言清理完成）。优先覆盖边界情况、错误路径、事件顺序、并发竞态，以及针对约定回归的永久测试（见 `packages/core/agent-loop/tests/contract-regressions.spec.ts`）。
|
||||
- **覆盖率门禁**（`pnpm run test:coverage`）：门禁级运行，对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码（应删除），而非需要补写的测试。行覆盖率是必要条件，但永远不是充分条件：它证明行被执行过，不证明功能按交付预期工作。`packages/shell/pwsh-local/src` 的按文件 100% 覆盖需要真实的 `pwsh`：缺少它时其执行器套件会自动跳过，`vitest.config.ts` 会豁免该文件以使无 pwsh 的主机保持绿色，而 CI runner 自带 pwsh，仍按完整标准执行门禁。
|
||||
- **真实 API e2e**（`pnpm run test:e2e`）：带密钥测试调用真实提供方 API，包括 DeepSeek 模型以及各提供方特有的冒烟测试；这些测试各自由自己的密钥控制（`EXA_API_KEY`、`PERPLEXITY_API_KEY` 等），缺少密钥时套件会自动跳过，使 keyless CI 保持绿色（[真实 API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.zh.md)）。
|
||||
- **快照**（`pnpm run test:snapshot`）：顶层场景的录制 `session.jsonl` 同时提供用户输入和模型回放，并作为持久化结果的预期值。进程级场景都通过 `dsh` 启动：headless 负责一次性行为，SDK 负责持久控制，ACP 负责自动化协议行为，Web 场景在同一规范会话旁保留浏览器与 ARIA 证据。`snapshot.yml` 声明随附 profile、组合与请求头类别、录制策略、例外回放或输入元数据以及工作区事实。带类型的身份 token 在父子日志之间保留引用关系；系统提示词和工具 schema 只存放在少量请求头 pin 指定的 sidecar 中。变更工作区的场景会独立比较完整的 `workspace.expected/` 目录，record 与 refresh 绝不改写该目录。当模型 transcript（文本记录）变化时使用 `pnpm run test:snapshot:record`，回放输入仍有效时使用 `pnpm run test:snapshot:refresh`；请审查每一处会话、sidecar、协议、UI 与工作区差异。
|
||||
- **所属位置的 golden**（`pnpm run test:golden`）：无录制会话往返的无密钥组装 CLI/进程预期。驱动使用 `*.golden.e2e.ts`，并与 `tests/goldens/` 同属一处；CI 针对构建产物运行。包/脚本 golden 使用 `test`，浏览器 golden 使用 `test:web`。
|
||||
- **快照**（`pnpm run test:snapshot`）：顶层场景的录制 `session.jsonl` 同时提供用户输入和模型回放，并作为持久化结果的预期值。进程级场景都通过 `dsh` 启动：headless 负责一次性行为，SDK 负责持久控制，ACP 负责自动化协议行为，Web 在同一会话旁保留浏览器与 ARIA 证据。`snapshot.yml` 声明 profile、组合与请求头类别、录制策略、例外回放或输入元数据以及工作区事实。带类型的 token 保留父子身份关系；只有请求头 pin 拥有提示词/schema sidecar。变更工作区的场景会独立比较完整的 `workspace.expected/` 目录，record 与 refresh 绝不改写该目录。当模型 transcript（文本记录）变化时使用 `test:snapshot:record`，回放输入仍有效时使用 `test:snapshot:refresh`；请审查所有结果差异。
|
||||
- **Web 浏览器快照**（`pnpm run test:web`；必需的 Linux PR（Pull Request）门禁）：Chromium 比较 `snapshots/web/` 下由会话驱动的输出，以及 `apps/web/tests/goldens/` 下仅含 UI 的输出。CI 强制只读的 `DSH_SNAPSHOT=replay`，绝不写入预期输出；record/refresh 留在本地，每处 diff 都须评审（[web e2e 车道](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md)、[CI 门禁决策](../.agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.zh.md)）。`test:web` 会[先构建](../.agents/notes/implemented/bug-fix/2026-07-28-themed-scrollbars-and-reserved-gutter.zh.md)以交付插件 CSS。
|
||||
|
||||
会话 fixture 保留 header 与 payload，但省略正文序号／时间 envelope。回放会合成这些字段；运行时持久化不变。fixture 使用规范打包行；[迁移器](../scripts/migrate-packed-session-fixtures.ts)会改写旧布局。
|
||||
|
|
|
|||
|
|
@ -1,91 +0,0 @@
|
|||
# ACP demo and snapshot-record patch over the shipped `acp` profile. The dsh
|
||||
# launcher owns environment loading, plugin resolution, and process shutdown;
|
||||
# stdout remains reserved for ACP JSON-RPC.
|
||||
|
||||
- id: llm-deepseek
|
||||
name: '@deepseek-ai/dsh-llm-deepseek'
|
||||
config:
|
||||
thinking: enabled
|
||||
reasoningEffort: max
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
- id: deepseek-v4-pro
|
||||
- id: deepseek-v4-flash-vision-exp
|
||||
inputModalities: [text, image]
|
||||
|
||||
- id: sandbox-policy
|
||||
name: '@deepseek-ai/dsh-sandbox-policy'
|
||||
config:
|
||||
mode: !!js "process.env.DSH_PERMISSION_MODE ?? (process.env.DSH_SNAPSHOT === undefined ? 'workspace-write' : 'danger-full-access')"
|
||||
workspaceRoot: !!js process.cwd()
|
||||
|
||||
- id: approval
|
||||
name: '@deepseek-ai/dsh-user-approval'
|
||||
config:
|
||||
policy: !!js "(process.env.DSH_PERMISSION_MODE ?? (process.env.DSH_SNAPSHOT === undefined ? 'workspace-write' : 'danger-full-access')) === 'danger-full-access' ? 'never' : 'ask'"
|
||||
|
||||
- id: session-persistence-jsonl
|
||||
name: '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
config:
|
||||
root: !!js process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? dshHomePath('sessions')
|
||||
compression: !!js "process.env.DSH_SNAPSHOT === undefined ? 'zstd' : 'none'"
|
||||
|
||||
- id: acp
|
||||
name: '@deepseek-ai/dsh-acp'
|
||||
config:
|
||||
provider: deepseek-official
|
||||
model: deepseek-v4-pro
|
||||
|
||||
- id: system-prompt
|
||||
name: '@deepseek-ai/dsh-system-prompt'
|
||||
config:
|
||||
persona: |
|
||||
You are a coding assistant powered by the {{model}} model. Your working directory is {{cwd}}. Your bash tool runs under a file sandbox — a `[sandbox: file access denied …]` result is policy, not a command bug.
|
||||
|
||||
Verify your work by running the code or tests. Keep answers brief and factual.
|
||||
|
||||
- id: agent-instructions
|
||||
name: '@deepseek-ai/dsh-agent-instructions'
|
||||
config:
|
||||
maxBytes: 65536
|
||||
|
||||
- id: tool-subagent
|
||||
name: '@deepseek-ai/dsh-tool-subagent'
|
||||
config:
|
||||
provider: spawn
|
||||
toolName: subagent
|
||||
enableModelSelection: true
|
||||
backgroundMode: continuable
|
||||
maxDepth: 1
|
||||
|
||||
# Fork omits model selection so provider/model stay equal to the parent and the
|
||||
# inherited history remains eligible for KV Cache reuse. It stays one-shot because
|
||||
# a continuable child's `report` tool and prompt section precede that history and
|
||||
# invalidate the same prefix. `run_in_background` is off as an explicit foreground-only
|
||||
# choice even though the shipped ACP profile mounts the generic Job runtime.
|
||||
# See .agents/notes/implemented/feature/2026-08-18-model-selected-subagent-routes.md
|
||||
# and .agents/notes/implemented/architecture/2026-08-10-fork-children-stay-one-shot.md.
|
||||
- id: tool-subagent-fork
|
||||
name: '@deepseek-ai/dsh-tool-subagent'
|
||||
config:
|
||||
provider: fork
|
||||
toolName: subagent_fork
|
||||
backgroundMode: one-shot
|
||||
enableRunInBackground: false
|
||||
maxDepth: 1
|
||||
|
||||
- id: fs-sandbox
|
||||
name: '@deepseek-ai/dsh-fs-sandbox'
|
||||
config:
|
||||
cwd: !!js process.cwd()
|
||||
|
||||
- insert:
|
||||
- id: hooks-claude-code
|
||||
name: '@deepseek-ai/dsh-hooks-claude-code'
|
||||
config:
|
||||
configPath: ./hooks.json
|
||||
|
||||
- id: hooks-codex
|
||||
name: '@deepseek-ai/dsh-hooks-codex'
|
||||
config:
|
||||
configPath: ./codex-hooks.json
|
||||
1
examples/acp-agent/cordis.yml
Symbolic link
1
examples/acp-agent/cordis.yml
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
../../snapshots/acp/escalation-approved/cordis.yml
|
||||
|
|
@ -25,7 +25,7 @@ const refreshing = process.env.DSH_SNAPSHOT === 'refresh'
|
|||
|
||||
const agent: AgentUnderTest = {
|
||||
binScript: fileURLToPath(new URL('../../../apps/cli/src/bin.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../../../snapshots/acp/escalation-approved/cordis.yml', import.meta.url)),
|
||||
profile: 'acp',
|
||||
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
|
||||
}
|
||||
|
|
@ -1,37 +1,17 @@
|
|||
import { fileURLToPath } from 'node:url'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { createServer } from 'node:http'
|
||||
import type { IncomingMessage, ServerResponse } from 'node:http'
|
||||
import { join } from 'node:path'
|
||||
import { expect, it } from 'vitest'
|
||||
import {
|
||||
defineAcpSnapshotSuite,
|
||||
runScenario,
|
||||
type InputScript,
|
||||
type Scenario,
|
||||
type SnapshotSuiteOptions,
|
||||
} from '@deepseek-ai/dsh-session-snapshot'
|
||||
import { resolvePwshPath } from '@deepseek-ai/dsh-pwsh-local'
|
||||
import { OFFLOADED_IMAGE_TEXT } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/**
|
||||
* The acp-agent example's snapshot suite: the scenario table for
|
||||
* `dsh-session-snapshot`'s ACP suite factory, which owns every compare/guard mechanic
|
||||
* (expected-output + re-persisted-log diffs, record/refresh write-back, the pinned-header
|
||||
* uniformity guard, the fixture guards). Fixtures live under `snapshots/<name>/`;
|
||||
* `pnpm run test:snapshot:record` re-records model transcripts against the real
|
||||
* API; `pnpm run test:snapshot:refresh` rewrites current replay expected outputs keyless.
|
||||
* See the package README (packages/test-support/session-snapshot) and the snapshot Agent Note,
|
||||
* .agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*/
|
||||
|
||||
// The dsh CLI, this example's profile patch, and
|
||||
// the repo-root tsconfig (four levels up from examples/acp-agent/tests) — all
|
||||
// ABSOLUTE: the subprocess cwd is a temp dir outside the repo.
|
||||
const AGENT = {
|
||||
binScript: fileURLToPath(new URL('../../../apps/cli/src/bin.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../../../snapshots/acp/escalation-approved/cordis.yml', import.meta.url)),
|
||||
profile: 'acp',
|
||||
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
|
||||
}
|
||||
|
|
@ -39,63 +19,6 @@ const IMAGE_OFFLOAD_CONFIG = fileURLToPath(new URL('./fixtures/image-offload.cor
|
|||
const SNAPSHOTS_DIR = fileURLToPath(new URL('../../../snapshots/acp/', import.meta.url))
|
||||
const READ_IMAGE_WORKSPACE = fileURLToPath(new URL('../../../snapshots/session/read-image/workspace/', import.meta.url))
|
||||
|
||||
function snapshotModeFromEnv(value: string | undefined): SnapshotSuiteOptions['mode'] {
|
||||
switch (value) {
|
||||
case undefined:
|
||||
case '':
|
||||
case 'replay':
|
||||
return 'replay'
|
||||
case 'record':
|
||||
return 'record'
|
||||
case 'refresh':
|
||||
return 'refresh'
|
||||
default:
|
||||
throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
|
||||
}
|
||||
}
|
||||
|
||||
const SCENARIOS: Scenario[] = [
|
||||
{ name: 'handshake', hasModelTurn: false, recorded: false, headerClass: 'sandbox' },
|
||||
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false, headerClass: 'sandbox' },
|
||||
{ name: 'cancel', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'sandbox' },
|
||||
{ name: 'cancel-tool-calls', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'sandbox', posixOnly: true },
|
||||
{
|
||||
name: 'escalation-approved',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
pinsHeader: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
{
|
||||
name: 'escalation-rejected',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
{
|
||||
name: 'fs-escalation-approved',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
]
|
||||
|
||||
// Hosts without a usable PowerShell skip the pwsh-tool-turn run (its fixtures
|
||||
// stay guarded); the probe follows the executor's own resolution so a Windows
|
||||
// host with only an install-location pwsh still runs the scenario.
|
||||
const hasPwsh = spawnSync(resolvePwshPath(), ['-NoLogo', '-NoProfile', '-NonInteractive', '-Command', '$true'], { encoding: 'utf8' }).status === 0
|
||||
|
||||
defineAcpSnapshotSuite({
|
||||
agent: AGENT,
|
||||
snapshotsDir: SNAPSHOTS_DIR,
|
||||
scenarios: SCENARIOS.filter(scenario => existsSync(join(SNAPSHOTS_DIR, scenario.name))),
|
||||
mode: snapshotModeFromEnv(process.env.DSH_SNAPSHOT),
|
||||
hasPwsh,
|
||||
})
|
||||
|
||||
it('pins native DeepSeek Files offload and inline fallback in assembled requests', async () => {
|
||||
const requests: Record<string, unknown>[] = []
|
||||
const fileRequests: Array<{ method: string; path: string; bytes: number }> = []
|
||||
|
|
@ -51,7 +51,6 @@
|
|||
"headless-agent/tests/fixtures/session-telemetry-otel-driver.ts",
|
||||
"headless-agent/tests/fixtures/telemetry-redact-rule.ts",
|
||||
"headless-agent/tests/fixtures/e2b/e2b/bin.ts",
|
||||
"acp-agent/tests/fixtures/subagent-settlement-marker.ts",
|
||||
"acp-agent/tests/fixtures/control-surface/control-surface-llm.ts",
|
||||
"acp-agent/tests/fixtures/subagent/subagent-acp/mock-delegating-llm.ts",
|
||||
"acp-agent/tests/fixtures/subagent/subagent-acp/driver.ts",
|
||||
|
|
@ -62,8 +61,7 @@
|
|||
"python-sdk-agent/tests/fixtures/subagent/subagent-dsh-sdk/driver.ts",
|
||||
"python-sdk-agent/tests/fixtures/subagent/subagent-dsh-sdk/child-mock-llm.ts",
|
||||
"python-sdk-agent/tests/fixtures/subagent/subagent-dsh-sdk/mock-delegating-llm.ts",
|
||||
"*/tests/**/*.e2e.ts",
|
||||
"*/tests/**/*.snapshot.ts"
|
||||
"*/tests/**/*.e2e.ts"
|
||||
],
|
||||
"project": [
|
||||
"**/*.ts"
|
||||
|
|
@ -650,8 +648,7 @@
|
|||
"apps/cli": {
|
||||
"entry": [
|
||||
"tests/**/*.spec.ts",
|
||||
"tests/**/*.e2e.ts",
|
||||
"tests/**/*.snapshot.ts"
|
||||
"tests/**/*.e2e.ts"
|
||||
],
|
||||
"project": [
|
||||
"src/**/*.ts",
|
||||
|
|
|
|||
|
|
@ -36,6 +36,8 @@
|
|||
"test:coverage": "vitest run --coverage",
|
||||
"test:coverage:partitioned": "tsx scripts/run-coverage-partitions.ts",
|
||||
"test:e2e": "vitest run --config vitest.e2e.config.ts",
|
||||
"test:golden": "vitest run --config vitest.golden.config.ts",
|
||||
"test:golden:refresh": "DSH_SNAPSHOT=refresh vitest run --config vitest.golden.config.ts",
|
||||
"test:issue-management": "node .github/issue-management/policy.test.mjs",
|
||||
"test:snapshot": "vitest run --config vitest.snapshot.config.ts",
|
||||
"test:snapshot:record": "DSH_SNAPSHOT=record vitest run --config vitest.snapshot.config.ts --update",
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/test-support/session-snapshot/README.md
|
||||
README.md: ad75128bc2c0af978b5defcbfd0efcf34554091e
|
||||
README.zh.md: f5167a26f186b3f71e6a39a84baa162fe3862bd8
|
||||
README.md: ff5acebe2a11eb9f1c2d6ef2717a7eada050a201
|
||||
README.zh.md: b5ff8b957d532bfd6f513bde32a7fd3ca9ee082c
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ defineAcpSnapshotSuite({
|
|||
})
|
||||
```
|
||||
|
||||
A scenario booting a differently composed profile sets its own `configPath` patch (its basename still ends in `cordis.yml`, so the launcher finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Default generated workspaces are stored in session fixtures as `{{cwd}}` so platform temp roots and random basenames do not affect recordings; `workspaceParent` moves the generated cwd outside the platform temp area when temporary-directory grants are themselves under test, keeps that explicit path in the fixture, and remains parent-owned while the harness removes only the generated child. A scenario's committed `workspace/` is copied into that child first, then `prepareWorkspace` runs against the generated cwd before the agent starts. Reserve this hook for fixtures Git cannot represent portably, keep ordinary seeds in `workspace/`, and pair it with `posixOnly` when the generated paths are invalid on Windows.
|
||||
A scenario booting a differently composed profile sets its own `configPath` patch (its basename still ends in `cordis.yml`, so the launcher finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario; the top-level profile corpora contain the current templates. Default generated workspaces are stored in session fixtures as `{{cwd}}` so platform temp roots and random basenames do not affect recordings; `workspaceParent` moves the generated cwd outside the platform temp area when temporary-directory grants are themselves under test, keeps that explicit path in the fixture, and remains parent-owned while the harness removes only the generated child. A scenario's committed `workspace/` is copied into that child first, then `prepareWorkspace` runs against the generated cwd before the agent starts. Reserve this hook for fixtures Git cannot represent portably, keep ordinary seeds in `workspace/`, and pair it with `posixOnly` when the generated paths are invalid on Windows.
|
||||
|
||||
A pin owns its generated `system-prompt.expected.md` or `tool-schemas.expected.json` by default; `systemPromptSource` and `toolSchemasSource` name another pin when the complete corresponding sequence is identical, so each distinct version is committed once. The pin's `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`; a shared source must declare the same count, and record/refresh rejects claimants that generate different bytes.
|
||||
|
||||
|
|
@ -66,7 +66,7 @@ A child session whose own scope composes a different request declares it per fix
|
|||
|
||||
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario requiring a non-Windows host declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere; examples include POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) and generated paths Windows cannot represent. A scenario whose composition needs a usable `pwsh` declares `pwshOnly`; the caller-supplied `hasPwsh` probe (the shipped acp-agent suite follows the executor's own resolution, so Program Files installs count) skips the run test when no usable `pwsh` resolves while the fixture guards keep covering its committed files everywhere.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay patch next to its live patch. The launcher applies the live base patch and the selected replay sibling under `DSH_SNAPSHOT=replay` ([single-source replay config Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)); [`dsh-llm-replay`](../llm-replay/README.md) serves fixtures named by the `DSH_SNAPSHOT_*` environment values. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay patch, and rewrites stdout, comparable session-log expected outputs, and owned prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
Each composition owner ships a `cordis.snapshot.yml` replay patch next to its live patch. The launcher applies the live base patch and the selected replay sibling under `DSH_SNAPSHOT=replay` ([single-source replay config Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)); [`dsh-llm-replay`](../llm-replay/README.md) serves fixtures named by the `DSH_SNAPSHOT_*` environment values. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay patch, and rewrites stdout, comparable session-log expected outputs, and owned prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` and `harness.ts` import vitest (the harness polls its durable-boundary waits through `vi.waitFor`), so the package entry is importable only inside a vitest run (the launcher and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the JSON-RPC and Web snapshot recorders. Input scripts cover initialization, fresh-session creation, shorthand text prompts, exact structured ACP prompt blocks, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ defineAcpSnapshotSuite({
|
|||
})
|
||||
```
|
||||
|
||||
启动不同 profile 组合的场景会设置自己的 `configPath` patch（其 basename 仍以 `cordis.yml` 结尾，使启动器可找到同级 `*cordis.snapshot.yml`）；当该组合改变请求 header 时，还会设置自己的 `headerClass` 和 pin 场景，acp-agent 示例的 Code Mode 与文件系统场景是模板。默认生成的 workspace 在会话 fixture 中存储为 `{{cwd}}`，使平台临时根目录和随机 basename 不影响录制结果；当临时目录授权自身待测时，`workspaceParent` 将生成 cwd 移出平台临时区域，在 fixture 中保留该显式路径，并仍归父级所有，而 harness 只移除生成的子级。场景签入的 `workspace/` 会先复制到该子级，随后 `prepareWorkspace` 在 agent 启动前针对生成 cwd 运行。此 hook 仅用于 Git 无法跨平台表示的 fixture；普通种子应留在 `workspace/` 中，而生成路径在 Windows 上无效时还必须搭配 `posixOnly`。
|
||||
启动不同 profile 组合的场景会设置自己的 `configPath` patch（其 basename 仍以 `cordis.yml` 结尾，使启动器可找到同级 `*cordis.snapshot.yml`）；当该组合改变请求 header 时，还会设置自己的 `headerClass` 和 pin 场景，顶层各 profile 语料包含当前模板。默认生成的 workspace 在会话 fixture 中存储为 `{{cwd}}`，使平台临时根目录和随机 basename 不影响录制结果；当临时目录授权自身待测时，`workspaceParent` 将生成 cwd 移出平台临时区域，在 fixture 中保留该显式路径，并仍归父级所有，而 harness 只移除生成的子级。场景签入的 `workspace/` 会先复制到该子级，随后 `prepareWorkspace` 在 agent 启动前针对生成 cwd 运行。此 hook 仅用于 Git 无法跨平台表示的 fixture；普通种子应留在 `workspace/` 中，而生成路径在 Windows 上无效时还必须搭配 `posixOnly`。
|
||||
|
||||
每个 pin 默认拥有其生成的 `system-prompt.expected.md` 或 `tool-schemas.expected.json`；当完整的对应序列相同时，`systemPromptSource` 和 `toolSchemasSource` 指定另一个 pin 作为来源，因此每个不同版本只提交一次。该 pin 的 `session.jsonl` 存储 `"system":"{{system}}","tools":"{{tools}}"`，同时保留配置、原因和任何模型可见前缀。具有合法运行中 header 变更的 pin 声明 `expectedHeaderChanges`；共享来源必须声明相同的 header 变更数量，录制/刷新会拒绝生成不同字节的共享引用方。
|
||||
|
||||
|
|
@ -66,7 +66,7 @@ defineAcpSnapshotSuite({
|
|||
|
||||
每个场景都比较 `stdout.expected.jsonl`，其中以 cwd 为根的分隔符规范化为 `/`。在 Windows 上，`pinsNativeWindowsStdout` 还会在共享预期输出之后比较完整 `stdout.expected.windows.jsonl`，并且仅在启用时要求存在该伴随文件。需要非 Windows 主机的场景声明 `posixOnly`，在 Windows 上跳过运行测试，但 fixture 保护仍在所有平台覆盖其已提交文件；示例包括 POSIX 进程语义（例如取消正在运行的 bash 调用会终止一个已脱离的进程组）和 Windows 无法表示的生成路径。组合需要可用 `pwsh` 的场景声明 `pwshOnly`；调用方提供的 `hasPwsh` 探测（随附的 acp-agent 套件遵循执行器自身的解析，因此 Program Files 安装也计入）在解析不到可用 `pwsh` 时跳过运行测试，而 fixture 保护仍处处覆盖其已提交文件。
|
||||
|
||||
示例还在 live patch 旁提供 `cordis.snapshot.yml` replay patch。`DSH_SNAPSHOT=replay` 下，启动器应用 live 基础 patch 和所选场景的 replay 同级文件（见[单源回放配置 Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)）；[`dsh-llm-replay`](../llm-replay/README.zh.md) 提供由 `DSH_SNAPSHOT_*` 环境值指向的 fixture。`pnpm run test:snapshot:record` 调用在线 LLM（大语言模型），并重写已记录场景的模型 fixture；`pnpm run test:snapshot:refresh` 保持无密钥，运行回放 overlay，并从已提交模型脚本重写 stdout、可比较会话日志预期输出，以及各 pin 自有的提示词与工具 schema 伴随文件。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.zh.md) 中。
|
||||
每个组合所有者都在 live patch 旁提供 `cordis.snapshot.yml` replay patch。`DSH_SNAPSHOT=replay` 下，启动器应用 live 基础 patch 和所选场景的 replay 同级文件（见[单源回放配置 Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)）；[`dsh-llm-replay`](../llm-replay/README.zh.md) 提供由 `DSH_SNAPSHOT_*` 环境值指向的 fixture。`pnpm run test:snapshot:record` 调用在线 LLM（大语言模型），并重写已记录场景的模型 fixture；`pnpm run test:snapshot:refresh` 保持无密钥，运行回放 overlay，并从已提交模型脚本重写 stdout、可比较会话日志预期输出，以及各 pin 自有的提示词与工具 schema 伴随文件。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.zh.md) 中。
|
||||
|
||||
约束：`suite.ts` 与 `harness.ts` 导入 vitest（harness 通过 `vi.waitFor` 轮询其持久边界等待），因此包入口只能在 vitest 运行中导入（启动器和规范化器没有此依赖，但从同一入口发布）。启动器和套件工厂按设计专用于 ACP，启动器使用 SDK 的 `ClientSideConnection`；规范化器是与传输无关的会话日志/文本辅助工具，还由 JSON-RPC 和 Web 快照录制器消费。输入脚本覆盖初始化、新建会话、文本提示简写、精确结构化 ACP 提示词块、取消、预期 RPC 失败和持久轮次边界等待。权限往返是选项类别选择（`allow_once`、`reject_once` 等）的 FIFO 队列，映射到 agent 发出的 `optionId`；缺少或耗尽的队列回答 `cancelled`，未提供类别会拒绝运行。
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
/**
|
||||
* Shared subprocess harness for ACP snapshot suites. A library module driven by
|
||||
* the suite factory in ./suite.ts (and directly by harness-level specs); each
|
||||
* example's `*.snapshot.ts` names its own agent-under-test paths.
|
||||
* profile adapter names its own agent-under-test paths.
|
||||
*
|
||||
* It boots the REAL agent bin subprocess via the cordis Loader (so the
|
||||
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
* factory ({@link defineAcpSnapshotSuite}) that registers a scenario table as a
|
||||
* full describe/it tree. Transport-neutral normalizers and fixture invariants
|
||||
* remain reusable by other profile adapters. Ordinary ACP e2e tests can use the launcher directly;
|
||||
* an example's `*.snapshot.ts` supplies only its {@link AgentUnderTest} paths,
|
||||
* the ACP corpus adapter supplies only its {@link AgentUnderTest} paths,
|
||||
* snapshots directory, and {@link Scenario} table.
|
||||
*
|
||||
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
|
||||
|
|
|
|||
|
|
@ -387,7 +387,7 @@ describe('Node 24 lane ownership', () => {
|
|||
const subject = withPnpmEntrypoint(() => gatesForMode('ci-consumers'))
|
||||
|
||||
expect(defaultConcurrency('ci-consumers', subject.length, 4)).toEqual({
|
||||
workers: 10,
|
||||
workers: 11,
|
||||
source: 'ci-consumers gate count',
|
||||
})
|
||||
expect(subject.map(item => item.id)).toEqual([
|
||||
|
|
@ -397,6 +397,7 @@ describe('Node 24 lane ownership', () => {
|
|||
'built-package-invariants',
|
||||
'lint-and-duplication',
|
||||
'snapshot',
|
||||
'golden',
|
||||
'web-snapshot',
|
||||
'doc-typecheck',
|
||||
'node-next-types',
|
||||
|
|
@ -413,6 +414,7 @@ describe('Node 24 lane ownership', () => {
|
|||
expect(subject.find(item => item.id === 'lint-and-duplication')?.needs).toEqual(['built-package-invariants'])
|
||||
for (const id of [
|
||||
'snapshot',
|
||||
'golden',
|
||||
'web-snapshot',
|
||||
'doc-typecheck',
|
||||
'node-next-types',
|
||||
|
|
@ -421,6 +423,7 @@ describe('Node 24 lane ownership', () => {
|
|||
expect(subject.find(item => item.id === id)?.needs).toEqual(['built-package-invariants'])
|
||||
}
|
||||
expect(subject.find(item => item.id === 'snapshot')?.env).toEqual({ DSH_EXAMPLE_MODE: 'lib' })
|
||||
expect(subject.find(item => item.id === 'golden')?.env).toEqual({ DSH_EXAMPLE_MODE: 'lib' })
|
||||
expect(subject.find(item => item.id === 'doc-typecheck')?.env).toEqual({
|
||||
DSH_DOC_TYPECHECK_USE_BUILD_OUTPUT: '1',
|
||||
})
|
||||
|
|
|
|||
|
|
@ -241,6 +241,7 @@ export function gatesForMode(selected: Mode): Gate[] {
|
|||
pnpmScript('issue-management', 'test:issue-management', { label: 'Issue management policy' }),
|
||||
pnpmScript('duplication', 'duplication'),
|
||||
snapshotGate(),
|
||||
goldenGate(),
|
||||
pnpmScript('build', 'build'),
|
||||
pnpmScript('build:web', 'build:web'),
|
||||
...hygieneLeafGates({ artifactNeeds: ['build'] }),
|
||||
|
|
@ -427,6 +428,7 @@ function ciConsumerGates(): Gate[] {
|
|||
needs: validatedBuild,
|
||||
}),
|
||||
snapshotGate(validatedBuild),
|
||||
goldenGate(validatedBuild),
|
||||
webSnapshotGate(validatedBuild),
|
||||
pnpmScript('doc-typecheck', 'doc-typecheck:contracts-ready', {
|
||||
needs: validatedBuild,
|
||||
|
|
@ -585,9 +587,8 @@ function coverageGates(): Gate[] {
|
|||
]
|
||||
}
|
||||
|
||||
// Example and package snapshots boot their bins in `lib` mode (built artifacts under plain Node,
|
||||
// plugins via real exports); script snapshots execute their real source entry path.
|
||||
// Callers wait either on `build` or on a validation gate that transitively owns that build.
|
||||
// Recorded-session adapters boot process scenarios in `lib` mode. Callers wait
|
||||
// either on `build` or on a validation gate that transitively owns that build.
|
||||
function snapshotGate(needs: string[] = ['build']): Gate {
|
||||
return pnpmScript('snapshot', 'test:snapshot', {
|
||||
env: { DSH_EXAMPLE_MODE: 'lib' },
|
||||
|
|
@ -595,6 +596,15 @@ function snapshotGate(needs: string[] = ['build']): Gate {
|
|||
})
|
||||
}
|
||||
|
||||
// Owner-local process goldens consume built package exports without entering
|
||||
// the recorded-session corpus or the credentialed provider lane.
|
||||
function goldenGate(needs: string[] = ['build']): Gate {
|
||||
return pnpmScript('golden', 'test:golden', {
|
||||
env: { DSH_EXAMPLE_MODE: 'lib' },
|
||||
needs,
|
||||
})
|
||||
}
|
||||
|
||||
function builtPackageInvariantsGate(needs?: string[]): Gate {
|
||||
return pnpmScript('built-package-invariants', 'verify-built-package-invariants', {
|
||||
label: 'built package invariants',
|
||||
|
|
|
|||
|
|
@ -1,17 +0,0 @@
|
|||
/** Repository-wide canonical-layout check for committed session snapshots. */
|
||||
|
||||
import { resolve } from 'node:path'
|
||||
import { expect, it } from 'vitest'
|
||||
import { inspectSessionFixtureLayouts } from './session-fixture-layout.ts'
|
||||
|
||||
const root = resolve(import.meta.dirname, '..')
|
||||
|
||||
it('keeps every session-format JSONL fixture projected into canonical packed layout', () => {
|
||||
const nonCanonical = inspectSessionFixtureLayouts(root)
|
||||
.filter(fixture => fixture.source !== fixture.canonical)
|
||||
.map(fixture => fixture.path)
|
||||
expect(
|
||||
nonCanonical,
|
||||
'Run `pnpm run migrate:packed-session-fixtures` and commit the mechanical fixture rewrite.',
|
||||
).toEqual([])
|
||||
})
|
||||
|
|
@ -1,9 +1,15 @@
|
|||
import { resolve } from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { type SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { parseSessionLog } from '@deepseek-ai/dsh-llm-replay'
|
||||
import { canonicalSessionFixture, isPhysicalSessionFixture } from './session-fixture-layout.ts'
|
||||
import {
|
||||
canonicalSessionFixture,
|
||||
inspectSessionFixtureLayouts,
|
||||
isPhysicalSessionFixture,
|
||||
} from './session-fixture-layout.ts'
|
||||
|
||||
const HEADER = ' {"type":"session","version":0,"id":"fixture","createdAt":1,"delegationDepth":0} '
|
||||
const root = resolve(import.meta.dirname, '..')
|
||||
|
||||
function chunkRun(): SessionEvent[] {
|
||||
return Array.from({ length: 4 }, (_, index) => ({
|
||||
|
|
@ -80,3 +86,13 @@ describe('isPhysicalSessionFixture', () => {
|
|||
expect(isPhysicalSessionFixture('apps/web/tests/snapshots/example/session.jsonl')).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps every session-format JSONL fixture projected into canonical packed layout', () => {
|
||||
const nonCanonical = inspectSessionFixtureLayouts(root)
|
||||
.filter(fixture => fixture.source !== fixture.canonical)
|
||||
.map(fixture => fixture.path)
|
||||
expect(
|
||||
nonCanonical,
|
||||
'Run `pnpm run migrate:packed-session-fixtures` and commit the mechanical fixture rewrite.',
|
||||
).toEqual([])
|
||||
})
|
||||
|
|
|
|||
|
|
@ -18,6 +18,13 @@ import {
|
|||
const repoRoot = resolve(import.meta.dirname, '..')
|
||||
const corpusRoot = join(repoRoot, 'snapshots')
|
||||
const profiles = ['acp', 'sdk', 'session', 'web'] as const
|
||||
const snapshotAdapters = [
|
||||
'apps/web/tests/message-feedback-protocol.snapshot.ts',
|
||||
'apps/web/tests/minimal-preset.snapshot.ts',
|
||||
'snapshots/acp/acp.snapshot.ts',
|
||||
'snapshots/sdk/sdk.snapshot.ts',
|
||||
'snapshots/session/headless.snapshot.ts',
|
||||
] as const
|
||||
|
||||
interface Scenario {
|
||||
readonly key: string
|
||||
|
|
@ -67,6 +74,28 @@ function referencedScenario(owner: Scenario, source: string): string {
|
|||
return source.includes('/') ? source : `${owner.profile}/${source}`
|
||||
}
|
||||
|
||||
async function snapshotNamedTests(): Promise<string[]> {
|
||||
const files: string[] = []
|
||||
const visit = async (directory: string, relativeDir: string): Promise<void> => {
|
||||
for (const entry of await readdir(directory, { withFileTypes: true })) {
|
||||
if (entry.isDirectory()) {
|
||||
if (['dist', 'lib', 'node_modules'].includes(entry.name)) continue
|
||||
await visit(join(directory, entry.name), join(relativeDir, entry.name))
|
||||
} else if (entry.isFile() && /\.snapshot\.tsx?$/u.test(entry.name)) {
|
||||
files.push(join(relativeDir, entry.name).split(/[/\\]/u).join('/'))
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const root of ['apps', 'examples', 'scripts', 'snapshots']) {
|
||||
await visit(join(repoRoot, root), root)
|
||||
}
|
||||
return files.sort()
|
||||
}
|
||||
|
||||
it('reserves the snapshot test suffix for recorded-session adapters', async () => {
|
||||
expect(await snapshotNamedTests()).toEqual([...snapshotAdapters])
|
||||
})
|
||||
|
||||
it('keeps every recorded session owned, pinned, redacted, and header-scrubbed', async () => {
|
||||
const all = await scenarios()
|
||||
const byKey = new Map(all.map(scenario => [scenario.key, scenario]))
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# AGENTS.md — Recorded-session snapshots
|
||||
|
||||
This tree contains only tests whose committed session JSONL is replay input and expected persisted output. Keep non-session ARIA, geometry, generator, CLI, and unit expected output with its owning app, script, or package as a golden.
|
||||
This tree contains only tests whose committed session JSONL is replay input and expected persisted output. Keep non-session ARIA, geometry, generator, CLI, and unit expected output with its owning app, script, or package as a golden; use `test:golden`, `test:web`, or `test` for its owning tier.
|
||||
|
||||
Every process under test starts through the `dsh` CLI with a shipped profile and optional scenario patches. Test clients may drive a public protocol or browser interface; do not add another application entrypoint, hidden CLI mode, or executable scenario driver.
|
||||
|
||||
|
|
|
|||
63
snapshots/acp/acp.snapshot.ts
Normal file
63
snapshots/acp/acp.snapshot.ts
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
/** Recorded ACP protocol behavior through the shipped `dsh --profile acp` interface. */
|
||||
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import {
|
||||
defineAcpSnapshotSuite,
|
||||
type Scenario,
|
||||
type SnapshotSuiteOptions,
|
||||
} from '@deepseek-ai/dsh-session-snapshot'
|
||||
|
||||
const corpusDir = fileURLToPath(new URL('./', import.meta.url))
|
||||
const compositionDir = fileURLToPath(new URL('./escalation-approved/', import.meta.url))
|
||||
|
||||
function snapshotMode(value: string | undefined): SnapshotSuiteOptions['mode'] {
|
||||
switch (value) {
|
||||
case undefined:
|
||||
case '':
|
||||
case 'replay': return 'replay'
|
||||
case 'record': return 'record'
|
||||
case 'refresh': return 'refresh'
|
||||
default: throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
|
||||
}
|
||||
}
|
||||
|
||||
const scenarios: Scenario[] = [
|
||||
{ name: 'handshake', hasModelTurn: false, recorded: false, headerClass: 'sandbox' },
|
||||
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false, headerClass: 'sandbox' },
|
||||
{ name: 'cancel', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'sandbox' },
|
||||
{ name: 'cancel-tool-calls', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'sandbox', posixOnly: true },
|
||||
{
|
||||
name: 'escalation-approved',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
pinsHeader: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
{
|
||||
name: 'escalation-rejected',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
{
|
||||
name: 'fs-escalation-approved',
|
||||
hasModelTurn: true,
|
||||
recorded: true,
|
||||
headerClass: 'sandbox',
|
||||
env: { DSH_PERMISSION_MODE: 'workspace-write' },
|
||||
},
|
||||
]
|
||||
|
||||
defineAcpSnapshotSuite({
|
||||
agent: {
|
||||
binScript: fileURLToPath(new URL('../../apps/cli/src/bin.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('./escalation-approved/cordis.yml', import.meta.url)),
|
||||
profile: 'acp',
|
||||
tsconfigPath: fileURLToPath(new URL('../../tsconfig.json', import.meta.url)),
|
||||
},
|
||||
snapshotsDir: corpusDir,
|
||||
scenarios,
|
||||
mode: snapshotMode(process.env.DSH_SNAPSHOT),
|
||||
})
|
||||
|
|
@ -41,4 +41,4 @@
|
|||
- id: deepseek-v4-pro
|
||||
|
||||
- id: subagent-settlement-marker
|
||||
name: './tests/fixtures/subagent-settlement-marker.ts'
|
||||
name: './subagent-settlement-marker.ts'
|
||||
91
snapshots/acp/escalation-approved/cordis.yml
Normal file
91
snapshots/acp/escalation-approved/cordis.yml
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
# ACP snapshot-record patch over the shipped `acp` profile. The dsh
|
||||
# launcher owns environment loading, plugin resolution, and process shutdown;
|
||||
# stdout remains reserved for ACP JSON-RPC.
|
||||
|
||||
- id: llm-deepseek
|
||||
name: '@deepseek-ai/dsh-llm-deepseek'
|
||||
config:
|
||||
thinking: enabled
|
||||
reasoningEffort: max
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
- id: deepseek-v4-pro
|
||||
- id: deepseek-v4-flash-vision-exp
|
||||
inputModalities: [text, image]
|
||||
|
||||
- id: sandbox-policy
|
||||
name: '@deepseek-ai/dsh-sandbox-policy'
|
||||
config:
|
||||
mode: !!js "process.env.DSH_PERMISSION_MODE ?? (process.env.DSH_SNAPSHOT === undefined ? 'workspace-write' : 'danger-full-access')"
|
||||
workspaceRoot: !!js process.cwd()
|
||||
|
||||
- id: approval
|
||||
name: '@deepseek-ai/dsh-user-approval'
|
||||
config:
|
||||
policy: !!js "(process.env.DSH_PERMISSION_MODE ?? (process.env.DSH_SNAPSHOT === undefined ? 'workspace-write' : 'danger-full-access')) === 'danger-full-access' ? 'never' : 'ask'"
|
||||
|
||||
- id: session-persistence-jsonl
|
||||
name: '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
config:
|
||||
root: !!js process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? dshHomePath('sessions')
|
||||
compression: !!js "process.env.DSH_SNAPSHOT === undefined ? 'zstd' : 'none'"
|
||||
|
||||
- id: acp
|
||||
name: '@deepseek-ai/dsh-acp'
|
||||
config:
|
||||
provider: deepseek-official
|
||||
model: deepseek-v4-pro
|
||||
|
||||
- id: system-prompt
|
||||
name: '@deepseek-ai/dsh-system-prompt'
|
||||
config:
|
||||
persona: |
|
||||
You are a coding assistant powered by the {{model}} model. Your working directory is {{cwd}}. Your bash tool runs under a file sandbox — a `[sandbox: file access denied …]` result is policy, not a command bug.
|
||||
|
||||
Verify your work by running the code or tests. Keep answers brief and factual.
|
||||
|
||||
- id: agent-instructions
|
||||
name: '@deepseek-ai/dsh-agent-instructions'
|
||||
config:
|
||||
maxBytes: 65536
|
||||
|
||||
- id: tool-subagent
|
||||
name: '@deepseek-ai/dsh-tool-subagent'
|
||||
config:
|
||||
provider: spawn
|
||||
toolName: subagent
|
||||
enableModelSelection: true
|
||||
backgroundMode: continuable
|
||||
maxDepth: 1
|
||||
|
||||
# Fork omits model selection so provider/model stay equal to the parent and the
|
||||
# inherited history remains eligible for KV Cache reuse. It stays one-shot because
|
||||
# a continuable child's `report` tool and prompt section precede that history and
|
||||
# invalidate the same prefix. `run_in_background` is off as an explicit foreground-only
|
||||
# choice even though the shipped ACP profile mounts the generic Job runtime.
|
||||
# See .agents/notes/implemented/feature/2026-08-18-model-selected-subagent-routes.md
|
||||
# and .agents/notes/implemented/architecture/2026-08-10-fork-children-stay-one-shot.md.
|
||||
- id: tool-subagent-fork
|
||||
name: '@deepseek-ai/dsh-tool-subagent'
|
||||
config:
|
||||
provider: fork
|
||||
toolName: subagent_fork
|
||||
backgroundMode: one-shot
|
||||
enableRunInBackground: false
|
||||
maxDepth: 1
|
||||
|
||||
- id: fs-sandbox
|
||||
name: '@deepseek-ai/dsh-fs-sandbox'
|
||||
config:
|
||||
cwd: !!js process.cwd()
|
||||
|
||||
- insert:
|
||||
- id: hooks-claude-code
|
||||
name: '@deepseek-ai/dsh-hooks-claude-code'
|
||||
config:
|
||||
configPath: ./hooks.json
|
||||
|
||||
- id: hooks-codex
|
||||
name: '@deepseek-ai/dsh-hooks-codex'
|
||||
config:
|
||||
configPath: ./codex-hooks.json
|
||||
|
|
@ -43,6 +43,7 @@ export default defineConfig({
|
|||
// apps/cli only, not apps/*: apps/web/tests/*.e2e.ts needs the built
|
||||
// frontend dist and runs under vitest.web.config.ts (the test:web job).
|
||||
include: ['packages/*/*/tests/**/*.e2e.ts', 'apps/cli/tests/**/*.e2e.ts', 'examples/*/tests/**/*.e2e.ts'],
|
||||
exclude: ['**/*.golden.e2e.ts'],
|
||||
// Real model calls: generous timeouts, and retries for transient flakes
|
||||
// (the shared internal key hits concurrency quotas). No coverage — the
|
||||
// unit suites own the coverage gate.
|
||||
|
|
|
|||
20
vitest.golden.config.ts
Normal file
20
vitest.golden.config.ts
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
import { availableParallelism } from 'node:os'
|
||||
import tsconfigPaths from 'vite-tsconfig-paths'
|
||||
import { defineConfig } from 'vitest/config'
|
||||
import { standardDecoratorPlugin, vitestExecArgv } from './vitest.shared.ts'
|
||||
|
||||
/** Owner-local assembled expected-output tests that do not use a recorded session as their input. */
|
||||
export default defineConfig({
|
||||
plugins: [tsconfigPaths({ projects: ['./tsconfig.base.json'] }), standardDecoratorPlugin()],
|
||||
test: {
|
||||
execArgv: vitestExecArgv,
|
||||
setupFiles: ['./scripts/test-invariants.ts'],
|
||||
include: [
|
||||
'apps/cli/tests/**/*.golden.e2e.ts',
|
||||
'examples/*/tests/**/*.golden.e2e.ts',
|
||||
],
|
||||
testTimeout: 120_000,
|
||||
hookTimeout: 30_000,
|
||||
maxWorkers: Math.min(5, availableParallelism()),
|
||||
},
|
||||
})
|
||||
|
|
@ -45,12 +45,10 @@ export default defineConfig({
|
|||
execArgv: vitestExecArgv,
|
||||
setupFiles: ['./scripts/test-invariants.ts'],
|
||||
include: [
|
||||
'scripts/**/*.snapshot.ts',
|
||||
'scripts/session-snapshot-corpus.corpus.ts',
|
||||
// The assembled Web snapshot executes generated client bundles; source
|
||||
// mode remains the zero-build path, while lib mode requires a prior build.
|
||||
...(process.env.DSH_EXAMPLE_MODE === 'lib' ? ['apps/web/tests/**/*.snapshot.ts'] : []),
|
||||
'apps/cli/tests/**/*.snapshot.ts',
|
||||
'examples/*/tests/**/*.snapshot.ts',
|
||||
'snapshots/**/*.snapshot.ts',
|
||||
],
|
||||
// Replay never writes committed outputs and every scenario owns its
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue