Merge pull request #3438 from deepseek-harness/fix/projcache-cross-version-read-compat
fix(session-projection-cache): survive upgrades across projcache domain versions
This commit is contained in:
commit
1915665e1e
28 changed files with 1500 additions and 53 deletions
|
|
@ -0,0 +1,6 @@
|
|||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md
|
||||
2026-09-02-projcache-cross-version-read-compat.md: 64b52ba482e5850e1d0c3d473ca87ba7822af67f
|
||||
2026-09-02-projcache-cross-version-read-compat.zh.md: c8adad9fd2b7184418a4da44e8b2e5d1f31feed7
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
# Agent Note: Projection-cache cross-version read compatibility (session_projcache v3/v4/v5)
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-09-02-projcache-cross-version-read-compat.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The `session_projcache` storage domain evolved through three on-disk generations across published releases. An upgraded DSH_HOME failed in two ways:
|
||||
|
||||
- **A v3 single-file home bricked startup after the upgrade**: the per-record layout's legacy bootstrap migrated the old whole-unit file without checking its `unit.version`, stamping the old records with the current version into the new tree; the domain layer's per-record zod validation at open then hit the missing now-required fields → `invalid-record` → the whole domain refused to open → the plugin tree failed to load. And because the bootstrap writes before validation runs, **the first boot permanently wrote the bad documents into the new tree** ("poisoning") — every later boot saw a non-empty tree, never took the legacy path again, and the home stayed unusable.
|
||||
- **A v4 per-record home lost its listing titles after the upgrade**: v4 documents were silently discarded by the version-stamp check (the per-record contract), and SessionList is a zero-I/O cache-only read, so a miss served the row without projections; titles only returned as each session was individually reopened.
|
||||
|
||||
The cache domain's own contract is "a stale or unreadable cache costs a longer tail replay, never a wrong value, never a refused load" — the hard failure and the wholesale discard each broke the first half of that contract or the product expectation.
|
||||
|
||||
## The three on-disk generations
|
||||
|
||||
| domain version | shipped in | layout | on-disk form | identity fields | row fields |
|
||||
|---|---|---|---|---|---|
|
||||
| 3 | 0.1.1-rc.2 | single | one file `storages/session_projcache.json` (`{unit:{name,version}, global, tables}`) | `createdAt`, `cwd?` | `ver`, `seq`, `val` |
|
||||
| 4 | 0.1.2-alpha.3 | per-record | one file per session `storages/session_projcache/sessions/<sessionId>.json` (`{version, record}`) | `createdAt`, `cwd?` | same |
|
||||
| 5 | 0.1.2-alpha.4 | per-record | same as v4 | + `isSeeded` (shipped required; now optional), `inheritedEventCount` (same) | same (`seq` numbers mean the same as v4; only type brands were added) |
|
||||
|
||||
The only substantive v4→v5 difference is the two new lineage identity fields; the `ver/seq/val` row shape is identical across all three generations, and `seq` numbering did not change ([the 2026-08-31 seq/offset brands note](2026-08-31-session-sequence-and-log-offset-brands.md) pins the on-disk numbers as unchanged). v3→v4 was a layout migration with identical record content.
|
||||
|
||||
One derived shape also exists: a v3 home that ran the v5 build once (the poisoned state) — its new tree holds documents **stamped 5 whose content is a v3 record** (no lineage fields).
|
||||
|
||||
## Decision
|
||||
|
||||
Declared read compatibility — reads tolerate vouched-for older versions, writes always stamp the current one:
|
||||
|
||||
1. **`DomainSpec.compatibleVersions` (new, optional)**: the domain owner declares "records stored under these older versions are also readable under the current record schemas" (typically by declaring the fields old records lack as optional). `defineDomain` validates each entry as a non-negative integer below the current version; `descriptorOf` projects the set onto the backend `KvUnitDescriptor`.
|
||||
2. **json backend per-record reads** accept version stamps in "current ∪ compatibleVersions"; anything outside the set is still discarded as foreign. **The write path always stamps the current version** (the first checkpoint after reading an old record naturally advances it). The `single` layout stays exact-version.
|
||||
3. **Legacy-bootstrap version gate (the actual bug fix)**: the old whole-unit file's `unit.version` must fall inside the accepted set to be migrated; otherwise the file is left alone and the unit reads empty — stamping records the owner never vouched for turns a discardable stale cache into hard schema failures at the domain layer.
|
||||
4. **The projcache domain declares `version: 5, compatibleVersions: [3, 4]`**, and the two lineage fields become `.optional()`. The single reader of stored identities, `identityMatches`, normalizes absence to the unseeded lineage (`?? false` / `?? 0`): exact for unforked sessions, while a forked session's expectation is seeded → natural mismatch → discard and cold rebuild, so the lineage binding loses none of its protection.
|
||||
5. **The poisoned state self-heals**: documents stamped 5 without lineage fields parse under the optional schema (their content is the real pre-upgrade cache data), so the home boots again and titles serve immediately.
|
||||
6. **Schema-validation backstop: `invalidRecords: 'backup-and-skip'` (declared by this domain only)**. A stored record that still fails to parse beyond read compatibility no longer refuses the whole domain: the domain layer calls the backend's `KvUnit.backupRecord` (json per-record implementation = rename the document to `<key>.json.bak.<YYYYMMDDHHmm>`, bytes kept, never read again), prints the concrete failure with `logger.error` (domain, table, key, destination, zod cause), and continues the open with the record absent; the next cold read rebuilds and rewrites that session's cache. **The policy is an explicit per-domain declaration and the default stays fail-loud** — other domains still refuse the whole load on invalid stored data, and a backend without `backupRecord` (single layout, row stores) also falls back to fail-loud. Naming history: quarantine → backup-and-skip (user ruling: the word must carry both "back up" and "skip", sharing its root with the `.bak` suffix; skip-backup was rejected because the CLI `--skip-X` convention reads it as "do not back up"). For this domain it supersedes the reset/destroy recovery path of the [2026-07-28 storage recovery proposal](../../proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md), which stays live for authoritative and whole-medium damage.
|
||||
|
||||
### Upgrade matrix
|
||||
|
||||
| home shape | behavior after the fix |
|
||||
|---|---|
|
||||
| v3 single-file (not poisoned) | bootstrap migrates (3 ∈ accepted set) → titles serve immediately |
|
||||
| v3 + poisoned new tree | new-tree documents read directly (optional tolerance) → boot restored, titles serve immediately |
|
||||
| v4 per-record | documents read directly (4 ∈ accepted set) → titles serve immediately |
|
||||
| v5 healthy | unaffected |
|
||||
| old records of forked (seeded) sessions | identity mismatch → discarded, cold rebuild when the session opens (safe side) |
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Discard-and-rebuild only** (bootstrap gate + bump to v6): fixes the boot, but every SessionList title is lost after the upgrade until each session is reopened — fails the upgrade-and-go product requirement.
|
||||
- **Schema `.default()` fills**: behaviorally equivalent to optional + reader normalization, but bakes the "absent = unseeded" interpretation into the durable schema's output type; ruled for optional — the schema honestly describes every accepted on-disk shape and the interpretation lives at the consumer (user ruling, 2026-09-02).
|
||||
- **Roll the domain version back 5→4**: the smallest diff (three lines), but breaks version monotonicity, depends on the "bootstrap skips no versions" bug itself, and drops every poisoned and healthy v5 home's cache.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A deployment routing this domain to the sqlite backend gets none of the tolerance: sqlite implements neither `compatibleVersions` nor `backupRecord`, so behavior degrades to the old strict-version semantics (a whole-unit version mismatch still refuses with `version-mismatch`; nothing loosens, nothing serves wrong values). Shipped compositions route this domain to json, so this stays a deployment-configuration risk only.
|
||||
- The optional lineage fields widen what a current-version document may omit: a v5-stamped record stripped of its lineage decodes as unseeded. The identity match still refuses it for seeded callers, and the per-row `ver` guard still screens every value, so the residual exposure is an unseeded caller reading an unseeded-shaped record — the same trust extended to genuine pre-lineage records.
|
||||
- `backupRecord` overwrites a same-minute backup of the same key (the newer bytes win); distinct minutes and distinct keys never collide.
|
||||
|
||||
## Testing
|
||||
|
||||
- `storage-json` unit tests: compat-stamped reads / out-of-set discards / writes stamping current; legacy bootstrap migrating only accepted versions (including the migrated-documents-stamp-current assertion); `backupRecord` move / absent read / rewrite / closed guard.
|
||||
- `storage-domain` unit tests: `compatibleVersions` / `invalidRecords` declaration validation; backup-and-skip falling back to fail-loud when the backend has no `backupRecord`.
|
||||
- `session-projection-cache` unit tests: records without lineage fields serve unseeded sessions verbatim and are discarded for seeded ones.
|
||||
- **Archived-fixture recovery tests** (`tests/fixtures.spec.ts` + `tests/fixtures/`): four media archives produced by the real released builds — `v3-single-unit.json` (the 0.1.1-rc.2 whole-unit file), `v4-session-doc.json` (0.1.2-alpha.3), `v5-session-doc.json` (current), `v5-lineageless-doc.json` (the unguarded bootstrap's poisoned shape, synthesized from the v3 record) — each opened through the real storage stack, asserting the listing serves the archived title and that a live write rewrites the document to the current version (v5 stamp + lineage fields + fresh value); plus the backup-and-skip behavior for a schema-failing record (boot survives, `.bak` lands, log is concrete, neighbor records unharmed).
|
||||
- End-to-end acceptance, executed against the real release artifacts: the published 0.1.1-rc.2 and 0.1.2-alpha.3 npm builds seeded homes through their own web apps (model turns plus a rename RPC), the published 0.1.2-alpha.4 build reproduced both failures (including the poisoned tree), and the fixed build served every home shape — pristine v3, poisoned v3, v4, and fresh — with the SessionList RPC returning the recorded titles verbatim.
|
||||
|
||||
Future bump procedure: when a new version's shape can tolerate old records through "optional fields + reader normalization", add the old version to `compatibleVersions`; otherwise bump normally (discard and rebuild) and remove the no-longer-compatible versions from the set. Either way, the package README requires the bump to land with archived fixtures and tests proving the chosen disposition.
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
# Agent Note: 投影缓存跨版本读兼容(session_projcache v3/v4/v5)
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-09-02-projcache-cross-version-read-compat.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`session_projcache` 存储域在已发布版本间演进了三代磁盘结构。升级后的 DSH_HOME 出现两类故障:
|
||||
|
||||
- **v3 单文件 home 升级后启动硬失败**:per-record 布局的 legacy bootstrap 迁移旧单文件时不检查其 `unit.version`,把旧记录原样打上当前版本戳写入新树;domain 层开域时逐条 zod 校验,旧记录缺新增必填字段 → `invalid-record` → 整个域拒开 → 插件树加载失败。且 bootstrap 先写盘后校验,**首次启动即把坏文档永久写入新树**("投毒")——此后每次启动新树非空、连 legacy 路径都不再走,home 持续不可用。
|
||||
- **v4 per-record home 升级后列表丢标题**:v4 文档被版本戳检查静默丢弃(per-record 契约),SessionList 是零 I/O 纯缓存读,miss 后整行不带投影;标题要等每个会话被逐个重新打开后才恢复。
|
||||
|
||||
缓存域自身的契约是"过期或不可读的缓存只付出更长的尾部重放,绝不给出错值、绝不拒载"——硬失败与整体丢弃都违背该契约的前半句或产品预期。
|
||||
|
||||
## 三代磁盘结构差异
|
||||
|
||||
| domain version | 携带发布 | 布局 | 磁盘形态 | identity 字段 | 行字段 |
|
||||
|---|---|---|---|---|---|
|
||||
| 3 | 0.1.1-rc.2 | single | 单文件 `storages/session_projcache.json`(`{unit:{name,version}, global, tables}`) | `createdAt`, `cwd?` | `ver`, `seq`, `val` |
|
||||
| 4 | 0.1.2-alpha.3 | per-record | 每会话一份 `storages/session_projcache/sessions/<sessionId>.json`(`{version, record}`) | `createdAt`, `cwd?` | 同上 |
|
||||
| 5 | 0.1.2-alpha.4 | per-record | 同 v4 | + `isSeeded`(v5 首发必填;现为 optional)、`inheritedEventCount`(同前) | 同上(`seq` 数值语义与 v4 相同,仅类型加 brand) |
|
||||
|
||||
v4→v5 的唯一实质差异是 identity 新增两个 lineage 字段;行内 `ver/seq/val` 三代一致,`seq` 的数值含义未变([2026-08-31 seq/offset brands note](2026-08-31-session-sequence-and-log-offset-brands.zh.md) 明确 on-disk 数值不变)。v3→v4 是布局迁移,记录内容结构一致。
|
||||
|
||||
另有一种衍生形态:跑过一次 v5 版本的 v3 home(投毒态)——新树里存在**版本戳为 5 但内容是 v3 记录**(缺 lineage 字段)的文档。
|
||||
|
||||
## 决策
|
||||
|
||||
声明式读兼容——读容忍 owner 背书过的旧版本,写恒戳当前版本:
|
||||
|
||||
1. **`DomainSpec.compatibleVersions`(新增,可选)**:域 owner 声明"这些旧版本的存量记录在当前记录 schema 下也可读"(典型手段:新增字段标 optional)。`defineDomain` 校验各项为小于当前 version 的非负整数;`descriptorOf` 透传到后端 `KvUnitDescriptor`。
|
||||
2. **json 后端 per-record 读**:接受"当前版本 ∪ compatibleVersions"内的版本戳,集合外照旧视为 foreign 丢弃;**写路径永远戳当前版本**(读到旧记录后的下一次 checkpoint 自然把它推进到当前版本)。single 布局维持 exact-version 不变。
|
||||
3. **legacy bootstrap 版本把关(bug 修复本体)**:旧单文件的 `unit.version` 必须落在接受集合内才迁移,否则视为空 unit 留在原地——为 owner 未背书的记录打当前版本戳,会把"可丢弃的过期缓存"变成 domain 层的 schema 硬失败。
|
||||
4. **projcache 域声明 `version: 5, compatibleVersions: [3, 4]`**;两个 lineage 字段改为 `.optional()`。唯一消费 stored identity 的读点 `identityMatches` 把缺失归一化为 unseeded lineage(`?? false` / `?? 0`):对非 fork 会话这是精确值;fork 会话的 expected 是 seeded → 天然 mismatch → 丢弃冷读重建,lineage 绑定的防护不放松。
|
||||
5. **投毒态自愈**:v5 戳缺 lineage 字段的文档被 optional schema 直接接受(内容本就是升级前的真实缓存数据),home 恢复可启动且标题立即可服务。
|
||||
6. **schema 校验兜底:`invalidRecords: 'backup-and-skip'`(仅本域声明)**。读兼容之外仍然解析失败的存量记录不再让整个域拒开:domain 层调用后端的 `KvUnit.backupRecord`(json per-record 实现=把文档改名为 `<key>.json.bak.<YYYYMMDDHHmm>`,字节留档、不再被读取),用 `logger.error` 打印具体失败信息(域名、表、键、移动去向、zod 失败原因),随后当该记录不存在继续启动;下一次冷读会重建并重写该会话的缓存。**该策略是域级显式声明,缺省仍为 fail-loud**——其他业务域的存量数据校验失败照旧整域拒载;后端没有 `backupRecord` 能力(single 布局、行存储)时也回退 fail-loud。命名沿革:quarantine → backup-and-skip(用户裁决:词要同时含"备份"与"跳过"两义,且与 `.bak` 后缀同源;skip-backup 因 CLI `--skip-X` 惯例存在"不备份"反读而弃用)。对本域而言,该策略取代了 [2026-07-28 存储恢复提案](../../proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md)中 reset/destroy 的恢复途径;该提案对权威介质与整介质损坏仍然有效。
|
||||
|
||||
### 升级矩阵
|
||||
|
||||
| home 形态 | 修复后行为 |
|
||||
|---|---|
|
||||
| v3 单文件(未投毒) | bootstrap 迁移(3 ∈ 接受集)→ 标题立即可服务 |
|
||||
| v3 + 投毒新树 | 新树文档直接读入(optional 容忍)→ 启动恢复、标题立即可服务 |
|
||||
| v4 per-record | 文档直接读入(4 ∈ 接受集)→ 标题立即可服务 |
|
||||
| v5 正常 | 不受影响 |
|
||||
| fork(seeded)会话的旧记录 | identity mismatch → 丢弃,打开会话时冷读重建(安全侧) |
|
||||
|
||||
## 备选方案
|
||||
|
||||
- **只丢弃重建**(bootstrap 把关 + bump v6):启动可修,但升级后 SessionList 标题全丢、要逐会话打开才恢复——不满足升级即用的产品要求。
|
||||
- **schema `.default()` 填缺省**:行为与 optional+读点归一化等价,但把"缺失=unseeded"的解释固化进 durable schema 的输出类型;拍板为 optional——schema 如实描述介质上所有被接受的形态,解释权在消费点(2026-09-02 用户裁决)。
|
||||
- **域版本回退 5→4**:改动最小(三行),但破坏版本单调性、依赖"bootstrap 不查版本"这个 bug 本身、且投毒态与正常 v5 home 的缓存全被丢弃。
|
||||
|
||||
## 影响
|
||||
|
||||
- 部署方若把本域路由到 sqlite 后端,得不到任何容忍能力:sqlite 既未实现 `compatibleVersions` 也没有 `backupRecord`,行为退化为原有的严格版本语义(整 unit 版本不匹配仍 `version-mismatch` 拒开;不放松、不出错值)。shipped 组合固定路由 json,此风险仅存在于部署配置层面。
|
||||
- optional lineage 字段放宽了当前版本文档可缺省的范围:被剥离 lineage 的 v5 戳记录会解码为 unseeded。身份比对仍会对 seeded 调用方拒收,逐行 `ver` 守卫仍筛查每个值,残余暴露面只是 unseeded 调用方读到 unseeded 形态的记录——与真实 pre-lineage 记录享有的信任完全相同。
|
||||
- `backupRecord` 对同一键的同一分钟内重复备份会覆盖前一份(新字节胜出);不同分钟、不同键永不冲突。
|
||||
|
||||
## 测试
|
||||
|
||||
- `storage-json` 单测:compat 版本戳读入/集合外丢弃/写恒当前版本;legacy bootstrap 仅在版本被接受时迁移(含迁移后文档戳当前版本断言);`backupRecord` 移档/读缺席/重写/封闭守卫。
|
||||
- `storage-domain` 单测:`compatibleVersions`/`invalidRecords` 声明校验;后端无 `backupRecord` 时 backup-and-skip 回退 fail-loud。
|
||||
- `session-projection-cache` 单测:缺 lineage 字段的记录对 unseeded 会话按原值服务、对 seeded 会话丢弃。
|
||||
- **归档 fixtures 独立恢复测试**(`tests/fixtures.spec.ts` + `tests/fixtures/`):真实发布物产出的四份介质存档——`v3-single-unit.json`(0.1.1-rc.2 整域单文件)、`v4-session-doc.json`(0.1.2-alpha.3)、`v5-session-doc.json`(当前版)、`v5-lineageless-doc.json`(无守卫 bootstrap 的投毒形态,由 v3 记录合成)——逐一走真实存储栈开域,断言列表读出归档标题、且 live 写把文档重写为当前版本(v5 戳 + lineage 字段 + 新值);外加 schema 失败记录的 backup-and-skip 行为(启动不失败、`.bak` 落盘、日志具体、邻居记录不受累)。
|
||||
- 端到端验收,以真实发布物执行:已发布的 0.1.1-rc.2 与 0.1.2-alpha.3 npm 包经各自 web app 造数(真实模型对话 + rename RPC),已发布的 0.1.2-alpha.4 包复现两类故障(含投毒树),修复后构建对纯净 v3、投毒 v3、v4、全新四种 home 形态经 SessionList RPC 原样返回记录在案的标题。
|
||||
|
||||
未来 bump 流程:新版本结构若可用"optional 字段 + 读点归一化"容忍旧记录,就把旧版本加入 `compatibleVersions`;否则正常 bump(丢弃重建),并把不再兼容的版本从集合中移除。无论哪条路,包 README 都要求 bump 随附归档 fixture 和论证所选处置方式的测试。
|
||||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md
|
||||
2026-07-28-storage-root-and-derived-medium-recovery.md: 1505be1c58d5cf829327b2919113bb2e42798ce7
|
||||
2026-07-28-storage-root-and-derived-medium-recovery.zh.md: 1bab5ab663df1419cc826c3d6acb59bd8bff7de0
|
||||
2026-07-28-storage-root-and-derived-medium-recovery.md: 68a6792c38a8fe097558de3d5a857ab4ea0d3533
|
||||
2026-07-28-storage-root-and-derived-medium-recovery.zh.md: 0c712b3b928f4b3df44e706b154c3a90a784403c
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ The persisted projection cache ([note](2026-07-27-session-projection-and-command
|
|||
|
||||
**Where the files actually live (root mismatch closed; resolve-once residual still open).** The shared base defaults the session store to the global harness home (`$DSH_HOME/sessions`, default `~/.dsh/sessions`), while the shipped Web overlay used to give the json backend the relative root `./.storages`: `workspace.json` and `session_projcache.json` landed under `<launch dir>/.storages/` — two launches from different directories shared their sessions yet saw different workspace registries and different projection caches, and the cache exists precisely to serve the cross-session cold listing, which missed for every session last cached under another launch directory. That mismatch is now closed: the overlay anchors `storage-json.root` to `$DSH_HOME/storages` with the same `!!js` expression the session root uses (`apps/cli/config/web.cordis.yml`). The residual hazard: `JsonStorageBackend` still never resolves its root — each unit open joins the path against whatever `process.cwd()` is at that moment (packages/storage/storage-json/src/index.ts); the shipped overlay root is already absolute and unaffected, but any relative root (bare Loader boots, tests) still splits on a later cwd change — the exact hazard the JSONL session backend resolves-once to prevent ("later process.cwd() changes cannot split one backend across roots", packages/session/session-persistence-jsonl/src/index.ts).
|
||||
|
||||
**Recovery behavior.** Inside a healthy medium the cache is fully self-healing by design: a `stateVersion`-mismatched row is discarded and refolded, a log shrunk below a row's watermark is detected by the anchored restore floor and answered with one full re-read, and every background write is fail-soft. But at the *medium* level there is no recovery at all: a truncated, hand-edited, or version-bumped `session_projcache.json` fails `openJsonUnit` with `malformed-medium`/`version-mismatch` (packages/storage/storage-json/src/format.ts), a schema-drifted record fails domain open with `invalid-record` (packages/storage/storage-domain/src/index.ts), the rejection propagates through `SessionProjectionCache[Service.init]`, and under the CLI's fail-loud boot the assembly refuses to start. A file whose entire content is rebuildable from session logs can brick boot. This contradicts the cache package's own stated stance ("a stale or unreadable cache costs a longer tail replay, never a wrong value") and the cache domain spec's JSDoc ("version bumps discard the whole medium"), which describes an aspiration, not the implementation. Partially superseded for the projection cache: [the per-session cache files note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.md) removed the global `session_projcache` domain, so the cache half of this proposal (recovery on that domain) no longer applies; the `workspace.json` half remains current. The same fail-loud path is *correct* for `workspace.json` — workspace records are authoritative, not derivable — so the missing concept is a per-domain declaration of authority, not a global behavior change.
|
||||
**Recovery behavior.** Inside a healthy medium the cache is fully self-healing by design: a `stateVersion`-mismatched row is discarded and refolded, a log shrunk below a row's watermark is detected by the anchored restore floor and answered with one full re-read, and every background write is fail-soft. But at the *medium* level there is no recovery at all: a truncated, hand-edited, or version-bumped `session_projcache.json` fails `openJsonUnit` with `malformed-medium`/`version-mismatch` (packages/storage/storage-json/src/format.ts), a schema-drifted record fails domain open with `invalid-record` (packages/storage/storage-domain/src/index.ts), the rejection propagates through `SessionProjectionCache[Service.init]`, and under the CLI's fail-loud boot the assembly refuses to start. A file whose entire content is rebuildable from session logs can brick boot. This contradicts the cache package's own stated stance ("a stale or unreadable cache costs a longer tail replay, never a wrong value") and the cache domain spec's JSDoc ("version bumps discard the whole medium"), which describes an aspiration, not the implementation. Partially superseded for the projection cache: [the per-session cache files note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.md) removed the global `session_projcache` domain, so the cache half of this proposal (recovery on that domain) no longer applies; the `workspace.json` half remains current. The `invalid-record` class for the per-record projection cache is now also superseded: the shipped domain declares `invalidRecords: 'backup-and-skip'` ([cross-version read-compat note](../../implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md)), which backs the failing record up and skips it at open, so the reset/destroy proposal below stays relevant only for whole-medium damage on authoritative or single-document media. The same fail-loud path is *correct* for `workspace.json` — workspace records are authoritative, not derivable — so the missing concept is a per-domain declaration of authority, not a global behavior change.
|
||||
|
||||
## Proposal
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Status: proposed
|
|||
|
||||
**文件到底存在哪(根错位已收口,resolve-once 残余仍开放)。** 共享 base 将会话存储默认为全局 harness home(`$DSH_HOME/sessions`,默认 `~/.dsh/sessions`),而出厂 Web overlay 曾给 json 后端相对根 `./.storages`:`workspace.json` 和 `session_projcache.json` 落在 `<启动目录>/.storages/` 下——从两个不同目录启动,会话相同,工作区注册表和投影缓存却各是一份,而缓存存在的意义恰恰是跨会话冷列表,凡上次在别的启动目录下缓存过的会话全部 miss。这一错位已消除:overlay 现以与会话根同一段 `!!js` 表达式把 `storage-json.root` 锚定到 `$DSH_HOME/storages`(`apps/cli/config/web.cordis.yml`)。残余隐患:`JsonStorageBackend` 仍从不 resolve 根——每次打开 unit 都把路径 join 到当时的 `process.cwd()` 上(packages/storage/storage-json/src/index.ts);出厂 overlay 的根已是绝对路径不受影响,但任何相对根(裸 Loader 启动、测试)仍会被后续 cwd 变化劈开,JSONL 会话后端用「构造时 resolve 一次」防住的正是它("later process.cwd() changes cannot split one backend across roots",packages/session/session-persistence-jsonl/src/index.ts)。
|
||||
|
||||
**恢复行为。** 在健康介质内部,缓存按设计完全自愈:`stateVersion` 不匹配的行被丢弃重折,日志缩短到行水位以下由带锚的 restore floor 检出并以一次全量重读回答,每次后台写都是 fail-soft。但在*介质*层面完全没有恢复:被截断、被手改或版本被 bump 的 `session_projcache.json` 会让 `openJsonUnit` 以 `malformed-medium`/`version-mismatch` 失败(packages/storage/storage-json/src/format.ts),schema 漂移的记录让域 open 以 `invalid-record` 失败(packages/storage/storage-domain/src/index.ts),拒绝一路穿过 `SessionProjectionCache[Service.init]`,在 CLI 的 fail-loud 启动下整个组装拒绝启动。一个内容完全可从会话日志重建的文件能把启动搞死。这与缓存包自己声明的立场("a stale or unreadable cache costs a longer tail replay, never a wrong value")和缓存域 spec 的 JSDoc("version bumps discard the whole medium")相矛盾——后者描述的是愿望而非实现。投影缓存半边已被[每会话缓存文件 note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md) 部分取代:全局 `session_projcache` domain 已移除,本提案的缓存恢复半边不再适用;`workspace.json` 半边仍然有效。同一条 fail-loud 路径对 `workspace.json` 却是*正确*的——工作区记录是权威数据,不可派生——所以缺的概念是按域声明权威性,而不是全局改行为。
|
||||
**恢复行为。** 在健康介质内部,缓存按设计完全自愈:`stateVersion` 不匹配的行被丢弃重折,日志缩短到行水位以下由带锚的 restore floor 检出并以一次全量重读回答,每次后台写都是 fail-soft。但在*介质*层面完全没有恢复:被截断、被手改或版本被 bump 的 `session_projcache.json` 会让 `openJsonUnit` 以 `malformed-medium`/`version-mismatch` 失败(packages/storage/storage-json/src/format.ts),schema 漂移的记录让域 open 以 `invalid-record` 失败(packages/storage/storage-domain/src/index.ts),拒绝一路穿过 `SessionProjectionCache[Service.init]`,在 CLI 的 fail-loud 启动下整个组装拒绝启动。一个内容完全可从会话日志重建的文件能把启动搞死。这与缓存包自己声明的立场("a stale or unreadable cache costs a longer tail replay, never a wrong value")和缓存域 spec 的 JSDoc("version bumps discard the whole medium")相矛盾——后者描述的是愿望而非实现。投影缓存半边已被[每会话缓存文件 note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md) 部分取代:全局 `session_projcache` domain 已移除,本提案的缓存恢复半边不再适用;`workspace.json` 半边仍然有效。per-record 投影缓存的 `invalid-record` 一类如今也已被取代:已发布的域声明了 `invalidRecords: 'backup-and-skip'`([跨版本读兼容 note](../../implemented/architecture/2026-09-02-projcache-cross-version-read-compat.zh.md)),open 时把失败记录备份后跳过,因此下文 reset/destroy 提案仅对权威介质或单文档介质的整介质损坏仍然相关。同一条 fail-loud 路径对 `workspace.json` 却是*正确*的——工作区记录是权威数据,不可派生——所以缺的概念是按域声明权威性,而不是全局改行为。
|
||||
|
||||
## 提案
|
||||
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/subsystems/storage.md
|
||||
storage.md: 1e4141e6ef1c6f8e1c2593e21e788b626d6b1ed7
|
||||
storage.zh.md: f0433c600674741c3de0ce3e99430297839ce124
|
||||
storage.md: e52f6d8869ee3092a5be99f0bd2873b410b45ed9
|
||||
storage.zh.md: f752862af9f10cfb32f39f2a1aa7801c922f841e
|
||||
|
|
|
|||
|
|
@ -65,6 +65,26 @@ interface DomainSpec {
|
|||
* (a stale record document is discarded, never migrated).
|
||||
*/
|
||||
readonly layout?: 'single' | 'per-record'
|
||||
/**
|
||||
* Older domain versions whose stored records the current record schemas
|
||||
* also accept (the declaring owner vouches for that, typically by
|
||||
* declaring the fields older records lack as optional). `per-record` backends
|
||||
* read documents stamped with a listed version instead of discarding them,
|
||||
* and accept a legacy whole-unit file so stamped for the one-time
|
||||
* bootstrap; writes always stamp {@link version}.
|
||||
*/
|
||||
readonly compatibleVersions?: readonly number[]
|
||||
/**
|
||||
* What `open` does with a stored table record that fails its zod schema.
|
||||
* Absent (the default), the whole open rejects with `invalid-record` —
|
||||
* right for authoritative data. `'backup-and-skip'` is for domains whose
|
||||
* records are disposable derived data: the backend moves the record's
|
||||
* document aside (`KvUnit.backupRecord`), the failure is logged with
|
||||
* its cause, and the open continues with the record absent. A backend
|
||||
* without `backupRecord` (no per-record document to move) falls back
|
||||
* to the rejecting default. The global slot always rejects.
|
||||
*/
|
||||
readonly invalidRecords?: 'backup-and-skip'
|
||||
/** Optional global singleton slot. */
|
||||
readonly global?: DomainGlobalSpec<unknown>
|
||||
/** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */
|
||||
|
|
@ -180,7 +200,10 @@ The mounted domain facility. Opens declared domains over routed backends; one fa
|
|||
* (`facet-unsupported`); open the unit projected from the spec (backend
|
||||
* `version-mismatch`/`malformed-medium` pass through); load and validate
|
||||
* every stored record against the spec's zod schemas (`invalid-record`
|
||||
* with the offending table and key); construct the domain.
|
||||
* with the offending table and key — unless the spec declares
|
||||
* `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in
|
||||
* which case the failing record is backed up, logged, and skipped);
|
||||
* construct the domain.
|
||||
*
|
||||
* Lifecycle: the CALLER owns the returned handle and closes it via
|
||||
* `Domain.close()` (typically as its own `ctx.effect` disposer) — the
|
||||
|
|
|
|||
|
|
@ -65,6 +65,26 @@ interface DomainSpec {
|
|||
* (a stale record document is discarded, never migrated).
|
||||
*/
|
||||
readonly layout?: 'single' | 'per-record'
|
||||
/**
|
||||
* Older domain versions whose stored records the current record schemas
|
||||
* also accept (the declaring owner vouches for that, typically by
|
||||
* declaring the fields older records lack as optional). `per-record` backends
|
||||
* read documents stamped with a listed version instead of discarding them,
|
||||
* and accept a legacy whole-unit file so stamped for the one-time
|
||||
* bootstrap; writes always stamp {@link version}.
|
||||
*/
|
||||
readonly compatibleVersions?: readonly number[]
|
||||
/**
|
||||
* What `open` does with a stored table record that fails its zod schema.
|
||||
* Absent (the default), the whole open rejects with `invalid-record` —
|
||||
* right for authoritative data. `'backup-and-skip'` is for domains whose
|
||||
* records are disposable derived data: the backend moves the record's
|
||||
* document aside (`KvUnit.backupRecord`), the failure is logged with
|
||||
* its cause, and the open continues with the record absent. A backend
|
||||
* without `backupRecord` (no per-record document to move) falls back
|
||||
* to the rejecting default. The global slot always rejects.
|
||||
*/
|
||||
readonly invalidRecords?: 'backup-and-skip'
|
||||
/** Optional global singleton slot. */
|
||||
readonly global?: DomainGlobalSpec<unknown>
|
||||
/** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */
|
||||
|
|
@ -180,7 +200,10 @@ The mounted domain facility. Opens declared domains over routed backends; one fa
|
|||
* (`facet-unsupported`); open the unit projected from the spec (backend
|
||||
* `version-mismatch`/`malformed-medium` pass through); load and validate
|
||||
* every stored record against the spec's zod schemas (`invalid-record`
|
||||
* with the offending table and key); construct the domain.
|
||||
* with the offending table and key — unless the spec declares
|
||||
* `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in
|
||||
* which case the failing record is backed up, logged, and skipped);
|
||||
* construct the domain.
|
||||
*
|
||||
* Lifecycle: the CALLER owns the returned handle and closes it via
|
||||
* `Domain.close()` (typically as its own `ctx.effect` disposer) — the
|
||||
|
|
|
|||
|
|
@ -2151,7 +2151,7 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [
|
|||
methods: [
|
||||
{
|
||||
signature: 'async open<S extends DomainSpec>(spec: S): Promise<Domain<S>>',
|
||||
description: 'Open one declared domain. Steps, each failing the whole call: reject a name that is already open (`already-open`); resolve the backend route (`backend-not-found` passes through from the hub); require its `kv` facet (`facet-unsupported`); open the unit projected from the spec (backend `version-mismatch`/`malformed-medium` pass through); load and validate every stored record against the spec\'s zod schemas (`invalid-record` with the offending table and key); construct the domain.\n\nLifecycle: the CALLER owns the returned handle and closes it via `Domain.close()` (typically as its own `ctx.effect` disposer) — the facility does not tie the domain to any consumer fiber. Domains still open when the facility unmounts are closed by the plugin disposer.',
|
||||
description: 'Open one declared domain. Steps, each failing the whole call: reject a name that is already open (`already-open`); resolve the backend route (`backend-not-found` passes through from the hub); require its `kv` facet (`facet-unsupported`); open the unit projected from the spec (backend `version-mismatch`/`malformed-medium` pass through); load and validate every stored record against the spec\'s zod schemas (`invalid-record` with the offending table and key — unless the spec declares `invalidRecords: \'backup-and-skip\'` and the unit can move documents aside, in which case the failing record is backed up, logged, and skipped); construct the domain.\n\nLifecycle: the CALLER owns the returned handle and closes it via `Domain.close()` (typically as its own `ctx.effect` disposer) — the facility does not tie the domain to any consumer fiber. Domains still open when the facility unmounts are closed by the plugin disposer.',
|
||||
parameters: [{ name: 'spec', description: 'The domain declaration, typically from `defineDomain`.' }],
|
||||
returns: 'the opened domain handle, typed by the spec.',
|
||||
},
|
||||
|
|
@ -3956,7 +3956,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
|||
},
|
||||
{
|
||||
name: 'DomainSpec',
|
||||
declaration: 'export interface DomainSpec {\n readonly name: string;\n readonly version: number;\n readonly layout?: \'single\' | \'per-record\';\n readonly global?: DomainGlobalSpec<unknown>;\n readonly tables: Record<string, DomainTableSpec>;\n}',
|
||||
declaration: 'export interface DomainSpec {\n readonly name: string;\n readonly version: number;\n readonly layout?: \'single\' | \'per-record\';\n readonly compatibleVersions?: readonly number[];\n readonly invalidRecords?: \'backup-and-skip\';\n readonly global?: DomainGlobalSpec<unknown>;\n readonly tables: Record<string, DomainTableSpec>;\n}',
|
||||
},
|
||||
{
|
||||
name: 'DomainTableSpec',
|
||||
|
|
@ -4264,11 +4264,11 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
|||
},
|
||||
{
|
||||
name: 'KvUnit',
|
||||
declaration: 'export interface KvUnit {\n loadAll(): Promise<{\n tables: Record<string, Record<string, unknown>>;\n global: unknown;\n }>;\n putRecord(table: string, key: string, value: unknown): Promise<void>;\n deleteRecord(table: string, key: string): Promise<void>;\n setGlobal(value: unknown): Promise<void>;\n close(): Promise<void>;\n}',
|
||||
declaration: 'export interface KvUnit {\n loadAll(): Promise<{\n tables: Record<string, Record<string, unknown>>;\n global: unknown;\n }>;\n putRecord(table: string, key: string, value: unknown): Promise<void>;\n deleteRecord(table: string, key: string): Promise<void>;\n backupRecord?(table: string, key: string): Promise<string>;\n setGlobal(value: unknown): Promise<void>;\n close(): Promise<void>;\n}',
|
||||
},
|
||||
{
|
||||
name: 'KvUnitDescriptor',
|
||||
declaration: 'export interface KvUnitDescriptor {\n readonly name: string;\n readonly version: number;\n readonly tables: readonly string[];\n readonly hasGlobal: boolean;\n readonly layout?: \'single\' | \'per-record\';\n}',
|
||||
declaration: 'export interface KvUnitDescriptor {\n readonly name: string;\n readonly version: number;\n readonly tables: readonly string[];\n readonly hasGlobal: boolean;\n readonly layout?: \'single\' | \'per-record\';\n readonly compatibleVersions?: readonly number[];\n}',
|
||||
},
|
||||
{
|
||||
name: 'LlmAdapter',
|
||||
|
|
|
|||
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session/session-projection-cache/README.md
|
||||
README.md: 51b9d86724af96304cf09a5c7c1b61b7394d336a
|
||||
README.zh.md: bb84b67bdde678884fd4f2be1b14b2161da8c2a0
|
||||
README.md: 9fd9766d75ab3f9a460b2802ac59810839f21ae4
|
||||
README.zh.md: 98725d9ce76ab44821adf3d43e3807932cf6b675
|
||||
|
|
|
|||
|
|
@ -64,6 +64,8 @@ Three mandatory points always write: session creation persists the seed-derived
|
|||
|
||||
The log leads and the cache follows: a live checkpoint flushes the session's buffered events durably before the cache row lands, so a crash can leave the cache behind the log but never ahead of it. Reads and writes share the storage domain's coherent in-memory state; the per-unit write chain mutates memory only after durability. Each version-stamped record must match the live unit schema and complete lifecycle identity (`createdAt`, `cwd`, `isSeeded`, and `inheritedEventCount`), so a row initialized under one fork cut cannot seed another. The JSON backend stores each record at `<root>/session_projcache/sessions/<id>.json` in an owner-only directory tree.
|
||||
|
||||
Upgrades never cost the boot or the listing: records stamped with a version in the spec's `compatibleVersions` stay readable (their absent lineage fields decode as the unseeded lineage — exact for unseeded sessions, while a seeded caller fails the identity match and refolds cold), and a stored record that still fails schema validation is moved aside as `<id>.json.bak.<stamp>` under the domain's `invalidRecords: 'backup-and-skip'` policy, logged with its cause, and rebuilt by the next checkpoint.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
|
|
@ -126,6 +128,7 @@ These limits define where the cache needs operational care. They are current pac
|
|||
- **No eviction or retention surface** — records accumulate per session; pruning stored checkpoints is out-of-band maintenance, same stance as session persistence itself.
|
||||
- **Interval throttle is per-session coarse** — the timer arms at the first dirty event after a clean write; a steady sub-threshold trickle writes once per interval, not a sliding window.
|
||||
- **No cache-side cold refold** — the cache serves and refreshes its rows but never reads the session log (it does not depend on the persistence layer); a consumer that needs a guaranteed cold snapshot refolds from the log itself.
|
||||
- **Every schema or domain-version change must prove its upgrade story** — a change to the stored record schema or the domain version lands in the same PR with an archived fixture of the previously shipped on-disk format under `tests/fixtures/` and test cases in `tests/fixtures.spec.ts` proving the chosen disposition: read-compat recovery (`compatibleVersions`), current-version rewrite, or backup-and-skip salvage. A bump whose old records are simply discarded still proves that the discard neither fails the boot nor poisons the tree.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
|
|
|||
|
|
@ -64,6 +64,8 @@ kind: "package-reference"
|
|||
|
||||
日志领先,缓存跟随:实时检查点先把会话的缓冲事件持久化,然后才保存缓存记录。因此崩溃可能让缓存落后于日志,但绝不会让缓存领先。读取和写入共享存储域内一致的内存状态;逐单元写入链只在持久化成功后修改内存。每个带版本戳的记录必须匹配实时单元 schema 与完整生命周期身份(`createdAt`、`cwd`、`isSeeded` 和 `inheritedEventCount`),因此在一个 fork 切点下初始化的行不能播种另一个切点。JSON 后端把每条记录存于仅所有者可访问的 `<root>/session_projcache/sessions/<id>.json` 目录树中。
|
||||
|
||||
升级绝不拖垮启动或列表:版本戳落在 spec `compatibleVersions` 集合内的记录保持可读(缺失的 lineage 字段解码为 unseeded lineage——对非 fork 会话精确无误,seeded 调用方则通不过身份比对、回落冷折叠),而仍然通不过 schema 校验的存量记录会按域的 `invalidRecords: 'backup-and-skip'` 策略移出为 `<id>.json.bak.<时间戳>`、连同原因写入日志,并由下一次检查点重建。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
|
|
@ -126,6 +128,7 @@ kind: "package-reference"
|
|||
- **无淘汰或保留接口**——记录按会话持续累积;清理已存储检查点属于带外维护,与会话持久化采用相同策略。
|
||||
- **间隔节流采用按会话的粗粒度控制**——一次无脏数据的写入完成后,计时器在首个脏事件到达时启动;持续但低于条数阈值的事件流每间隔写入一次,而非滑动窗口。
|
||||
- **缓存侧不做冷重折叠**——缓存只服务并刷新自己的记录,从不读取会话日志,因为它不依赖持久化层;需要保证冷快照的消费方自行从日志重新折叠。
|
||||
- **每次 schema 或域版本变更都必须论证升级路径**——改动存储记录 schema 或域版本时,同一 PR 必须在 `tests/fixtures/` 下归档此前已发布的磁盘格式样本,并在 `tests/fixtures.spec.ts` 中用测试论证所选的处置方式:读兼容恢复(`compatibleVersions`)、当前版本重写,或 backup-and-skip 抢救。即便选择直接丢弃旧记录的 bump,也要证明丢弃既不炸启动、也不污染缓存树。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
|
|
|||
|
|
@ -367,12 +367,18 @@ function identityOf(
|
|||
}
|
||||
}
|
||||
|
||||
/** Whether a stored record's bound identity names the caller's lifecycle. */
|
||||
/**
|
||||
* Whether a stored record's bound identity names the caller's lifecycle.
|
||||
* Absent lineage fields (records admitted via `compatibleVersions` predate
|
||||
* them) read as the unseeded lineage: exact for an unseeded caller, and a
|
||||
* seeded caller's expectation then fails the match, discarding the record to
|
||||
* a cold rebuild.
|
||||
*/
|
||||
function identityMatches(stored: CheckpointIdentity, expected: CheckpointIdentity): boolean {
|
||||
return stored.createdAt === expected.createdAt
|
||||
&& stored.cwd === expected.cwd
|
||||
&& stored.isSeeded === expected.isSeeded
|
||||
&& stored.inheritedEventCount === expected.inheritedEventCount
|
||||
&& (stored.isSeeded ?? false) === expected.isSeeded
|
||||
&& (stored.inheritedEventCount ?? 0) === expected.inheritedEventCount
|
||||
}
|
||||
|
||||
export default SessionProjectionCache
|
||||
|
|
|
|||
|
|
@ -38,12 +38,19 @@ export const checkpointRow = z.object({
|
|||
* old record pass every watermark check and seed state folded from an
|
||||
* unrelated log. Reads validate this against the live header (listing) or
|
||||
* the stored header (cold read) before accepting any record.
|
||||
*
|
||||
* The lineage fields are optional because records admitted through
|
||||
* `compatibleVersions` predate them. The reader (`identityMatches`)
|
||||
* interprets their absence as the unseeded lineage — exact for an unseeded
|
||||
* session, while a seeded expectation fails the match and the record is
|
||||
* discarded to a cold rebuild. Current-version writes always store both
|
||||
* fields.
|
||||
*/
|
||||
export const checkpointIdentity = z.object({
|
||||
createdAt: z.number().int().nonnegative(),
|
||||
cwd: z.string().optional(),
|
||||
isSeeded: z.boolean(),
|
||||
inheritedEventCount: z.number().int().nonnegative().transform(SessionLogOffset),
|
||||
isSeeded: z.boolean().optional(),
|
||||
inheritedEventCount: z.number().int().nonnegative().transform(SessionLogOffset).optional(),
|
||||
})
|
||||
|
||||
/** The identity fields a record is bound to, inferred from {@link checkpointIdentity}. */
|
||||
|
|
@ -68,11 +75,27 @@ export type CheckpointRecord = z.infer<typeof checkpointRecord>
|
|||
* bumps per session: after a bump, a stale session document is discarded on
|
||||
* open (cache semantics — a stale or unreadable cache costs a longer tail
|
||||
* replay, never a wrong value) while the rest of the domain stays usable,
|
||||
* instead of rejecting the whole medium.
|
||||
* instead of rejecting the whole medium. The `compatibleVersions` entries
|
||||
* are declared because those records differ from the current version only
|
||||
* by the absent optional lineage fields, so upgraded homes keep serving
|
||||
* their cached listing projections instead of dropping every title until
|
||||
* each session is reopened; the per-record version map lives in the
|
||||
* read-compat Agent Note
|
||||
* (.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md).
|
||||
* The per-row `ver` guard and the identity match still discard anything the
|
||||
* current fold semantics cannot vouch for.
|
||||
*
|
||||
* `invalidRecords: 'backup-and-skip'`: a stored record that fails the schema
|
||||
* anyway is disposable derived data, so it must never cost the boot — the
|
||||
* domain layer moves the document aside as `<key>.json.bak.<stamp>`, logs
|
||||
* the concrete validation failure, and serves the session as uncached (a
|
||||
* cold read rebuilds and rewrites it).
|
||||
*/
|
||||
export const projectionCacheDomainSpec = defineDomain({
|
||||
name: 'session_projcache',
|
||||
version: 5,
|
||||
compatibleVersions: [3, 4],
|
||||
invalidRecords: 'backup-and-skip',
|
||||
layout: 'per-record',
|
||||
tables: { sessions: domainTable<SessionId, CheckpointRecord>(checkpointRecord) },
|
||||
})
|
||||
|
|
|
|||
|
|
@ -42,9 +42,11 @@ declare module '@deepseek-ai/dsh-session-projection/types' {
|
|||
'cache-test/marks2': Map<string, string>
|
||||
'cache-test/count': number
|
||||
'cache-test/secret': string
|
||||
'cache-test/marks3': MarksState
|
||||
}
|
||||
interface SessionProjectionMap {
|
||||
'cache-test/marks': { marks: string[] }
|
||||
'cache-test/marks3': { marks: string[] }
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -71,6 +73,18 @@ const marksUnit = (stateVersion = 1) => ({
|
|||
stateVersion,
|
||||
}) satisfies ProjectionDefinition<'cache-test/marks', MarksState>
|
||||
|
||||
const marks3Unit = {
|
||||
key: 'cache-test/marks3',
|
||||
stateSchema: z.object({ marks: z.array(z.string()) }).nullable(),
|
||||
init: () => null,
|
||||
apply: state => state,
|
||||
wire: {
|
||||
viewSchema: z.object({ marks: z.array(z.string()) }),
|
||||
view: state => state ?? { marks: [] },
|
||||
},
|
||||
stateVersion: 1,
|
||||
} satisfies ProjectionDefinition<'cache-test/marks3', MarksState>
|
||||
|
||||
const secretUnit = {
|
||||
key: 'cache-test/secret',
|
||||
stateSchema: z.string(),
|
||||
|
|
@ -376,6 +390,26 @@ describe('SessionProjectionCache listing read', () => {
|
|||
.toBeUndefined()
|
||||
})
|
||||
|
||||
it('carries ONE cut across multiple served rows: the lowest watermark wins', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-'))
|
||||
roots.push(root)
|
||||
// Equal watermarks: whichever row is visited second cannot lower the cut,
|
||||
// so the one-cut fold sees both a lowering and a non-lowering row in
|
||||
// every iteration order.
|
||||
await seedRecord(root, 'multi-row', {
|
||||
'cache-test/marks': { ver: 1, seq: SessionSeq(4), val: { marks: ['a'] } },
|
||||
'cache-test/marks3': { ver: 1, seq: SessionSeq(4), val: { marks: ['b'] } },
|
||||
})
|
||||
const { ctx, cache } = await harness({ root })
|
||||
ctx.sessionProjections.register(marks3Unit)
|
||||
const block = cache.cachedSnapshot(headerOf(SessionId('multi-row')), SessionLogOffset(0))
|
||||
expect(block?.values).toEqual({
|
||||
'cache-test/marks': { marks: ['a'] },
|
||||
'cache-test/marks3': { marks: ['b'] },
|
||||
})
|
||||
expect(block?.asOfSeq).toBe(4)
|
||||
})
|
||||
|
||||
it('returns undefined when the stored record is version-mismatched', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-'))
|
||||
roots.push(root)
|
||||
|
|
@ -394,6 +428,30 @@ describe('SessionProjectionCache listing read', () => {
|
|||
.toBeUndefined()
|
||||
})
|
||||
|
||||
it('serves a pre-lineage record (accepted old version) to an unseeded caller only', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-'))
|
||||
roots.push(root)
|
||||
// A document stamped with an accepted older version whose identity
|
||||
// predates the lineage fields: absent lineage reads as unseeded.
|
||||
const path = recordPath(root, SessionId('pre-lineage'))
|
||||
await mkdir(dirname(path), { recursive: true })
|
||||
await writeFile(path, JSON.stringify({
|
||||
version: 4,
|
||||
record: {
|
||||
identity: { createdAt: 0 },
|
||||
rows: { 'cache-test/marks': { ver: 1, seq: 4, val: { marks: ['kept'] } } },
|
||||
},
|
||||
}))
|
||||
const { cache } = await harness({ root })
|
||||
const id = SessionId('pre-lineage')
|
||||
// Unseeded caller: the absent lineage is exactly its identity — served.
|
||||
expect(cache.cachedSnapshot(headerOf(id), SessionLogOffset(0)))
|
||||
.toEqual({ asOfSeq: 4, values: { 'cache-test/marks': { marks: ['kept'] } } })
|
||||
// Seeded caller: the lineage-less record cannot vouch for the cut — refused.
|
||||
expect(cache.cachedSnapshot({ ...headerOf(id), isSeeded: true }, SessionLogOffset(2)))
|
||||
.toBeUndefined()
|
||||
})
|
||||
|
||||
it('returns undefined when every stored row is version-mismatched', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-'))
|
||||
roots.push(root)
|
||||
|
|
|
|||
242
packages/session/session-projection-cache/tests/fixtures.spec.ts
Normal file
242
packages/session/session-projection-cache/tests/fixtures.spec.ts
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
/**
|
||||
* Cross-version recovery over archived on-disk artifacts. `fixtures/` holds
|
||||
* real `session_projcache` media, each produced by driving the named release
|
||||
* through its own web app (session created over RPC, real model turns, a
|
||||
* rename): the v3 whole-unit file (published 0.1.1-rc.2), a v4 per-record
|
||||
* document (published 0.1.2-alpha.3), a current v5 document, and the
|
||||
* v5-stamped lineage-less document reproducing byte-for-byte what the
|
||||
* formerly unguarded legacy bootstrap wrote over v3 records. Each must
|
||||
* recover through the real storage stack — the domain opens and the listing
|
||||
* read serves the archived title — and a record that fails schema validation
|
||||
* anyway is backed up and skipped instead of failing the boot.
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { cp, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { z } from 'zod'
|
||||
import SessionStore, { SessionId, SessionLogOffset } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import SessionProjectionRegistry from '@deepseek-ai/dsh-session-projection'
|
||||
import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection'
|
||||
import Storage from '@deepseek-ai/dsh-storage'
|
||||
import {
|
||||
apply as storageJsonApply, Config as storageJsonConfig, inject as storageJsonInject, name as storageJsonName,
|
||||
} from '@deepseek-ai/dsh-storage-json'
|
||||
import {
|
||||
apply as storageDomainApply, Config as storageDomainConfig, inject as storageDomainInject, name as storageDomainName,
|
||||
} from '@deepseek-ai/dsh-storage-domain'
|
||||
import SessionProjectionCache from '../src/index.ts'
|
||||
import { projectionCacheDomainSpec } from '../src/spec.ts'
|
||||
|
||||
// Declarations must match the shipped title unit's exactly (the repo-wide
|
||||
// compile face sees both).
|
||||
declare module '@deepseek-ai/dsh-session-projection/types' {
|
||||
interface SessionProjectionStateMap {
|
||||
title: string | null
|
||||
}
|
||||
interface SessionProjectionMap {
|
||||
title: string | null
|
||||
}
|
||||
}
|
||||
|
||||
declare module '@deepseek-ai/dsh-session/types' {
|
||||
interface SessionEventMap {
|
||||
'fixtures-test/set-title': { title: string }
|
||||
}
|
||||
|
||||
interface OutOfBandSessionEventMap {
|
||||
'fixtures-test/set-title': true
|
||||
}
|
||||
}
|
||||
|
||||
// Mirrors the shipped title unit's storage face: stateVersion 1, bare-string
|
||||
// state (the fixture rows carry exactly this shape in every archived
|
||||
// version), folding a test event so the rewrite path has fresh data.
|
||||
const titleUnit = {
|
||||
key: 'title',
|
||||
stateSchema: z.string().nullable(),
|
||||
init: () => null,
|
||||
apply: (state, event) => (event.type === 'fixtures-test/set-title' ? event.data.title : state),
|
||||
wire: { viewSchema: z.string().nullable(), view: state => state },
|
||||
stateVersion: 1,
|
||||
} satisfies ProjectionDefinition<'title', string | null>
|
||||
|
||||
const FIXTURES = fileURLToPath(new URL('./fixtures/', import.meta.url))
|
||||
|
||||
/** One archived per-record document (`{version, record}`). */
|
||||
interface FixtureDoc {
|
||||
version: number
|
||||
record: {
|
||||
identity: { createdAt: number; cwd?: string }
|
||||
rows: Record<string, { ver: number; seq: number; val: unknown }>
|
||||
}
|
||||
}
|
||||
|
||||
async function fixtureJson<T>(name: string): Promise<T> {
|
||||
return JSON.parse(await readFile(join(FIXTURES, name), 'utf8')) as T
|
||||
}
|
||||
|
||||
/** Header for the session a fixture record is bound to (identity witness). */
|
||||
function headerFor(id: SessionId, identity: FixtureDoc['record']['identity']): SessionHeader {
|
||||
return {
|
||||
version: 0,
|
||||
id,
|
||||
createdAt: identity.createdAt,
|
||||
isSeeded: false,
|
||||
...identity.cwd === undefined ? {} : { cwd: identity.cwd },
|
||||
}
|
||||
}
|
||||
|
||||
const contexts: Context[] = []
|
||||
const roots: string[] = []
|
||||
|
||||
async function harness(root: string) {
|
||||
roots.push(root)
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(Storage)
|
||||
await ctx.plugin({ name: storageJsonName, inject: storageJsonInject, apply: storageJsonApply, Config: storageJsonConfig }, { root })
|
||||
await ctx.plugin({ name: storageDomainName, inject: storageDomainInject, apply: storageDomainApply, Config: storageDomainConfig }, { backend: 'json' })
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
ctx.sessionProjections.register(titleUnit)
|
||||
await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 })
|
||||
return { ctx, cache: ctx.sessionProjectionCache }
|
||||
}
|
||||
|
||||
/** Lay one per-record fixture document into a fresh backend root. */
|
||||
async function placeDoc(root: string, id: string, name: string): Promise<FixtureDoc> {
|
||||
const path = join(root, projectionCacheDomainSpec.name, 'sessions', `${id}.json`)
|
||||
await mkdir(dirname(path), { recursive: true })
|
||||
await cp(join(FIXTURES, name), path)
|
||||
return fixtureJson<FixtureDoc>(name)
|
||||
}
|
||||
|
||||
/**
|
||||
* Drive a live write over a recovered session id and assert the archived
|
||||
* document is replaced by a current-version one: v5 stamp, lineage present,
|
||||
* and the freshly folded title — the write path never keeps the old format.
|
||||
*/
|
||||
async function assertRewrite(ctx: Context, root: string, id: SessionId): Promise<void> {
|
||||
const session = ctx.sessions.create(id)
|
||||
session.append('fixtures-test/set-title', { title: '重写标题' })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
const path = join(root, projectionCacheDomainSpec.name, 'sessions', `${id}.json`)
|
||||
await vi.waitFor(async () => {
|
||||
const doc = JSON.parse(await readFile(path, 'utf8')) as FixtureDoc
|
||||
expect(doc.version).toBe(projectionCacheDomainSpec.version)
|
||||
expect(doc.record.identity).toMatchObject({ isSeeded: false, inheritedEventCount: 0 })
|
||||
expect(doc.record.rows['title']?.val).toBe('重写标题')
|
||||
}, { timeout: 5_000 })
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(contexts.splice(0).map(ctx => ctx.fiber.dispose()))
|
||||
await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 })))
|
||||
})
|
||||
|
||||
describe('archived version recovery', () => {
|
||||
it('recovers the v3 whole-unit archive through the legacy bootstrap', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-'))
|
||||
await cp(join(FIXTURES, 'v3-single-unit.json'), join(root, `${projectionCacheDomainSpec.name}.json`))
|
||||
type SingleUnit = {
|
||||
unit: { version: number }
|
||||
tables: { sessions: Record<string, FixtureDoc['record']> }
|
||||
}
|
||||
const archive = await fixtureJson<SingleUnit>('v3-single-unit.json')
|
||||
expect(archive.unit.version).toBe(3) // the fixture IS the old format
|
||||
const [sid, record] = Object.entries(archive.tables.sessions)[0]!
|
||||
|
||||
const { ctx, cache } = await harness(root)
|
||||
const snapshot = cache.cachedSnapshot(headerFor(SessionId(sid), record.identity), SessionLogOffset(0), ['title'])
|
||||
expect(snapshot?.values.title).toBe(record.rows['title']!.val)
|
||||
|
||||
// The one-time bootstrap materialized a current-version document.
|
||||
const migrated = JSON.parse(
|
||||
await readFile(join(root, projectionCacheDomainSpec.name, 'sessions', `${sid}.json`), 'utf8'),
|
||||
) as { version: number }
|
||||
expect(migrated.version).toBe(projectionCacheDomainSpec.version)
|
||||
|
||||
await assertRewrite(ctx, root, SessionId(sid))
|
||||
})
|
||||
|
||||
for (const [fixture, storedVersion] of [
|
||||
['v4-session-doc.json', 4],
|
||||
['v5-session-doc.json', 5],
|
||||
['v5-lineageless-doc.json', 5],
|
||||
] as const) {
|
||||
it(`serves the archived title from ${fixture}, then rewrites it current`, async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-'))
|
||||
const id = SessionId('fixture-session')
|
||||
const doc = await placeDoc(root, id, fixture)
|
||||
expect(doc.version).toBe(storedVersion)
|
||||
|
||||
const { ctx, cache } = await harness(root)
|
||||
const snapshot = cache.cachedSnapshot(headerFor(id, doc.record.identity), SessionLogOffset(0), ['title'])
|
||||
expect(snapshot?.values.title).toBe(doc.record.rows['title']!.val)
|
||||
|
||||
await assertRewrite(ctx, root, id)
|
||||
})
|
||||
}
|
||||
|
||||
it('refuses a lineage-less archive for a seeded caller (identity mismatch, cold rebuild)', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-'))
|
||||
const id = SessionId('fixture-seeded')
|
||||
const doc = await placeDoc(root, id, 'v5-lineageless-doc.json')
|
||||
|
||||
const { cache } = await harness(root)
|
||||
const seeded = { ...headerFor(id, doc.record.identity), isSeeded: true }
|
||||
expect(cache.cachedSnapshot(seeded, SessionLogOffset(2), ['title'])).toBeUndefined()
|
||||
})
|
||||
|
||||
it('backs up and skips a record that fails schema validation instead of failing the boot', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-'))
|
||||
roots.push(root)
|
||||
const sessionsDir = join(root, projectionCacheDomainSpec.name, 'sessions')
|
||||
await mkdir(sessionsDir, { recursive: true })
|
||||
// Current-version stamp, hopeless record content: no compat rung can save it.
|
||||
await writeFile(join(sessionsDir, 'broken.json'), JSON.stringify({
|
||||
version: projectionCacheDomainSpec.version,
|
||||
record: { identity: { createdAt: 'not-a-number' }, rows: 'not-an-object' },
|
||||
}))
|
||||
const good = await placeDoc(root, SessionId('survivor'), 'v5-session-doc.json')
|
||||
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(Storage)
|
||||
await ctx.plugin({ name: storageJsonName, inject: storageJsonInject, apply: storageJsonApply, Config: storageJsonConfig }, { root })
|
||||
await ctx.plugin({ name: storageDomainName, inject: storageDomainInject, apply: storageDomainApply, Config: storageDomainConfig }, { backend: 'json' })
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
ctx.sessionProjections.register(titleUnit)
|
||||
const error = vi.spyOn(ctx.logger, 'error').mockImplementation(() => {})
|
||||
// The boot survives the broken record — this line rejecting IS the fixed bug.
|
||||
await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 })
|
||||
|
||||
// Concrete console diagnostics: which record, where it went, and why.
|
||||
expect(error).toHaveBeenCalledWith(expect.stringContaining("record 'broken'"))
|
||||
expect(error).toHaveBeenCalledWith(expect.stringContaining('.json.bak.'))
|
||||
|
||||
// The document was moved aside as <key>.json.bak.<YYYYMMDDHHmm>, bytes intact.
|
||||
const entries = await readdir(sessionsDir)
|
||||
expect(entries).not.toContain('broken.json')
|
||||
const backup = entries.find(name => /^broken\.json\.bak\.\d{12}$/.test(name))
|
||||
expect(backup).toBeDefined()
|
||||
expect(JSON.parse(await readFile(join(sessionsDir, backup!), 'utf8')))
|
||||
.toMatchObject({ record: { rows: 'not-an-object' } })
|
||||
|
||||
// The broken record reads as absent; its neighbors still serve.
|
||||
const cache = ctx.sessionProjectionCache
|
||||
expect(cache.cachedSnapshot(headerFor(SessionId('broken'), { createdAt: 0 }), SessionLogOffset(0)))
|
||||
.toBeUndefined()
|
||||
expect(cache.cachedSnapshot(
|
||||
headerFor(SessionId('survivor'), good.record.identity),
|
||||
SessionLogOffset(0),
|
||||
['title'],
|
||||
)?.values.title).toBe(good.record.rows['title']!.val)
|
||||
})
|
||||
})
|
||||
136
packages/session/session-projection-cache/tests/fixtures/v3-single-unit.json
vendored
Normal file
136
packages/session/session-projection-cache/tests/fixtures/v3-single-unit.json
vendored
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
{
|
||||
"unit": {
|
||||
"name": "session_projcache",
|
||||
"version": 3
|
||||
},
|
||||
"global": null,
|
||||
"tables": {
|
||||
"sessions": {
|
||||
"session-1374fa81-15da-44ca-be12-b4c6fe8076a3": {
|
||||
"identity": {
|
||||
"createdAt": 1788286864454,
|
||||
"cwd": "/tmp"
|
||||
},
|
||||
"rows": {
|
||||
"sessionStats": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"turns": 2,
|
||||
"steps": 2,
|
||||
"llmMs": 2169,
|
||||
"toolMs": 0,
|
||||
"ttftMs": 1872,
|
||||
"ttftSteps": 2,
|
||||
"decodeMs": 297,
|
||||
"decodeTokens": 20,
|
||||
"lastTurn": 2,
|
||||
"openStep": null,
|
||||
"pendingCalls": {}
|
||||
}
|
||||
},
|
||||
"title": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": "验收标题-rc2"
|
||||
},
|
||||
"goal": {
|
||||
"ver": 4,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"tokenUsage": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"totals": {
|
||||
"uncachedInputTokens": 6083,
|
||||
"outputTokens": 20,
|
||||
"cacheReadTokens": 9728,
|
||||
"cacheWriteTokens": 0
|
||||
},
|
||||
"last": {
|
||||
"turn": 2,
|
||||
"step": 1,
|
||||
"buckets": {
|
||||
"uncachedInputTokens": 112,
|
||||
"outputTokens": 4,
|
||||
"cacheReadTokens": 7808,
|
||||
"cacheWriteTokens": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"contextPressure": {
|
||||
"ver": 4,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"surfaceTokens": 178,
|
||||
"contextWindow": 1000000,
|
||||
"pressureTokens": 7920,
|
||||
"sampledSurfaceTokens": 168
|
||||
}
|
||||
},
|
||||
"contextBreakdown": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"systemTokens": 1620,
|
||||
"toolsTokens": 6475,
|
||||
"messageTokens": 178
|
||||
}
|
||||
},
|
||||
"subagentTiming": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"descriptorSeen": false,
|
||||
"settledMs": 0
|
||||
}
|
||||
},
|
||||
"subagent": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {}
|
||||
},
|
||||
"permissions": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"preset": "workspace-write",
|
||||
"sandbox": "workspace-write",
|
||||
"approval": "ask"
|
||||
}
|
||||
},
|
||||
"sessionListMetadata": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"blank": false,
|
||||
"lastPromptAt": 1788286867410
|
||||
}
|
||||
},
|
||||
"imageLimits": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"todos": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"plan": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"active": false,
|
||||
"wanted": null,
|
||||
"running": null
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
212
packages/session/session-projection-cache/tests/fixtures/v4-session-doc.json
vendored
Normal file
212
packages/session/session-projection-cache/tests/fixtures/v4-session-doc.json
vendored
Normal file
|
|
@ -0,0 +1,212 @@
|
|||
{
|
||||
"version": 4,
|
||||
"record": {
|
||||
"identity": {
|
||||
"createdAt": 1788286912530,
|
||||
"cwd": "/tmp"
|
||||
},
|
||||
"rows": {
|
||||
"title": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": "验收标题-alpha3"
|
||||
},
|
||||
"titleInput": {
|
||||
"ver": 3,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"first": {
|
||||
"seq": 7,
|
||||
"text": "请只回复一个词:pong"
|
||||
},
|
||||
"count": 2,
|
||||
"lastSeq": 58
|
||||
}
|
||||
},
|
||||
"llmRetry": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": {}
|
||||
},
|
||||
"sandboxMode": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": "workspace-write"
|
||||
},
|
||||
"goal": {
|
||||
"ver": 6,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"current": null,
|
||||
"seenGoalIds": [],
|
||||
"failure": null
|
||||
}
|
||||
},
|
||||
"tokenUsage": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"totals": {
|
||||
"uncachedInputTokens": 8192,
|
||||
"outputTokens": 35,
|
||||
"cacheReadTokens": 8064,
|
||||
"cacheWriteTokens": 0
|
||||
},
|
||||
"last": {
|
||||
"turn": 2,
|
||||
"step": 1,
|
||||
"buckets": {
|
||||
"uncachedInputTokens": 86,
|
||||
"outputTokens": 4,
|
||||
"cacheReadTokens": 8064,
|
||||
"cacheWriteTokens": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"contextPressure": {
|
||||
"ver": 4,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"surfaceTokens": 193,
|
||||
"contextWindow": 1000000,
|
||||
"pressureTokens": 8150,
|
||||
"sampledSurfaceTokens": 183
|
||||
}
|
||||
},
|
||||
"contextBreakdown": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"systemTokens": 1760,
|
||||
"toolsTokens": 6579,
|
||||
"messageTokens": 193
|
||||
}
|
||||
},
|
||||
"turnBoundary": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"openTurnStartSeq": null,
|
||||
"lastStepStartSeq": 57,
|
||||
"lastStepBoundary": {
|
||||
"kind": "end",
|
||||
"seq": 67
|
||||
},
|
||||
"lastTurn": 2
|
||||
}
|
||||
},
|
||||
"sessionStats": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"turns": 2,
|
||||
"steps": 2,
|
||||
"llmMs": 3486,
|
||||
"toolMs": 0,
|
||||
"ttftMs": 3199,
|
||||
"ttftSteps": 2,
|
||||
"decodeMs": 287,
|
||||
"decodeTokens": 35,
|
||||
"lastTurn": 2,
|
||||
"openStep": null,
|
||||
"pendingCalls": {}
|
||||
}
|
||||
},
|
||||
"turnOutline": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"turns": [
|
||||
{
|
||||
"turn": 1,
|
||||
"seq": 4,
|
||||
"prompt": "请只回复一个词:pong",
|
||||
"response": "pong"
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"seq": 55,
|
||||
"prompt": "请只回复一个词:pong2",
|
||||
"response": "pong2"
|
||||
}
|
||||
],
|
||||
"draft": ""
|
||||
}
|
||||
},
|
||||
"agentPreset": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": "standard"
|
||||
},
|
||||
"subagentTiming": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"descriptorSeen": false,
|
||||
"settledMs": 0
|
||||
}
|
||||
},
|
||||
"subagent": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {}
|
||||
},
|
||||
"permissions": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"preset": "workspace-write",
|
||||
"sandbox": "workspace-write",
|
||||
"approval": "ask",
|
||||
"seeded": false
|
||||
}
|
||||
},
|
||||
"modelSelection": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"lastUsed": {
|
||||
"provider": "deepseek-official",
|
||||
"model": "deepseek-v4-flash",
|
||||
"reasoningEffort": "high"
|
||||
},
|
||||
"pending": null
|
||||
}
|
||||
},
|
||||
"sessionListMetadata": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"blank": false,
|
||||
"lastPromptAt": 1788286917834
|
||||
}
|
||||
},
|
||||
"imageLimits": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": null
|
||||
},
|
||||
"todos": {
|
||||
"ver": 2,
|
||||
"seq": 68,
|
||||
"val": null
|
||||
},
|
||||
"plan": {
|
||||
"ver": 3,
|
||||
"seq": 68,
|
||||
"val": {
|
||||
"active": false,
|
||||
"wanted": null,
|
||||
"running": null,
|
||||
"activeAtLastHeader": false
|
||||
}
|
||||
},
|
||||
"subagentModelSelectionPolicy": {
|
||||
"ver": 1,
|
||||
"seq": 68,
|
||||
"val": null
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
128
packages/session/session-projection-cache/tests/fixtures/v5-lineageless-doc.json
vendored
Normal file
128
packages/session/session-projection-cache/tests/fixtures/v5-lineageless-doc.json
vendored
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
{
|
||||
"version": 5,
|
||||
"record": {
|
||||
"identity": {
|
||||
"createdAt": 1788286864454,
|
||||
"cwd": "/tmp"
|
||||
},
|
||||
"rows": {
|
||||
"sessionStats": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"turns": 2,
|
||||
"steps": 2,
|
||||
"llmMs": 2169,
|
||||
"toolMs": 0,
|
||||
"ttftMs": 1872,
|
||||
"ttftSteps": 2,
|
||||
"decodeMs": 297,
|
||||
"decodeTokens": 20,
|
||||
"lastTurn": 2,
|
||||
"openStep": null,
|
||||
"pendingCalls": {}
|
||||
}
|
||||
},
|
||||
"title": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": "\u9a8c\u6536\u6807\u9898-rc2"
|
||||
},
|
||||
"goal": {
|
||||
"ver": 4,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"tokenUsage": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"totals": {
|
||||
"uncachedInputTokens": 6083,
|
||||
"outputTokens": 20,
|
||||
"cacheReadTokens": 9728,
|
||||
"cacheWriteTokens": 0
|
||||
},
|
||||
"last": {
|
||||
"turn": 2,
|
||||
"step": 1,
|
||||
"buckets": {
|
||||
"uncachedInputTokens": 112,
|
||||
"outputTokens": 4,
|
||||
"cacheReadTokens": 7808,
|
||||
"cacheWriteTokens": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"contextPressure": {
|
||||
"ver": 4,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"surfaceTokens": 178,
|
||||
"contextWindow": 1000000,
|
||||
"pressureTokens": 7920,
|
||||
"sampledSurfaceTokens": 168
|
||||
}
|
||||
},
|
||||
"contextBreakdown": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"systemTokens": 1620,
|
||||
"toolsTokens": 6475,
|
||||
"messageTokens": 178
|
||||
}
|
||||
},
|
||||
"subagentTiming": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"descriptorSeen": false,
|
||||
"settledMs": 0
|
||||
}
|
||||
},
|
||||
"subagent": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {}
|
||||
},
|
||||
"permissions": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"preset": "workspace-write",
|
||||
"sandbox": "workspace-write",
|
||||
"approval": "ask"
|
||||
}
|
||||
},
|
||||
"sessionListMetadata": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"blank": false,
|
||||
"lastPromptAt": 1788286867410
|
||||
}
|
||||
},
|
||||
"imageLimits": {
|
||||
"ver": 1,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"todos": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": null
|
||||
},
|
||||
"plan": {
|
||||
"ver": 2,
|
||||
"seq": 54,
|
||||
"val": {
|
||||
"active": false,
|
||||
"wanted": null,
|
||||
"running": null
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
220
packages/session/session-projection-cache/tests/fixtures/v5-session-doc.json
vendored
Normal file
220
packages/session/session-projection-cache/tests/fixtures/v5-session-doc.json
vendored
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
{
|
||||
"version": 5,
|
||||
"record": {
|
||||
"identity": {
|
||||
"createdAt": 1788286864454,
|
||||
"cwd": "/tmp",
|
||||
"isSeeded": false,
|
||||
"inheritedEventCount": 0
|
||||
},
|
||||
"rows": {
|
||||
"title": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": "验收标题-rc2"
|
||||
},
|
||||
"titleInput": {
|
||||
"ver": 3,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"first": {
|
||||
"seq": 7,
|
||||
"text": "请只回复一个词:pong"
|
||||
},
|
||||
"count": 3,
|
||||
"lastSeq": 60
|
||||
}
|
||||
},
|
||||
"llmRetry": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": {}
|
||||
},
|
||||
"sandboxMode": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": "workspace-write"
|
||||
},
|
||||
"goal": {
|
||||
"ver": 6,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"current": null,
|
||||
"seenGoalIds": [],
|
||||
"failure": null
|
||||
}
|
||||
},
|
||||
"tokenUsage": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"totals": {
|
||||
"uncachedInputTokens": 14219,
|
||||
"outputTokens": 24,
|
||||
"cacheReadTokens": 9728,
|
||||
"cacheWriteTokens": 0
|
||||
},
|
||||
"last": {
|
||||
"turn": 3,
|
||||
"step": 1,
|
||||
"buckets": {
|
||||
"uncachedInputTokens": 8136,
|
||||
"outputTokens": 4,
|
||||
"cacheReadTokens": 0,
|
||||
"cacheWriteTokens": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"contextPressure": {
|
||||
"ver": 4,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"surfaceTokens": 200,
|
||||
"contextWindow": 1000000,
|
||||
"pressureTokens": 8136,
|
||||
"sampledSurfaceTokens": 190
|
||||
}
|
||||
},
|
||||
"contextBreakdown": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"systemTokens": 1729,
|
||||
"toolsTokens": 6611,
|
||||
"messageTokens": 200
|
||||
}
|
||||
},
|
||||
"sessionStats": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"turns": 3,
|
||||
"steps": 3,
|
||||
"llmMs": 4368,
|
||||
"toolMs": 0,
|
||||
"ttftMs": 4030,
|
||||
"ttftSteps": 3,
|
||||
"decodeMs": 338,
|
||||
"decodeTokens": 24,
|
||||
"lastTurn": 3,
|
||||
"openStep": null,
|
||||
"pendingCalls": {}
|
||||
}
|
||||
},
|
||||
"agentPreset": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": "standard"
|
||||
},
|
||||
"subagentTiming": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"descriptorSeen": false,
|
||||
"settledMs": 0
|
||||
}
|
||||
},
|
||||
"subagent": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {}
|
||||
},
|
||||
"turnBoundary": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"openTurnStartSeq": null,
|
||||
"lastStepStartSeq": 59,
|
||||
"lastStepBoundary": {
|
||||
"kind": "end",
|
||||
"seq": 70
|
||||
},
|
||||
"lastTurn": 3
|
||||
}
|
||||
},
|
||||
"turnOutline": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"turns": [
|
||||
{
|
||||
"turn": 1,
|
||||
"seq": 4,
|
||||
"prompt": "请只回复一个词:pong",
|
||||
"response": "pong"
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"seq": 40,
|
||||
"prompt": "请只回复一个词:pong2",
|
||||
"response": "pong2"
|
||||
},
|
||||
{
|
||||
"turn": 3,
|
||||
"seq": 57,
|
||||
"prompt": "请只回复一个词:pong3",
|
||||
"response": "pong3"
|
||||
}
|
||||
],
|
||||
"draft": ""
|
||||
}
|
||||
},
|
||||
"permissions": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"preset": "workspace-write",
|
||||
"sandbox": "workspace-write",
|
||||
"approval": "ask",
|
||||
"seeded": true
|
||||
}
|
||||
},
|
||||
"modelSelection": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"lastUsed": {
|
||||
"provider": "deepseek-official",
|
||||
"model": "deepseek-v4-flash",
|
||||
"reasoningEffort": "high"
|
||||
},
|
||||
"pending": null
|
||||
}
|
||||
},
|
||||
"sessionListMetadata": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"blank": false,
|
||||
"lastPromptAt": 1788287660106
|
||||
}
|
||||
},
|
||||
"imageLimits": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": null
|
||||
},
|
||||
"todos": {
|
||||
"ver": 2,
|
||||
"seq": 71,
|
||||
"val": null
|
||||
},
|
||||
"plan": {
|
||||
"ver": 3,
|
||||
"seq": 71,
|
||||
"val": {
|
||||
"active": false,
|
||||
"wanted": null,
|
||||
"running": null,
|
||||
"activeAtLastHeader": false
|
||||
}
|
||||
},
|
||||
"subagentModelSelectionPolicy": {
|
||||
"ver": 1,
|
||||
"seq": 71,
|
||||
"val": null
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -88,7 +88,10 @@ export class DomainFacility {
|
|||
* (`facet-unsupported`); open the unit projected from the spec (backend
|
||||
* `version-mismatch`/`malformed-medium` pass through); load and validate
|
||||
* every stored record against the spec's zod schemas (`invalid-record`
|
||||
* with the offending table and key); construct the domain.
|
||||
* with the offending table and key — unless the spec declares
|
||||
* `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in
|
||||
* which case the failing record is backed up, logged, and skipped);
|
||||
* construct the domain.
|
||||
*
|
||||
* Lifecycle: the CALLER owns the returned handle and closes it via
|
||||
* `Domain.close()` (typically as its own `ctx.effect` disposer) — the
|
||||
|
|
@ -118,7 +121,23 @@ export class DomainFacility {
|
|||
for (const [table, tableSpec] of Object.entries(spec.tables)) {
|
||||
const records = new Map<string, unknown>()
|
||||
for (const [key, raw] of Object.entries(snapshot.tables[table] ?? {})) {
|
||||
records.set(key, parseRecord(spec.name, table, key, () => tableSpec.valueSchema.parse(raw)))
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = parseRecord(spec.name, table, key, () => tableSpec.valueSchema.parse(raw))
|
||||
} catch (error) {
|
||||
// Backup-and-skip policy (disposable derived data): move the record's
|
||||
// document aside, log the concrete failure, and open without the
|
||||
// record. Backends that cannot move a document keep the loud path.
|
||||
if (spec.invalidRecords !== 'backup-and-skip' || unit.backupRecord === undefined) throw error
|
||||
const moved = await unit.backupRecord(table, key)
|
||||
// parseRecord always wraps the zod failure as the cause.
|
||||
this.ctx.logger.error(
|
||||
`domain '${spec.name}': stored record '${key}' in table '${table}' failed schema validation; `
|
||||
+ `moved to '${moved}' and treated as absent. Cause: ${String((error as DomainError).cause)}`,
|
||||
)
|
||||
continue
|
||||
}
|
||||
records.set(key, parsed)
|
||||
}
|
||||
tables.set(table, records)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -45,6 +45,26 @@ export interface DomainSpec {
|
|||
* (a stale record document is discarded, never migrated).
|
||||
*/
|
||||
readonly layout?: 'single' | 'per-record'
|
||||
/**
|
||||
* Older domain versions whose stored records the current record schemas
|
||||
* also accept (the declaring owner vouches for that, typically by
|
||||
* declaring the fields older records lack as optional). `per-record` backends
|
||||
* read documents stamped with a listed version instead of discarding them,
|
||||
* and accept a legacy whole-unit file so stamped for the one-time
|
||||
* bootstrap; writes always stamp {@link version}.
|
||||
*/
|
||||
readonly compatibleVersions?: readonly number[]
|
||||
/**
|
||||
* What `open` does with a stored table record that fails its zod schema.
|
||||
* Absent (the default), the whole open rejects with `invalid-record` —
|
||||
* right for authoritative data. `'backup-and-skip'` is for domains whose
|
||||
* records are disposable derived data: the backend moves the record's
|
||||
* document aside (`KvUnit.backupRecord`), the failure is logged with
|
||||
* its cause, and the open continues with the record absent. A backend
|
||||
* without `backupRecord` (no per-record document to move) falls back
|
||||
* to the rejecting default. The global slot always rejects.
|
||||
*/
|
||||
readonly invalidRecords?: 'backup-and-skip'
|
||||
/** Optional global singleton slot. */
|
||||
readonly global?: DomainGlobalSpec<unknown>
|
||||
/** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */
|
||||
|
|
@ -91,6 +111,13 @@ export function defineDomain<S extends DomainSpec>(spec: S): S {
|
|||
if (!Number.isInteger(spec.version) || spec.version < 0) {
|
||||
throw new Error(`domain '${spec.name}' version must be a non-negative integer, got ${spec.version}`)
|
||||
}
|
||||
for (const compat of spec.compatibleVersions ?? []) {
|
||||
if (!Number.isInteger(compat) || compat < 0 || compat >= spec.version) {
|
||||
throw new Error(
|
||||
`domain '${spec.name}' compatibleVersions entries must be non-negative integers below version ${spec.version}, got ${compat}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
if (spec.layout !== undefined) {
|
||||
// Runtime boundary: the union type is compile-time only — a spec built
|
||||
// from config could carry any value, and a bad one must fail loud here.
|
||||
|
|
@ -99,6 +126,12 @@ export function defineDomain<S extends DomainSpec>(spec: S): S {
|
|||
throw new Error(`domain '${spec.name}' layout must be 'single' or 'per-record', got ${layout}`)
|
||||
}
|
||||
}
|
||||
if (spec.invalidRecords !== undefined) {
|
||||
const policy: string = spec.invalidRecords
|
||||
if (policy !== 'backup-and-skip') {
|
||||
throw new Error(`domain '${spec.name}' invalidRecords must be 'backup-and-skip' when present, got ${policy}`)
|
||||
}
|
||||
}
|
||||
for (const table of Object.keys(spec.tables)) {
|
||||
if (!UNIT_NAME_RE.test(table)) {
|
||||
throw new Error(`domain '${spec.name}' table name '${table}' must match ${UNIT_NAME_RE}`)
|
||||
|
|
@ -125,5 +158,6 @@ export function descriptorOf(spec: DomainSpec): KvUnitDescriptor {
|
|||
tables: Object.keys(spec.tables),
|
||||
hasGlobal: spec.global !== undefined,
|
||||
...spec.layout === undefined ? {} : { layout: spec.layout },
|
||||
...spec.compatibleVersions === undefined ? {} : { compatibleVersions: spec.compatibleVersions },
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -58,6 +58,25 @@ describe('defineDomain', () => {
|
|||
})).toThrow(/must not accept null/)
|
||||
})
|
||||
|
||||
it('validates compatibleVersions entries and projects them onto the descriptor', () => {
|
||||
expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [1.5], tables: {} }))
|
||||
.toThrow(/compatibleVersions/)
|
||||
expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [2], tables: {} }))
|
||||
.toThrow(/below version/)
|
||||
expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [-1], tables: {} }))
|
||||
.toThrow(/compatibleVersions/)
|
||||
expect(descriptorOf(defineDomain({ name: 'ok', version: 2, compatibleVersions: [0, 1], tables: {} })))
|
||||
.toMatchObject({ compatibleVersions: [0, 1] })
|
||||
// An undeclared set is absent from the descriptor.
|
||||
expect(descriptorOf(spec)).not.toHaveProperty('compatibleVersions')
|
||||
})
|
||||
|
||||
it('rejects an unknown invalidRecords policy', () => {
|
||||
expect(() => defineDomain({
|
||||
name: 'ok', version: 1, invalidRecords: 'zap' as 'backup-and-skip', tables: {},
|
||||
})).toThrow(/invalidRecords/)
|
||||
})
|
||||
|
||||
it('rejects an invalid layout and projects the declared one onto the descriptor', () => {
|
||||
// A spec built from config can carry any value; the union type is
|
||||
// compile-time only, so the runtime boundary check must reject it.
|
||||
|
|
@ -139,6 +158,28 @@ describe('DomainFacility.open', () => {
|
|||
})
|
||||
})
|
||||
|
||||
it('keeps the rejecting default under backup-and-skip when the backend cannot move documents', async () => {
|
||||
// The memory backend has no backupRecord, so the declared policy cannot
|
||||
// apply and the open falls back to failing loud.
|
||||
const salvageSpec = defineDomain({
|
||||
name: 'salvage',
|
||||
version: 1,
|
||||
invalidRecords: 'backup-and-skip',
|
||||
tables: { items: domainTable<string, Item>(itemSchema) },
|
||||
})
|
||||
const pool = new MemoryMediaPool()
|
||||
{
|
||||
const { facility } = await harness({ pool })
|
||||
await (await facility.open(salvageSpec)).table('items').put('bad', { label: 'x', count: 2 })
|
||||
}
|
||||
pool.media.get('salvage')!.tables.get('items')!.set('bad', { label: 'x', count: 'NaN' })
|
||||
const { facility } = await harness({ pool })
|
||||
await expect(facility.open(salvageSpec)).rejects.toMatchObject({
|
||||
code: 'invalid-record',
|
||||
detail: { table: 'items', key: 'bad' },
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects a stored global that fails its schema with the global marker', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
pool.versions.set('demo', 1)
|
||||
|
|
|
|||
|
|
@ -100,16 +100,18 @@ export function serializeRecord(version: number, value: unknown): string {
|
|||
|
||||
/**
|
||||
* Parse one per-record document, validating its version stamp. A document
|
||||
* that is malformed or stamped with a different version is FOREIGN and reads
|
||||
* as absent — the per-record contract: one bad or stale record file must not
|
||||
* brick the whole unit, and a version bump discards stale records instead of
|
||||
* migrating them (the whole-unit format rejects instead, because there is
|
||||
* exactly one document).
|
||||
* that is malformed or stamped with an unaccepted version is FOREIGN and
|
||||
* reads as absent — the per-record contract: one bad or stale record file
|
||||
* must not brick the whole unit, and a version bump discards stale records
|
||||
* instead of migrating them (the whole-unit format rejects instead, because
|
||||
* there is exactly one document).
|
||||
* @param text - Raw per-record document content.
|
||||
* @param version - Expected unit version; a mismatch discards the document.
|
||||
* @param versions - Accepted unit versions (the current one plus the
|
||||
* descriptor's compatibleVersions); any other stamp discards the
|
||||
* document.
|
||||
* @returns the record value, or `undefined` for a foreign document.
|
||||
*/
|
||||
export function parseRecord(text: string, version: number): unknown {
|
||||
export function parseRecord(text: string, versions: readonly number[]): unknown {
|
||||
let document: unknown
|
||||
try {
|
||||
document = JSON.parse(text)
|
||||
|
|
@ -118,6 +120,6 @@ export function parseRecord(text: string, version: number): unknown {
|
|||
}
|
||||
if (typeof document !== 'object' || document === null) return undefined
|
||||
const { version: stamped, record } = document as Record<string, unknown>
|
||||
if (stamped !== version) return undefined
|
||||
if (typeof stamped !== 'number' || !versions.includes(stamped)) return undefined
|
||||
return record
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,20 +10,23 @@
|
|||
* memory unchanged.
|
||||
*
|
||||
* Per-record contract: a record document that is malformed or stamped with a
|
||||
* different version reads as an absent record — one bad or stale file never
|
||||
* bricks the whole unit, and a version bump discards stale records instead
|
||||
* of migrating them. Record keys become path segments, so they must be
|
||||
* path-safe (`[a-zA-Z0-9_-]+`); an unsafe key rejects at write.
|
||||
* version outside the accepted set (the descriptor's current version plus
|
||||
* its `compatibleVersions`) reads as an absent record — one bad or stale
|
||||
* file never bricks the whole unit, and a version bump discards stale
|
||||
* records instead of migrating them. Record keys become path segments, so
|
||||
* they must be path-safe (`[a-zA-Z0-9_-]+`); an unsafe key rejects at write.
|
||||
*
|
||||
* Legacy bootstrap: when the new tree has no document path, a legacy
|
||||
* whole-unit file `<root>/<name>.json` (the pre-per-record layout) seeds
|
||||
* per-record documents. Any new document path, including one whose contents
|
||||
* are unreadable or stale, suppresses the bootstrap for the whole unit. The
|
||||
* per-record documents, provided its stored unit version is in the accepted
|
||||
* set — a legacy file stamped with any other version is left alone and reads
|
||||
* as the empty unit. Any new document path, including one whose contents are
|
||||
* unreadable or stale, suppresses the bootstrap for the whole unit. The
|
||||
* legacy file is never changed or deleted.
|
||||
* @module @deepseek-ai/dsh-storage-json/src/per-record-unit
|
||||
*/
|
||||
|
||||
import { mkdir, readFile, readdir, rm } from 'node:fs/promises'
|
||||
import { mkdir, readFile, readdir, rename, rm } from 'node:fs/promises'
|
||||
import { dirname, join } from 'node:path'
|
||||
import type { Dirent } from 'node:fs'
|
||||
import { StorageError } from '@deepseek-ai/dsh-storage'
|
||||
|
|
@ -64,6 +67,7 @@ export async function openPerRecordUnit(
|
|||
* @returns the authoritative state reconstructed from the tree.
|
||||
*/
|
||||
async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Promise<UnitState> {
|
||||
const versions = acceptedStamps(descriptor)
|
||||
const state: UnitState = {
|
||||
version: descriptor.version,
|
||||
global: null,
|
||||
|
|
@ -83,11 +87,11 @@ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Pr
|
|||
if (entry.isDirectory()) {
|
||||
const records = state.tables.get(entry.name)
|
||||
if (records !== undefined) {
|
||||
return loadTableRecords(records, descriptor.version, join(dir, entry.name))
|
||||
return loadTableRecords(records, versions, join(dir, entry.name))
|
||||
}
|
||||
}
|
||||
if (entry.name === 'global.json' && descriptor.hasGlobal) {
|
||||
const global = await readRecord(join(dir, entry.name), descriptor.version)
|
||||
const global = await readRecord(join(dir, entry.name), versions)
|
||||
if (global !== undefined) state.global = global
|
||||
return true
|
||||
}
|
||||
|
|
@ -97,12 +101,21 @@ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Pr
|
|||
return state
|
||||
}
|
||||
|
||||
/** The version stamps this unit reads as its own: current plus declared compatible versions. */
|
||||
function acceptedStamps(descriptor: KvUnitDescriptor): readonly number[] {
|
||||
return [descriptor.version, ...descriptor.compatibleVersions ?? []]
|
||||
}
|
||||
|
||||
/**
|
||||
* Bootstrap an empty per-record tree from a legacy whole-unit file
|
||||
* (`<root>/<name>.json`, the pre-per-record layout). Every declared-table
|
||||
* record is copied into a current-version document, while the legacy file is
|
||||
* retained unchanged. A missing, foreign (another unit's name), malformed,
|
||||
* or non-unit legacy file is left alone; other read failures propagate.
|
||||
* or non-unit legacy file is left alone, and so is one whose stored unit
|
||||
* version is outside the accepted set — migrating records the owner never
|
||||
* vouched for would stamp them with the current version and turn a
|
||||
* discardable stale cache into schema failures at the domain layer. Other
|
||||
* read failures propagate.
|
||||
* @param descriptor - Static identity and shape of the unit.
|
||||
* @param dir - The per-record unit directory (`<root>/<name>`).
|
||||
* @param state - The empty tree state; bootstrapped records are added.
|
||||
|
|
@ -116,16 +129,18 @@ async function bootstrapLegacyUnit(descriptor: KvUnitDescriptor, dir: string, st
|
|||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
return
|
||||
}
|
||||
// The legacy document is runtime data: only `unit.name` and the tables map
|
||||
// shape are checked here — the record values are migrated as-is and the
|
||||
// domain layer's schemas judge them.
|
||||
let document: { unit?: { name?: unknown }; tables?: unknown }
|
||||
// The legacy document is runtime data: only `unit.name`, `unit.version`,
|
||||
// and the tables map shape are checked here — the record values are
|
||||
// migrated as-is and the domain layer's schemas judge them.
|
||||
let document: { unit?: { name?: unknown; version?: unknown }; tables?: unknown }
|
||||
try {
|
||||
document = JSON.parse(text) as { unit?: { name?: unknown }; tables?: unknown }
|
||||
document = JSON.parse(text) as { unit?: { name?: unknown; version?: unknown }; tables?: unknown }
|
||||
} catch {
|
||||
return // Malformed legacy file: not ours to interpret or delete.
|
||||
}
|
||||
if (document.unit?.name !== descriptor.name) return
|
||||
const stamped = document.unit.version
|
||||
if (typeof stamped !== 'number' || !acceptedStamps(descriptor).includes(stamped)) return
|
||||
const tables = document.tables
|
||||
if (typeof tables !== 'object' || tables === null) return
|
||||
const recordsByTable = tables as Record<string, Record<string, unknown>>
|
||||
|
|
@ -146,14 +161,14 @@ async function bootstrapLegacyUnit(descriptor: KvUnitDescriptor, dir: string, st
|
|||
* @returns whether the directory contains any `.json` document path,
|
||||
* independently of key safety, readability, or stored version.
|
||||
*/
|
||||
async function loadTableRecords(records: Map<string, unknown>, version: number, dir: string): Promise<boolean> {
|
||||
async function loadTableRecords(records: Map<string, unknown>, versions: readonly number[], dir: string): Promise<boolean> {
|
||||
const files = await readdir(dir, { withFileTypes: true })
|
||||
const hasDocuments = files.some(file => file.name.endsWith('.json'))
|
||||
const loaded = await Promise.all(files.map(async (file) => {
|
||||
if (!file.name.endsWith('.json')) return
|
||||
const key = file.name.slice(0, -'.json'.length)
|
||||
if (!SAFE_KEY_RE.test(key)) return
|
||||
const record = await readRecord(join(dir, file.name), version)
|
||||
const record = await readRecord(join(dir, file.name), versions)
|
||||
if (record !== undefined) return [key, record] as const
|
||||
}))
|
||||
for (const record of loaded) {
|
||||
|
|
@ -163,9 +178,9 @@ async function loadTableRecords(records: Map<string, unknown>, version: number,
|
|||
}
|
||||
|
||||
/** Read one record document; a foreign (unreadable or stale) one reads as absent. */
|
||||
async function readRecord(path: string, version: number): Promise<unknown> {
|
||||
async function readRecord(path: string, versions: readonly number[]): Promise<unknown> {
|
||||
try {
|
||||
return parseRecord(await readFile(path, 'utf8'), version)
|
||||
return parseRecord(await readFile(path, 'utf8'), versions)
|
||||
} catch {
|
||||
return undefined
|
||||
}
|
||||
|
|
@ -213,6 +228,22 @@ export class PerRecordJsonUnit implements KvUnit {
|
|||
await this.tracked(rm(join(this.tableDir(table), `${key}.json`), { force: true }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Move one record's document aside as `<key>.json.bak.<YYYYMMDDHHmm>`. The
|
||||
* moved file no longer ends in `.json`, so every later read ignores it; the
|
||||
* bytes stay on disk for inspection. A same-minute backup of the same
|
||||
* key overwrites the previous backup (the newer bytes are the ones worth
|
||||
* keeping).
|
||||
*/
|
||||
async backupRecord(table: string, key: string): Promise<string> {
|
||||
this.assertOpen()
|
||||
assertSafeKey(this.descriptor.name, key)
|
||||
const path = join(this.tableDir(table), `${key}.json`)
|
||||
const moved = `${path}.bak.${backupStamp(new Date())}`
|
||||
await this.tracked(rename(path, moved))
|
||||
return moved
|
||||
}
|
||||
|
||||
/** Durably replace the global singleton. Only valid when declared. */
|
||||
async setGlobal(value: unknown): Promise<void> {
|
||||
this.assertOpen()
|
||||
|
|
@ -267,6 +298,12 @@ export class PerRecordJsonUnit implements KvUnit {
|
|||
}
|
||||
}
|
||||
|
||||
/** Local-time `YYYYMMDDHHmm` suffix for backed-up documents. */
|
||||
function backupStamp(now: Date): string {
|
||||
const pad = (value: number): string => String(value).padStart(2, '0')
|
||||
return `${String(now.getFullYear())}${pad(now.getMonth() + 1)}${pad(now.getDate())}${pad(now.getHours())}${pad(now.getMinutes())}`
|
||||
}
|
||||
|
||||
/** Reject a record key that would be unsafe as a path segment. */
|
||||
function assertSafeKey(unit: string, key: string): void {
|
||||
if (!SAFE_KEY_RE.test(key)) {
|
||||
|
|
|
|||
|
|
@ -329,10 +329,10 @@ describe('per-record layout', () => {
|
|||
|
||||
it('bootstraps an empty per-record tree from a legacy whole-unit file and preserves it', async () => {
|
||||
const root = await freshRoot()
|
||||
// A legacy single-layout file for the same unit (any older version);
|
||||
// the extra table is not declared and must be skipped.
|
||||
// A legacy single-layout file for the same unit, stamped with the current
|
||||
// version; the extra table is not declared and must be skipped.
|
||||
const legacy = JSON.stringify({
|
||||
unit: { name: 'recs', version: 3 },
|
||||
unit: { name: 'recs', version: 2 },
|
||||
global: null,
|
||||
tables: { t: { old1: { v: 1 }, old2: { v: 2 } }, undeclared: { k: { v: 0 } } },
|
||||
})
|
||||
|
|
@ -347,6 +347,75 @@ describe('per-record layout', () => {
|
|||
await backend.close()
|
||||
})
|
||||
|
||||
it('bootstraps from a legacy file only when its stored version is accepted', async () => {
|
||||
// Version 3 is neither current (2) nor declared compat: the legacy file
|
||||
// is left alone and the unit reads empty — migrating unvouched records
|
||||
// would stamp them current and surface as schema failures at the domain
|
||||
// layer instead of a discardable stale cache.
|
||||
const root = await freshRoot()
|
||||
const legacy = JSON.stringify({
|
||||
unit: { name: 'recs', version: 3 },
|
||||
global: null,
|
||||
tables: { t: { old: { v: 1 } } },
|
||||
})
|
||||
await writeFile(join(root, 'recs.json'), legacy, 'utf8')
|
||||
const backend = new JsonStorageBackend(root)
|
||||
const unit = await backend.kv.open(descriptor)
|
||||
expect(await unit.loadAll()).toEqual({ tables: { t: {} }, global: null })
|
||||
await expect(readFile(join(root, 'recs.json'), 'utf8')).resolves.toBe(legacy)
|
||||
await unit.close()
|
||||
await backend.close()
|
||||
|
||||
// The same file bootstraps once version 3 is declared read-compatible…
|
||||
const root2 = await freshRoot()
|
||||
await writeFile(join(root2, 'recs.json'), legacy, 'utf8')
|
||||
const backend2 = new JsonStorageBackend(root2)
|
||||
const compat = { ...descriptor, version: 4, compatibleVersions: [3] }
|
||||
const unit2 = await backend2.kv.open(compat)
|
||||
expect(await unit2.loadAll()).toEqual({ tables: { t: { old: { v: 1 } } }, global: null })
|
||||
// …and the migrated documents are stamped with the CURRENT version.
|
||||
expect(JSON.parse(await readFile(join(root2, 'recs', 't', 'old.json'), 'utf8')))
|
||||
.toEqual({ version: 4, record: { v: 1 } })
|
||||
await unit2.close()
|
||||
await backend2.close()
|
||||
})
|
||||
|
||||
it('backupRecord moves the document aside; reads see it absent and a write recreates it', async () => {
|
||||
const root = await freshRoot()
|
||||
const backend = new JsonStorageBackend(root)
|
||||
const unit = await backend.kv.open(descriptor)
|
||||
await unit.putRecord('t', 'k', { v: 1 })
|
||||
const moved = await unit.backupRecord!('t', 'k')
|
||||
expect(moved).toMatch(/k\.json\.bak\.\d{12}$/)
|
||||
expect(JSON.parse(await readFile(moved, 'utf8'))).toEqual({ version: 2, record: { v: 1 } })
|
||||
await expect(readFile(recordPath(root, 'k'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' })
|
||||
// The moved file no longer ends in .json, so it reads as absent…
|
||||
expect(await unit.loadAll()).toEqual({ tables: { t: {} }, global: null })
|
||||
// …and the key is free for a fresh write.
|
||||
await unit.putRecord('t', 'k', { v: 2 })
|
||||
expect(await unit.loadAll()).toEqual({ tables: { t: { k: { v: 2 } } }, global: null })
|
||||
await expect(unit.backupRecord!('t', 'a/b')).rejects.toThrow(/not path-safe/)
|
||||
await unit.close()
|
||||
await expect(unit.backupRecord!('t', 'k')).rejects.toMatchObject({ code: 'closed' })
|
||||
await backend.close()
|
||||
})
|
||||
|
||||
it('reads per-record documents stamped with a declared compat version and stamps writes current', async () => {
|
||||
const root = await freshRoot()
|
||||
const backend = new JsonStorageBackend(root)
|
||||
const compat = { ...descriptor, compatibleVersions: [1] }
|
||||
await mkdir(join(root, 'recs', 't'), { recursive: true })
|
||||
await writeFile(recordPath(root, 'oldrec'), JSON.stringify({ version: 1, record: { v: 'old' } }), 'utf8')
|
||||
await writeFile(recordPath(root, 'ancient'), JSON.stringify({ version: 0, record: { v: 'no' } }), 'utf8')
|
||||
const unit = await backend.kv.open(compat)
|
||||
// Version 1 is declared compat and served; version 0 is not and discards.
|
||||
expect(await unit.loadAll()).toEqual({ tables: { t: { oldrec: { v: 'old' } } }, global: null })
|
||||
await unit.putRecord('t', 'oldrec', { v: 'new' })
|
||||
expect(JSON.parse(await readFile(recordPath(root, 'oldrec'), 'utf8')))
|
||||
.toEqual({ version: 2, record: { v: 'new' } })
|
||||
await backend.close()
|
||||
})
|
||||
|
||||
it('ignores the legacy whole-unit file when any new document path exists', async () => {
|
||||
const root = await freshRoot()
|
||||
const legacy = JSON.stringify({
|
||||
|
|
@ -400,7 +469,8 @@ describe('per-record layout', () => {
|
|||
await backend4.close()
|
||||
|
||||
const root5 = await freshRoot()
|
||||
await writeFile(join(root5, 'recs.json'), JSON.stringify({ unit: { name: 'recs' }, tables: 'not an object' }), 'utf8')
|
||||
// A current-version stamp so the shapeless `tables` is what stops the bootstrap.
|
||||
await writeFile(join(root5, 'recs.json'), JSON.stringify({ unit: { name: 'recs', version: 2 }, tables: 'not an object' }), 'utf8')
|
||||
const backend5 = new JsonStorageBackend(root5)
|
||||
const unit5 = await backend5.kv.open(descriptor)
|
||||
expect(await unit5.loadAll()).toEqual({ tables: { t: {} }, global: null })
|
||||
|
|
|
|||
|
|
@ -61,6 +61,16 @@ export interface KvUnitDescriptor {
|
|||
* foreign documents.
|
||||
*/
|
||||
readonly layout?: 'single' | 'per-record'
|
||||
/**
|
||||
* Older unit versions whose stored records are also readable under the
|
||||
* declaring owner's current record schemas (the owner vouches for that —
|
||||
* typically by declaring the fields old records lack as optional). Reads of
|
||||
* a `per-record` unit accept documents stamped with any listed version, and
|
||||
* the legacy whole-unit bootstrap accepts a legacy file stamped with one;
|
||||
* writes always stamp {@link version}. `single`-layout reads stay
|
||||
* exact-version.
|
||||
*/
|
||||
readonly compatibleVersions?: readonly number[]
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -99,6 +109,19 @@ export interface KvUnit {
|
|||
*/
|
||||
deleteRecord(table: string, key: string): Promise<void>
|
||||
|
||||
/**
|
||||
* Move one record's stored document out of the unit's readable set,
|
||||
* preserving its bytes for inspection instead of deleting them. Backends
|
||||
* whose medium has no per-record document to move (the `single` layout, a
|
||||
* row store) omit this member, and the caller falls back to its
|
||||
* reject-loud path. Absent after the move: a later {@link loadAll} reads
|
||||
* the key as missing and a later {@link putRecord} recreates it fresh.
|
||||
* @param table - Declared table name.
|
||||
* @param key - Record key.
|
||||
* @returns the medium location the document was moved to (diagnostics).
|
||||
*/
|
||||
backupRecord?(table: string, key: string): Promise<string>
|
||||
|
||||
/**
|
||||
* Write the global singleton durably. Only valid when the descriptor
|
||||
* declared `hasGlobal`.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue