diff --git a/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.i18n.yaml new file mode 100644 index 0000000000..78edca4d40 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md +2026-09-02-projcache-cross-version-read-compat.md: 64b52ba482e5850e1d0c3d473ca87ba7822af67f +2026-09-02-projcache-cross-version-read-compat.zh.md: c8adad9fd2b7184418a4da44e8b2e5d1f31feed7 diff --git a/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md new file mode 100644 index 0000000000..64b52ba482 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md @@ -0,0 +1,69 @@ +# Agent Note: Projection-cache cross-version read compatibility (session_projcache v3/v4/v5) + +Status: implemented + +English | [中文](2026-09-02-projcache-cross-version-read-compat.zh.md) + +## Problem + +The `session_projcache` storage domain evolved through three on-disk generations across published releases. An upgraded DSH_HOME failed in two ways: + +- **A v3 single-file home bricked startup after the upgrade**: the per-record layout's legacy bootstrap migrated the old whole-unit file without checking its `unit.version`, stamping the old records with the current version into the new tree; the domain layer's per-record zod validation at open then hit the missing now-required fields → `invalid-record` → the whole domain refused to open → the plugin tree failed to load. And because the bootstrap writes before validation runs, **the first boot permanently wrote the bad documents into the new tree** ("poisoning") — every later boot saw a non-empty tree, never took the legacy path again, and the home stayed unusable. +- **A v4 per-record home lost its listing titles after the upgrade**: v4 documents were silently discarded by the version-stamp check (the per-record contract), and SessionList is a zero-I/O cache-only read, so a miss served the row without projections; titles only returned as each session was individually reopened. + +The cache domain's own contract is "a stale or unreadable cache costs a longer tail replay, never a wrong value, never a refused load" — the hard failure and the wholesale discard each broke the first half of that contract or the product expectation. + +## The three on-disk generations + +| domain version | shipped in | layout | on-disk form | identity fields | row fields | +|---|---|---|---|---|---| +| 3 | 0.1.1-rc.2 | single | one file `storages/session_projcache.json` (`{unit:{name,version}, global, tables}`) | `createdAt`, `cwd?` | `ver`, `seq`, `val` | +| 4 | 0.1.2-alpha.3 | per-record | one file per session `storages/session_projcache/sessions/.json` (`{version, record}`) | `createdAt`, `cwd?` | same | +| 5 | 0.1.2-alpha.4 | per-record | same as v4 | + `isSeeded` (shipped required; now optional), `inheritedEventCount` (same) | same (`seq` numbers mean the same as v4; only type brands were added) | + +The only substantive v4→v5 difference is the two new lineage identity fields; the `ver/seq/val` row shape is identical across all three generations, and `seq` numbering did not change ([the 2026-08-31 seq/offset brands note](2026-08-31-session-sequence-and-log-offset-brands.md) pins the on-disk numbers as unchanged). v3→v4 was a layout migration with identical record content. + +One derived shape also exists: a v3 home that ran the v5 build once (the poisoned state) — its new tree holds documents **stamped 5 whose content is a v3 record** (no lineage fields). + +## Decision + +Declared read compatibility — reads tolerate vouched-for older versions, writes always stamp the current one: + +1. **`DomainSpec.compatibleVersions` (new, optional)**: the domain owner declares "records stored under these older versions are also readable under the current record schemas" (typically by declaring the fields old records lack as optional). `defineDomain` validates each entry as a non-negative integer below the current version; `descriptorOf` projects the set onto the backend `KvUnitDescriptor`. +2. **json backend per-record reads** accept version stamps in "current ∪ compatibleVersions"; anything outside the set is still discarded as foreign. **The write path always stamps the current version** (the first checkpoint after reading an old record naturally advances it). The `single` layout stays exact-version. +3. **Legacy-bootstrap version gate (the actual bug fix)**: the old whole-unit file's `unit.version` must fall inside the accepted set to be migrated; otherwise the file is left alone and the unit reads empty — stamping records the owner never vouched for turns a discardable stale cache into hard schema failures at the domain layer. +4. **The projcache domain declares `version: 5, compatibleVersions: [3, 4]`**, and the two lineage fields become `.optional()`. The single reader of stored identities, `identityMatches`, normalizes absence to the unseeded lineage (`?? false` / `?? 0`): exact for unforked sessions, while a forked session's expectation is seeded → natural mismatch → discard and cold rebuild, so the lineage binding loses none of its protection. +5. **The poisoned state self-heals**: documents stamped 5 without lineage fields parse under the optional schema (their content is the real pre-upgrade cache data), so the home boots again and titles serve immediately. +6. **Schema-validation backstop: `invalidRecords: 'backup-and-skip'` (declared by this domain only)**. A stored record that still fails to parse beyond read compatibility no longer refuses the whole domain: the domain layer calls the backend's `KvUnit.backupRecord` (json per-record implementation = rename the document to `.json.bak.`, bytes kept, never read again), prints the concrete failure with `logger.error` (domain, table, key, destination, zod cause), and continues the open with the record absent; the next cold read rebuilds and rewrites that session's cache. **The policy is an explicit per-domain declaration and the default stays fail-loud** — other domains still refuse the whole load on invalid stored data, and a backend without `backupRecord` (single layout, row stores) also falls back to fail-loud. Naming history: quarantine → backup-and-skip (user ruling: the word must carry both "back up" and "skip", sharing its root with the `.bak` suffix; skip-backup was rejected because the CLI `--skip-X` convention reads it as "do not back up"). For this domain it supersedes the reset/destroy recovery path of the [2026-07-28 storage recovery proposal](../../proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md), which stays live for authoritative and whole-medium damage. + +### Upgrade matrix + +| home shape | behavior after the fix | +|---|---| +| v3 single-file (not poisoned) | bootstrap migrates (3 ∈ accepted set) → titles serve immediately | +| v3 + poisoned new tree | new-tree documents read directly (optional tolerance) → boot restored, titles serve immediately | +| v4 per-record | documents read directly (4 ∈ accepted set) → titles serve immediately | +| v5 healthy | unaffected | +| old records of forked (seeded) sessions | identity mismatch → discarded, cold rebuild when the session opens (safe side) | + +## Alternatives considered + +- **Discard-and-rebuild only** (bootstrap gate + bump to v6): fixes the boot, but every SessionList title is lost after the upgrade until each session is reopened — fails the upgrade-and-go product requirement. +- **Schema `.default()` fills**: behaviorally equivalent to optional + reader normalization, but bakes the "absent = unseeded" interpretation into the durable schema's output type; ruled for optional — the schema honestly describes every accepted on-disk shape and the interpretation lives at the consumer (user ruling, 2026-09-02). +- **Roll the domain version back 5→4**: the smallest diff (three lines), but breaks version monotonicity, depends on the "bootstrap skips no versions" bug itself, and drops every poisoned and healthy v5 home's cache. + +## Consequences + +- A deployment routing this domain to the sqlite backend gets none of the tolerance: sqlite implements neither `compatibleVersions` nor `backupRecord`, so behavior degrades to the old strict-version semantics (a whole-unit version mismatch still refuses with `version-mismatch`; nothing loosens, nothing serves wrong values). Shipped compositions route this domain to json, so this stays a deployment-configuration risk only. +- The optional lineage fields widen what a current-version document may omit: a v5-stamped record stripped of its lineage decodes as unseeded. The identity match still refuses it for seeded callers, and the per-row `ver` guard still screens every value, so the residual exposure is an unseeded caller reading an unseeded-shaped record — the same trust extended to genuine pre-lineage records. +- `backupRecord` overwrites a same-minute backup of the same key (the newer bytes win); distinct minutes and distinct keys never collide. + +## Testing + +- `storage-json` unit tests: compat-stamped reads / out-of-set discards / writes stamping current; legacy bootstrap migrating only accepted versions (including the migrated-documents-stamp-current assertion); `backupRecord` move / absent read / rewrite / closed guard. +- `storage-domain` unit tests: `compatibleVersions` / `invalidRecords` declaration validation; backup-and-skip falling back to fail-loud when the backend has no `backupRecord`. +- `session-projection-cache` unit tests: records without lineage fields serve unseeded sessions verbatim and are discarded for seeded ones. +- **Archived-fixture recovery tests** (`tests/fixtures.spec.ts` + `tests/fixtures/`): four media archives produced by the real released builds — `v3-single-unit.json` (the 0.1.1-rc.2 whole-unit file), `v4-session-doc.json` (0.1.2-alpha.3), `v5-session-doc.json` (current), `v5-lineageless-doc.json` (the unguarded bootstrap's poisoned shape, synthesized from the v3 record) — each opened through the real storage stack, asserting the listing serves the archived title and that a live write rewrites the document to the current version (v5 stamp + lineage fields + fresh value); plus the backup-and-skip behavior for a schema-failing record (boot survives, `.bak` lands, log is concrete, neighbor records unharmed). +- End-to-end acceptance, executed against the real release artifacts: the published 0.1.1-rc.2 and 0.1.2-alpha.3 npm builds seeded homes through their own web apps (model turns plus a rename RPC), the published 0.1.2-alpha.4 build reproduced both failures (including the poisoned tree), and the fixed build served every home shape — pristine v3, poisoned v3, v4, and fresh — with the SessionList RPC returning the recorded titles verbatim. + +Future bump procedure: when a new version's shape can tolerate old records through "optional fields + reader normalization", add the old version to `compatibleVersions`; otherwise bump normally (discard and rebuild) and remove the no-longer-compatible versions from the set. Either way, the package README requires the bump to land with archived fixtures and tests proving the chosen disposition. diff --git a/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.zh.md b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.zh.md new file mode 100644 index 0000000000..c8adad9fd2 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.zh.md @@ -0,0 +1,69 @@ +# Agent Note: 投影缓存跨版本读兼容(session_projcache v3/v4/v5) + +Status: implemented + +[English](2026-09-02-projcache-cross-version-read-compat.md) | 中文 + +## 问题 + +`session_projcache` 存储域在已发布版本间演进了三代磁盘结构。升级后的 DSH_HOME 出现两类故障: + +- **v3 单文件 home 升级后启动硬失败**:per-record 布局的 legacy bootstrap 迁移旧单文件时不检查其 `unit.version`,把旧记录原样打上当前版本戳写入新树;domain 层开域时逐条 zod 校验,旧记录缺新增必填字段 → `invalid-record` → 整个域拒开 → 插件树加载失败。且 bootstrap 先写盘后校验,**首次启动即把坏文档永久写入新树**("投毒")——此后每次启动新树非空、连 legacy 路径都不再走,home 持续不可用。 +- **v4 per-record home 升级后列表丢标题**:v4 文档被版本戳检查静默丢弃(per-record 契约),SessionList 是零 I/O 纯缓存读,miss 后整行不带投影;标题要等每个会话被逐个重新打开后才恢复。 + +缓存域自身的契约是"过期或不可读的缓存只付出更长的尾部重放,绝不给出错值、绝不拒载"——硬失败与整体丢弃都违背该契约的前半句或产品预期。 + +## 三代磁盘结构差异 + +| domain version | 携带发布 | 布局 | 磁盘形态 | identity 字段 | 行字段 | +|---|---|---|---|---|---| +| 3 | 0.1.1-rc.2 | single | 单文件 `storages/session_projcache.json`(`{unit:{name,version}, global, tables}`) | `createdAt`, `cwd?` | `ver`, `seq`, `val` | +| 4 | 0.1.2-alpha.3 | per-record | 每会话一份 `storages/session_projcache/sessions/.json`(`{version, record}`) | `createdAt`, `cwd?` | 同上 | +| 5 | 0.1.2-alpha.4 | per-record | 同 v4 | + `isSeeded`(v5 首发必填;现为 optional)、`inheritedEventCount`(同前) | 同上(`seq` 数值语义与 v4 相同,仅类型加 brand) | + +v4→v5 的唯一实质差异是 identity 新增两个 lineage 字段;行内 `ver/seq/val` 三代一致,`seq` 的数值含义未变([2026-08-31 seq/offset brands note](2026-08-31-session-sequence-and-log-offset-brands.zh.md) 明确 on-disk 数值不变)。v3→v4 是布局迁移,记录内容结构一致。 + +另有一种衍生形态:跑过一次 v5 版本的 v3 home(投毒态)——新树里存在**版本戳为 5 但内容是 v3 记录**(缺 lineage 字段)的文档。 + +## 决策 + +声明式读兼容——读容忍 owner 背书过的旧版本,写恒戳当前版本: + +1. **`DomainSpec.compatibleVersions`(新增,可选)**:域 owner 声明"这些旧版本的存量记录在当前记录 schema 下也可读"(典型手段:新增字段标 optional)。`defineDomain` 校验各项为小于当前 version 的非负整数;`descriptorOf` 透传到后端 `KvUnitDescriptor`。 +2. **json 后端 per-record 读**:接受"当前版本 ∪ compatibleVersions"内的版本戳,集合外照旧视为 foreign 丢弃;**写路径永远戳当前版本**(读到旧记录后的下一次 checkpoint 自然把它推进到当前版本)。single 布局维持 exact-version 不变。 +3. **legacy bootstrap 版本把关(bug 修复本体)**:旧单文件的 `unit.version` 必须落在接受集合内才迁移,否则视为空 unit 留在原地——为 owner 未背书的记录打当前版本戳,会把"可丢弃的过期缓存"变成 domain 层的 schema 硬失败。 +4. **projcache 域声明 `version: 5, compatibleVersions: [3, 4]`**;两个 lineage 字段改为 `.optional()`。唯一消费 stored identity 的读点 `identityMatches` 把缺失归一化为 unseeded lineage(`?? false` / `?? 0`):对非 fork 会话这是精确值;fork 会话的 expected 是 seeded → 天然 mismatch → 丢弃冷读重建,lineage 绑定的防护不放松。 +5. **投毒态自愈**:v5 戳缺 lineage 字段的文档被 optional schema 直接接受(内容本就是升级前的真实缓存数据),home 恢复可启动且标题立即可服务。 +6. **schema 校验兜底:`invalidRecords: 'backup-and-skip'`(仅本域声明)**。读兼容之外仍然解析失败的存量记录不再让整个域拒开:domain 层调用后端的 `KvUnit.backupRecord`(json per-record 实现=把文档改名为 `.json.bak.`,字节留档、不再被读取),用 `logger.error` 打印具体失败信息(域名、表、键、移动去向、zod 失败原因),随后当该记录不存在继续启动;下一次冷读会重建并重写该会话的缓存。**该策略是域级显式声明,缺省仍为 fail-loud**——其他业务域的存量数据校验失败照旧整域拒载;后端没有 `backupRecord` 能力(single 布局、行存储)时也回退 fail-loud。命名沿革:quarantine → backup-and-skip(用户裁决:词要同时含"备份"与"跳过"两义,且与 `.bak` 后缀同源;skip-backup 因 CLI `--skip-X` 惯例存在"不备份"反读而弃用)。对本域而言,该策略取代了 [2026-07-28 存储恢复提案](../../proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md)中 reset/destroy 的恢复途径;该提案对权威介质与整介质损坏仍然有效。 + +### 升级矩阵 + +| home 形态 | 修复后行为 | +|---|---| +| v3 单文件(未投毒) | bootstrap 迁移(3 ∈ 接受集)→ 标题立即可服务 | +| v3 + 投毒新树 | 新树文档直接读入(optional 容忍)→ 启动恢复、标题立即可服务 | +| v4 per-record | 文档直接读入(4 ∈ 接受集)→ 标题立即可服务 | +| v5 正常 | 不受影响 | +| fork(seeded)会话的旧记录 | identity mismatch → 丢弃,打开会话时冷读重建(安全侧) | + +## 备选方案 + +- **只丢弃重建**(bootstrap 把关 + bump v6):启动可修,但升级后 SessionList 标题全丢、要逐会话打开才恢复——不满足升级即用的产品要求。 +- **schema `.default()` 填缺省**:行为与 optional+读点归一化等价,但把"缺失=unseeded"的解释固化进 durable schema 的输出类型;拍板为 optional——schema 如实描述介质上所有被接受的形态,解释权在消费点(2026-09-02 用户裁决)。 +- **域版本回退 5→4**:改动最小(三行),但破坏版本单调性、依赖"bootstrap 不查版本"这个 bug 本身、且投毒态与正常 v5 home 的缓存全被丢弃。 + +## 影响 + +- 部署方若把本域路由到 sqlite 后端,得不到任何容忍能力:sqlite 既未实现 `compatibleVersions` 也没有 `backupRecord`,行为退化为原有的严格版本语义(整 unit 版本不匹配仍 `version-mismatch` 拒开;不放松、不出错值)。shipped 组合固定路由 json,此风险仅存在于部署配置层面。 +- optional lineage 字段放宽了当前版本文档可缺省的范围:被剥离 lineage 的 v5 戳记录会解码为 unseeded。身份比对仍会对 seeded 调用方拒收,逐行 `ver` 守卫仍筛查每个值,残余暴露面只是 unseeded 调用方读到 unseeded 形态的记录——与真实 pre-lineage 记录享有的信任完全相同。 +- `backupRecord` 对同一键的同一分钟内重复备份会覆盖前一份(新字节胜出);不同分钟、不同键永不冲突。 + +## 测试 + +- `storage-json` 单测:compat 版本戳读入/集合外丢弃/写恒当前版本;legacy bootstrap 仅在版本被接受时迁移(含迁移后文档戳当前版本断言);`backupRecord` 移档/读缺席/重写/封闭守卫。 +- `storage-domain` 单测:`compatibleVersions`/`invalidRecords` 声明校验;后端无 `backupRecord` 时 backup-and-skip 回退 fail-loud。 +- `session-projection-cache` 单测:缺 lineage 字段的记录对 unseeded 会话按原值服务、对 seeded 会话丢弃。 +- **归档 fixtures 独立恢复测试**(`tests/fixtures.spec.ts` + `tests/fixtures/`):真实发布物产出的四份介质存档——`v3-single-unit.json`(0.1.1-rc.2 整域单文件)、`v4-session-doc.json`(0.1.2-alpha.3)、`v5-session-doc.json`(当前版)、`v5-lineageless-doc.json`(无守卫 bootstrap 的投毒形态,由 v3 记录合成)——逐一走真实存储栈开域,断言列表读出归档标题、且 live 写把文档重写为当前版本(v5 戳 + lineage 字段 + 新值);外加 schema 失败记录的 backup-and-skip 行为(启动不失败、`.bak` 落盘、日志具体、邻居记录不受累)。 +- 端到端验收,以真实发布物执行:已发布的 0.1.1-rc.2 与 0.1.2-alpha.3 npm 包经各自 web app 造数(真实模型对话 + rename RPC),已发布的 0.1.2-alpha.4 包复现两类故障(含投毒树),修复后构建对纯净 v3、投毒 v3、v4、全新四种 home 形态经 SessionList RPC 原样返回记录在案的标题。 + +未来 bump 流程:新版本结构若可用"optional 字段 + 读点归一化"容忍旧记录,就把旧版本加入 `compatibleVersions`;否则正常 bump(丢弃重建),并把不再兼容的版本从集合中移除。无论哪条路,包 README 都要求 bump 随附归档 fixture 和论证所选处置方式的测试。 diff --git a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.i18n.yaml b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.i18n.yaml index ea0fe4b44b..96ea4961d6 100644 --- a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.i18n.yaml +++ b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md -2026-07-28-storage-root-and-derived-medium-recovery.md: 1505be1c58d5cf829327b2919113bb2e42798ce7 -2026-07-28-storage-root-and-derived-medium-recovery.zh.md: 1bab5ab663df1419cc826c3d6acb59bd8bff7de0 +2026-07-28-storage-root-and-derived-medium-recovery.md: 68a6792c38a8fe097558de3d5a857ab4ea0d3533 +2026-07-28-storage-root-and-derived-medium-recovery.zh.md: 0c712b3b928f4b3df44e706b154c3a90a784403c diff --git a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md index 1505be1c58..68a6792c38 100644 --- a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md +++ b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.md @@ -10,7 +10,7 @@ The persisted projection cache ([note](2026-07-27-session-projection-and-command **Where the files actually live (root mismatch closed; resolve-once residual still open).** The shared base defaults the session store to the global harness home (`$DSH_HOME/sessions`, default `~/.dsh/sessions`), while the shipped Web overlay used to give the json backend the relative root `./.storages`: `workspace.json` and `session_projcache.json` landed under `/.storages/` — two launches from different directories shared their sessions yet saw different workspace registries and different projection caches, and the cache exists precisely to serve the cross-session cold listing, which missed for every session last cached under another launch directory. That mismatch is now closed: the overlay anchors `storage-json.root` to `$DSH_HOME/storages` with the same `!!js` expression the session root uses (`apps/cli/config/web.cordis.yml`). The residual hazard: `JsonStorageBackend` still never resolves its root — each unit open joins the path against whatever `process.cwd()` is at that moment (packages/storage/storage-json/src/index.ts); the shipped overlay root is already absolute and unaffected, but any relative root (bare Loader boots, tests) still splits on a later cwd change — the exact hazard the JSONL session backend resolves-once to prevent ("later process.cwd() changes cannot split one backend across roots", packages/session/session-persistence-jsonl/src/index.ts). -**Recovery behavior.** Inside a healthy medium the cache is fully self-healing by design: a `stateVersion`-mismatched row is discarded and refolded, a log shrunk below a row's watermark is detected by the anchored restore floor and answered with one full re-read, and every background write is fail-soft. But at the *medium* level there is no recovery at all: a truncated, hand-edited, or version-bumped `session_projcache.json` fails `openJsonUnit` with `malformed-medium`/`version-mismatch` (packages/storage/storage-json/src/format.ts), a schema-drifted record fails domain open with `invalid-record` (packages/storage/storage-domain/src/index.ts), the rejection propagates through `SessionProjectionCache[Service.init]`, and under the CLI's fail-loud boot the assembly refuses to start. A file whose entire content is rebuildable from session logs can brick boot. This contradicts the cache package's own stated stance ("a stale or unreadable cache costs a longer tail replay, never a wrong value") and the cache domain spec's JSDoc ("version bumps discard the whole medium"), which describes an aspiration, not the implementation. Partially superseded for the projection cache: [the per-session cache files note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.md) removed the global `session_projcache` domain, so the cache half of this proposal (recovery on that domain) no longer applies; the `workspace.json` half remains current. The same fail-loud path is *correct* for `workspace.json` — workspace records are authoritative, not derivable — so the missing concept is a per-domain declaration of authority, not a global behavior change. +**Recovery behavior.** Inside a healthy medium the cache is fully self-healing by design: a `stateVersion`-mismatched row is discarded and refolded, a log shrunk below a row's watermark is detected by the anchored restore floor and answered with one full re-read, and every background write is fail-soft. But at the *medium* level there is no recovery at all: a truncated, hand-edited, or version-bumped `session_projcache.json` fails `openJsonUnit` with `malformed-medium`/`version-mismatch` (packages/storage/storage-json/src/format.ts), a schema-drifted record fails domain open with `invalid-record` (packages/storage/storage-domain/src/index.ts), the rejection propagates through `SessionProjectionCache[Service.init]`, and under the CLI's fail-loud boot the assembly refuses to start. A file whose entire content is rebuildable from session logs can brick boot. This contradicts the cache package's own stated stance ("a stale or unreadable cache costs a longer tail replay, never a wrong value") and the cache domain spec's JSDoc ("version bumps discard the whole medium"), which describes an aspiration, not the implementation. Partially superseded for the projection cache: [the per-session cache files note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.md) removed the global `session_projcache` domain, so the cache half of this proposal (recovery on that domain) no longer applies; the `workspace.json` half remains current. The `invalid-record` class for the per-record projection cache is now also superseded: the shipped domain declares `invalidRecords: 'backup-and-skip'` ([cross-version read-compat note](../../implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md)), which backs the failing record up and skips it at open, so the reset/destroy proposal below stays relevant only for whole-medium damage on authoritative or single-document media. The same fail-loud path is *correct* for `workspace.json` — workspace records are authoritative, not derivable — so the missing concept is a per-domain declaration of authority, not a global behavior change. ## Proposal diff --git a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md index 1bab5ab663..0c712b3b92 100644 --- a/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md +++ b/.agents/notes/proposed/architecture/2026-07-28-storage-root-and-derived-medium-recovery.zh.md @@ -10,7 +10,7 @@ Status: proposed **文件到底存在哪(根错位已收口,resolve-once 残余仍开放)。** 共享 base 将会话存储默认为全局 harness home(`$DSH_HOME/sessions`,默认 `~/.dsh/sessions`),而出厂 Web overlay 曾给 json 后端相对根 `./.storages`:`workspace.json` 和 `session_projcache.json` 落在 `<启动目录>/.storages/` 下——从两个不同目录启动,会话相同,工作区注册表和投影缓存却各是一份,而缓存存在的意义恰恰是跨会话冷列表,凡上次在别的启动目录下缓存过的会话全部 miss。这一错位已消除:overlay 现以与会话根同一段 `!!js` 表达式把 `storage-json.root` 锚定到 `$DSH_HOME/storages`(`apps/cli/config/web.cordis.yml`)。残余隐患:`JsonStorageBackend` 仍从不 resolve 根——每次打开 unit 都把路径 join 到当时的 `process.cwd()` 上(packages/storage/storage-json/src/index.ts);出厂 overlay 的根已是绝对路径不受影响,但任何相对根(裸 Loader 启动、测试)仍会被后续 cwd 变化劈开,JSONL 会话后端用「构造时 resolve 一次」防住的正是它("later process.cwd() changes cannot split one backend across roots",packages/session/session-persistence-jsonl/src/index.ts)。 -**恢复行为。** 在健康介质内部,缓存按设计完全自愈:`stateVersion` 不匹配的行被丢弃重折,日志缩短到行水位以下由带锚的 restore floor 检出并以一次全量重读回答,每次后台写都是 fail-soft。但在*介质*层面完全没有恢复:被截断、被手改或版本被 bump 的 `session_projcache.json` 会让 `openJsonUnit` 以 `malformed-medium`/`version-mismatch` 失败(packages/storage/storage-json/src/format.ts),schema 漂移的记录让域 open 以 `invalid-record` 失败(packages/storage/storage-domain/src/index.ts),拒绝一路穿过 `SessionProjectionCache[Service.init]`,在 CLI 的 fail-loud 启动下整个组装拒绝启动。一个内容完全可从会话日志重建的文件能把启动搞死。这与缓存包自己声明的立场("a stale or unreadable cache costs a longer tail replay, never a wrong value")和缓存域 spec 的 JSDoc("version bumps discard the whole medium")相矛盾——后者描述的是愿望而非实现。投影缓存半边已被[每会话缓存文件 note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md) 部分取代:全局 `session_projcache` domain 已移除,本提案的缓存恢复半边不再适用;`workspace.json` 半边仍然有效。同一条 fail-loud 路径对 `workspace.json` 却是*正确*的——工作区记录是权威数据,不可派生——所以缺的概念是按域声明权威性,而不是全局改行为。 +**恢复行为。** 在健康介质内部,缓存按设计完全自愈:`stateVersion` 不匹配的行被丢弃重折,日志缩短到行水位以下由带锚的 restore floor 检出并以一次全量重读回答,每次后台写都是 fail-soft。但在*介质*层面完全没有恢复:被截断、被手改或版本被 bump 的 `session_projcache.json` 会让 `openJsonUnit` 以 `malformed-medium`/`version-mismatch` 失败(packages/storage/storage-json/src/format.ts),schema 漂移的记录让域 open 以 `invalid-record` 失败(packages/storage/storage-domain/src/index.ts),拒绝一路穿过 `SessionProjectionCache[Service.init]`,在 CLI 的 fail-loud 启动下整个组装拒绝启动。一个内容完全可从会话日志重建的文件能把启动搞死。这与缓存包自己声明的立场("a stale or unreadable cache costs a longer tail replay, never a wrong value")和缓存域 spec 的 JSDoc("version bumps discard the whole medium")相矛盾——后者描述的是愿望而非实现。投影缓存半边已被[每会话缓存文件 note](../../implemented/architecture/2026-08-19-projection-cache-per-session-files.zh.md) 部分取代:全局 `session_projcache` domain 已移除,本提案的缓存恢复半边不再适用;`workspace.json` 半边仍然有效。per-record 投影缓存的 `invalid-record` 一类如今也已被取代:已发布的域声明了 `invalidRecords: 'backup-and-skip'`([跨版本读兼容 note](../../implemented/architecture/2026-09-02-projcache-cross-version-read-compat.zh.md)),open 时把失败记录备份后跳过,因此下文 reset/destroy 提案仅对权威介质或单文档介质的整介质损坏仍然相关。同一条 fail-loud 路径对 `workspace.json` 却是*正确*的——工作区记录是权威数据,不可派生——所以缺的概念是按域声明权威性,而不是全局改行为。 ## 提案 diff --git a/docs/subsystems/storage.i18n.yaml b/docs/subsystems/storage.i18n.yaml index 1e9a7e221c..d234297a9d 100644 --- a/docs/subsystems/storage.i18n.yaml +++ b/docs/subsystems/storage.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/storage.md -storage.md: 1e4141e6ef1c6f8e1c2593e21e788b626d6b1ed7 -storage.zh.md: f0433c600674741c3de0ce3e99430297839ce124 +storage.md: e52f6d8869ee3092a5be99f0bd2873b410b45ed9 +storage.zh.md: f752862af9f10cfb32f39f2a1aa7801c922f841e diff --git a/docs/subsystems/storage.md b/docs/subsystems/storage.md index 1e4141e6ef..e52f6d8869 100644 --- a/docs/subsystems/storage.md +++ b/docs/subsystems/storage.md @@ -65,6 +65,26 @@ interface DomainSpec { * (a stale record document is discarded, never migrated). */ readonly layout?: 'single' | 'per-record' + /** + * Older domain versions whose stored records the current record schemas + * also accept (the declaring owner vouches for that, typically by + * declaring the fields older records lack as optional). `per-record` backends + * read documents stamped with a listed version instead of discarding them, + * and accept a legacy whole-unit file so stamped for the one-time + * bootstrap; writes always stamp {@link version}. + */ + readonly compatibleVersions?: readonly number[] + /** + * What `open` does with a stored table record that fails its zod schema. + * Absent (the default), the whole open rejects with `invalid-record` — + * right for authoritative data. `'backup-and-skip'` is for domains whose + * records are disposable derived data: the backend moves the record's + * document aside (`KvUnit.backupRecord`), the failure is logged with + * its cause, and the open continues with the record absent. A backend + * without `backupRecord` (no per-record document to move) falls back + * to the rejecting default. The global slot always rejects. + */ + readonly invalidRecords?: 'backup-and-skip' /** Optional global singleton slot. */ readonly global?: DomainGlobalSpec /** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */ @@ -180,7 +200,10 @@ The mounted domain facility. Opens declared domains over routed backends; one fa * (`facet-unsupported`); open the unit projected from the spec (backend * `version-mismatch`/`malformed-medium` pass through); load and validate * every stored record against the spec's zod schemas (`invalid-record` - * with the offending table and key); construct the domain. + * with the offending table and key — unless the spec declares + * `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in + * which case the failing record is backed up, logged, and skipped); + * construct the domain. * * Lifecycle: the CALLER owns the returned handle and closes it via * `Domain.close()` (typically as its own `ctx.effect` disposer) — the diff --git a/docs/subsystems/storage.zh.md b/docs/subsystems/storage.zh.md index f0433c6006..f752862af9 100644 --- a/docs/subsystems/storage.zh.md +++ b/docs/subsystems/storage.zh.md @@ -65,6 +65,26 @@ interface DomainSpec { * (a stale record document is discarded, never migrated). */ readonly layout?: 'single' | 'per-record' + /** + * Older domain versions whose stored records the current record schemas + * also accept (the declaring owner vouches for that, typically by + * declaring the fields older records lack as optional). `per-record` backends + * read documents stamped with a listed version instead of discarding them, + * and accept a legacy whole-unit file so stamped for the one-time + * bootstrap; writes always stamp {@link version}. + */ + readonly compatibleVersions?: readonly number[] + /** + * What `open` does with a stored table record that fails its zod schema. + * Absent (the default), the whole open rejects with `invalid-record` — + * right for authoritative data. `'backup-and-skip'` is for domains whose + * records are disposable derived data: the backend moves the record's + * document aside (`KvUnit.backupRecord`), the failure is logged with + * its cause, and the open continues with the record absent. A backend + * without `backupRecord` (no per-record document to move) falls back + * to the rejecting default. The global slot always rejects. + */ + readonly invalidRecords?: 'backup-and-skip' /** Optional global singleton slot. */ readonly global?: DomainGlobalSpec /** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */ @@ -180,7 +200,10 @@ The mounted domain facility. Opens declared domains over routed backends; one fa * (`facet-unsupported`); open the unit projected from the spec (backend * `version-mismatch`/`malformed-medium` pass through); load and validate * every stored record against the spec's zod schemas (`invalid-record` - * with the offending table and key); construct the domain. + * with the offending table and key — unless the spec declares + * `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in + * which case the failing record is backed up, logged, and skipped); + * construct the domain. * * Lifecycle: the CALLER owns the returned handle and closes it via * `Domain.close()` (typically as its own `ctx.effect` disposer) — the diff --git a/packages/extensions/tool-cordis/src/api-catalog.ts b/packages/extensions/tool-cordis/src/api-catalog.ts index 55761bf933..3c81eb2b3b 100644 --- a/packages/extensions/tool-cordis/src/api-catalog.ts +++ b/packages/extensions/tool-cordis/src/api-catalog.ts @@ -2151,7 +2151,7 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [ methods: [ { signature: 'async open(spec: S): Promise>', - description: 'Open one declared domain. Steps, each failing the whole call: reject a name that is already open (`already-open`); resolve the backend route (`backend-not-found` passes through from the hub); require its `kv` facet (`facet-unsupported`); open the unit projected from the spec (backend `version-mismatch`/`malformed-medium` pass through); load and validate every stored record against the spec\'s zod schemas (`invalid-record` with the offending table and key); construct the domain.\n\nLifecycle: the CALLER owns the returned handle and closes it via `Domain.close()` (typically as its own `ctx.effect` disposer) — the facility does not tie the domain to any consumer fiber. Domains still open when the facility unmounts are closed by the plugin disposer.', + description: 'Open one declared domain. Steps, each failing the whole call: reject a name that is already open (`already-open`); resolve the backend route (`backend-not-found` passes through from the hub); require its `kv` facet (`facet-unsupported`); open the unit projected from the spec (backend `version-mismatch`/`malformed-medium` pass through); load and validate every stored record against the spec\'s zod schemas (`invalid-record` with the offending table and key — unless the spec declares `invalidRecords: \'backup-and-skip\'` and the unit can move documents aside, in which case the failing record is backed up, logged, and skipped); construct the domain.\n\nLifecycle: the CALLER owns the returned handle and closes it via `Domain.close()` (typically as its own `ctx.effect` disposer) — the facility does not tie the domain to any consumer fiber. Domains still open when the facility unmounts are closed by the plugin disposer.', parameters: [{ name: 'spec', description: 'The domain declaration, typically from `defineDomain`.' }], returns: 'the opened domain handle, typed by the spec.', }, @@ -3956,7 +3956,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [ }, { name: 'DomainSpec', - declaration: 'export interface DomainSpec {\n readonly name: string;\n readonly version: number;\n readonly layout?: \'single\' | \'per-record\';\n readonly global?: DomainGlobalSpec;\n readonly tables: Record;\n}', + declaration: 'export interface DomainSpec {\n readonly name: string;\n readonly version: number;\n readonly layout?: \'single\' | \'per-record\';\n readonly compatibleVersions?: readonly number[];\n readonly invalidRecords?: \'backup-and-skip\';\n readonly global?: DomainGlobalSpec;\n readonly tables: Record;\n}', }, { name: 'DomainTableSpec', @@ -4264,11 +4264,11 @@ export const TYPE_API: readonly TypeApiEntry[] = [ }, { name: 'KvUnit', - declaration: 'export interface KvUnit {\n loadAll(): Promise<{\n tables: Record>;\n global: unknown;\n }>;\n putRecord(table: string, key: string, value: unknown): Promise;\n deleteRecord(table: string, key: string): Promise;\n setGlobal(value: unknown): Promise;\n close(): Promise;\n}', + declaration: 'export interface KvUnit {\n loadAll(): Promise<{\n tables: Record>;\n global: unknown;\n }>;\n putRecord(table: string, key: string, value: unknown): Promise;\n deleteRecord(table: string, key: string): Promise;\n backupRecord?(table: string, key: string): Promise;\n setGlobal(value: unknown): Promise;\n close(): Promise;\n}', }, { name: 'KvUnitDescriptor', - declaration: 'export interface KvUnitDescriptor {\n readonly name: string;\n readonly version: number;\n readonly tables: readonly string[];\n readonly hasGlobal: boolean;\n readonly layout?: \'single\' | \'per-record\';\n}', + declaration: 'export interface KvUnitDescriptor {\n readonly name: string;\n readonly version: number;\n readonly tables: readonly string[];\n readonly hasGlobal: boolean;\n readonly layout?: \'single\' | \'per-record\';\n readonly compatibleVersions?: readonly number[];\n}', }, { name: 'LlmAdapter', diff --git a/packages/session/session-projection-cache/README.i18n.yaml b/packages/session/session-projection-cache/README.i18n.yaml index e4fe8490a2..4dd1489d4e 100644 --- a/packages/session/session-projection-cache/README.i18n.yaml +++ b/packages/session/session-projection-cache/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/session/session-projection-cache/README.md -README.md: 51b9d86724af96304cf09a5c7c1b61b7394d336a -README.zh.md: bb84b67bdde678884fd4f2be1b14b2161da8c2a0 +README.md: 9fd9766d75ab3f9a460b2802ac59810839f21ae4 +README.zh.md: 98725d9ce76ab44821adf3d43e3807932cf6b675 diff --git a/packages/session/session-projection-cache/README.md b/packages/session/session-projection-cache/README.md index 51b9d86724..9fd9766d75 100644 --- a/packages/session/session-projection-cache/README.md +++ b/packages/session/session-projection-cache/README.md @@ -64,6 +64,8 @@ Three mandatory points always write: session creation persists the seed-derived The log leads and the cache follows: a live checkpoint flushes the session's buffered events durably before the cache row lands, so a crash can leave the cache behind the log but never ahead of it. Reads and writes share the storage domain's coherent in-memory state; the per-unit write chain mutates memory only after durability. Each version-stamped record must match the live unit schema and complete lifecycle identity (`createdAt`, `cwd`, `isSeeded`, and `inheritedEventCount`), so a row initialized under one fork cut cannot seed another. The JSON backend stores each record at `/session_projcache/sessions/.json` in an owner-only directory tree. +Upgrades never cost the boot or the listing: records stamped with a version in the spec's `compatibleVersions` stay readable (their absent lineage fields decode as the unseeded lineage — exact for unseeded sessions, while a seeded caller fails the identity match and refolds cold), and a stored record that still fails schema validation is moved aside as `.json.bak.` under the domain's `invalidRecords: 'backup-and-skip'` policy, logged with its cause, and rebuilt by the next checkpoint. + ----- @@ -126,6 +128,7 @@ These limits define where the cache needs operational care. They are current pac - **No eviction or retention surface** — records accumulate per session; pruning stored checkpoints is out-of-band maintenance, same stance as session persistence itself. - **Interval throttle is per-session coarse** — the timer arms at the first dirty event after a clean write; a steady sub-threshold trickle writes once per interval, not a sliding window. - **No cache-side cold refold** — the cache serves and refreshes its rows but never reads the session log (it does not depend on the persistence layer); a consumer that needs a guaranteed cold snapshot refolds from the log itself. +- **Every schema or domain-version change must prove its upgrade story** — a change to the stored record schema or the domain version lands in the same PR with an archived fixture of the previously shipped on-disk format under `tests/fixtures/` and test cases in `tests/fixtures.spec.ts` proving the chosen disposition: read-compat recovery (`compatibleVersions`), current-version rewrite, or backup-and-skip salvage. A bump whose old records are simply discarded still proves that the discard neither fails the boot nor poisons the tree. ### Dev Note diff --git a/packages/session/session-projection-cache/README.zh.md b/packages/session/session-projection-cache/README.zh.md index bb84b67bdd..98725d9ce7 100644 --- a/packages/session/session-projection-cache/README.zh.md +++ b/packages/session/session-projection-cache/README.zh.md @@ -64,6 +64,8 @@ kind: "package-reference" 日志领先,缓存跟随:实时检查点先把会话的缓冲事件持久化,然后才保存缓存记录。因此崩溃可能让缓存落后于日志,但绝不会让缓存领先。读取和写入共享存储域内一致的内存状态;逐单元写入链只在持久化成功后修改内存。每个带版本戳的记录必须匹配实时单元 schema 与完整生命周期身份(`createdAt`、`cwd`、`isSeeded` 和 `inheritedEventCount`),因此在一个 fork 切点下初始化的行不能播种另一个切点。JSON 后端把每条记录存于仅所有者可访问的 `/session_projcache/sessions/.json` 目录树中。 +升级绝不拖垮启动或列表:版本戳落在 spec `compatibleVersions` 集合内的记录保持可读(缺失的 lineage 字段解码为 unseeded lineage——对非 fork 会话精确无误,seeded 调用方则通不过身份比对、回落冷折叠),而仍然通不过 schema 校验的存量记录会按域的 `invalidRecords: 'backup-and-skip'` 策略移出为 `.json.bak.<时间戳>`、连同原因写入日志,并由下一次检查点重建。 + ----- @@ -126,6 +128,7 @@ kind: "package-reference" - **无淘汰或保留接口**——记录按会话持续累积;清理已存储检查点属于带外维护,与会话持久化采用相同策略。 - **间隔节流采用按会话的粗粒度控制**——一次无脏数据的写入完成后,计时器在首个脏事件到达时启动;持续但低于条数阈值的事件流每间隔写入一次,而非滑动窗口。 - **缓存侧不做冷重折叠**——缓存只服务并刷新自己的记录,从不读取会话日志,因为它不依赖持久化层;需要保证冷快照的消费方自行从日志重新折叠。 +- **每次 schema 或域版本变更都必须论证升级路径**——改动存储记录 schema 或域版本时,同一 PR 必须在 `tests/fixtures/` 下归档此前已发布的磁盘格式样本,并在 `tests/fixtures.spec.ts` 中用测试论证所选的处置方式:读兼容恢复(`compatibleVersions`)、当前版本重写,或 backup-and-skip 抢救。即便选择直接丢弃旧记录的 bump,也要证明丢弃既不炸启动、也不污染缓存树。 ### 开发备注 diff --git a/packages/session/session-projection-cache/src/index.ts b/packages/session/session-projection-cache/src/index.ts index 35bf5d9eed..22828a103c 100644 --- a/packages/session/session-projection-cache/src/index.ts +++ b/packages/session/session-projection-cache/src/index.ts @@ -367,12 +367,18 @@ function identityOf( } } -/** Whether a stored record's bound identity names the caller's lifecycle. */ +/** + * Whether a stored record's bound identity names the caller's lifecycle. + * Absent lineage fields (records admitted via `compatibleVersions` predate + * them) read as the unseeded lineage: exact for an unseeded caller, and a + * seeded caller's expectation then fails the match, discarding the record to + * a cold rebuild. + */ function identityMatches(stored: CheckpointIdentity, expected: CheckpointIdentity): boolean { return stored.createdAt === expected.createdAt && stored.cwd === expected.cwd - && stored.isSeeded === expected.isSeeded - && stored.inheritedEventCount === expected.inheritedEventCount + && (stored.isSeeded ?? false) === expected.isSeeded + && (stored.inheritedEventCount ?? 0) === expected.inheritedEventCount } export default SessionProjectionCache diff --git a/packages/session/session-projection-cache/src/spec.ts b/packages/session/session-projection-cache/src/spec.ts index 35351917a5..1830853f6c 100644 --- a/packages/session/session-projection-cache/src/spec.ts +++ b/packages/session/session-projection-cache/src/spec.ts @@ -38,12 +38,19 @@ export const checkpointRow = z.object({ * old record pass every watermark check and seed state folded from an * unrelated log. Reads validate this against the live header (listing) or * the stored header (cold read) before accepting any record. + * + * The lineage fields are optional because records admitted through + * `compatibleVersions` predate them. The reader (`identityMatches`) + * interprets their absence as the unseeded lineage — exact for an unseeded + * session, while a seeded expectation fails the match and the record is + * discarded to a cold rebuild. Current-version writes always store both + * fields. */ export const checkpointIdentity = z.object({ createdAt: z.number().int().nonnegative(), cwd: z.string().optional(), - isSeeded: z.boolean(), - inheritedEventCount: z.number().int().nonnegative().transform(SessionLogOffset), + isSeeded: z.boolean().optional(), + inheritedEventCount: z.number().int().nonnegative().transform(SessionLogOffset).optional(), }) /** The identity fields a record is bound to, inferred from {@link checkpointIdentity}. */ @@ -68,11 +75,27 @@ export type CheckpointRecord = z.infer * bumps per session: after a bump, a stale session document is discarded on * open (cache semantics — a stale or unreadable cache costs a longer tail * replay, never a wrong value) while the rest of the domain stays usable, - * instead of rejecting the whole medium. + * instead of rejecting the whole medium. The `compatibleVersions` entries + * are declared because those records differ from the current version only + * by the absent optional lineage fields, so upgraded homes keep serving + * their cached listing projections instead of dropping every title until + * each session is reopened; the per-record version map lives in the + * read-compat Agent Note + * (.agents/notes/implemented/architecture/2026-09-02-projcache-cross-version-read-compat.md). + * The per-row `ver` guard and the identity match still discard anything the + * current fold semantics cannot vouch for. + * + * `invalidRecords: 'backup-and-skip'`: a stored record that fails the schema + * anyway is disposable derived data, so it must never cost the boot — the + * domain layer moves the document aside as `.json.bak.`, logs + * the concrete validation failure, and serves the session as uncached (a + * cold read rebuilds and rewrites it). */ export const projectionCacheDomainSpec = defineDomain({ name: 'session_projcache', version: 5, + compatibleVersions: [3, 4], + invalidRecords: 'backup-and-skip', layout: 'per-record', tables: { sessions: domainTable(checkpointRecord) }, }) diff --git a/packages/session/session-projection-cache/tests/cache.spec.ts b/packages/session/session-projection-cache/tests/cache.spec.ts index bfdca18567..4a5ce73d28 100644 --- a/packages/session/session-projection-cache/tests/cache.spec.ts +++ b/packages/session/session-projection-cache/tests/cache.spec.ts @@ -42,9 +42,11 @@ declare module '@deepseek-ai/dsh-session-projection/types' { 'cache-test/marks2': Map 'cache-test/count': number 'cache-test/secret': string + 'cache-test/marks3': MarksState } interface SessionProjectionMap { 'cache-test/marks': { marks: string[] } + 'cache-test/marks3': { marks: string[] } } } @@ -71,6 +73,18 @@ const marksUnit = (stateVersion = 1) => ({ stateVersion, }) satisfies ProjectionDefinition<'cache-test/marks', MarksState> +const marks3Unit = { + key: 'cache-test/marks3', + stateSchema: z.object({ marks: z.array(z.string()) }).nullable(), + init: () => null, + apply: state => state, + wire: { + viewSchema: z.object({ marks: z.array(z.string()) }), + view: state => state ?? { marks: [] }, + }, + stateVersion: 1, +} satisfies ProjectionDefinition<'cache-test/marks3', MarksState> + const secretUnit = { key: 'cache-test/secret', stateSchema: z.string(), @@ -376,6 +390,26 @@ describe('SessionProjectionCache listing read', () => { .toBeUndefined() }) + it('carries ONE cut across multiple served rows: the lowest watermark wins', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-')) + roots.push(root) + // Equal watermarks: whichever row is visited second cannot lower the cut, + // so the one-cut fold sees both a lowering and a non-lowering row in + // every iteration order. + await seedRecord(root, 'multi-row', { + 'cache-test/marks': { ver: 1, seq: SessionSeq(4), val: { marks: ['a'] } }, + 'cache-test/marks3': { ver: 1, seq: SessionSeq(4), val: { marks: ['b'] } }, + }) + const { ctx, cache } = await harness({ root }) + ctx.sessionProjections.register(marks3Unit) + const block = cache.cachedSnapshot(headerOf(SessionId('multi-row')), SessionLogOffset(0)) + expect(block?.values).toEqual({ + 'cache-test/marks': { marks: ['a'] }, + 'cache-test/marks3': { marks: ['b'] }, + }) + expect(block?.asOfSeq).toBe(4) + }) + it('returns undefined when the stored record is version-mismatched', async () => { const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-')) roots.push(root) @@ -394,6 +428,30 @@ describe('SessionProjectionCache listing read', () => { .toBeUndefined() }) + it('serves a pre-lineage record (accepted old version) to an unseeded caller only', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-')) + roots.push(root) + // A document stamped with an accepted older version whose identity + // predates the lineage fields: absent lineage reads as unseeded. + const path = recordPath(root, SessionId('pre-lineage')) + await mkdir(dirname(path), { recursive: true }) + await writeFile(path, JSON.stringify({ + version: 4, + record: { + identity: { createdAt: 0 }, + rows: { 'cache-test/marks': { ver: 1, seq: 4, val: { marks: ['kept'] } } }, + }, + })) + const { cache } = await harness({ root }) + const id = SessionId('pre-lineage') + // Unseeded caller: the absent lineage is exactly its identity — served. + expect(cache.cachedSnapshot(headerOf(id), SessionLogOffset(0))) + .toEqual({ asOfSeq: 4, values: { 'cache-test/marks': { marks: ['kept'] } } }) + // Seeded caller: the lineage-less record cannot vouch for the cut — refused. + expect(cache.cachedSnapshot({ ...headerOf(id), isSeeded: true }, SessionLogOffset(2))) + .toBeUndefined() + }) + it('returns undefined when every stored row is version-mismatched', async () => { const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-')) roots.push(root) diff --git a/packages/session/session-projection-cache/tests/fixtures.spec.ts b/packages/session/session-projection-cache/tests/fixtures.spec.ts new file mode 100644 index 0000000000..e2486b7a9d --- /dev/null +++ b/packages/session/session-projection-cache/tests/fixtures.spec.ts @@ -0,0 +1,242 @@ +/** + * Cross-version recovery over archived on-disk artifacts. `fixtures/` holds + * real `session_projcache` media, each produced by driving the named release + * through its own web app (session created over RPC, real model turns, a + * rename): the v3 whole-unit file (published 0.1.1-rc.2), a v4 per-record + * document (published 0.1.2-alpha.3), a current v5 document, and the + * v5-stamped lineage-less document reproducing byte-for-byte what the + * formerly unguarded legacy bootstrap wrote over v3 records. Each must + * recover through the real storage stack — the domain opens and the listing + * read serves the archived title — and a record that fails schema validation + * anyway is backed up and skipped instead of failing the boot. + */ + +import { afterEach, describe, expect, it, vi } from 'vitest' +import { cp, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { Context } from '@deepseek-ai/cordis' +import { z } from 'zod' +import SessionStore, { SessionId, SessionLogOffset } from '@deepseek-ai/dsh-session' +import type { SessionHeader } from '@deepseek-ai/dsh-session' +import SessionProjectionRegistry from '@deepseek-ai/dsh-session-projection' +import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection' +import Storage from '@deepseek-ai/dsh-storage' +import { + apply as storageJsonApply, Config as storageJsonConfig, inject as storageJsonInject, name as storageJsonName, +} from '@deepseek-ai/dsh-storage-json' +import { + apply as storageDomainApply, Config as storageDomainConfig, inject as storageDomainInject, name as storageDomainName, +} from '@deepseek-ai/dsh-storage-domain' +import SessionProjectionCache from '../src/index.ts' +import { projectionCacheDomainSpec } from '../src/spec.ts' + +// Declarations must match the shipped title unit's exactly (the repo-wide +// compile face sees both). +declare module '@deepseek-ai/dsh-session-projection/types' { + interface SessionProjectionStateMap { + title: string | null + } + interface SessionProjectionMap { + title: string | null + } +} + +declare module '@deepseek-ai/dsh-session/types' { + interface SessionEventMap { + 'fixtures-test/set-title': { title: string } + } + + interface OutOfBandSessionEventMap { + 'fixtures-test/set-title': true + } +} + +// Mirrors the shipped title unit's storage face: stateVersion 1, bare-string +// state (the fixture rows carry exactly this shape in every archived +// version), folding a test event so the rewrite path has fresh data. +const titleUnit = { + key: 'title', + stateSchema: z.string().nullable(), + init: () => null, + apply: (state, event) => (event.type === 'fixtures-test/set-title' ? event.data.title : state), + wire: { viewSchema: z.string().nullable(), view: state => state }, + stateVersion: 1, +} satisfies ProjectionDefinition<'title', string | null> + +const FIXTURES = fileURLToPath(new URL('./fixtures/', import.meta.url)) + +/** One archived per-record document (`{version, record}`). */ +interface FixtureDoc { + version: number + record: { + identity: { createdAt: number; cwd?: string } + rows: Record + } +} + +async function fixtureJson(name: string): Promise { + return JSON.parse(await readFile(join(FIXTURES, name), 'utf8')) as T +} + +/** Header for the session a fixture record is bound to (identity witness). */ +function headerFor(id: SessionId, identity: FixtureDoc['record']['identity']): SessionHeader { + return { + version: 0, + id, + createdAt: identity.createdAt, + isSeeded: false, + ...identity.cwd === undefined ? {} : { cwd: identity.cwd }, + } +} + +const contexts: Context[] = [] +const roots: string[] = [] + +async function harness(root: string) { + roots.push(root) + const ctx = new Context() + contexts.push(ctx) + await ctx.plugin(Storage) + await ctx.plugin({ name: storageJsonName, inject: storageJsonInject, apply: storageJsonApply, Config: storageJsonConfig }, { root }) + await ctx.plugin({ name: storageDomainName, inject: storageDomainInject, apply: storageDomainApply, Config: storageDomainConfig }, { backend: 'json' }) + await ctx.plugin(SessionStore) + await ctx.plugin(SessionProjectionRegistry) + ctx.sessionProjections.register(titleUnit) + await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 }) + return { ctx, cache: ctx.sessionProjectionCache } +} + +/** Lay one per-record fixture document into a fresh backend root. */ +async function placeDoc(root: string, id: string, name: string): Promise { + const path = join(root, projectionCacheDomainSpec.name, 'sessions', `${id}.json`) + await mkdir(dirname(path), { recursive: true }) + await cp(join(FIXTURES, name), path) + return fixtureJson(name) +} + +/** + * Drive a live write over a recovered session id and assert the archived + * document is replaced by a current-version one: v5 stamp, lineage present, + * and the freshly folded title — the write path never keeps the old format. + */ +async function assertRewrite(ctx: Context, root: string, id: SessionId): Promise { + const session = ctx.sessions.create(id) + session.append('fixtures-test/set-title', { title: '重写标题' }) + session.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + const path = join(root, projectionCacheDomainSpec.name, 'sessions', `${id}.json`) + await vi.waitFor(async () => { + const doc = JSON.parse(await readFile(path, 'utf8')) as FixtureDoc + expect(doc.version).toBe(projectionCacheDomainSpec.version) + expect(doc.record.identity).toMatchObject({ isSeeded: false, inheritedEventCount: 0 }) + expect(doc.record.rows['title']?.val).toBe('重写标题') + }, { timeout: 5_000 }) +} + +afterEach(async () => { + await Promise.all(contexts.splice(0).map(ctx => ctx.fiber.dispose())) + await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }))) +}) + +describe('archived version recovery', () => { + it('recovers the v3 whole-unit archive through the legacy bootstrap', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-')) + await cp(join(FIXTURES, 'v3-single-unit.json'), join(root, `${projectionCacheDomainSpec.name}.json`)) + type SingleUnit = { + unit: { version: number } + tables: { sessions: Record } + } + const archive = await fixtureJson('v3-single-unit.json') + expect(archive.unit.version).toBe(3) // the fixture IS the old format + const [sid, record] = Object.entries(archive.tables.sessions)[0]! + + const { ctx, cache } = await harness(root) + const snapshot = cache.cachedSnapshot(headerFor(SessionId(sid), record.identity), SessionLogOffset(0), ['title']) + expect(snapshot?.values.title).toBe(record.rows['title']!.val) + + // The one-time bootstrap materialized a current-version document. + const migrated = JSON.parse( + await readFile(join(root, projectionCacheDomainSpec.name, 'sessions', `${sid}.json`), 'utf8'), + ) as { version: number } + expect(migrated.version).toBe(projectionCacheDomainSpec.version) + + await assertRewrite(ctx, root, SessionId(sid)) + }) + + for (const [fixture, storedVersion] of [ + ['v4-session-doc.json', 4], + ['v5-session-doc.json', 5], + ['v5-lineageless-doc.json', 5], + ] as const) { + it(`serves the archived title from ${fixture}, then rewrites it current`, async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-')) + const id = SessionId('fixture-session') + const doc = await placeDoc(root, id, fixture) + expect(doc.version).toBe(storedVersion) + + const { ctx, cache } = await harness(root) + const snapshot = cache.cachedSnapshot(headerFor(id, doc.record.identity), SessionLogOffset(0), ['title']) + expect(snapshot?.values.title).toBe(doc.record.rows['title']!.val) + + await assertRewrite(ctx, root, id) + }) + } + + it('refuses a lineage-less archive for a seeded caller (identity mismatch, cold rebuild)', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-')) + const id = SessionId('fixture-seeded') + const doc = await placeDoc(root, id, 'v5-lineageless-doc.json') + + const { cache } = await harness(root) + const seeded = { ...headerFor(id, doc.record.identity), isSeeded: true } + expect(cache.cachedSnapshot(seeded, SessionLogOffset(2), ['title'])).toBeUndefined() + }) + + it('backs up and skips a record that fails schema validation instead of failing the boot', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-projcache-fx-')) + roots.push(root) + const sessionsDir = join(root, projectionCacheDomainSpec.name, 'sessions') + await mkdir(sessionsDir, { recursive: true }) + // Current-version stamp, hopeless record content: no compat rung can save it. + await writeFile(join(sessionsDir, 'broken.json'), JSON.stringify({ + version: projectionCacheDomainSpec.version, + record: { identity: { createdAt: 'not-a-number' }, rows: 'not-an-object' }, + })) + const good = await placeDoc(root, SessionId('survivor'), 'v5-session-doc.json') + + const ctx = new Context() + contexts.push(ctx) + await ctx.plugin(Storage) + await ctx.plugin({ name: storageJsonName, inject: storageJsonInject, apply: storageJsonApply, Config: storageJsonConfig }, { root }) + await ctx.plugin({ name: storageDomainName, inject: storageDomainInject, apply: storageDomainApply, Config: storageDomainConfig }, { backend: 'json' }) + await ctx.plugin(SessionStore) + await ctx.plugin(SessionProjectionRegistry) + ctx.sessionProjections.register(titleUnit) + const error = vi.spyOn(ctx.logger, 'error').mockImplementation(() => {}) + // The boot survives the broken record — this line rejecting IS the fixed bug. + await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 }) + + // Concrete console diagnostics: which record, where it went, and why. + expect(error).toHaveBeenCalledWith(expect.stringContaining("record 'broken'")) + expect(error).toHaveBeenCalledWith(expect.stringContaining('.json.bak.')) + + // The document was moved aside as .json.bak., bytes intact. + const entries = await readdir(sessionsDir) + expect(entries).not.toContain('broken.json') + const backup = entries.find(name => /^broken\.json\.bak\.\d{12}$/.test(name)) + expect(backup).toBeDefined() + expect(JSON.parse(await readFile(join(sessionsDir, backup!), 'utf8'))) + .toMatchObject({ record: { rows: 'not-an-object' } }) + + // The broken record reads as absent; its neighbors still serve. + const cache = ctx.sessionProjectionCache + expect(cache.cachedSnapshot(headerFor(SessionId('broken'), { createdAt: 0 }), SessionLogOffset(0))) + .toBeUndefined() + expect(cache.cachedSnapshot( + headerFor(SessionId('survivor'), good.record.identity), + SessionLogOffset(0), + ['title'], + )?.values.title).toBe(good.record.rows['title']!.val) + }) +}) diff --git a/packages/session/session-projection-cache/tests/fixtures/v3-single-unit.json b/packages/session/session-projection-cache/tests/fixtures/v3-single-unit.json new file mode 100644 index 0000000000..4b36243611 --- /dev/null +++ b/packages/session/session-projection-cache/tests/fixtures/v3-single-unit.json @@ -0,0 +1,136 @@ +{ + "unit": { + "name": "session_projcache", + "version": 3 + }, + "global": null, + "tables": { + "sessions": { + "session-1374fa81-15da-44ca-be12-b4c6fe8076a3": { + "identity": { + "createdAt": 1788286864454, + "cwd": "/tmp" + }, + "rows": { + "sessionStats": { + "ver": 1, + "seq": 54, + "val": { + "turns": 2, + "steps": 2, + "llmMs": 2169, + "toolMs": 0, + "ttftMs": 1872, + "ttftSteps": 2, + "decodeMs": 297, + "decodeTokens": 20, + "lastTurn": 2, + "openStep": null, + "pendingCalls": {} + } + }, + "title": { + "ver": 1, + "seq": 54, + "val": "验收标题-rc2" + }, + "goal": { + "ver": 4, + "seq": 54, + "val": null + }, + "tokenUsage": { + "ver": 1, + "seq": 54, + "val": { + "totals": { + "uncachedInputTokens": 6083, + "outputTokens": 20, + "cacheReadTokens": 9728, + "cacheWriteTokens": 0 + }, + "last": { + "turn": 2, + "step": 1, + "buckets": { + "uncachedInputTokens": 112, + "outputTokens": 4, + "cacheReadTokens": 7808, + "cacheWriteTokens": 0 + } + } + } + }, + "contextPressure": { + "ver": 4, + "seq": 54, + "val": { + "surfaceTokens": 178, + "contextWindow": 1000000, + "pressureTokens": 7920, + "sampledSurfaceTokens": 168 + } + }, + "contextBreakdown": { + "ver": 2, + "seq": 54, + "val": { + "systemTokens": 1620, + "toolsTokens": 6475, + "messageTokens": 178 + } + }, + "subagentTiming": { + "ver": 2, + "seq": 54, + "val": { + "descriptorSeen": false, + "settledMs": 0 + } + }, + "subagent": { + "ver": 2, + "seq": 54, + "val": {} + }, + "permissions": { + "ver": 1, + "seq": 54, + "val": { + "preset": "workspace-write", + "sandbox": "workspace-write", + "approval": "ask" + } + }, + "sessionListMetadata": { + "ver": 1, + "seq": 54, + "val": { + "blank": false, + "lastPromptAt": 1788286867410 + } + }, + "imageLimits": { + "ver": 1, + "seq": 54, + "val": null + }, + "todos": { + "ver": 2, + "seq": 54, + "val": null + }, + "plan": { + "ver": 2, + "seq": 54, + "val": { + "active": false, + "wanted": null, + "running": null + } + } + } + } + } + } +} diff --git a/packages/session/session-projection-cache/tests/fixtures/v4-session-doc.json b/packages/session/session-projection-cache/tests/fixtures/v4-session-doc.json new file mode 100644 index 0000000000..56d6868405 --- /dev/null +++ b/packages/session/session-projection-cache/tests/fixtures/v4-session-doc.json @@ -0,0 +1,212 @@ +{ + "version": 4, + "record": { + "identity": { + "createdAt": 1788286912530, + "cwd": "/tmp" + }, + "rows": { + "title": { + "ver": 1, + "seq": 68, + "val": "验收标题-alpha3" + }, + "titleInput": { + "ver": 3, + "seq": 68, + "val": { + "first": { + "seq": 7, + "text": "请只回复一个词:pong" + }, + "count": 2, + "lastSeq": 58 + } + }, + "llmRetry": { + "ver": 1, + "seq": 68, + "val": {} + }, + "sandboxMode": { + "ver": 1, + "seq": 68, + "val": "workspace-write" + }, + "goal": { + "ver": 6, + "seq": 68, + "val": { + "current": null, + "seenGoalIds": [], + "failure": null + } + }, + "tokenUsage": { + "ver": 2, + "seq": 68, + "val": { + "totals": { + "uncachedInputTokens": 8192, + "outputTokens": 35, + "cacheReadTokens": 8064, + "cacheWriteTokens": 0 + }, + "last": { + "turn": 2, + "step": 1, + "buckets": { + "uncachedInputTokens": 86, + "outputTokens": 4, + "cacheReadTokens": 8064, + "cacheWriteTokens": 0 + } + } + } + }, + "contextPressure": { + "ver": 4, + "seq": 68, + "val": { + "surfaceTokens": 193, + "contextWindow": 1000000, + "pressureTokens": 8150, + "sampledSurfaceTokens": 183 + } + }, + "contextBreakdown": { + "ver": 2, + "seq": 68, + "val": { + "systemTokens": 1760, + "toolsTokens": 6579, + "messageTokens": 193 + } + }, + "turnBoundary": { + "ver": 2, + "seq": 68, + "val": { + "openTurnStartSeq": null, + "lastStepStartSeq": 57, + "lastStepBoundary": { + "kind": "end", + "seq": 67 + }, + "lastTurn": 2 + } + }, + "sessionStats": { + "ver": 1, + "seq": 68, + "val": { + "turns": 2, + "steps": 2, + "llmMs": 3486, + "toolMs": 0, + "ttftMs": 3199, + "ttftSteps": 2, + "decodeMs": 287, + "decodeTokens": 35, + "lastTurn": 2, + "openStep": null, + "pendingCalls": {} + } + }, + "turnOutline": { + "ver": 2, + "seq": 68, + "val": { + "turns": [ + { + "turn": 1, + "seq": 4, + "prompt": "请只回复一个词:pong", + "response": "pong" + }, + { + "turn": 2, + "seq": 55, + "prompt": "请只回复一个词:pong2", + "response": "pong2" + } + ], + "draft": "" + } + }, + "agentPreset": { + "ver": 1, + "seq": 68, + "val": "standard" + }, + "subagentTiming": { + "ver": 2, + "seq": 68, + "val": { + "descriptorSeen": false, + "settledMs": 0 + } + }, + "subagent": { + "ver": 2, + "seq": 68, + "val": {} + }, + "permissions": { + "ver": 2, + "seq": 68, + "val": { + "preset": "workspace-write", + "sandbox": "workspace-write", + "approval": "ask", + "seeded": false + } + }, + "modelSelection": { + "ver": 2, + "seq": 68, + "val": { + "lastUsed": { + "provider": "deepseek-official", + "model": "deepseek-v4-flash", + "reasoningEffort": "high" + }, + "pending": null + } + }, + "sessionListMetadata": { + "ver": 1, + "seq": 68, + "val": { + "blank": false, + "lastPromptAt": 1788286917834 + } + }, + "imageLimits": { + "ver": 1, + "seq": 68, + "val": null + }, + "todos": { + "ver": 2, + "seq": 68, + "val": null + }, + "plan": { + "ver": 3, + "seq": 68, + "val": { + "active": false, + "wanted": null, + "running": null, + "activeAtLastHeader": false + } + }, + "subagentModelSelectionPolicy": { + "ver": 1, + "seq": 68, + "val": null + } + } + } +} diff --git a/packages/session/session-projection-cache/tests/fixtures/v5-lineageless-doc.json b/packages/session/session-projection-cache/tests/fixtures/v5-lineageless-doc.json new file mode 100644 index 0000000000..f887e980e2 --- /dev/null +++ b/packages/session/session-projection-cache/tests/fixtures/v5-lineageless-doc.json @@ -0,0 +1,128 @@ +{ + "version": 5, + "record": { + "identity": { + "createdAt": 1788286864454, + "cwd": "/tmp" + }, + "rows": { + "sessionStats": { + "ver": 1, + "seq": 54, + "val": { + "turns": 2, + "steps": 2, + "llmMs": 2169, + "toolMs": 0, + "ttftMs": 1872, + "ttftSteps": 2, + "decodeMs": 297, + "decodeTokens": 20, + "lastTurn": 2, + "openStep": null, + "pendingCalls": {} + } + }, + "title": { + "ver": 1, + "seq": 54, + "val": "\u9a8c\u6536\u6807\u9898-rc2" + }, + "goal": { + "ver": 4, + "seq": 54, + "val": null + }, + "tokenUsage": { + "ver": 1, + "seq": 54, + "val": { + "totals": { + "uncachedInputTokens": 6083, + "outputTokens": 20, + "cacheReadTokens": 9728, + "cacheWriteTokens": 0 + }, + "last": { + "turn": 2, + "step": 1, + "buckets": { + "uncachedInputTokens": 112, + "outputTokens": 4, + "cacheReadTokens": 7808, + "cacheWriteTokens": 0 + } + } + } + }, + "contextPressure": { + "ver": 4, + "seq": 54, + "val": { + "surfaceTokens": 178, + "contextWindow": 1000000, + "pressureTokens": 7920, + "sampledSurfaceTokens": 168 + } + }, + "contextBreakdown": { + "ver": 2, + "seq": 54, + "val": { + "systemTokens": 1620, + "toolsTokens": 6475, + "messageTokens": 178 + } + }, + "subagentTiming": { + "ver": 2, + "seq": 54, + "val": { + "descriptorSeen": false, + "settledMs": 0 + } + }, + "subagent": { + "ver": 2, + "seq": 54, + "val": {} + }, + "permissions": { + "ver": 1, + "seq": 54, + "val": { + "preset": "workspace-write", + "sandbox": "workspace-write", + "approval": "ask" + } + }, + "sessionListMetadata": { + "ver": 1, + "seq": 54, + "val": { + "blank": false, + "lastPromptAt": 1788286867410 + } + }, + "imageLimits": { + "ver": 1, + "seq": 54, + "val": null + }, + "todos": { + "ver": 2, + "seq": 54, + "val": null + }, + "plan": { + "ver": 2, + "seq": 54, + "val": { + "active": false, + "wanted": null, + "running": null + } + } + } + } +} diff --git a/packages/session/session-projection-cache/tests/fixtures/v5-session-doc.json b/packages/session/session-projection-cache/tests/fixtures/v5-session-doc.json new file mode 100644 index 0000000000..9b822870a8 --- /dev/null +++ b/packages/session/session-projection-cache/tests/fixtures/v5-session-doc.json @@ -0,0 +1,220 @@ +{ + "version": 5, + "record": { + "identity": { + "createdAt": 1788286864454, + "cwd": "/tmp", + "isSeeded": false, + "inheritedEventCount": 0 + }, + "rows": { + "title": { + "ver": 1, + "seq": 71, + "val": "验收标题-rc2" + }, + "titleInput": { + "ver": 3, + "seq": 71, + "val": { + "first": { + "seq": 7, + "text": "请只回复一个词:pong" + }, + "count": 3, + "lastSeq": 60 + } + }, + "llmRetry": { + "ver": 1, + "seq": 71, + "val": {} + }, + "sandboxMode": { + "ver": 1, + "seq": 71, + "val": "workspace-write" + }, + "goal": { + "ver": 6, + "seq": 71, + "val": { + "current": null, + "seenGoalIds": [], + "failure": null + } + }, + "tokenUsage": { + "ver": 2, + "seq": 71, + "val": { + "totals": { + "uncachedInputTokens": 14219, + "outputTokens": 24, + "cacheReadTokens": 9728, + "cacheWriteTokens": 0 + }, + "last": { + "turn": 3, + "step": 1, + "buckets": { + "uncachedInputTokens": 8136, + "outputTokens": 4, + "cacheReadTokens": 0, + "cacheWriteTokens": 0 + } + } + } + }, + "contextPressure": { + "ver": 4, + "seq": 71, + "val": { + "surfaceTokens": 200, + "contextWindow": 1000000, + "pressureTokens": 8136, + "sampledSurfaceTokens": 190 + } + }, + "contextBreakdown": { + "ver": 2, + "seq": 71, + "val": { + "systemTokens": 1729, + "toolsTokens": 6611, + "messageTokens": 200 + } + }, + "sessionStats": { + "ver": 1, + "seq": 71, + "val": { + "turns": 3, + "steps": 3, + "llmMs": 4368, + "toolMs": 0, + "ttftMs": 4030, + "ttftSteps": 3, + "decodeMs": 338, + "decodeTokens": 24, + "lastTurn": 3, + "openStep": null, + "pendingCalls": {} + } + }, + "agentPreset": { + "ver": 1, + "seq": 71, + "val": "standard" + }, + "subagentTiming": { + "ver": 2, + "seq": 71, + "val": { + "descriptorSeen": false, + "settledMs": 0 + } + }, + "subagent": { + "ver": 2, + "seq": 71, + "val": {} + }, + "turnBoundary": { + "ver": 2, + "seq": 71, + "val": { + "openTurnStartSeq": null, + "lastStepStartSeq": 59, + "lastStepBoundary": { + "kind": "end", + "seq": 70 + }, + "lastTurn": 3 + } + }, + "turnOutline": { + "ver": 2, + "seq": 71, + "val": { + "turns": [ + { + "turn": 1, + "seq": 4, + "prompt": "请只回复一个词:pong", + "response": "pong" + }, + { + "turn": 2, + "seq": 40, + "prompt": "请只回复一个词:pong2", + "response": "pong2" + }, + { + "turn": 3, + "seq": 57, + "prompt": "请只回复一个词:pong3", + "response": "pong3" + } + ], + "draft": "" + } + }, + "permissions": { + "ver": 2, + "seq": 71, + "val": { + "preset": "workspace-write", + "sandbox": "workspace-write", + "approval": "ask", + "seeded": true + } + }, + "modelSelection": { + "ver": 2, + "seq": 71, + "val": { + "lastUsed": { + "provider": "deepseek-official", + "model": "deepseek-v4-flash", + "reasoningEffort": "high" + }, + "pending": null + } + }, + "sessionListMetadata": { + "ver": 1, + "seq": 71, + "val": { + "blank": false, + "lastPromptAt": 1788287660106 + } + }, + "imageLimits": { + "ver": 1, + "seq": 71, + "val": null + }, + "todos": { + "ver": 2, + "seq": 71, + "val": null + }, + "plan": { + "ver": 3, + "seq": 71, + "val": { + "active": false, + "wanted": null, + "running": null, + "activeAtLastHeader": false + } + }, + "subagentModelSelectionPolicy": { + "ver": 1, + "seq": 71, + "val": null + } + } + } +} diff --git a/packages/storage/storage-domain/src/index.ts b/packages/storage/storage-domain/src/index.ts index d2c16a3d69..7e0be5e151 100644 --- a/packages/storage/storage-domain/src/index.ts +++ b/packages/storage/storage-domain/src/index.ts @@ -88,7 +88,10 @@ export class DomainFacility { * (`facet-unsupported`); open the unit projected from the spec (backend * `version-mismatch`/`malformed-medium` pass through); load and validate * every stored record against the spec's zod schemas (`invalid-record` - * with the offending table and key); construct the domain. + * with the offending table and key — unless the spec declares + * `invalidRecords: 'backup-and-skip'` and the unit can move documents aside, in + * which case the failing record is backed up, logged, and skipped); + * construct the domain. * * Lifecycle: the CALLER owns the returned handle and closes it via * `Domain.close()` (typically as its own `ctx.effect` disposer) — the @@ -118,7 +121,23 @@ export class DomainFacility { for (const [table, tableSpec] of Object.entries(spec.tables)) { const records = new Map() for (const [key, raw] of Object.entries(snapshot.tables[table] ?? {})) { - records.set(key, parseRecord(spec.name, table, key, () => tableSpec.valueSchema.parse(raw))) + let parsed: unknown + try { + parsed = parseRecord(spec.name, table, key, () => tableSpec.valueSchema.parse(raw)) + } catch (error) { + // Backup-and-skip policy (disposable derived data): move the record's + // document aside, log the concrete failure, and open without the + // record. Backends that cannot move a document keep the loud path. + if (spec.invalidRecords !== 'backup-and-skip' || unit.backupRecord === undefined) throw error + const moved = await unit.backupRecord(table, key) + // parseRecord always wraps the zod failure as the cause. + this.ctx.logger.error( + `domain '${spec.name}': stored record '${key}' in table '${table}' failed schema validation; ` + + `moved to '${moved}' and treated as absent. Cause: ${String((error as DomainError).cause)}`, + ) + continue + } + records.set(key, parsed) } tables.set(table, records) } diff --git a/packages/storage/storage-domain/src/spec.ts b/packages/storage/storage-domain/src/spec.ts index 74348918aa..f9267bcba5 100644 --- a/packages/storage/storage-domain/src/spec.ts +++ b/packages/storage/storage-domain/src/spec.ts @@ -45,6 +45,26 @@ export interface DomainSpec { * (a stale record document is discarded, never migrated). */ readonly layout?: 'single' | 'per-record' + /** + * Older domain versions whose stored records the current record schemas + * also accept (the declaring owner vouches for that, typically by + * declaring the fields older records lack as optional). `per-record` backends + * read documents stamped with a listed version instead of discarding them, + * and accept a legacy whole-unit file so stamped for the one-time + * bootstrap; writes always stamp {@link version}. + */ + readonly compatibleVersions?: readonly number[] + /** + * What `open` does with a stored table record that fails its zod schema. + * Absent (the default), the whole open rejects with `invalid-record` — + * right for authoritative data. `'backup-and-skip'` is for domains whose + * records are disposable derived data: the backend moves the record's + * document aside (`KvUnit.backupRecord`), the failure is logged with + * its cause, and the open continues with the record absent. A backend + * without `backupRecord` (no per-record document to move) falls back + * to the rejecting default. The global slot always rejects. + */ + readonly invalidRecords?: 'backup-and-skip' /** Optional global singleton slot. */ readonly global?: DomainGlobalSpec /** Table declarations keyed by table name; each name must match `UNIT_NAME_RE`. */ @@ -91,6 +111,13 @@ export function defineDomain(spec: S): S { if (!Number.isInteger(spec.version) || spec.version < 0) { throw new Error(`domain '${spec.name}' version must be a non-negative integer, got ${spec.version}`) } + for (const compat of spec.compatibleVersions ?? []) { + if (!Number.isInteger(compat) || compat < 0 || compat >= spec.version) { + throw new Error( + `domain '${spec.name}' compatibleVersions entries must be non-negative integers below version ${spec.version}, got ${compat}`, + ) + } + } if (spec.layout !== undefined) { // Runtime boundary: the union type is compile-time only — a spec built // from config could carry any value, and a bad one must fail loud here. @@ -99,6 +126,12 @@ export function defineDomain(spec: S): S { throw new Error(`domain '${spec.name}' layout must be 'single' or 'per-record', got ${layout}`) } } + if (spec.invalidRecords !== undefined) { + const policy: string = spec.invalidRecords + if (policy !== 'backup-and-skip') { + throw new Error(`domain '${spec.name}' invalidRecords must be 'backup-and-skip' when present, got ${policy}`) + } + } for (const table of Object.keys(spec.tables)) { if (!UNIT_NAME_RE.test(table)) { throw new Error(`domain '${spec.name}' table name '${table}' must match ${UNIT_NAME_RE}`) @@ -125,5 +158,6 @@ export function descriptorOf(spec: DomainSpec): KvUnitDescriptor { tables: Object.keys(spec.tables), hasGlobal: spec.global !== undefined, ...spec.layout === undefined ? {} : { layout: spec.layout }, + ...spec.compatibleVersions === undefined ? {} : { compatibleVersions: spec.compatibleVersions }, } } diff --git a/packages/storage/storage-domain/tests/domain.spec.ts b/packages/storage/storage-domain/tests/domain.spec.ts index 3b3c8fa679..b21ec0faf8 100644 --- a/packages/storage/storage-domain/tests/domain.spec.ts +++ b/packages/storage/storage-domain/tests/domain.spec.ts @@ -58,6 +58,25 @@ describe('defineDomain', () => { })).toThrow(/must not accept null/) }) + it('validates compatibleVersions entries and projects them onto the descriptor', () => { + expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [1.5], tables: {} })) + .toThrow(/compatibleVersions/) + expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [2], tables: {} })) + .toThrow(/below version/) + expect(() => defineDomain({ name: 'ok', version: 2, compatibleVersions: [-1], tables: {} })) + .toThrow(/compatibleVersions/) + expect(descriptorOf(defineDomain({ name: 'ok', version: 2, compatibleVersions: [0, 1], tables: {} }))) + .toMatchObject({ compatibleVersions: [0, 1] }) + // An undeclared set is absent from the descriptor. + expect(descriptorOf(spec)).not.toHaveProperty('compatibleVersions') + }) + + it('rejects an unknown invalidRecords policy', () => { + expect(() => defineDomain({ + name: 'ok', version: 1, invalidRecords: 'zap' as 'backup-and-skip', tables: {}, + })).toThrow(/invalidRecords/) + }) + it('rejects an invalid layout and projects the declared one onto the descriptor', () => { // A spec built from config can carry any value; the union type is // compile-time only, so the runtime boundary check must reject it. @@ -139,6 +158,28 @@ describe('DomainFacility.open', () => { }) }) + it('keeps the rejecting default under backup-and-skip when the backend cannot move documents', async () => { + // The memory backend has no backupRecord, so the declared policy cannot + // apply and the open falls back to failing loud. + const salvageSpec = defineDomain({ + name: 'salvage', + version: 1, + invalidRecords: 'backup-and-skip', + tables: { items: domainTable(itemSchema) }, + }) + const pool = new MemoryMediaPool() + { + const { facility } = await harness({ pool }) + await (await facility.open(salvageSpec)).table('items').put('bad', { label: 'x', count: 2 }) + } + pool.media.get('salvage')!.tables.get('items')!.set('bad', { label: 'x', count: 'NaN' }) + const { facility } = await harness({ pool }) + await expect(facility.open(salvageSpec)).rejects.toMatchObject({ + code: 'invalid-record', + detail: { table: 'items', key: 'bad' }, + }) + }) + it('rejects a stored global that fails its schema with the global marker', async () => { const pool = new MemoryMediaPool() pool.versions.set('demo', 1) diff --git a/packages/storage/storage-json/src/format.ts b/packages/storage/storage-json/src/format.ts index 045485eaba..22cbe8f703 100644 --- a/packages/storage/storage-json/src/format.ts +++ b/packages/storage/storage-json/src/format.ts @@ -100,16 +100,18 @@ export function serializeRecord(version: number, value: unknown): string { /** * Parse one per-record document, validating its version stamp. A document - * that is malformed or stamped with a different version is FOREIGN and reads - * as absent — the per-record contract: one bad or stale record file must not - * brick the whole unit, and a version bump discards stale records instead of - * migrating them (the whole-unit format rejects instead, because there is - * exactly one document). + * that is malformed or stamped with an unaccepted version is FOREIGN and + * reads as absent — the per-record contract: one bad or stale record file + * must not brick the whole unit, and a version bump discards stale records + * instead of migrating them (the whole-unit format rejects instead, because + * there is exactly one document). * @param text - Raw per-record document content. - * @param version - Expected unit version; a mismatch discards the document. + * @param versions - Accepted unit versions (the current one plus the + * descriptor's compatibleVersions); any other stamp discards the + * document. * @returns the record value, or `undefined` for a foreign document. */ -export function parseRecord(text: string, version: number): unknown { +export function parseRecord(text: string, versions: readonly number[]): unknown { let document: unknown try { document = JSON.parse(text) @@ -118,6 +120,6 @@ export function parseRecord(text: string, version: number): unknown { } if (typeof document !== 'object' || document === null) return undefined const { version: stamped, record } = document as Record - if (stamped !== version) return undefined + if (typeof stamped !== 'number' || !versions.includes(stamped)) return undefined return record } diff --git a/packages/storage/storage-json/src/per-record-unit.ts b/packages/storage/storage-json/src/per-record-unit.ts index b75f9f9452..346c7a0d6a 100644 --- a/packages/storage/storage-json/src/per-record-unit.ts +++ b/packages/storage/storage-json/src/per-record-unit.ts @@ -10,20 +10,23 @@ * memory unchanged. * * Per-record contract: a record document that is malformed or stamped with a - * different version reads as an absent record — one bad or stale file never - * bricks the whole unit, and a version bump discards stale records instead - * of migrating them. Record keys become path segments, so they must be - * path-safe (`[a-zA-Z0-9_-]+`); an unsafe key rejects at write. + * version outside the accepted set (the descriptor's current version plus + * its `compatibleVersions`) reads as an absent record — one bad or stale + * file never bricks the whole unit, and a version bump discards stale + * records instead of migrating them. Record keys become path segments, so + * they must be path-safe (`[a-zA-Z0-9_-]+`); an unsafe key rejects at write. * * Legacy bootstrap: when the new tree has no document path, a legacy * whole-unit file `/.json` (the pre-per-record layout) seeds - * per-record documents. Any new document path, including one whose contents - * are unreadable or stale, suppresses the bootstrap for the whole unit. The + * per-record documents, provided its stored unit version is in the accepted + * set — a legacy file stamped with any other version is left alone and reads + * as the empty unit. Any new document path, including one whose contents are + * unreadable or stale, suppresses the bootstrap for the whole unit. The * legacy file is never changed or deleted. * @module @deepseek-ai/dsh-storage-json/src/per-record-unit */ -import { mkdir, readFile, readdir, rm } from 'node:fs/promises' +import { mkdir, readFile, readdir, rename, rm } from 'node:fs/promises' import { dirname, join } from 'node:path' import type { Dirent } from 'node:fs' import { StorageError } from '@deepseek-ai/dsh-storage' @@ -64,6 +67,7 @@ export async function openPerRecordUnit( * @returns the authoritative state reconstructed from the tree. */ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Promise { + const versions = acceptedStamps(descriptor) const state: UnitState = { version: descriptor.version, global: null, @@ -83,11 +87,11 @@ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Pr if (entry.isDirectory()) { const records = state.tables.get(entry.name) if (records !== undefined) { - return loadTableRecords(records, descriptor.version, join(dir, entry.name)) + return loadTableRecords(records, versions, join(dir, entry.name)) } } if (entry.name === 'global.json' && descriptor.hasGlobal) { - const global = await readRecord(join(dir, entry.name), descriptor.version) + const global = await readRecord(join(dir, entry.name), versions) if (global !== undefined) state.global = global return true } @@ -97,12 +101,21 @@ async function loadPerRecordState(descriptor: KvUnitDescriptor, dir: string): Pr return state } +/** The version stamps this unit reads as its own: current plus declared compatible versions. */ +function acceptedStamps(descriptor: KvUnitDescriptor): readonly number[] { + return [descriptor.version, ...descriptor.compatibleVersions ?? []] +} + /** * Bootstrap an empty per-record tree from a legacy whole-unit file * (`/.json`, the pre-per-record layout). Every declared-table * record is copied into a current-version document, while the legacy file is * retained unchanged. A missing, foreign (another unit's name), malformed, - * or non-unit legacy file is left alone; other read failures propagate. + * or non-unit legacy file is left alone, and so is one whose stored unit + * version is outside the accepted set — migrating records the owner never + * vouched for would stamp them with the current version and turn a + * discardable stale cache into schema failures at the domain layer. Other + * read failures propagate. * @param descriptor - Static identity and shape of the unit. * @param dir - The per-record unit directory (`/`). * @param state - The empty tree state; bootstrapped records are added. @@ -116,16 +129,18 @@ async function bootstrapLegacyUnit(descriptor: KvUnitDescriptor, dir: string, st if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error return } - // The legacy document is runtime data: only `unit.name` and the tables map - // shape are checked here — the record values are migrated as-is and the - // domain layer's schemas judge them. - let document: { unit?: { name?: unknown }; tables?: unknown } + // The legacy document is runtime data: only `unit.name`, `unit.version`, + // and the tables map shape are checked here — the record values are + // migrated as-is and the domain layer's schemas judge them. + let document: { unit?: { name?: unknown; version?: unknown }; tables?: unknown } try { - document = JSON.parse(text) as { unit?: { name?: unknown }; tables?: unknown } + document = JSON.parse(text) as { unit?: { name?: unknown; version?: unknown }; tables?: unknown } } catch { return // Malformed legacy file: not ours to interpret or delete. } if (document.unit?.name !== descriptor.name) return + const stamped = document.unit.version + if (typeof stamped !== 'number' || !acceptedStamps(descriptor).includes(stamped)) return const tables = document.tables if (typeof tables !== 'object' || tables === null) return const recordsByTable = tables as Record> @@ -146,14 +161,14 @@ async function bootstrapLegacyUnit(descriptor: KvUnitDescriptor, dir: string, st * @returns whether the directory contains any `.json` document path, * independently of key safety, readability, or stored version. */ -async function loadTableRecords(records: Map, version: number, dir: string): Promise { +async function loadTableRecords(records: Map, versions: readonly number[], dir: string): Promise { const files = await readdir(dir, { withFileTypes: true }) const hasDocuments = files.some(file => file.name.endsWith('.json')) const loaded = await Promise.all(files.map(async (file) => { if (!file.name.endsWith('.json')) return const key = file.name.slice(0, -'.json'.length) if (!SAFE_KEY_RE.test(key)) return - const record = await readRecord(join(dir, file.name), version) + const record = await readRecord(join(dir, file.name), versions) if (record !== undefined) return [key, record] as const })) for (const record of loaded) { @@ -163,9 +178,9 @@ async function loadTableRecords(records: Map, version: number, } /** Read one record document; a foreign (unreadable or stale) one reads as absent. */ -async function readRecord(path: string, version: number): Promise { +async function readRecord(path: string, versions: readonly number[]): Promise { try { - return parseRecord(await readFile(path, 'utf8'), version) + return parseRecord(await readFile(path, 'utf8'), versions) } catch { return undefined } @@ -213,6 +228,22 @@ export class PerRecordJsonUnit implements KvUnit { await this.tracked(rm(join(this.tableDir(table), `${key}.json`), { force: true })) } + /** + * Move one record's document aside as `.json.bak.`. The + * moved file no longer ends in `.json`, so every later read ignores it; the + * bytes stay on disk for inspection. A same-minute backup of the same + * key overwrites the previous backup (the newer bytes are the ones worth + * keeping). + */ + async backupRecord(table: string, key: string): Promise { + this.assertOpen() + assertSafeKey(this.descriptor.name, key) + const path = join(this.tableDir(table), `${key}.json`) + const moved = `${path}.bak.${backupStamp(new Date())}` + await this.tracked(rename(path, moved)) + return moved + } + /** Durably replace the global singleton. Only valid when declared. */ async setGlobal(value: unknown): Promise { this.assertOpen() @@ -267,6 +298,12 @@ export class PerRecordJsonUnit implements KvUnit { } } +/** Local-time `YYYYMMDDHHmm` suffix for backed-up documents. */ +function backupStamp(now: Date): string { + const pad = (value: number): string => String(value).padStart(2, '0') + return `${String(now.getFullYear())}${pad(now.getMonth() + 1)}${pad(now.getDate())}${pad(now.getHours())}${pad(now.getMinutes())}` +} + /** Reject a record key that would be unsafe as a path segment. */ function assertSafeKey(unit: string, key: string): void { if (!SAFE_KEY_RE.test(key)) { diff --git a/packages/storage/storage-json/tests/json-backend.spec.ts b/packages/storage/storage-json/tests/json-backend.spec.ts index bafbf92ef4..57ad9d7633 100644 --- a/packages/storage/storage-json/tests/json-backend.spec.ts +++ b/packages/storage/storage-json/tests/json-backend.spec.ts @@ -329,10 +329,10 @@ describe('per-record layout', () => { it('bootstraps an empty per-record tree from a legacy whole-unit file and preserves it', async () => { const root = await freshRoot() - // A legacy single-layout file for the same unit (any older version); - // the extra table is not declared and must be skipped. + // A legacy single-layout file for the same unit, stamped with the current + // version; the extra table is not declared and must be skipped. const legacy = JSON.stringify({ - unit: { name: 'recs', version: 3 }, + unit: { name: 'recs', version: 2 }, global: null, tables: { t: { old1: { v: 1 }, old2: { v: 2 } }, undeclared: { k: { v: 0 } } }, }) @@ -347,6 +347,75 @@ describe('per-record layout', () => { await backend.close() }) + it('bootstraps from a legacy file only when its stored version is accepted', async () => { + // Version 3 is neither current (2) nor declared compat: the legacy file + // is left alone and the unit reads empty — migrating unvouched records + // would stamp them current and surface as schema failures at the domain + // layer instead of a discardable stale cache. + const root = await freshRoot() + const legacy = JSON.stringify({ + unit: { name: 'recs', version: 3 }, + global: null, + tables: { t: { old: { v: 1 } } }, + }) + await writeFile(join(root, 'recs.json'), legacy, 'utf8') + const backend = new JsonStorageBackend(root) + const unit = await backend.kv.open(descriptor) + expect(await unit.loadAll()).toEqual({ tables: { t: {} }, global: null }) + await expect(readFile(join(root, 'recs.json'), 'utf8')).resolves.toBe(legacy) + await unit.close() + await backend.close() + + // The same file bootstraps once version 3 is declared read-compatible… + const root2 = await freshRoot() + await writeFile(join(root2, 'recs.json'), legacy, 'utf8') + const backend2 = new JsonStorageBackend(root2) + const compat = { ...descriptor, version: 4, compatibleVersions: [3] } + const unit2 = await backend2.kv.open(compat) + expect(await unit2.loadAll()).toEqual({ tables: { t: { old: { v: 1 } } }, global: null }) + // …and the migrated documents are stamped with the CURRENT version. + expect(JSON.parse(await readFile(join(root2, 'recs', 't', 'old.json'), 'utf8'))) + .toEqual({ version: 4, record: { v: 1 } }) + await unit2.close() + await backend2.close() + }) + + it('backupRecord moves the document aside; reads see it absent and a write recreates it', async () => { + const root = await freshRoot() + const backend = new JsonStorageBackend(root) + const unit = await backend.kv.open(descriptor) + await unit.putRecord('t', 'k', { v: 1 }) + const moved = await unit.backupRecord!('t', 'k') + expect(moved).toMatch(/k\.json\.bak\.\d{12}$/) + expect(JSON.parse(await readFile(moved, 'utf8'))).toEqual({ version: 2, record: { v: 1 } }) + await expect(readFile(recordPath(root, 'k'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' }) + // The moved file no longer ends in .json, so it reads as absent… + expect(await unit.loadAll()).toEqual({ tables: { t: {} }, global: null }) + // …and the key is free for a fresh write. + await unit.putRecord('t', 'k', { v: 2 }) + expect(await unit.loadAll()).toEqual({ tables: { t: { k: { v: 2 } } }, global: null }) + await expect(unit.backupRecord!('t', 'a/b')).rejects.toThrow(/not path-safe/) + await unit.close() + await expect(unit.backupRecord!('t', 'k')).rejects.toMatchObject({ code: 'closed' }) + await backend.close() + }) + + it('reads per-record documents stamped with a declared compat version and stamps writes current', async () => { + const root = await freshRoot() + const backend = new JsonStorageBackend(root) + const compat = { ...descriptor, compatibleVersions: [1] } + await mkdir(join(root, 'recs', 't'), { recursive: true }) + await writeFile(recordPath(root, 'oldrec'), JSON.stringify({ version: 1, record: { v: 'old' } }), 'utf8') + await writeFile(recordPath(root, 'ancient'), JSON.stringify({ version: 0, record: { v: 'no' } }), 'utf8') + const unit = await backend.kv.open(compat) + // Version 1 is declared compat and served; version 0 is not and discards. + expect(await unit.loadAll()).toEqual({ tables: { t: { oldrec: { v: 'old' } } }, global: null }) + await unit.putRecord('t', 'oldrec', { v: 'new' }) + expect(JSON.parse(await readFile(recordPath(root, 'oldrec'), 'utf8'))) + .toEqual({ version: 2, record: { v: 'new' } }) + await backend.close() + }) + it('ignores the legacy whole-unit file when any new document path exists', async () => { const root = await freshRoot() const legacy = JSON.stringify({ @@ -400,7 +469,8 @@ describe('per-record layout', () => { await backend4.close() const root5 = await freshRoot() - await writeFile(join(root5, 'recs.json'), JSON.stringify({ unit: { name: 'recs' }, tables: 'not an object' }), 'utf8') + // A current-version stamp so the shapeless `tables` is what stops the bootstrap. + await writeFile(join(root5, 'recs.json'), JSON.stringify({ unit: { name: 'recs', version: 2 }, tables: 'not an object' }), 'utf8') const backend5 = new JsonStorageBackend(root5) const unit5 = await backend5.kv.open(descriptor) expect(await unit5.loadAll()).toEqual({ tables: { t: {} }, global: null }) diff --git a/packages/storage/storage/src/backend.ts b/packages/storage/storage/src/backend.ts index 9cba748655..4a8e0ac2d8 100644 --- a/packages/storage/storage/src/backend.ts +++ b/packages/storage/storage/src/backend.ts @@ -61,6 +61,16 @@ export interface KvUnitDescriptor { * foreign documents. */ readonly layout?: 'single' | 'per-record' + /** + * Older unit versions whose stored records are also readable under the + * declaring owner's current record schemas (the owner vouches for that — + * typically by declaring the fields old records lack as optional). Reads of + * a `per-record` unit accept documents stamped with any listed version, and + * the legacy whole-unit bootstrap accepts a legacy file stamped with one; + * writes always stamp {@link version}. `single`-layout reads stay + * exact-version. + */ + readonly compatibleVersions?: readonly number[] } /** @@ -99,6 +109,19 @@ export interface KvUnit { */ deleteRecord(table: string, key: string): Promise + /** + * Move one record's stored document out of the unit's readable set, + * preserving its bytes for inspection instead of deleting them. Backends + * whose medium has no per-record document to move (the `single` layout, a + * row store) omit this member, and the caller falls back to its + * reject-loud path. Absent after the move: a later {@link loadAll} reads + * the key as missing and a later {@link putRecord} recreates it fresh. + * @param table - Declared table name. + * @param key - Record key. + * @returns the medium location the document was moved to (diagnostics). + */ + backupRecord?(table: string, key: string): Promise + /** * Write the global singleton durably. Only valid when the descriptor * declared `hasGlobal`.