deepseek-harness/packages/session/session-persistence-sqlite/README.md

63 lines
8.3 KiB
Markdown
Raw Normal View History

feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
# @deepseek-ai/dsh-session-persistence-sqlite
2026-07-26 05:03:53 +08:00
English | [中文](README.zh.md)
A SQLite durable session-persistence backend — a second `SessionPersistence` provider ([session persistence](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)) satisfying the same contract as `dsh-session-persistence-jsonl` (append-only, contiguous-seq, lazy materialization, interrupted-turn close on load), expressed over `node:sqlite` rows instead of file bytes.
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
`locate(meta)` returns `undefined`: all sessions share one database, so there is no honest independent per-session transcript path.
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
## Storage model
Each `SessionEvent` maps 1:1 onto a row in an `events` table `(session_id, seq, type, time, data, source_event_seqs, surface_op)` — `data` is the event payload as JSON text, so the row shape is the event verbatim (including `assistant/chunk`, keeping `seq` contiguous). The two `TEXT` columns `source_event_seqs` and `surface_op` are nullable; they store the event's optional surface-metadata fields (see [session surface](../../../.agents/notes/implemented/architecture/2026-06-18-session-surface.md)). Out-of-log metadata (`SessionHeader`), a per-materialization incarnation id, and a monotonic per-log revision live in a `sessions` row; `createdAt` is a non-negative safe integer stored in a strict `INTEGER` column. A singleton state row carries the immutable store id. A `sessions` row is written only by the first `append` — its existence is the lazy-materialization signal (`list` reports exactly the sessions that have a row).
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
The repository's Node range supports unflagged `node:sqlite`. The database enables foreign keys and uses the configured journal mode (`wal` by default; use a rollback mode where WAL shared-memory files are unsuitable). `PRAGMA application_id` identifies the canonical persistence database, and `PRAGMA user_version` stores its layout version. A fresh database must have no application identity or user-defined schema objects; initialization creates every table and stamps both pragmas in one transaction. Non-pristine unversioned databases, foreign application identities, and every non-current version reject before journal-mode mutation because this unreleased format has no migrations.
On filesystems with POSIX modes, the backend requests mode `0700` for missing directories and exclusively creates a missing database with mode `0600` before SQLite opens it; the process umask may further restrict both. New WAL, shared-memory, and persistent rollback-journal sidecars receive the database's resulting owner-only mode. Existing directories, database files, and sidecars keep their modes; filesystem setup errors other than an existing database fail initialization. These defaults prevent incidental exposure through a permissive process umask, but do not protect database confidentiality or integrity when another principal can replace the database entry in its parent directory.
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
## Contract semantics over rows
- **Append = a transaction.** `append` runs `BEGIN`/`COMMIT` around the batch: it materializes the `sessions` row (if still lazy) and INSERTs every event, asserting the contiguous-seq contract first (the first event's `seq` must equal the stored next-seq). A mid-batch failure (a UNIQUE violation on a duplicated seq) rolls back entirely, so the stored log and the in-memory cursor stay consistent. (`load()` already balanced the stored log, so `append` never has to repair a crash tail.)
- **Lazy materialization.** `create()` records intent in memory only — no row is written until the first `append`. A created-but-never-appended session has no `sessions` row, so it is absent from `list()` (which reports exactly the sessions that have a row).
2026-07-19 22:50:49 +08:00
- **Interrupted-turn close on load.** `load()` implements the shared [crash-recovery contract](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md): preserve the valid interrupted turn, append its synthetic closing events in one transaction, and remove only a torn tail row. Committed parse errors or sequence gaps make the session unloadable. Because recovery mutates stored rows, the next append starts from a balanced log and accurate cursor.
- **Non-mutating inspection.** `inspect()` returns an immutable balanced logical view and may synthesize recovery closers in memory, without deleting a torn tail row, appending recovery rows, or changing the lightweight revision.
- **Lightweight revisions.** `listSnapshots(signal?)` combines the immutable store and database-file identity, a per-materialization incarnation id, and a per-session counter incremented in each mutating transaction. A full-prefix read captures that revision and its event rows in one read transaction, while `readStoredRevision()` queries only the session row to validate retained preparations. This keeps unchanged observations stable without parsing event rows and distinguishes independent stores and recreated same-id logs. It checks cancellation before and after shared readiness and the synchronous metadata query; the query itself is non-preemptible.
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
## Configuration (schemastery)
```ts
interface Config {
path: string // SQLite database file path, or ':memory:' for an in-process DB
Expose audited hardcoded tunables as plugin config The audit swept every packages/*/* plugin for the new AGENTS.md convention (no hardcoded tunables in plugins) and exposes each finding as a defaulted, validated Config field. Defaults are the previously hardcoded values throughout, so no deployment or golden changes. - tool-fs (had NO Config): readLimit, readMaxLineLength, readMaxBytes, readStreamMinSize. The caps thread through ReadToolCaps/ReadWindow — read-render already documented that the consumer applies the caps, so they become explicit per-request fields. - tool-web: searchMaxResults (WEB_SEARCH_MAX_RESULTS stays as the schemastery default). Also fixes the stale GREP_LIMIT references in search.ts and the web-capability-seam RFC (no such constant exists). - bash-local: graceMs (SIGTERM->SIGKILL escalation grace). The RunInternals.graceMs test seam is gone: graceMs is now a required SpawnSpec field filled from config, so tests exercise the real config path and the defaults live in exactly one place. - subagent-acp: disposeEofGraceMs / disposeGraceMs. The AcpRunSpec fields become required for the same one-defaulting-layer reason. - session-persistence-sqlite: journalMode ('wal' default; the rollback-journal modes serve filesystems where WAL's shared-memory files do not work, e.g. network mounts). - hooks-claude + hooks-codex: stderrSummaryMaxChars for the persisted hook/result stderr summary. The duplicated summarize() helpers merge into hook-protocol's summarizeStderr(stderr, maxChars), beside the HookResultRecord field it feeds, with the bound parameterized the same way runHook's defaultTimeoutMs already is. - compact-basic: charsPerToken for the token estimator (default 4, the English-text heuristic; CJK-heavy deployments need ~1-2 or compaction fires far too late). Also corrects the BasicCompactService class doc, which claimed defaults the required-field config never had. - fs-local: deletes the dead STREAM_MIN_SIZE constant and the dead FsIoInternals.streamMinSize seam — the read-routing bound lives in the consumer (tool-fs), where it is now config. This is item 1 of the proposed prune-write-only-fs-surface RFC, annotated accordingly. Every new field gets range validation (following the existing assertPositiveFinite pattern), a README row, and tests covering the configured behavior, the schema default, and load-time rejection.
2026-07-04 17:37:23 +08:00
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist' // journal_mode pragma; default 'wal'
preparedSessionCacheSize?: number // positive integer; default 5
writeBatchMaxDelayMs?: number // positive integer; default 200; maximum 2_147_483_647
feat(session-persistence-sqlite): second backend validating the abstraction Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics".
2026-06-15 21:45:21 +08:00
}
```
## Write path
Like the JSONL backend, the plugin copies each frozen `session/event` into one controller per live session. The first pending event starts the configured fixed batching window, and later events join without resetting it. Expiry starts one transaction; events admitted during that write form a separately bounded follow-up batch. `session/flush` cancels the wait and drains current and pending batches. The controller persists a fork's seed once, keeps a write cursor so resume never re-appends stored events, and seeds live sessions on apply because HMR does not replay `session/created`. Dispose drains every retained controller before closing the database. Every event remains a separate SQLite row; batching only groups more INSERTs into one transaction and revision increment.
Add a gated Known Limitations and Deferred Work section to every package README Every packages/*/* README now carries a canonical '## Known Limitations and Deferred Work' section: condensed, evidence-backed bullets for consumer-visible gaps (unimplemented features, platform caveats, MVP cuts) and consciously postponed work (TODO/FIXME/XXX markers, RFC deferrals still open). The ten pre-existing ad-hoc variants ('What is NOT here (TODO)', 'Deferred', 'Limitations (MVP)', 'Known limitations (tracked TODOs)', ...) are normalized into the canonical heading. A new doc-sync gate, scripts/verify-readme-limitations.ts, enforces the shape: exactly one limitations-like heading per package README, byte-equal to the canonical h2, with at least one bullet; near-miss headings fail so variants cannot creep back. Packages with genuinely nothing to declare (dsh-brand, dsh-timeout, dsh-subagent-mock, dsh-app-boot) are whitelisted in the script and must NOT carry the section; whitelist entries are validated against the scanned package set so a rename fails loud. Wired into the doc-sync chain (package.json) and the run-gates doc-sync leaf set; the standing rule lands in packages/AGENTS.md and the adding-a-package cookbook; decision record in docs/rfc/implemented/process/2026-07-10-readme-known-limitations-gate.md (RFC index regenerated). Also fixes two stale '(deferred)' markers claiming dsh-compact-basic is unimplemented (the dsh-compact seam README's package table and the seam's module doc comment).
2026-07-10 01:51:50 +08:00
2026-07-12 02:55:26 +08:00
## Model Experience
### Resumed conversation history
#### What the model sees
SQLite storage contributes no live prompt or schema. Loading restores the same surface history as JSONL and preserves prior headers for reconstruction; the new loop composes its current envelope. Recovery balances an assistant request without a durable call with `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, which tells the model to retry only read-only or idempotent work and to verify possible side effects or ask the user. Row metadata and raw chunks are not messages.
2026-07-12 02:55:26 +08:00
#### Token effect
Zero live-request tokens. Resume restores retained history and pays the current envelope, plus the quoted repair result for each interrupted call.
#### KV Cache effect
SQLite storage does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append.
Add a gated Known Limitations and Deferred Work section to every package README Every packages/*/* README now carries a canonical '## Known Limitations and Deferred Work' section: condensed, evidence-backed bullets for consumer-visible gaps (unimplemented features, platform caveats, MVP cuts) and consciously postponed work (TODO/FIXME/XXX markers, RFC deferrals still open). The ten pre-existing ad-hoc variants ('What is NOT here (TODO)', 'Deferred', 'Limitations (MVP)', 'Known limitations (tracked TODOs)', ...) are normalized into the canonical heading. A new doc-sync gate, scripts/verify-readme-limitations.ts, enforces the shape: exactly one limitations-like heading per package README, byte-equal to the canonical h2, with at least one bullet; near-miss headings fail so variants cannot creep back. Packages with genuinely nothing to declare (dsh-brand, dsh-timeout, dsh-subagent-mock, dsh-app-boot) are whitelisted in the script and must NOT carry the section; whitelist entries are validated against the scanned package set so a rename fails loud. Wired into the doc-sync chain (package.json) and the run-gates doc-sync leaf set; the standing rule lands in packages/AGENTS.md and the adding-a-package cookbook; decision record in docs/rfc/implemented/process/2026-07-10-readme-known-limitations-gate.md (RFC index regenerated). Also fixes two stale '(deferred)' markers claiming dsh-compact-basic is unimplemented (the dsh-compact seam README's package table and the seam's module doc comment).
2026-07-10 01:51:50 +08:00
## Known Limitations and Deferred Work
- **`DatabaseSync` is synchronous** — every append transaction blocks the event loop for its duration; acceptable for local stores, a throughput ceiling for busy multi-session servers.
2026-07-13 14:51:00 +08:00
- **Write contention has no wait or retry policy** — the backend sets no busy timeout and retries no locked-database error, so another connection holding a write transaction makes the operation reject immediately.
- **Only a pristine new database or the current owned `SCHEMA_VERSION` opens** — unversioned schema objects, foreign application identities, and every other schema version are rejected rather than migrated (unreleased software; no persisted user data to preserve).
- **Nothing deletes stored sessions** — rows accumulate until removed externally (the seam has no deletion API; `ON DELETE CASCADE` is wired for such out-of-band cleanup).
- **TODO:** this backend talks to `node:sqlite` directly. If a cordis database service (`cordis/db` / a `@cordisjs` SQL driver plugin) is adopted, route through that instead of holding a raw `DatabaseSync` here — the contract surface (`SessionPersistence`) would not change, only the storage driver.