bec6805d6a
The persistence seam is now create/open/stat/list returning per-session SessionHandles (read/append/flush/close); every log read and write flows through the owning handle. The seam package exports only the service and handle contracts, consumer-visible errors, and pure durable-data validation helpers; each backend owns its complete storage runtime, and the shared contract suites pin equivalent observable behavior. The backend routes published sessions' live events by id into the active write handle; agent-loop only acquires, seeds, and closes the handle. Resume appends interruptedTurnClosers through its write handle; session-query owns the revision-keyed cold cache. Legacy-only surfaces are removed in the same swap: locate/readRaw/supportsRawArtifacts, the legacy event-shape read migration, zstd torn-frame salvage, DSH_SESSION_JSONL, and hook transcript_path population; a torn final zstd frame is discarded whole; the session-list cold blank probe returns on stat metadata (eventCount derived from the last physical row, sizeBytes). The WebUI ZIP export serializes the logical log from a read handle, so both backends export identically. Refs #3245
36 lines
7.7 KiB
Markdown
36 lines
7.7 KiB
Markdown
# Agent Note: Session persistence as an abstract service over the existing `SessionEvent`
|
|
|
|
Status: implemented
|
|
|
|
English | [中文](2026-06-14-session-persistence.zh.md)
|
|
|
|
## Problem
|
|
|
|
Sessions lived only in memory. The example `session-jsonl.ts` plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered `session/event` and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume, durable forking, and host-side session browsing were all impossible.
|
|
|
|
The [event-sourced model](2026-06-11-event-sourced-sessions.md) makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing `SessionEvent` directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.
|
|
|
|
## Decision
|
|
|
|
Persistence is a **capability seam** with an abstract Service Definition ([capability seams](2026-06-13-capability-seams.md), the `dsh-shell` template), not loop or core logic:
|
|
|
|
1. **Interface** (`dsh-session-persistence`, `ctx.sessionPersistence`) — an abstract `SessionPersistence` service: `create`/`open`/`stat`/`list`/`export`, with `create`/`open` returning per-session `SessionHandle`s that carry `read`/`append`/`flush`/`close` ([handle-based seam](2026-08-27-handle-based-session-persistence.md)). Its persisted unit IS the existing `SessionEvent` (`{ type, seq, time, data }`), reused verbatim — no conversion type.
|
|
2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only logical JSONL log per session: a `SessionHeader` line followed by storage records that losslessly represent the contiguous `SessionEvent` stream. Eligible `assistant/chunk` delta runs use packed rows by default; [checksummed Zstandard frames](2026-07-19-zstandard-jsonl-session-logs.md) are the default physical encoding, with raw lines configurable.
|
|
|
|
Key durable, contested choices:
|
|
|
|
- **The canonical durable log persists every `SessionEvent` losslessly, including `assistant/chunk`.** JSONL storage may encode a consecutive delta run as one packed row, but logical readers reconstruct the exact event boundaries, sequence numbers, and timestamps. `deriveMessages()` skips chunks, and a chunk-filtered rollout (Codex's `policy.rs`) is tempting — but `seq = log.length` and validation of `events[i].seq === i` require a *contiguous* logical log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
|
|
- **Append-only; a crashed turn is closed, never truncated.** Flushed events are never rewritten. The [semantic checkpoint policy](../bug-fix/2026-07-21-semantic-session-checkpoints.md) drains the request before model dispatch, a recorded top-level call before tool dispatch, and the complete response/result batch after a step; the loop drains the final turn boundary. Because one interrupted turn may contain substantial valid work, persistence returns its contiguous, parseable events unmodified; the reader owns balancing — resume computes risk-classified error results for unanswered assistant calls, a missing `step/end`, and `turn/end` with `{ kind: 'interrupted' }` (`interruptedTurnClosers`) and appends them through its write handle, while read-only observers add the same closers in memory. The synthetic results keep resumed provider transcripts valid. Only the incomplete fragment of a torn final append is discarded — complete records recovered from it are durably rewritten by the write path before its first new append; a parse error or sequence gap in the committed prefix is corruption and makes the session unloadable.
|
|
- **The file backend is canonical while the service remains extensible.** `dsh-session-persistence-jsonl` is the sole first-party provider and passes `runPersistenceContract`; the abstract service remains available to out-of-tree providers. The [JSONL-only persistence decision](../simplification/2026-08-30-jsonl-only-session-persistence.md) owns removal of the first-party database provider and its deliberate compatibility cut.
|
|
- **Metadata is out-of-log.** Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a `SessionHeader` owned by `dsh-session` and attached to a `Session` via a new readonly `session.header` — never in `SessionEventMap`, never reaching `deriveMessages()`. `createdAt` is non-negative safe-integer Unix epoch milliseconds: live creation and persistence registration reject fractional values, and JSONL validates the decoded header. The alternative (a merge-extensible `session/meta` event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header boundary is the cleaner cost. (The header was originally split into an immutable `SessionHeader` plus a mutable `SessionSummary` whose union was `SessionMeta`; the mutable summary was later removed as dead state — see [Drop the mutable session summary](../simplification/2026-06-19-drop-mutable-session-summary.md).)
|
|
- **`ctx.agents.create()` and `ctx.agents.resume()` are async factories; resume additionally crosses the persistence boundary.** `ctx.agents.resume({ resumeSessionId })` opens the session's write handle, reads the stored log, and publishes the prepared Session under the persisted id, continuing its projections. The [Session preparation decision](2026-08-05-session-preparation.md) owns the unpublished-Session ownership window. The agent-loop does NOT hard-inject `sessionPersistence` (that would pend non-persistent demos forever); `resume` rejects with a clear error when it is absent.
|
|
|
|
## Alternatives considered
|
|
|
|
Each key choice above records its rejected alternative where the choice is stated: a **chunk-filtered canonical log** (Codex's `policy.rs` shape) — breaks the contiguous-seq contract; **truncating a crashed turn** — silently destroys a long autonomous run's real work; an **in-log `session/meta` event as log line 0** — metadata is not replayable state; **finite fractional `createdAt` values** — have no producer and diverge from integer Unix-millisecond storage; **hard-injecting `sessionPersistence` into the loop** — would pend non-persistent demos forever.
|
|
|
|
Format versioning: the header carries a `version`; cold reads reject any non-current version. The pre-release session format stays pinned at `SESSION_FORMAT_VERSION = 0` and carries no compatibility promise: reads validate current v0 records only, and retired same-version shapes refuse fail-closed ([export and pre-release trims](../simplification/2026-08-27-persistence-export-and-pre-release-trims.md)). Append-only + flush is robust to partial trailing writes (tolerated during cold preparation) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option there.
|
|
|
|
## Consequences
|
|
|
|
The Service Definition, JSONL provider, and metadata contract in `dsh-session` (`session.header`, the `create(id?, options?)` signature) buy durable resume/fork, a read/replay path, crash tolerance, and host-side session access over the existing event-sourced log. The reusable `runPersistenceContract` suite holds the provider and future implementations to the same append-only, contiguous-seq, lazy-materialization, logical-recovery, integer-metadata, and serializability semantics. Persisting the full logical log also settles event fidelity: every `assistant/chunk` survives exactly even when JSONL packs several into one storage row.
|