deepseek-harness/.agents/notes/implemented/architecture/2026-06-14-session-persistence.md

Agent Note: Session persistence as an abstract service over the existing SessionEvent

Status: implemented

English | 中文

Problem

Sessions lived only in memory. The example session-jsonl.ts plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered session/event and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume, durable forking, and host-side session browsing were all impossible.

The event-sourced model makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing SessionEvent directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.

Decision

Persistence is a capability seam with an abstract Service Definition (capability seams, the dsh-shell template), not loop or core logic:

1. Interface (dsh-session-persistence, ctx.sessionPersistence) — an abstract SessionPersistence service: create/open/stat/list/export, with create/open returning per-session SessionHandles that carry read/append/flush/close (handle-based seam). Its persisted unit IS the existing SessionEvent ({ type, seq, time, data }), reused verbatim — no conversion type.
2. Implementation (dsh-session-persistence-jsonl) — an append-only logical JSONL log per session: a SessionHeader line followed by storage records that losslessly represent the contiguous SessionEvent stream. Eligible assistant/chunk delta runs use packed rows by default; checksummed Zstandard frames are the default physical encoding, with raw lines configurable.

Key durable, contested choices:

• The canonical durable log persists every SessionEvent losslessly, including assistant/chunk. JSONL storage may encode a consecutive delta run as one packed row, but logical readers reconstruct the exact event boundaries, sequence numbers, and timestamps. deriveMessages() skips chunks, and a chunk-filtered rollout (Codex's policy.rs) is tempting — but seq = log.length and validation of events[i].seq === i require a contiguous logical log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
• Append-only; a crashed turn is closed, never truncated. Flushed events are never rewritten. The semantic checkpoint policy drains the request before model dispatch, a recorded top-level call before tool dispatch, and the complete response/result batch after a step; the loop drains the final turn boundary. Because one interrupted turn may contain substantial valid work, persistence returns its contiguous, parseable events unmodified; the reader owns balancing — resume computes risk-classified error results for unanswered assistant calls, a missing step/end, and turn/end with { kind: 'interrupted' } (interruptedTurnClosers) and appends them through its write handle, while read-only observers add the same closers in memory. The synthetic results keep resumed provider transcripts valid. Only the incomplete fragment of a torn final append is discarded — complete records recovered from it are durably rewritten by the write path before its first new append; a parse error or sequence gap in the committed prefix is corruption and makes the session unloadable.
• The file backend is canonical while the service remains extensible. dsh-session-persistence-jsonl is the sole first-party provider and passes runPersistenceContract; the abstract service remains available to out-of-tree providers. The JSONL-only persistence decision owns removal of the first-party database provider and its deliberate compatibility cut.
• Metadata is out-of-log. Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a SessionHeader owned by dsh-session and attached to a Session via a new readonly session.header — never in SessionEventMap, never reaching deriveMessages(). createdAt is non-negative safe-integer Unix epoch milliseconds: live creation and persistence registration reject fractional values, and JSONL validates the decoded header. The alternative (a merge-extensible session/meta event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header boundary is the cleaner cost. (The header was originally split into an immutable SessionHeader plus a mutable SessionSummary whose union was SessionMeta; the mutable summary was later removed as dead state — see Drop the mutable session summary.)
• ctx.agents.create() and ctx.agents.resume() are async factories; resume additionally crosses the persistence boundary. ctx.agents.resume({ resumeSessionId }) opens the session's write handle, reads the stored log, and publishes the prepared Session under the persisted id, continuing its projections. The Session preparation decision owns the unpublished-Session ownership window. The agent-loop does NOT hard-inject sessionPersistence (that would pend non-persistent demos forever); resume rejects with a clear error when it is absent.

Alternatives considered

Each key choice above records its rejected alternative where the choice is stated: a chunk-filtered canonical log (Codex's policy.rs shape) — breaks the contiguous-seq contract; truncating a crashed turn — silently destroys a long autonomous run's real work; an in-log session/meta event as log line 0 — metadata is not replayable state; finite fractional createdAt values — have no producer and diverge from integer Unix-millisecond storage; hard-injecting sessionPersistence into the loop — would pend non-persistent demos forever.

Format versioning: the header carries a version; cold reads reject any non-current version. The pre-release session format stays pinned at SESSION_FORMAT_VERSION = 0 and carries no compatibility promise: reads validate current v0 records only, and retired same-version shapes refuse fail-closed (export and pre-release trims). Append-only + flush is robust to partial trailing writes (tolerated during cold preparation) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option there.

Consequences

The Service Definition, JSONL provider, and metadata contract in dsh-session (session.header, the create(id?, options?) signature) buy durable resume/fork, a read/replay path, crash tolerance, and host-side session access over the existing event-sourced log. The reusable runPersistenceContract suite holds the provider and future implementations to the same append-only, contiguous-seq, lazy-materialization, logical-recovery, integer-metadata, and serializability semantics. Persisting the full logical log also settles event fidelity: every assistant/chunk survives exactly even when JSONL packs several into one storage row.