deepseek-harness/docs/core-data-structures/persistence.md

102 lines
6.9 KiB
Markdown
Raw Normal View History

# Session Persistence
The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log. The event vocabulary the log carries is enumerated, member by member, in the generated [persistence log event catalog](../persistence-catalog.md).
2026-07-19 22:50:49 +08:00
The seam is a textbook [capability seam](../../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence/session-persistence), `ctx.sessionPersistence`) defining locate/create/append/load/list over the existing `SessionEvent` — **no parallel persisted type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence Agent Note](../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md).
## The flush checkpoint
`session/event` is a *synchronous* notification; persistence plugins buffer it (write-behind) until `session/flush`. The loop awaits an ordinary turn's checkpoint before claiming the next queue item; synchronous idle `inject()` schedules its checkpoint without blocking `send()`, and disposal still drains it. A successful flush durably commits the closed turn as one unit; a rejecting flush is reported through `agent/error` and the logger — never as a session event past the closed turn — while the backend keeps its buffered events for the next flush.
## Crash recovery preserves an interrupted turn
A backend that reloads a log crashed mid-turn finds an open `turn/start` with no `turn/end`. It does **not** truncate — a single turn can be huge in a long-horizon task (many steps, large tool output), and those events were durably appended before the crash. Instead it closes the orphaned turn with a synthetic `turn/end { reason: { kind: 'interrupted' } }`, keeping the log balanced and the turn-enclosure invariant intact. `interrupted` is the one `TurnEndReason` no loop emits (see [session.md](session.md#why-a-turn-ended-turnendreasonmap)).
## `SessionLocation` — optional per-session artifact target
`SessionPersistence.locate(meta)` synchronously resolves a backend-owned independent artifact without reading, creating, or flushing it. JSONL returns its absolute target path; SQLite returns `undefined` because sessions share one database. A returned path can therefore name a file that does not yet exist or lacks the current unflushed turn; it is a location hint, not authorization or a freshness guarantee.
```ts type-equiv
/**
* A backend-resolved, per-session local artifact location. The path is an
* absolute target path and can name an artifact that has not materialized yet.
* Consumers must treat it as a location hint, never as an authorization token.
*/
interface SessionLocation {
/** Backend-specific artifact kind, for example `jsonl`. */
readonly kind: string
/** Absolute path to this session's backend-owned artifact. */
readonly path: string
}
```
## `SessionHeader` — metadata beside the log
2026-07-01 12:56:13 +08:00
Per-session metadata travels **separately** from the event log: format version, cwd, lineage, and the seed boundary are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The header is attached to a `Session` via `session.header`.
Source: [`packages/core/session/src/types.ts`](../../packages/core/session/src/types.ts)
```ts type-equiv
/**
* Immutable validated storage metadata, kept outside the conversation event log.
*/
interface SessionHeader {
fix review findings: bump session format version + restore late turn-end warn Codex review of the trace-event fold found two merge-blockers. Blocker #1 — format version. Folding usage onto assistant/message and removing the standalone usage/error events changed the persisted SessionEventMap shape, which per the AGENTS.md "bump the version and reject — don't migrate" policy requires a backend to reject any non-current log. Centralize the version in an exported SESSION_FORMAT_VERSION constant (dsh-session), read by both write sites (Session constructor default, SessionStore.prepare header) and the coordinator's load-time assertVersion check. The constant is pinned at 0: while unreleased the on-disk format is unstable/pre-release, so breaking shape churn is absorbed at v0 (no monotonic bump until the first tagged release) and any non-0 log is rejected on load — no migration. Update every test/fixture/doc that stamps a currently-written header to the constant, bump the ACP snapshot fixture + golden headers to v0, and keep the version-rejection test meaningful by switching its bad value to a clearly non-current 99. AGENTS.md documents both the monotonic (SQLite SCHEMA_VERSION) and pinned-0 (session log) pre-release stances. Blocker #2 — restore the late turn-end warn. failTurn now sets the error reason only while the turn is still open; once turn/end is appended (a throwing agent/turn-end listener after closeTurn) the reason can no longer reach the durable log, so the late throw is logged via ctx.logger.warn instead of vanishing into a futile post-close assignment. A regression test asserts the warn fires. Also guard the normal-step assistant/message append with the same content-or-usage condition as the max-tokens branch (a content-less, usage-less step records no trace-only row), with a covering test.
2026-06-21 11:08:10 +08:00
/**
* On-disk format version, stamped from {@link SESSION_FORMAT_VERSION} when the
* session is created. A persistence backend rejects any other version on load
* (no migration — see the constant).
*/
readonly version: number
/** The session's id (mirrors the {@link Session}'s id). */
readonly id: SessionId
/** Unix epoch milliseconds when the session was created. */
readonly createdAt: number
/** Absolute working directory the session was created in (if any). */
readonly cwd?: string
/** The session this one was forked from (seed lineage), if any. */
readonly parentSession?: SessionId
Persist the seed boundary so fork-child replay routes correctly A fork subagent seeds its child session with a prefix of the parent's log, and that seed becomes the child's persisted log — so a fork child's .jsonl begins with the PARENT's events, including the parent's assistant/chunk events. The snapshot replay harness derived a child's script from its whole log, which would replay the parent's recorded responses as the child's model calls. Spawn-only scenarios never hit it, but a fork snapshot would mis-route silently. Record the seed boundary and skip the inherited prefix at replay: - SessionHeader gains an optional `seedLength` (how many leading events were inherited via a seed), threaded through CreateSessionOptions/CreateAgentOptions meta and stamped by the fork backend (= seeded-prefix length; absent for spawn). It is EXPLICIT, never inferred from seed.length: a resume seeds the whole stored log, so the resume path passes the persisted boundary back. - Both persistence backends round-trip it: JSONL header line, SQLite seed_length column. The SQLite table change bumps SCHEMA_VERSION 2->3; per the pre-release stance the backend rejects an older user_version on open with NO migration. - llm-replay's parseSessionHeader reads seedLength and loadSessionScripts derives a child script from events AFTER the boundary. seedLength is 0 for spawn, so spawn replay is byte-for-byte unchanged. Closes the routing-correctness gap the per-session snapshot replay RFC under- stated; a recorded fork scenario remains a future addition but now derives correctly. RFC: docs/rfc/implemented/testing/2026-06-22-fork-child-replay-seed-boundary.md. Regression coverage: a fork child fixture whose seeded prefix carries a parent chunk (derived script must exclude it, proven red without the slice); a seedLength persistence round-trip through the shared coordinator contract (both backends); the fork backend stamping it; resume preserving it from the persisted header.
2026-06-22 20:55:32 +08:00
/**
* How many leading events were inherited through a seed. Persisting this
* boundary lets resume and replay distinguish parent history from child work.
Persist the seed boundary so fork-child replay routes correctly A fork subagent seeds its child session with a prefix of the parent's log, and that seed becomes the child's persisted log — so a fork child's .jsonl begins with the PARENT's events, including the parent's assistant/chunk events. The snapshot replay harness derived a child's script from its whole log, which would replay the parent's recorded responses as the child's model calls. Spawn-only scenarios never hit it, but a fork snapshot would mis-route silently. Record the seed boundary and skip the inherited prefix at replay: - SessionHeader gains an optional `seedLength` (how many leading events were inherited via a seed), threaded through CreateSessionOptions/CreateAgentOptions meta and stamped by the fork backend (= seeded-prefix length; absent for spawn). It is EXPLICIT, never inferred from seed.length: a resume seeds the whole stored log, so the resume path passes the persisted boundary back. - Both persistence backends round-trip it: JSONL header line, SQLite seed_length column. The SQLite table change bumps SCHEMA_VERSION 2->3; per the pre-release stance the backend rejects an older user_version on open with NO migration. - llm-replay's parseSessionHeader reads seedLength and loadSessionScripts derives a child script from events AFTER the boundary. seedLength is 0 for spawn, so spawn replay is byte-for-byte unchanged. Closes the routing-correctness gap the per-session snapshot replay RFC under- stated; a recorded fork scenario remains a future addition but now derives correctly. RFC: docs/rfc/implemented/testing/2026-06-22-fork-child-replay-seed-boundary.md. Regression coverage: a fork child fixture whose seeded prefix carries a parent chunk (derived script must exclude it, proven red without the slice); a seedLength persistence round-trip through the shared coordinator contract (both backends); the fork backend stamping it; resume preserving it from the persisted header.
2026-06-22 20:55:32 +08:00
*/
readonly seedLength?: number
}
```
## `CreateSessionOptions` — seeding and metadata
Persist the seed boundary so fork-child replay routes correctly A fork subagent seeds its child session with a prefix of the parent's log, and that seed becomes the child's persisted log — so a fork child's .jsonl begins with the PARENT's events, including the parent's assistant/chunk events. The snapshot replay harness derived a child's script from its whole log, which would replay the parent's recorded responses as the child's model calls. Spawn-only scenarios never hit it, but a fork snapshot would mis-route silently. Record the seed boundary and skip the inherited prefix at replay: - SessionHeader gains an optional `seedLength` (how many leading events were inherited via a seed), threaded through CreateSessionOptions/CreateAgentOptions meta and stamped by the fork backend (= seeded-prefix length; absent for spawn). It is EXPLICIT, never inferred from seed.length: a resume seeds the whole stored log, so the resume path passes the persisted boundary back. - Both persistence backends round-trip it: JSONL header line, SQLite seed_length column. The SQLite table change bumps SCHEMA_VERSION 2->3; per the pre-release stance the backend rejects an older user_version on open with NO migration. - llm-replay's parseSessionHeader reads seedLength and loadSessionScripts derives a child script from events AFTER the boundary. seedLength is 0 for spawn, so spawn replay is byte-for-byte unchanged. Closes the routing-correctness gap the per-session snapshot replay RFC under- stated; a recorded fork scenario remains a future addition but now derives correctly. RFC: docs/rfc/implemented/testing/2026-06-22-fork-child-replay-seed-boundary.md. Regression coverage: a fork child fixture whose seeded prefix carries a parent chunk (derived script must exclude it, proven red without the slice); a seedLength persistence round-trip through the shared coordinator contract (both backends); the fork backend stamping it; resume preserving it from the persisted header.
2026-06-22 20:55:32 +08:00
Creating a `Session` through the store takes a `seed` (replay/fork an existing event log) and `meta` (the storage-level fields the store folds into a `SessionHeader`). The store fills in `version`/`id` and defaults `createdAt`; the caller supplies the validated absolute `cwd`, the `parentSession` lineage, the `seedLength` seed boundary, and — only when reconstructing a persisted session — the original `createdAt` to preserve it.
```ts type-equiv
/**
* Options for creating a {@link Session} via the store. `seed` replays/forks
* an existing event log; `meta` carries the caller-supplied storage fields the
* store folds into a {@link SessionHeader}.
*/
interface CreateSessionOptions {
/** Events to seed the new session with (replay/fork). */
readonly seed?: readonly SessionEvent[]
/**
* Storage metadata read once before publication. `seedLength` is explicit
* because a resumed seed contains the full stored log, not only its inherited prefix.
*/
readonly meta?: {
readonly cwd?: string
readonly parentSession?: SessionId
readonly createdAt?: number
readonly seedLength?: number
}
}
```
Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resuming a *persisted* session into a live agent is `ctx.agents.resume({ resumeSessionId })`.
## The backends
Both implement the same abstract `SessionPersistence` (locate/create/append/load/list over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic:
- **[dsh-session-persistence-jsonl](../../packages/session-persistence/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path.
- **[dsh-session-persistence-sqlite](../../packages/session-persistence/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data, source_event_seqs, surface_op)` maps 1:1 onto the event, including optional surface metadata, so there is no parallel persisted schema to keep in sync.
2026-07-19 22:50:49 +08:00
Multiple backends sharing one on-disk session coordinate writes through the [shared persistence write-coordinator](../../.agents/notes/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md).