deepseek-harness/.agents/notes/implemented/architecture/2026-08-10-session-log-version-mechanism.md

Agent Note: Session log versioning — one integer, an upgrade chain, and a per-event ignorable marker

Status: implemented

English | 中文

Problem

Session logs must be upgradable after release, and the runtime that ships first is the floor for every later decision: whatever refusal and degradation behavior is missing from the first released reader can never be added to the copies users already run. Release issue #1901 required at minimum that an old runtime reading a newer session format reports "unsupported" instead of misreading it. The pre-change reader did the opposite on both axes: assertVersion rejected any version mismatch with one direction-blind message, and the JSONL decoder passed unknown event types through untouched, so reconstruction silently skipped them — resuming a gutted session with no diagnostic at all.

Decision

One monotonic integer, no major/minor split. Whether a version step is auto-upgradable is a property of that step — expressed by whether its upgrader exists — not something a two-level numbering scheme should promise in advance (you rarely know at design time whether the next change will turn out "major"). This matches the SQLite backend's SCHEMA_VERSION precedent.

The writer decides bumps, not the reader. A bump is required exactly when an old runtime could no longer handle a new log with full semantic correctness. "Parses without error" is not the bar: silently skipping content that shapes reconstruction is a wrong read. Only structural changes qualify — header shape, event envelope, core event semantics, the surface mechanism (SurfaceEventType set, SurfaceOp variants). When unsure, bump: a near-identity upgrader is almost free, a missed bump silently corrupts old readers.

Read rules by direction. Equal version: decode normally. Newer than the reader: refuse, name the direction ("written by a newer harness — upgrade"), and point at the raw log artifact so the user can still see the text (SessionFormatUnsupportedError, distinct from SessionPersistenceCorruptionError because nothing is damaged). Older than the reader: require a complete chain of static n→n+1 SessionFormatSteps; a missing step refuses the read and names the gap. The registry is part of the build rather than Cordis composition, so one build has the same durable read capability under every plugin set.

Format migration is the decoder, not a Coordinator repair branch. Backends expose parsed durable data as unknown through a repeatable StoredSessionSource: one raw header, one exact revision, and readEvents() factories that create independently consumable AsyncIterable streams bound to that revision. Each step implements migrateHeader() and lazy migrateEvents() transforms, must advance exactly one version, and may not change the session id or cwd. Any version conversion reads the complete event stream and applies the requested suffix only after all steps; an equal-version read retains backend suffix seek. The decoder validates each output header version, then applies current SessionHeader and SessionEvent validation only after the complete chain.

A future format bump adds one format-owned step. The change adds format-migrations/vN-to-vN+1.ts, exports that step from the static SESSION_FORMAT_STEPS array, and increments SESSION_FORMAT_VERSION. The step owns every old header and event variant it accepts, cross-event state inside its iterator transform, and explicit failure for malformed input. Backends and the Coordinator do not gain version-specific branches. Historical variants that never changed the version remain isolated in the format-v0 compatibility decoder and are not a template for later version steps. This decoder maps the historical compact/start, compact/summary, compact/end, and compact/prune names to canonical compaction/* events while preserving the rest of each record.

Recovery and writeback consume current-format data. inspect() and readFrom() decode only in memory. Cold prepare()/load() first decode the whole source, add the current recovery closers, and replace the exact old revision with that complete balanced current-format stream. Live HMR adoption uses the same replacement primitive after seed verification but does not synthesize closers for a turn still owned by the live Session. A successful replacement or revision conflict discards the prepared object and reopens the stored source before continuing.

Replacement is an internal backend compare-and-swap. replaceStored(expectedRevision, meta, events) accepts a streaming current-format log and checks storage identity plus the source revision at the commit boundary. JSONL writes and fsyncs a sibling temporary artifact, rechecks the source revision immediately before the atomic replace, atomically replaces the path (using the Windows write-through replacement primitive there), and syncs the parent directory on POSIX; like every other coordinator freshness check, the recheck adds no cross-process writer exclusion — JSONL assumes one live writer per session. SQLite stages the event iterator, then rechecks and replaces the header and event rows in one transaction. A failed commit leaves one complete old or new log; retaining a permanent pre-upgrade copy is a separate recovery policy, not part of the format migration API.

A per-event ignorable marker covers vocabulary growth, so ordinary event additions never bump the version. The event vocabulary is decided by which plugins are mounted, which a single version integer cannot describe. A reader meeting an unrecognized event type refuses to interpret the log unless the event carries ignorable: true in its envelope. The default is required: forgetting the marker over-refuses a resumable session (an inconvenience), while a default of ignorable would make the same mistake silently resume a gutted one (a safety failure). The architecture makes this sound: model-visible content flows only through the three surfaceOp-marked surface event types plus the request/header/request/context folds, so the dangerous unknowns are exactly the non-surface events that change how the rest of the log is read (session/end-seed is the existing example).

Consequences

Format v0 carries direction-aware refusal with the raw-log path; the unknown-event guard against a generated known-vocabulary list (KNOWN_SESSION_EVENT_TYPES, emitted by gen-persistence-catalog from every SessionEventMap merge and kept fresh by verify-persistence-catalog); the ignorable envelope field accepted by seed validation, both backends (a dedicated SQLite column, SCHEMA_VERSION 15), and the BFF wire schema; and the static streaming migration decoder with an empty adjacent-version registry. SESSION_FORMAT_VERSION remains 0 until a real v0→v1 step lands. The decoder and backend replacement APIs therefore have direct tests without manufacturing a format bump. Writers do not yet set ignorable because no producer needs it. Until a registration surface exists, an out-of-repo plugin's events refuse resume under first-party readers; the refusal is loud rather than silent. The unknown-type guard is read-side only: appendCore keeps rejecting retired legacy shapes but does not vocabulary-check new types, because an append-time refusal would stall a live session's durability mid-flight, which costs more than a loud refusal at the log's next load. The JSONL backend additionally refuses a foreign version from the raw header line before validating today's header fields or decoding any event record, so a structurally different future format still reports the upgrade direction instead of "corrupt"; SQLite gates whole-file structure through its own SCHEMA_VERSION pragma first.

Alternatives considered

• Major/minor versioning — the "is it convertible" bit lives on each step's upgrader, and pre-committing it into a number shape invites wrong promises.
• Default-ignorable unknown events — inverts the failure mode of a forgotten marker from visible over-refusal into silent corruption.
• Auto-migrating on view — rewriting the artifact on open turns a read into a destructive write: a converter bug corrupts logs at browse time, and a same-directory older runtime loses access because a newer one merely looked.
• Per-plugin runtime registration of known event types — would make the known set composition-dependent, so a leaner same-version composition would refuse logs a fuller one wrote. The generated repo-wide list keeps same-version reads uniform; out-of-repo plugin events are outside it by construction, and a registration surface for them is deferred until such a consumer exists.
• Materializing migrations as header and event arrays — makes the framework proportional to complete log size in memory even when each transformation is record-local. Repeatable revision-bound readers plus iterator transforms preserve retry semantics without imposing that allocation.
• Version-specific conversion in PersistenceCoordinator — mixes format decoding with operation-specific crash recovery and duplicates behavior across inspect, suffix read, cold continuation, and live adoption. The shared decoder produces only current-format data; each consumer retains its own recovery intent.
• A mandatory permanent backup for every upgrade — is not needed for atomicity and cannot promise the same physical representation across JSONL and SQLite. Backends may add recovery copies as a separate product policy without changing migrations.