kevchess15/deepseek-harness
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
deepseek-harness/docs/rfc/001-property-based-testing.md

25 lines
2.1 KiB
Markdown
Raw Normal View History

Add RFCs for the remaining quality-proposal ideas Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
2026-06-11 15:27:07 +08:00
# RFC 001: Property-based testing for protocol-shaped code
Status: proposed
## Problem
docs: unwrap hard-wrapped Markdown to one line per paragraph Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. Reflow all tracked non-vendor Markdown (plus vendor/AGENTS.md) so each prose paragraph is a single line; soft-wrapping is the editor's job. Fenced code, tables, and list structure are preserved (wrapped list items fold to one line per bullet). Documents the convention in AGENTS.md.
2026-06-13 18:39:20 +08:00
Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for (the `streamBlocks` ordering bug survived 100% line coverage of the happy paths).
Add RFCs for the remaining quality-proposal ideas Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
2026-06-11 15:27:07 +08:00
## Proposal
Adopt fast-check (vitest integration) with generators for our vocabulary:
docs: unwrap hard-wrapped Markdown to one line per paragraph Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. Reflow all tracked non-vendor Markdown (plus vendor/AGENTS.md) so each prose paragraph is a single line; soft-wrapping is the editor's job. Fenced code, tables, and list structure are preserved (wrapped list items fold to one line per bullet). Documents the convention in AGENTS.md.
2026-06-13 18:39:20 +08:00
- **BlockAssembler**: arbitrary chunk sequences (valid and malformed — duplicate indices, stragglers after block-end, missing block-start). Invariants: `flushReady() + flushRemaining() ≡ blocks()` in order; `streamBlocks ≡ generate().message.content`; memory bounded (partials map size ≤ distinct indices); idempotent re-assembly.
- **Session**: arbitrary event logs (seeded generators over SessionEventMap). Invariants: `deriveMessages` deterministic; replay-from-seed produces identical derivation; seq strictly monotonic; derived history unaffected by non-message events.
- **Schema DSL**: arbitrary SchemaSpecs. Invariants: generated JSON Schema's `required` array equals the `required: true` keys at every nesting level; conversion is total (never throws); generated args satisfying `InferArgs` validate against the generated schema (once RFC 005's validator exists — the two RFCs compose).
- **Inbox/loop**: arbitrary send/steer/abort schedules against a scripted adapter. Invariants: no message lost (every send/steer appears in the log exactly once), turn numbers strictly increase, status transitions follow idle→running→idle/disposed.
Add RFCs for the remaining quality-proposal ideas Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
2026-06-11 15:27:07 +08:00
## Plan
docs: unwrap hard-wrapped Markdown to one line per paragraph Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. Reflow all tracked non-vendor Markdown (plus vendor/AGENTS.md) so each prose paragraph is a single line; soft-wrapping is the editor's job. Fenced code, tables, and list structure are preserved (wrapped list items fold to one line per bullet). Documents the convention in AGENTS.md.
2026-06-13 18:39:20 +08:00
One `tests/properties.spec.ts` per package; fast-check as devDependency; numRuns tuned so the suite stays under ~10s locally, with a nightly CI job running 100× the iterations. Failures persist their seed in the report so agents can reproduce deterministically.
Add RFCs for the remaining quality-proposal ideas Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
2026-06-11 15:27:07 +08:00
## Risks
docs: unwrap hard-wrapped Markdown to one line per paragraph Hard line breaks mid-paragraph make docs harder to edit and diff — a one-word change reflows and re-diffs the whole paragraph. Reflow all tracked non-vendor Markdown (plus vendor/AGENTS.md) so each prose paragraph is a single line; soft-wrapping is the editor's job. Fenced code, tables, and list structure are preserved (wrapped list items fold to one line per bullet). Documents the convention in AGENTS.md.
2026-06-13 18:39:20 +08:00
Generator quality determines value — invest in generators that produce *realistic-but-adversarial* streams, not uniform noise. Property flake from timeouts must be treated as a finding, not retried away.
Reference in a new issue Copy permalink