deepseek-harness/packages/web/web-fetch-local/README.md

# @deepseek-ai/dsh-web-fetch-local

English | [中文](README.zh.md)

An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.

This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).

## Responsibility split

The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.

The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers and misconfigured deployments, not the model-facing tool-call budget. [`dsh-timeout-policy`](../../timeout/timeout-policy/README.md) owns the `web_fetch` tool-call budget by arming `exec.signal`.

A shipping web-tool deployment sets the provider backstop above the tool budget, so model calls normally return `TOOL_TIMEOUT`. If the outer deadline reaches the provider first, the provider reports `WEB_ABORTED` and the outer policy replaces it with `TOOL_TIMEOUT`. `WEB_FETCH_TIMEOUT` therefore identifies a direct seam caller whose provider budget elapsed.

## Transport hygiene

- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
- Sends an explicit product `User-Agent`, never a browser disguise.
- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.

## Config

| Key | Default | Meaning |
|---|---|---|
| `maxUrlLength` | `2048` | Maximum accepted request URL length. |
| `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. |
| `maxBodyChars` | `100_000` | Maximum decoded body length in characters. |
| `timeoutMs` | `30_000` | Fetch timeout within Node's timer range — a resource backstop for direct `ctx.web.fetch()` callers, not the model-facing tool-call budget (that is `dsh-timeout-policy`). |
| `maxRedirects` | `5` | Maximum same-origin redirect hops (`0` follows none). |
| `userAgent` | `deepseek-harness/…` | `User-Agent` header. |

The numeric limits are validated at plugin construction: every cap except `maxRedirects` must be a positive finite number, and `maxRedirects` must be a non-negative integer. An invalid value throws rather than silently constructing a provider with nonsensical limits.

## Model Experience

Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which places this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.

#### KV Cache effect

No direct invalidation; the named consumer owns any request-prefix changes.

## Known Limitations and Deferred Work

- **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
- **Only textual content decodes** — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00			`# @deepseek-ai/dsh-web-fetch-local`

docs: translate remaining READMEs 2026-07-26 05:03:53 +08:00			`English \| [中文](README.zh.md)`

Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00			An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.

			This is an implementation package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).

			`## Responsibility split`

feat(timeout): add tools/execute seam + tool-timeout policy plugin Model-facing tool-call budgets were tangled into each capability's schema (bash timeoutMs, web_fetch timeout_ms) with no shared home. Add a tools/execute around-dispatch waterfall to dsh-tools whose base next() is the dispatch-with-normalization thunk, and a new @deepseek-ai/dsh-timeout-policy plugin (packages/timeout/) that arms a per-tool deadline on exec.signal and returns a structured TOOL_TIMEOUT when it wins. Migrate web_fetch (drop the model-facing timeout_ms) and web_search onto it; the fetch provider keeps its timeout only as a resource backstop for direct callers. bash and hook command execution keep BASH_TIMEOUT unchanged. Named the plugin timeout-policy (not the RFC's tool-timeout) so it does not trip the gen-tool-catalog packages//tool- completeness guard, and replace exec.signal by in-place mutation before next() since cordis waterfall next() ignores passed arguments. RFC moved to implemented/ recording both deviations. 2026-07-08 10:06:07 +08:00			The provider owns safe resource retrieval: URL validation, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns presentation (HTML→markdown, truncation formatting). A non-2xx HTTP response is a result (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.

Merge remote-tracking branch 'origin/master' into codex/simp-prune-web-seam-fields # Conflicts: # docs/config-catalog.md # docs/cordis-catalog/services.md # docs/core-data-structures/web.md # docs/rfc/implemented/simplification/2026-07-04-drop-unconsumed-web-observation-surface.md # packages/web/tool-web/tests/integration.spec.ts # packages/web/web-fetch-local/README.md # packages/web/web-fetch-local/src/provider.ts # packages/web/web/src/types.ts 2026-07-14 17:57:50 +08:00			The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers and misconfigured deployments, not the model-facing tool-call budget. [`dsh-timeout-policy`](../../timeout/timeout-policy/README.md) owns the `web_fetch` tool-call budget by arming `exec.signal`.
docs: rebalance prose cleanup and add trimming skill 2026-07-13 23:27:00 +08:00
			A shipping web-tool deployment sets the provider backstop above the tool budget, so model calls normally return `TOOL_TIMEOUT`. If the outer deadline reaches the provider first, the provider reports `WEB_ABORTED` and the outer policy replaces it with `TOOL_TIMEOUT`. `WEB_FETCH_TIMEOUT` therefore identifies a direct seam caller whose provider budget elapsed.
Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00
			`## Transport hygiene`

			- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
			- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
			- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
			- Follows only same-origin redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
			- Sends an explicit product `User-Agent`, never a browser disguise.
			- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.

			`## Config`

			`\| Key \| Default \| Meaning \|`
			`\|---\|---\|---\|`
			\| `maxUrlLength` \| `2048` \| Maximum accepted request URL length. \|
			\| `maxResponseBytes` \| `5_000_000` \| Maximum response body size in bytes. \|
			\| `maxBodyChars` \| `100_000` \| Maximum decoded body length in characters. \|
fix: bound local fetch timer config 2026-07-14 05:06:01 +08:00			\| `timeoutMs` \| `30_000` \| Fetch timeout within Node's timer range — a resource backstop for direct `ctx.web.fetch()` callers, not the model-facing tool-call budget (that is `dsh-timeout-policy`). \|
fix: address codex review findings on web seam - search providers (exa/perplexity/deepseek): map the parsed response INSIDE the parse try, so a well-formed body of the wrong shape surfaces as WEB_PROVIDER_ERROR instead of escaping as a raw TypeError; a WebError the mapper throws on purpose is re-thrown untouched - web-fetch-local: validate numeric limits at plugin construction (positive finite caps; non-negative integer maxRedirects) rather than constructing a provider with nonsensical values - web-fetch-local: enforce the redirect budget BEFORE resolving each hop, so maxRedirects:N follows exactly N redirects and an over-limit hop reports "exceeded the maximum" rather than misdiagnosing a cross-origin block - drop the stale dsh-tool-web/search and /fetch path aliases (the package no longer declares those subpath exports) - strip trailing EOF blank lines flagged by git diff --check Each fix carries a regression test. 2026-06-29 17:23:11 +08:00			\| `maxRedirects` \| `5` \| Maximum same-origin redirect hops (`0` follows none). \|
Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00			\| `userAgent` \| `deepseek-harness/…` \| `User-Agent` header. \|

fix: address codex review findings on web seam - search providers (exa/perplexity/deepseek): map the parsed response INSIDE the parse try, so a well-formed body of the wrong shape surfaces as WEB_PROVIDER_ERROR instead of escaping as a raw TypeError; a WebError the mapper throws on purpose is re-thrown untouched - web-fetch-local: validate numeric limits at plugin construction (positive finite caps; non-negative integer maxRedirects) rather than constructing a provider with nonsensical values - web-fetch-local: enforce the redirect budget BEFORE resolving each hop, so maxRedirects:N follows exactly N redirects and an over-limit hop reports "exceeded the maximum" rather than misdiagnosing a cross-origin block - drop the stale dsh-tool-web/search and /fetch path aliases (the package no longer declares those subpath exports) - strip trailing EOF blank lines flagged by git diff --check Each fix carries a regression test. 2026-06-29 17:23:11 +08:00			The numeric limits are validated at plugin construction: every cap except `maxRedirects` must be a positive finite number, and `maxRedirects` must be a non-negative integer. An invalid value throws rather than silently constructing a provider with nonsensical limits.

docs: address model experience review 2026-07-12 02:55:26 +08:00			`## Model Experience`
Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00
Merge remote-tracking branch 'origin/master' into codex/trim-ai-prose # Conflicts: # docs/AGENTS.md # docs/config-catalog.md # packages/bash/bash-sandbox/src/index.ts # packages/bash/bash/src/session-mode.ts # packages/bash/tool-bash/README.md # packages/code-runtime/code-runtime-worker/README.md # packages/compact/compact/src/index.ts # packages/core/agent-core/README.md # packages/hooks/hooks-claude/src/config.ts # packages/hooks/hooks-claude/src/index.ts # packages/hooks/hooks-codex/src/config.ts # packages/hooks/hooks-codex/src/index.ts # packages/llm/llm/README.md # packages/session-persistence/session-persistence-jsonl/README.md # packages/session-persistence/session-persistence/README.md # packages/skill/skill-local/README.md # packages/support/acp-snapshot/README.md # packages/support/invariants/src/index.ts # packages/ui/acp/README.md # packages/ui/jsonrpc-agent/README.md # packages/ui/jsonrpc/README.md # packages/ui/permission/README.md # packages/ui/user-approval/README.md # packages/ui/user-interaction/README.md # packages/web/web-search-deepseek/README.md 2026-07-14 14:37:16 +08:00			Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which places this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.
docs: address model experience review 2026-07-12 02:55:26 +08:00
docs: structure model experience fields 2026-07-19 18:08:42 +08:00			`#### KV Cache effect`

			`No direct invalidation; the named consumer owns any request-prefix changes.`
docs: document package KV cache effects 2026-07-19 17:39:50 +08:00
Add a gated Known Limitations and Deferred Work section to every package README Every packages// README now carries a canonical '## Known Limitations and Deferred Work' section: condensed, evidence-backed bullets for consumer-visible gaps (unimplemented features, platform caveats, MVP cuts) and consciously postponed work (TODO/FIXME/XXX markers, RFC deferrals still open). The ten pre-existing ad-hoc variants ('What is NOT here (TODO)', 'Deferred', 'Limitations (MVP)', 'Known limitations (tracked TODOs)', ...) are normalized into the canonical heading. A new doc-sync gate, scripts/verify-readme-limitations.ts, enforces the shape: exactly one limitations-like heading per package README, byte-equal to the canonical h2, with at least one bullet; near-miss headings fail so variants cannot creep back. Packages with genuinely nothing to declare (dsh-brand, dsh-timeout, dsh-subagent-mock, dsh-app-boot) are whitelisted in the script and must NOT carry the section; whitelist entries are validated against the scanned package set so a rename fails loud. Wired into the doc-sync chain (package.json) and the run-gates doc-sync leaf set; the standing rule lands in packages/AGENTS.md and the adding-a-package cookbook; decision record in docs/rfc/implemented/process/2026-07-10-readme-known-limitations-gate.md (RFC index regenerated). Also fixes two stale '(deferred)' markers claiming dsh-compact-basic is unimplemented (the dsh-compact seam README's package table and the seam's module doc comment). 2026-07-10 01:51:50 +08:00			`## Known Limitations and Deferred Work`
Add web capability seam: ctx.web, search/fetch providers, web tools Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6). 2026-06-25 15:04:12 +08:00
Rename RFCs to Agent Notes 2026-07-19 22:50:49 +08:00			`- SSRF / private-network protection is deferred — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and must not be enabled in a deployment that can reach sensitive internal network targets.`
Add a gated Known Limitations and Deferred Work section to every package README Every packages// README now carries a canonical '## Known Limitations and Deferred Work' section: condensed, evidence-backed bullets for consumer-visible gaps (unimplemented features, platform caveats, MVP cuts) and consciously postponed work (TODO/FIXME/XXX markers, RFC deferrals still open). The ten pre-existing ad-hoc variants ('What is NOT here (TODO)', 'Deferred', 'Limitations (MVP)', 'Known limitations (tracked TODOs)', ...) are normalized into the canonical heading. A new doc-sync gate, scripts/verify-readme-limitations.ts, enforces the shape: exactly one limitations-like heading per package README, byte-equal to the canonical h2, with at least one bullet; near-miss headings fail so variants cannot creep back. Packages with genuinely nothing to declare (dsh-brand, dsh-timeout, dsh-subagent-mock, dsh-app-boot) are whitelisted in the script and must NOT carry the section; whitelist entries are validated against the scanned package set so a rename fails loud. Wired into the doc-sync chain (package.json) and the run-gates doc-sync leaf set; the standing rule lands in packages/AGENTS.md and the adding-a-package cookbook; decision record in docs/rfc/implemented/process/2026-07-10-readme-known-limitations-gate.md (RFC index regenerated). Also fixes two stale '(deferred)' markers claiming dsh-compact-basic is unimplemented (the dsh-compact seam README's package table and the seam's module doc comment). 2026-07-10 01:51:50 +08:00			- Only textual content decodes — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
			- Charset comes only from the `Content-Type` header (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.