deepseek-harness/packages/web/web-fetch-http/README.md

@deepseek-ai/dsh-web-fetch-http

English | 中文

An anonymous public HTTP(S) WebFetchProvider for the harness web capability seam (ctx.web). It retrieves a concrete URL and returns a status code plus bounded decoded content.

This is an implementation package: it registers a provider into ctx.web, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (inject: ['web']). The separate dsh-web-fetch-approval-policy plugin reuses its network-free URL validation before asking users about restricted web_fetch calls.

Responsibility split

The provider owns safe resource retrieval: URL validation, public-address resolution and connection pinning, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. @deepseek-ai/dsh-tool-web owns presentation (HTML→markdown, truncation formatting). A non-2xx HTTP response is a result (status code + decoded body), not an error; WebError is reserved for failures to safely retrieve or represent the resource.

The provider's timeoutMs is a resource backstop for direct ctx.web.fetch() callers and misconfigured deployments, not the model-facing tool-call budget. dsh-tool-call-timeout-policy owns the web_fetch tool-call budget by arming exec.signal.

A shipping web-tool deployment sets the provider backstop above the tool budget, so model calls normally return TOOL_TIMEOUT. If the outer deadline reaches the provider first, the provider reports WEB_ABORTED and the outer policy replaces it with TOOL_TIMEOUT. WEB_FETCH_TIMEOUT therefore identifies a direct service caller whose provider budget elapsed.

Transport hygiene

• Accepts only http: and https: URLs; rejects credentials in URLs (WEB_BLOCKED_URL) and URLs over the fixed 2,048-character security limit or otherwise malformed (WEB_INVALID_URL).
• Resolves each hostname once, rejects the complete answer set if any IPv4 or IPv6 destination is not public unicast (WEB_BLOCKED_URL), and pins the connection to that validated set. For IPv6 answers it discovers the active DNS64 prefix through ipv4only.arpa and rejects NAT64 translations to non-public IPv4. This blocks loopback, private, link-local, carrier-grade NAT, multicast, reserved, transition, translation, and private IPv4-mapped IPv6 destinations without resolving the target hostname twice.
• Enforces the URL limit, response byte cap (WEB_FETCH_TOO_LARGE), decoded body character cap, timeout (WEB_FETCH_TIMEOUT), and redirect hop cap.
• Propagates the caller's abort signal (WEB_ABORTED) into the network request and the streaming read.
• Follows only same-origin redirects; each followed hop repeats public-address resolution and pinning, while a cross-origin redirect fails with WEB_REDIRECT_BLOCKED and requires a fresh tool call (the model of Claude Code's WebFetch).
• Sends an explicit product User-Agent, never a browser disguise.
• Rejects unsupported (e.g. binary) content types with WEB_UNSUPPORTED_CONTENT_TYPE.

validateFetchApprovalUrl() exposes network-free URL syntax, length, credentials, and literal-IP checks to permission consumers. Hostname resolution remains exclusively in the provider after consent, where the result is enforced and pinned rather than reused as an authorization token.

Direct HttpFetchProvider construction may inject an HttpFetchResolver for alternate trusted assemblies and deterministic tests. That resolver must reject every non-public destination before returning addresses; the shipped plugin always uses the built-in public-address resolver.

Config

Key	Default	Meaning
maxResponseBytes	5_000_000	Maximum response body size in bytes.
maxBodyChars	100_000	Maximum decoded body length in characters.
timeoutMs	30_000	Fetch timeout within Node's timer range — a resource backstop for direct ctx.web.fetch() callers, not the model-facing tool-call budget (that is dsh-tool-call-timeout-policy).
maxRedirects	5	Maximum same-origin redirect hops (0 follows none).
userAgent	deepseek-harness/…	User-Agent header.

The configurable numeric limits are validated at plugin construction: every cap except maxRedirects must be a positive finite number, and maxRedirects must be a non-negative integer. An invalid value throws rather than silently constructing a provider with nonsensical limits.

Model Experience

Indirectly, through dsh-tool-web, which places this provider's maxBodyChars-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.

KV Cache effect

No direct invalidation; the named consumer owns any request-prefix changes.

Known Limitations and Deferred Work

• Only textual content decodes — html/xhtml and text/*-plus-JSON/XML families; a missing Content-Type or any binary type throws WEB_UNSUPPORTED_CONTENT_TYPE, and text-extractable PDF decoding is named deferred work.
• Charset comes only from the Content-Type header (UTF-8 default) — an HTML <meta charset> declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.