deepseek-harness/.agents/notes/implemented/feature/2026-08-20-unified-image-request-pipeline.md

Agent Note: Unified normalized attachments, request versions, and provider files

Status: implemented

English | 中文

Problem

Durable image history, provider resolution, inline request size, and remote file reuse have different limits. Treating an admitted image as the bytes sent on every later request forced one byte cap and one raster to serve all four concerns. Large but ordinary input was refused, clean 16-bit PNG could pass into history and fail at DeepSeek, repeated base64 expanded long requests, and a provider rejection repeated because the same durable image stayed in every future request.

Decision

The image path has two explicit versions. The attachment backend owns a provider-independent durable normalized attachment. Each image-capable model route owns a deterministic request policy, and the attachment backend derives and caches the exact request version from that attachment. Session history contains only the normalized attachment reference; inline bytes and provider file ids remain transient request projections.

Provider-independent normalized attachment

Admission accepts at most 20 images and 200MiB of encoded source bytes per message. Each source is fully decoded under configurable 20MiB, 64,000,000-pixel, and 8192px-per-side limits. Normalization applies EXIF orientation, removes metadata and color profiles, converts to 8-bit sRGB/sRGBA, and preserves aspect ratio while scaling into the normalizedImageMaxPixels total-pixel budget (2048x2048 by default) under a normalizedImageMaxDimension long-edge cap (8192px by default). When scaling reduces the raster, originalDimensions records its orientation-applied width and height before normalization.

The normalized attachment has an independent normalizedImageMaxBytes encoded-byte target, 4MiB by default. Alpha is never flattened. Codec routing is by the decoded alpha fact alone — alpha input encodes as WebP (effort 0) and opaque input as JPEG, each down the shared 85/75/60 quality ladder; when every quality exceeds the target the smallest output is kept, per the superseding alpha-routed quality ladders note. A clean, single-frame 8-bit sRGB/sRGBA PNG, JPEG, or WebP within every normalization limit passes through byte-identically and retains content-addressed deduplication. GIF, animation, metadata, orientation, 16-bit PNG, and incompatible color spaces force conversion. The source and a converted output are each fully decoded once; the output must match its format, dimensions, depth, color space, and alpha facts before its digest enters the reference.

Batch admission prepares and verifies every normalized attachment once before publishing any member. Validation failure starts no writes. Publication uses those prepared bytes directly, so a large batch does not repeat full decoding and encoding during commit. A later storage failure returns no partial references; already published immutable objects may remain unreachable under the existing storage rule.

Deterministic request versions

AttachmentStore.readImageRequest derives a request version under route-owned total-pixel and encoded-byte budgets. Scaling is min(1, sqrt(maxPixels / (width * height))), with no enlargement, followed by inward integer rounding so the encoded raster never exceeds the total-pixel cap. DeepSeek V4 Flash Vision Exp uses 640,000 total pixels and a 1MiB raw encoded-byte target by default. Its catalog uses one imagePixelBudget field: a positive integer selects an exact total-pixel budget, low selects 512 by 512 total pixels, and omission selects the route default. A 2048 by 1024 normalized attachment projects to 1130 by 565 under the hard cap. Request encoding uses the same alpha routing and 85/75/60 quality ladder as normalization, executed lazily; a target no quality meets keeps the smallest ladder output (see the alpha-routed quality ladders note). The same derivation is used by normal agent turns, direct ctx.llm.stream calls, compaction, and other auxiliary streams.

The variantId and cache path cover the normalized attachment id, transform version, route pixel and byte budgets, and fixed encoder parameters. A new cache entry is fully decoded before publication. Cache hits use a header probe to check format, 8-bit sRGB/sRGBA facts, dimensions, and alpha without decoding the complete raster again; a mismatch regenerates the entry. DeepSeek Files and pi-ai inline base64 therefore use the same deterministic bytes for the same policy. Inline accounting uses the derived byte length after base64 expansion, not the normalized attachment byte count. Equal in-process variantId calls share one transform and cache write. Each caller can cancel its own wait; the shared transform is aborted only after every waiter has cancelled. Callers preserve order by applying Promise.all to singular readImageRequest calls. The local implementation runs normalization and request transforms through one FIFO limiter; imageCompressionConcurrency is configurable from 1 through 8 and defaults to 2. Batch publication remains sequential after every normalized attachment has been prepared.

Request-size offload is a deterministic oldest-first projection. Before reading attachments, each route uses min(attachmentBytes, requestVersionMaxBytes) as a conservative upper bound and removes the oldest over-budget prefix. Only retained attachments are read and transformed, so an omitted missing or corrupt object cannot block the request. A second projection uses exact derived lengths without bringing omitted images back. DeepSeek defaults to 128MiB and 600 referenced images. Its removed prefix advances past successive 64MiB byte boundaries and in 20-image count quanta, so 129 one-megabyte images remove the oldest 65, retain 64MiB, and keep that prefix stable until total history passes 192MiB. Pi-ai retains a configurable base64 request bound. Each omitted image becomes a per-image placeholder that retains its identity and access resolved for the current tool execution world, including nested tool-result images, while append-only session history keeps the original references.

Stable handles

Every retained request image is preceded by its display name or complete attachment id and actual request dimensions. The attachment provider can supply its host object location; the LLM consumer combines it with the current filesystem mapping before adding an absolute read-only path, normalized dimensions, and media type. The descriptor states that normalization or request projection may have resized or re-encoded the upload, so the model cannot infer original upload properties from either representation. User messages, tool results, agent-loop requests, compaction, and direct ctx.llm.stream calls share this projection. The path is resolved independently from the deterministic request version and does not enter its variantId, the durable reference, or the session log.

DeepSeek Files lifecycle

The direct deepseek-official adapter normally uploads every retained request version through the OpenAI-compatible Files API and sends file_id content blocks. A bounded inline fallback sends the same deterministic request versions when file resolution fails. The default catalog advertises deepseek-v4-flash-vision-exp as image-capable. Uploaded ids are indexed by endpoint and API-key scope plus variantId. Uploads request seven days by default and record the returned expires_at; a mapping with no more than one hour remaining is replaced without a preceding retrieve call. The index never stores the API key.

An upload is indexed only after the response returns a complete file object, matching byte count, and expires_at. A missing or inconsistent response leaves no local mapping, so a later request uploads again. Concurrent upload resolution for one scoped variantId shares one provider operation; one waiter cannot cancel another, and the upload stops when every waiter has cancelled. A malformed upload index is an empty cache and is replaced on the next successful upload; filesystem I/O failures remain errors. If chat reports expired, deleted, missing, or invalid ids and names one or more ids used by the request, only those mappings are removed. A stale-file response without a specific id removes every mapping used by that chat attempt. The affected request bytes are uploaded again and chat is retried once. A second stale rejection clears the mappings identified by its response and returns the error without a third chat attempt. One upload quota error first lists the configured number of oldest harness-owned dsh- files, then deletes that collected set and retries once; deleting after pagination keeps provider cursors valid. Public file operations expose list, retrieve, delete, one-variant release, and namespace-wide release. Every Files request carries the shared Harness User-Agent. The client enforces the documented 128MiB upload limit, 32MiB chat-image limit, 10,000-file and 25GiB quotas, and one-hour to 30-day expiry range.

Diagnostics

A 16-bit RGB or RGBA PNG is normal admitted input and converts to 8-bit sRGB/sRGBA. If local conversion fails, read_image names the path, detected 16-bit PNG, required normalized form, and manual conversion remedy. If DeepSeek rejects a normalized request version, the primary error names the attachment or display name, durable message and image position, normalized media type, 8-bit sRGB/sRGBA depth, dimensions, and provider message. An ambiguous multi-image rejection lists every candidate. The raw provider body remains the error cause rather than the only visible message.

Historical attachment objects that later disappear or fail integrity verification remain fail-loud. Durable quarantine and verified recovery require session events and are tracked by Quarantine unreadable historical attachments.

Alternatives considered

Use one 1MiB normalized attachment for storage and requests. This makes model resolution determine durable image detail and combines local storage, inline expansion, Files quota, and model pixels into one setting. Independent normalization and request policies keep those responsibilities explicit.

Reject images above provider dimensions or at the encoding quality floor. A provider limit is route-specific and future requests may use another model. Proportional normalization and request projection accept ordinary large images while bounding each later representation.

Treat PNG as a screenshot and reject 16-bit PNG. File format does not reveal pixel complexity, and 16-bit RGB/RGBA is a convertible sample depth rather than an unsupported image type. Pixel sampling and post-conversion probes give the required facts.

Keep DeepSeek data URLs as the primary transport. Inline base64 repeats bytes on every request and caps usable image history by request-body size. Files API references reuse uploaded deterministic request bytes and provide explicit expiry and deletion; the bounded fallback uses data URLs only when file resolution fails.

Trust a locally indexed file id indefinitely. Remote expiry, deletion, and lost upload responses make local and provider state diverge. Response-directed invalidation and one re-upload recover without an unbounded retry loop; an ambiguous stale-file response must invalidate every file used by that attempt because it provides no safe exact target.

Refuse text-only model selection after any image. Durable history can outlive the model that first consumed it. Request-local placeholders keep the session usable without rewriting history.

Remove one image whenever a request crosses its limit. That changes an early request message after nearly every new upload. Quantized removed prefixes keep cache invalidation occasional while honoring the configured high bound.

Verification

Package tests generate 16-bit RGB and RGBA PNG fixtures, prove 8-bit conversion and clean 8-bit passthrough, retain alpha under byte pressure, stop lazy encoding after the first fitting candidate, keep the smallest ladder output above an unreachable byte target, cover square and wide 640,000-pixel projections, singleflight equal variants and uploads without shared-cancellation leaks, bound transform concurrency, preserve cache and upload identity, skip attachment reads for conservatively offloaded history, prepare batches once, reject inconsistent Files responses, refresh near-expiry ids without retrieve, recover once from single-id, multiple-id, and ambiguous stale responses, fall back to bounded all-inline requests after file resolution failure, paginate before quota deletion, normalize provider diagnostics, project text-only history, and share normal/compaction request bytes. Keyless assembled snapshots cover the real tool schemas and image request path. A credentialed test uses the built-in deepseek-official route and its configured endpoint, never a custom provider entry.

Consequences

Normalized attachments consume up to the independent local safety cap, while request caches and remote Files consume additional derived storage. Deterministic identities and singleflight make that work reusable across turns and sessions sharing the same DSH home. Two simultaneous transforms reduce batch latency while increasing peak RSS relative to serial execution; deployments with tighter memory can set the limit to one. Encoder or transform-version changes create new future identities without rewriting existing history. DeepSeek image requests prefer Files reuse; bounded stale-id recovery handles inconsistent remote state, while file-resolution failures use the smaller inline budget. Missing or corrupt durable attachments still require the separate quarantine design.