forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.
A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.
Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.
With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
handlePassthrough checked only the provider allowlist, never the model
or the path. A tenant scoped to one cheap model could reach a
provider's entire native API (files, batches, fine-tuning) on the
operator's key, and client identity headers reached keyless in-cluster
MCP/A2A targets unfiltered. This change closes both gaps, additively:
every new field defaults to reproducing prior behavior exactly.
Model enforcement: when a passthrough JSON body carries a top-level
"model" field, handlePassthrough checks it against the group's model
glob, in both the bare and provider-prefixed form (the same
dual-candidate matcher the unified route's resolveAgainst already
applies). A body with no inspectable model (Gemini's URL-embedded
model, a multipart upload, a non-JSON body) falls back to
provider-only authorization, unchanged from before this round.
Per-provider toggle: ProviderConfig.Passthrough, nil or true by
default, lets an operator disable a provider's native passthrough
route. A disabled provider now reports the ordinary unknown-route 404,
checked in ServeHTTP's own route gate, before auth even runs.
Per-group path allowlist: GroupConfig.PassthroughPaths, empty by
default (matching every other allow-list field's empty-means-all
semantics), restricts which native rest path a group may address.
Dangerous-header stripping: proxyUpstream now additionally strips
Forwarded, X-Forwarded-*, X-Auth-*, X-Remote-User, X-Remote-Groups, and
Cookie on both the native-passthrough and MCP/A2A target-proxy paths,
plus OpenAI-Organization, OpenAI-Project, and anthropic-beta on native
passthrough only. This is an additive denylist expansion, not an
allowlist inversion; ordinary headers still pass through.
A dedicated test asserts a config with none of the new fields set
behaves identically to before this change. The operator's home group
(providers/models/mcpServers/agents/passthroughPaths all empty) stays
fully unrestricted, and macstudio-rerank/parakeet-mlx/piper-tts stay
reachable through native passthrough by default.
Track per-provider and per-(provider,upstream-model) upstream-attempt
outcomes using the existing fixed-window counter machinery (limits.go),
following mcp_a2a.go's per-target counter pattern: kind-embedded keys
(kindProvider="prov", kindProviderModel="provmodel") keep a provider
scope collision-safe against a user/group/MCP-target/agent of the same
name.
limiter.recordProviderAttempt(provider, model, resp, err) increments an
"attempts" counter on every call, and a "failures" counter only for a
provider-fault outcome — reusing retry.go's own isTransient(resp, err)
classification exactly (transport/connection error, HTTP 429, or a 5xx),
not forking it. Both counters are written at minute and day granularity
only; provider health is a now-and-today question, not a billing one, so
there is no month window. limiter.providerUsage batches every scope's
read into one storeGetMulti round trip, mirroring targetUsage's own
single-batch discipline.
Wired at every chokepoint an upstream HTTP attempt resolves:
- providers.go's upstreamJSON/upstreamRawBytes path (chat, embeddings,
images, audio) via retry.go's retryPolicy.do, which now looks up an
optional attemptRecorder from ctx and invokes it once per retry
attempt (attempt-accounting: a request retried twice before
succeeding reports three attempts, not one).
- routes_unified.go's runUnified (chat/embeddings, streaming and
non-streaming alike — both funnel through the same upstreamJSON call).
- routes_media.go's three media handlers (images.generations,
audio.speech, audio.transcriptions).
- routes_passthrough.go's proxyUpstream (native provider passthrough) —
a single un-retried attempt, provider-level only: the upstream model
lives in the response body, read only after the attempt resolves.
The attemptRecorder is threaded via context (providers.go's
withAttemptRecorder/attemptRecorderFromContext), not a new function
parameter on every call site — proxyUpstream and retryPolicy.do are
both shared with paths that must NOT record provider accounting
(registry.go's discovery listModels calls, and the MCP/A2A target
proxy, mcp_a2a.go's handleTargetProxy), so scoping via context keeps
their signatures unchanged and those callers correctly opt out by
simply never wrapping their context.
lukaszraczylo