forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.
A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.
Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.
With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
The metadata fetcher understood only LM Studio's field names, so a
vLLM-backed provider reported no per-model context and every model had to
have its window pinned by hand in modelMeta. That pin then goes stale
silently: a renamed served model ships no context at all and clients fall
back to their own default, and a changed window advertises a size the
engine no longer accepts. Both happened on a live deployment today.
vLLM reports max_model_len at its OpenAI-compatible /v1/models and neither
LM Studio field, verified against vLLM 0.28. Reading it means pointing
MetadataPath at /v1/models is enough for such a provider.
Precedence is unchanged where it already applied: loaded_context_length
still wins as the window usable right now, then max_context_length, and
max_model_len is consulted only when neither is present. An upstream
reports one dialect or the other, so the added branch cannot change any
existing capture.
Renames the payload type, which now carries two dialects rather than LM
Studio's alone.
Adapters previously issued upstream requests with no timeout at all.
A provider that accepted a connection and then stalled hung the
gateway request forever. newAdapterHTTPClient now sets
Transport.ResponseHeaderTimeout, and every response body read
(providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream)
runs through a watchdogBody that resets on every successful read and
cancels the request once timeout passes with no progress.
The idle-progress design lets a long stream that keeps emitting tokens
run indefinitely. Only silence, never total duration, ends a request.
Config.RequestTimeout sets the global default (five minutes when
unset). ProviderConfig.RequestTimeout overrides it per provider. Both
are Go duration strings, validated at construction; a zero or negative
value is rejected rather than treated as unlimited, since that would
reintroduce the hang this feature fixes.
Native passthrough (routes_passthrough.go) and the MCP/A2A target
proxy (mcp_a2a.go) share the identical watchdog mechanism through
proxyUpstream's new timeout parameter.
A watchdog-fired timeout wraps errProviderTimeout and names the
provider. It never wraps context.Canceled, so it stays distinguishable
from a genuine client disconnect in every existing log classification.
Config gets two new surfaces (llmgateway.go): Config.ModelMeta
(map[string]*ModelMetaConfig, keyed by exact "provider/model" or a
bare model id) for operator overrides, and ProviderConfig.MetadataPath
for openai-type providers whose discovery should also capture per-model
context (LM Studio's native /api/v0/models shape).
modelmeta.go adds the layering itself: resolveModelMeta applies config
override > discovered context > builtin table > absent for context, and
config override/free > builtin > absent for cost (discovery never
reports cost). resolveAliasModelMeta implements alias inheritance: an
alias inherits its target's fully resolved metadata unless the alias
itself has its own modelMeta entry, applied per field. validateModelMeta
rejects negative values and a free:true/non-zero-cost contradiction at
construction.
registry.go wires this into the request path: providerState gains a
discoveredContext map (stale-while-error, mirroring discovered's own
convention) with contextFor/setDiscoveredContext; a new optional
modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted,
documented as yaegi-safe against the Trap-4 restriction) rather than
added to providerAdapter, lets warmFill/refreshProvider capture metadata
alongside listModels without touching anthropic/gemini adapters at all.
modelObject/listFor now attach resolved context_window/pricing to every
GET /v1/models entry, including aliases (resolveMetaFor/
resolveMetaForAlias/resolveMetaForAliasName).
provider_openai.go implements modelMetadataFetcher: fetchModelMetadata
reads ProviderConfig.MetadataPath, preferring loaded_context_length over
max_context_length when the LM Studio response reports both. providers.go
wires MetadataPath onto the adapter in buildAdapters.
tools/yaegi-check/main.go adds one cheap modelMeta config-override
assertion to the existing attempt-accounting harness, proving
resolveModelMeta's config-override layer and the context_window/pricing
extension fields both run correctly under the real Yaegi interpreter.
Controller-ruled small fixes accumulated from T3/T4 reviews:
1. routes_media.go: multipart Content-Type check is now an exact
mediaType != "multipart/form-data" comparison (HasPrefix admitted
"multipart/form-datax"); a malformed-tail multipart body (fails to
parse at all) now reports "malformed multipart body", distinct from
"model is required" (a body that parses fine but carries no model
part) and "duplicate model field".
2. provider_openai.go: audioTranscription's doc comment no longer claims
the body is always "replayed unchanged"/the boundary always
"verbatim" — it now reflects the rewrite path routes_media.go
already takes for an aliased model id.
3. rewriteMultipartModel: buf.Grow(len(body)+512) up front, avoiding
repeated reallocate-and-copy growth while rebuilding an aliased
multipart body.
4. limits.go currentUsage: guards len(vals) == len(keys) before indexing
storeGetMulti's result, so a non-conforming counterStore's short slice
reports storeDown instead of panicking.
5. admin.go dashboard JS: the 5s poll no longer wipes an existing
auth-error message when no key is stored (only a successful load or a
new submission clears it now); clearStoredKey on a 401/403 now only
fires when the failing request's own key still matches what's
currently stored, guarding against a late response for a key the user
has since replaced.
6. Spec §4's duplicated /admin/api/overview bullet and the
nosniff/no-store sentence's placement were already fixed upstream
(docs/superpowers/specs/2026-08-20-llmgateway-v02-design.md, commit
c02010d) — verified identical, nothing further to commit here.
7. Added TestAdmin_JSONSecurityHeaders (nosniff/no-store/CSP on both
admin JSON APIs) and TestHandleAudioTranscriptions_MalformedTail_400
(covers fix 1's new malformed-body branch).
go test ./... -count=1 -race, gofmt, vet, make yaegi-check, golangci-lint
(incl. gosec) all pass.
Adds spec §1's opt-in retry engine: RetryConfig (top-level, zero value
disabled) and retryPolicy (retry.go), wired into upstreamJSON — the
single HTTP chokepoint every adapter's chat/embeddings/listModels call
already funneled through. A transient failure (net error, 429, 5xx)
retries up to Attempts times with exponential backoff capped at 2s,
honoring a qualifying Retry-After header on 429, draining and closing
each failed attempt's body before retrying. Never retries a non-429 4xx,
a translate error, or after context cancellation.
Because upstreamJSON returns before any adapter forwards a byte to the
client (forwardStream/forwardJSON run only after it returns), the
'zero bytes reached client' safety rule and the 'streaming retries only
the initial exchange' rule both hold by construction — no adapter
streaming logic needed to change.
buildAdapters(cfg) keeps its v0.1 signature: it derives the retryPolicy
from cfg.Retry internally rather than taking it as a parameter, so
every existing direct call (15+ across provider_openai_test.go) keeps
compiling unmodified. Every v0.1 test passes byte-identically, without
edits.
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
Implements Task 12: POST /v1/chat/completions and POST /v1/embeddings
through a shared pipeline (routes_unified.go: handleChat, handleEmbeddings,
runUnified) wired into ServeHTTP's dispatch in llmgateway.go.
Pipeline: decode (10MiB cap) -> resolve model -> enforce user/group
limits -> call adapter -> account usage, translating every failure mode
into its HTTP envelope per the task brief's carried rulings:
- ALIAS ECHO (ruling a): the client-requested model id is threaded to the
adapter via the internal gatewayAliasKey convention and echoed back in
anthropic/gemini's translated response model field; every adapter
strips the key before building its upstream wire body so it never
leaks to a real provider.
- Limit violations (ruling b): storeDown -> 503 server_error; otherwise
429 rate_limit_error with Retry-After when the violation names a
window.
- Stream-safe error handling (ruling c): statusTrackingWriter.wroteHeader
gates whether an adapter error still gets an envelope; providerHTTPError
wraps into {error:{message,type,code,upstream}}; translateError maps to
400/501; context.Canceled logs quietly; anything else is a 502.
- Usage/accounting (ruling d): zero non-streaming usage gets a
ceil(len(body)/4) prompt-token estimate; zero streaming usage logs once
and accounts the request only. unifiedCostMicros extends costMicros'
single-id lookup to canonical-then-bare without changing its signature.
- Scopes (ruling e): user scope only when u.limits is set, group scope
only when grp.limits is set, user first.
routes_unified_test.go drives every case end-to-end through
Gateway.ServeHTTP against httptest fake upstreams (openai- and
anthropic-shaped): happy path with counters asserted via the limiter's
own store, incremental stream flushing, 401/403/404/429/503, embeddings,
alias echo through a provider-prefixed anthropic request, wrapped
upstream error envelopes, and a mid-stream drop that must not append a
second envelope to an already-committed response.
Gates: gofmt, go vet, fieldalignment, golangci-lint (incl. gosec) all
clean; go test ./... -race -count=1 passes.
Content-Type header values are case-insensitive per RFC 7231, but
chatCompletion's streaming-vs-JSON-fallback gate used a plain
strings.Contains against "text/event-stream", so an upstream that
sent "Text/Event-Stream" or similar would silently fall through to
the non-streaming passthrough path. Carried item (a) from Task 9's
brief.
lukaszraczylo