forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.
A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.
Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.
With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
Add time-to-first-byte and total-duration histograms for every upstream
call, observation only. No throttling, priorities, or control logic
reads this data. The operator wants strain-based throttling later and
needs real traffic data first.
Extend watchdogBody (timeout.go) rather than add a new interception
layer: it already wraps every upstream response body for provider
adapters, native passthrough, and the MCP/A2A target proxy, and already
observes the first successful read. armLatency arms a watchdogBody
after construction, so newWatchdogBody keeps its existing signature and
behavior; its own pinned tests still pass unchanged. A timestamp plus a
few atomics are the only added cost; Read and Close each pay one nil
check when latency collection is disabled.
Expose llmgateway_upstream_ttfb_seconds and
llmgateway_upstream_duration_seconds on the existing Prometheus
endpoint (metrics.go). Both are in-process, per-replica histograms,
never store-backed, so sum() is always the correct cross-replica
aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and
short completions finely, cover the bulk of real chat completions, and
bracket the five-minute default request timeout. Labels are provider
and stream always, model only when Config.Metrics.ModelLabel is set,
matching the existing per-model opt-in convention. Add a compact
per-provider, per-stream average summary to GET /admin/api/overview,
sourced from the same in-process store, without a full histogram or a
per-model breakdown.
Every write path is gated on metricsEnabled(cfg), so a deployment with
metrics disabled collects nothing and the histogram families never
render. No behavior change to request handling: same responses, status
codes, headers, and accounting.
Fix a Yaegi-only bug this feature's own context-threading change
exposed: two distinct empty struct types used as context.WithValue keys
compare equal to each other under the interpreter's interface equality,
though never under compiled Go. Threading a second recorder
(latencyRecorder) alongside the existing attemptRecorder made this
observable for the first time, since compiled tests never see it and
the plugin previously carried only one such key. Give
latencyRecorderCtxKey a field so its size differs from
attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction
before and after the fix.
Fix a second bug, reproducible under compiled Go too: io.Reader may
legally return its final data together with io.EOF in one call, and
Go's real net/http body reader does this for small, fully-buffered
responses. The original TTFB capture lived inside the err == nil
branch, so it never fired for a typical small non-streaming completion.
Move the check ahead of the error branch so it fires on any n > 0,
leaving the pinned timer-reset and Close semantics untouched.
Tests mutate the real source per case and confirm each one fails first,
recorded in each test's own doc comment.
F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a
plain `now` variable while a background refresh goroutine read it
through nowFn, racing under -race in about 14% of full-suite runs once
this branch's timer-heavy tests shared the parallel phase. Guards the
clock with a mutex instead. registry.go is untouched.
F4: a mid-body watchdog timeout never reached provider-health
accounting. limiter.recordProviderAttempt only ran at header-arrival
time, so a provider that reliably stalled mid-body reported
attempts=1 failures=0 forever. watchdogBody.fire now invokes the same
attemptRecorder a second time, as a failure, when it fires. Reuses the
existing recorder rather than adding new plumbing; this does mean two
attempts are recorded for one logical request (one success at header
time, one failure at watchdog-fire time), not one attempt marked
failed outright.
F6: watchdogBody.Read reset the timer on any err == nil read,
regardless of n. A reader returning (0, nil) forever would have held
the watchdog open indefinitely. Reset now only fires when n > 0.
F9 (three items):
- proxyUpstream passed logPrefix as a bare providerName, which
newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an
MCP/A2A target as a provider. watchdogBody now takes an
already-formatted label from the caller instead.
- errUpstream was wrapped twice: once in watchdogBody's own error, once
again by every caller that already wraps it. watchdogBody now wraps
only errProviderTimeout; callers' existing wraps still make
errors.Is(err, errUpstream) hold.
- timeout_test.go read wb.timedOut directly in one place while every
other access is atomic; switched to atomic.LoadInt32.
Also rewords errProviderTimeout's message from "provider request timed
out" to "upstream progress stalled" (finding F3, documented not fixed
per coordinator ruling): the watchdog measures time since the last
successful upstream read, which can go quiet because of a slow client
write between reads, not only a slow provider. The new wording no
longer asserts which side caused it.
New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure,
TestWatchdogBody_ZeroByteRead_DoesNotResetTimer,
TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider.
Each mutation-verified per its own doc comment.
Adapters previously issued upstream requests with no timeout at all.
A provider that accepted a connection and then stalled hung the
gateway request forever. newAdapterHTTPClient now sets
Transport.ResponseHeaderTimeout, and every response body read
(providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream)
runs through a watchdogBody that resets on every successful read and
cancels the request once timeout passes with no progress.
The idle-progress design lets a long stream that keeps emitting tokens
run indefinitely. Only silence, never total duration, ends a request.
Config.RequestTimeout sets the global default (five minutes when
unset). ProviderConfig.RequestTimeout overrides it per provider. Both
are Go duration strings, validated at construction; a zero or negative
value is rejected rather than treated as unlimited, since that would
reintroduce the hang this feature fixes.
Native passthrough (routes_passthrough.go) and the MCP/A2A target
proxy (mcp_a2a.go) share the identical watchdog mechanism through
proxyUpstream's new timeout parameter.
A watchdog-fired timeout wraps errProviderTimeout and names the
provider. It never wraps context.Canceled, so it stays distinguishable
from a genuine client disconnect in every existing log classification.
handlePassthrough checked only the provider allowlist, never the model
or the path. A tenant scoped to one cheap model could reach a
provider's entire native API (files, batches, fine-tuning) on the
operator's key, and client identity headers reached keyless in-cluster
MCP/A2A targets unfiltered. This change closes both gaps, additively:
every new field defaults to reproducing prior behavior exactly.
Model enforcement: when a passthrough JSON body carries a top-level
"model" field, handlePassthrough checks it against the group's model
glob, in both the bare and provider-prefixed form (the same
dual-candidate matcher the unified route's resolveAgainst already
applies). A body with no inspectable model (Gemini's URL-embedded
model, a multipart upload, a non-JSON body) falls back to
provider-only authorization, unchanged from before this round.
Per-provider toggle: ProviderConfig.Passthrough, nil or true by
default, lets an operator disable a provider's native passthrough
route. A disabled provider now reports the ordinary unknown-route 404,
checked in ServeHTTP's own route gate, before auth even runs.
Per-group path allowlist: GroupConfig.PassthroughPaths, empty by
default (matching every other allow-list field's empty-means-all
semantics), restricts which native rest path a group may address.
Dangerous-header stripping: proxyUpstream now additionally strips
Forwarded, X-Forwarded-*, X-Auth-*, X-Remote-User, X-Remote-Groups, and
Cookie on both the native-passthrough and MCP/A2A target-proxy paths,
plus OpenAI-Organization, OpenAI-Project, and anthropic-beta on native
passthrough only. This is an additive denylist expansion, not an
allowlist inversion; ordinary headers still pass through.
A dedicated test asserts a config with none of the new fields set
behaves identically to before this change. The operator's home group
(providers/models/mcpServers/agents/passthroughPaths all empty) stays
fully unrestricted, and macstudio-rerank/parakeet-mlx/piper-tts stay
reachable through native passthrough by default.
fetchModelMetadata (provider_openai.go) builds its request URL by plain
string concatenation, a.baseURL+a.metadataPath. A metadataPath missing
its leading slash would silently glue onto baseURL's own path with no
separator (e.g. "http://host:1234api/v0/models"), a malformed URL that
only ever surfaced as a confusing runtime fetch failure. buildAdapters
now rejects a non-empty metadataPath without a leading "/" at
construction, naming the provider, the same fail-fast treatment every
other malformed config value already gets. Empty (the default) is
always valid. Folded review minor.
Config gets two new surfaces (llmgateway.go): Config.ModelMeta
(map[string]*ModelMetaConfig, keyed by exact "provider/model" or a
bare model id) for operator overrides, and ProviderConfig.MetadataPath
for openai-type providers whose discovery should also capture per-model
context (LM Studio's native /api/v0/models shape).
modelmeta.go adds the layering itself: resolveModelMeta applies config
override > discovered context > builtin table > absent for context, and
config override/free > builtin > absent for cost (discovery never
reports cost). resolveAliasModelMeta implements alias inheritance: an
alias inherits its target's fully resolved metadata unless the alias
itself has its own modelMeta entry, applied per field. validateModelMeta
rejects negative values and a free:true/non-zero-cost contradiction at
construction.
registry.go wires this into the request path: providerState gains a
discoveredContext map (stale-while-error, mirroring discovered's own
convention) with contextFor/setDiscoveredContext; a new optional
modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted,
documented as yaegi-safe against the Trap-4 restriction) rather than
added to providerAdapter, lets warmFill/refreshProvider capture metadata
alongside listModels without touching anthropic/gemini adapters at all.
modelObject/listFor now attach resolved context_window/pricing to every
GET /v1/models entry, including aliases (resolveMetaFor/
resolveMetaForAlias/resolveMetaForAliasName).
provider_openai.go implements modelMetadataFetcher: fetchModelMetadata
reads ProviderConfig.MetadataPath, preferring loaded_context_length over
max_context_length when the LM Studio response reports both. providers.go
wires MetadataPath onto the adapter in buildAdapters.
tools/yaegi-check/main.go adds one cheap modelMeta config-override
assertion to the existing attempt-accounting harness, proving
resolveModelMeta's config-override layer and the context_window/pricing
extension fields both run correctly under the real Yaegi interpreter.
Track per-provider and per-(provider,upstream-model) upstream-attempt
outcomes using the existing fixed-window counter machinery (limits.go),
following mcp_a2a.go's per-target counter pattern: kind-embedded keys
(kindProvider="prov", kindProviderModel="provmodel") keep a provider
scope collision-safe against a user/group/MCP-target/agent of the same
name.
limiter.recordProviderAttempt(provider, model, resp, err) increments an
"attempts" counter on every call, and a "failures" counter only for a
provider-fault outcome — reusing retry.go's own isTransient(resp, err)
classification exactly (transport/connection error, HTTP 429, or a 5xx),
not forking it. Both counters are written at minute and day granularity
only; provider health is a now-and-today question, not a billing one, so
there is no month window. limiter.providerUsage batches every scope's
read into one storeGetMulti round trip, mirroring targetUsage's own
single-batch discipline.
Wired at every chokepoint an upstream HTTP attempt resolves:
- providers.go's upstreamJSON/upstreamRawBytes path (chat, embeddings,
images, audio) via retry.go's retryPolicy.do, which now looks up an
optional attemptRecorder from ctx and invokes it once per retry
attempt (attempt-accounting: a request retried twice before
succeeding reports three attempts, not one).
- routes_unified.go's runUnified (chat/embeddings, streaming and
non-streaming alike — both funnel through the same upstreamJSON call).
- routes_media.go's three media handlers (images.generations,
audio.speech, audio.transcriptions).
- routes_passthrough.go's proxyUpstream (native provider passthrough) —
a single un-retried attempt, provider-level only: the upstream model
lives in the response body, read only after the attempt resolves.
The attemptRecorder is threaded via context (providers.go's
withAttemptRecorder/attemptRecorderFromContext), not a new function
parameter on every call site — proxyUpstream and retryPolicy.do are
both shared with paths that must NOT record provider accounting
(registry.go's discovery listModels calls, and the MCP/A2A target
proxy, mcp_a2a.go's handleTargetProxy), so scoping via context keeps
their signatures unchanged and those callers correctly opt out by
simply never wrapping their context.
Critical: attempts now counts retries performed AFTER the first try
(default 1 -> two tries total), not the total try count. The prior
reading made retry:{enabled:true}'s own documented default perform zero
retries. (*retryPolicy).totalTries() is the single conversion point,
clamping any attempts<1 to 1 so do()'s loop bound can never compute <=0
and skip calling call() entirely (no (nil,nil) return path).
Backoff waits are now ctx-aware: sleepFn func(time.Duration) is replaced
by waitFn func(context.Context, time.Duration) bool, whose production
implementation (defaultWaitFn) selects between time.After(d) and
ctx.Done(). A context canceled mid-wait aborts the remaining retries
immediately instead of blocking out the full backoff.
Also: classify a request-build failure (invalid method/URL) as
non-transient via errRequestBuildFailed, since retrying an identically
malformed request only wastes attempts; drop the drain cap to a
dedicated 64KiB constant instead of the 32MiB response cap; fix a
t.Fatal call from a non-test goroutine in the mid-stream-death test;
drop the misleading omitempty on Config.Retry (a struct value is never
'empty' to encoding/json); document the Retry-After >2s-falls-back-to-
backoff behavior and cover it with a direct table test.
README: add the retry config table, a RetryConfig field reference, and
a Retry section documenting the semantics. examples/kubernetes.yaml
gains a retry block alongside its other optional sections.
Gate: go test -race, gofmt, vet, yaegi-check, golangci-lint (incl a
fieldalignment fix on two new test structs) all clean. 295 subtests
pass, 0 failed. integration/ untouched.
Adds spec §1's opt-in retry engine: RetryConfig (top-level, zero value
disabled) and retryPolicy (retry.go), wired into upstreamJSON — the
single HTTP chokepoint every adapter's chat/embeddings/listModels call
already funneled through. A transient failure (net error, 429, 5xx)
retries up to Attempts times with exponential backoff capped at 2s,
honoring a qualifying Retry-After header on 429, draining and closing
each failed attempt's body before retrying. Never retries a non-429 4xx,
a translate error, or after context cancellation.
Because upstreamJSON returns before any adapter forwards a byte to the
client (forwardStream/forwardJSON run only after it returns), the
'zero bytes reached client' safety rule and the 'streaming retries only
the initial exchange' rule both hold by construction — no adapter
streaming logic needed to change.
buildAdapters(cfg) keeps its v0.1 signature: it derives the retryPolicy
from cfg.Retry internally rather than taking it as a parameter, so
every existing direct call (15+ across provider_openai_test.go) keeps
compiling unmodified. Every v0.1 test passes byte-identically, without
edits.
configNamePattern (^[a-zA-Z0-9._-]+$) permits "." and ".." as valid
provider/mcpServers/agents names, since a bare dot is in the allowed
character class. Both names read as directory-traversal segments once
embedded as a path segment in the gateway's own routes
(passthroughRoute's "/{provider}/*", targetRoute's "/mcp/{name}/*" and
"/a2a/{name}/*"). Nothing in this package's current path handling
mis-resolves them, but disallowing them at construction is cheap and
removes any future dependence on that staying true.
- providers.go: add dotOnlyConfigNames ({".", ".."}), checked in
validateConfigName between the character-class check and the
reserved-name check.
- provider_openai_test.go: extend TestValidateConfigName with "." and
".." cases (rejected) across provider/mcpServers/agents kinds, plus
"..." (accepted — the rule is an exact match on "." and "..", not "any
dots-only string").
Gates: go test ./... -count=1, go test ./... -race -count=1, gofmt -l .,
go vet ./..., make yaegi-check, golangci-lint run ./..., golangci-lint
run --enable=gosec ./..., fieldalignment ./... — all clean.
- providers.go: validate provider names against ^[a-zA-Z0-9._-]+$ and
reject reserved names (v1, mcp, a2a) that would shadow gateway routes;
reject nil ProviderConfig map values at construction instead of
panicking.
- mcp_a2a.go: apply the same name-pattern and reserved-name validation to
mcpServers/agents keys, and reject nil TargetConfig/AgentConfig map
values at construction.
- auth.go: reject nil GroupConfig map values at construction.
- provider_openai_test.go, mcp_a2a_test.go, auth_test.go: table-driven
constructor-error coverage for the above.
- integration/integration_test.go: point the package doc at
`make integration-up`/`integration-keep` instead of a bare
`docker compose up -d --build`, which skips the dynamic.yml.tmpl
render step and can run against a stale or absent dynamic.yml; add
max_tokens to the gemini chat request so translate_gemini.go's
hasMaxTokens=true branch is actually exercised under real Yaegi.
- translate_gemini.go: soften the max_tokens comment's verification claim
to match what the integration suite actually now exercises.
Addresses coordinator review of the Task 15 integration suite: 1 critical
ruling, 5 important, 3 folded. make integration now exits 0.
- integration_test.go: TestStreamingIncrementalityProbe restructured —
content assertion (5 deltas + [DONE], in order) always runs; the raw-TCP
timing assertion (unchanged, >=3 gaps >=100ms) is gated behind
INTEGRATION_STREAMING=1 with t.Skipf naming traefik/traefik#10269,
verified to still genuinely fail when re-armed (not silently disabled).
- Makefile: integration-up/integration-wait moved from prerequisites into
the integration recipe body as go test ./... -count=1
ok github.com/lukaszraczylo/traefik-llmgateway 6.065s sub-invocations with per-stage
status capture, so integration-down always runs even when bring-up or
the health wait fails; verified with a synthetic failure. Added
.NOTPARALLEL (the integration targets are stateful).
- translate_gemini.go: fixed the same yaegi multi-value-assignment panic
shape as the prior sse.go fix, on the live max_tokens path. Added a
gemini chat completion assertion to integration_test.go (renamed
TestUnifiedChatAllProviders) so this path now runs under real Yaegi.
- providers.go, translate_anthropic.go: documented the never-wrap
contract on providerHTTPError/translateError (routes_unified.go asserts
their concrete type directly; errors.As panics under yaegi on these).
- integration_test.go: TestCrossReplicaLimits now FLUSHALLs Redis first,
for idempotence under integration-keep.
- mock/main.go: probe's header-terminator search now accumulates a
rolling buffer across reads instead of only checking the latest one.
- Folded: hot-reload deadline 10s -> 20s; dynamic.yml's real-upstream
baseUrl is now a template (dynamic.yml.tmpl) rendered via
INTEGRATION_REAL_BASEURL (default https://llmgw.example.com), not a
hardcoded-only hostname.
See task-15-report.md's addendum for full verification detail.
Review round 1 fixes (1 Critical + 5 Important) on the anthropic
adapter/translator.
- CRITICAL: consecutive role:"tool" messages each became their own
Anthropic user turn. Anthropic requires alternating user/assistant
turns and 400s on two consecutive same-role messages; parallel tool
calls (one assistant message with N tool_calls followed by N
role:"tool" messages) are the standard agentic transcript shape.
anthropicRequestFromOpenAI now accumulates consecutive tool messages
into one Anthropic user turn carrying all of their tool_result
blocks (anthropicToolResultBlock replaces anthropicToolResultMessage).
Added chat_parallel_tool_results fixture.
- Streaming tool_calls[].index passed Anthropic's raw content-block
index through, so a tool_use block after a text block emitted
index:1 instead of the required contiguous 0-based ordinal.
anthropicStreamState now tracks a per-stream tool-call ordinal
counter plus a content-block-index -> ordinal map, populated on
content_block_start and consulted on input_json_delta. Added the
stream_text_then_tool fixture (text block at index 0, tool_use at
index 1) proving the emitted index is 0.
- A system message with array-form content ([{"type":"text",...}])
was silently dropped (only a string content was handled). Added
systemTextFromContentParts: text parts join with "\n"; a non-text
part returns a *translateError instead of being dropped. Added
chat_system_array_content (success) and
chat_system_array_content_error fixtures.
- Malformed JSON on a recognized stream event (message_start,
content_block_start, content_block_delta, message_delta) was
swallowed as (nil, nil), which could silently corrupt a client's
reassembled tool-call arguments. All four now return an error
wrapping errUpstream; forwardStream already propagates that and
withholds the terminal [DONE]. The unrecognized-event-type and
error-event branches are unchanged. Added
TestAnthropicStreamState_MalformedEventReturnsError.
- providerAdapter.injectAuth's doc comment claimed a universal
empty-key no-op; updated to state the per-type divergence
(openai-type: keyless is a no-op; anthropic-type: the constructor
rejects an empty key, so injectAuth is never called with one).
- Added case-insensitive-Content-Type coverage for both adapters'
streaming-vs-JSON gate (previously untested): one subtest each in
provider_openai_test.go / provider_anthropic_test.go using a fake
upstream that sends "Text/Event-Stream; charset=utf-8", asserting
the streaming path (not the JSON fallback) is taken.
Gate: go test ./... -count=1 and -race both green; gofmt, go vet,
fieldalignment, golangci-lint (incl. gosec), govulncheck all clean.
Implements Task 9: the anthropic-type providerAdapter and its
request/response/stream translation to and from OpenAI's wire shape.
- translate_anthropic.go: anthropicRequestFromOpenAI (system messages
joined, content blocks incl. base64 data-URI images, tool_calls ->
tool_use, tool messages -> tool_result, max_tokens defaulting,
stop/stop_sequences, tool_choice incl. the none-drops-tools rule,
n>1/logit_bias/logprobs rejected), openAIResponseFromAnthropic
(content blocks joined/tool_calls re-marshaled, stop_reason mapping,
usage), and anthropicStreamState.translate mapping each SSE event to
zero or more OpenAI chat.completion.chunk payloads. translateError
carries a notSupported flag so the caller (Task 12) can tell a 400
from a 501.
- provider_anthropic.go: anthropicAdapter implementing providerAdapter.
newAnthropicAdapter errors on an empty API key per ruling (c), unlike
openai-type's keyless ruling. Headers: x-api-key, anthropic-version.
embeddings always returns a notSupported translateError without
calling upstream (Anthropic has no embeddings endpoint). listModels
GETs /v1/models and returns any non-2xx (incl. 404) as-is.
- providers.go: wires the anthropic buildAdapters case to
newAnthropicAdapter, propagating its constructor error.
- provider_openai_test.go: replaces the now-stale placeholder
TestBuildAdapters subtest with one asserting a keyed anthropic
provider builds successfully and a keyless one errors.
- translate_anthropic_test.go: golden-file harness walking
testdata/anthropic/{chat,resp}_*_in.json against sibling _want.json
(success) or _wanterr.json (*translateError) fixtures, plus a stream
harness driving each *.sse fixture through anthropicStreamState and
asserting the chunk list and usage().
- provider_anthropic_test.go: httptest-backed adapter wiring tests
(streaming + non-streaming), asserting OpenAI-shaped output on the
wire, per-event flushing, and the required auth headers.
Gate: go test ./... -count=1 and -race both green; gofmt, go vet,
fieldalignment, golangci-lint (incl. gosec) all clean.
lukaszraczylo