Chart and ranking fetches carry a request id, and a superseded response
is dropped. Changing a selection clears the old data at once, so old
numbers never show under new labels or units. The auto-refresh tick
skips while a fetch is still in flight. "Loading..." shows only until
the first result for a selection arrives. Dashboard polls do not
overlap.
Users in several groups are counted and searched in every group. The
Models tab labels its windows as the current UTC bucket. Hour labels
are marked UTC. The Members column sorts numerically. Target access
shows "All groups" for null and "No groups" for []. The UI shows the
server's own error message. Cost axis ticks use the currency
formatter. npm run dev proxies /admin/api. Nine unused shadcn
components are removed.
admin_assets_gen.go is regenerated from the new build.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
A go test run over ./... stops at nested modules, so CI never ran
tools/pricing-sync or the Yaegi interpreter check. A new job runs both
with Go 1.25.x. The go.sum of tools/yaegi-check now holds its test
dependencies, and its go.mod lists the prometheus modules as direct
dependencies. The .gitattributes comment now says that the
export-ignore rules for tools/, webui/, examples/ and testdata/ do take
effect.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
cmd/gateway runs the middleware as its own HTTP server behind Traefik.
It reads a YAML config, converts it to JSON and decodes it with
DisallowUnknownFields, so a misspelled key stops startup. It warns
when metrics.allowedCIDRs is set, because behind a proxy the source
address is the proxy's.
The Makefile gains build-binary, docker-build and run-local targets.
The Dockerfile builds from the committed vendor tree and stamps
main.version from the VERSION build argument. go.mod lists
go.yaml.in/yaml/v3 as a direct dependency.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Federated tools/call re-initializes a session only when the backend
signals a missing session: HTTP 400 or 404, or a JSON-RPC error that
names a session. A failing tool is no longer called twice.
tools/list now:
- passes each tool object through and rewrites only its name
- follows nextCursor up to a page cap and keeps the earlier pages if a
later page fails
- skips tools with no name
- resolves name collisions to the server that tools/call routes to
The handshake sends notifications/initialized. tools/call forwards
_meta. Multi-line SSE data is joined. A2A target URLs are joined with
url.JoinPath and keep their query string. agents[].card must start with
"/". Logged target URLs and errors are scrubbed of credentials.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
The gateway refuses a request for an unpriced model when the caller has a
cost budget. An unpriced model cost 0, so the budget counter was never
written and the budget could never fire. The new
allowUnpricedWithCostBudget setting restores the old behaviour.
The retry loop honours an upstream Retry-After of up to 30s and jitters
its own computed backoff. SSE parsing bounds the total size of one event.
The Redis client can use TLS. Provider base URLs and errors are scrubbed
of credentials before they reach logs or the admin view. The gateway
warns when failover and retry settings multiply into a large number of
outbound calls.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
encodeCommands wrote one fmt.Appendf call per header and per argument. A
counter pipeline is almost entirely INCRBY and EXPIRE commands, which
carry three arguments each, so a metered request paid four Appendf calls
for every counter it wrote.
A command with exactly three arguments now goes through one Appendf with
the whole template. Every other arity keeps the per-argument loop. The
bytes on the wire do not change.
A new test compares arities two, three, four and five against the
previous encoder, so a wrong arity guard cannot emit a three-argument
header for a longer command.
Measured under yaegi as above: about 3% fewer bytes allocated per
request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
incrMulti and incrAndGetMulti took the expire mutex twice per counter:
once to ask needsExpire whether the key still wants an EXPIRE, and once
after the reply to commit or forget it. A metered request writes 28
counters, so each request took that mutex about 56 times.
needsExpireBatch now answers for the whole batch under one lock, and
settleExpireBatch applies the commit and forget decisions under one more.
The decisions are unchanged: needsExpire is read for every entry before
any commit, a confirmed EXPIRE commits the key, and an INCRBY reply equal
to the delta forgets it. The FIFO eviction bound is unchanged.
applyIncrReplies decodes the INCRBY replies and reports how many entries
it examined, so the caller settles exactly that prefix on every path. A
reply error used to leave the already-decided entries unsettled, which
kept a key marked as having an EXPIRE it never received. That key could
then never regain a TTL, because its INCRBY reply can never equal the
delta again. A provider failover answering READONLY mid-batch reaches
this path. Two tests reproduce it and fail without the settle.
Measured under yaegi as above: this step removed a further 4% of the
bytes allocated per request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
Each counter entry called newCounterIncr, which called windowKey, which
called fmt.Sprintf and formatted the window bucket from the clock. A
metered request writes 28 counters, so every request formatted 28 bucket
strings and built 28 keys through Sprintf.
bucketsFor now formats the four window buckets once per limiter call, and
enforceTTLsFor reads the four enforcement floors from new constants
instead of calling enforceTTLFor per entry. The call sites build
counterIncr literals directly from those values, and windowKeyForBucket
joins the key parts. Keys on the wire do not change, and windowKey keeps
its old signature for every other caller.
The inline literals pair the window, the bucket, the TTL and the
enforcement floor by hand, which newCounterIncr used to derive. A new
test walks every entry the four metered methods produce and asserts the
enforcement floor matches the window in the key, and that the key ends
with that window's bucket. The test fails if any pair drifts.
Measured under yaegi with a harness that runs POST /v1/embeddings against
Redis with the cache and metrics on: this step alone removed about 6% of
the bytes allocated per request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.
A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.
Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.
With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
Under yaegi v0.16.1, the Redis write path allocated heavily.
encodeCommand wrote each argument with fmt.Fprintf into a
strings.Builder. attemptPipelineOn copied every command into a second
builder, and io.WriteString converted the result to bytes again. setEx
also copied the cached response with string(val).
encodeCommands now appends one fmt.Appendf call per header and argument
to one slice, and the connection writes that slice directly. pipeline
encodes once, so a retry sends the same bytes. setEx frames SET directly
from the value bytes. The bytes on the wire do not change.
Two other encoders were measured and rejected. Under the interpreter, a
presized buffer and direct byte appends cost more than native slice
growth inside fmt.Appendf.
Tests compare encodeCommands and the setEx wire bytes with the previous
encoder. The cases include binary values, empty arguments and a
100000-byte value.
A yaegi harness runs POST /v1/embeddings with Redis, cache and metrics
on. This change lowers allocation from 925587 to 783189 bytes per
request (-15%).
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
A user entry can now list several groups in "groups", next to the
existing "group" field. A user can also hold a personal grant with its
own "providers" and optional "models" lists. Users with one group and
no personal grant behave exactly as before.
A request is allowed only when one grant allows the provider, the model
and, for passthrough, the path. Grants never merge, so one grant cannot
supply the provider while another supplies the model or the path. The
personal grant uses the path list of the first group. A personal
"models" list without "providers" is a configuration error.
Each member group applies its own limits. Caching stays on unless a
member group turns it off, and the shortest group TTL wins. The
/v1/models cache uses a stable, length-prefixed key, so reloads do not
grow it.
Tests cover the grant union, cross-grant denials, passthrough path and
model coupling, reloads and the cache key. The yaegi-check harness
asserts the same cases under the interpreter.
Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3
OpenAI's /v1/models schema carries no context field, so every client that
wants one reads a vendor dialect, and a client whose dialect is absent does
not ask — it substitutes its own default. GET /v1/models now emits the same
resolved integer under "context_window", "context_length" (the OpenRouter
spelling) and "max_model_len" (the vLLM spelling). All three are omitted
together when the window is unknown, exactly as the single field was.
Measured case: omp 18.1.15's openai-models-list discovery resolves
`max_model_len ?? context_length ?? <bundled catalog> ?? 128000` and never
reads context_window, so a 1048576-context local model reached the client as
128000, which also clamped its maxTokens.
The metadata fetcher already parses all of these dialects on the inbound
side (provider_openai.go), so this makes the outbound direction speak what
the inbound one already understands.
Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp
resolve gives an exact alias match precedence over every discovered bare
model id, but listFor did the opposite: it skipped the alias whenever a
discovered model of the same bare id had already been listed. The catalog
then advertised the shadowed provider as the owner, with that provider's
metadata, for an id every request routed to the alias target instead.
A live fleet shows the effect. The alias "deepseek-v4-flash-vision-exp"
targets "gx10/current" (context 1048576), while a cloud provider discovers
a model of the identical bare id. GET /v1/models returned owned_by
"deepseek" and no context_window at all, so clients fell back to their own
128k default while the request ran against a 1M-context local model.
listFor now resolves every usable alias up front and suppresses the bare
entry of any id an alias shadows. The shadowed model stays listed in
"provider/id" form, which is how a client must address it once the alias
owns the bare id, so no model disappears from the catalog.
Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp
The metadata fetcher understood only LM Studio's field names, so a
vLLM-backed provider reported no per-model context and every model had to
have its window pinned by hand in modelMeta. That pin then goes stale
silently: a renamed served model ships no context at all and clients fall
back to their own default, and a changed window advertises a size the
engine no longer accepts. Both happened on a live deployment today.
vLLM reports max_model_len at its OpenAI-compatible /v1/models and neither
LM Studio field, verified against vLLM 0.28. Reading it means pointing
MetadataPath at /v1/models is enough for such a provider.
Precedence is unchanged where it already applied: loaded_context_length
still wins as the window usable right now, then max_context_length, and
max_model_len is consulted only when neither is present. An upstream
reports one dialect or the other, so the added branch cannot change any
existing capture.
Renames the payload type, which now carries two dialects rather than LM
Studio's alone.
Add per-replica health tracking for every configured MCP server and A2A
agent. Real proxied and federated traffic always feeds the signal. An
opt-in lazy probe sweep, off by default, fills the gaps for targets that
see no traffic. Expose the result on GET /admin/api/targets, as the
llmgateway_target_healthy gauge, and as a badge in the dashboard.
Federation routing is unchanged: an unhealthy server is still attempted.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The zero-time sentinel was module private, so the spec redeclared the
literal once a third consumer started depending on it. Export it and
import it in the spec. Regenerate the baked admin assets.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The warmup waited for one target while the same sweep probed the others
concurrently. A late probe could overwrite another target's traffic
sourced record and fail a later assertion.
Wait for every configured MCP target to report the probe source, and
correct the comment that claimed a guarantee the code did not give.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The note claimed fieldalignment reports no change from pinning the sweep
timestamp first. The analyser is in fact clean before the move and
reports 192 against an optimal 184 leading pointer bytes after it.
State the measured truth. The struct stays 264 bytes either way, so the
exception costs no padding. Moving the sweep flag up to close the
pointer-bytes gap was tried and made the report worse. Atomic alignment
on 32-bit platforms outranks the linter hint.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The previous round suppressed every context-caused federation error. A
server that accepted the connection and then never answered therefore
read "unknown" forever whenever federation carried the only traffic,
which is the outage shape this feature exists to surface.
Split the two causes instead. A server still waiting for a fan-out slot
when the budget expires is never dialed, so the semaphore acquire now
selects on the fan-out context and records nothing for it. A server this
gateway did dial records a failure on a deadline, because it had its
chance to answer. Only a caller hanging up stays unattributable.
Document the split rule, correct a stale comment about a second record
site, and state that "unknown" reads two ways rather than three.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Go's net/http masks a password in the dialed URL before it builds the
*url.Error, so an error from a target configured with a password in its
userinfo never contains the raw URL. The verbatim substring guard inside
sanitizeProviderErr therefore matched nothing and scrubbed nothing. Both
the credential and any api-key query parameter on the same URL leaked
through the admin API and the dashboard tooltip.
Add sanitizeTargetErr. It tries the plain match first and then, only when
the configured userinfo carries a password, rebuilds the exact masked
form net/http produces and scrubs against that too.
Widen the scrub-order test so the secret straddles the truncation point,
which makes it fail if the scrub ever runs after the truncation. Drive
every new credential case from a real http.Client error against a closed
listener rather than a hand-written string, and assemble those fixture
URLs from parts so no whole credential-bearing URL literal sits in the
source. Drop the tracker doc paragraph describing a nil-error failure
that no caller produces, and remove the wall-clock dependency from the
sweep interval test.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a Health column to the MCP servers and A2A agents tables in
TargetsView. An unhealthy target renders a destructive badge with the
age of its last check and carries the failure count, source, and
scrubbed last error in the title. An unknown target renders an outline
badge and, when it has failed but never succeeded, the same failure
details in its title. A healthy target renders muted text with its last
observed latency and no badge, matching the providers' convention.
Add AdminTargetHealthView to the API types with the wire contract the
Go side emits, a bare formatElapsedAgo helper next to formatAgo, and
column-cell tests for all three states. Regenerate the baked admin
assets.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
targetHealthLastSweepUnixNano moves to Gateway's first field. It is
read and written only through sync/atomic's 64-bit functions, and
the sync/atomic package guarantees 64-bit alignment on ARM, x86-32,
and 32-bit MIPS only for a struct's first word — Traefik ships armv7
and 386 builds alongside amd64/arm64. Every other field keeps
golangci-lint fieldalignment's pointer-grouped ordering; verified
with the standalone fieldalignment analyzer that this is the only
field out of place and adds no struct-size padding, only a documented
+8 GC-scan-prefix cost.
docs: catches up README's admin `health` object and Target health
"State" bullet to F4 (previous commit, a0c8607) — three ways to
read `state: "unknown"`, and that consecutiveFailures/lastCheck/
lastError/latencyMs/source stay present when a target has been
observed but never succeeded.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
writeTargetMetrics now returns before emitting even
llmgateway_target_healthy's own HELP/TYPE header when neither an
MCP server nor an agent is configured, mirroring
writeStoreHealthMetrics' identical behaviour when Redis is
unconfigured.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
exerciseTargetHealth's traffic-sourced assertion (mcpProbeServerName
must read healthy/traffic, F5 review) raced its own trigger: GET
/admin/api/targets was the first-ever sweep-capable route this
harness hit, so the background sweep it started could beat the
response encode and overwrite source from traffic to probe under
Yaegi's interpreter overhead. exerciseTargetHealthProbeWarmup now
runs first, before exerciseHandler, against a dedicated, never-
proxied MCP target — its health can only ever be written by the
probe, so polling it to healthy/probe both proves the interpreted
active-probe path end to end and spends this gateway's one
probeInterval-bounded sweep before any traffic runs, making every
later target-health assertion in this harness deterministic.
probeInterval moves from 10s to 60s so no later exercise can trigger
a second sweep mid-run. TARGETHEALTH-1's expected state also updates
to unknown, reflecting F4 (a2d152b1): mcpProbeServerName's only
observation is a failed tools/call, so it has never once succeeded.
mcp_a2a_test.go's ClientCanceled_RecordsNothing test replaces its
30ms-cancel-vs-200ms-sleep timing margin with a channel handshake
(cancel only once the handler is genuinely reached, then release
it), removing any wall-clock dependence.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
mcpFederatedToolsList's fan-out and mcpFederatedToolsCall both
derive their outbound context from r.Context(). A client cancel
(context.Canceled) or the shared fan-out deadline expiring
(context.DeadlineExceeded) is not evidence about a server's own
health, so neither is recorded any more, mirroring
recordTargetProxyHealth's own client-cancel rule. A server that is
genuinely hung is still caught by the active probe sweep or its own
per-server proxy health.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Combines three closely-coupled target_health.go fixes that all touch
targetHealthTracker.record/stateForLocked in the same commit, since
splitting them would leave an intermediate commit that fails to
compile or a test asserting behaviour its own commit does not yet
implement.
F1: record gains a rawURL parameter and scrubs err's text through
sanitizeProviderErr before storing lastError, so a target URL
configured with an embedded credential can never leak through GET
/admin/api/targets. The scrub runs before truncateRunes so a long
embedded URL cannot survive truncation unscrubbed. Every probe, the
per-server proxy hook (mcp_a2a.go), and both federation record
sites (mcp_federation.go) now pass their target's configured URL
through.
F3: maybeSweepTargetHealth re-checks the elapsed interval after
winning the single-flight CAS, closing a window where a caller's
own stale interval read could still win the CAS once a separate,
faster sweep had already started and finished within the same
interval.
F4: targetHealthEntry gains everOK, set on record's success branch.
stateForLocked now reads a target that has never once succeeded as
unknown rather than healthy, even below failureThreshold — the
threshold still protects against a single transient first failure,
it just does not promote a target this tracker has never seen
answer to healthy. targetHealthSnapshot gains observed so the admin
view can still tell apart never-observed (omit every other field)
from observed-but-never-succeeded (keep them) despite both reading
state unknown. The now-redundant lastOK field is removed.
Also included: target_health_test.go's own test-hygiene fixes
(F6) — an idempotent, t.Cleanup-registered unblock ahead of
srv.Close so a failing assertion cannot leave a released server
handler blocked until the test binary's own timeout, and an atomic
hit counter in the disabled-sweep test — since they land in the
same file as the tests above.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add the targetHealth config row and a TargetHealthConfig reference
section, a Target health section under MCP and A2A describing the
passive recording rules and the active probe rules per target kind,
the health field on GET /admin/api/targets, and the
llmgateway_target_healthy gauge with its aggregation-across-replicas
note.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a second MCP server (target-health-down) pointed at a closed
listener, and enable targetHealth in the harness config. After the
existing federated tools/call probe, exerciseTargetHealth reads GET
/admin/api/targets and asserts the probe server reads healthy with
source traffic, then drives targetHealthFailureThreshold consecutive
failures through the direct per-server proxy path against the dead
server and asserts it flips to unhealthy on both GET
/admin/api/targets and llmgateway_target_healthy on /metrics.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a health object to each row of GET /admin/api/targets
(adminTargetHealthView, admin.go): state, an RFC3339 lastCheck,
lastError, source, and latencyMs, every optional field omitted while
the target has never been observed. Both serveAdminTargets and the
MCP/agent listing routes (handleMCPServers, handleAgents) trigger the
active probe sweep before responding, so admin polling and client
listings double as the heartbeat that drives it.
Add llmgateway_target_healthy to /metrics (metrics.go): a 1/0 gauge
labeled by kind (mcp or agent) and target name, with no sample at all
for a target the tracker has never observed. serveMetrics triggers
the sweep too, so a Prometheus scrape becomes a heartbeat as well.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a per-replica, in-memory health tracker for MCP servers and A2A
agents (target_health.go). The tracker records observations from real
proxied traffic (handleTargetProxy, mcp_a2a.go) and federated calls
(mcpFederatedToolsList/mcpFederatedToolsCall, mcp_federation.go), and
derives a three-state readout (unknown, healthy, unhealthy) from a
configurable consecutive-failure threshold. Add TargetHealthConfig to
Config, validated at construction. The tracker never gates routing:
federation and the target proxy keep contacting every configured
server regardless of its recorded health.
Extend proxyResult (routes_passthrough.go) with status and
clientCanceled so handleTargetProxy can classify a proxied response
without proxyUpstream knowing anything about target health.
Add maybeSweepTargetHealth, an opt-in, lazy, single-flight active
probe sweep with no timer and no long-lived goroutine, matching this
plugin's lack of a Close hook under Traefik hot reload. A probe uses
the initialize/close handshake for a Streamable HTTP MCP server, a
plain unread GET for a legacy /sse server, and a GET of the agent card
path for an A2A agent. handleMCPServers and handleAgents now trigger a
sweep at the top of each call.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The Charts view could break usage down by user, group or total, but not
by model. Per-model data existed only as day-window attempt and failure
counts, which measure availability rather than delivered usage.
A served model now accumulates the same four metrics a user or group
does. The scope kind is "model" and its id is the canonical
"provider/model", so a request that fails over counts against the
provider that answered it, and one bare id served by two providers ranks
as two rows.
The request counter is written after the response, in the batch that
already writes tokens and cost. That is the first point at which the
serving model is known, so no extra round trip is needed and admission
never double counts.
GET /admin/api/usage/models ranks models for one metric and window. It
returns only models with non-zero usage, by operator requirement: the
catalog runs to hundreds of models and a padded ranking is unreadable.
The read stays one storeGetMulti over the catalog.
The Charts view gains a Models tab with its own metric picker, and the
scope picker gains the models that have traffic.
bareWinner picked the first provider in global sorted order that served
a bare model id, ignoring which providers the caller's group could
actually use. resolveAgainst then denied the request whenever that
provider fell outside the group, even when another provider serving the
identical id was already permitted. A caller entitled to a model could
receive errModelDenied purely because an unrelated, invisible provider
sorted first alphabetically.
bareWinner now takes the group and prefers the sorted-first provider it
may use, falling back to the global sorted-first provider only when the
group is authorized for none of them. That fallback keeps a genuinely
unauthorized request returning errModelDenied, not the wrong
errModelUnknown a "no visible owner" result would produce. resolve,
resolveAliasTarget, and the alias-shadow collision check all route
through this corrected selection.
listFor now computes each bare id's owner from the group's own visible
providers and adds "provider/model" disambiguating entries only when the
group can see two or more owners of that id. A provider invisible to the
group never appears in that group's catalog, bare or prefixed.
warnCollisionOnce keeps logging the global collision unchanged, since a
provider collision is a fact about the fleet's configuration that every
operator must see regardless of which group lists models first. Its
message now says winner is the global default, because a group's own
request can resolve to a different provider than the one logged.
Adds registry_test.go coverage for the production repro, the
both-allowed and neither-allowed non-regression cases, the listing
behavior, and the alias path. Adds a yaegi-check probe proving the fix
resolves and serves under the real interpreter, and updates one existing
listFor test that asserted the pre-fix prefixed-form behavior.
The README claimed discovery runs again on discoveryInterval "in the
background", which reads as a periodic timer. There is none. maybeRefresh
is called from ServeHTTP only, so the interval is checked per incoming
request and a refresh is spawned when it has elapsed.
That distinction is what an operator actually needs, because it has three
consequences the old wording hid: an idle gateway never refreshes at all,
a newly published upstream model takes up to discoveryInterval to appear,
and there is no manual trigger since the admin API is read-only.
The plugin runs inside the Traefik process, so declining to start a
periodic goroutine is deliberate rather than an oversight. Documented as
behaviour, with the option of lowering discoveryInterval on the providers
whose catalogues actually change. Also corrected the two provider config
table rows that carried the same implication. Documentation only.
The federated /mcp endpoint ignored the Accept header and always
returned application/json. A real MCP client that sends Accept:
application/json, text/event-stream expects SSE framing. It fails with
no data line in SSE response when it receives plain JSON instead.
handleMCPFederated now negotiates the response format once per request
from the Accept header. It threads the result through
writeJSONRPCEnvelope, writeJSONRPCResult, and writeJSONRPCErrorResponse
as an explicit parameter. A request with no Accept header, or one that
omits text/event-stream, still gets byte-identical plain JSON. Only an
explicit text/event-stream token in the Accept list switches the
response to SSE framing. SSE framing is one data line carrying the
JSON-RPC envelope, terminated by the required blank line.
Error responses and every method path honor the negotiated format.
Notification responses stay 202 with no body regardless of Accept.
The SSE branch reuses sse.go newSSEWriter and writeData instead of a
second encoder. writeData already writes the single-frame, correctly
terminated shape this route needs.
Adds a compiled test file with a captured pre-change JSON byte
baseline. It also adds explicit SSE negotiation and framing tests. A
new yaegi-check probe exercises both framings under the real
interpreter.
Add a provenance-est provider whose upstream answers with no "usage"
field at all, driving a real, non-streaming chat completion against it
under Yaegi. exerciseUsageProvenance then parses GET /metrics with the
real Prometheus TextParser and confirms both new families
(llmgateway_usage_provenance_requests_total and
-_tokens_total) render as COUNTER type with a distinct "estimated"
series for this provider and a distinct "reported" series for openai's
own already-driven traffic, then confirms GET /admin/api/overview
surfaces the identical estimated observation and never leaks a
"reported" key into the compact admin summary.
Run at the end of run(), after every other probe that already reads
/metrics or /admin/api/overview, so it never perturbs a counter an
earlier probe asserted on.
Rebuilds webui/dist/ via vite build and re-runs webui/generate.mjs to
bake the usage-provenance badge changes into admin_assets_gen.go, the
file admin.go's serveAdminPage/serveAdminAsset actually serve.
Add AdminProvenanceView and AdminProviderView.provenance to types/api.ts,
mirroring admin.go's new fields, then surface two badges beside the
existing health and latency badges in ProvidersView.vue: "estimated N"
(secondary variant) and "unbilled N" (destructive variant), each with a
hover title explaining what happened and why it matters. Neither badge
renders when that provenance kind has no observations yet, matching the
existing health/latency badges' own convention.
Estimated and unbilled stay two separate badges, never merged into one
indicator: collapsing them would hide the one thing this feature exists
to surface, that a provider serving completions entirely free against
every budget is a materially different failure than an approximated
bill.
Built with shadcn-vue's existing Badge component; no hand-rolled markup.
Add adminProviderView.Provenance to GET /admin/api/overview: a compact,
per-provider map keyed by provenance kind, populated only for
"estimated" and "unbilled" observations. "reported" never appears here,
since it is the healthy default and AttemptsDay/FailuresDay already
describe it; the full reported/estimated/unbilled breakdown, by request
and by token, stays on GET /metrics.
buildAdminProvenanceViews mirrors buildAdminLatencyViews' own shape:
one batched g.provenance.snapshot() read per response, grouped by
provider, so this adds no additional store round trip and keeps the
already-large overview response small, matching the existing latency
summary's own size budget.
A test proves the reported/estimated/unbilled filtering, verified
against a real mutation of the exclusion guard before being kept.
Token accounting has three provenances today: reported (real provider
usage), estimated (a non-streaming response with no usage, prompt
substituted from request body size), and unbilled (a streaming response
with no usage, request counted but zero tokens billed). Nothing on any
read surface distinguished them before this change.
Add a per-process provenanceStore (metrics.go), recorded in
runUnified right after the existing estimated/unbilled classification
(routes_unified.go), for a candidate whose response actually completed.
A failed-over candidate is never recorded, so ordinary failover noise
never pollutes the unbilled signal.
Expose two new counter families on GET /metrics:
llmgateway_usage_provenance_requests_total and
llmgateway_usage_provenance_tokens_total, both labeled by provider and
provenance. Together they answer the three operator questions this
change targets: what fraction of billed tokens came from real reported
usage, which providers produce estimated or unbilled accounting, and
how often the unbilled case happens at all. Both families are
in-process only, so sum() is the correct cross-replica aggregation,
matching this file's other per-process families.
Billing itself is unchanged: this only records which provenance an
already-billed number took, never a new number.
Tests cover reported/estimated/unbilled classification, that billing
matches the pre-existing contract exactly, and that estimated and
unbilled never collapse into one series. Each behavioral assertion was
verified against a real mutation of the source before being kept.
Add streaming and non-streaming latency badges beside the discovery
health badge on each provider row. Streaming shows average time to
first byte, labeled ttfb, since it is the only load-sensitive figure
this data provides. Non-streaming shows average total duration,
labeled avg, never latency or ttfb, because a non-streaming provider
buffers the whole completion before sending anything and its ttfb
carries the same output-length contamination as its duration.
Badges omit entirely when a provider has no observations yet, so
absence never reads as a fabricated zero. Every badge title states the
per-replica, in-process scope explicitly: this data comes from the
same in-process accumulator the rate-limit counters use, never Redis,
so a single poll reflects only whichever replica answered it.
Mirror admin.go's adminLatencyView type in types/api.ts, keyed by the
literal streaming/non-streaming union admin.go emits. Add
formatLatencyMs to lib/format.ts with spec tests alongside its
siblings in format.spec.ts.
Add exerciseLatencyMetrics, proving the new upstream-latency histograms
render correctly under the real Yaegi interpreter, not merely that a
compiled go test accepts the same bytes. Compiled tests have repeatedly
passed on this codebase while the interpreted shape failed, which is
exactly what this probe caught: two bugs in the core feature commit,
found and fixed only because this harness exists.
Parse the rendered body with prometheus/common/expfmt's real
TextParser, the same library a real Prometheus server's scrape loop
uses, instead of this project's own hand-rolled parser. A malformed
histogram fails to parse outright under a real parser, matching the
failure mode a live scrape target actually hits, rather than merely
failing one hand-written assertion.
No new traffic is driven here: the existing exerciseAttemptAccounting
sub-probe already posts one real, non-streaming chat completion against
the openai provider, so the duration histogram already carries a real
observation by the time this probe runs.
Add github.com/prometheus/common/expfmt as a dependency of this
harness module only, isolated from the plugin's own stdlib-only go.mod
by design (this package's own doc comment). Go raised this module's go
directive from 1.22 to 1.25.0 to satisfy the new dependency; the
plugin module itself is untouched.
Add time-to-first-byte and total-duration histograms for every upstream
call, observation only. No throttling, priorities, or control logic
reads this data. The operator wants strain-based throttling later and
needs real traffic data first.
Extend watchdogBody (timeout.go) rather than add a new interception
layer: it already wraps every upstream response body for provider
adapters, native passthrough, and the MCP/A2A target proxy, and already
observes the first successful read. armLatency arms a watchdogBody
after construction, so newWatchdogBody keeps its existing signature and
behavior; its own pinned tests still pass unchanged. A timestamp plus a
few atomics are the only added cost; Read and Close each pay one nil
check when latency collection is disabled.
Expose llmgateway_upstream_ttfb_seconds and
llmgateway_upstream_duration_seconds on the existing Prometheus
endpoint (metrics.go). Both are in-process, per-replica histograms,
never store-backed, so sum() is always the correct cross-replica
aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and
short completions finely, cover the bulk of real chat completions, and
bracket the five-minute default request timeout. Labels are provider
and stream always, model only when Config.Metrics.ModelLabel is set,
matching the existing per-model opt-in convention. Add a compact
per-provider, per-stream average summary to GET /admin/api/overview,
sourced from the same in-process store, without a full histogram or a
per-model breakdown.
Every write path is gated on metricsEnabled(cfg), so a deployment with
metrics disabled collects nothing and the histogram families never
render. No behavior change to request handling: same responses, status
codes, headers, and accounting.
Fix a Yaegi-only bug this feature's own context-threading change
exposed: two distinct empty struct types used as context.WithValue keys
compare equal to each other under the interpreter's interface equality,
though never under compiled Go. Threading a second recorder
(latencyRecorder) alongside the existing attemptRecorder made this
observable for the first time, since compiled tests never see it and
the plugin previously carried only one such key. Give
latencyRecorderCtxKey a field so its size differs from
attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction
before and after the fix.
Fix a second bug, reproducible under compiled Go too: io.Reader may
legally return its final data together with io.EOF in one call, and
Go's real net/http body reader does this for small, fully-buffered
responses. The original TTFB capture lived inside the err == nil
branch, so it never fired for a typical small non-streaming completion.
Move the check ahead of the error branch so it fires on any n > 0,
leaving the pinned timer-reset and Close semantics untouched.
Tests mutate the real source per case and confirm each one fails first,
recorded in each test's own doc comment.
The Traefik Plugin Catalog analyzer rejected v1.0.43:
import "github.com/lukaszraczylo/oss-telemetry" error:
unable to find source related to: "github.com/lukaszraczylo/oss-telemetry"
piceus fetches the source archive at the released tag and interprets it
with Yaegi. It never runs go mod download, so every non-stdlib import in
production code must exist in the archive own vendor/ tree.
.gitattributes carried "vendor/ export-ignore" with an explicit warning
to remove it the moment production code imported a third-party package.
That happened when llmgateway.go took on oss-telemetry for anonymous
usage reporting, and the warning was missed. The tag itself always had
vendor/; git archive stripped it, so the tarball shipped a go.mod
requiring a module whose source was absent.
Verified by reproducing the analyzer: the v1.0.43 archive contains 0
vendor entries, the fixed archive contains 71, and interpreting the
extracted archive under yaegi passes.
F2: the idle timer resets on DECODED body bytes, after net/http's own
gzip decompression. A gzip'd response whose compressed bytes arrive
steadily but whose deflate stream yields nothing for the whole window
still gets aborted. The previous wording claimed a slow-but-progressing
body is always unaffected; adds a qualifying paragraph explaining the
gzip case and why fixing it (measuring wire bytes) is out of scope at
the five-minute default.
F3: reworded the "silence ends a request, never elapsed time" framing.
A streaming forwarder reads a chunk, writes it to the client, then
reads again; the write is not watched, so time blocked on a slow
client counts against the same idle window as a silent provider.
Aborting is still correct, but the log line no longer claims the
provider specifically caused it.
F5: documents that neither timeout is ever retried, and why: a
ResponseHeaderTimeout failure wraps context.DeadlineExceeded, which
Retry's own transient classification excludes; a mid-body watchdog
timeout fires after the retry mechanism has already returned.
F8: softened "Every upstream request this gateway makes" to the
routes actually covered, and added a "Federated MCP" note: the
federated /mcp endpoint's two direct calls are not wrapped by the
idle-progress watchdog, but are not unbounded either — they still get
ResponseHeaderTimeout from the shared client, plus their own 20s/120s/
5s context.WithTimeout budgets. Worst case is a bounded stall, not a
forever one.
exerciseRequestTimeout only drove a provider that never sends headers,
so it exercised Transport.ResponseHeaderTimeout alone.
watchdogBody.fire — the interpreted method value handed to
time.AfterFunc, plus context.CancelFunc and sync/atomic's function API
— never ran under that probe, since the request never got far enough
to construct a watchdogBody at all (finding F7, coordinator
adversarial review, 2026-08-23).
Adds hungbody, a provider whose upstream sends headers immediately and
then stalls, and exerciseRequestTimeoutMidBodyStall to drive it.
Verified the two probes exercise different code paths: removing
watchdogBody's construction from upstreamBytes left the existing
header-timeout probe passing unchanged, while the new probe failed
with status 200 instead of 502. Reverted before this commit.
F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a
plain `now` variable while a background refresh goroutine read it
through nowFn, racing under -race in about 14% of full-suite runs once
this branch's timer-heavy tests shared the parallel phase. Guards the
clock with a mutex instead. registry.go is untouched.
F4: a mid-body watchdog timeout never reached provider-health
accounting. limiter.recordProviderAttempt only ran at header-arrival
time, so a provider that reliably stalled mid-body reported
attempts=1 failures=0 forever. watchdogBody.fire now invokes the same
attemptRecorder a second time, as a failure, when it fires. Reuses the
existing recorder rather than adding new plumbing; this does mean two
attempts are recorded for one logical request (one success at header
time, one failure at watchdog-fire time), not one attempt marked
failed outright.
F6: watchdogBody.Read reset the timer on any err == nil read,
regardless of n. A reader returning (0, nil) forever would have held
the watchdog open indefinitely. Reset now only fires when n > 0.
F9 (three items):
- proxyUpstream passed logPrefix as a bare providerName, which
newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an
MCP/A2A target as a provider. watchdogBody now takes an
already-formatted label from the caller instead.
- errUpstream was wrapped twice: once in watchdogBody's own error, once
again by every caller that already wraps it. watchdogBody now wraps
only errProviderTimeout; callers' existing wraps still make
errors.Is(err, errUpstream) hold.
- timeout_test.go read wb.timedOut directly in one place while every
other access is atomic; switched to atomic.LoadInt32.
Also rewords errProviderTimeout's message from "provider request timed
out" to "upstream progress stalled" (finding F3, documented not fixed
per coordinator ruling): the watchdog measures time since the last
successful upstream read, which can go quiet because of a slow client
write between reads, not only a slow provider. The new wording no
longer asserts which side caused it.
New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure,
TestWatchdogBody_ZeroByteRead_DoesNotResetTimer,
TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider.
Each mutation-verified per its own doc comment.
The VMServiceScrape example scraped Traefik's pods directly, so the
request carried Host: <pod-ip>:<port>. Traefik evaluates a router's
match rule before any middleware runs, and the only router in this
file requires Host(`llm.example.com`). The scrape request never
matched it, the llmgateway middleware never ran, and /metrics 404ed
before the plugin ever saw the request.
Add a second, Host-free IngressRoute matching PathPrefix(`/metrics`)
and carrying the same middleware, so an in-cluster scraper reaches the
route regardless of what Host header it sends. Document the
reachability trade-off this introduces (this route is not
Host-restricted, though /metrics stays gated by its own auth inside
the plugin) and name the alternative fix the review also offered:
relabelConfigs rewriting __address__ to the ingress hostname before
the request is sent, keeping one Host-scoped router instead.
Three blocking fixes from a second round of adversarial verification,
plus two should-fix items bundled in since they touch the same code.
storeDown detection covered only the fail-CLOSED path. failOpen
defaults to true (newConfiguredLimiter), so on the shipped default a
broken store made storeGetMulti return ok=true after silently reading
the empty in-process fallback, bypassing every storeDown guard in
metrics.go. Add limiter.configuredStoreDown, checked alongside ok in
currentUsage and providerUsage, so a fail-open read served from the
fallback is treated the same as a fail-closed refusal. The pinning
test previously hand-set failOpen=false, which meant it only ever
exercised the case that was already correct; it now leaves failOpen at
its true default.
The escaping rewrite from the previous round achieved injectivity by
copying an invalid UTF-8 byte through raw, which can make the output
itself invalid UTF-8. Measured against a real Prometheus scrape parser
this round: invalid UTF-8 in a label value is fatal, the scrape target
goes down, which is worse than the duplicate series the previous round
was fixing (measured non-fatal, dropped and logged, target stays up).
Replace an invalid byte with a deterministic \xHH marker instead,
proven injective and always valid UTF-8 by an exhaustive test over
every 1- and 2-byte input, not merely spot-checked. Also adds the fast
path a benchmark showed the byte-oriented rewrite needed: unescaped,
already-valid-UTF-8 input, the common case, returns unchanged with
zero allocations instead of paying a full copy.
llmgateway_rate_limit_rejections_total's HELP text claimed it "resets
only on restart", which stopped being true once the previous round
added generation-rotation bounding and reload-triggered eviction: a
scope resets when its user is removed, and every tracked scope resets
together in bulk the instant the tracked-scope cap is exceeded. The
HELP text and README now say so.
Adds llmgateway_limit_store_up, a per-replica gauge for this process's
own connection health to the configured limit store: with the store
flapping faster than its fail latch, every store-backed family can go
silent scrape after scrape with nothing on the endpoint itself saying
why. Also adds a test pinning that Gateway.pruneRejectionsAfterReload
runs only when authStore.maybeReload reports a real reload, never on
every request.
Fix one must-fix and one should-fix finding from re-verification of
d45125e, plus two smaller findings from the same pass.
MUST FIX: the F2 header clear ran on the first candidate attempt too,
so it ran on the default, failover-disabled path and deleted response
headers a different, earlier Traefik middleware had already staged
before this plugin ran (CORS, HSTS, request-id, and similar). Scope the
clear to i > 0, the only case that ever needs it. Add a test proving a
header staged before the gateway runs survives the default path, on
both a cache hit and a cache miss.
SHOULD FIX, coordinator ruling: drop the all-candidates cache pre-pass.
It silently served a different candidate's cached response while the
primary was perfectly healthy, which is failover with no failure. Check
each candidate's own cache entry immediately before that candidate's
own attempt instead. Request-path health, now fixed in the prior round,
is the correct mechanism for skipping a genuinely dead primary: once it
is marked unhealthy it is excluded from the candidate list entirely, so
a later request still reaches the next candidate's cache without the
dead one being dialed. Update the existing cache regression test to
drive enough requests to trip the health breaker before asserting no
repeat call, and add a dedicated test proving a healthy primary is never
bypassed for another candidate's cached response.
Guard the request-health record call with the enabled check. That state
is never read on the disabled path, so recording it there was dead work
plus a new contention point on requestHealthTracker's tracker-wide mutex
on every metered request, disabled or not. Add a test asserting the
tracker's internal state stays empty across the disabled path.
Add a test proving the 404 loud-log line is not rate-limited: drive two
404 failovers through the same Gateway in one request and assert both
lines appear, since a version that wrapped the 404 line in the same
rate limiter as the generic line would still pass every other test
(each of which only ever drives one 404 event against a fresh gate).
Document, but do not fix, that the cost guard cannot see a model
declared free via modelMeta: it resolves pricing only through the same
source real billing uses (pricing and the built-in table), which is
correct and consistent, but means an operator who declares a model free
the modelMeta way loses it as a failover candidate without realizing
why. README now says to use pricing instead for that case.
Every fix is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
Adds a "Request timeout" section explaining the progress-based design,
the two mechanisms, the global/per-provider override rule, and error
classification. States plainly that this changes behavior for every
deployment, since no timeout existed before.
Adds requestTimeout rows to the Config and ProviderConfig reference
tables.
Adds exerciseRequestTimeout: a provider (hungtimeout) with an 80ms
ProviderConfig.RequestTimeout, backed by an upstream that stays silent
for 600ms. The request carries no client-side deadline of its own, so
a pass proves the gateway's own adapter-level timeout aborts the hang,
not a caller-supplied context.
This complements the existing slowProviderName probe, which proves a
CLIENT-set deadline classifies correctly. It does not prove the
adapter's own timeout mechanism runs correctly interpreted, since it
never exercises it without client help.
Verified the probe is not a no-op: temporarily removing
Transport.ResponseHeaderTimeout from newAdapterHTTPClient made make
yaegi-check fail with status 200 instead of 502. Reverted before this
commit.
timeoutProviderModel uses a distinct id, not testDataWantModel:
reusing it made the new provider win bare-id model resolution and
silently hijacked an unrelated existing probe.
TestWatchdogBody_NoBlockingGoroutineLeak_AcrossManyRequests drove its
timed-out iterations against a server that never wrote headers. That
hits Transport.ResponseHeaderTimeout inside client.Do and returns
before a response ever exists, so watchdogBody.fire never runs and the
test could not have caught a leak in fire itself.
The mutation-verification pass (manually adding a permanently blocked
goroutine to fire, then reverting) confirmed this: the test passed
even with that goroutine present. Fixing the stalling server to write
headers first, mirroring the mid-body-stall tests above it, makes
fire run on every iteration. Re-running the same mutation now fails
the test as intended.
Adapters previously issued upstream requests with no timeout at all.
A provider that accepted a connection and then stalled hung the
gateway request forever. newAdapterHTTPClient now sets
Transport.ResponseHeaderTimeout, and every response body read
(providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream)
runs through a watchdogBody that resets on every successful read and
cancels the request once timeout passes with no progress.
The idle-progress design lets a long stream that keeps emitting tokens
run indefinitely. Only silence, never total duration, ends a request.
Config.RequestTimeout sets the global default (five minutes when
unset). ProviderConfig.RequestTimeout overrides it per provider. Both
are Go duration strings, validated at construction; a zero or negative
value is rejected rather than treated as unlimited, since that would
reintroduce the hang this feature fixes.
Native passthrough (routes_passthrough.go) and the MCP/A2A target
proxy (mcp_a2a.go) share the identical watchdog mechanism through
proxyUpstream's new timeout parameter.
A watchdog-fired timeout wraps errProviderTimeout and names the
provider. It never wraps context.Canceled, so it stays distinguishable
from a genuine client disconnect in every existing log classification.
Flip FailoverConfig.Enabled to default false. A version upgrade with no
config change must preserve prior behaviour, and failover changes which
provider serves a request. Update tests, README, and the yaegi-check
probe to opt in explicitly.
Add a mandatory cost guard. A candidate is skipped unless its input AND
output cost per token are each no greater than the primary candidate's,
resolved through the same canonical-then-bare pricing path the real
billing code uses. Unknown pricing is never treated as free: a known
primary against an unknown candidate skips, an unknown primary against a
known non-zero candidate skips, everything else that is equal is
allowed. Not configurable.
Fix four blocking findings from adversarial review of commit 5d62479.
F1: openai-type adapters mutate the request map in place to force
stream usage reporting. Reusing one map across candidates let a later
candidate read back an earlier candidate's own injection and leak a
usage-only SSE chunk to a client that never asked for it. Every
candidate now attempts against a fresh copy of the original request.
F2: a streaming candidate that fails before writing anything still
stages SSE-only response headers. Those headers now get cleared before
every candidate attempt, so they cannot survive into a later
candidate's plain JSON response.
F3: the request-health gate used the retry-shaped isTransient
classification, which treats a non-429 4xx as a successful outcome. A
provider with a dead API key was never skipped and was retried forever.
The gate now reuses failoverEligible's own classification instead.
F4: a shared request context that an earlier candidate exhausted was
attributed to whichever candidate happened to run next, opening a
healthy provider's breaker for a timeout that was not its fault. A
candidate is no longer attempted once the shared context is already
done; the loop stops and reports the last real error instead.
Also fix three should-fix findings from the same review.
F5: a failure arriving while the request-health breaker is already open
but not yet past its cooldown window is now a no-op, instead of
resetting backoff and sliding the open window forward on every such
failure.
F6: the request-health gate now records at most one outcome per
candidate per logical request, not one per same-provider retry attempt,
so retry.attempts alone can no longer open the breaker on a single
request.
F7: every candidate's cache entry is now checked before any candidate
is attempted, so a cached failover response no longer costs a round
trip to a still-broken primary first.
F10: the generic "failing over" log line is now rate-limited to once
per 30 seconds during a sustained outage, reusing the existing logGate
type. The 404 loud-log line stays unthrottled, as ruled.
F9 (optional): skip building the failover candidate pool entirely when
failover is disabled.
Correct an overstated test comment on
TestHandleChat_Failover_DiscoveryUnhealthy_StillTried: it proves alpha
is still tried, not that ordering places it second.
Document, but do not fix here, that this plugin's HTTP client sets no
per-attempt timeout, so a provider that accepts a connection and then
hangs will never trigger failover. This is a pre-existing property of
the shared HTTP client, not specific to failover.
Deferred: F8 (surfacing request-health state on the admin dashboard)
touches admin.go, which a sibling branch is also editing concurrently;
left for a follow-up to avoid a merge conflict in a file this change
does not otherwise need.
Every fix above is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
The metrics feature shipped in code only: no README section, no
metrics block in .traefik.yml testData, nothing under examples/. The
Go doc comments were the entire manual for a human operator or a
config-generating assistant.
Add a README Metrics section covering the served path, both auth
modes (admin bearer token, optional CIDR allowlist), the CIDR-sizing
warning with the live-reproduced SNAT failure mode, the modelLabel
cardinality trade-off, the exposed metric families, the cross-replica
aggregation contract, label escaping, and store-outage behaviour. Add
a MetricsConfig reference table matching the existing AdminConfig
one, and a metrics row in the top-level Config table.
Add metrics: {enabled: true} to .traefik.yml's testData, the
catalog-facing smoke-test config, deliberately without allowedCIDRs
so a copy-pasted example never skips the README's own sizing
guidance.
Add a metrics block to the Kubernetes Middleware example and a
VMServiceScrape resource: the target cluster runs VictoriaMetrics, not
Prometheus, so the CRD differs even though the exposition format does
not. Verified the VMServiceScrape shape (apiVersion, selector,
endpoints, path) against the victoriametrics/operator documentation
rather than assuming it. Notes that /metrics is served by Traefik
itself, not by any backend service, since this middleware intercepts
the path before a backend is ever reached.
Add a third configured user whose name contains a double quote, a
backslash, and a newline, then assert GET /metrics renders its
Prometheus-escaped form and never the raw name. auth.go validates a
user name as non-empty only, so this is a real, legal configuration,
not a contrived one.
The compiled suite already proves escapeLabelValue correct
(TestMetrics_LabelValuesWithSpecialChars_Escaped,
TestEscapeLabelValue_InvalidUTF8_Injective), but this codebase's own
history is that a compiled pass has repeatedly said nothing about the
interpreted shape. json.Marshal produces the JSON-escaped literal for
the harness's own hand-built config document, kept separate from the
Prometheus escaping the probe actually verifies in the rendered body.
Three fixes from adversarial verification.
writeProviderMetrics rendered a storeDown providerCounters entry as a
hard zero, the same fabricated-reset hazard the request/token/cost
families already guarded against. Skip a storeDown provider or
provider-model entry instead of emitting it.
escapeLabelValue ranged over the string as runes, and Go substitutes
U+FFFD for every invalid UTF-8 byte. Two distinct configured names
that both contain invalid UTF-8 could therefore render as the
identical escaped value, colliding two series into one duplicate
series and failing the whole scrape target. Escaping is now
byte-oriented, which is injective and also avoids the per-rune decode
cost on a large value.
Every store-backed family's HELP text now states its aggregation
contract: with Redis configured, the value is the gateway-wide total
and identical on every replica, so summing across replicas overcounts
by the replica count and a cost alert fires early. Aggregate with
max() instead. Without Redis, the same family becomes genuinely
per-process and sum() becomes correct instead. The rejection counter
and the provider-healthy gauge get their own notes: the former is
always per-process regardless of Redis, the latter reflects each
replica's own independently-running discovery breaker.
Also replaces the AllowedCIDRs doc example (10.0.0.0/8) with a
cluster pod CIDR (10.42.0.0/16) and states plainly that a whole
RFC1918 range is never correct, since a SNAT-ing router or load
balancer makes external traffic arrive from inside it — reproduced
live against the target cluster.
Adds tests pinning three properties mutation-proofed against this
suite: every storeDown skip (deleting any of them now fails
TestMetrics_StoreDown_SkipsRatherThanFabricatesZero), the CIDR-check
running before auth.identify (TestMetrics_
AllowlistedRequest_NeverTouchesAuthFailureTracking), and
escapeLabelValue's injectivity under invalid UTF-8
(TestEscapeLabelValue_InvalidUTF8_Injective). Also covers the new
rejection-counter pruning end to end through a real file-sourced
reload.
Two correctness fixes from adversarial verification.
providerUsage returned every counter at its zero value on a
failed-closed store read, with no signal distinguishing that from a
real zero. metrics.go rendered the zero as a hard value. A single
Redis blip therefore read as a counter reset, and the recovery scrape
counted the whole prior total as a fresh increase, painting a phantom
multi-thousand-request spike from one transient error. Add storeDown
to providerCounters and set it on the failed-closed path, mirroring
scopeUsage's existing contract.
limiter.rejections grew without bound: nothing ever removed a scope
whose user was renamed or deleted from a hot-reloaded users file, so a
long-running process kept exporting a series for every name ever seen.
Bound the map with the same current/previous generation-rotation
authFailureTracker already uses, and evict a user's entry when
authStore.maybeReload performs a real reload. maybeReload now returns
whether it actually reloaded, so ServeHTTP only runs the eviction walk
on that rare event, never on every request.
Also corrects a stale comment: the rejection counter's store-down
branch fires on every request during a full outage, not rarely, as an
earlier comment claimed.
Cover an undefended half of runMeteredCall's own cacheable guard
(routes_unified.go): "!streaming" was asserted nowhere end to end.
Existing streaming tests only checked chunk delivery and mid-stream
accounting, never the cache header or store state.
Add a test that drives a real streaming chat-completion request
through a Gateway with caching enabled and proves no X-Llmgw-Cache
header appears, and that the cache holds no entry under the key a
non-streaming request for the same body would use.
Confirmed to fail against a real mutation (dropping "!streaming" from
the cacheable expression) before being kept; the mutation and its
revert are recorded in the test's own doc comment.
Cover failPolicyIncrBy (0% covered), and add direct tests for
storeIncrBy and storeGet against a real store, not only the nil-store
fallback path the existing suite exercised.
Each direct test proves four branches: a nil store always uses the
in-process fallback, a configured store's success returns the store's
own value rather than the fallback's, a store error records the
failure and falls through to the fail-open/fail-closed policy, and an
open store-down latch skips the network call entirely until it
expires.
Add a direct test for logSpawnDrop (0% covered), proving it logs once
per storeErrorLogEvery like its sibling logStoreError, sharing the
same rate-limit gate by design.
Every test was confirmed to fail against a real mutation of the
target function before being kept; each mutation and its revert are
recorded in the test's own doc comment.
Cover four zero-coverage functions on the /v1/messages route Claude
Code drives on nearly every call: openAIToolsFromAnthropic,
openAIToolChoiceFromAnthropic, responseTranslationError.Error, and
writeAnthropicProviderUpstreamError.
Add table-driven tests for every tool_choice form (auto, any,
tool-by-name, unrecognized, absent, non-map, nil), a multi-tool
mapping test that checks field order, description omission, and
malformed-entry skipping, a full request-to-response round trip
through a forced tool call, and a test proving no-tools/no-tool-choice
omits both keys rather than sending an empty array.
Add direct tests for the Anthropic-shaped upstream error envelope
(status passthrough, error.code, decoded error.upstream) and for the
translation-failure error prefix.
Every test was confirmed to fail against a real mutation of the
target function before being kept; each mutation and its revert are
recorded in the test's own doc comment.
Add failover.go with the full candidate pipeline. resolveWithCandidates
resolves a request exactly like resolve does, then adds every other
provider sharing the identical bare model id. orderedFailoverCandidates
applies a hard skip for request-path-unhealthy providers, deprioritizes
discovery-unhealthy providers without dropping them, fails open when
every candidate reads unhealthy, and caps the list at maxAttempts.
Add requestHealthTracker, a separate per-pod request-path health signal
fed by the same isTransient/isDeadlineExceeded classification
recordProviderAttempt already uses. It never widens the existing
discovery circuit breaker, per that breaker's own documented scope.
Rewrite runMeteredCall (routes_unified.go) to loop over candidates
instead of resolving once. Each attempt rebinds the attempt recorder to
its own provider, recomputes the cache key before the response is
cached, and re-injects the alias echo key an adapter may have deleted on
a prior failed attempt. Failover never runs once any byte has reached
the client, since statusTrackingWriter already tracks that state.
A failure falls through on everything except HTTP 400, matching the
brief exactly. A 404 failover logs loudly, naming both providers and the
model, since it usually signals a real configuration mistake.
Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to
true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A
single-provider deployment, or one with no overlapping bare model ids,
behaves exactly as before regardless of this setting.
Add a yaegi-check probe (tools/yaegi-check/main.go) driving two
providers under the real interpreter, one failing and one serving,
proving the candidate loop runs correctly interpreted.
Update README.md with a Failover section and correct the Retry
section's stale claim that no cross-provider failover exists.
Media routes (images, audio speech, audio transcriptions) do not get
failover in this change. Each has its own request-shape handling that
needs its own scoped follow-up.
Compiled tests have repeatedly passed on this codebase while the
interpreted shape failed, so the metrics feature needs its own probe
in this harness, not just go test coverage.
Enable metrics in the harness config, with modelLabel on so the
opt-in provider-model breakdown runs interpreted too. Add
exerciseMetricsRoute, driving four real requests: unauthenticated
from outside the allowed CIDR block returns 401, a non-admin key
returns 403, the admin key returns 200 with the expected TYPE lines
and live provider and model data in the body, and a keyless request
from inside the allowed CIDR block returns 200.
Run the new probe last, after the breaker probe settles, since GET
/metrics triggers the same maybeRefresh call every route does and
the breaker probe's own hit counting is timing-sensitive.
Serve a hand-written Prometheus text-exposition document at a
configurable path, defaulting to /metrics. The plugin is stdlib-only
under Yaegi, so this cannot use a client library.
Add MetricsConfig: Enabled gates the route, Path overrides it,
AllowedCIDRs lets an unauthenticated caller scrape from a trusted
source address, and ModelLabel opts into a per-(provider,model)
breakdown that defaults off to avoid a series-count explosion.
Reuse the existing admin bearer-token auth path instead of adding a
second credential check. Reuse the batched, chunked reads GET
/admin/api/overview and GET /admin/api/usage already use, so a scrape
costs no extra store round trip.
Expose request, token, and cost counters per scope, provider attempt
and failure counters, a provider-health gauge from the discovery
circuit breaker, and a budget-consumed-ratio gauge per limited scope
for Prometheus alerting. Add a rate-limit-rejection counter in
limits.go: this is the one number with no existing accounting to read
back, so checkAndCount now increments a small in-process,
per-process-lifetime counter on every violation.
Escape every label value for backslash, double quote, and newline. An
unescaped quote in a label value, such as a discovered model id, makes
the whole scrape unparseable, and Prometheus reports an unparseable
scrape as the target being down rather than as one bad line.
Cover the auth gate matrix, exposition structure (TYPE per family,
counters ending _total, no duplicate series), label escaping
round-trips, the model-label opt-in, and counters reflecting real
seeded usage rather than zeros.
Replace the alias-echo rewrite on the Anthropic passthrough branch
with a byte-preserving field replacement. The prior version decoded
the response into a Go map and re-marshaled it. This silently
corrupted a JSON integer beyond float64's exact range. It also
rewrote 1.0 to 1, HTML-escaped angle brackets and ampersands, and
re-sorted every key. A tool_use block's "input" is arbitrary
model-generated JSON. A long numeric id inside tool arguments could
be silently corrupted on a route documented as passthrough.
rewriteModelField now walks the response with a JSON decoder and
copies every field but "model" as raw bytes.
Add a test with a 20-digit integer, a 1.0 float, and a literal "<"
character inside tool_use input. It asserts the response is
byte-identical to the source apart from the model field.
Add a negative test for the anthropic-beta header allowlist. It
asserts Authorization and Cookie never reach the upstream. It also
asserts the upstream sees exactly one x-api-key value: the gateway's
own key, never the client's. A single Get() call could miss a second,
leaked value, since Set() and Add() interact in a way a naive check
would not catch. The test checks the Values() length explicitly.
Add a test case for the native passthrough route's prompt-cache
billing. It fails if cache_creation_input_tokens or
cache_read_input_tokens drop out of the billed prompt count. This
closes a production billing change that previously carried no
coverage at all.
Decouple response delivery from usage extraction on the passthrough
branch. A response body read failure, or a malformed usage field, no
longer both discard the response and misreport the failure as an
upstream connection error. A read failure still returns an error, now
correctly classified as a translation failure. A malformed usage
field no longer blocks the response at all: the client still gets
it, and the existing zero-usage estimate takes over for accounting.
Return the adapter's own usage value on a chatCompletion error in the
translated path, instead of discarding it. This has no effect while
streaming stays rejected up front. It prevents the same critical bug
from resurfacing once streaming support lands.
Normalize tool_result content to a plain string before applying the
error prefix. The prior version only prefixed a Go string. Anthropic's
array content form, which Claude Code sends, lost the is_error signal
entirely. Absent content now becomes an empty string instead of the
JSON null OpenAI rejects.
Treat "stream": null the same as an absent key. The prior fix for a
non-boolean stream value also rejected null. An SDK that serializes an
unset optional field as null must still get an ordinary non-streaming
response.
Drop a yaegi-check assertion that could not fail. The fixture
configures no cache, so the header it checked was never set regardless
of what the code under test did. A mutation pass confirmed this: it
deleted the line the assertion was meant to guard, and make
yaegi-check still reported OK.
Document that this route now intercepts POST /v1/messages
unconditionally, even under passthroughUnknown, where it previously
fell through to the next handler. Add the note to both the new README
section and Known limitations. Correct the passthrough table's claim
about response bytes to describe the actual guarantee.
ProvidersView.vue's accordion detail row hardcoded text-destructive red
for any non-closed health state, while the trigger row's Badge
deliberately used grey (secondary) for half-open and red (destructive)
only for open. Add healthDetailClass so the detail row's color always
matches the badge's own severity, instead of overstating a half-open
probe as an error.
Add a describe block for formatUntil to format.spec.ts, covering the
unset sentinel, undefined, a future timestamp, and a past one. It had
no test coverage since being added in the previous round.
The Provider health section claimed a broken provider stops being
re-probed "every single interval forever", while its own third bullet
said the opposite: under stock config the interval remained the
binding constraint and nothing was actually suppressed. Rewrite the
opening paragraph and the bullet to state the real relationship
plainly: the backoff window only ever reduces re-probe frequency once
it exceeds discoveryInterval, and the shipped defaults are chosen so
that holds.
Update every remaining reference to the old 30m maxOpenDuration default
to the new 6h value, in the BreakerConfig reference table and the
Provider health section, and explain what a misconfigured relationship
between discoveryInterval and maxOpenDuration does.
Address the design finding and remaining defects from round 3 of
adversarial review.
Raise defaultBreakerMaxOpenDuration from 30m to 6h. Round 2 fixed the
breaker so it never retries more than plain interval throttling, but
left it cadence-neutral: with maxOpenDuration shorter than the 1h
default interval, the interval always dominated and backoff never
suppressed anything. Measured over 24h with stock config, admits
equaled the interval-only baseline exactly. At 6h, a permanently broken
provider now backs off 1h, 2h, 4h, 6h, 6h, roughly 5 attempts a day
instead of 24. The 24h cadence test now asserts admits are strictly
fewer than baseline, not merely no worse.
Correct the emitted openUntil to account for the interval floor.
Because the interval gate applies unconditionally, the real next probe
time is max(openUntil, lastRefresh+interval), not the raw backoff value
alone. On pure defaults a fresh trip set the raw value to +1m while the
real next probe was at +1h, a 59 minute skew the dashboard rendered as
a countdown that reached zero and then silently vanished. snapshot now
emits the corrected value while open; closed and half-open are
unaffected.
Revert a canceled half-open probe to open instead of leaving it at
half-open. A half-open breaker reports healthy by design, so a canceled
probe previously left a still-broken provider reading healthy until the
next attempt happened to land. Reverting costs nothing extra, since the
interval gate already governs when that next attempt is admitted either
way.
Rewrite the discovery-suppression claims in doc comments: the earlier
wording claimed re-probing stops "every single interval forever" under
all conditions, which the interval-floor fix from round 2 already made
false, and which contradicted this file's own surrounding explanation.
Both now state plainly that suppression only occurs once backoff
exceeds discoveryInterval.
Add a sentence to recordHealthLocked's doc comment stating that
context.DeadlineExceeded counts as a plain failure, matching
recordProviderAttempt's SHOULD-1 ruling, and a named test proving it.
Tests: two new snapshot tests pin the openUntil correction in both
directions. The cadence test's baseline comparison is now strict. A new
test proves the zero value truly disables the breaker rather than
tripping on the first failure. A new test proves a canceled context
never gets misclassified as a failure, completing the bidirectional
check the existing success-direction test already had. The concurrent
test now asserts backoff stays within its ceiling and health stays a
valid state on every iteration, rather than only checking for panics
and data races.
Add healthState and openUntil to AdminProviderView, and a formatUntil
helper alongside the existing formatAgo/formatTimestamp pair, rendering
a future ISO timestamp as "in Ns".
ProvidersView.vue shows a Badge next to a provider's other status
indicators, destructive for open and secondary for half-open, using the
existing shadcn-vue Badge component rather than a hand-rolled
indicator. A closed provider, the healthy common case, renders no badge
at all, matching the existing convention for the per-model rate badge.
The expanded accordion content adds a matching detail row.
This was the brief's stated motivation for the admin surfacing work: a
broken provider looked identical to a working one in the dashboard.
That gap was previously closed at the API layer only; this closes it in
the UI clients actually see.
Regenerates admin_assets_gen.go from the rebuilt webui via make
admin-ui, per this repository's own convention of committing the
generated asset alongside its source change.
Add BreakerConfig to the full configuration reference, alongside
RetryConfig and CacheConfig, and add a breaker row to the top-level
Config table. Add a new Provider health section explaining the three
states, the failure classification, the neutral treatment of a
canceled context, and the scope limitation: the signal reflects the
discovery endpoint only and must not be the sole reason to refuse
routing to a provider.
Also document the new healthState and openUntil fields GET
/admin/api/overview returns per provider, in the same sentence that
already lists discoveryEnabled, modelRates, and modelMeta.
Add a breaker probe to the yaegi-check gate, alongside the existing
handler coverage rather than replacing it. Drives a provider whose
/v1/models 401s, the xiaomi production case, through the full closed to
open to half-open to closed cycle against the real interpreter.
This closes a gap adversarial review found: registry.go added a new
interpreted named int32 type with a String method and a new
classification call in recordHealthLocked, and nothing exercised either
under Yaegi. limits.go's matchesSentinel already documents that this
class of divergence between compiled and interpreted behavior is
invisible to go test and only an interpreted probe can catch it.
The probe configures a short breaker window (2 failures, 3 to 6 second
backoff) and a 1 millisecond discovery interval on a dedicated "brk"
provider, so the interval floor never binds and the breaker's own
timing is what the assertions observe. It proves the breaker opens
after sustained failures, suppresses discovery traffic for the whole
open window, and closes again once the provider recovers.
Address five findings from adversarial review of the discovery circuit
breaker. All are behavioral corrections to registry.go; none touch the
request path.
Interval floor (blocking). tryBeginRefresh let an open breaker's own
backoff window replace the plain discoveryInterval gate instead of
adding to it. Since defaultBreakerMaxOpenDuration (30m) is shorter than
defaultDiscoveryInterval (1h), a permanently broken provider was probed
roughly twice as often as before this feature existed. The interval
gate now applies unconditionally, closed or open alike, so backoff can
only ever lengthen the retry cadence, never shorten it.
Explicit failure classification. recordHealthLocked reused isTransient's
retry-shaped semantics wholesale, including its exclusion of
errRequestBuildFailed. A provider whose baseURL is malformed produced
that exact error on every discovery cycle and never opened its breaker,
because the old code routed it to a successful outcome. Health
classification is now explicit: any non-nil, non-canceled error counts
as a failure.
Neutral cancellation. A context.Canceled outcome no longer counts as
success or failure. Treating it as success previously closed an
already-open breaker and wiped its backoff whenever a half-open probe
was interrupted at shutdown, reporting a still-broken provider healthy.
Zero-value safety. A providerState assembled without going through
newModelRegistry now behaves like plain interval-only gating instead of
opening on the first failure with a zero backoff, matching the type's
own documented claim.
Backoff overflow ceiling. Repeated doubling with no ceiling on
maxOpenDuration could wrap the backoff negative after roughly 28
probes, silently disabling it. validateBreakerConfig now caps
maxOpenDuration at 24 hours.
Also renames modelRegistry.providerUsable and providerState.usable to
discoveryHealthy, and documents that the signal reflects the discovery
endpoint only. It must never be the sole reason to refuse routing to a
provider whose request path works fine. feat/failover should combine it
with a separate request-path signal fed by recordProviderAttempt.
Tests: a 24-hour simulated cadence test proves the fixed gate admits no
more attempts than plain interval throttling did before this feature.
A concurrent test exercises the breaker fields under -race for the
first time. Two tests replace a vacuous predecessor and distinguish
neutral cancellation from treating it as success. One test proves
errRequestBuildFailed opens the breaker. One test proves the zero value
never opens. A rewritten healthy-provider test asserts behavior instead
of reading the internal openUntil field directly.
Preserve usage when response translation fails after a successful,
billed upstream call. The route previously discarded already-billed
tokens on that path. A client could induce unlimited free usage
against any budget via a truncated tool-call response.
Reject a streaming request with 400 instead of 501. Check it before
the rate limiter runs. A 501 made anthropic-sdk-python retry the
request, since it retries any status of 500 or higher. Each retry
also counted against the caller's rate limit before this fix.
Preserve block order when translating an Anthropic message to
OpenAI's shape. A tool_result block before other content must stay
before it. OpenAI otherwise rejects the translated conversation.
Echo the client's requested model alias in both the passthrough and
the translated response paths. The passthrough branch previously
returned the upstream's own model id instead.
Forward the anthropic-beta header to an anthropic-type upstream
through an explicit allowlist. Never forward Authorization,
x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own
credential injection win regardless.
Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads
cache_creation_input_tokens and cache_read_input_tokens too. It folds
all three into the billed prompt count. This applies to this route,
the existing outbound Anthropic path, and the native passthrough
route.
Extract the metered-request pipeline shared by the unified route and
this route into runMeteredCall. Thread an envelope writer through it
so each route answers in its own error shape. This removes the
duplication the original brief warned against. It also lets the
rate-limit and body-admission error paths answer in this route's own
shape.
Treat a non-boolean "stream" value as a streaming request. A string
or number value previously bypassed the streaming check silently.
Emit max_completion_tokens instead of max_tokens for OpenAI's
reasoning models. Those models reject max_tokens with a 400.
Preserve tool_result.is_error by prefixing string content with
"Error: ". OpenAI's tool message carries no error field of its own.
Log a warning when the thinking field is dropped instead of
discarding it silently.
Classify a response-translation failure distinctly in the log. An
operator must never see a genuine translation bug reported as an
upstream connection error.
Document the route in README.md. Cover both auth headers, the
non-streaming limitation, and which providers translate versus pass
through. Add the reviewer's yaegi-check probes to
tools/yaegi-check/main.go, adjusted for the fixes above.
Add HealthState and OpenUntil to adminProviderView, GET
/admin/api/overview's per-provider row. HealthState mirrors the
discovery circuit breaker's state: closed, open, or half-open.
OpenUntil is the zero time.Time unless the breaker is open.
A provider with discovery disabled, or one that has never failed a
refresh, always reads closed. This keeps the dashboard's default view
unchanged for a healthy deployment.
Both fields come from providerSnapshot, read in the same locked pass
as the existing lastRefresh/lastErr fields, so a concurrent
finishRefresh can never hand the dashboard a breaker state that
disagrees with the rest of the row.
Add a three-state circuit breaker to providerState: closed, open, and
half-open. It is driven only by discovery-refresh outcomes, never by
request-path traffic.
A provider whose refresh fails breakerThreshold times in a row opens
the breaker and backs off before its next half-open probe, instead of
retrying every discovery interval forever. A failed probe reopens the
breaker with a doubled backoff, capped at breakerOpenMax. A successful
probe closes the breaker again, so a recovered provider never stays
dead.
The failure classification reuses limits.go's isTransient and
isDeadlineExceeded, the same rule recordProviderAttempt already
applies to request-path attempts. It does not add a second
classification.
Add Config.Breaker (BreakerConfig), validated and defaulted by
validateBreakerConfig. Every default preserves current behavior for a
provider that never fails: a closed breaker never changes the existing
interval-based refresh gating.
Export modelRegistry.providerUsable(name) as the queryable health
predicate. A sibling branch (feat/failover) will use it to skip
unusable providers before routing. This commit does not touch the
request path or routes_unified.go.
Add POST /v1/messages, serving Anthropic SDK clients directly instead
of outbound-only support. A request resolved to an anthropic-type
provider passes through untranslated. A request resolved to any other
provider type translates to and from OpenAI's chat-completion shape
through the resolved adapter's own chatCompletion, since that method's
contract is OpenAI-shape in and out regardless of provider type.
The route reuses admitRequest, readAndDecodeUnifiedBody,
registry.resolve, withAttemptRecorder, and limiter.account exactly as
routes_unified.go does, so groups, model resolution, limits,
accounting, and caching semantics stay identical to the unified route.
Client-facing errors use Anthropic's envelope shape. A streaming
request is rejected explicitly with a 501, never silently answered
non-streamed; streaming support is a follow-up branch.
Auth reuses authStore.identify and logAuthEvent unchanged.
presentedKey already reads both x-api-key and Authorization: Bearer,
so no new credential-checking code was needed.
Add openAIRequestFromAnthropic and anthropicResponseFromOpenAI to
translate_anthropic.go, the mirror of the existing outbound
translators, plus their supporting helpers and unit tests.
Release builds send one anonymous "plugin loaded" ping per process, so
the project can see roughly how many installs exist and which versions
are in use. The payload is project name, version and timestamp. Nothing
else leaves the pod: no identifiers, no hostname, no configuration, no
provider names, no keys, no request data.
Three things keep it from firing when it should not. Builds from source
carry the devPluginVersion sentinel and stay silent, so developer
checkouts and every test run never phone home. The ping is sent only
after the plugin constructs successfully, so a rejected configuration is
never counted as an install. A sync.Once holds it to one per process,
because Traefik calls New once per route and oss-telemetry does not
deduplicate on the client side.
Operators of a release build opt out with DO_NOT_TRACK=1,
OSS_TELEMETRY_DISABLED=1, or TRAEFIK_LLMGATEWAY_DISABLE_TELEMETRY=1.
README.md documents the full payload and the opt-out; a plugin that
phones home without saying so in its own README would be indefensible.
oss-telemetry is the plugin's first non-stdlib RUNTIME dependency, which
has consequences for the Yaegi gate:
- vendor/ is no longer excluded from yaegi-check's GOPATH copy. It was
excluded on the reasoning that it held only test-only testify and
_test.go files are never interpreted. That reasoning no longer holds:
leaving vendor out would make the interpreted copy fail to resolve an
import the real plugin tree resolves fine, so the gate would stop
representing production.
- yaegi-check now stamps a fake release version into its GOPATH copy so
New's telemetry gate is PASSED rather than short-circuited, and the
vendored package is actually called under the interpreter. Import
resolution alone is not coverage; a cross-package call into interpreted
third-party code is its own risk. Telemetry is suppressed by env vars
set before anything is interpreted, so the gate never emits a real
ping.
- A new assertion reads the version back through GET /admin/api/overview.
Without it a silently failed stamp would leave the dev sentinel in
place and the harness would still print OK while covering nothing.
Verified by disabling the stamp and watching the gate fail.
shouldSendTelemetry is split out from sendStartupTelemetry so the
decision is table-testable without a network, since oss-telemetry's
endpoint is deliberately not overridable from consuming code.
Also brings the version-gated documentation up to date, since the same
README edit touches it. The latest release is v1.0.3, so the pre-1.0
experimental warning has met its own stated condition and is replaced by
a semver stability statement in README.md and dropped from
.traefik.yml's catalog summary. The catalog now serves the released tags
(verified: the v1.0.3 download endpoint returns 200), so the "not yet
published" and "analyzer has not been run" claims are corrected, the
catalog form of the static config is documented as usable, and
examples/kubernetes.yaml pins v1.0.3 instead of v0.1.0.
A model id served by two configured providers is resolved
deterministically by the registry, the winner is named in the message,
and no request is affected. Logging it at ERROR put 144 ERROR lines in a
healthy pod's boot log on a cluster with overlapping openai,
openai-audio, copilot and macstudio catalogs, which is exactly the noise
that hides a real failure.
Adds Gateway.warnf beside logf and errorf, and a warn field on
modelRegistry carrying it. The field is set by the production caller
after newModelRegistry returns rather than added as a fourth constructor
parameter: 45 call sites pass a logger today, almost all of them tests
passing a no-op, and threading a parameter through all of them would
bury a one-line severity change in unrelated churn. A nil warn falls
back to log, so a registry built without one keeps its current behaviour
instead of panicking or dropping the line.
registry.warnf takes an already-formatted string so both call shapes
stay single-variadic-argument. Yaegi v0.16.1's CFG builder panics on a
call to a struct FIELD of variadic func type with two or more variadic
arguments, which is the restriction the log field beside it already
documents.
Verified under the real interpreter, not only compiled: yaegi-check
drives a genuine two-provider collision and now prints WARN. The
severity test fails if the call site reverts to log, so the level cannot
drift back unnoticed.
Only this message moves. Genuine failures — discovery errors, metadata
capture panics, encoding failures — stay at ERROR.
Federation's tools/call oversize branch matched a string marker with
strings.Contains instead of wrapping a sentinel. The stated reason was
that no gate exercised this path under Yaegi, so errors.Is was unproven
here.
The caution was misplaced. errors.New returns a compiled
*errors.errorString and fmt.Errorf a compiled *fmt.wrapError, so
declaring the sentinel variable in interpreted code leaves the entire
chain errors.Is walks compiled. Neither documented trap applies: the
matchesSentinel trip-wire covers a plugin-declared error TYPE, and the
comma-ok trap needs a plugin-declared interface. retry.go has matched
errRequestBuildFailed this way through a multi-%w wrap since v0.2.0.
The string check was also strictly worse: err.Error() is itself an
interpreted-to-compiled dispatch, so it verified nothing extra, and
matching on text that partly comes from the backend admits false
positives a sentinel cannot have.
Rather than settle this by argument, tools/yaegi-check now drives POST
/mcp tools/call against a backend whose response exceeds the cap, and
asserts the interpreted plugin reports "response too large". Breaking
the match makes the gate fail with "upstream error", so the probe has
teeth. This closes the objection permanently instead of answering it in
a comment.
Also in doBackendJSONRPC, check the HTTP status before the size cap. A
backend failing with a large HTML error page now reports "upstream
returned HTTP 500", which is actionable, rather than sending the
operator after a payload-size problem that does not exist.
Two comment corrections carried forward from review: the fan-out
semaphore is not the same shape as limiter.spawnTokens (spawnTokens
drops work via a default case, this one blocks and must never skip a
server), and the shared fan-out deadline gives later batches the
remaining budget, not a fresh one, so uniformly slow backends yield a
partial list.
Addresses critical finding 3 and important findings 3-4 from security
review round 2, 2026-08-22.
Critical finding 3 — the tools/list-per-backend weight charged against
the caller's own requests-per-minute/day budget (security audit finding
1b) was not default-preserving: both live groups have an empty MCP
allow-list (all 11 servers), so it silently cut every configured
requests-per-minute budget by ~11x with no config change, and
examples/kubernetes.yaml's shipped requests-per-minute: 10 would 429 the
very first tools/list call ever made. Reverted: checkAndCountWeighted
removed from limits.go, handleMCPFederated back to a plain checkAndCount.
The real amplification stays fully visible to an operator via
countTargetRequests' existing per-server 'mcp' target-scope counters —
a separate scope from the user/group budget — rather than silently
consuming tenant quota. The fan-out semaphore (below) is the actual
protection against the amplification finding 1 described.
Important finding 3 — the fan-out semaphore (4 concurrent backends)
tripled worst-case latency (907ms measured at 300ms/server, was ~300ms)
and pushed the hung-backend worst case to ceil(11/4)x20s=60s, past the
~30s timeout many MCP clients use. Raised to 8, and every backend in one
tools/list call now shares ONE overall toolsListBackendTimeout deadline
(fanoutCtx) instead of each getting a fresh one from its own staggered
start — restoring this function's original 'bounds the whole call
regardless of server count' invariant even with bounded concurrency.
Important finding 4 — the 4MiB response cap (sized for the tools/list
fan-out's memory-amplification risk) silently applied to tools/call too,
where exactly one backend is ever contacted and a legitimate result (an
image, an extracted document) can be much larger. tools/call now uses
mcpBackendCallResponseMaxBytes (10MiB, the pre-finding-1c budget);
tools/list keeps the 4MiB cap. A response that hits either cap is
detected via a cap+1 read (never silently truncated into a confusing
parse failure) and reported to the client as 'response too large'.
Fixes critical finding 1 from security review round 2, 2026-08-22:
identify checked the per-source-IP throttle BEFORE verifying the
presented key. This plugin runs behind a Kubernetes Service, and with
externalTrafficPolicy: Cluster (this operator's own deployment),
r.RemoteAddr is the node's own SNAT address shared by every external
caller reaching that node — not a per-tenant address. 20 failed requests
from anywhere on the internet sharing that node locked out all 10 live
users, including the admin dashboard's own poll.
- identify now verifies the key FIRST, unconditionally: a valid key
always succeeds before any throttle state is read. Every failure path
still records against the caller's IP, but throttling is now pure
observability (bounding log volume and tracker memory), never
enforcement — an attacker with no valid key already got 'false'
either way, so nothing is lost against brute force.
- Replaces the per-source-IP tracker's backing store: the general-
purpose memoryStore (limits.go) has no size cap and a full mutex-held
sweep that scales with tracked-IP count (~83ms at 1M IPs). authFailureTracker
is a bounded, two-generation, O(1)-rotation structure capped at
2x50,000 entries (important finding 1).
- logAuthEvent now emits a distinct, separately rate-limited
'auth throttle:' line the first time a source crosses the limit
(important finding 5), and both the routine and throttle-engaged
lines report how many events they suppressed since the last line
actually written, so several distinct attacker IPs in one window no
longer collapse into a single indistinguishable line (important
finding 6).
- The success path no longer touches the throttle tracker's mutex at
all (important finding 2).
Fixes finding 5 from the 2026-08-22 security audit: newAuthStore and
replaceFileUsers rejected duplicate API keys but not duplicate names, and
counters are keyed on u.name (routes_unified.go), so two users sharing a
name silently shared llmgw:user:{name}:* counters and both appeared in
buildAdminUsage; an empty name collapsed every such user into
llmgw:user::*.
- buildEntry now rejects an empty user name outright — the source of
both bugs, since duplicate names and an empty name are the same root
cause (two config entries resolving to the same counter key).
- newAuthStore and replaceFileUsers each reject a duplicate name WITHIN
their own source (two inline users, or two file users) — but not
across sources: a file user overriding an inline user of the same name
stays intentional and unaffected, per replaceFileUsers' own documented
merge semantics.
- Deliberately does NOT extend configNamePattern's character
restriction to user names: that pattern exists because provider/
mcpServers/agents names are embedded as URL path segments, which a
user name never is here. Restricting the charset further risked
rejecting real operator-chosen names this round has no visibility
into; the live users file's steve-cw/kevinsandom already pass the
non-empty check unaffected.
Also adds TestDefaultPreserving_LiveClusterShape_LoadsAndServesIdentically
(admin_test.go), modeling the live cluster's 5 friends + 4 home users and
11 MCP servers to prove all five fixes in this round load and serve
identically at that scale.
Fixes finding 4 from the 2026-08-22 security audit: buildAdminUsage sent
every user's, group's, and the total scope's counters (8 keys each) to
limiter.currentUsage in ONE storeGetMulti round trip. With 1,000 users
that is ~8,000 GETs pipelined through the single mutex-guarded store
connection, during which every concurrent checkAndCount from live LLM
traffic queues behind it (up to respCallTimeout) — and the admin
dashboard polls this endpoint every 5s.
Add chunkedCurrentUsage: splits the scopes slice into batches of at most
adminUsageChunkScopes (200) before calling currentUsage, letting live
traffic interleave between chunks while each chunk still batches its own
scopes in one round trip. At or under 200 scopes (the live cluster's 9
users) this makes exactly one call, identical to today's behavior — the
payload shape is unchanged, so no webui coordination is needed.
Fixes finding 3 from the 2026-08-22 security audit: authStore.identify
had no counter, no backoff, and no lockout on failed attempts, and
logger.go wrote one unbuffered stderr line per attempt including the
caller's RemoteAddr. An unauthenticated attacker got unlimited digest
guesses at line rate and could drive synchronous stderr writes on the
shared Traefik ingress.
- Add per-source-IP failure throttling to identify, reusing the existing
in-process memoryStore (limits.go) as authStore's own dedicated
instance rather than a new dependency: authFailureLimit (20) failed
attempts per authFailureWindow (1 minute) throttles further attempts
from that IP outright, including a subsequently valid key, and resets
once the window elapses.
- clientIP reads r.RemoteAddr only, never a client-supplied header,
matching the same trust boundary the rest of this package already
assumes (routes_passthrough.go strips X-Forwarded-*/X-Auth-* before any
outbound request).
- Rate-limit logAuthEvent's failure line via authStore.shouldLogAuthFailure,
the identical technique limiter.logStoreError already applies to store
errors. Successful-auth logging stays unbounded (bounded by legitimate,
already-authenticated traffic elsewhere).
Fixes finding 2 from the 2026-08-22 security audit: the per-server
goroutines in mcpFederatedToolsList carried only defer wg.Done(), unlike
every other off-request goroutine in this package (limiter.spawn,
modelRegistry.refreshProvider/captureModelMetadata), which all guard
against a panic crashing the whole shared Traefik process. This codebase
has repeatedly hit interpreter-level panics (registry.go reflect.Set,
cache.go CFG index-out-of-range, providers.go errors.As), so an
unrecovered panic here is not hypothetical.
Add the same defer-recover pattern used at the other three sites: a
panicking backend now degrades to "that server failed" and is logged,
exactly like an HTTP error from it, instead of taking down the process.
lukaszraczylo
