A go test run over ./... stops at nested modules, so CI never ran
tools/pricing-sync or the Yaegi interpreter check. A new job runs both
with Go 1.25.x. The go.sum of tools/yaegi-check now holds its test
dependencies, and its go.mod lists the prometheus modules as direct
dependencies. The .gitattributes comment now says that the
export-ignore rules for tools/, webui/, examples/ and testdata/ do take
effect.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
cmd/gateway runs the middleware as its own HTTP server behind Traefik.
It reads a YAML config, converts it to JSON and decodes it with
DisallowUnknownFields, so a misspelled key stops startup. It warns
when metrics.allowedCIDRs is set, because behind a proxy the source
address is the proxy's.
The Makefile gains build-binary, docker-build and run-local targets.
The Dockerfile builds from the committed vendor tree and stamps
main.version from the VERSION build argument. go.mod lists
go.yaml.in/yaml/v3 as a direct dependency.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Federated tools/call re-initializes a session only when the backend
signals a missing session: HTTP 400 or 404, or a JSON-RPC error that
names a session. A failing tool is no longer called twice.
tools/list now:
- passes each tool object through and rewrites only its name
- follows nextCursor up to a page cap and keeps the earlier pages if a
later page fails
- skips tools with no name
- resolves name collisions to the server that tools/call routes to
The handshake sends notifications/initialized. tools/call forwards
_meta. Multi-line SSE data is joined. A2A target URLs are joined with
url.JoinPath and keep their query string. agents[].card must start with
"/". Logged target URLs and errors are scrubbed of credentials.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
The gateway refuses a request for an unpriced model when the caller has a
cost budget. An unpriced model cost 0, so the budget counter was never
written and the budget could never fire. The new
allowUnpricedWithCostBudget setting restores the old behaviour.
The retry loop honours an upstream Retry-After of up to 30s and jitters
its own computed backoff. SSE parsing bounds the total size of one event.
The Redis client can use TLS. Provider base URLs and errors are scrubbed
of credentials before they reach logs or the admin view. The gateway
warns when failover and retry settings multiply into a large number of
outbound calls.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
encodeCommands wrote one fmt.Appendf call per header and per argument. A
counter pipeline is almost entirely INCRBY and EXPIRE commands, which
carry three arguments each, so a metered request paid four Appendf calls
for every counter it wrote.
A command with exactly three arguments now goes through one Appendf with
the whole template. Every other arity keeps the per-argument loop. The
bytes on the wire do not change.
A new test compares arities two, three, four and five against the
previous encoder, so a wrong arity guard cannot emit a three-argument
header for a longer command.
Measured under yaegi as above: about 3% fewer bytes allocated per
request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
incrMulti and incrAndGetMulti took the expire mutex twice per counter:
once to ask needsExpire whether the key still wants an EXPIRE, and once
after the reply to commit or forget it. A metered request writes 28
counters, so each request took that mutex about 56 times.
needsExpireBatch now answers for the whole batch under one lock, and
settleExpireBatch applies the commit and forget decisions under one more.
The decisions are unchanged: needsExpire is read for every entry before
any commit, a confirmed EXPIRE commits the key, and an INCRBY reply equal
to the delta forgets it. The FIFO eviction bound is unchanged.
applyIncrReplies decodes the INCRBY replies and reports how many entries
it examined, so the caller settles exactly that prefix on every path. A
reply error used to leave the already-decided entries unsettled, which
kept a key marked as having an EXPIRE it never received. That key could
then never regain a TTL, because its INCRBY reply can never equal the
delta again. A provider failover answering READONLY mid-batch reaches
this path. Two tests reproduce it and fail without the settle.
Measured under yaegi as above: this step removed a further 4% of the
bytes allocated per request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
Each counter entry called newCounterIncr, which called windowKey, which
called fmt.Sprintf and formatted the window bucket from the clock. A
metered request writes 28 counters, so every request formatted 28 bucket
strings and built 28 keys through Sprintf.
bucketsFor now formats the four window buckets once per limiter call, and
enforceTTLsFor reads the four enforcement floors from new constants
instead of calling enforceTTLFor per entry. The call sites build
counterIncr literals directly from those values, and windowKeyForBucket
joins the key parts. Keys on the wire do not change, and windowKey keeps
its old signature for every other caller.
The inline literals pair the window, the bucket, the TTL and the
enforcement floor by hand, which newCounterIncr used to derive. A new
test walks every entry the four metered methods produce and asserts the
enforcement floor matches the window in the key, and that the key ends
with that window's bucket. The test fails if any pair drifts.
Measured under yaegi with a harness that runs POST /v1/embeddings against
Redis with the cache and metrics on: this step alone removed about 6% of
the bytes allocated per request.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.
A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.
Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.
With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
Under yaegi v0.16.1, the Redis write path allocated heavily.
encodeCommand wrote each argument with fmt.Fprintf into a
strings.Builder. attemptPipelineOn copied every command into a second
builder, and io.WriteString converted the result to bytes again. setEx
also copied the cached response with string(val).
encodeCommands now appends one fmt.Appendf call per header and argument
to one slice, and the connection writes that slice directly. pipeline
encodes once, so a retry sends the same bytes. setEx frames SET directly
from the value bytes. The bytes on the wire do not change.
Two other encoders were measured and rejected. Under the interpreter, a
presized buffer and direct byte appends cost more than native slice
growth inside fmt.Appendf.
Tests compare encodeCommands and the setEx wire bytes with the previous
encoder. The cases include binary values, empty arguments and a
100000-byte value.
A yaegi harness runs POST /v1/embeddings with Redis, cache and metrics
on. This change lowers allocation from 925587 to 783189 bytes per
request (-15%).
Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
A user entry can now list several groups in "groups", next to the
existing "group" field. A user can also hold a personal grant with its
own "providers" and optional "models" lists. Users with one group and
no personal grant behave exactly as before.
A request is allowed only when one grant allows the provider, the model
and, for passthrough, the path. Grants never merge, so one grant cannot
supply the provider while another supplies the model or the path. The
personal grant uses the path list of the first group. A personal
"models" list without "providers" is a configuration error.
Each member group applies its own limits. Caching stays on unless a
member group turns it off, and the shortest group TTL wins. The
/v1/models cache uses a stable, length-prefixed key, so reloads do not
grow it.
Tests cover the grant union, cross-grant denials, passthrough path and
model coupling, reloads and the cache key. The yaegi-check harness
asserts the same cases under the interpreter.
Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3
OpenAI's /v1/models schema carries no context field, so every client that
wants one reads a vendor dialect, and a client whose dialect is absent does
not ask — it substitutes its own default. GET /v1/models now emits the same
resolved integer under "context_window", "context_length" (the OpenRouter
spelling) and "max_model_len" (the vLLM spelling). All three are omitted
together when the window is unknown, exactly as the single field was.
Measured case: omp 18.1.15's openai-models-list discovery resolves
`max_model_len ?? context_length ?? <bundled catalog> ?? 128000` and never
reads context_window, so a 1048576-context local model reached the client as
128000, which also clamped its maxTokens.
The metadata fetcher already parses all of these dialects on the inbound
side (provider_openai.go), so this makes the outbound direction speak what
the inbound one already understands.
Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp
resolve gives an exact alias match precedence over every discovered bare
model id, but listFor did the opposite: it skipped the alias whenever a
discovered model of the same bare id had already been listed. The catalog
then advertised the shadowed provider as the owner, with that provider's
metadata, for an id every request routed to the alias target instead.
A live fleet shows the effect. The alias "deepseek-v4-flash-vision-exp"
targets "gx10/current" (context 1048576), while a cloud provider discovers
a model of the identical bare id. GET /v1/models returned owned_by
"deepseek" and no context_window at all, so clients fell back to their own
128k default while the request ran against a 1M-context local model.
listFor now resolves every usable alias up front and suppresses the bare
entry of any id an alias shadows. The shadowed model stays listed in
"provider/id" form, which is how a client must address it once the alias
owns the bare id, so no model disappears from the catalog.
Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp
The metadata fetcher understood only LM Studio's field names, so a
vLLM-backed provider reported no per-model context and every model had to
have its window pinned by hand in modelMeta. That pin then goes stale
silently: a renamed served model ships no context at all and clients fall
back to their own default, and a changed window advertises a size the
engine no longer accepts. Both happened on a live deployment today.
vLLM reports max_model_len at its OpenAI-compatible /v1/models and neither
LM Studio field, verified against vLLM 0.28. Reading it means pointing
MetadataPath at /v1/models is enough for such a provider.
Precedence is unchanged where it already applied: loaded_context_length
still wins as the window usable right now, then max_context_length, and
max_model_len is consulted only when neither is present. An upstream
reports one dialect or the other, so the added branch cannot change any
existing capture.
Renames the payload type, which now carries two dialects rather than LM
Studio's alone.
Add per-replica health tracking for every configured MCP server and A2A
agent. Real proxied and federated traffic always feeds the signal. An
opt-in lazy probe sweep, off by default, fills the gaps for targets that
see no traffic. Expose the result on GET /admin/api/targets, as the
llmgateway_target_healthy gauge, and as a badge in the dashboard.
Federation routing is unchanged: an unhealthy server is still attempted.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The zero-time sentinel was module private, so the spec redeclared the
literal once a third consumer started depending on it. Export it and
import it in the spec. Regenerate the baked admin assets.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The warmup waited for one target while the same sweep probed the others
concurrently. A late probe could overwrite another target's traffic
sourced record and fail a later assertion.
Wait for every configured MCP target to report the probe source, and
correct the comment that claimed a guarantee the code did not give.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The note claimed fieldalignment reports no change from pinning the sweep
timestamp first. The analyser is in fact clean before the move and
reports 192 against an optimal 184 leading pointer bytes after it.
State the measured truth. The struct stays 264 bytes either way, so the
exception costs no padding. Moving the sweep flag up to close the
pointer-bytes gap was tried and made the report worse. Atomic alignment
on 32-bit platforms outranks the linter hint.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The previous round suppressed every context-caused federation error. A
server that accepted the connection and then never answered therefore
read "unknown" forever whenever federation carried the only traffic,
which is the outage shape this feature exists to surface.
Split the two causes instead. A server still waiting for a fan-out slot
when the budget expires is never dialed, so the semaphore acquire now
selects on the fan-out context and records nothing for it. A server this
gateway did dial records a failure on a deadline, because it had its
chance to answer. Only a caller hanging up stays unattributable.
Document the split rule, correct a stale comment about a second record
site, and state that "unknown" reads two ways rather than three.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Go's net/http masks a password in the dialed URL before it builds the
*url.Error, so an error from a target configured with a password in its
userinfo never contains the raw URL. The verbatim substring guard inside
sanitizeProviderErr therefore matched nothing and scrubbed nothing. Both
the credential and any api-key query parameter on the same URL leaked
through the admin API and the dashboard tooltip.
Add sanitizeTargetErr. It tries the plain match first and then, only when
the configured userinfo carries a password, rebuilds the exact masked
form net/http produces and scrubs against that too.
Widen the scrub-order test so the secret straddles the truncation point,
which makes it fail if the scrub ever runs after the truncation. Drive
every new credential case from a real http.Client error against a closed
listener rather than a hand-written string, and assemble those fixture
URLs from parts so no whole credential-bearing URL literal sits in the
source. Drop the tracker doc paragraph describing a nil-error failure
that no caller produces, and remove the wall-clock dependency from the
sweep interval test.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a Health column to the MCP servers and A2A agents tables in
TargetsView. An unhealthy target renders a destructive badge with the
age of its last check and carries the failure count, source, and
scrubbed last error in the title. An unknown target renders an outline
badge and, when it has failed but never succeeded, the same failure
details in its title. A healthy target renders muted text with its last
observed latency and no badge, matching the providers' convention.
Add AdminTargetHealthView to the API types with the wire contract the
Go side emits, a bare formatElapsedAgo helper next to formatAgo, and
column-cell tests for all three states. Regenerate the baked admin
assets.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
targetHealthLastSweepUnixNano moves to Gateway's first field. It is
read and written only through sync/atomic's 64-bit functions, and
the sync/atomic package guarantees 64-bit alignment on ARM, x86-32,
and 32-bit MIPS only for a struct's first word — Traefik ships armv7
and 386 builds alongside amd64/arm64. Every other field keeps
golangci-lint fieldalignment's pointer-grouped ordering; verified
with the standalone fieldalignment analyzer that this is the only
field out of place and adds no struct-size padding, only a documented
+8 GC-scan-prefix cost.
docs: catches up README's admin `health` object and Target health
"State" bullet to F4 (previous commit, a0c8607) — three ways to
read `state: "unknown"`, and that consecutiveFailures/lastCheck/
lastError/latencyMs/source stay present when a target has been
observed but never succeeded.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
writeTargetMetrics now returns before emitting even
llmgateway_target_healthy's own HELP/TYPE header when neither an
MCP server nor an agent is configured, mirroring
writeStoreHealthMetrics' identical behaviour when Redis is
unconfigured.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
exerciseTargetHealth's traffic-sourced assertion (mcpProbeServerName
must read healthy/traffic, F5 review) raced its own trigger: GET
/admin/api/targets was the first-ever sweep-capable route this
harness hit, so the background sweep it started could beat the
response encode and overwrite source from traffic to probe under
Yaegi's interpreter overhead. exerciseTargetHealthProbeWarmup now
runs first, before exerciseHandler, against a dedicated, never-
proxied MCP target — its health can only ever be written by the
probe, so polling it to healthy/probe both proves the interpreted
active-probe path end to end and spends this gateway's one
probeInterval-bounded sweep before any traffic runs, making every
later target-health assertion in this harness deterministic.
probeInterval moves from 10s to 60s so no later exercise can trigger
a second sweep mid-run. TARGETHEALTH-1's expected state also updates
to unknown, reflecting F4 (a2d152b1): mcpProbeServerName's only
observation is a failed tools/call, so it has never once succeeded.
mcp_a2a_test.go's ClientCanceled_RecordsNothing test replaces its
30ms-cancel-vs-200ms-sleep timing margin with a channel handshake
(cancel only once the handler is genuinely reached, then release
it), removing any wall-clock dependence.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
mcpFederatedToolsList's fan-out and mcpFederatedToolsCall both
derive their outbound context from r.Context(). A client cancel
(context.Canceled) or the shared fan-out deadline expiring
(context.DeadlineExceeded) is not evidence about a server's own
health, so neither is recorded any more, mirroring
recordTargetProxyHealth's own client-cancel rule. A server that is
genuinely hung is still caught by the active probe sweep or its own
per-server proxy health.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Combines three closely-coupled target_health.go fixes that all touch
targetHealthTracker.record/stateForLocked in the same commit, since
splitting them would leave an intermediate commit that fails to
compile or a test asserting behaviour its own commit does not yet
implement.
F1: record gains a rawURL parameter and scrubs err's text through
sanitizeProviderErr before storing lastError, so a target URL
configured with an embedded credential can never leak through GET
/admin/api/targets. The scrub runs before truncateRunes so a long
embedded URL cannot survive truncation unscrubbed. Every probe, the
per-server proxy hook (mcp_a2a.go), and both federation record
sites (mcp_federation.go) now pass their target's configured URL
through.
F3: maybeSweepTargetHealth re-checks the elapsed interval after
winning the single-flight CAS, closing a window where a caller's
own stale interval read could still win the CAS once a separate,
faster sweep had already started and finished within the same
interval.
F4: targetHealthEntry gains everOK, set on record's success branch.
stateForLocked now reads a target that has never once succeeded as
unknown rather than healthy, even below failureThreshold — the
threshold still protects against a single transient first failure,
it just does not promote a target this tracker has never seen
answer to healthy. targetHealthSnapshot gains observed so the admin
view can still tell apart never-observed (omit every other field)
from observed-but-never-succeeded (keep them) despite both reading
state unknown. The now-redundant lastOK field is removed.
Also included: target_health_test.go's own test-hygiene fixes
(F6) — an idempotent, t.Cleanup-registered unblock ahead of
srv.Close so a failing assertion cannot leave a released server
handler blocked until the test binary's own timeout, and an atomic
hit counter in the disabled-sweep test — since they land in the
same file as the tests above.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add the targetHealth config row and a TargetHealthConfig reference
section, a Target health section under MCP and A2A describing the
passive recording rules and the active probe rules per target kind,
the health field on GET /admin/api/targets, and the
llmgateway_target_healthy gauge with its aggregation-across-replicas
note.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a second MCP server (target-health-down) pointed at a closed
listener, and enable targetHealth in the harness config. After the
existing federated tools/call probe, exerciseTargetHealth reads GET
/admin/api/targets and asserts the probe server reads healthy with
source traffic, then drives targetHealthFailureThreshold consecutive
failures through the direct per-server proxy path against the dead
server and asserts it flips to unhealthy on both GET
/admin/api/targets and llmgateway_target_healthy on /metrics.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a health object to each row of GET /admin/api/targets
(adminTargetHealthView, admin.go): state, an RFC3339 lastCheck,
lastError, source, and latencyMs, every optional field omitted while
the target has never been observed. Both serveAdminTargets and the
MCP/agent listing routes (handleMCPServers, handleAgents) trigger the
active probe sweep before responding, so admin polling and client
listings double as the heartbeat that drives it.
Add llmgateway_target_healthy to /metrics (metrics.go): a 1/0 gauge
labeled by kind (mcp or agent) and target name, with no sample at all
for a target the tracker has never observed. serveMetrics triggers
the sweep too, so a Prometheus scrape becomes a heartbeat as well.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a per-replica, in-memory health tracker for MCP servers and A2A
agents (target_health.go). The tracker records observations from real
proxied traffic (handleTargetProxy, mcp_a2a.go) and federated calls
(mcpFederatedToolsList/mcpFederatedToolsCall, mcp_federation.go), and
derives a three-state readout (unknown, healthy, unhealthy) from a
configurable consecutive-failure threshold. Add TargetHealthConfig to
Config, validated at construction. The tracker never gates routing:
federation and the target proxy keep contacting every configured
server regardless of its recorded health.
Extend proxyResult (routes_passthrough.go) with status and
clientCanceled so handleTargetProxy can classify a proxied response
without proxyUpstream knowing anything about target health.
Add maybeSweepTargetHealth, an opt-in, lazy, single-flight active
probe sweep with no timer and no long-lived goroutine, matching this
plugin's lack of a Close hook under Traefik hot reload. A probe uses
the initialize/close handshake for a Streamable HTTP MCP server, a
plain unread GET for a legacy /sse server, and a GET of the agent card
path for an A2A agent. handleMCPServers and handleAgents now trigger a
sweep at the top of each call.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The Charts view could break usage down by user, group or total, but not
by model. Per-model data existed only as day-window attempt and failure
counts, which measure availability rather than delivered usage.
A served model now accumulates the same four metrics a user or group
does. The scope kind is "model" and its id is the canonical
"provider/model", so a request that fails over counts against the
provider that answered it, and one bare id served by two providers ranks
as two rows.
The request counter is written after the response, in the batch that
already writes tokens and cost. That is the first point at which the
serving model is known, so no extra round trip is needed and admission
never double counts.
GET /admin/api/usage/models ranks models for one metric and window. It
returns only models with non-zero usage, by operator requirement: the
catalog runs to hundreds of models and a padded ranking is unreadable.
The read stays one storeGetMulti over the catalog.
The Charts view gains a Models tab with its own metric picker, and the
scope picker gains the models that have traffic.
bareWinner picked the first provider in global sorted order that served
a bare model id, ignoring which providers the caller's group could
actually use. resolveAgainst then denied the request whenever that
provider fell outside the group, even when another provider serving the
identical id was already permitted. A caller entitled to a model could
receive errModelDenied purely because an unrelated, invisible provider
sorted first alphabetically.
bareWinner now takes the group and prefers the sorted-first provider it
may use, falling back to the global sorted-first provider only when the
group is authorized for none of them. That fallback keeps a genuinely
unauthorized request returning errModelDenied, not the wrong
errModelUnknown a "no visible owner" result would produce. resolve,
resolveAliasTarget, and the alias-shadow collision check all route
through this corrected selection.
listFor now computes each bare id's owner from the group's own visible
providers and adds "provider/model" disambiguating entries only when the
group can see two or more owners of that id. A provider invisible to the
group never appears in that group's catalog, bare or prefixed.
warnCollisionOnce keeps logging the global collision unchanged, since a
provider collision is a fact about the fleet's configuration that every
operator must see regardless of which group lists models first. Its
message now says winner is the global default, because a group's own
request can resolve to a different provider than the one logged.
Adds registry_test.go coverage for the production repro, the
both-allowed and neither-allowed non-regression cases, the listing
behavior, and the alias path. Adds a yaegi-check probe proving the fix
resolves and serves under the real interpreter, and updates one existing
listFor test that asserted the pre-fix prefixed-form behavior.
The README claimed discovery runs again on discoveryInterval "in the
background", which reads as a periodic timer. There is none. maybeRefresh
is called from ServeHTTP only, so the interval is checked per incoming
request and a refresh is spawned when it has elapsed.
That distinction is what an operator actually needs, because it has three
consequences the old wording hid: an idle gateway never refreshes at all,
a newly published upstream model takes up to discoveryInterval to appear,
and there is no manual trigger since the admin API is read-only.
The plugin runs inside the Traefik process, so declining to start a
periodic goroutine is deliberate rather than an oversight. Documented as
behaviour, with the option of lowering discoveryInterval on the providers
whose catalogues actually change. Also corrected the two provider config
table rows that carried the same implication. Documentation only.
The federated /mcp endpoint ignored the Accept header and always
returned application/json. A real MCP client that sends Accept:
application/json, text/event-stream expects SSE framing. It fails with
no data line in SSE response when it receives plain JSON instead.
handleMCPFederated now negotiates the response format once per request
from the Accept header. It threads the result through
writeJSONRPCEnvelope, writeJSONRPCResult, and writeJSONRPCErrorResponse
as an explicit parameter. A request with no Accept header, or one that
omits text/event-stream, still gets byte-identical plain JSON. Only an
explicit text/event-stream token in the Accept list switches the
response to SSE framing. SSE framing is one data line carrying the
JSON-RPC envelope, terminated by the required blank line.
Error responses and every method path honor the negotiated format.
Notification responses stay 202 with no body regardless of Accept.
The SSE branch reuses sse.go newSSEWriter and writeData instead of a
second encoder. writeData already writes the single-frame, correctly
terminated shape this route needs.
Adds a compiled test file with a captured pre-change JSON byte
baseline. It also adds explicit SSE negotiation and framing tests. A
new yaegi-check probe exercises both framings under the real
interpreter.
Add a provenance-est provider whose upstream answers with no "usage"
field at all, driving a real, non-streaming chat completion against it
under Yaegi. exerciseUsageProvenance then parses GET /metrics with the
real Prometheus TextParser and confirms both new families
(llmgateway_usage_provenance_requests_total and
-_tokens_total) render as COUNTER type with a distinct "estimated"
series for this provider and a distinct "reported" series for openai's
own already-driven traffic, then confirms GET /admin/api/overview
surfaces the identical estimated observation and never leaks a
"reported" key into the compact admin summary.
Run at the end of run(), after every other probe that already reads
/metrics or /admin/api/overview, so it never perturbs a counter an
earlier probe asserted on.
Rebuilds webui/dist/ via vite build and re-runs webui/generate.mjs to
bake the usage-provenance badge changes into admin_assets_gen.go, the
file admin.go's serveAdminPage/serveAdminAsset actually serve.
Add AdminProvenanceView and AdminProviderView.provenance to types/api.ts,
mirroring admin.go's new fields, then surface two badges beside the
existing health and latency badges in ProvidersView.vue: "estimated N"
(secondary variant) and "unbilled N" (destructive variant), each with a
hover title explaining what happened and why it matters. Neither badge
renders when that provenance kind has no observations yet, matching the
existing health/latency badges' own convention.
Estimated and unbilled stay two separate badges, never merged into one
indicator: collapsing them would hide the one thing this feature exists
to surface, that a provider serving completions entirely free against
every budget is a materially different failure than an approximated
bill.
Built with shadcn-vue's existing Badge component; no hand-rolled markup.
Add adminProviderView.Provenance to GET /admin/api/overview: a compact,
per-provider map keyed by provenance kind, populated only for
"estimated" and "unbilled" observations. "reported" never appears here,
since it is the healthy default and AttemptsDay/FailuresDay already
describe it; the full reported/estimated/unbilled breakdown, by request
and by token, stays on GET /metrics.
buildAdminProvenanceViews mirrors buildAdminLatencyViews' own shape:
one batched g.provenance.snapshot() read per response, grouped by
provider, so this adds no additional store round trip and keeps the
already-large overview response small, matching the existing latency
summary's own size budget.
A test proves the reported/estimated/unbilled filtering, verified
against a real mutation of the exclusion guard before being kept.
Token accounting has three provenances today: reported (real provider
usage), estimated (a non-streaming response with no usage, prompt
substituted from request body size), and unbilled (a streaming response
with no usage, request counted but zero tokens billed). Nothing on any
read surface distinguished them before this change.
Add a per-process provenanceStore (metrics.go), recorded in
runUnified right after the existing estimated/unbilled classification
(routes_unified.go), for a candidate whose response actually completed.
A failed-over candidate is never recorded, so ordinary failover noise
never pollutes the unbilled signal.
Expose two new counter families on GET /metrics:
llmgateway_usage_provenance_requests_total and
llmgateway_usage_provenance_tokens_total, both labeled by provider and
provenance. Together they answer the three operator questions this
change targets: what fraction of billed tokens came from real reported
usage, which providers produce estimated or unbilled accounting, and
how often the unbilled case happens at all. Both families are
in-process only, so sum() is the correct cross-replica aggregation,
matching this file's other per-process families.
Billing itself is unchanged: this only records which provenance an
already-billed number took, never a new number.
Tests cover reported/estimated/unbilled classification, that billing
matches the pre-existing contract exactly, and that estimated and
unbilled never collapse into one series. Each behavioral assertion was
verified against a real mutation of the source before being kept.
Add streaming and non-streaming latency badges beside the discovery
health badge on each provider row. Streaming shows average time to
first byte, labeled ttfb, since it is the only load-sensitive figure
this data provides. Non-streaming shows average total duration,
labeled avg, never latency or ttfb, because a non-streaming provider
buffers the whole completion before sending anything and its ttfb
carries the same output-length contamination as its duration.
Badges omit entirely when a provider has no observations yet, so
absence never reads as a fabricated zero. Every badge title states the
per-replica, in-process scope explicitly: this data comes from the
same in-process accumulator the rate-limit counters use, never Redis,
so a single poll reflects only whichever replica answered it.
Mirror admin.go's adminLatencyView type in types/api.ts, keyed by the
literal streaming/non-streaming union admin.go emits. Add
formatLatencyMs to lib/format.ts with spec tests alongside its
siblings in format.spec.ts.
Add exerciseLatencyMetrics, proving the new upstream-latency histograms
render correctly under the real Yaegi interpreter, not merely that a
compiled go test accepts the same bytes. Compiled tests have repeatedly
passed on this codebase while the interpreted shape failed, which is
exactly what this probe caught: two bugs in the core feature commit,
found and fixed only because this harness exists.
Parse the rendered body with prometheus/common/expfmt's real
TextParser, the same library a real Prometheus server's scrape loop
uses, instead of this project's own hand-rolled parser. A malformed
histogram fails to parse outright under a real parser, matching the
failure mode a live scrape target actually hits, rather than merely
failing one hand-written assertion.
No new traffic is driven here: the existing exerciseAttemptAccounting
sub-probe already posts one real, non-streaming chat completion against
the openai provider, so the duration histogram already carries a real
observation by the time this probe runs.
Add github.com/prometheus/common/expfmt as a dependency of this
harness module only, isolated from the plugin's own stdlib-only go.mod
by design (this package's own doc comment). Go raised this module's go
directive from 1.22 to 1.25.0 to satisfy the new dependency; the
plugin module itself is untouched.
Add time-to-first-byte and total-duration histograms for every upstream
call, observation only. No throttling, priorities, or control logic
reads this data. The operator wants strain-based throttling later and
needs real traffic data first.
Extend watchdogBody (timeout.go) rather than add a new interception
layer: it already wraps every upstream response body for provider
adapters, native passthrough, and the MCP/A2A target proxy, and already
observes the first successful read. armLatency arms a watchdogBody
after construction, so newWatchdogBody keeps its existing signature and
behavior; its own pinned tests still pass unchanged. A timestamp plus a
few atomics are the only added cost; Read and Close each pay one nil
check when latency collection is disabled.
Expose llmgateway_upstream_ttfb_seconds and
llmgateway_upstream_duration_seconds on the existing Prometheus
endpoint (metrics.go). Both are in-process, per-replica histograms,
never store-backed, so sum() is always the correct cross-replica
aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and
short completions finely, cover the bulk of real chat completions, and
bracket the five-minute default request timeout. Labels are provider
and stream always, model only when Config.Metrics.ModelLabel is set,
matching the existing per-model opt-in convention. Add a compact
per-provider, per-stream average summary to GET /admin/api/overview,
sourced from the same in-process store, without a full histogram or a
per-model breakdown.
Every write path is gated on metricsEnabled(cfg), so a deployment with
metrics disabled collects nothing and the histogram families never
render. No behavior change to request handling: same responses, status
codes, headers, and accounting.
Fix a Yaegi-only bug this feature's own context-threading change
exposed: two distinct empty struct types used as context.WithValue keys
compare equal to each other under the interpreter's interface equality,
though never under compiled Go. Threading a second recorder
(latencyRecorder) alongside the existing attemptRecorder made this
observable for the first time, since compiled tests never see it and
the plugin previously carried only one such key. Give
latencyRecorderCtxKey a field so its size differs from
attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction
before and after the fix.
Fix a second bug, reproducible under compiled Go too: io.Reader may
legally return its final data together with io.EOF in one call, and
Go's real net/http body reader does this for small, fully-buffered
responses. The original TTFB capture lived inside the err == nil
branch, so it never fired for a typical small non-streaming completion.
Move the check ahead of the error branch so it fires on any n > 0,
leaving the pinned timer-reset and Close semantics untouched.
Tests mutate the real source per case and confirm each one fails first,
recorded in each test's own doc comment.
lukaszraczylo
