Counters (always on when admin is enabled):
- last seen per user and group, written asynchronously after
admission, throttled to one write per scope per minute per replica
- hourly provider and model attempts, failures and timeouts
- failovers per provider, rejections per hour, and unpriced (402)
refusals
- cache hits, misses and cost saved
- request counts per target and caller
Opt-in counters under admin.stats (both default off):
- userModel: usage per user and model
- latency: latency histograms per provider and model
All counters ride the existing Redis batches. A test measures three
round trips per request with defaults and with every statistic on. A
cache hit takes two. A model request is no longer counted for failed
failover candidates.
New admin endpoints:
- usage/series and usage/totals
- performance, with percentiles from the histograms
- catalog, with the price billing actually uses and its source
- consumers, with key source, grants and model access
- config, with secrets redacted
The overview gains features and a pricing summary. Usage gains
lastSeen. usage/models gains offset and more detail fields. Reads are
chunked at 1600 keys per round trip and capped at 64000 keys per
request.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
GET /admin/api/events returns recent failures, newest first. It covers
rate limits, budget and unpriced-model refusals, upstream errors,
timeouts and capacity rejections. Events go into an in-process ring of
200 entries and are copied to a Redis list (llmgw:events, LPUSH plus
LTRIM to 200). The copy happens only on the error path, is throttled to
50 per second per replica, and never blocks a request. When Redis is
absent or down, the feed falls back to this replica's ring. Messages
are fixed strings or scrubbed provider errors. Fields are assigned one
at a time, because Yaegi drops values from a multi-value assignment
that contains a call.
Rejections per day are counted per scope and in total, in the same
store call that already rolls back a rejected request. /admin/api/usage
returns them as rejectionsPerDay. /admin/api/usage/models accepts span,
which sums that many buckets, and prefix, which filters model ids
before the limit applies. The overview reports the replica id, the
instance name and the warnings logged while the middleware was built.
The failover and retry fan-out warning is now logged at WARN.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Release builds send one anonymous "plugin loaded" ping per process, so
the project can see roughly how many installs exist and which versions
are in use. The payload is project name, version and timestamp. Nothing
else leaves the pod: no identifiers, no hostname, no configuration, no
provider names, no keys, no request data.
Three things keep it from firing when it should not. Builds from source
carry the devPluginVersion sentinel and stay silent, so developer
checkouts and every test run never phone home. The ping is sent only
after the plugin constructs successfully, so a rejected configuration is
never counted as an install. A sync.Once holds it to one per process,
because Traefik calls New once per route and oss-telemetry does not
deduplicate on the client side.
Operators of a release build opt out with DO_NOT_TRACK=1,
OSS_TELEMETRY_DISABLED=1, or TRAEFIK_LLMGATEWAY_DISABLE_TELEMETRY=1.
README.md documents the full payload and the opt-out; a plugin that
phones home without saying so in its own README would be indefensible.
oss-telemetry is the plugin's first non-stdlib RUNTIME dependency, which
has consequences for the Yaegi gate:
- vendor/ is no longer excluded from yaegi-check's GOPATH copy. It was
excluded on the reasoning that it held only test-only testify and
_test.go files are never interpreted. That reasoning no longer holds:
leaving vendor out would make the interpreted copy fail to resolve an
import the real plugin tree resolves fine, so the gate would stop
representing production.
- yaegi-check now stamps a fake release version into its GOPATH copy so
New's telemetry gate is PASSED rather than short-circuited, and the
vendored package is actually called under the interpreter. Import
resolution alone is not coverage; a cross-package call into interpreted
third-party code is its own risk. Telemetry is suppressed by env vars
set before anything is interpreted, so the gate never emits a real
ping.
- A new assertion reads the version back through GET /admin/api/overview.
Without it a silently failed stamp would leave the dev sentinel in
place and the harness would still print OK while covering nothing.
Verified by disabling the stamp and watching the gate fail.
shouldSendTelemetry is split out from sendStartupTelemetry so the
decision is table-testable without a network, since oss-telemetry's
endpoint is deliberately not overridable from consuming code.
Also brings the version-gated documentation up to date, since the same
README edit touches it. The latest release is v1.0.3, so the pre-1.0
experimental warning has met its own stated condition and is replaced by
a semver stability statement in README.md and dropped from
.traefik.yml's catalog summary. The catalog now serves the released tags
(verified: the v1.0.3 download endpoint returns 200), so the "not yet
published" and "analyzer has not been run" claims are corrected, the
catalog form of the static config is documented as usable, and
examples/kubernetes.yaml pins v1.0.3 instead of v0.1.0.
respClient held a single net.Conn behind one mutex, serializing every
caller's whole round trip. Interpreted-harness audit measured this as
the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on
memoryStore for the same workload, latency scaling near-linearly with
concurrency (869us@conc=1 -> 10.81ms@conc=16).
Replace the single connection with a pool of independently usable
connections (respConn slots in a buffered channel), each acquired for
one caller's whole round trip and returned via a deferred send back
onto the pool. A broken connection is discarded (closeConn) but its
slot always returns to the pool, so pool size never shrinks after an
error. Default size is GOMAXPROCS clamped to [4, 8]
(defaultRespPoolSize); RedisConfig.PoolSize lets an operator override
it explicitly, per the house rule that an explicit override always
wins over self-tuning.
Preserves exactly: the 2s respCallTimeout call deadline, reconnect-
once-on-error semantics, nested-array rejection, the RESP size bounds,
and every existing pipelining contract (incrMulti, incrAndGetMulti,
getMulti, cache get/set) — nothing outside resp.go references a
connection field directly.
Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured
at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%),
throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1
control matched HEAD exactly, confirming the win is parallelism, not
refactor noise.
cache_test.go and llmgateway_test.go: adjust white-box tests that
issue multiple sequential calls against one respClient scripted with
runFakeRESPServer's single-connection model — force pool size 1 there
(newRESPClientPool) so the test still pins one connection's exact wire
sequence; production callers are unaffected.
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
Bound respClient's worst-case latency to one respCallTimeout per
do/pipeline call instead of per read/write/handshake step. Compute a
single absolute deadline at call entry, apply it once via
conn.SetDeadline after connecting (covering the AUTH/SELECT handshake
and the call's own command I/O), and skip the reconnect-retry when the
failing attempt was a timeout (errors.As -> net.Error, Timeout()) — a
peer that accepted the connection and went silent must not be asked
the same non-question twice. Previously each read, write, and dial got
its own fresh 2s budget and every caller queued behind the client's
single mutex, so a hung (not merely refused) Redis-compatible server
could stall one checkAndCount call for 16-32s+.
Cap decodeBulk/decodeArray's wire-declared lengths (respMaxBulkLen
64MiB, respMaxArrayLen 1<<20) before allocating: an unguarded
"$9223372036854775807\r\n" overflows n+2 into a negative slice length
and panics make([]byte, n+2); a merely huge-but-valid length forces an
outsized allocation either way. decodeBulk also now verifies its two
trailing bytes are CRLF. Bound readLine to respMaxLineLen (64KiB),
reading byte-by-byte so it can bail before ever buffering an unbounded
line from a peer that never sends '\n'.
Fix TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation: the
erroringStore it used fails incrBy too, so checkAndCount's own
increment loop already returned a storeDown violation before
evaluateScope (and therefore budgetViolation) ever ran, leaving
budgetViolation's storeDown branch uncovered. Replace it with a stub
that only fails get, so the test's TokensPerDay limit actually drives
evaluateScope into budgetViolation's fail-closed path.
Document redis_store.go's incrBy as deliberately at-least-once: the
reconnect-retry can re-send INCRBY after a lost reply (possible
over-count), and fail-open can additionally count in the fallback —
both fail-safe (more restrictive), never fail-open in the unsafe
direction.
Validate Redis.DB >= 0 in newConfiguredLimiter alongside Address, so a
negative db is a constructor error rather than a SELECT -1 on the wire.
New tests: a hung (accepts, never replies) fake RESP server proving a
respClient call and a failOpen limiter call both stay bounded rather
than stacking timeouts; decodeReply subtests for an oversized bulk
length, a length just over the cap, a huge array length, a bad bulk
trailer, and an oversized line; a negative-DB constructor-rejection
test.
Add resp.go: a minimal stdlib-only RESP2 client (respClient) for a
single Redis-compatible server. It lazily dials one mutex-guarded
connection, applies 2s dial/read/write deadlines, sends AUTH (if a
password is set) then SELECT on every new connection, and reconnects
once on any I/O error before giving up. do and pipeline encode
commands as "*N\r\n$len\r\n..." and decode +/-/:/$/* replies.
Add redis_store.go: redisStore implements counterStore over
respClient. incrBy pipelines INCRBY and EXPIRE in one round trip and
returns INCRBY's value; get treats a missing key (RESP null bulk) as
0.
Wire the limiter's fail-open/fail-closed policy in limits.go, which
Task 5 stored but never read: a configured store's error now routes
through storeIncrBy/storeGet. failOpen=true falls back to the
in-process memoryStore transparently; failOpen=false makes
checkAndCount return a storeDown violation and makes account drop the
sample. Both paths log through an injectable, 30s-rate-limited logf.
Wire newGateway (llmgateway.go) to build a redisStore-backed limiter
when Config.Redis is set (password through resolveSecret, failOpen
defaulting to true unless RedisConfig.FailOpen says otherwise) and a
memory-only limiter otherwise, and to route pricing.go's unknown-model
warnings through the gateway's own logf.
Validate LimitsConfig at construction: a new LimitsConfig.validate
rejects a negative int64 count limit or a negative/NaN/Inf cost limit,
called from newAuthStore for every group and every user (inline and
file-sourced) so a malformed limit is a constructor error instead of
silently behaving as unlimited.
Add RequestsPerDay, TokensPerMonth, and CostPerMonthUSD enforcement
tests alongside the existing per-minute/day/cost-per-day cases, plus
an erroringStore-backed test pair proving the fail-open (fallback
counted) and fail-closed (storeDown violation) branches directly.
Folded the validation work into this single commit rather than a
second one: it interleaves with the fail-open changes across
limits.go, limits_test.go, and llmgateway_test.go closely enough that
splitting by hunk would be artificial.
Add usersFile (newUsersFile/load) for the JSON users-file shape
{"users":[{name,group,apiKey,limits?}]}, and authStore.maybeReload
for throttled hot reload: checks mtime at most every reloadEvery
(5s), reloads on change, keeps the last good set and logs via
errorf on any error (stat, parse, or a rejected replace), and logs
via logf when the reload changes the file-sourced user count.
nowFn is injectable for tests.
Wire-up in llmgateway.go: newGateway does a synchronous initial
load when Users.File is set (a bad initial file fails the
constructor); ServeHTTP calls g.auth.maybeReload() at entry
(no-op when no file is configured).
Also fixes replaceFileUsers's merge semantics: a file user now
overrides an inline user of the same name, even when their keys
differ, dropping the inline entry from the rebuilt set. The prior
digest-only keying let a same-named inline/file pair with
different keys coexist instead of the file version winning.
lukaszraczylo