GET /admin/api/events returns recent failures, newest first. It covers
rate limits, budget and unpriced-model refusals, upstream errors,
timeouts and capacity rejections. Events go into an in-process ring of
200 entries and are copied to a Redis list (llmgw:events, LPUSH plus
LTRIM to 200). The copy happens only on the error path, is throttled to
50 per second per replica, and never blocks a request. When Redis is
absent or down, the feed falls back to this replica's ring. Messages
are fixed strings or scrubbed provider errors. Fields are assigned one
at a time, because Yaegi drops values from a multi-value assignment
that contains a call.
Rejections per day are counted per scope and in total, in the same
store call that already rolls back a rejected request. /admin/api/usage
returns them as rejectionsPerDay. /admin/api/usage/models accepts span,
which sums that many buckets, and prefix, which filters model ids
before the limit applies. The overview reports the replica id, the
instance name and the warnings logged while the middleware was built.
The failover and retry fan-out warning is now logged at WARN.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
A model id served by two configured providers is resolved
deterministically by the registry, the winner is named in the message,
and no request is affected. Logging it at ERROR put 144 ERROR lines in a
healthy pod's boot log on a cluster with overlapping openai,
openai-audio, copilot and macstudio catalogs, which is exactly the noise
that hides a real failure.
Adds Gateway.warnf beside logf and errorf, and a warn field on
modelRegistry carrying it. The field is set by the production caller
after newModelRegistry returns rather than added as a fourth constructor
parameter: 45 call sites pass a logger today, almost all of them tests
passing a no-op, and threading a parameter through all of them would
bury a one-line severity change in unrelated churn. A nil warn falls
back to log, so a registry built without one keeps its current behaviour
instead of panicking or dropping the line.
registry.warnf takes an already-formatted string so both call shapes
stay single-variadic-argument. Yaegi v0.16.1's CFG builder panics on a
call to a struct FIELD of variadic func type with two or more variadic
arguments, which is the restriction the log field beside it already
documents.
Verified under the real interpreter, not only compiled: yaegi-check
drives a genuine two-provider collision and now prints WARN. The
severity test fails if the call site reverts to log, so the level cannot
drift back unnoticed.
Only this message moves. Genuine failures — discovery errors, metadata
capture panics, encoding failures — stay at ERROR.
Fixes critical finding 1 from security review round 2, 2026-08-22:
identify checked the per-source-IP throttle BEFORE verifying the
presented key. This plugin runs behind a Kubernetes Service, and with
externalTrafficPolicy: Cluster (this operator's own deployment),
r.RemoteAddr is the node's own SNAT address shared by every external
caller reaching that node — not a per-tenant address. 20 failed requests
from anywhere on the internet sharing that node locked out all 10 live
users, including the admin dashboard's own poll.
- identify now verifies the key FIRST, unconditionally: a valid key
always succeeds before any throttle state is read. Every failure path
still records against the caller's IP, but throttling is now pure
observability (bounding log volume and tracker memory), never
enforcement — an attacker with no valid key already got 'false'
either way, so nothing is lost against brute force.
- Replaces the per-source-IP tracker's backing store: the general-
purpose memoryStore (limits.go) has no size cap and a full mutex-held
sweep that scales with tracked-IP count (~83ms at 1M IPs). authFailureTracker
is a bounded, two-generation, O(1)-rotation structure capped at
2x50,000 entries (important finding 1).
- logAuthEvent now emits a distinct, separately rate-limited
'auth throttle:' line the first time a source crosses the limit
(important finding 5), and both the routine and throttle-engaged
lines report how many events they suppressed since the last line
actually written, so several distinct attacker IPs in one window no
longer collapse into a single indistinguishable line (important
finding 6).
- The success path no longer touches the throttle tracker's mutex at
all (important finding 2).
Fixes finding 3 from the 2026-08-22 security audit: authStore.identify
had no counter, no backoff, and no lockout on failed attempts, and
logger.go wrote one unbuffered stderr line per attempt including the
caller's RemoteAddr. An unauthenticated attacker got unlimited digest
guesses at line rate and could drive synchronous stderr writes on the
shared Traefik ingress.
- Add per-source-IP failure throttling to identify, reusing the existing
in-process memoryStore (limits.go) as authStore's own dedicated
instance rather than a new dependency: authFailureLimit (20) failed
attempts per authFailureWindow (1 minute) throttles further attempts
from that IP outright, including a subsequently valid key, and resets
once the window elapses.
- clientIP reads r.RemoteAddr only, never a client-supplied header,
matching the same trust boundary the rest of this package already
assumes (routes_passthrough.go strips X-Forwarded-*/X-Auth-* before any
outbound request).
- Rate-limit logAuthEvent's failure line via authStore.shouldLogAuthFailure,
the identical technique limiter.logStoreError already applies to store
errors. Successful-auth logging stays unbounded (bounded by legitimate,
already-authenticated traffic elsewhere).
logAuthEvent (logger.go) formatted r.URL.Path with %s in both the
failure and success log lines. An unauthenticated client controls that
path and can embed a raw newline (%0A) in it, letting a single request
forge additional, attacker-chosen log lines. Switched both to %q, which
escapes control characters (newlines included) while still rendering
the path as a quoted string.
No test changes needed: both existing tests (llmgateway_test.go) assert
via strings.Contains on the bare path substring, which %q's quoted
output still contains.
Gates: go test ./... -count=1 green, gofmt -l . empty, go vet clean.
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
lukaszraczylo