mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
5cc681496c010e3b7741d2bedf51a7a71d8df9ad
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
5cc681496c |
perf(limits): single-batch counter writes, bounded fallback retention
Review fix round (2 Important + 5 folded) on 08cfdb4/b3fb705.
1. counterStore gains incrMulti(entries []counterIncr) ([]int64, error).
redisStore.incrMulti sends every entry's INCRBY+EXPIRE pair as ONE
pipeline (ttlToSeconds extracted from incrBy, now shared);
memoryStore.incrMulti loops incrBy for interface conformance.
checkAndCount batches every scope's req:min/req:day/req:hour into one
storeIncrMulti call (down from up to 9 round trips for a 3-scope
request) and reads req:min/req:day back from the reply, no extra
GETs. account batches every scope's nonzero tok-in/tok-out/cost
entries into one call (down from up to 27). Violation math and
fail-open/fail-closed/latched semantics are unchanged — only the
round-trip count moves.
2. memoryStoreMaxTTL (48h) caps every TTL memoryStore actually applies,
clamped inside incrBy so incrMulti's loop inherits it for free. The
in-process fallback is continuity, not history: applying the real
hour/day/month TTLs there was growing live-key count ~30x, multiplying
sweepEvery's own documented "167x incrBy cliff" by the same factor.
Sweep cadence is unchanged (still 30s-gated); the clamp shrinks what
there is to sweep instead.
Folded: historyMaxSpan's margin comment corrected (~4 days, not a
month); checkAndCount's godoc names req:hour; the integration
current-hour assertion sums every point instead of reading only the
last one (an hour-rollover race the suite cannot control); a new
exact-boundary test proves tokin+tokout landing exactly on a
TokensPerDay limit still violates; README documents the history route
as unthrottled (admin-gate is its only trust boundary) and the
fallback's 48h retention ceiling.
Tests: countingIncrStore proves exactly one incrMulti call per
checkAndCount and per account, with correct per-scope/metric/window
values and unchanged violation math; a fake-RESP-server test pins
incrMulti's wire format (one pipeline, entry-ordered replies); TTL
clamp tests cover both the ceiling and the pass-through case at the
boundary.
Gates: go test ./... -count=1, -race, gofmt, vet, golangci-lint,
make yaegi-check all green. make integration: every test passes
except the pre-existing TestUsersFileHotReload flake, which is NOT
caused by this change — verified via an isolated worktree at the
prior commit (
|
||
|
lukaszraczylo
|
08cfdb4d99 |
feat: split token counters and hour-resolution usage history
Data layer for the upcoming Vue admin charts (v0.2 data-layer task).
Tokens-in/out split (limits.go): metricTok replaced by metricTokIn
("tokin") and metricTokOut ("tokout"). account() records usage.prompt
and usage.completion independently, skipping a direction only when its
own value is zero. A TokensPerDay/TokensPerMonth limit still enforces
one TOTAL budget: tokenBudgetViolation reads both counters in one
storeGetMulti batch and sums them; the violation message still says
"tokens". Estimated usage (the request-body-size fallback) lands
entirely in tokin, documented at its call site.
Hour window (limits.go): windowHour ("hour"), bucket format
"2006010215" UTC, hourWindowTTL (48h). checkAndCount/account both write
it on every metered route; no LimitsConfig field ever names it, so
evaluateScope never evaluates it. dayWindowTTL/monthWindowTTL bumped to
35d/400d so the usage-history API's largest span always finds a live
key.
Total scope (limits.go, routes_unified.go): a synthetic
{kind:"total", id:"all"} scope, limits always nil. withTotalScope
wraps buildLimitScopes' result at every metered route (unified,
media, passthrough, mcp/a2a) so none can forget it; handleAdminAPI's
own accounting stays on buildLimitScopes alone, so admin traffic never
contributes to it.
History API (admin.go): GET /admin/api/usage/history?scope=&metric=&
window=&span= — admin-gated like the other two JSON routes, but
skips checkAndCount ahead of that same change landing for every
/admin/api/* route in a later task. Validates scope kind/metric/window
(400), span bounds per window (400), and unknown user/group id (404).
Computes every bucket key up front and reads them in ONE storeGetMulti
batch; a storeDown read is 503, never a silently-zero series. Works
against memoryStore too.
Admin usage API (admin.go): scopeUsage/adminUsageEntryView now expose
tokensInPerDay/tokensOutPerDay and their per-month counterparts in
place of the combined tokensPerDay/tokensPerMonth pair; buildAdminUsage
adds a "total" row to the response, still one storeGetMulti round trip
overall. usageKeysPerScope names the per-scope key stride (was a bare
6, now 8), closing a deferred review follow-up while the count changed
anyway.
Tests: TDD throughout — bucket/window-key math incl. UTC and hour
boundaries, the month step-back day-31 AddDate trap, split accounting,
token budget summing both directions, total-scope counting-without-
evaluating on every metered route, and the full history API (shape,
oldest-first, span bounds, validation table, 404, 503, one-batch
assertion via a counting stub). Integration: TestAdminDashboard now
fetches an hour history series for user alice and for total and
asserts a non-zero current-hour bucket.
Migration: none — old "tok" counter keys simply expire on their
existing TTL; nothing is deployed yet.
|
||
|
lukaszraczylo
|
657480348b |
fix: v0.2 final review wave
Transcriptions Yaegi coverage via integration mock, flattened admin usage batch, hardened CSP, sanitized provider lastErr, source-archive prefix template, vendor export-ignore warning, release paths. |
||
|
lukaszraczylo
|
63459b0c24 |
fix: v0.2 review sweep
Controller-ruled small fixes accumulated from T3/T4 reviews: 1. routes_media.go: multipart Content-Type check is now an exact mediaType != "multipart/form-data" comparison (HasPrefix admitted "multipart/form-datax"); a malformed-tail multipart body (fails to parse at all) now reports "malformed multipart body", distinct from "model is required" (a body that parses fine but carries no model part) and "duplicate model field". 2. provider_openai.go: audioTranscription's doc comment no longer claims the body is always "replayed unchanged"/the boundary always "verbatim" — it now reflects the rewrite path routes_media.go already takes for an aliased model id. 3. rewriteMultipartModel: buf.Grow(len(body)+512) up front, avoiding repeated reallocate-and-copy growth while rebuilding an aliased multipart body. 4. limits.go currentUsage: guards len(vals) == len(keys) before indexing storeGetMulti's result, so a non-conforming counterStore's short slice reports storeDown instead of panicking. 5. admin.go dashboard JS: the 5s poll no longer wipes an existing auth-error message when no key is stored (only a successful load or a new submission clears it now); clearStoredKey on a 401/403 now only fires when the failing request's own key still matches what's currently stored, guarding against a late response for a key the user has since replaced. 6. Spec §4's duplicated /admin/api/overview bullet and the nosniff/no-store sentence's placement were already fixed upstream (docs/superpowers/specs/2026-08-20-llmgateway-v02-design.md, commit c02010d) — verified identical, nothing further to commit here. 7. Added TestAdmin_JSONSecurityHeaders (nosniff/no-store/CSP on both admin JSON APIs) and TestHandleAudioTranscriptions_MalformedTail_400 (covers fix 1's new malformed-body branch). go test ./... -count=1 -race, gofmt, vet, make yaegi-check, golangci-lint (incl. gosec) all pass. |
||
|
lukaszraczylo
|
33bd8bc2cc |
fix(admin): browser key entry, batched usage reads, hardened responses
Review fixes (2 Important + 4 folded): 1. GET /admin now serves the HTML shell with no authentication - a browser navigating directly to the URL cannot attach a custom header. The page reads an admin key from sessionStorage and sends it as x-api-key on /admin/api/* fetches, falling back to an inline key-entry form on a missing key or any 401/403. Both JSON routes keep their original 401/403/200 gate. Spec ss4 amended in place. 2. GET /admin/api/usage no longer serializes 6 GETs per user/group scope on the shared Redis connection. respClient gains getBatch (one pipelined round trip); counterStore gains getMulti; limiter gains storeGetMulti mirroring storeGet's fail-open/fail-closed policy. currentUsage now issues one getMulti per scope. 3. Provider baseUrl is stripped of userinfo and query before it is ever echoed in the overview JSON (sanitizeBaseURL), stronger than net/url's own Redacted (which leaves a bare username in place). 4. limiter.redisStatus now also returns the last failure's timestamp, so the dashboard can render "last error (Ns ago)" instead of a bare message that reads as currently-broken indefinitely. 5. Both JSON admin responses now set X-Content-Type-Options: nosniff, Cache-Control: no-store, and the same CSP the HTML page uses. 6. Added a test proving a file-sourced (replaceFileUsers) admin user gets the same access an inline admin user does. go test ./... -count=1 -race, gofmt, vet, make yaegi-check all pass. |
||
|
lukaszraczylo
|
547b61f5a9 |
feat: read-only admin dashboard and JSON API
Adds GET /admin (single-page dashboard), GET /admin/api/overview, and GET /admin/api/usage per spec §4: providers/redis/cache/groups summary and per-user/group current-window usage+limits. Gated by Config.Admin (default off, not registered when disabled) and UserConfig.Admin. Gate order: disabled -> 404 fall-through; unauthenticated -> 401; authenticated non-admin -> 403; admin -> serve, counted against the admin's own request limits like any other route. Adds small, read-only, mutex-guarded hooks: modelRegistry.snapshot(), limiter.currentUsage()/redisStatus(), authStore.snapshot(). No secret field (API key, digest, redis password) is ever read into any admin response type; verified by a canary-literal redaction test. HTML/CSS/JS ship as a single Go raw-string const (Yaegi forbids go:embed); confirmed interpretable via make yaegi-check. |
||
|
lukaszraczylo
|
4b849fd6c7 |
fix: force include_usage, auth event logging, credential-strip coverage
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
|
||
|
lukaszraczylo
|
0e73558f5c |
ci: yaegi catalog-analyzer gate harness
Add tools/yaegi-check, a separate Go module (llmgw-yaegi-check,
never a dependency of the plugin module) that replicates the
Traefik Plugin Catalog analyzer (piceus)'s yaegiMiddlewareCheck:
copy the repo into a temp GOPATH, interp.New + i.Use(stdlib.Symbols),
import the plugin package, eval CreateConfig(), decode the plugin's
real .traefik.yml testData into it via a hand-rolled YAML-subset
parser plus a JSON round-trip, reflect-Call New from outside the
interpreter, then exercise the returned handler with two real HTTP
requests (authenticated GET /v1/models against the testData key, and
the same route unauthenticated). Pins github.com/traefik/yaegi to
v0.16.1, confirmed as Traefik v3.5.0's own vendored version by
downloading its go.mod.
Wire cd tools/yaegi-check && GOWORK=off go run . /Users/nvm/Documents/projects/private/traefik-llmgateway
yaegi-check: OK to run the harness from repo root.
The harness caught a real Yaegi v0.16.1 limitation: calling a struct
FIELD of variadic func type (e.g. modelRegistry.log
func(string, ...any)) with two or more variadic arguments crashes
Yaegi's CFG builder ('index out of range'). The identical call shape
through a method or interface method is unaffected — bisected with a
standalone yaegi repro to confirm the exact trigger before touching
plugin code. Fixed by pre-formatting with fmt.Sprintf at
modelRegistry's three affected call sites (registry.go) so the field
is always called with exactly one variadic argument; documented the
constraint on both modelRegistry.log and limiter.logf so it is not
reintroduced. Verified the gate itself is load-bearing by temporarily
renaming CreateConfig (harness failed non-zero) and restoring it
(harness passed again).
Also fixes two golangci-lint findings in the new harness module
(govet shadow, gosec G301 directory permissions).
|
||
|
lukaszraczylo
|
3ed473bdfe |
fix(routes): bill partial stream usage on error, pin openai alias stripping
Review fixes for Task 12: 1. (CRITICAL) runUnified now calls g.limiter.account BEFORE handling an adapter error, not after returning on it. All three adapters' forwardStream deliberately return captured usage alongside a mid-stream error, so a client that aborts right after the usage frame arrives was getting tokens delivered but never billed - repeated aborts bypassed every token/cost budget. A non-streaming error path always carries zero usage by contract, so accounting it is a no-op (account skips writes once both total tokens and cost are zero). The zero-usage-stream logf and the zero-usage-non-stream estimate now only fire when callErr is nil, so a genuine error no longer gets a fabricated token estimate. New test: TestHandleChat_MidStreamDropAfterUsageChunk_AccountsPartialUsage (hijack+close after the usage chunk; asserts the limiter's counter shows the captured tokens). Verified this test fails against the pre-fix ordering and the two new alias tests below fail when their guarded delete() call is removed, before restoring and re-running clean. 2. Added TestHandleChat_OpenAI_AliasKeyStrippedFromUpstreamBody and TestHandleEmbeddings_OpenAI_AliasKeyStrippedFromUpstreamBody: the openai-type adapter marshals req verbatim as the upstream wire body, so it is the one path where a leftover gatewayAliasKey entry actually reaches a real provider. The only existing alias-leak assertion covered anthropic, which rebuilds the body field-by-field and was never at risk. 3. Reworded the stale comment on limiter.evaluateScope (limits.go): it described a nil-limits scope's req counters as a normal, expected case. Ruling (e), implemented in routes_unified.go's buildLimitScopes, means the unified route never builds a nil-limits scope in the first place - the nil check remains a defensive no-op, not a path any current caller exercises. Gates: gofmt, go vet, fieldalignment, golangci-lint (incl. gosec) all clean; go test ./... -race -count=1 passes (full suite, including all Task 12 tests). |
||
|
lukaszraczylo
|
2acc0a2a84 |
fix(resp,limits): reject nested arrays, store-down latch, clamp dial to call budget
Reject a nested array reply instead of recursing into it. decodeReply now takes a depth parameter; a '*' type byte is accepted only at depth 0, so an element that is itself an array (decodeArray calls decodeReply at depth+1) is a protocol error rather than a recursive call. A crafted "*1\r\n*1\r\n..." stream repeated could otherwise recurse to Go's stack limit — a fatal, unrecoverable error, not a panic recoverPanic can catch — from only a few megabytes of wire input. Also lower respMaxArrayLen from 1<<20 to 1024: none of this client's commands (AUTH, SELECT, INCRBY, EXPIRE, GET) ever return an array reply at all, so even 1024 is generous headroom, not a limit tuned to real traffic. Add a store-down latch to the limiter (limits.go): after any store operation fails, record the failure time; for storeDownLatchFor (5s) afterward, every storeIncrBy/storeGet call short-circuits straight to the fail-open/fail-closed policy without attempting the store at all. One request can touch several counters (its own and its group's, request/token/cost, day/month) — each respClient call is already bounded to ~respCallTimeout on its own, but without the latch every one of those still paid a fresh bounded wait against a store that had just proven unreachable. After the window, the next operation probes the real store again: a natural half-open retry, no separate recovery state. Guarded by the same logMu mutex lastLogAt already uses for this kind of small timestamp state. Clamp respClient's dial timeout to the remaining call budget: dialTimeoutFor returns min(respDialTimeout, time.Until(deadline)), with a respDeadlineExceededErr (implements net.Error, Timeout() true) returned outright if the deadline has already elapsed — e.g. this call queued behind the client's mutex for its whole budget while an earlier call was stuck. Without this, a future increase to respDialTimeout alone, leaving respCallTimeout untouched, could silently let one dial attempt consume more of a call's total budget than the call-level deadline is supposed to allow. New tests: a nested-array reply and an array length one over the new 1024 cap, both asserted as protocol errors; a countingErrorStore stub proving a second checkAndCount within the latch window makes zero additional store calls, and that the store is probed again once the window elapses. Tightened TestLimiter_FailOpenTrue_ HungRedisServer_FallsBackQuickly's bound from <6s to <3s now that the latch collapses its two store ops into one real network attempt. |
||
|
lukaszraczylo
|
5eab51bdef |
feat: stdlib RESP client and redis-backed limit store with fail-open
Add resp.go: a minimal stdlib-only RESP2 client (respClient) for a single Redis-compatible server. It lazily dials one mutex-guarded connection, applies 2s dial/read/write deadlines, sends AUTH (if a password is set) then SELECT on every new connection, and reconnects once on any I/O error before giving up. do and pipeline encode commands as "*N\r\n$len\r\n..." and decode +/-/:/$/* replies. Add redis_store.go: redisStore implements counterStore over respClient. incrBy pipelines INCRBY and EXPIRE in one round trip and returns INCRBY's value; get treats a missing key (RESP null bulk) as 0. Wire the limiter's fail-open/fail-closed policy in limits.go, which Task 5 stored but never read: a configured store's error now routes through storeIncrBy/storeGet. failOpen=true falls back to the in-process memoryStore transparently; failOpen=false makes checkAndCount return a storeDown violation and makes account drop the sample. Both paths log through an injectable, 30s-rate-limited logf. Wire newGateway (llmgateway.go) to build a redisStore-backed limiter when Config.Redis is set (password through resolveSecret, failOpen defaulting to true unless RedisConfig.FailOpen says otherwise) and a memory-only limiter otherwise, and to route pricing.go's unknown-model warnings through the gateway's own logf. Validate LimitsConfig at construction: a new LimitsConfig.validate rejects a negative int64 count limit or a negative/NaN/Inf cost limit, called from newAuthStore for every group and every user (inline and file-sourced) so a malformed limit is a constructor error instead of silently behaving as unlimited. Add RequestsPerDay, TokensPerMonth, and CostPerMonthUSD enforcement tests alongside the existing per-minute/day/cost-per-day cases, plus an erroringStore-backed test pair proving the fail-open (fallback counted) and fail-closed (storeDown violation) branches directly. Folded the validation work into this single commit rather than a second one: it interleaves with the fail-open changes across limits.go, limits_test.go, and llmgateway_test.go closely enough that splitting by hunk would be artificial. |
||
|
lukaszraczylo
|
ed1b1d7b2f | fix(limits): cap unknown-model warnings, time-gated sweep, rounded micro-USD conversion | ||
|
lukaszraczylo
|
ac11023a45 | feat: limit engine with fixed windows, token and cost budgets |