mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
41bff22cb5cbe36bffe501e647ea3e94dc5fc086
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
4c01c75276 |
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
4fb5315bb3 |
perf(store): settle expire bookkeeping once per pipeline
incrMulti and incrAndGetMulti took the expire mutex twice per counter: once to ask needsExpire whether the key still wants an EXPIRE, and once after the reply to commit or forget it. A metered request writes 28 counters, so each request took that mutex about 56 times. needsExpireBatch now answers for the whole batch under one lock, and settleExpireBatch applies the commit and forget decisions under one more. The decisions are unchanged: needsExpire is read for every entry before any commit, a confirmed EXPIRE commits the key, and an INCRBY reply equal to the delta forgets it. The FIFO eviction bound is unchanged. applyIncrReplies decodes the INCRBY replies and reports how many entries it examined, so the caller settles exactly that prefix on every path. A reply error used to leave the already-decided entries unsettled, which kept a key marked as having an EXPIRE it never received. That key could then never regain a TTL, because its INCRBY reply can never equal the delta again. A provider failover answering READONLY mid-batch reaches this path. Two tests reproduce it and fail without the settle. Measured under yaegi as above: this step removed a further 4% of the bytes allocated per request. Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad |
||
|
lukaszraczylo
|
90eb58171e |
perf(redis_store): send EXPIRE only once per key per process lifetime
31 of 74 commands per request in the interpreted-harness audit were EXPIRE, emitted unconditionally after every INCRBY. Deleting EXPIRE entirely (upper bound) measured 74->43 cmds/req, p50 867.9->697.9us (-19.6%), conc=16 throughput 1,477->2,336 req/s (+58%), allocs -26%. Route chosen: a bounded first-touch-per-key cache (expireOnce), not EVALSHA+Lua. windowKey (limits.go) already embeds each counter's own time bucket in its key string, and every TTL callers pass is deliberately far longer than its window's own natural length purely as a stale-key GC margin (limits.go's own doc comment on dayWindowTTL/monthWindowTTL/etc.), not a sliding-window mechanism — so once EXPIRE has been sent for an exact key, every further INCRBY against that SAME key happens well within its own bucket's window, comfortably inside the TTL's margin. Skipping the redundant resend cannot leave a key un-expiring: a bucket's key simply stops being written once its window rolls over (a new bucket string takes its place). expireOnce is bounded (expireOnceCacheMax=4096, FIFO eviction) so a long-lived process's ever-rolling buckets cannot grow it unboundedly; an evicted key just gets one redundant-but-harmless EXPIRE again, exactly as if never touched before. incrBy/incrMulti/incrAndGetMulti now build a variable-stride command list (each entry contributes its own INCRBY, plus EXPIRE only when expireOnce(key) is true) with an explicit incrReplyIdx/base tracking each entry's actual reply position — no longer a fixed i*2 stride, since a skipped EXPIRE shifts every later command left by one. EVALSHA+Lua was rejected as the shippable route here: it would rewrite every command-building and reply-index path, requires NOSCRIPT fallback handling, and every existing wire-level test (dozens of INCRBY+EXPIRE-scripted fakeRESPServer cases) would need rebuilding around SHA1 digests for no additional correctness benefit over the simpler first-touch cache. |
||
|
lukaszraczylo
|
311dd20a8e |
fix(limits): correct dangling comments and redisStore edge cases
Independent review round approved the fusion itself (11 to 3 RTT/req, precedence verified branch by branch, latch/fail-open checked under Yaegi against a dead Redis). This commit addresses the follow-up items raised against that same change. Six doc comments in limits.go still named evaluateScope, budgetViolation, or tokenBudgetViolation as if they were live functions to consult. All three were removed by the fusion commit; the comments now point at checkAndCount and buildBudgetProbes, where the same logic lives today. One further comment claimed checkAndCount still slices a "storeIncrMulti result" — it now slices storeIncrAndGetMulti's own incrVals. TestCheckAndCount_FusedViolationPrecedence's doc comment claimed every subtest engineers two simultaneous violation conditions. The seventh engineers only one (the group violates; the user carries no limits at all) — it pins a different claim, that a later scope is still evaluated when an earlier one has nothing to check. The comment now says so. redisStore.incrAndGetMulti gained the same len()==0 guard incrMulti already has, and a table-driven test proves a RESP error or a non-integer value on a read key fails the whole call, matching respClient.getBatch's own equivalent table. Re-ran the interpreted Yaegi harness after these changes: RTT/req stays at 3.00, cmds/req stays at 74.00 — the fusion's throughput win is unaffected by this round's fixes. |
||
|
lukaszraczylo
|
a418814b34 |
perf(limits): fuse admission reads into one round trip
checkAndCount already batched its request-counter increments into one pipeline. evaluateScope then read every scope's token and cost budgets through up to four further serial round trips per scope. A request with a fully-limited user and group scope paid nine round trips before the upstream call started. This change fuses every budget read into the SAME pipeline as the increments, through a new incrAndGetMulti method on counterStore. The reads are safe to fuse because they touch disjoint keys (req:* versus tokin:*/tokout:*/cost:*), and tokens and cost are only ever written by account() after the response completes, never by checkAndCount itself. Violation precedence stays identical to the pre-fusion implementation: a user's own violation still wins over their group's, and within one scope the order stays requests-per-minute, requests-per-day, tokens-per-day, tokens-per-month, cost-per-day, cost-per-month. New table-driven tests pin this precedence directly, engineering two simultaneous violation conditions per case and asserting only the higher-precedence one is reported. evaluateScope, budgetViolation, and tokenBudgetViolation are removed; buildBudgetProbes plus a loop inline the same logic into checkAndCount, which is what lets their reads share the fused round trip. Measured on the interpreted Yaegi harness (yaegi v0.16.1, the runtime this plugin actually ships under): RTT/req for a fully-limited chat request drops from 10.99 to 3.00, with cmds/req unchanged at 74 (this is a round-trip fix, not a command-count or allocation change). Single-request throughput rises from 112 to 335 req/s; at concurrency 16, from 121 to 380 req/s. counterStore.incrAndGetMulti is a required interface method, not an optional capability behind a type assertion: Yaegi can silently return ok=false when a comma-ok assertion checks a compiled concrete value against a plugin-declared interface (Trap 4, documented in matchesSentinel's own doc comment). A plain interface method call carries no such risk and is the same mechanism incrMulti already uses in production. |
||
|
lukaszraczylo
|
5cc681496c |
perf(limits): single-batch counter writes, bounded fallback retention
Review fix round (2 Important + 5 folded) on 08cfdb4/b3fb705.
1. counterStore gains incrMulti(entries []counterIncr) ([]int64, error).
redisStore.incrMulti sends every entry's INCRBY+EXPIRE pair as ONE
pipeline (ttlToSeconds extracted from incrBy, now shared);
memoryStore.incrMulti loops incrBy for interface conformance.
checkAndCount batches every scope's req:min/req:day/req:hour into one
storeIncrMulti call (down from up to 9 round trips for a 3-scope
request) and reads req:min/req:day back from the reply, no extra
GETs. account batches every scope's nonzero tok-in/tok-out/cost
entries into one call (down from up to 27). Violation math and
fail-open/fail-closed/latched semantics are unchanged — only the
round-trip count moves.
2. memoryStoreMaxTTL (48h) caps every TTL memoryStore actually applies,
clamped inside incrBy so incrMulti's loop inherits it for free. The
in-process fallback is continuity, not history: applying the real
hour/day/month TTLs there was growing live-key count ~30x, multiplying
sweepEvery's own documented "167x incrBy cliff" by the same factor.
Sweep cadence is unchanged (still 30s-gated); the clamp shrinks what
there is to sweep instead.
Folded: historyMaxSpan's margin comment corrected (~4 days, not a
month); checkAndCount's godoc names req:hour; the integration
current-hour assertion sums every point instead of reading only the
last one (an hour-rollover race the suite cannot control); a new
exact-boundary test proves tokin+tokout landing exactly on a
TokensPerDay limit still violates; README documents the history route
as unthrottled (admin-gate is its only trust boundary) and the
fallback's 48h retention ceiling.
Tests: countingIncrStore proves exactly one incrMulti call per
checkAndCount and per account, with correct per-scope/metric/window
values and unchanged violation math; a fake-RESP-server test pins
incrMulti's wire format (one pipeline, entry-ordered replies); TTL
clamp tests cover both the ceiling and the pass-through case at the
boundary.
Gates: go test ./... -count=1, -race, gofmt, vet, golangci-lint,
make yaegi-check all green. make integration: every test passes
except the pre-existing TestUsersFileHotReload flake, which is NOT
caused by this change — verified via an isolated worktree at the
prior commit (
|
||
|
lukaszraczylo
|
33bd8bc2cc |
fix(admin): browser key entry, batched usage reads, hardened responses
Review fixes (2 Important + 4 folded): 1. GET /admin now serves the HTML shell with no authentication - a browser navigating directly to the URL cannot attach a custom header. The page reads an admin key from sessionStorage and sends it as x-api-key on /admin/api/* fetches, falling back to an inline key-entry form on a missing key or any 401/403. Both JSON routes keep their original 401/403/200 gate. Spec ss4 amended in place. 2. GET /admin/api/usage no longer serializes 6 GETs per user/group scope on the shared Redis connection. respClient gains getBatch (one pipelined round trip); counterStore gains getMulti; limiter gains storeGetMulti mirroring storeGet's fail-open/fail-closed policy. currentUsage now issues one getMulti per scope. 3. Provider baseUrl is stripped of userinfo and query before it is ever echoed in the overview JSON (sanitizeBaseURL), stronger than net/url's own Redacted (which leaves a bare username in place). 4. limiter.redisStatus now also returns the last failure's timestamp, so the dashboard can render "last error (Ns ago)" instead of a bare message that reads as currently-broken indefinitely. 5. Both JSON admin responses now set X-Content-Type-Options: nosniff, Cache-Control: no-store, and the same CSP the HTML page uses. 6. Added a test proving a file-sourced (replaceFileUsers) admin user gets the same access an inline admin user does. go test ./... -count=1 -race, gofmt, vet, make yaegi-check all pass. |
||
|
lukaszraczylo
|
7ee635cd27 |
fix(resp): bound store-call latency, cap wire lengths, cover fail-closed budget path
Bound respClient's worst-case latency to one respCallTimeout per do/pipeline call instead of per read/write/handshake step. Compute a single absolute deadline at call entry, apply it once via conn.SetDeadline after connecting (covering the AUTH/SELECT handshake and the call's own command I/O), and skip the reconnect-retry when the failing attempt was a timeout (errors.As -> net.Error, Timeout()) — a peer that accepted the connection and went silent must not be asked the same non-question twice. Previously each read, write, and dial got its own fresh 2s budget and every caller queued behind the client's single mutex, so a hung (not merely refused) Redis-compatible server could stall one checkAndCount call for 16-32s+. Cap decodeBulk/decodeArray's wire-declared lengths (respMaxBulkLen 64MiB, respMaxArrayLen 1<<20) before allocating: an unguarded "$9223372036854775807\r\n" overflows n+2 into a negative slice length and panics make([]byte, n+2); a merely huge-but-valid length forces an outsized allocation either way. decodeBulk also now verifies its two trailing bytes are CRLF. Bound readLine to respMaxLineLen (64KiB), reading byte-by-byte so it can bail before ever buffering an unbounded line from a peer that never sends '\n'. Fix TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation: the erroringStore it used fails incrBy too, so checkAndCount's own increment loop already returned a storeDown violation before evaluateScope (and therefore budgetViolation) ever ran, leaving budgetViolation's storeDown branch uncovered. Replace it with a stub that only fails get, so the test's TokensPerDay limit actually drives evaluateScope into budgetViolation's fail-closed path. Document redis_store.go's incrBy as deliberately at-least-once: the reconnect-retry can re-send INCRBY after a lost reply (possible over-count), and fail-open can additionally count in the fallback — both fail-safe (more restrictive), never fail-open in the unsafe direction. Validate Redis.DB >= 0 in newConfiguredLimiter alongside Address, so a negative db is a constructor error rather than a SELECT -1 on the wire. New tests: a hung (accepts, never replies) fake RESP server proving a respClient call and a failOpen limiter call both stay bounded rather than stacking timeouts; decodeReply subtests for an oversized bulk length, a length just over the cap, a huge array length, a bad bulk trailer, and an oversized line; a negative-DB constructor-rejection test. |
||
|
lukaszraczylo
|
5eab51bdef |
feat: stdlib RESP client and redis-backed limit store with fail-open
Add resp.go: a minimal stdlib-only RESP2 client (respClient) for a single Redis-compatible server. It lazily dials one mutex-guarded connection, applies 2s dial/read/write deadlines, sends AUTH (if a password is set) then SELECT on every new connection, and reconnects once on any I/O error before giving up. do and pipeline encode commands as "*N\r\n$len\r\n..." and decode +/-/:/$/* replies. Add redis_store.go: redisStore implements counterStore over respClient. incrBy pipelines INCRBY and EXPIRE in one round trip and returns INCRBY's value; get treats a missing key (RESP null bulk) as 0. Wire the limiter's fail-open/fail-closed policy in limits.go, which Task 5 stored but never read: a configured store's error now routes through storeIncrBy/storeGet. failOpen=true falls back to the in-process memoryStore transparently; failOpen=false makes checkAndCount return a storeDown violation and makes account drop the sample. Both paths log through an injectable, 30s-rate-limited logf. Wire newGateway (llmgateway.go) to build a redisStore-backed limiter when Config.Redis is set (password through resolveSecret, failOpen defaulting to true unless RedisConfig.FailOpen says otherwise) and a memory-only limiter otherwise, and to route pricing.go's unknown-model warnings through the gateway's own logf. Validate LimitsConfig at construction: a new LimitsConfig.validate rejects a negative int64 count limit or a negative/NaN/Inf cost limit, called from newAuthStore for every group and every user (inline and file-sourced) so a malformed limit is a constructor error instead of silently behaving as unlimited. Add RequestsPerDay, TokensPerMonth, and CostPerMonthUSD enforcement tests alongside the existing per-minute/day/cost-per-day cases, plus an erroringStore-backed test pair proving the fail-open (fallback counted) and fail-closed (storeDown violation) branches directly. Folded the validation work into this single commit rather than a second one: it interleaves with the fail-open changes across limits.go, limits_test.go, and llmgateway_test.go closely enough that splitting by hunk would be artificial. |