mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-17 05:37:43 +00:00
perf(limits): single-batch counter writes, bounded fallback retention
Review fix round (2 Important + 5 folded) on 08cfdb4/b3fb705.
1. counterStore gains incrMulti(entries []counterIncr) ([]int64, error).
redisStore.incrMulti sends every entry's INCRBY+EXPIRE pair as ONE
pipeline (ttlToSeconds extracted from incrBy, now shared);
memoryStore.incrMulti loops incrBy for interface conformance.
checkAndCount batches every scope's req:min/req:day/req:hour into one
storeIncrMulti call (down from up to 9 round trips for a 3-scope
request) and reads req:min/req:day back from the reply, no extra
GETs. account batches every scope's nonzero tok-in/tok-out/cost
entries into one call (down from up to 27). Violation math and
fail-open/fail-closed/latched semantics are unchanged — only the
round-trip count moves.
2. memoryStoreMaxTTL (48h) caps every TTL memoryStore actually applies,
clamped inside incrBy so incrMulti's loop inherits it for free. The
in-process fallback is continuity, not history: applying the real
hour/day/month TTLs there was growing live-key count ~30x, multiplying
sweepEvery's own documented "167x incrBy cliff" by the same factor.
Sweep cadence is unchanged (still 30s-gated); the clamp shrinks what
there is to sweep instead.
Folded: historyMaxSpan's margin comment corrected (~4 days, not a
month); checkAndCount's godoc names req:hour; the integration
current-hour assertion sums every point instead of reading only the
last one (an hour-rollover race the suite cannot control); a new
exact-boundary test proves tokin+tokout landing exactly on a
TokensPerDay limit still violates; README documents the history route
as unthrottled (admin-gate is its only trust boundary) and the
fallback's 48h retention ceiling.
Tests: countingIncrStore proves exactly one incrMulti call per
checkAndCount and per account, with correct per-scope/metric/window
values and unchanged violation math; a fake-RESP-server test pins
incrMulti's wire format (one pipeline, entry-ordered replies); TTL
clamp tests cover both the ceiling and the pass-through case at the
boundary.
Gates: go test ./... -count=1, -race, gofmt, vet, golangci-lint,
make yaegi-check all green. make integration: every test passes
except the pre-existing TestUsersFileHotReload flake, which is NOT
caused by this change — verified via an isolated worktree at the
prior commit (b3fb705) run under identical conditions (full suite,
same host), which fails identically. This host's Docker Desktop is
currently slow to propagate the bind-mounted users.json change into
the containers; auth.go/users_file.go are untouched by this diff.
This commit is contained in:
@@ -509,26 +509,38 @@ http:
|
||||
accuracy.
|
||||
- **Storage**: `redis` configured wires a hand-rolled, stdlib-only RESP2
|
||||
client (`resp.go`) — no `go-redis`, which is not Yaegi-interpretable —
|
||||
against Redis, Valkey, or Dragonfly. Semantics are deliberately
|
||||
**at-least-once, never under-counting**: a lost reply after the server
|
||||
already applied an `INCRBY` can cause a resend that over-counts by one
|
||||
request's worth on that key. This is a fail-*safe* trade-off (more
|
||||
restrictive than reality), never a fail-*open* one. Without `redis`
|
||||
configured, or when it errors and `failOpen: true`, counters live in an
|
||||
in-process map — correct for one Traefik replica, only approximate
|
||||
across several, since each replica counts independently. Exact limits
|
||||
across multiple Traefik replicas require the shared Redis. Every
|
||||
counter, including the fallback in-process one, works as the usage-
|
||||
history API's data source: a fallback-only deployment gets charts of
|
||||
whatever the process itself accumulated, not an empty series.
|
||||
- **Retention** (counter key TTL, distinct from a window's own length):
|
||||
a minute key lives 2 minutes, an hour key 48 hours, a day key 35 days,
|
||||
a month key 400 days — long enough for the usage-history API's largest
|
||||
span (48 hourly / 35 daily / 13 monthly buckets) to always find a live
|
||||
key, not just long enough to survive that bucket's own single rollover.
|
||||
**Migration**: none — a deployed process's old, unsplit `tok` counter
|
||||
keys simply expire on their existing TTL and are never read again;
|
||||
nothing needs backfilling.
|
||||
against Redis, Valkey, or Dragonfly. Every counter write (`checkAndCount`,
|
||||
`account`) goes through ONE pipelined round trip for the whole request,
|
||||
regardless of how many user/group/total counters it touches (perf
|
||||
review, 2026-08-21) — a 3-scope request previously paid up to 9 (request
|
||||
counting) or 27 (usage accounting) separate round trips. Semantics are
|
||||
deliberately **at-least-once, never under-counting**: a lost reply after
|
||||
the server already applied the pipeline can cause a resend that
|
||||
over-counts every counter in it by its own delta. This is a
|
||||
fail-*safe* trade-off (more restrictive than reality), never a
|
||||
fail-*open* one.
|
||||
- **Fallback** (no `redis` configured, or a Redis error with
|
||||
`failOpen: true`): counters live in an in-process map — correct for one
|
||||
Traefik replica, only approximate across several, since each replica
|
||||
counts independently. Exact limits across multiple Traefik replicas
|
||||
require the shared Redis. Every counter, including this fallback one,
|
||||
works as the usage-history API's data source, but its retention is
|
||||
capped at **48 hours regardless of window** (`memoryStoreMaxTTL`,
|
||||
limits.go — ruling, 2026-08-21: the in-process fallback is continuity,
|
||||
not history; applying the real day/month TTLs to it would grow its live
|
||||
key count roughly 30x, multiplying the cost of its own periodic sweep by
|
||||
the same factor). A fallback-only deployment's usage-history charts
|
||||
therefore show at most the trailing 48 hours, never the full day/month
|
||||
span Redis-backed deployments get — full retention needs the shared
|
||||
Redis.
|
||||
- **Retention** (Redis counter key TTL, distinct from a window's own
|
||||
length): a minute key lives 2 minutes, an hour key 48 hours, a day key
|
||||
35 days, a month key 400 days — long enough for the usage-history API's
|
||||
largest span (48 hourly / 35 daily / 13 monthly buckets) to always find
|
||||
a live key, not just long enough to survive that bucket's own single
|
||||
rollover. **Migration**: none — a deployed process's old, unsplit `tok`
|
||||
counter keys simply expire on their existing TTL and are never read
|
||||
again; nothing needs backfilling.
|
||||
|
||||
## Retry
|
||||
|
||||
@@ -773,7 +785,11 @@ request to any of them falls through to the plugin's existing
|
||||
`{"scope","metric","window","points":[{"bucket":"2026082114","value":123},...]}`.
|
||||
Unlike the other two JSON routes, this one **never counts request
|
||||
statistics** — it skips `checkAndCount` entirely, ahead of the same
|
||||
change landing for every `/admin/api/*` route in a later change.
|
||||
change landing for every `/admin/api/*` route in a later change. It is
|
||||
therefore also **unthrottled**: no `requestsPerMinute`/`requestsPerDay`
|
||||
limit ever applies to it, so its only trust boundary is the admin gate
|
||||
itself (a valid admin key). An admin key holder can poll it as fast as
|
||||
they like.
|
||||
- **What's exposed**: provider names, types, base URLs (with any
|
||||
userinfo/query string stripped before it's ever echoed), model counts,
|
||||
discovery status, group/user names, membership, limits, and live usage
|
||||
|
||||
Reference in New Issue
Block a user