mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
fix(limits): enforcement-aware fallback TTL clamp
Re-review round on 5cc6814 found round 1's 48h fallback clamp
defective: it broke TTL > window-length for the MONTH window on
memoryStore, turning tokensPerMonth/costPerMonthUSD into a rolling
~48h budget (fail-open) whenever Redis is absent or down.
1. counterIncr gains enforceTTL time.Duration — the minimum retention
window's own enforcement correctness requires (enforceTTLFor:
min->2m, hour->2h, day->25h, month->32*24h, matching each window's
pre-history-bump TTL). newCounterIncr builds every checkAndCount/
account entry through one helper so a (window, ttl) pair can never
reach incrMulti without its matching floor. memoryStore's clampTTL
now takes (ttl, enforceTTL) and applies max(min(ttl,
memoryStoreMaxTTL), enforceTTL): the 48h ceiling still bounds
min/hour/day (unaffected/still-capped as before), but the floor
wins for month, landing it back at ~32 days (1-2 live keys per
scope/metric — negligible against the day-key savings that
motivated the ceiling). redisStore ignores enforceTTL; incrBy's own
single-key path passes enforceTTL=0 (ceiling only, unchanged
behavior — it has no window to derive a floor from and has no
production caller since the batching, see (3) below).
2. limits.go's TTL-invariant comment restated precisely: the consts
still satisfy TTL > window-length exactly as applied to Redis; that
did not automatically extend to memoryStore's own effective TTL
after its ceiling, which is what round 1 broke and round 2 fixes.
README: the Limits and accounting "Fallback" bullet now says
enforcement stays correct on the fallback at every window
(including tokensPerMonth/costPerMonthUSD), and that the 48h
ceiling applies to usage-history CHARTING retention only.
LimitsConfig's window table (tokensPerDay/tokensPerMonth rows)
cross-links it. The "resets on a rebuild" bullet's stale
limiter.storeIncrBy reference is corrected to storeIncrMulti and
now distinguishes rebuild-loss from the fallback's own per-window
TTL expiry.
3. counterStore.incrBy, limiter.storeIncrBy, and limiter.incrCounter's
doc comments now state plainly they have no production caller since
the batching (perf review, 2026-08-21) and are kept for direct
counter seeding in tests — 5 call sites, all in admin_test.go's
TestAdminUsage_MathAgainstSeededCounters, correcting the prior fix
report's "dozens of tests" overclaim to the actual count.
Tests: TestClampTTL_ExactMathTable pins the ceiling-vs-floor
precedence for every window's real (ttl, enforceTTL) pair.
TestEnforceTTLFor_MatchesTheOriginalPreHistoryBumpValues pins the four
floor values directly. TestMemoryStore_IncrMulti_
MonthCounterSurvivesPast48hAndStillEnforces drives a real
TokensPerMonth budget through account/checkAndCount on the fallback
5 days in and confirms both survival and enforcement.
TestMemoryStore_IncrMulti_DayCounterStillClampsAt48h confirms day
still resets at the 48h ceiling (its own floor sits below it).
Gate: go test ./... -count=1, -race, gofmt, vet, make yaegi-check —
all green.
This commit is contained in:
@@ -229,6 +229,11 @@ that is negative, `NaN`, or `±Inf`, is a construction error.
|
||||
| `costPerDayUSD` | `float64` | Fixed calendar-day window, UTC. |
|
||||
| `costPerMonthUSD` | `float64` | Fixed calendar-month window, UTC. |
|
||||
|
||||
Every window enforces correctly on the in-process fallback too, not only Redis —
|
||||
see [Limits and accounting](#limits-and-accounting)'s "Fallback" bullet for the
|
||||
retention floor that keeps `tokensPerMonth`/`costPerMonthUSD` a true calendar-month
|
||||
budget rather than a rolling one when Redis is absent or down.
|
||||
|
||||
A user's own `limits`, when set, are checked **in addition to** their
|
||||
group's limits, not instead of them — a request is refused the moment it
|
||||
breaches whichever of the two is tighter. A user with no `limits` of their
|
||||
@@ -523,16 +528,21 @@ http:
|
||||
`failOpen: true`): counters live in an in-process map — correct for one
|
||||
Traefik replica, only approximate across several, since each replica
|
||||
counts independently. Exact limits across multiple Traefik replicas
|
||||
require the shared Redis. Every counter, including this fallback one,
|
||||
works as the usage-history API's data source, but its retention is
|
||||
capped at **48 hours regardless of window** (`memoryStoreMaxTTL`,
|
||||
limits.go — ruling, 2026-08-21: the in-process fallback is continuity,
|
||||
not history; applying the real day/month TTLs to it would grow its live
|
||||
key count roughly 30x, multiplying the cost of its own periodic sweep by
|
||||
the same factor). A fallback-only deployment's usage-history charts
|
||||
therefore show at most the trailing 48 hours, never the full day/month
|
||||
span Redis-backed deployments get — full retention needs the shared
|
||||
Redis.
|
||||
require the shared Redis. **Enforcement stays correct on the fallback at
|
||||
every window**, including `tokensPerMonth`/`costPerMonthUSD`: a
|
||||
window's counter always outlives its own natural length
|
||||
(`enforceTTLFor`, limits.go), so a month budget is still a true
|
||||
calendar-month budget, never a rolling one, whether Redis is configured
|
||||
or not. **Usage-history CHARTING retention is shorter on the fallback,
|
||||
though**: capped at 48 hours regardless of window
|
||||
(`memoryStoreMaxTTL`, limits.go — ruling, 2026-08-21: the in-process
|
||||
fallback is continuity for charting, not full history; applying the
|
||||
real day/month retention TTLs to it purely for charting purposes would
|
||||
grow its live key count roughly 30x, multiplying the cost of its own
|
||||
periodic sweep by the same factor). A month counter itself still lives
|
||||
its full ~32 days on the fallback (enforcement needs that), it just
|
||||
cannot be charted past the trailing 48 hours there — full
|
||||
usage-history retention needs the shared Redis.
|
||||
- **Retention** (Redis counter key TTL, distinct from a window's own
|
||||
length): a minute key lives 2 minutes, an hour key 48 hours, a day key
|
||||
35 days, a month key 400 days — long enough for the usage-history API's
|
||||
@@ -989,11 +999,15 @@ A rebuilt instance keeps some state and resets the rest:
|
||||
counters the old one wrote. The response cache's keys work the same way.
|
||||
- **Resets on a rebuild**: the in-process fallback counters (used whenever
|
||||
no Redis is configured, and also transparently during a Redis outage
|
||||
when `failOpen` is true — the default; see `limiter.storeIncrBy`,
|
||||
when `failOpen` is true — the default; see `limiter.storeIncrMulti`,
|
||||
`limits.go`), the limiter's and cache's own Redis-outage down-latches,
|
||||
and the model registry's discovery cache — a new instance always runs
|
||||
its own full warm-fill before serving, rather than inheriting the old
|
||||
instance's listing.
|
||||
instance's listing. This is separate from the fallback's own
|
||||
TTL-bounded retention (a window's counter still expires and resets on
|
||||
its own, per-window schedule, even between rebuilds — see [Limits and
|
||||
accounting](#limits-and-accounting)'s "Fallback" bullet): a rebuild
|
||||
loses every fallback counter at once, TTL or not.
|
||||
|
||||
Two practical implications follow:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user