mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-16 05:37:28 +00:00
feat(admin): add fleet statistics and admin read endpoints
Counters (always on when admin is enabled): - last seen per user and group, written asynchronously after admission, throttled to one write per scope per minute per replica - hourly provider and model attempts, failures and timeouts - failovers per provider, rejections per hour, and unpriced (402) refusals - cache hits, misses and cost saved - request counts per target and caller Opt-in counters under admin.stats (both default off): - userModel: usage per user and model - latency: latency histograms per provider and model All counters ride the existing Redis batches. A test measures three round trips per request with defaults and with every statistic on. A cache hit takes two. A model request is no longer counted for failed failover candidates. New admin endpoints: - usage/series and usage/totals - performance, with percentiles from the histograms - catalog, with the price billing actually uses and its source - consumers, with key source, grants and model access - config, with secrets redacted The overview gains features and a pricing summary. Usage gains lastSeen. usage/models gains offset and more detail fields. Reads are chunked at 1600 keys per round trip and capped at 64000 keys per request. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
+175
-76
@@ -1,92 +1,191 @@
|
||||
package traefikllmgateway
|
||||
|
||||
import "time"
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// This file restores single-key test conveniences for limiter/memoryStore
|
||||
// counter operations that no longer have any production caller (deadcode
|
||||
// audit, 2026-09 review): checkAndCount/account moved onto the batched
|
||||
// incrMulti/getMulti/incrAndGetMulti paths in the 2026-08-21/22 perf
|
||||
// reviews, which left incrBy/get — and the limiter-level helpers built on
|
||||
// them (storeIncrBy, storeGet, failPolicyIncrBy, failPolicyGet, incrCounter,
|
||||
// getCounter) — with no caller anywhere in production. They were removed
|
||||
// from limits.go and from the counterStore interface. Every helper here is
|
||||
// implemented via the SAME batched production path a single-key call would
|
||||
// have used (a one-entry counterIncr slice, or a one-key string slice), so
|
||||
// it exercises the identical fail-open/fail-closed/latch policy and EXPIRE-
|
||||
// confirmation logic the removed originals did — not a parallel
|
||||
// reimplementation. This keeps the ~130 existing test call sites (across
|
||||
// limits_test.go and admin_test.go) compiling and passing unchanged.
|
||||
// This file holds test-only helpers for limits.go functions that
|
||||
// admin-redesign WP-G's deadcode cleanup removed from production: every
|
||||
// production call site was rewritten to call the more general
|
||||
// replacement directly (accountWith, countTargetRequestsBy/
|
||||
// countTargetRequestBy, admin.go's chunkedModelSpanTotals/
|
||||
// chunkedModelSpanTotalsMulti), leaving these exact shapes reachable only
|
||||
// from tests. Moving them here — same signatures, same bodies — keeps
|
||||
// `deadcode ./...` clean (they are gone from the production build graph)
|
||||
// while `deadcode -test ./...` stays clean too (tests still call them
|
||||
// directly) and every pre-existing test keeps its original coverage of
|
||||
// the behavior these names document.
|
||||
//
|
||||
// storeIncrBy/storeGet/incrCounter below are a separate, older instance
|
||||
// of the identical pattern (counterStore's own doc comment, limits.go:
|
||||
// "counterStore had single-key incrBy/get methods until the 2026-09
|
||||
// deadcode audit ... the tests that needed a single-key call now build a
|
||||
// one-entry incrMulti batch or a one-key getMulti read instead (see
|
||||
// limits_helpers_test.go ...)"), naming this exact file as their home;
|
||||
// this pass fills that gap.
|
||||
|
||||
// incrBy is a test-only single-key counterpart to memoryStore.incrMulti,
|
||||
// restoring the pre-batching direct-increment API for memoryStore-level
|
||||
// unit tests that need a bare key with no window-derived enforceTTL floor
|
||||
// (incrMulti's own callers always supply one via newCounterIncr; a bare
|
||||
// test key has none, matching the removed production incrBy's own
|
||||
// no-floor clamp).
|
||||
// enforceTTLFor returns the minimum TTL window's own counter must stay
|
||||
// alive for to enforce correctly — see clampTTL's own doc comment
|
||||
// (limits.go) for why the floor must always win a conflict with the
|
||||
// ceiling. Production code builds all four windows' floors at once via
|
||||
// enforceTTLsFor (limits.go) instead of calling this per window; the
|
||||
// tests below pin enforceTTLsFor's four values against this switch
|
||||
// directly, and clampTTL's own tests use it to build per-window floor
|
||||
// values as clampTTL inputs.
|
||||
func enforceTTLFor(window string) time.Duration {
|
||||
switch window {
|
||||
case windowMin:
|
||||
return enforceTTLMinWindow
|
||||
case windowHour:
|
||||
return enforceTTLHourWindow
|
||||
case windowDay:
|
||||
return enforceTTLDayWindow
|
||||
case windowMonth:
|
||||
return enforceTTLMonthWindow
|
||||
default:
|
||||
panic(fmt.Sprintf("llmgateway: enforceTTLFor: unknown window %q", window))
|
||||
}
|
||||
}
|
||||
|
||||
// account records u's prompt/completion tokens and costMicros with no
|
||||
// extra counter families — accountWith(scopes, u, costMicros,
|
||||
// accountExtras{}). Every production call site now calls accountWith
|
||||
// directly; this wrapper keeps every pre-existing test's three-argument
|
||||
// call site compiling and behaving byte-for-byte the same as before
|
||||
// accountExtras existed.
|
||||
func (l *limiter) account(scopes []limitScope, u usage, costMicros int64) {
|
||||
l.accountWith(scopes, u, costMicros, accountExtras{})
|
||||
}
|
||||
|
||||
// countTargetRequests increments MULTIPLE target scopes' per-target
|
||||
// request counters with no caller identity attached —
|
||||
// countTargetRequestsBy(limits.go) with caller="". Every production call
|
||||
// site now names a caller and calls countTargetRequestsBy directly; this
|
||||
// wrapper keeps tests exercising the caller-less shape unchanged.
|
||||
func (l *limiter) countTargetRequests(scopes []limitScope) {
|
||||
l.countTargetRequestsBy("", scopes)
|
||||
}
|
||||
|
||||
// countTargetRequest is countTargetRequests for exactly one target scope.
|
||||
func (l *limiter) countTargetRequest(kind, id string) {
|
||||
l.countTargetRequests([]limitScope{{kind: kind, id: id}})
|
||||
}
|
||||
|
||||
// modelSpanTotals reads span counter buckets per id, positionally
|
||||
// (result[i] is the sum for ids[i]) — a single-metric convenience over
|
||||
// the production spanTotalsMulti reader (limits.go; P12 fix, admin
|
||||
// dashboard redesign verify round: this used to be its own separate
|
||||
// unchunked implementation, exercising nothing chunkedModelSpanTotals
|
||||
// (admin.go) actually runs in production — spanTotalsMulti is now the
|
||||
// ONE production reader both call). Kept here as a test convenience
|
||||
// purely so limits_test.go's many single-metric call sites did not all
|
||||
// need rewriting to unpack spanTotalsMulti's [][]int64 by hand.
|
||||
func (l *limiter) modelSpanTotals(ids []string, metric, window string, now time.Time, span int) ([]int64, bool) {
|
||||
out, ok := l.spanTotalsMulti(kindModel, ids, []string{metric}, window, now, span, 0)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
return out[0], true
|
||||
}
|
||||
|
||||
// modelTotals reads ONE counter per id — metric at window's CURRENT
|
||||
// bucket — the modelSpanTotals span=1 special case.
|
||||
func (l *limiter) modelTotals(ids []string, metric, window string) ([]int64, bool) {
|
||||
return l.modelSpanTotals(ids, metric, window, l.now(), 1)
|
||||
}
|
||||
|
||||
// incrBy restores memoryStore's single-key increment API removed from
|
||||
// production in the 2026-09 deadcode audit (applyIncr's own doc comment,
|
||||
// limits.go): checkAndCount/accountWith moved onto the batched incrMulti
|
||||
// in the 2026-08-21 perf review, leaving incrBy with no production
|
||||
// caller. Clamped with no enforcement floor (clampTTL(ttl, 0)) — a bare
|
||||
// key carries no (kind, id, metric, window) tuple to derive one from,
|
||||
// unlike incrMulti's own per-entry enforceTTL floor.
|
||||
func (m *memoryStore) incrBy(key string, n int64, ttl time.Duration) (int64, error) {
|
||||
return m.applyIncr(key, n, clampTTL(ttl, 0)), nil
|
||||
}
|
||||
|
||||
// storeIncrBy is a test-only single-key counterpart to storeIncrMulti,
|
||||
// letting the limiter's fail-open/fail-closed/latch tests exercise that
|
||||
// policy against one key without building a one-entry counterIncr slice at
|
||||
// every call site. Delegates to storeIncrMulti itself, so it drives the
|
||||
// exact same policy branches (nil store / latched / store error / store
|
||||
// success) a single-entry batch would.
|
||||
func (l *limiter) storeIncrBy(key string, n int64, ttl time.Duration) (v int64, ok bool) {
|
||||
vs, ok := l.storeIncrMulti([]counterIncr{{key: key, delta: n, ttl: ttl}})
|
||||
if !ok || len(vs) == 0 {
|
||||
return 0, ok
|
||||
}
|
||||
return vs[0], true
|
||||
}
|
||||
|
||||
// storeGet mirrors storeIncrBy for a read, delegating to storeGetMulti.
|
||||
func (l *limiter) storeGet(key string) (v int64, ok bool) {
|
||||
vs, ok := l.storeGetMulti([]string{key})
|
||||
if !ok || len(vs) == 0 {
|
||||
return 0, ok
|
||||
}
|
||||
return vs[0], true
|
||||
}
|
||||
|
||||
// failPolicyIncrBy applies the limiter's fail-open/fail-closed policy for
|
||||
// an incrBy the caller has decided not to attempt against the configured
|
||||
// store (it just failed, or the store-down latch is open): failOpen=true
|
||||
// counts key in the fallback instead; failOpen=false reports the
|
||||
// operation as failed. Exercised directly (not via storeIncrBy/
|
||||
// storeIncrMulti) by tests that isolate this branching from the store-call
|
||||
// decision above it.
|
||||
func (l *limiter) failPolicyIncrBy(key string, n int64, ttl time.Duration) (int64, bool) {
|
||||
if !l.failOpen {
|
||||
return 0, false
|
||||
}
|
||||
v, _ := l.fallback.incrBy(key, n, ttl)
|
||||
return v, true
|
||||
}
|
||||
|
||||
// failPolicyGet mirrors failPolicyIncrBy for a read.
|
||||
// failPolicyGet is failPolicyGetMulti's single-key special case, isolating
|
||||
// the fail-open/fail-closed branch itself (TestLimiter_FailPolicyGet) from
|
||||
// storeGet's own error/latched paths that already drive it indirectly.
|
||||
func (l *limiter) failPolicyGet(key string) (int64, bool) {
|
||||
if !l.failOpen {
|
||||
v, ok := l.failPolicyGetMulti([]string{key})
|
||||
if !ok || len(v) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
v, _ := l.fallback.get(key)
|
||||
return v, true
|
||||
return v[0], true
|
||||
}
|
||||
|
||||
// incrCounter increments the (kind, id, metric, window) counter at t by n
|
||||
// and returns its new value, together with whether the operation
|
||||
// succeeded under the limiter's fail-open/fail-closed policy — the
|
||||
// limiter-level single-key seeding helper tests use to seed counters
|
||||
// directly (admin_test.go's TestAdminUsage_MathAgainstSeededCounters and
|
||||
// others), implemented via the batched storeIncrMulti path.
|
||||
func (l *limiter) incrCounter(kind, id, metric, window string, t time.Time, n int64, ttl time.Duration) (int64, bool) {
|
||||
return l.storeIncrBy(windowKey(kind, id, metric, window, t), n, ttl)
|
||||
// failPolicyIncrBy mirrors failPolicyGet for failPolicyIncrMulti
|
||||
// (TestLimiter_FailPolicyIncrBy).
|
||||
func (l *limiter) failPolicyIncrBy(key string, delta int64, ttl time.Duration) (int64, bool) {
|
||||
v, ok := l.failPolicyIncrMulti([]counterIncr{{key: key, delta: delta, ttl: ttl}})
|
||||
if !ok || len(v) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return v[0], true
|
||||
}
|
||||
|
||||
// getCounter returns the (kind, id, metric, window) counter's value at t,
|
||||
// together with whether the read succeeded under the limiter's fail-open/
|
||||
// fail-closed policy, implemented via the batched storeGetMulti path.
|
||||
func (l *limiter) getCounter(kind, id, metric, window string, t time.Time) (int64, bool) {
|
||||
return l.storeGet(windowKey(kind, id, metric, window, t))
|
||||
// storeIncrBy is storeIncrMulti's single-entry special case: build one
|
||||
// counterIncr{key, delta, ttl} (enforceTTL left at its zero value —
|
||||
// "ceiling only, no floor", clampTTL's own doc comment) and apply the
|
||||
// identical fail-open/fail-closed/latched policy storeIncrMulti already
|
||||
// applies to a batch. No production caller needs a single-key increment
|
||||
// on its own (checkAndCount/accountWith/countTargetRequestsBy/
|
||||
// recordProviderAttempt all batch); this exists purely so
|
||||
// TestLimiter_StoreIncrBy_DirectPaths and incrCounter (below) can drive
|
||||
// and seed one counter at a time without building a one-element slice
|
||||
// literal at every call site.
|
||||
func (l *limiter) storeIncrBy(key string, delta int64, ttl time.Duration) (int64, bool) {
|
||||
v, ok := l.storeIncrMulti([]counterIncr{{key: key, delta: delta, ttl: ttl}})
|
||||
if !ok || len(v) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return v[0], true
|
||||
}
|
||||
|
||||
// storeGet is storeGetMulti's single-key special case, mirroring
|
||||
// storeIncrBy for the read side — TestLimiter_StoreGet_DirectPaths' own
|
||||
// subject.
|
||||
func (l *limiter) storeGet(key string) (int64, bool) {
|
||||
v, ok := l.storeGetMulti([]string{key})
|
||||
if !ok || len(v) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return v[0], true
|
||||
}
|
||||
|
||||
// incrCounter seeds exactly one (kind, id, metric, window) counter's
|
||||
// bucket for the instant now falls in, incrementing it by n with TTL
|
||||
// ttl — windowKey (limits.go) plus a single storeIncrBy call. This is
|
||||
// every test file's own direct counter-seeding primitive (admin_test.go,
|
||||
// limits_test.go, metrics_test.go): production code never seeds a
|
||||
// counter outside of a real accountWith/checkAndCount/
|
||||
// countTargetRequestsBy/recordProviderAttempt call, so there is nothing
|
||||
// for this to mirror there — it exists purely to put a known value at a
|
||||
// known key without a test reaching past the limiter into windowKey and
|
||||
// storeIncrBy itself at every call site. The store-failure return
|
||||
// storeIncrBy carries is intentionally discarded: every caller seeds
|
||||
// against a nil-store or fixedStore limiter where the write cannot fail,
|
||||
// and a seeding helper silently doing nothing on an unexpected failure
|
||||
// would only surface as a confusing assertion mismatch two lines later,
|
||||
// not here.
|
||||
func (l *limiter) incrCounter(kind, id, metric, window string, now time.Time, n int64, ttl time.Duration) {
|
||||
key := windowKey(kind, id, metric, window, now)
|
||||
_, _ = l.storeIncrBy(key, n, ttl)
|
||||
}
|
||||
|
||||
// getCounter reads exactly one (kind, id, metric, window) counter's
|
||||
// bucket for the instant now falls in — windowKey (limits.go) plus a
|
||||
// single storeGet call, incrCounter's read-side mirror and this whole
|
||||
// test suite's dominant assertion primitive: "did the right counter end
|
||||
// up with the right value after this request". ok is false only in the
|
||||
// fail-closed case, exactly like storeGet/storeGetMulti's own contract;
|
||||
// every real assertion above passes ok through require.True/assert
|
||||
// rather than ignoring it, the same discipline production callers
|
||||
// already apply to storeGetMulti's own ok.
|
||||
func (l *limiter) getCounter(kind, id, metric, window string, now time.Time) (int64, bool) {
|
||||
key := windowKey(kind, id, metric, window, now)
|
||||
return l.storeGet(key)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user