feat(admin): add fleet statistics and admin read endpoints

Counters (always on when admin is enabled):
- last seen per user and group, written asynchronously after
  admission, throttled to one write per scope per minute per replica
- hourly provider and model attempts, failures and timeouts
- failovers per provider, rejections per hour, and unpriced (402)
  refusals
- cache hits, misses and cost saved
- request counts per target and caller

Opt-in counters under admin.stats (both default off):
- userModel: usage per user and model
- latency: latency histograms per provider and model

All counters ride the existing Redis batches. A test measures three
round trips per request with defaults and with every statistic on. A
cache hit takes two. A model request is no longer counted for failed
failover candidates.

New admin endpoints:
- usage/series and usage/totals
- performance, with percentiles from the histograms
- catalog, with the price billing actually uses and its source
- consumers, with key source, grants and model access
- config, with secrets redacted

The overview gains features and a pricing summary. Usage gains
lastSeen. usage/models gains offset and more detail fields. Reads are
chunked at 1600 keys per round trip and capped at 64000 keys per
request.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 16:40:45 +01:00
parent 8ac20d8d5e
commit 65b2f121c4
40 changed files with 7723 additions and 506 deletions
+22 -5
View File
@@ -871,16 +871,33 @@ func newLatencyHistogram() *latencyHistogram {
// first bound the observation is <= is its bucket, matching Prometheus's
// own "le" (less-than-or-equal) histogram semantics exactly.
func observeLatencyBucket(buckets []int64, overflow *int64, sum *float64, count *int64, d time.Duration) {
v := d.Seconds()
*sum += v
*sum += d.Seconds()
*count++
idx, isOverflow := latencyBucketIndex(d)
if isOverflow {
*overflow++
return
}
buckets[idx]++
}
// latencyBucketIndex returns d's index into latencyBucketBounds — the
// first bound d.Seconds() is <= to — or overflow=true, with idx equal to
// len(latencyBucketBounds), when d exceeds every configured bound. Shared
// by observeLatencyBucket (this in-process Prometheus histogram, above)
// and limits.go's accountWith/latencyCounterEntries (admin-redesign WP-A,
// admin.stats.latency's own opt-in per-bucket counter family), so the two
// can never disagree about which bucket a given duration belongs to —
// accountWith's own counter names (latencyDurationMetric/
// latencyTTFBMetric, limits.go) are built directly from this same index.
func latencyBucketIndex(d time.Duration) (idx int, overflow bool) {
v := d.Seconds()
for i, bound := range latencyBucketBounds {
if v <= bound {
buckets[i]++
return
return i, false
}
}
*overflow++
return len(latencyBucketBounds), true
}
// latencyStore is the Gateway's in-process, per-replica upstream-latency