mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-15 05:29:28 +00:00
feat(admin): add fleet statistics and admin read endpoints
Counters (always on when admin is enabled): - last seen per user and group, written asynchronously after admission, throttled to one write per scope per minute per replica - hourly provider and model attempts, failures and timeouts - failovers per provider, rejections per hour, and unpriced (402) refusals - cache hits, misses and cost saved - request counts per target and caller Opt-in counters under admin.stats (both default off): - userModel: usage per user and model - latency: latency histograms per provider and model All counters ride the existing Redis batches. A test measures three round trips per request with defaults and with every statistic on. A cache hit takes two. A model request is no longer counted for failed failover candidates. New admin endpoints: - usage/series and usage/totals - performance, with percentiles from the histograms - catalog, with the price billing actually uses and its source - consumers, with key source, grants and model access - config, with secrets redacted The overview gains features and a pricing summary. Usage gains lastSeen. usage/models gains offset and more detail fields. Reads are chunked at 1600 keys per round trip and capped at 64000 keys per request. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
@@ -349,6 +349,29 @@ func (c *responseCache) store(key string, status int, contentType string, body [
|
||||
}
|
||||
}
|
||||
|
||||
// cachedUsage extracts the prompt/completion token counts a cached
|
||||
// response body's own top-level "usage" object reports — the OpenAI-
|
||||
// compatible shape every non-stream JSON response this gateway's adapters
|
||||
// produce already carries — used only by a response-cache HIT
|
||||
// (routes_unified.go's own call site, admin.stats + cache both enabled)
|
||||
// to attribute the avoided cost (recordCacheHit, limits.go) against the
|
||||
// same numbers a MISS would have billed. A body with no "usage" object,
|
||||
// or one that fails to unmarshal, reports a zero usage{}: a hit still
|
||||
// counts (chit), it simply has nothing to value (csave stays 0 and is
|
||||
// skipped, mirroring account's own "skip a zero direction" rule).
|
||||
func cachedUsage(body []byte) usage {
|
||||
var parsed struct {
|
||||
Usage *struct {
|
||||
Prompt int64 `json:"prompt_tokens"`
|
||||
Completion int64 `json:"completion_tokens"`
|
||||
} `json:"usage"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &parsed); err != nil || parsed.Usage == nil {
|
||||
return usage{}
|
||||
}
|
||||
return usage{prompt: parsed.Usage.Prompt, completion: parsed.Usage.Completion}
|
||||
}
|
||||
|
||||
// cacheEndpointChat and cacheEndpointEmbeddings are cacheKey's endpoint
|
||||
// argument, one per cacheable route (routes_unified.go's handleChat/
|
||||
// handleEmbeddings) — keeping /v1/chat/completions and /v1/embeddings
|
||||
|
||||
Reference in New Issue
Block a user