feat(admin): add events feed, fleet-wide rejections and span rankings

GET /admin/api/events returns recent failures, newest first. It covers
rate limits, budget and unpriced-model refusals, upstream errors,
timeouts and capacity rejections. Events go into an in-process ring of
200 entries and are copied to a Redis list (llmgw:events, LPUSH plus
LTRIM to 200). The copy happens only on the error path, is throttled to
50 per second per replica, and never blocks a request. When Redis is
absent or down, the feed falls back to this replica's ring. Messages
are fixed strings or scrubbed provider errors. Fields are assigned one
at a time, because Yaegi drops values from a multi-value assignment
that contains a call.

Rejections per day are counted per scope and in total, in the same
store call that already rolls back a rejected request. /admin/api/usage
returns them as rejectionsPerDay. /admin/api/usage/models accepts span,
which sums that many buckets, and prefix, which filters model ids
before the limit applies. The overview reports the replica id, the
instance name and the warnings logged while the middleware was built.
The failover and retry fan-out warning is now logged at WARN.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 13:09:38 +01:00
parent 41bff22cb5
commit 6776981675
21 changed files with 2641 additions and 142 deletions
+63 -1
View File
@@ -21,8 +21,60 @@ func (g *Gateway) logf(format string, args ...any) {
// lines were ERROR until 2026-08-22, which put 144 of them in a healthy
// pod's boot log on a cluster with overlapping catalogs and made real
// errors harder to find. Reserve errorf for something actually broken.
//
// Also calls noteConfigWarning (F10, v0.3 dashboard task): while
// newGateway is still constructing g (collectingWarnings true), the
// formatted message is additionally collected into g.configWarnings —
// GET /admin/api/overview's Warnings field. A warnf call from
// request-handling code, after construction finished, is NOT collected —
// see noteConfigWarning's own doc comment for why.
func (g *Gateway) warnf(format string, args ...any) {
fmt.Fprintf(os.Stderr, "llmgw[%s] WARN %s\n", g.name, fmt.Sprintf(format, args...))
msg := fmt.Sprintf(format, args...)
fmt.Fprintf(os.Stderr, "llmgw[%s] WARN %s\n", g.name, msg)
g.noteConfigWarning(msg)
}
// configWarningsCap bounds how many construction-time warnings
// g.configWarnings ever holds (F10, v0.3 dashboard task) — a
// misconfigured deployment logging hundreds of warnings during
// construction must not make GET /admin/api/overview's response itself
// unbounded. configWarningsDropped counts what did not fit, so the
// dashboard can say "and N more" instead of silently truncating.
const configWarningsCap = 50
// noteConfigWarning appends msg to g.configWarnings while newGateway is
// still collecting them — collectingWarnings is set true right after g is
// constructed and false just before newGateway returns (llmgateway.go),
// so every warnf call in that window is a CONSTRUCTION-time warning,
// exactly what GET /admin/api/overview's Warnings field surfaces. Once
// collectingWarnings is false (every warnf call from request-handling
// code, for the rest of the process lifetime), this is a no-op: such a
// warning already reached the operator via the stderr line warnf itself
// wrote, and an unbounded stream of runtime warnings must never keep
// growing this slice for the life of the process.
func (g *Gateway) noteConfigWarning(msg string) {
g.warnMu.Lock()
defer g.warnMu.Unlock()
if !g.collectingWarnings {
return
}
if len(g.configWarnings) >= configWarningsCap {
g.configWarningsDropped++
return
}
g.configWarnings = append(g.configWarnings, msg)
}
// configWarningsSnapshot returns a fresh copy of g.configWarnings — never
// nil, matching GET /admin/api/overview's own "never nil" contract for
// its Warnings field — and the dropped count, safe for concurrent use
// alongside noteConfigWarning.
func (g *Gateway) configWarningsSnapshot() (warnings []string, dropped int) {
g.warnMu.Lock()
defer g.warnMu.Unlock()
out := make([]string, len(g.configWarnings))
copy(out, g.configWarnings)
return out, g.configWarningsDropped
}
// errorf writes an error-level log line to stderr. No timestamps — Traefik
@@ -47,6 +99,16 @@ func (g *Gateway) pricingWarn(msg string) {
return
}
g.logf("pricing: no price configured for model %q; cost will be recorded as 0", msg)
// F3 hook 4 (v0.3 dashboard task): msg is the bare unpriced model id
// here (warnUnknownModelFor's own "warn(model)" call, pricing.go) —
// the cap-message branch above is a process-wide meta-warning, not
// about any one model, so it is deliberately excluded from the event
// feed. Route is the fixed "pricing" label: this fires deep inside
// cost accounting, with no request-scoped route name available.
g.recordEvent(gatewayEvent{
Model: msg, Route: routePricing, Kind: eventKindUnpriced,
Message: fmt.Sprintf("no price configured for model %q; cost recorded as 0", msg),
})
}
// logAuthEvent logs one authentication attempt, at the point one of