mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-13 04:58:50 +00:00
fix(providers): deadline failures, cheaper polling, async writes
Review round (v0.22): 1 must-fix + 5 should-fix. This covers the three backend items. MUST-1's WCAG fix lives in the webui commit; this is the three provider/ admin-side SHOULD items. SHOULD-1 (timeout blindness): a hung upstream previously read as 100% healthy — isTransient (retry.go) deliberately excludes context.DeadlineExceeded from what it retries (retrying past the gateway's own deadline is pointless), and recordProviderAttempt reused that same classification unforked, so a timed-out attempt recorded no failure. Fixed narrowly in recordProviderAttempt only, per the ruling: isTransient itself stays untouched and shared. A new isDeadlineExceeded adds exactly one extra condition — a context.Canceled error (the client walked away) still counts as an attempt only, from both functions alike. The sentinel match (matchesSentinel) is a hand-rolled errors.Unwrap walk, not errors.Is: providerHTTPError's own doc comment already documents that errors.As panics under Yaegi checking whether an interpreted pointer type implements error; errors.Is walks the identical internal machinery and carries the same risk, so this sidesteps it with a bare `err.(sentinelUnwrapper)` comma-ok assertion instead — plain interface satisfaction, not reflection over a possibly-interpreted type. SHOULD-2 (poll fan-out): a (provider, model) scope's minute-window counters fed nothing but a badge's title text on the Providers tab, so a dashboard with N configured models paid N wasted minute-window reads on every 5s poll. providerCounterKeys/providerUsage now read a kindProviderModel scope's day window only (2 keys, down from 4); a kindProvider scope is unaffected (still 4 — one scope per provider, not per model, so the read amplification never applied there). adminModelRateView drops AttemptsMinute/FailuresMinute entirely, not just unpopulated fields — the webui side (separate commit) falls back to a day-window-only title for a per-model badge. SHOULD-5 (TTFB): recordProviderAttempt's store write is telemetry-only — nothing in the request path gates on it — so it now runs via a new limiter.spawn dependency-injection field (mirrors nowFn/waitFn's existing pattern), defaulting to a real goroutine in production and overridden to run synchronously in every test that asserts on the resulting counter immediately afterward. This removes the synchronous Redis round trip this counter family added to the streaming TTFB path entirely. Loss on process shutdown is accepted (telemetry only).
This commit is contained in:
@@ -308,14 +308,18 @@ type adminProviderView struct {
|
||||
|
||||
// adminModelRateView is one upstream model's current-window attempt/
|
||||
// failure counters within its provider — adminProviderView.ModelRates'
|
||||
// value type (Feature A, v0.22). Field shape and meaning are identical to
|
||||
// adminProviderView's own Attempts*/Failures* fields, just scoped to one
|
||||
// (provider, model) pair instead of the whole provider.
|
||||
// value type (Feature A, v0.22). Day window only, no AttemptsMinute/
|
||||
// FailuresMinute (SHOULD-2, v0.22 review round — removed, not merely
|
||||
// left unpopulated): a dashboard with N configured models was paying N
|
||||
// wasted minute-window store reads on every 5s poll for a figure that
|
||||
// only ever fed a badge's title text, never its own displayed tier — see
|
||||
// limits.go's providerCounterKeys/providerScopeKeyCount for the read-side
|
||||
// half of this same ruling. The Providers tab's per-model badge falls
|
||||
// back to a day-window-only title when these are absent; see webui's
|
||||
// lib/provider-rate.ts (dayRateTitle) and ProviderRateBadge.vue.
|
||||
type adminModelRateView struct {
|
||||
AttemptsDay int64 `json:"attemptsDay"`
|
||||
FailuresDay int64 `json:"failuresDay"`
|
||||
AttemptsMinute int64 `json:"attemptsMinute"`
|
||||
FailuresMinute int64 `json:"failuresMinute"`
|
||||
AttemptsDay int64 `json:"attemptsDay"`
|
||||
FailuresDay int64 `json:"failuresDay"`
|
||||
}
|
||||
|
||||
// adminRedisView is the redis status line in GET /admin/api/overview.
|
||||
@@ -423,10 +427,8 @@ func (g *Gateway) buildAdminOverview() adminOverviewResponse {
|
||||
mc := modelCounters[mi]
|
||||
mi++
|
||||
modelRates[model] = adminModelRateView{
|
||||
AttemptsDay: mc.attemptsDay,
|
||||
FailuresDay: mc.failuresDay,
|
||||
AttemptsMinute: mc.attemptsMinute,
|
||||
FailuresMinute: mc.failuresMinute,
|
||||
AttemptsDay: mc.attemptsDay,
|
||||
FailuresDay: mc.failuresDay,
|
||||
}
|
||||
}
|
||||
providers[i] = adminProviderView{
|
||||
|
||||
Reference in New Issue
Block a user