mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
fix(failover): default off, add cost guard, fix adversarial-review findings
Flip FailoverConfig.Enabled to default false. A version upgrade with no
config change must preserve prior behaviour, and failover changes which
provider serves a request. Update tests, README, and the yaegi-check
probe to opt in explicitly.
Add a mandatory cost guard. A candidate is skipped unless its input AND
output cost per token are each no greater than the primary candidate's,
resolved through the same canonical-then-bare pricing path the real
billing code uses. Unknown pricing is never treated as free: a known
primary against an unknown candidate skips, an unknown primary against a
known non-zero candidate skips, everything else that is equal is
allowed. Not configurable.
Fix four blocking findings from adversarial review of commit 5d62479.
F1: openai-type adapters mutate the request map in place to force
stream usage reporting. Reusing one map across candidates let a later
candidate read back an earlier candidate's own injection and leak a
usage-only SSE chunk to a client that never asked for it. Every
candidate now attempts against a fresh copy of the original request.
F2: a streaming candidate that fails before writing anything still
stages SSE-only response headers. Those headers now get cleared before
every candidate attempt, so they cannot survive into a later
candidate's plain JSON response.
F3: the request-health gate used the retry-shaped isTransient
classification, which treats a non-429 4xx as a successful outcome. A
provider with a dead API key was never skipped and was retried forever.
The gate now reuses failoverEligible's own classification instead.
F4: a shared request context that an earlier candidate exhausted was
attributed to whichever candidate happened to run next, opening a
healthy provider's breaker for a timeout that was not its fault. A
candidate is no longer attempted once the shared context is already
done; the loop stops and reports the last real error instead.
Also fix three should-fix findings from the same review.
F5: a failure arriving while the request-health breaker is already open
but not yet past its cooldown window is now a no-op, instead of
resetting backoff and sliding the open window forward on every such
failure.
F6: the request-health gate now records at most one outcome per
candidate per logical request, not one per same-provider retry attempt,
so retry.attempts alone can no longer open the breaker on a single
request.
F7: every candidate's cache entry is now checked before any candidate
is attempted, so a cached failover response no longer costs a round
trip to a still-broken primary first.
F10: the generic "failing over" log line is now rate-limited to once
per 30 seconds during a sustained outage, reusing the existing logGate
type. The 404 loud-log line stays unthrottled, as ruled.
F9 (optional): skip building the failover candidate pool entirely when
failover is disabled.
Correct an overstated test comment on
TestHandleChat_Failover_DiscoveryUnhealthy_StillTried: it proves alpha
is still tried, not that ordering places it second.
Document, but do not fix here, that this plugin's HTTP client sets no
per-attempt timeout, so a provider that accepts a connection and then
hangs will never trigger failover. This is a pre-existing property of
the shared HTTP client, not specific to failover.
Deferred: F8 (surfacing request-health state on the admin dashboard)
touches admin.go, which a sibling branch is also editing concurrently;
left for a follow-up to avoid a merge conflict in a file this change
does not otherwise need.
Every fix above is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
This commit is contained in:
@@ -371,7 +371,12 @@ func run() error {
|
||||
}))
|
||||
defer failoverBUpstream.Close()
|
||||
|
||||
attemptAccountingOverride := `{"breaker":{"failureThreshold":2,"openDuration":"3s","maxOpenDuration":"6s"},` +
|
||||
// failover.enabled must be set explicitly here: FailoverConfig.Enabled
|
||||
// defaults to false (coordinator ruling — a version upgrade with no
|
||||
// config change must preserve prior behavior), so without this the
|
||||
// exerciseFailover probe below would find failover-a's 500 returned
|
||||
// to the client unchanged, never reaching failover-b at all.
|
||||
attemptAccountingOverride := `{"failover":{"enabled":true},"breaker":{"failureThreshold":2,"openDuration":"3s","maxOpenDuration":"6s"},` +
|
||||
`"providers":{"openai":{"type":"openai","baseUrl":"` + upstream.URL + `","apiKey":"sk-up","models":["` + testDataWantModel + `","` + builtinLookupModelID + `"]},` +
|
||||
`"brk":{"type":"openai","baseUrl":"` + brkUpstream.URL + `","apiKey":"sk-up","discovery":true,"discoveryInterval":"1ms"},"` +
|
||||
slowProviderName + `":{"type":"openai","baseUrl":"` + slowUpstream.URL + `","apiKey":"sk-up","models":["` + slowProviderModel + `"]},` +
|
||||
|
||||
Reference in New Issue
Block a user