mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-16 05:37:28 +00:00
fix(failover): default off, add cost guard, fix adversarial-review findings
Flip FailoverConfig.Enabled to default false. A version upgrade with no
config change must preserve prior behaviour, and failover changes which
provider serves a request. Update tests, README, and the yaegi-check
probe to opt in explicitly.
Add a mandatory cost guard. A candidate is skipped unless its input AND
output cost per token are each no greater than the primary candidate's,
resolved through the same canonical-then-bare pricing path the real
billing code uses. Unknown pricing is never treated as free: a known
primary against an unknown candidate skips, an unknown primary against a
known non-zero candidate skips, everything else that is equal is
allowed. Not configurable.
Fix four blocking findings from adversarial review of commit 5d62479.
F1: openai-type adapters mutate the request map in place to force
stream usage reporting. Reusing one map across candidates let a later
candidate read back an earlier candidate's own injection and leak a
usage-only SSE chunk to a client that never asked for it. Every
candidate now attempts against a fresh copy of the original request.
F2: a streaming candidate that fails before writing anything still
stages SSE-only response headers. Those headers now get cleared before
every candidate attempt, so they cannot survive into a later
candidate's plain JSON response.
F3: the request-health gate used the retry-shaped isTransient
classification, which treats a non-429 4xx as a successful outcome. A
provider with a dead API key was never skipped and was retried forever.
The gate now reuses failoverEligible's own classification instead.
F4: a shared request context that an earlier candidate exhausted was
attributed to whichever candidate happened to run next, opening a
healthy provider's breaker for a timeout that was not its fault. A
candidate is no longer attempted once the shared context is already
done; the loop stops and reports the last real error instead.
Also fix three should-fix findings from the same review.
F5: a failure arriving while the request-health breaker is already open
but not yet past its cooldown window is now a no-op, instead of
resetting backoff and sliding the open window forward on every such
failure.
F6: the request-health gate now records at most one outcome per
candidate per logical request, not one per same-provider retry attempt,
so retry.attempts alone can no longer open the breaker on a single
request.
F7: every candidate's cache entry is now checked before any candidate
is attempted, so a cached failover response no longer costs a round
trip to a still-broken primary first.
F10: the generic "failing over" log line is now rate-limited to once
per 30 seconds during a sustained outage, reusing the existing logGate
type. The 404 loud-log line stays unthrottled, as ruled.
F9 (optional): skip building the failover candidate pool entirely when
failover is disabled.
Correct an overstated test comment on
TestHandleChat_Failover_DiscoveryUnhealthy_StillTried: it proves alpha
is still tried, not that ordering places it second.
Document, but do not fix here, that this plugin's HTTP client sets no
per-attempt timeout, so a provider that accepts a connection and then
hangs will never trigger failover. This is a pre-existing property of
the shared HTTP client, not specific to failover.
Deferred: F8 (surfacing request-health state on the admin dashboard)
touches admin.go, which a sibling branch is also editing concurrently;
left for a follow-up to avoid a merge conflict in a file this change
does not otherwise need.
Every fix above is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
This commit is contained in:
@@ -193,7 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
|
||||
| `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
|
||||
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
|
||||
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
|
||||
| `failover` | `FailoverConfig` | `{}` (enabled) | Cross-provider failover for a bare model id more than one configured provider serves — see [Failover](#failover). `enabled: false` turns it off; a deployment with no overlapping providers behaves exactly as before regardless of this setting. |
|
||||
| `failover` | `FailoverConfig` | `{}` (disabled) | Cross-provider failover for a bare model id more than one configured provider serves — see [Failover](#failover). Omitted or `enabled: false` means no failover, byte-identical to a gateway built before this field existed. `enabled: true` opts in; a deployment with no overlapping providers behaves exactly as before regardless of this setting either way. |
|
||||
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
|
||||
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
|
||||
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
|
||||
@@ -851,10 +851,14 @@ below for the separate, cross-provider mechanism.
|
||||
|
||||
Cross-provider failover for a bare model id more than one configured
|
||||
provider serves — for example `whisper-1` configured on both `openai` and
|
||||
`openai-audio`. On by default (`failover.enabled: true` unless set
|
||||
otherwise); a deployment with no overlapping providers, or with a single
|
||||
provider, behaves exactly as before regardless of this setting, since
|
||||
there is never a second candidate to try.
|
||||
`openai-audio`. **Off by default** (`failover.enabled: false` unless set
|
||||
otherwise): a version upgrade with no config change preserves prior
|
||||
behavior exactly, since failover changes which provider serves a request
|
||||
and can shift cost attribution or mask a broken provider key behind an
|
||||
apparent success. Set `failover.enabled: true` to opt in. A deployment
|
||||
with no overlapping providers, or with a single provider, behaves
|
||||
exactly as before regardless of this setting either way, since there is
|
||||
never a second candidate to try.
|
||||
|
||||
- **Scope**: `POST /v1/chat/completions`, `POST /v1/embeddings`, and
|
||||
`POST /v1/messages` only. The image/audio endpoints
|
||||
@@ -871,9 +875,30 @@ there is never a second candidate to try.
|
||||
- **What falls through**: everything except HTTP 400 — connection
|
||||
failures, timeouts, HTTP 429/401/403/404, and any 5xx. A 400 never
|
||||
falls through: the request is malformed and every provider would reject
|
||||
it identically. A failover past a 404 is logged loudly, naming both
|
||||
providers and the model — a 404 usually means a real configuration
|
||||
mistake, not a transient outage.
|
||||
it identically. A failover past a 404 is logged loudly, always, naming
|
||||
both providers and the model — a 404 usually means a real configuration
|
||||
mistake, not a transient outage. The routine "failing over from X to Y"
|
||||
line is rate-limited to once per 30s during a sustained outage (with a
|
||||
suppressed-count note on the next line), so it cannot itself become a
|
||||
blocking synchronous write on every request; the 404 line is exempt
|
||||
from that limit.
|
||||
- **Cost guard (not optional)**: failover never routes to a candidate
|
||||
priced higher than the provider it is failing over FROM (the first
|
||||
candidate actually attempted, never the cheapest in the list), checked
|
||||
on BOTH input and output cost per token — a candidate cheaper on input
|
||||
but dearer on output is still rejected. Pricing is resolved exactly the
|
||||
way real billing resolves it: the canonical `provider/model` id in
|
||||
`pricing` first, falling back to the bare model id, so a per-provider
|
||||
`pricing` override correctly makes the identical model id cost
|
||||
different amounts on different providers. Unknown pricing is never
|
||||
treated as free: if the primary's price is known and a candidate's is
|
||||
not, the candidate is skipped (an unknown price cannot be proven
|
||||
cheaper); if the primary's price is unknown and a candidate's is known
|
||||
and non-zero, the candidate is skipped too (moving from unbilled to
|
||||
billed is itself a spend increase); both unknown, or both an explicit
|
||||
known-zero price, are treated as equal and allowed. There is no config
|
||||
flag for this — it applies unconditionally whenever failover is
|
||||
enabled.
|
||||
- **Never mid-stream**: failover only happens while nothing has reached
|
||||
the client yet. Once the first byte of a response is written, status
|
||||
and body are committed and the request is never retried elsewhere.
|
||||
@@ -882,9 +907,12 @@ there is never a second candidate to try.
|
||||
whose `/v1/models` is unhealthy is tried later, never skipped outright,
|
||||
since plenty of providers serve chat completions fine while their model
|
||||
listing is broken. A separate, in-memory, per-pod request-path health
|
||||
signal (not exposed to config) is the actual routing gate: a provider
|
||||
that has failed several consecutive live requests is skipped without
|
||||
being called until it recovers.
|
||||
signal (not yet exposed on the admin dashboard) is the actual routing
|
||||
gate: a provider is skipped without being called once it has failed
|
||||
three consecutive LOGICAL requests (same-provider retries all count as
|
||||
one outcome, not one each) — everything except HTTP 400 counts as a
|
||||
failure for this gate, including 401/403/404, so a provider with a dead
|
||||
API key is eventually routed around instead of retried forever.
|
||||
- **Accounting**: a failover chain still counts as exactly one client
|
||||
request against `requestsPerMinute`/`requestsPerDay` — never once per
|
||||
attempt. Per-provider attempt/failure counters (the admin dashboard's
|
||||
@@ -892,10 +920,25 @@ there is never a second candidate to try.
|
||||
actually made it.
|
||||
- **Caching**: a cacheable response produced by a failover provider is
|
||||
stored under that provider's own cache key, never the first candidate's.
|
||||
Every candidate's cache entry is checked before any of them is actually
|
||||
attempted, so a cached response from a failover provider is served
|
||||
without a wasted round trip to a still-broken primary.
|
||||
- **Configurable**: `failover.maxAttempts` caps how many different
|
||||
providers one request will try in total, primary included (default 3,
|
||||
maximum 10) — distinct from `retry.attempts`, which retries the SAME
|
||||
provider. `failover.enabled: false` turns the whole mechanism off.
|
||||
provider. `failover.enabled: false` (the default) turns the whole
|
||||
mechanism off.
|
||||
- **Known limitation — no per-attempt timeout**: `newAdapterHTTPClient`
|
||||
sets no client timeout and no response-header timeout; the only bound
|
||||
on an upstream call is the incoming request's own context, shared
|
||||
identically across every candidate. A provider that accepts the TCP
|
||||
connection and then simply never responds will hang the request
|
||||
indefinitely and never trigger failover — only a failure fast enough to
|
||||
leave time on that shared context (a dial failure, a fast non-2xx
|
||||
response, or the context's own deadline finally expiring) does. This is
|
||||
a pre-existing property of the plugin's HTTP client, not specific to
|
||||
failover, and applies to every route, not only this one; it is not
|
||||
addressed in this change.
|
||||
|
||||
## Caching
|
||||
|
||||
|
||||
Reference in New Issue
Block a user