fix(failover): default off, add cost guard, fix adversarial-review findings

Flip FailoverConfig.Enabled to default false. A version upgrade with no
config change must preserve prior behaviour, and failover changes which
provider serves a request. Update tests, README, and the yaegi-check
probe to opt in explicitly.

Add a mandatory cost guard. A candidate is skipped unless its input AND
output cost per token are each no greater than the primary candidate's,
resolved through the same canonical-then-bare pricing path the real
billing code uses. Unknown pricing is never treated as free: a known
primary against an unknown candidate skips, an unknown primary against a
known non-zero candidate skips, everything else that is equal is
allowed. Not configurable.

Fix four blocking findings from adversarial review of commit 5d62479.

F1: openai-type adapters mutate the request map in place to force
stream usage reporting. Reusing one map across candidates let a later
candidate read back an earlier candidate's own injection and leak a
usage-only SSE chunk to a client that never asked for it. Every
candidate now attempts against a fresh copy of the original request.

F2: a streaming candidate that fails before writing anything still
stages SSE-only response headers. Those headers now get cleared before
every candidate attempt, so they cannot survive into a later
candidate's plain JSON response.

F3: the request-health gate used the retry-shaped isTransient
classification, which treats a non-429 4xx as a successful outcome. A
provider with a dead API key was never skipped and was retried forever.
The gate now reuses failoverEligible's own classification instead.

F4: a shared request context that an earlier candidate exhausted was
attributed to whichever candidate happened to run next, opening a
healthy provider's breaker for a timeout that was not its fault. A
candidate is no longer attempted once the shared context is already
done; the loop stops and reports the last real error instead.

Also fix three should-fix findings from the same review.

F5: a failure arriving while the request-health breaker is already open
but not yet past its cooldown window is now a no-op, instead of
resetting backoff and sliding the open window forward on every such
failure.

F6: the request-health gate now records at most one outcome per
candidate per logical request, not one per same-provider retry attempt,
so retry.attempts alone can no longer open the breaker on a single
request.

F7: every candidate's cache entry is now checked before any candidate
is attempted, so a cached failover response no longer costs a round
trip to a still-broken primary first.

F10: the generic "failing over" log line is now rate-limited to once
per 30 seconds during a sustained outage, reusing the existing logGate
type. The 404 loud-log line stays unthrottled, as ruled.

F9 (optional): skip building the failover candidate pool entirely when
failover is disabled.

Correct an overstated test comment on
TestHandleChat_Failover_DiscoveryUnhealthy_StillTried: it proves alpha
is still tried, not that ordering places it second.

Document, but do not fix here, that this plugin's HTTP client sets no
per-attempt timeout, so a provider that accepts a connection and then
hangs will never trigger failover. This is a pre-existing property of
the shared HTTP client, not specific to failover.

Deferred: F8 (surfacing request-health state on the admin dashboard)
touches admin.go, which a sibling branch is also editing concurrently;
left for a follow-up to avoid a merge conflict in a file this change
does not otherwise need.

Every fix above is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
This commit is contained in:
2026-08-23 01:18:58 +01:00
parent 5d624791fa
commit d45125ee07
6 changed files with 1174 additions and 126 deletions
+55 -12
View File
@@ -193,7 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
| `failover` | `FailoverConfig` | `{}` (enabled) | Cross-provider failover for a bare model id more than one configured provider serves — see [Failover](#failover). `enabled: false` turns it off; a deployment with no overlapping providers behaves exactly as before regardless of this setting. |
| `failover` | `FailoverConfig` | `{}` (disabled) | Cross-provider failover for a bare model id more than one configured provider serves — see [Failover](#failover). Omitted or `enabled: false` means no failover, byte-identical to a gateway built before this field existed. `enabled: true` opts in; a deployment with no overlapping providers behaves exactly as before regardless of this setting either way. |
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
@@ -851,10 +851,14 @@ below for the separate, cross-provider mechanism.
Cross-provider failover for a bare model id more than one configured
provider serves — for example `whisper-1` configured on both `openai` and
`openai-audio`. On by default (`failover.enabled: true` unless set
otherwise); a deployment with no overlapping providers, or with a single
provider, behaves exactly as before regardless of this setting, since
there is never a second candidate to try.
`openai-audio`. **Off by default** (`failover.enabled: false` unless set
otherwise): a version upgrade with no config change preserves prior
behavior exactly, since failover changes which provider serves a request
and can shift cost attribution or mask a broken provider key behind an
apparent success. Set `failover.enabled: true` to opt in. A deployment
with no overlapping providers, or with a single provider, behaves
exactly as before regardless of this setting either way, since there is
never a second candidate to try.
- **Scope**: `POST /v1/chat/completions`, `POST /v1/embeddings`, and
`POST /v1/messages` only. The image/audio endpoints
@@ -871,9 +875,30 @@ there is never a second candidate to try.
- **What falls through**: everything except HTTP 400 — connection
failures, timeouts, HTTP 429/401/403/404, and any 5xx. A 400 never
falls through: the request is malformed and every provider would reject
it identically. A failover past a 404 is logged loudly, naming both
providers and the model — a 404 usually means a real configuration
mistake, not a transient outage.
it identically. A failover past a 404 is logged loudly, always, naming
both providers and the model — a 404 usually means a real configuration
mistake, not a transient outage. The routine "failing over from X to Y"
line is rate-limited to once per 30s during a sustained outage (with a
suppressed-count note on the next line), so it cannot itself become a
blocking synchronous write on every request; the 404 line is exempt
from that limit.
- **Cost guard (not optional)**: failover never routes to a candidate
priced higher than the provider it is failing over FROM (the first
candidate actually attempted, never the cheapest in the list), checked
on BOTH input and output cost per token — a candidate cheaper on input
but dearer on output is still rejected. Pricing is resolved exactly the
way real billing resolves it: the canonical `provider/model` id in
`pricing` first, falling back to the bare model id, so a per-provider
`pricing` override correctly makes the identical model id cost
different amounts on different providers. Unknown pricing is never
treated as free: if the primary's price is known and a candidate's is
not, the candidate is skipped (an unknown price cannot be proven
cheaper); if the primary's price is unknown and a candidate's is known
and non-zero, the candidate is skipped too (moving from unbilled to
billed is itself a spend increase); both unknown, or both an explicit
known-zero price, are treated as equal and allowed. There is no config
flag for this — it applies unconditionally whenever failover is
enabled.
- **Never mid-stream**: failover only happens while nothing has reached
the client yet. Once the first byte of a response is written, status
and body are committed and the request is never retried elsewhere.
@@ -882,9 +907,12 @@ there is never a second candidate to try.
whose `/v1/models` is unhealthy is tried later, never skipped outright,
since plenty of providers serve chat completions fine while their model
listing is broken. A separate, in-memory, per-pod request-path health
signal (not exposed to config) is the actual routing gate: a provider
that has failed several consecutive live requests is skipped without
being called until it recovers.
signal (not yet exposed on the admin dashboard) is the actual routing
gate: a provider is skipped without being called once it has failed
three consecutive LOGICAL requests (same-provider retries all count as
one outcome, not one each) — everything except HTTP 400 counts as a
failure for this gate, including 401/403/404, so a provider with a dead
API key is eventually routed around instead of retried forever.
- **Accounting**: a failover chain still counts as exactly one client
request against `requestsPerMinute`/`requestsPerDay` — never once per
attempt. Per-provider attempt/failure counters (the admin dashboard's
@@ -892,10 +920,25 @@ there is never a second candidate to try.
actually made it.
- **Caching**: a cacheable response produced by a failover provider is
stored under that provider's own cache key, never the first candidate's.
Every candidate's cache entry is checked before any of them is actually
attempted, so a cached response from a failover provider is served
without a wasted round trip to a still-broken primary.
- **Configurable**: `failover.maxAttempts` caps how many different
providers one request will try in total, primary included (default 3,
maximum 10) — distinct from `retry.attempts`, which retries the SAME
provider. `failover.enabled: false` turns the whole mechanism off.
provider. `failover.enabled: false` (the default) turns the whole
mechanism off.
- **Known limitation — no per-attempt timeout**: `newAdapterHTTPClient`
sets no client timeout and no response-header timeout; the only bound
on an upstream call is the incoming request's own context, shared
identically across every candidate. A provider that accepts the TCP
connection and then simply never responds will hang the request
indefinitely and never trigger failover — only a failure fast enough to
leave time on that shared context (a dial failure, a fast non-2xx
response, or the context's own deadline finally expiring) does. This is
a pre-existing property of the plugin's HTTP client, not specific to
failover, and applies to every route, not only this one; it is not
addressed in this change.
## Caching