feat(failover): add cross-provider failover for overlapping models

Add failover.go with the full candidate pipeline. resolveWithCandidates resolves a request exactly like resolve does, then adds every other provider sharing the identical bare model id. orderedFailoverCandidates applies a hard skip for request-path-unhealthy providers, deprioritizes discovery-unhealthy providers without dropping them, fails open when every candidate reads unhealthy, and caps the list at maxAttempts. Add requestHealthTracker, a separate per-pod request-path health signal fed by the same isTransient/isDeadlineExceeded classification recordProviderAttempt already uses. It never widens the existing discovery circuit breaker, per that breaker's own documented scope. Rewrite runMeteredCall (routes_unified.go) to loop over candidates instead of resolving once. Each attempt rebinds the attempt recorder to its own provider, recomputes the cache key before the response is cached, and re-injects the alias echo key an adapter may have deleted on a prior failed attempt. Failover never runs once any byte has reached the client, since statusTrackingWriter already tracks that state. A failure falls through on everything except HTTP 400, matching the brief exactly. A 404 failover logs loudly, naming both providers and the model, since it usually signals a real configuration mistake. Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A single-provider deployment, or one with no overlapping bare model ids, behaves exactly as before regardless of this setting. Add a yaegi-check probe (tools/yaegi-check/main.go) driving two providers under the real interpreter, one failing and one serving, proving the candidate loop runs correctly interpreted. Update README.md with a Failover section and correct the Retry section's stale claim that no cross-provider failover exists. Media routes (images, audio speech, audio transcriptions) do not get failover in this change. Each has its own request-shape handling that needs its own scoped follow-up.
2026-10-12 04:54:45 +00:00 · 2026-08-23 00:32:12 +01:00
parent 208084ac83
commit 5d624791fa
6 changed files with 1866 additions and 119 deletions
@@ -193,6 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
 | `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
 | `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
 | `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
+| `failover` | `FailoverConfig` | `{}` (enabled) | Cross-provider failover for a bare model id more than one configured provider serves — see [Failover](#failover). `enabled: false` turns it off; a deployment with no overlapping providers behaves exactly as before regardless of this setting. |
 | `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
 | `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
 | `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
@@ -807,10 +808,10 @@ released, so this never becomes a ceiling on total concurrent requests.

 ## Retry

-Same-provider retry for a transient upstream failure — no cross-provider
-failover. Off by default (`retry.enabled: false`); every config that
-predates this field keeps making exactly one upstream attempt per
-request.
+Same-provider retry for a transient upstream failure. Off by default
+(`retry.enabled: false`); every config that predates this field keeps
+making exactly one upstream attempt per request. See [Failover](#failover)
+below for the separate, cross-provider mechanism.

 - **Scope**: every adapter upstream call — chat, embeddings, model
  listing, and the three media endpoints (`images/generations`,
@@ -846,6 +847,56 @@ request.
  is never double-counted. Request counters still increment once per
  client request, unchanged.

+## Failover
+
+Cross-provider failover for a bare model id more than one configured
+provider serves — for example `whisper-1` configured on both `openai` and
+`openai-audio`. On by default (`failover.enabled: true` unless set
+otherwise); a deployment with no overlapping providers, or with a single
+provider, behaves exactly as before regardless of this setting, since
+there is never a second candidate to try.
+
+- **Scope**: `POST /v1/chat/completions`, `POST /v1/embeddings`, and
+  `POST /v1/messages` only. The image/audio endpoints
+  (`images/generations`, `audio/speech`, `audio/transcriptions`) are not
+  wired into failover yet — each has its own request-shape handling
+  (multipart rebuild, binary/streaming responses) that needs its own
+  scoped follow-up.
+- **Eligible candidates**: a bare model id resolved via the collision
+  winner (the same mechanism `GET /v1/models` documents for overlapping
+  catalogs) offers every OTHER configured provider that also serves that
+  id and that the caller's group is authorized to use. A `provider/model`
+  request, or an alias whose target is `provider/model`, is an explicit,
+  exact choice and is never eligible for substitution.
+- **What falls through**: everything except HTTP 400 — connection
+  failures, timeouts, HTTP 429/401/403/404, and any 5xx. A 400 never
+  falls through: the request is malformed and every provider would reject
+  it identically. A failover past a 404 is logged loudly, naming both
+  providers and the model — a 404 usually means a real configuration
+  mistake, not a transient outage.
+- **Never mid-stream**: failover only happens while nothing has reached
+  the client yet. Once the first byte of a response is written, status
+  and body are committed and the request is never retried elsewhere.
+- **Health signals, kept separate**: the discovery-endpoint health
+  surfaced on the admin dashboard is an ordering hint only — a provider
+  whose `/v1/models` is unhealthy is tried later, never skipped outright,
+  since plenty of providers serve chat completions fine while their model
+  listing is broken. A separate, in-memory, per-pod request-path health
+  signal (not exposed to config) is the actual routing gate: a provider
+  that has failed several consecutive live requests is skipped without
+  being called until it recovers.
+- **Accounting**: a failover chain still counts as exactly one client
+  request against `requestsPerMinute`/`requestsPerDay` — never once per
+  attempt. Per-provider attempt/failure counters (the admin dashboard's
+  success-rate badge) attribute every attempt to the provider that
+  actually made it.
+- **Caching**: a cacheable response produced by a failover provider is
+  stored under that provider's own cache key, never the first candidate's.
+- **Configurable**: `failover.maxAttempts` caps how many different
+  providers one request will try in total, primary included (default 3,
+  maximum 10) — distinct from `retry.attempts`, which retries the SAME
+  provider. `failover.enabled: false` turns the whole mechanism off.
+
 ## Caching

 An opt-in, Redis-backed response cache for the unified, non-streaming
@@ -0,0 +1,487 @@
+package traefikllmgateway
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net/http"
+	"sync"
+	"time"
+)
+
+// This file implements feat/failover: cross-provider failover for a bare
+// model id more than one configured provider serves (registry.go's
+// bareWinner collision case — production logs show whisper-1, tts-1,
+// tts-1-hd, text-embedding-3-small, text-embedding-3-large, and
+// text-embedding-ada-002 served by two or more of openai/openai-audio/
+// copilot today). Before this feature, registry.go's resolve returns
+// exactly one provider adapter and retry.go's retryPolicy only ever
+// retries the SAME provider — so a down provider fails every request for
+// its models even when another configured provider serves the identical
+// model. This file turns that already-existing redundancy into a hot
+// standby, hooked into routes_unified.go's runMeteredCall, the ONE shared
+// metered-call path both /v1/chat/completions and /v1/messages already go
+// through.
+//
+// SCOPE: routes_media.go's three media routes (images/generations,
+// audio/speech, audio/transcriptions) are deliberately NOT wired into
+// failover in this change, even though two of the brief's own named
+// overlapping models (whisper-1, tts-1/tts-1-hd) are media-route
+// traffic. Each of those three handlers has its own request-shape
+// preprocessing (multipart rebuild for transcriptions, a JSON
+// re-marshal for speech, a binary/streaming response for speech) that
+// would need its own careful "nothing written yet" analysis and its own
+// test coverage, rather than reusing runMeteredCall's shared shape — the
+// brief's own hook-point instruction ("do not fork a second
+// implementation") is about not duplicating the chat/messages pipeline,
+// not about extending every possible call site in one pass. Left as a
+// scoped follow-up; see this branch's own report for the explicit
+// call-out.
+
+// FailoverConfig configures feat/failover. Enabled is a *bool, mirroring
+// ProviderConfig.Passthrough's own nil-means-default-true convention
+// (providers.go): nil or true (the default) turns an overlapping
+// multi-provider deployment's already-existing redundancy into a hot
+// standby with NO config change — the entire point of this feature (see
+// this file's own package doc and the brief's "Why" section). Set false
+// to opt out entirely. A single-provider deployment, or one where no two
+// providers ever serve the identical bare model id, behaves exactly as
+// before regardless of this setting: there is never more than one
+// candidate to try (orderedFailoverCandidates' own doc comment).
+type FailoverConfig struct {
+	Enabled *bool `json:"enabled,omitempty"`
+	// MaxAttempts caps how many DIFFERENT PROVIDERS runMeteredCall tries
+	// in total for one logical request, the primary included — distinct
+	// from RetryConfig.Attempts (retry.go), which retries the SAME
+	// provider. 0 (the default) uses defaultFailoverMaxAttempts. A value
+	// outside 1..maxFailoverMaxAttempts is a construction error.
+	MaxAttempts int `json:"maxAttempts,omitempty"`
+}
+
+// defaultFailoverMaxAttempts is applied when FailoverConfig.MaxAttempts is
+// left at 0. maxFailoverMaxAttempts bounds an explicit override, the same
+// "cheap sanity ceiling on an operator value" precedent
+// maxBreakerFailureThreshold/maxRetryAttempts already set (registry.go/
+// retry.go).
+const (
+	defaultFailoverMaxAttempts = 3
+	maxFailoverMaxAttempts     = 10
+)
+
+// failoverConfig is FailoverConfig, validated and defaulted once at
+// construction (newGateway, llmgateway.go) — mirrors breakerConfig/
+// retryPolicy's own resolved-value-not-raw-config shape (registry.go/
+// retry.go).
+type failoverConfig struct {
+	maxAttempts int
+	enabled     bool
+}
+
+// validateFailoverConfig validates fc and returns the resolved
+// failoverConfig newGateway attaches to the Gateway.
+func validateFailoverConfig(fc FailoverConfig) (failoverConfig, error) {
+	enabled := true
+	if fc.Enabled != nil {
+		enabled = *fc.Enabled
+	}
+
+	maxAttempts := fc.MaxAttempts
+	if maxAttempts == 0 {
+		maxAttempts = defaultFailoverMaxAttempts
+	}
+	if maxAttempts < 1 || maxAttempts > maxFailoverMaxAttempts {
+		return failoverConfig{}, fmt.Errorf("llmgateway: failover.maxAttempts must be between 1 and %d, got %d", maxFailoverMaxAttempts, maxAttempts)
+	}
+
+	return failoverConfig{enabled: enabled, maxAttempts: maxAttempts}, nil
+}
+
+// resolveCandidate is one provider that can serve a client-requested model
+// id — modelRegistry.resolve's own (adapter, upstreamModel, canonical)
+// triple, tagged with the provider's own name so a failover attempt can
+// rebind its cache key, attempt recorder, and error log to the ACTUAL
+// provider that served or failed the request, never the first one resolve
+// happened to pick.
+type resolveCandidate struct {
+	adapter       providerAdapter
+	providerName  string
+	upstreamModel string
+	canonical     string
+}
+
+// resolveWithCandidates resolves id exactly as modelRegistry.resolve
+// (registry.go) does — identical winner, identical error — and
+// additionally reports every OTHER configured provider that also serves
+// the same bare upstream model id and remains authorized for grp, as
+// feat/failover's candidate pool.
+//
+// primary/err are byte-identical to what resolve(id, grp) alone would
+// return: this method calls resolve internally and never second-guesses
+// its winner or its error — feat/failover only ADDS candidates after an
+// already-successful resolution, it never changes which model or error a
+// request resolves to (in particular, authorization denial/unknown-model
+// behavior for the primary winner is completely unchanged by this
+// branch). extra is nil whenever there is nothing to add:
+//
+//   - id resolved via an explicit "provider/model" prefix, or an alias
+//     whose target is itself provider-prefixed: the requester's own
+//     explicit provider choice is never a candidate for pool
+//     substitution — ruling, this feature's own design: "provider/model"
+//     addressing is exact, not eligible for failover.
+//   - id resolved via a bare-id collision winner (bareWinner), or an
+//     alias whose target is itself a bare-id collision winner: extra is
+//     every OTHER configured provider whose known model set also
+//     contains that identical bare id string, in
+//     modelRegistry.providerNames (sorted) order, and authorized for grp
+//     on that provider.
+func (m *modelRegistry) resolveWithCandidates(id string, grp *group) (resolveCandidate, []resolveCandidate, error) {
+	adapter, upstreamModel, canonical, err := m.resolve(id, grp)
+	if err != nil {
+		return resolveCandidate{}, nil, err
+	}
+	primary := resolveCandidate{adapter: adapter, providerName: adapter.name(), upstreamModel: upstreamModel, canonical: canonical}
+
+	bareID := id
+	if target, isAlias := m.aliases[id]; isAlias {
+		if _, _, ok := m.splitConfiguredProvider(target); ok {
+			return primary, nil, nil
+		}
+		bareID = target
+	} else if _, _, ok := m.splitConfiguredProvider(id); ok {
+		return primary, nil, nil
+	}
+
+	return primary, m.failoverCandidates(bareID, primary.providerName, grp), nil
+}
+
+// failoverCandidates returns every configured provider other than
+// primaryProvider whose known model set contains bareID and that grp is
+// authorized to use, in modelRegistry.providerNames (sorted) order — the
+// raw candidate pool orderedFailoverCandidates (below) narrows and
+// reorders further by health.
+//
+// Only grp.allowsProvider is re-checked per candidate: grp.allowsModel is
+// already known true from the primary winner's own successful resolve
+// call, and — for the identical bare id string every candidate here
+// shares by construction — allowsModel's result does not vary by
+// provider (auth.go's allowsModel takes no provider argument), so
+// re-checking it per candidate would be redundant work, not a
+// correctness gap.
+func (m *modelRegistry) failoverCandidates(bareID, primaryProvider string, grp *group) []resolveCandidate {
+	var extra []resolveCandidate
+	for _, name := range m.providerNames {
+		if name == primaryProvider {
+			continue
+		}
+		if !m.states[name].hasModel(bareID) || !grp.allowsProvider(name) {
+			continue
+		}
+		// canonical assigned to a local before the struct literal, matching
+		// resolveAgainst's own documented Yaegi precedent (registry.go): a
+		// defensive, cheap precaution against the same class of interpreter
+		// trap, not a proven necessity for a struct-literal field (only a
+		// multi-value RETURN tuple is the proven-bad shape there).
+		canonical := name + "/" + bareID
+		extra = append(extra, resolveCandidate{
+			adapter:       m.adapters[name],
+			providerName:  name,
+			upstreamModel: bareID,
+			canonical:     canonical,
+		})
+	}
+	return extra
+}
+
+// orderedFailoverCandidates builds runMeteredCall's actual try list from
+// resolveWithCandidates' own (primary, extra) split.
+//
+// Failover is a no-op (returns exactly [primary]) whenever it is disabled
+// (g.failover.enabled false) or there is nothing to add (len(extra) ==
+// 0) — the second case is what makes "single-provider deployment behaves
+// exactly as before" hold unconditionally, regardless of the enabled
+// switch: there is never more than one candidate to try when no other
+// configured provider shares this bare model id.
+//
+// Otherwise, two health signals apply, deliberately kept separate
+// (modelRegistry.discoveryHealthy's own doc comment, registry.go, is
+// binding on this):
+//
+//   - REQUEST-PATH health (g.failoverHealth, this file) is a HARD SKIP
+//     gate: a candidate it reports unhealthy is excluded outright,
+//     never attempted. If that filtering would remove every candidate
+//     (this local, per-pod signal reporting the whole fleet down
+//     simultaneously is more likely stale than true), it is discarded
+//     and every original candidate is tried anyway — failing open
+//     against the gateway's own imperfect signal rather than refusing
+//     to route a request at all.
+//   - DISCOVERY health (g.registry.discoveryHealthy) is an ORDERING HINT
+//     ONLY, exactly as its own doc comment requires: a discovery-
+//     unhealthy candidate is moved after every discovery-healthy/
+//     half-open one (a stable partition, not a full sort — relative
+//     order within each bucket is preserved), never dropped.
+//
+// Finally the list is capped at g.failover.maxAttempts entries.
+func (g *Gateway) orderedFailoverCandidates(primary resolveCandidate, extra []resolveCandidate) []resolveCandidate {
+	if !g.failover.enabled || len(extra) == 0 {
+		return []resolveCandidate{primary}
+	}
+
+	all := make([]resolveCandidate, 0, len(extra)+1)
+	all = append(all, primary)
+	all = append(all, extra...)
+
+	healthy := make([]resolveCandidate, 0, len(all))
+	for _, c := range all {
+		if g.failoverHealth.healthy(c.providerName) {
+			healthy = append(healthy, c)
+		}
+	}
+	if len(healthy) == 0 {
+		healthy = all
+	}
+
+	ordered := make([]resolveCandidate, 0, len(healthy))
+	var deprioritized []resolveCandidate
+	for _, c := range healthy {
+		if g.registry.discoveryHealthy(c.providerName) {
+			ordered = append(ordered, c)
+		} else {
+			deprioritized = append(deprioritized, c)
+		}
+	}
+	ordered = append(ordered, deprioritized...)
+
+	if max := g.failover.maxAttempts; max > 0 && len(ordered) > max {
+		ordered = ordered[:max]
+	}
+	return ordered
+}
+
+// failoverEligible reports whether callErr — runMeteredCall's own
+// adapterCall return, evaluated only once nothing has reached the client
+// yet (the hard constraint this whole feature is built around: once an
+// adapter has written anything, status and body are committed and can
+// never be retried elsewhere) — should try the next candidate instead of
+// answering the client with callErr.
+//
+// Ruling (brief, "Which failures fall through" — decided, not
+// relitigated here): every failure except HTTP 400 falls through. 5xx,
+// connection failures, timeouts, 429, 401, 403, and 404 all do; 400
+// never does, because the request itself is malformed and every provider
+// would reject it identically, so retrying just multiplies latency and
+// burns budget.
+//
+// Two failure shapes are excluded for the SAME "retrying changes
+// nothing" reasoning 400 itself is excluded for, even though neither is
+// literally an HTTP 400: context.Canceled (the client walked away —
+// failing over would only waste an attempt against a connection nobody
+// is waiting on) and errRequestBuildFailed (upstreamBytes, providers.go
+// — the identical malformed request shape reaches every candidate the
+// same way, differing only in which operator-configured base URL it is
+// built against). A *translateError or *responseTranslationError is this
+// gateway's OWN translation code failing on an already-decoded request —
+// not a signal about the PROVIDER's availability at all — so neither is
+// eligible either.
+func failoverEligible(callErr error) bool {
+	if callErr == nil {
+		return false
+	}
+	if errors.Is(callErr, context.Canceled) || errors.Is(callErr, errRequestBuildFailed) {
+		return false
+	}
+	// Plain type assertions, not errors.As — matching this package's
+	// established Yaegi-safe convention for these exact two pointer error
+	// types (providerHTTPError's own doc comment, providers.go;
+	// handleAdapterErrorEnvelope, routes_unified.go, does the identical
+	// assertion for the identical reason).
+	if perr, ok := callErr.(*providerHTTPError); ok {
+		return perr.status != http.StatusBadRequest
+	}
+	if _, ok := callErr.(*translateError); ok {
+		return false
+	}
+	if _, ok := callErr.(*responseTranslationError); ok {
+		return false
+	}
+	return true
+}
+
+// isProviderNotFoundError reports whether err is a *providerHTTPError
+// carrying HTTP 404 — routes_unified.go's runMeteredCall uses this to
+// decide when a failover needs its own loud, non-suppressed log line
+// (brief: "Failover after a 404 must log loudly ... a 404 usually means
+// a real configuration mistake, and silently succeeding elsewhere would
+// hide it").
+func isProviderNotFoundError(err error) bool {
+	perr, ok := err.(*providerHTTPError)
+	return ok && perr.status == http.StatusNotFound
+}
+
+// requestBreakerFailureThreshold/requestBreakerOpenDuration/
+// requestBreakerMaxOpenDuration size requestHealthTracker's own breaker —
+// deliberately separate constants from registry.go's discovery breaker
+// (defaultBreakerFailureThreshold/defaultBreakerOpenDuration/
+// defaultBreakerMaxOpenDuration): a request-path outage needs to be
+// noticed and routed around within seconds, not the discovery breaker's
+// minutes-to-hours cadence, and modelRegistry.discoveryHealthy's own doc
+// comment forbids widening that breaker to cover this outcome stream at
+// all. Fixed constants, not configurable — unlike FailoverConfig above,
+// the brief only asks for max-attempts and an on/off switch to be
+// operator-configurable; this breaker's thresholds are an internal
+// implementation detail of the routing-gate signal, not a documented
+// tuning surface.
+const (
+	requestBreakerFailureThreshold = 3
+	requestBreakerOpenDuration     = 30 * time.Second
+	requestBreakerMaxOpenDuration  = 5 * time.Minute
+)
+
+// requestHealthState is one provider's REQUEST-PATH health (feat/
+// failover), mu-guarded — mirrors providerState's per-provider,
+// individually-mutexed shape (registry.go) at a much smaller scale: this
+// tracks exactly one outcome stream (request-path attempts), never
+// discovery. The zero value is closed/healthy, so a provider that never
+// fails never touches anything below open/backoff/openUntil.
+//
+// Field order below is fieldalignment-verified (golangci-lint's govet
+// enable-all, run with -fix against a scratch copy to derive the exact
+// zero-waste sequence, then hand-applied here so this doc comment
+// survives — the same convention providerState's own doc comment
+// explains, registry.go).
+type requestHealthState struct {
+	openUntil           time.Time
+	backoff             time.Duration
+	consecutiveFailures int
+	mu                  sync.Mutex
+	open                bool
+}
+
+// requestHealthTracker is feat/failover's own per-pod, in-memory,
+// request-path health signal — deliberately separate from registry.go's
+// discovery circuit breaker (modelRegistry.discoveryHealthy's own doc
+// comment forbids widening that one to cover request-path traffic). Fed
+// by the SAME failure classification limiter.recordProviderAttempt uses
+// (isTransient || isDeadlineExceeded — limits.go/retry.go), via record's
+// own callers in routes_unified.go, but keeps its own independent state
+// per provider name.
+//
+// Every method is nil-receiver-safe (both report "healthy"/no-op on a
+// nil *requestHealthTracker), matching this package's own nil-means-
+// disabled convention for an optional subsystem (responseCache's own doc
+// comment, llmgateway.go's Gateway.cache field) — a Gateway assembled
+// directly in a test without going through newGateway, or any future
+// caller that does the same, degrades to "failover never skips a
+// candidate for request health" rather than a nil-pointer panic.
+type requestHealthTracker struct {
+	states map[string]*requestHealthState
+	nowFn  func() time.Time
+	mu     sync.Mutex
+}
+
+// newRequestHealthTracker returns an empty tracker — every provider name
+// reads healthy until its first recorded failure.
+func newRequestHealthTracker() *requestHealthTracker {
+	return &requestHealthTracker{states: make(map[string]*requestHealthState), nowFn: time.Now}
+}
+
+// stateFor returns provider's requestHealthState, creating it on first
+// use.
+func (t *requestHealthTracker) stateFor(provider string) *requestHealthState {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+	st, ok := t.states[provider]
+	if !ok {
+		st = &requestHealthState{}
+		t.states[provider] = st
+	}
+	return st
+}
+
+// record accounts one upstream request-path attempt's outcome for
+// provider: success=false must be EXACTLY isTransient(resp, err) ||
+// isDeadlineExceeded(err) — the identical classification
+// limiter.recordProviderAttempt applies (limits.go/retry.go) — never a
+// wider one; routes_unified.go's attemptRecorder closure computes it
+// once and feeds both.
+//
+// requestBreakerFailureThreshold consecutive failures opens the breaker
+// for requestBreakerOpenDuration. While open, healthy (below) reports
+// unhealthy until that window passes; once it has, the NEXT record call
+// for this provider — success or failure — is what healthy already
+// exposed as a post-cooldown probe attempt (probed, below): success
+// closes the breaker outright, failure re-opens it with a doubled
+// backoff (capped at requestBreakerMaxOpenDuration). This is the same
+// probe-then-escalate-or-close shape providerState's discovery breaker
+// uses (registry.go's recordHealthLocked/openBreakerLocked),
+// reimplemented independently rather than shared — the two track
+// different outcome streams, and discoveryHealthy's own doc comment
+// forbids widening it to cover this one.
+//
+// Deliberately no single-flight guard on the probe window (unlike
+// registry.go's tryBeginRefresh inFlight flag for its own, much lower-
+// volume, background-goroutine probe): this signal sits on the live
+// request path, where many concurrent requests can observe the same
+// post-cooldown window at once. Serializing them would need its own
+// lock-and-CAS dance for a benefit — one broken provider getting at most
+// one real probe instead of a short burst of them right as its backoff
+// expires — this per-pod, best-effort signal does not need to pay for.
+func (t *requestHealthTracker) record(provider string, success bool) {
+	if t == nil || provider == "" {
+		return
+	}
+	st := t.stateFor(provider)
+	st.mu.Lock()
+	defer st.mu.Unlock()
+	now := t.nowFn()
+
+	probed := st.open && !now.Before(st.openUntil)
+
+	if success {
+		st.open = false
+		st.consecutiveFailures = 0
+		st.backoff = 0
+		st.openUntil = time.Time{}
+		return
+	}
+
+	if probed {
+		if st.backoff <= 0 {
+			st.backoff = requestBreakerOpenDuration
+		} else {
+			st.backoff *= 2
+		}
+		if st.backoff > requestBreakerMaxOpenDuration {
+			st.backoff = requestBreakerMaxOpenDuration
+		}
+		st.openUntil = now.Add(st.backoff)
+		return
+	}
+
+	st.consecutiveFailures++
+	if st.consecutiveFailures >= requestBreakerFailureThreshold {
+		st.open = true
+		st.backoff = requestBreakerOpenDuration
+		st.openUntil = now.Add(st.backoff)
+	}
+}
+
+// healthy reports whether provider's request-path breaker currently
+// permits routing to it — feat/failover's hard SKIP gate
+// (orderedFailoverCandidates, above), distinct from
+// modelRegistry.discoveryHealthy's deprioritize-only contract. An
+// unrecorded provider (never failed, or never attempted at all)
+// reads healthy — the same "never seen a failure" default the discovery
+// breaker's own zero value already applies (providerState's doc
+// comment, registry.go).
+func (t *requestHealthTracker) healthy(provider string) bool {
+	if t == nil || provider == "" {
+		return true
+	}
+	st := t.stateFor(provider)
+	st.mu.Lock()
+	defer st.mu.Unlock()
+	if !st.open {
+		return true
+	}
+	return !t.nowFn().Before(st.openUntil)
+}
@@ -70,7 +70,13 @@ type Config struct {
 	// at all) resolves to every documented default and behaves exactly
 	// like a deployment with no circuit breaker at all for a provider
 	// that never fails — see validateBreakerConfig's own doc comment.
-	Breaker BreakerConfig `json:"breaker"`
+	// Failover configures feat/failover — cross-provider failover for a
+	// bare model id more than one configured provider serves. See
+	// FailoverConfig's own doc comment (failover.go) for the full
+	// contract; a struct value, not a pointer, for the same
+	// zero-means-default reason as Breaker/Retry/Cache below.
+	Failover FailoverConfig `json:"failover"`
+	Breaker  BreakerConfig  `json:"breaker"`
 	// Retry is a struct value, not a pointer, because its own Enabled
 	// field is the on/off signal (unlike Redis/Users, where the block's
 	// mere presence is the signal) — so its tag omits "omitempty":
@@ -493,18 +499,24 @@ func defaultBodyAdmissionCap() int {
 const maxExplicitBodyAdmissionCap = 10_000

 // Gateway is the Traefik middleware handler.
+//
+// Field order below is fieldalignment-verified (golangci-lint's govet
+// enable-all, run with -fix against a scratch copy to derive the exact
+// zero-waste sequence, then hand-applied here so every field keeps its
+// original doc comment) — see providerState's own doc comment
+// (registry.go) for the general convention.
 type Gateway struct {
-	next     http.Handler
-	cfg      *Config
-	auth     *authStore
-	limiter  *limiter
-	registry *modelRegistry
-	adapters map[string]providerAdapter
+	next http.Handler
 	// targetClient is the shared, connection-pooled *http.Client the
 	// MCP/A2A target proxy (mcp_a2a.go) issues every upstream request
 	// through — built once via newAdapterHTTPClient, the same constructor
 	// each provider adapter uses for its own client.
 	targetClient *http.Client
+	auth         *authStore
+	limiter      *limiter
+	registry     *modelRegistry
+	adapters     map[string]providerAdapter
+	cfg          *Config
 	// cache is nil whenever response caching is not configured or not
 	// usable (cfg.Cache.Enabled is false, or true with no config.Redis —
 	// see buildResponseCache, cache.go). Every call site checks for nil
@@ -525,7 +537,17 @@ type Gateway struct {
 	// resized afterward, matching a Go channel's own fixed-capacity
 	// contract.
 	bodyAdmission chan struct{}
-	name          string
+	// failoverHealth is feat/failover's own per-pod, in-memory
+	// request-path health signal (failover.go), built once, here, by
+	// newGateway. Every method on it is nil-receiver-safe, so a Gateway
+	// assembled directly (bypassing newGateway, as a few older tests do)
+	// degrades to "failover never skips a candidate for request health"
+	// rather than a nil-pointer panic.
+	failoverHealth *requestHealthTracker
+	name           string
+	// failover is Config.Failover, validated and resolved once by
+	// newGateway (validateFailoverConfig, failover.go).
+	failover failoverConfig
 }

 // telemetryStartupOnce keeps the anonymous "plugin loaded" ping to one per
@@ -634,6 +656,16 @@ func newGateway(ctx context.Context, next http.Handler, config *Config, name str
 	}
 	g := &Gateway{next: next, name: name, cfg: config, auth: auth}

+	// feat/failover: validated once, here — see FailoverConfig's own doc
+	// comment (failover.go) for the default (enabled) and what "single-
+	// provider deployment behaves exactly as before" means in practice.
+	failoverCfg, err := validateFailoverConfig(config.Failover)
+	if err != nil {
+		return nil, err
+	}
+	g.failover = failoverCfg
+	g.failoverHealth = newRequestHealthTracker()
+
 	// bodyAdmissionCap: an explicit MaxInFlightBodyRequests always wins
 	// over the self-tuned default (house rule: explicit override always
 	// wins) — see Config.MaxInFlightBodyRequests' own doc comment. 0 (the
@@ -256,123 +256,223 @@ func (g *Gateway) runUnified(w http.ResponseWriter, r *http.Request, u *user, gr
 // envelope shape they answer in. logPrefix names the route in every log
 // line this function writes, so a log reader can always tell which route
 // produced a given line without the classification logic existing twice.
+//
+// feat/failover: this function now tries a LIST of candidate providers
+// (g.registry.resolveWithCandidates/g.orderedFailoverCandidates,
+// failover.go), not just the one resolve alone would pick, and is the
+// ONE hook point both /v1/chat/completions (runUnified, above) and
+// /v1/messages (handleMessages, routes_messages.go) share — per the
+// brief's own instruction, failover is wired here once rather than
+// forked into a second implementation. THE HARD CONSTRAINT this loop is
+// built around: call(...) writes directly to respWriter, which tees
+// straight through to sw; the very first byte written commits the
+// response, so a candidate is only ever retried while sw.wroteHeader is
+// still false. Every step that was a one-shot computation before this
+// feature (the cache key, the attempt recorder's bound provider name,
+// the usage/canonical id billed) now happens fresh INSIDE the loop, once
+// per candidate — this is what "rebind the attempt recorder per
+// attempt" and "cache under the winning provider's key" actually mean
+// in code, not just in the brief's prose.
 func (g *Gateway) runMeteredCall(sw *statusTrackingWriter, r *http.Request, scopes []limitScope, body []byte, req map[string]any, requestedModel string, grp *group, endpoint, logPrefix string, envelope envelopeWriter, writeUpstream providerUpstreamErrorWriter, call adapterCall) {
-	adapter, upstreamModel, canonical, err := g.registry.resolve(requestedModel, grp)
+	primary, extra, err := g.registry.resolveWithCandidates(requestedModel, grp)
 	if err != nil {
 		writeModelResolveErrorEnvelope(sw, err, envelope)
 		return
 	}
+	candidates := g.orderedFailoverCandidates(primary, extra)

-	req["model"] = upstreamModel
-
-	// cacheable gates every cache step below on spec §2's scope: a
-	// non-streaming request, with a working cache (g.cache is nil
-	// whenever caching is off or Redis was absent — buildResponseCache,
-	// cache.go), for a group that has not opted out (groupCacheEnabled).
-	// cacheKey is computed here — after "model" is rewritten to
-	// upstreamModel but deliberately BEFORE gatewayAliasKey is injected
-	// below — so the hashed request body reflects exactly what goes
-	// upstream. requestedModel (the client's own, un-rewritten alias
-	// string) and endpoint are passed as separate cacheKey arguments,
-	// not left in req: a translating adapter bakes the echoed alias
-	// straight into the cached response body's own "model" field, so two
-	// clients requesting the same upstream model under different alias
-	// forms (e.g. "claude-x" vs "anthropic/claude-x") must never collide
-	// into one cache entry (cacheKey's own doc comment, cache.go).
+	// cacheable/streaming gate every cache step below on spec §2's scope,
+	// exactly as before this feature: a non-streaming request, with a
+	// working cache (g.cache is nil whenever caching is off or Redis was
+	// absent — buildResponseCache, cache.go), for a group that has not
+	// opted out (groupCacheEnabled). Computed once, outside the loop: it
+	// depends only on req["stream"]/grp, neither of which a failover
+	// attempt changes.
 	streaming, _ := req["stream"].(bool)
 	cacheable := !streaming && g.cache != nil && groupCacheEnabled(grp)
-	var cacheKeyStr string
-	if cacheable {
-		cacheKeyStr = cacheKey(adapter.name(), upstreamModel, requestedModel, endpoint, req)
-		if cached, hit := g.cache.lookup(cacheKeyStr); hit {
-			// A cache hit accounts the request only — checkAndCount
-			// above already counted it — never token/cost, and never
-			// runs the zero-usage estimate branch below: the client
-			// never reached the upstream provider its body size would
-			// be estimating against.
-			sw.Header().Set("X-Llmgw-Cache", "hit")
-			if cached.ContentType != "" {
-				sw.Header().Set("Content-Type", cached.ContentType)
+
+	for i, cand := range candidates {
+		if i > 0 {
+			// checkAndCount above (admitRequest, this file) already
+			// counted this request exactly once — never double-count: a
+			// failover attempt is a NEW upstream attempt, not a new
+			// logical request, so nothing here calls checkAndCount
+			// again. Per-provider ATTEMPT counters, below, still record
+			// every attempt independently via the freshly-bound
+			// recorder.
+			g.warnf("%s: failing over from provider %q to %q for model %q", logPrefix, candidates[i-1].providerName, cand.providerName, requestedModel)
+		}
+
+		// req["model"] is reset on EVERY iteration: cand.upstreamModel is
+		// the identical string across every candidate in a failover chain
+		// by construction (failoverCandidates, failover.go, only ever
+		// adds providers sharing the SAME bare id resolve's primary
+		// already resolved to), so this is a no-op rewrite on the common
+		// single-candidate path, not new cost.
+		req["model"] = cand.upstreamModel
+
+		// cacheKey is computed fresh per candidate — cache correctness
+		// (brief): a response produced by a failover provider must be
+		// stored under THAT provider's own key, never the first
+		// candidate's. Called here, BEFORE gatewayAliasKey is injected
+		// below, preserving the ordering cacheKey's own doc comment
+		// (cache.go) requires of every caller: gatewayAliasKey is a
+		// purely internal echo-back mechanism cacheKey does not itself
+		// strip, so computing the key after injecting it would hash an
+		// extra, noisy field.
+		var cacheKeyStr string
+		if cacheable {
+			cacheKeyStr = cacheKey(cand.providerName, cand.upstreamModel, requestedModel, endpoint, req)
+			if cached, hit := g.cache.lookup(cacheKeyStr); hit {
+				// A cache hit accounts the request only — checkAndCount
+				// above already counted it — never token/cost, and never
+				// runs the zero-usage estimate branch below: the client
+				// never reached the upstream provider its body size would
+				// be estimating against.
+				sw.Header().Set("X-Llmgw-Cache", "hit")
+				if cached.ContentType != "" {
+					sw.Header().Set("Content-Type", cached.ContentType)
+				}
+				sw.WriteHeader(cached.Status)
+				_, _ = sw.Write(cached.Body)
+				return
 			}
-			sw.WriteHeader(cached.Status)
-			_, _ = sw.Write(cached.Body)
+			// Set before call() below writes anything — headers must
+			// precede the body a miss is about to produce (spec §2). A
+			// later iteration's own miss overwrites this harmlessly: it
+			// is never observable until SOME candidate actually writes,
+			// by which point the loop has already committed to that
+			// candidate's outcome (THE HARD CONSTRAINT, this function's
+			// own doc comment).
+			sw.Header().Set("X-Llmgw-Cache", "miss")
+		}
+
+		// gatewayAliasKey is (re-)injected on every iteration, AFTER the
+		// cacheKey computation above: every providerAdapter implementation
+		// unconditionally deletes it before it ever touches the network
+		// (its own doc comment, providers.go) — a failed first attempt
+		// leaves req without it, and the next candidate needs it set
+		// again to echo the client's own requested alias correctly
+		// (anthropic/gemini-type adapters read it to build their own
+		// response envelope).
+		req[gatewayAliasKey] = requestedModel
+
+		// respWriter is sw, wrapped in a capture tee only when this
+		// request is cacheable — cacheCaptureWriter buffers everything
+		// written so a 200 non-stream response can be stored after
+		// call() returns, without any adapter knowing caching exists
+		// (adapterCall's contract is unchanged either way). A fresh
+		// capture per candidate: reusing one across attempts would let a
+		// candidate that never wrote anything (the common failover case)
+		// leave a stale buffer behind — capture.status stays 0 either
+		// way, so this is defensive clarity, not a proven bug fix.
+		var respWriter http.ResponseWriter = sw
+		var capture *cacheCaptureWriter
+		if cacheable {
+			capture = newCacheCaptureWriter(sw, g.cache.maxBodyBytes)
+			respWriter = capture
+		}
+
+		// Feature A (v0.22) attempt recorder, REBOUND every iteration to
+		// THIS candidate's own provider name (feat/failover's own "never
+		// misattribute a retried attempt to the first provider"
+		// requirement) — every upstream attempt call() makes, via
+		// upstreamJSON's retryPolicy.do (providers.go), reports through
+		// here. The same closure also feeds g.failoverHealth (failover.go),
+		// feat/failover's own request-path health signal, with the
+		// IDENTICAL isTransient||isDeadlineExceeded classification
+		// limiter.recordProviderAttempt already applies — see
+		// requestHealthTracker.record's own doc comment for why this must
+		// stay the same classification, not a wider one.
+		providerName := cand.providerName
+		upstreamModel := cand.upstreamModel
+		ctx := withAttemptRecorder(r.Context(), func(resp *http.Response, attemptErr error) {
+			g.limiter.recordProviderAttempt(providerName, upstreamModel, resp, attemptErr)
+			g.failoverHealth.record(providerName, !isTransient(resp, attemptErr) && !isDeadlineExceeded(attemptErr))
+		})
+		result, callErr := call(cand.adapter, ctx, respWriter, req)
+
+		// Usage is accounted before the error branch below runs, not
+		// after: every adapter that can fail mid-stream (forwardStream
+		// in each of the three provider files) still returns whatever
+		// usage it had already captured alongside the error — a usage
+		// chunk that arrived just before a dropped connection must
+		// still be billed, or a client that aborts right after that
+		// frame arrives could repeat the trick to dodge every
+		// token/cost budget. A non-streaming failure (providerHTTPError,
+		// translateError, or a connection failure before any write)
+		// always carries zero usage by contract, so accounting it here
+		// is a no-op — account skips every write once both total tokens
+		// and cost are zero. routes_messages.go's
+		// callTranslatedMessages/callAnthropicMessagesPassthrough follow
+		// the identical contract for their own error paths (item 1 fix,
+		// 2026-08-22 review). This still runs exactly once per
+		// CANDIDATE, including a candidate that goes on to fail over —
+		// its own (zero, by contract) usage is billed as a no-op, never
+		// skipped, so a later successful candidate's real usage is never
+		// silently doubled with a phantom first entry either.
+		if callErr == nil && result.total() == 0 {
+			if streaming {
+				g.logf("%s: zero usage reported for a streaming response from model %q; accounting the request only", logPrefix, cand.canonical)
+			} else {
+				result.prompt = int64(math.Ceil(float64(len(body)) / 4))
+				result.estimated = true
+			}
+		}
+		g.limiter.account(scopes, result, unifiedCostMicros(cand.canonical, cand.upstreamModel, result, g.cfg.Pricing))
+		if result.estimated {
+			g.logf("%s: usage for model %q logged as estimated (%d prompt tokens derived from request body size, not the provider's reported usage)", logPrefix, cand.canonical, result.prompt)
+		}
+
+		// A miss stores the response after everything above has already
+		// run — accounting must never be skipped or delayed waiting on a
+		// cache write. Only a genuine upstream 200 is stored (spec §2's
+		// "on 200 non-stream, tee and SET"): a non-2xx status never
+		// reaches here as capture.status (forwardJSON/translate error
+		// paths return *providerHTTPError/*translateError instead of
+		// writing through respWriter, so callErr is non-nil and
+		// capture.status stays 0), and callErr == nil is checked
+		// directly regardless. capture.oversize excludes a response
+		// cacheCaptureWriter stopped buffering past maxBodyBytes:
+		// store's own maxBodyBytes check would reject it too, but only
+		// after being handed a silently truncated body — skip the call
+		// outright instead of ever constructing a corrupt cache entry.
+		if cacheable && callErr == nil && capture.status == http.StatusOK && !capture.oversize {
+			g.cache.store(cacheKeyStr, capture.status, capture.contentType, capture.buf.Bytes(), effectiveTTL(g.cache, grp))
+		}
+
+		if callErr == nil {
 			return
 		}
-		// Set before call() below writes anything — headers must precede
-		// the body a miss is about to produce (spec §2).
-		sw.Header().Set("X-Llmgw-Cache", "miss")
-	}

-	req[gatewayAliasKey] = requestedModel
-
-	// respWriter is sw, wrapped in a capture tee only when this request
-	// is cacheable — cacheCaptureWriter buffers everything written so a
-	// 200 non-stream response can be stored after call() returns, without
-	// any adapter knowing caching exists (adapterCall's contract is
-	// unchanged either way).
-	var respWriter http.ResponseWriter = sw
-	var capture *cacheCaptureWriter
-	if cacheable {
-		capture = newCacheCaptureWriter(sw, g.cache.maxBodyBytes)
-		respWriter = capture
-	}
-
-	// Feature A (v0.22): every upstream attempt call() makes — via
-	// upstreamJSON's retryPolicy.do, providers.go — reports through to
-	// this recorder, closing over the adapter/model this request already
-	// resolved to. See attemptRecorder's own doc comment (providers.go).
-	ctx := withAttemptRecorder(r.Context(), func(resp *http.Response, attemptErr error) {
-		g.limiter.recordProviderAttempt(adapter.name(), upstreamModel, resp, attemptErr)
-	})
-	result, callErr := call(adapter, ctx, respWriter, req)
-
-	// Usage is accounted before the error branch below runs, not after:
-	// every adapter that can fail mid-stream (forwardStream in each of the
-	// three provider files) still returns whatever usage it had already
-	// captured alongside the error — a usage chunk that arrived just before
-	// a dropped connection must still be billed, or a client that aborts
-	// right after that frame arrives could repeat the trick to dodge every
-	// token/cost budget. A non-streaming failure (providerHTTPError,
-	// translateError, or a connection failure before any write) always
-	// carries zero usage by contract, so accounting it here is a no-op —
-	// account skips every write once both total tokens and cost are zero.
-	// routes_messages.go's callTranslatedMessages/
-	// callAnthropicMessagesPassthrough follow the identical contract for
-	// their own error paths (item 1 fix, 2026-08-22 review: a translation
-	// failure AFTER a successful, billed upstream call must still return
-	// that call's real usage, not usage{}).
-	if callErr == nil && result.total() == 0 {
-		if streaming {
-			g.logf("%s: zero usage reported for a streaming response from model %q; accounting the request only", logPrefix, canonical)
-		} else {
-			result.prompt = int64(math.Ceil(float64(len(body)) / 4))
-			result.estimated = true
+		// THE HARD CONSTRAINT: once sw has written anything, status and
+		// body are committed and can never be retried elsewhere — a
+		// mid-stream failure (headers, or a partial SSE body, already
+		// reached the client) always falls straight through to the
+		// existing, unchanged error classification, exactly as it did
+		// before this feature existed, regardless of how many candidates
+		// remain.
+		if sw.wroteHeader {
+			g.handleAdapterErrorEnvelope(sw, callErr, providerName, logPrefix, envelope, writeUpstream)
+			return
 		}
-	}
-	g.limiter.account(scopes, result, unifiedCostMicros(canonical, upstreamModel, result, g.cfg.Pricing))
-	if result.estimated {
-		g.logf("%s: usage for model %q logged as estimated (%d prompt tokens derived from request body size, not the provider's reported usage)", logPrefix, canonical, result.prompt)
-	}

-	// A miss stores the response after everything above has already run —
-	// accounting must never be skipped or delayed waiting on a cache
-	// write. Only a genuine upstream 200 is stored (spec §2's "on 200
-	// non-stream, tee and SET"): a non-2xx status never reaches here as
-	// capture.status (forwardJSON/translate error paths return
-	// *providerHTTPError/*translateError instead of writing through
-	// respWriter, so callErr is non-nil and capture.status stays 0), and
-	// callErr == nil is checked directly regardless. capture.oversize
-	// excludes a response cacheCaptureWriter stopped buffering past
-	// maxBodyBytes: store's own maxBodyBytes check would reject it too,
-	// but only after being handed a silently truncated body — skip the
-	// call outright instead of ever constructing a corrupt cache entry.
-	if cacheable && callErr == nil && capture.status == http.StatusOK && !capture.oversize {
-		g.cache.store(cacheKeyStr, capture.status, capture.contentType, capture.buf.Bytes(), effectiveTTL(g.cache, grp))
-	}
-
-	if callErr != nil {
-		g.handleAdapterErrorEnvelope(sw, callErr, adapter.name(), logPrefix, envelope, writeUpstream)
-		return
+		hasNext := i < len(candidates)-1 && failoverEligible(callErr)
+		if hasNext && isProviderNotFoundError(callErr) {
+			// Loud, non-suppressed log line (brief: "Failover after a 404
+			// must log loudly ... a 404 usually means a real
+			// configuration mistake, and silently succeeding elsewhere
+			// would hide it") — deliberately g.errorf, not g.logf, and
+			// deliberately its own line rather than folded into the
+			// generic "failing over" line above the next iteration
+			// already prints.
+			g.errorf("%s: provider %q returned 404 for model %q; failing over to %q — this usually means a real provider/model configuration mistake, not a transient outage", logPrefix, providerName, requestedModel, candidates[i+1].providerName)
+		}
+		if !hasNext {
+			g.handleAdapterErrorEnvelope(sw, callErr, providerName, logPrefix, envelope, writeUpstream)
+			return
+		}
+		// Falls through to the next candidate.
 	}
 }

@@ -346,6 +346,31 @@ func run() error {
 	}))
 	defer brkUpstream.Close()

+	// failoverAUpstream/failoverBUpstream back exerciseFailover (feat/
+	// failover, below): two providers configured with the IDENTICAL bare
+	// model id (failoverModelID) — the exact shape registry.go's bareWinner
+	// collision handles, and the shape production logs show for whisper-1/
+	// tts-1/tts-1-hd/the embeddings models across openai/openai-audio/
+	// copilot. A always answers 500; B always answers 200. This proves the
+	// failover candidate loop (runMeteredCall, routes_unified.go) falls
+	// through and serves correctly under the REAL interpreter, not merely
+	// compiled — this package's own doc comment is exactly why that
+	// distinction matters on this codebase.
+	failoverAUpstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusInternalServerError)
+		_, _ = w.Write([]byte(`{"error":{"message":"failover-a is down","type":"server_error"}}`))
+	}))
+	defer failoverAUpstream.Close()
+	failoverBHits := new(int64)
+	failoverBUpstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		atomic.AddInt64(failoverBHits, 1)
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte(`{"id":"c-failover","object":"chat.completion","model":"` + failoverModelID + `","choices":[{"index":0,"message":{"role":"assistant","content":"served by failover-b"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1}}`))
+	}))
+	defer failoverBUpstream.Close()
+
 	attemptAccountingOverride := `{"breaker":{"failureThreshold":2,"openDuration":"3s","maxOpenDuration":"6s"},` +
 		`"providers":{"openai":{"type":"openai","baseUrl":"` + upstream.URL + `","apiKey":"sk-up","models":["` + testDataWantModel + `","` + builtinLookupModelID + `"]},` +
 		`"brk":{"type":"openai","baseUrl":"` + brkUpstream.URL + `","apiKey":"sk-up","discovery":true,"discoveryInterval":"1ms"},"` +
@@ -354,7 +379,9 @@ func run() error {
 		// (exerciseMessagesRoute, below): a real anthropic-type provider,
 		// interpreted end to end through
 		// callAnthropicMessagesPassthrough.
-		`"anthropic":{"type":"anthropic","baseUrl":"` + anthropicProbeUpstream.URL + `","apiKey":"sk-anth","models":["claude-test"]}},` + // #nosec G101 -- test fixture literal, not a real credential
+		`"anthropic":{"type":"anthropic","baseUrl":"` + anthropicProbeUpstream.URL + `","apiKey":"sk-anth","models":["claude-test"]},` + // #nosec G101 -- test fixture literal, not a real credential
+		`"failover-a":{"type":"openai","baseUrl":"` + failoverAUpstream.URL + `","apiKey":"sk-up","models":["` + failoverModelID + `"]},` +
+		`"failover-b":{"type":"openai","baseUrl":"` + failoverBUpstream.URL + `","apiKey":"sk-up","models":["` + failoverModelID + `"]}},` +
 		// modelMeta (feature v0.23): a config-override entry for
 		// testDataWantModel, so exerciseHandler's GET /v1/models
 		// assertion below proves resolveModelMeta's config-override
@@ -405,7 +432,7 @@ func run() error {
 		return fmt.Errorf("%s.New's returned value does not implement http.Handler", pkgName)
 	}

-	if err := exerciseHandler(handler, builtinLookupContextTokens); err != nil {
+	if err := exerciseHandler(handler, builtinLookupContextTokens, failoverBHits); err != nil {
 		return err
 	}
 	// exerciseBreaker (feat/provider-health, adversarial-review round 2):
@@ -429,6 +456,11 @@ func run() error {
 // the config-override path testDataWantModel already exercises.
 const builtinLookupModelID = "gpt-4o"

+// failoverModelID is the bare model id both "failover-a" and "failover-b"
+// (run(), above) are configured with — the run()-owned upstream servers
+// exerciseFailover (below) drives.
+const failoverModelID = "failover-shared"
+
 // mcpProbeServerName is the federated MCP server run() configures against
 // mcpProbeUpstream, and mcpProbeToolName is a tool id carrying its
 // "<server>_" prefix so resolveFederatedTool routes POST /mcp's
@@ -634,8 +666,8 @@ func exerciseBreaker(handler http.Handler, hits, healthy *int64) error {
 // for real under Yaegi, not merely that the package's imports resolve.
 // builtinContextTokens is builtinLookupModelID's own real, extracted
 // builtinModelMetaTable value (SHOULD-6) — see that constant's own doc
-// comment.
-func exerciseHandler(handler http.Handler, builtinContextTokens int) error {
+// comment. failoverBHits backs exerciseFailover's own call, below.
+func exerciseHandler(handler http.Handler, builtinContextTokens int, failoverBHits *int64) error {
 	authedReq := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
 	authedReq.Header.Set("Authorization", "Bearer "+testDataUserAPIKey)
 	authedRec := httptest.NewRecorder()
@@ -702,6 +734,10 @@ func exerciseHandler(handler http.Handler, builtinContextTokens int) error {
 		return err
 	}

+	if err := exerciseFailover(handler, failoverBHits); err != nil {
+		return err
+	}
+
 	if err := exerciseFederatedTooLarge(handler); err != nil {
 		return err
 	}
@@ -1013,6 +1049,38 @@ func exerciseFederatedTooLarge(handler http.Handler) error {
 	return nil
 }

+// exerciseFailover drives one real POST /v1/chat/completions against the
+// interpreted handler for failoverModelID — a bare model id both
+// "failover-a" (always 500s) and "failover-b" (always 200s) are
+// configured with — proving the failover candidate loop (runMeteredCall,
+// routes_unified.go/failover.go) falls through a genuine provider
+// failure and serves the second provider's response under the REAL
+// interpreter. Compiled tests already prove this (failover_test.go); this
+// harness exists because compiled tests have repeatedly passed on this
+// codebase while the interpreted shape failed (this package's own doc
+// comment).
+func exerciseFailover(handler http.Handler, bHits *int64) error {
+	req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(
+		`{"model":"`+failoverModelID+`","messages":[{"role":"user","content":"hi"}]}`,
+	))
+	req.Header.Set("Authorization", "Bearer "+testDataUserAPIKey)
+	req.Header.Set("Content-Type", "application/json")
+	rec := httptest.NewRecorder()
+	handler.ServeHTTP(rec, req)
+
+	if rec.Code != http.StatusOK {
+		return fmt.Errorf("POST /v1/chat/completions (feat/failover harness): status = %d, want 200 (failover-a's 500 must fall through to failover-b), body=%s", rec.Code, rec.Body.String())
+	}
+	if !strings.Contains(rec.Body.String(), "served by failover-b") {
+		return fmt.Errorf("POST /v1/chat/completions (feat/failover harness): body does not contain failover-b's own content, want the SECOND provider's response: %s", rec.Body.String())
+	}
+	if atomic.LoadInt64(bHits) != 1 {
+		return fmt.Errorf("feat/failover harness: failover-b upstream hit count = %d, want exactly 1", atomic.LoadInt64(bHits))
+	}
+	fmt.Println("yaegi-check: failover fell through from failover-a to failover-b and served its response")
+	return nil
+}
+
 func exerciseAttemptAccounting(handler http.Handler) error {
 	chatReq := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(
 		`{"model":"`+testDataWantModel+`","messages":[{"role":"user","content":"hi"}]}`,