mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
feat(failover): add cross-provider failover for overlapping models
Add failover.go with the full candidate pipeline. resolveWithCandidates resolves a request exactly like resolve does, then adds every other provider sharing the identical bare model id. orderedFailoverCandidates applies a hard skip for request-path-unhealthy providers, deprioritizes discovery-unhealthy providers without dropping them, fails open when every candidate reads unhealthy, and caps the list at maxAttempts. Add requestHealthTracker, a separate per-pod request-path health signal fed by the same isTransient/isDeadlineExceeded classification recordProviderAttempt already uses. It never widens the existing discovery circuit breaker, per that breaker's own documented scope. Rewrite runMeteredCall (routes_unified.go) to loop over candidates instead of resolving once. Each attempt rebinds the attempt recorder to its own provider, recomputes the cache key before the response is cached, and re-injects the alias echo key an adapter may have deleted on a prior failed attempt. Failover never runs once any byte has reached the client, since statusTrackingWriter already tracks that state. A failure falls through on everything except HTTP 400, matching the brief exactly. A 404 failover logs loudly, naming both providers and the model, since it usually signals a real configuration mistake. Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A single-provider deployment, or one with no overlapping bare model ids, behaves exactly as before regardless of this setting. Add a yaegi-check probe (tools/yaegi-check/main.go) driving two providers under the real interpreter, one failing and one serving, proving the candidate loop runs correctly interpreted. Update README.md with a Failover section and correct the Retry section's stale claim that no cross-provider failover exists. Media routes (images, audio speech, audio transcriptions) do not get failover in this change. Each has its own request-shape handling that needs its own scoped follow-up.
This commit is contained in:
+487
@@ -0,0 +1,487 @@
|
||||
package traefikllmgateway
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// This file implements feat/failover: cross-provider failover for a bare
|
||||
// model id more than one configured provider serves (registry.go's
|
||||
// bareWinner collision case — production logs show whisper-1, tts-1,
|
||||
// tts-1-hd, text-embedding-3-small, text-embedding-3-large, and
|
||||
// text-embedding-ada-002 served by two or more of openai/openai-audio/
|
||||
// copilot today). Before this feature, registry.go's resolve returns
|
||||
// exactly one provider adapter and retry.go's retryPolicy only ever
|
||||
// retries the SAME provider — so a down provider fails every request for
|
||||
// its models even when another configured provider serves the identical
|
||||
// model. This file turns that already-existing redundancy into a hot
|
||||
// standby, hooked into routes_unified.go's runMeteredCall, the ONE shared
|
||||
// metered-call path both /v1/chat/completions and /v1/messages already go
|
||||
// through.
|
||||
//
|
||||
// SCOPE: routes_media.go's three media routes (images/generations,
|
||||
// audio/speech, audio/transcriptions) are deliberately NOT wired into
|
||||
// failover in this change, even though two of the brief's own named
|
||||
// overlapping models (whisper-1, tts-1/tts-1-hd) are media-route
|
||||
// traffic. Each of those three handlers has its own request-shape
|
||||
// preprocessing (multipart rebuild for transcriptions, a JSON
|
||||
// re-marshal for speech, a binary/streaming response for speech) that
|
||||
// would need its own careful "nothing written yet" analysis and its own
|
||||
// test coverage, rather than reusing runMeteredCall's shared shape — the
|
||||
// brief's own hook-point instruction ("do not fork a second
|
||||
// implementation") is about not duplicating the chat/messages pipeline,
|
||||
// not about extending every possible call site in one pass. Left as a
|
||||
// scoped follow-up; see this branch's own report for the explicit
|
||||
// call-out.
|
||||
|
||||
// FailoverConfig configures feat/failover. Enabled is a *bool, mirroring
|
||||
// ProviderConfig.Passthrough's own nil-means-default-true convention
|
||||
// (providers.go): nil or true (the default) turns an overlapping
|
||||
// multi-provider deployment's already-existing redundancy into a hot
|
||||
// standby with NO config change — the entire point of this feature (see
|
||||
// this file's own package doc and the brief's "Why" section). Set false
|
||||
// to opt out entirely. A single-provider deployment, or one where no two
|
||||
// providers ever serve the identical bare model id, behaves exactly as
|
||||
// before regardless of this setting: there is never more than one
|
||||
// candidate to try (orderedFailoverCandidates' own doc comment).
|
||||
type FailoverConfig struct {
|
||||
Enabled *bool `json:"enabled,omitempty"`
|
||||
// MaxAttempts caps how many DIFFERENT PROVIDERS runMeteredCall tries
|
||||
// in total for one logical request, the primary included — distinct
|
||||
// from RetryConfig.Attempts (retry.go), which retries the SAME
|
||||
// provider. 0 (the default) uses defaultFailoverMaxAttempts. A value
|
||||
// outside 1..maxFailoverMaxAttempts is a construction error.
|
||||
MaxAttempts int `json:"maxAttempts,omitempty"`
|
||||
}
|
||||
|
||||
// defaultFailoverMaxAttempts is applied when FailoverConfig.MaxAttempts is
|
||||
// left at 0. maxFailoverMaxAttempts bounds an explicit override, the same
|
||||
// "cheap sanity ceiling on an operator value" precedent
|
||||
// maxBreakerFailureThreshold/maxRetryAttempts already set (registry.go/
|
||||
// retry.go).
|
||||
const (
|
||||
defaultFailoverMaxAttempts = 3
|
||||
maxFailoverMaxAttempts = 10
|
||||
)
|
||||
|
||||
// failoverConfig is FailoverConfig, validated and defaulted once at
|
||||
// construction (newGateway, llmgateway.go) — mirrors breakerConfig/
|
||||
// retryPolicy's own resolved-value-not-raw-config shape (registry.go/
|
||||
// retry.go).
|
||||
type failoverConfig struct {
|
||||
maxAttempts int
|
||||
enabled bool
|
||||
}
|
||||
|
||||
// validateFailoverConfig validates fc and returns the resolved
|
||||
// failoverConfig newGateway attaches to the Gateway.
|
||||
func validateFailoverConfig(fc FailoverConfig) (failoverConfig, error) {
|
||||
enabled := true
|
||||
if fc.Enabled != nil {
|
||||
enabled = *fc.Enabled
|
||||
}
|
||||
|
||||
maxAttempts := fc.MaxAttempts
|
||||
if maxAttempts == 0 {
|
||||
maxAttempts = defaultFailoverMaxAttempts
|
||||
}
|
||||
if maxAttempts < 1 || maxAttempts > maxFailoverMaxAttempts {
|
||||
return failoverConfig{}, fmt.Errorf("llmgateway: failover.maxAttempts must be between 1 and %d, got %d", maxFailoverMaxAttempts, maxAttempts)
|
||||
}
|
||||
|
||||
return failoverConfig{enabled: enabled, maxAttempts: maxAttempts}, nil
|
||||
}
|
||||
|
||||
// resolveCandidate is one provider that can serve a client-requested model
|
||||
// id — modelRegistry.resolve's own (adapter, upstreamModel, canonical)
|
||||
// triple, tagged with the provider's own name so a failover attempt can
|
||||
// rebind its cache key, attempt recorder, and error log to the ACTUAL
|
||||
// provider that served or failed the request, never the first one resolve
|
||||
// happened to pick.
|
||||
type resolveCandidate struct {
|
||||
adapter providerAdapter
|
||||
providerName string
|
||||
upstreamModel string
|
||||
canonical string
|
||||
}
|
||||
|
||||
// resolveWithCandidates resolves id exactly as modelRegistry.resolve
|
||||
// (registry.go) does — identical winner, identical error — and
|
||||
// additionally reports every OTHER configured provider that also serves
|
||||
// the same bare upstream model id and remains authorized for grp, as
|
||||
// feat/failover's candidate pool.
|
||||
//
|
||||
// primary/err are byte-identical to what resolve(id, grp) alone would
|
||||
// return: this method calls resolve internally and never second-guesses
|
||||
// its winner or its error — feat/failover only ADDS candidates after an
|
||||
// already-successful resolution, it never changes which model or error a
|
||||
// request resolves to (in particular, authorization denial/unknown-model
|
||||
// behavior for the primary winner is completely unchanged by this
|
||||
// branch). extra is nil whenever there is nothing to add:
|
||||
//
|
||||
// - id resolved via an explicit "provider/model" prefix, or an alias
|
||||
// whose target is itself provider-prefixed: the requester's own
|
||||
// explicit provider choice is never a candidate for pool
|
||||
// substitution — ruling, this feature's own design: "provider/model"
|
||||
// addressing is exact, not eligible for failover.
|
||||
// - id resolved via a bare-id collision winner (bareWinner), or an
|
||||
// alias whose target is itself a bare-id collision winner: extra is
|
||||
// every OTHER configured provider whose known model set also
|
||||
// contains that identical bare id string, in
|
||||
// modelRegistry.providerNames (sorted) order, and authorized for grp
|
||||
// on that provider.
|
||||
func (m *modelRegistry) resolveWithCandidates(id string, grp *group) (resolveCandidate, []resolveCandidate, error) {
|
||||
adapter, upstreamModel, canonical, err := m.resolve(id, grp)
|
||||
if err != nil {
|
||||
return resolveCandidate{}, nil, err
|
||||
}
|
||||
primary := resolveCandidate{adapter: adapter, providerName: adapter.name(), upstreamModel: upstreamModel, canonical: canonical}
|
||||
|
||||
bareID := id
|
||||
if target, isAlias := m.aliases[id]; isAlias {
|
||||
if _, _, ok := m.splitConfiguredProvider(target); ok {
|
||||
return primary, nil, nil
|
||||
}
|
||||
bareID = target
|
||||
} else if _, _, ok := m.splitConfiguredProvider(id); ok {
|
||||
return primary, nil, nil
|
||||
}
|
||||
|
||||
return primary, m.failoverCandidates(bareID, primary.providerName, grp), nil
|
||||
}
|
||||
|
||||
// failoverCandidates returns every configured provider other than
|
||||
// primaryProvider whose known model set contains bareID and that grp is
|
||||
// authorized to use, in modelRegistry.providerNames (sorted) order — the
|
||||
// raw candidate pool orderedFailoverCandidates (below) narrows and
|
||||
// reorders further by health.
|
||||
//
|
||||
// Only grp.allowsProvider is re-checked per candidate: grp.allowsModel is
|
||||
// already known true from the primary winner's own successful resolve
|
||||
// call, and — for the identical bare id string every candidate here
|
||||
// shares by construction — allowsModel's result does not vary by
|
||||
// provider (auth.go's allowsModel takes no provider argument), so
|
||||
// re-checking it per candidate would be redundant work, not a
|
||||
// correctness gap.
|
||||
func (m *modelRegistry) failoverCandidates(bareID, primaryProvider string, grp *group) []resolveCandidate {
|
||||
var extra []resolveCandidate
|
||||
for _, name := range m.providerNames {
|
||||
if name == primaryProvider {
|
||||
continue
|
||||
}
|
||||
if !m.states[name].hasModel(bareID) || !grp.allowsProvider(name) {
|
||||
continue
|
||||
}
|
||||
// canonical assigned to a local before the struct literal, matching
|
||||
// resolveAgainst's own documented Yaegi precedent (registry.go): a
|
||||
// defensive, cheap precaution against the same class of interpreter
|
||||
// trap, not a proven necessity for a struct-literal field (only a
|
||||
// multi-value RETURN tuple is the proven-bad shape there).
|
||||
canonical := name + "/" + bareID
|
||||
extra = append(extra, resolveCandidate{
|
||||
adapter: m.adapters[name],
|
||||
providerName: name,
|
||||
upstreamModel: bareID,
|
||||
canonical: canonical,
|
||||
})
|
||||
}
|
||||
return extra
|
||||
}
|
||||
|
||||
// orderedFailoverCandidates builds runMeteredCall's actual try list from
|
||||
// resolveWithCandidates' own (primary, extra) split.
|
||||
//
|
||||
// Failover is a no-op (returns exactly [primary]) whenever it is disabled
|
||||
// (g.failover.enabled false) or there is nothing to add (len(extra) ==
|
||||
// 0) — the second case is what makes "single-provider deployment behaves
|
||||
// exactly as before" hold unconditionally, regardless of the enabled
|
||||
// switch: there is never more than one candidate to try when no other
|
||||
// configured provider shares this bare model id.
|
||||
//
|
||||
// Otherwise, two health signals apply, deliberately kept separate
|
||||
// (modelRegistry.discoveryHealthy's own doc comment, registry.go, is
|
||||
// binding on this):
|
||||
//
|
||||
// - REQUEST-PATH health (g.failoverHealth, this file) is a HARD SKIP
|
||||
// gate: a candidate it reports unhealthy is excluded outright,
|
||||
// never attempted. If that filtering would remove every candidate
|
||||
// (this local, per-pod signal reporting the whole fleet down
|
||||
// simultaneously is more likely stale than true), it is discarded
|
||||
// and every original candidate is tried anyway — failing open
|
||||
// against the gateway's own imperfect signal rather than refusing
|
||||
// to route a request at all.
|
||||
// - DISCOVERY health (g.registry.discoveryHealthy) is an ORDERING HINT
|
||||
// ONLY, exactly as its own doc comment requires: a discovery-
|
||||
// unhealthy candidate is moved after every discovery-healthy/
|
||||
// half-open one (a stable partition, not a full sort — relative
|
||||
// order within each bucket is preserved), never dropped.
|
||||
//
|
||||
// Finally the list is capped at g.failover.maxAttempts entries.
|
||||
func (g *Gateway) orderedFailoverCandidates(primary resolveCandidate, extra []resolveCandidate) []resolveCandidate {
|
||||
if !g.failover.enabled || len(extra) == 0 {
|
||||
return []resolveCandidate{primary}
|
||||
}
|
||||
|
||||
all := make([]resolveCandidate, 0, len(extra)+1)
|
||||
all = append(all, primary)
|
||||
all = append(all, extra...)
|
||||
|
||||
healthy := make([]resolveCandidate, 0, len(all))
|
||||
for _, c := range all {
|
||||
if g.failoverHealth.healthy(c.providerName) {
|
||||
healthy = append(healthy, c)
|
||||
}
|
||||
}
|
||||
if len(healthy) == 0 {
|
||||
healthy = all
|
||||
}
|
||||
|
||||
ordered := make([]resolveCandidate, 0, len(healthy))
|
||||
var deprioritized []resolveCandidate
|
||||
for _, c := range healthy {
|
||||
if g.registry.discoveryHealthy(c.providerName) {
|
||||
ordered = append(ordered, c)
|
||||
} else {
|
||||
deprioritized = append(deprioritized, c)
|
||||
}
|
||||
}
|
||||
ordered = append(ordered, deprioritized...)
|
||||
|
||||
if max := g.failover.maxAttempts; max > 0 && len(ordered) > max {
|
||||
ordered = ordered[:max]
|
||||
}
|
||||
return ordered
|
||||
}
|
||||
|
||||
// failoverEligible reports whether callErr — runMeteredCall's own
|
||||
// adapterCall return, evaluated only once nothing has reached the client
|
||||
// yet (the hard constraint this whole feature is built around: once an
|
||||
// adapter has written anything, status and body are committed and can
|
||||
// never be retried elsewhere) — should try the next candidate instead of
|
||||
// answering the client with callErr.
|
||||
//
|
||||
// Ruling (brief, "Which failures fall through" — decided, not
|
||||
// relitigated here): every failure except HTTP 400 falls through. 5xx,
|
||||
// connection failures, timeouts, 429, 401, 403, and 404 all do; 400
|
||||
// never does, because the request itself is malformed and every provider
|
||||
// would reject it identically, so retrying just multiplies latency and
|
||||
// burns budget.
|
||||
//
|
||||
// Two failure shapes are excluded for the SAME "retrying changes
|
||||
// nothing" reasoning 400 itself is excluded for, even though neither is
|
||||
// literally an HTTP 400: context.Canceled (the client walked away —
|
||||
// failing over would only waste an attempt against a connection nobody
|
||||
// is waiting on) and errRequestBuildFailed (upstreamBytes, providers.go
|
||||
// — the identical malformed request shape reaches every candidate the
|
||||
// same way, differing only in which operator-configured base URL it is
|
||||
// built against). A *translateError or *responseTranslationError is this
|
||||
// gateway's OWN translation code failing on an already-decoded request —
|
||||
// not a signal about the PROVIDER's availability at all — so neither is
|
||||
// eligible either.
|
||||
func failoverEligible(callErr error) bool {
|
||||
if callErr == nil {
|
||||
return false
|
||||
}
|
||||
if errors.Is(callErr, context.Canceled) || errors.Is(callErr, errRequestBuildFailed) {
|
||||
return false
|
||||
}
|
||||
// Plain type assertions, not errors.As — matching this package's
|
||||
// established Yaegi-safe convention for these exact two pointer error
|
||||
// types (providerHTTPError's own doc comment, providers.go;
|
||||
// handleAdapterErrorEnvelope, routes_unified.go, does the identical
|
||||
// assertion for the identical reason).
|
||||
if perr, ok := callErr.(*providerHTTPError); ok {
|
||||
return perr.status != http.StatusBadRequest
|
||||
}
|
||||
if _, ok := callErr.(*translateError); ok {
|
||||
return false
|
||||
}
|
||||
if _, ok := callErr.(*responseTranslationError); ok {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// isProviderNotFoundError reports whether err is a *providerHTTPError
|
||||
// carrying HTTP 404 — routes_unified.go's runMeteredCall uses this to
|
||||
// decide when a failover needs its own loud, non-suppressed log line
|
||||
// (brief: "Failover after a 404 must log loudly ... a 404 usually means
|
||||
// a real configuration mistake, and silently succeeding elsewhere would
|
||||
// hide it").
|
||||
func isProviderNotFoundError(err error) bool {
|
||||
perr, ok := err.(*providerHTTPError)
|
||||
return ok && perr.status == http.StatusNotFound
|
||||
}
|
||||
|
||||
// requestBreakerFailureThreshold/requestBreakerOpenDuration/
|
||||
// requestBreakerMaxOpenDuration size requestHealthTracker's own breaker —
|
||||
// deliberately separate constants from registry.go's discovery breaker
|
||||
// (defaultBreakerFailureThreshold/defaultBreakerOpenDuration/
|
||||
// defaultBreakerMaxOpenDuration): a request-path outage needs to be
|
||||
// noticed and routed around within seconds, not the discovery breaker's
|
||||
// minutes-to-hours cadence, and modelRegistry.discoveryHealthy's own doc
|
||||
// comment forbids widening that breaker to cover this outcome stream at
|
||||
// all. Fixed constants, not configurable — unlike FailoverConfig above,
|
||||
// the brief only asks for max-attempts and an on/off switch to be
|
||||
// operator-configurable; this breaker's thresholds are an internal
|
||||
// implementation detail of the routing-gate signal, not a documented
|
||||
// tuning surface.
|
||||
const (
|
||||
requestBreakerFailureThreshold = 3
|
||||
requestBreakerOpenDuration = 30 * time.Second
|
||||
requestBreakerMaxOpenDuration = 5 * time.Minute
|
||||
)
|
||||
|
||||
// requestHealthState is one provider's REQUEST-PATH health (feat/
|
||||
// failover), mu-guarded — mirrors providerState's per-provider,
|
||||
// individually-mutexed shape (registry.go) at a much smaller scale: this
|
||||
// tracks exactly one outcome stream (request-path attempts), never
|
||||
// discovery. The zero value is closed/healthy, so a provider that never
|
||||
// fails never touches anything below open/backoff/openUntil.
|
||||
//
|
||||
// Field order below is fieldalignment-verified (golangci-lint's govet
|
||||
// enable-all, run with -fix against a scratch copy to derive the exact
|
||||
// zero-waste sequence, then hand-applied here so this doc comment
|
||||
// survives — the same convention providerState's own doc comment
|
||||
// explains, registry.go).
|
||||
type requestHealthState struct {
|
||||
openUntil time.Time
|
||||
backoff time.Duration
|
||||
consecutiveFailures int
|
||||
mu sync.Mutex
|
||||
open bool
|
||||
}
|
||||
|
||||
// requestHealthTracker is feat/failover's own per-pod, in-memory,
|
||||
// request-path health signal — deliberately separate from registry.go's
|
||||
// discovery circuit breaker (modelRegistry.discoveryHealthy's own doc
|
||||
// comment forbids widening that one to cover request-path traffic). Fed
|
||||
// by the SAME failure classification limiter.recordProviderAttempt uses
|
||||
// (isTransient || isDeadlineExceeded — limits.go/retry.go), via record's
|
||||
// own callers in routes_unified.go, but keeps its own independent state
|
||||
// per provider name.
|
||||
//
|
||||
// Every method is nil-receiver-safe (both report "healthy"/no-op on a
|
||||
// nil *requestHealthTracker), matching this package's own nil-means-
|
||||
// disabled convention for an optional subsystem (responseCache's own doc
|
||||
// comment, llmgateway.go's Gateway.cache field) — a Gateway assembled
|
||||
// directly in a test without going through newGateway, or any future
|
||||
// caller that does the same, degrades to "failover never skips a
|
||||
// candidate for request health" rather than a nil-pointer panic.
|
||||
type requestHealthTracker struct {
|
||||
states map[string]*requestHealthState
|
||||
nowFn func() time.Time
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// newRequestHealthTracker returns an empty tracker — every provider name
|
||||
// reads healthy until its first recorded failure.
|
||||
func newRequestHealthTracker() *requestHealthTracker {
|
||||
return &requestHealthTracker{states: make(map[string]*requestHealthState), nowFn: time.Now}
|
||||
}
|
||||
|
||||
// stateFor returns provider's requestHealthState, creating it on first
|
||||
// use.
|
||||
func (t *requestHealthTracker) stateFor(provider string) *requestHealthState {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
st, ok := t.states[provider]
|
||||
if !ok {
|
||||
st = &requestHealthState{}
|
||||
t.states[provider] = st
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// record accounts one upstream request-path attempt's outcome for
|
||||
// provider: success=false must be EXACTLY isTransient(resp, err) ||
|
||||
// isDeadlineExceeded(err) — the identical classification
|
||||
// limiter.recordProviderAttempt applies (limits.go/retry.go) — never a
|
||||
// wider one; routes_unified.go's attemptRecorder closure computes it
|
||||
// once and feeds both.
|
||||
//
|
||||
// requestBreakerFailureThreshold consecutive failures opens the breaker
|
||||
// for requestBreakerOpenDuration. While open, healthy (below) reports
|
||||
// unhealthy until that window passes; once it has, the NEXT record call
|
||||
// for this provider — success or failure — is what healthy already
|
||||
// exposed as a post-cooldown probe attempt (probed, below): success
|
||||
// closes the breaker outright, failure re-opens it with a doubled
|
||||
// backoff (capped at requestBreakerMaxOpenDuration). This is the same
|
||||
// probe-then-escalate-or-close shape providerState's discovery breaker
|
||||
// uses (registry.go's recordHealthLocked/openBreakerLocked),
|
||||
// reimplemented independently rather than shared — the two track
|
||||
// different outcome streams, and discoveryHealthy's own doc comment
|
||||
// forbids widening it to cover this one.
|
||||
//
|
||||
// Deliberately no single-flight guard on the probe window (unlike
|
||||
// registry.go's tryBeginRefresh inFlight flag for its own, much lower-
|
||||
// volume, background-goroutine probe): this signal sits on the live
|
||||
// request path, where many concurrent requests can observe the same
|
||||
// post-cooldown window at once. Serializing them would need its own
|
||||
// lock-and-CAS dance for a benefit — one broken provider getting at most
|
||||
// one real probe instead of a short burst of them right as its backoff
|
||||
// expires — this per-pod, best-effort signal does not need to pay for.
|
||||
func (t *requestHealthTracker) record(provider string, success bool) {
|
||||
if t == nil || provider == "" {
|
||||
return
|
||||
}
|
||||
st := t.stateFor(provider)
|
||||
st.mu.Lock()
|
||||
defer st.mu.Unlock()
|
||||
now := t.nowFn()
|
||||
|
||||
probed := st.open && !now.Before(st.openUntil)
|
||||
|
||||
if success {
|
||||
st.open = false
|
||||
st.consecutiveFailures = 0
|
||||
st.backoff = 0
|
||||
st.openUntil = time.Time{}
|
||||
return
|
||||
}
|
||||
|
||||
if probed {
|
||||
if st.backoff <= 0 {
|
||||
st.backoff = requestBreakerOpenDuration
|
||||
} else {
|
||||
st.backoff *= 2
|
||||
}
|
||||
if st.backoff > requestBreakerMaxOpenDuration {
|
||||
st.backoff = requestBreakerMaxOpenDuration
|
||||
}
|
||||
st.openUntil = now.Add(st.backoff)
|
||||
return
|
||||
}
|
||||
|
||||
st.consecutiveFailures++
|
||||
if st.consecutiveFailures >= requestBreakerFailureThreshold {
|
||||
st.open = true
|
||||
st.backoff = requestBreakerOpenDuration
|
||||
st.openUntil = now.Add(st.backoff)
|
||||
}
|
||||
}
|
||||
|
||||
// healthy reports whether provider's request-path breaker currently
|
||||
// permits routing to it — feat/failover's hard SKIP gate
|
||||
// (orderedFailoverCandidates, above), distinct from
|
||||
// modelRegistry.discoveryHealthy's deprioritize-only contract. An
|
||||
// unrecorded provider (never failed, or never attempted at all)
|
||||
// reads healthy — the same "never seen a failure" default the discovery
|
||||
// breaker's own zero value already applies (providerState's doc
|
||||
// comment, registry.go).
|
||||
func (t *requestHealthTracker) healthy(provider string) bool {
|
||||
if t == nil || provider == "" {
|
||||
return true
|
||||
}
|
||||
st := t.stateFor(provider)
|
||||
st.mu.Lock()
|
||||
defer st.mu.Unlock()
|
||||
if !st.open {
|
||||
return true
|
||||
}
|
||||
return !t.nowFn().Before(st.openUntil)
|
||||
}
|
||||
Reference in New Issue
Block a user