mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
feat(failover): add cross-provider failover for overlapping models
Add failover.go with the full candidate pipeline. resolveWithCandidates resolves a request exactly like resolve does, then adds every other provider sharing the identical bare model id. orderedFailoverCandidates applies a hard skip for request-path-unhealthy providers, deprioritizes discovery-unhealthy providers without dropping them, fails open when every candidate reads unhealthy, and caps the list at maxAttempts. Add requestHealthTracker, a separate per-pod request-path health signal fed by the same isTransient/isDeadlineExceeded classification recordProviderAttempt already uses. It never widens the existing discovery circuit breaker, per that breaker's own documented scope. Rewrite runMeteredCall (routes_unified.go) to loop over candidates instead of resolving once. Each attempt rebinds the attempt recorder to its own provider, recomputes the cache key before the response is cached, and re-injects the alias echo key an adapter may have deleted on a prior failed attempt. Failover never runs once any byte has reached the client, since statusTrackingWriter already tracks that state. A failure falls through on everything except HTTP 400, matching the brief exactly. A 404 failover logs loudly, naming both providers and the model, since it usually signals a real configuration mistake. Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A single-provider deployment, or one with no overlapping bare model ids, behaves exactly as before regardless of this setting. Add a yaegi-check probe (tools/yaegi-check/main.go) driving two providers under the real interpreter, one failing and one serving, proving the candidate loop runs correctly interpreted. Update README.md with a Failover section and correct the Retry section's stale claim that no cross-provider failover exists. Media routes (images, audio speech, audio transcriptions) do not get failover in this change. Each has its own request-shape handling that needs its own scoped follow-up.
This commit is contained in:
+40
-8
@@ -70,7 +70,13 @@ type Config struct {
|
||||
// at all) resolves to every documented default and behaves exactly
|
||||
// like a deployment with no circuit breaker at all for a provider
|
||||
// that never fails — see validateBreakerConfig's own doc comment.
|
||||
Breaker BreakerConfig `json:"breaker"`
|
||||
// Failover configures feat/failover — cross-provider failover for a
|
||||
// bare model id more than one configured provider serves. See
|
||||
// FailoverConfig's own doc comment (failover.go) for the full
|
||||
// contract; a struct value, not a pointer, for the same
|
||||
// zero-means-default reason as Breaker/Retry/Cache below.
|
||||
Failover FailoverConfig `json:"failover"`
|
||||
Breaker BreakerConfig `json:"breaker"`
|
||||
// Retry is a struct value, not a pointer, because its own Enabled
|
||||
// field is the on/off signal (unlike Redis/Users, where the block's
|
||||
// mere presence is the signal) — so its tag omits "omitempty":
|
||||
@@ -493,18 +499,24 @@ func defaultBodyAdmissionCap() int {
|
||||
const maxExplicitBodyAdmissionCap = 10_000
|
||||
|
||||
// Gateway is the Traefik middleware handler.
|
||||
//
|
||||
// Field order below is fieldalignment-verified (golangci-lint's govet
|
||||
// enable-all, run with -fix against a scratch copy to derive the exact
|
||||
// zero-waste sequence, then hand-applied here so every field keeps its
|
||||
// original doc comment) — see providerState's own doc comment
|
||||
// (registry.go) for the general convention.
|
||||
type Gateway struct {
|
||||
next http.Handler
|
||||
cfg *Config
|
||||
auth *authStore
|
||||
limiter *limiter
|
||||
registry *modelRegistry
|
||||
adapters map[string]providerAdapter
|
||||
next http.Handler
|
||||
// targetClient is the shared, connection-pooled *http.Client the
|
||||
// MCP/A2A target proxy (mcp_a2a.go) issues every upstream request
|
||||
// through — built once via newAdapterHTTPClient, the same constructor
|
||||
// each provider adapter uses for its own client.
|
||||
targetClient *http.Client
|
||||
auth *authStore
|
||||
limiter *limiter
|
||||
registry *modelRegistry
|
||||
adapters map[string]providerAdapter
|
||||
cfg *Config
|
||||
// cache is nil whenever response caching is not configured or not
|
||||
// usable (cfg.Cache.Enabled is false, or true with no config.Redis —
|
||||
// see buildResponseCache, cache.go). Every call site checks for nil
|
||||
@@ -525,7 +537,17 @@ type Gateway struct {
|
||||
// resized afterward, matching a Go channel's own fixed-capacity
|
||||
// contract.
|
||||
bodyAdmission chan struct{}
|
||||
name string
|
||||
// failoverHealth is feat/failover's own per-pod, in-memory
|
||||
// request-path health signal (failover.go), built once, here, by
|
||||
// newGateway. Every method on it is nil-receiver-safe, so a Gateway
|
||||
// assembled directly (bypassing newGateway, as a few older tests do)
|
||||
// degrades to "failover never skips a candidate for request health"
|
||||
// rather than a nil-pointer panic.
|
||||
failoverHealth *requestHealthTracker
|
||||
name string
|
||||
// failover is Config.Failover, validated and resolved once by
|
||||
// newGateway (validateFailoverConfig, failover.go).
|
||||
failover failoverConfig
|
||||
}
|
||||
|
||||
// telemetryStartupOnce keeps the anonymous "plugin loaded" ping to one per
|
||||
@@ -634,6 +656,16 @@ func newGateway(ctx context.Context, next http.Handler, config *Config, name str
|
||||
}
|
||||
g := &Gateway{next: next, name: name, cfg: config, auth: auth}
|
||||
|
||||
// feat/failover: validated once, here — see FailoverConfig's own doc
|
||||
// comment (failover.go) for the default (enabled) and what "single-
|
||||
// provider deployment behaves exactly as before" means in practice.
|
||||
failoverCfg, err := validateFailoverConfig(config.Failover)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
g.failover = failoverCfg
|
||||
g.failoverHealth = newRequestHealthTracker()
|
||||
|
||||
// bodyAdmissionCap: an explicit MaxInFlightBodyRequests always wins
|
||||
// over the self-tuned default (house rule: explicit override always
|
||||
// wins) — see Config.MaxInFlightBodyRequests' own doc comment. 0 (the
|
||||
|
||||
Reference in New Issue
Block a user