mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-21 05:57:53 +00:00
feat: same-provider retry for transient upstream failures
Adds spec §1's opt-in retry engine: RetryConfig (top-level, zero value disabled) and retryPolicy (retry.go), wired into upstreamJSON — the single HTTP chokepoint every adapter's chat/embeddings/listModels call already funneled through. A transient failure (net error, 429, 5xx) retries up to Attempts times with exponential backoff capped at 2s, honoring a qualifying Retry-After header on 429, draining and closing each failed attempt's body before retrying. Never retries a non-429 4xx, a translate error, or after context cancellation. Because upstreamJSON returns before any adapter forwards a byte to the client (forwardStream/forwardJSON run only after it returns), the 'zero bytes reached client' safety rule and the 'streaming retries only the initial exchange' rule both hold by construction — no adapter streaming logic needed to change. buildAdapters(cfg) keeps its v0.1 signature: it derives the retryPolicy from cfg.Retry internally rather than taking it as a parameter, so every existing direct call (15+ across provider_openai_test.go) keeps compiling unmodified. Every v0.1 test passes byte-identically, without edits.
This commit is contained in:
@@ -22,6 +22,7 @@ type Config struct {
|
||||
Agents map[string]*AgentConfig `json:"agents,omitempty"`
|
||||
Users *UsersConfig `json:"users,omitempty"`
|
||||
Redis *RedisConfig `json:"redis,omitempty"`
|
||||
Retry RetryConfig `json:"retry,omitempty"`
|
||||
PassthroughUnknown bool `json:"passthroughUnknown,omitempty"`
|
||||
}
|
||||
|
||||
@@ -77,6 +78,18 @@ type RedisConfig struct {
|
||||
DB int `json:"db,omitempty"`
|
||||
}
|
||||
|
||||
// RetryConfig configures same-provider retry on transient upstream
|
||||
// failures (connection errors, HTTP 429, HTTP 5xx) — spec §1. The zero
|
||||
// value (Enabled: false) disables retry, preserving v0.1 behavior
|
||||
// exactly: every request still makes exactly one upstream attempt.
|
||||
// Attempts and Backoff are validated (and defaulted, when left zero) by
|
||||
// newRetryPolicy (retry.go) only when Enabled is true.
|
||||
type RetryConfig struct {
|
||||
Backoff string `json:"backoff,omitempty"`
|
||||
Attempts int `json:"attempts,omitempty"`
|
||||
Enabled bool `json:"enabled,omitempty"`
|
||||
}
|
||||
|
||||
// ModelPricing overrides the built-in per-model price table.
|
||||
type ModelPricing struct {
|
||||
InputPerM float64 `json:"inputPerM"`
|
||||
|
||||
Reference in New Issue
Block a user