docs: document the discovery circuit breaker

Add BreakerConfig to the full configuration reference, alongside
RetryConfig and CacheConfig, and add a breaker row to the top-level
Config table. Add a new Provider health section explaining the three
states, the failure classification, the neutral treatment of a
canceled context, and the scope limitation: the signal reflects the
discovery endpoint only and must not be the sole reason to refuse
routing to a provider.

Also document the new healthState and openUntil fields GET
/admin/api/overview returns per provider, in the same sentence that
already lists discoveryEnabled, modelRates, and modelMeta.
This commit is contained in:
2026-08-22 22:56:56 +01:00
parent 92b1af0705
commit e82dabb1dc
+67 -2
View File
@@ -190,6 +190,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
| `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. |
@@ -308,6 +309,14 @@ own is governed purely by their group's — see
| `ttl` | `string` (Go duration) | `5m` | How long a cached response stays valid. Invalid duration string is a construction error. |
| `maxBodyBytes` | `int` | `1048576` (1MiB) | Largest response body still eligible for caching; a larger one is skipped (never stored, never an error). Maximum accepted value is `8388608` (8MiB) — above that is a construction error. |
### `BreakerConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `failureThreshold` | `int` | `3` | Consecutive failed discovery refreshes that open the breaker — see [Provider health](#provider-health-discovery-circuit-breaker). `0` uses the default. Must be between `1` and `100`, or construction fails. |
| `openDuration` | `string` (Go duration) | `1m` | Base backoff a newly opened breaker waits before its first half-open probe; doubles on every further failed probe, capped at `maxOpenDuration`. Invalid or non-positive duration string is a construction error. |
| `maxOpenDuration` | `string` (Go duration) | `30m` | Ceiling on the backoff `openDuration` doubles into. Must be `>= openDuration` and no more than `24h`, or construction fails. |
### `AdminConfig`
| Field | Type | Default | Semantics |
@@ -897,6 +906,58 @@ cached bodies for the same request.
rate-limited) and never fails the request; the upstream call still
happens and the client still gets a correct answer.
## Provider health (discovery circuit breaker)
A per-provider, in-memory, three-state circuit breaker (`closed` →
`open` → `half-open`) driven entirely by `discovery` refresh outcomes —
never by `/v1/chat/completions` or any other request-path traffic. It
exists so a provider whose discovery endpoint fails on every cycle (an
expired API key, a typo'd `baseUrl`, an upstream outage) backs off
instead of being re-probed on every single `discoveryInterval` forever,
and so the admin dashboard can show that provider as visibly broken
instead of indistinguishable from a healthy one.
- **States**: `closed` is normal — refreshes run on the provider's own
`discoveryInterval`, exactly as if this feature did not exist.
`breaker.failureThreshold` (default `3`) CONSECUTIVE failed refreshes
open it; any success in between resets the count to zero. `open` skips
refresh attempts until a backoff window elapses, starting at
`breaker.openDuration` (default `1m`) and doubling on every further
failed probe, capped at `breaker.maxOpenDuration` (default `30m`).
`half-open` is exactly one probe refresh, deciding whether to close the
breaker again (success) or reopen it with a doubled backoff (failure).
- **Never retries more often than plain interval throttling.** The
backoff window is an ADDITIONAL gate on top of `discoveryInterval`, not
a replacement for it — whichever of the two is currently longer wins.
With every default, `breaker.maxOpenDuration` (30m) is shorter than the
default `discoveryInterval` (1h), so under default settings the plain
interval remains the sole binding constraint and a persistently broken
provider is refreshed at exactly the same cadence a deployment without
this feature would already see — never more.
- **Classification**: any refresh outcome with a non-nil error counts as
a failure, including a malformed-`baseUrl` request-build failure — this
is deliberately NOT the same classification `retry` above uses for
deciding whether to retry a request, since a permanently broken
provider must still be able to trip the breaker. A client-canceled
context (plugin shutdown, an outer context canceled upstream) is
NEUTRAL: it counts as neither a failure nor a success, and never resets
or closes an open breaker.
- **A provider that never fails is completely unaffected** — every
default is chosen so `closed` never diverges from plain
`discoveryInterval` gating.
- **Scope**: this reflects the DISCOVERY endpoint only. A provider whose
model listing 401s while its actual completion endpoint works fine
(common behind a proxy that exposes chat but not model listing) is
reported unhealthy here despite being entirely usable for real traffic
— this signal must never be the sole reason to refuse routing to a
provider.
- **Per-pod, not distributed.** Each replica learns a provider's
discovery health independently — a latency optimization, not a
correctness guarantee. Three replicas disagreeing briefly after a
provider starts failing is expected, not a bug.
- **Dashboard**: the Providers tab shows a provider's current state and,
while open, when it will next be probed — see [Admin](#admin).
## Unified vs. passthrough
| | Unified (`/v1/...`) | Native passthrough (`/{provider}/...`) |
@@ -1057,8 +1118,12 @@ or in CI.
per-model minute figures, since nothing displays them; the dashboard's
per-model badge falls back to a day-window-only detail instead), and
`modelMeta` (the resolved context window and cost per known model —
see [Model metadata](#model-metadata)); each alias entry carries its
own `modelMeta` too. `usage`
see [Model metadata](#model-metadata)), and `healthState`/`openUntil`
(the discovery circuit breaker's current state — `"closed"`, `"open"`,
or `"half-open"` — and, while open, when it will next attempt a
half-open probe; see [Provider health](#provider-health-discovery-circuit-breaker)).
`openUntil` is the zero time when `healthState` is not `"open"`. Each
alias entry carries its own `modelMeta` too. `usage`
returns every user's and every group's
current-window counter values — `requestsPerMinute`, `requestsPerDay`,
`tokensInPerDay`/`tokensOutPerDay`, `tokensInPerMonth`/