docs: document the request timeout feature and its default

Adds a "Request timeout" section explaining the progress-based design,
the two mechanisms, the global/per-provider override rule, and error
classification. States plainly that this changes behavior for every
deployment, since no timeout existed before.

Adds requestTimeout rows to the Config and ProviderConfig reference
tables.
This commit is contained in:
2026-08-23 01:46:49 +01:00
parent 5062017cf1
commit 834238d141
+45
View File
@@ -193,6 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
| `requestTimeout` | `string` (Go duration) | `5m` | Progress-based upstream request timeout, applied to every provider adapter call, native passthrough, and the MCP/A2A target proxy — see [Request timeout](#request-timeout). **Behavior change for every deployment**: before this field existed, no timeout was set at all, so a provider that accepted a connection and then went silent hung the request forever. A provider's own `requestTimeout` (`ProviderConfig`) overrides this. Empty means the default; a value that fails to parse, or parses to zero or a negative duration, is a construction error. |
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
@@ -232,6 +233,7 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`,
| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. |
| `metadataPath` | `string` | `""` (no metadata capture) | Only read by `openai`-type providers. A second endpoint, fetched alongside `discovery`, that reports per-model context length — for example `/api/v0/models` on an LM Studio server. Must start with `/` when set (checked at construction). See [Model metadata](#model-metadata). |
| `passthrough` | `*bool` | `nil` (enabled) | `nil` or `true`: this provider's native passthrough route (`/{name}/...`) stays reachable, unchanged from before this field existed. `false`: the route is disabled — `ServeHTTP` treats it exactly like an unconfigured provider name, falling through to the ordinary unknown-route 404 without even reaching authentication. Does not affect the MCP/A2A target proxy, which has its own routing prefix. |
| `requestTimeout` | `string` (Go duration) | Inherits the top-level `requestTimeout` (`5m` if that is also unset) | Overrides the global request timeout for this provider alone, including its native passthrough route — see [Request timeout](#request-timeout). An explicit override always wins over the global default. Empty inherits; a value that fails to parse, or parses to zero or a negative duration, is a construction error. |
### `GroupConfig`
@@ -846,6 +848,49 @@ request.
is never double-counted. Request counters still increment once per
client request, unchanged.
## Request timeout
Every upstream request this gateway makes — provider adapter calls, native
passthrough (`/{provider}/...`), and the MCP/A2A target proxy — is bounded
by a progress-based timeout: silence ends a request, never elapsed time.
**Behavior change for every deployment, not an invisible bug fix.** Before
this feature, an adapter's `*http.Client` set no timeout of any kind: a
provider that accepted the connection and then went silent — sent no
headers, or sent headers and then stalled mid-body — hung the gateway
request forever. Default, when neither `requestTimeout` field is set:
**five minutes**.
- **Two mechanisms, one config value**: time to the first response header
(`Transport.ResponseHeaderTimeout`), and idle time WITHIN the response
body once headers arrive. The second is what catches a provider that
answers promptly and then stalls mid-stream or mid-JSON-body —
`ResponseHeaderTimeout` alone only bounds the wait for headers.
- **Progress-based, not total-duration.** A streaming response resets the
idle timer on every chunk it forwards, so a long generation that keeps
producing tokens completes even if the whole exchange runs for 20
minutes — a total-duration cap was deliberately rejected for exactly
this reason. Non-streaming responses share the identical mechanism: a
large or slow-but-progressing body is unaffected; only a body that goes
quiet for the full timeout is aborted.
- **Scope and override**: `requestTimeout` (top level) sets the global
default. A provider's own `requestTimeout` overrides it — an explicit
override always wins. Native passthrough for a provider uses that same
provider's resolved value, so a client hitting `/{provider}/...` gets
the identical bound as a request through `/v1/chat/completions`. The
MCP/A2A target proxy has no per-target override; it always uses the
global default.
- **Config form and validation**: a Go duration string (`"5m"`, `"90s"`),
matching `retry.backoff`/`cache.ttl`'s own convention. Empty means the
default. A value that fails to parse, or parses to zero or a negative
duration, is a construction error — unlike `cache.ttl`, zero/negative is
never accepted here as "no timeout": that would silently reintroduce the
exact hang this feature fixes.
- **Classification**: a timeout firing is always logged and reported to
the client as a provider failure (`502`, `server_error`), naming the
provider, and is never classified as a client disconnect
(`context.Canceled`) — the two stay distinguishable in every log line.
## Caching
An opt-in, Redis-backed response cache for the unified, non-streaming