mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
docs: document the request timeout feature and its default
Adds a "Request timeout" section explaining the progress-based design, the two mechanisms, the global/per-provider override rule, and error classification. States plainly that this changes behavior for every deployment, since no timeout existed before. Adds requestTimeout rows to the Config and ProviderConfig reference tables.
This commit is contained in:
@@ -193,6 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
|
||||
| `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
|
||||
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
|
||||
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
|
||||
| `requestTimeout` | `string` (Go duration) | `5m` | Progress-based upstream request timeout, applied to every provider adapter call, native passthrough, and the MCP/A2A target proxy — see [Request timeout](#request-timeout). **Behavior change for every deployment**: before this field existed, no timeout was set at all, so a provider that accepted a connection and then went silent hung the request forever. A provider's own `requestTimeout` (`ProviderConfig`) overrides this. Empty means the default; a value that fails to parse, or parses to zero or a negative duration, is a construction error. |
|
||||
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
|
||||
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
|
||||
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
|
||||
@@ -232,6 +233,7 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`,
|
||||
| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. |
|
||||
| `metadataPath` | `string` | `""` (no metadata capture) | Only read by `openai`-type providers. A second endpoint, fetched alongside `discovery`, that reports per-model context length — for example `/api/v0/models` on an LM Studio server. Must start with `/` when set (checked at construction). See [Model metadata](#model-metadata). |
|
||||
| `passthrough` | `*bool` | `nil` (enabled) | `nil` or `true`: this provider's native passthrough route (`/{name}/...`) stays reachable, unchanged from before this field existed. `false`: the route is disabled — `ServeHTTP` treats it exactly like an unconfigured provider name, falling through to the ordinary unknown-route 404 without even reaching authentication. Does not affect the MCP/A2A target proxy, which has its own routing prefix. |
|
||||
| `requestTimeout` | `string` (Go duration) | Inherits the top-level `requestTimeout` (`5m` if that is also unset) | Overrides the global request timeout for this provider alone, including its native passthrough route — see [Request timeout](#request-timeout). An explicit override always wins over the global default. Empty inherits; a value that fails to parse, or parses to zero or a negative duration, is a construction error. |
|
||||
|
||||
### `GroupConfig`
|
||||
|
||||
@@ -846,6 +848,49 @@ request.
|
||||
is never double-counted. Request counters still increment once per
|
||||
client request, unchanged.
|
||||
|
||||
## Request timeout
|
||||
|
||||
Every upstream request this gateway makes — provider adapter calls, native
|
||||
passthrough (`/{provider}/...`), and the MCP/A2A target proxy — is bounded
|
||||
by a progress-based timeout: silence ends a request, never elapsed time.
|
||||
|
||||
**Behavior change for every deployment, not an invisible bug fix.** Before
|
||||
this feature, an adapter's `*http.Client` set no timeout of any kind: a
|
||||
provider that accepted the connection and then went silent — sent no
|
||||
headers, or sent headers and then stalled mid-body — hung the gateway
|
||||
request forever. Default, when neither `requestTimeout` field is set:
|
||||
**five minutes**.
|
||||
|
||||
- **Two mechanisms, one config value**: time to the first response header
|
||||
(`Transport.ResponseHeaderTimeout`), and idle time WITHIN the response
|
||||
body once headers arrive. The second is what catches a provider that
|
||||
answers promptly and then stalls mid-stream or mid-JSON-body —
|
||||
`ResponseHeaderTimeout` alone only bounds the wait for headers.
|
||||
- **Progress-based, not total-duration.** A streaming response resets the
|
||||
idle timer on every chunk it forwards, so a long generation that keeps
|
||||
producing tokens completes even if the whole exchange runs for 20
|
||||
minutes — a total-duration cap was deliberately rejected for exactly
|
||||
this reason. Non-streaming responses share the identical mechanism: a
|
||||
large or slow-but-progressing body is unaffected; only a body that goes
|
||||
quiet for the full timeout is aborted.
|
||||
- **Scope and override**: `requestTimeout` (top level) sets the global
|
||||
default. A provider's own `requestTimeout` overrides it — an explicit
|
||||
override always wins. Native passthrough for a provider uses that same
|
||||
provider's resolved value, so a client hitting `/{provider}/...` gets
|
||||
the identical bound as a request through `/v1/chat/completions`. The
|
||||
MCP/A2A target proxy has no per-target override; it always uses the
|
||||
global default.
|
||||
- **Config form and validation**: a Go duration string (`"5m"`, `"90s"`),
|
||||
matching `retry.backoff`/`cache.ttl`'s own convention. Empty means the
|
||||
default. A value that fails to parse, or parses to zero or a negative
|
||||
duration, is a construction error — unlike `cache.ttl`, zero/negative is
|
||||
never accepted here as "no timeout": that would silently reintroduce the
|
||||
exact hang this feature fixes.
|
||||
- **Classification**: a timeout firing is always logged and reported to
|
||||
the client as a provider failure (`502`, `server_error`), naming the
|
||||
provider, and is never classified as a client disconnect
|
||||
(`context.Canceled`) — the two stay distinguishable in every log line.
|
||||
|
||||
## Caching
|
||||
|
||||
An opt-in, Redis-backed response cache for the unified, non-streaming
|
||||
|
||||
Reference in New Issue
Block a user