diff --git a/README.md b/README.md index a2f0871..f29665e 100644 --- a/README.md +++ b/README.md @@ -193,6 +193,7 @@ config accepts them as YAML, which decodes to the same JSON shape. | `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. | | `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). | | `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. | +| `requestTimeout` | `string` (Go duration) | `5m` | Progress-based upstream request timeout, applied to every provider adapter call, native passthrough, and the MCP/A2A target proxy — see [Request timeout](#request-timeout). **Behavior change for every deployment**: before this field existed, no timeout was set at all, so a provider that accepted a connection and then went silent hung the request forever. A provider's own `requestTimeout` (`ProviderConfig`) overrides this. Empty means the default; a value that fails to parse, or parses to zero or a negative duration, is a construction error. | | `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. | | `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. | | `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. | @@ -232,6 +233,7 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`, | `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. | | `metadataPath` | `string` | `""` (no metadata capture) | Only read by `openai`-type providers. A second endpoint, fetched alongside `discovery`, that reports per-model context length — for example `/api/v0/models` on an LM Studio server. Must start with `/` when set (checked at construction). See [Model metadata](#model-metadata). | | `passthrough` | `*bool` | `nil` (enabled) | `nil` or `true`: this provider's native passthrough route (`/{name}/...`) stays reachable, unchanged from before this field existed. `false`: the route is disabled — `ServeHTTP` treats it exactly like an unconfigured provider name, falling through to the ordinary unknown-route 404 without even reaching authentication. Does not affect the MCP/A2A target proxy, which has its own routing prefix. | +| `requestTimeout` | `string` (Go duration) | Inherits the top-level `requestTimeout` (`5m` if that is also unset) | Overrides the global request timeout for this provider alone, including its native passthrough route — see [Request timeout](#request-timeout). An explicit override always wins over the global default. Empty inherits; a value that fails to parse, or parses to zero or a negative duration, is a construction error. | ### `GroupConfig` @@ -846,6 +848,49 @@ request. is never double-counted. Request counters still increment once per client request, unchanged. +## Request timeout + +Every upstream request this gateway makes — provider adapter calls, native +passthrough (`/{provider}/...`), and the MCP/A2A target proxy — is bounded +by a progress-based timeout: silence ends a request, never elapsed time. + +**Behavior change for every deployment, not an invisible bug fix.** Before +this feature, an adapter's `*http.Client` set no timeout of any kind: a +provider that accepted the connection and then went silent — sent no +headers, or sent headers and then stalled mid-body — hung the gateway +request forever. Default, when neither `requestTimeout` field is set: +**five minutes**. + +- **Two mechanisms, one config value**: time to the first response header + (`Transport.ResponseHeaderTimeout`), and idle time WITHIN the response + body once headers arrive. The second is what catches a provider that + answers promptly and then stalls mid-stream or mid-JSON-body — + `ResponseHeaderTimeout` alone only bounds the wait for headers. +- **Progress-based, not total-duration.** A streaming response resets the + idle timer on every chunk it forwards, so a long generation that keeps + producing tokens completes even if the whole exchange runs for 20 + minutes — a total-duration cap was deliberately rejected for exactly + this reason. Non-streaming responses share the identical mechanism: a + large or slow-but-progressing body is unaffected; only a body that goes + quiet for the full timeout is aborted. +- **Scope and override**: `requestTimeout` (top level) sets the global + default. A provider's own `requestTimeout` overrides it — an explicit + override always wins. Native passthrough for a provider uses that same + provider's resolved value, so a client hitting `/{provider}/...` gets + the identical bound as a request through `/v1/chat/completions`. The + MCP/A2A target proxy has no per-target override; it always uses the + global default. +- **Config form and validation**: a Go duration string (`"5m"`, `"90s"`), + matching `retry.backoff`/`cache.ttl`'s own convention. Empty means the + default. A value that fails to parse, or parses to zero or a negative + duration, is a construction error — unlike `cache.ttl`, zero/negative is + never accepted here as "no timeout": that would silently reintroduce the + exact hang this feature fixes. +- **Classification**: a timeout firing is always logged and reported to + the client as a provider failure (`502`, `server_error`), naming the + provider, and is never classified as a client disconnect + (`context.Canceled`) — the two stay distinguishable in every log line. + ## Caching An opt-in, Redis-backed response cache for the unified, non-streaming