From 55e44aadca269feaffe905593f0506918a982c0b Mon Sep 17 00:00:00 2001 From: Lukasz Raczylo Date: Thu, 27 Aug 2026 19:07:14 +0100 Subject: [PATCH] docs: correct how discovery refresh is actually triggered The README claimed discovery runs again on discoveryInterval "in the background", which reads as a periodic timer. There is none. maybeRefresh is called from ServeHTTP only, so the interval is checked per incoming request and a refresh is spawned when it has elapsed. That distinction is what an operator actually needs, because it has three consequences the old wording hid: an idle gateway never refreshes at all, a newly published upstream model takes up to discoveryInterval to appear, and there is no manual trigger since the admin API is read-only. The plugin runs inside the Traefik process, so declining to start a periodic goroutine is deliberate rather than an oversight. Documented as behaviour, with the option of lowering discoveryInterval on the providers whose catalogues actually change. Also corrected the two provider config table rows that carried the same implication. Documentation only. --- README.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 17d4212..fbcb7ac 100644 --- a/README.md +++ b/README.md @@ -230,9 +230,9 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`, | `type` | `string` | — | Required: `openai`, `anthropic`, or `gemini`. `openai` also serves any OpenAI-wire-compatible provider (Grok, DeepSeek, a local vLLM/Ollama gateway, ...) via `baseUrl`. | | `baseUrl` | `string` | Per type: `https://api.openai.com`, `https://api.anthropic.com`, `https://generativelanguage.googleapis.com` | Trailing slash trimmed. | | `apiKey` | `string` | — | Secret form: literal, `env:VAR`, or `file:/path` — see [Secret forms](#secret-forms). An `openai`-type provider resolving to an empty key is a valid **keyless** upstream (no `Authorization` header sent) — useful for a local model server with no auth. `anthropic` and `gemini` reject an empty resolved key at construction. | -| `discoveryInterval` | `string` (Go duration) | `1h` | Only meaningful when `discovery: true`. Invalid duration string is a construction error. | +| `discoveryInterval` | `string` (Go duration) | `1h` | Minimum time between discovery refreshes for this provider. Only meaningful when `discovery: true`. Refresh is checked on incoming requests, not by a background timer, so this is a floor rather than a schedule — see [Model routing](#model-routing). Invalid duration string is a construction error. | | `models` | `[]string` | `[]` | Explicit model ids this provider serves. Combined with any discovered ids. | -| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. | +| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup, then no more often than `discoveryInterval` when requests arrive. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. | | `metadataPath` | `string` | `""` (no metadata capture) | Only read by `openai`-type providers. A second endpoint, fetched alongside `discovery`, that reports per-model context length — for example `/api/v0/models` on an LM Studio server. Must start with `/` when set (checked at construction). See [Model metadata](#model-metadata). | | `passthrough` | `*bool` | `nil` (enabled) | `nil` or `true`: this provider's native passthrough route (`/{name}/...`) stays reachable, unchanged from before this field existed. `false`: the route is disabled — `ServeHTTP` treats it exactly like an unconfigured provider name, falling through to the ordinary unknown-route 404 without even reaching authentication. Does not affect the MCP/A2A target proxy, which has its own routing prefix. | | `requestTimeout` | `string` (Go duration) | Inherits the top-level `requestTimeout` (`5m` if that is also unset) | Overrides the global request timeout for this provider alone, including its native passthrough route — see [Request timeout](#request-timeout). An explicit override always wins over the global default. Empty inherits; a value that fails to parse, or parses to zero or a negative duration, is a construction error. | @@ -445,11 +445,32 @@ kept — a bad edit to the file never breaks already-authenticated traffic. - **Discovery**: `GET {baseUrl}/v1/models` for an `openai`-type provider, `GET {baseUrl}/v1/models` for `anthropic`, `GET {baseUrl}/v1beta/models` for `gemini`. Runs once at plugin construction (bounded to 5s per - provider) and again on `discoveryInterval` in the background. A failed + provider), then again no more often than `discoveryInterval`. A failed refresh is logged and keeps the provider's last-known discovered set — **stale-while-error**, never an empty list just because one refresh attempt failed. + **Refresh is request-driven, not a background timer.** The plugin runs + inside Traefik's own process, so it deliberately starts no periodic + ticker of its own: the interval is checked on each incoming request + (`ServeHTTP`), and a refresh is spawned only when it has elapsed. Three + consequences worth knowing: + + - **An idle gateway never refreshes.** With no traffic at all, the + discovered set stays as it was, indefinitely. + - **A new model upstream takes up to `discoveryInterval` to appear** — + an hour by default. Lower `discoveryInterval` on the providers whose + catalogues actually change if you want it sooner; it costs one extra + model-listing call per provider per interval. + - **There is no manual refresh trigger.** The admin API is read-only + (`overview`, `targets`, `usage`, `usage/history`); restarting the + plugin is the only way to force discovery immediately. + + This is why `GET /v1/models` can lag a provider that just published a + model, and it is behaviour, not a fault. Discovery health itself is + visible per provider in the admin dashboard and as + `llmgateway_provider_healthy` on `/metrics`. + ## Model aliases Operator-defined indirection: a client-facing alias id that resolves to a