mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
fix(hot-reload): honest reload docs, unconditional settle, isolated probe budget
Review response (5 Important + 3 folded):
1. README's cacheTTL row/bullet implied per-group scoping; cacheTTL sets
the TTL written when a group's own request populates a shared cache
entry, it never scopes the entry itself. Reworded only, cacheKey
unchanged.
2. README's fallback-counter claim ("used only when redis is not
configured") was wrong: limits.go also falls back during a Redis
outage whenever failOpen is true (the default). Reworded against
limiter.storeIncrBy/storeGet.
3. Dropped the unevidenced "in-flight requests finish on the old
instance" claim; replaced with what TestConfigHotReload actually
observes (zero non-200s across the rewrite window, every run).
4. Real bug: cleanup's settle poll checked for the injected alias to
disappear, which trivially no-ops on the skip path (the alias was
never live, so "absent" was already true on the first check) —
exactly the unguarded window that caused the earlier 404 cascade.
Replaced with settleToBaseline: captures the baseline /v1/models id
set up front, then polls unconditionally (both paths) for 3
consecutive matching polls, bounded 20s. Verified against a live run
that actually hit the skip path.
5. The ~20rps probe loop shared aliceKey/"eng" (1000/min) with most of
this suite's other tests; a budget-exhaustion 429 would look identical
to a reload blip. Added a dedicated hotreload group
(requestsPerMinute: 100000) and hotreload-probe user to
dynamic.yml.tmpl; every request TestConfigHotReload issues now uses
that key.
6. examples/docker-compose.yml still shipped providers.file.filename,
contradicting the new guidance; switched to providers.file.directory.
Verified by actually running the demo stack.
7. waitForModelAliasBounded and the new settle poll now use
doJSONNonFatal (returns an error instead of calling t.Fatalf), so a
transient transport error resets that poll's retry state instead of
aborting the whole test.
8. Reworded the skip-path Skip message: zero failures there proves only
that the stack survived an unreloaded rewrite attempt, not that a
successful reload is blip-free. Re-added the old single-file path to
.gitignore alongside the new directory entry.
Gates: go build/vet/gofmt clean on integration/; root go test -race,
gofmt, vet, yaegi-check all clean; make integration run cold twice, both
exit 0, plus a verbose run that landed on the skip path (the exact
scenario item 4 fixes) with the full suite, including
TestUsersFileHotReload, passing immediately after.
This commit is contained in:
@@ -211,7 +211,7 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`,
|
||||
| `providers` | `[]string` | `[]` (all) | Glob-matched (`path.Match`) against a provider's configured name. |
|
||||
| `models` | `[]string` | `[]` (all) | Glob-matched, exactly as given — no automatic `provider/` prefix stripping. Both the bare and provider-prefixed forms of a model id are checked, so a pattern like `gpt-*` matches a client request for either `gpt-5-mini` or `openai/gpt-5-mini`. |
|
||||
| `cache` | `*bool` | `nil` (inherit) | Overrides the top-level `cache.enabled` setting for this group's requests — see [Caching](#caching). `nil` inherits the global setting; `false` opts the group out even when caching is globally on; `true` opts the group in, but only when the top-level `cache` block is actually configured (a construction error otherwise — there is nothing to inherit `ttl`/`maxBodyBytes` from). |
|
||||
| `cacheTTL` | `string` | `""` (inherit) | Overrides the top-level `cache.ttl` for this group's cache entries — see [Caching](#caching). Empty inherits the global TTL. A set value must be a valid, positive Go duration (`"30s"`, `"5m"`), and requires the top-level `cache` block to be configured — a construction error otherwise, the same "nothing to inherit from" reasoning as `cache: true` above. |
|
||||
| `cacheTTL` | `string` | `""` (inherit) | Sets the TTL written when this group's own request populates a cache entry — see [Caching](#caching). It does not scope entries to a group: cache entries are shared across every group that can reach the model (see the Caching section's "User identity is not part of the key" note, which applies to group identity too), so this controls a write's TTL only, not which group can read the entry. Empty inherits the global TTL. A set value must be a valid, positive Go duration (`"30s"`, `"5m"`), and requires the top-level `cache` block to be configured — a construction error otherwise, the same "nothing to inherit from" reasoning as `cache: true` above. |
|
||||
| `mcpServers` | `[]string` | `[]` (all) | Glob-matched against a configured MCP server name. |
|
||||
| `agents` | `[]string` | `[]` (all) | Glob-matched against a configured agent name. |
|
||||
|
||||
@@ -585,11 +585,16 @@ cached bodies for the same request.
|
||||
enabled — the block omitted entirely, or present with `enabled: false`,
|
||||
both read the same way — is a construction error: there is nothing
|
||||
configured to inherit `ttl`/`maxBodyBytes` from.
|
||||
- **Group TTL override**: `GroupConfig.cacheTTL` overrides the top-level
|
||||
`cache.ttl` for one group's cache entries only, independently of that
|
||||
group's own `cache` setting — an empty value inherits the global TTL, a
|
||||
set value must parse as a positive Go duration and requires the
|
||||
top-level `cache` block to be configured, or construction fails.
|
||||
- **Group TTL override**: `GroupConfig.cacheTTL` sets the TTL written when
|
||||
this group's own request populates a cache entry — it is not a per-group
|
||||
scope. Cache entries are shared across groups, exactly like the "User
|
||||
identity is not part of the key" bullet above: whichever group's request
|
||||
happens to populate (or later refresh) an entry decides that entry's
|
||||
TTL, and a later request from a different group that hits the same
|
||||
entry reads it with the TTL already stored there, unaffected by its own
|
||||
`cacheTTL`. An empty value inherits the global TTL for that group's own
|
||||
writes; a set value must parse as a positive Go duration and requires
|
||||
the top-level `cache` block to be configured, or construction fails.
|
||||
- **Storage**: values are stored via the same hand-rolled RESP2 client
|
||||
(`resp.go`) the limiter's distributed counters use — the identical
|
||||
Redis/Valkey/Dragonfly connection, not a second one. A Redis error
|
||||
@@ -887,9 +892,13 @@ manual reload step. The Kubernetes CRD provider watches `Middleware`
|
||||
resources natively: on a change, Traefik builds a whole new plugin
|
||||
instance from the updated config, lets it warm-fill discovery for every
|
||||
provider with `discovery: true` (see [Model routing](#model-routing)), then
|
||||
swaps it in for new requests. Requests already in flight keep running
|
||||
against the old instance until they finish; only new requests reach the
|
||||
new one.
|
||||
swaps it in for new requests. This repository does not carry evidence of
|
||||
Traefik's exact draining mechanism for requests already in flight on the
|
||||
old instance at swap time. What is verified: the integration suite's
|
||||
sustained-traffic reload test (`TestConfigHotReload`,
|
||||
`integration/config_hot_reload_test.go`) sends requests throughout the
|
||||
whole rewrite-and-propagation window and sees zero non-200 responses, on
|
||||
every run.
|
||||
|
||||
A rebuilt instance keeps some state and resets the rest:
|
||||
|
||||
@@ -899,11 +908,13 @@ A rebuilt instance keeps some state and resets the rest:
|
||||
[Limits and accounting](#limits-and-accounting)) — never from anything
|
||||
specific to one Gateway instance — so a new instance reads the exact
|
||||
counters the old one wrote. The response cache's keys work the same way.
|
||||
- **Resets on a rebuild**: the in-process fallback counters used only when
|
||||
`redis` is not configured, the limiter's and cache's Redis-outage
|
||||
down-latches, and the model registry's discovery cache — a new instance
|
||||
always runs its own full warm-fill before serving, rather than
|
||||
inheriting the old instance's listing.
|
||||
- **Resets on a rebuild**: the in-process fallback counters (used whenever
|
||||
no Redis is configured, and also transparently during a Redis outage
|
||||
when `failOpen` is true — the default; see `limiter.storeIncrBy`,
|
||||
`limits.go`), the limiter's and cache's own Redis-outage down-latches,
|
||||
and the model registry's discovery cache — a new instance always runs
|
||||
its own full warm-fill before serving, rather than inheriting the old
|
||||
instance's listing.
|
||||
|
||||
Two practical implications follow:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user