mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
dded3a56fe1df462046011fa38a5bbc6aa8f09c9
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
c1bb9fe87a |
fix(hot-reload): honest reload docs, unconditional settle, isolated probe budget
Review response (5 Important + 3 folded):
1. README's cacheTTL row/bullet implied per-group scoping; cacheTTL sets
the TTL written when a group's own request populates a shared cache
entry, it never scopes the entry itself. Reworded only, cacheKey
unchanged.
2. README's fallback-counter claim ("used only when redis is not
configured") was wrong: limits.go also falls back during a Redis
outage whenever failOpen is true (the default). Reworded against
limiter.storeIncrBy/storeGet.
3. Dropped the unevidenced "in-flight requests finish on the old
instance" claim; replaced with what TestConfigHotReload actually
observes (zero non-200s across the rewrite window, every run).
4. Real bug: cleanup's settle poll checked for the injected alias to
disappear, which trivially no-ops on the skip path (the alias was
never live, so "absent" was already true on the first check) —
exactly the unguarded window that caused the earlier 404 cascade.
Replaced with settleToBaseline: captures the baseline /v1/models id
set up front, then polls unconditionally (both paths) for 3
consecutive matching polls, bounded 20s. Verified against a live run
that actually hit the skip path.
5. The ~20rps probe loop shared aliceKey/"eng" (1000/min) with most of
this suite's other tests; a budget-exhaustion 429 would look identical
to a reload blip. Added a dedicated hotreload group
(requestsPerMinute: 100000) and hotreload-probe user to
dynamic.yml.tmpl; every request TestConfigHotReload issues now uses
that key.
6. examples/docker-compose.yml still shipped providers.file.filename,
contradicting the new guidance; switched to providers.file.directory.
Verified by actually running the demo stack.
7. waitForModelAliasBounded and the new settle poll now use
doJSONNonFatal (returns an error instead of calling t.Fatalf), so a
transient transport error resets that poll's retry state instead of
aborting the whole test.
8. Reworded the skip-path Skip message: zero failures there proves only
that the stack survived an unreloaded rewrite attempt, not that a
successful reload is blip-free. Re-added the old single-file path to
.gitignore alongside the new directory entry.
Gates: go build/vet/gofmt clean on integration/; root go test -race,
gofmt, vet, yaegi-check all clean; make integration run cold twice, both
exit 0, plus a verbose run that landed on the skip path (the exact
scenario item 4 fixes) with the full suite, including
TestUsersFileHotReload, passing immediately after.
|
||
|
lukaszraczylo
|
945f41b9b0 |
docs: kubernetes hot-reload guidance
README Kubernetes section: what a Middleware CR edit actually does (Traefik builds a new instance, warm-fills discovery, swaps it in; requests in flight finish on the old one), what survives a rebuild (Redis-backed counters and cache — windowKey's key never carries instance identity) vs what resets (in-memory fallback counters, down-latches, the discovery cache), why new models via discovery need no config edit, why the users Secret must be a whole-volume mount (subPath never receives a kubelet update) and how the plugin's own 5s mtime poll picks up the resulting symlink swap, and why a file-provider dynamic config needs directory: rather than filename: in Kubernetes (ConfigMap symlink swap) and under Docker Desktop (VirtioFS fsnotify) alike. examples/kubernetes.yaml: annotated the existing (already-correct, no-subPath) users Secret volume mount explaining why subPath would silently break Secret updates, so a future edit doesn't 'simplify' it into one. |
||
|
lukaszraczylo
|
f49c51b341 |
fix(docs,integration): example env completeness, flaky reset, cache key assertion
Task 5 re-review fixes: 1 Important + 3 folded. 1. (Important) examples/kubernetes.yaml was construction-broken on copy-paste: the inline dashboard-admin user resolves env:LLMGW_ADMIN_KEY at construction time regardless of admin.enabled, but the Helm env snippet never listed it. Added it there, alongside a comment on the admin user explaining the env var is required either way. Fixed the two reversed above/below cross-references (kubernetes.yaml:90/:115) and reworded the misleading "self-contained" comment on the inline admin user. Verified with a throwaway round-trip test covering: no env vars set fails, every documented env var set succeeds (admin.enabled staying false as committed), and — the actual bug — every var set EXCEPT LLMGW_ADMIN_KEY still fails even with admin disabled. 2. README: Admin section's "(disabled still 404s)" now points at the same 404/passthroughUnknown wording the section already uses above it. Caching section's "block absent entirely is a construction error" now says "not enabled" — auth.go's actual check is !cfg.Cache.Enabled, which is identically true whether the block is omitted or present with enabled: false. 3. integration/mock/main.go: added POST /flaky/reset (handleFlakyReset), reachable through the "flaky" provider's own native passthrough route, resetting the atomic hit counter on demand. TestRetryFlakyRecovers now calls it at the start of the test, removing the make integration-keep re-run trap the old doc comment warned about instead of fixing. dynamic.yml.tmpl's stale "test 8" comment now reads "task 5". 4. integration_test.go: added redisKeys (docker compose exec redis-cli KEYS, mirroring flushRedis's own pattern) and a TestResponseCache assertion that redis holds at least one llmgw:cache:* key immediately after the miss request. go test ./... -count=1, gofmt, vet, make yaegi-check all pass; make integration exits 0 across two fresh full runs. |
||
|
lukaszraczylo
|
2fe71c25eb |
docs: v0.2 features
README, examples, and a comment update covering v0.2's four features:
- README: new Caching section (key semantics incl. requestedModel +
endpoint, hit/miss header, count-requests-free-tokens, group override,
redis-required) and Admin section (enable flag, admin:true users,
browser key-entry flow, sessionStorage, what's exposed/redacted,
poll-counts-against-quota, unauthenticated-shell). Retry and Image/audio
sections already existed from T3/T4; left as-is. Config reference gains
cache/admin rows at the top level, GroupConfig.cache, UserConfig.admin,
and full CacheConfig/AdminConfig tables. Known limitations gains
images/audio-not-cost-accounted and cache-shares-entries-across-users.
- examples/kubernetes.yaml: adds cache{enabled:true,ttl:5m} (this example
already configures redis) and admin{enabled:false} with a comment on
enabling it, plus an inline admin:true user gated behind that flag.
Round-trips through New() cleanly (verified with a throwaway test).
- examples/docker-compose.yml: adds retry, cache (left disabled — this
minimal demo runs no Redis), and admin (left disabled, with a comment
on enabling it). Round-trips through New() cleanly (verified the same
way).
- admin.go: pluginVersion's doc comment now says to keep it in sync with
the release tag.
|
||
|
lukaszraczylo
|
722e6f3d29 |
fix(retry): retries-after-first semantics, ctx-aware backoff, hardened classification
Critical: attempts now counts retries performed AFTER the first try
(default 1 -> two tries total), not the total try count. The prior
reading made retry:{enabled:true}'s own documented default perform zero
retries. (*retryPolicy).totalTries() is the single conversion point,
clamping any attempts<1 to 1 so do()'s loop bound can never compute <=0
and skip calling call() entirely (no (nil,nil) return path).
Backoff waits are now ctx-aware: sleepFn func(time.Duration) is replaced
by waitFn func(context.Context, time.Duration) bool, whose production
implementation (defaultWaitFn) selects between time.After(d) and
ctx.Done(). A context canceled mid-wait aborts the remaining retries
immediately instead of blocking out the full backoff.
Also: classify a request-build failure (invalid method/URL) as
non-transient via errRequestBuildFailed, since retrying an identically
malformed request only wastes attempts; drop the drain cap to a
dedicated 64KiB constant instead of the 32MiB response cap; fix a
t.Fatal call from a non-test goroutine in the mid-stream-death test;
drop the misleading omitempty on Config.Retry (a struct value is never
'empty' to encoding/json); document the Retry-After >2s-falls-back-to-
backoff behavior and cover it with a direct table test.
README: add the retry config table, a RetryConfig field reference, and
a Retry section documenting the semantics. examples/kubernetes.yaml
gains a retry block alongside its other optional sections.
Gate: go test -race, gofmt, vet, yaegi-check, golangci-lint (incl a
fieldalignment fix on two new test structs) all clean. 295 subtests
pass, 0 failed. integration/ untouched.
|
||
|
lukaszraczylo
|
5a11cc85be |
docs: correct security claims, pipeline diagram, k8s example completeness
Fixes from review, verified against source before rewriting:
- README.md Security notes: "auth events are logged" was false — grepped
auth.go, zero logging calls exist anywhere in it. A lookup miss is a
plain 401 envelope with no log line at all, not "logged without key
material". Also corrected the adjacent providerHTTPError claim: the
struct retains the upstream response body as a field (surfaced
deliberately to the client under error.upstream); only its Error()
string omits the body, and nothing in this codebase currently logs a
providerHTTPError via a bare %v anyway.
- README.md mermaid diagram: the previous version showed a single global
Auth gate ahead of routing. ServeHTTP (llmgateway.go) actually dispatches
on path+method first and calls identify() separately inside each
recognized-route branch; an unrecognized route (passthroughUnknown/404
fallback) never calls identify() at all. Redrawn Route -> per-branch
Auth, with the unauthenticated fallback branching directly off Route.
Also fixed the 401 node's error type label from the nonexistent
"invalid_api_key" to the actual envelope type, "authentication_error"
(llmgateway.go, routes_unified.go). Re-rendered with mmdc before and
after the fix.
- README.md + examples/kubernetes.yaml: narrowed the bare-{} empty-group
claim. Re-verified this session with an isolated, minimal repro
(Traefik file provider only, no docker-compose configs: mechanism
involved) that a bare `default: {}` still fails plugin construction
with the file provider specifically: "expected a map, got 'string'".
Scoped the claim to what is actually verified (file provider, twice)
and explicitly flagged Docker/ECS label providers and the Kubernetes
CRD provider as untested, rather than asserting the file provider is
unaffected or that the CRD provider shares its decode path — the
latter was an unverified inference in the prior text, now hedged.
- examples/kubernetes.yaml: the IngressRoute referenced an undefined
Service (llmgateway-dummy-backend), so the manifest did not actually
apply as advertised. Added the missing Deployment + Service (traefik/
whoami, the same image already exercised end-to-end in
examples/docker-compose.yml this session). Re-validated: 6-document
YAML parses, Middleware's plugin config still round-trips through the
real Config struct and New() (throwaway test, deleted after).
- README.md Known limitations: dropped the standalone path-traversal
bullet — it duplicated content already stated in the "request rejected
before it reaches a provider" bullet immediately above it.
NOTE for the reviewer: the ruling on the bare-{} finding stated "file
provider observed working" and named Docker/ECS label providers as the
affected ones. I re-tested this directly (twice, including a standalone
repro outside the compose demo) and the file provider fails on a bare
{} exactly as originally reported — I could not reproduce "file provider
unaffected" and have not tested the label providers at all. Narrowed the
doc wording to only what I verified rather than asserting either
provider's behaviour without evidence; flagging the discrepancy in case
the review had independent grounds I don't have visibility into.
Gates: go test ./... -count=1, gofmt -l ., go vet ./..., make
yaegi-check — all clean. Also re-ran go test -race, golangci-lint (incl.
gosec), and fieldalignment, matching the standard this task already set;
all clean.
|
||
|
lukaszraczylo
|
08dca6ad98 |
docs: README, kubernetes and compose examples, MIT license
- README.md: what-it-is, quickstart (localPlugins today, catalog form
once published), full config reference table for every Config/
ProviderConfig/GroupConfig/LimitsConfig/UsersConfig/UserConfig/
RedisConfig/ModelPricing/TargetConfig/AgentConfig field, model routing,
limits and accounting semantics, unified vs passthrough differences,
MCP/A2A, security notes, known limitations (streaming buffering under
Yaegi, quota-on-reject nuances, identity-encoding, path-traversal
rejects), Kubernetes deployment notes, and development/catalog-
submission status. Every claim grounded against the actual source read
this session; two draft claims (user-limits "replace" group limits, an
unqualified "go-redis is documented as unsuitable in resp.go/
providers.go") were caught and corrected against the real
buildLimitScopes semantics and an actual grep before committing.
- examples/kubernetes.yaml: Secret (users.json) + Middleware CR +
IngressRoute + a Helm values snippet. The Middleware's plugin config
round-trips through the real Config struct and New() (verified via a
throwaway, since-deleted test), not just YAML-syntax-checked.
- examples/docker-compose.yml: minimal localPlugins-based demo (traefik +
whoami + inline file-provider config via compose's inline configs.content).
Brought up and exercised end-to-end against real OpenAI this session:
auth 401/200, /v1/models, /v1/chat/completions (upstream 401 envelope
with a fake key, proving passthrough+key-injection+wrapping all work),
and the unknown-route 404 envelope. Found and fixed a real bug in the
process: a bare "{}" empty group value fails Traefik's own dynamic-
config decode with "expected a map, got 'string'"; fixed to an
explicit empty list and documented the gotcha in both example files and
the README.
- LICENSE: MIT, copyright 2026 Lukasz Raczylo.
|