SHOULD-7, round 3, 2026-08-22 review -- a report-correction gap, not a
code defect (the reviewer verified peekPassthroughModel/
scanTopLevelModel and allowsPassthroughPath directly via the repo's own
Yaegi harness). Every group/user this integration suite drove native
passthrough through ("eng", "limited", "hotreload") left models and
passthroughPaths unset, so hasModelRestriction() was always false and
allowsPassthroughPath's non-empty-list branch was never reached -- the
suite itself never exercised security findings 2/3's enforcement code
under a real Traefik+Yaegi process, contrary to an earlier claim.
Added a dedicated "restricted" group (models: ["gpt-mock"],
passthroughPaths: ["v1/chat/completions"]) and its own user (dave),
isolated from every other group/test in this file, plus
TestPassthroughModelAndPathRestriction: an allowed model on the allowed
path (200, proxied for real), a denied model on that same allowed path
(403 -- proves the model peek runs), and an ALLOWED model on a path
outside the allowlist (403 -- proves the path check alone, not the
model check, is what fires there). Verified passing via `make
integration` against the real docker-compose stack.
mocktool's 'tool' mode gains two federated-/mcp JSON-RPC endpoints:
'/' (lenient — every bare, session-less call succeeds, matching 8 of
11 real servers the live probe found) and '/strict' (session-required
— rejects a bare tools/list/tools/call with a JSON-RPC error until a
prior 'initialize' call's Mcp-Session-Id header accompanies it,
matching readitall/fetch's real behavior). A new 'toolstrict'
MCPServers entry in dynamic.yml.tmpl points at '/strict'.
TestFederatedMCP drives POST /mcp against real Traefik+Yaegi: a
federated tools/list must include both tool_echo (bare path) and
toolstrict_echo (reachable only via the plugin's own
bare-first-with-fallback handshake retry, mcp_federation.go's
mcpBackendCall), and a tools/call to toolstrict_echo must route
correctly with the prefix stripped. Closes the review gap ('only
goroutine in the package, zero real-interpreter coverage') the
go-test-only suite left: the handshake fallback, and the plugin's
goroutine fan-out in mcpFederatedToolsList, now both run for real
under Yaegi, not just against go test's in-process mocks.
Verified: make integration — ok github.com/lukaszraczylo/
traefik-llmgateway/integration 40.113s, exit 0, no FAIL/panic/skip in
the full run.
buildLimitScopes previously omitted a user or group scope entirely
whenever that entity had no configured Limits, so a limit-less
user/group never accumulated ANY usage counters. Root-caused live:
the admin dashboard's total kept climbing while every per-user row
stayed zero until an operator added phantom, unreachable
requestsPerDay limits everywhere just to make counters build at all.
buildLimitScopes now always builds a user and a group scope. Limits
still gate enforcement only: evaluateScope already no-ops on a nil
Limits field, so a limit-less scope is counted but never violates.
Update stale doc comments coupling accounting to configured limits
(evaluateScope, the integration alice fixture, routes_media_test.go,
routes_unified_test.go, admin_test.go's admin1 assertion) and add:
- TestBuildLimitScopes_AlwaysBuildsBothScopes: table-driven over
every nil/non-nil user/group limits combination.
- TestAdminUsage_LimitlessUserAndGroup_AccumulateRealTraffic:
end-to-end regression proving a limit-less user and group
accumulate real request/token counters visible via
GET /admin/api/usage.
Review response (5 Important + 3 folded):
1. README's cacheTTL row/bullet implied per-group scoping; cacheTTL sets
the TTL written when a group's own request populates a shared cache
entry, it never scopes the entry itself. Reworded only, cacheKey
unchanged.
2. README's fallback-counter claim ("used only when redis is not
configured") was wrong: limits.go also falls back during a Redis
outage whenever failOpen is true (the default). Reworded against
limiter.storeIncrBy/storeGet.
3. Dropped the unevidenced "in-flight requests finish on the old
instance" claim; replaced with what TestConfigHotReload actually
observes (zero non-200s across the rewrite window, every run).
4. Real bug: cleanup's settle poll checked for the injected alias to
disappear, which trivially no-ops on the skip path (the alias was
never live, so "absent" was already true on the first check) —
exactly the unguarded window that caused the earlier 404 cascade.
Replaced with settleToBaseline: captures the baseline /v1/models id
set up front, then polls unconditionally (both paths) for 3
consecutive matching polls, bounded 20s. Verified against a live run
that actually hit the skip path.
5. The ~20rps probe loop shared aliceKey/"eng" (1000/min) with most of
this suite's other tests; a budget-exhaustion 429 would look identical
to a reload blip. Added a dedicated hotreload group
(requestsPerMinute: 100000) and hotreload-probe user to
dynamic.yml.tmpl; every request TestConfigHotReload issues now uses
that key.
6. examples/docker-compose.yml still shipped providers.file.filename,
contradicting the new guidance; switched to providers.file.directory.
Verified by actually running the demo stack.
7. waitForModelAliasBounded and the new settle poll now use
doJSONNonFatal (returns an error instead of calling t.Fatalf), so a
transient transport error resets that poll's retry state instead of
aborting the whole test.
8. Reworded the skip-path Skip message: zero failures there proves only
that the stack survived an unreloaded rewrite attempt, not that a
successful reload is blip-free. Re-added the old single-file path to
.gitignore alongside the new directory entry.
Gates: go build/vet/gofmt clean on integration/; root go test -race,
gofmt, vet, yaegi-check all clean; make integration run cold twice, both
exit 0, plus a verbose run that landed on the skip path (the exact
scenario item 4 fixes) with the full suite, including
TestUsersFileHotReload, passing immediately after.
Discovery failures are non-fatal (explicit model keeps resolving
offline), and with discovery on, INTEGRATION_REAL=1 runs serve the
gateway's full live catalog from /v1/models instead of one pinned id.
Spec §5 (v0.2): Config.ModelAliases map[string]string, alias-first
resolution precedence ahead of provider-prefix splitting, alias-OR-target
authorization, lazy target resolution with a 404 naming both ids,
/v1/models listing, admin overview alias table (JSON + dashboard), and
construction-time validation (shape, prefix-shadow, explicit-model
collision, chain, empty target).
routes_unified.go/routes_media.go needed no resolution-call-site changes
- both already pass the client's raw requested id through to
registry.resolve, so aliases inherit the __alias echo and multipart
rewrite machinery for free. cache.go needed no changes either - cacheKey
already hashes requestedModel, so two aliases resolving to the same
target already produce distinct cache entries.
Adds registry_test.go validation table + precedence/authz/lazy-404/
listing coverage, a dedicated cache-key split test, chat/embeddings/
images/audio-transcriptions end-to-end tests, an extended admin overview
test, and an integration test covering the openai-type vs anthropic-type
echo asymmetry under real Traefik+Yaegi.
Task 5 re-review fixes: 1 Important + 3 folded.
1. (Important) examples/kubernetes.yaml was construction-broken on
copy-paste: the inline dashboard-admin user resolves env:LLMGW_ADMIN_KEY
at construction time regardless of admin.enabled, but the Helm env
snippet never listed it. Added it there, alongside a comment on the
admin user explaining the env var is required either way. Fixed the
two reversed above/below cross-references (kubernetes.yaml:90/:115)
and reworded the misleading "self-contained" comment on the inline
admin user. Verified with a throwaway round-trip test covering: no env
vars set fails, every documented env var set succeeds (admin.enabled
staying false as committed), and — the actual bug — every var set
EXCEPT LLMGW_ADMIN_KEY still fails even with admin disabled.
2. README: Admin section's "(disabled still 404s)" now points at the
same 404/passthroughUnknown wording the section already uses above it.
Caching section's "block absent entirely is a construction error" now
says "not enabled" — auth.go's actual check is !cfg.Cache.Enabled,
which is identically true whether the block is omitted or present with
enabled: false.
3. integration/mock/main.go: added POST /flaky/reset (handleFlakyReset),
reachable through the "flaky" provider's own native passthrough route,
resetting the atomic hit counter on demand. TestRetryFlakyRecovers now
calls it at the start of the test, removing the make integration-keep
re-run trap the old doc comment warned about instead of fixing.
dynamic.yml.tmpl's stale "test 8" comment now reads "task 5".
4. integration_test.go: added redisKeys (docker compose exec redis-cli
KEYS, mirroring flushRedis's own pattern) and a
TestResponseCache assertion that redis holds at least one
llmgw:cache:* key immediately after the miss request.
go test ./... -count=1, gofmt, vet, make yaegi-check all pass; make
integration exits 0 across two fresh full runs.
Extends the integration suite (task 5) to exercise retry, cache, images/audio, and admin under real Traefik+Yaegi:
- mock/main.go: a /flaky/v1/chat/completions endpoint failing the first
request per process with 503 then succeeding (atomic counter, response
carries mock_attempt for verification); /v1/images/generations returning
a canned OpenAI images response; /v1/audio/speech returning binary bytes
under audio/mpeg; gemini mode's :predict returning canned Imagen
predictions.
- dynamic.yml.tmpl: retry{enabled:true,attempts:1}, cache{enabled:true,
ttl:2m}, admin{enabled:true}; a flaky provider pointed at the new mock
endpoint; an admin:true inline user; alice and admin both given a
generous-but-real per-user limits block so their own request counters
are trackable via the usage API.
- integration_test.go: TestRetryFlakyRecovers (client 200, exactly two
upstream hits proven via mock_attempt), TestResponseCache (miss then
hit, tokensPerDay unchanged across the hit via the admin usage API),
TestImagesGenerations (openai native forward + gemini Imagen
translation), TestAudioSpeech (binary round trip, Content-Type
forwarded), TestAdminDashboard (unauthenticated shell, admin vs
non-admin gate, usage reflecting this suite's own traffic).
make integration exits 0 (verified across multiple fresh runs).
The upstream gateway's model ids carry a uni/ prefix; a provider named
uni made the plugin strip it as a provider prefix. Renamed to real and
listed the full slash-bearing id, exercising bare-id resolution.
Addresses coordinator review of the Task 15 integration suite: 1 critical
ruling, 5 important, 3 folded. make integration now exits 0.
- integration_test.go: TestStreamingIncrementalityProbe restructured —
content assertion (5 deltas + [DONE], in order) always runs; the raw-TCP
timing assertion (unchanged, >=3 gaps >=100ms) is gated behind
INTEGRATION_STREAMING=1 with t.Skipf naming traefik/traefik#10269,
verified to still genuinely fail when re-armed (not silently disabled).
- Makefile: integration-up/integration-wait moved from prerequisites into
the integration recipe body as go test ./... -count=1
ok github.com/lukaszraczylo/traefik-llmgateway 6.065s sub-invocations with per-stage
status capture, so integration-down always runs even when bring-up or
the health wait fails; verified with a synthetic failure. Added
.NOTPARALLEL (the integration targets are stateful).
- translate_gemini.go: fixed the same yaegi multi-value-assignment panic
shape as the prior sse.go fix, on the live max_tokens path. Added a
gemini chat completion assertion to integration_test.go (renamed
TestUnifiedChatAllProviders) so this path now runs under real Yaegi.
- providers.go, translate_anthropic.go: documented the never-wrap
contract on providerHTTPError/translateError (routes_unified.go asserts
their concrete type directly; errors.As panics under yaegi on these).
- integration_test.go: TestCrossReplicaLimits now FLUSHALLs Redis first,
for idempotence under integration-keep.
- mock/main.go: probe's header-terminator search now accumulates a
rolling buffer across reads instead of only checking the latest one.
- Folded: hot-reload deadline 10s -> 20s; dynamic.yml's real-upstream
baseUrl is now a template (dynamic.yml.tmpl) rendered via
INTEGRATION_REAL_BASEURL (default https://llmgw.example.com), not a
hardcoded-only hostname.
See task-15-report.md's addendum for full verification detail.
lukaszraczylo