mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
ce39b8eb37ab1f5cd0f727dd9439b78ee72030b4
285
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
e82dabb1dc |
docs: document the discovery circuit breaker
Add BreakerConfig to the full configuration reference, alongside RetryConfig and CacheConfig, and add a breaker row to the top-level Config table. Add a new Provider health section explaining the three states, the failure classification, the neutral treatment of a canceled context, and the scope limitation: the signal reflects the discovery endpoint only and must not be the sole reason to refuse routing to a provider. Also document the new healthState and openUntil fields GET /admin/api/overview returns per provider, in the same sentence that already lists discoveryEnabled, modelRates, and modelMeta. |
||
|
lukaszraczylo
|
92b1af0705 |
test(yaegi-check): exercise the discovery circuit breaker interpreted
Add a breaker probe to the yaegi-check gate, alongside the existing handler coverage rather than replacing it. Drives a provider whose /v1/models 401s, the xiaomi production case, through the full closed to open to half-open to closed cycle against the real interpreter. This closes a gap adversarial review found: registry.go added a new interpreted named int32 type with a String method and a new classification call in recordHealthLocked, and nothing exercised either under Yaegi. limits.go's matchesSentinel already documents that this class of divergence between compiled and interpreted behavior is invisible to go test and only an interpreted probe can catch it. The probe configures a short breaker window (2 failures, 3 to 6 second backoff) and a 1 millisecond discovery interval on a dedicated "brk" provider, so the interval floor never binds and the breaker's own timing is what the assertions observe. It proves the breaker opens after sustained failures, suppresses discovery traffic for the whole open window, and closes again once the provider recovers. |
||
|
lukaszraczylo
|
24167b16f6 |
fix(registry): correct breaker cadence, classification, and cancellation
Address five findings from adversarial review of the discovery circuit breaker. All are behavioral corrections to registry.go; none touch the request path. Interval floor (blocking). tryBeginRefresh let an open breaker's own backoff window replace the plain discoveryInterval gate instead of adding to it. Since defaultBreakerMaxOpenDuration (30m) is shorter than defaultDiscoveryInterval (1h), a permanently broken provider was probed roughly twice as often as before this feature existed. The interval gate now applies unconditionally, closed or open alike, so backoff can only ever lengthen the retry cadence, never shorten it. Explicit failure classification. recordHealthLocked reused isTransient's retry-shaped semantics wholesale, including its exclusion of errRequestBuildFailed. A provider whose baseURL is malformed produced that exact error on every discovery cycle and never opened its breaker, because the old code routed it to a successful outcome. Health classification is now explicit: any non-nil, non-canceled error counts as a failure. Neutral cancellation. A context.Canceled outcome no longer counts as success or failure. Treating it as success previously closed an already-open breaker and wiped its backoff whenever a half-open probe was interrupted at shutdown, reporting a still-broken provider healthy. Zero-value safety. A providerState assembled without going through newModelRegistry now behaves like plain interval-only gating instead of opening on the first failure with a zero backoff, matching the type's own documented claim. Backoff overflow ceiling. Repeated doubling with no ceiling on maxOpenDuration could wrap the backoff negative after roughly 28 probes, silently disabling it. validateBreakerConfig now caps maxOpenDuration at 24 hours. Also renames modelRegistry.providerUsable and providerState.usable to discoveryHealthy, and documents that the signal reflects the discovery endpoint only. It must never be the sole reason to refuse routing to a provider whose request path works fine. feat/failover should combine it with a separate request-path signal fed by recordProviderAttempt. Tests: a 24-hour simulated cadence test proves the fixed gate admits no more attempts than plain interval throttling did before this feature. A concurrent test exercises the breaker fields under -race for the first time. Two tests replace a vacuous predecessor and distinguish neutral cancellation from treating it as success. One test proves errRequestBuildFailed opens the breaker. One test proves the zero value never opens. A rewritten healthy-provider test asserts behavior instead of reading the internal openUntil field directly. |
||
|
lukaszraczylo
|
25e73cb585 |
fix(routes): address review findings on the Anthropic Messages route
Preserve usage when response translation fails after a successful, billed upstream call. The route previously discarded already-billed tokens on that path. A client could induce unlimited free usage against any budget via a truncated tool-call response. Reject a streaming request with 400 instead of 501. Check it before the rate limiter runs. A 501 made anthropic-sdk-python retry the request, since it retries any status of 500 or higher. Each retry also counted against the caller's rate limit before this fix. Preserve block order when translating an Anthropic message to OpenAI's shape. A tool_result block before other content must stay before it. OpenAI otherwise rejects the translated conversation. Echo the client's requested model alias in both the passthrough and the translated response paths. The passthrough branch previously returned the upstream's own model id instead. Forward the anthropic-beta header to an anthropic-type upstream through an explicit allowlist. Never forward Authorization, x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own credential injection win regardless. Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads cache_creation_input_tokens and cache_read_input_tokens too. It folds all three into the billed prompt count. This applies to this route, the existing outbound Anthropic path, and the native passthrough route. Extract the metered-request pipeline shared by the unified route and this route into runMeteredCall. Thread an envelope writer through it so each route answers in its own error shape. This removes the duplication the original brief warned against. It also lets the rate-limit and body-admission error paths answer in this route's own shape. Treat a non-boolean "stream" value as a streaming request. A string or number value previously bypassed the streaming check silently. Emit max_completion_tokens instead of max_tokens for OpenAI's reasoning models. Those models reject max_tokens with a 400. Preserve tool_result.is_error by prefixing string content with "Error: ". OpenAI's tool message carries no error field of its own. Log a warning when the thinking field is dropped instead of discarding it silently. Classify a response-translation failure distinctly in the log. An operator must never see a genuine translation bug reported as an upstream connection error. Document the route in README.md. Cover both auth headers, the non-streaming limitation, and which providers translate versus pass through. Add the reviewer's yaegi-check probes to tools/yaegi-check/main.go, adjusted for the fixes above. |
||
|
lukaszraczylo
|
619931b2b2 |
feat(admin): surface provider health state in overview
Add HealthState and OpenUntil to adminProviderView, GET /admin/api/overview's per-provider row. HealthState mirrors the discovery circuit breaker's state: closed, open, or half-open. OpenUntil is the zero time.Time unless the breaker is open. A provider with discovery disabled, or one that has never failed a refresh, always reads closed. This keeps the dashboard's default view unchanged for a healthy deployment. Both fields come from providerSnapshot, read in the same locked pass as the existing lastRefresh/lastErr fields, so a concurrent finishRefresh can never hand the dashboard a breaker state that disagrees with the rest of the row. |
||
|
lukaszraczylo
|
e9ce6b873d |
feat(registry): add per-provider discovery circuit breaker
Add a three-state circuit breaker to providerState: closed, open, and half-open. It is driven only by discovery-refresh outcomes, never by request-path traffic. A provider whose refresh fails breakerThreshold times in a row opens the breaker and backs off before its next half-open probe, instead of retrying every discovery interval forever. A failed probe reopens the breaker with a doubled backoff, capped at breakerOpenMax. A successful probe closes the breaker again, so a recovered provider never stays dead. The failure classification reuses limits.go's isTransient and isDeadlineExceeded, the same rule recordProviderAttempt already applies to request-path attempts. It does not add a second classification. Add Config.Breaker (BreakerConfig), validated and defaulted by validateBreakerConfig. Every default preserves current behavior for a provider that never fails: a closed breaker never changes the existing interval-based refresh gating. Export modelRegistry.providerUsable(name) as the queryable health predicate. A sibling branch (feat/failover) will use it to skip unusable providers before routing. This commit does not touch the request path or routes_unified.go. |
||
|
lukaszraczylo
|
d47dd06a46 |
feat(routes): serve inbound Anthropic Messages API
Add POST /v1/messages, serving Anthropic SDK clients directly instead of outbound-only support. A request resolved to an anthropic-type provider passes through untranslated. A request resolved to any other provider type translates to and from OpenAI's chat-completion shape through the resolved adapter's own chatCompletion, since that method's contract is OpenAI-shape in and out regardless of provider type. The route reuses admitRequest, readAndDecodeUnifiedBody, registry.resolve, withAttemptRecorder, and limiter.account exactly as routes_unified.go does, so groups, model resolution, limits, accounting, and caching semantics stay identical to the unified route. Client-facing errors use Anthropic's envelope shape. A streaming request is rejected explicitly with a 501, never silently answered non-streamed; streaming support is a follow-up branch. Auth reuses authStore.identify and logAuthEvent unchanged. presentedKey already reads both x-api-key and Authorization: Bearer, so no new credential-checking code was needed. Add openAIRequestFromAnthropic and anthropicResponseFromOpenAI to translate_anthropic.go, the mirror of the existing outbound translators, plus their supporting helpers and unit tests. |
||
|
lukaszraczylo
|
9dc67dedbb |
feat: add anonymous usage reporting, matching traefikoidc
Release builds send one anonymous "plugin loaded" ping per process, so the project can see roughly how many installs exist and which versions are in use. The payload is project name, version and timestamp. Nothing else leaves the pod: no identifiers, no hostname, no configuration, no provider names, no keys, no request data. Three things keep it from firing when it should not. Builds from source carry the devPluginVersion sentinel and stay silent, so developer checkouts and every test run never phone home. The ping is sent only after the plugin constructs successfully, so a rejected configuration is never counted as an install. A sync.Once holds it to one per process, because Traefik calls New once per route and oss-telemetry does not deduplicate on the client side. Operators of a release build opt out with DO_NOT_TRACK=1, OSS_TELEMETRY_DISABLED=1, or TRAEFIK_LLMGATEWAY_DISABLE_TELEMETRY=1. README.md documents the full payload and the opt-out; a plugin that phones home without saying so in its own README would be indefensible. oss-telemetry is the plugin's first non-stdlib RUNTIME dependency, which has consequences for the Yaegi gate: - vendor/ is no longer excluded from yaegi-check's GOPATH copy. It was excluded on the reasoning that it held only test-only testify and _test.go files are never interpreted. That reasoning no longer holds: leaving vendor out would make the interpreted copy fail to resolve an import the real plugin tree resolves fine, so the gate would stop representing production. - yaegi-check now stamps a fake release version into its GOPATH copy so New's telemetry gate is PASSED rather than short-circuited, and the vendored package is actually called under the interpreter. Import resolution alone is not coverage; a cross-package call into interpreted third-party code is its own risk. Telemetry is suppressed by env vars set before anything is interpreted, so the gate never emits a real ping. - A new assertion reads the version back through GET /admin/api/overview. Without it a silently failed stamp would leave the dev sentinel in place and the harness would still print OK while covering nothing. Verified by disabling the stamp and watching the gate fail. shouldSendTelemetry is split out from sendStartupTelemetry so the decision is table-testable without a network, since oss-telemetry's endpoint is deliberately not overridable from consuming code. Also brings the version-gated documentation up to date, since the same README edit touches it. The latest release is v1.0.3, so the pre-1.0 experimental warning has met its own stated condition and is replaced by a semver stability statement in README.md and dropped from .traefik.yml's catalog summary. The catalog now serves the released tags (verified: the v1.0.3 download endpoint returns 200), so the "not yet published" and "analyzer has not been run" claims are corrected, the catalog form of the static config is documented as usable, and examples/kubernetes.yaml pins v1.0.3 instead of v0.1.0. |
||
|
lukaszraczylo
|
0346c3ed39 |
fix(registry): log model-id collisions at WARN, not ERROR
A model id served by two configured providers is resolved deterministically by the registry, the winner is named in the message, and no request is affected. Logging it at ERROR put 144 ERROR lines in a healthy pod's boot log on a cluster with overlapping openai, openai-audio, copilot and macstudio catalogs, which is exactly the noise that hides a real failure. Adds Gateway.warnf beside logf and errorf, and a warn field on modelRegistry carrying it. The field is set by the production caller after newModelRegistry returns rather than added as a fourth constructor parameter: 45 call sites pass a logger today, almost all of them tests passing a no-op, and threading a parameter through all of them would bury a one-line severity change in unrelated churn. A nil warn falls back to log, so a registry built without one keeps its current behaviour instead of panicking or dropping the line. registry.warnf takes an already-formatted string so both call shapes stay single-variadic-argument. Yaegi v0.16.1's CFG builder panics on a call to a struct FIELD of variadic func type with two or more variadic arguments, which is the restriction the log field beside it already documents. Verified under the real interpreter, not only compiled: yaegi-check drives a genuine two-provider collision and now prints WARN. The severity test fails if the call site reverts to log, so the level cannot drift back unnoticed. Only this message moves. Genuine failures — discovery errors, metadata capture panics, encoding failures — stay at ERROR. |
||
|
lukaszraczylo
|
5a34c5a52c | merge: federated MCP auth and fan-out hardening (audit round 2) | ||
|
lukaszraczylo
|
de42192426 |
refactor(mcp): restore errors.Is for the oversize-response sentinel
Federation's tools/call oversize branch matched a string marker with strings.Contains instead of wrapping a sentinel. The stated reason was that no gate exercised this path under Yaegi, so errors.Is was unproven here. The caution was misplaced. errors.New returns a compiled *errors.errorString and fmt.Errorf a compiled *fmt.wrapError, so declaring the sentinel variable in interpreted code leaves the entire chain errors.Is walks compiled. Neither documented trap applies: the matchesSentinel trip-wire covers a plugin-declared error TYPE, and the comma-ok trap needs a plugin-declared interface. retry.go has matched errRequestBuildFailed this way through a multi-%w wrap since v0.2.0. The string check was also strictly worse: err.Error() is itself an interpreted-to-compiled dispatch, so it verified nothing extra, and matching on text that partly comes from the backend admits false positives a sentinel cannot have. Rather than settle this by argument, tools/yaegi-check now drives POST /mcp tools/call against a backend whose response exceeds the cap, and asserts the interpreted plugin reports "response too large". Breaking the match makes the gate fail with "upstream error", so the probe has teeth. This closes the objection permanently instead of answering it in a comment. Also in doBackendJSONRPC, check the HTTP status before the size cap. A backend failing with a large HTML error page now reports "upstream returned HTTP 500", which is actionable, rather than sending the operator after a payload-size problem that does not exist. Two comment corrections carried forward from review: the fan-out semaphore is not the same shape as limiter.spawnTokens (spawnTokens drops work via a default case, this one blocks and must never skip a server), and the shared fan-out deadline gives later batches the remaining budget, not a fresh one, so uniformly slow backends yield a partial list. |
||
|
lukaszraczylo
|
452be5151c |
fix(mcp): stop weighting tools/list budget, widen fan-out, split caps
Addresses critical finding 3 and important findings 3-4 from security review round 2, 2026-08-22. Critical finding 3 — the tools/list-per-backend weight charged against the caller's own requests-per-minute/day budget (security audit finding 1b) was not default-preserving: both live groups have an empty MCP allow-list (all 11 servers), so it silently cut every configured requests-per-minute budget by ~11x with no config change, and examples/kubernetes.yaml's shipped requests-per-minute: 10 would 429 the very first tools/list call ever made. Reverted: checkAndCountWeighted removed from limits.go, handleMCPFederated back to a plain checkAndCount. The real amplification stays fully visible to an operator via countTargetRequests' existing per-server 'mcp' target-scope counters — a separate scope from the user/group budget — rather than silently consuming tenant quota. The fan-out semaphore (below) is the actual protection against the amplification finding 1 described. Important finding 3 — the fan-out semaphore (4 concurrent backends) tripled worst-case latency (907ms measured at 300ms/server, was ~300ms) and pushed the hung-backend worst case to ceil(11/4)x20s=60s, past the ~30s timeout many MCP clients use. Raised to 8, and every backend in one tools/list call now shares ONE overall toolsListBackendTimeout deadline (fanoutCtx) instead of each getting a fresh one from its own staggered start — restoring this function's original 'bounds the whole call regardless of server count' invariant even with bounded concurrency. Important finding 4 — the 4MiB response cap (sized for the tools/list fan-out's memory-amplification risk) silently applied to tools/call too, where exactly one backend is ever contacted and a legitimate result (an image, an extracted document) can be much larger. tools/call now uses mcpBackendCallResponseMaxBytes (10MiB, the pre-finding-1c budget); tools/list keeps the 4MiB cap. A response that hits either cap is detected via a cap+1 read (never silently truncated into a confusing parse failure) and reported to the client as 'response too large'. |
||
|
lukaszraczylo
|
3536c32667 |
fix(auth): let a valid key always win over IP throttle
Fixes critical finding 1 from security review round 2, 2026-08-22: identify checked the per-source-IP throttle BEFORE verifying the presented key. This plugin runs behind a Kubernetes Service, and with externalTrafficPolicy: Cluster (this operator's own deployment), r.RemoteAddr is the node's own SNAT address shared by every external caller reaching that node — not a per-tenant address. 20 failed requests from anywhere on the internet sharing that node locked out all 10 live users, including the admin dashboard's own poll. - identify now verifies the key FIRST, unconditionally: a valid key always succeeds before any throttle state is read. Every failure path still records against the caller's IP, but throttling is now pure observability (bounding log volume and tracker memory), never enforcement — an attacker with no valid key already got 'false' either way, so nothing is lost against brute force. - Replaces the per-source-IP tracker's backing store: the general- purpose memoryStore (limits.go) has no size cap and a full mutex-held sweep that scales with tracked-IP count (~83ms at 1M IPs). authFailureTracker is a bounded, two-generation, O(1)-rotation structure capped at 2x50,000 entries (important finding 1). - logAuthEvent now emits a distinct, separately rate-limited 'auth throttle:' line the first time a source crosses the limit (important finding 5), and both the routine and throttle-engaged lines report how many events they suppressed since the last line actually written, so several distinct attacker IPs in one window no longer collapse into a single indistinguishable line (important finding 6). - The success path no longer touches the throttle tracker's mutex at all (important finding 2). |
||
|
lukaszraczylo
|
d2a0c4860f |
fix(auth): reject duplicate and empty user names
Fixes finding 5 from the 2026-08-22 security audit: newAuthStore and
replaceFileUsers rejected duplicate API keys but not duplicate names, and
counters are keyed on u.name (routes_unified.go), so two users sharing a
name silently shared llmgw:user:{name}:* counters and both appeared in
buildAdminUsage; an empty name collapsed every such user into
llmgw:user::*.
- buildEntry now rejects an empty user name outright — the source of
both bugs, since duplicate names and an empty name are the same root
cause (two config entries resolving to the same counter key).
- newAuthStore and replaceFileUsers each reject a duplicate name WITHIN
their own source (two inline users, or two file users) — but not
across sources: a file user overriding an inline user of the same name
stays intentional and unaffected, per replaceFileUsers' own documented
merge semantics.
- Deliberately does NOT extend configNamePattern's character
restriction to user names: that pattern exists because provider/
mcpServers/agents names are embedded as URL path segments, which a
user name never is here. Restricting the charset further risked
rejecting real operator-chosen names this round has no visibility
into; the live users file's steve-cw/kevinsandom already pass the
non-empty check unaffected.
Also adds TestDefaultPreserving_LiveClusterShape_LoadsAndServesIdentically
(admin_test.go), modeling the live cluster's 5 friends + 4 home users and
11 MCP servers to prove all five fixes in this round load and serve
identically at that scale.
|
||
|
lukaszraczylo
|
800f63ac53 |
fix(admin): chunk usage batch reads to bound connection contention
Fixes finding 4 from the 2026-08-22 security audit: buildAdminUsage sent every user's, group's, and the total scope's counters (8 keys each) to limiter.currentUsage in ONE storeGetMulti round trip. With 1,000 users that is ~8,000 GETs pipelined through the single mutex-guarded store connection, during which every concurrent checkAndCount from live LLM traffic queues behind it (up to respCallTimeout) — and the admin dashboard polls this endpoint every 5s. Add chunkedCurrentUsage: splits the scopes slice into batches of at most adminUsageChunkScopes (200) before calling currentUsage, letting live traffic interleave between chunks while each chunk still batches its own scopes in one round trip. At or under 200 scopes (the live cluster's 9 users) this makes exactly one call, identical to today's behavior — the payload shape is unchanged, so no webui coordination is needed. |
||
|
lukaszraczylo
|
87ebee9003 |
fix(auth): throttle repeated auth failures, rate-limit failure log
Fixes finding 3 from the 2026-08-22 security audit: authStore.identify had no counter, no backoff, and no lockout on failed attempts, and logger.go wrote one unbuffered stderr line per attempt including the caller's RemoteAddr. An unauthenticated attacker got unlimited digest guesses at line rate and could drive synchronous stderr writes on the shared Traefik ingress. - Add per-source-IP failure throttling to identify, reusing the existing in-process memoryStore (limits.go) as authStore's own dedicated instance rather than a new dependency: authFailureLimit (20) failed attempts per authFailureWindow (1 minute) throttles further attempts from that IP outright, including a subsequently valid key, and resets once the window elapses. - clientIP reads r.RemoteAddr only, never a client-supplied header, matching the same trust boundary the rest of this package already assumes (routes_passthrough.go strips X-Forwarded-*/X-Auth-* before any outbound request). - Rate-limit logAuthEvent's failure line via authStore.shouldLogAuthFailure, the identical technique limiter.logStoreError already applies to store errors. Successful-auth logging stays unbounded (bounded by legitimate, already-authenticated traffic elsewhere). |
||
|
lukaszraczylo
|
8bc54d15b5 |
fix(mcp): recover panics in tools/list fan-out goroutines
Fixes finding 2 from the 2026-08-22 security audit: the per-server goroutines in mcpFederatedToolsList carried only defer wg.Done(), unlike every other off-request goroutine in this package (limiter.spawn, modelRegistry.refreshProvider/captureModelMetadata), which all guard against a panic crashing the whole shared Traefik process. This codebase has repeatedly hit interpreter-level panics (registry.go reflect.Set, cache.go CFG index-out-of-range, providers.go errors.As), so an unrecovered panic here is not hypothetical. Add the same defer-recover pattern used at the other three sites: a panicking backend now degrades to "that server failed" and is logged, exactly like an HTTP error from it, instead of taking down the process. |
||
|
lukaszraczylo
|
26ff1a8b5b |
fix(mcp): bound federation tools/list fan-out and weight admission by backend count
Fixes finding 1 from the 2026-08-22 security audit: mcpFederatedToolsList spawned one unbounded goroutine per allowed MCP server, each able to issue up to 4 sequential HTTP calls reading up to 10MiB per response, while checkAndCount charged the caller only 1 request regardless of how many backends the fan-out actually contacted. With the 11 production servers, one client tools/list call could drive up to 44 outbound requests and hold ~110MiB concurrently, all for the cost of a single request against the caller's own rate limit. - Bound the fan-out to mcpFederatedFanoutConcurrency (4) concurrent backends via a buffered-channel semaphore, the same shape limiter.spawnTokens already uses. - Add limiter.checkAndCountWeighted so a tools/list call charges len(allowedMCPServerNames) requests against the caller's requests-per-minute/day budget instead of undercounting the fan-out as one; every other route keeps weight 1 via checkAndCount's own unchanged wrapper. - Shrink the per-backend response cap from maxRequestBytes (10MiB, sized for a client's chat/embeddings body) to mcpBackendResponseMaxBytes (4MiB, sized for a JSON-RPC reply), bounding worst-case concurrent memory to 4 x 4MiB = 16MiB. |
||
|
lukaszraczylo
|
8ab617767c |
fix(limits): cut body-admission bounds to [8,64] on measured heap cost
Round 3 sized these from the isolated json.Unmarshal peak (~51.5MB). Measured end-to-end through the real handler it is ~80-108MB per slot, so a 256 ceiling allows a 21GB peak under adversarial load — above any realistic pod budget, and the reference deployment limits Traefik to 512Mi. The slot is now held only across read+decode, so a smaller bound costs legitimate throughput almost nothing. |
||
|
lukaszraczylo
|
f405340d32 |
test(integration): exercise passthrough model/path restriction under Yaegi
SHOULD-7, round 3, 2026-08-22 review -- a report-correction gap, not a
code defect (the reviewer verified peekPassthroughModel/
scanTopLevelModel and allowsPassthroughPath directly via the repo's own
Yaegi harness). Every group/user this integration suite drove native
passthrough through ("eng", "limited", "hotreload") left models and
passthroughPaths unset, so hasModelRestriction() was always false and
allowsPassthroughPath's non-empty-list branch was never reached -- the
suite itself never exercised security findings 2/3's enforcement code
under a real Traefik+Yaegi process, contrary to an earlier claim.
Added a dedicated "restricted" group (models: ["gpt-mock"],
passthroughPaths: ["v1/chat/completions"]) and its own user (dave),
isolated from every other group/test in this file, plus
TestPassthroughModelAndPathRestriction: an allowed model on the allowed
path (200, proxied for real), a denied model on that same allowed path
(403 -- proves the model peek runs), and an ALLOWED model on a path
outside the allowlist (403 -- proves the path check alone, not the
model check, is what fires there). Verified passing via `make
integration` against the real docker-compose stack.
|
||
|
lukaszraczylo
|
03fe1d6589 |
docs(readme): document body-admission limit and the narrower unified cap
Round-3, 2026-08-22 review (SHOULD-6): maxInFlightBodyRequests, the new 503, and the unified routes' narrower 4MiB decode cap appeared nowhere in the README -- a gap on a public Catalog plugin. Also corrected the now-stale "images/generations and audio/speech share the unified routes' 10MiB JSON cap" line: the unified chat/embeddings routes moved to 4MiB while the media routes kept 10MiB, so they no longer share a cap at all. Added: - maxInFlightBodyRequests to the top-level Config reference table. - A new "Body-admission limit" subsection under Limits and accounting, covering the semaphore's scope (read+decode only, not the whole request), the 503+Retry-After behavior, the self-tuning default, the 4MiB unified cap, and the explicit note that native passthrough is not gated by this at all (streams rather than decodes). |
||
|
lukaszraczylo
|
64126db7d4 |
docs(passthrough): drop out-of-scope "." rejection, correct %25 comment
Two doc/scope corrections from the round-3, 2026-08-22 review (FOLDED +
SHOULD-5), no behavior change to the traversal bypasses finding 4 itself
fixed:
- hasTraversalSegment rejected an exact "." segment alongside "..",
outside finding 4's actual scope: "v1/x/./y" is a legal, non-traversal
path and was silently getting 400'd. Dropped the "." check; only ".."
is rejected again, matching every other traversal rule this function
enforces.
- The double-encoding rejection's own comment claimed it "rejects an
intentionally double-encoded %25" -- overstated. "%25" is the correct,
single, standards-conforming encoding of a literal "%" (RFC 3986);
after exactly one unescape pass, a legitimate single-encoded literal
"%" is byte-for-byte indistinguishable from a genuine double-encoding.
The behavior (fail closed) is still correct and unchanged -- refusing
an occasional literal "%" is a far smaller cost than admitting a real
double-encoded traversal -- but the comment now says so honestly,
including the concrete false-positive example ("v1/100%25done" decodes
to "v1/100%done" and is rejected even though it names a real,
non-malicious resource).
Tests: added "v1/x/./y" (must not 400) and "v1/100%25done" (the
documented false positive, still 403s) to hasTraversalSegment's table.
|
||
|
lukaszraczylo
|
c69210fd91 |
fix(routes): release body-admission slot after decode, not whole handler
Fix an availability regression the round-2 body-admission semaphore shipped (MUST-1, round 3, 2026-08-22 review): `defer release()` spanned the WHOLE handler in runUnified and all three media handlers, including the upstream round trip and response streaming. The stated goal was bounding decoded-body memory during the read+json.Unmarshal step; the shipped code instead bounded total in-flight requests, since an LLM completion is I/O-bound for seconds to minutes. Measured, zero-config, 400 concurrent chat requests: 220 succeeded, 180 got 503, on a cap a 2-vCPU pod would size at 16. Narrowed the held window to read+decode only, via new helpers that do the acquire/decode/release internally and return before the caller ever resolves a model or calls the adapter: - routes_unified.go: readAndDecodeUnifiedBody (runUnified no longer acquires/defers at all). - routes_media.go: decodeMediaJSONRequest now claims/releases its own slot (images.generations, audio.speech); a new readAdmittedCapped helper does the same around audio.transcriptions' readCapped call — narrower still, since that route never json.Unmarshal's into a map[string]any at all. Every release still goes through `defer` inside these narrow helpers, not a bare post-call statement, so a slot is released exactly once on every path including a panic unwinding through the read/decode itself. Since the held window shrank from "whole request" (seconds-minutes) to "one decode" (low milliseconds), raised the self-tuned default: bodyAdmissionCapPerCPU 8->24, floor 16->32, ceiling 128->256 — justified in llmgateway.go's doc comment against the measured adversarial-shape decode peak (~51.5MB LIVE heap, ~189MB TotalAlloc for a 4MiB adversarial body), not the old, no-longer-accurate ~170MB whole-pipeline estimate. Also clamped an explicit Config.MaxInFlightBodyRequests override to 10,000 (previously accepted verbatim, so e.g. a fat-fingered MaxInt would have silently disabled the bound). Corrected maxUnifiedRequestBytes' own DoS-math doc comment (SHOULD-3): the prior "~11x general case" framing was wrong — a realistic single inline base64 image measures ~1.0x (no amplification at all), while the adversarial many-small-values shape is what actually amplifies (12.3x LIVE / ~45x TotalAlloc, both now stated with real measured numbers instead of a single misleading multiplier). Documented two real, previously-silent behavior changes from finding 1a's admission-before-decode reordering (SHOULD-4): an over-budget caller whose request would otherwise have 400'd/404'd now gets 429 instead, and a malformed/unroutable request now consumes rate-limit budget it previously did not. Both are arguably improvements but were not visible from the diff alone. Replaced the sequential zero-config admission test, which could not actually exercise contention (MUST-2: the reviewer reran its exact shape with MaxInFlightBodyRequests=1 and it still passed), with a genuinely concurrent one: n = defaultBodyAdmissionCap()+20 real goroutines against a gated upstream, asserting (a) zero 503s across all n and (b) the semaphore holds zero slots while all n are parked in the upstream -- the regression's own regression test. |
||
|
lukaszraczylo
|
60036c43df |
fix(passthrough): strip dangerous upstream response headers
Close the response-header relay hole (security review finding 5, round 3, 2026-08-22): proxyUpstream copied every upstream response header to the client except hop-by-hop ones -- an upstream provider, or an in-cluster MCP/A2A target reached with no credential trust boundary of its own, could set Set-Cookie on the gateway's own origin (the same origin serving the WAN-exposed /admin dashboard), or return Access-Control-*/Strict-Transport-Security/Content-Security-Policy headers to rewrite the browser's security or CORS policy toward that origin. Added dangerousResponseHeaders (exact-match: Set-Cookie, Strict-Transport-Security, Content-Security-Policy) and dangerousResponseHeaderPrefixes (Access-Control-*), stripped in proxyUpstream right where hop-by-hop headers already are -- so both callers (handlePassthrough here, handleTargetProxy in mcp_a2a.go) get the fix for free. A deny-list, not an allowlist: grepping this repo finds it only ever reads Content-Type and Retry-After off a response (the latter only for the unified routes' own internal retry decision, never from passthrough). Everything else -- including provider rate-limit telemetry headers a client's own SDK commonly reads -- is opaque to this gateway but real passthrough transparency; an allowlist would silently break that for every header this package doesn't already know to name. This mirrors the request side's own already-hardened deny-list shape (dangerousClientHeaders/providerCredentialRetargetHeaders). Test: an upstream setting all four dangerous headers plus Content-Type, Retry-After, and an ordinary custom header proves the four are stripped while the rest reach the client unchanged. |
||
|
lukaszraczylo
|
ff819e01d7 |
fix(passthrough): catch path-parameter, backslash, and double-encoded traversal
Close three further traversal bypasses hasTraversalSegment's single-pass
decode-and-split missed (security review finding 4, round 3,
2026-08-22), each a real upstream convention:
- Path parameters ("..;/x", the Tomcat/Spring RFC 3986 Section 3.3
convention): a segment's identity is everything before its first ";".
Every segment now has its ";"-suffix stripped before the traversal
comparison.
- Backslash normalization ("..%5c..%5cx", decoding to "..\..\x"):
Windows/.NET and other backslash-normalizing upstreams treat "\" as an
equivalent separator. The decoded string now has every "\" replaced
with "/" before splitting.
- Double-encoding ("%252e%252e/x"): one unescape pass decodes this to
"%2e%2e/x" -- still containing "%". Rather than decode a second time
(which invites an unbounded decode loop), a rest that still contains
"%" after the one legitimate unescape pass is now rejected outright.
Also now rejects an exact "." segment, not just "..", closing the same
class of ambiguous-normalization bypass on a self-reference segment.
Literal ".." and single-encoded "..%2f" keep their existing, already
correct behavior; a legitimate encoded "/" within one segment (a2Fb) and
a genuine, non-traversal use of ";" on an unrelated segment are both
still forwarded.
Tests: hasTraversalSegment's table gains one case per new bypass plus
adjacent non-traversal cases proving the fix doesn't over-reject, and
the end-to-end ServeHTTP traversal test gains the same three variants
driven through the full request path.
|
||
|
lukaszraczylo
|
d70e5fc1df |
fix(auth): decode rest before matching the passthrough path allowlist
Close the %2f allowlist bypass (security review finding 3, round 3,
2026-08-22): allowsPassthroughPath and hasTraversalSegment (routes_
passthrough.go) reasoned about rest asymmetrically -- the traversal check
decodes it first, the allowlist matcher ran path.Match against the raw
escaped form. path.Match("v1/*", "v1/chat%2fcompletions") reports true
(one escaped segment, no literal "/" to split it), while the decoded
equivalent, "v1/chat/completions", is two segments and does not match
"v1/*". A group restricted to PassthroughPaths: ["v1/*"] therefore let
"/openai/v1/fine_tuning%2Fjobs" through -- a path an upstream that
itself decodes %2F would read as a different, unauthorized endpoint.
allowsPassthroughPath now url.PathUnescape's rest before matching -- the
same decode hasTraversalSegment already applies -- and denies a rest
that fails to unescape at all, mirroring hasTraversalSegment's own
fail-closed rule for malformed encoding. The empty-pattern fast path
(matchesGlob's own empty-means-all contract) is checked first, before
any decoding: a group with no PassthroughPaths configured -- every live
group today -- keeps allowing every request exactly as before, malformed
encoding included; hasTraversalSegment (unconditional, later in
handlePassthrough) remains the gate for that case.
Tests: a direct unit test proving the decoded three-segment form is
denied (both via the encoded and literal spellings), a single-segment
encoded value still matches, malformed encoding denies for a restricted
group and is unaffected for an unrestricted one, and an end-to-end
ServeHTTP test reproducing the exact finding scenario.
|
||
|
lukaszraczylo
|
a221b4d2eb |
fix(passthrough): fail closed on a model peek that never closes the object
Close the residual gap in the duplicate-"model" fix (security review
finding 2, round 3, coordinator ruling): round 2 made a duplicate
top-level "model" key WITHIN the 64KiB peek window fail closed, but a
client can still pad the body so a second "model" sits just PAST
maxModelPeekBytes -- `{"model":"cheap", <64KiB padding>,
"model":"expensive"}`. The peek reports "cheap" (the only key it saw),
the body is forwarded byte-for-byte, and the upstream (last-wins on
every mainstream JSON parser) executes "expensive".
scanTopLevelModel now also reports whether it actually confirmed the
top-level object's closing '}' within the scanned window (closed).
peekPassthroughModel combines that with its own knowledge of whether the
read hit the byte cap (more of the body remains unread) to report
truncated: true only when both hold, meaning a later duplicate cannot be
ruled out. handlePassthrough denies (403, a message distinct from "model
could not be determined") whenever truncated is true, checked before
even looking at the found model -- a found model under truncation can
never be trusted.
Scoped exactly as ruled: only affects a group with a non-empty Models
list (grp.hasModelRestriction) whose body is large enough to hit
maxModelPeekBytes -- an unrestricted group never calls
peekPassthroughModel at all, unchanged from round 2 and pinned by a new
test. No live group is model-restricted today. The unified /v1/* route
remains the supported path for a legitimately large restricted-group
body.
Tests: the coordinator's own exploit body end-to-end (403, upstream
never called); the general non-duplicate case (any restricted-group body
that never closes within the window denies, not just a provably
duplicate one); the unrestricted-group-unaffected case explicitly, using
the identical exploit body; and closed's own truncation-tracking table
against scanTopLevelModel directly. One pre-existing test
(TestHandlePassthrough_BodyIntegrity_ByteIdentical_LargeBody) used a
restricted group incidentally while testing an unrelated property (byte-
for-byte forwarding) -- switched to an unrestricted group so it tests
only what its name says.
|
||
|
lukaszraczylo
|
f736b231a3 |
fix(routes): admit before decode, bound in-flight bodies, narrow unified cap
Close a HIGH-severity OOM path on the unified and media JSON routes (security review finding 1). Three parts: - checkAndCount now runs via a new admitRequest helper BEFORE the request body is read at all, on both routes_unified.go's runUnified and every routes_media.go handler. The rate-limit scopes need only the already-resolved user/group, so a caller already over their requestsPerMinute is refused without paying for the read+decode of a body that was always going to be discarded. Measured: an encoding/json decode into map[string]any costs ~11x the wire body in live heap, so the prior ordering let an already-limited caller burn ~110MiB per request before ever seeing the 429. - A new bounded, buffered-channel semaphore (Gateway.bodyAdmission, mirroring limits.go's spawnTokens shape) caps how many unified/media requests may concurrently hold a decoded body in memory. Acquired non-blockingly at each route's entry via acquireBodyAdmission; on exhaustion, returns 503 with Retry-After rather than blocking. Sized by defaultBodyAdmissionCap (GOMAXPROCS * 8, clamped to [16, 128]); Config.MaxInFlightBodyRequests, when set, always overrides the self-tuned default. - The unified chat/embeddings routes now decode against a new, narrower maxUnifiedRequestBytes (4MiB) instead of the shared maxRequestBytes (10MiB), which the media JSON routes, audio/transcriptions' multipart cap, and the MCP federation JSON-RPC routes keep unchanged. 1MiB (the audit's own first suggestion) was evaluated and rejected: this package documents and exercises inline base64 image_url content parts in chat messages (translate_anthropic.go, translate_gemini.go), and a single moderately-sized photo commonly exceeds 1MiB once base64-encoded. 4MiB preserves that documented single-inline-image case while still cutting worst-case per-request decode amplification from ~110MiB to ~44MiB. routes_media.go's resolveMediaRequest is split: model resolution stays (renamed resolveMediaModel), the limit check moves into the shared admitRequest. Tests: admission-before-decode is asserted directly (a 429'd request's body is never Read), semaphore exhaustion is asserted deterministically by filling the channel directly, the self-tune/override contract and its clamp bounds are pinned, and a zero-config run of ordinary traffic is asserted to see no new 503s. A dedicated test proves the chosen 4MiB cap still accepts a realistic ~1.5MiB inline image — the case 1MiB would have broken. |
||
|
lukaszraczylo
|
bf33732123 |
perf(registry): cache the encoded /v1/models response per group
GET /v1/models cost 15.9ms, 21.3MB, 552,884 allocs per call at the
live 1,033-model catalog (registry.go's listFor plus routes_unified.go's
json.Encoder), uncached — and OpenAI SDK clients poll this endpoint per
session, so an unchanged catalog was being re-walked and re-marshaled
on every request.
Add modelRegistry.modelsJSON(grp), caching the fully encoded
{object:list,data:[...]} body per (*group, modelsGen). modelsGen
is an atomic counter bumped by a new modelRegistry.finishRefresh
wrapper around providerState.finishRefresh — the only two places that
ever mutate a provider's discovered set or discovery-captured metadata
(warmFill, refreshProvider) already funnel through providerState.
finishRefresh once per refresh cycle, so wrapping it is sufficient to
invalidate the cache on every possible change, including a
metadata-only update on an otherwise-failed refresh (captureModelMetadata
runs unconditionally alongside finishRefresh in both call sites).
Bumped unconditionally rather than only on success, since a failed
listModels fetch can still pair with a successful metadata capture.
Cached per *group pointer: authStore builds every group once, at
construction, and never rebuilds or replaces that map afterward — a
config reload constructs an entirely new Gateway (and modelRegistry)
with its own empty cache, so a *group key can never alias a different
group's catalog either within one registry's lifetime or across a
reload.
routes_unified.go's handleModels call site shrinks to a single write of
modelsJSON's result, replacing the previous inline listFor+json.Encode;
routes_unified.go is otherwise untouched (owned by a sibling change in
this same effort).
|
||
|
lukaszraczylo
|
90eb58171e |
perf(redis_store): send EXPIRE only once per key per process lifetime
31 of 74 commands per request in the interpreted-harness audit were EXPIRE, emitted unconditionally after every INCRBY. Deleting EXPIRE entirely (upper bound) measured 74->43 cmds/req, p50 867.9->697.9us (-19.6%), conc=16 throughput 1,477->2,336 req/s (+58%), allocs -26%. Route chosen: a bounded first-touch-per-key cache (expireOnce), not EVALSHA+Lua. windowKey (limits.go) already embeds each counter's own time bucket in its key string, and every TTL callers pass is deliberately far longer than its window's own natural length purely as a stale-key GC margin (limits.go's own doc comment on dayWindowTTL/monthWindowTTL/etc.), not a sliding-window mechanism — so once EXPIRE has been sent for an exact key, every further INCRBY against that SAME key happens well within its own bucket's window, comfortably inside the TTL's margin. Skipping the redundant resend cannot leave a key un-expiring: a bucket's key simply stops being written once its window rolls over (a new bucket string takes its place). expireOnce is bounded (expireOnceCacheMax=4096, FIFO eviction) so a long-lived process's ever-rolling buckets cannot grow it unboundedly; an evicted key just gets one redundant-but-harmless EXPIRE again, exactly as if never touched before. incrBy/incrMulti/incrAndGetMulti now build a variable-stride command list (each entry contributes its own INCRBY, plus EXPIRE only when expireOnce(key) is true) with an explicit incrReplyIdx/base tracking each entry's actual reply position — no longer a fixed i*2 stride, since a skipped EXPIRE shifts every later command left by one. EVALSHA+Lua was rejected as the shippable route here: it would rewrite every command-building and reply-index path, requires NOSCRIPT fallback handling, and every existing wire-level test (dozens of INCRBY+EXPIRE-scripted fakeRESPServer cases) would need rebuilding around SHA1 digests for no additional correctness benefit over the simpler first-touch cache. |
||
|
lukaszraczylo
|
2309a59b40 |
perf(resp): pool redis connections instead of one mutex-guarded conn
respClient held a single net.Conn behind one mutex, serializing every caller's whole round trip. Interpreted-harness audit measured this as the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on memoryStore for the same workload, latency scaling near-linearly with concurrency (869us@conc=1 -> 10.81ms@conc=16). Replace the single connection with a pool of independently usable connections (respConn slots in a buffered channel), each acquired for one caller's whole round trip and returned via a deferred send back onto the pool. A broken connection is discarded (closeConn) but its slot always returns to the pool, so pool size never shrinks after an error. Default size is GOMAXPROCS clamped to [4, 8] (defaultRespPoolSize); RedisConfig.PoolSize lets an operator override it explicitly, per the house rule that an explicit override always wins over self-tuning. Preserves exactly: the 2s respCallTimeout call deadline, reconnect- once-on-error semantics, nested-array rejection, the RESP size bounds, and every existing pipelining contract (incrMulti, incrAndGetMulti, getMulti, cache get/set) — nothing outside resp.go references a connection field directly. Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%), throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1 control matched HEAD exactly, confirming the win is parallelism, not refactor noise. cache_test.go and llmgateway_test.go: adjust white-box tests that issue multiple sequential calls against one respClient scripted with runFakeRESPServer's single-connection model — force pool size 1 there (newRESPClientPool) so the test still pins one connection's exact wire sequence; production callers are unaffected. |
||
|
lukaszraczylo
|
df39ad3739 |
chore(admin): regenerate baked admin assets
Regenerated via `make admin-ui` against the new webui/generate.mjs chunking. Decoded asset bytes are byte-identical to the prior baked file (SHA-256 over adminIndexHTML + every adminAssets[name].body, sorted by name: 7bf96c68a64d26586057c714ba8b6ee393ad361a47ab1acb21d1d260955d3edd before and after — verified with a throwaway test against both generator outputs on the same webui/dist/ build). contentType strings and asset map keys are unchanged. Measured under Yaegi (github.com/traefik/yaegi, the interpreter this plugin actually runs under) with a standalone import-timing harness pointed at this tree: package import 233.8ms -> 39.8ms (median of 3 runs each), heap after import 32.0MB -> 22.8MB. |
||
|
lukaszraczylo
|
e5c3cd4b97 |
perf(admin): stop wrapping baked assets in a 7500-literal + chain
webui/generate.mjs split each admin-panel asset's base64 into 120-char Go string literals joined with `+` (~7,550 literals, ~7,538 `+` ops across index.html + the JS/CSS bundle). A compiled Go binary folds that at compile time for free, but this plugin loads under Yaegi (github.com/traefik/yaegi), which folds a chain of N literal `+` operations at import time by repeatedly reallocating and copying the whole already-folded prefix: O(N^2). Measured: ~186ms of a ~223-240ms package import was this folding alone, versus ~3-4ms to decode the asset bytes. Replace the 120-char LINE_WIDTH with a 1,000,000-char CHUNK_CHARS cap. Every asset this panel ships today (largest ~516,000 base64 chars) fits in a single literal, so goBase64Expr now emits one decodeAdminAsset call with zero `+` folds per asset. The cap still bounds a future, much larger bundle to a small number of large chunks rather than one unbounded line; single-line Go string literals in this size class were verified against gofmt/go vet/go build before picking the constant. No behavior change: decoded asset bytes, contentType strings, and the asset map keys are unaffected by how the base64 is chunked into source literals — see the following regen commit for the byte-identity proof. |
||
|
lukaszraczylo
|
e06d8bf5cd |
fix(passthrough): fail closed on a duplicate top-level model key
RFC 8259 permits duplicate names and every mainstream upstream parser
resolves them last-wins, while a passthrough body is forwarded verbatim.
Reporting the first occurrence let a model-restricted caller authorize
{"model":"allowed","model":"expensive"} against the allowed one and
execute the expensive one. Refuse to answer instead, matching
extractMultipartModel's existing duplicate rule.
Found by the security audit; a padded-duplicate variant that hides the
second key beyond the 64KiB peek window is tracked separately.
|
||
|
lukaszraczylo
|
e57a97d63e |
fix(passthrough): close Content-Type bypass and matcher-parity bug
Independent review measured two real gaps in the round-1 model
enforcement and fixed both here, plus documentation and comment
regressions the round introduced.
MUST FIX: peekPassthroughModel's Content-Type gate was an ALLOW-list
("application/json" or skip") and fully client-controlled. The
reviewer sent "Content-Type: text/plain" with {"model":"EXPENSIVE"}
and got a 200 from a group scoped to a cheap model. This commit
replaces the whole gating approach:
1. A group with an empty Models list has nothing for allowsModel to
reject, so the request body is never read at all
(group.hasModelRestriction). This is the overwhelming majority of
traffic, including the operator's own "home" group, and removes
the memory-buffering concern entirely for it.
2. For a group WITH a restricted Models list, the Content-Type gate
is now a SKIP-list of genuinely binary types (multipart/, audio/,
image/, video/, application/octet-stream). Every other
Content-Type, including "text/plain" or none at all, is
inspected. The attacker no longer controls whether the check
runs.
3. The peek reads at most 64KiB (maxModelPeekBytes), restored via
io.MultiReader so the remainder streams straight to the upstream
instead of a second full buffer. A bounded JSON token walk
(scanTopLevelModel) finds a top-level "model" key even when the
rest of the body is truncated past the cap.
4. A restricted group whose peek window never finds "model" is
denied (403), fail closed. This also means a Gemini passthrough
request (its model lives in the URL, never the body) is always
denied for a model-restricted group; documented in
maxModelPeekBytes' own comment, the handler's doc comment, and
the README.
Matcher parity: the round-1 version glued providerName+"/" onto
whatever the body sent unconditionally, so an already-prefixed body
("openai/x") produced "openai/openai/x" and false-403'd a legitimate
client. allowsPassthroughModel now reuses modelRegistry.
splitConfiguredProvider — the same rule resolve/resolveAgainst apply
to a client-given id — to strip a leading segment only when it names
an actually configured provider, before generating candidates.
Also in this commit:
- Restored ~26 lines of doc comments fieldalignment -fix silently
dropped when it reordered the group struct (auth.go: cache,
cacheTTL) and ProviderConfig (llmgateway.go: MetadataPath, and the
Passthrough field's own new comment). Verified nothing else was
reordered by re-running fieldalignment: 0 issues.
- Documented the path.Match footgun a non-empty PassthroughPaths
inherits: "*" does not cross "/", so ["*"] denies almost all real
multi-segment passthrough traffic rather than allowing it. Empty is
the actual allow-all value. Documented on the config field, the
allowsPassthroughPath method, and the handler's own doc comment.
- Renamed the local "model" that shadowed the outer request-model
variable near extractPassthroughUsage's call site to "respModel" —
the new gating already scopes the outer one to a narrower block, so
this was no longer a live shadow, but the rename removes any future
ambiguity between the request's model and the response's.
- README: rewrote the two "Unified vs. passthrough" table cells that
described the OPPOSITE of this feature's actual behavior, added a
"Native passthrough model authorization" section, and added
providers.<name>.passthrough / groups.<name>.passthroughPaths rows
to the config tables.
New tests: the Content-Type-bypass regression (text/plain, absent,
form-urlencoded, all now 403), the matcher-parity case (already-
prefixed body allowed), fail-closed on a binary body for a restricted
group, skip-entirely for an unrestricted group, byte-identical body
forwarding for a body larger than the peek cap and for a
ContentLength<0 (chunked) request, and a scanTopLevelModel unit table
covering truncation before/after the model field, nested values, and
malformed input.
Gate re-run in full after these changes: gofmt, go vet, fieldalignment,
golangci-lint (0 issues), go test -race (ok), make yaegi-check (OK),
make integration (ok, real Traefik + Redis). Re-ran the interpreted
Yaegi harness: RTT/req unchanged at 3.00, cmds/req unchanged at 74.00.
|
||
|
lukaszraczylo
|
311dd20a8e |
fix(limits): correct dangling comments and redisStore edge cases
Independent review round approved the fusion itself (11 to 3 RTT/req, precedence verified branch by branch, latch/fail-open checked under Yaegi against a dead Redis). This commit addresses the follow-up items raised against that same change. Six doc comments in limits.go still named evaluateScope, budgetViolation, or tokenBudgetViolation as if they were live functions to consult. All three were removed by the fusion commit; the comments now point at checkAndCount and buildBudgetProbes, where the same logic lives today. One further comment claimed checkAndCount still slices a "storeIncrMulti result" — it now slices storeIncrAndGetMulti's own incrVals. TestCheckAndCount_FusedViolationPrecedence's doc comment claimed every subtest engineers two simultaneous violation conditions. The seventh engineers only one (the group violates; the user carries no limits at all) — it pins a different claim, that a later scope is still evaluated when an earlier one has nothing to check. The comment now says so. redisStore.incrAndGetMulti gained the same len()==0 guard incrMulti already has, and a table-driven test proves a RESP error or a non-integer value on a read key fails the whole call, matching respClient.getBatch's own equivalent table. Re-ran the interpreted Yaegi harness after these changes: RTT/req stays at 3.00, cmds/req stays at 74.00 — the fusion's throughput win is unaffected by this round's fixes. |
||
|
lukaszraczylo
|
e3ef4084d2 |
feat(passthrough): add model, path, and header authorization hardening
handlePassthrough checked only the provider allowlist, never the model or the path. A tenant scoped to one cheap model could reach a provider's entire native API (files, batches, fine-tuning) on the operator's key, and client identity headers reached keyless in-cluster MCP/A2A targets unfiltered. This change closes both gaps, additively: every new field defaults to reproducing prior behavior exactly. Model enforcement: when a passthrough JSON body carries a top-level "model" field, handlePassthrough checks it against the group's model glob, in both the bare and provider-prefixed form (the same dual-candidate matcher the unified route's resolveAgainst already applies). A body with no inspectable model (Gemini's URL-embedded model, a multipart upload, a non-JSON body) falls back to provider-only authorization, unchanged from before this round. Per-provider toggle: ProviderConfig.Passthrough, nil or true by default, lets an operator disable a provider's native passthrough route. A disabled provider now reports the ordinary unknown-route 404, checked in ServeHTTP's own route gate, before auth even runs. Per-group path allowlist: GroupConfig.PassthroughPaths, empty by default (matching every other allow-list field's empty-means-all semantics), restricts which native rest path a group may address. Dangerous-header stripping: proxyUpstream now additionally strips Forwarded, X-Forwarded-*, X-Auth-*, X-Remote-User, X-Remote-Groups, and Cookie on both the native-passthrough and MCP/A2A target-proxy paths, plus OpenAI-Organization, OpenAI-Project, and anthropic-beta on native passthrough only. This is an additive denylist expansion, not an allowlist inversion; ordinary headers still pass through. A dedicated test asserts a config with none of the new fields set behaves identically to before this change. The operator's home group (providers/models/mcpServers/agents/passthroughPaths all empty) stays fully unrestricted, and macstudio-rerank/parakeet-mlx/piper-tts stay reachable through native passthrough by default. |
||
|
lukaszraczylo
|
a418814b34 |
perf(limits): fuse admission reads into one round trip
checkAndCount already batched its request-counter increments into one pipeline. evaluateScope then read every scope's token and cost budgets through up to four further serial round trips per scope. A request with a fully-limited user and group scope paid nine round trips before the upstream call started. This change fuses every budget read into the SAME pipeline as the increments, through a new incrAndGetMulti method on counterStore. The reads are safe to fuse because they touch disjoint keys (req:* versus tokin:*/tokout:*/cost:*), and tokens and cost are only ever written by account() after the response completes, never by checkAndCount itself. Violation precedence stays identical to the pre-fusion implementation: a user's own violation still wins over their group's, and within one scope the order stays requests-per-minute, requests-per-day, tokens-per-day, tokens-per-month, cost-per-day, cost-per-month. New table-driven tests pin this precedence directly, engineering two simultaneous violation conditions per case and asserting only the higher-precedence one is reported. evaluateScope, budgetViolation, and tokenBudgetViolation are removed; buildBudgetProbes plus a loop inline the same logic into checkAndCount, which is what lets their reads share the fused round trip. Measured on the interpreted Yaegi harness (yaegi v0.16.1, the runtime this plugin actually ships under): RTT/req for a fully-limited chat request drops from 10.99 to 3.00, with cmds/req unchanged at 74 (this is a round-trip fix, not a command-count or allocation change). Single-request throughput rises from 112 to 335 req/s; at concurrency 16, from 121 to 380 req/s. counterStore.incrAndGetMulti is a required interface method, not an optional capability behind a type assertion: Yaegi can silently return ok=false when a comma-ok assertion checks a compiled concrete value against a plugin-declared interface (Trap 4, documented in matchesSentinel's own doc comment). A plain interface method call carries no such risk and is the same mechanism incrMulti already uses in production. |
||
|
lukaszraczylo
|
2d10509e20 |
docs: render correctly on the Traefik plugin catalog
The catalog uses a plain CommonMark renderer, not GitHub's: the "[!WARNING]" alert syntax printed literally and the mermaid block rendered as a wall of diagram source. Use a portable bold blockquote and fold the diagram into <details> so plain renderers collapse it while GitHub still draws it. |
||
|
lukaszraczylo
|
1875ee7b8b |
fix(registry): defer context cancels on the refresh panic path
A panic in listModels skipped the inline cancel, holding the timeout timer until it fired. Review follow-up from the v0.2.3 final verdict. |
||
|
lukaszraczylo
|
0a37fa1b47 |
chore(admin): regenerate baked admin assets
make admin-ui: vue-tsc -b && vite build, then webui/generate.mjs against webui/dist/, gofmt -w'd. Reflects the review-round webui fixes: the compact-format precision corrections, the ARIA-valid hover-text components, and the ProvidersView.vue polish. |
||
|
lukaszraczylo
|
171f6d04c9 |
docs(readme): document model metadata (review SHOULD-7)
New 'Model metadata' section: the context/cost layering order, the modelMeta config override example (mirrors the operator's own cluster snippet), alias inheritance, the ":free" naming convention, the metadataPath discovery capture (including the loaded-vs-max-context volatility note — a discovery refresh can legitimately report a different context window across reloads), the pricing-sync/built-in table workflow, and the two GET /v1/models extension fields with a worked example. Config reference tables gain a ModelMetaConfig subsection and modelMeta/metadataPath rows on Config/ProviderConfig; the Admin section notes the per-provider and per-alias modelMeta fields on GET /admin/api/overview. |
||
|
lukaszraczylo
|
fb1b2a36ff |
test(yaegi-check): exercise the builtin table lookup (review SHOULD-6)
The harness previously only ever proved resolveModelMeta's CONFIG- OVERRIDE layer runs interpreted (testDataWantModel carries an explicit modelMeta entry). It never exercised an interpreted LOOKUP into the real, 236-entry builtinModelMetaTable map literal at all. Adds "gpt-4o" to the harness's openai provider with NO modelMeta override of its own, so its GET /v1/models context_window can only come from the builtin-table branch. Its expected value is read directly out of pricing_data_gen.go at harness run time (readBuiltinContextTokens), not hardcoded, so this assertion self-updates across a `make pricing-sync` regeneration instead of pinning a number that could legitimately drift when upstream LiteLLM data changes. Also switched both existing GET /v1/models assertions from raw strings.Contains substring matching to a structured JSON decode keyed by model id — a bare substring check could not tell which model entry a given context_window value actually belonged to, which now matters with two entries in play. |
||
|
lukaszraczylo
|
9b60c0afae |
fix(webui): ARIA-valid hover text, alias metadata guard, context-chip polish
CompactNumber.vue and ProviderRateBadge.vue both put aria-label on a
plain <span> — axe's aria-prohibited-attr rule: a <span>'s implicit
ARIA role ("generic") does not support accessible-name computation at
all, so the attribute was simply invalid there, not just discouraged.
Both now use the visible-text-hidden-plus-sr-only-full-text pattern
instead: the compact/label text is marked aria-hidden="true", and a
sr-only sibling span carries the exact value/detail for assistive tech.
title stays on the outer element for mouse hover, unchanged. Folded
review minor.
ProvidersView.vue: the alias table's ModelChip cell read
row.original.modelMeta.contextTokens without a guard, unlike the
provider path's own modelMetaFor fallback — a response from a server
build older than this field (version skew) would throw instead of just
omitting the hover detail. Now optional-chained to match. The visible
context chip beside each model gains title="context window" and
text-xs (was an arbitrary text-[10px]), matching the rest of the panel's
muted-label sizing convention. Folded review minors.
|
||
|
lukaszraczylo
|
5e3cf919d9 |
fix(webui): compact-format precision bugs (review IMPORTANT-2/3)
formatCompactCount: Math.floor(scaled*factor)/factor under-reported by one unit whenever the intermediate scaled=abs/divisor value was not exactly representable in binary floating point (8_700_000 -> "8.69M", not "8.7M" — 8.7 itself has no exact double representation, so the original two-step division/floor landed on 869 instead of 870). Reordered to floor(abs*factor/divisor) — multiplying the original INTEGER first keeps every intermediate value exact for any realistic count, since abs*factor stays within Number.MAX_SAFE_INTEGER. New rows: 8_700_000 -> "8.7M", 1_130_000 -> "1.13M". (review IMPORTANT-2) formatContextWindow: always divided by 1024 (binary-K), which is right for a genuinely binary context size (262144 -> "256k") but wrong for a decimal, marketing-style round count (128000 -> "125k", an odd number nobody would recognize as "128K"). A plain "tokens % 1024 == 0" check cannot separate these — 128000 is ALSO an exact multiple of 1024 (125 x 1024) despite not being a power of two. The actual, consistent discriminator is POWER OF TWO: 262144 = 2^18, 128000 = 2^10 x 5^3 (not a power of two). Binary-K/M now applies only when tokens is an exact power of two; decimal-K/M otherwise. Fixed rows: 128000 -> "128k" (was "125k"), 1_000_000 -> "1M" (was "977k"), 2_000_000 -> "2M" (was "1953k"); 262144 -> "256k" and 32768 -> "32k" unchanged. (review IMPORTANT-3) |
||
|
lukaszraczylo
|
e3996d88bf |
fix(pricing): content-digest snapshot instead of a wall-clock date
The generated pricing_data_gen.go header used to stamp today's date on every `make pricing-sync` run, so a re-run against genuinely unchanged upstream data still touched the file and produced a spurious daily diff with nothing real to review. The header now records a short SHA-256 digest of the FETCHED bytes instead — identical upstream data always reproduces a byte-identical generated file. Folded review minor. |
||
|
lukaszraczylo
|
962f5eeeed |
fix(providers): validate metadataPath has a leading slash
fetchModelMetadata (provider_openai.go) builds its request URL by plain string concatenation, a.baseURL+a.metadataPath. A metadataPath missing its leading slash would silently glue onto baseURL's own path with no separator (e.g. "http://host:1234api/v0/models"), a malformed URL that only ever surfaced as a confusing runtime fetch failure. buildAdapters now rejects a non-empty metadataPath without a leading "/" at construction, naming the provider, the same fail-fast treatment every other malformed config value already gets. Empty (the default) is always valid. Folded review minor. |
||
|
lukaszraczylo
|
82dcaa5309 |
fix(registry): metadata capture gets its own timeout + panic isolation
warmFill/refreshProvider used to reuse listModels' own context for the subsequent captureModelMetadata call — a listModels attempt that took real, non-trivial time (or timed out outright) guaranteed the metadata fetch an already-spent, near-expired context, producing a spurious deadline error and log line every warm fill or refresh for an otherwise healthy provider. Both call sites now give captureModelMetadata its own fresh context.WithTimeout budget, computed after listModels returns. (review SHOULD-4) captureModelMetadata also gains its own deferred recover, isolating a panic inside fetchModelMetadata (or a future modelMetadataFetcher implementation) from refreshProvider's OUTER recover — which would otherwise overwrite the (already-nil, already-correct) err variable from a successful listModels call and cause finishRefresh to discard a freshly fetched, genuinely good model list over an unrelated bug in the metadata path. "Metadata must never fail discovery" is now enforced on the panic path, not just the plain-error-return one already handled. (review SHOULD-5) New regression tests: a fresh-timeout-budget check (a slow simulated listModels must not shrink the metadata fetch's own deadline) and a panic-in-fetchModelMetadata check (discovery must still succeed and record both models). |
||
|
lukaszraczylo
|
933d869e32 |
fix(modelmeta): free-tier suffix layer (review IMPORTANT-1)
A ":free"-suffixed model id (a real provider naming convention, e.g. OpenRouter's own live API) is free by naming convention alone, independent of whatever the built-in table does or does not track for that exact id. resolveModelMeta's cost switch gains a new case, ranked above the built-in table and below an operator's own config override — so a provider's free-tier naming is never shadowed by a stale or coincidental paid built-in price under the identical key, and an operator can still override it explicitly if they choose. Context is untouched by this rule: it keeps resolving through the normal config/ discovery/builtin/absent layers, never inherited from an unrelated entry. Also makes resolveModelMeta's own Free case assign 0, 0, true explicitly (matching resolveAliasModelMeta's existing style) instead of relying on the zero-value default — defensive consistency, folded review minor. Independently reviewed and reproduced against the same live LiteLLM fetch: zero ":free"-suffixed keys exist anywhere in the 3111-entry upstream file, confirming the earlier grounding note in the v0.23 report. This layer is a general resolution rule for a naming convention provider APIs use in live discovery, not something the pruned build-time table needs to carry any entries for. |
||
|
lukaszraczylo
|
002ee3b143 |
chore(admin): regenerate baked admin assets
make admin-ui: vue-tsc -b && vite build, then webui/generate.mjs against webui/dist/, gofmt -w'd. Reflects every webui/ change in this branch: feature v0.23's metadata hover/chips and the compact-count formatting addition. |