mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
063a05e9a7e895d6b8b68f6fd08afbe3be237254
219
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
063a05e9a7 |
feat(metrics): instrument upstream latency
Add time-to-first-byte and total-duration histograms for every upstream call, observation only. No throttling, priorities, or control logic reads this data. The operator wants strain-based throttling later and needs real traffic data first. Extend watchdogBody (timeout.go) rather than add a new interception layer: it already wraps every upstream response body for provider adapters, native passthrough, and the MCP/A2A target proxy, and already observes the first successful read. armLatency arms a watchdogBody after construction, so newWatchdogBody keeps its existing signature and behavior; its own pinned tests still pass unchanged. A timestamp plus a few atomics are the only added cost; Read and Close each pay one nil check when latency collection is disabled. Expose llmgateway_upstream_ttfb_seconds and llmgateway_upstream_duration_seconds on the existing Prometheus endpoint (metrics.go). Both are in-process, per-replica histograms, never store-backed, so sum() is always the correct cross-replica aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and short completions finely, cover the bulk of real chat completions, and bracket the five-minute default request timeout. Labels are provider and stream always, model only when Config.Metrics.ModelLabel is set, matching the existing per-model opt-in convention. Add a compact per-provider, per-stream average summary to GET /admin/api/overview, sourced from the same in-process store, without a full histogram or a per-model breakdown. Every write path is gated on metricsEnabled(cfg), so a deployment with metrics disabled collects nothing and the histogram families never render. No behavior change to request handling: same responses, status codes, headers, and accounting. Fix a Yaegi-only bug this feature's own context-threading change exposed: two distinct empty struct types used as context.WithValue keys compare equal to each other under the interpreter's interface equality, though never under compiled Go. Threading a second recorder (latencyRecorder) alongside the existing attemptRecorder made this observable for the first time, since compiled tests never see it and the plugin previously carried only one such key. Give latencyRecorderCtxKey a field so its size differs from attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction before and after the fix. Fix a second bug, reproducible under compiled Go too: io.Reader may legally return its final data together with io.EOF in one call, and Go's real net/http body reader does this for small, fully-buffered responses. The original TTFB capture lived inside the err == nil branch, so it never fired for a typical small non-streaming completion. Move the check ahead of the error branch so it fires on any n > 0, leaving the pinned timer-reset and Close semantics untouched. Tests mutate the real source per case and confirm each one fails first, recorded in each test's own doc comment. |
||
|
lukaszraczylo
|
2a5b086020 |
fix: ship vendor/ in the release archive so the catalog can import
The Traefik Plugin Catalog analyzer rejected v1.0.43: import "github.com/lukaszraczylo/oss-telemetry" error: unable to find source related to: "github.com/lukaszraczylo/oss-telemetry" piceus fetches the source archive at the released tag and interprets it with Yaegi. It never runs go mod download, so every non-stdlib import in production code must exist in the archive own vendor/ tree. .gitattributes carried "vendor/ export-ignore" with an explicit warning to remove it the moment production code imported a third-party package. That happened when llmgateway.go took on oss-telemetry for anonymous usage reporting, and the warning was missed. The tag itself always had vendor/; git archive stripped it, so the tarball shipped a go.mod requiring a module whose source was absent. Verified by reproducing the analyzer: the v1.0.43 archive contains 0 vendor entries, the fixed archive contains 71, and interpreting the extracted archive under yaegi passes. |
||
|
lukaszraczylo
|
1089a2b41a |
merge: progress-based upstream request timeout, 5m default
# Conflicts: # README.md # llmgateway.go # tools/yaegi-check/main.go |
||
|
lukaszraczylo
|
edd618279f |
docs: correct request-timeout claims found by adversarial review
F2: the idle timer resets on DECODED body bytes, after net/http's own gzip decompression. A gzip'd response whose compressed bytes arrive steadily but whose deflate stream yields nothing for the whole window still gets aborted. The previous wording claimed a slow-but-progressing body is always unaffected; adds a qualifying paragraph explaining the gzip case and why fixing it (measuring wire bytes) is out of scope at the five-minute default. F3: reworded the "silence ends a request, never elapsed time" framing. A streaming forwarder reads a chunk, writes it to the client, then reads again; the write is not watched, so time blocked on a slow client counts against the same idle window as a silent provider. Aborting is still correct, but the log line no longer claims the provider specifically caused it. F5: documents that neither timeout is ever retried, and why: a ResponseHeaderTimeout failure wraps context.DeadlineExceeded, which Retry's own transient classification excludes; a mid-body watchdog timeout fires after the retry mechanism has already returned. F8: softened "Every upstream request this gateway makes" to the routes actually covered, and added a "Federated MCP" note: the federated /mcp endpoint's two direct calls are not wrapped by the idle-progress watchdog, but are not unbounded either — they still get ResponseHeaderTimeout from the shared client, plus their own 20s/120s/ 5s context.WithTimeout budgets. Worst case is a bounded stall, not a forever one. |
||
|
lukaszraczylo
|
c811de3dfb |
test(yaegi-check): probe the mid-body-stall half of the watchdog too
exerciseRequestTimeout only drove a provider that never sends headers, so it exercised Transport.ResponseHeaderTimeout alone. watchdogBody.fire — the interpreted method value handed to time.AfterFunc, plus context.CancelFunc and sync/atomic's function API — never ran under that probe, since the request never got far enough to construct a watchdogBody at all (finding F7, coordinator adversarial review, 2026-08-23). Adds hungbody, a provider whose upstream sends headers immediately and then stalls, and exerciseRequestTimeoutMidBodyStall to drive it. Verified the two probes exercise different code paths: removing watchdogBody's construction from upstreamBytes left the existing header-timeout probe passing unchanged, while the new probe failed with status 200 instead of 502. Reverted before this commit. |
||
|
lukaszraczylo
|
51e675acb0 |
fix(providers): address adversarial review findings F1, F4, F6, F9
F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a plain `now` variable while a background refresh goroutine read it through nowFn, racing under -race in about 14% of full-suite runs once this branch's timer-heavy tests shared the parallel phase. Guards the clock with a mutex instead. registry.go is untouched. F4: a mid-body watchdog timeout never reached provider-health accounting. limiter.recordProviderAttempt only ran at header-arrival time, so a provider that reliably stalled mid-body reported attempts=1 failures=0 forever. watchdogBody.fire now invokes the same attemptRecorder a second time, as a failure, when it fires. Reuses the existing recorder rather than adding new plumbing; this does mean two attempts are recorded for one logical request (one success at header time, one failure at watchdog-fire time), not one attempt marked failed outright. F6: watchdogBody.Read reset the timer on any err == nil read, regardless of n. A reader returning (0, nil) forever would have held the watchdog open indefinitely. Reset now only fires when n > 0. F9 (three items): - proxyUpstream passed logPrefix as a bare providerName, which newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an MCP/A2A target as a provider. watchdogBody now takes an already-formatted label from the caller instead. - errUpstream was wrapped twice: once in watchdogBody's own error, once again by every caller that already wraps it. watchdogBody now wraps only errProviderTimeout; callers' existing wraps still make errors.Is(err, errUpstream) hold. - timeout_test.go read wb.timedOut directly in one place while every other access is atomic; switched to atomic.LoadInt32. Also rewords errProviderTimeout's message from "provider request timed out" to "upstream progress stalled" (finding F3, documented not fixed per coordinator ruling): the watchdog measures time since the last successful upstream read, which can go quiet because of a slow client write between reads, not only a slow provider. The new wording no longer asserts which side caused it. New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure, TestWatchdogBody_ZeroByteRead_DoesNotResetTimer, TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider. Each mutation-verified per its own doc comment. |
||
|
lukaszraczylo
|
e6ce1cc4c5 |
merge: Prometheus metrics endpoint with admin-key and CIDR auth
# Conflicts: # llmgateway.go # tools/yaegi-check/main.go |
||
|
lukaszraczylo
|
95039d96c9 |
fix(examples): make the metrics scrape example actually reachable
The VMServiceScrape example scraped Traefik's pods directly, so the request carried Host: <pod-ip>:<port>. Traefik evaluates a router's match rule before any middleware runs, and the only router in this file requires Host(`llm.example.com`). The scrape request never matched it, the llmgateway middleware never ran, and /metrics 404ed before the plugin ever saw the request. Add a second, Host-free IngressRoute matching PathPrefix(`/metrics`) and carrying the same middleware, so an in-cluster scraper reaches the route regardless of what Host header it sends. Document the reachability trade-off this introduces (this route is not Host-restricted, though /metrics stays gated by its own auth inside the plugin) and name the alternative fix the review also offered: relabelConfigs rewriting __address__ to the ingress hostname before the request is sent, keeping one Host-scoped router instead. |
||
|
lukaszraczylo
|
47e22e627c |
fix(metrics): close the default-config store-down gap, fix escaping
Three blocking fixes from a second round of adversarial verification, plus two should-fix items bundled in since they touch the same code. storeDown detection covered only the fail-CLOSED path. failOpen defaults to true (newConfiguredLimiter), so on the shipped default a broken store made storeGetMulti return ok=true after silently reading the empty in-process fallback, bypassing every storeDown guard in metrics.go. Add limiter.configuredStoreDown, checked alongside ok in currentUsage and providerUsage, so a fail-open read served from the fallback is treated the same as a fail-closed refusal. The pinning test previously hand-set failOpen=false, which meant it only ever exercised the case that was already correct; it now leaves failOpen at its true default. The escaping rewrite from the previous round achieved injectivity by copying an invalid UTF-8 byte through raw, which can make the output itself invalid UTF-8. Measured against a real Prometheus scrape parser this round: invalid UTF-8 in a label value is fatal, the scrape target goes down, which is worse than the duplicate series the previous round was fixing (measured non-fatal, dropped and logged, target stays up). Replace an invalid byte with a deterministic \xHH marker instead, proven injective and always valid UTF-8 by an exhaustive test over every 1- and 2-byte input, not merely spot-checked. Also adds the fast path a benchmark showed the byte-oriented rewrite needed: unescaped, already-valid-UTF-8 input, the common case, returns unchanged with zero allocations instead of paying a full copy. llmgateway_rate_limit_rejections_total's HELP text claimed it "resets only on restart", which stopped being true once the previous round added generation-rotation bounding and reload-triggered eviction: a scope resets when its user is removed, and every tracked scope resets together in bulk the instant the tracked-scope cap is exceeded. The HELP text and README now say so. Adds llmgateway_limit_store_up, a per-replica gauge for this process's own connection health to the configured limit store: with the store flapping faster than its fail latch, every store-backed family can go silent scrape after scrape with nothing on the endpoint itself saying why. Also adds a test pinning that Gateway.pruneRejectionsAfterReload runs only when authStore.maybeReload reports a real reload, never on every request. |
||
|
lukaszraczylo
|
1f409507a3 | merge: cross-provider failover with cost guard, default off | ||
|
lukaszraczylo
|
0cf23345b1 |
fix(failover): scope header clear to retries, fix cache pre-pass, gate health record
Fix one must-fix and one should-fix finding from re-verification of
|
||
|
lukaszraczylo
|
834238d141 |
docs: document the request timeout feature and its default
Adds a "Request timeout" section explaining the progress-based design, the two mechanisms, the global/per-provider override rule, and error classification. States plainly that this changes behavior for every deployment, since no timeout existed before. Adds requestTimeout rows to the Config and ProviderConfig reference tables. |
||
|
lukaszraczylo
|
5062017cf1 |
test(yaegi-check): probe a hung provider aborted under the interpreter
Adds exerciseRequestTimeout: a provider (hungtimeout) with an 80ms ProviderConfig.RequestTimeout, backed by an upstream that stays silent for 600ms. The request carries no client-side deadline of its own, so a pass proves the gateway's own adapter-level timeout aborts the hang, not a caller-supplied context. This complements the existing slowProviderName probe, which proves a CLIENT-set deadline classifies correctly. It does not prove the adapter's own timeout mechanism runs correctly interpreted, since it never exercises it without client help. Verified the probe is not a no-op: temporarily removing Transport.ResponseHeaderTimeout from newAdapterHTTPClient made make yaegi-check fail with status 200 instead of 502. Reverted before this commit. timeoutProviderModel uses a distinct id, not testDataWantModel: reusing it made the new provider win bare-id model resolution and silently hijacked an unrelated existing probe. |
||
|
lukaszraczylo
|
057c061326 |
test(providers): exercise fire, not ResponseHeaderTimeout, in the leak test
TestWatchdogBody_NoBlockingGoroutineLeak_AcrossManyRequests drove its timed-out iterations against a server that never wrote headers. That hits Transport.ResponseHeaderTimeout inside client.Do and returns before a response ever exists, so watchdogBody.fire never runs and the test could not have caught a leak in fire itself. The mutation-verification pass (manually adding a permanently blocked goroutine to fire, then reverting) confirmed this: the test passed even with that goroutine present. Fixing the stalling server to write headers first, mirroring the mid-body-stall tests above it, makes fire run on every iteration. Re-running the same mutation now fails the test as intended. |
||
|
lukaszraczylo
|
503fc33933 |
feat(providers): add progress-based upstream request timeout
Adapters previously issued upstream requests with no timeout at all. A provider that accepted a connection and then stalled hung the gateway request forever. newAdapterHTTPClient now sets Transport.ResponseHeaderTimeout, and every response body read (providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream) runs through a watchdogBody that resets on every successful read and cancels the request once timeout passes with no progress. The idle-progress design lets a long stream that keeps emitting tokens run indefinitely. Only silence, never total duration, ends a request. Config.RequestTimeout sets the global default (five minutes when unset). ProviderConfig.RequestTimeout overrides it per provider. Both are Go duration strings, validated at construction; a zero or negative value is rejected rather than treated as unlimited, since that would reintroduce the hang this feature fixes. Native passthrough (routes_passthrough.go) and the MCP/A2A target proxy (mcp_a2a.go) share the identical watchdog mechanism through proxyUpstream's new timeout parameter. A watchdog-fired timeout wraps errProviderTimeout and names the provider. It never wraps context.Canceled, so it stays distinguishable from a genuine client disconnect in every existing log classification. |
||
|
lukaszraczylo
|
d45125ee07 |
fix(failover): default off, add cost guard, fix adversarial-review findings
Flip FailoverConfig.Enabled to default false. A version upgrade with no
config change must preserve prior behaviour, and failover changes which
provider serves a request. Update tests, README, and the yaegi-check
probe to opt in explicitly.
Add a mandatory cost guard. A candidate is skipped unless its input AND
output cost per token are each no greater than the primary candidate's,
resolved through the same canonical-then-bare pricing path the real
billing code uses. Unknown pricing is never treated as free: a known
primary against an unknown candidate skips, an unknown primary against a
known non-zero candidate skips, everything else that is equal is
allowed. Not configurable.
Fix four blocking findings from adversarial review of commit
|
||
|
lukaszraczylo
|
935df1a3f0 |
docs(metrics): add README section, testData, and a scrape example
The metrics feature shipped in code only: no README section, no
metrics block in .traefik.yml testData, nothing under examples/. The
Go doc comments were the entire manual for a human operator or a
config-generating assistant.
Add a README Metrics section covering the served path, both auth
modes (admin bearer token, optional CIDR allowlist), the CIDR-sizing
warning with the live-reproduced SNAT failure mode, the modelLabel
cardinality trade-off, the exposed metric families, the cross-replica
aggregation contract, label escaping, and store-outage behaviour. Add
a MetricsConfig reference table matching the existing AdminConfig
one, and a metrics row in the top-level Config table.
Add metrics: {enabled: true} to .traefik.yml's testData, the
catalog-facing smoke-test config, deliberately without allowedCIDRs
so a copy-pasted example never skips the README's own sizing
guidance.
Add a metrics block to the Kubernetes Middleware example and a
VMServiceScrape resource: the target cluster runs VictoriaMetrics, not
Prometheus, so the CRD differs even though the exposition format does
not. Verified the VMServiceScrape shape (apiVersion, selector,
endpoints, path) against the victoriametrics/operator documentation
rather than assuming it. Notes that /metrics is served by Traefik
itself, not by any backend service, since this middleware intercepts
the path before a backend is ever reached.
|
||
|
lukaszraczylo
|
244dd6185f |
test(yaegi-check): verify label escaping under the interpreter
Add a third configured user whose name contains a double quote, a backslash, and a newline, then assert GET /metrics renders its Prometheus-escaped form and never the raw name. auth.go validates a user name as non-empty only, so this is a real, legal configuration, not a contrived one. The compiled suite already proves escapeLabelValue correct (TestMetrics_LabelValuesWithSpecialChars_Escaped, TestEscapeLabelValue_InvalidUTF8_Injective), but this codebase's own history is that a compiled pass has repeatedly said nothing about the interpreted shape. json.Marshal produces the JSON-escaped literal for the harness's own hand-built config document, kept separate from the Prometheus escaping the probe actually verifies in the rendered body. |
||
|
lukaszraczylo
|
d352379375 |
fix(metrics): honest rendering, injective escaping, aggregation docs
Three fixes from adversarial verification. writeProviderMetrics rendered a storeDown providerCounters entry as a hard zero, the same fabricated-reset hazard the request/token/cost families already guarded against. Skip a storeDown provider or provider-model entry instead of emitting it. escapeLabelValue ranged over the string as runes, and Go substitutes U+FFFD for every invalid UTF-8 byte. Two distinct configured names that both contain invalid UTF-8 could therefore render as the identical escaped value, colliding two series into one duplicate series and failing the whole scrape target. Escaping is now byte-oriented, which is injective and also avoids the per-rune decode cost on a large value. Every store-backed family's HELP text now states its aggregation contract: with Redis configured, the value is the gateway-wide total and identical on every replica, so summing across replicas overcounts by the replica count and a cost alert fires early. Aggregate with max() instead. Without Redis, the same family becomes genuinely per-process and sum() becomes correct instead. The rejection counter and the provider-healthy gauge get their own notes: the former is always per-process regardless of Redis, the latter reflects each replica's own independently-running discovery breaker. Also replaces the AllowedCIDRs doc example (10.0.0.0/8) with a cluster pod CIDR (10.42.0.0/16) and states plainly that a whole RFC1918 range is never correct, since a SNAT-ing router or load balancer makes external traffic arrive from inside it — reproduced live against the target cluster. Adds tests pinning three properties mutation-proofed against this suite: every storeDown skip (deleting any of them now fails TestMetrics_StoreDown_SkipsRatherThanFabricatesZero), the CIDR-check running before auth.identify (TestMetrics_ AllowlistedRequest_NeverTouchesAuthFailureTracking), and escapeLabelValue's injectivity under invalid UTF-8 (TestEscapeLabelValue_InvalidUTF8_Injective). Also covers the new rejection-counter pruning end to end through a real file-sourced reload. |
||
|
lukaszraczylo
|
5700dd482d |
fix(limits): never fabricate zero counters, bound the rejection counter
Two correctness fixes from adversarial verification. providerUsage returned every counter at its zero value on a failed-closed store read, with no signal distinguishing that from a real zero. metrics.go rendered the zero as a hard value. A single Redis blip therefore read as a counter reset, and the recovery scrape counted the whole prior total as a fresh increase, painting a phantom multi-thousand-request spike from one transient error. Add storeDown to providerCounters and set it on the failed-closed path, mirroring scopeUsage's existing contract. limiter.rejections grew without bound: nothing ever removed a scope whose user was renamed or deleted from a hot-reloaded users file, so a long-running process kept exporting a series for every name ever seen. Bound the map with the same current/previous generation-rotation authFailureTracker already uses, and evict a user's entry when authStore.maybeReload performs a real reload. maybeReload now returns whether it actually reloaded, so ServeHTTP only runs the eviction walk on that rare event, never on every request. Also corrects a stale comment: the rejection counter's store-down branch fires on every request during a full outage, not rarely, as an earlier comment claimed. |
||
|
lukaszraczylo
|
a12b02188f | merge: mutation-verified test hardening | ||
|
lukaszraczylo
|
49505227db |
test(cache): pin streaming requests as never cacheable
Cover an undefended half of runMeteredCall's own cacheable guard (routes_unified.go): "!streaming" was asserted nowhere end to end. Existing streaming tests only checked chunk delivery and mid-stream accounting, never the cache header or store state. Add a test that drives a real streaming chat-completion request through a Gateway with caching enabled and proves no X-Llmgw-Cache header appears, and that the cache holds no entry under the key a non-streaming request for the same body would use. Confirmed to fail against a real mutation (dropping "!streaming" from the cacheable expression) before being kept; the mutation and its revert are recorded in the test's own doc comment. |
||
|
lukaszraczylo
|
a45bd291ea |
test(limits): pin store fail-open/fail-closed policy and latch skip
Cover failPolicyIncrBy (0% covered), and add direct tests for storeIncrBy and storeGet against a real store, not only the nil-store fallback path the existing suite exercised. Each direct test proves four branches: a nil store always uses the in-process fallback, a configured store's success returns the store's own value rather than the fallback's, a store error records the failure and falls through to the fail-open/fail-closed policy, and an open store-down latch skips the network call entirely until it expires. Add a direct test for logSpawnDrop (0% covered), proving it logs once per storeErrorLogEvery like its sibling logStoreError, sharing the same rate-limit gate by design. Every test was confirmed to fail against a real mutation of the target function before being kept; each mutation and its revert are recorded in the test's own doc comment. |
||
|
lukaszraczylo
|
c6e4450686 |
test(anthropic): pin tool translation and upstream error shapes
Cover four zero-coverage functions on the /v1/messages route Claude Code drives on nearly every call: openAIToolsFromAnthropic, openAIToolChoiceFromAnthropic, responseTranslationError.Error, and writeAnthropicProviderUpstreamError. Add table-driven tests for every tool_choice form (auto, any, tool-by-name, unrecognized, absent, non-map, nil), a multi-tool mapping test that checks field order, description omission, and malformed-entry skipping, a full request-to-response round trip through a forced tool call, and a test proving no-tools/no-tool-choice omits both keys rather than sending an empty array. Add direct tests for the Anthropic-shaped upstream error envelope (status passthrough, error.code, decoded error.upstream) and for the translation-failure error prefix. Every test was confirmed to fail against a real mutation of the target function before being kept; each mutation and its revert are recorded in the test's own doc comment. |
||
|
lukaszraczylo
|
5d624791fa |
feat(failover): add cross-provider failover for overlapping models
Add failover.go with the full candidate pipeline. resolveWithCandidates resolves a request exactly like resolve does, then adds every other provider sharing the identical bare model id. orderedFailoverCandidates applies a hard skip for request-path-unhealthy providers, deprioritizes discovery-unhealthy providers without dropping them, fails open when every candidate reads unhealthy, and caps the list at maxAttempts. Add requestHealthTracker, a separate per-pod request-path health signal fed by the same isTransient/isDeadlineExceeded classification recordProviderAttempt already uses. It never widens the existing discovery circuit breaker, per that breaker's own documented scope. Rewrite runMeteredCall (routes_unified.go) to loop over candidates instead of resolving once. Each attempt rebinds the attempt recorder to its own provider, recomputes the cache key before the response is cached, and re-injects the alias echo key an adapter may have deleted on a prior failed attempt. Failover never runs once any byte has reached the client, since statusTrackingWriter already tracks that state. A failure falls through on everything except HTTP 400, matching the brief exactly. A 404 failover logs loudly, naming both providers and the model, since it usually signals a real configuration mistake. Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A single-provider deployment, or one with no overlapping bare model ids, behaves exactly as before regardless of this setting. Add a yaegi-check probe (tools/yaegi-check/main.go) driving two providers under the real interpreter, one failing and one serving, proving the candidate loop runs correctly interpreted. Update README.md with a Failover section and correct the Retry section's stale claim that no cross-provider failover exists. Media routes (images, audio speech, audio transcriptions) do not get failover in this change. Each has its own request-shape handling that needs its own scoped follow-up. |
||
|
lukaszraczylo
|
d421f49ff1 |
test(yaegi-check): verify the metrics endpoint under the interpreter
Compiled tests have repeatedly passed on this codebase while the interpreted shape failed, so the metrics feature needs its own probe in this harness, not just go test coverage. Enable metrics in the harness config, with modelLabel on so the opt-in provider-model breakdown runs interpreted too. Add exerciseMetricsRoute, driving four real requests: unauthenticated from outside the allowed CIDR block returns 401, a non-admin key returns 403, the admin key returns 200 with the expected TYPE lines and live provider and model data in the body, and a keyless request from inside the allowed CIDR block returns 200. Run the new probe last, after the breaker probe settles, since GET /metrics triggers the same maybeRefresh call every route does and the breaker probe's own hit counting is timing-sensitive. |
||
|
lukaszraczylo
|
d58eef9aa7 |
feat(metrics): add Prometheus text-exposition endpoint
Serve a hand-written Prometheus text-exposition document at a configurable path, defaulting to /metrics. The plugin is stdlib-only under Yaegi, so this cannot use a client library. Add MetricsConfig: Enabled gates the route, Path overrides it, AllowedCIDRs lets an unauthenticated caller scrape from a trusted source address, and ModelLabel opts into a per-(provider,model) breakdown that defaults off to avoid a series-count explosion. Reuse the existing admin bearer-token auth path instead of adding a second credential check. Reuse the batched, chunked reads GET /admin/api/overview and GET /admin/api/usage already use, so a scrape costs no extra store round trip. Expose request, token, and cost counters per scope, provider attempt and failure counters, a provider-health gauge from the discovery circuit breaker, and a budget-consumed-ratio gauge per limited scope for Prometheus alerting. Add a rate-limit-rejection counter in limits.go: this is the one number with no existing accounting to read back, so checkAndCount now increments a small in-process, per-process-lifetime counter on every violation. Escape every label value for backslash, double quote, and newline. An unescaped quote in a label value, such as a discovered model id, makes the whole scrape unparseable, and Prometheus reports an unparseable scrape as the target being down rather than as one bad line. Cover the auth gate matrix, exposition structure (TYPE per family, counters ending _total, no duplicate series), label escaping round-trips, the model-label opt-in, and counters reflecting real seeded usage rather than zeros. |
||
|
lukaszraczylo
|
208084ac83 |
merge: serve the Anthropic Messages API inbound
# Conflicts: # tools/yaegi-check/main.go |
||
|
lukaszraczylo
|
941a72bf7e | merge: per-provider discovery circuit breaker and health surfacing | ||
|
lukaszraczylo
|
3e1f188426 |
fix(routes): preserve response bytes and close remaining test gaps
Replace the alias-echo rewrite on the Anthropic passthrough branch with a byte-preserving field replacement. The prior version decoded the response into a Go map and re-marshaled it. This silently corrupted a JSON integer beyond float64's exact range. It also rewrote 1.0 to 1, HTML-escaped angle brackets and ampersands, and re-sorted every key. A tool_use block's "input" is arbitrary model-generated JSON. A long numeric id inside tool arguments could be silently corrupted on a route documented as passthrough. rewriteModelField now walks the response with a JSON decoder and copies every field but "model" as raw bytes. Add a test with a 20-digit integer, a 1.0 float, and a literal "<" character inside tool_use input. It asserts the response is byte-identical to the source apart from the model field. Add a negative test for the anthropic-beta header allowlist. It asserts Authorization and Cookie never reach the upstream. It also asserts the upstream sees exactly one x-api-key value: the gateway's own key, never the client's. A single Get() call could miss a second, leaked value, since Set() and Add() interact in a way a naive check would not catch. The test checks the Values() length explicitly. Add a test case for the native passthrough route's prompt-cache billing. It fails if cache_creation_input_tokens or cache_read_input_tokens drop out of the billed prompt count. This closes a production billing change that previously carried no coverage at all. Decouple response delivery from usage extraction on the passthrough branch. A response body read failure, or a malformed usage field, no longer both discard the response and misreport the failure as an upstream connection error. A read failure still returns an error, now correctly classified as a translation failure. A malformed usage field no longer blocks the response at all: the client still gets it, and the existing zero-usage estimate takes over for accounting. Return the adapter's own usage value on a chatCompletion error in the translated path, instead of discarding it. This has no effect while streaming stays rejected up front. It prevents the same critical bug from resurfacing once streaming support lands. Normalize tool_result content to a plain string before applying the error prefix. The prior version only prefixed a Go string. Anthropic's array content form, which Claude Code sends, lost the is_error signal entirely. Absent content now becomes an empty string instead of the JSON null OpenAI rejects. Treat "stream": null the same as an absent key. The prior fix for a non-boolean stream value also rejected null. An SDK that serializes an unset optional field as null must still get an ordinary non-streaming response. Drop a yaegi-check assertion that could not fail. The fixture configures no cache, so the header it checked was never set regardless of what the code under test did. A mutation pass confirmed this: it deleted the line the assertion was meant to guard, and make yaegi-check still reported OK. Document that this route now intercepts POST /v1/messages unconditionally, even under passthroughUnknown, where it previously fell through to the next handler. Add the note to both the new README section and Known limitations. Correct the passthrough table's claim about response bytes to describe the actual guarantee. |
||
|
lukaszraczylo
|
d0b55a37e9 |
fix(webui): match the health detail row color to its badge, add tests
ProvidersView.vue's accordion detail row hardcoded text-destructive red for any non-closed health state, while the trigger row's Badge deliberately used grey (secondary) for half-open and red (destructive) only for open. Add healthDetailClass so the detail row's color always matches the badge's own severity, instead of overstating a half-open probe as an error. Add a describe block for formatUntil to format.spec.ts, covering the unset sentinel, undefined, a future timestamp, and a past one. It had no test coverage since being added in the previous round. |
||
|
lukaszraczylo
|
46a3577162 |
docs: fix the provider health section and update the 6h default
The Provider health section claimed a broken provider stops being re-probed "every single interval forever", while its own third bullet said the opposite: under stock config the interval remained the binding constraint and nothing was actually suppressed. Rewrite the opening paragraph and the bullet to state the real relationship plainly: the backoff window only ever reduces re-probe frequency once it exceeds discoveryInterval, and the shipped defaults are chosen so that holds. Update every remaining reference to the old 30m maxOpenDuration default to the new 6h value, in the BreakerConfig reference table and the Provider health section, and explain what a misconfigured relationship between discoveryInterval and maxOpenDuration does. |
||
|
lukaszraczylo
|
a18580e930 |
fix(registry): make the breaker actually suppress re-probing
Address the design finding and remaining defects from round 3 of adversarial review. Raise defaultBreakerMaxOpenDuration from 30m to 6h. Round 2 fixed the breaker so it never retries more than plain interval throttling, but left it cadence-neutral: with maxOpenDuration shorter than the 1h default interval, the interval always dominated and backoff never suppressed anything. Measured over 24h with stock config, admits equaled the interval-only baseline exactly. At 6h, a permanently broken provider now backs off 1h, 2h, 4h, 6h, 6h, roughly 5 attempts a day instead of 24. The 24h cadence test now asserts admits are strictly fewer than baseline, not merely no worse. Correct the emitted openUntil to account for the interval floor. Because the interval gate applies unconditionally, the real next probe time is max(openUntil, lastRefresh+interval), not the raw backoff value alone. On pure defaults a fresh trip set the raw value to +1m while the real next probe was at +1h, a 59 minute skew the dashboard rendered as a countdown that reached zero and then silently vanished. snapshot now emits the corrected value while open; closed and half-open are unaffected. Revert a canceled half-open probe to open instead of leaving it at half-open. A half-open breaker reports healthy by design, so a canceled probe previously left a still-broken provider reading healthy until the next attempt happened to land. Reverting costs nothing extra, since the interval gate already governs when that next attempt is admitted either way. Rewrite the discovery-suppression claims in doc comments: the earlier wording claimed re-probing stops "every single interval forever" under all conditions, which the interval-floor fix from round 2 already made false, and which contradicted this file's own surrounding explanation. Both now state plainly that suppression only occurs once backoff exceeds discoveryInterval. Add a sentence to recordHealthLocked's doc comment stating that context.DeadlineExceeded counts as a plain failure, matching recordProviderAttempt's SHOULD-1 ruling, and a named test proving it. Tests: two new snapshot tests pin the openUntil correction in both directions. The cadence test's baseline comparison is now strict. A new test proves the zero value truly disables the breaker rather than tripping on the first failure. A new test proves a canceled context never gets misclassified as a failure, completing the bidirectional check the existing success-direction test already had. The concurrent test now asserts backoff stays within its ceiling and health stays a valid state on every iteration, rather than only checking for panics and data races. |
||
|
lukaszraczylo
|
8a3f9e7d00 |
feat(webui): surface discovery health state in the Providers tab
Add healthState and openUntil to AdminProviderView, and a formatUntil helper alongside the existing formatAgo/formatTimestamp pair, rendering a future ISO timestamp as "in Ns". ProvidersView.vue shows a Badge next to a provider's other status indicators, destructive for open and secondary for half-open, using the existing shadcn-vue Badge component rather than a hand-rolled indicator. A closed provider, the healthy common case, renders no badge at all, matching the existing convention for the per-model rate badge. The expanded accordion content adds a matching detail row. This was the brief's stated motivation for the admin surfacing work: a broken provider looked identical to a working one in the dashboard. That gap was previously closed at the API layer only; this closes it in the UI clients actually see. Regenerates admin_assets_gen.go from the rebuilt webui via make admin-ui, per this repository's own convention of committing the generated asset alongside its source change. |
||
|
lukaszraczylo
|
e82dabb1dc |
docs: document the discovery circuit breaker
Add BreakerConfig to the full configuration reference, alongside RetryConfig and CacheConfig, and add a breaker row to the top-level Config table. Add a new Provider health section explaining the three states, the failure classification, the neutral treatment of a canceled context, and the scope limitation: the signal reflects the discovery endpoint only and must not be the sole reason to refuse routing to a provider. Also document the new healthState and openUntil fields GET /admin/api/overview returns per provider, in the same sentence that already lists discoveryEnabled, modelRates, and modelMeta. |
||
|
lukaszraczylo
|
92b1af0705 |
test(yaegi-check): exercise the discovery circuit breaker interpreted
Add a breaker probe to the yaegi-check gate, alongside the existing handler coverage rather than replacing it. Drives a provider whose /v1/models 401s, the xiaomi production case, through the full closed to open to half-open to closed cycle against the real interpreter. This closes a gap adversarial review found: registry.go added a new interpreted named int32 type with a String method and a new classification call in recordHealthLocked, and nothing exercised either under Yaegi. limits.go's matchesSentinel already documents that this class of divergence between compiled and interpreted behavior is invisible to go test and only an interpreted probe can catch it. The probe configures a short breaker window (2 failures, 3 to 6 second backoff) and a 1 millisecond discovery interval on a dedicated "brk" provider, so the interval floor never binds and the breaker's own timing is what the assertions observe. It proves the breaker opens after sustained failures, suppresses discovery traffic for the whole open window, and closes again once the provider recovers. |
||
|
lukaszraczylo
|
24167b16f6 |
fix(registry): correct breaker cadence, classification, and cancellation
Address five findings from adversarial review of the discovery circuit breaker. All are behavioral corrections to registry.go; none touch the request path. Interval floor (blocking). tryBeginRefresh let an open breaker's own backoff window replace the plain discoveryInterval gate instead of adding to it. Since defaultBreakerMaxOpenDuration (30m) is shorter than defaultDiscoveryInterval (1h), a permanently broken provider was probed roughly twice as often as before this feature existed. The interval gate now applies unconditionally, closed or open alike, so backoff can only ever lengthen the retry cadence, never shorten it. Explicit failure classification. recordHealthLocked reused isTransient's retry-shaped semantics wholesale, including its exclusion of errRequestBuildFailed. A provider whose baseURL is malformed produced that exact error on every discovery cycle and never opened its breaker, because the old code routed it to a successful outcome. Health classification is now explicit: any non-nil, non-canceled error counts as a failure. Neutral cancellation. A context.Canceled outcome no longer counts as success or failure. Treating it as success previously closed an already-open breaker and wiped its backoff whenever a half-open probe was interrupted at shutdown, reporting a still-broken provider healthy. Zero-value safety. A providerState assembled without going through newModelRegistry now behaves like plain interval-only gating instead of opening on the first failure with a zero backoff, matching the type's own documented claim. Backoff overflow ceiling. Repeated doubling with no ceiling on maxOpenDuration could wrap the backoff negative after roughly 28 probes, silently disabling it. validateBreakerConfig now caps maxOpenDuration at 24 hours. Also renames modelRegistry.providerUsable and providerState.usable to discoveryHealthy, and documents that the signal reflects the discovery endpoint only. It must never be the sole reason to refuse routing to a provider whose request path works fine. feat/failover should combine it with a separate request-path signal fed by recordProviderAttempt. Tests: a 24-hour simulated cadence test proves the fixed gate admits no more attempts than plain interval throttling did before this feature. A concurrent test exercises the breaker fields under -race for the first time. Two tests replace a vacuous predecessor and distinguish neutral cancellation from treating it as success. One test proves errRequestBuildFailed opens the breaker. One test proves the zero value never opens. A rewritten healthy-provider test asserts behavior instead of reading the internal openUntil field directly. |
||
|
lukaszraczylo
|
25e73cb585 |
fix(routes): address review findings on the Anthropic Messages route
Preserve usage when response translation fails after a successful, billed upstream call. The route previously discarded already-billed tokens on that path. A client could induce unlimited free usage against any budget via a truncated tool-call response. Reject a streaming request with 400 instead of 501. Check it before the rate limiter runs. A 501 made anthropic-sdk-python retry the request, since it retries any status of 500 or higher. Each retry also counted against the caller's rate limit before this fix. Preserve block order when translating an Anthropic message to OpenAI's shape. A tool_result block before other content must stay before it. OpenAI otherwise rejects the translated conversation. Echo the client's requested model alias in both the passthrough and the translated response paths. The passthrough branch previously returned the upstream's own model id instead. Forward the anthropic-beta header to an anthropic-type upstream through an explicit allowlist. Never forward Authorization, x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own credential injection win regardless. Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads cache_creation_input_tokens and cache_read_input_tokens too. It folds all three into the billed prompt count. This applies to this route, the existing outbound Anthropic path, and the native passthrough route. Extract the metered-request pipeline shared by the unified route and this route into runMeteredCall. Thread an envelope writer through it so each route answers in its own error shape. This removes the duplication the original brief warned against. It also lets the rate-limit and body-admission error paths answer in this route's own shape. Treat a non-boolean "stream" value as a streaming request. A string or number value previously bypassed the streaming check silently. Emit max_completion_tokens instead of max_tokens for OpenAI's reasoning models. Those models reject max_tokens with a 400. Preserve tool_result.is_error by prefixing string content with "Error: ". OpenAI's tool message carries no error field of its own. Log a warning when the thinking field is dropped instead of discarding it silently. Classify a response-translation failure distinctly in the log. An operator must never see a genuine translation bug reported as an upstream connection error. Document the route in README.md. Cover both auth headers, the non-streaming limitation, and which providers translate versus pass through. Add the reviewer's yaegi-check probes to tools/yaegi-check/main.go, adjusted for the fixes above. |
||
|
lukaszraczylo
|
619931b2b2 |
feat(admin): surface provider health state in overview
Add HealthState and OpenUntil to adminProviderView, GET /admin/api/overview's per-provider row. HealthState mirrors the discovery circuit breaker's state: closed, open, or half-open. OpenUntil is the zero time.Time unless the breaker is open. A provider with discovery disabled, or one that has never failed a refresh, always reads closed. This keeps the dashboard's default view unchanged for a healthy deployment. Both fields come from providerSnapshot, read in the same locked pass as the existing lastRefresh/lastErr fields, so a concurrent finishRefresh can never hand the dashboard a breaker state that disagrees with the rest of the row. |
||
|
lukaszraczylo
|
e9ce6b873d |
feat(registry): add per-provider discovery circuit breaker
Add a three-state circuit breaker to providerState: closed, open, and half-open. It is driven only by discovery-refresh outcomes, never by request-path traffic. A provider whose refresh fails breakerThreshold times in a row opens the breaker and backs off before its next half-open probe, instead of retrying every discovery interval forever. A failed probe reopens the breaker with a doubled backoff, capped at breakerOpenMax. A successful probe closes the breaker again, so a recovered provider never stays dead. The failure classification reuses limits.go's isTransient and isDeadlineExceeded, the same rule recordProviderAttempt already applies to request-path attempts. It does not add a second classification. Add Config.Breaker (BreakerConfig), validated and defaulted by validateBreakerConfig. Every default preserves current behavior for a provider that never fails: a closed breaker never changes the existing interval-based refresh gating. Export modelRegistry.providerUsable(name) as the queryable health predicate. A sibling branch (feat/failover) will use it to skip unusable providers before routing. This commit does not touch the request path or routes_unified.go. |
||
|
lukaszraczylo
|
d47dd06a46 |
feat(routes): serve inbound Anthropic Messages API
Add POST /v1/messages, serving Anthropic SDK clients directly instead of outbound-only support. A request resolved to an anthropic-type provider passes through untranslated. A request resolved to any other provider type translates to and from OpenAI's chat-completion shape through the resolved adapter's own chatCompletion, since that method's contract is OpenAI-shape in and out regardless of provider type. The route reuses admitRequest, readAndDecodeUnifiedBody, registry.resolve, withAttemptRecorder, and limiter.account exactly as routes_unified.go does, so groups, model resolution, limits, accounting, and caching semantics stay identical to the unified route. Client-facing errors use Anthropic's envelope shape. A streaming request is rejected explicitly with a 501, never silently answered non-streamed; streaming support is a follow-up branch. Auth reuses authStore.identify and logAuthEvent unchanged. presentedKey already reads both x-api-key and Authorization: Bearer, so no new credential-checking code was needed. Add openAIRequestFromAnthropic and anthropicResponseFromOpenAI to translate_anthropic.go, the mirror of the existing outbound translators, plus their supporting helpers and unit tests. |
||
|
lukaszraczylo
|
9dc67dedbb |
feat: add anonymous usage reporting, matching traefikoidc
Release builds send one anonymous "plugin loaded" ping per process, so the project can see roughly how many installs exist and which versions are in use. The payload is project name, version and timestamp. Nothing else leaves the pod: no identifiers, no hostname, no configuration, no provider names, no keys, no request data. Three things keep it from firing when it should not. Builds from source carry the devPluginVersion sentinel and stay silent, so developer checkouts and every test run never phone home. The ping is sent only after the plugin constructs successfully, so a rejected configuration is never counted as an install. A sync.Once holds it to one per process, because Traefik calls New once per route and oss-telemetry does not deduplicate on the client side. Operators of a release build opt out with DO_NOT_TRACK=1, OSS_TELEMETRY_DISABLED=1, or TRAEFIK_LLMGATEWAY_DISABLE_TELEMETRY=1. README.md documents the full payload and the opt-out; a plugin that phones home without saying so in its own README would be indefensible. oss-telemetry is the plugin's first non-stdlib RUNTIME dependency, which has consequences for the Yaegi gate: - vendor/ is no longer excluded from yaegi-check's GOPATH copy. It was excluded on the reasoning that it held only test-only testify and _test.go files are never interpreted. That reasoning no longer holds: leaving vendor out would make the interpreted copy fail to resolve an import the real plugin tree resolves fine, so the gate would stop representing production. - yaegi-check now stamps a fake release version into its GOPATH copy so New's telemetry gate is PASSED rather than short-circuited, and the vendored package is actually called under the interpreter. Import resolution alone is not coverage; a cross-package call into interpreted third-party code is its own risk. Telemetry is suppressed by env vars set before anything is interpreted, so the gate never emits a real ping. - A new assertion reads the version back through GET /admin/api/overview. Without it a silently failed stamp would leave the dev sentinel in place and the harness would still print OK while covering nothing. Verified by disabling the stamp and watching the gate fail. shouldSendTelemetry is split out from sendStartupTelemetry so the decision is table-testable without a network, since oss-telemetry's endpoint is deliberately not overridable from consuming code. Also brings the version-gated documentation up to date, since the same README edit touches it. The latest release is v1.0.3, so the pre-1.0 experimental warning has met its own stated condition and is replaced by a semver stability statement in README.md and dropped from .traefik.yml's catalog summary. The catalog now serves the released tags (verified: the v1.0.3 download endpoint returns 200), so the "not yet published" and "analyzer has not been run" claims are corrected, the catalog form of the static config is documented as usable, and examples/kubernetes.yaml pins v1.0.3 instead of v0.1.0. |
||
|
lukaszraczylo
|
0346c3ed39 |
fix(registry): log model-id collisions at WARN, not ERROR
A model id served by two configured providers is resolved deterministically by the registry, the winner is named in the message, and no request is affected. Logging it at ERROR put 144 ERROR lines in a healthy pod's boot log on a cluster with overlapping openai, openai-audio, copilot and macstudio catalogs, which is exactly the noise that hides a real failure. Adds Gateway.warnf beside logf and errorf, and a warn field on modelRegistry carrying it. The field is set by the production caller after newModelRegistry returns rather than added as a fourth constructor parameter: 45 call sites pass a logger today, almost all of them tests passing a no-op, and threading a parameter through all of them would bury a one-line severity change in unrelated churn. A nil warn falls back to log, so a registry built without one keeps its current behaviour instead of panicking or dropping the line. registry.warnf takes an already-formatted string so both call shapes stay single-variadic-argument. Yaegi v0.16.1's CFG builder panics on a call to a struct FIELD of variadic func type with two or more variadic arguments, which is the restriction the log field beside it already documents. Verified under the real interpreter, not only compiled: yaegi-check drives a genuine two-provider collision and now prints WARN. The severity test fails if the call site reverts to log, so the level cannot drift back unnoticed. Only this message moves. Genuine failures — discovery errors, metadata capture panics, encoding failures — stay at ERROR. |
||
|
lukaszraczylo
|
5a34c5a52c | merge: federated MCP auth and fan-out hardening (audit round 2) | ||
|
lukaszraczylo
|
de42192426 |
refactor(mcp): restore errors.Is for the oversize-response sentinel
Federation's tools/call oversize branch matched a string marker with strings.Contains instead of wrapping a sentinel. The stated reason was that no gate exercised this path under Yaegi, so errors.Is was unproven here. The caution was misplaced. errors.New returns a compiled *errors.errorString and fmt.Errorf a compiled *fmt.wrapError, so declaring the sentinel variable in interpreted code leaves the entire chain errors.Is walks compiled. Neither documented trap applies: the matchesSentinel trip-wire covers a plugin-declared error TYPE, and the comma-ok trap needs a plugin-declared interface. retry.go has matched errRequestBuildFailed this way through a multi-%w wrap since v0.2.0. The string check was also strictly worse: err.Error() is itself an interpreted-to-compiled dispatch, so it verified nothing extra, and matching on text that partly comes from the backend admits false positives a sentinel cannot have. Rather than settle this by argument, tools/yaegi-check now drives POST /mcp tools/call against a backend whose response exceeds the cap, and asserts the interpreted plugin reports "response too large". Breaking the match makes the gate fail with "upstream error", so the probe has teeth. This closes the objection permanently instead of answering it in a comment. Also in doBackendJSONRPC, check the HTTP status before the size cap. A backend failing with a large HTML error page now reports "upstream returned HTTP 500", which is actionable, rather than sending the operator after a payload-size problem that does not exist. Two comment corrections carried forward from review: the fan-out semaphore is not the same shape as limiter.spawnTokens (spawnTokens drops work via a default case, this one blocks and must never skip a server), and the shared fan-out deadline gives later batches the remaining budget, not a fresh one, so uniformly slow backends yield a partial list. |
||
|
lukaszraczylo
|
452be5151c |
fix(mcp): stop weighting tools/list budget, widen fan-out, split caps
Addresses critical finding 3 and important findings 3-4 from security review round 2, 2026-08-22. Critical finding 3 — the tools/list-per-backend weight charged against the caller's own requests-per-minute/day budget (security audit finding 1b) was not default-preserving: both live groups have an empty MCP allow-list (all 11 servers), so it silently cut every configured requests-per-minute budget by ~11x with no config change, and examples/kubernetes.yaml's shipped requests-per-minute: 10 would 429 the very first tools/list call ever made. Reverted: checkAndCountWeighted removed from limits.go, handleMCPFederated back to a plain checkAndCount. The real amplification stays fully visible to an operator via countTargetRequests' existing per-server 'mcp' target-scope counters — a separate scope from the user/group budget — rather than silently consuming tenant quota. The fan-out semaphore (below) is the actual protection against the amplification finding 1 described. Important finding 3 — the fan-out semaphore (4 concurrent backends) tripled worst-case latency (907ms measured at 300ms/server, was ~300ms) and pushed the hung-backend worst case to ceil(11/4)x20s=60s, past the ~30s timeout many MCP clients use. Raised to 8, and every backend in one tools/list call now shares ONE overall toolsListBackendTimeout deadline (fanoutCtx) instead of each getting a fresh one from its own staggered start — restoring this function's original 'bounds the whole call regardless of server count' invariant even with bounded concurrency. Important finding 4 — the 4MiB response cap (sized for the tools/list fan-out's memory-amplification risk) silently applied to tools/call too, where exactly one backend is ever contacted and a legitimate result (an image, an extracted document) can be much larger. tools/call now uses mcpBackendCallResponseMaxBytes (10MiB, the pre-finding-1c budget); tools/list keeps the 4MiB cap. A response that hits either cap is detected via a cap+1 read (never silently truncated into a confusing parse failure) and reported to the client as 'response too large'. |
||
|
lukaszraczylo
|
3536c32667 |
fix(auth): let a valid key always win over IP throttle
Fixes critical finding 1 from security review round 2, 2026-08-22: identify checked the per-source-IP throttle BEFORE verifying the presented key. This plugin runs behind a Kubernetes Service, and with externalTrafficPolicy: Cluster (this operator's own deployment), r.RemoteAddr is the node's own SNAT address shared by every external caller reaching that node — not a per-tenant address. 20 failed requests from anywhere on the internet sharing that node locked out all 10 live users, including the admin dashboard's own poll. - identify now verifies the key FIRST, unconditionally: a valid key always succeeds before any throttle state is read. Every failure path still records against the caller's IP, but throttling is now pure observability (bounding log volume and tracker memory), never enforcement — an attacker with no valid key already got 'false' either way, so nothing is lost against brute force. - Replaces the per-source-IP tracker's backing store: the general- purpose memoryStore (limits.go) has no size cap and a full mutex-held sweep that scales with tracked-IP count (~83ms at 1M IPs). authFailureTracker is a bounded, two-generation, O(1)-rotation structure capped at 2x50,000 entries (important finding 1). - logAuthEvent now emits a distinct, separately rate-limited 'auth throttle:' line the first time a source crosses the limit (important finding 5), and both the routine and throttle-engaged lines report how many events they suppressed since the last line actually written, so several distinct attacker IPs in one window no longer collapse into a single indistinguishable line (important finding 6). - The success path no longer touches the throttle tracker's mutex at all (important finding 2). |
||
|
lukaszraczylo
|
d2a0c4860f |
fix(auth): reject duplicate and empty user names
Fixes finding 5 from the 2026-08-22 security audit: newAuthStore and
replaceFileUsers rejected duplicate API keys but not duplicate names, and
counters are keyed on u.name (routes_unified.go), so two users sharing a
name silently shared llmgw:user:{name}:* counters and both appeared in
buildAdminUsage; an empty name collapsed every such user into
llmgw:user::*.
- buildEntry now rejects an empty user name outright — the source of
both bugs, since duplicate names and an empty name are the same root
cause (two config entries resolving to the same counter key).
- newAuthStore and replaceFileUsers each reject a duplicate name WITHIN
their own source (two inline users, or two file users) — but not
across sources: a file user overriding an inline user of the same name
stays intentional and unaffected, per replaceFileUsers' own documented
merge semantics.
- Deliberately does NOT extend configNamePattern's character
restriction to user names: that pattern exists because provider/
mcpServers/agents names are embedded as URL path segments, which a
user name never is here. Restricting the charset further risked
rejecting real operator-chosen names this round has no visibility
into; the live users file's steve-cw/kevinsandom already pass the
non-empty check unaffected.
Also adds TestDefaultPreserving_LiveClusterShape_LoadsAndServesIdentically
(admin_test.go), modeling the live cluster's 5 friends + 4 home users and
11 MCP servers to prove all five fixes in this round load and serve
identically at that scale.
|
||
|
lukaszraczylo
|
800f63ac53 |
fix(admin): chunk usage batch reads to bound connection contention
Fixes finding 4 from the 2026-08-22 security audit: buildAdminUsage sent every user's, group's, and the total scope's counters (8 keys each) to limiter.currentUsage in ONE storeGetMulti round trip. With 1,000 users that is ~8,000 GETs pipelined through the single mutex-guarded store connection, during which every concurrent checkAndCount from live LLM traffic queues behind it (up to respCallTimeout) — and the admin dashboard polls this endpoint every 5s. Add chunkedCurrentUsage: splits the scopes slice into batches of at most adminUsageChunkScopes (200) before calling currentUsage, letting live traffic interleave between chunks while each chunk still batches its own scopes in one round trip. At or under 200 scopes (the live cluster's 9 users) this makes exactly one call, identical to today's behavior — the payload shape is unchanged, so no webui coordination is needed. |
||
|
lukaszraczylo
|
87ebee9003 |
fix(auth): throttle repeated auth failures, rate-limit failure log
Fixes finding 3 from the 2026-08-22 security audit: authStore.identify had no counter, no backoff, and no lockout on failed attempts, and logger.go wrote one unbuffered stderr line per attempt including the caller's RemoteAddr. An unauthenticated attacker got unlimited digest guesses at line rate and could drive synchronous stderr writes on the shared Traefik ingress. - Add per-source-IP failure throttling to identify, reusing the existing in-process memoryStore (limits.go) as authStore's own dedicated instance rather than a new dependency: authFailureLimit (20) failed attempts per authFailureWindow (1 minute) throttles further attempts from that IP outright, including a subsequently valid key, and resets once the window elapses. - clientIP reads r.RemoteAddr only, never a client-supplied header, matching the same trust boundary the rest of this package already assumes (routes_passthrough.go strips X-Forwarded-*/X-Auth-* before any outbound request). - Rate-limit logAuthEvent's failure line via authStore.shouldLogAuthFailure, the identical technique limiter.logStoreError already applies to store errors. Successful-auth logging stays unbounded (bounded by legitimate, already-authenticated traffic elsewhere). |