mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
ce39b8eb37ab1f5cd0f727dd9439b78ee72030b4
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
65b2f121c4 |
feat(admin): add fleet statistics and admin read endpoints
Counters (always on when admin is enabled): - last seen per user and group, written asynchronously after admission, throttled to one write per scope per minute per replica - hourly provider and model attempts, failures and timeouts - failovers per provider, rejections per hour, and unpriced (402) refusals - cache hits, misses and cost saved - request counts per target and caller Opt-in counters under admin.stats (both default off): - userModel: usage per user and model - latency: latency histograms per provider and model All counters ride the existing Redis batches. A test measures three round trips per request with defaults and with every statistic on. A cache hit takes two. A model request is no longer counted for failed failover candidates. New admin endpoints: - usage/series and usage/totals - performance, with percentiles from the histograms - catalog, with the price billing actually uses and its source - consumers, with key source, grants and model access - config, with secrets redacted The overview gains features and a pricing summary. Usage gains lastSeen. usage/models gains offset and more detail fields. Reads are chunked at 1600 keys per round trip and capped at 64000 keys per request. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
6776981675 |
feat(admin): add events feed, fleet-wide rejections and span rankings
GET /admin/api/events returns recent failures, newest first. It covers rate limits, budget and unpriced-model refusals, upstream errors, timeouts and capacity rejections. Events go into an in-process ring of 200 entries and are copied to a Redis list (llmgw:events, LPUSH plus LTRIM to 200). The copy happens only on the error path, is throttled to 50 per second per replica, and never blocks a request. When Redis is absent or down, the feed falls back to this replica's ring. Messages are fixed strings or scrubbed provider errors. Fields are assigned one at a time, because Yaegi drops values from a multi-value assignment that contains a call. Rejections per day are counted per scope and in total, in the same store call that already rolls back a rejected request. /admin/api/usage returns them as rejectionsPerDay. /admin/api/usage/models accepts span, which sums that many buckets, and prefix, which filters model ids before the limit applies. The overview reports the replica id, the instance name and the warnings logged while the middleware was built. The failover and retry fan-out warning is now logged at WARN. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
4c01c75276 |
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
72b0df5b0c |
feat(auth): let a user join several groups and hold a personal grant
A user entry can now list several groups in "groups", next to the existing "group" field. A user can also hold a personal grant with its own "providers" and optional "models" lists. Users with one group and no personal grant behave exactly as before. A request is allowed only when one grant allows the provider, the model and, for passthrough, the path. Grants never merge, so one grant cannot supply the provider while another supplies the model or the path. The personal grant uses the path list of the first group. A personal "models" list without "providers" is a configuration error. Each member group applies its own limits. Caching stays on unless a member group turns it off, and the shortest group TTL wins. The /v1/models cache uses a stable, length-prefixed key, so reloads do not grow it. Tests cover the grant union, cross-grant denials, passthrough path and model coupling, reloads and the cache key. The yaegi-check harness asserts the same cases under the interpreter. Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3 |
||
|
lukaszraczylo
|
d6eebd5a84 |
feat(target-health): track MCP and agent target health
Add a per-replica, in-memory health tracker for MCP servers and A2A agents (target_health.go). The tracker records observations from real proxied traffic (handleTargetProxy, mcp_a2a.go) and federated calls (mcpFederatedToolsList/mcpFederatedToolsCall, mcp_federation.go), and derives a three-state readout (unknown, healthy, unhealthy) from a configurable consecutive-failure threshold. Add TargetHealthConfig to Config, validated at construction. The tracker never gates routing: federation and the target proxy keep contacting every configured server regardless of its recorded health. Extend proxyResult (routes_passthrough.go) with status and clientCanceled so handleTargetProxy can classify a proxied response without proxyUpstream knowing anything about target health. Add maybeSweepTargetHealth, an opt-in, lazy, single-flight active probe sweep with no timer and no long-lived goroutine, matching this plugin's lack of a Close hook under Traefik hot reload. A probe uses the initialize/close handshake for a Streamable HTTP MCP server, a plain unread GET for a legacy /sse server, and a GET of the agent card path for an A2A agent. handleMCPServers and handleAgents now trigger a sweep at the top of each call. Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w |
||
|
lukaszraczylo
|
4bb49b4fe9 |
feat(admin): add per-model usage statistics to the charts section
The Charts view could break usage down by user, group or total, but not by model. Per-model data existed only as day-window attempt and failure counts, which measure availability rather than delivered usage. A served model now accumulates the same four metrics a user or group does. The scope kind is "model" and its id is the canonical "provider/model", so a request that fails over counts against the provider that answered it, and one bare id served by two providers ranks as two rows. The request counter is written after the response, in the batch that already writes tokens and cost. That is the first point at which the serving model is known, so no extra round trip is needed and admission never double counts. GET /admin/api/usage/models ranks models for one metric and window. It returns only models with non-zero usage, by operator requirement: the catalog runs to hundreds of models and a padded ranking is unreadable. The read stays one storeGetMulti over the catalog. The Charts view gains a Models tab with its own metric picker, and the scope picker gains the models that have traffic. |
||
|
lukaszraczylo
|
063a05e9a7 |
feat(metrics): instrument upstream latency
Add time-to-first-byte and total-duration histograms for every upstream call, observation only. No throttling, priorities, or control logic reads this data. The operator wants strain-based throttling later and needs real traffic data first. Extend watchdogBody (timeout.go) rather than add a new interception layer: it already wraps every upstream response body for provider adapters, native passthrough, and the MCP/A2A target proxy, and already observes the first successful read. armLatency arms a watchdogBody after construction, so newWatchdogBody keeps its existing signature and behavior; its own pinned tests still pass unchanged. A timestamp plus a few atomics are the only added cost; Read and Close each pay one nil check when latency collection is disabled. Expose llmgateway_upstream_ttfb_seconds and llmgateway_upstream_duration_seconds on the existing Prometheus endpoint (metrics.go). Both are in-process, per-replica histograms, never store-backed, so sum() is always the correct cross-replica aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and short completions finely, cover the bulk of real chat completions, and bracket the five-minute default request timeout. Labels are provider and stream always, model only when Config.Metrics.ModelLabel is set, matching the existing per-model opt-in convention. Add a compact per-provider, per-stream average summary to GET /admin/api/overview, sourced from the same in-process store, without a full histogram or a per-model breakdown. Every write path is gated on metricsEnabled(cfg), so a deployment with metrics disabled collects nothing and the histogram families never render. No behavior change to request handling: same responses, status codes, headers, and accounting. Fix a Yaegi-only bug this feature's own context-threading change exposed: two distinct empty struct types used as context.WithValue keys compare equal to each other under the interpreter's interface equality, though never under compiled Go. Threading a second recorder (latencyRecorder) alongside the existing attemptRecorder made this observable for the first time, since compiled tests never see it and the plugin previously carried only one such key. Give latencyRecorderCtxKey a field so its size differs from attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction before and after the fix. Fix a second bug, reproducible under compiled Go too: io.Reader may legally return its final data together with io.EOF in one call, and Go's real net/http body reader does this for small, fully-buffered responses. The original TTFB capture lived inside the err == nil branch, so it never fired for a typical small non-streaming completion. Move the check ahead of the error branch so it fires on any n > 0, leaving the pinned timer-reset and Close semantics untouched. Tests mutate the real source per case and confirm each one fails first, recorded in each test's own doc comment. |
||
|
lukaszraczylo
|
51e675acb0 |
fix(providers): address adversarial review findings F1, F4, F6, F9
F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a plain `now` variable while a background refresh goroutine read it through nowFn, racing under -race in about 14% of full-suite runs once this branch's timer-heavy tests shared the parallel phase. Guards the clock with a mutex instead. registry.go is untouched. F4: a mid-body watchdog timeout never reached provider-health accounting. limiter.recordProviderAttempt only ran at header-arrival time, so a provider that reliably stalled mid-body reported attempts=1 failures=0 forever. watchdogBody.fire now invokes the same attemptRecorder a second time, as a failure, when it fires. Reuses the existing recorder rather than adding new plumbing; this does mean two attempts are recorded for one logical request (one success at header time, one failure at watchdog-fire time), not one attempt marked failed outright. F6: watchdogBody.Read reset the timer on any err == nil read, regardless of n. A reader returning (0, nil) forever would have held the watchdog open indefinitely. Reset now only fires when n > 0. F9 (three items): - proxyUpstream passed logPrefix as a bare providerName, which newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an MCP/A2A target as a provider. watchdogBody now takes an already-formatted label from the caller instead. - errUpstream was wrapped twice: once in watchdogBody's own error, once again by every caller that already wraps it. watchdogBody now wraps only errProviderTimeout; callers' existing wraps still make errors.Is(err, errUpstream) hold. - timeout_test.go read wb.timedOut directly in one place while every other access is atomic; switched to atomic.LoadInt32. Also rewords errProviderTimeout's message from "provider request timed out" to "upstream progress stalled" (finding F3, documented not fixed per coordinator ruling): the watchdog measures time since the last successful upstream read, which can go quiet because of a slow client write between reads, not only a slow provider. The new wording no longer asserts which side caused it. New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure, TestWatchdogBody_ZeroByteRead_DoesNotResetTimer, TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider. Each mutation-verified per its own doc comment. |
||
|
lukaszraczylo
|
503fc33933 |
feat(providers): add progress-based upstream request timeout
Adapters previously issued upstream requests with no timeout at all. A provider that accepted a connection and then stalled hung the gateway request forever. newAdapterHTTPClient now sets Transport.ResponseHeaderTimeout, and every response body read (providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream) runs through a watchdogBody that resets on every successful read and cancels the request once timeout passes with no progress. The idle-progress design lets a long stream that keeps emitting tokens run indefinitely. Only silence, never total duration, ends a request. Config.RequestTimeout sets the global default (five minutes when unset). ProviderConfig.RequestTimeout overrides it per provider. Both are Go duration strings, validated at construction; a zero or negative value is rejected rather than treated as unlimited, since that would reintroduce the hang this feature fixes. Native passthrough (routes_passthrough.go) and the MCP/A2A target proxy (mcp_a2a.go) share the identical watchdog mechanism through proxyUpstream's new timeout parameter. A watchdog-fired timeout wraps errProviderTimeout and names the provider. It never wraps context.Canceled, so it stays distinguishable from a genuine client disconnect in every existing log classification. |
||
|
lukaszraczylo
|
25e73cb585 |
fix(routes): address review findings on the Anthropic Messages route
Preserve usage when response translation fails after a successful, billed upstream call. The route previously discarded already-billed tokens on that path. A client could induce unlimited free usage against any budget via a truncated tool-call response. Reject a streaming request with 400 instead of 501. Check it before the rate limiter runs. A 501 made anthropic-sdk-python retry the request, since it retries any status of 500 or higher. Each retry also counted against the caller's rate limit before this fix. Preserve block order when translating an Anthropic message to OpenAI's shape. A tool_result block before other content must stay before it. OpenAI otherwise rejects the translated conversation. Echo the client's requested model alias in both the passthrough and the translated response paths. The passthrough branch previously returned the upstream's own model id instead. Forward the anthropic-beta header to an anthropic-type upstream through an explicit allowlist. Never forward Authorization, x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own credential injection win regardless. Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads cache_creation_input_tokens and cache_read_input_tokens too. It folds all three into the billed prompt count. This applies to this route, the existing outbound Anthropic path, and the native passthrough route. Extract the metered-request pipeline shared by the unified route and this route into runMeteredCall. Thread an envelope writer through it so each route answers in its own error shape. This removes the duplication the original brief warned against. It also lets the rate-limit and body-admission error paths answer in this route's own shape. Treat a non-boolean "stream" value as a streaming request. A string or number value previously bypassed the streaming check silently. Emit max_completion_tokens instead of max_tokens for OpenAI's reasoning models. Those models reject max_tokens with a 400. Preserve tool_result.is_error by prefixing string content with "Error: ". OpenAI's tool message carries no error field of its own. Log a warning when the thinking field is dropped instead of discarding it silently. Classify a response-translation failure distinctly in the log. An operator must never see a genuine translation bug reported as an upstream connection error. Document the route in README.md. Cover both auth headers, the non-streaming limitation, and which providers translate versus pass through. Add the reviewer's yaegi-check probes to tools/yaegi-check/main.go, adjusted for the fixes above. |
||
|
lukaszraczylo
|
64126db7d4 |
docs(passthrough): drop out-of-scope "." rejection, correct %25 comment
Two doc/scope corrections from the round-3, 2026-08-22 review (FOLDED +
SHOULD-5), no behavior change to the traversal bypasses finding 4 itself
fixed:
- hasTraversalSegment rejected an exact "." segment alongside "..",
outside finding 4's actual scope: "v1/x/./y" is a legal, non-traversal
path and was silently getting 400'd. Dropped the "." check; only ".."
is rejected again, matching every other traversal rule this function
enforces.
- The double-encoding rejection's own comment claimed it "rejects an
intentionally double-encoded %25" -- overstated. "%25" is the correct,
single, standards-conforming encoding of a literal "%" (RFC 3986);
after exactly one unescape pass, a legitimate single-encoded literal
"%" is byte-for-byte indistinguishable from a genuine double-encoding.
The behavior (fail closed) is still correct and unchanged -- refusing
an occasional literal "%" is a far smaller cost than admitting a real
double-encoded traversal -- but the comment now says so honestly,
including the concrete false-positive example ("v1/100%25done" decodes
to "v1/100%done" and is rejected even though it names a real,
non-malicious resource).
Tests: added "v1/x/./y" (must not 400) and "v1/100%25done" (the
documented false positive, still 403s) to hasTraversalSegment's table.
|
||
|
lukaszraczylo
|
60036c43df |
fix(passthrough): strip dangerous upstream response headers
Close the response-header relay hole (security review finding 5, round 3, 2026-08-22): proxyUpstream copied every upstream response header to the client except hop-by-hop ones -- an upstream provider, or an in-cluster MCP/A2A target reached with no credential trust boundary of its own, could set Set-Cookie on the gateway's own origin (the same origin serving the WAN-exposed /admin dashboard), or return Access-Control-*/Strict-Transport-Security/Content-Security-Policy headers to rewrite the browser's security or CORS policy toward that origin. Added dangerousResponseHeaders (exact-match: Set-Cookie, Strict-Transport-Security, Content-Security-Policy) and dangerousResponseHeaderPrefixes (Access-Control-*), stripped in proxyUpstream right where hop-by-hop headers already are -- so both callers (handlePassthrough here, handleTargetProxy in mcp_a2a.go) get the fix for free. A deny-list, not an allowlist: grepping this repo finds it only ever reads Content-Type and Retry-After off a response (the latter only for the unified routes' own internal retry decision, never from passthrough). Everything else -- including provider rate-limit telemetry headers a client's own SDK commonly reads -- is opaque to this gateway but real passthrough transparency; an allowlist would silently break that for every header this package doesn't already know to name. This mirrors the request side's own already-hardened deny-list shape (dangerousClientHeaders/providerCredentialRetargetHeaders). Test: an upstream setting all four dangerous headers plus Content-Type, Retry-After, and an ordinary custom header proves the four are stripped while the rest reach the client unchanged. |
||
|
lukaszraczylo
|
ff819e01d7 |
fix(passthrough): catch path-parameter, backslash, and double-encoded traversal
Close three further traversal bypasses hasTraversalSegment's single-pass
decode-and-split missed (security review finding 4, round 3,
2026-08-22), each a real upstream convention:
- Path parameters ("..;/x", the Tomcat/Spring RFC 3986 Section 3.3
convention): a segment's identity is everything before its first ";".
Every segment now has its ";"-suffix stripped before the traversal
comparison.
- Backslash normalization ("..%5c..%5cx", decoding to "..\..\x"):
Windows/.NET and other backslash-normalizing upstreams treat "\" as an
equivalent separator. The decoded string now has every "\" replaced
with "/" before splitting.
- Double-encoding ("%252e%252e/x"): one unescape pass decodes this to
"%2e%2e/x" -- still containing "%". Rather than decode a second time
(which invites an unbounded decode loop), a rest that still contains
"%" after the one legitimate unescape pass is now rejected outright.
Also now rejects an exact "." segment, not just "..", closing the same
class of ambiguous-normalization bypass on a self-reference segment.
Literal ".." and single-encoded "..%2f" keep their existing, already
correct behavior; a legitimate encoded "/" within one segment (a2Fb) and
a genuine, non-traversal use of ";" on an unrelated segment are both
still forwarded.
Tests: hasTraversalSegment's table gains one case per new bypass plus
adjacent non-traversal cases proving the fix doesn't over-reject, and
the end-to-end ServeHTTP traversal test gains the same three variants
driven through the full request path.
|
||
|
lukaszraczylo
|
a221b4d2eb |
fix(passthrough): fail closed on a model peek that never closes the object
Close the residual gap in the duplicate-"model" fix (security review
finding 2, round 3, coordinator ruling): round 2 made a duplicate
top-level "model" key WITHIN the 64KiB peek window fail closed, but a
client can still pad the body so a second "model" sits just PAST
maxModelPeekBytes -- `{"model":"cheap", <64KiB padding>,
"model":"expensive"}`. The peek reports "cheap" (the only key it saw),
the body is forwarded byte-for-byte, and the upstream (last-wins on
every mainstream JSON parser) executes "expensive".
scanTopLevelModel now also reports whether it actually confirmed the
top-level object's closing '}' within the scanned window (closed).
peekPassthroughModel combines that with its own knowledge of whether the
read hit the byte cap (more of the body remains unread) to report
truncated: true only when both hold, meaning a later duplicate cannot be
ruled out. handlePassthrough denies (403, a message distinct from "model
could not be determined") whenever truncated is true, checked before
even looking at the found model -- a found model under truncation can
never be trusted.
Scoped exactly as ruled: only affects a group with a non-empty Models
list (grp.hasModelRestriction) whose body is large enough to hit
maxModelPeekBytes -- an unrestricted group never calls
peekPassthroughModel at all, unchanged from round 2 and pinned by a new
test. No live group is model-restricted today. The unified /v1/* route
remains the supported path for a legitimately large restricted-group
body.
Tests: the coordinator's own exploit body end-to-end (403, upstream
never called); the general non-duplicate case (any restricted-group body
that never closes within the window denies, not just a provably
duplicate one); the unrestricted-group-unaffected case explicitly, using
the identical exploit body; and closed's own truncation-tracking table
against scanTopLevelModel directly. One pre-existing test
(TestHandlePassthrough_BodyIntegrity_ByteIdentical_LargeBody) used a
restricted group incidentally while testing an unrelated property (byte-
for-byte forwarding) -- switched to an unrestricted group so it tests
only what its name says.
|
||
|
lukaszraczylo
|
e06d8bf5cd |
fix(passthrough): fail closed on a duplicate top-level model key
RFC 8259 permits duplicate names and every mainstream upstream parser
resolves them last-wins, while a passthrough body is forwarded verbatim.
Reporting the first occurrence let a model-restricted caller authorize
{"model":"allowed","model":"expensive"} against the allowed one and
execute the expensive one. Refuse to answer instead, matching
extractMultipartModel's existing duplicate rule.
Found by the security audit; a padded-duplicate variant that hides the
second key beyond the 64KiB peek window is tracked separately.
|
||
|
lukaszraczylo
|
e57a97d63e |
fix(passthrough): close Content-Type bypass and matcher-parity bug
Independent review measured two real gaps in the round-1 model
enforcement and fixed both here, plus documentation and comment
regressions the round introduced.
MUST FIX: peekPassthroughModel's Content-Type gate was an ALLOW-list
("application/json" or skip") and fully client-controlled. The
reviewer sent "Content-Type: text/plain" with {"model":"EXPENSIVE"}
and got a 200 from a group scoped to a cheap model. This commit
replaces the whole gating approach:
1. A group with an empty Models list has nothing for allowsModel to
reject, so the request body is never read at all
(group.hasModelRestriction). This is the overwhelming majority of
traffic, including the operator's own "home" group, and removes
the memory-buffering concern entirely for it.
2. For a group WITH a restricted Models list, the Content-Type gate
is now a SKIP-list of genuinely binary types (multipart/, audio/,
image/, video/, application/octet-stream). Every other
Content-Type, including "text/plain" or none at all, is
inspected. The attacker no longer controls whether the check
runs.
3. The peek reads at most 64KiB (maxModelPeekBytes), restored via
io.MultiReader so the remainder streams straight to the upstream
instead of a second full buffer. A bounded JSON token walk
(scanTopLevelModel) finds a top-level "model" key even when the
rest of the body is truncated past the cap.
4. A restricted group whose peek window never finds "model" is
denied (403), fail closed. This also means a Gemini passthrough
request (its model lives in the URL, never the body) is always
denied for a model-restricted group; documented in
maxModelPeekBytes' own comment, the handler's doc comment, and
the README.
Matcher parity: the round-1 version glued providerName+"/" onto
whatever the body sent unconditionally, so an already-prefixed body
("openai/x") produced "openai/openai/x" and false-403'd a legitimate
client. allowsPassthroughModel now reuses modelRegistry.
splitConfiguredProvider — the same rule resolve/resolveAgainst apply
to a client-given id — to strip a leading segment only when it names
an actually configured provider, before generating candidates.
Also in this commit:
- Restored ~26 lines of doc comments fieldalignment -fix silently
dropped when it reordered the group struct (auth.go: cache,
cacheTTL) and ProviderConfig (llmgateway.go: MetadataPath, and the
Passthrough field's own new comment). Verified nothing else was
reordered by re-running fieldalignment: 0 issues.
- Documented the path.Match footgun a non-empty PassthroughPaths
inherits: "*" does not cross "/", so ["*"] denies almost all real
multi-segment passthrough traffic rather than allowing it. Empty is
the actual allow-all value. Documented on the config field, the
allowsPassthroughPath method, and the handler's own doc comment.
- Renamed the local "model" that shadowed the outer request-model
variable near extractPassthroughUsage's call site to "respModel" —
the new gating already scopes the outer one to a narrower block, so
this was no longer a live shadow, but the rename removes any future
ambiguity between the request's model and the response's.
- README: rewrote the two "Unified vs. passthrough" table cells that
described the OPPOSITE of this feature's actual behavior, added a
"Native passthrough model authorization" section, and added
providers.<name>.passthrough / groups.<name>.passthroughPaths rows
to the config tables.
New tests: the Content-Type-bypass regression (text/plain, absent,
form-urlencoded, all now 403), the matcher-parity case (already-
prefixed body allowed), fail-closed on a binary body for a restricted
group, skip-entirely for an unrestricted group, byte-identical body
forwarding for a body larger than the peek cap and for a
ContentLength<0 (chunked) request, and a scanTopLevelModel unit table
covering truncation before/after the model field, nested values, and
malformed input.
Gate re-run in full after these changes: gofmt, go vet, fieldalignment,
golangci-lint (0 issues), go test -race (ok), make yaegi-check (OK),
make integration (ok, real Traefik + Redis). Re-ran the interpreted
Yaegi harness: RTT/req unchanged at 3.00, cmds/req unchanged at 74.00.
|
||
|
lukaszraczylo
|
e3ef4084d2 |
feat(passthrough): add model, path, and header authorization hardening
handlePassthrough checked only the provider allowlist, never the model or the path. A tenant scoped to one cheap model could reach a provider's entire native API (files, batches, fine-tuning) on the operator's key, and client identity headers reached keyless in-cluster MCP/A2A targets unfiltered. This change closes both gaps, additively: every new field defaults to reproducing prior behavior exactly. Model enforcement: when a passthrough JSON body carries a top-level "model" field, handlePassthrough checks it against the group's model glob, in both the bare and provider-prefixed form (the same dual-candidate matcher the unified route's resolveAgainst already applies). A body with no inspectable model (Gemini's URL-embedded model, a multipart upload, a non-JSON body) falls back to provider-only authorization, unchanged from before this round. Per-provider toggle: ProviderConfig.Passthrough, nil or true by default, lets an operator disable a provider's native passthrough route. A disabled provider now reports the ordinary unknown-route 404, checked in ServeHTTP's own route gate, before auth even runs. Per-group path allowlist: GroupConfig.PassthroughPaths, empty by default (matching every other allow-list field's empty-means-all semantics), restricts which native rest path a group may address. Dangerous-header stripping: proxyUpstream now additionally strips Forwarded, X-Forwarded-*, X-Auth-*, X-Remote-User, X-Remote-Groups, and Cookie on both the native-passthrough and MCP/A2A target-proxy paths, plus OpenAI-Organization, OpenAI-Project, and anthropic-beta on native passthrough only. This is an additive denylist expansion, not an allowlist inversion; ordinary headers still pass through. A dedicated test asserts a config with none of the new fields set behaves identically to before this change. The operator's home group (providers/models/mcpServers/agents/passthroughPaths all empty) stays fully unrestricted, and macstudio-rerank/parakeet-mlx/piper-tts stay reachable through native passthrough by default. |
||
|
lukaszraczylo
|
aafd2a0433 |
feat(providers): success-rate accounting
Track per-provider and per-(provider,upstream-model) upstream-attempt outcomes using the existing fixed-window counter machinery (limits.go), following mcp_a2a.go's per-target counter pattern: kind-embedded keys (kindProvider="prov", kindProviderModel="provmodel") keep a provider scope collision-safe against a user/group/MCP-target/agent of the same name. limiter.recordProviderAttempt(provider, model, resp, err) increments an "attempts" counter on every call, and a "failures" counter only for a provider-fault outcome — reusing retry.go's own isTransient(resp, err) classification exactly (transport/connection error, HTTP 429, or a 5xx), not forking it. Both counters are written at minute and day granularity only; provider health is a now-and-today question, not a billing one, so there is no month window. limiter.providerUsage batches every scope's read into one storeGetMulti round trip, mirroring targetUsage's own single-batch discipline. Wired at every chokepoint an upstream HTTP attempt resolves: - providers.go's upstreamJSON/upstreamRawBytes path (chat, embeddings, images, audio) via retry.go's retryPolicy.do, which now looks up an optional attemptRecorder from ctx and invokes it once per retry attempt (attempt-accounting: a request retried twice before succeeding reports three attempts, not one). - routes_unified.go's runUnified (chat/embeddings, streaming and non-streaming alike — both funnel through the same upstreamJSON call). - routes_media.go's three media handlers (images.generations, audio.speech, audio.transcriptions). - routes_passthrough.go's proxyUpstream (native provider passthrough) — a single un-retried attempt, provider-level only: the upstream model lives in the response body, read only after the attempt resolves. The attemptRecorder is threaded via context (providers.go's withAttemptRecorder/attemptRecorderFromContext), not a new function parameter on every call site — proxyUpstream and retryPolicy.do are both shared with paths that must NOT record provider accounting (registry.go's discovery listModels calls, and the MCP/A2A target proxy, mcp_a2a.go's handleTargetProxy), so scoping via context keeps their signatures unchanged and those callers correctly opt out by simply never wrapping their context. |
||
|
lukaszraczylo
|
08cfdb4d99 |
feat: split token counters and hour-resolution usage history
Data layer for the upcoming Vue admin charts (v0.2 data-layer task).
Tokens-in/out split (limits.go): metricTok replaced by metricTokIn
("tokin") and metricTokOut ("tokout"). account() records usage.prompt
and usage.completion independently, skipping a direction only when its
own value is zero. A TokensPerDay/TokensPerMonth limit still enforces
one TOTAL budget: tokenBudgetViolation reads both counters in one
storeGetMulti batch and sums them; the violation message still says
"tokens". Estimated usage (the request-body-size fallback) lands
entirely in tokin, documented at its call site.
Hour window (limits.go): windowHour ("hour"), bucket format
"2006010215" UTC, hourWindowTTL (48h). checkAndCount/account both write
it on every metered route; no LimitsConfig field ever names it, so
evaluateScope never evaluates it. dayWindowTTL/monthWindowTTL bumped to
35d/400d so the usage-history API's largest span always finds a live
key.
Total scope (limits.go, routes_unified.go): a synthetic
{kind:"total", id:"all"} scope, limits always nil. withTotalScope
wraps buildLimitScopes' result at every metered route (unified,
media, passthrough, mcp/a2a) so none can forget it; handleAdminAPI's
own accounting stays on buildLimitScopes alone, so admin traffic never
contributes to it.
History API (admin.go): GET /admin/api/usage/history?scope=&metric=&
window=&span= — admin-gated like the other two JSON routes, but
skips checkAndCount ahead of that same change landing for every
/admin/api/* route in a later task. Validates scope kind/metric/window
(400), span bounds per window (400), and unknown user/group id (404).
Computes every bucket key up front and reads them in ONE storeGetMulti
batch; a storeDown read is 503, never a silently-zero series. Works
against memoryStore too.
Admin usage API (admin.go): scopeUsage/adminUsageEntryView now expose
tokensInPerDay/tokensOutPerDay and their per-month counterparts in
place of the combined tokensPerDay/tokensPerMonth pair; buildAdminUsage
adds a "total" row to the response, still one storeGetMulti round trip
overall. usageKeysPerScope names the per-scope key stride (was a bare
6, now 8), closing a deferred review follow-up while the count changed
anyway.
Tests: TDD throughout — bucket/window-key math incl. UTC and hour
boundaries, the month step-back day-31 AddDate trap, split accounting,
token budget summing both directions, total-scope counting-without-
evaluating on every metered route, and the full history API (shape,
oldest-first, span bounds, validation table, 404, 503, one-batch
assertion via a counting stub). Integration: TestAdminDashboard now
fetches an hour history series for user alice and for total and
asserts a non-zero current-hour bucket.
Migration: none — old "tok" counter keys simply expire on their
existing TTL; nothing is deployed yet.
|
||
|
lukaszraczylo
|
b879b21975 |
feat: MCP/A2A registry and authed streaming proxy
Add GET /v1/mcp/servers and GET /v1/agents: group-filtered,
name-sorted registry listings of configured MCP servers and A2A
agents, with their gateway-relative proxy (and, for agents, card)
URLs.
Add handleTargetProxy for /mcp/{name}/{rest...} and
/a2a/{name}/{rest...}: reverse-proxies to the target's configured
URL. Checks run unknown name (404), group deny via allowsMCP/
allowsAgent (403), request-rate limits (429/503), then Upgrade
(501) and path traversal (400) around the proxy call itself. No
upstream credential is injected — MCP servers and A2A agents are
in-cluster targets, unlike native provider passthrough.
Extract proxyUpstream from routes_passthrough.go's handlePassthrough
into a shared reverse-proxy core (header stripping, SSE-safe
flushWriter streaming, dead-upstream 502, canceled-client log-only,
optional JSON usage teeing) used by both native passthrough and the
new target proxy. Behavior-preserving for Task 13: its tests pass
unmodified.
Validate every configured MCP-server and agent target URL at
construction (url.Parse, http/https scheme only) — a malformed or
non-HTTP target is a constructor error, not a per-request 502.
|
||
|
lukaszraczylo
|
3ce3715c79 | fix(passthrough): decompression-safe accounting, escaped-path proxying, streaming tee | ||
|
lukaszraczylo
|
8d470b173b | feat: native provider passthrough with key injection and best-effort accounting |