Counters (always on when admin is enabled):
- last seen per user and group, written asynchronously after
admission, throttled to one write per scope per minute per replica
- hourly provider and model attempts, failures and timeouts
- failovers per provider, rejections per hour, and unpriced (402)
refusals
- cache hits, misses and cost saved
- request counts per target and caller
Opt-in counters under admin.stats (both default off):
- userModel: usage per user and model
- latency: latency histograms per provider and model
All counters ride the existing Redis batches. A test measures three
round trips per request with defaults and with every statistic on. A
cache hit takes two. A model request is no longer counted for failed
failover candidates.
New admin endpoints:
- usage/series and usage/totals
- performance, with percentiles from the histograms
- catalog, with the price billing actually uses and its source
- consumers, with key source, grants and model access
- config, with secrets redacted
The overview gains features and a pricing summary. Usage gains
lastSeen. usage/models gains offset and more detail fields. Reads are
chunked at 1600 keys per round trip and capped at 64000 keys per
request.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
GET /admin/api/events returns recent failures, newest first. It covers
rate limits, budget and unpriced-model refusals, upstream errors,
timeouts and capacity rejections. Events go into an in-process ring of
200 entries and are copied to a Redis list (llmgw:events, LPUSH plus
LTRIM to 200). The copy happens only on the error path, is throttled to
50 per second per replica, and never blocks a request. When Redis is
absent or down, the feed falls back to this replica's ring. Messages
are fixed strings or scrubbed provider errors. Fields are assigned one
at a time, because Yaegi drops values from a multi-value assignment
that contains a call.
Rejections per day are counted per scope and in total, in the same
store call that already rolls back a rejected request. /admin/api/usage
returns them as rejectionsPerDay. /admin/api/usage/models accepts span,
which sums that many buckets, and prefix, which filters model ids
before the limit applies. The overview reports the replica id, the
instance name and the warnings logged while the middleware was built.
The failover and retry fan-out warning is now logged at WARN.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Federated tools/call re-initializes a session only when the backend
signals a missing session: HTTP 400 or 404, or a JSON-RPC error that
names a session. A failing tool is no longer called twice.
tools/list now:
- passes each tool object through and rewrites only its name
- follows nextCursor up to a page cap and keeps the earlier pages if a
later page fails
- skips tools with no name
- resolves name collisions to the server that tools/call routes to
The handshake sends notifications/initialized. tools/call forwards
_meta. Multi-line SSE data is joined. A2A target URLs are joined with
url.JoinPath and keep their query string. agents[].card must start with
"/". Logged target URLs and errors are scrubbed of credentials.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
The previous round suppressed every context-caused federation error. A
server that accepted the connection and then never answered therefore
read "unknown" forever whenever federation carried the only traffic,
which is the outage shape this feature exists to surface.
Split the two causes instead. A server still waiting for a fan-out slot
when the budget expires is never dialed, so the semaphore acquire now
selects on the fan-out context and records nothing for it. A server this
gateway did dial records a failure on a deadline, because it had its
chance to answer. Only a caller hanging up stays unattributable.
Document the split rule, correct a stale comment about a second record
site, and state that "unknown" reads two ways rather than three.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
mcpFederatedToolsList's fan-out and mcpFederatedToolsCall both
derive their outbound context from r.Context(). A client cancel
(context.Canceled) or the shared fan-out deadline expiring
(context.DeadlineExceeded) is not evidence about a server's own
health, so neither is recorded any more, mirroring
recordTargetProxyHealth's own client-cancel rule. A server that is
genuinely hung is still caught by the active probe sweep or its own
per-server proxy health.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Combines three closely-coupled target_health.go fixes that all touch
targetHealthTracker.record/stateForLocked in the same commit, since
splitting them would leave an intermediate commit that fails to
compile or a test asserting behaviour its own commit does not yet
implement.
F1: record gains a rawURL parameter and scrubs err's text through
sanitizeProviderErr before storing lastError, so a target URL
configured with an embedded credential can never leak through GET
/admin/api/targets. The scrub runs before truncateRunes so a long
embedded URL cannot survive truncation unscrubbed. Every probe, the
per-server proxy hook (mcp_a2a.go), and both federation record
sites (mcp_federation.go) now pass their target's configured URL
through.
F3: maybeSweepTargetHealth re-checks the elapsed interval after
winning the single-flight CAS, closing a window where a caller's
own stale interval read could still win the CAS once a separate,
faster sweep had already started and finished within the same
interval.
F4: targetHealthEntry gains everOK, set on record's success branch.
stateForLocked now reads a target that has never once succeeded as
unknown rather than healthy, even below failureThreshold — the
threshold still protects against a single transient first failure,
it just does not promote a target this tracker has never seen
answer to healthy. targetHealthSnapshot gains observed so the admin
view can still tell apart never-observed (omit every other field)
from observed-but-never-succeeded (keep them) despite both reading
state unknown. The now-redundant lastOK field is removed.
Also included: target_health_test.go's own test-hygiene fixes
(F6) — an idempotent, t.Cleanup-registered unblock ahead of
srv.Close so a failing assertion cannot leave a released server
handler blocked until the test binary's own timeout, and an atomic
hit counter in the disabled-sweep test — since they land in the
same file as the tests above.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
Add a per-replica, in-memory health tracker for MCP servers and A2A
agents (target_health.go). The tracker records observations from real
proxied traffic (handleTargetProxy, mcp_a2a.go) and federated calls
(mcpFederatedToolsList/mcpFederatedToolsCall, mcp_federation.go), and
derives a three-state readout (unknown, healthy, unhealthy) from a
configurable consecutive-failure threshold. Add TargetHealthConfig to
Config, validated at construction. The tracker never gates routing:
federation and the target proxy keep contacting every configured
server regardless of its recorded health.
Extend proxyResult (routes_passthrough.go) with status and
clientCanceled so handleTargetProxy can classify a proxied response
without proxyUpstream knowing anything about target health.
Add maybeSweepTargetHealth, an opt-in, lazy, single-flight active
probe sweep with no timer and no long-lived goroutine, matching this
plugin's lack of a Close hook under Traefik hot reload. A probe uses
the initialize/close handshake for a Streamable HTTP MCP server, a
plain unread GET for a legacy /sse server, and a GET of the agent card
path for an A2A agent. handleMCPServers and handleAgents now trigger a
sweep at the top of each call.
Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
The federated /mcp endpoint ignored the Accept header and always
returned application/json. A real MCP client that sends Accept:
application/json, text/event-stream expects SSE framing. It fails with
no data line in SSE response when it receives plain JSON instead.
handleMCPFederated now negotiates the response format once per request
from the Accept header. It threads the result through
writeJSONRPCEnvelope, writeJSONRPCResult, and writeJSONRPCErrorResponse
as an explicit parameter. A request with no Accept header, or one that
omits text/event-stream, still gets byte-identical plain JSON. Only an
explicit text/event-stream token in the Accept list switches the
response to SSE framing. SSE framing is one data line carrying the
JSON-RPC envelope, terminated by the required blank line.
Error responses and every method path honor the negotiated format.
Notification responses stay 202 with no body regardless of Accept.
The SSE branch reuses sse.go newSSEWriter and writeData instead of a
second encoder. writeData already writes the single-frame, correctly
terminated shape this route needs.
Adds a compiled test file with a captured pre-change JSON byte
baseline. It also adds explicit SSE negotiation and framing tests. A
new yaegi-check probe exercises both framings under the real
interpreter.
Federation's tools/call oversize branch matched a string marker with
strings.Contains instead of wrapping a sentinel. The stated reason was
that no gate exercised this path under Yaegi, so errors.Is was unproven
here.
The caution was misplaced. errors.New returns a compiled
*errors.errorString and fmt.Errorf a compiled *fmt.wrapError, so
declaring the sentinel variable in interpreted code leaves the entire
chain errors.Is walks compiled. Neither documented trap applies: the
matchesSentinel trip-wire covers a plugin-declared error TYPE, and the
comma-ok trap needs a plugin-declared interface. retry.go has matched
errRequestBuildFailed this way through a multi-%w wrap since v0.2.0.
The string check was also strictly worse: err.Error() is itself an
interpreted-to-compiled dispatch, so it verified nothing extra, and
matching on text that partly comes from the backend admits false
positives a sentinel cannot have.
Rather than settle this by argument, tools/yaegi-check now drives POST
/mcp tools/call against a backend whose response exceeds the cap, and
asserts the interpreted plugin reports "response too large". Breaking
the match makes the gate fail with "upstream error", so the probe has
teeth. This closes the objection permanently instead of answering it in
a comment.
Also in doBackendJSONRPC, check the HTTP status before the size cap. A
backend failing with a large HTML error page now reports "upstream
returned HTTP 500", which is actionable, rather than sending the
operator after a payload-size problem that does not exist.
Two comment corrections carried forward from review: the fan-out
semaphore is not the same shape as limiter.spawnTokens (spawnTokens
drops work via a default case, this one blocks and must never skip a
server), and the shared fan-out deadline gives later batches the
remaining budget, not a fresh one, so uniformly slow backends yield a
partial list.
Addresses critical finding 3 and important findings 3-4 from security
review round 2, 2026-08-22.
Critical finding 3 — the tools/list-per-backend weight charged against
the caller's own requests-per-minute/day budget (security audit finding
1b) was not default-preserving: both live groups have an empty MCP
allow-list (all 11 servers), so it silently cut every configured
requests-per-minute budget by ~11x with no config change, and
examples/kubernetes.yaml's shipped requests-per-minute: 10 would 429 the
very first tools/list call ever made. Reverted: checkAndCountWeighted
removed from limits.go, handleMCPFederated back to a plain checkAndCount.
The real amplification stays fully visible to an operator via
countTargetRequests' existing per-server 'mcp' target-scope counters —
a separate scope from the user/group budget — rather than silently
consuming tenant quota. The fan-out semaphore (below) is the actual
protection against the amplification finding 1 described.
Important finding 3 — the fan-out semaphore (4 concurrent backends)
tripled worst-case latency (907ms measured at 300ms/server, was ~300ms)
and pushed the hung-backend worst case to ceil(11/4)x20s=60s, past the
~30s timeout many MCP clients use. Raised to 8, and every backend in one
tools/list call now shares ONE overall toolsListBackendTimeout deadline
(fanoutCtx) instead of each getting a fresh one from its own staggered
start — restoring this function's original 'bounds the whole call
regardless of server count' invariant even with bounded concurrency.
Important finding 4 — the 4MiB response cap (sized for the tools/list
fan-out's memory-amplification risk) silently applied to tools/call too,
where exactly one backend is ever contacted and a legitimate result (an
image, an extracted document) can be much larger. tools/call now uses
mcpBackendCallResponseMaxBytes (10MiB, the pre-finding-1c budget);
tools/list keeps the 4MiB cap. A response that hits either cap is
detected via a cap+1 read (never silently truncated into a confusing
parse failure) and reported to the client as 'response too large'.
Fixes finding 2 from the 2026-08-22 security audit: the per-server
goroutines in mcpFederatedToolsList carried only defer wg.Done(), unlike
every other off-request goroutine in this package (limiter.spawn,
modelRegistry.refreshProvider/captureModelMetadata), which all guard
against a panic crashing the whole shared Traefik process. This codebase
has repeatedly hit interpreter-level panics (registry.go reflect.Set,
cache.go CFG index-out-of-range, providers.go errors.As), so an
unrecovered panic here is not hypothetical.
Add the same defer-recover pattern used at the other three sites: a
panicking backend now degrades to "that server failed" and is logged,
exactly like an HTTP error from it, instead of taking down the process.
Fixes finding 1 from the 2026-08-22 security audit: mcpFederatedToolsList
spawned one unbounded goroutine per allowed MCP server, each able to issue
up to 4 sequential HTTP calls reading up to 10MiB per response, while
checkAndCount charged the caller only 1 request regardless of how many
backends the fan-out actually contacted. With the 11 production servers,
one client tools/list call could drive up to 44 outbound requests and
hold ~110MiB concurrently, all for the cost of a single request against
the caller's own rate limit.
- Bound the fan-out to mcpFederatedFanoutConcurrency (4) concurrent
backends via a buffered-channel semaphore, the same shape
limiter.spawnTokens already uses.
- Add limiter.checkAndCountWeighted so a tools/list call charges
len(allowedMCPServerNames) requests against the caller's
requests-per-minute/day budget instead of undercounting the fan-out as
one; every other route keeps weight 1 via checkAndCount's own
unchanged wrapper.
- Shrink the per-backend response cap from maxRequestBytes (10MiB, sized
for a client's chat/embeddings body) to mcpBackendResponseMaxBytes
(4MiB, sized for a JSON-RPC reply), bounding worst-case concurrent
memory to 4 x 4MiB = 16MiB.
Review round 2 fix batch for the federated /mcp endpoint, driven by a
live probe against all 11 production MCP servers behind this gateway
through the byte-pipe proxy (session-less tools/list): 8 answer bare,
readitall and fetch reject it and need a session handshake, playwright
is a legacy HTTP+SSE-transport server (400 Missing sessionId) that no
handshake fixes.
MF1: every backend call now runs under its own context.WithTimeout
derived from r.Context() — toolsListBackendTimeout (20s) per fan-out
call, toolsCallBackendTimeout (120s) for tools/call, named constants.
MF2: doBackendJSONRPC rejects any non-2xx upstream status as an error
(previously unchecked — a non-JSON-RPC error body only failed by
accident of not parsing). parseBackendJSONRPC additionally rejects a
validly-JSON body that carries neither "result" nor "error" — the
exact silent-failure shape that would otherwise decode into an
all-zero jsonrpcResponse and hand the client a bare
{"jsonrpc":"2.0","id":1}. mcpFederatedToolsCall carries a
defensive final guard against the same shape before relaying.
MF3: mcpBackendCall is now bare-first-with-fallback — attempt the
bare call; on a JSON-RPC error OR HTTP 4xx (never a network failure,
timeout, or 5xx), retry once via initialize -> capture Mcp-Session-Id
-> retry with the header -> best-effort DELETE close. A legacy
HTTP+SSE-transport server (URL path ending "/sse") is excluded from
federation entirely before ever reaching this — isLegacySSETransportURL,
folded into allowedMCPServerNames — since no handshake fixes a
different transport; it stays reachable via the per-server proxy. When
every attempted server fails, tools/list returns a JSON-RPC internal
error ("no MCP server reachable") instead of a silently-empty list.
SF6: defaultMCPProtocolVersion -> "2025-11-25"; a caller's own
protocolVersion is echoed back only from a small explicit allowlist
(knownMCPProtocolVersions), falling back to the default otherwise —
never echoes an arbitrary caller string as this gateway's own claimed
capability.
SF8: GET/DELETE/etc. on exactly /mcp is now 405 with Allow: POST,
checked before auth (llmgateway.go).
Folded minors: countTargetRequests batches per-target counter writes
for tools/list's whole fan-out into ONE storeIncrMulti call after
wg.Wait() (was one call per server); every "actually contacted" doc
comment now says "attempted" (attribution is attempt-, not
success-, accounting); lastSSEDataLine documents its no-cross-message-
id-correlation limitation; routes_unified.go's stale buildLimitScopes/
withTotalScope comments (which claimed handleAdminAPI calls
buildLimitScopes directly — it never has) are corrected; jsonrpcRequest
documents why its "jsonrpc" field is decoded but never validated.
Tests: ~30 new/updated cases covering per-backend timeout derivation
(via the shorter-of-two-context-deadlines technique, no real 20s/120s
waits), non-2xx rejection, the empty-envelope guard end-to-end,
handshake-trigger classification (table-driven: JSON-RPC error and
HTTP 4xx trigger, HTTP 500 and network failure do not), the full
handshake sequence (bare -> initialize -> retry -> close, asserted
call-by-call), handshake-gives-up-too, all-servers-failed, legacy-SSE
exclusion (never contacted, in both tools/list and tools/call),
protocol-version allowlist (all 4 versions + one rejected), 405/Allow,
and best-effort session-close failure never surfacing to the caller.
New POST /mcp: a single JSON-RPC 2.0 endpoint aggregating every MCP
server the caller's group can reach, for clients built against the
earlier pre-plugin gateway's one aggregated endpoint (pugbot's
mcpclient, agentkit) rather than this plugin's per-server
/mcp/{name}/... proxy.
Unlike the per-server proxy, this cannot be a byte-level reverse
proxy: aggregating N servers' tools/list results, or routing one
tools/call to the single right server, both require actually parsing
the JSON-RPC envelope. The gateway answers initialize/ping locally
and, for tools/list/tools/call, issues its own outbound JSON-RPC
call(s) to the relevant backend server(s) per incoming request — no
session state persisted across separate client requests, matching the
per-server proxy's own complete absence of gateway-side session
tracking. A backend may answer bare JSON or an SSE stream; both are
accepted (parseBackendJSONRPC).
- tools/list fans out concurrently to every allowed server and merges
results, prefixing each tool "<serverName>_<toolName>" — the
underscore convention the earlier agentgateway used and that
pugbot/agentkit already persist in their own tool-id records (e.g.
brave-search_brave_web_search, verified live). A failing/unreachable
server is skipped, not a whole-call failure.
- tools/call resolves the target by LONGEST matching
"<serverName>_" prefix against the caller's allowed servers
(disambiguates a server name that prefixes another), strips it,
forwards, and relays the result/error under the CLIENT's own id. An
unresolvable prefix (unconfigured, or authz-excluded) is a JSON-RPC
invalid-params error, not a 404.
- Reuses Feature B's countTargetRequest for per-target attribution —
every backend server actually contacted increments its own scope.
- notifications/* -> 202 no body; unknown method -> JSON-RPC
method-not-found; malformed body -> JSON-RPC parse error with
id:null (spec-required, not omitted).
No new limits config surface. Tests cover prefix resolution
(table-driven, incl. the ambiguous-prefix case), SSE/JSON backend
parsing, group-restricted aggregation (a denied server is never even
contacted), call routing/prefix-stripping, backend error relay,
unreachable-server handling, unknown-prefix errors, and per-target
counter attribution. README's MCP and A2A section documents the
federation contract and naming convention; the architecture diagram
and the total-scope/rejected-before-quota notes are updated to
include the new route.
lukaszraczylo