lukaszraczylo/traefik-llmgateway
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/traefik-llmgateway.git synced 2026-10-16 05:37:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(target-health): track MCP and agent target health

Browse Source 
Add a per-replica, in-memory health tracker for MCP servers and A2A
agents (target_health.go). The tracker records observations from real
proxied traffic (handleTargetProxy, mcp_a2a.go) and federated calls
(mcpFederatedToolsList/mcpFederatedToolsCall, mcp_federation.go), and
derives a three-state readout (unknown, healthy, unhealthy) from a
configurable consecutive-failure threshold. Add TargetHealthConfig to
Config, validated at construction. The tracker never gates routing:
federation and the target proxy keep contacting every configured
server regardless of its recorded health.

Extend proxyResult (routes_passthrough.go) with status and
clientCanceled so handleTargetProxy can classify a proxied response
without proxyUpstream knowing anything about target health.

Add maybeSweepTargetHealth, an opt-in, lazy, single-flight active
probe sweep with no timer and no long-lived goroutine, matching this
plugin's lack of a Close hook under Traefik hot reload. A probe uses
the initialize/close handshake for a Streamable HTTP MCP server, a
plain unread GET for a legacy /sse server, and a GET of the agent card
path for an A2A agent. handleMCPServers and handleAgents now trigger a
sweep at the top of each call.

Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
This commit is contained in:
lukaszraczylo
2026-09-03 01:42:21 +01:00
parent 4bb49b4fe9
commit d6eebd5a84
8 changed files with 1413 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mcp_federation.go
+12
View File
@@ -974,7 +974,13 @@ func (g *Gateway) mcpFederatedToolsList(w http.ResponseWriter, format mcpRespons
defer func() { <-sem }()
targetURL := g.cfg.MCPServers[name].URL
// feat/target-health: passive recording (target_health.go) —
// err != nil is a failure; a JSON-RPC-level resp.Error is NOT
// (the server answered, it just reported its own error), so
// it is recorded separately, below, once the shape is known.
probeStart := time.Now()
resp, err := g.mcpBackendCall(fanoutCtx, targetURL, "tools/list", struct{}{}, mcpBackendResponseMaxBytes)
g.targetHealth.record(targetKindMCP, name, err == nil, err, time.Since(probeStart), targetHealthSourceTraffic)
mu.Lock()
defer mu.Unlock()
@@ -1091,7 +1097,13 @@ func (g *Gateway) mcpFederatedToolsCall(w http.ResponseWriter, format mcpRespons
// important finding 4): tools/call always resolves to exactly ONE
// backend, so the fan-out memory-amplification argument does not
// apply here — see mcpBackendCallResponseMaxBytes' own doc comment.
// feat/target-health: passive recording (target_health.go) — measured
// around the whole outbound call, same rule as tools/list's own fan-
// out above: err != nil is a failure, a JSON-RPC-level resp.Error is
// not.
probeStart := time.Now()
resp, err := g.mcpBackendCall(ctx, targetURL, "tools/call", mcpToolCallParams{Name: toolName, Arguments: params.Arguments}, mcpBackendCallResponseMaxBytes)
g.targetHealth.record(targetKindMCP, serverName, err == nil, err, time.Since(probeStart), targetHealthSourceTraffic)
g.limiter.countTargetRequest(targetKindMCP, serverName)
if err != nil {
g.logf("federated tools/call: server %q: %v", serverName, err)