fix(target-health): record a hung federated server as a failure

The previous round suppressed every context-caused federation error. A
server that accepted the connection and then never answered therefore
read "unknown" forever whenever federation carried the only traffic,
which is the outage shape this feature exists to surface.

Split the two causes instead. A server still waiting for a fan-out slot
when the budget expires is never dialed, so the semaphore acquire now
selects on the fan-out context and records nothing for it. A server this
gateway did dial records a failure on a deadline, because it had its
chance to answer. Only a caller hanging up stays unattributable.

Document the split rule, correct a stale comment about a second record
site, and state that "unknown" reads two ways rather than three.

Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w

mcp_federation.go

@@ -970,28 +970,53 @@ func (g *Gateway) mcpFederatedToolsList(w http.ResponseWriter, format mcpRespons
 				}
 			}()
 
-			sem <- struct{}{}
+			// G4, feat/target-health review round 2: acquiring the
+			// semaphore via select, not a bare blocking send, lets a
+			// server still parked here when fanoutCtx expires return
+			// WITHOUT ever calling mcpBackendCall — it was never dialed,
+			// so it has nothing to say about that server's own health
+			// (below). Without this, a bare `sem <- struct{}{}` still
+			// unblocks once fanoutCtx dies (an earlier holder's own call
+			// fails fast against the already-expired context and frees
+			// its slot), but by then mcpBackendCall would be called with
+			// a context that is already done — indistinguishable from a
+			// server that WAS dialed and genuinely timed out, which is
+			// exactly the over-suppression this fix removes.
+			select {
+			case sem <- struct{}{}:
+			case <-fanoutCtx.Done():
+				mu.Lock()
+				failed = append(failed, name)
+				mu.Unlock()
+				return
+			}
 			defer func() { <-sem }()
 
 			targetURL := g.cfg.MCPServers[name].URL
 			// feat/target-health: passive recording (target_health.go) —
 			// err != nil is a failure; a JSON-RPC-level resp.Error is NOT
-			// (the server answered, it just reported its own error), so
-			// it is recorded separately, below, once the shape is known.
+			// (the server answered, it just reported its own error). The
+			// record call right below already covers both: it reads ok
+			// from err == nil alone, so a non-nil resp.Error alongside a
+			// nil err still records success here — the failed-list
+			// append further down (used only for the merged tools/list
+			// result and the loud-failure short-circuit) is a separate
+			// concern and never touches targetHealth itself.
 			probeStart := time.Now()
 			resp, err := g.mcpBackendCall(fanoutCtx, targetURL, "tools/list", struct{}{}, mcpBackendResponseMaxBytes)
-			// F2: fanoutCtx is derived from r.Context() (above), so a
-			// client hang-up (context.Canceled) surfaces here as EVERY
-			// still-in-flight server's own error, and the shared
-			// toolsListBackendTimeout expiry (context.DeadlineExceeded)
-			// surfaces identically for a server that was genuinely slow
-			// AND one still parked on sem, never dialed at all — neither
-			// says anything about that server's own health, so neither is
-			// recorded, mirroring recordTargetProxyHealth's identical
-			// client-cancel rule (target_health.go). A server that is
-			// really hung is instead caught by the active probe sweep or
-			// its own per-request proxy health, not by this fan-out.
-			if err == nil || (!errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded)) {
+			// G4: this goroutine WON the select above, so it was actually
+			// dialed — only a client hang-up (context.Canceled,
+			// propagating from r.Context() into fanoutCtx) says nothing
+			// about this server's own health, mirroring
+			// recordTargetProxyHealth's identical client-cancel rule
+			// (target_health.go). context.DeadlineExceeded here means the
+			// shared toolsListBackendTimeout fired while THIS dialed
+			// server was still in flight — it did not answer within the
+			// fan-out's own budget, which is a real failure worth
+			// recording, not a client artifact (F2 over-suppressed this
+			// case; a genuinely hung server used to read "unknown"
+			// forever under federation-only traffic).
+			if err == nil || !errors.Is(err, context.Canceled) {
 				g.targetHealth.record(targetKindMCP, name, targetURL, err == nil, err, time.Since(probeStart), targetHealthSourceTraffic)
 			}
 
@@ -1116,11 +1141,15 @@ func (g *Gateway) mcpFederatedToolsCall(w http.ResponseWriter, format mcpRespons
 	// not.
 	probeStart := time.Now()
 	resp, err := g.mcpBackendCall(ctx, targetURL, "tools/call", mcpToolCallParams{Name: toolName, Arguments: params.Arguments}, mcpBackendCallResponseMaxBytes)
-	// F2: same context-caused-error guard as mcpFederatedToolsList's own
-	// fan-out, above — ctx here derives from r.Context() too, so a client
-	// cancel or the shared toolsCallBackendTimeout expiry must record
-	// nothing rather than a false failure.
-	if err == nil || (!errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded)) {
+	// G4: same dialed-server rule as mcpFederatedToolsList's own fan-out,
+	// above — ctx here derives from r.Context() too, so a client cancel
+	// (context.Canceled) must record nothing. Unlike the fan-out,
+	// tools/call has no semaphore to wait behind: the call above is
+	// always actually dialed, so context.DeadlineExceeded here means
+	// THIS server did not answer within toolsCallBackendTimeout — a real
+	// failure worth recording, not a client artifact (F2 over-suppressed
+	// this case).
+	if err == nil || !errors.Is(err, context.Canceled) {
 		g.targetHealth.record(targetKindMCP, serverName, targetURL, err == nil, err, time.Since(probeStart), targetHealthSourceTraffic)
 	}
 	g.limiter.countTargetRequest(targetKindMCP, serverName)