fix(target-health): record a hung federated server as a failure

The previous round suppressed every context-caused federation error. A
server that accepted the connection and then never answered therefore
read "unknown" forever whenever federation carried the only traffic,
which is the outage shape this feature exists to surface.

Split the two causes instead. A server still waiting for a fan-out slot
when the budget expires is never dialed, so the semaphore acquire now
selects on the fan-out context and records nothing for it. A server this
gateway did dial records a failure on a deadline, because it had its
chance to answer. Only a caller hanging up stays unattributable.

Document the split rule, correct a stale comment about a second record
site, and state that "unknown" reads two ways rather than three.

Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w
This commit is contained in:
2026-09-03 08:56:48 +01:00
parent 9864fcec0e
commit 5d82a6ce65
3 changed files with 156 additions and 49 deletions
+13 -10
View File
@@ -1898,11 +1898,14 @@ plugin's config.
only when the outbound call itself fails (`err != nil` from the
backend call) — a JSON-RPC-level `error` in the backend's own
response still counts as reachable, since the server answered. A
context-caused failure (the caller canceling, or the shared fan-out
deadline expiring) records nothing at all, the same client-cancel
rule the per-server proxy above applies; a genuinely hung server is
still caught by the active probe sweep or its own per-server proxy
health.
context-caused failure is split two ways: the caller hanging up
(`context.Canceled`) records nothing, the same client-cancel rule the
per-server proxy above applies, but a server this gateway actually
dialed that never answers before the shared fan-out budget expires
(`context.DeadlineExceeded`) records a real failure — it had a chance
to answer and did not. A server still waiting for a free fan-out slot
when that same budget expires — never dialed at all — records
nothing, since this gateway never gave it the chance.
- **Active probes are opt-in** (`targetHealth.enabled: true`) and lazy:
no timer, no background goroutine running on its own. A probe sweep
is triggered by `/metrics`, `GET /admin/api/targets`, `GET
@@ -1929,8 +1932,7 @@ plugin's config.
whether or not the target has ever succeeded). A target with one or
two recent failures after a success still reads `"healthy"` until the
threshold is actually crossed.
- **`"unknown"` reads three ways**, all sharing one JSON `state`
value:
- **`"unknown"` reads two ways**, both sharing one JSON `state` value:
1. never observed at all — no probe or passive traffic has reached
this target yet. `GET /admin/api/targets` omits every health field
but `state`/`consecutiveFailures` for this case.
@@ -1941,9 +1943,10 @@ plugin's config.
has never seen answer is not `"healthy"` either. Every other
health field is still present here, since there is a real
observation to show.
3. either way, `llmgateway_target_healthy` emits no sample while
`state` is `"unknown"` — a gap in the series is honest, a
fabricated `1` is not.
Either way, `llmgateway_target_healthy` emits no sample while `state`
is `"unknown"` — a gap in the series is honest, a fabricated `1` is
not.
## Security notes