fix(failover): scope header clear to retries, fix cache pre-pass, gate health record

Fix one must-fix and one should-fix finding from re-verification of
d45125e, plus two smaller findings from the same pass.

MUST FIX: the F2 header clear ran on the first candidate attempt too,
so it ran on the default, failover-disabled path and deleted response
headers a different, earlier Traefik middleware had already staged
before this plugin ran (CORS, HSTS, request-id, and similar). Scope the
clear to i > 0, the only case that ever needs it. Add a test proving a
header staged before the gateway runs survives the default path, on
both a cache hit and a cache miss.

SHOULD FIX, coordinator ruling: drop the all-candidates cache pre-pass.
It silently served a different candidate's cached response while the
primary was perfectly healthy, which is failover with no failure. Check
each candidate's own cache entry immediately before that candidate's
own attempt instead. Request-path health, now fixed in the prior round,
is the correct mechanism for skipping a genuinely dead primary: once it
is marked unhealthy it is excluded from the candidate list entirely, so
a later request still reaches the next candidate's cache without the
dead one being dialed. Update the existing cache regression test to
drive enough requests to trip the health breaker before asserting no
repeat call, and add a dedicated test proving a healthy primary is never
bypassed for another candidate's cached response.

Guard the request-health record call with the enabled check. That state
is never read on the disabled path, so recording it there was dead work
plus a new contention point on requestHealthTracker's tracker-wide mutex
on every metered request, disabled or not. Add a test asserting the
tracker's internal state stays empty across the disabled path.

Add a test proving the 404 loud-log line is not rate-limited: drive two
404 failovers through the same Gateway in one request and assert both
lines appear, since a version that wrapped the 404 line in the same
rate limiter as the generic line would still pass every other test
(each of which only ever drives one 404 event against a fresh gate).

Document, but do not fix, that the cost guard cannot see a model
declared free via modelMeta: it resolves pricing only through the same
source real billing uses (pricing and the built-in table), which is
correct and consistent, but means an operator who declares a model free
the modelMeta way loses it as a failover candidate without realizing
why. README now says to use pricing instead for that case.

Every fix is proven by a dedicated regression test, and each was
verified by reverting the fix in place and confirming the corresponding
test fails before restoring it.
This commit is contained in:
2026-08-23 01:48:04 +01:00
parent d45125ee07
commit 0cf23345b1
3 changed files with 340 additions and 61 deletions
+23 -3
View File
@@ -899,6 +899,18 @@ never a second candidate to try.
known-zero price, are treated as equal and allowed. There is no config
flag for this — it applies unconditionally whenever failover is
enabled.
- **The cost guard cannot see `modelMeta`'s `free: true`.** It resolves
pricing only through `pricing` and the built-in table — the same
source `unifiedCostMicros` actually bills against — deliberately never
through `modelMeta` (see [Model metadata](#model-metadata) /
`ModelMetaConfig`), since `modelMeta` drives GET `/v1/models` display
only and never affects billing. A model declared free the `modelMeta`
way is therefore treated as unknown-priced by the cost guard and loses
redundancy: it is skipped as a failover candidate whenever the primary
has a known price (fail-safe, and logged like any other cost skip).
Declare the price through `pricing` (e.g. `{"inputPerM": 0,
"outputPerM": 0}`) if you want a genuinely free model to remain
eligible as a cheaper failover candidate.
- **Never mid-stream**: failover only happens while nothing has reached
the client yet. Once the first byte of a response is written, status
and body are committed and the request is never retried elsewhere.
@@ -920,9 +932,17 @@ never a second candidate to try.
actually made it.
- **Caching**: a cacheable response produced by a failover provider is
stored under that provider's own cache key, never the first candidate's.
Every candidate's cache entry is checked before any of them is actually
attempted, so a cached response from a failover provider is served
without a wasted round trip to a still-broken primary.
Each candidate's own cache entry is checked immediately before that
candidate's own attempt, in try order — never for every candidate up
front. Checking every candidate's cache before any of them was
attempted was tried and reverted: it silently served a DIFFERENT
candidate's cached response even while the actual primary was
perfectly healthy, which is failover with no failure. The
request-path health signal above is what actually keeps a genuinely
dead primary from being dialed again — once it is marked unhealthy it
is excluded from the candidate list entirely, so a later request
reaches the next candidate's cache without the dead one ever being
attempted.
- **Configurable**: `failover.maxAttempts` caps how many different
providers one request will try in total, primary included (default 3,
maximum 10) — distinct from `retry.attempts`, which retries the SAME