fix(providers): multi-wrap deadline detection was dead on 3 of 4 routes

BLOCKER (v0.22 review round, round 2): matchesSentinel's hand-rolled
Unwrap walk (round 1's SHOULD-1 fix) stopped dead the moment it hit
providers.go's own multi-%w wrapping. fmt.Errorf with more than one %w
verb builds a *fmt.wrapErrors implementing Unwrap() []error, not
Unwrap() error — a comma-ok assertion against a single-error Unwrap
interface alone never reaches past it. All three of providers.go's own
wrap sites use multi-%w (upstreamJSON's encode-failure path, "%w:
encode request body: %w"; upstreamBytes' build-request path, "%w: %w:
build request: %w"; and upstreamBytes' client.Do path, "%w: %w" — the
one a real request-context deadline reaches through). Result: a hung
upstream on the unified chat/embeddings/media routes recorded as 100%
healthy; only native passthrough (whose error is never re-wrapped by
upstreamBytes) worked.

Round 2's first attempt added a second hand-rolled interface
(multiUnwrapper, Unwrap() []error) and recursed into it — verified
correct against every wrap shape under `go test`, INCLUDING the exact
double-/triple-%w shapes providers.go itself produces. It still did not
work. tools/yaegi-check's own SHOULD-A extension (below) caught it: a
type assertion of a COMPILED concrete value (*fmt.wrapErrors, built by
the real fmt package) against an INTERPRETED interface type
(multiUnwrapper, declared in this plugin's own source) silently returns
ok=false under Yaegi, even though %T correctly reports *fmt.wrapErrors
and that type genuinely implements Unwrap() []error in real Go — the
reverse direction of the already-documented errors.As-on-interpreted-
types trap, not previously known to this codebase.

Fix: matchesSentinel now calls real errors.Is instead of hand-rolling
anything. This is safe here specifically, despite the "avoid errors.Is/
As" guidance elsewhere in this codebase: the panic risk that guidance
targets is errors.As checking whether an INTERPRETED type implements
error, which never arises for a plain sentinel comparison against
compiled-origin values (context.DeadlineExceeded, context.Canceled, and
every err this function ever sees) — the identical safe shape retry.go's
isTransient already relies on for these same two sentinels, proven
correct under Yaegi across every prior round. Empirically verified: this
is what made the extended yaegi-check harness pass; the hand-rolled
version did not, despite passing every compiled `go test` row.

Also in this commit:
- Route-level regression tests (TestHandleChat_
  ContextDeadlineExceeded_RecordsProviderFailure, routes_unified_test.go;
  TestHandlePassthrough_ContextDeadlineExceeded_RecordsProviderFailure,
  routes_passthrough_test.go) — a real httptest upstream sleeping past a
  real short request-context deadline, adapted from a reviewer-provided
  probe. These are what exposed the blocker in the first place; the
  sentinel-level table test alone (hand-built error shapes) is exactly
  what let it ship.
- Prod-shape rows (double-/triple-%w, byte-for-byte providers.go's own
  format strings) folded into the existing classification table
  (renamed TestIsDeadlineExceeded_Classification — its own doc comment
  now records that this table alone is NOT proof under the interpreter,
  after round 2's own history).
- tools/yaegi-check/main.go (SHOULD-A): the harness's claim to prove the
  interpreted deadline path was never actually true — its only upstream
  answered instantly, so isDeadlineExceeded was never exercised with a
  non-nil error at all. Added a second, deliberately slow provider
  ("slow", provider-prefixed model id so it cannot collide with the
  existing fast one) driven with a short request-context deadline,
  asserting failuresDay==1 under the real interpreter. This is the
  harness that caught the blocker above.
- FOLDED-1: recordProviderAttempt's fire-and-forget store write
  (SHOULD-5, round 1) is now bounded — limiter.spawnTokens, a per-limiter
  buffered-channel semaphore (providerAttemptSpawnCap=64) the production
  spawn closure acquires non-blockingly; a write that cannot start
  immediately is DROPPED, not queued. Bounded and lossy-under-pressure by
  design: telemetry only, enforcement never goes through spawn.
  Per-limiter, not package-level, so the test suite's many parallel
  *limiter instances cannot starve each other.
- FOLDED-2: TestRecordProviderAttempt_ProductionSpawn_
  WriteEventuallyLandsAsync leaves limiter.spawn at its real production
  default (every other test in this file overrides it to run
  synchronously) and polls (waitUntil, deadline-bounded, not a fixed
  sleep) for the counter to land — closing the -race async-path gap the
  synchronous overrides otherwise leave entirely untested.
This commit is contained in:
2026-08-22 00:32:00 +01:00
parent 1661cd5144
commit eddcbe35bd
5 changed files with 436 additions and 96 deletions
+64
View File
@@ -1830,3 +1830,67 @@ func TestUnifiedCostMicros(t *testing.T) {
})
}
}
// TestHandleChat_ContextDeadlineExceeded_RecordsProviderFailure is the
// route-level regression test the review's blocker (v0.22, round 2)
// specifically demanded: TestIsDeadlineExceeded_YaegiSafeUnwrap's
// hand-built error shapes (limits_test.go) proved matchesSentinel's own
// unwrap logic works in isolation, but that alone is exactly what let the
// real bug ship — every hand-built shape there was single-%w, while the
// live chain through upstreamBytes' client.Do call is providers.go's
// own double-%w `fmt.Errorf("%w: %w", errUpstream, err)`, one layer
// short of what a table test alone would ever exercise. This drives a
// REAL request whose context deadline expires while a REAL httptest
// upstream is still sleeping — client.Do, *url.Error, and the wrap chain
// providers.go actually builds, all for real — and asserts the resulting
// provider counters directly, not a synthetic error value.
func TestHandleChat_ContextDeadlineExceeded_RecordsProviderFailure(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(600 * time.Millisecond) // well past the request's own 60ms deadline below
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"c1","object":"chat.completion","model":"gpt-test","choices":[],"usage":{}}`))
}))
defer srv.Close()
cfg := CreateConfig()
cfg.Providers = map[string]*ProviderConfig{
"openai": {Type: "openai", BaseURL: srv.URL, APIKey: "sk-up", Models: []string{"gpt-test"}},
}
cfg.Groups = map[string]*GroupConfig{"default": {}}
cfg.Users = &UsersConfig{Inline: []*UserConfig{
{Name: "alice", Group: "default", APIKey: "sk-alice", Limits: &LimitsConfig{}},
}}
next := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})
h, err := New(context.Background(), next, cfg, "llmgw")
if err != nil {
t.Fatalf("New: %v", err)
}
gw, ok := h.(*Gateway)
if !ok {
t.Fatal("handler is not *Gateway")
}
// SHOULD-5: run recordProviderAttempt's store write synchronously so
// the counter read below is deterministic (see limiter.spawn's own
// doc comment, limits.go). FOLDED-2's own test (limits_test.go)
// covers the production async default separately.
gw.limiter.spawn = func(f func()) { f() }
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Millisecond)
defer cancel()
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions",
strings.NewReader(`{"model":"gpt-test","messages":[{"role":"user","content":"hi"}]}`)).WithContext(ctx)
req.Header.Set("Authorization", "Bearer sk-alice")
req.Header.Set("Content-Type", "application/json")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
now := time.Now()
attempts, _ := gw.limiter.getCounter(kindProvider, "openai", metricProvAttempt, windowDay, now)
if attempts != 1 {
t.Errorf("provider attempts/day = %d, want 1", attempts)
}
fails, _ := gw.limiter.getCounter(kindProvider, "openai", metricProvFail, windowDay, now)
if fails != 1 {
t.Errorf("provider fails/day = %d, want 1 — a real context-deadline timeout must count as a provider-health failure (SHOULD-1)", fails)
}
}