fix(providers): multi-wrap deadline detection was dead on 3 of 4 routes

BLOCKER (v0.22 review round, round 2): matchesSentinel's hand-rolled
Unwrap walk (round 1's SHOULD-1 fix) stopped dead the moment it hit
providers.go's own multi-%w wrapping. fmt.Errorf with more than one %w
verb builds a *fmt.wrapErrors implementing Unwrap() []error, not
Unwrap() error — a comma-ok assertion against a single-error Unwrap
interface alone never reaches past it. All three of providers.go's own
wrap sites use multi-%w (upstreamJSON's encode-failure path, "%w:
encode request body: %w"; upstreamBytes' build-request path, "%w: %w:
build request: %w"; and upstreamBytes' client.Do path, "%w: %w" — the
one a real request-context deadline reaches through). Result: a hung
upstream on the unified chat/embeddings/media routes recorded as 100%
healthy; only native passthrough (whose error is never re-wrapped by
upstreamBytes) worked.

Round 2's first attempt added a second hand-rolled interface
(multiUnwrapper, Unwrap() []error) and recursed into it — verified
correct against every wrap shape under `go test`, INCLUDING the exact
double-/triple-%w shapes providers.go itself produces. It still did not
work. tools/yaegi-check's own SHOULD-A extension (below) caught it: a
type assertion of a COMPILED concrete value (*fmt.wrapErrors, built by
the real fmt package) against an INTERPRETED interface type
(multiUnwrapper, declared in this plugin's own source) silently returns
ok=false under Yaegi, even though %T correctly reports *fmt.wrapErrors
and that type genuinely implements Unwrap() []error in real Go — the
reverse direction of the already-documented errors.As-on-interpreted-
types trap, not previously known to this codebase.

Fix: matchesSentinel now calls real errors.Is instead of hand-rolling
anything. This is safe here specifically, despite the "avoid errors.Is/
As" guidance elsewhere in this codebase: the panic risk that guidance
targets is errors.As checking whether an INTERPRETED type implements
error, which never arises for a plain sentinel comparison against
compiled-origin values (context.DeadlineExceeded, context.Canceled, and
every err this function ever sees) — the identical safe shape retry.go's
isTransient already relies on for these same two sentinels, proven
correct under Yaegi across every prior round. Empirically verified: this
is what made the extended yaegi-check harness pass; the hand-rolled
version did not, despite passing every compiled `go test` row.

Also in this commit:
- Route-level regression tests (TestHandleChat_
  ContextDeadlineExceeded_RecordsProviderFailure, routes_unified_test.go;
  TestHandlePassthrough_ContextDeadlineExceeded_RecordsProviderFailure,
  routes_passthrough_test.go) — a real httptest upstream sleeping past a
  real short request-context deadline, adapted from a reviewer-provided
  probe. These are what exposed the blocker in the first place; the
  sentinel-level table test alone (hand-built error shapes) is exactly
  what let it ship.
- Prod-shape rows (double-/triple-%w, byte-for-byte providers.go's own
  format strings) folded into the existing classification table
  (renamed TestIsDeadlineExceeded_Classification — its own doc comment
  now records that this table alone is NOT proof under the interpreter,
  after round 2's own history).
- tools/yaegi-check/main.go (SHOULD-A): the harness's claim to prove the
  interpreted deadline path was never actually true — its only upstream
  answered instantly, so isDeadlineExceeded was never exercised with a
  non-nil error at all. Added a second, deliberately slow provider
  ("slow", provider-prefixed model id so it cannot collide with the
  existing fast one) driven with a short request-context deadline,
  asserting failuresDay==1 under the real interpreter. This is the
  harness that caught the blocker above.
- FOLDED-1: recordProviderAttempt's fire-and-forget store write
  (SHOULD-5, round 1) is now bounded — limiter.spawnTokens, a per-limiter
  buffered-channel semaphore (providerAttemptSpawnCap=64) the production
  spawn closure acquires non-blockingly; a write that cannot start
  immediately is DROPPED, not queued. Bounded and lossy-under-pressure by
  design: telemetry only, enforcement never goes through spawn.
  Per-limiter, not package-level, so the test suite's many parallel
  *limiter instances cannot starve each other.
- FOLDED-2: TestRecordProviderAttempt_ProductionSpawn_
  WriteEventuallyLandsAsync leaves limiter.spawn at its real production
  default (every other test in this file overrides it to run
  synchronously) and polls (waitUntil, deadline-bounded, not a fixed
  sleep) for the counter to land — closing the -race async-path gap the
  synchronous overrides otherwise leave entirely untested.
This commit is contained in:
2026-08-22 00:32:00 +01:00
parent 1661cd5144
commit eddcbe35bd
5 changed files with 436 additions and 96 deletions
+83 -9
View File
@@ -1421,7 +1421,7 @@ func TestRecordProviderAttempt_ClassifiesUsingIsTransient(t *testing.T) {
{name: "network error", err: errors.New("dial tcp: connection refused"), wantFail: true},
{name: "context canceled (not a provider fault)", err: context.Canceled, wantFail: false},
{name: "context deadline exceeded (hung upstream timed out — SHOULD-1)", err: context.DeadlineExceeded, wantFail: true},
{name: "wrapped context deadline exceeded (proves the manual Unwrap walk)", err: fmt.Errorf("%w: dial timeout", context.DeadlineExceeded), wantFail: true},
{name: "wrapped context deadline exceeded (proves errors.Is walks the chain)", err: fmt.Errorf("%w: dial timeout", context.DeadlineExceeded), wantFail: true},
{name: "wrapped context canceled stays non-failure too", err: fmt.Errorf("%w: client hung up", context.Canceled), wantFail: false},
}
@@ -1494,14 +1494,63 @@ func TestRecordProviderAttempt_EmptyProvider_NoOp(t *testing.T) {
l.recordProviderAttempt("", "some-model", &http.Response{StatusCode: http.StatusOK}, nil) // must not panic
}
// TestIsDeadlineExceeded_YaegiSafeUnwrap exercises matchesSentinel's
// hand-rolled Unwrap walk directly (SHOULD-1, v0.22 review round): a bare
// context.DeadlineExceeded, one wrapped once, one wrapped twice (proving
// the loop actually loops, not just unwraps one level), a non-matching
// error that implements Unwrap (must walk to its end and report false,
// not stop after one failed comparison), and a plain error with no
// Unwrap method at all (the comma-ok assertion's negative case).
func TestIsDeadlineExceeded_YaegiSafeUnwrap(t *testing.T) {
// TestRecordProviderAttempt_ProductionSpawn_WriteEventuallyLandsAsync
// leaves limiter.spawn at its real production default (newLimiter's own
// bounded, goroutine-spawning closure) instead of overriding it to run
// synchronously like every other recordProviderAttempt test in this file
// — FOLDED-2 (v0.22 review round, round 2): every other test here proves
// the WRITE is correct, but only by bypassing the actual async code path
// (limiter.spawn's own doc comment) entirely; this is the one test that
// exercises it for real, closing a real -race async-path gap. Polls
// (waitUntil, registry_test.go) rather than sleeping a fixed duration —
// a fixed sleep is either flaky (too short, especially under `-race`,
// which distorts goroutine scheduling) or wastefully slow (too long); a
// deadline-bounded poll is neither.
func TestRecordProviderAttempt_ProductionSpawn_WriteEventuallyLandsAsync(t *testing.T) {
l := newLimiter(nil, true) // production default spawn — deliberately NOT newSyncLimiter
now := time.Date(2026, 8, 21, 12, 0, 0, 0, time.UTC)
l.nowFn = func() time.Time { return now }
l.recordProviderAttempt("openai", "gpt-4o", &http.Response{StatusCode: http.StatusOK}, nil)
waitUntil(t, time.Second, func() bool {
attempts, _ := l.getCounter(kindProvider, "openai", metricProvAttempt, windowDay, now)
return attempts == 1
})
fails, _ := l.getCounter(kindProvider, "openai", metricProvFail, windowDay, now)
if fails != 0 {
t.Errorf("fails/day = %d, want 0 (a 200 response is never a failure)", fails)
}
}
// TestIsDeadlineExceeded_Classification exercises isDeadlineExceeded's
// observable classification across every wrap shape that matters: bare,
// wrapped once, wrapped twice, an unrelated sentinel (context.Canceled,
// which must never match), an unrelated wrapped error, a plain error
// with no Unwrap method — PLUS the prod-shape double-%w/triple-%w rows a
// review round-2 blocker demanded, byte-for-byte the exact format
// strings providers.go's three upstreamJSON/upstreamBytes call sites use.
//
// A load-bearing caveat this test's own name used to hide (it was
// TestIsDeadlineExceeded_YaegiSafeUnwrap): every case here runs COMPILED,
// under `go test`, and compiled-Go correctness is not the same claim as
// Yaegi-interpreted correctness. Round 2's original fix — a hand-rolled
// Unwrap walk against two locally declared interfaces, matched with bare
// comma-ok type assertions — passed every one of these exact rows
// (including the double-/triple-%w ones) under `go test`, while
// SHOULD-A's yaegi-check harness (tools/yaegi-check/main.go's
// exerciseAttemptAccounting) proved it silently returned false for the
// double-%w case under the REAL interpreter: an interpreted interface
// type does not correctly match a compiled concrete value's method set
// in Yaegi, the reverse direction of the already-documented errors.As-
// on-interpreted-types trap. matchesSentinel now calls real errors.Is
// instead (limits.go's own doc comment there has the full account of why
// that is safe here specifically, unlike errors.As on an interpreted
// type) — this table still pins the OBSERVABLE behavior, but
// tools/yaegi-check is what actually proves it under the interpreter;
// this table alone would not have caught the round-2 regression, and a
// future change here must keep running yaegi-check, not just `go test`.
func TestIsDeadlineExceeded_Classification(t *testing.T) {
tests := []struct {
err error
name string
@@ -1514,6 +1563,31 @@ func TestIsDeadlineExceeded_YaegiSafeUnwrap(t *testing.T) {
{name: "unrelated sentinel (context.Canceled)", err: context.Canceled, want: false},
{name: "unrelated wrapped error", err: fmt.Errorf("dial: %w", errors.New("connection refused")), want: false},
{name: "plain error, no Unwrap method", err: errors.New("boom"), want: false},
{
name: "double-%w, upstreamBytes' client.Do shape — the blocker's own reproduction",
err: fmt.Errorf("%w: %w", errUpstream, context.DeadlineExceeded),
want: true,
},
{
name: "double-%w, upstreamJSON's encode-failure shape",
err: fmt.Errorf("%w: encode request body: %w", errUpstream, context.DeadlineExceeded),
want: true,
},
{
name: "triple-%w, upstreamBytes' build-request shape",
err: fmt.Errorf("%w: %w: build request: %w", errUpstream, errRequestBuildFailed, context.DeadlineExceeded),
want: true,
},
{
name: "double-%w wrapping context.Canceled must still NOT match — multi-wrap does not blur the two sentinels apart",
err: fmt.Errorf("%w: %w", errUpstream, context.Canceled),
want: false,
},
{
name: "double-%w wrapping an unrelated error must still NOT match",
err: fmt.Errorf("%w: %w", errUpstream, errors.New("connection refused")),
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {