Fix 6 issues surfaced by adversarial agent team (round 35)

Concurrency/lifecycle (P1, highest-value):
- singleton_resources.go: RegisterBackgroundTask now REPLACES a stopped
  task instead of reusing it. BackgroundTask.Start is sync.Once-guarded,
  so a task stopped on instance close and re-registered (Traefik
  close+recreate for the same provider) was permanently dead: IsTaskRunning
  returned false (started && !stopped) so callers retried a start that
  was a no-op. Metadata refresh, token cleanup, and health checks now
  resume across a recreate cycle.
- error_recovery.go: the shared graceful-degradation health task was
  bound to the FIRST instance's performHealthChecks (later instances'
  checks never ran) and any instance's Close stopped it for all. Added a
  global live-instance registry; the shared task runs every instance's
  checks and is stopped only when the last instance closes (mirrors the
  singleton-token-cleanup gate).

Storage/cache:
- hybrid.go recordL2Error: Store(now) then immediately compared
  time.Since(t)<1s against its own write => ALWAYS fired, so a single
  transient L2 error flipped the whole cache into fallback (full L2
  bypass -> upstream stampede). Now enters fallback only on a PRIOR
  error within the window (true "multiple errors"). Updated two tests
  that had encoded the old single-error-fallback bug.
- redis_pool.go: ConnectionPool.Put checked closed but sent on
  p.connections without holding p.mu; Close closed it under p.mu.
  Window => 'send on closed channel' panic at shutdown. Put now holds
  p.mu (the lock only Close used).

HTTP/transport:
- jwk.go fetchJWKS: added an explicit >1 MiB guard (was silent partial
  read, could yield a truncated keyset and spurious missing-kid).
- http_client_pool.go configKey: now includes every transport-behavior
  field it actually consumes (compression, HTTP2, keep-alives, timeouts,
  buffer sizes, dial) so behaviorally different configs no longer share
  one transport built with the first config's settings.

Each fix has a regression test. Verified: full source-order suite green,
golangci-lint 0 issues, 3x full -race -count=1 -shuffle=on ./... all exit 0.
This commit is contained in:
2026-08-12 23:55:55 +01:00
parent 01ad33c879
commit 35d26078f0
9 changed files with 361 additions and 27 deletions
+31 -2
View File
@@ -800,7 +800,8 @@ func TestHybridBackend_Ping_RecoverFromFallback(t *testing.T) {
ctx := context.Background()
// First ping fails L2, enters fallback
// Repeated L2 failures (two errors within the window) enter fallback.
hybrid.Ping(ctx)
hybrid.Ping(ctx)
assert.True(t, hybrid.fallbackMode.Load())
@@ -1184,7 +1185,13 @@ func TestHybridBackend_GetMany_BatchBackendError(t *testing.T) {
assert.Len(t, results, 1)
assert.Equal(t, []byte("value1"), results["key1"])
// Batch error should trigger fallback mode
// A single batch error is not enough to enter fallback (avoids a cache
// stampede on one transient L2 error); a second error within the window is.
assert.False(t, hybrid.fallbackMode.Load())
_, err = hybrid.GetMany(ctx, []string{"key2"})
assert.NoError(t, err)
// Repeated batch error should trigger fallback mode
time.Sleep(50 * time.Millisecond)
assert.True(t, hybrid.fallbackMode.Load())
}
@@ -1488,3 +1495,25 @@ func TestHybridBackend_RecordL2Error_EntersFallbackMode(t *testing.T) {
// Should enter fallback mode
assert.True(t, hybrid.fallbackMode.Load())
}
func TestHybridBackend_RecordL2Error_RequiresRepeatedErrors(t *testing.T) {
config := &HybridConfig{
Primary: newMockBackend(),
Secondary: newMockBackend(),
}
hybrid, err := NewHybridBackend(config)
require.NoError(t, err)
defer hybrid.Close()
// A SINGLE transient L2 error must NOT flip the whole cache into fallback
// mode (which bypasses L2 for reads and writes -> cache stampede).
hybrid.recordL2Error()
assert.False(t, hybrid.fallbackMode.Load(),
"a single L2 error must not enter fallback mode")
// A second error within the 1s window (repeated failures) does.
hybrid.recordL2Error()
assert.True(t, hybrid.fallbackMode.Load(),
"repeated L2 errors within the window must enter fallback mode")
}