mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
fix(resp): bound store-call latency, cap wire lengths, cover fail-closed budget path
Bound respClient's worst-case latency to one respCallTimeout per do/pipeline call instead of per read/write/handshake step. Compute a single absolute deadline at call entry, apply it once via conn.SetDeadline after connecting (covering the AUTH/SELECT handshake and the call's own command I/O), and skip the reconnect-retry when the failing attempt was a timeout (errors.As -> net.Error, Timeout()) — a peer that accepted the connection and went silent must not be asked the same non-question twice. Previously each read, write, and dial got its own fresh 2s budget and every caller queued behind the client's single mutex, so a hung (not merely refused) Redis-compatible server could stall one checkAndCount call for 16-32s+. Cap decodeBulk/decodeArray's wire-declared lengths (respMaxBulkLen 64MiB, respMaxArrayLen 1<<20) before allocating: an unguarded "$9223372036854775807\r\n" overflows n+2 into a negative slice length and panics make([]byte, n+2); a merely huge-but-valid length forces an outsized allocation either way. decodeBulk also now verifies its two trailing bytes are CRLF. Bound readLine to respMaxLineLen (64KiB), reading byte-by-byte so it can bail before ever buffering an unbounded line from a peer that never sends '\n'. Fix TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation: the erroringStore it used fails incrBy too, so checkAndCount's own increment loop already returned a storeDown violation before evaluateScope (and therefore budgetViolation) ever ran, leaving budgetViolation's storeDown branch uncovered. Replace it with a stub that only fails get, so the test's TokensPerDay limit actually drives evaluateScope into budgetViolation's fail-closed path. Document redis_store.go's incrBy as deliberately at-least-once: the reconnect-retry can re-send INCRBY after a lost reply (possible over-count), and fail-open can additionally count in the fallback — both fail-safe (more restrictive), never fail-open in the unsafe direction. Validate Redis.DB >= 0 in newConfiguredLimiter alongside Address, so a negative db is a constructor error rather than a SELECT -1 on the wire. New tests: a hung (accepts, never replies) fake RESP server proving a respClient call and a failOpen limiter call both stay bounded rather than stacking timeouts; decodeReply subtests for an oversized bulk length, a length just over the cap, a huge array length, a bad bulk trailer, and an oversized line; a negative-DB constructor-rejection test.
This commit is contained in:
+23
-3
@@ -433,18 +433,38 @@ func TestLimiter_FailClosed_StoreErrorReturnsStoreDownViolation(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// succeedIncrFailGetStore is a counterStore stub whose incrBy always
|
||||
// succeeds and whose get always errors. It isolates budgetViolation's own
|
||||
// storeDown branch (limits.go) from checkAndCount's earlier incrCounter
|
||||
// one: an erroringStore that fails both methods makes checkAndCount's
|
||||
// initial req:min/req:day increments fail closed and return before
|
||||
// evaluateScope — and therefore budgetViolation — ever runs, so a test
|
||||
// built on it cannot actually prove budgetViolation's own branch works.
|
||||
type succeedIncrFailGetStore struct {
|
||||
getErr error
|
||||
}
|
||||
|
||||
func (s *succeedIncrFailGetStore) incrBy(string, int64, time.Duration) (int64, error) {
|
||||
return 1, nil
|
||||
}
|
||||
func (s *succeedIncrFailGetStore) get(string) (int64, error) { return 0, s.getErr }
|
||||
|
||||
// TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation covers the
|
||||
// budgetViolation fail-closed path specifically (as opposed to the
|
||||
// request-counter incrCounter path covered above): a store error on a
|
||||
// request-counter incrCounter path TestLimiter_FailClosed_
|
||||
// StoreErrorReturnsStoreDownViolation covers above): a store error on a
|
||||
// token/cost budget read also refuses the request when failOpen is false.
|
||||
// The store's incrBy succeeds so checkAndCount's req:min/req:day
|
||||
// increments pass and evaluateScope actually reaches budgetViolation; a
|
||||
// TokensPerDay limit makes evaluateScope call it.
|
||||
func TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation(t *testing.T) {
|
||||
store := &erroringStore{err: errors.New("boom")}
|
||||
store := &succeedIncrFailGetStore{getErr: errors.New("boom")}
|
||||
l := newLimiter(store, false)
|
||||
scopes := []limitScope{{kind: "user", id: "u", limits: &LimitsConfig{TokensPerDay: 100}}}
|
||||
|
||||
v := l.checkAndCount(scopes)
|
||||
if v == nil {
|
||||
t.Fatal("want a storeDown violation on a store error with failOpen=false")
|
||||
t.Fatal("want a storeDown violation from the budget-read path with failOpen=false")
|
||||
}
|
||||
if !v.storeDown {
|
||||
t.Error("v.storeDown = false, want true")
|
||||
|
||||
Reference in New Issue
Block a user