mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
perf(resp): pool redis connections instead of one mutex-guarded conn
respClient held a single net.Conn behind one mutex, serializing every caller's whole round trip. Interpreted-harness audit measured this as the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on memoryStore for the same workload, latency scaling near-linearly with concurrency (869us@conc=1 -> 10.81ms@conc=16). Replace the single connection with a pool of independently usable connections (respConn slots in a buffered channel), each acquired for one caller's whole round trip and returned via a deferred send back onto the pool. A broken connection is discarded (closeConn) but its slot always returns to the pool, so pool size never shrinks after an error. Default size is GOMAXPROCS clamped to [4, 8] (defaultRespPoolSize); RedisConfig.PoolSize lets an operator override it explicitly, per the house rule that an explicit override always wins over self-tuning. Preserves exactly: the 2s respCallTimeout call deadline, reconnect- once-on-error semantics, nested-array rejection, the RESP size bounds, and every existing pipelining contract (incrMulti, incrAndGetMulti, getMulti, cache get/set) — nothing outside resp.go references a connection field directly. Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%), throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1 control matched HEAD exactly, confirming the win is parallelism, not refactor noise. cache_test.go and llmgateway_test.go: adjust white-box tests that issue multiple sequential calls against one respClient scripted with runFakeRESPServer's single-connection model — force pool size 1 there (newRESPClientPool) so the test still pins one connection's exact wire sequence; production callers are unaffected.
This commit is contained in:
+15
-2
@@ -301,9 +301,19 @@ func TestBuildResponseCache_EnabledWithRedis_ReturnsWorkingCache(t *testing.T) {
|
||||
// newTestResponseCache builds a responseCache over ln's fake server, with
|
||||
// nowFn pinned to a fixed instant so logCacheError's rate-limit window is
|
||||
// deterministic. logged collects every logf call's formatted message.
|
||||
//
|
||||
// Pool size 1 (newRESPClientPool, not newRESPClient's self-tuned
|
||||
// default): a caller issuing more than one lookup/store against the
|
||||
// returned cache expects every command on the SAME connection, matching
|
||||
// runFakeRESPServer's strict one-connection-at-a-time script — with the
|
||||
// default pool (size >1, finding 1), a second call could acquire a
|
||||
// different, not-yet-dialled slot instead of reusing the first
|
||||
// connection, sending a second SELECT the fixed script never expects.
|
||||
// Production callers (llmgateway.go's buildRedisClient) are unaffected —
|
||||
// they always go through newRESPClient's own default.
|
||||
func newTestResponseCache(t *testing.T, ln net.Listener) (c *responseCache, logged *[]string) {
|
||||
t.Helper()
|
||||
client := newRESPClient(ln.Addr().String(), "", 0)
|
||||
client := newRESPClientPool(ln.Addr().String(), "", 0, 1)
|
||||
msgs := []string{}
|
||||
spy := func(format string, args ...any) { msgs = append(msgs, fmt.Sprintf(format, args...)) }
|
||||
c = newResponseCache(client, time.Minute, defaultCacheMaxBodyBytes, spy, spy)
|
||||
@@ -678,7 +688,10 @@ func TestResponseCache_Store_EffectiveTTLDiffersPerGroup_SETCarriesGroupTTL(t *t
|
||||
{wantArgs: []string{"SET", "k-override", string(val1), "EX", "30"}, reply: []byte("+OK\r\n")},
|
||||
{wantArgs: []string{"SET", "k-inherit", string(val2), "EX", "60"}, reply: []byte("+OK\r\n")},
|
||||
})
|
||||
client := newRESPClient(ln.Addr().String(), "", 0)
|
||||
// Pool size 1 — see newTestResponseCache's own doc comment: two
|
||||
// sequential store() calls below must land on runFakeRESPServer's one
|
||||
// scripted connection, not cycle across separate pool slots.
|
||||
client := newRESPClientPool(ln.Addr().String(), "", 0, 1)
|
||||
noop := func(string, ...any) {}
|
||||
c := newResponseCache(client, time.Minute, defaultCacheMaxBodyBytes, noop, noop) // global ttl = 60s
|
||||
c.nowFn = time.Now
|
||||
|
||||
Reference in New Issue
Block a user