perf(resp): pool redis connections instead of one mutex-guarded conn

respClient held a single net.Conn behind one mutex, serializing every
caller's whole round trip. Interpreted-harness audit measured this as
the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on
memoryStore for the same workload, latency scaling near-linearly with
concurrency (869us@conc=1 -> 10.81ms@conc=16).

Replace the single connection with a pool of independently usable
connections (respConn slots in a buffered channel), each acquired for
one caller's whole round trip and returned via a deferred send back
onto the pool. A broken connection is discarded (closeConn) but its
slot always returns to the pool, so pool size never shrinks after an
error. Default size is GOMAXPROCS clamped to [4, 8]
(defaultRespPoolSize); RedisConfig.PoolSize lets an operator override
it explicitly, per the house rule that an explicit override always
wins over self-tuning.

Preserves exactly: the 2s respCallTimeout call deadline, reconnect-
once-on-error semantics, nested-array rejection, the RESP size bounds,
and every existing pipelining contract (incrMulti, incrAndGetMulti,
getMulti, cache get/set) — nothing outside resp.go references a
connection field directly.

Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured
at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%),
throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1
control matched HEAD exactly, confirming the win is parallelism, not
refactor noise.

cache_test.go and llmgateway_test.go: adjust white-box tests that
issue multiple sequential calls against one respClient scripted with
runFakeRESPServer's single-connection model — force pool size 1 there
(newRESPClientPool) so the test still pins one connection's exact wire
sequence; production callers are unaffected.
This commit is contained in:
2026-08-22 14:36:41 +01:00
parent df39ad3739
commit 2309a59b40
5 changed files with 766 additions and 136 deletions
+15 -2
View File
@@ -301,9 +301,19 @@ func TestBuildResponseCache_EnabledWithRedis_ReturnsWorkingCache(t *testing.T) {
// newTestResponseCache builds a responseCache over ln's fake server, with
// nowFn pinned to a fixed instant so logCacheError's rate-limit window is
// deterministic. logged collects every logf call's formatted message.
//
// Pool size 1 (newRESPClientPool, not newRESPClient's self-tuned
// default): a caller issuing more than one lookup/store against the
// returned cache expects every command on the SAME connection, matching
// runFakeRESPServer's strict one-connection-at-a-time script — with the
// default pool (size >1, finding 1), a second call could acquire a
// different, not-yet-dialled slot instead of reusing the first
// connection, sending a second SELECT the fixed script never expects.
// Production callers (llmgateway.go's buildRedisClient) are unaffected —
// they always go through newRESPClient's own default.
func newTestResponseCache(t *testing.T, ln net.Listener) (c *responseCache, logged *[]string) {
t.Helper()
client := newRESPClient(ln.Addr().String(), "", 0)
client := newRESPClientPool(ln.Addr().String(), "", 0, 1)
msgs := []string{}
spy := func(format string, args ...any) { msgs = append(msgs, fmt.Sprintf(format, args...)) }
c = newResponseCache(client, time.Minute, defaultCacheMaxBodyBytes, spy, spy)
@@ -678,7 +688,10 @@ func TestResponseCache_Store_EffectiveTTLDiffersPerGroup_SETCarriesGroupTTL(t *t
{wantArgs: []string{"SET", "k-override", string(val1), "EX", "30"}, reply: []byte("+OK\r\n")},
{wantArgs: []string{"SET", "k-inherit", string(val2), "EX", "60"}, reply: []byte("+OK\r\n")},
})
client := newRESPClient(ln.Addr().String(), "", 0)
// Pool size 1 — see newTestResponseCache's own doc comment: two
// sequential store() calls below must land on runFakeRESPServer's one
// scripted connection, not cycle across separate pool slots.
client := newRESPClientPool(ln.Addr().String(), "", 0, 1)
noop := func(string, ...any) {}
c := newResponseCache(client, time.Minute, defaultCacheMaxBodyBytes, noop, noop) // global ttl = 60s
c.nowFn = time.Now