respClient held a single net.Conn behind one mutex, serializing every
caller's whole round trip. Interpreted-harness audit measured this as
the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on
memoryStore for the same workload, latency scaling near-linearly with
concurrency (869us@conc=1 -> 10.81ms@conc=16).
Replace the single connection with a pool of independently usable
connections (respConn slots in a buffered channel), each acquired for
one caller's whole round trip and returned via a deferred send back
onto the pool. A broken connection is discarded (closeConn) but its
slot always returns to the pool, so pool size never shrinks after an
error. Default size is GOMAXPROCS clamped to [4, 8]
(defaultRespPoolSize); RedisConfig.PoolSize lets an operator override
it explicitly, per the house rule that an explicit override always
wins over self-tuning.
Preserves exactly: the 2s respCallTimeout call deadline, reconnect-
once-on-error semantics, nested-array rejection, the RESP size bounds,
and every existing pipelining contract (incrMulti, incrAndGetMulti,
getMulti, cache get/set) — nothing outside resp.go references a
connection field directly.
Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured
at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%),
throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1
control matched HEAD exactly, confirming the win is parallelism, not
refactor noise.
cache_test.go and llmgateway_test.go: adjust white-box tests that
issue multiple sequential calls against one respClient scripted with
runFakeRESPServer's single-connection model — force pool size 1 there
(newRESPClientPool) so the test still pins one connection's exact wire
sequence; production callers are unaffected.
GroupConfig gains CacheTTL (json cacheTTL, empty inherits the global
cache.ttl). Parsed and validated at construction (newAuthStore,
groupCacheTTL in cache.go): a malformed duration, a duration <= 0, or
CacheTTL set while the global cache block is not configured is a
constructor error naming the group, mirroring Cache:true's existing
"nothing to inherit from" check.
group carries the resolved time.Duration (0 means inherit).
responseCache.store gains an explicit ttl parameter; routes_unified.go
computes effectiveTTL(g.cache, grp) immediately before each store call,
so setEx's Redis-native EX seconds reflect the group's own override.
Tests: constructor-error table, empty-inherits and valid-override
resolution, effectiveTTL inheritance/override, a scripted-RESP-server
proof that store's SET...EX carries a different value per group, and an
end-to-end Gateway.ServeHTTP test with two groups asserting the actual
wire EX value (30 vs 60) differs per group.
README: GroupConfig.cacheTTL config row and a Caching-section paragraph.
Spec §5 (v0.2): Config.ModelAliases map[string]string, alias-first
resolution precedence ahead of provider-prefix splitting, alias-OR-target
authorization, lazy target resolution with a 404 naming both ids,
/v1/models listing, admin overview alias table (JSON + dashboard), and
construction-time validation (shape, prefix-shadow, explicit-model
collision, chain, empty target).
routes_unified.go/routes_media.go needed no resolution-call-site changes
- both already pass the client's raw requested id through to
registry.resolve, so aliases inherit the __alias echo and multipart
rewrite machinery for free. cache.go needed no changes either - cacheKey
already hashes requestedModel, so two aliases resolving to the same
target already produce distinct cache entries.
Adds registry_test.go validation table + precedence/authz/lazy-404/
listing coverage, a dedicated cache-key split test, chat/embeddings/
images/audio-transcriptions end-to-end tests, an extended admin overview
test, and an integration test covering the openai-type vs anthropic-type
echo asymmetry under real Traefik+Yaegi.
lukaszraczylo