respClient held a single net.Conn behind one mutex, serializing every
caller's whole round trip. Interpreted-harness audit measured this as
the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on
memoryStore for the same workload, latency scaling near-linearly with
concurrency (869us@conc=1 -> 10.81ms@conc=16).
Replace the single connection with a pool of independently usable
connections (respConn slots in a buffered channel), each acquired for
one caller's whole round trip and returned via a deferred send back
onto the pool. A broken connection is discarded (closeConn) but its
slot always returns to the pool, so pool size never shrinks after an
error. Default size is GOMAXPROCS clamped to [4, 8]
(defaultRespPoolSize); RedisConfig.PoolSize lets an operator override
it explicitly, per the house rule that an explicit override always
wins over self-tuning.
Preserves exactly: the 2s respCallTimeout call deadline, reconnect-
once-on-error semantics, nested-array rejection, the RESP size bounds,
and every existing pipelining contract (incrMulti, incrAndGetMulti,
getMulti, cache get/set) — nothing outside resp.go references a
connection field directly.
Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured
at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%),
throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1
control matched HEAD exactly, confirming the win is parallelism, not
refactor noise.
cache_test.go and llmgateway_test.go: adjust white-box tests that
issue multiple sequential calls against one respClient scripted with
runFakeRESPServer's single-connection model — force pool size 1 there
(newRESPClientPool) so the test still pins one connection's exact wire
sequence; production callers are unaffected.
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
Bound respClient's worst-case latency to one respCallTimeout per
do/pipeline call instead of per read/write/handshake step. Compute a
single absolute deadline at call entry, apply it once via
conn.SetDeadline after connecting (covering the AUTH/SELECT handshake
and the call's own command I/O), and skip the reconnect-retry when the
failing attempt was a timeout (errors.As -> net.Error, Timeout()) — a
peer that accepted the connection and went silent must not be asked
the same non-question twice. Previously each read, write, and dial got
its own fresh 2s budget and every caller queued behind the client's
single mutex, so a hung (not merely refused) Redis-compatible server
could stall one checkAndCount call for 16-32s+.
Cap decodeBulk/decodeArray's wire-declared lengths (respMaxBulkLen
64MiB, respMaxArrayLen 1<<20) before allocating: an unguarded
"$9223372036854775807\r\n" overflows n+2 into a negative slice length
and panics make([]byte, n+2); a merely huge-but-valid length forces an
outsized allocation either way. decodeBulk also now verifies its two
trailing bytes are CRLF. Bound readLine to respMaxLineLen (64KiB),
reading byte-by-byte so it can bail before ever buffering an unbounded
line from a peer that never sends '\n'.
Fix TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation: the
erroringStore it used fails incrBy too, so checkAndCount's own
increment loop already returned a storeDown violation before
evaluateScope (and therefore budgetViolation) ever ran, leaving
budgetViolation's storeDown branch uncovered. Replace it with a stub
that only fails get, so the test's TokensPerDay limit actually drives
evaluateScope into budgetViolation's fail-closed path.
Document redis_store.go's incrBy as deliberately at-least-once: the
reconnect-retry can re-send INCRBY after a lost reply (possible
over-count), and fail-open can additionally count in the fallback —
both fail-safe (more restrictive), never fail-open in the unsafe
direction.
Validate Redis.DB >= 0 in newConfiguredLimiter alongside Address, so a
negative db is a constructor error rather than a SELECT -1 on the wire.
New tests: a hung (accepts, never replies) fake RESP server proving a
respClient call and a failOpen limiter call both stay bounded rather
than stacking timeouts; decodeReply subtests for an oversized bulk
length, a length just over the cap, a huge array length, a bad bulk
trailer, and an oversized line; a negative-DB constructor-rejection
test.
Add resp.go: a minimal stdlib-only RESP2 client (respClient) for a
single Redis-compatible server. It lazily dials one mutex-guarded
connection, applies 2s dial/read/write deadlines, sends AUTH (if a
password is set) then SELECT on every new connection, and reconnects
once on any I/O error before giving up. do and pipeline encode
commands as "*N\r\n$len\r\n..." and decode +/-/:/$/* replies.
Add redis_store.go: redisStore implements counterStore over
respClient. incrBy pipelines INCRBY and EXPIRE in one round trip and
returns INCRBY's value; get treats a missing key (RESP null bulk) as
0.
Wire the limiter's fail-open/fail-closed policy in limits.go, which
Task 5 stored but never read: a configured store's error now routes
through storeIncrBy/storeGet. failOpen=true falls back to the
in-process memoryStore transparently; failOpen=false makes
checkAndCount return a storeDown violation and makes account drop the
sample. Both paths log through an injectable, 30s-rate-limited logf.
Wire newGateway (llmgateway.go) to build a redisStore-backed limiter
when Config.Redis is set (password through resolveSecret, failOpen
defaulting to true unless RedisConfig.FailOpen says otherwise) and a
memory-only limiter otherwise, and to route pricing.go's unknown-model
warnings through the gateway's own logf.
Validate LimitsConfig at construction: a new LimitsConfig.validate
rejects a negative int64 count limit or a negative/NaN/Inf cost limit,
called from newAuthStore for every group and every user (inline and
file-sourced) so a malformed limit is a constructor error instead of
silently behaving as unlimited.
Add RequestsPerDay, TokensPerMonth, and CostPerMonthUSD enforcement
tests alongside the existing per-minute/day/cost-per-day cases, plus
an erroringStore-backed test pair proving the fail-open (fallback
counted) and fail-closed (storeDown violation) branches directly.
Folded the validation work into this single commit rather than a
second one: it interleaves with the fail-open changes across
limits.go, limits_test.go, and llmgateway_test.go closely enough that
splitting by hunk would be artificial.
Add usersFile (newUsersFile/load) for the JSON users-file shape
{"users":[{name,group,apiKey,limits?}]}, and authStore.maybeReload
for throttled hot reload: checks mtime at most every reloadEvery
(5s), reloads on change, keeps the last good set and logs via
errorf on any error (stat, parse, or a rejected replace), and logs
via logf when the reload changes the file-sourced user count.
nowFn is injectable for tests.
Wire-up in llmgateway.go: newGateway does a synchronous initial
load when Users.File is set (a bad initial file fails the
constructor); ServeHTTP calls g.auth.maybeReload() at entry
(no-op when no file is configured).
Also fixes replaceFileUsers's merge semantics: a file user now
overrides an inline user of the same name, even when their keys
differ, dropping the inline entry from the rebuilt set. The prior
digest-only keying let a same-named inline/file pair with
different keys coexist instead of the file version winning.
lukaszraczylo