respClient held a single net.Conn behind one mutex, serializing every
caller's whole round trip. Interpreted-harness audit measured this as
the throughput ceiling: 1,458 req/s Redis-backed vs 7,811 req/s on
memoryStore for the same workload, latency scaling near-linearly with
concurrency (869us@conc=1 -> 10.81ms@conc=16).
Replace the single connection with a pool of independently usable
connections (respConn slots in a buffered channel), each acquired for
one caller's whole round trip and returned via a deferred send back
onto the pool. A broken connection is discarded (closeConn) but its
slot always returns to the pool, so pool size never shrinks after an
error. Default size is GOMAXPROCS clamped to [4, 8]
(defaultRespPoolSize); RedisConfig.PoolSize lets an operator override
it explicitly, per the house rule that an explicit override always
wins over self-tuning.
Preserves exactly: the 2s respCallTimeout call deadline, reconnect-
once-on-error semantics, nested-array rejection, the RESP size bounds,
and every existing pipelining contract (incrMulti, incrAndGetMulti,
getMulti, cache get/set) — nothing outside resp.go references a
connection field directly.
Interpreted-harness prototype (variants/resp_pool8.go, pool=8) measured
at a realistic 200us Redis RTT, conc=16: p50 24.06ms -> 3.24ms (-86%),
throughput 663 -> 4,670 req/s (+599%, +-1% over 3 reps); a pool=1
control matched HEAD exactly, confirming the win is parallelism, not
refactor noise.
cache_test.go and llmgateway_test.go: adjust white-box tests that
issue multiple sequential calls against one respClient scripted with
runFakeRESPServer's single-connection model — force pool size 1 there
(newRESPClientPool) so the test still pins one connection's exact wire
sequence; production callers are unaffected.
Review fixes (2 Important + 4 folded):
1. GET /admin now serves the HTML shell with no authentication -
a browser navigating directly to the URL cannot attach a custom
header. The page reads an admin key from sessionStorage and sends
it as x-api-key on /admin/api/* fetches, falling back to an inline
key-entry form on a missing key or any 401/403. Both JSON routes
keep their original 401/403/200 gate. Spec ss4 amended in place.
2. GET /admin/api/usage no longer serializes 6 GETs per user/group
scope on the shared Redis connection. respClient gains getBatch
(one pipelined round trip); counterStore gains getMulti; limiter
gains storeGetMulti mirroring storeGet's fail-open/fail-closed
policy. currentUsage now issues one getMulti per scope.
3. Provider baseUrl is stripped of userinfo and query before it is
ever echoed in the overview JSON (sanitizeBaseURL), stronger than
net/url's own Redacted (which leaves a bare username in place).
4. limiter.redisStatus now also returns the last failure's timestamp,
so the dashboard can render "last error (Ns ago)" instead of a
bare message that reads as currently-broken indefinitely.
5. Both JSON admin responses now set X-Content-Type-Options: nosniff,
Cache-Control: no-store, and the same CSP the HTML page uses.
6. Added a test proving a file-sourced (replaceFileUsers) admin user
gets the same access an inline admin user does.
go test ./... -count=1 -race, gofmt, vet, make yaegi-check all pass.
Adds integration/ (separate module): docker compose stack with real Traefik
v3.5 + localPlugins, mock openai/anthropic/gemini/tool providers, shared
Redis, and a raw-TCP streaming timing probe. 6/7 tests pass; streaming
incrementality fails on a confirmed external yaegi/Traefik bug (see
task-15-report.md), not a plugin defect.
Fixes 4 real Yaegi-interpretation bugs surfaced by running the plugin under
real Traefik for the first time (none caught by go test or yaegi-check,
which never exercises chat/embeddings/streaming/Redis):
- .traefik.yml: add basePkg (Traefik derives the yaegi binding identifier
from the module path, not the package's own declared name; without this
override the plugin never loads at all).
- registry.go: resolveAgainst's multi-value return panicked yaegi on an
inline string concatenation; assign to a local first.
- routes_unified.go: errors.As panicked yaegi on interpreted pointer error
types; use plain type assertions (both types are always returned bare,
never wrapped).
- sse.go: readSSE's dispatch closure hit the same multi-value-assignment
panic as registry.go; split into separate statements.
- resp.go: respErr as a defined string type was indistinguishable from a
plain string in a yaegi type assertion, misreading every successful
Redis SELECT reply as failed and silently disabling rate limiting;
changed to a struct.
Reject a nested array reply instead of recursing into it. decodeReply
now takes a depth parameter; a '*' type byte is accepted only at
depth 0, so an element that is itself an array (decodeArray calls
decodeReply at depth+1) is a protocol error rather than a recursive
call. A crafted "*1\r\n*1\r\n..." stream repeated could otherwise
recurse to Go's stack limit — a fatal, unrecoverable error, not a
panic recoverPanic can catch — from only a few megabytes of wire
input. Also lower respMaxArrayLen from 1<<20 to 1024: none of this
client's commands (AUTH, SELECT, INCRBY, EXPIRE, GET) ever return an
array reply at all, so even 1024 is generous headroom, not a limit
tuned to real traffic.
Add a store-down latch to the limiter (limits.go): after any store
operation fails, record the failure time; for storeDownLatchFor (5s)
afterward, every storeIncrBy/storeGet call short-circuits straight to
the fail-open/fail-closed policy without attempting the store at all.
One request can touch several counters (its own and its group's,
request/token/cost, day/month) — each respClient call is already
bounded to ~respCallTimeout on its own, but without the latch every
one of those still paid a fresh bounded wait against a store that had
just proven unreachable. After the window, the next operation probes
the real store again: a natural half-open retry, no separate recovery
state. Guarded by the same logMu mutex lastLogAt already uses for this
kind of small timestamp state.
Clamp respClient's dial timeout to the remaining call budget:
dialTimeoutFor returns min(respDialTimeout, time.Until(deadline)),
with a respDeadlineExceededErr (implements net.Error, Timeout() true)
returned outright if the deadline has already elapsed — e.g. this
call queued behind the client's mutex for its whole budget while an
earlier call was stuck. Without this, a future increase to
respDialTimeout alone, leaving respCallTimeout untouched, could
silently let one dial attempt consume more of a call's total budget
than the call-level deadline is supposed to allow.
New tests: a nested-array reply and an array length one over the new
1024 cap, both asserted as protocol errors; a countingErrorStore stub
proving a second checkAndCount within the latch window makes zero
additional store calls, and that the store is probed again once the
window elapses. Tightened TestLimiter_FailOpenTrue_
HungRedisServer_FallsBackQuickly's bound from <6s to <3s now that the
latch collapses its two store ops into one real network attempt.
Bound respClient's worst-case latency to one respCallTimeout per
do/pipeline call instead of per read/write/handshake step. Compute a
single absolute deadline at call entry, apply it once via
conn.SetDeadline after connecting (covering the AUTH/SELECT handshake
and the call's own command I/O), and skip the reconnect-retry when the
failing attempt was a timeout (errors.As -> net.Error, Timeout()) — a
peer that accepted the connection and went silent must not be asked
the same non-question twice. Previously each read, write, and dial got
its own fresh 2s budget and every caller queued behind the client's
single mutex, so a hung (not merely refused) Redis-compatible server
could stall one checkAndCount call for 16-32s+.
Cap decodeBulk/decodeArray's wire-declared lengths (respMaxBulkLen
64MiB, respMaxArrayLen 1<<20) before allocating: an unguarded
"$9223372036854775807\r\n" overflows n+2 into a negative slice length
and panics make([]byte, n+2); a merely huge-but-valid length forces an
outsized allocation either way. decodeBulk also now verifies its two
trailing bytes are CRLF. Bound readLine to respMaxLineLen (64KiB),
reading byte-by-byte so it can bail before ever buffering an unbounded
line from a peer that never sends '\n'.
Fix TestLimiter_FailClosed_BudgetReadReturnsStoreDownViolation: the
erroringStore it used fails incrBy too, so checkAndCount's own
increment loop already returned a storeDown violation before
evaluateScope (and therefore budgetViolation) ever ran, leaving
budgetViolation's storeDown branch uncovered. Replace it with a stub
that only fails get, so the test's TokensPerDay limit actually drives
evaluateScope into budgetViolation's fail-closed path.
Document redis_store.go's incrBy as deliberately at-least-once: the
reconnect-retry can re-send INCRBY after a lost reply (possible
over-count), and fail-open can additionally count in the fallback —
both fail-safe (more restrictive), never fail-open in the unsafe
direction.
Validate Redis.DB >= 0 in newConfiguredLimiter alongside Address, so a
negative db is a constructor error rather than a SELECT -1 on the wire.
New tests: a hung (accepts, never replies) fake RESP server proving a
respClient call and a failOpen limiter call both stay bounded rather
than stacking timeouts; decodeReply subtests for an oversized bulk
length, a length just over the cap, a huge array length, a bad bulk
trailer, and an oversized line; a negative-DB constructor-rejection
test.
Add resp.go: a minimal stdlib-only RESP2 client (respClient) for a
single Redis-compatible server. It lazily dials one mutex-guarded
connection, applies 2s dial/read/write deadlines, sends AUTH (if a
password is set) then SELECT on every new connection, and reconnects
once on any I/O error before giving up. do and pipeline encode
commands as "*N\r\n$len\r\n..." and decode +/-/:/$/* replies.
Add redis_store.go: redisStore implements counterStore over
respClient. incrBy pipelines INCRBY and EXPIRE in one round trip and
returns INCRBY's value; get treats a missing key (RESP null bulk) as
0.
Wire the limiter's fail-open/fail-closed policy in limits.go, which
Task 5 stored but never read: a configured store's error now routes
through storeIncrBy/storeGet. failOpen=true falls back to the
in-process memoryStore transparently; failOpen=false makes
checkAndCount return a storeDown violation and makes account drop the
sample. Both paths log through an injectable, 30s-rate-limited logf.
Wire newGateway (llmgateway.go) to build a redisStore-backed limiter
when Config.Redis is set (password through resolveSecret, failOpen
defaulting to true unless RedisConfig.FailOpen says otherwise) and a
memory-only limiter otherwise, and to route pricing.go's unknown-model
warnings through the gateway's own logf.
Validate LimitsConfig at construction: a new LimitsConfig.validate
rejects a negative int64 count limit or a negative/NaN/Inf cost limit,
called from newAuthStore for every group and every user (inline and
file-sourced) so a malformed limit is a constructor error instead of
silently behaving as unlimited.
Add RequestsPerDay, TokensPerMonth, and CostPerMonthUSD enforcement
tests alongside the existing per-minute/day/cost-per-day cases, plus
an erroringStore-backed test pair proving the fail-open (fallback
counted) and fail-closed (storeDown violation) branches directly.
Folded the validation work into this single commit rather than a
second one: it interleaves with the fail-open changes across
limits.go, limits_test.go, and llmgateway_test.go closely enough that
splitting by hunk would be artificial.
lukaszraczylo