An audit against the code found stale defaults, wrong numbers, an
invented error-response example, and capability claims for features that
do not exist (query-complexity limiting, IP-based and per-endpoint rate
limits, config hot-reload).
Correct the env-var defaults, the degraded-retry and keepalive numbers,
and the circuit-open response example. Reframe the cache-key isolation as
a correctness boundary, not an authentication one, because the proxy does
not verify JWT signatures. Document the admin dashboard exposure, the API
key scope, the WebSocket ban and rate-limit checks, and the new
BIND_ADDRESS option.
The /api/connection-pool/health endpoint hardcoded a keepalive interval
of 15s while the pool actually uses 45s, so it reported a false value.
The endpoint now reads the live keepAliveInterval field. A startup
backoff comment that claimed jitter is also corrected; the backoff has
no jitter.
The WebSocket route skipped the ban and rate-limit checks that the HTTP
path enforced, and returned errors set no custom status, so circuit-open
became HTTP 500. The route now runs the same ban and rate-limit checks
before the upgrade, and a custom error handler maps errors to the
correct status, including 503 for circuit-open. The API and admin
listeners honour the BIND_ADDRESS option.
Invalid numeric environment values parsed to zero, an invalid prefixed
value fell through to a stale unprefixed value, and the prefixed boolean
parser rejected on and yes that the unprefixed parser accepted. A
non-positive cache TTL or retry-budget rate could crash or disable a
subsystem.
Parsing now warns and uses the coded default for an invalid value,
prefixed and unprefixed booleans accept the same tokens, and the cache
TTL and retry rate are clamped. A dead per-endpoint circuit config was
removed and a BIND_ADDRESS option was added, defaulting to the current
all-interfaces bind. The pprof server now stops on shutdown.
The circuit breaker ignored the trip-on-timeout, trip-on-4xx, and
trip-on-5xx flags and never cleared its counts, so the failure ratio
became lifetime-cumulative. The backoff-multiplier and max-backoff
settings did nothing. Circuit-open surfaced as HTTP 500.
Failure classification now honours the trip flags, the breaker uses a
rolling count window, and repeated trips extend the open state up to the
configured maximum. A helper maps circuit-open to HTTP 503. The request
body is copied once per request instead of once per retry. Defaults
reproduce the prior behaviour: the breaker is off by default and a
backoff multiplier of one keeps a constant open window.
Burst and the sustained rate were compared against the same window, so
a burst above the sustained rate never took effect. Burst is now
measured over a short window on top of the sustained limit. An unset
burst preserves the previous rate-only behaviour.
Shutdown closed the Redis client while the publisher goroutine could
still use it, so the instance key sometimes lingered in the cluster set,
and the goroutine could read a package global after shutdown returned.
Shutdown now removes the instance key, joins the publisher goroutine,
then closes the client, and clears the singleton so a later
initialisation creates a fresh instance.
get_metrics_name called os.Hostname on every metrics call and keyed a
never-evicted cache by label values. The hostname is now resolved once
and the cache is keyed by label names, so it cannot grow with high
cardinality values. The metrics listener also honours a graceful
shutdown context.
SetFieldName wrote a package-global map with no lock while log read it
from many goroutines, and the level, caller, and time-format setters
mutated fields without synchronisation. A runtime setter call could
crash the process with a concurrent map write.
The scalar fields now use atomics on the hot log path and the global
field-name map uses a read-write mutex. Behaviour for init-time
configuration is unchanged.
The count-based evictOldest used an O(n squared) selection sort with a
candidate cap over an unordered map range, so it could evict the wrong
entries. It now collects all entries and sorts by expiry, matching the
memory-based evictor.
A non-positive CACHE_TTL behaved differently per backend: the standard
memory cache panicked, Redis stored a permanent key, and the LRU cache
expired entries immediately. Each backend now clamps a non-positive TTL
to a shared default at its Set boundary. The nil-logger fallback in the
Redis wrapper now uses the constructor so it cannot panic.
The manager ran every component concurrently, so on SIGTERM the cache
and connection backends closed while the proxy was still draining
in-flight requests, which then failed with closed clients.
Add an ordered phase to each component. Run phases in ascending order,
components within a phase concurrently, under one timeout budget. Drain
the proxy in an earlier phase than the backend closers.
The refill goroutine started only when the budget was enabled at
construction. Enabling it later through UpdateConfig left a bucket that
never refilled.
Start the refill loop from UpdateConfig, guarded by sync.Once so it
starts at most once, sharing the existing shutdown lifecycle.
The JSON log path re-marshaled redacted output with no size cap, so a
large body was logged in full. An XML-form secret whose closing tag
fell past the truncation cut also leaked its prefix.
Cap the re-marshaled JSON at MaxLogBodySize after redaction. Strip a
trailing unterminated sensitive tag. Fold tag names with a
length-preserving ASCII lower so the cut index stays valid.
Clear reset the counters with StoreInt64(0) after deleting entries, so
a Set racing the reset left a live entry while counters read zero, and
the next Delete drove them permanently negative.
Rewrite Clear to LoadAndDelete each key and subtract the linearized
returned entry, matching the rest of the file. Size the key slice from
the live entry count.
The SCAN MATCH pattern used the raw prefix, so a prefix containing
Redis glob metacharacters cleared the wrong keys or none. A Del error
also aborted the clear and discarded the remaining batches.
Escape the prefix for MATCH, continue past a per-batch Del error, and
return the joined error. Apply the same escaping to CountQueries.
truncateUTF8 validated the whole string each iteration. One
invalid byte emptied the log line at quadratic cost. A secret
whose closing quote fell past the truncation cut also leaked its
leading characters, because the redaction pattern needs a
terminator.
Back off over the last rune only, bounded by the maximum rune
width, and trim the partial rune before redaction runs. Strip a
trailing unterminated quoted value after the cut.
When the primary request panicked, waiters received a nil response
and nil error, which the proxy turned into an empty HTTP 200.
Recover in the deferred release, store an error response for the
waiters, then re-panic so the primary panic semantics are
unchanged. The test gates the panic on the live waiter count, so
every waiter is coalesced before the release.
The Set mutex serialised writes but left Delete, expiry, and
eviction racing the same counters, so double subtraction remained.
Eviction also pre-allocated from the configured maximum, which
could allocate hundreds of megabytes on a near-empty cache.
Use sync.Map.Swap in Set and adjust counters from the linearized
previous value. Size the eviction slice from the live entry count.
CleanExpiredEntries now subtracts the size of the entry it
actually deleted. A concurrency test recomputes expected counters
from the final map contents.
The handler returned while the ping goroutine could still write to
a connection that gofiber had recycled into its pool. This caused
a nil dereference panic or a ping on an unrelated socket.
Join the ping goroutine through a done channel before the handler
returns. Compute effective keepalive values once in the
constructor, clamp pongTimeout to at least pingInterval with one
warning, and pass the effective values to both proxy directions.
A negative interval or timeout disables keepalive. New test drives
the real HandleWebSocket wiring.
The read-error path in loadBannedUsers returned while it held the
shared file lock. One transient read failure blocked every later
ban write for the process lifetime. The reload also applied a stale
snapshot over a ban that landed between the read and the replace.
Acquire bannedUsersMu first, then the file lock, for the whole
read and replace. Release both locks with defer on every path.
A deterministic test proves the lock is released after a read
error.
The parser does not reject fragment cycles. A 60-byte query with a
self-referential fragment caused unbounded recursion and a stack
overflow, which recover cannot catch. This was a remote DoS when
BLOCK_SCHEMA_INTROSPECTION was enabled.
Thread a visited set through checkSelections so each fragment is
entered at most once per operation. Tests cover self cycles, mutual
cycles, and a deep acyclic chain that must still detect
introspection.
The shutdown component could observe the app before Listen ran.
fasthttp then treated Shutdown as a no-op and reset its stop flag.
The proxy started to serve after shutdown was requested.
Publish httpProxyApp from the OnListen hook and clear it after
Listen returns. Honor the shutdown context and retry until the
drain completes or the context expires. Gate the tests on the
publish signal with a bounded context.
cache.Shutdown only called Close() on an io.Closer, which only the Redis
backend (CacheWrapper) satisfied -- the in-memory Cache's Shutdown
(cancels ctx, stopping its cleanupRoutine background ticker) was never
reached, so that goroutine leaked until process exit (and with the
default set-up, and whenever Redis is unreachable and EnableCache falls
back to memory).
Add Close() error to Cache (calls Shutdown, idempotent, leaves the
cache usable) so cache.Shutdown handles both backends symmetrically.
Update Shutdown doc; add test TestMemoryCacheClose_IdempotentAndStillUsable.
PingInterval/PongTimeout were configured but unused, so a half-open peer
(TCP open but silent) left both proxy goroutines and the handler blocked
in ReadMessage forever. Wire both directions: read deadline refreshed by
real traffic or pong, plus a sidecar ping goroutine (WriteControl is
safe to call concurrently with WriteMessage). Add regression test
TestWebSocketProxy_ReapsSilentBackend_Keepalive proving a silent backend
is reaped via the read deadline instead of hanging.
- graphql.go: parseGraphQLQuery now indexes top-level FragmentDefinition nodes
and checkSelections resolves FragmentSpread by name, closing an
introspection-blocking bypass (`{ ...F }` where F spreads __schema was
allowed because the spread carried no selection set and definitions loop
skipped fragments).
- proxy.go/server.go: handleCaching stores the computed cache hash on the
request context; performProxyRequest reuses it instead of recomputing a
full MD5 of the request body per proxied request (fallback preserved for
direct callers).
- cache/memory: Set serializes the exists-check + counter update + store
with the embedded mutex so concurrent Sets of the same new key can't
double-increment entryCount/memoryUsage; regression test added.
Verification: go build, go vet, gofmt clean; full go test ./... and
go test -race ./... green (9 packages); golangci-lint 0 issues; trufflehog clean.
- tracing: validateEndpoint used fmt.Sscanf("%s:%d") which treats %s as
greedy, so every valid host:port endpoint failed and traced==nil disabled
tracing entirely. Validate with net.SplitHostPort (pure helper, port range
0-65535); test TestValidateEndpoint.
- proxy: createFasthttpClient used clientTimeout for every timeout slot,
ignoring the parsed CLIENT_READ_TIMEOUT/CLIENT_WRITE_TIMEOUT (silent
config loss, documented separately in README). Honour per-side read/write
timeouts, falling back to client timeout when unset.
- cache/memory: evictToFreeMemory capped candidates at maxCacheSize/5 and
stopped at the subset end, so a large write could leave the cache over
its memory limit. Collect all entries and sort.Slice (correct + O(n log n)
vs O(cap^2)), evict until the target is met or the cache is drained.
- cache/redis: the Redis cache client was never closed at shutdown (unlike
the metrics aggregator). Add RedisConfig.Close, CacheWrapper.Close, a
cache.Shutdown and register it with the ShutdownManager.
- ratelimit: custom UnmarshalJSON temp struct only carried interval/req,
silently dropping the configured 'burst' and 'endpoints' fields. Burst is
consumed at checkRateLimit (burst allowance) but could never be set via
config file. Carry all fields in the temp struct.
- sanitization: a large non-parsable body was truncated before redaction,
potentially leaking a secret in the first MaxLogBodySize bytes, and the
truncation could split a multi-byte UTF-8 rune. Redact the retained
prefix and truncate on a rune boundary (sanitizeTruncatedBody).
- tests: TestRateLimitConfigUnmarshalKeepsBurstAndEndpoints,
TestSanitizeTruncatedBodyRedactsAndKeepsUTF8, TestTruncateUTF8RuneBoundary.
The generic typed env reader only handled string/int/bool; any float64 key
fell through to default, so CIRCUIT_FAILURE_RATIO, CIRCUIT_BACKOFF_
MULTIPLIER and RETRY_BUDGET_TOKENS_PER_SEC were silently impossible to
override via environment (always the compile-time default).
Add a float64 case mirroring the int one: parse the GMP_ prefixed key,
then the plain key, else the default (envutil has no GetFloat).
Test: extended Test_getDetailsFromEnv with float-valid and float-malformed
cases.
bannedUsersIDs (sync.Map) was modified by admin Store/Delete (apiBanUser/
apiUnbanUser) and by loadBannedUsers->replaceBannedUsers, which does a
clear-then-rebuild (Range-Delete then Store). The flock only serialized
FILE I/O, not the in-memory map, so a reload could clear a just-applied
ban before storeBannedUsers snapshotted it (or persist the wiped set to
disk).
Add bannedUsersMu (mutex) held across both the map update and the file
write in the admin path, and across the clear-then-rebuild in
replaceBannedUsers. Lock ordering: bannedUsersMu before the file lock;
loadBannedUsers now releases its file read lock before applying the
snapshot, so no ordering inversion and no deadlock.
Also harden the connection-pool health type assertion against a missing
map key (apiConnectionPoolHealth).
Test: TestBannedUsersConcurrentModifyAndReload (concurrent modify +
reload must not lose a persisted ban; race-free under -race).
1) connection_pool.go: GetConnectionStats read lastRecoveryAttempt (a
multi-word time.Time) without any lock while checkAndRecover wrote it
under recoveryMutex -> data race on live admin API endpoints. Snapshot it
under recoveryMutex (the writer's lock). Regression test
TestGetConnectionStatsRace drives the read path concurrent with the locked
write under -race.
2) cache/memory memory.go: Cache.Get expired path used an unconditional
Delete + atomic decrement, so concurrent Gets on the same expired entry
each decremented entryCount/memoryUsage -> permanent negative drift. Use
LoadAndDelete-with-guard (as evictOldest/evictToFreeMemory already do) so
only the goroutine that actually removes the entry decrements. Regression
test TestMemoryCacheConcurrentGetOnExpiredEntry_CountersConsistent.
StartHealthChecking's goroutine captures startupProbe=true and blocks on
readinessChan until the readiness probe completes. WaitForBackendReady only
closed readinessChan on SUCCESS; on timeout it returned an error without
closing it.
main proceeds anyway after a timeout ('Staring proxy anyway'), so the
health-check goroutine stayed blocked forever on readinessChan: periodic
checks never began and bhm.IsHealthy() stayed false permanently, so the
backend could never be marked healthy again even after it recovered.
Release readinessChan on the timeout path too (via sync.Once for safety),
so the health-check goroutine proceeds to periodic checking.
Test: TestHealthChecksStartAfterStartupTimeout (after WaitForBackendReady
times out, readinessChan is closed).
GetGzipReader got a reader from the pool, Reset it, and on error returned a
fresh gzip.NewReader while never returning the pooled reader to the pool —
one leaked reader per reset error (non-gzip input), silently draining the
pool. Return the pooled reader without Close (Close panics because the
decompressor is nil until a header parses); the next caller's Reset
reinitialises it.
Test: TestGetGzipReader_ResetErrorPath (error path leaves the pool usable
for subsequent valid gzip reads).
In RequestCoalescer.Do, the primary path calls fn() before storing the
response, deleting the inflight key and calling wg.Done(). If fn() panics,
all three steps are skipped: the sync.Map entry leaks and every coalesced
waiter remains blocked on wg.Wait() forever (permanent deadlock for that
key). Route release through a defer so waiters are always released and the
entry cleared, even on panic.
Test: TestRequestCoalescer_PanicSafety (a panicking primary must not poison
the key for subsequent calls).
IsRetryable and GetStatusCode used a direct type assertion, so a ProxyError
wrapped with %w was never detected (status fell back to 500, retryable to
false). Use errors.As, which dereferences the existing Unwrap chain and
matches the file's own Unwrap method.
Test: Test_IsRetryable_GetStatusCode_Wrapped (wrapped error preserves both
flags).
Note: these helpers currently have no production callers (test-only
surface), so this is a correctness hardening of the error API.
Two defects in the bidirectional proxy:
1. Goroutine/socket leak on client disconnect. proxyClientToBackend and
proxyBackendToClient each block on their own Read; when the client drops,
the client-direction goroutine returns but the backend-direction goroutine
stays blocked on backend.ReadMessage, so wg.Wait() never returns and the
goroutine plus backend socket leak until the backend independently closes
or sends. Each direction now closes the opposite connection on exit,
unblocking the peer.
2. Backend->client messages were unbounded. client.SetReadLimit only bound
what the client could send; a large or misbehaving backend could push
arbitrarily large frames through. backend.SetReadLimit now bounds it.
Tests: TestWebSocketProxy_TeardownOnClientDisconnect (backend must be closed
after client disconnect), TestWebSocketProxy_BackendReadLimit (connection
torn down when backend exceeds the limit).
On signal, main waited on wg for StartHTTPProxy's server.Listen, which never
returns because nothing called server.Shutdown(); it timed out after 10s and
process-exited, abruptly killing in-flight GraphQL requests.
Expose the proxy Fiber app and register an http-proxy Shutdown component so
server.Listen returns after Shutdown, letting connections drain. Guard the
shared app handle with a mutex (written by the StartHTTPProxy goroutine, read
by the shutdown component). Verified with build, vet, full test and full
race suite.
Clear() used FlushDB, wiping every key in the selected Redis DB even if
that database is shared with other tenants. Switch to a prefix-scoped
SCAN+DEL so only this cache's keys (prefixed) are removed, matching the
memory cache's Clear semantics.
GetMemoryUsage made an Info() round-trip purely to return 0, stalling
the admin dashboard when Redis is slow; return 0 without the call.
UpdateConfig writes tokensPerSecond/maxTokens/enabled as plain fields while
the refill goroutine and AllowRetry readers touch them concurrently. Guard
all reads/writes of the mutable config with an RWMutex.
Adds TestRetryBudget_ConcurrentUpdateConfig, which races UpdateConfig
against AllowRetry/GetStats and is verified clean under -race.
Also drop the package-level min() that shadowed the Go 1.25 builtin.
doProxyRequestWithTimeout returns an error for every non-200 status and
executeProxyAttempt returned it as a plain, retryable error, so 4xx
client errors (400/404/405/...) were retried with exponential backoff
and the 4xx branch of isRetryableStatusCode was dead code.
Additionally, retry-go's finite-attempt loop only consults the custom
RetryIf, which overrode the default IsRecoverable gate, so retry.
Unrecoverable errors (non-retryable 4xx AND timeouts) were still
retried. Honor IsRecoverable in RetryIf so these never retry.
Now 5xx/429 are retried, other 4xx and timeouts fail fast.
Adds Test_proxyRetryPolicyByStatusCode locking in the policy.
The test hit live external endpoints (telegram-bot.app, google.com) and
failed once telegram-bot.app changed /v1/graphql to return 405. Replace
real hosts with local ok (200) and err (404) servers, preserving the
query->read-only / mutation->write routing semantics under test.
- Document WebSocket-streamed live metrics (2s) with 5s polling fallback, the overview KPIs and live charts, and the cluster-view toggle (aggregated vs this-node, disabled in single-node).
- Complete the API endpoint list (stats, cache, ws/stats, cluster endpoints, cache/clear) to match the registered routes. - Regenerate static/admin-dashboard.png for the redesigned dashboard.
Fixes the stale 'auto-refreshes every 5 seconds' claim.
BREAKING CHANGE: upgrades the HTTP framework from gofiber/fiber/v2 to gofiber/fiber/v3, and gofiber/websocket/v2 to gofiber/contrib/v3/websocket.
- Handlers now take fiber.Ctx (interface) instead of *fiber.Ctx. - DisableStartupMessage moved from fiber.Config to fiber.ListenConfig on the proxy/api/monitoring Listen calls. - cors AllowOrigins is now []string; c.BodyParser -> c.Bind().Body(). - app.Test takes fiber.TestConfig instead of an int timeout (tests updated). - WebSocket swapped to contrib/v3/websocket (same Conn/New/Config/IsCloseError/IsWebSocketUpgrade API; Conn.Query preserved). - Adopts fasthttp v1.71.0 (pulled by fiber v3), resolving the v1.71 Host-header enforcement that broke the prior pinned-v1.69 workaround.
fix(connpool): the background keep-alive goroutine no longer reads the mutable global cfg (captures HostGraphQL/HealthcheckGraphQL at pool creation). Under v3 timing it raced with parseConfig/test cfg reassignment; go test -race ./... is now clean (0 data races).
test: make the circuit-breaker cache-fallback and request-coalescing integration tests deterministic (cache TTL outlives the retry loop; coalescing asserts against a measured single-request baseline instead of a hard-coded count); drop the flaky timing-ratio assertion in TestTimingAttackResistance (constant-time guarantee comes from subtle.ConstantTimeCompare, not wall-clock timing).
Verified: go build, go vet, golangci-lint (0 issues), go test -race ./... (all packages pass, 0 races), govulncheck (no vulnerabilities). App + dashboard + WebSocket verified live under v3 in single-node and cluster modes.
Bump golang.org/x/net v0.52.0 -> v0.56.0, fixing GO-2026-5026 (idna Punycode label rejection) and GO-2026-4918 (HTTP/2 transport infinite loop). govulncheck now reports no vulnerabilities. Also bumps x/sync, x/sys, x/term, x/text, otel v1.43->v1.44, grpc v1.80->v1.81 and genproto.
Hold github.com/valyala/fasthttp at v1.69.0: v1.71.0 changes request handling such that the API auth/integration security suites fail (app.Test errors on benign requests), violating the behaviour-preserving upgrade rule. fasthttp had no flagged vulnerability, so pinning it back loses no security fix. Adopting v1.71.0 needs separate investigation.
Verified: go build, go vet, golangci-lint, go test -race ./... all pass; govulncheck clean.
Starting with ENABLE_CIRCUIT_BREAKER=true panicked at boot: parseConfig -> initCircuitBreaker registers a metrics gauge, but cfg.Monitoring is assigned later by StartMonitoringServer, so the registration dereferenced a nil *MetricsSetup.
Add nil-receiver guards to the MetricsSetup registration methods (RegisterMetricsGauge/GaugeFunc/Counter, RegisterFloatCounter, RegisterMetricsSummary/Histogram), matching their existing 'return a dummy to prevent panics' contract. The guard is a no-op for non-nil receivers, so existing callers are unaffected. The breaker and the admin dashboard (which reads gobreaker state directly) keep working.
Add regression tests for the nil-receiver methods and for initCircuitBreaker with nil monitoring.
RecordRequest() only ran inside proxyTheRequest, but cache hits return from handleCaching before reaching it. As a result the dashboard's current RPS read 0 whenever traffic was served from cache (e.g. a repeated query at 100% hit rate). Record the request on the cache-hit branch too; misses and proxied/GET requests still record in proxyTheRequest, so there is no double counting.
Verified: 100% cache-hit load now reports ~16-20k req/s instead of a flat 0.
Replace the AI-slop visual (indigo->purple gradient, glassmorphism, uniform rounded cards, hover-lift) with a deliberate warm-paper 'instrument panel': monospace tabular telemetry, GraphQL brand magenta accent, flat hairline cards, real type hierarchy, WCAG-AA contrast, focus-visible, prefers-reduced-motion and aria-live status. No new font/CDN deps.
Make the cluster-view toggle functional: the WebSocket stats stream now honours a ?view=local|cluster query param, so toggling streams this-node-only vs aggregated metrics instead of being cosmetic (the stream previously always sent cluster-aware data when an aggregator existed). The toggle is always visible -- disabled with a 'single node' hint when Redis cluster mode is off -- so it is discoverable.
Fix cluster-mode rendering bugs: derive backend health from healthy/total instances (health card no longer stuck on 'Loading'), show 'n/a' for Redis memory instead of a misleading 0 MB, render aggregated CB/retry/coalescing/connections on the polling path too, and surface the proxy version and backend-health detail.
Tests assert the served dashboard against stable markers instead of the cosmetic page title.
The repo's go.mod has required go 1.25.0 since the perf+coverage pass,
but the release workflow still pinned setup-go to 1.24 — the latest
1.24.X tool refuses to compile a 1.25 module with GOTOOLCHAIN=local,
breaking auto-release on every push.
The released v0.45.1 binary shipped with the source default
appVersion="dev" because .goreleaser.yaml had ldflags="-s -w" only,
so every startup ping was rejected by the receiver with HTTP 400
(invalid version: regex requires leading digit).
Two-layer fix:
1. .goreleaser.yaml now passes -X main.appVersion={{.Version}} so
goreleaser-built binaries report the actual release version.
2. Switch to telemetry.SendForModule which prefers
debug.ReadBuildInfo Main/Deps when available, falling back to
appVersion. This means `go install github.com/lukaszraczylo/
[email protected]` users also get correct versions
without relying on the ldflag.
Bumps oss-telemetry to v0.2.1 for SendForModule.
lukaszraczylo