Three blocking fixes from a second round of adversarial verification,
plus two should-fix items bundled in since they touch the same code.
storeDown detection covered only the fail-CLOSED path. failOpen
defaults to true (newConfiguredLimiter), so on the shipped default a
broken store made storeGetMulti return ok=true after silently reading
the empty in-process fallback, bypassing every storeDown guard in
metrics.go. Add limiter.configuredStoreDown, checked alongside ok in
currentUsage and providerUsage, so a fail-open read served from the
fallback is treated the same as a fail-closed refusal. The pinning
test previously hand-set failOpen=false, which meant it only ever
exercised the case that was already correct; it now leaves failOpen at
its true default.
The escaping rewrite from the previous round achieved injectivity by
copying an invalid UTF-8 byte through raw, which can make the output
itself invalid UTF-8. Measured against a real Prometheus scrape parser
this round: invalid UTF-8 in a label value is fatal, the scrape target
goes down, which is worse than the duplicate series the previous round
was fixing (measured non-fatal, dropped and logged, target stays up).
Replace an invalid byte with a deterministic \xHH marker instead,
proven injective and always valid UTF-8 by an exhaustive test over
every 1- and 2-byte input, not merely spot-checked. Also adds the fast
path a benchmark showed the byte-oriented rewrite needed: unescaped,
already-valid-UTF-8 input, the common case, returns unchanged with
zero allocations instead of paying a full copy.
llmgateway_rate_limit_rejections_total's HELP text claimed it "resets
only on restart", which stopped being true once the previous round
added generation-rotation bounding and reload-triggered eviction: a
scope resets when its user is removed, and every tracked scope resets
together in bulk the instant the tracked-scope cap is exceeded. The
HELP text and README now say so.
Adds llmgateway_limit_store_up, a per-replica gauge for this process's
own connection health to the configured limit store: with the store
flapping faster than its fail latch, every store-backed family can go
silent scrape after scrape with nothing on the endpoint itself saying
why. Also adds a test pinning that Gateway.pruneRejectionsAfterReload
runs only when authStore.maybeReload reports a real reload, never on
every request.
Three fixes from adversarial verification.
writeProviderMetrics rendered a storeDown providerCounters entry as a
hard zero, the same fabricated-reset hazard the request/token/cost
families already guarded against. Skip a storeDown provider or
provider-model entry instead of emitting it.
escapeLabelValue ranged over the string as runes, and Go substitutes
U+FFFD for every invalid UTF-8 byte. Two distinct configured names
that both contain invalid UTF-8 could therefore render as the
identical escaped value, colliding two series into one duplicate
series and failing the whole scrape target. Escaping is now
byte-oriented, which is injective and also avoids the per-rune decode
cost on a large value.
Every store-backed family's HELP text now states its aggregation
contract: with Redis configured, the value is the gateway-wide total
and identical on every replica, so summing across replicas overcounts
by the replica count and a cost alert fires early. Aggregate with
max() instead. Without Redis, the same family becomes genuinely
per-process and sum() becomes correct instead. The rejection counter
and the provider-healthy gauge get their own notes: the former is
always per-process regardless of Redis, the latter reflects each
replica's own independently-running discovery breaker.
Also replaces the AllowedCIDRs doc example (10.0.0.0/8) with a
cluster pod CIDR (10.42.0.0/16) and states plainly that a whole
RFC1918 range is never correct, since a SNAT-ing router or load
balancer makes external traffic arrive from inside it — reproduced
live against the target cluster.
Adds tests pinning three properties mutation-proofed against this
suite: every storeDown skip (deleting any of them now fails
TestMetrics_StoreDown_SkipsRatherThanFabricatesZero), the CIDR-check
running before auth.identify (TestMetrics_
AllowlistedRequest_NeverTouchesAuthFailureTracking), and
escapeLabelValue's injectivity under invalid UTF-8
(TestEscapeLabelValue_InvalidUTF8_Injective). Also covers the new
rejection-counter pruning end to end through a real file-sourced
reload.
Serve a hand-written Prometheus text-exposition document at a
configurable path, defaulting to /metrics. The plugin is stdlib-only
under Yaegi, so this cannot use a client library.
Add MetricsConfig: Enabled gates the route, Path overrides it,
AllowedCIDRs lets an unauthenticated caller scrape from a trusted
source address, and ModelLabel opts into a per-(provider,model)
breakdown that defaults off to avoid a series-count explosion.
Reuse the existing admin bearer-token auth path instead of adding a
second credential check. Reuse the batched, chunked reads GET
/admin/api/overview and GET /admin/api/usage already use, so a scrape
costs no extra store round trip.
Expose request, token, and cost counters per scope, provider attempt
and failure counters, a provider-health gauge from the discovery
circuit breaker, and a budget-consumed-ratio gauge per limited scope
for Prometheus alerting. Add a rate-limit-rejection counter in
limits.go: this is the one number with no existing accounting to read
back, so checkAndCount now increments a small in-process,
per-process-lifetime counter on every violation.
Escape every label value for backslash, double quote, and newline. An
unescaped quote in a label value, such as a discovered model id, makes
the whole scrape unparseable, and Prometheus reports an unparseable
scrape as the target being down rather than as one bad line.
Cover the auth gate matrix, exposition structure (TYPE per family,
counters ending _total, no duplicate series), label escaping
round-trips, the model-label opt-in, and counters reflecting real
seeded usage rather than zeros.
lukaszraczylo