mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
fix(metrics): close the default-config store-down gap, fix escaping
Three blocking fixes from a second round of adversarial verification, plus two should-fix items bundled in since they touch the same code. storeDown detection covered only the fail-CLOSED path. failOpen defaults to true (newConfiguredLimiter), so on the shipped default a broken store made storeGetMulti return ok=true after silently reading the empty in-process fallback, bypassing every storeDown guard in metrics.go. Add limiter.configuredStoreDown, checked alongside ok in currentUsage and providerUsage, so a fail-open read served from the fallback is treated the same as a fail-closed refusal. The pinning test previously hand-set failOpen=false, which meant it only ever exercised the case that was already correct; it now leaves failOpen at its true default. The escaping rewrite from the previous round achieved injectivity by copying an invalid UTF-8 byte through raw, which can make the output itself invalid UTF-8. Measured against a real Prometheus scrape parser this round: invalid UTF-8 in a label value is fatal, the scrape target goes down, which is worse than the duplicate series the previous round was fixing (measured non-fatal, dropped and logged, target stays up). Replace an invalid byte with a deterministic \xHH marker instead, proven injective and always valid UTF-8 by an exhaustive test over every 1- and 2-byte input, not merely spot-checked. Also adds the fast path a benchmark showed the byte-oriented rewrite needed: unescaped, already-valid-UTF-8 input, the common case, returns unchanged with zero allocations instead of paying a full copy. llmgateway_rate_limit_rejections_total's HELP text claimed it "resets only on restart", which stopped being true once the previous round added generation-rotation bounding and reload-triggered eviction: a scope resets when its user is removed, and every tracked scope resets together in bulk the instant the tracked-scope cap is exceeded. The HELP text and README now say so. Adds llmgateway_limit_store_up, a per-replica gauge for this process's own connection health to the configured limit store: with the store flapping faster than its fail latch, every store-backed family can go silent scrape after scrape with nothing on the endpoint itself saying why. Also adds a test pinning that Gateway.pruneRejectionsAfterReload runs only when authStore.maybeReload reports a real reload, never on every request.
This commit is contained in:
@@ -1406,9 +1406,26 @@ plugin's config.
|
||||
- `llmgateway_provider_healthy{provider}` — `1`/`0` gauge from the
|
||||
discovery circuit breaker (`healthState` in the admin API); see
|
||||
[Provider health](#provider-health-discovery-circuit-breaker).
|
||||
- `llmgateway_limit_store_up` — `1`/`0` gauge for this replica's own
|
||||
connection to the configured limit store (Redis); absent entirely
|
||||
when no store is configured. Every store-backed family above simply
|
||||
goes silent, scope by scope, for whatever it cannot currently read —
|
||||
the honest choice for those families, but on its own that gives a
|
||||
store outage no independent signal on this endpoint (with
|
||||
`redis.failOpen: false` and a store flapping faster than its 5s fail
|
||||
latch, every store-backed family can be absent scrape after scrape
|
||||
with nothing else here saying why). This gauge is that signal.
|
||||
- `llmgateway_rate_limit_rejections_total{scope_kind,scope_id}` — how
|
||||
many requests a rate or budget limit actually refused, by scope. This
|
||||
one has no equivalent in the admin JSON API: nothing else tracks it.
|
||||
It does **not** reset only on process restart: a scope's own count
|
||||
also resets to `0` when that user is removed via a hot-reloaded users
|
||||
file, and **every** tracked scope resets together, in one bulk cliff,
|
||||
the instant more than 2000 distinct scopes have ever been seen in
|
||||
this process's lifetime (a bounded-memory safeguard, not a graceful
|
||||
per-scope decay). A value that unexpectedly drops is one of these
|
||||
events, not a scrape anomaly — see the live `# HELP` line for the
|
||||
exact current threshold.
|
||||
|
||||
- **Aggregation across replicas — read this before wiring an alert**: with
|
||||
`redis` configured, **every replica reads the SAME shared counters**
|
||||
@@ -1425,7 +1442,7 @@ plugin's config.
|
||||
identical families become genuinely **per-replica** instead — each
|
||||
replica counts only the traffic it personally handled — and `sum()`
|
||||
becomes the correct aggregation instead. This flips on that one config
|
||||
bit; know which one you are running before wiring an alert. Two
|
||||
bit; know which one you are running before wiring an alert. Three
|
||||
families do **not** follow this rule:
|
||||
- `llmgateway_rate_limit_rejections_total` lives only in each replica's
|
||||
own process memory, **never** in Redis, regardless of the `redis`
|
||||
@@ -1437,24 +1454,41 @@ plugin's config.
|
||||
possible; if you must aggregate, `min()` surfaces "at least one
|
||||
replica sees this provider as unhealthy" — `sum()` is meaningless for
|
||||
a 0/1 gauge.
|
||||
- `llmgateway_limit_store_up` reflects each replica's own, locally
|
||||
-discovered connection health to the configured store — it cannot
|
||||
itself be read from the store it describes, so it is inherently
|
||||
per-replica too. Same guidance as `llmgateway_provider_healthy`:
|
||||
prefer per-instance, `min()` if you must aggregate.
|
||||
|
||||
- **Labels are escaped**: a label value that is user- or
|
||||
upstream-controlled (a user/group name, a model id) is escaped for
|
||||
backslash, double quote, and newline, byte-for-byte — the exact rule
|
||||
the text-exposition format requires. This matters more than it looks:
|
||||
Prometheus reports an unparseable scrape as the **whole target being
|
||||
down**, not as one bad line, so a single unescaped quote in a model id
|
||||
would otherwise take the entire endpoint offline from Prometheus's point
|
||||
of view, not just that one series.
|
||||
backslash, double quote, and newline — the exact rule the
|
||||
text-exposition format requires. This matters more than it looks: an
|
||||
unescaped quote or newline breaks the exposition format's own lexical
|
||||
structure (an unterminated quoted string), and Prometheus reports an
|
||||
unparseable scrape as the **whole target being down**, not as one bad
|
||||
line — so a single unescaped quote in a model id would otherwise take
|
||||
the entire endpoint offline from Prometheus's point of view, not just
|
||||
that one series. A raw, invalid UTF-8 byte is escaped too, as a
|
||||
deterministic `\xHH` marker rather than passed through unchanged: an
|
||||
invalid-UTF-8 label value is independently fatal to a real Prometheus
|
||||
scrape (measured against `promparse.go`), so the escaping keeps every
|
||||
distinct input distinguishable (no two different names collapse onto
|
||||
one series) while guaranteeing the output stays valid UTF-8 either way.
|
||||
|
||||
- **A store outage never fabricates a zero**: if the configured Redis is
|
||||
unreachable and `redis.failOpen` is `false`, an affected scope's
|
||||
request/token/cost/provider counters are **skipped entirely** for that
|
||||
scrape rather than reported as `0` — a temporary gap in the series is
|
||||
the honest signal; a fabricated `0` would read to Prometheus as a
|
||||
counter reset followed by a phantom spike the instant the store
|
||||
recovers. `llmgateway_provider_healthy` is unaffected either way — it
|
||||
never reads from the store.
|
||||
unreachable, an affected scope's request/token/cost/provider counters
|
||||
are **skipped entirely** for that scrape rather than reported as `0` —
|
||||
a temporary gap in the series is the honest signal; a fabricated `0`
|
||||
would read to Prometheus as a counter reset followed by a phantom spike
|
||||
the instant the store recovers. This holds regardless of
|
||||
`redis.failOpen`: with `failOpen: false` (or unset) the request itself
|
||||
is refused during the outage, and with `failOpen: true` (the default)
|
||||
requests keep flowing against the in-process fallback, but `/metrics`
|
||||
still treats the affected scopes as unreadable rather than presenting
|
||||
the fallback's own, non-authoritative zero as real gateway-wide usage.
|
||||
`llmgateway_provider_healthy` is unaffected either way — it never reads
|
||||
from the store.
|
||||
|
||||
## MCP and A2A
|
||||
|
||||
@@ -1586,6 +1620,11 @@ plugin's config.
|
||||
envelope. A successful attempt logs the resolved user's name and the
|
||||
route. Neither line ever includes the presented API key — only the
|
||||
identity it resolved to, never the key material itself.
|
||||
- **`GET /metrics` (see [Metrics](#metrics)) exposes per-user and
|
||||
per-group cost and usage data**, the identical sensitivity as the admin
|
||||
dashboard's JSON API, so it is gated the same way: an admin bearer
|
||||
token by default, with an optional, narrowly-scoped `allowedCIDRs`
|
||||
bypass for a co-located scraper — never leave it reachable with neither.
|
||||
|
||||
## Anonymous usage reporting
|
||||
|
||||
@@ -1741,8 +1780,11 @@ struct — every field in that example round-trips through `New()`
|
||||
cleanly), a dummy backing `Deployment`/`Service` (Traefik requires a real
|
||||
router backend even though this middleware never actually reaches it —
|
||||
see [Quickstart](#quickstart)), an `IngressRoute` wiring it all to a
|
||||
host, and a Traefik Helm values snippet for loading the plugin and
|
||||
wiring provider keys in from Secrets via environment variables.
|
||||
host, a second, Host-free `IngressRoute` plus a `VMServiceScrape` for
|
||||
scraping [`/metrics`](#metrics) (a `ServiceMonitor` works the same way on
|
||||
a Prometheus-operator cluster — only the CRD kind differs), and a Traefik
|
||||
Helm values snippet for loading the plugin and wiring provider keys in
|
||||
from Secrets via environment variables.
|
||||
|
||||
The short version:
|
||||
|
||||
@@ -1758,6 +1800,11 @@ The short version:
|
||||
- Point `redis.address` at your cluster's existing Redis/Valkey/Dragonfly
|
||||
rather than standing up a dedicated instance for this plugin — the
|
||||
distributed limiter is a client, not a datastore.
|
||||
- [`/metrics`](#metrics) is served by Traefik itself, not by any backend
|
||||
Service, so scraping it needs its own Host-free route (see the example
|
||||
manifest's own comment on this — a Host-restricted router alone 404s an
|
||||
in-cluster scrape) and its own CIDR/admin-key auth — never point a
|
||||
scraper at it without one or the other configured.
|
||||
|
||||
### Config hot-reload
|
||||
|
||||
|
||||
Reference in New Issue
Block a user