Three fixes from adversarial verification.
writeProviderMetrics rendered a storeDown providerCounters entry as a
hard zero, the same fabricated-reset hazard the request/token/cost
families already guarded against. Skip a storeDown provider or
provider-model entry instead of emitting it.
escapeLabelValue ranged over the string as runes, and Go substitutes
U+FFFD for every invalid UTF-8 byte. Two distinct configured names
that both contain invalid UTF-8 could therefore render as the
identical escaped value, colliding two series into one duplicate
series and failing the whole scrape target. Escaping is now
byte-oriented, which is injective and also avoids the per-rune decode
cost on a large value.
Every store-backed family's HELP text now states its aggregation
contract: with Redis configured, the value is the gateway-wide total
and identical on every replica, so summing across replicas overcounts
by the replica count and a cost alert fires early. Aggregate with
max() instead. Without Redis, the same family becomes genuinely
per-process and sum() becomes correct instead. The rejection counter
and the provider-healthy gauge get their own notes: the former is
always per-process regardless of Redis, the latter reflects each
replica's own independently-running discovery breaker.
Also replaces the AllowedCIDRs doc example (10.0.0.0/8) with a
cluster pod CIDR (10.42.0.0/16) and states plainly that a whole
RFC1918 range is never correct, since a SNAT-ing router or load
balancer makes external traffic arrive from inside it — reproduced
live against the target cluster.
Adds tests pinning three properties mutation-proofed against this
suite: every storeDown skip (deleting any of them now fails
TestMetrics_StoreDown_SkipsRatherThanFabricatesZero), the CIDR-check
running before auth.identify (TestMetrics_
AllowlistedRequest_NeverTouchesAuthFailureTracking), and
escapeLabelValue's injectivity under invalid UTF-8
(TestEscapeLabelValue_InvalidUTF8_Injective). Also covers the new
rejection-counter pruning end to end through a real file-sourced
reload.
Serve a hand-written Prometheus text-exposition document at a
configurable path, defaulting to /metrics. The plugin is stdlib-only
under Yaegi, so this cannot use a client library.
Add MetricsConfig: Enabled gates the route, Path overrides it,
AllowedCIDRs lets an unauthenticated caller scrape from a trusted
source address, and ModelLabel opts into a per-(provider,model)
breakdown that defaults off to avoid a series-count explosion.
Reuse the existing admin bearer-token auth path instead of adding a
second credential check. Reuse the batched, chunked reads GET
/admin/api/overview and GET /admin/api/usage already use, so a scrape
costs no extra store round trip.
Expose request, token, and cost counters per scope, provider attempt
and failure counters, a provider-health gauge from the discovery
circuit breaker, and a budget-consumed-ratio gauge per limited scope
for Prometheus alerting. Add a rate-limit-rejection counter in
limits.go: this is the one number with no existing accounting to read
back, so checkAndCount now increments a small in-process,
per-process-lifetime counter on every violation.
Escape every label value for backslash, double quote, and newline. An
unescaped quote in a label value, such as a discovered model id, makes
the whole scrape unparseable, and Prometheus reports an unparseable
scrape as the target being down rather than as one bad line.
Cover the auth gate matrix, exposition structure (TYPE per family,
counters ending _total, no duplicate series), label escaping
round-trips, the model-label opt-in, and counters reflecting real
seeded usage rather than zeros.
lukaszraczylo