lukaszraczylo/traefik-llmgateway
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/traefik-llmgateway.git synced 2026-10-12 04:54:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(metrics): add README section, testData, and a scrape example

Browse Source 
The metrics feature shipped in code only: no README section, no
metrics block in .traefik.yml testData, nothing under examples/. The
Go doc comments were the entire manual for a human operator or a
config-generating assistant.

Add a README Metrics section covering the served path, both auth
modes (admin bearer token, optional CIDR allowlist), the CIDR-sizing
warning with the live-reproduced SNAT failure mode, the modelLabel
cardinality trade-off, the exposed metric families, the cross-replica
aggregation contract, label escaping, and store-outage behaviour. Add
a MetricsConfig reference table matching the existing AdminConfig
one, and a metrics row in the top-level Config table.

Add metrics: {enabled: true} to .traefik.yml's testData, the
catalog-facing smoke-test config, deliberately without allowedCIDRs
so a copy-pasted example never skips the README's own sizing
guidance.

Add a metrics block to the Kubernetes Middleware example and a
VMServiceScrape resource: the target cluster runs VictoriaMetrics, not
Prometheus, so the CRD differs even though the exposition format does
not. Verified the VMServiceScrape shape (apiVersion, selector,
endpoints, path) against the victoriametrics/operator documentation
rather than assuming it. Notes that /metrics is served by Traefik
itself, not by any backend service, since this middleware intercepts
the path before a backend is ever reached.
This commit is contained in:
lukaszraczylo
2026-08-23 00:59:13 +01:00
parent 244dd6185f
commit 935df1a3f0
3 changed files with 204 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.traefik.yml
+7
View File
@@ -28,3 +28,10 @@ testData:
- name: tester
group: default
apiKey: test-user-key
# Prometheus text-exposition endpoint — see README.md#metrics. No
# allowedCIDRs here deliberately: this is the catalog-facing smoke-test
# config, and README.md#metrics has the full CIDR-sizing guidance
# (narrowest block only, never a whole RFC1918 range) that a
# copy-pasted example here would risk skipping past.
metrics:
enabled: true
README.md
+144
View File
@@ -196,6 +196,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
| `breaker` | `BreakerConfig` | `{}` (every default) | Per-provider discovery circuit breaker — see [Provider health (discovery circuit breaker)](#provider-health-discovery-circuit-breaker). Every field is individually zero-means-default; a provider whose discovery never fails is unaffected regardless of what this block contains. |
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
| `metrics` | `*MetricsConfig` | `nil` (disabled) | Prometheus text-exposition endpoint — see [Metrics](#metrics). `nil` or `enabled: false` means the metrics route is not registered at all. |
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
| `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. |
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
@@ -327,6 +328,15 @@ own is governed purely by their group's — see
|---|---|---|---|
| `enabled` | `bool` | `false` | `false`: the `/admin*` routes are not registered at all — a request to any of them falls through to the plugin's existing 404/`passthroughUnknown` handling like any other unrecognized path. `true`: the read-only dashboard per [Admin](#admin) below. |
### `MetricsConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `enabled` | `bool` | `false` | `false`: the metrics route is not registered at all. `true`: the Prometheus text-exposition endpoint per [Metrics](#metrics) below. |
| `path` | `string` | `/metrics` | Overrides the served path. |
| `allowedCIDRs` | `[]string` | `[]` (no bypass) | CIDR blocks whose source address may scrape without an admin key — e.g. `10.42.0.0/16` for a cluster's pod network. **Never use a whole RFC1918 range like `10.0.0.0/8`** — see [Metrics](#metrics)'s own warning for why that specific mistake is live-reproducible, not theoretical. |
| `modelLabel` | `bool` | `false` | Adds a `model` label to the provider attempt/failure counters, breaking them out per `(provider, model)` pair. Off by default: a deployment with hundreds of discovered model ids is a series-count explosion waiting to happen. |
### `ModelPricing`
| Field | Type | Semantics |
@@ -1312,6 +1322,140 @@ or in CI.
for asset responses to be the exception. The dashboard loads no
external asset of any kind and works in an air-gapped cluster.
## Metrics
A hand-written Prometheus text-exposition endpoint — the plugin is
stdlib-only under Yaegi, so it cannot use `client_golang` or any other
metrics library. Off by default (`metrics.enabled: false`, or the
`metrics` block omitted entirely): the route is not registered at all,
the same nil-disables convention [Admin](#admin) uses. Served at
`/metrics` by default, overridable via `metrics.path`. Works identically
with any scraper that speaks the standard text exposition format —
Prometheus with a `ServiceMonitor`/`PodMonitor`, or
[VictoriaMetrics](https://victoriametrics.com) with a `VMServiceScrape`;
the format is the same either way, only the CRD kind that points a
scraper at this endpoint differs, and that is entirely outside this
plugin's config.
- **This endpoint exposes per-user and per-group cost and usage data**
(everything [Admin](#admin)'s `GET /admin/api/usage` exposes, in
Prometheus form), so it is never reachable unauthenticated by default.
Two ways in, either is sufficient on its own:
- **Admin bearer token** (the default, and always works): the identical
gate `GET /admin/api/*` uses — present a key belonging to a user
flagged `admin: true` as `Authorization: Bearer <key>` or
`x-api-key: <key>`. Unauthenticated → 401; authenticated non-admin →
403; admin → served. There is no second credential path — this reuses
`authStore.identify` exactly.
- **`metrics.allowedCIDRs`** (optional, off by default): CIDR blocks
whose **source address** may scrape with no key at all — for a
Prometheus/VictoriaMetrics server that should not need to carry a
credential. Checked against the request's source address exactly as
Traefik's own `net/http` server sees it (`RemoteAddr`, never a
client-supplied header like `X-Forwarded-For`) — **only meaningful for
a caller whose real source address actually survives to this
gateway**. Behind a Kubernetes `Service` or a second reverse proxy
that does not preserve the original address, every caller looks
identical to this check, and the allowlist becomes either "everyone"
or "no one" depending on what address arrives; confirm `RemoteAddr`
reflects the scraper's own address in your deployment before relying
on this instead of a key.
**On CIDR sizing — this is the mistake that actually breaks security,
live-reproduced, not a theoretical warning**: always scope
`allowedCIDRs` to the **narrowest block that covers only your actual
scrapers** — a cluster's Prometheus/VictoriaMetrics pod network, e.g.
`10.42.0.0/16`. **Never configure a whole RFC1918 range** such as
`10.0.0.0/8`, `172.16.0.0/12`, or `192.168.0.0/16`. The failure mode is
not "the allowlist does not work" — it is "the allowlist works exactly
as configured, and what survives to `RemoteAddr` is an address you never
intended to trust": a SNAT-ing router or load balancer makes
**external** traffic arrive from an address inside that same private
range. `allowedCIDRs: ["10.0.0.0/8"]` paired with a router's own SNAT
address has been reproduced returning full per-user cost data to an
external caller with no key at all.
- **`metrics.modelLabel`** (`false` by default) adds a `model` label to
the provider attempt/failure counters
(`llmgateway_provider_model_attempts_total`/
`llmgateway_provider_model_failures_total`), breaking them out per
`(provider, model)` pair in addition to the always-on per-provider
totals. Left off by default because a deployment with hundreds of
discovered model ids across several providers is a series-count
explosion waiting to happen — enable it only when your scraper's
retention and cardinality budget can absorb one series per
`(provider, model)` pair actually seen.
- **What's exposed** (families; see each one's own `# HELP` line in the
live output for the exact contract):
- `llmgateway_requests_total{scope_kind,scope_id}`,
`llmgateway_tokens_total{scope_kind,scope_id,direction}`,
`llmgateway_cost_micro_usd_total{scope_kind,scope_id}` — the
identical per-user/per-group/total accounting `GET /admin/api/usage`
already exposes as JSON, today's UTC-calendar-day window (resets at
UTC midnight).
- `llmgateway_budget_consumed_ratio{scope_kind,scope_id,budget}` —
fraction of a configured limit already consumed in its own window
(`0.0`–`1.0`+; above `1.0` means the limit has been breached), for
Prometheus's own alerting to threshold on. Emitted only for a scope
with that particular limit configured.
- `llmgateway_provider_attempts_total{provider}` /
`llmgateway_provider_failures_total{provider}` (+ opt-in
`_model_` variants, above) — the identical `attemptsDay`/`failuresDay`
counters `GET /admin/api/overview` already exposes.
- `llmgateway_provider_healthy{provider}` — `1`/`0` gauge from the
discovery circuit breaker (`healthState` in the admin API); see
[Provider health](#provider-health-discovery-circuit-breaker).
- `llmgateway_rate_limit_rejections_total{scope_kind,scope_id}` — how
many requests a rate or budget limit actually refused, by scope. This
one has no equivalent in the admin JSON API: nothing else tracks it.
- **Aggregation across replicas — read this before wiring an alert**: with
`redis` configured, **every replica reads the SAME shared counters**
(the underlying key carries no per-instance component), so
`llmgateway_requests_total`, `llmgateway_tokens_total`,
`llmgateway_cost_micro_usd_total`, `llmgateway_budget_consumed_ratio`,
and both `llmgateway_provider_*_total` families are the
**gateway-wide total, identical on every replica's own `/metrics`
response**. Prometheus scrapes each replica as its own target, so
`sum(rate(llmgateway_requests_total[5m]))` (or any other `sum()`) across
a 3-replica deployment **triple-counts** — a cost alert built on `sum()`
fires at a third of its intended threshold. **Aggregate these with
`max()`, never `sum()`.** Without `redis` (the in-process fallback), the
identical families become genuinely **per-replica** instead — each
replica counts only the traffic it personally handled — and `sum()`
becomes the correct aggregation instead. This flips on that one config
bit; know which one you are running before wiring an alert. Two
families do **not** follow this rule:
- `llmgateway_rate_limit_rejections_total` lives only in each replica's
own process memory, **never** in Redis, regardless of the `redis`
config — `sum()` is always correct for it.
- `llmgateway_provider_healthy` reflects each replica's own,
independently-running discovery circuit breaker — never shared via
Redis either — so two replicas can legitimately disagree about one
provider's health at the same instant. Read it per-instance where
possible; if you must aggregate, `min()` surfaces "at least one
replica sees this provider as unhealthy" — `sum()` is meaningless for
a 0/1 gauge.
- **Labels are escaped**: a label value that is user- or
upstream-controlled (a user/group name, a model id) is escaped for
backslash, double quote, and newline, byte-for-byte — the exact rule
the text-exposition format requires. This matters more than it looks:
Prometheus reports an unparseable scrape as the **whole target being
down**, not as one bad line, so a single unescaped quote in a model id
would otherwise take the entire endpoint offline from Prometheus's point
of view, not just that one series.
- **A store outage never fabricates a zero**: if the configured Redis is
unreachable and `redis.failOpen` is `false`, an affected scope's
request/token/cost/provider counters are **skipped entirely** for that
scrape rather than reported as `0` — a temporary gap in the series is
the honest signal; a fabricated `0` would read to Prometheus as a
counter reset followed by a phantom spike the instant the store
recovers. `llmgateway_provider_healthy` is unaffected either way — it
never reads from the store.
## MCP and A2A
- **Registry** (group-filtered, sorted by name):
examples/kubernetes.yaml
+53
View File
@@ -125,6 +125,23 @@ spec:
# ../README.md#admin.
admin:
enabled: false
# Prometheus text-exposition endpoint — see ../README.md#metrics.
# allowedCIDRs is the cluster's OWN pod network (a placeholder here;
# replace 10.42.0.0/16 with your cluster CIDR, e.g. `kubectl get
# nodes -o jsonpath='{.items[0].spec.podCIDR}'`), letting an
# in-cluster scraper (VMServiceScrape/ServiceMonitor below) hit
# /metrics with no credential. NEVER widen this to a whole RFC1918
# range like 10.0.0.0/8 — see ../README.md#metrics for why that
# specific mistake is live-reproducible, not theoretical: a
# SNAT-ing router or load balancer makes EXTERNAL traffic arrive
# from an address inside that same private range too. The
# dashboard-admin key above (users.inline) works here regardless,
# for a scraper that lives outside allowedCIDRs.
metrics:
enabled: true
allowedCIDRs:
- 10.42.0.0/16
modelLabel: false
pricing:
gpt-5-mini: {inputPerM: 0.25, outputPerM: 2.0}
mcpServers:
@@ -193,6 +210,42 @@ spec:
- name: llmgateway-dummy-backend
port: 80
---
# Scraping /metrics (../README.md#metrics): this cluster runs
# VictoriaMetrics, not Prometheus, so the CRD is VMServiceScrape
# (victoriametrics/operator), not a Prometheus-operator ServiceMonitor —
# the exposition FORMAT this plugin emits is identical either way (a
# ServiceMonitor with the same selector/port/path works unchanged if
# your cluster runs Prometheus instead); only the CRD kind differs, and
# that is entirely outside this plugin's own config.
#
# IMPORTANT: /metrics is served by TRAEFIK ITSELF, not by
# llmgateway-dummy-backend above — this middleware intercepts the path
# before any backend service is ever reached (same as every other route
# this plugin owns). Point this scrape at YOUR Traefik installation's own
# Service (commonly named "traefik" — verify against your own Helm
# release), never at a backend service this file defines. Adjust
# selector/namespaceSelector/port to match your Traefik Service's real
# labels and named port.
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
name: llmgateway-metrics
namespace: traefik
spec:
namespaceSelector:
matchNames:
- traefik
selector:
matchLabels:
app.kubernetes.io/name: traefik
endpoints:
- port: websecure # the Traefik Service's own named entryPoint port llmgateway is exposed on
path: /metrics # the default; matches metrics.path left unset above
scheme: https
tlsConfig:
insecureSkipVerify: true # adjust for your cluster's own CA/cert setup
---
# Helm values snippet (traefik/traefik-helm-chart) — illustrative, verify
# field names against the chart version you deploy. Two things are