# traefik-llmgateway
A [Traefik](https://traefik.io) middleware plugin that turns a Traefik
instance into a multi-provider LLM gateway: one endpoint, several upstream
providers, per-user API keys organised into groups, request/token/cost
limits, and a registry plus authenticated proxy for in-cluster MCP servers
and A2A agents.
The plugin is self-contained. It has no sidecar and no companion service.
It runs as a [Yaegi](https://github.com/traefik/yaegi)-interpreted Go
module, the same way [`traefikoidc`](https://github.com/lukaszraczylo/traefikoidc)
does, configured per-middleware in Traefik's dynamic configuration.
**Status:** feature-complete, all gates green (unit tests, `-race`,
`yaegi-check`, integration). Not yet published to the Traefik Plugin
Catalog — see [Development](#development) for what that needs.
## What it is
- A unified, OpenAI-compatible surface (`/v1/chat/completions`,
`/v1/embeddings`, `/v1/models`) that translates requests and responses to
and from Anthropic's and Google Gemini's own wire formats, so any
OpenAI-SDK-compatible client can talk to all three providers through one
API shape.
- Native passthrough per provider (`/{provider}/...`) for callers that want
a provider's own wire format untouched, with the gateway injecting that
provider's own API key and still enforcing auth and limits.
- Per-user API keys, organised into groups that control which providers,
models, MCP servers, and agents a user can reach, plus request/token/cost
limits enforced per user and per group.
- A registry and authenticated reverse proxy for MCP servers and A2A
agents, so those can sit behind the same gateway, the same auth, and the
same group-based visibility rules as the LLM traffic.
```mermaid
flowchart LR
Req["incoming request"] --> Route{"route
ServeHTTP: path+method dispatch"}
Route -->|"a recognized route:
/v1/models, /v1/chat/completions,
/v1/embeddings, /v1/mcp/servers,
/v1/agents, /mcp/name/...,
/a2a/name/..., or /providerName/...
for a known provider"| Auth["auth (per branch)
identify(): key digest lookup"]
Route -->|"no route matches"| Fallback["passthroughUnknown ? next : 404
never calls identify()"]
Auth -->|"no match"| E401["401 authentication_error"]
Auth -->|"user, group"| Branch{"branch"}
Branch --> Models["GET /v1/models"]
Branch --> Unified["POST /v1/chat/completions
POST /v1/embeddings"]
Branch --> Registry["GET /v1/mcp/servers, /v1/agents"]
Branch --> MCP["MCP/A2A proxy"]
Branch --> Passthrough["native passthrough"]
Unified --> Resolve["resolve model
registry.resolve"]
Resolve -->|"unknown/denied"| E404["404 / 403"]
Resolve --> Limit1["limiter.checkAndCount"]
MCP --> Limit2["limiter.checkAndCount"]
Passthrough --> Limit3["limiter.checkAndCount"]
Limit1 -->|"over budget"| E429["429 / 503"]
Limit2 -->|"over budget"| E429
Limit3 -->|"over budget"| E429
Limit1 --> Exec["execute: translate + call adapter"]
Limit2 --> ExecProxy["execute: reverse proxy"]
Limit3 --> ExecProxy
Exec --> Account["account: usage, cost"]
Account --> Resp["respond to client"]
ExecProxy --> Resp
Models --> Resp
Registry --> Resp
```
Every route the plugin recognises is handled entirely inside the
middleware — `next` (the router's configured backing service) is only
reached when a request matches none of the plugin's routes and
`passthroughUnknown` is `true`.
## Quickstart
### Static configuration
The plugin is not yet in the Traefik Plugin Catalog (see
[Development](#development)), so the catalog form of static configuration
(`experimental.plugins.llmgateway`) is not usable yet. Until then, load it
as a [`localPlugin`](https://doc.traefik.io/traefik/plugins/#defining-a-local-plugin),
mounting this repository's source into the Traefik container at
`/src/github.com/lukaszraczylo/traefik-llmgateway` — see
[`examples/docker-compose.yml`](examples/docker-compose.yml) for a
working, tested setup that does exactly this:
```yaml
# traefik static config (traefik.yml / --experimental.localPlugins.*)
experimental:
localPlugins:
llmgateway:
moduleName: github.com/lukaszraczylo/traefik-llmgateway
```
Once the plugin is published, the catalog form replaces this:
```yaml
experimental:
plugins:
llmgateway:
moduleName: github.com/lukaszraczylo/traefik-llmgateway
version: "v0.1.0" # substitute the tag you actually deploy
```
### Dynamic configuration
A minimal single-provider middleware, one group, one user:
```yaml
http:
middlewares:
llmgateway:
plugin:
llmgateway:
providers:
openai:
type: openai
apiKey: "env:OPENAI_API_KEY"
models: ["gpt-5-mini", "gpt-4o"]
groups:
default:
providers: [] # empty = every configured provider is allowed
users:
inline:
- name: alice
group: default
apiKey: "env:LLMGW_KEY_ALICE"
```
> Write an "allow everything" group as `providers: []` (an explicit empty
> list), not a bare `{}`. See the note at the top of
> [`examples/kubernetes.yaml`](examples/kubernetes.yaml) — confirmed
> against Traefik's **file provider**, twice, with this exact plugin build:
> a bare `default: {}` fails plugin construction with
> `expected a map, got 'string'`; `default: {providers: []}` loads cleanly.
> Not tested against the Docker/ECS label providers or the Kubernetes CRD
> provider — no claim either way for those. An explicit empty list means
> the same thing as an empty object (all values allowed), so writing it
> this way costs nothing even where the bug does not apply.
Attach the middleware to a router the same way as any other Traefik
middleware. The router's backing service is never actually reached unless
`passthroughUnknown: true` and the request matches none of the plugin's
routes — a small dummy service (`traefik/whoami`, or any always-up
internal service) is enough.
Call it:
```sh
curl http://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_KEY_ALICE" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}]}'
```
## Full configuration reference
Every limit or visibility list follows one convention throughout: **empty
or omitted means unlimited / all**. Field names below are the exact JSON
keys the plugin's Go structs expect (`llmgateway.go`); Traefik's dynamic
config accepts them as YAML, which decodes to the same JSON shape.
### `Config` (top level)
| Field | Type | Default | Semantics |
|---|---|---|---|
| `providers` | `map[string]ProviderConfig` | — | Required, at least one entry, or the plugin refuses to construct. |
| `groups` | `map[string]GroupConfig` | `{}` | Optional. A user references one group by name; an unknown reference is a construction error. |
| `pricing` | `map[string]ModelPricing` | `{}` | Per-model USD/1M-token overrides, keyed by model id (bare or `provider/model`). Merged over, not replacing, the built-in table (`pricing.go`). |
| `mcpServers` | `map[string]TargetConfig` | `{}` | MCP server registry; see [MCP/A2A](#mcp-and-a2a). |
| `agents` | `map[string]AgentConfig` | `{}` | A2A agent registry; see [MCP/A2A](#mcp-and-a2a). |
| `users` | `UsersConfig` | — | Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401. |
| `redis` | `RedisConfig` | — | Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances). |
| `retry` | `RetryConfig` | `{}` (disabled) | Same-provider retry for transient upstream failures — see [Retry](#retry). Omitted or `enabled: false` means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed. |
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
> **Upgrading from v0.1**: `POST /v1/images/generations`, `POST
> /v1/audio/speech`, and `POST /v1/audio/transcriptions` are now handled by
> the gateway itself, unconditionally — see [Image and audio
> endpoints](#image-and-audio-endpoints). They no longer fall through to
> `next` even when `passthroughUnknown: true`. A v0.1 deployment that
> served its own backend on any of these three exact paths, relying on
> passthrough to reach it, must move that backend to a different path
> before upgrading to v0.2 — the gateway now intercepts these paths first,
> every time.
Provider, `mcpServers`, and `agents` map **keys** (names) must match
`^[a-zA-Z0-9._-]+$` and must not be `v1`, `mcp`, or `a2a` — those names are
reserved because they would shadow the gateway's own fixed routes
(`/v1/...`, `/mcp/{name}/...`, `/a2a/{name}/...`); a name that collides is
a construction error, not a silently-unreachable provider. A `nil` map
*value* under any of `providers`, `groups`, `mcpServers`, or `agents` is
also a construction error, never a panic (`providers.go`, `mcp_a2a.go`,
`auth.go`).
### `ProviderConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `type` | `string` | — | Required: `openai`, `anthropic`, or `gemini`. `openai` also serves any OpenAI-wire-compatible provider (Grok, DeepSeek, a local vLLM/Ollama gateway, ...) via `baseUrl`. |
| `baseUrl` | `string` | Per type: `https://api.openai.com`, `https://api.anthropic.com`, `https://generativelanguage.googleapis.com` | Trailing slash trimmed. |
| `apiKey` | `string` | — | Secret form: literal, `env:VAR`, or `file:/path` — see [Secret forms](#secret-forms). An `openai`-type provider resolving to an empty key is a valid **keyless** upstream (no `Authorization` header sent) — useful for a local model server with no auth. `anthropic` and `gemini` reject an empty resolved key at construction. |
| `discoveryInterval` | `string` (Go duration) | `1h` | Only meaningful when `discovery: true`. Invalid duration string is a construction error. |
| `models` | `[]string` | `[]` | Explicit model ids this provider serves. Combined with any discovered ids. |
| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. |
### `GroupConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `limits` | `*LimitsConfig` | `nil` (unlimited) | |
| `providers` | `[]string` | `[]` (all) | Glob-matched (`path.Match`) against a provider's configured name. |
| `models` | `[]string` | `[]` (all) | Glob-matched, exactly as given — no automatic `provider/` prefix stripping. Both the bare and provider-prefixed forms of a model id are checked, so a pattern like `gpt-*` matches a client request for either `gpt-5-mini` or `openai/gpt-5-mini`. |
| `cache` | `*bool` | `nil` (inherit) | Overrides the top-level `cache.enabled` setting for this group's requests — see [Caching](#caching). `nil` inherits the global setting; `false` opts the group out even when caching is globally on; `true` opts the group in, but only when the top-level `cache` block is actually configured (a construction error otherwise — there is nothing to inherit `ttl`/`maxBodyBytes` from). |
| `mcpServers` | `[]string` | `[]` (all) | Glob-matched against a configured MCP server name. |
| `agents` | `[]string` | `[]` (all) | Glob-matched against a configured agent name. |
### `LimitsConfig`
All six fields default to `0` (unlimited). A negative count, or a cost
that is negative, `NaN`, or `±Inf`, is a construction error.
| Field | Type | Window |
|---|---|---|
| `requestsPerMinute` | `int64` | Fixed 1-minute window, UTC. |
| `requestsPerDay` | `int64` | Fixed calendar-day window, UTC midnight to midnight. |
| `tokensPerDay` | `int64` | Fixed calendar-day window, UTC. |
| `tokensPerMonth` | `int64` | Fixed calendar-month window, UTC. |
| `costPerDayUSD` | `float64` | Fixed calendar-day window, UTC. |
| `costPerMonthUSD` | `float64` | Fixed calendar-month window, UTC. |
A user's own `limits`, when set, are checked **in addition to** their
group's limits, not instead of them — a request is refused the moment it
breaches whichever of the two is tighter. A user with no `limits` of their
own is governed purely by their group's — see
[UserConfig](#userconfig).
### `UsersConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `file` | `string` | — | Path to a JSON users file, merged over `inline` and hot-reloaded — see [Users file](#users-file-format). |
| `inline` | `[]UserConfig` | `[]` | Resolved once at plugin construction. |
### `UserConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `name` | `string` | — | Required. |
| `group` | `string` | — | Required; must reference a configured group, or construction fails. |
| `apiKey` | `string` | — | Secret form (see [Secret forms](#secret-forms)); must resolve non-empty and unique across every inline **and** file-sourced user. |
| `limits` | `*LimitsConfig` | `nil` (governed by group's limits alone) | When set, enforced alongside — not instead of — the group's own limits; both scopes are checked, and the request is refused by whichever is breached first. |
| `admin` | `bool` | `false` | Grants access to the read-only admin dashboard — see [Admin](#admin). Works the same for an inline or a file-sourced user. An admin user is otherwise ordinary: their own keys, group, and limits still apply, including to the admin routes themselves. |
### `RedisConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `address` | `string` | — | Required when the `redis` block is present at all. `host:port`, RESP protocol — works against Redis, [Valkey](https://valkey.io), or [Dragonfly](https://www.dragonflydb.io). |
| `password` | `string` | `""` (no `AUTH`) | Secret form. |
| `db` | `int` | `0` | Must be `≥ 0`. Sent via `SELECT` on every new connection. |
| `failOpen` | `*bool` | `true` | `true`: a Redis error switches that operation to the in-process fallback store and logs (rate-limited to once per 30s). `false`: a Redis error returns 503 instead of enforcing against a non-shared fallback. |
### `RetryConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `enabled` | `bool` | `false` | `false`: no retry, one upstream attempt per request. `true`: retry per [Retry](#retry) below. |
| `attempts` | `int` | `1` | Retries performed **after** the first try, not the total try count. `1` (default) allows one retry — two tries total. The maximum, `3`, allows three retries — four tries total. A value outside `1`-`3` is a construction error. |
| `backoff` | `string` (Go duration) | `250ms` | Base wait before the first retry; doubles on each further retry, capped at 2s per wait. Invalid duration string is a construction error. |
### `CacheConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `enabled` | `bool` | `false` | `false`: no caching, `/v1/chat/completions` and `/v1/embeddings` behave byte-identically to a gateway built before this field existed. `true`: caching per [Caching](#caching) below — requires `redis` to be configured too, or it silently stays off (one warning logged). |
| `ttl` | `string` (Go duration) | `5m` | How long a cached response stays valid. Invalid duration string is a construction error. |
| `maxBodyBytes` | `int` | `1048576` (1MiB) | Largest response body still eligible for caching; a larger one is skipped (never stored, never an error). Maximum accepted value is `8388608` (8MiB) — above that is a construction error. |
### `AdminConfig`
| Field | Type | Default | Semantics |
|---|---|---|---|
| `enabled` | `bool` | `false` | `false`: the `/admin*` routes are not registered at all — a request to any of them falls through to the plugin's existing 404/`passthroughUnknown` handling like any other unrecognized path. `true`: the read-only dashboard per [Admin](#admin) below. |
### `ModelPricing`
| Field | Type | Semantics |
|---|---|---|
| `inputPerM` | `float64` | USD per 1,000,000 input/prompt tokens. |
| `outputPerM` | `float64` | USD per 1,000,000 output/completion tokens. |
### `TargetConfig` (an `mcpServers` entry)
| Field | Type | Semantics |
|---|---|---|
| `url` | `string` | Required; must parse with an `http` or `https` scheme, checked at construction. |
### `AgentConfig` (an `agents` entry)
| Field | Type | Default | Semantics |
|---|---|---|---|
| `url` | `string` | — | Required; same URL validation as `TargetConfig`. |
| `card` | `string` | `/.well-known/agent-card.json` | Path appended to the agent's gateway-relative proxy URL for its A2A agent-card listing. |
### Secret forms
Every `apiKey`/`password` field accepts one of three forms
(`secrets.go`):
- **Literal** — the value itself, used as-is. Fine for local/dev; avoid for
anything real.
- **`env:VAR`** — read from environment variable `VAR` on the Traefik
process. Errors if unset or empty.
- **`file:/path`** — read from a file, trimmed of surrounding whitespace.
Errors if unreadable or the trimmed content is empty.
Provider keys, Redis's password, and inline users' keys are resolved
**once**, at plugin construction. A file-sourced user's `apiKey` is
re-resolved on every hot reload (see below), so it can use `env:`/`file:`
forms too and pick up a rotated value without a Traefik restart.
### Users file format
One JSON object, one `users` array, mounted from a single Kubernetes
Secret (`users_file.go`):
```json
{
"users": [
{"name": "alice", "group": "engineering", "apiKey": "sk-llmgw-..."},
{"name": "bob", "group": "general", "apiKey": "env:LLMGW_KEY_BOB", "limits": {"requestsPerDay": 100}}
]
}
```
A file user with the same `name` as an inline user overrides it (the
inline entry is dropped from the merged set). Hot reload polls the file's
mtime, throttled to once per 5 seconds on the request path, and is
best-effort: a reload already in progress is skipped rather than
serialising every concurrent request behind it, so a change lands within a
few request-driven checks, not necessarily on the very next one. A
reload's own error (unreadable file, malformed JSON, a user referencing an
unknown group, a duplicate key) is logged and the last good user set is
kept — a bad edit to the file never breaks already-authenticated traffic.
## Model routing
- A model id in **`provider/model`** form resolves directly against that
provider, but only when the part before the slash names an actually
configured provider. Any other id — including one that merely contains a
slash, such as an upstream's own `uni/deepseek-v4-flash-0731` naming — is
treated as a single **bare** id.
- A bare id resolves against the first configured provider, in sorted
provider-name order, whose known model set (explicit `models` plus any
discovered ids) contains it. When more than one provider knows the same
bare id, the sorted-first provider wins it and the gateway logs a
collision warning once per id; every provider's copy stays reachable
through the explicit `provider/model` form regardless of who won the
bare id, and `GET /v1/models` lists both the bare (winner's) form and
every `provider/model` form.
- **On the unified routes** (`modelRegistry.resolve`, used by
`/v1/chat/completions`, `/v1/embeddings`, and the three media
endpoints), authorization requires both `group.allowsModel` and
`group.allowsProvider` for the resolved provider — a model that exists
but the caller's group cannot reach returns 403, distinct from a model no
configured provider knows at all (404). Native passthrough
(`/{provider}/...`) resolves no model at all and checks
`group.allowsProvider` only — see the authorization row in [Unified vs.
passthrough](#unified-vs-passthrough).
- **Discovery**: `GET {baseUrl}/v1/models` for an `openai`-type provider,
`GET {baseUrl}/v1/models` for `anthropic`, `GET {baseUrl}/v1beta/models`
for `gemini`. Runs once at plugin construction (bounded to 5s per
provider) and again on `discoveryInterval` in the background. A failed
refresh is logged and keeps the provider's last-known discovered set —
**stale-while-error**, never an empty list just because one refresh
attempt failed.
## Model aliases
Operator-defined indirection: a client-facing alias id that resolves to a
real target model id. `modelAliases` is empty by default, and an empty map
changes nothing about model resolution — this is purely additive.
```yaml
http:
middlewares:
llmgateway:
plugin:
llmgateway:
# ... providers, groups, users unchanged ...
modelAliases:
# Client-facing name -> real target. The target can itself be
# bare ("gpt-5-mini") or provider-prefixed
# ("anthropic/claude-sonnet-4-5"); either resolves through the
# normal rules in [Model routing](#model-routing) above.
aliased/coding: "anthropic/claude-sonnet-4-5"
aliased/fast: "gpt-5-mini"
```
- **Swapping the underlying model**: point an alias at a different target
and let Traefik's dynamic-config reload apply it — no gateway restart,
no client-side change. A client that has always addressed
`aliased/coding` keeps working unmodified after the operator moves that
alias to a newer model.
- **Precedence**: an EXACT alias match wins resolution before any other
rule, including `provider/model` prefix splitting — checked first in
`modelRegistry.resolve`. The target then resolves through the same
rules as any other id (bare, `provider/model`, or a discovered id).
Because of this, an alias whose id contains a `/` is rejected at
construction if the part before the `/` names an actually configured
provider (it would be shadowed by prefix resolution) — see the
validation rules below.
- **Authorization**: a group may use an alias when its `models` glob
matches the **alias name itself**, OR when it already grants access to
the **resolved target** — either is enough. `providers` still gates the
target's own provider as usual.
- **Validation at construction** (a bad alias config fails the same way a
bad provider config does — no gateway starts with a broken alias
table):
- the alias must be non-empty and contain no control characters;
- when the alias contains `/`, its prefix must not name a configured
provider (would be shadowed by prefix resolution);
- the alias must not equal an explicit `models` entry of any provider
(ambiguous — which one wins?);
- the target must not itself be another alias (no alias chains);
- the target must not be empty.
A target naming a model that is not yet known (awaiting a provider's
first discovery fetch, or simply never configured) is accepted at
construction — it resolves lazily. An unresolvable target at request
time returns 404 with a message naming both the alias and the missing
target. A target that is later discovered and collides with an
already-configured alias id is not a construction error either — the
alias always wins that resolution, and the gateway logs one warning
per such id.
- **`GET /v1/models` listing**: every alias the caller's group may use is
listed as a model object alongside real models — `id` is the alias,
`owned_by` is the target's resolved provider name. An alias whose
target does not resolve yet is omitted from the listing until it does.
- **Echo behavior**: exactly like the existing provider-prefix alias echo
(`__alias`, [Unified vs. passthrough](#unified-vs-passthrough)) —
`anthropic`- and `gemini`-type targets build their own response
envelope and echo the client's exact alias string back in its `model`
field. An `openai`-type target's response is a **verbatim passthrough**
of the upstream body, so its `model` field carries whatever the
upstream itself returned (typically the bare upstream model id) —
**not** the alias — the same documented v0.1 passthrough asymmetry,
unchanged by this feature.
- **Applies everywhere a model routes**: unified chat and embeddings,
`/v1/images/generations`, `/v1/audio/speech`, and
`/v1/audio/transcriptions` (including its multipart rewrite — the
upstream `model` field receives the resolved target's own upstream id,
never the alias, exactly like a provider-prefixed id already does).
- **Admin overview**: `GET /admin/api/overview` includes an `aliases`
array (`{alias, target}`, sorted by alias) — see [Admin](#admin). No
secrets involved.
## Limits and accounting
- Counters use **fixed windows**, keyed by UTC calendar boundaries (the
minute, the day, the month) — not a sliding window. Requests-per-minute
and requests-per-day counters increment **before** evaluation, so every
request counts toward its scopes' rate tracking even if it is ultimately
refused by a different scope's limit.
- Token and cost budgets are checked against the value **already
accumulated** from prior requests, before this request's own usage is
known (usage only arrives after the upstream call completes). A request
that would start already over budget is refused; a request that starts
under budget but pushes the scope over it is still allowed to complete —
an accepted at-most-one-request overshoot, not an exactly-enforced cap.
- **Accounting** parses usage from the provider's own response: OpenAI-type
`usage`, Anthropic `usage` (from `message_start`/`message_delta`), Gemini
`usageMetadata`. When a **non-streaming** unified-route response reports
no usage at all, the gateway falls back to a rough estimate —
`ceil(request-body-bytes / 4)` counted as prompt tokens, no completion
estimate — and marks it `estimated`. A **streaming** unified-route
response with no usage is logged and the request itself is still
counted, but no token estimate is applied.
- Cost is `tokens × price`, computed in integer **micro-USD**
(1,000,000ths of a dollar) throughout to avoid float drift, from the
built-in per-model table (`pricing.go`) or a configured `pricing`
override. An unpriced model costs 0 and logs a once-per-model warning
(capped at 128 distinct unpriced model names per process lifetime, then
one summary warning). The built-in table is **approximate** — list
prices as each provider published them, recorded 2026-08, and not kept
in sync automatically. Set `pricing` overrides for billing-grade
accuracy.
- **Storage**: `redis` configured wires a hand-rolled, stdlib-only RESP2
client (`resp.go`) — no `go-redis`, which is not Yaegi-interpretable —
against Redis, Valkey, or Dragonfly. Semantics are deliberately
**at-least-once, never under-counting**: a lost reply after the server
already applied an `INCRBY` can cause a resend that over-counts by one
request's worth on that key. This is a fail-*safe* trade-off (more
restrictive than reality), never a fail-*open* one. Without `redis`
configured, or when it errors and `failOpen: true`, counters live in an
in-process map — correct for one Traefik replica, only approximate
across several, since each replica counts independently. Exact limits
across multiple Traefik replicas require the shared Redis.
## Retry
Same-provider retry for a transient upstream failure — no cross-provider
failover. Off by default (`retry.enabled: false`); every config that
predates this field keeps making exactly one upstream attempt per
request.
- **Scope**: every adapter upstream call — chat, embeddings, model
listing, and the three media endpoints (`images/generations`,
`audio/speech`, `audio/transcriptions`) — across all three provider
types. Native passthrough (`/{provider}/...`) and the MCP/A2A proxy are
raw reverse proxies and are never retried; the client owns retry
semantics there.
- **Attempts is retries, not tries**: `attempts` counts retries
performed **after** the first try. The default, `1`, allows one retry
— two tries total, not one. The maximum, `3`, allows three retries —
four tries total.
- **Transient classification**: a connection error (dial/EOF/reset), HTTP
429, or HTTP 5xx. Never a non-429 4xx, a translation error, a request
that failed to build in the first place (a malformed method or URL
fails identically on every attempt), or a context
cancellation/deadline.
- **Zero-bytes rule**: retry only runs before any response byte reaches
the client. A non-streaming call may retry its whole exchange. A
streaming call retries only the initial request, up to the point the
upstream's headers and status arrive — once the first chunk is
forwarded, a stream that then dies mid-flight is not retried, since the
client already holds partial data.
- **Waiting**: a 429 response carrying a `Retry-After` header that parses
as a non-negative whole-second count no greater than 2s waits exactly
that long. Every other case — no header, an unparseable value (for
example the HTTP-date form), a negative value, a value over 2s, or a
`Retry-After` on a non-429 5xx — falls back to exponential backoff
(`backoff × 2^(attempt-1)`), capped at 2s per wait either way. A
context canceled or timed out during a wait aborts the remaining
retries immediately, rather than blocking out the wait in full.
- **Accounting**: only the final attempt's usage is recorded — a failed
attempt's body is drained and discarded, never parsed for usage, so it
is never double-counted. Request counters still increment once per
client request, unchanged.
## Caching
An opt-in, Redis-backed response cache for the unified, non-streaming
`POST /v1/chat/completions` and `POST /v1/embeddings` routes. Off by
default (`cache.enabled: false`); every config that predates this field
keeps calling the upstream on every request. Caching also requires
`redis` to be configured — without it, `cache.enabled: true` is silently
disabled with one warning logged at construction, never a per-replica
in-process fallback: divergent replicas would otherwise serve different
cached bodies for the same request.
- **What's cacheable**: a non-streaming request to either route, whose
upstream response was HTTP 200 and no larger than `cache.maxBodyBytes`.
Never cached: a streaming request, native passthrough, MCP/A2A, the
image/audio endpoints (see [Image and audio endpoints](#image-and-audio-endpoints)),
or any non-200 upstream response.
- **Key**: `llmgw:cache:` plus a hex SHA-256 digest of the provider name,
the resolved upstream model id, the client's own exactly-as-sent
`model` string (`requestedModel`), the endpoint (`chat` or
`embeddings`), and the canonical JSON of the request body with
`stream_options` and `user` removed. `requestedModel` and the endpoint
are both part of the key so that two different alias forms of the same
upstream model (`claude-x` versus `anthropic/claude-x`) never collide
into one cache entry: a translating adapter (`anthropic`, `gemini`)
bakes the client's own alias into the cached response body's `model`
field, so a second client using the other alias form would otherwise
receive an id it never sent.
- **User identity is not part of the key.** Identical requests share one
cache entry across every user and group that can reach the model — by
design, since a provider's response to an identical request is
provider-deterministic and carries no per-user data.
- **Hit/miss header**: every cacheable request's response carries
`X-Llmgw-Cache: hit` or `X-Llmgw-Cache: miss`, set before the body is
written either way.
- **Accounting**: a cache hit increments request counters exactly like
any other request — `limiter.checkAndCount` runs before the cache
lookup — but never touches token or cost counters, and never runs the
streaming-response-with-no-usage estimate. Only a genuine upstream call
(a miss) accounts real tokens and cost. This means a cache hit is
request-quota-counted but token/cost-free.
- **Group override**: `GroupConfig.cache` (a `*bool`) overrides the
top-level `cache.enabled` setting per group — `nil` inherits it, `false`
opts the group out even when caching is globally on, `true` opts the
group in. Setting `true` on a group when the top-level `cache` is not
enabled — the block omitted entirely, or present with `enabled: false`,
both read the same way — is a construction error: there is nothing
configured to inherit `ttl`/`maxBodyBytes` from.
- **Storage**: values are stored via the same hand-rolled RESP2 client
(`resp.go`) the limiter's distributed counters use — the identical
Redis/Valkey/Dragonfly connection, not a second one. A Redis error
during a cache read or write is treated as a miss (logged,
rate-limited) and never fails the request; the upstream call still
happens and the client still gets a correct answer.
## Unified vs. passthrough
| | Unified (`/v1/...`) | Native passthrough (`/{provider}/...`) |
|---|---|---|
| Wire format | Always OpenAI-shaped in, OpenAI-shaped out. | The provider's own native format, untouched. |
| Authorization | `group.allowsModel` AND `group.allowsProvider`, both checked against the resolved provider (`modelRegistry.resolve` — see [Model routing](#model-routing)). | `group.allowsProvider` only. Passthrough resolves no model at all, so there is nothing for `allowsModel` to gate. |
| Translation | `openai`-type: body forwarded verbatim (model id rewritten). `anthropic`/`gemini`: full bidirectional translation, including streaming, chunk by chunk. | None — a raw reverse proxy. |
| Model alias in the response | Translated providers (`anthropic`, `gemini`) echo back the client's exact requested model string in the response's `model` field, even though the upstream call used the resolved provider model id. An `openai`-type response is a verbatim passthrough of the upstream body, so it carries whatever model id the upstream itself returned — this asymmetry is intentional, not a bug. | The upstream's own `model` field, verbatim — there is no alias to echo. |
| Embeddings | `openai`-type: passthrough. `gemini`: mapped to `:embedContent`/`:batchEmbedContents`. `anthropic`: **501** — Anthropic's API has no embeddings endpoint. | Whatever the provider itself supports at that path; the gateway does not gate it. |
| Unsupported parameters | A semantically meaningful field the target provider cannot express (e.g. `n > 1`, `logit_bias`, `logprobs` on Anthropic) is a **400**, never silently dropped. | Not applicable — the request reaches the provider exactly as sent. |
| Non-2xx upstream response | Wrapped in the gateway's own error envelope, with the provider's own body embedded under `error.upstream`. | Forwarded to the client exactly as the upstream sent it — status, headers, and body. |
## Image and audio endpoints
`POST /v1/images/generations`, `POST /v1/audio/speech`, and
`POST /v1/audio/transcriptions` follow the same pipeline as
`/v1/chat/completions` — auth, model routing, group authorization, limits,
adapter call — minus response caching and token/cost accounting (below).
> **Upgrading from v0.1**: these three routes are now handled by the
> gateway unconditionally. They are matched and served before
> `passthroughUnknown` is ever consulted, so a request to any of them
> **never** falls through to `next` — not even with `passthroughUnknown:
> true`. If a v0.1 deployment pointed its router's backing service at one
> of these exact paths and relied on passthrough to reach it, that backend
> must move to a different path before upgrading.
| | `openai`-type | `gemini` | `anthropic` |
|---|---|---|---|
| `images/generations` | Native forward. | Translated to Imagen's `:predict` (below). | **501**. |
| `audio/speech` | Native forward; the binary response is streamed to the client as it arrives. | **501** — no OpenAI-compatible text-to-speech endpoint. | **501**. |
| `audio/transcriptions` | Native forward; see below for how the multipart body is forwarded. | **501** — no OpenAI-compatible speech-to-text endpoint. | **501**. |
- **Model routing**: `images/generations` and `audio/speech` read `model`
from the JSON request body. `audio/transcriptions` reads it from the
multipart request's `model` form field instead — the field can appear
in any position among the request's parts, and a body with more than
one `model` field is a **400**.
- **Multipart forwarding**: when the client's `model` field already
equals the resolved upstream model id (a bare id, the common case), the
original multipart body is replayed byte-for-byte, unchanged. When it
does not — a provider-prefixed id (`openai/whisper-1`) or another alias
resolves to a different upstream model string — the body is rebuilt
with every other part copied verbatim and only the `model` part's value
rewritten, so the alias never reaches the real provider. The rebuilt
body uses a new multipart boundary; the client's exact boundary is not
preserved, only an equivalent body.
- **Gemini image translation**: `{model, prompt, n, size,
response_format}` maps to Imagen's `{instances:[{prompt}], parameters:
{sampleCount, aspectRatio}}`. `size` maps to `aspectRatio`:
`1024x1024`/`512x512` → `1:1`, `1792x1024` → `16:9`, `1024x1792` →
`9:16`; an absent `size` defaults to `1:1`; any other value is a
**400**. `response_format: "url"` is a **400** — the gateway stores
nothing to host a URL from. A truthy `quality` or `style` is a **400**,
the same convention chat's translators use for an unsupported field.
Imagen's `predictions[].bytesBase64Encoded` maps back to OpenAI's
`data[].b64_json`.
- **Never cached**: image and audio responses are excluded from the
response cache (`cache.enabled`) regardless of its configuration.
- **Accounting**: request counters (`requestsPerMinute`/`requestsPerDay`)
increment the same as any other route. Token counters never move, and
cost is always recorded as 0 — images and audio are not
cost-accounted.
- **Body limits**: `images/generations` and `audio/speech` share the
unified routes' 10MiB JSON cap. `audio/transcriptions` enforces the
same 10MiB cap explicitly, returning **413** over it — unlike an
oversized JSON body, which is silently truncated into a parse failure,
a truncated multipart body can still parse a valid prefix and would
forward cut-off audio upstream if it were not rejected outright.
- **Retry** (above) applies to all three the same as chat/embeddings; a
streaming `audio/speech` response is retried only before the first
response byte reaches the client, the same zero-bytes rule as chat's
streaming path.
## Admin
A read-only dashboard and JSON API for operational visibility — providers,
groups, and per-user/per-group usage against configured limits. No
mutation of any kind; config stays owned by GitOps/Traefik as usual. Off
by default (`admin.enabled: false`, or the `admin` block omitted
entirely): the three routes below are not registered at all, and a
request to any of them falls through to the plugin's existing
404/`passthroughUnknown` handling.
- **Enabling it**: set `admin.enabled: true`, then flag at least one user
`admin: true` (`UserConfig.admin`) so someone can actually reach
`/admin/api/*`. An admin user is otherwise ordinary — their own keys,
group, and limits still apply, including to the admin routes
themselves.
- **`GET /admin`** serves the dashboard's HTML/CSS/JS shell. This route is
deliberately served **with no authentication at all** once
`admin.enabled` is true (disabled still falls through to the same
404/`passthroughUnknown` handling described above): a browser navigating
straight to the URL has no way to attach a custom `Authorization` or
`x-api-key` header, so gating the page itself would make it unreachable
from a browser in the first place. This is safe because the shell
carries **zero data of its own** — every value is fetched client-side
from the two JSON routes below, which stay fully gated.
- **Browser key-entry flow**: the page's script reads an admin API key
from the current tab's `sessionStorage`. Absent a stored key, or on any
`401`/`403` from either JSON route, it shows an inline key-entry form
instead of the dashboard. A submitted key is kept **only in
`sessionStorage`** — never a cookie, `localStorage`, or any persistent
store — so it disappears when the tab closes, and it is sent only as
the `x-api-key` header on this page's own `/admin/api/*` fetches.
- **`GET /admin/api/overview`** and **`GET /admin/api/usage`** share one
gate: unauthenticated → 401, authenticated non-admin → 403, admin →
serve. `overview` returns the provider list, groups, the configured
model alias table (`alias`/`target` pairs, sorted by alias — see
[Model aliases](#model-aliases)), redis/cache/retry status (`retry`'s
`enabled`/`attempts`/`backoff` are the EFFECTIVE values after
[Retry](#retry)'s own defaulting, not the raw config — `attempts` and
`backoff` are both omitted when retry is disabled), and the plugin
version string. `usage` returns every user's and every group's
current-window counter values (req/min, req/day, tok/day, tok/month,
cost/day, cost/month) alongside their configured limits.
- **What's exposed**: provider names, types, base URLs (with any
userinfo/query string stripped before it's ever echoed), model counts,
discovery status, group/user names, membership, limits, and live usage
counters. **Never exposed**: API keys (not even digests), provider
keys, the Redis password, or users-file path contents — every
secret-bearing field is redacted from every response.
- **Poll counts against quota**: the dashboard's own JavaScript polls
both JSON routes every 5 seconds while open. Each poll is a real
authenticated request and increments the admin user's own request
counters exactly like any other route — leaving a dashboard tab open
against a tightly limited admin user can itself exhaust their
`requestsPerMinute`/`requestsPerDay` budget.
- **`GET /admin` counts nothing**: it is unauthenticated, so there is no
identified user to count a request against.
- **Response headers**: both JSON routes set
`X-Content-Type-Options: nosniff` and `Cache-Control: no-store`, and all
three routes share one `Content-Security-Policy` header
(`default-src 'none'; script-src 'unsafe-inline'; style-src
'unsafe-inline'; connect-src 'self'`) — the dashboard loads no external
asset of any kind and works in an air-gapped cluster.
## MCP and A2A
- **Registry** (group-filtered, sorted by name):
`GET /v1/mcp/servers` → `{"servers":[{"name","url"}]}`;
`GET /v1/agents` → `{"agents":[{"name","url","card"}]}`.
- **Proxy**: `/mcp/{name}/{rest...}` and `/a2a/{name}/{rest...}`
reverse-proxy `{rest...}` to the configured target's `url`, streaming
(SSE) preserved via a flush-per-write proxy. The gateway's own
`Authorization`/`x-api-key` credential is stripped before proxying, and
— unlike native provider passthrough — **no credential is injected** on
the way out: an MCP server or A2A agent is treated as an in-cluster
target that trusts the gateway's network position, not a per-provider
key the gateway holds on the caller's behalf.
- Request-rate limits (`requestsPerMinute`/`requestsPerDay`) apply to
target-proxy calls the same as everywhere else; token/cost metrics do
not, since there is no usage to parse from an arbitrary MCP/A2A
response.
- Group visibility (`mcpServers`/`agents` glob lists) gates both the
registry listing and the proxy path itself — a denied name is a 403 at
the proxy, not just hidden from the listing.
## Security notes
- API keys are never retained in memory as plaintext after construction —
only their SHA-256 digests. Lookup is by digest, verified with
`crypto/subtle.ConstantTimeCompare` against the stored digest, so
matching does not leak timing information about a near-miss key.
- Provider API keys are injected server-side (native passthrough's
`injectAuth`) and never reach the client; a client's own gateway key is
stripped before any upstream or target request and never reaches a
provider, MCP server, or A2A agent.
- Neither kind of key is ever written to a log line — the logging helpers
(`logger.go`) carry an explicit "never log key material" contract.
`providerHTTPError`'s `Error()` string (`providers.go`) separately omits
the upstream response body it still holds as a struct field, so a future
caller that ever formats the error bare (`%v`) cannot accidentally echo
provider response content into a log line; today's only caller
(`writeProviderUpstreamError`) reads that body directly and puts it in
the client-facing response, never in a log.
- **Every authentication attempt is logged**, through one shared helper
(`logAuthEvent`, `logger.go`) called at each of the six routes that
authenticate a request (`llmgateway.go`'s `ServeHTTP`, and
`routes_unified.go`'s `handleModels`). A failed attempt logs the request
method, path, and remote address, and returns the usual 401 JSON
envelope. A successful attempt logs the resolved user's name and the
route. Neither line ever includes the presented API key — only the
identity it resolved to, never the key material itself.
## Known limitations
- **Streaming is delivered buffered, not token-by-token, under the current
Traefik + Yaegi combination.** The plugin's own code streams correctly
(proven by its unit tests, which drive the exact same code compiled, not
interpreted): content is always complete and correct, but a Yaegi
limitation strips the `http.Flusher` interface from the
`http.ResponseWriter` crossing from Traefik's compiled dispatcher into
the interpreted plugin, so no per-chunk flush actually reaches the
client — the whole response arrives in one batch once the handler
returns. This is a confirmed, external, still-open upstream bug
([traefik/traefik#10269](https://github.com/traefik/traefik/issues/10269),
labelled `kind/bug/confirmed` by Traefik's own maintainers; see also
[traefik/yaegi#1600](https://github.com/traefik/yaegi/issues/1600)), not
a defect in this plugin, and no code-level workaround is known. The
integration suite's true incremental-delivery timing assertion is gated
behind `INTEGRATION_STREAMING=1` and skipped by default for exactly this
reason — re-run it once the upstream issue is fixed.
- **A request rejected before it reaches a provider never consumes
request-rate quota.** On the unified routes, a malformed body, a missing
`model` field, or an unknown/denied model (404/403) is rejected before
`limiter.checkAndCount` runs. On native passthrough and the MCP/A2A
proxy, denied provider/target access (403), an unsupported `Upgrade`
request (501), and a path-traversal attempt (400) are all checked before
the limiter too. Once a request *does* reach a provider adapter,
though, a translate-time 400 (an unsupported parameter combination) or
Anthropic's embeddings 501 still count toward the caller's request-rate
limit — the limiter already ran by then — though neither ever bills a
token or cost, since usage stays at zero either way.
- **Gateway-to-client responses on passthrough and MCP/A2A routes are
identity-encoded.** The gateway strips a client's own `Accept-Encoding`
header before forwarding to the upstream, so Go's `http.Transport` adds
its own `Accept-Encoding: gzip` and, per its documented contract,
transparently decompresses a gzip response and strips its
`Content-Encoding`/`Content-Length` headers before handing it back — the
client sees plain, uncompressed bytes with no `Content-Encoding` header,
regardless of what it originally asked for. This covers the gzip case
Go's `Transport` auto-negotiates; an upstream that used a different
encoding on its own initiative (never observed against OpenAI,
Anthropic, or Gemini) would pass through unmodified instead.
- **Image and audio responses are never cost-accounted.** Request
counters (`requestsPerMinute`/`requestsPerDay`) increment as usual, but
`images/generations`, `audio/speech`, and `audio/transcriptions` never
move a token counter and always record cost as 0, regardless of what a
provider itself would charge for them — a real gap for an operator
billing on these endpoints, not just an accounting nuance.
- **An upstream response over 32MiB (`maxResponseBytes`, `providers.go`)
is silently truncated, not rejected.** Every adapter response read — the
unified chat/embeddings JSON, and the three media endpoints — is capped
at this limit. `audio/speech`'s streaming copy (`openaiAdapter.audioSpeech`,
`provider_openai.go`) already committed the upstream's 200 status before
it starts copying bytes, so a response over the cap simply ends early: no
error, no truncation signal of any kind, a shorter audio file
indistinguishable from a genuinely short one. `images/generations` and
`audio/transcriptions` (`forwardMediaBody`) read their whole response
before writing a status, so they hit the identical cap with the same
silent-truncation-into-200 outcome. 32MiB is generous for any of these
payloads in practice, but nothing enforces or signals the cutoff.
- **The response cache shares entries across every user and group that
can reach a model.** `cache`'s key (see [Caching](#caching)) carries no
user identity by design, so two different callers sending the same
request receive the exact same cached response, including the exact
same `id`/timing-shaped fields a provider embeds in it. This is
intentional — a provider's response to an identical request is
provider-deterministic — but it means the cache is not a per-tenant
isolation boundary: don't enable it for a deployment where two callers
must never observe evidence that another caller made the identical
request.
## Kubernetes
See [`examples/kubernetes.yaml`](examples/kubernetes.yaml) for a complete,
YAML-validated example manifest that `kubectl apply -f` applies cleanly
once Traefik's CRDs (`Middleware`, `IngressRoute`) are installed in the
cluster: a `Secret` holding the users file, a `Middleware` custom resource
carrying the plugin's own configuration (validated against the real `Config`
struct — every field in that example round-trips through `New()`
cleanly), a dummy backing `Deployment`/`Service` (Traefik requires a real
router backend even though this middleware never actually reaches it —
see [Quickstart](#quickstart)), an `IngressRoute` wiring it all to a
host, and a Traefik Helm values snippet for loading the plugin and
wiring provider keys in from Secrets via environment variables.
The short version:
- Provider keys and the Redis password live in `env:VAR` form inside the
`Middleware`, sourced from Kubernetes `Secret`s mounted as environment
variables on the Traefik pod — never as literals in the CR.
- The users file lives in its own `Secret`, mounted as a volume, referenced
by `users.file` in the `Middleware` spec. Editing that `Secret` (and
letting it propagate to the mounted file, or restarting the pod if your
cluster's kubelet doesn't sync `Secret` volumes fast enough for your
needs) updates the live user set within the 5-second reload window — no
Traefik restart required.
- Point `redis.address` at your cluster's existing Redis/Valkey/Dragonfly
rather than standing up a dedicated instance for this plugin — the
distributed limiter is a client, not a datastore.
## Development
```sh
make test # go test ./... -count=1
make lint # gofmt -l . && go vet ./...
make yaegi-check # interprets the plugin with Yaegi, exercises .traefik.yml's testData
make integration # docker-compose: real Traefik, real Redis, mock providers; brings the
# stack up, waits for it, runs the suite, always tears it down after
make integration-keep # same run, but leaves the stack up afterward for debugging —
# tear it down yourself with `make integration-down` when done
```
Also run before committing (not wired into `make lint`, since it needs a
separate binary): `golangci-lint run ./...` and
`golangci-lint run --enable=gosec ./...` (gosec applies to `_test.go`
files too), plus `fieldalignment ./...`
([`golang.org/x/tools/go/analysis/passes/fieldalignment`](https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fieldalignment)).
All are 0-issues on this repository as of this commit.
The plugin ships stdlib-only (no third-party dependencies at all, see
`go.mod`) — a hard constraint of Yaegi interpretation, not a style
preference. [`resp.go`](resp.go) is the clearest example: a hand-rolled,
~400-line RESP2 client instead of `go-redis`, which the design doc records
as unsuitable for Yaegi interpretation
(`docs/superpowers/specs/2026-08-19-traefik-llmgateway-design.md`, §7).
### Catalog submission
[`.traefik.yml`](.traefik.yml) is in place, with a `basePkg` override
(`traefikllmgateway`) required because Traefik's Yaegi middleware loader
otherwise binds the interpreted package under
`strings.ReplaceAll(path.Base(import), "-", "_")` — `traefik_llmgateway`
— which does not match this package's actual `package traefikllmgateway`
declaration. This was discovered running the plugin in real Traefik for
the first time; see the comment in `.traefik.yml` itself. **The Plugin
Catalog's own analyzer (piceus) has not yet been run against this
repository** — whether it accepts the same `basePkg` override the same
way real Traefik does is unverified until actual submission.
## License
[MIT](LICENSE) — Copyright (c) 2026 Lukasz Raczylo.