traefik-llmgateway

A Traefik middleware plugin that turns a Traefik instance into a multi-provider LLM gateway: one endpoint, several upstream providers, per-user API keys organised into groups, request/token/cost limits, and a registry plus authenticated proxy for in-cluster MCP servers and A2A agents.

The plugin is self-contained. It has no sidecar and no companion service. It runs as a Yaegi-interpreted Go module, the same way traefikoidc does, configured per-middleware in Traefik's dynamic configuration.

Status: feature-complete, all gates green (unit tests, -race, yaegi-check, integration). Not yet published to the Traefik Plugin Catalog — see Development for what that needs.

What it is

• A unified, OpenAI-compatible surface (/v1/chat/completions, /v1/embeddings, /v1/models) that translates requests and responses to and from Anthropic's and Google Gemini's own wire formats, so any OpenAI-SDK-compatible client can talk to all three providers through one API shape.
• Native passthrough per provider (/{provider}/...) for callers that want a provider's own wire format untouched, with the gateway injecting that provider's own API key and still enforcing auth and limits.
• Per-user API keys, organised into groups that control which providers, models, MCP servers, and agents a user can reach, plus request/token/cost limits enforced per user and per group.
• A registry and authenticated reverse proxy for MCP servers and A2A agents, so those can sit behind the same gateway, the same auth, and the same group-based visibility rules as the LLM traffic.

flowchart LR
    Req["incoming request"] --> Route{"route<br/>ServeHTTP: path+method dispatch"}
    Route -->|"a recognized route:<br/>/v1/models, /v1/chat/completions,<br/>/v1/embeddings, /v1/mcp/servers,<br/>/v1/agents, /mcp/name/...,<br/>/a2a/name/..., or /providerName/...<br/>for a known provider"| Auth["auth (per branch)<br/>identify(): key digest lookup"]
    Route -->|"no route matches"| Fallback["passthroughUnknown ? next : 404<br/>never calls identify()"]
    Auth -->|"no match"| E401["401 authentication_error"]
    Auth -->|"user, group"| Branch{"branch"}
    Branch --> Models["GET /v1/models"]
    Branch --> Unified["POST /v1/chat/completions<br/>POST /v1/embeddings"]
    Branch --> Registry["GET /v1/mcp/servers, /v1/agents"]
    Branch --> MCP["MCP/A2A proxy"]
    Branch --> Passthrough["native passthrough"]
    Unified --> Resolve["resolve model<br/>registry.resolve"]
    Resolve -->|"unknown/denied"| E404["404 / 403"]
    Resolve --> Limit1["limiter.checkAndCount"]
    MCP --> Limit2["limiter.checkAndCount"]
    Passthrough --> Limit3["limiter.checkAndCount"]
    Limit1 -->|"over budget"| E429["429 / 503"]
    Limit2 -->|"over budget"| E429
    Limit3 -->|"over budget"| E429
    Limit1 --> Exec["execute: translate + call adapter"]
    Limit2 --> ExecProxy["execute: reverse proxy"]
    Limit3 --> ExecProxy
    Exec --> Account["account: usage, cost"]
    Account --> Resp["respond to client"]
    ExecProxy --> Resp
    Models --> Resp
    Registry --> Resp

Every route the plugin recognises is handled entirely inside the middleware — next (the router's configured backing service) is only reached when a request matches none of the plugin's routes and passthroughUnknown is true.

Quickstart

Static configuration

The plugin is not yet in the Traefik Plugin Catalog (see Development), so the catalog form of static configuration (experimental.plugins.llmgateway) is not usable yet. Until then, load it as a localPlugin, mounting this repository's source into the Traefik container at <plugins-local dir>/src/github.com/lukaszraczylo/traefik-llmgateway — see examples/docker-compose.yml for a working, tested setup that does exactly this:

# traefik static config (traefik.yml / --experimental.localPlugins.*)
experimental:
  localPlugins:
    llmgateway:
      moduleName: github.com/lukaszraczylo/traefik-llmgateway

Once the plugin is published, the catalog form replaces this:

experimental:
  plugins:
    llmgateway:
      moduleName: github.com/lukaszraczylo/traefik-llmgateway
      version: "v0.1.0" # substitute the tag you actually deploy

Dynamic configuration

A minimal single-provider middleware, one group, one user:

http:
  middlewares:
    llmgateway:
      plugin:
        llmgateway:
          providers:
            openai:
              type: openai
              apiKey: "env:OPENAI_API_KEY"
              models: ["gpt-5-mini", "gpt-4o"]
          groups:
            default:
              providers: [] # empty = every configured provider is allowed
          users:
            inline:
              - name: alice
                group: default
                apiKey: "env:LLMGW_KEY_ALICE"

Write an "allow everything" group as providers: [] (an explicit empty list), not a bare {}. See the note at the top of examples/kubernetes.yaml — confirmed against Traefik's file provider, twice, with this exact plugin build: a bare default: {} fails plugin construction with expected a map, got 'string'; default: {providers: []} loads cleanly. Not tested against the Docker/ECS label providers or the Kubernetes CRD provider — no claim either way for those. An explicit empty list means the same thing as an empty object (all values allowed), so writing it this way costs nothing even where the bug does not apply.

Attach the middleware to a router the same way as any other Traefik middleware. The router's backing service is never actually reached unless passthroughUnknown: true and the request matches none of the plugin's routes — a small dummy service (traefik/whoami, or any always-up internal service) is enough.

Call it:

curl http://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_KEY_ALICE" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}]}'

Full configuration reference

Every limit or visibility list follows one convention throughout: empty or omitted means unlimited / all. Field names below are the exact JSON keys the plugin's Go structs expect (llmgateway.go); Traefik's dynamic config accepts them as YAML, which decodes to the same JSON shape.

Config (top level)

Field	Type	Default	Semantics
providers	map[string]ProviderConfig	—	Required, at least one entry, or the plugin refuses to construct.
groups	map[string]GroupConfig	{}	Optional. A user references one group by name; an unknown reference is a construction error.
pricing	map[string]ModelPricing	{}	Per-model USD/1M-token overrides, keyed by model id (bare or provider/model). Merged over, not replacing, the built-in table (pricing.go).
mcpServers	map[string]TargetConfig	{}	MCP server registry; see MCP/A2A.
agents	map[string]AgentConfig	{}	A2A agent registry; see MCP/A2A.
users	UsersConfig	—	Inline and/or file-backed API-key holders. With none configured, every request is unauthenticated and gets 401.
redis	RedisConfig	—	Distributed limit-counter backend. Omitted means in-process counters only (per-replica, approximate across multiple Traefik instances).
retry	RetryConfig	{} (disabled)	Same-provider retry for transient upstream failures — see Retry. Omitted or enabled: false means no retry: every request makes exactly one upstream attempt, byte-identical to a gateway built before this field existed.
cache	CacheConfig	{} (disabled)	Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see Caching. Omitted or enabled: false means no caching, byte-identical to a gateway built before this field existed.
admin	*AdminConfig	nil (disabled)	Read-only admin dashboard — see Admin. nil or enabled: false means the /admin* routes are not registered at all.
modelAliases	map[string]string	{}	Operator-defined alias id → target model id — see Model aliases. Omitted or empty means no aliases, byte-identical to a gateway built before this field existed.
passthroughUnknown	bool	false	false: a request matching none of the plugin's routes gets a 404 JSON envelope. true: it falls through to the router's own backing service.

Provider, mcpServers, and agents map keys (names) must match ^[a-zA-Z0-9._-]+$ and must not be v1, mcp, or a2a — those names are reserved because they would shadow the gateway's own fixed routes (/v1/..., /mcp/{name}/..., /a2a/{name}/...); a name that collides is a construction error, not a silently-unreachable provider. A nil map value under any of providers, groups, mcpServers, or agents is also a construction error, never a panic (providers.go, mcp_a2a.go, auth.go).

ProviderConfig

Field	Type	Default	Semantics
type	string	—	Required: openai, anthropic, or gemini. openai also serves any OpenAI-wire-compatible provider (Grok, DeepSeek, a local vLLM/Ollama gateway, ...) via baseUrl.
baseUrl	string	Per type: https://api.openai.com, https://api.anthropic.com, https://generativelanguage.googleapis.com	Trailing slash trimmed.
apiKey	string	—	Secret form: literal, env:VAR, or file:/path — see Secret forms. An openai-type provider resolving to an empty key is a valid keyless upstream (no Authorization header sent) — useful for a local model server with no auth. anthropic and gemini reject an empty resolved key at construction.
discoveryInterval	string (Go duration)	1h	Only meaningful when discovery: true. Invalid duration string is a construction error.
models	[]string	[]	Explicit model ids this provider serves. Combined with any discovered ids.
discovery	bool	false	Pull the provider's own model-listing endpoint at startup and on discoveryInterval. A discovery failure is logged and non-fatal; construction still succeeds on models alone.

GroupConfig

Field	Type	Default	Semantics
limits	*LimitsConfig	nil (unlimited)
providers	[]string	[] (all)	Glob-matched (path.Match) against a provider's configured name.
models	[]string	[] (all)	Glob-matched, exactly as given — no automatic provider/ prefix stripping. Both the bare and provider-prefixed forms of a model id are checked, so a pattern like gpt-* matches a client request for either gpt-5-mini or openai/gpt-5-mini.
cache	*bool	nil (inherit)	Overrides the top-level cache.enabled setting for this group's requests — see Caching. nil inherits the global setting; false opts the group out even when caching is globally on; true opts the group in, but only when the top-level cache block is actually configured (a construction error otherwise — there is nothing to inherit ttl/maxBodyBytes from).
mcpServers	[]string	[] (all)	Glob-matched against a configured MCP server name.
agents	[]string	[] (all)	Glob-matched against a configured agent name.

LimitsConfig

All six fields default to 0 (unlimited). A negative count, or a cost that is negative, NaN, or ±Inf, is a construction error.

Field	Type	Window
requestsPerMinute	int64	Fixed 1-minute window, UTC.
requestsPerDay	int64	Fixed calendar-day window, UTC midnight to midnight.
tokensPerDay	int64	Fixed calendar-day window, UTC.
tokensPerMonth	int64	Fixed calendar-month window, UTC.
costPerDayUSD	float64	Fixed calendar-day window, UTC.
costPerMonthUSD	float64	Fixed calendar-month window, UTC.

A user's own limits, when set, are checked in addition to their group's limits, not instead of them — a request is refused the moment it breaches whichever of the two is tighter. A user with no limits of their own is governed purely by their group's — see UserConfig.

UsersConfig

Field	Type	Default	Semantics
file	string	—	Path to a JSON users file, merged over inline and hot-reloaded — see Users file.
inline	[]UserConfig	[]	Resolved once at plugin construction.

UserConfig

Field	Type	Default	Semantics
name	string	—	Required.
group	string	—	Required; must reference a configured group, or construction fails.
apiKey	string	—	Secret form (see Secret forms); must resolve non-empty and unique across every inline and file-sourced user.
limits	*LimitsConfig	nil (governed by group's limits alone)	When set, enforced alongside — not instead of — the group's own limits; both scopes are checked, and the request is refused by whichever is breached first.
admin	bool	false	Grants access to the read-only admin dashboard — see Admin. Works the same for an inline or a file-sourced user. An admin user is otherwise ordinary: their own keys, group, and limits still apply, including to the admin routes themselves.

RedisConfig

Field	Type	Default	Semantics
address	string	—	Required when the redis block is present at all. host:port, RESP protocol — works against Redis, Valkey, or Dragonfly.
password	string	"" (no AUTH)	Secret form.
db	int	0	Must be ≥ 0. Sent via SELECT on every new connection.
failOpen	*bool	true	true: a Redis error switches that operation to the in-process fallback store and logs (rate-limited to once per 30s). false: a Redis error returns 503 instead of enforcing against a non-shared fallback.

RetryConfig

Field	Type	Default	Semantics
enabled	bool	false	false: no retry, one upstream attempt per request. true: retry per Retry below.
attempts	int	1	Retries performed after the first try, not the total try count. 1 (default) allows one retry — two tries total. The maximum, 3, allows three retries — four tries total. A value outside 1-3 is a construction error.
backoff	string (Go duration)	250ms	Base wait before the first retry; doubles on each further retry, capped at 2s per wait. Invalid duration string is a construction error.

CacheConfig

Field	Type	Default	Semantics
enabled	bool	false	false: no caching, /v1/chat/completions and /v1/embeddings behave byte-identically to a gateway built before this field existed. true: caching per Caching below — requires redis to be configured too, or it silently stays off (one warning logged).
ttl	string (Go duration)	5m	How long a cached response stays valid. Invalid duration string is a construction error.
maxBodyBytes	int	1048576 (1MiB)	Largest response body still eligible for caching; a larger one is skipped (never stored, never an error). Maximum accepted value is 8388608 (8MiB) — above that is a construction error.

AdminConfig

Field	Type	Default	Semantics
enabled	bool	false	false: the /admin* routes are not registered at all — a request to any of them falls through to the plugin's existing 404/passthroughUnknown handling like any other unrecognized path. true: the read-only dashboard per Admin below.

ModelPricing

Field	Type	Semantics
inputPerM	float64	USD per 1,000,000 input/prompt tokens.
outputPerM	float64	USD per 1,000,000 output/completion tokens.

TargetConfig (an mcpServers entry)

Field	Type	Semantics
url	string	Required; must parse with an http or https scheme, checked at construction.

AgentConfig (an agents entry)

Field	Type	Default	Semantics
url	string	—	Required; same URL validation as TargetConfig.
card	string	/.well-known/agent-card.json	Path appended to the agent's gateway-relative proxy URL for its A2A agent-card listing.

Secret forms

Every apiKey/password field accepts one of three forms (secrets.go):

• Literal — the value itself, used as-is. Fine for local/dev; avoid for anything real.
• env:VAR — read from environment variable VAR on the Traefik process. Errors if unset or empty.
• file:/path — read from a file, trimmed of surrounding whitespace. Errors if unreadable or the trimmed content is empty.

Provider keys, Redis's password, and inline users' keys are resolved once, at plugin construction. A file-sourced user's apiKey is re-resolved on every hot reload (see below), so it can use env:/file: forms too and pick up a rotated value without a Traefik restart.

Users file format

One JSON object, one users array, mounted from a single Kubernetes Secret (users_file.go):

{
  "users": [
    {"name": "alice", "group": "engineering", "apiKey": "sk-llmgw-..."},
    {"name": "bob", "group": "general", "apiKey": "env:LLMGW_KEY_BOB", "limits": {"requestsPerDay": 100}}
  ]
}

A file user with the same name as an inline user overrides it (the inline entry is dropped from the merged set). Hot reload polls the file's mtime, throttled to once per 5 seconds on the request path, and is best-effort: a reload already in progress is skipped rather than serialising every concurrent request behind it, so a change lands within a few request-driven checks, not necessarily on the very next one. A reload's own error (unreadable file, malformed JSON, a user referencing an unknown group, a duplicate key) is logged and the last good user set is kept — a bad edit to the file never breaks already-authenticated traffic.

Model routing

• A model id in provider/model form resolves directly against that provider, but only when the part before the slash names an actually configured provider. Any other id — including one that merely contains a slash, such as an upstream's own uni/deepseek-v4-flash-0731 naming — is treated as a single bare id.
• A bare id resolves against the first configured provider, in sorted provider-name order, whose known model set (explicit models plus any discovered ids) contains it. When more than one provider knows the same bare id, the sorted-first provider wins it and the gateway logs a collision warning once per id; every provider's copy stays reachable through the explicit provider/model form regardless of who won the bare id, and GET /v1/models lists both the bare (winner's) form and every provider/model form.
• Authorization always requires both group.allowsModel and group.allowsProvider for the resolved provider — a model that exists but the caller's group cannot reach returns 403, distinct from a model no configured provider knows at all (404).
• Discovery: GET {baseUrl}/v1/models for an openai-type provider, GET {baseUrl}/v1/models for anthropic, GET {baseUrl}/v1beta/models for gemini. Runs once at plugin construction (bounded to 5s per provider) and again on discoveryInterval in the background. A failed refresh is logged and keeps the provider's last-known discovered set — stale-while-error, never an empty list just because one refresh attempt failed.

Model aliases

Operator-defined indirection: a client-facing alias id that resolves to a real target model id. modelAliases is empty by default, and an empty map changes nothing about model resolution — this is purely additive.

http:
  middlewares:
    llmgateway:
      plugin:
        llmgateway:
          # ... providers, groups, users unchanged ...
          modelAliases:
            # Client-facing name -> real target. The target can itself be
            # bare ("gpt-5-mini") or provider-prefixed
            # ("anthropic/claude-sonnet-4-5"); either resolves through the
            # normal rules in [Model routing](#model-routing) above.
            aliased/coding: "anthropic/claude-sonnet-4-5"
            aliased/fast: "gpt-5-mini"

• Swapping the underlying model: point an alias at a different target and let Traefik's dynamic-config reload apply it — no gateway restart, no client-side change. A client that has always addressed aliased/coding keeps working unmodified after the operator moves that alias to a newer model.
• Precedence: an EXACT alias match wins resolution before any other rule, including provider/model prefix splitting — checked first in modelRegistry.resolve. The target then resolves through the same rules as any other id (bare, provider/model, or a discovered id). Because of this, an alias whose id contains a / is rejected at construction if the part before the / names an actually configured provider (it would be shadowed by prefix resolution) — see the validation rules below.
• Authorization: a group may use an alias when its models glob matches the alias name itself, OR when it already grants access to the resolved target — either is enough. providers still gates the target's own provider as usual.
• Validation at construction (a bad alias config fails the same way a bad provider config does — no gateway starts with a broken alias table):
  • the alias must be non-empty and contain no control characters;
  • when the alias contains /, its prefix must not name a configured provider (would be shadowed by prefix resolution);
  • the alias must not equal an explicit models entry of any provider (ambiguous — which one wins?);
  • the target must not itself be another alias (no alias chains);
  • the target must not be empty. A target naming a model that is not yet known (awaiting a provider's first discovery fetch, or simply never configured) is accepted at construction — it resolves lazily. An unresolvable target at request time returns 404 with a message naming both the alias and the missing target. A target that is later discovered and collides with an already-configured alias id is not a construction error either — the alias always wins that resolution, and the gateway logs one warning per such id.
• GET /v1/models listing: every alias the caller's group may use is listed as a model object alongside real models — id is the alias, owned_by is the target's resolved provider name. An alias whose target does not resolve yet is omitted from the listing until it does.
• Echo behavior: exactly like the existing provider-prefix alias echo (__alias, Unified vs. passthrough) — anthropic- and gemini-type targets build their own response envelope and echo the client's exact alias string back in its model field. An openai-type target's response is a verbatim passthrough of the upstream body, so its model field carries whatever the upstream itself returned (typically the bare upstream model id) — not the alias — the same documented v0.1 passthrough asymmetry, unchanged by this feature.
• Applies everywhere a model routes: unified chat and embeddings, /v1/images/generations, /v1/audio/speech, and /v1/audio/transcriptions (including its multipart rewrite — the upstream model field receives the resolved target's own upstream id, never the alias, exactly like a provider-prefixed id already does).
• Admin overview: GET /admin/api/overview includes an aliases array ({alias, target}, sorted by alias) — see Admin. No secrets involved.

Limits and accounting

• Counters use fixed windows, keyed by UTC calendar boundaries (the minute, the day, the month) — not a sliding window. Requests-per-minute and requests-per-day counters increment before evaluation, so every request counts toward its scopes' rate tracking even if it is ultimately refused by a different scope's limit.
• Token and cost budgets are checked against the value already accumulated from prior requests, before this request's own usage is known (usage only arrives after the upstream call completes). A request that would start already over budget is refused; a request that starts under budget but pushes the scope over it is still allowed to complete — an accepted at-most-one-request overshoot, not an exactly-enforced cap.
• Accounting parses usage from the provider's own response: OpenAI-type usage, Anthropic usage (from message_start/message_delta), Gemini usageMetadata. When a non-streaming unified-route response reports no usage at all, the gateway falls back to a rough estimate — ceil(request-body-bytes / 4) counted as prompt tokens, no completion estimate — and marks it estimated. A streaming unified-route response with no usage is logged and the request itself is still counted, but no token estimate is applied.
• Cost is tokens × price, computed in integer micro-USD (1,000,000ths of a dollar) throughout to avoid float drift, from the built-in per-model table (pricing.go) or a configured pricing override. An unpriced model costs 0 and logs a once-per-model warning (capped at 128 distinct unpriced model names per process lifetime, then one summary warning). The built-in table is approximate — list prices as each provider published them, recorded 2026-08, and not kept in sync automatically. Set pricing overrides for billing-grade accuracy.
• Storage: redis configured wires a hand-rolled, stdlib-only RESP2 client (resp.go) — no go-redis, which is not Yaegi-interpretable — against Redis, Valkey, or Dragonfly. Semantics are deliberately at-least-once, never under-counting: a lost reply after the server already applied an INCRBY can cause a resend that over-counts by one request's worth on that key. This is a fail-safe trade-off (more restrictive than reality), never a fail-open one. Without redis configured, or when it errors and failOpen: true, counters live in an in-process map — correct for one Traefik replica, only approximate across several, since each replica counts independently. Exact limits across multiple Traefik replicas require the shared Redis.

Retry

Same-provider retry for a transient upstream failure — no cross-provider failover. Off by default (retry.enabled: false); every config that predates this field keeps making exactly one upstream attempt per request.

• Scope: every adapter upstream call — chat, embeddings, model listing — across all three provider types. Native passthrough (/{provider}/...) and the MCP/A2A proxy are raw reverse proxies and are never retried; the client owns retry semantics there.
• Attempts is retries, not tries: attempts counts retries performed after the first try. The default, 1, allows one retry — two tries total, not one. The maximum, 3, allows three retries — four tries total.
• Transient classification: a connection error (dial/EOF/reset), HTTP 429, or HTTP 5xx. Never a non-429 4xx, a translation error, a request that failed to build in the first place (a malformed method or URL fails identically on every attempt), or a context cancellation/deadline.
• Zero-bytes rule: retry only runs before any response byte reaches the client. A non-streaming call may retry its whole exchange. A streaming call retries only the initial request, up to the point the upstream's headers and status arrive — once the first chunk is forwarded, a stream that then dies mid-flight is not retried, since the client already holds partial data.
• Waiting: a 429 response carrying a Retry-After header that parses as a non-negative whole-second count no greater than 2s waits exactly that long. Every other case — no header, an unparseable value (for example the HTTP-date form), a negative value, a value over 2s, or a Retry-After on a non-429 5xx — falls back to exponential backoff (backoff × 2^(attempt-1)), capped at 2s per wait either way. A context canceled or timed out during a wait aborts the remaining retries immediately, rather than blocking out the wait in full.
• Accounting: only the final attempt's usage is recorded — a failed attempt's body is drained and discarded, never parsed for usage, so it is never double-counted. Request counters still increment once per client request, unchanged.

Caching

An opt-in, Redis-backed response cache for the unified, non-streaming POST /v1/chat/completions and POST /v1/embeddings routes. Off by default (cache.enabled: false); every config that predates this field keeps calling the upstream on every request. Caching also requires redis to be configured — without it, cache.enabled: true is silently disabled with one warning logged at construction, never a per-replica in-process fallback: divergent replicas would otherwise serve different cached bodies for the same request.

• What's cacheable: a non-streaming request to either route, whose upstream response was HTTP 200 and no larger than cache.maxBodyBytes. Never cached: a streaming request, native passthrough, MCP/A2A, the image/audio endpoints (see Image and audio endpoints), or any non-200 upstream response.
• Key: llmgw:cache: plus a hex SHA-256 digest of the provider name, the resolved upstream model id, the client's own exactly-as-sent model string (requestedModel), the endpoint (chat or embeddings), and the canonical JSON of the request body with stream_options and user removed. requestedModel and the endpoint are both part of the key so that two different alias forms of the same upstream model (claude-x versus anthropic/claude-x) never collide into one cache entry: a translating adapter (anthropic, gemini) bakes the client's own alias into the cached response body's model field, so a second client using the other alias form would otherwise receive an id it never sent.
• User identity is not part of the key. Identical requests share one cache entry across every user and group that can reach the model — by design, since a provider's response to an identical request is provider-deterministic and carries no per-user data.
• Hit/miss header: every cacheable request's response carries X-Llmgw-Cache: hit or X-Llmgw-Cache: miss, set before the body is written either way.
• Accounting: a cache hit increments request counters exactly like any other request — limiter.checkAndCount runs before the cache lookup — but never touches token or cost counters, and never runs the streaming-response-with-no-usage estimate. Only a genuine upstream call (a miss) accounts real tokens and cost. This means a cache hit is request-quota-counted but token/cost-free.
• Group override: GroupConfig.cache (a *bool) overrides the top-level cache.enabled setting per group — nil inherits it, false opts the group out even when caching is globally on, true opts the group in. Setting true on a group when the top-level cache is not enabled — the block omitted entirely, or present with enabled: false, both read the same way — is a construction error: there is nothing configured to inherit ttl/maxBodyBytes from.
• Storage: values are stored via the same hand-rolled RESP2 client (resp.go) the limiter's distributed counters use — the identical Redis/Valkey/Dragonfly connection, not a second one. A Redis error during a cache read or write is treated as a miss (logged, rate-limited) and never fails the request; the upstream call still happens and the client still gets a correct answer.

Unified vs. passthrough

	Unified (/v1/...)	Native passthrough (/{provider}/...)
Wire format	Always OpenAI-shaped in, OpenAI-shaped out.	The provider's own native format, untouched.
Translation	openai-type: body forwarded verbatim (model id rewritten). anthropic/gemini: full bidirectional translation, including streaming, chunk by chunk.	None — a raw reverse proxy.
Model alias in the response	Translated providers (anthropic, gemini) echo back the client's exact requested model string in the response's model field, even though the upstream call used the resolved provider model id. An openai-type response is a verbatim passthrough of the upstream body, so it carries whatever model id the upstream itself returned — this asymmetry is intentional, not a bug.	The upstream's own model field, verbatim — there is no alias to echo.
Embeddings	openai-type: passthrough. gemini: mapped to :embedContent/:batchEmbedContents. anthropic: 501 — Anthropic's API has no embeddings endpoint.	Whatever the provider itself supports at that path; the gateway does not gate it.
Unsupported parameters	A semantically meaningful field the target provider cannot express (e.g. n > 1, logit_bias, logprobs on Anthropic) is a 400, never silently dropped.	Not applicable — the request reaches the provider exactly as sent.
Non-2xx upstream response	Wrapped in the gateway's own error envelope, with the provider's own body embedded under error.upstream.	Forwarded to the client exactly as the upstream sent it — status, headers, and body.

Image and audio endpoints

POST /v1/images/generations, POST /v1/audio/speech, and POST /v1/audio/transcriptions follow the same pipeline as /v1/chat/completions — auth, model routing, group authorization, limits, adapter call — minus response caching and token/cost accounting (below).

	openai-type	gemini	anthropic
images/generations	Native forward.	Translated to Imagen's :predict (below).	501.
audio/speech	Native forward; the binary response is streamed to the client as it arrives.	501 — no OpenAI-compatible text-to-speech endpoint.	501.
audio/transcriptions	Native forward; see below for how the multipart body is forwarded.	501 — no OpenAI-compatible speech-to-text endpoint.	501.

• Model routing: images/generations and audio/speech read model from the JSON request body. audio/transcriptions reads it from the multipart request's model form field instead — the field can appear in any position among the request's parts, and a body with more than one model field is a 400.
• Multipart forwarding: when the client's model field already equals the resolved upstream model id (a bare id, the common case), the original multipart body is replayed byte-for-byte, unchanged. When it does not — a provider-prefixed id (openai/whisper-1) or another alias resolves to a different upstream model string — the body is rebuilt with every other part copied verbatim and only the model part's value rewritten, so the alias never reaches the real provider. The rebuilt body uses a new multipart boundary; the client's exact boundary is not preserved, only an equivalent body.
• Gemini image translation: {model, prompt, n, size, response_format} maps to Imagen's {instances:[{prompt}], parameters: {sampleCount, aspectRatio}}. size maps to aspectRatio: 1024x1024/512x512 → 1:1, 1792x1024 → 16:9, 1024x1792 → 9:16; an absent size defaults to 1:1; any other value is a 400. response_format: "url" is a 400 — the gateway stores nothing to host a URL from. A truthy quality or style is a 400, the same convention chat's translators use for an unsupported field. Imagen's predictions[].bytesBase64Encoded maps back to OpenAI's data[].b64_json.
• Never cached: image and audio responses are excluded from the response cache (cache.enabled) regardless of its configuration.
• Accounting: request counters (requestsPerMinute/requestsPerDay) increment the same as any other route. Token counters never move, and cost is always recorded as 0 — images and audio are not cost-accounted.
• Body limits: images/generations and audio/speech share the unified routes' 10MiB JSON cap. audio/transcriptions enforces the same 10MiB cap explicitly, returning 413 over it — unlike an oversized JSON body, which is silently truncated into a parse failure, a truncated multipart body can still parse a valid prefix and would forward cut-off audio upstream if it were not rejected outright.
• Retry (above) applies to all three the same as chat/embeddings; a streaming audio/speech response is retried only before the first response byte reaches the client, the same zero-bytes rule as chat's streaming path.

Admin

A read-only dashboard and JSON API for operational visibility — providers, groups, and per-user/per-group usage against configured limits. No mutation of any kind; config stays owned by GitOps/Traefik as usual. Off by default (admin.enabled: false, or the admin block omitted entirely): the three routes below are not registered at all, and a request to any of them falls through to the plugin's existing 404/passthroughUnknown handling.

• Enabling it: set admin.enabled: true, then flag at least one user admin: true (UserConfig.admin) so someone can actually reach /admin/api/*. An admin user is otherwise ordinary — their own keys, group, and limits still apply, including to the admin routes themselves.
• GET /admin serves the dashboard's HTML/CSS/JS shell. This route is deliberately served with no authentication at all once admin.enabled is true (disabled still falls through to the same 404/passthroughUnknown handling described above): a browser navigating straight to the URL has no way to attach a custom Authorization or x-api-key header, so gating the page itself would make it unreachable from a browser in the first place. This is safe because the shell carries zero data of its own — every value is fetched client-side from the two JSON routes below, which stay fully gated.
• Browser key-entry flow: the page's script reads an admin API key from the current tab's sessionStorage. Absent a stored key, or on any 401/403 from either JSON route, it shows an inline key-entry form instead of the dashboard. A submitted key is kept only in sessionStorage — never a cookie, localStorage, or any persistent store — so it disappears when the tab closes, and it is sent only as the x-api-key header on this page's own /admin/api/* fetches.
• GET /admin/api/overview and GET /admin/api/usage share one gate: unauthenticated → 401, authenticated non-admin → 403, admin → serve. overview returns the provider list, groups, the configured model alias table (alias/target pairs, sorted by alias — see Model aliases), redis/cache status, and the plugin version string. usage returns every user's and every group's current-window counter values (req/min, req/day, tok/day, tok/month, cost/day, cost/month) alongside their configured limits.
• What's exposed: provider names, types, base URLs (with any userinfo/query string stripped before it's ever echoed), model counts, discovery status, group/user names, membership, limits, and live usage counters. Never exposed: API keys (not even digests), provider keys, the Redis password, or users-file path contents — every secret-bearing field is redacted from every response.
• Poll counts against quota: the dashboard's own JavaScript polls both JSON routes every 5 seconds while open. Each poll is a real authenticated request and increments the admin user's own request counters exactly like any other route — leaving a dashboard tab open against a tightly limited admin user can itself exhaust their requestsPerMinute/requestsPerDay budget.
• GET /admin counts nothing: it is unauthenticated, so there is no identified user to count a request against.
• Response headers: both JSON routes set X-Content-Type-Options: nosniff and Cache-Control: no-store, and all three routes share one Content-Security-Policy header (default-src 'none'; script-src 'unsafe-inline'; style-src 'unsafe-inline'; connect-src 'self') — the dashboard loads no external asset of any kind and works in an air-gapped cluster.

MCP and A2A

• Registry (group-filtered, sorted by name): GET /v1/mcp/servers → {"servers":[{"name","url"}]}; GET /v1/agents → {"agents":[{"name","url","card"}]}.
• Proxy: /mcp/{name}/{rest...} and /a2a/{name}/{rest...} reverse-proxy {rest...} to the configured target's url, streaming (SSE) preserved via a flush-per-write proxy. The gateway's own Authorization/x-api-key credential is stripped before proxying, and — unlike native provider passthrough — no credential is injected on the way out: an MCP server or A2A agent is treated as an in-cluster target that trusts the gateway's network position, not a per-provider key the gateway holds on the caller's behalf.
• Request-rate limits (requestsPerMinute/requestsPerDay) apply to target-proxy calls the same as everywhere else; token/cost metrics do not, since there is no usage to parse from an arbitrary MCP/A2A response.
• Group visibility (mcpServers/agents glob lists) gates both the registry listing and the proxy path itself — a denied name is a 403 at the proxy, not just hidden from the listing.

Security notes

• API keys are never retained in memory as plaintext after construction — only their SHA-256 digests. Lookup is by digest, verified with crypto/subtle.ConstantTimeCompare against the stored digest, so matching does not leak timing information about a near-miss key.
• Provider API keys are injected server-side (native passthrough's injectAuth) and never reach the client; a client's own gateway key is stripped before any upstream or target request and never reaches a provider, MCP server, or A2A agent.
• Neither kind of key is ever written to a log line — the logging helpers (logger.go) carry an explicit "never log key material" contract. providerHTTPError's Error() string (providers.go) separately omits the upstream response body it still holds as a struct field, so a future caller that ever formats the error bare (%v) cannot accidentally echo provider response content into a log line; today's only caller (writeProviderUpstreamError) reads that body directly and puts it in the client-facing response, never in a log.
• Every authentication attempt is logged, through one shared helper (logAuthEvent, logger.go) called at each of the six routes that authenticate a request (llmgateway.go's ServeHTTP, and routes_unified.go's handleModels). A failed attempt logs the request method, path, and remote address, and returns the usual 401 JSON envelope. A successful attempt logs the resolved user's name and the route. Neither line ever includes the presented API key — only the identity it resolved to, never the key material itself.

Known limitations

• Streaming is delivered buffered, not token-by-token, under the current Traefik + Yaegi combination. The plugin's own code streams correctly (proven by its unit tests, which drive the exact same code compiled, not interpreted): content is always complete and correct, but a Yaegi limitation strips the http.Flusher interface from the http.ResponseWriter crossing from Traefik's compiled dispatcher into the interpreted plugin, so no per-chunk flush actually reaches the client — the whole response arrives in one batch once the handler returns. This is a confirmed, external, still-open upstream bug (traefik/traefik#10269, labelled kind/bug/confirmed by Traefik's own maintainers; see also traefik/yaegi#1600), not a defect in this plugin, and no code-level workaround is known. The integration suite's true incremental-delivery timing assertion is gated behind INTEGRATION_STREAMING=1 and skipped by default for exactly this reason — re-run it once the upstream issue is fixed.
• A request rejected before it reaches a provider never consumes request-rate quota. On the unified routes, a malformed body, a missing model field, or an unknown/denied model (404/403) is rejected before limiter.checkAndCount runs. On native passthrough and the MCP/A2A proxy, denied provider/target access (403), an unsupported Upgrade request (501), and a path-traversal attempt (400) are all checked before the limiter too. Once a request does reach a provider adapter, though, a translate-time 400 (an unsupported parameter combination) or Anthropic's embeddings 501 still count toward the caller's request-rate limit — the limiter already ran by then — though neither ever bills a token or cost, since usage stays at zero either way.
• Gateway-to-client responses on passthrough and MCP/A2A routes are identity-encoded. The gateway strips a client's own Accept-Encoding header before forwarding to the upstream, so Go's http.Transport adds its own Accept-Encoding: gzip and, per its documented contract, transparently decompresses a gzip response and strips its Content-Encoding/Content-Length headers before handing it back — the client sees plain, uncompressed bytes with no Content-Encoding header, regardless of what it originally asked for. This covers the gzip case Go's Transport auto-negotiates; an upstream that used a different encoding on its own initiative (never observed against OpenAI, Anthropic, or Gemini) would pass through unmodified instead.
• Image and audio responses are never cost-accounted. Request counters (requestsPerMinute/requestsPerDay) increment as usual, but images/generations, audio/speech, and audio/transcriptions never move a token counter and always record cost as 0, regardless of what a provider itself would charge for them — a real gap for an operator billing on these endpoints, not just an accounting nuance.
• The response cache shares entries across every user and group that can reach a model. cache's key (see Caching) carries no user identity by design, so two different callers sending the same request receive the exact same cached response, including the exact same id/timing-shaped fields a provider embeds in it. This is intentional — a provider's response to an identical request is provider-deterministic — but it means the cache is not a per-tenant isolation boundary: don't enable it for a deployment where two callers must never observe evidence that another caller made the identical request.

Kubernetes

See examples/kubernetes.yaml for a complete, YAML-validated example manifest that kubectl apply -f applies cleanly once Traefik's CRDs (Middleware, IngressRoute) are installed in the cluster: a Secret holding the users file, a Middleware custom resource carrying the plugin's own configuration (validated against the real Config struct — every field in that example round-trips through New() cleanly), a dummy backing Deployment/Service (Traefik requires a real router backend even though this middleware never actually reaches it — see Quickstart), an IngressRoute wiring it all to a host, and a Traefik Helm values snippet for loading the plugin and wiring provider keys in from Secrets via environment variables.

The short version:

• Provider keys and the Redis password live in env:VAR form inside the Middleware, sourced from Kubernetes Secrets mounted as environment variables on the Traefik pod — never as literals in the CR.
• The users file lives in its own Secret, mounted as a volume, referenced by users.file in the Middleware spec. Editing that Secret (and letting it propagate to the mounted file, or restarting the pod if your cluster's kubelet doesn't sync Secret volumes fast enough for your needs) updates the live user set within the 5-second reload window — no Traefik restart required.
• Point redis.address at your cluster's existing Redis/Valkey/Dragonfly rather than standing up a dedicated instance for this plugin — the distributed limiter is a client, not a datastore.

Development

make test            # go test ./... -count=1
make lint            # gofmt -l . && go vet ./...
make yaegi-check      # interprets the plugin with Yaegi, exercises .traefik.yml's testData
make integration      # docker-compose: real Traefik, real Redis, mock providers; brings the
                      # stack up, waits for it, runs the suite, always tears it down after
make integration-keep # same run, but leaves the stack up afterward for debugging —
                      # tear it down yourself with `make integration-down` when done

Also run before committing (not wired into make lint, since it needs a separate binary): golangci-lint run ./... and golangci-lint run --enable=gosec ./... (gosec applies to _test.go files too), plus fieldalignment ./... (golang.org/x/tools/go/analysis/passes/fieldalignment). All are 0-issues on this repository as of this commit.

The plugin ships stdlib-only (no third-party dependencies at all, see go.mod) — a hard constraint of Yaegi interpretation, not a style preference. resp.go is the clearest example: a hand-rolled, ~400-line RESP2 client instead of go-redis, which the design doc records as unsuitable for Yaegi interpretation (docs/superpowers/specs/2026-08-19-traefik-llmgateway-design.md, §7).

Catalog submission

.traefik.yml is in place, with a basePkg override (traefikllmgateway) required because Traefik's Yaegi middleware loader otherwise binds the interpreted package under strings.ReplaceAll(path.Base(import), "-", "_") — traefik_llmgateway — which does not match this package's actual package traefikllmgateway declaration. This was discovered running the plugin in real Traefik for the first time; see the comment in .traefik.yml itself. The Plugin Catalog's own analyzer (piceus) has not yet been run against this repository — whether it accepts the same basePkg override the same way real Traefik does is unverified until actual submission.

License

MIT — Copyright (c) 2026 Lukasz Raczylo.