mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-13 04:58:50 +00:00
lukaszraczylo
5a11cc85be
Fixes from review, verified against source before rewriting:
- README.md Security notes: "auth events are logged" was false — grepped
auth.go, zero logging calls exist anywhere in it. A lookup miss is a
plain 401 envelope with no log line at all, not "logged without key
material". Also corrected the adjacent providerHTTPError claim: the
struct retains the upstream response body as a field (surfaced
deliberately to the client under error.upstream); only its Error()
string omits the body, and nothing in this codebase currently logs a
providerHTTPError via a bare %v anyway.
- README.md mermaid diagram: the previous version showed a single global
Auth gate ahead of routing. ServeHTTP (llmgateway.go) actually dispatches
on path+method first and calls identify() separately inside each
recognized-route branch; an unrecognized route (passthroughUnknown/404
fallback) never calls identify() at all. Redrawn Route -> per-branch
Auth, with the unauthenticated fallback branching directly off Route.
Also fixed the 401 node's error type label from the nonexistent
"invalid_api_key" to the actual envelope type, "authentication_error"
(llmgateway.go, routes_unified.go). Re-rendered with mmdc before and
after the fix.
- README.md + examples/kubernetes.yaml: narrowed the bare-{} empty-group
claim. Re-verified this session with an isolated, minimal repro
(Traefik file provider only, no docker-compose configs: mechanism
involved) that a bare `default: {}` still fails plugin construction
with the file provider specifically: "expected a map, got 'string'".
Scoped the claim to what is actually verified (file provider, twice)
and explicitly flagged Docker/ECS label providers and the Kubernetes
CRD provider as untested, rather than asserting the file provider is
unaffected or that the CRD provider shares its decode path — the
latter was an unverified inference in the prior text, now hedged.
- examples/kubernetes.yaml: the IngressRoute referenced an undefined
Service (llmgateway-dummy-backend), so the manifest did not actually
apply as advertised. Added the missing Deployment + Service (traefik/
whoami, the same image already exercised end-to-end in
examples/docker-compose.yml this session). Re-validated: 6-document
YAML parses, Middleware's plugin config still round-trips through the
real Config struct and New() (throwaway test, deleted after).
- README.md Known limitations: dropped the standalone path-traversal
bullet — it duplicated content already stated in the "request rejected
before it reaches a provider" bullet immediately above it.
NOTE for the reviewer: the ruling on the bare-{} finding stated "file
provider observed working" and named Docker/ECS label providers as the
affected ones. I re-tested this directly (twice, including a standalone
repro outside the compose demo) and the file provider fails on a bare
{} exactly as originally reported — I could not reproduce "file provider
unaffected" and have not tested the label providers at all. Narrowed the
doc wording to only what I verified rather than asserting either
provider's behaviour without evidence; flagging the discrepancy in case
the review had independent grounds I don't have visibility into.
Gates: go test ./... -count=1, gofmt -l ., go vet ./..., make
yaegi-check — all clean. Also re-ran go test -race, golangci-lint (incl.
gosec), and fieldalignment, matching the standard this task already set;
all clean.
199 lines
6.4 KiB
YAML
199 lines
6.4 KiB
YAML
# Example Kubernetes deployment of the LLM Gateway Traefik plugin: a
|
|
# Secret holding the file-backed users list, a Middleware custom resource
|
|
# carrying the plugin's own configuration, and a router wiring both onto a
|
|
# host. See ../README.md for the full configuration reference and
|
|
# ../.traefik.yml for the catalog-facing plugin manifest.
|
|
#
|
|
# Provider API keys are NOT put in this file. They are read by the plugin
|
|
# at startup via "env:VAR_NAME" (see the Middleware below), which means
|
|
# they must be set as environment variables on the Traefik pod itself —
|
|
# see the Helm values snippet at the bottom of this file.
|
|
#
|
|
# The empty-map gotcha: write an empty group as `providers: []`, never a
|
|
# bare `{}`. Confirmed against Traefik's file provider, twice, with this
|
|
# exact plugin build (examples/docker-compose.yml, plus an isolated
|
|
# minimal repro outside that demo's compose-specific config mechanism): a
|
|
# bare `default: {}` fails plugin construction with
|
|
# "expected a map, got 'string'"; an explicit empty list decodes
|
|
# correctly and means the same thing (all values allowed). Whether the
|
|
# Kubernetes CRD provider used below hits the same decode path as the
|
|
# file provider is NOT verified — this file uses the empty-list form
|
|
# defensively, not because the CRD path is confirmed affected.
|
|
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: llmgw-users
|
|
namespace: traefik
|
|
type: Opaque
|
|
stringData:
|
|
# Read by the plugin via users.file (mounted below), and hot-reloaded —
|
|
# no Traefik restart needed to add, remove, or rotate a user. Shape is
|
|
# fixed by users_file.go: {"users":[{"name","group","apiKey","limits"}]}.
|
|
# limits is optional per user; omit it to inherit the user's group limits.
|
|
users.json: |
|
|
{
|
|
"users": [
|
|
{"name": "alice", "group": "engineering", "apiKey": "sk-llmgw-alice-CHANGE-ME"},
|
|
{"name": "bob", "group": "general", "apiKey": "sk-llmgw-bob-CHANGE-ME"}
|
|
]
|
|
}
|
|
|
|
---
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: Middleware
|
|
metadata:
|
|
name: llmgateway
|
|
namespace: traefik
|
|
spec:
|
|
plugin:
|
|
llmgateway:
|
|
providers:
|
|
openai:
|
|
type: openai
|
|
apiKey: "env:OPENAI_API_KEY"
|
|
discovery: true
|
|
discoveryInterval: 1h
|
|
anthropic:
|
|
type: anthropic
|
|
apiKey: "env:ANTHROPIC_API_KEY"
|
|
models: ["claude-opus-4-5", "claude-sonnet-4-5", "claude-haiku-4-5"]
|
|
gemini:
|
|
type: gemini
|
|
apiKey: "env:GEMINI_API_KEY"
|
|
models: ["gemini-2.5-pro", "gemini-2.5-flash"]
|
|
groups:
|
|
engineering:
|
|
# Empty lists, not omitted fields, so this block is a real map
|
|
# entry (see the empty-map gotcha note above) — every provider
|
|
# and model stays allowed either way (empty = all).
|
|
providers: []
|
|
models: []
|
|
limits:
|
|
requestsPerMinute: 60
|
|
requestsPerDay: 5000
|
|
tokensPerDay: 1000000
|
|
tokensPerMonth: 20000000
|
|
costPerDayUSD: 25.0
|
|
costPerMonthUSD: 250.0
|
|
mcpServers: []
|
|
agents: []
|
|
general:
|
|
providers: ["openai"]
|
|
models: ["gpt-5-mini", "gpt-4o"]
|
|
limits:
|
|
requestsPerMinute: 10
|
|
requestsPerDay: 500
|
|
users:
|
|
file: /secrets/llmgw-users/users.json
|
|
redis:
|
|
address: redis.shared-resources.svc:6379
|
|
password: "env:REDIS_PASSWORD"
|
|
failOpen: true
|
|
pricing:
|
|
gpt-5-mini: {inputPerM: 0.25, outputPerM: 2.0}
|
|
mcpServers:
|
|
github:
|
|
url: http://mcp-github.tools.svc:8080
|
|
agents:
|
|
summarizer:
|
|
url: http://summarizer.agents.svc:8080
|
|
passthroughUnknown: false
|
|
|
|
---
|
|
# Traefik requires a real backing Service for a router even though this
|
|
# middleware short-circuits every path it recognizes; whoami (the same
|
|
# image already exercised in ../examples/docker-compose.yml) is a small,
|
|
# always-up dummy, only ever reached via passthroughUnknown (default
|
|
# false, which 404s instead). Any always-up internal service works here.
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: llmgateway-dummy-backend
|
|
namespace: traefik
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: llmgateway-dummy-backend
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: llmgateway-dummy-backend
|
|
spec:
|
|
containers:
|
|
- name: whoami
|
|
image: traefik/whoami:v1.10
|
|
ports:
|
|
- containerPort: 80
|
|
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: llmgateway-dummy-backend
|
|
namespace: traefik
|
|
spec:
|
|
selector:
|
|
app: llmgateway-dummy-backend
|
|
ports:
|
|
- port: 80
|
|
targetPort: 80
|
|
|
|
---
|
|
apiVersion: traefik.io/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: llmgateway
|
|
namespace: traefik
|
|
spec:
|
|
entryPoints:
|
|
- websecure
|
|
routes:
|
|
- match: Host(`llm.example.com`)
|
|
kind: Rule
|
|
middlewares:
|
|
- name: llmgateway
|
|
services:
|
|
- name: llmgateway-dummy-backend
|
|
port: 80
|
|
|
|
---
|
|
# Helm values snippet (traefik/traefik-helm-chart) — illustrative, verify
|
|
# field names against the chart version you deploy. Two things are
|
|
# required beyond a stock install:
|
|
#
|
|
# 1. Load the plugin. Either the catalog form (once this plugin is listed
|
|
# in the Traefik Plugin Catalog, see ../.traefik.yml) or localPlugins,
|
|
# mounting this repository's source into the Traefik pod the same way
|
|
# ../examples/docker-compose.yml does for its local demo:
|
|
#
|
|
# experimental:
|
|
# plugins:
|
|
# llmgateway:
|
|
# moduleName: "github.com/lukaszraczylo/traefik-llmgateway"
|
|
# version: "v0.1.0"
|
|
#
|
|
# 2. Provider keys as env vars, sourced from Secrets — never inlined into
|
|
# the Middleware CR above, which is why every apiKey field there reads
|
|
# "env:VAR_NAME" instead of a literal:
|
|
#
|
|
# env:
|
|
# - name: OPENAI_API_KEY
|
|
# valueFrom: {secretKeyRef: {name: llmgw-provider-keys, key: openai}}
|
|
# - name: ANTHROPIC_API_KEY
|
|
# valueFrom: {secretKeyRef: {name: llmgw-provider-keys, key: anthropic}}
|
|
# - name: GEMINI_API_KEY
|
|
# valueFrom: {secretKeyRef: {name: llmgw-provider-keys, key: gemini}}
|
|
# - name: REDIS_PASSWORD
|
|
# valueFrom: {secretKeyRef: {name: llmgw-redis, key: password}}
|
|
#
|
|
# additionalVolumes:
|
|
# - name: llmgw-users
|
|
# secret: {secretName: llmgw-users}
|
|
# deployment:
|
|
# additionalVolumeMounts:
|
|
# - name: llmgw-users
|
|
# mountPath: /secrets/llmgw-users
|
|
# readOnly: true
|