mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
docs: kubernetes hot-reload guidance
README Kubernetes section: what a Middleware CR edit actually does (Traefik builds a new instance, warm-fills discovery, swaps it in; requests in flight finish on the old one), what survives a rebuild (Redis-backed counters and cache — windowKey's key never carries instance identity) vs what resets (in-memory fallback counters, down-latches, the discovery cache), why new models via discovery need no config edit, why the users Secret must be a whole-volume mount (subPath never receives a kubelet update) and how the plugin's own 5s mtime poll picks up the resulting symlink swap, and why a file-provider dynamic config needs directory: rather than filename: in Kubernetes (ConfigMap symlink swap) and under Docker Desktop (VirtioFS fsnotify) alike. examples/kubernetes.yaml: annotated the existing (already-correct, no-subPath) users Secret volume mount explaining why subPath would silently break Secret updates, so a future edit doesn't 'simplify' it into one.
This commit is contained in:
@@ -880,6 +880,60 @@ The short version:
|
|||||||
rather than standing up a dedicated instance for this plugin — the
|
rather than standing up a dedicated instance for this plugin — the
|
||||||
distributed limiter is a client, not a datastore.
|
distributed limiter is a client, not a datastore.
|
||||||
|
|
||||||
|
### Config hot-reload
|
||||||
|
|
||||||
|
Editing the `Middleware` custom resource applies with no pod restart and no
|
||||||
|
manual reload step. The Kubernetes CRD provider watches `Middleware`
|
||||||
|
resources natively: on a change, Traefik builds a whole new plugin
|
||||||
|
instance from the updated config, lets it warm-fill discovery for every
|
||||||
|
provider with `discovery: true` (see [Model routing](#model-routing)), then
|
||||||
|
swaps it in for new requests. Requests already in flight keep running
|
||||||
|
against the old instance until they finish; only new requests reach the
|
||||||
|
new one.
|
||||||
|
|
||||||
|
A rebuilt instance keeps some state and resets the rest:
|
||||||
|
|
||||||
|
- **Survives a rebuild**: request/token/cost counters and cached responses,
|
||||||
|
both stored in Redis. A counter's key comes from `windowKey`'s
|
||||||
|
`(kind, id, metric, window, time)` alone (see
|
||||||
|
[Limits and accounting](#limits-and-accounting)) — never from anything
|
||||||
|
specific to one Gateway instance — so a new instance reads the exact
|
||||||
|
counters the old one wrote. The response cache's keys work the same way.
|
||||||
|
- **Resets on a rebuild**: the in-process fallback counters used only when
|
||||||
|
`redis` is not configured, the limiter's and cache's Redis-outage
|
||||||
|
down-latches, and the model registry's discovery cache — a new instance
|
||||||
|
always runs its own full warm-fill before serving, rather than
|
||||||
|
inheriting the old instance's listing.
|
||||||
|
|
||||||
|
Two practical implications follow:
|
||||||
|
|
||||||
|
- **New models need no config edit at all.** A provider's own
|
||||||
|
`discoveryInterval` polls the upstream directly on its own schedule;
|
||||||
|
adding a model upstream needs neither a `Middleware` change nor a
|
||||||
|
rebuild.
|
||||||
|
- **The users `Secret` must be a whole-volume mount, never `subPath`.** A
|
||||||
|
`subPath` volume mount binds once at pod start — the kubelet never
|
||||||
|
updates it again, so a `Secret` change behind a `subPath` mount stays
|
||||||
|
invisible to the running pod for its whole lifetime (a documented
|
||||||
|
kubelet limitation, not a plugin one). Mount the whole `Secret` as its
|
||||||
|
own volume instead, exactly as `examples/kubernetes.yaml`'s
|
||||||
|
`additionalVolumeMounts` snippet does: a whole-volume mount updates by
|
||||||
|
the kubelet swapping a symlink, and the plugin's own 5-second mtime poll
|
||||||
|
(`reloadEvery`, `users_file.go`) picks that swap up with no pod restart.
|
||||||
|
|
||||||
|
If you run Traefik's file provider in Kubernetes instead of the CRD
|
||||||
|
provider — a `ConfigMap`-mounted dynamic config file rather than
|
||||||
|
`Middleware` resources — point it at the mount with `directory:`, never
|
||||||
|
`filename:`. A `ConfigMap` update reaches the pod the same way a `Secret`
|
||||||
|
does: the kubelet swaps a symlink for the whole mounted directory, which
|
||||||
|
Traefik's single-file `filename:` watch never observes; `directory:`
|
||||||
|
watches the directory itself and picks the swap up. The same
|
||||||
|
`directory:` requirement also holds for local/Docker Desktop development
|
||||||
|
against this plugin: a single file's fsnotify watch does not fire
|
||||||
|
reliably over Docker Desktop's VirtioFS bind mount, which is why this
|
||||||
|
repository's own integration stack (`integration/traefik/traefik.yml`)
|
||||||
|
uses `directory:` rather than `filename:`.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
@@ -234,6 +234,14 @@ spec:
|
|||||||
# secret: {secretName: llmgw-users}
|
# secret: {secretName: llmgw-users}
|
||||||
# deployment:
|
# deployment:
|
||||||
# additionalVolumeMounts:
|
# additionalVolumeMounts:
|
||||||
|
# # Whole-volume mount — no subPath — is required, not a style
|
||||||
|
# # choice: a subPath mount binds once at pod start and the
|
||||||
|
# # kubelet never updates it again, so editing the llmgw-users
|
||||||
|
# # Secret above would never reach this pod. A whole-volume mount
|
||||||
|
# # updates via the kubelet swapping a symlink for the whole
|
||||||
|
# # directory, which users.file's own 5-second mtime poll
|
||||||
|
# # (reloadEvery, users_file.go) picks up with no pod restart —
|
||||||
|
# # see ../README.md#config-hot-reload.
|
||||||
# - name: llmgw-users
|
# - name: llmgw-users
|
||||||
# mountPath: /secrets/llmgw-users
|
# mountPath: /secrets/llmgw-users
|
||||||
# readOnly: true
|
# readOnly: true
|
||||||
|
|||||||
Reference in New Issue
Block a user