mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
docs(readme): document model metadata (review SHOULD-7)
New 'Model metadata' section: the context/cost layering order, the modelMeta config override example (mirrors the operator's own cluster snippet), alias inheritance, the ":free" naming convention, the metadataPath discovery capture (including the loaded-vs-max-context volatility note — a discovery refresh can legitimately report a different context window across reloads), the pricing-sync/built-in table workflow, and the two GET /v1/models extension fields with a worked example. Config reference tables gain a ModelMetaConfig subsection and modelMeta/metadataPath rows on Config/ProviderConfig; the Admin section notes the per-provider and per-alias modelMeta fields on GET /admin/api/overview.
This commit is contained in:
@@ -183,6 +183,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
|
|||||||
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
|
| `cache` | `CacheConfig` | `{}` (disabled) | Opt-in Redis-backed response cache for unified non-streaming chat/embeddings — see [Caching](#caching). Omitted or `enabled: false` means no caching, byte-identical to a gateway built before this field existed. |
|
||||||
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
|
| `admin` | `*AdminConfig` | `nil` (disabled) | Read-only admin dashboard — see [Admin](#admin). `nil` or `enabled: false` means the `/admin*` routes are not registered at all. |
|
||||||
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
|
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
|
||||||
|
| `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. |
|
||||||
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
|
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
|
||||||
|
|
||||||
> **Upgrading from v0.1**: `POST /v1/images/generations`, `POST
|
> **Upgrading from v0.1**: `POST /v1/images/generations`, `POST
|
||||||
@@ -214,6 +215,7 @@ also a construction error, never a panic (`providers.go`, `mcp_a2a.go`,
|
|||||||
| `discoveryInterval` | `string` (Go duration) | `1h` | Only meaningful when `discovery: true`. Invalid duration string is a construction error. |
|
| `discoveryInterval` | `string` (Go duration) | `1h` | Only meaningful when `discovery: true`. Invalid duration string is a construction error. |
|
||||||
| `models` | `[]string` | `[]` | Explicit model ids this provider serves. Combined with any discovered ids. |
|
| `models` | `[]string` | `[]` | Explicit model ids this provider serves. Combined with any discovered ids. |
|
||||||
| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. |
|
| `discovery` | `bool` | `false` | Pull the provider's own model-listing endpoint at startup and on `discoveryInterval`. A discovery failure is logged and non-fatal; construction still succeeds on `models` alone. |
|
||||||
|
| `metadataPath` | `string` | `""` (no metadata capture) | Only read by `openai`-type providers. A second endpoint, fetched alongside `discovery`, that reports per-model context length — for example `/api/v0/models` on an LM Studio server. Must start with `/` when set (checked at construction). See [Model metadata](#model-metadata). |
|
||||||
|
|
||||||
### `GroupConfig`
|
### `GroupConfig`
|
||||||
|
|
||||||
@@ -307,6 +309,20 @@ own is governed purely by their group's — see
|
|||||||
| `inputPerM` | `float64` | USD per 1,000,000 input/prompt tokens. |
|
| `inputPerM` | `float64` | USD per 1,000,000 input/prompt tokens. |
|
||||||
| `outputPerM` | `float64` | USD per 1,000,000 output/completion tokens. |
|
| `outputPerM` | `float64` | USD per 1,000,000 output/completion tokens. |
|
||||||
|
|
||||||
|
### `ModelMetaConfig` (a `modelMeta` entry)
|
||||||
|
|
||||||
|
Overrides one model's **context window and per-token cost for display**
|
||||||
|
— see [Model metadata](#model-metadata). Distinct from `ModelPricing`
|
||||||
|
above, which drives request-cost **accounting**: `modelMeta` never
|
||||||
|
changes what a request is billed.
|
||||||
|
|
||||||
|
| Field | Type | Default | Semantics |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `contextTokens` | `int` | `0` (not overridden) | A value of `0` falls through to the next layer (discovery, then the built-in table) instead of overriding with an unknown value. Must not be negative. |
|
||||||
|
| `inputCostPerMTokMicroUsd` | `int64` | `0` (not overridden) | Input cost per 1,000,000 tokens, in micro-USD (1 USD = 1,000,000). `0` falls through to the next layer unless `free` is set. Must not be negative. |
|
||||||
|
| `outputCostPerMTokMicroUsd` | `int64` | `0` (not overridden) | Output cost per 1,000,000 tokens, in micro-USD. Same fallthrough rule as `inputCostPerMTokMicroUsd`. |
|
||||||
|
| `free` | `bool` | `false` | `true` sets both cost fields to an explicit, known zero — distinct from leaving them at `0`, which means "not overridden here, keep looking." Rejected at construction when combined with a non-zero cost field (contradictory). |
|
||||||
|
|
||||||
### `TargetConfig` (an `mcpServers` entry)
|
### `TargetConfig` (an `mcpServers` entry)
|
||||||
|
|
||||||
| Field | Type | Semantics |
|
| Field | Type | Semantics |
|
||||||
@@ -472,6 +488,130 @@ http:
|
|||||||
array (`{alias, target}`, sorted by alias) — see [Admin](#admin). No
|
array (`{alias, target}`, sorted by alias) — see [Admin](#admin). No
|
||||||
secrets involved.
|
secrets involved.
|
||||||
|
|
||||||
|
## Model metadata
|
||||||
|
|
||||||
|
Per-model context window and per-token cost, resolved for display —
|
||||||
|
`GET /v1/models` and the admin dashboard show what an operator or client
|
||||||
|
actually knows about a model, without inventing a number nobody
|
||||||
|
configured or reported. `modelMeta` (config overrides) is a **separate**
|
||||||
|
config surface from `pricing` (request-cost accounting, see
|
||||||
|
[`ModelPricing`](#modelpricing)) — the two never affect each other.
|
||||||
|
|
||||||
|
Context and cost each resolve through their own layer stack
|
||||||
|
(`modelmeta.go`'s `resolveModelMeta`), most specific first:
|
||||||
|
|
||||||
|
1. **Context**: a `modelMeta` config override → a provider's own
|
||||||
|
discovery-captured context (`metadataPath`, below) → the built-in
|
||||||
|
table → unknown.
|
||||||
|
2. **Cost**: a `modelMeta` config override (or its `free: true`) → a
|
||||||
|
`":free"`-suffixed model id (see the free-tier note below) → the
|
||||||
|
built-in table → unknown.
|
||||||
|
|
||||||
|
A field genuinely unknown at every layer stays unknown — never invented
|
||||||
|
— and is omitted from `GET /v1/models`, not sent as a misleading zero.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
http:
|
||||||
|
middlewares:
|
||||||
|
llmgateway:
|
||||||
|
plugin:
|
||||||
|
llmgateway:
|
||||||
|
# ... providers, groups, users unchanged ...
|
||||||
|
modelMeta:
|
||||||
|
# Exact "provider/model" key.
|
||||||
|
"uni/deepseek-v4-flash-0731":
|
||||||
|
contextTokens: 524288
|
||||||
|
free: true
|
||||||
|
# Bare model id — applies wherever that bare id resolves.
|
||||||
|
"ornith-397b/qwen3.6-27b/btl-4":
|
||||||
|
contextTokens: 262144
|
||||||
|
free: true
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Alias inheritance**: a [model alias](#model-aliases) inherits its
|
||||||
|
target's fully resolved metadata, unless the alias id itself has its
|
||||||
|
own `modelMeta` entry — that entry's fields win, independently per
|
||||||
|
field (setting only `contextTokens` on the alias still inherits the
|
||||||
|
target's cost).
|
||||||
|
- **Free-tier naming convention**: a model id ending in `:free` (a real
|
||||||
|
provider naming convention some upstreams use for their own free-tier
|
||||||
|
models) resolves to a known, zero cost even with no `modelMeta` entry
|
||||||
|
and no built-in table row for that exact id — ranked above the
|
||||||
|
built-in table, below an explicit `modelMeta` override. Context is
|
||||||
|
unaffected by this rule; it still resolves through the normal layers
|
||||||
|
above.
|
||||||
|
|
||||||
|
### Discovery-captured context (`metadataPath`)
|
||||||
|
|
||||||
|
Set `metadataPath` on an `openai`-type provider to fetch a second
|
||||||
|
endpoint alongside `discovery`, one that reports per-model context
|
||||||
|
length — LM Studio's own, non-OpenAI-compatible endpoint is the
|
||||||
|
supported shape:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
providers:
|
||||||
|
macstudio:
|
||||||
|
type: openai
|
||||||
|
baseUrl: "http://macstudio.local:1234"
|
||||||
|
discovery: true
|
||||||
|
metadataPath: "/api/v0/models"
|
||||||
|
```
|
||||||
|
|
||||||
|
The response's `loaded_context_length` field is preferred over
|
||||||
|
`max_context_length` when both are present — it reflects the context
|
||||||
|
actually usable right now, which an operator can configure smaller than
|
||||||
|
the model's own maximum under VRAM-constrained runtime settings. This
|
||||||
|
means the reported context window can **change** across a discovery
|
||||||
|
refresh (a model reloaded with a different context setting reports a
|
||||||
|
different value) — it is not a fixed, one-time fact the way the built-in
|
||||||
|
table's numbers are. A failed or absent metadata fetch is always
|
||||||
|
non-fatal: no metadata is captured for that refresh, and `discovery`
|
||||||
|
itself is unaffected.
|
||||||
|
|
||||||
|
### Built-in table
|
||||||
|
|
||||||
|
`pricing_data_gen.go` is a generated, checked-in table synced from
|
||||||
|
[LiteLLM's own pricing
|
||||||
|
data](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json),
|
||||||
|
pruned to the provider types this gateway can meaningfully match
|
||||||
|
(`openai`, `anthropic`, `deepseek`, `xai`, `minimax`, `cerebras`,
|
||||||
|
`dashscope`, `moonshot`) and to chat/embedding models only — the full
|
||||||
|
upstream table is over 3,000 entries, most of them for providers this
|
||||||
|
gateway has no adapter for.
|
||||||
|
|
||||||
|
Run `make pricing-sync` to regenerate it against the current upstream
|
||||||
|
data. This fetches over the network and writes `pricing_data_gen.go` at
|
||||||
|
build time only — the running plugin never fetches anything itself. The
|
||||||
|
generated file's header records a content digest of the fetched bytes,
|
||||||
|
not a date, so a re-run against unchanged upstream data reproduces an
|
||||||
|
identical file with nothing to review.
|
||||||
|
|
||||||
|
### `GET /v1/models` extension fields
|
||||||
|
|
||||||
|
Two OpenAI-compat-safe extra fields on each model object, present only
|
||||||
|
when known:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "openai/gpt-5",
|
||||||
|
"object": "model",
|
||||||
|
"owned_by": "openai",
|
||||||
|
"context_window": 400000,
|
||||||
|
"pricing": {"input_per_mtok_usd": 1.25, "output_per_mtok_usd": 10}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `context_window` — an integer token count, omitted when unknown.
|
||||||
|
- `pricing` — `{input_per_mtok_usd, output_per_mtok_usd}`, both floats
|
||||||
|
in USD per 1,000,000 tokens. Omitted when unknown; present with `0`
|
||||||
|
values for an explicitly free model — a known zero, not an absent
|
||||||
|
field.
|
||||||
|
|
||||||
|
The admin dashboard's `GET /admin/api/overview` exposes the identical
|
||||||
|
resolved values per provider (`providers[].modelMeta`, keyed by upstream
|
||||||
|
model id) and per alias (`aliases[].modelMeta`) — see
|
||||||
|
[Admin](#admin).
|
||||||
|
|
||||||
## Limits and accounting
|
## Limits and accounting
|
||||||
|
|
||||||
- Counters use **fixed windows**, keyed by UTC calendar boundaries (the
|
- Counters use **fixed windows**, keyed by UTC calendar boundaries (the
|
||||||
@@ -842,10 +982,13 @@ or in CI.
|
|||||||
otherwise show the identical zero `lastRefresh`), `attemptsDay`/
|
otherwise show the identical zero `lastRefresh`), `attemptsDay`/
|
||||||
`failuresDay`/`attemptsMinute`/`failuresMinute` (the provider's own
|
`failuresDay`/`attemptsMinute`/`failuresMinute` (the provider's own
|
||||||
current-window success-rate counters — see [Limits and
|
current-window success-rate counters — see [Limits and
|
||||||
accounting](#limits-and-accounting)), and `modelRates` (the identical
|
accounting](#limits-and-accounting)), `modelRates` (the identical
|
||||||
`attemptsDay`/`failuresDay` pair per known model, keyed by model id — no
|
`attemptsDay`/`failuresDay` pair per known model, keyed by model id — no
|
||||||
per-model minute figures, since nothing displays them; the dashboard's
|
per-model minute figures, since nothing displays them; the dashboard's
|
||||||
per-model badge falls back to a day-window-only detail instead). `usage`
|
per-model badge falls back to a day-window-only detail instead), and
|
||||||
|
`modelMeta` (the resolved context window and cost per known model —
|
||||||
|
see [Model metadata](#model-metadata)); each alias entry carries its
|
||||||
|
own `modelMeta` too. `usage`
|
||||||
returns every user's and every group's
|
returns every user's and every group's
|
||||||
current-window counter values — `requestsPerMinute`, `requestsPerDay`,
|
current-window counter values — `requestsPerMinute`, `requestsPerDay`,
|
||||||
`tokensInPerDay`/`tokensOutPerDay`, `tokensInPerMonth`/
|
`tokensInPerDay`/`tokensOutPerDay`, `tokensInPerMonth`/
|
||||||
|
|||||||
Reference in New Issue
Block a user