mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
main
26
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
ce39b8eb37 |
fix(registry): retry failed startup discovery once in the background
A provider whose discovery failed during the warm fill at pod start went without discovered models for a full discovery interval, which is 1 hour by default. It also stayed that way until a request arrived after the interval. Production logs showed this for openai-audio and anthropic when outbound calls were slow at startup. A provider that fails the warm fill now gets one background retry after 10s. The retry has its own timeout and uses the same refresh bookkeeping. It is skipped when a refresh is already in flight or has finished since the warm fill, and it is not scheduled when the parent context was canceled. A recovery logs one INFO line. If the retry also fails, the existing breaker behaviour applies unchanged. The retry delay and the warm-fill timeout can be set per registry for tests. yaegi-check probes the retry under the interpreter. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
4c01c75276 |
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
b3b8ae6ad5 |
fix(security): harden retry, SSE, Redis and cost-budget paths
The gateway refuses a request for an unpriced model when the caller has a cost budget. An unpriced model cost 0, so the budget counter was never written and the budget could never fire. The new allowUnpricedWithCostBudget setting restores the old behaviour. The retry loop honours an upstream Retry-After of up to 30s and jitters its own computed backoff. SSE parsing bounds the total size of one event. The Redis client can use TLS. Provider base URLs and errors are scrubbed of credentials before they reach logs or the admin view. The gateway warns when failover and retry settings multiply into a large number of outbound calls. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv |
||
|
lukaszraczylo
|
72b0df5b0c |
feat(auth): let a user join several groups and hold a personal grant
A user entry can now list several groups in "groups", next to the existing "group" field. A user can also hold a personal grant with its own "providers" and optional "models" lists. Users with one group and no personal grant behave exactly as before. A request is allowed only when one grant allows the provider, the model and, for passthrough, the path. Grants never merge, so one grant cannot supply the provider while another supplies the model or the path. The personal grant uses the path list of the first group. A personal "models" list without "providers" is a configuration error. Each member group applies its own limits. Caching stays on unless a member group turns it off, and the shortest group TTL wins. The /v1/models cache uses a stable, length-prefixed key, so reloads do not grow it. Tests cover the grant union, cross-grant denials, passthrough path and model coupling, reloads and the cache key. The yaegi-check harness asserts the same cases under the interpreter. Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3 |
||
|
lukaszraczylo
|
8b49bbf2a7 |
feat(models): emit the context window under three client dialects
OpenAI's /v1/models schema carries no context field, so every client that wants one reads a vendor dialect, and a client whose dialect is absent does not ask — it substitutes its own default. GET /v1/models now emits the same resolved integer under "context_window", "context_length" (the OpenRouter spelling) and "max_model_len" (the vLLM spelling). All three are omitted together when the window is unknown, exactly as the single field was. Measured case: omp 18.1.15's openai-models-list discovery resolves `max_model_len ?? context_length ?? <bundled catalog> ?? 128000` and never reads context_window, so a 1048576-context local model reached the client as 128000, which also clamped its maxTokens. The metadata fetcher already parses all of these dialects on the inbound side (provider_openai.go), so this makes the outbound direction speak what the inbound one already understands. Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp |
||
|
lukaszraczylo
|
ffa27ce831 |
fix(registry): let an alias win its own /v1/models entry
resolve gives an exact alias match precedence over every discovered bare model id, but listFor did the opposite: it skipped the alias whenever a discovered model of the same bare id had already been listed. The catalog then advertised the shadowed provider as the owner, with that provider's metadata, for an id every request routed to the alias target instead. A live fleet shows the effect. The alias "deepseek-v4-flash-vision-exp" targets "gx10/current" (context 1048576), while a cloud provider discovers a model of the identical bare id. GET /v1/models returned owned_by "deepseek" and no context_window at all, so clients fell back to their own 128k default while the request ran against a 1M-context local model. listFor now resolves every usable alias up front and suppresses the bare entry of any id an alias shadows. The shadowed model stays listed in "provider/id" form, which is how a client must address it once the alias owns the bare id, so no model disappears from the catalog. Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp |
||
|
lukaszraczylo
|
0c6d52da9a |
fix(registry): resolve bare model ids against the caller's own group
bareWinner picked the first provider in global sorted order that served a bare model id, ignoring which providers the caller's group could actually use. resolveAgainst then denied the request whenever that provider fell outside the group, even when another provider serving the identical id was already permitted. A caller entitled to a model could receive errModelDenied purely because an unrelated, invisible provider sorted first alphabetically. bareWinner now takes the group and prefers the sorted-first provider it may use, falling back to the global sorted-first provider only when the group is authorized for none of them. That fallback keeps a genuinely unauthorized request returning errModelDenied, not the wrong errModelUnknown a "no visible owner" result would produce. resolve, resolveAliasTarget, and the alias-shadow collision check all route through this corrected selection. listFor now computes each bare id's owner from the group's own visible providers and adds "provider/model" disambiguating entries only when the group can see two or more owners of that id. A provider invisible to the group never appears in that group's catalog, bare or prefixed. warnCollisionOnce keeps logging the global collision unchanged, since a provider collision is a fact about the fleet's configuration that every operator must see regardless of which group lists models first. Its message now says winner is the global default, because a group's own request can resolve to a different provider than the one logged. Adds registry_test.go coverage for the production repro, the both-allowed and neither-allowed non-regression cases, the listing behavior, and the alias path. Adds a yaegi-check probe proving the fix resolves and serves under the real interpreter, and updates one existing listFor test that asserted the pre-fix prefixed-form behavior. |
||
|
lukaszraczylo
|
a18580e930 |
fix(registry): make the breaker actually suppress re-probing
Address the design finding and remaining defects from round 3 of adversarial review. Raise defaultBreakerMaxOpenDuration from 30m to 6h. Round 2 fixed the breaker so it never retries more than plain interval throttling, but left it cadence-neutral: with maxOpenDuration shorter than the 1h default interval, the interval always dominated and backoff never suppressed anything. Measured over 24h with stock config, admits equaled the interval-only baseline exactly. At 6h, a permanently broken provider now backs off 1h, 2h, 4h, 6h, 6h, roughly 5 attempts a day instead of 24. The 24h cadence test now asserts admits are strictly fewer than baseline, not merely no worse. Correct the emitted openUntil to account for the interval floor. Because the interval gate applies unconditionally, the real next probe time is max(openUntil, lastRefresh+interval), not the raw backoff value alone. On pure defaults a fresh trip set the raw value to +1m while the real next probe was at +1h, a 59 minute skew the dashboard rendered as a countdown that reached zero and then silently vanished. snapshot now emits the corrected value while open; closed and half-open are unaffected. Revert a canceled half-open probe to open instead of leaving it at half-open. A half-open breaker reports healthy by design, so a canceled probe previously left a still-broken provider reading healthy until the next attempt happened to land. Reverting costs nothing extra, since the interval gate already governs when that next attempt is admitted either way. Rewrite the discovery-suppression claims in doc comments: the earlier wording claimed re-probing stops "every single interval forever" under all conditions, which the interval-floor fix from round 2 already made false, and which contradicted this file's own surrounding explanation. Both now state plainly that suppression only occurs once backoff exceeds discoveryInterval. Add a sentence to recordHealthLocked's doc comment stating that context.DeadlineExceeded counts as a plain failure, matching recordProviderAttempt's SHOULD-1 ruling, and a named test proving it. Tests: two new snapshot tests pin the openUntil correction in both directions. The cadence test's baseline comparison is now strict. A new test proves the zero value truly disables the breaker rather than tripping on the first failure. A new test proves a canceled context never gets misclassified as a failure, completing the bidirectional check the existing success-direction test already had. The concurrent test now asserts backoff stays within its ceiling and health stays a valid state on every iteration, rather than only checking for panics and data races. |
||
|
lukaszraczylo
|
24167b16f6 |
fix(registry): correct breaker cadence, classification, and cancellation
Address five findings from adversarial review of the discovery circuit breaker. All are behavioral corrections to registry.go; none touch the request path. Interval floor (blocking). tryBeginRefresh let an open breaker's own backoff window replace the plain discoveryInterval gate instead of adding to it. Since defaultBreakerMaxOpenDuration (30m) is shorter than defaultDiscoveryInterval (1h), a permanently broken provider was probed roughly twice as often as before this feature existed. The interval gate now applies unconditionally, closed or open alike, so backoff can only ever lengthen the retry cadence, never shorten it. Explicit failure classification. recordHealthLocked reused isTransient's retry-shaped semantics wholesale, including its exclusion of errRequestBuildFailed. A provider whose baseURL is malformed produced that exact error on every discovery cycle and never opened its breaker, because the old code routed it to a successful outcome. Health classification is now explicit: any non-nil, non-canceled error counts as a failure. Neutral cancellation. A context.Canceled outcome no longer counts as success or failure. Treating it as success previously closed an already-open breaker and wiped its backoff whenever a half-open probe was interrupted at shutdown, reporting a still-broken provider healthy. Zero-value safety. A providerState assembled without going through newModelRegistry now behaves like plain interval-only gating instead of opening on the first failure with a zero backoff, matching the type's own documented claim. Backoff overflow ceiling. Repeated doubling with no ceiling on maxOpenDuration could wrap the backoff negative after roughly 28 probes, silently disabling it. validateBreakerConfig now caps maxOpenDuration at 24 hours. Also renames modelRegistry.providerUsable and providerState.usable to discoveryHealthy, and documents that the signal reflects the discovery endpoint only. It must never be the sole reason to refuse routing to a provider whose request path works fine. feat/failover should combine it with a separate request-path signal fed by recordProviderAttempt. Tests: a 24-hour simulated cadence test proves the fixed gate admits no more attempts than plain interval throttling did before this feature. A concurrent test exercises the breaker fields under -race for the first time. Two tests replace a vacuous predecessor and distinguish neutral cancellation from treating it as success. One test proves errRequestBuildFailed opens the breaker. One test proves the zero value never opens. A rewritten healthy-provider test asserts behavior instead of reading the internal openUntil field directly. |
||
|
lukaszraczylo
|
e9ce6b873d |
feat(registry): add per-provider discovery circuit breaker
Add a three-state circuit breaker to providerState: closed, open, and half-open. It is driven only by discovery-refresh outcomes, never by request-path traffic. A provider whose refresh fails breakerThreshold times in a row opens the breaker and backs off before its next half-open probe, instead of retrying every discovery interval forever. A failed probe reopens the breaker with a doubled backoff, capped at breakerOpenMax. A successful probe closes the breaker again, so a recovered provider never stays dead. The failure classification reuses limits.go's isTransient and isDeadlineExceeded, the same rule recordProviderAttempt already applies to request-path attempts. It does not add a second classification. Add Config.Breaker (BreakerConfig), validated and defaulted by validateBreakerConfig. Every default preserves current behavior for a provider that never fails: a closed breaker never changes the existing interval-based refresh gating. Export modelRegistry.providerUsable(name) as the queryable health predicate. A sibling branch (feat/failover) will use it to skip unusable providers before routing. This commit does not touch the request path or routes_unified.go. |
||
|
lukaszraczylo
|
0346c3ed39 |
fix(registry): log model-id collisions at WARN, not ERROR
A model id served by two configured providers is resolved deterministically by the registry, the winner is named in the message, and no request is affected. Logging it at ERROR put 144 ERROR lines in a healthy pod's boot log on a cluster with overlapping openai, openai-audio, copilot and macstudio catalogs, which is exactly the noise that hides a real failure. Adds Gateway.warnf beside logf and errorf, and a warn field on modelRegistry carrying it. The field is set by the production caller after newModelRegistry returns rather than added as a fourth constructor parameter: 45 call sites pass a logger today, almost all of them tests passing a no-op, and threading a parameter through all of them would bury a one-line severity change in unrelated churn. A nil warn falls back to log, so a registry built without one keeps its current behaviour instead of panicking or dropping the line. registry.warnf takes an already-formatted string so both call shapes stay single-variadic-argument. Yaegi v0.16.1's CFG builder panics on a call to a struct FIELD of variadic func type with two or more variadic arguments, which is the restriction the log field beside it already documents. Verified under the real interpreter, not only compiled: yaegi-check drives a genuine two-provider collision and now prints WARN. The severity test fails if the call site reverts to log, so the level cannot drift back unnoticed. Only this message moves. Genuine failures — discovery errors, metadata capture panics, encoding failures — stay at ERROR. |
||
|
lukaszraczylo
|
bf33732123 |
perf(registry): cache the encoded /v1/models response per group
GET /v1/models cost 15.9ms, 21.3MB, 552,884 allocs per call at the
live 1,033-model catalog (registry.go's listFor plus routes_unified.go's
json.Encoder), uncached — and OpenAI SDK clients poll this endpoint per
session, so an unchanged catalog was being re-walked and re-marshaled
on every request.
Add modelRegistry.modelsJSON(grp), caching the fully encoded
{object:list,data:[...]} body per (*group, modelsGen). modelsGen
is an atomic counter bumped by a new modelRegistry.finishRefresh
wrapper around providerState.finishRefresh — the only two places that
ever mutate a provider's discovered set or discovery-captured metadata
(warmFill, refreshProvider) already funnel through providerState.
finishRefresh once per refresh cycle, so wrapping it is sufficient to
invalidate the cache on every possible change, including a
metadata-only update on an otherwise-failed refresh (captureModelMetadata
runs unconditionally alongside finishRefresh in both call sites).
Bumped unconditionally rather than only on success, since a failed
listModels fetch can still pair with a successful metadata capture.
Cached per *group pointer: authStore builds every group once, at
construction, and never rebuilds or replaces that map afterward — a
config reload constructs an entirely new Gateway (and modelRegistry)
with its own empty cache, so a *group key can never alias a different
group's catalog either within one registry's lifetime or across a
reload.
routes_unified.go's handleModels call site shrinks to a single write of
modelsJSON's result, replacing the previous inline listFor+json.Encode;
routes_unified.go is otherwise untouched (owned by a sibling change in
this same effort).
|
||
|
lukaszraczylo
|
1875ee7b8b |
fix(registry): defer context cancels on the refresh panic path
A panic in listModels skipped the inline cancel, holding the timeout timer until it fired. Review follow-up from the v0.2.3 final verdict. |
||
|
lukaszraczylo
|
82dcaa5309 |
fix(registry): metadata capture gets its own timeout + panic isolation
warmFill/refreshProvider used to reuse listModels' own context for the subsequent captureModelMetadata call — a listModels attempt that took real, non-trivial time (or timed out outright) guaranteed the metadata fetch an already-spent, near-expired context, producing a spurious deadline error and log line every warm fill or refresh for an otherwise healthy provider. Both call sites now give captureModelMetadata its own fresh context.WithTimeout budget, computed after listModels returns. (review SHOULD-4) captureModelMetadata also gains its own deferred recover, isolating a panic inside fetchModelMetadata (or a future modelMetadataFetcher implementation) from refreshProvider's OUTER recover — which would otherwise overwrite the (already-nil, already-correct) err variable from a successful listModels call and cause finishRefresh to discard a freshly fetched, genuinely good model list over an unrelated bug in the metadata path. "Metadata must never fail discovery" is now enforced on the panic path, not just the plain-error-return one already handled. (review SHOULD-5) New regression tests: a fresh-timeout-budget check (a slow simulated listModels must not shrink the metadata fetch's own deadline) and a panic-in-fetchModelMetadata check (discovery must still succeed and record both models). |
||
|
lukaszraczylo
|
0c61296c73 |
feat(registry): model metadata capture and resolution
Config gets two new surfaces (llmgateway.go): Config.ModelMeta (map[string]*ModelMetaConfig, keyed by exact "provider/model" or a bare model id) for operator overrides, and ProviderConfig.MetadataPath for openai-type providers whose discovery should also capture per-model context (LM Studio's native /api/v0/models shape). modelmeta.go adds the layering itself: resolveModelMeta applies config override > discovered context > builtin table > absent for context, and config override/free > builtin > absent for cost (discovery never reports cost). resolveAliasModelMeta implements alias inheritance: an alias inherits its target's fully resolved metadata unless the alias itself has its own modelMeta entry, applied per field. validateModelMeta rejects negative values and a free:true/non-zero-cost contradiction at construction. registry.go wires this into the request path: providerState gains a discoveredContext map (stale-while-error, mirroring discovered's own convention) with contextFor/setDiscoveredContext; a new optional modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted, documented as yaegi-safe against the Trap-4 restriction) rather than added to providerAdapter, lets warmFill/refreshProvider capture metadata alongside listModels without touching anthropic/gemini adapters at all. modelObject/listFor now attach resolved context_window/pricing to every GET /v1/models entry, including aliases (resolveMetaFor/ resolveMetaForAlias/resolveMetaForAliasName). provider_openai.go implements modelMetadataFetcher: fetchModelMetadata reads ProviderConfig.MetadataPath, preferring loaded_context_length over max_context_length when the LM Studio response reports both. providers.go wires MetadataPath onto the adapter in buildAdapters. tools/yaegi-check/main.go adds one cheap modelMeta config-override assertion to the existing attempt-accounting harness, proving resolveModelMeta's config-override layer and the context_window/pricing extension fields both run correctly under the real Yaegi interpreter. |
||
|
lukaszraczylo
|
bf23b41568 |
feat(admin): rates + discovery flag in API
Extend GET /admin/api/overview's existing providers payload rather than adding a sibling endpoint: the route already returns one providers array per request on the dashboard's 5s poll, is already admin-gated and stats-excluded, and the new fields are one more batched limiter read sharing that same request — a second endpoint would duplicate all three properties for no benefit. adminProviderView gains AttemptsDay/FailuresDay/AttemptsMinute/ FailuresMinute (the provider's own current-window counters) and ModelRates (a per-model breakdown, keyed by upstream model id, via the new adminModelRateView) — Feature A. buildAdminOverview builds every provider's and every (provider, model) pair's limitScope in two passes and reads them all through limiter.providerUsage in ONE storeGetMulti call, then slices the flat result back by index. adminProviderView also gains DiscoveryEnabled, mirroring a new providerSnapshot.discoveryEnabled field (registry.go) read straight off providerState — it is set once at construction and never mutated afterward, so it needs neither providerState's mutex nor a place in its existing snapshot() method's mutable-field return shape. Feature B: this lets the webui tell "discovery is off" apart from "discovery is on but has not refreshed yet", which previously both showed the same zero lastRefresh as an identical "refreshed never". |
||
|
lukaszraczylo
|
b39d4e92c9 |
fix(webui): query-safe expand state, prefixed search, accessible row toggle
Review fix round (5 Important + 3 folded) against the provider-model
accordion (
|
||
|
lukaszraczylo
|
f9b289f1ea |
feat(webui): provider model accordion
Clicking a provider row in the dashboard expands to show its model
list, with a search filter and copy-to-clipboard on each model.
Backend (registry.go/admin.go): providerSnapshot gains models []string
- the sorted explicit union discovered id set, computed in the SAME
providerState.mu critical section as lastRefresh/lastErr (not via a
separate knownIDs() call, which takes the same non-reentrant lock and
would deadlock) so the three values are never assembled from two
separate locked passes a concurrent finishRefresh could interleave
between. adminProviderView exposes it as models (never nil - a real,
if empty, JSON array). New tests: overview shape asserts sorted
models per provider; a new TestModelRegistry_Snapshot_ModelsStaleWhileError
proves a failed refresh after an earlier success keeps the previous
discovered set (existing stale-while-error semantics, no special
handling added); TestAdmin_SecretRedaction unaffected (model ids are
not secrets, still passes as-is).
Frontend: the accordion is NOT shadcn-vue's Accordion or Collapsible -
both wrap trigger+content in one div, which cannot legally sit
between two tr elements inside a table/tbody. Installed
Collapsible via the CLI to check (verified components.json's now-
defused font/iconLibrary omission holds under a real install: main.css
stayed byte-for-byte unchanged) and inspected its source: reka-ui's
CollapsibleContent does support an "as" prop that could force it onto
a tr, but its animation-measurement code (getBoundingClientRect +
inline transition styles) is untested in that shape and not worth
taking on for an instant show/hide with no transition - removed the
installed files, kept a plain reactive Set<string> + v-if second
TableRow, which reuses Accordion's own visual language (rotating
chevron, keyboard-accessible trigger row: role=button, tabindex,
Enter/Space handlers) without misusing a component built for block
content.
Model chips (new ModelChip.vue) show the full ROUTABLE id -
"{providerName}/{modelId}" unconditionally, so a discovered id that
already contains a slash (uni/deepseek-v4-flash-0731 on provider
"real") renders as real/uni/deepseek-v4-flash-0731 and stays
copy-paste-valid - and copy that id to the clipboard on click
(lib/clipboard.ts: navigator.clipboard.writeText with a
select-the-text fallback when the Clipboard API is unavailable, never
the deprecated execCommand('copy')), with a brief checkmark
copied-state affordance. No CSP change: Clipboard access is a browser
permission, not a CSP directive.
Search input (OverviewView.vue) filters the providers table to
providers with a matching model (case-insensitive substring) and the
aliases table to aliases whose own name or target matches, auto-
expanding matching providers while a query is active and filtering
each expanded provider's own chip list down to just its matches (so a
72-model provider does not stay a 72-chip wall for a 1-model search).
Manual expand/collapse state is tracked separately from the search's
own auto-expand set, so clearing the query restores exactly what the
user had open by hand - nothing to explicitly "restore", since the
auto-expand set was never written into the manual one.
Integration: TestAdminDashboard's overview assertion extended
(adminProviderModels helper) to check the openai provider's models
array contains dynamic.yml.tmpl's explicit "gpt-mock" model - stays
API-level (asserts the JSON payload), not the rendered HTML, so the
prefixed-display-form request is left at the frontend only per scope.
Regenerated via make admin-ui; admin_assets_gen.go included.
|
||
|
lukaszraczylo
|
ed2119cc99 |
fix(aliases): dedupe listing against emitted entries, not raw owners
listFor's alias dedupe keyed on the raw owners map, so a colliding discovered model whose provider is group-denied caused BOTH the real entry and the alias to be skipped - the id vanished from the listing entirely even though resolve still serves it via the alias. Track ids actually appended to out in a listed map instead, and skip the alias only when listed[alias] is true. Extends TestModelRegistry_ListFor_Alias_DedupedAgainstDiscoveredCollision with a provider-denied variant: group providers:["openai"], collision on the denied "acme" provider -> alias listed exactly once, owned_by openai. |
||
|
lukaszraczylo
|
f0730a70d7 |
fix(aliases): deterministic integration gate, warn-first fast path, deduped listing
Review fix, task 6 (1 Important + 3 folded): 1. TestModelAliases now polls GET /v1/models (waitForModelAlias) until "aliased/mock" appears, bounded 30s, before asserting - removes the stale-dynamic-config propagation window a plain healthy-200 check cannot rule out. Confirmed config-propagation, not alias-logic, per reviewer's config+error-text analysis. 2. registry.go resolve(): check the warned-set (aliasShadowWarned) before paying bareWinner's O(providerNames) scan on every aliased request - only the first request for a colliding alias pays that cost now. 3. listFor: skip an alias entry whose id already exists in owners (a real model already owns it) - previously produced duplicate ids when a discovered model collided with a configured alias. resolve's own alias-wins precedence is unchanged; only the listing is deduped. New test: TestModelRegistry_ListFor_Alias_DedupedAgainstDiscoveredCollision. 4. newAliasTestRegistry doc comment: "grp's given aliases" -> "the given aliases" (no grp parameter exists). |
||
|
lukaszraczylo
|
2090b31ae2 |
feat: operator-defined model aliases
Spec §5 (v0.2): Config.ModelAliases map[string]string, alias-first resolution precedence ahead of provider-prefix splitting, alias-OR-target authorization, lazy target resolution with a 404 naming both ids, /v1/models listing, admin overview alias table (JSON + dashboard), and construction-time validation (shape, prefix-shadow, explicit-model collision, chain, empty target). routes_unified.go/routes_media.go needed no resolution-call-site changes - both already pass the client's raw requested id through to registry.resolve, so aliases inherit the __alias echo and multipart rewrite machinery for free. cache.go needed no changes either - cacheKey already hashes requestedModel, so two aliases resolving to the same target already produce distinct cache entries. Adds registry_test.go validation table + precedence/authz/lazy-404/ listing coverage, a dedicated cache-key split test, chat/embeddings/ images/audio-transcriptions end-to-end tests, an extended admin overview test, and an integration test covering the openai-type vs anthropic-type echo asymmetry under real Traefik+Yaegi. |
||
|
lukaszraczylo
|
547b61f5a9 |
feat: read-only admin dashboard and JSON API
Adds GET /admin (single-page dashboard), GET /admin/api/overview, and GET /admin/api/usage per spec §4: providers/redis/cache/groups summary and per-user/group current-window usage+limits. Gated by Config.Admin (default off, not registered when disabled) and UserConfig.Admin. Gate order: disabled -> 404 fall-through; unauthenticated -> 401; authenticated non-admin -> 403; admin -> serve, counted against the admin's own request limits like any other route. Adds small, read-only, mutex-guarded hooks: modelRegistry.snapshot(), limiter.currentUsage()/redisStatus(), authStore.snapshot(). No secret field (API key, digest, redis password) is ever read into any admin response type; verified by a canary-literal redaction test. HTML/CSS/JS ship as a single Go raw-string const (Yaegi forbids go:embed); confirmed interpretable via make yaegi-check. |
||
|
lukaszraczylo
|
7939c3e74b |
test: integration suite with real Traefik, redis-shared limits, streaming assertions
Adds integration/ (separate module): docker compose stack with real Traefik v3.5 + localPlugins, mock openai/anthropic/gemini/tool providers, shared Redis, and a raw-TCP streaming timing probe. 6/7 tests pass; streaming incrementality fails on a confirmed external yaegi/Traefik bug (see task-15-report.md), not a plugin defect. Fixes 4 real Yaegi-interpretation bugs surfaced by running the plugin under real Traefik for the first time (none caught by go test or yaegi-check, which never exercises chat/embeddings/streaming/Redis): - .traefik.yml: add basePkg (Traefik derives the yaegi binding identifier from the module path, not the package's own declared name; without this override the plugin never loads at all). - registry.go: resolveAgainst's multi-value return panicked yaegi on an inline string concatenation; assign to a local first. - routes_unified.go: errors.As panicked yaegi on interpreted pointer error types; use plain type assertions (both types are always returned bare, never wrapped). - sse.go: readSSE's dispatch closure hit the same multi-value-assignment panic as registry.go; split into separate statements. - resp.go: respErr as a defined string type was indistinguishable from a plain string in a yaegi type assertion, misreading every successful Redis SELECT reply as failed and silently disabling rate limiting; changed to a struct. |
||
|
lukaszraczylo
|
0e73558f5c |
ci: yaegi catalog-analyzer gate harness
Add tools/yaegi-check, a separate Go module (llmgw-yaegi-check,
never a dependency of the plugin module) that replicates the
Traefik Plugin Catalog analyzer (piceus)'s yaegiMiddlewareCheck:
copy the repo into a temp GOPATH, interp.New + i.Use(stdlib.Symbols),
import the plugin package, eval CreateConfig(), decode the plugin's
real .traefik.yml testData into it via a hand-rolled YAML-subset
parser plus a JSON round-trip, reflect-Call New from outside the
interpreter, then exercise the returned handler with two real HTTP
requests (authenticated GET /v1/models against the testData key, and
the same route unauthenticated). Pins github.com/traefik/yaegi to
v0.16.1, confirmed as Traefik v3.5.0's own vendored version by
downloading its go.mod.
Wire cd tools/yaegi-check && GOWORK=off go run . /Users/nvm/Documents/projects/private/traefik-llmgateway
yaegi-check: OK to run the harness from repo root.
The harness caught a real Yaegi v0.16.1 limitation: calling a struct
FIELD of variadic func type (e.g. modelRegistry.log
func(string, ...any)) with two or more variadic arguments crashes
Yaegi's CFG builder ('index out of range'). The identical call shape
through a method or interface method is unaffected — bisected with a
standalone yaegi repro to confirm the exact trigger before touching
plugin code. Fixed by pre-formatting with fmt.Sprintf at
modelRegistry's three affected call sites (registry.go) so the field
is always called with exactly one variadic argument; documented the
constraint on both modelRegistry.log and limiter.logf so it is not
reintroduced. Verified the gate itself is load-bearing by temporarily
renaming CreateConfig (harness failed non-zero) and restoring it
(harness passed again).
Also fixes two golangci-lint findings in the new harness module
(govet shadow, gosec G301 directory permissions).
|
||
|
lukaszraczylo
|
e91f0e36ff |
fix(registry): recover refresh goroutine, per-provider warm-fill budget, exact-glob model authorization
1. refreshProvider (the plugin's only background goroutine) had no recover(): a panicking adapter's listModels would crash the whole Traefik process. Restructured around deferred recover + finishRefresh, so a panic is turned into a logged error, inFlight always releases, and the next interval retries. Documented as a scoped exception to limits.go's no-background-goroutine rule. 2. newGateway wrapped ctx in an outer 5s WithTimeout before warmFill, but warmFill already derives a fresh 5s budget per provider from whatever ctx it receives. The outer wrap made that a single budget shared across every discovery-enabled provider instead of a per-provider one - providers after the first could get a near-expired context and fail instantly. Dropped the outer wrapper; construction's worst case is now N * 5s, accepted. 3. group.allowsModel stripped at any first slash, so a glob like 'deepseek-*' could spuriously match a bare model id that merely contains a slash (e.g. an upstream's own 'uni/deepseek-v4-flash-0731' naming) even though the leading segment names no configured provider. allowsModel is now an exact glob with no internal splitting; resolveAgainst and listFor generate the provider-prefixed and bare-suffix candidates themselves, only when a prefix legitimately names a configured provider. Updated the Task-3 TestIdentify assertion to match. 4. Annotated the discarded json.Encode error in handleModels. Gate: go test ./... -count=1, -race, gofmt, vet, fieldalignment, golangci-lint (incl gosec) all clean. |
||
|
lukaszraczylo
|
fdb81640ec |
feat: model registry with discovery and aggregated /v1/models
Adds modelRegistry: per-provider explicit+discovered model sets, bare-id and provider-prefixed resolution with sorted-name collision precedence, group authorization pairing (allowsModel AND allowsProvider), and a group-filtered OpenAI-compatible /v1/models catalog. Discovery is lazy and throttled per provider: newGateway runs a synchronous 5s-bounded warm fill for discovery-enabled providers (non-fatal on error), and Gateway.ServeHTTP triggers a throttled, inFlight-guarded background refresh per provider per interval (default 1h, per-provider DiscoveryInterval parsed at construction). A failed refresh keeps the previous discovered set (stale-while-error). Wires GET /v1/models into ServeHTTP (auth required, 401 envelope otherwise) and gives Gateway its adapters/registry fields, both built in newGateway. newModelRegistry's signature gained an error return beyond the task brief's literal listing: an invalid DiscoveryInterval must be a constructor error per the controller ruling, which the brief's bare *modelRegistry signature could not express. |