mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-14 05:18:54 +00:00
9dc67dedbb2651295d897e0a53fca84424ee3ec3
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 lukaszraczylo
|
9dc67dedbb |
feat: add anonymous usage reporting, matching traefikoidc
Release builds send one anonymous "plugin loaded" ping per process, so the project can see roughly how many installs exist and which versions are in use. The payload is project name, version and timestamp. Nothing else leaves the pod: no identifiers, no hostname, no configuration, no provider names, no keys, no request data. Three things keep it from firing when it should not. Builds from source carry the devPluginVersion sentinel and stay silent, so developer checkouts and every test run never phone home. The ping is sent only after the plugin constructs successfully, so a rejected configuration is never counted as an install. A sync.Once holds it to one per process, because Traefik calls New once per route and oss-telemetry does not deduplicate on the client side. Operators of a release build opt out with DO_NOT_TRACK=1, OSS_TELEMETRY_DISABLED=1, or TRAEFIK_LLMGATEWAY_DISABLE_TELEMETRY=1. README.md documents the full payload and the opt-out; a plugin that phones home without saying so in its own README would be indefensible. oss-telemetry is the plugin's first non-stdlib RUNTIME dependency, which has consequences for the Yaegi gate: - vendor/ is no longer excluded from yaegi-check's GOPATH copy. It was excluded on the reasoning that it held only test-only testify and _test.go files are never interpreted. That reasoning no longer holds: leaving vendor out would make the interpreted copy fail to resolve an import the real plugin tree resolves fine, so the gate would stop representing production. - yaegi-check now stamps a fake release version into its GOPATH copy so New's telemetry gate is PASSED rather than short-circuited, and the vendored package is actually called under the interpreter. Import resolution alone is not coverage; a cross-package call into interpreted third-party code is its own risk. Telemetry is suppressed by env vars set before anything is interpreted, so the gate never emits a real ping. - A new assertion reads the version back through GET /admin/api/overview. Without it a silently failed stamp would leave the dev sentinel in place and the harness would still print OK while covering nothing. Verified by disabling the stamp and watching the gate fail. shouldSendTelemetry is split out from sendStartupTelemetry so the decision is table-testable without a network, since oss-telemetry's endpoint is deliberately not overridable from consuming code. Also brings the version-gated documentation up to date, since the same README edit touches it. The latest release is v1.0.3, so the pre-1.0 experimental warning has met its own stated condition and is replaced by a semver stability statement in README.md and dropped from .traefik.yml's catalog summary. The catalog now serves the released tags (verified: the v1.0.3 download endpoint returns 200), so the "not yet published" and "analyzer has not been run" claims are corrected, the catalog form of the static config is documented as usable, and examples/kubernetes.yaml pins v1.0.3 instead of v0.1.0. |
||
|
lukaszraczylo
|
de42192426 |
refactor(mcp): restore errors.Is for the oversize-response sentinel
Federation's tools/call oversize branch matched a string marker with strings.Contains instead of wrapping a sentinel. The stated reason was that no gate exercised this path under Yaegi, so errors.Is was unproven here. The caution was misplaced. errors.New returns a compiled *errors.errorString and fmt.Errorf a compiled *fmt.wrapError, so declaring the sentinel variable in interpreted code leaves the entire chain errors.Is walks compiled. Neither documented trap applies: the matchesSentinel trip-wire covers a plugin-declared error TYPE, and the comma-ok trap needs a plugin-declared interface. retry.go has matched errRequestBuildFailed this way through a multi-%w wrap since v0.2.0. The string check was also strictly worse: err.Error() is itself an interpreted-to-compiled dispatch, so it verified nothing extra, and matching on text that partly comes from the backend admits false positives a sentinel cannot have. Rather than settle this by argument, tools/yaegi-check now drives POST /mcp tools/call against a backend whose response exceeds the cap, and asserts the interpreted plugin reports "response too large". Breaking the match makes the gate fail with "upstream error", so the probe has teeth. This closes the objection permanently instead of answering it in a comment. Also in doBackendJSONRPC, check the HTTP status before the size cap. A backend failing with a large HTML error page now reports "upstream returned HTTP 500", which is actionable, rather than sending the operator after a payload-size problem that does not exist. Two comment corrections carried forward from review: the fan-out semaphore is not the same shape as limiter.spawnTokens (spawnTokens drops work via a default case, this one blocks and must never skip a server), and the shared fan-out deadline gives later batches the remaining budget, not a fresh one, so uniformly slow backends yield a partial list. |
||
|
lukaszraczylo
|
fb1b2a36ff |
test(yaegi-check): exercise the builtin table lookup (review SHOULD-6)
The harness previously only ever proved resolveModelMeta's CONFIG- OVERRIDE layer runs interpreted (testDataWantModel carries an explicit modelMeta entry). It never exercised an interpreted LOOKUP into the real, 236-entry builtinModelMetaTable map literal at all. Adds "gpt-4o" to the harness's openai provider with NO modelMeta override of its own, so its GET /v1/models context_window can only come from the builtin-table branch. Its expected value is read directly out of pricing_data_gen.go at harness run time (readBuiltinContextTokens), not hardcoded, so this assertion self-updates across a `make pricing-sync` regeneration instead of pinning a number that could legitimately drift when upstream LiteLLM data changes. Also switched both existing GET /v1/models assertions from raw strings.Contains substring matching to a structured JSON decode keyed by model id — a bare substring check could not tell which model entry a given context_window value actually belonged to, which now matters with two entries in play. |
||
|
lukaszraczylo
|
e3996d88bf |
fix(pricing): content-digest snapshot instead of a wall-clock date
The generated pricing_data_gen.go header used to stamp today's date on every `make pricing-sync` run, so a re-run against genuinely unchanged upstream data still touched the file and produced a spurious daily diff with nothing real to review. The header now records a short SHA-256 digest of the FETCHED bytes instead — identical upstream data always reproduces a byte-identical generated file. Folded review minor. |
||
|
lukaszraczylo
|
0c61296c73 |
feat(registry): model metadata capture and resolution
Config gets two new surfaces (llmgateway.go): Config.ModelMeta (map[string]*ModelMetaConfig, keyed by exact "provider/model" or a bare model id) for operator overrides, and ProviderConfig.MetadataPath for openai-type providers whose discovery should also capture per-model context (LM Studio's native /api/v0/models shape). modelmeta.go adds the layering itself: resolveModelMeta applies config override > discovered context > builtin table > absent for context, and config override/free > builtin > absent for cost (discovery never reports cost). resolveAliasModelMeta implements alias inheritance: an alias inherits its target's fully resolved metadata unless the alias itself has its own modelMeta entry, applied per field. validateModelMeta rejects negative values and a free:true/non-zero-cost contradiction at construction. registry.go wires this into the request path: providerState gains a discoveredContext map (stale-while-error, mirroring discovered's own convention) with contextFor/setDiscoveredContext; a new optional modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted, documented as yaegi-safe against the Trap-4 restriction) rather than added to providerAdapter, lets warmFill/refreshProvider capture metadata alongside listModels without touching anthropic/gemini adapters at all. modelObject/listFor now attach resolved context_window/pricing to every GET /v1/models entry, including aliases (resolveMetaFor/ resolveMetaForAlias/resolveMetaForAliasName). provider_openai.go implements modelMetadataFetcher: fetchModelMetadata reads ProviderConfig.MetadataPath, preferring loaded_context_length over max_context_length when the LM Studio response reports both. providers.go wires MetadataPath onto the adapter in buildAdapters. tools/yaegi-check/main.go adds one cheap modelMeta config-override assertion to the existing attempt-accounting harness, proving resolveModelMeta's config-override layer and the context_window/pricing extension fields both run correctly under the real Yaegi interpreter. |
||
|
lukaszraczylo
|
e8398e6d49 |
feat(pricing): litellm sync generator + table
Add tools/pricing-sync: a standalone, compiled Go tool that fetches
LiteLLM's model_prices_and_context_window.json at build time and
regenerates pricing_data_gen.go, a plain map[string]builtinModelMeta
literal keyed by bare model id (context window, input/output cost in
micro-USD per million tokens).
Pruning rule (documented on the tool's own package doc comment): keep
only entries whose litellm_provider matches this gateway's own provider
types (openai, anthropic, deepseek, xai, minimax, cerebras, dashscope
for qwen/alibaba, and moonshot for Kimi models) and whose mode is chat
or embedding, dropping everything else — a 3111-entry table is a real
startup cost under Yaegi interpretation, so the generated table stays at
236 entries. Bare-id collisions across LiteLLM keys resolve
deterministically: a bare key beats a provider-prefixed one, then
provider priority, then lexicographic order.
Naming bridges: a small, data-driven table (namingBridges) duplicates an
entry under a second bare id when a provider's own live API disagrees
with LiteLLM's naming for the identical model — seeded with MiniMax's
lightning/highspeed dual naming, verified against a live deployment.
Coverage-audit follow-up (verified against a live fetch of the same
source, 2026-08): moonshot's own "litellm_provider" tag carries 23
real Kimi entries (kimi-k2.5, kimi-k3, ...), added to the allowlist as a
first-class provider rather than a per-id bridge, since the table is
already keyed by bare id regardless of provider. Two claims from a
coverage-audit request could NOT be verified against this same fetched
file and were deliberately NOT implemented: no key anywhere in the
3111-entry source ends in ":free" (an openrouter free-tier naming
convention the request assumed exists in this file), and no
"qwen/"-prefixed key exists at all — see the report for the full
grounding note. Also confirmed already-correct by design, no change
needed: audio/responses-mode OpenAI entries ("codex-mini-latest" and
friends) are excluded by the existing mode allowlist, not by any
provider special-case; image/video media models are excluded the same
way.
builtinModelMeta itself lives in its own file (builtin_model_meta.go),
separate from modelmeta.go's resolution logic, so this commit's table
and generator compile standalone.
make pricing-sync regenerates the table; add its gitignore entry for the
tool's own compiled binary.
|
||
|
lukaszraczylo
|
eddcbe35bd |
fix(providers): multi-wrap deadline detection was dead on 3 of 4 routes
BLOCKER (v0.22 review round, round 2): matchesSentinel's hand-rolled
Unwrap walk (round 1's SHOULD-1 fix) stopped dead the moment it hit
providers.go's own multi-%w wrapping. fmt.Errorf with more than one %w
verb builds a *fmt.wrapErrors implementing Unwrap() []error, not
Unwrap() error — a comma-ok assertion against a single-error Unwrap
interface alone never reaches past it. All three of providers.go's own
wrap sites use multi-%w (upstreamJSON's encode-failure path, "%w:
encode request body: %w"; upstreamBytes' build-request path, "%w: %w:
build request: %w"; and upstreamBytes' client.Do path, "%w: %w" — the
one a real request-context deadline reaches through). Result: a hung
upstream on the unified chat/embeddings/media routes recorded as 100%
healthy; only native passthrough (whose error is never re-wrapped by
upstreamBytes) worked.
Round 2's first attempt added a second hand-rolled interface
(multiUnwrapper, Unwrap() []error) and recursed into it — verified
correct against every wrap shape under `go test`, INCLUDING the exact
double-/triple-%w shapes providers.go itself produces. It still did not
work. tools/yaegi-check's own SHOULD-A extension (below) caught it: a
type assertion of a COMPILED concrete value (*fmt.wrapErrors, built by
the real fmt package) against an INTERPRETED interface type
(multiUnwrapper, declared in this plugin's own source) silently returns
ok=false under Yaegi, even though %T correctly reports *fmt.wrapErrors
and that type genuinely implements Unwrap() []error in real Go — the
reverse direction of the already-documented errors.As-on-interpreted-
types trap, not previously known to this codebase.
Fix: matchesSentinel now calls real errors.Is instead of hand-rolling
anything. This is safe here specifically, despite the "avoid errors.Is/
As" guidance elsewhere in this codebase: the panic risk that guidance
targets is errors.As checking whether an INTERPRETED type implements
error, which never arises for a plain sentinel comparison against
compiled-origin values (context.DeadlineExceeded, context.Canceled, and
every err this function ever sees) — the identical safe shape retry.go's
isTransient already relies on for these same two sentinels, proven
correct under Yaegi across every prior round. Empirically verified: this
is what made the extended yaegi-check harness pass; the hand-rolled
version did not, despite passing every compiled `go test` row.
Also in this commit:
- Route-level regression tests (TestHandleChat_
ContextDeadlineExceeded_RecordsProviderFailure, routes_unified_test.go;
TestHandlePassthrough_ContextDeadlineExceeded_RecordsProviderFailure,
routes_passthrough_test.go) — a real httptest upstream sleeping past a
real short request-context deadline, adapted from a reviewer-provided
probe. These are what exposed the blocker in the first place; the
sentinel-level table test alone (hand-built error shapes) is exactly
what let it ship.
- Prod-shape rows (double-/triple-%w, byte-for-byte providers.go's own
format strings) folded into the existing classification table
(renamed TestIsDeadlineExceeded_Classification — its own doc comment
now records that this table alone is NOT proof under the interpreter,
after round 2's own history).
- tools/yaegi-check/main.go (SHOULD-A): the harness's claim to prove the
interpreted deadline path was never actually true — its only upstream
answered instantly, so isDeadlineExceeded was never exercised with a
non-nil error at all. Added a second, deliberately slow provider
("slow", provider-prefixed model id so it cannot collide with the
existing fast one) driven with a short request-context deadline,
asserting failuresDay==1 under the real interpreter. This is the
harness that caught the blocker above.
- FOLDED-1: recordProviderAttempt's fire-and-forget store write
(SHOULD-5, round 1) is now bounded — limiter.spawnTokens, a per-limiter
buffered-channel semaphore (providerAttemptSpawnCap=64) the production
spawn closure acquires non-blockingly; a write that cannot start
immediately is DROPPED, not queued. Bounded and lossy-under-pressure by
design: telemetry only, enforcement never goes through spawn.
Per-limiter, not package-level, so the test suite's many parallel
*limiter instances cannot starve each other.
- FOLDED-2: TestRecordProviderAttempt_ProductionSpawn_
WriteEventuallyLandsAsync leaves limiter.spawn at its real production
default (every other test in this file overrides it to run
synchronously) and polls (waitUntil, deadline-bounded, not a fixed
sleep) for the counter to land — closing the -race async-path gap the
synchronous overrides otherwise leave entirely untested.
|
||
|
lukaszraczylo
|
56817b41d1 |
test(yaegi): prove attempt-accounting under the interpreter
SHOULD-4 (v0.22 review round): tools/yaegi-check now drives a real chat completion against a local httptest upstream, then GET /admin/api/overview, asserting the resulting provider reports exactly one attempt and zero failures — end to end under Yaegi, not merely compiled: runUnified's attemptRecorder -> retryPolicy.do's context lookup -> limiter.recordProviderAttempt's isTransient/isDeadlineExceeded classification -> its fire-and-forget spawn -> buildAdminOverview's own batched providerUsage read. This also proves the SHOULD-1 hand-rolled errors.Unwrap walk (matchesSentinel/isDeadlineExceeded, limits.go) runs correctly interpreted, which `go test` cannot: that class of bug is exactly what this harness exists to catch. Adapted from a reviewer-provided reference harness (a second json.Unmarshal layering a mock-upstream provider and an admin-flagged user on top of the plugin's own .traefik.yml testData) rather than reinvented — reworked to poll GET /admin/api/overview briefly (SHOULD-5's fire-and-forget write can lag a single immediate read) and to assert on the decoded attemptsDay/failuresDay values instead of only printing them, so a regression here actually fails the gate. |
||
|
lukaszraczylo
|
655bbcbf2c |
feat(webui): vue admin panel source
Vue 3 + TypeScript + Pinia + Tailwind v4 + shadcn-vue + Chart.js admin panel source under webui/, built with Vite. Views: Overview (providers, redis/cache/retry status, model aliases), Usage (per-user/per-group/total tables with tokensIn/tokensOut/cost + limits), Charts (scope selector, req/tokens(stacked in-out)/cost metric tabs, 24h/30d/12mo window switcher, fed by GET /admin/api/usage/history). Pinia stores: auth (sessionStorage key gate, x-api-key header, 401/403 re-prompt), dashboard (5s poll), history (fetch-on-selection + 30s refresh). Design: deliberate neutral warm-neutral palette with a signature teal accent (no default Inter/purple-gradient/glassmorphism), real typographic hierarchy via the platform UI/mono font stacks (zero external assets - no Google Fonts), dark+light via prefers-color-scheme (no class toggle, no theme switcher). Charts: zero-based bars, honest axes, Okabe-Ito colorblind-safe series, tokens-in/out as a distinguishable stacked pair. vite.config.ts: base '/admin/' (matches admin.go's route layout) and build.modulePreload.polyfill:false (no inline bootstrap script, so the tightened admin CSP needs no 'unsafe-inline'). Code-split: Chart.js (ChartsView) loads lazily, not in the initial bundle. generate.mjs reads dist/ after `vite build` and writes ../admin_assets_gen.go (not committed by this change) - base64-encodes every asset, decoded once at package-var init; documented rationale in the script's own header comment. Hygiene: webui/go.mod walls webui/ off from the repo root's `go build`/ `vet`/`test ./...` (npm's node_modules bundles a stray .go file that would otherwise be swept in); tools/yaegi-check/main.go excludes webui/ from its GOPATH copy for the same reason. webui/ export-ignored (.gitattributes) from release source tarballs; webui/node_modules/ gitignored. tsconfig.app.json/tsconfig.node.json's JSONC comments dropped (TS itself is fine with either; the global pre-commit JSON validator is not) - no config values changed. |
||
|
lukaszraczylo
|
f7d9208bde |
chore(deps): vendor testify
Add github.com/stretchr/testify (test-only) as a direct dependency and vendor it, matching the traefikoidc precedent: _test.go files are never Yaegi-interpreted, so vendoring a test-only dependency does not affect the plugin's runtime under Traefik or the catalog analyzer. tools/yaegi-check/main.go: add vendor to the harness's repo-copy exclude list, since vendor/'s nested module layout (go.yaml.in/yaml/v3 alongside testify) has nothing to do with the plugin package Yaegi imports. Verified: make yaegi-check stays green; go build/vet/test unaffected. |
||
|
lukaszraczylo
|
0e73558f5c |
ci: yaegi catalog-analyzer gate harness
Add tools/yaegi-check, a separate Go module (llmgw-yaegi-check,
never a dependency of the plugin module) that replicates the
Traefik Plugin Catalog analyzer (piceus)'s yaegiMiddlewareCheck:
copy the repo into a temp GOPATH, interp.New + i.Use(stdlib.Symbols),
import the plugin package, eval CreateConfig(), decode the plugin's
real .traefik.yml testData into it via a hand-rolled YAML-subset
parser plus a JSON round-trip, reflect-Call New from outside the
interpreter, then exercise the returned handler with two real HTTP
requests (authenticated GET /v1/models against the testData key, and
the same route unauthenticated). Pins github.com/traefik/yaegi to
v0.16.1, confirmed as Traefik v3.5.0's own vendored version by
downloading its go.mod.
Wire cd tools/yaegi-check && GOWORK=off go run . /Users/nvm/Documents/projects/private/traefik-llmgateway
yaegi-check: OK to run the harness from repo root.
The harness caught a real Yaegi v0.16.1 limitation: calling a struct
FIELD of variadic func type (e.g. modelRegistry.log
func(string, ...any)) with two or more variadic arguments crashes
Yaegi's CFG builder ('index out of range'). The identical call shape
through a method or interface method is unaffected — bisected with a
standalone yaegi repro to confirm the exact trigger before touching
plugin code. Fixed by pre-formatting with fmt.Sprintf at
modelRegistry's three affected call sites (registry.go) so the field
is always called with exactly one variadic argument; documented the
constraint on both modelRegistry.log and limiter.logf so it is not
reintroduced. Verified the gate itself is load-bearing by temporarily
renaming CreateConfig (harness failed non-zero) and restoring it
(harness passed again).
Also fixes two golangci-lint findings in the new harness module
(govet shadow, gosec G301 directory permissions).
|