5 Commits

Include renames

Author	SHA1	Message	Date
lukaszraczylo	4c01c75276	fix(gateway): close billing, limit, routing and translation gaps ... Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv	2026-09-23 09:44:31 +01:00
lukaszraczylo	72b0df5b0c	feat(auth): let a user join several groups and hold a personal grant ... A user entry can now list several groups in "groups", next to the existing "group" field. A user can also hold a personal grant with its own "providers" and optional "models" lists. Users with one group and no personal grant behave exactly as before. A request is allowed only when one grant allows the provider, the model and, for passthrough, the path. Grants never merge, so one grant cannot supply the provider while another supplies the model or the path. The personal grant uses the path list of the first group. A personal "models" list without "providers" is a configuration error. Each member group applies its own limits. Caching stays on unless a member group turns it off, and the shortest group TTL wins. The /v1/models cache uses a stable, length-prefixed key, so reloads do not grow it. Tests cover the grant union, cross-grant denials, passthrough path and model coupling, reloads and the cache key. The yaegi-check harness asserts the same cases under the interpreter. Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3	2026-09-11 22:45:45 +01:00
lukaszraczylo	0cf23345b1	fix(failover): scope header clear to retries, fix cache pre-pass, gate health record ... Fix one must-fix and one should-fix finding from re-verification of d45125e, plus two smaller findings from the same pass. MUST FIX: the F2 header clear ran on the first candidate attempt too, so it ran on the default, failover-disabled path and deleted response headers a different, earlier Traefik middleware had already staged before this plugin ran (CORS, HSTS, request-id, and similar). Scope the clear to i > 0, the only case that ever needs it. Add a test proving a header staged before the gateway runs survives the default path, on both a cache hit and a cache miss. SHOULD FIX, coordinator ruling: drop the all-candidates cache pre-pass. It silently served a different candidate's cached response while the primary was perfectly healthy, which is failover with no failure. Check each candidate's own cache entry immediately before that candidate's own attempt instead. Request-path health, now fixed in the prior round, is the correct mechanism for skipping a genuinely dead primary: once it is marked unhealthy it is excluded from the candidate list entirely, so a later request still reaches the next candidate's cache without the dead one being dialed. Update the existing cache regression test to drive enough requests to trip the health breaker before asserting no repeat call, and add a dedicated test proving a healthy primary is never bypassed for another candidate's cached response. Guard the request-health record call with the enabled check. That state is never read on the disabled path, so recording it there was dead work plus a new contention point on requestHealthTracker's tracker-wide mutex on every metered request, disabled or not. Add a test asserting the tracker's internal state stays empty across the disabled path. Add a test proving the 404 loud-log line is not rate-limited: drive two 404 failovers through the same Gateway in one request and assert both lines appear, since a version that wrapped the 404 line in the same rate limiter as the generic line would still pass every other test (each of which only ever drives one 404 event against a fresh gate). Document, but do not fix, that the cost guard cannot see a model declared free via modelMeta: it resolves pricing only through the same source real billing uses (pricing and the built-in table), which is correct and consistent, but means an operator who declares a model free the modelMeta way loses it as a failover candidate without realizing why. README now says to use pricing instead for that case. Every fix is proven by a dedicated regression test, and each was verified by reverting the fix in place and confirming the corresponding test fails before restoring it.	2026-08-23 01:48:04 +01:00
lukaszraczylo	d45125ee07	fix(failover): default off, add cost guard, fix adversarial-review findings ... Flip FailoverConfig.Enabled to default false. A version upgrade with no config change must preserve prior behaviour, and failover changes which provider serves a request. Update tests, README, and the yaegi-check probe to opt in explicitly. Add a mandatory cost guard. A candidate is skipped unless its input AND output cost per token are each no greater than the primary candidate's, resolved through the same canonical-then-bare pricing path the real billing code uses. Unknown pricing is never treated as free: a known primary against an unknown candidate skips, an unknown primary against a known non-zero candidate skips, everything else that is equal is allowed. Not configurable. Fix four blocking findings from adversarial review of commit 5d62479. F1: openai-type adapters mutate the request map in place to force stream usage reporting. Reusing one map across candidates let a later candidate read back an earlier candidate's own injection and leak a usage-only SSE chunk to a client that never asked for it. Every candidate now attempts against a fresh copy of the original request. F2: a streaming candidate that fails before writing anything still stages SSE-only response headers. Those headers now get cleared before every candidate attempt, so they cannot survive into a later candidate's plain JSON response. F3: the request-health gate used the retry-shaped isTransient classification, which treats a non-429 4xx as a successful outcome. A provider with a dead API key was never skipped and was retried forever. The gate now reuses failoverEligible's own classification instead. F4: a shared request context that an earlier candidate exhausted was attributed to whichever candidate happened to run next, opening a healthy provider's breaker for a timeout that was not its fault. A candidate is no longer attempted once the shared context is already done; the loop stops and reports the last real error instead. Also fix three should-fix findings from the same review. F5: a failure arriving while the request-health breaker is already open but not yet past its cooldown window is now a no-op, instead of resetting backoff and sliding the open window forward on every such failure. F6: the request-health gate now records at most one outcome per candidate per logical request, not one per same-provider retry attempt, so retry.attempts alone can no longer open the breaker on a single request. F7: every candidate's cache entry is now checked before any candidate is attempted, so a cached failover response no longer costs a round trip to a still-broken primary first. F10: the generic "failing over" log line is now rate-limited to once per 30 seconds during a sustained outage, reusing the existing logGate type. The 404 loud-log line stays unthrottled, as ruled. F9 (optional): skip building the failover candidate pool entirely when failover is disabled. Correct an overstated test comment on TestHandleChat_Failover_DiscoveryUnhealthy_StillTried: it proves alpha is still tried, not that ordering places it second. Document, but do not fix here, that this plugin's HTTP client sets no per-attempt timeout, so a provider that accepts a connection and then hangs will never trigger failover. This is a pre-existing property of the shared HTTP client, not specific to failover. Deferred: F8 (surfacing request-health state on the admin dashboard) touches admin.go, which a sibling branch is also editing concurrently; left for a follow-up to avoid a merge conflict in a file this change does not otherwise need. Every fix above is proven by a dedicated regression test, and each was verified by reverting the fix in place and confirming the corresponding test fails before restoring it.	2026-08-23 01:18:58 +01:00
lukaszraczylo	5d624791fa	feat(failover): add cross-provider failover for overlapping models ... Add failover.go with the full candidate pipeline. resolveWithCandidates resolves a request exactly like resolve does, then adds every other provider sharing the identical bare model id. orderedFailoverCandidates applies a hard skip for request-path-unhealthy providers, deprioritizes discovery-unhealthy providers without dropping them, fails open when every candidate reads unhealthy, and caps the list at maxAttempts. Add requestHealthTracker, a separate per-pod request-path health signal fed by the same isTransient/isDeadlineExceeded classification recordProviderAttempt already uses. It never widens the existing discovery circuit breaker, per that breaker's own documented scope. Rewrite runMeteredCall (routes_unified.go) to loop over candidates instead of resolving once. Each attempt rebinds the attempt recorder to its own provider, recomputes the cache key before the response is cached, and re-injects the alias echo key an adapter may have deleted on a prior failed attempt. Failover never runs once any byte has reached the client, since statusTrackingWriter already tracks that state. A failure falls through on everything except HTTP 400, matching the brief exactly. A 404 failover logs loudly, naming both providers and the model, since it usually signals a real configuration mistake. Add FailoverConfig (Config.Failover): an Enabled pointer defaulting to true, and a MaxAttempts bound between 1 and 10, defaulting to 3. A single-provider deployment, or one with no overlapping bare model ids, behaves exactly as before regardless of this setting. Add a yaegi-check probe (tools/yaegi-check/main.go) driving two providers under the real interpreter, one failing and one serving, proving the candidate loop runs correctly interpreted. Update README.md with a Failover section and correct the Retry section's stale claim that no cross-provider failover exists. Media routes (images, audio speech, audio transcriptions) do not get failover in this change. Each has its own request-shape handling that needs its own scoped follow-up.	2026-08-23 00:32:12 +01:00