mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
+79
-23
@@ -222,11 +222,16 @@ func (m *modelRegistry) resolvePrimaryOnly(id string, grp *group) (resolveCandid
|
||||
// Only grp.allowsProvider is re-checked per candidate FOR A SINGLE-GRANT
|
||||
// PRINCIPAL (grp.grantCount() == 1 — the overwhelmingly common case,
|
||||
// unchanged from before the multi-group/personal-grant feature existed):
|
||||
// grp.allowsModel is already known true from the primary winner's own
|
||||
// successful resolve call, and — for the identical bare id string every
|
||||
// candidate here shares by construction — allowsModel's result does not
|
||||
// vary by provider (auth.go's allowsModel takes no provider argument),
|
||||
// so re-checking it per candidate would be redundant work, not a
|
||||
// the bare id's own model-glob match is already known true from the
|
||||
// primary winner's own successful resolve call (which goes through
|
||||
// grp.allowsProviderModel, auth.go — the standalone group.allowsModel
|
||||
// this comment used to name was removed as dead code, review-auth
|
||||
// finding F6: every production model check already went through a
|
||||
// per-grant method), and — for the identical bare id string every
|
||||
// candidate here shares by construction — a single grant's model-glob
|
||||
// match does not vary by provider (allowsProviderModel matches grp.
|
||||
// providers and grp.models independently for the single-grant case), so
|
||||
// re-checking it per candidate would be redundant work, not a
|
||||
// correctness gap.
|
||||
//
|
||||
// A MULTI-GRANT principal (more than one member group and/or a personal
|
||||
@@ -501,6 +506,51 @@ func failoverEligible(callErr error) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// requestHealthOutcome classifies callErr for requestHealthTracker.record
|
||||
// (finding 16 fix, review-routes.md): record is boolean (success/
|
||||
// failure), but !failoverEligible(callErr) alone — the previous call-site
|
||||
// shape — conflates two different meanings of "not eligible for
|
||||
// failover" into one "success". A context.Canceled means the CLIENT
|
||||
// walked away; it says nothing about whether the provider would have
|
||||
// succeeded, so recording it as a provider success can reset
|
||||
// consecutiveFailures and close an open breaker for a provider that is,
|
||||
// in fact, still failing every request an attentive client stays around
|
||||
// for. A *translateError or *responseTranslationError is this GATEWAY's
|
||||
// own code failing on an already-decoded/already-received request — not
|
||||
// a signal about the provider's availability at all, per
|
||||
// failoverEligible's own doc comment. errRequestBuildFailed (verify-core
|
||||
// fix, round 4) is the SAME "says nothing about the provider" shape:
|
||||
// http.NewRequestWithContext rejecting this candidate's own configured
|
||||
// base URL/method is a construction-time defect in THIS gateway's config
|
||||
// for that candidate, not a network round trip the provider ever saw —
|
||||
// before this fix, !failoverEligible(callErr) below returned true for
|
||||
// it (failoverEligible itself reports false — not eligible for
|
||||
// failover — for the OPPOSITE reason: retrying an identically malformed
|
||||
// request wastes attempts, not because it is evidence of health),
|
||||
// inverting to record it as a provider SUCCESS and resetting
|
||||
// consecutiveFailures/closing an open breaker for a candidate that
|
||||
// never actually got called. record reports false, meaning "do not
|
||||
// record anything" (a tri-state without a third bool: the caller skips
|
||||
// the record call entirely), for all three cases; every other outcome
|
||||
// keeps using failoverEligible's existing classification unchanged,
|
||||
// since a genuine HTTP 400 (or a candidate that succeeded outright)
|
||||
// legitimately says something about the provider.
|
||||
func requestHealthOutcome(callErr error) (success, record bool) {
|
||||
if callErr == nil {
|
||||
return true, true
|
||||
}
|
||||
if errors.Is(callErr, context.Canceled) || errors.Is(callErr, errRequestBuildFailed) {
|
||||
return false, false
|
||||
}
|
||||
if _, ok := callErr.(*translateError); ok {
|
||||
return false, false
|
||||
}
|
||||
if _, ok := callErr.(*responseTranslationError); ok {
|
||||
return false, false
|
||||
}
|
||||
return !failoverEligible(callErr), true
|
||||
}
|
||||
|
||||
// isProviderNotFoundError reports whether err is a *providerHTTPError
|
||||
// carrying HTTP 404 — routes_unified.go's runMeteredCall uses this to
|
||||
// decide when a failover needs its own loud, non-suppressed log line
|
||||
@@ -597,24 +647,30 @@ func (t *requestHealthTracker) stateFor(provider string) *requestHealthState {
|
||||
// exactly once per candidate per client request, after call() has
|
||||
// already exhausted whatever same-provider retries retryPolicy.do
|
||||
// performed internally (routes_unified.go's runMeteredCall), never once
|
||||
// per raw upstream attempt. success=false must be
|
||||
// !failoverEligible(callErr) (routes_unified.go/failover.go's own
|
||||
// classification of the FINAL adapter-level error) — adversarial-review
|
||||
// ruling (F3/F6): an EARLIER version fed this from isTransient(resp,
|
||||
// err) || isDeadlineExceeded(err), limits.go/retry.go's own
|
||||
// retry-shaped classification, once per RAW attempt inside
|
||||
// retryPolicy.do. That was wrong on two counts. isTransient treats a
|
||||
// non-429 4xx as "the provider answered correctly to a request it did
|
||||
// not like" — correct for deciding whether retrying the SAME provider
|
||||
// is worth it, wrong for deciding whether the POOL should route around
|
||||
// it: a provider whose API key is dead answers 401 to every request and
|
||||
// must eventually be skipped, not retried forever, which is exactly the
|
||||
// "masking a broken provider key" risk this feature exists to reduce.
|
||||
// And recording per raw attempt let ONE client request with
|
||||
// retry.enabled (attempts: 3) report FOUR failures against a threshold
|
||||
// documented as three REQUESTS. failoverEligible already classifies
|
||||
// "everything except HTTP 400" as a failure, matching what this gate
|
||||
// needs with no separate function required.
|
||||
// per raw upstream attempt. Callers derive success from
|
||||
// requestHealthOutcome(callErr), above (finding 16 fix, review-
|
||||
// routes.md): that helper's own success return is
|
||||
// !failoverEligible(callErr) — adversarial-review ruling (F3/F6): an
|
||||
// EARLIER version fed this from isTransient(resp, err) ||
|
||||
// isDeadlineExceeded(err), limits.go/retry.go's own retry-shaped
|
||||
// classification, once per RAW attempt inside retryPolicy.do. That was
|
||||
// wrong on two counts. isTransient treats a non-429 4xx as "the provider
|
||||
// answered correctly to a request it did not like" — correct for
|
||||
// deciding whether retrying the SAME provider is worth it, wrong for
|
||||
// deciding whether the POOL should route around it: a provider whose API
|
||||
// key is dead answers 401 to every request and must eventually be
|
||||
// skipped, not retried forever, which is exactly the "masking a broken
|
||||
// provider key" risk this feature exists to reduce. And recording per
|
||||
// raw attempt let ONE client request with retry.enabled (attempts: 3)
|
||||
// report FOUR failures against a threshold documented as three REQUESTS.
|
||||
// failoverEligible already classifies "everything except HTTP 400" as a
|
||||
// failure, matching what this gate needs with no separate function
|
||||
// required — requestHealthOutcome's own additional filtering (skip a
|
||||
// client cancel or a gateway-side translation error entirely, rather
|
||||
// than recording either as a provider success) is what THIS record
|
||||
// method must never see: it trusts success at face value and has no way
|
||||
// to tell "the provider genuinely succeeded" apart from "the caller
|
||||
// decided not to blame the provider" on its own.
|
||||
//
|
||||
// requestBreakerFailureThreshold consecutive failures opens the breaker
|
||||
// for requestBreakerOpenDuration. While open, healthy (below) reports
|
||||
|
||||
Reference in New Issue
Block a user