fix(gateway): close billing, limit, routing and translation gaps

Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
  The gateway falls back to a request-size estimate only on a 2xx
  stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
  billed at zero.
- Billing uses the built-in LiteLLM price table and honours
  modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
  saturates on overflow.
- Gemini thinking tokens count as completion tokens.

Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
  user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
  repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.

Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
  owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.

Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
  path.
- Retry-After is never jittered. A cancelled wait returns the
  context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
  as provider successes.

Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 09:44:31 +01:00
parent b3b8ae6ad5
commit 4c01c75276
62 changed files with 6064 additions and 1182 deletions
+71 -60
View File
@@ -2,7 +2,6 @@ package traefikllmgateway
import (
"crypto/sha256"
"crypto/subtle"
"fmt"
"net"
"net/http"
@@ -107,32 +106,20 @@ func (grp *group) allowsProvider(name string) bool {
return matchesGlob(grp.providers, name)
}
// allowsModel reports whether id matches one of the group's model glob
// patterns, exactly as given. An empty pattern list allows every model.
//
// allowsModel does no "provider/model" prefix-splitting of its own — a
// pattern like "deepseek-*" must not spuriously match a bare model id that
// merely happens to contain a slash (e.g. an upstream's own
// "uni/deepseek-v4-flash-0731" naming, where "uni" is not a configured
// provider: there is no legitimate prefix to strip there at all). A caller
// that genuinely knows an id's leading segment names a configured provider
// (modelRegistry.resolveAgainst, modelRegistry.listFor) generates both the
// full and bare-suffix candidate strings itself and calls allowsModel once
// per candidate.
func (grp *group) allowsModel(id string) bool {
return matchesGlob(grp.models, id)
}
// hasModelRestriction reports whether grp's Models glob is non-empty —
// whether there is anything for allowsModel to actually reject (security
// review fix, 2026-08-22, round 2). handlePassthrough (routes_
// passthrough.go) checks this BEFORE peeking a passthrough request's body
// at all: a group with an empty Models list (matchesGlob's own
// empty-means-all contract — the live cluster's "home" group and every
// other group that has not opted into model restrictions) has nothing
// allowsModel could ever deny, so there is no reason to read, buffer, or
// even look at the request body for model enforcement — zero risk, and
// exactly today's behavior for the overwhelming majority of traffic.
// whether there is anything for allowsProviderModel to actually reject on
// the model side (security review fix, 2026-08-22, round 2 — updated
// review-auth finding F6, 2026-09 audit: the standalone allowsModel this
// comment originally referenced was removed as dead code; every
// production model check goes through allowsProviderModel). handlePassthrough
// (routes_passthrough.go) checks this BEFORE peeking a passthrough
// request's body at all: a group with an empty Models list (matchesGlob's
// own empty-means-all contract — the live cluster's "home" group and
// every other group that has not opted into model restrictions) has
// nothing to deny on the model side, so there is no reason to read,
// buffer, or even look at the request body for model enforcement — zero
// risk, and exactly today's behavior for the overwhelming majority of
// traffic.
func (grp *group) hasModelRestriction() bool {
return len(grp.models) > 0
}
@@ -697,21 +684,36 @@ func (g *logGate) shouldLog(now time.Time) (log bool, suppressed int64) {
// Gateway even has a limiter wired yet. failureLog/throttleLog back
// logAuthEvent's two distinct rate-limited lines (logger.go).
type authStore struct {
lastModTime time.Time
lastCheck time.Time
log gatewayLogger
failures *authFailureTracker
usersFile *usersFile
nowFn func() time.Time
groups map[string]*group
byDigest map[[32]byte]*authEntry
inline map[[32]byte]*authEntry
failureLog logGate
throttleLog logGate
fileUserCount int
mu sync.RWMutex
buildMu sync.Mutex
reloadMu sync.Mutex
lastModTime time.Time
lastCheck time.Time
// lastReloadErrModTime and lastReloadErrMsg dedup maybeReload's own
// error logging (users_file.go — review-auth finding F11, 2026-09
// audit): a users file left broken (bad JSON, a stat failure, a
// rejected replaceFileUsers) never advances lastModTime, so without
// this a broken file gets re-attempted AND re-logged every
// reloadEvery for as long as it stays broken — roughly 17k lines/day
// at the default 5s throttle. A later log fires again only when the
// error TEXT changes, or the mtime the attempt was made against
// changes (an operator rewrote the file, even to an identically
// broken state) — "once per distinct error / mtime change". Both are
// touched only from inside maybeReload, which its own TryLock already
// serializes to at most one call in flight, so no separate mutex
// guards them.
lastReloadErrModTime time.Time
log gatewayLogger
failures *authFailureTracker
usersFile *usersFile
nowFn func() time.Time
groups map[string]*group
byDigest map[[32]byte]*authEntry
inline map[[32]byte]*authEntry
lastReloadErrMsg string
failureLog logGate
throttleLog logGate
fileUserCount int
mu sync.RWMutex
buildMu sync.Mutex
reloadMu sync.Mutex
}
// newAuthStore builds an authStore from cfg's groups and inline users. It
@@ -1168,8 +1170,7 @@ func (a *authStore) shouldLogThrottleEngaged() (log bool, suppressed int64) {
// identify resolves r's presented API key to a user and their group. It
// reads "Authorization: Bearer <key>" (case-insensitive "bearer" prefix) or
// "x-api-key: <key>"; Bearer wins when both are present. The presented key
// is looked up by SHA-256 digest, then verified with a constant-time
// compare — the key itself is never retained.
// is looked up by its SHA-256 digest — the key itself is never retained.
//
// The presented key is verified FIRST, UNCONDITIONALLY (security review
// round 2, 2026-08-22, critical finding 1): a request carrying a VALID
@@ -1186,22 +1187,32 @@ func (a *authStore) shouldLogThrottleEngaged() (log bool, suppressed int64) {
// already-valid request can never be rejected by another caller's failed
// attempts, from the same address or not.
//
// Every failure path below — no key presented, key not found, or a
// (practically unreachable, see subtle.ConstantTimeCompare's own call
// site) digest mismatch — records one failure for the caller's source IP
// via recordAuthFailure; logAuthEvent (logger.go) separately decides, per
// call, whether to emit a routine failure line or a distinct throttle-
// engaged line by consulting authThrottled itself. No branch in this
// function ever changes its own return value based on throttle state —
// throttling is pure observability now (bounding the log's write volume
// and the tracker's own memory; authFailureTracker's doc comment,
// above), never enforcement: an attacker with no valid key gains nothing
// from this change (every failure path here already returned false
// either way, throttled or not), and the raw per-attempt cost this
// function pays (one SHA-256 digest, one map lookup, one
// ConstantTimeCompare) is already cheap enough — nanoseconds — that
// gating it early bought negligible protection against the cost itself,
// only the false-positive risk the paragraph above describes.
// Every failure path below — no key presented, or key not found — records
// one failure for the caller's source IP via recordAuthFailure;
// logAuthEvent (logger.go) separately decides, per call, whether to emit
// a routine failure line or a distinct throttle-engaged line by
// consulting authThrottled itself. No branch in this function ever
// changes its own return value based on throttle state — throttling is
// pure observability now (bounding the log's write volume and the
// tracker's own memory; authFailureTracker's doc comment, above), never
// enforcement: an attacker with no valid key gains nothing from this
// change (every failure path here already returned false either way,
// throttled or not), and the raw per-attempt cost this function pays (one
// SHA-256 digest, one map lookup) is already cheap enough — nanoseconds —
// that gating it early bought negligible protection against the cost
// itself, only the false-positive risk the paragraph above describes.
//
// No constant-time compare runs here (review-auth finding F7, 2026-09
// audit — a prior revision ran subtle.ConstantTimeCompare(digest[:],
// entry.digest[:]) after this exact lookup, but byDigest is populated
// ONLY as a.byDigest[entry.digest] = entry (newAuthStore/
// replaceFileUsers), so a map hit's entry.digest is the map key itself by
// construction: the compare was a tautology, always true when found is
// true, comparing digest against a value guaranteed identical to it. The
// real secret-matching step already happened inside the Go runtime's own
// map lookup, on the presented key's SHA-256 digest, not the key itself
// — there is no separate secret-bytes compare here for a non-constant-time
// implementation to leak timing from).
func (a *authStore) identify(r *http.Request) (*user, *group, bool) {
key, ok := presentedKey(r)
if ok {
@@ -1209,7 +1220,7 @@ func (a *authStore) identify(r *http.Request) (*user, *group, bool) {
a.mu.RLock()
entry, found := a.byDigest[digest]
a.mu.RUnlock()
if found && subtle.ConstantTimeCompare(digest[:], entry.digest[:]) == 1 {
if found {
return entry.user, entry.group, true
}
}