Files
traefik-llmgateway/limits_helpers_test.go
T
lukaszraczylo 4c01c75276 fix(gateway): close billing, limit, routing and translation gaps
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
  The gateway falls back to a request-size estimate only on a 2xx
  stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
  billed at zero.
- Billing uses the built-in LiteLLM price table and honours
  modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
  saturates on overflow.
- Gemini thinking tokens count as completion tokens.

Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
  user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
  repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.

Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
  owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.

Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
  path.
- Retry-After is never jittered. A cancelled wait returns the
  context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
  as provider successes.

Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
2026-09-23 09:44:31 +01:00

93 lines
4.2 KiB
Go

package traefikllmgateway
import "time"
// This file restores single-key test conveniences for limiter/memoryStore
// counter operations that no longer have any production caller (deadcode
// audit, 2026-09 review): checkAndCount/account moved onto the batched
// incrMulti/getMulti/incrAndGetMulti paths in the 2026-08-21/22 perf
// reviews, which left incrBy/get — and the limiter-level helpers built on
// them (storeIncrBy, storeGet, failPolicyIncrBy, failPolicyGet, incrCounter,
// getCounter) — with no caller anywhere in production. They were removed
// from limits.go and from the counterStore interface. Every helper here is
// implemented via the SAME batched production path a single-key call would
// have used (a one-entry counterIncr slice, or a one-key string slice), so
// it exercises the identical fail-open/fail-closed/latch policy and EXPIRE-
// confirmation logic the removed originals did — not a parallel
// reimplementation. This keeps the ~130 existing test call sites (across
// limits_test.go and admin_test.go) compiling and passing unchanged.
// incrBy is a test-only single-key counterpart to memoryStore.incrMulti,
// restoring the pre-batching direct-increment API for memoryStore-level
// unit tests that need a bare key with no window-derived enforceTTL floor
// (incrMulti's own callers always supply one via newCounterIncr; a bare
// test key has none, matching the removed production incrBy's own
// no-floor clamp).
func (m *memoryStore) incrBy(key string, n int64, ttl time.Duration) (int64, error) {
return m.applyIncr(key, n, clampTTL(ttl, 0)), nil
}
// storeIncrBy is a test-only single-key counterpart to storeIncrMulti,
// letting the limiter's fail-open/fail-closed/latch tests exercise that
// policy against one key without building a one-entry counterIncr slice at
// every call site. Delegates to storeIncrMulti itself, so it drives the
// exact same policy branches (nil store / latched / store error / store
// success) a single-entry batch would.
func (l *limiter) storeIncrBy(key string, n int64, ttl time.Duration) (v int64, ok bool) {
vs, ok := l.storeIncrMulti([]counterIncr{{key: key, delta: n, ttl: ttl}})
if !ok || len(vs) == 0 {
return 0, ok
}
return vs[0], true
}
// storeGet mirrors storeIncrBy for a read, delegating to storeGetMulti.
func (l *limiter) storeGet(key string) (v int64, ok bool) {
vs, ok := l.storeGetMulti([]string{key})
if !ok || len(vs) == 0 {
return 0, ok
}
return vs[0], true
}
// failPolicyIncrBy applies the limiter's fail-open/fail-closed policy for
// an incrBy the caller has decided not to attempt against the configured
// store (it just failed, or the store-down latch is open): failOpen=true
// counts key in the fallback instead; failOpen=false reports the
// operation as failed. Exercised directly (not via storeIncrBy/
// storeIncrMulti) by tests that isolate this branching from the store-call
// decision above it.
func (l *limiter) failPolicyIncrBy(key string, n int64, ttl time.Duration) (int64, bool) {
if !l.failOpen {
return 0, false
}
v, _ := l.fallback.incrBy(key, n, ttl)
return v, true
}
// failPolicyGet mirrors failPolicyIncrBy for a read.
func (l *limiter) failPolicyGet(key string) (int64, bool) {
if !l.failOpen {
return 0, false
}
v, _ := l.fallback.get(key)
return v, true
}
// incrCounter increments the (kind, id, metric, window) counter at t by n
// and returns its new value, together with whether the operation
// succeeded under the limiter's fail-open/fail-closed policy — the
// limiter-level single-key seeding helper tests use to seed counters
// directly (admin_test.go's TestAdminUsage_MathAgainstSeededCounters and
// others), implemented via the batched storeIncrMulti path.
func (l *limiter) incrCounter(kind, id, metric, window string, t time.Time, n int64, ttl time.Duration) (int64, bool) {
return l.storeIncrBy(windowKey(kind, id, metric, window, t), n, ttl)
}
// getCounter returns the (kind, id, metric, window) counter's value at t,
// together with whether the read succeeded under the limiter's fail-open/
// fail-closed policy, implemented via the batched storeGetMulti path.
func (l *limiter) getCounter(kind, id, metric, window string, t time.Time) (int64, bool) {
return l.storeGet(windowKey(kind, id, metric, window, t))
}