mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-16 05:37:28 +00:00
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
@@ -39,6 +39,20 @@ func validateCacheConfig(cc CacheConfig) (time.Duration, int, error) {
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("llmgateway: cache.ttl: %w", err)
|
||||
}
|
||||
// verify-core fix (round 4): F8 (2026-09 review) turned a zero/
|
||||
// negative ttl into a hard construction error here, matching
|
||||
// groupCacheTTL's own per-group-override validation below — but that
|
||||
// broke a previously WORKING config on upgrade: setEx (resp.go)
|
||||
// already floors a non-positive TTL to 1s rather than erroring, so a
|
||||
// "0s" cache.ttl silently ran as a 1-second cache before F8 existed,
|
||||
// not as "off". Refusing to build the whole middleware over a config
|
||||
// that merely caches less usefully than the operator probably
|
||||
// intended violates the house rule that an upgrade with no config
|
||||
// change preserves prior behavior. Restored to accept-and-warn:
|
||||
// buildResponseCache (below) logs a warning naming the effective 1s
|
||||
// floor once client (the constructed logf) is available; ttl itself
|
||||
// is returned here exactly as parsed, unclamped — newResponseCache/
|
||||
// setEx's own floor is what actually takes effect at write time.
|
||||
|
||||
maxBody := cc.MaxBodyBytes
|
||||
if maxBody == 0 {
|
||||
@@ -78,7 +92,7 @@ type cacheStore interface {
|
||||
// is deliberately NOT a construction error, unlike the case above.
|
||||
// - cc.Enabled is true and valid and client is non-nil: a working
|
||||
// *responseCache over client.
|
||||
func buildResponseCache(cc CacheConfig, client *respClient, logf, errorf func(format string, args ...any)) (*responseCache, error) {
|
||||
func buildResponseCache(cc CacheConfig, client *respClient, logf, warnf, errorf func(format string, args ...any)) (*responseCache, error) {
|
||||
if !cc.Enabled {
|
||||
return nil, nil
|
||||
}
|
||||
@@ -86,6 +100,17 @@ func buildResponseCache(cc CacheConfig, client *respClient, logf, errorf func(fo
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// verify-core fix (round 4): a zero/negative cache.ttl is accepted
|
||||
// (validateCacheConfig's own doc comment above explains why this was
|
||||
// restored from a hard error), but it is not silently free of
|
||||
// consequence either — setEx (resp.go) floors it to a 1-second TTL,
|
||||
// so entries expire almost immediately and every lookup effectively
|
||||
// misses while still paying a SET. Warn once at construction so an
|
||||
// operator who meant "0s" as "disable the cache" notices, rather than
|
||||
// silently running a cache that never actually caches anything.
|
||||
if ttl <= 0 {
|
||||
warnf("cache: ttl %q is zero or negative; every cached entry will expire after 1 second (the store's own floor), not stay disabled — set cache.enabled=false to actually turn caching off", cc.TTL)
|
||||
}
|
||||
if client == nil {
|
||||
logf("cache: enabled but redis is not configured; response caching disabled")
|
||||
return nil, nil
|
||||
|
||||
Reference in New Issue
Block a user