mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-12 04:54:45 +00:00
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
115 lines
5.0 KiB
Go
115 lines
5.0 KiB
Go
package traefikllmgateway
|
|
|
|
import (
|
|
"fmt"
|
|
"net/http"
|
|
"os"
|
|
)
|
|
|
|
// logf writes an info-level log line to stderr. No timestamps — Traefik
|
|
// adds its own. Never log key material.
|
|
func (g *Gateway) logf(format string, args ...any) {
|
|
fmt.Fprintf(os.Stderr, "llmgw[%s] INFO %s\n", g.name, fmt.Sprintf(format, args...))
|
|
}
|
|
|
|
// warnf writes a warning-level log line to stderr. No timestamps —
|
|
// Traefik adds its own. Never log key material.
|
|
//
|
|
// Use this for a condition an operator may want to know about but that
|
|
// the gateway resolved by itself, deterministically, with no request
|
|
// affected — a model id served by two providers, for instance. Those
|
|
// lines were ERROR until 2026-08-22, which put 144 of them in a healthy
|
|
// pod's boot log on a cluster with overlapping catalogs and made real
|
|
// errors harder to find. Reserve errorf for something actually broken.
|
|
func (g *Gateway) warnf(format string, args ...any) {
|
|
fmt.Fprintf(os.Stderr, "llmgw[%s] WARN %s\n", g.name, fmt.Sprintf(format, args...))
|
|
}
|
|
|
|
// errorf writes an error-level log line to stderr. No timestamps — Traefik
|
|
// adds its own. Never log key material.
|
|
func (g *Gateway) errorf(format string, args ...any) {
|
|
fmt.Fprintf(os.Stderr, "llmgw[%s] ERROR %s\n", g.name, fmt.Sprintf(format, args...))
|
|
}
|
|
|
|
// pricingWarn is g's own per-instance pricing-warning sink, passed to
|
|
// costMicrosKnownFor (pricing.go) so an unpriced-model warning always logs
|
|
// under this instance's `llmgw[name]` prefix (finding F11 / review-auth
|
|
// F8, 2026-09 review: a package-level warn func replaced on every New()
|
|
// call made the most recently constructed instance log every other
|
|
// instance's warnings).
|
|
//
|
|
// A plain method, not a stored closure field: it reads g.name fresh on
|
|
// every call, exactly like logf/warnf/errorf above, so there is nothing
|
|
// new to keep in sync across a config reload that rebuilds g.
|
|
func (g *Gateway) pricingWarn(msg string) {
|
|
if msg == warnCapMessage {
|
|
g.logf("%s", msg)
|
|
return
|
|
}
|
|
g.logf("pricing: no price configured for model %q; cost will be recorded as 0", msg)
|
|
}
|
|
|
|
// logAuthEvent logs one authentication attempt, at the point one of
|
|
// authStore.identify's six call sites resolves it. A success logs the
|
|
// resolved user's name alongside the request's method and path. Neither
|
|
// line this function ever writes includes the presented API key itself —
|
|
// identify resolves a key to a *user and hands this function only that
|
|
// user's name, never the key it authenticated.
|
|
//
|
|
// A failure logs one of two distinct lines (security review round 2,
|
|
// 2026-08-22, important finding 5): if the caller's source IP
|
|
// (clientIP) is currently authThrottled, a distinct "throttle engaged"
|
|
// line naming the IP and the limit it exceeded — an operator seeing this
|
|
// knows immediately that a SOURCE, not just an individual request, needs
|
|
// attention, which the routine per-attempt line alone does not convey.
|
|
// Otherwise the routine failure line (method, path, remote address).
|
|
// Both lines are independently rate-limited (authStore.shouldLog
|
|
// ThrottleEngaged / shouldLogAuthFailure, auth.go — the identical
|
|
// technique limiter.logStoreError, limits.go, already applies to
|
|
// store-error lines) so an unauthenticated caller driving unlimited
|
|
// failed attempts can never drive unlimited synchronous stderr writes on
|
|
// the shared Traefik ingress — a slow log pipe would otherwise stall
|
|
// request handling for every tenant, not just the one failing auth. Each
|
|
// gate reports how many events it suppressed since its last written line
|
|
// (security review round 2, important finding 6): a single global gate
|
|
// would otherwise make several DISTINCT attacker IPs failing within one
|
|
// interval indistinguishable from one noisy IP — suppressedSuffix
|
|
// appends that count to the line when non-zero. The success line stays
|
|
// completely unbounded — see shouldLogAuthFailure's own doc comment for
|
|
// why that is safe.
|
|
func (g *Gateway) logAuthEvent(ok bool, userName string, r *http.Request) {
|
|
if !ok {
|
|
ip := clientIP(r)
|
|
if g.auth.authThrottled(ip) {
|
|
if log, suppressed := g.auth.shouldLogThrottleEngaged(); log {
|
|
g.logf("auth throttle: source %s exceeded %d failed attempts in the last %s%s", ip, authFailureLimit, authFailureWindow, suppressedSuffix(suppressed))
|
|
}
|
|
return
|
|
}
|
|
if log, suppressed := g.auth.shouldLogAuthFailure(); log {
|
|
g.logf("auth failed: %s %q from %s%s", r.Method, r.URL.Path, r.RemoteAddr, suppressedSuffix(suppressed))
|
|
}
|
|
return
|
|
}
|
|
g.logf("auth ok: user %q %s %q", userName, r.Method, r.URL.Path)
|
|
}
|
|
|
|
// suppressedSuffix formats n (a logGate's own suppressed-event count,
|
|
// auth.go) as a log-line suffix, or "" when there is nothing to report.
|
|
func suppressedSuffix(n int64) string {
|
|
if n <= 0 {
|
|
return ""
|
|
}
|
|
return fmt.Sprintf(" (%d more suppressed since last log)", n)
|
|
}
|
|
|
|
// authEventUserName extracts u's name for logAuthEvent's userName
|
|
// argument, or "" when u is nil — identify's failure path, which never
|
|
// resolves a user.
|
|
func authEventUserName(u *user) string {
|
|
if u == nil {
|
|
return ""
|
|
}
|
|
return u.name
|
|
}
|