mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-21 05:57:53 +00:00
fix(gateway): close billing, limit, routing and translation gaps
Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
+8
-1
@@ -40,6 +40,9 @@ const maxMultipartModelFieldBytes = 4096
|
||||
// audio-speech routes, whose request bodies are both single JSON objects
|
||||
// carrying a top-level "model" field, unlike audio-transcriptions'
|
||||
// multipart body. A read failure or invalid/missing JSON is a 400,
|
||||
// an oversize body is a 413 (finding 14 fix, review-routes.md — see
|
||||
// readAndDecodeUnifiedBody's identical fix, routes_unified.go, for why
|
||||
// readCapped replaces a plain io.LimitReader-and-decode here too), both
|
||||
// already written to sw; ok reports whether the caller may proceed.
|
||||
func (g *Gateway) decodeMediaJSONRequest(sw *statusTrackingWriter, r *http.Request) (req map[string]any, model string, ok bool) {
|
||||
release, admitted := g.acquireBodyAdmission(sw, writeOAIError)
|
||||
@@ -48,11 +51,15 @@ func (g *Gateway) decodeMediaJSONRequest(sw *statusTrackingWriter, r *http.Reque
|
||||
return nil, "", false
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, maxRequestBytes))
|
||||
body, oversize, err := readCapped(r.Body, maxRequestBytes)
|
||||
if err != nil {
|
||||
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "cannot read request body")
|
||||
return nil, "", false
|
||||
}
|
||||
if oversize {
|
||||
writeOAIError(sw, http.StatusRequestEntityTooLarge, "invalid_request_error", "request body too large")
|
||||
return nil, "", false
|
||||
}
|
||||
if err = json.Unmarshal(body, &req); err != nil {
|
||||
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "invalid JSON body")
|
||||
return nil, "", false
|
||||
|
||||
Reference in New Issue
Block a user