fix(gateway): close billing, limit, routing and translation gaps

Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
  The gateway falls back to a request-size estimate only on a 2xx
  stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
  billed at zero.
- Billing uses the built-in LiteLLM price table and honours
  modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
  saturates on overflow.
- Gemini thinking tokens count as completion tokens.

Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
  user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
  repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.

Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
  owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.

Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
  path.
- Retry-After is never jittered. A cancelled wait returns the
  context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
  as provider successes.

Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 09:44:31 +01:00
parent b3b8ae6ad5
commit 4c01c75276
62 changed files with 6064 additions and 1182 deletions
+8 -1
View File
@@ -40,6 +40,9 @@ const maxMultipartModelFieldBytes = 4096
// audio-speech routes, whose request bodies are both single JSON objects
// carrying a top-level "model" field, unlike audio-transcriptions'
// multipart body. A read failure or invalid/missing JSON is a 400,
// an oversize body is a 413 (finding 14 fix, review-routes.md — see
// readAndDecodeUnifiedBody's identical fix, routes_unified.go, for why
// readCapped replaces a plain io.LimitReader-and-decode here too), both
// already written to sw; ok reports whether the caller may proceed.
func (g *Gateway) decodeMediaJSONRequest(sw *statusTrackingWriter, r *http.Request) (req map[string]any, model string, ok bool) {
release, admitted := g.acquireBodyAdmission(sw, writeOAIError)
@@ -48,11 +51,15 @@ func (g *Gateway) decodeMediaJSONRequest(sw *statusTrackingWriter, r *http.Reque
return nil, "", false
}
body, err := io.ReadAll(io.LimitReader(r.Body, maxRequestBytes))
body, oversize, err := readCapped(r.Body, maxRequestBytes)
if err != nil {
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "cannot read request body")
return nil, "", false
}
if oversize {
writeOAIError(sw, http.StatusRequestEntityTooLarge, "invalid_request_error", "request body too large")
return nil, "", false
}
if err = json.Unmarshal(body, &req); err != nil {
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "invalid JSON body")
return nil, "", false