fix(gateway): close billing, limit, routing and translation gaps

Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
  The gateway falls back to a request-size estimate only on a 2xx
  stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
  billed at zero.
- Billing uses the built-in LiteLLM price table and honours
  modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
  saturates on overflow.
- Gemini thinking tokens count as completion tokens.

Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
  user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
  repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.

Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
  owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.

Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
  path.
- Retry-After is never jittered. A cancelled wait returns the
  context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
  as provider successes.

Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 09:44:31 +01:00
parent b3b8ae6ad5
commit 4c01c75276
62 changed files with 6064 additions and 1182 deletions
+8
View File
@@ -0,0 +1,8 @@
{
"model": "claude-opus-5",
"messages": [
{"role": "developer", "content": "Always answer in one sentence."},
{"role": "user", "content": "Hello, how are you?"}
],
"max_tokens": 100
}
+8
View File
@@ -0,0 +1,8 @@
{
"model": "claude-opus-5",
"system": "Always answer in one sentence.",
"messages": [
{"role": "user", "content": "Hello, how are you?"}
],
"max_tokens": 100
}
+10
View File
@@ -0,0 +1,10 @@
{
"model": "claude-opus-5",
"messages": [
{"role": "user", "content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;charset=binary;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAAB"}}
]}
],
"max_tokens": 300
}
+10
View File
@@ -0,0 +1,10 @@
{
"model": "claude-opus-5",
"messages": [
{"role": "user", "content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAAB"}}
]}
],
"max_tokens": 300
}
@@ -0,0 +1,15 @@
{
"model": "claude-opus-5",
"messages": [
{"role": "user", "content": "What's the weather in Paris?"},
{"role": "assistant", "content": null, "tool_calls": [
{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\":\"Paris\"}"}}
]},
{"role": "tool", "tool_call_id": "call_1", "content": "72F and sunny"}
],
"max_tokens": 200,
"tools": [
{"type": "function", "function": {"name": "get_weather", "description": "gets the weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}}
],
"tool_choice": "none"
}
@@ -0,0 +1,17 @@
{
"model": "claude-opus-5",
"messages": [
{"role": "user", "content": "What's the weather in Paris?"},
{"role": "assistant", "content": [
{"type": "tool_use", "id": "call_1", "name": "get_weather", "input": {"location": "Paris"}}
]},
{"role": "user", "content": [
{"type": "tool_result", "tool_use_id": "call_1", "content": "72F and sunny"}
]}
],
"max_tokens": 200,
"tools": [
{"name": "get_weather", "description": "gets the weather", "input_schema": {"type": "object", "properties": {"location": {"type": "string"}}}}
],
"tool_choice": {"type": "none"}
}
+12
View File
@@ -0,0 +1,12 @@
{
"id": "msg_01REFUSE",
"type": "message",
"role": "assistant",
"content": [
{"type": "text", "text": "I can't help with that."}
],
"model": "claude-opus-5",
"stop_reason": "refusal",
"stop_sequence": null,
"usage": {"input_tokens": 9, "output_tokens": 6}
}
+14
View File
@@ -0,0 +1,14 @@
{
"id": "chatcmpl-msg_01REFUSE",
"object": "chat.completion",
"created": 1734000000,
"model": "gw-anthropic-model",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "I can't help with that."},
"finish_reason": "content_filter"
}
],
"usage": {"prompt_tokens": 9, "completion_tokens": 6, "total_tokens": 15}
}
+7
View File
@@ -0,0 +1,7 @@
{
"model": "gemini-2.5-pro",
"messages": [
{"role": "developer", "content": "Always answer in one sentence."},
{"role": "user", "content": "Hello there"}
]
}
+6
View File
@@ -0,0 +1,6 @@
{
"systemInstruction": {"parts": [{"text": "Always answer in one sentence."}]},
"contents": [
{"role": "user", "parts": [{"text": "Hello there"}]}
]
}
+19
View File
@@ -0,0 +1,19 @@
{
"candidates": [
{
"content": {
"role": "model",
"parts": [{"text": "The answer is 42."}]
},
"finishReason": "STOP"
}
],
"usageMetadata": {
"promptTokenCount": 10,
"candidatesTokenCount": 6,
"thoughtsTokenCount": 500,
"totalTokenCount": 516
},
"modelVersion": "gemini-2.5-pro-001",
"responseId": "resp_THINK1"
}
+14
View File
@@ -0,0 +1,14 @@
{
"id": "chatcmpl-resp_THINK1",
"object": "chat.completion",
"created": 1734000000,
"model": "gw-gemini-model",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "The answer is 42."},
"finish_reason": "stop"
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 506, "total_tokens": 516}
}