mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
fix(routes): address review findings on the Anthropic Messages route
Preserve usage when response translation fails after a successful, billed upstream call. The route previously discarded already-billed tokens on that path. A client could induce unlimited free usage against any budget via a truncated tool-call response. Reject a streaming request with 400 instead of 501. Check it before the rate limiter runs. A 501 made anthropic-sdk-python retry the request, since it retries any status of 500 or higher. Each retry also counted against the caller's rate limit before this fix. Preserve block order when translating an Anthropic message to OpenAI's shape. A tool_result block before other content must stay before it. OpenAI otherwise rejects the translated conversation. Echo the client's requested model alias in both the passthrough and the translated response paths. The passthrough branch previously returned the upstream's own model id instead. Forward the anthropic-beta header to an anthropic-type upstream through an explicit allowlist. Never forward Authorization, x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own credential injection win regardless. Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads cache_creation_input_tokens and cache_read_input_tokens too. It folds all three into the billed prompt count. This applies to this route, the existing outbound Anthropic path, and the native passthrough route. Extract the metered-request pipeline shared by the unified route and this route into runMeteredCall. Thread an envelope writer through it so each route answers in its own error shape. This removes the duplication the original brief warned against. It also lets the rate-limit and body-admission error paths answer in this route's own shape. Treat a non-boolean "stream" value as a streaming request. A string or number value previously bypassed the streaming check silently. Emit max_completion_tokens instead of max_tokens for OpenAI's reasoning models. Those models reject max_tokens with a 400. Preserve tool_result.is_error by prefixing string content with "Error: ". OpenAI's tool message carries no error field of its own. Log a warning when the thinking field is dropped instead of discarding it silently. Classify a response-translation failure distinctly in the log. An operator must never see a genuine translation bug reported as an upstream connection error. Document the route in README.md. Cover both auth headers, the non-streaming limitation, and which providers translate versus pass through. Add the reviewer's yaegi-check probes to tools/yaegi-check/main.go, adjusted for the fixes above.
This commit is contained in:
+12
-1
@@ -426,6 +426,17 @@ type passthroughUsagePayload struct {
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
InputTokens int64 `json:"input_tokens"`
|
||||
OutputTokens int64 `json:"output_tokens"`
|
||||
// CacheCreationInputTokens/CacheReadInputTokens (item 6 fix,
|
||||
// 2026-08-22 review — applied "for the anthropic provider
|
||||
// generally", not only translate_anthropic.go's own
|
||||
// anthropicUsagePayload): Anthropic's native passthrough
|
||||
// response reports these two prompt-cache counters separately
|
||||
// from InputTokens, and extractPassthroughUsage's own
|
||||
// providerTypeAnthropic case folds them in below so a
|
||||
// cache-heavy caller's usage is not silently under-billed on
|
||||
// this route either.
|
||||
CacheCreationInputTokens int64 `json:"cache_creation_input_tokens"`
|
||||
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
|
||||
} `json:"usage"`
|
||||
UsageMetadata struct {
|
||||
PromptTokenCount int64 `json:"promptTokenCount"`
|
||||
@@ -459,7 +470,7 @@ func extractPassthroughUsage(typeName, providerName string, body []byte) (usage,
|
||||
case providerTypeOpenAI:
|
||||
return usage{prompt: payload.Usage.PromptTokens, completion: payload.Usage.CompletionTokens}, model, err
|
||||
case providerTypeAnthropic:
|
||||
return usage{prompt: payload.Usage.InputTokens, completion: payload.Usage.OutputTokens}, model, err
|
||||
return usage{prompt: payload.Usage.InputTokens + payload.Usage.CacheCreationInputTokens + payload.Usage.CacheReadInputTokens, completion: payload.Usage.OutputTokens}, model, err
|
||||
case providerTypeGemini:
|
||||
return usage{prompt: payload.UsageMetadata.PromptTokenCount, completion: payload.UsageMetadata.CandidatesTokenCount}, model, err
|
||||
default:
|
||||
|
||||
Reference in New Issue
Block a user