mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
fix(routes): address review findings on the Anthropic Messages route
Preserve usage when response translation fails after a successful, billed upstream call. The route previously discarded already-billed tokens on that path. A client could induce unlimited free usage against any budget via a truncated tool-call response. Reject a streaming request with 400 instead of 501. Check it before the rate limiter runs. A 501 made anthropic-sdk-python retry the request, since it retries any status of 500 or higher. Each retry also counted against the caller's rate limit before this fix. Preserve block order when translating an Anthropic message to OpenAI's shape. A tool_result block before other content must stay before it. OpenAI otherwise rejects the translated conversation. Echo the client's requested model alias in both the passthrough and the translated response paths. The passthrough branch previously returned the upstream's own model id instead. Forward the anthropic-beta header to an anthropic-type upstream through an explicit allowlist. Never forward Authorization, x-api-key, Cookie, or hop-by-hop headers. Let the adapter's own credential injection win regardless. Bill Anthropic's prompt-cache tokens. anthropicUsagePayload now reads cache_creation_input_tokens and cache_read_input_tokens too. It folds all three into the billed prompt count. This applies to this route, the existing outbound Anthropic path, and the native passthrough route. Extract the metered-request pipeline shared by the unified route and this route into runMeteredCall. Thread an envelope writer through it so each route answers in its own error shape. This removes the duplication the original brief warned against. It also lets the rate-limit and body-admission error paths answer in this route's own shape. Treat a non-boolean "stream" value as a streaming request. A string or number value previously bypassed the streaming check silently. Emit max_completion_tokens instead of max_tokens for OpenAI's reasoning models. Those models reject max_tokens with a 400. Preserve tool_result.is_error by prefixing string content with "Error: ". OpenAI's tool message carries no error field of its own. Log a warning when the thinking field is dropped instead of discarding it silently. Classify a response-translation failure distinctly in the log. An operator must never see a genuine translation bug reported as an upstream connection error. Document the route in README.md. Cover both auth headers, the non-streaming limitation, and which providers translate versus pass through. Add the reviewer's yaegi-check probes to tools/yaegi-check/main.go, adjusted for the fixes above.
This commit is contained in:
+202
-198
@@ -4,10 +4,8 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"strconv"
|
||||
)
|
||||
@@ -30,31 +28,50 @@ const messagesPath = "/v1/messages"
|
||||
// cache hit across them would hand a client the wrong envelope entirely.
|
||||
const cacheEndpointMessages = "messages"
|
||||
|
||||
// anthropicPassthroughForwardedHeaders is the ALLOWLIST of client request
|
||||
// headers callAnthropicMessagesPassthrough forwards to an anthropic-type
|
||||
// upstream (item 5 fix, 2026-08-22 review): Claude Code and the Anthropic
|
||||
// SDKs send anthropic-beta to opt into features this gateway must not
|
||||
// silently disable on a route named "passthrough" — 1M context,
|
||||
// token-efficient tools, computer use, among others. Deliberately an
|
||||
// allowlist, never a blanket forward: Authorization, x-api-key, Cookie,
|
||||
// and every hop-by-hop header must never reach an upstream this gateway
|
||||
// itself authenticates against on the caller's behalf. injectAuth is
|
||||
// always applied AFTER this copy (callAnthropicMessagesPassthrough,
|
||||
// below), so it always wins if a future entry here ever collided with
|
||||
// what injectAuth itself sets.
|
||||
var anthropicPassthroughForwardedHeaders = []string{"anthropic-beta"}
|
||||
|
||||
// handleMessages implements POST /v1/messages: the Anthropic Messages API
|
||||
// shape, inbound. It shares every pipeline step routes_unified.go's
|
||||
// runUnified already established for /v1/chat/completions and
|
||||
// /v1/embeddings — body-admission-gated decode, model resolution,
|
||||
// request-rate limiting, response caching, usage accounting, and
|
||||
// per-provider attempt recording — by calling the exact same functions
|
||||
// runUnified calls, rather than reimplementing any of them: g.admitRequest,
|
||||
// g.readAndDecodeUnifiedBody, g.registry.resolve, withAttemptRecorder,
|
||||
// g.limiter.account, unifiedCostMicros, cacheKey/groupCacheEnabled/
|
||||
// effectiveTTL, and cacheCaptureWriter. Only two things are genuinely
|
||||
// specific to this route and are NOT shared: which wire shape a request/
|
||||
// response is in (Anthropic, not OpenAI — see callAnthropicMessagesPassthrough/
|
||||
// callTranslatedMessages below) and which error envelope shape a failure
|
||||
// gets written in (Anthropic's {"type":"error","error":{...}}, not
|
||||
// OpenAI's {"error":{...}} — see writeAnthropicError and its callers,
|
||||
// below).
|
||||
// shape, inbound. From model resolution through response caching, usage
|
||||
// accounting, and adapter-error handling, this route shares ONE
|
||||
// implementation with routes_unified.go's runUnified — runMeteredCall,
|
||||
// routes_unified.go (item 7 fix, 2026-08-22 review) — rather than
|
||||
// duplicating that pipeline. Only two things are genuinely specific to
|
||||
// this route: which wire shape a request/response is in (Anthropic, not
|
||||
// OpenAI — see callAnthropicMessagesPassthrough/callTranslatedMessages
|
||||
// below) and which error envelope shape a failure gets written in
|
||||
// (Anthropic's {"type":"error","error":{...}}, not OpenAI's
|
||||
// {"error":{...}} — see writeAnthropicError and its callers, below).
|
||||
//
|
||||
// ORDERING (item 2 fix, 2026-08-22 review): body admission + decode run
|
||||
// BEFORE admitRequest (the rate-limit check) on THIS route — the
|
||||
// opposite order from runUnified's own finding 1a. anthropic-sdk-python
|
||||
// retries any status >= 500, and Claude Code streams on every call: the
|
||||
// previous ordering (admitRequest first, streaming rejected only after
|
||||
// decode) meant every real client hitting this not-yet-streaming-capable
|
||||
// route burned three requests of rate-limit budget per logical call —
|
||||
// verified repro: requestsPerMinute 2, three attempts -> 501, 501, 429,
|
||||
// day counter reads 3. Deciding "is this streaming" requires the decoded
|
||||
// body, so decode has to happen first; the trade-off finding 1a accepts
|
||||
// for runUnified (an adversarial caller paying nothing to retry a
|
||||
// malformed body) does not apply here, since the common case hitting
|
||||
// this ordering is not adversarial retries but every legitimate
|
||||
// streaming-capable client that exists today.
|
||||
func (g *Gateway) handleMessages(w http.ResponseWriter, r *http.Request, u *user, grp *group) {
|
||||
sw := &statusTrackingWriter{ResponseWriter: w}
|
||||
|
||||
scopes, ok := g.admitRequest(sw, u, grp)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
body, req, ok := g.readAndDecodeUnifiedBody(sw, r)
|
||||
body, req, ok := g.readAndDecodeUnifiedBody(sw, r, writeAnthropicError)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
@@ -65,99 +82,89 @@ func (g *Gateway) handleMessages(w http.ResponseWriter, r *http.Request, u *user
|
||||
return
|
||||
}
|
||||
|
||||
// Streaming is rejected cleanly and explicitly here, before model
|
||||
// resolution or any upstream call: this cut of the route ships
|
||||
// non-streaming only (brief scope) — a follow-up branch adds the
|
||||
// Anthropic SSE event translator (message_start/content_block_delta/
|
||||
// message_stop), a genuinely different event model from OpenAI's
|
||||
// stream shape this route already knows how to forward. A caller
|
||||
// that asked for streaming must see a clear, typed error, never a
|
||||
// silently-ignored flag answered non-streamed.
|
||||
if streaming, _ := req["stream"].(bool); streaming {
|
||||
writeAnthropicError(sw, http.StatusNotImplemented, "invalid_request_error", "streaming is not yet supported on /v1/messages")
|
||||
// Streaming is rejected cleanly and explicitly here, before the
|
||||
// rate-limit check, model resolution, or any upstream call: this cut
|
||||
// of the route ships non-streaming only (brief scope) — a follow-up
|
||||
// branch adds the Anthropic SSE event translator (message_start/
|
||||
// content_block_delta/message_stop), a genuinely different event
|
||||
// model from OpenAI's stream shape this route already knows how to
|
||||
// forward. A caller that asked for streaming must see a clear, typed
|
||||
// error, never a silently-ignored flag answered non-streamed.
|
||||
//
|
||||
// 400, not 501 (item 2 fix, 2026-08-22 review): anthropic-sdk-python's
|
||||
// retry classifies purely on status code and retries anything >= 500
|
||||
// — a 501 here turned every streaming call into an SDK-driven retry
|
||||
// storm against this route's own rate limit (this function's own doc
|
||||
// comment above has the verified repro). 400 maps to the SDK's
|
||||
// non-retryable BadRequestError, and invalid_request_error is already
|
||||
// the correct error type for a 400.
|
||||
if isStreamingRequested(req) {
|
||||
writeAnthropicError(sw, http.StatusBadRequest, "invalid_request_error", "streaming is not yet supported on /v1/messages")
|
||||
return
|
||||
}
|
||||
|
||||
adapter, upstreamModel, canonical, err := g.registry.resolve(requestedModel, grp)
|
||||
if err != nil {
|
||||
writeAnthropicModelResolveError(sw, err)
|
||||
scopes, ok := g.admitRequest(sw, u, grp, writeAnthropicError)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
req["model"] = upstreamModel
|
||||
delete(req, gatewayAliasKey) // defensive: this route never sets it, but a client-sent "__alias" field must never reach either upstream shape
|
||||
|
||||
cacheable := g.cache != nil && groupCacheEnabled(grp)
|
||||
var cacheKeyStr string
|
||||
if cacheable {
|
||||
cacheKeyStr = cacheKey(adapter.name(), upstreamModel, requestedModel, cacheEndpointMessages, req)
|
||||
if cached, hit := g.cache.lookup(cacheKeyStr); hit {
|
||||
sw.Header().Set("X-Llmgw-Cache", "hit")
|
||||
if cached.ContentType != "" {
|
||||
sw.Header().Set("Content-Type", cached.ContentType)
|
||||
g.runMeteredCall(sw, r, scopes, body, req, requestedModel, grp, cacheEndpointMessages, "messages route", writeAnthropicError, writeAnthropicProviderUpstreamError,
|
||||
func(a providerAdapter, ctx context.Context, w http.ResponseWriter, req map[string]any) (usage, error) {
|
||||
if a.typeName() == providerTypeAnthropic {
|
||||
return g.callAnthropicMessagesPassthrough(ctx, w, r.Header, a, req, requestedModel)
|
||||
}
|
||||
sw.WriteHeader(cached.Status)
|
||||
_, _ = sw.Write(cached.Body)
|
||||
return
|
||||
}
|
||||
sw.Header().Set("X-Llmgw-Cache", "miss")
|
||||
}
|
||||
|
||||
var respWriter http.ResponseWriter = sw
|
||||
var capture *cacheCaptureWriter
|
||||
if cacheable {
|
||||
capture = newCacheCaptureWriter(sw, g.cache.maxBodyBytes)
|
||||
respWriter = capture
|
||||
}
|
||||
|
||||
ctx := withAttemptRecorder(r.Context(), func(resp *http.Response, attemptErr error) {
|
||||
g.limiter.recordProviderAttempt(adapter.name(), upstreamModel, resp, attemptErr)
|
||||
})
|
||||
|
||||
var result usage
|
||||
var callErr error
|
||||
if adapter.typeName() == providerTypeAnthropic {
|
||||
result, callErr = g.callAnthropicMessagesPassthrough(ctx, respWriter, adapter, req)
|
||||
} else {
|
||||
result, callErr = g.callTranslatedMessages(ctx, respWriter, adapter, req, requestedModel)
|
||||
}
|
||||
|
||||
// Zero-usage fallback, mirroring runUnified's own (routes_unified.go):
|
||||
// a non-streaming failure always carries zero usage by contract, so
|
||||
// this is a no-op whenever callErr != nil (account below skips every
|
||||
// write once both total tokens and cost are zero). Unlike runUnified,
|
||||
// there is no streaming branch to skip here — streaming was already
|
||||
// rejected above, so every call into this route that reaches here is
|
||||
// non-streaming.
|
||||
if callErr == nil && result.total() == 0 {
|
||||
result.prompt = int64(math.Ceil(float64(len(body)) / 4))
|
||||
result.estimated = true
|
||||
}
|
||||
g.limiter.account(scopes, result, unifiedCostMicros(canonical, upstreamModel, result, g.cfg.Pricing))
|
||||
if result.estimated {
|
||||
g.logf("messages route: usage for model %q logged as estimated (%d prompt tokens derived from request body size, not the provider's reported usage)", canonical, result.prompt)
|
||||
}
|
||||
|
||||
if cacheable && callErr == nil && capture.status == http.StatusOK && !capture.oversize {
|
||||
g.cache.store(cacheKeyStr, capture.status, capture.contentType, capture.buf.Bytes(), effectiveTTL(g.cache, grp))
|
||||
}
|
||||
|
||||
if callErr != nil {
|
||||
g.handleMessagesAdapterError(sw, callErr, adapter.name())
|
||||
return
|
||||
}
|
||||
return g.callTranslatedMessages(ctx, w, a, req, requestedModel)
|
||||
})
|
||||
}
|
||||
|
||||
// isStreamingRequested reports whether req's "stream" field is anything
|
||||
// other than the JSON literal false or an absent key (item 9 fix,
|
||||
// 2026-08-22 review). A bare req["stream"].(bool) assertion silently
|
||||
// treats a non-bool value ("stream":"true", "stream":1) as false and lets
|
||||
// it through to an upstream that might itself honor a loosely-typed
|
||||
// truthy value — exactly the "silently answered non-streamed" outcome
|
||||
// this route must reject explicitly instead of allowing by accident.
|
||||
func isStreamingRequested(req map[string]any) bool {
|
||||
v, ok := req["stream"]
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return v != false
|
||||
}
|
||||
|
||||
// responseTranslationError wraps a failure translating an already-
|
||||
// successful, already-billed upstream response into this route's own
|
||||
// client-facing Anthropic shape (item 1/12 fix, 2026-08-22 review): the
|
||||
// upstream call itself succeeded, so its usage travels back to the
|
||||
// caller as call's own first return value regardless of this error (see
|
||||
// callTranslatedMessages/callAnthropicMessagesPassthrough, below) — a
|
||||
// client who can induce a translation failure (verified repro: a
|
||||
// provider response with usage prompt=5000/completion=900,
|
||||
// finish_reason "length", and a truncated tool-call arguments string)
|
||||
// must not get unlimited free tokens against their budget just because
|
||||
// this gateway's own re-encoding step broke after the provider already
|
||||
// did, and was paid for, the work. Kept distinct from an errUpstream-
|
||||
// wrapped error (a genuine connectivity/protocol failure) so
|
||||
// handleAdapterErrorEnvelope's (routes_unified.go) generic "upstream
|
||||
// connection error" branch never misclassifies a translation bug as a
|
||||
// network problem in the log — see that function's own
|
||||
// *responseTranslationError branch.
|
||||
type responseTranslationError struct {
|
||||
err error
|
||||
}
|
||||
|
||||
// Error implements the error interface.
|
||||
func (e *responseTranslationError) Error() string { return "translate response: " + e.err.Error() }
|
||||
|
||||
// callAnthropicMessagesPassthrough handles a /v1/messages request already
|
||||
// resolved to an anthropic-type provider: forward req to that provider's
|
||||
// own Messages endpoint UNTRANSLATED (brief: "pass through, no
|
||||
// translation of the message body") and copy its response back to w
|
||||
// exactly as Anthropic sent it. req's only mutation, applied by the
|
||||
// caller (handleMessages) before this function ever runs, is the "model"
|
||||
// translation of the message body") and copy its response back to w. req's
|
||||
// only mutation, applied by the shared pipeline (runMeteredCall,
|
||||
// routes_unified.go) before this function ever runs, is the "model"
|
||||
// field rewrite to the resolved upstream model id — the same rewrite
|
||||
// every route in this package applies before any adapter call
|
||||
// (runUnified, resolveMediaModel); it is request ROUTING, not message
|
||||
// translation.
|
||||
// every route in this package applies before any adapter call; it is
|
||||
// request ROUTING, not message translation.
|
||||
//
|
||||
// This deliberately bypasses providerAdapter.chatCompletion:
|
||||
// chatCompletion's contract (providers.go) is OpenAI-shape in, OpenAI-
|
||||
@@ -166,19 +173,27 @@ func (g *Gateway) handleMessages(w http.ResponseWriter, r *http.Request, u *user
|
||||
// would corrupt an already-Anthropic-shaped body instead of passing it
|
||||
// through.
|
||||
//
|
||||
// The response is written back byte-for-byte: this function reads
|
||||
// Anthropic's raw JSON only to extract usage (input_tokens/output_tokens,
|
||||
// via the existing anthropicResponseBody type, translate_anthropic.go —
|
||||
// reused rather than declared again) for accounting, and never re-
|
||||
// encodes what reaches the client. One accepted consequence: the
|
||||
// response's own "model" field carries the upstream model id Anthropic
|
||||
// actually served, not the client's requested alias — unlike every
|
||||
// translating adapter's gatewayAliasKey echo (routes_unified.go's own
|
||||
// doc comment on that mechanism), rewriting it here would mean decoding
|
||||
// and re-marshaling the full response body, which is itself a
|
||||
// translation step the brief's "no translation of the message body" rule
|
||||
// for this branch rules out.
|
||||
func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.ResponseWriter, adapter providerAdapter, req map[string]any) (usage, error) {
|
||||
// clientHeaders is the client's own incoming request headers, forwarded
|
||||
// through anthropicPassthroughForwardedHeaders' allowlist only (item 5
|
||||
// fix, above).
|
||||
//
|
||||
// The response body reaches the client almost byte-for-byte: this
|
||||
// function decodes it twice, read-only, to (a) extract usage
|
||||
// (input_tokens/output_tokens/cache_*_tokens, via the existing
|
||||
// anthropicResponseBody type, translate_anthropic.go — reused rather
|
||||
// than declared again) for accounting, and (b) rewrite only the "model"
|
||||
// field to requestedModel — the client's own requested alias — before
|
||||
// re-encoding and writing it (item 4 fix, 2026-08-22 review: the
|
||||
// PREVIOUS version left the upstream's own real model id in the
|
||||
// response, inconsistent with callTranslatedMessages' own alias echo on
|
||||
// the same route, and leaking the operator's real upstream model id,
|
||||
// which gatewayAliasKey exists specifically to prevent). Every other
|
||||
// field survives the round trip unchanged, since the rewrite decodes
|
||||
// into a generic map rather than a narrow typed struct — rewriting one
|
||||
// field this way is routing, not translating the message body.
|
||||
func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.ResponseWriter, clientHeaders http.Header, adapter providerAdapter, req map[string]any, requestedModel string) (usage, error) {
|
||||
delete(req, gatewayAliasKey) // this route never sets it via req itself, but a client-sent "__alias" field must never reach the real Anthropic API
|
||||
|
||||
// Plain type assertion, not an added providerAdapter interface method:
|
||||
// this branch only ever runs when the caller already confirmed
|
||||
// adapter.typeName() == providerTypeAnthropic, and buildAdapters
|
||||
@@ -195,6 +210,15 @@ func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.R
|
||||
|
||||
hdr := http.Header{}
|
||||
hdr.Set("Content-Type", "application/json")
|
||||
for _, name := range anthropicPassthroughForwardedHeaders {
|
||||
for _, v := range clientHeaders.Values(name) {
|
||||
hdr.Add(name, v)
|
||||
}
|
||||
}
|
||||
// injectAuth runs AFTER the allowlist copy above, so it always wins
|
||||
// (item 5 ruling) — in practice the two never collide (the allowlist
|
||||
// carries no auth-shaped header), but the ordering is deliberate and
|
||||
// documented, not incidental.
|
||||
adapter.injectAuth(&http.Request{Header: hdr})
|
||||
|
||||
resp, err := upstreamJSON(ctx, adapter.httpClient(), http.MethodPost, adapter.base()+anthropicMessagesPath, hdr, req, retry)
|
||||
@@ -213,8 +237,29 @@ func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.R
|
||||
}
|
||||
|
||||
var parsed anthropicResponseBody
|
||||
if err := json.Unmarshal(raw, &parsed); err != nil {
|
||||
return usage{}, fmt.Errorf("%w: decode anthropic response: %w", errUpstream, err)
|
||||
if unmarshalErr := json.Unmarshal(raw, &parsed); unmarshalErr != nil {
|
||||
return usage{}, fmt.Errorf("%w: decode anthropic response: %w", errUpstream, unmarshalErr)
|
||||
}
|
||||
// totalInputTokens folds Anthropic's prompt-cache counters
|
||||
// (cache_creation_input_tokens/cache_read_input_tokens) into the
|
||||
// billed prompt count (item 6 fix, 2026-08-22 review) — see its own
|
||||
// doc comment, translate_anthropic.go, for why dropping them
|
||||
// under-bills a cache-heavy caller (the common case for Claude Code)
|
||||
// by orders of magnitude.
|
||||
billed := usage{prompt: parsed.Usage.totalInputTokens(), completion: parsed.Usage.OutputTokens}
|
||||
|
||||
// The upstream call already succeeded and billed above by this
|
||||
// point — any failure from here on is a *responseTranslationError,
|
||||
// not a connectivity problem, so billed still reaches the caller
|
||||
// even if the field rewrite below somehow fails (item 1/12 fix).
|
||||
var envelope map[string]any
|
||||
if unmarshalErr := json.Unmarshal(raw, &envelope); unmarshalErr != nil {
|
||||
return billed, &responseTranslationError{err: fmt.Errorf("decode anthropic response envelope: %w", unmarshalErr)}
|
||||
}
|
||||
envelope["model"] = requestedModel
|
||||
out, err := json.Marshal(envelope)
|
||||
if err != nil {
|
||||
return billed, &responseTranslationError{err: fmt.Errorf("re-marshal anthropic response: %w", err)}
|
||||
}
|
||||
|
||||
if ct := resp.Header.Get("Content-Type"); ct != "" {
|
||||
@@ -223,9 +268,9 @@ func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.R
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
}
|
||||
w.WriteHeader(resp.StatusCode)
|
||||
_, _ = w.Write(raw)
|
||||
_, _ = w.Write(out)
|
||||
|
||||
return usage{prompt: parsed.Usage.InputTokens, completion: parsed.Usage.OutputTokens}, nil
|
||||
return billed, nil
|
||||
}
|
||||
|
||||
// callTranslatedMessages handles a /v1/messages request resolved to a
|
||||
@@ -239,11 +284,34 @@ func (g *Gateway) callAnthropicMessagesPassthrough(ctx context.Context, w http.R
|
||||
// instead of letting it reach the client directly, then translate that
|
||||
// captured OpenAI-shaped response back to Anthropic's shape
|
||||
// (anthropicResponseFromOpenAI) before writing the result to w.
|
||||
//
|
||||
// USAGE ON A TRANSLATION FAILURE (item 1 fix, 2026-08-22 review):
|
||||
// chatCompletion's own returned usage (result, below) is what the
|
||||
// upstream actually reported and this gateway already billed the
|
||||
// provider for — it is preserved and returned even when translating
|
||||
// THAT successful response into Anthropic's shape fails afterward
|
||||
// (verified repro: a response with usage prompt=5000/completion=900,
|
||||
// finish_reason "length", and a tool-call whose arguments string was
|
||||
// truncated mid-JSON by the provider's own max_tokens cutoff). The
|
||||
// PREVIOUS version of this function returned usage{} on that path,
|
||||
// discarding tokens the provider had already charged for — a client who
|
||||
// could reliably induce a truncated tool-call response got unlimited
|
||||
// free tokens against any budget, and this route was strictly less
|
||||
// robust than /v1/chat/completions, which forwards that same malformed
|
||||
// response to the client successfully instead of erroring on it.
|
||||
func (g *Gateway) callTranslatedMessages(ctx context.Context, w http.ResponseWriter, adapter providerAdapter, req map[string]any, requestedModel string) (usage, error) {
|
||||
openaiReq, err := openAIRequestFromAnthropic(req)
|
||||
openaiReq, dropped, err := openAIRequestFromAnthropic(req)
|
||||
if err != nil {
|
||||
return usage{}, err
|
||||
}
|
||||
// thinking (Anthropic's extended-thinking config) and any other
|
||||
// field this translator recognizes but cannot map has no OpenAI
|
||||
// equivalent — logged rather than silently discarded (item 11 fix,
|
||||
// 2026-08-22 review), so an operator can see a client's request was
|
||||
// only partially honored instead of the drop being invisible.
|
||||
for _, field := range dropped {
|
||||
g.warnf("messages route: request field %q has no OpenAI equivalent for provider %q and was dropped in translation", field, adapter.name())
|
||||
}
|
||||
|
||||
rec := newMessagesResponseRecorder()
|
||||
result, err := adapter.chatCompletion(ctx, rec, openaiReq)
|
||||
@@ -253,11 +321,11 @@ func (g *Gateway) callTranslatedMessages(ctx context.Context, w http.ResponseWri
|
||||
|
||||
out, err := anthropicResponseFromOpenAI(rec.body.Bytes(), requestedModel)
|
||||
if err != nil {
|
||||
return usage{}, err
|
||||
return result, &responseTranslationError{err: err}
|
||||
}
|
||||
b, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
return usage{}, fmt.Errorf("%w: marshal translated response: %w", errUpstream, err)
|
||||
return result, &responseTranslationError{err: fmt.Errorf("marshal translated response: %w", err)}
|
||||
}
|
||||
|
||||
status := rec.status
|
||||
@@ -280,8 +348,9 @@ func (g *Gateway) callTranslatedMessages(ctx context.Context, w http.ResponseWri
|
||||
// so the adapter's write is captured here and never forwarded; the real
|
||||
// ResponseWriter only ever sees the translated bytes callTranslatedMessages
|
||||
// writes afterward. Streaming can never reach this recorder: handleMessages
|
||||
// rejects "stream": true before either call path runs, so chatCompletion's
|
||||
// non-streaming forwardJSON is the only path that ever writes to it.
|
||||
// rejects a streaming request before either call path runs, so
|
||||
// chatCompletion's non-streaming forwardJSON is the only path that ever
|
||||
// writes to it.
|
||||
type messagesResponseRecorder struct {
|
||||
header http.Header
|
||||
body bytes.Buffer
|
||||
@@ -311,6 +380,8 @@ func (r *messagesResponseRecorder) Write(b []byte) (int, error) {
|
||||
// rate_limit_error, server_error), so an operator reading logs sees one
|
||||
// consistent taxonomy regardless of which route answered; only the JSON
|
||||
// shape around it differs, to match what this route's own clients expect.
|
||||
// Matches envelopeWriter's signature (routes_unified.go), so this route's
|
||||
// error paths thread straight into the shared pipeline there.
|
||||
func writeAnthropicError(w http.ResponseWriter, status int, errType, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
@@ -320,81 +391,14 @@ func writeAnthropicError(w http.ResponseWriter, status int, errType, msg string)
|
||||
})
|
||||
}
|
||||
|
||||
// writeAnthropicModelResolveError is writeModelResolveError's
|
||||
// (routes_unified.go) Anthropic-shaped counterpart: the SAME status-code
|
||||
// decision (aliasTargetError/errModelUnknown/errModelDenied), written
|
||||
// through writeAnthropicError instead of writeOAIError. Kept as its own
|
||||
// small function, rather than a routes_unified.go refactor threading an
|
||||
// envelope-writer parameter through, so this route's addition stays
|
||||
// isolated to its own file while a sibling branch is concurrently editing
|
||||
// registry.go/admin.go — the decision logic below is intentionally
|
||||
// identical to writeModelResolveError's, not independently re-derived.
|
||||
func writeAnthropicModelResolveError(w http.ResponseWriter, err error) {
|
||||
if aerr, ok := err.(*aliasTargetError); ok {
|
||||
writeAnthropicError(w, http.StatusNotFound, "invalid_request_error", aerr.Error())
|
||||
return
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, errModelUnknown):
|
||||
writeAnthropicError(w, http.StatusNotFound, "invalid_request_error", "unknown model")
|
||||
case errors.Is(err, errModelDenied):
|
||||
writeAnthropicError(w, http.StatusForbidden, "invalid_request_error", "model access denied")
|
||||
default:
|
||||
writeAnthropicError(w, http.StatusInternalServerError, "server_error", "internal error")
|
||||
}
|
||||
}
|
||||
|
||||
// handleMessagesAdapterError is handleAdapterError's (routes_unified.go)
|
||||
// Anthropic-shaped counterpart, covering the identical set of adapter
|
||||
// error cases with the identical yaegi-safety rationale (plain type
|
||||
// assertions for providerHTTPError/translateError, never errors.As —
|
||||
// see providerHTTPError's own doc comment, providers.go) but writing
|
||||
// through writeAnthropicError/writeAnthropicProviderUpstreamError instead
|
||||
// of writeOAIError/writeProviderUpstreamError, and its own "messages
|
||||
// route:" log prefix so a log line always names which route it came from.
|
||||
func (g *Gateway) handleMessagesAdapterError(sw *statusTrackingWriter, err error, providerName string) {
|
||||
sw.Header().Del("X-Llmgw-Cache")
|
||||
|
||||
if sw.wroteHeader {
|
||||
g.errorf("messages route: adapter error after response started (provider %q): %v", providerName, err)
|
||||
return
|
||||
}
|
||||
|
||||
if perr, ok := err.(*providerHTTPError); ok {
|
||||
writeAnthropicProviderUpstreamError(sw, providerName, perr)
|
||||
return
|
||||
}
|
||||
|
||||
if terr, ok := err.(*translateError); ok {
|
||||
status := http.StatusBadRequest
|
||||
if terr.notSupported {
|
||||
status = http.StatusNotImplemented
|
||||
}
|
||||
writeAnthropicError(sw, status, "invalid_request_error", terr.msg)
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, context.Canceled) {
|
||||
g.logf("messages route: client canceled request to provider %q: %v", providerName, err)
|
||||
return
|
||||
}
|
||||
|
||||
g.errorf("messages route: upstream connection error (provider %q): %v", providerName, err)
|
||||
writeAnthropicError(sw, http.StatusBadGateway, "server_error", "upstream connection error")
|
||||
}
|
||||
|
||||
// writeAnthropicProviderUpstreamError is writeProviderUpstreamError's
|
||||
// (routes_unified.go) Anthropic-shaped counterpart: perr's upstream body
|
||||
// is embedded under error.upstream (decoded to a JSON value when it
|
||||
// parses as one, left as a raw string otherwise) inside the Anthropic
|
||||
// {"type":"error","error":{...}} envelope instead of the OpenAI one.
|
||||
// is embedded under error.upstream (decodeUpstreamErrorBody,
|
||||
// routes_unified.go — shared with writeProviderUpstreamError, item 7/8
|
||||
// fix) inside the Anthropic {"type":"error","error":{...}} envelope
|
||||
// instead of the OpenAI one. Matches providerUpstreamErrorWriter's
|
||||
// signature (routes_unified.go).
|
||||
func writeAnthropicProviderUpstreamError(w http.ResponseWriter, providerName string, perr *providerHTTPError) {
|
||||
var upstream any = string(perr.body)
|
||||
var parsed any
|
||||
if json.Unmarshal(perr.body, &parsed) == nil {
|
||||
upstream = parsed
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(perr.status)
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{ // headers already committed; nothing useful to do on encode failure
|
||||
@@ -403,7 +407,7 @@ func writeAnthropicProviderUpstreamError(w http.ResponseWriter, providerName str
|
||||
"type": "upstream_error",
|
||||
"message": providerName + " upstream error",
|
||||
"code": strconv.Itoa(perr.status),
|
||||
"upstream": upstream,
|
||||
"upstream": decodeUpstreamErrorBody(perr),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user