mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-20 05:56:56 +00:00
merge: serve the Anthropic Messages API inbound
# Conflicts: # tools/yaegi-check/main.go
This commit is contained in:
@@ -28,6 +28,10 @@ combination.
|
||||
and from Anthropic's and Google Gemini's own wire formats, so any
|
||||
OpenAI-SDK-compatible client can talk to all three providers through one
|
||||
API shape.
|
||||
- An Anthropic-compatible surface (`POST /v1/messages`) for Anthropic SDK
|
||||
clients — including Claude Code — to talk to any configured provider in
|
||||
Anthropic's own Messages API shape; see [Anthropic Messages API
|
||||
(`/v1/messages`)](#anthropic-messages-api-v1messages).
|
||||
- Native passthrough per provider (`/{provider}/...`) for callers that want
|
||||
a provider's own wire format untouched, with the gateway injecting that
|
||||
provider's own API key and still enforcing auth and limits.
|
||||
@@ -998,6 +1002,99 @@ only once it has something to authorize:
|
||||
will see every Gemini passthrough request denied. Route Gemini traffic
|
||||
for a model-restricted group through the unified `/v1/*` API instead.
|
||||
|
||||
## Anthropic Messages API (`/v1/messages`)
|
||||
|
||||
`POST /v1/messages` serves Anthropic SDK clients — including Claude
|
||||
Code — directly, in Anthropic's own Messages API request/response shape:
|
||||
the inbound counterpart to the outbound-only Anthropic support the
|
||||
unified routes already had. From model resolution through response
|
||||
caching, usage accounting, and adapter-error handling, this route shares
|
||||
one implementation with `/v1/chat/completions`
|
||||
(`runMeteredCall`, `routes_unified.go`) — the same group authorization
|
||||
(see [`GroupConfig`](#groupconfig)), [limits and
|
||||
accounting](#limits-and-accounting), and [caching](#caching) semantics
|
||||
apply. Only the wire shape and the client-facing error envelope are
|
||||
specific to this route.
|
||||
|
||||
### Authentication
|
||||
|
||||
Anthropic SDKs authenticate with the `x-api-key` header, not
|
||||
`Authorization: Bearer`. This route accepts **both** — the identical
|
||||
credential lookup (`authStore.identify`, `auth.go`) every other route
|
||||
uses already reads either header, `Authorization: Bearer` taking
|
||||
priority if a request somehow carries both — so no separate credential
|
||||
path exists for this route. A failed auth answers in Anthropic's own
|
||||
error shape (`{"type":"error","error":{"type":"authentication_error",
|
||||
"message":"..."}}`), never the OpenAI-shaped
|
||||
`{"error":{"type":...,"code":...}}` every other route uses.
|
||||
|
||||
### Translation vs. passthrough
|
||||
|
||||
Behavior depends entirely on the resolved provider's configured `type`:
|
||||
|
||||
| Resolved provider `type` | Behavior |
|
||||
|---|---|
|
||||
| `anthropic` | **Passthrough.** The request reaches that provider's own `/v1/messages` endpoint untranslated — no message-body translation of any kind. The response reaches the client BYTE-FOR-BYTE identical, with one exception: `"model"` is rewritten to the client's own requested alias, the same routing rewrite every route applies before any adapter call — not a translation of the response body. This is a real byte-level guarantee, not an approximation: every other field's value is copied from the upstream's raw bytes without ever passing through a Go value, so a JSON integer beyond `float64`'s exact range (inside, say, a `tool_use` block's arbitrary `input`) survives intact instead of being silently rounded. |
|
||||
| `openai` or `gemini` | **Full bidirectional translation**, through the same `providerAdapter.chatCompletion` a `/v1/chat/completions` request against that provider already uses — its contract is OpenAI-shape in, OpenAI-shape out regardless of provider type. The Anthropic-shaped request is translated to OpenAI's shape before the call; that provider's OpenAI-shaped response is translated back to Anthropic's shape before it reaches the client. |
|
||||
|
||||
A few translation specifics worth knowing:
|
||||
|
||||
- **`anthropic-beta` is forwarded** to an `anthropic`-type upstream (an
|
||||
explicit allowlist — `Authorization`, `x-api-key`, `Cookie`, and every
|
||||
hop-by-hop header are never forwarded, and the gateway's own credential
|
||||
injection always overrides whatever the allowlist copied), so Claude
|
||||
Code and the SDKs can still opt into 1M context, token-efficient
|
||||
tools, computer use, and other beta features through this route.
|
||||
- **Prompt-cache tokens are billed.** Anthropic's
|
||||
`cache_creation_input_tokens` and `cache_read_input_tokens` fold into
|
||||
the same combined prompt-token count every other route's budgets
|
||||
track, alongside `input_tokens` — not dropped, and not priced at a
|
||||
separate cache-tier rate.
|
||||
- **`tool_result.is_error`** has no native OpenAI tool-message field.
|
||||
`content` — a string, Anthropic's content-block array form (Claude
|
||||
Code sends this), or absent — is normalized to a plain string first
|
||||
(OpenAI's own tool message requires one; a `null` content is
|
||||
rejected), then gets an `"Error: "` prefix, so an OpenAI-backed
|
||||
agentic loop still sees the failure signal regardless of which shape
|
||||
the client sent.
|
||||
- **Reasoning models** (OpenAI's `o1`/`o3`/`o4`/`gpt-5` families) get
|
||||
`max_completion_tokens` instead of `max_tokens` — the field those
|
||||
models reject with a 400.
|
||||
- **`thinking`** (Anthropic's extended-thinking config) has no OpenAI
|
||||
equivalent and is dropped when translating to an `openai`/`gemini`
|
||||
target; the gateway logs a warning naming the dropped field rather
|
||||
than discarding it silently.
|
||||
|
||||
### Streaming is not yet supported
|
||||
|
||||
`"stream"` set to anything other than the JSON literal `false`, `null`,
|
||||
or an absent key gets a clean **400** `invalid_request_error` — never
|
||||
silently ignored, never answered non-streamed. `null` is treated the
|
||||
same as unset, so an SDK that serializes an unset optional field as
|
||||
explicit `null` never gets a false-positive rejection on an ordinary
|
||||
non-streaming call. This is deliberately a 400, not a 501: Anthropic's
|
||||
own SDKs retry any status ≥ 500, so a 501 here would turn every
|
||||
streaming call — the default for Claude Code — into a multi-request
|
||||
retry storm against the caller's own request-rate limit. The rejection
|
||||
happens *before* the rate limiter runs, so a streaming-rejected request
|
||||
never consumes budget either. A follow-up release adds the Anthropic SSE
|
||||
event translator (`message_start`/`content_block_delta`/`message_stop`)
|
||||
— a genuinely different event model from OpenAI's own stream shape this
|
||||
plugin already forwards on `/v1/chat/completions`.
|
||||
|
||||
### Route takeover
|
||||
|
||||
On a deployment running `passthroughUnknown: true`, `POST /v1/messages`
|
||||
used to fall through to `next` (the router's own backing service) before
|
||||
this route existed — nothing in this plugin recognized the path. Now the
|
||||
gateway intercepts it unconditionally: `next` is never reached for this
|
||||
path again, regardless of `passthroughUnknown`. This is intended — the
|
||||
whole point of this route is for the gateway itself to answer
|
||||
`/v1/messages` — but it is a real behavior change for any deployment
|
||||
currently proxying `/v1/messages` downstream through
|
||||
`passthroughUnknown`. Route that traffic elsewhere, or expect the
|
||||
gateway to answer it directly, before upgrading.
|
||||
|
||||
## Image and audio endpoints
|
||||
|
||||
`POST /v1/images/generations`, `POST /v1/audio/speech`, and
|
||||
@@ -1416,13 +1513,29 @@ the plugin's only non-standard-library runtime dependency.
|
||||
integration suite's true incremental-delivery timing assertion is gated
|
||||
behind `INTEGRATION_STREAMING=1` and skipped by default for exactly this
|
||||
reason — re-run it once the upstream issue is fixed.
|
||||
- **`POST /v1/messages` now intercepts unconditionally, even with
|
||||
`passthroughUnknown: true`.** Before this route existed, that path
|
||||
fell through to `next` on a `passthroughUnknown` deployment like any
|
||||
other unrecognized path. The gateway now answers it directly and
|
||||
`next` is never reached for it again — intended, but a real behavior
|
||||
change for a deployment currently proxying `/v1/messages` downstream.
|
||||
See [Anthropic Messages API — Route
|
||||
takeover](#route-takeover).
|
||||
- **A request rejected before it reaches a provider never consumes
|
||||
request-rate quota.** On the unified routes, a malformed body, a missing
|
||||
`model` field, or an unknown/denied model (404/403) is rejected before
|
||||
`limiter.checkAndCount` runs. On native passthrough and the MCP/A2A
|
||||
proxy, denied provider/target access (403), an unsupported `Upgrade`
|
||||
request (501), and a path-traversal attempt (400) are all checked before
|
||||
the limiter too. Once a request *does* reach a provider adapter,
|
||||
`limiter.checkAndCount` runs. `/v1/messages` goes further for one
|
||||
specific field: it decodes the body and rejects a streaming request
|
||||
(400) *before* the rate limiter runs too, the opposite relative order
|
||||
from the unified routes' own admission-before-decode default — a
|
||||
deliberate, route-scoped exception (see [Anthropic Messages
|
||||
API](#anthropic-messages-api-v1messages)), since Anthropic SDKs retry
|
||||
any 5xx and would otherwise burn multiple requests of budget per
|
||||
logical streaming call against a route that does not support streaming
|
||||
yet. On native passthrough and the MCP/A2A proxy, denied provider/target
|
||||
access (403), an unsupported `Upgrade` request (501), and a
|
||||
path-traversal attempt (400) are all checked before the limiter too.
|
||||
Once a request *does* reach a provider adapter,
|
||||
though, a translate-time 400 (an unsupported parameter combination) or
|
||||
Anthropic's embeddings 501 still count toward the caller's request-rate
|
||||
limit — the limiter already ran by then — though neither ever bills a
|
||||
|
||||
Reference in New Issue
Block a user