Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
The metadata fetcher understood only LM Studio's field names, so a
vLLM-backed provider reported no per-model context and every model had to
have its window pinned by hand in modelMeta. That pin then goes stale
silently: a renamed served model ships no context at all and clients fall
back to their own default, and a changed window advertises a size the
engine no longer accepts. Both happened on a live deployment today.
vLLM reports max_model_len at its OpenAI-compatible /v1/models and neither
LM Studio field, verified against vLLM 0.28. Reading it means pointing
MetadataPath at /v1/models is enough for such a provider.
Precedence is unchanged where it already applied: loaded_context_length
still wins as the window usable right now, then max_context_length, and
max_model_len is consulted only when neither is present. An upstream
reports one dialect or the other, so the added branch cannot change any
existing capture.
Renames the payload type, which now carries two dialects rather than LM
Studio's alone.
fetchModelMetadata (provider_openai.go) builds its request URL by plain
string concatenation, a.baseURL+a.metadataPath. A metadataPath missing
its leading slash would silently glue onto baseURL's own path with no
separator (e.g. "http://host:1234api/v0/models"), a malformed URL that
only ever surfaced as a confusing runtime fetch failure. buildAdapters
now rejects a non-empty metadataPath without a leading "/" at
construction, naming the provider, the same fail-fast treatment every
other malformed config value already gets. Empty (the default) is
always valid. Folded review minor.
Config gets two new surfaces (llmgateway.go): Config.ModelMeta
(map[string]*ModelMetaConfig, keyed by exact "provider/model" or a
bare model id) for operator overrides, and ProviderConfig.MetadataPath
for openai-type providers whose discovery should also capture per-model
context (LM Studio's native /api/v0/models shape).
modelmeta.go adds the layering itself: resolveModelMeta applies config
override > discovered context > builtin table > absent for context, and
config override/free > builtin > absent for cost (discovery never
reports cost). resolveAliasModelMeta implements alias inheritance: an
alias inherits its target's fully resolved metadata unless the alias
itself has its own modelMeta entry, applied per field. validateModelMeta
rejects negative values and a free:true/non-zero-cost contradiction at
construction.
registry.go wires this into the request path: providerState gains a
discoveredContext map (stale-while-error, mirroring discovered's own
convention) with contextFor/setDiscoveredContext; a new optional
modelMetadataFetcher interface, type-asserted (interpreted-to-interpreted,
documented as yaegi-safe against the Trap-4 restriction) rather than
added to providerAdapter, lets warmFill/refreshProvider capture metadata
alongside listModels without touching anthropic/gemini adapters at all.
modelObject/listFor now attach resolved context_window/pricing to every
GET /v1/models entry, including aliases (resolveMetaFor/
resolveMetaForAlias/resolveMetaForAliasName).
provider_openai.go implements modelMetadataFetcher: fetchModelMetadata
reads ProviderConfig.MetadataPath, preferring loaded_context_length over
max_context_length when the LM Studio response reports both. providers.go
wires MetadataPath onto the adapter in buildAdapters.
tools/yaegi-check/main.go adds one cheap modelMeta config-override
assertion to the existing attempt-accounting harness, proving
resolveModelMeta's config-override layer and the context_window/pricing
extension fields both run correctly under the real Yaegi interpreter.
C1 (quota bypass): a client sending stream_options.include_usage=false
suppressed upstream usage reporting and escaped every token/cost budget,
since the non-streaming T12 estimate never covers a streaming response.
chatCompletion (provider_openai.go) now always forces include_usage=true
on the outgoing request, merged into a copy of the client's own
stream_options map so the caller's map is never mutated in place.
clientAskedUsage is derived from the client's own value via isTruthy, so
the usage-only chunk is still suppressed for a client that never asked
for it.
I1 (vacuous security tests): the X-Api-Key credential-strip set had zero
effective coverage — existing tests either never sent x-api-key, or
targeted anthropic, whose injectAuth overwrites x-api-key regardless of
whether the strip ran. Added one openai-passthrough and one MCP-proxy
test that authenticate via x-api-key itself and assert it arrives empty
upstream. Mutation-checked: both fail when "X-Api-Key": true is removed
from gatewayCredentialHeaders (verified, then restored).
I3 (auth-event logging): added Gateway.logAuthEvent (logger.go), called
at all six authStore.identify call sites (llmgateway.go's ServeHTTP,
routes_unified.go's handleModels). A failed attempt logs method, path,
and remote address; a success logs the resolved user's name and route.
Neither line ever includes the presented API key.
M2 (usage.estimated): runUnified now reads result.estimated and logs the
estimate explicitly when a non-streaming response's usage falls back to
the body-size heuristic, making the field genuinely read — removes the
stale //nolint:unused on limits.go's usage struct.
M8 (gosec consistency): secrets.go's file: secret path read now carries
the same "#nosec G304 -- operator-supplied file path from middleware
config" annotation as users_file.go's equivalent read.
Doc nit: mcp_a2a.go's validateTargetURLs comment now also names the
dot-only ("."/"..") rule validateConfigName enforces, not just character
set and reserved names.
Gates: go test ./... -count=1 (-race) green, gofmt -l . empty, go vet
clean, make yaegi-check OK, golangci-lint (gosec+staticcheck+unused+
errcheck+govet, global config) 0 issues.
configNamePattern (^[a-zA-Z0-9._-]+$) permits "." and ".." as valid
provider/mcpServers/agents names, since a bare dot is in the allowed
character class. Both names read as directory-traversal segments once
embedded as a path segment in the gateway's own routes
(passthroughRoute's "/{provider}/*", targetRoute's "/mcp/{name}/*" and
"/a2a/{name}/*"). Nothing in this package's current path handling
mis-resolves them, but disallowing them at construction is cheap and
removes any future dependence on that staying true.
- providers.go: add dotOnlyConfigNames ({".", ".."}), checked in
validateConfigName between the character-class check and the
reserved-name check.
- provider_openai_test.go: extend TestValidateConfigName with "." and
".." cases (rejected) across provider/mcpServers/agents kinds, plus
"..." (accepted — the rule is an exact match on "." and "..", not "any
dots-only string").
Gates: go test ./... -count=1, go test ./... -race -count=1, gofmt -l .,
go vet ./..., make yaegi-check, golangci-lint run ./..., golangci-lint
run --enable=gosec ./..., fieldalignment ./... — all clean.
- providers.go: validate provider names against ^[a-zA-Z0-9._-]+$ and
reject reserved names (v1, mcp, a2a) that would shadow gateway routes;
reject nil ProviderConfig map values at construction instead of
panicking.
- mcp_a2a.go: apply the same name-pattern and reserved-name validation to
mcpServers/agents keys, and reject nil TargetConfig/AgentConfig map
values at construction.
- auth.go: reject nil GroupConfig map values at construction.
- provider_openai_test.go, mcp_a2a_test.go, auth_test.go: table-driven
constructor-error coverage for the above.
- integration/integration_test.go: point the package doc at
`make integration-up`/`integration-keep` instead of a bare
`docker compose up -d --build`, which skips the dynamic.yml.tmpl
render step and can run against a stale or absent dynamic.yml; add
max_tokens to the gemini chat request so translate_gemini.go's
hasMaxTokens=true branch is actually exercised under real Yaegi.
- translate_gemini.go: soften the max_tokens comment's verification claim
to match what the integration suite actually now exercises.
Review round 1 fixes (1 Critical + 5 Important) on the anthropic
adapter/translator.
- CRITICAL: consecutive role:"tool" messages each became their own
Anthropic user turn. Anthropic requires alternating user/assistant
turns and 400s on two consecutive same-role messages; parallel tool
calls (one assistant message with N tool_calls followed by N
role:"tool" messages) are the standard agentic transcript shape.
anthropicRequestFromOpenAI now accumulates consecutive tool messages
into one Anthropic user turn carrying all of their tool_result
blocks (anthropicToolResultBlock replaces anthropicToolResultMessage).
Added chat_parallel_tool_results fixture.
- Streaming tool_calls[].index passed Anthropic's raw content-block
index through, so a tool_use block after a text block emitted
index:1 instead of the required contiguous 0-based ordinal.
anthropicStreamState now tracks a per-stream tool-call ordinal
counter plus a content-block-index -> ordinal map, populated on
content_block_start and consulted on input_json_delta. Added the
stream_text_then_tool fixture (text block at index 0, tool_use at
index 1) proving the emitted index is 0.
- A system message with array-form content ([{"type":"text",...}])
was silently dropped (only a string content was handled). Added
systemTextFromContentParts: text parts join with "\n"; a non-text
part returns a *translateError instead of being dropped. Added
chat_system_array_content (success) and
chat_system_array_content_error fixtures.
- Malformed JSON on a recognized stream event (message_start,
content_block_start, content_block_delta, message_delta) was
swallowed as (nil, nil), which could silently corrupt a client's
reassembled tool-call arguments. All four now return an error
wrapping errUpstream; forwardStream already propagates that and
withholds the terminal [DONE]. The unrecognized-event-type and
error-event branches are unchanged. Added
TestAnthropicStreamState_MalformedEventReturnsError.
- providerAdapter.injectAuth's doc comment claimed a universal
empty-key no-op; updated to state the per-type divergence
(openai-type: keyless is a no-op; anthropic-type: the constructor
rejects an empty key, so injectAuth is never called with one).
- Added case-insensitive-Content-Type coverage for both adapters'
streaming-vs-JSON gate (previously untested): one subtest each in
provider_openai_test.go / provider_anthropic_test.go using a fake
upstream that sends "Text/Event-Stream; charset=utf-8", asserting
the streaming path (not the JSON fallback) is taken.
Gate: go test ./... -count=1 and -race both green; gofmt, go vet,
fieldalignment, golangci-lint (incl. gosec), govulncheck all clean.
Implements Task 9: the anthropic-type providerAdapter and its
request/response/stream translation to and from OpenAI's wire shape.
- translate_anthropic.go: anthropicRequestFromOpenAI (system messages
joined, content blocks incl. base64 data-URI images, tool_calls ->
tool_use, tool messages -> tool_result, max_tokens defaulting,
stop/stop_sequences, tool_choice incl. the none-drops-tools rule,
n>1/logit_bias/logprobs rejected), openAIResponseFromAnthropic
(content blocks joined/tool_calls re-marshaled, stop_reason mapping,
usage), and anthropicStreamState.translate mapping each SSE event to
zero or more OpenAI chat.completion.chunk payloads. translateError
carries a notSupported flag so the caller (Task 12) can tell a 400
from a 501.
- provider_anthropic.go: anthropicAdapter implementing providerAdapter.
newAnthropicAdapter errors on an empty API key per ruling (c), unlike
openai-type's keyless ruling. Headers: x-api-key, anthropic-version.
embeddings always returns a notSupported translateError without
calling upstream (Anthropic has no embeddings endpoint). listModels
GETs /v1/models and returns any non-2xx (incl. 404) as-is.
- providers.go: wires the anthropic buildAdapters case to
newAnthropicAdapter, propagating its constructor error.
- provider_openai_test.go: replaces the now-stale placeholder
TestBuildAdapters subtest with one asserting a keyed anthropic
provider builds successfully and a keyless one errors.
- translate_anthropic_test.go: golden-file harness walking
testdata/anthropic/{chat,resp}_*_in.json against sibling _want.json
(success) or _wanterr.json (*translateError) fixtures, plus a stream
harness driving each *.sse fixture through anthropicStreamState and
asserting the chunk list and usage().
- provider_anthropic_test.go: httptest-backed adapter wiring tests
(streaming + non-streaming), asserting OpenAI-shaped output on the
wire, per-event flushing, and the required auth headers.
Gate: go test ./... -count=1 and -race both green; gofmt, go vet,
fieldalignment, golangci-lint (incl. gosec) all clean.
lukaszraczylo