A provider whose discovery failed during the warm fill at pod start
went without discovered models for a full discovery interval, which is
1 hour by default. It also stayed that way until a request arrived
after the interval. Production logs showed this for openai-audio and
anthropic when outbound calls were slow at startup.
A provider that fails the warm fill now gets one background retry
after 10s. The retry has its own timeout and uses the same refresh
bookkeeping. It is skipped when a refresh is already in flight or has
finished since the warm fill, and it is not scheduled when the parent
context was canceled. A recovery logs one INFO line. If the retry also
fails, the existing breaker behaviour applies unchanged.
The retry delay and the warm-fill timeout can be set per registry for
tests. yaegi-check probes the retry under the interpreter.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
The gateway falls back to a request-size estimate only on a 2xx
stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
billed at zero.
- Billing uses the built-in LiteLLM price table and honours
modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
saturates on overflow.
- Gemini thinking tokens count as completion tokens.
Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.
Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.
Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
path.
- Retry-After is never jittered. A cancelled wait returns the
context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
as provider successes.
Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.
Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
A user entry can now list several groups in "groups", next to the
existing "group" field. A user can also hold a personal grant with its
own "providers" and optional "models" lists. Users with one group and
no personal grant behave exactly as before.
A request is allowed only when one grant allows the provider, the model
and, for passthrough, the path. Grants never merge, so one grant cannot
supply the provider while another supplies the model or the path. The
personal grant uses the path list of the first group. A personal
"models" list without "providers" is a configuration error.
Each member group applies its own limits. Caching stays on unless a
member group turns it off, and the shortest group TTL wins. The
/v1/models cache uses a stable, length-prefixed key, so reloads do not
grow it.
Tests cover the grant union, cross-grant denials, passthrough path and
model coupling, reloads and the cache key. The yaegi-check harness
asserts the same cases under the interpreter.
Claude-Session: https://claude.ai/code/session_019uJHQKcNLMACkXssb3L7Q3
resolve gives an exact alias match precedence over every discovered bare
model id, but listFor did the opposite: it skipped the alias whenever a
discovered model of the same bare id had already been listed. The catalog
then advertised the shadowed provider as the owner, with that provider's
metadata, for an id every request routed to the alias target instead.
A live fleet shows the effect. The alias "deepseek-v4-flash-vision-exp"
targets "gx10/current" (context 1048576), while a cloud provider discovers
a model of the identical bare id. GET /v1/models returned owned_by
"deepseek" and no context_window at all, so clients fell back to their own
128k default while the request ran against a 1M-context local model.
listFor now resolves every usable alias up front and suppresses the bare
entry of any id an alias shadows. The shadowed model stays listed in
"provider/id" form, which is how a client must address it once the alias
owns the bare id, so no model disappears from the catalog.
Claude-Session: https://claude.ai/code/session_0168ukk3uwp8PgiuRAzeLPXp
bareWinner picked the first provider in global sorted order that served
a bare model id, ignoring which providers the caller's group could
actually use. resolveAgainst then denied the request whenever that
provider fell outside the group, even when another provider serving the
identical id was already permitted. A caller entitled to a model could
receive errModelDenied purely because an unrelated, invisible provider
sorted first alphabetically.
bareWinner now takes the group and prefers the sorted-first provider it
may use, falling back to the global sorted-first provider only when the
group is authorized for none of them. That fallback keeps a genuinely
unauthorized request returning errModelDenied, not the wrong
errModelUnknown a "no visible owner" result would produce. resolve,
resolveAliasTarget, and the alias-shadow collision check all route
through this corrected selection.
listFor now computes each bare id's owner from the group's own visible
providers and adds "provider/model" disambiguating entries only when the
group can see two or more owners of that id. A provider invisible to the
group never appears in that group's catalog, bare or prefixed.
warnCollisionOnce keeps logging the global collision unchanged, since a
provider collision is a fact about the fleet's configuration that every
operator must see regardless of which group lists models first. Its
message now says winner is the global default, because a group's own
request can resolve to a different provider than the one logged.
Adds registry_test.go coverage for the production repro, the
both-allowed and neither-allowed non-regression cases, the listing
behavior, and the alias path. Adds a yaegi-check probe proving the fix
resolves and serves under the real interpreter, and updates one existing
listFor test that asserted the pre-fix prefixed-form behavior.
F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a
plain `now` variable while a background refresh goroutine read it
through nowFn, racing under -race in about 14% of full-suite runs once
this branch's timer-heavy tests shared the parallel phase. Guards the
clock with a mutex instead. registry.go is untouched.
F4: a mid-body watchdog timeout never reached provider-health
accounting. limiter.recordProviderAttempt only ran at header-arrival
time, so a provider that reliably stalled mid-body reported
attempts=1 failures=0 forever. watchdogBody.fire now invokes the same
attemptRecorder a second time, as a failure, when it fires. Reuses the
existing recorder rather than adding new plumbing; this does mean two
attempts are recorded for one logical request (one success at header
time, one failure at watchdog-fire time), not one attempt marked
failed outright.
F6: watchdogBody.Read reset the timer on any err == nil read,
regardless of n. A reader returning (0, nil) forever would have held
the watchdog open indefinitely. Reset now only fires when n > 0.
F9 (three items):
- proxyUpstream passed logPrefix as a bare providerName, which
newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an
MCP/A2A target as a provider. watchdogBody now takes an
already-formatted label from the caller instead.
- errUpstream was wrapped twice: once in watchdogBody's own error, once
again by every caller that already wraps it. watchdogBody now wraps
only errProviderTimeout; callers' existing wraps still make
errors.Is(err, errUpstream) hold.
- timeout_test.go read wb.timedOut directly in one place while every
other access is atomic; switched to atomic.LoadInt32.
Also rewords errProviderTimeout's message from "provider request timed
out" to "upstream progress stalled" (finding F3, documented not fixed
per coordinator ruling): the watchdog measures time since the last
successful upstream read, which can go quiet because of a slow client
write between reads, not only a slow provider. The new wording no
longer asserts which side caused it.
New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure,
TestWatchdogBody_ZeroByteRead_DoesNotResetTimer,
TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider.
Each mutation-verified per its own doc comment.
Adapters previously issued upstream requests with no timeout at all.
A provider that accepted a connection and then stalled hung the
gateway request forever. newAdapterHTTPClient now sets
Transport.ResponseHeaderTimeout, and every response body read
(providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream)
runs through a watchdogBody that resets on every successful read and
cancels the request once timeout passes with no progress.
The idle-progress design lets a long stream that keeps emitting tokens
run indefinitely. Only silence, never total duration, ends a request.
Config.RequestTimeout sets the global default (five minutes when
unset). ProviderConfig.RequestTimeout overrides it per provider. Both
are Go duration strings, validated at construction; a zero or negative
value is rejected rather than treated as unlimited, since that would
reintroduce the hang this feature fixes.
Native passthrough (routes_passthrough.go) and the MCP/A2A target
proxy (mcp_a2a.go) share the identical watchdog mechanism through
proxyUpstream's new timeout parameter.
A watchdog-fired timeout wraps errProviderTimeout and names the
provider. It never wraps context.Canceled, so it stays distinguishable
from a genuine client disconnect in every existing log classification.
GET /v1/models cost 15.9ms, 21.3MB, 552,884 allocs per call at the
live 1,033-model catalog (registry.go's listFor plus routes_unified.go's
json.Encoder), uncached — and OpenAI SDK clients poll this endpoint per
session, so an unchanged catalog was being re-walked and re-marshaled
on every request.
Add modelRegistry.modelsJSON(grp), caching the fully encoded
{object:list,data:[...]} body per (*group, modelsGen). modelsGen
is an atomic counter bumped by a new modelRegistry.finishRefresh
wrapper around providerState.finishRefresh — the only two places that
ever mutate a provider's discovered set or discovery-captured metadata
(warmFill, refreshProvider) already funnel through providerState.
finishRefresh once per refresh cycle, so wrapping it is sufficient to
invalidate the cache on every possible change, including a
metadata-only update on an otherwise-failed refresh (captureModelMetadata
runs unconditionally alongside finishRefresh in both call sites).
Bumped unconditionally rather than only on success, since a failed
listModels fetch can still pair with a successful metadata capture.
Cached per *group pointer: authStore builds every group once, at
construction, and never rebuilds or replaces that map afterward — a
config reload constructs an entirely new Gateway (and modelRegistry)
with its own empty cache, so a *group key can never alias a different
group's catalog either within one registry's lifetime or across a
reload.
routes_unified.go's handleModels call site shrinks to a single write of
modelsJSON's result, replacing the previous inline listFor+json.Encode;
routes_unified.go is otherwise untouched (owned by a sibling change in
this same effort).
Extend GET /admin/api/overview's existing providers payload rather than
adding a sibling endpoint: the route already returns one providers array
per request on the dashboard's 5s poll, is already admin-gated and
stats-excluded, and the new fields are one more batched limiter read
sharing that same request — a second endpoint would duplicate all three
properties for no benefit.
adminProviderView gains AttemptsDay/FailuresDay/AttemptsMinute/
FailuresMinute (the provider's own current-window counters) and
ModelRates (a per-model breakdown, keyed by upstream model id, via the
new adminModelRateView) — Feature A. buildAdminOverview builds every
provider's and every (provider, model) pair's limitScope in two passes
and reads them all through limiter.providerUsage in ONE storeGetMulti
call, then slices the flat result back by index.
adminProviderView also gains DiscoveryEnabled, mirroring a new
providerSnapshot.discoveryEnabled field (registry.go) read straight off
providerState — it is set once at construction and never mutated
afterward, so it needs neither providerState's mutex nor a place in its
existing snapshot() method's mutable-field return shape. Feature B: this
lets the webui tell "discovery is off" apart from "discovery is on but
has not refreshed yet", which previously both showed the same zero
lastRefresh as an identical "refreshed never".
Clicking a provider row in the dashboard expands to show its model
list, with a search filter and copy-to-clipboard on each model.
Backend (registry.go/admin.go): providerSnapshot gains models []string
- the sorted explicit union discovered id set, computed in the SAME
providerState.mu critical section as lastRefresh/lastErr (not via a
separate knownIDs() call, which takes the same non-reentrant lock and
would deadlock) so the three values are never assembled from two
separate locked passes a concurrent finishRefresh could interleave
between. adminProviderView exposes it as models (never nil - a real,
if empty, JSON array). New tests: overview shape asserts sorted
models per provider; a new TestModelRegistry_Snapshot_ModelsStaleWhileError
proves a failed refresh after an earlier success keeps the previous
discovered set (existing stale-while-error semantics, no special
handling added); TestAdmin_SecretRedaction unaffected (model ids are
not secrets, still passes as-is).
Frontend: the accordion is NOT shadcn-vue's Accordion or Collapsible -
both wrap trigger+content in one div, which cannot legally sit
between two tr elements inside a table/tbody. Installed
Collapsible via the CLI to check (verified components.json's now-
defused font/iconLibrary omission holds under a real install: main.css
stayed byte-for-byte unchanged) and inspected its source: reka-ui's
CollapsibleContent does support an "as" prop that could force it onto
a tr, but its animation-measurement code (getBoundingClientRect +
inline transition styles) is untested in that shape and not worth
taking on for an instant show/hide with no transition - removed the
installed files, kept a plain reactive Set<string> + v-if second
TableRow, which reuses Accordion's own visual language (rotating
chevron, keyboard-accessible trigger row: role=button, tabindex,
Enter/Space handlers) without misusing a component built for block
content.
Model chips (new ModelChip.vue) show the full ROUTABLE id -
"{providerName}/{modelId}" unconditionally, so a discovered id that
already contains a slash (uni/deepseek-v4-flash-0731 on provider
"real") renders as real/uni/deepseek-v4-flash-0731 and stays
copy-paste-valid - and copy that id to the clipboard on click
(lib/clipboard.ts: navigator.clipboard.writeText with a
select-the-text fallback when the Clipboard API is unavailable, never
the deprecated execCommand('copy')), with a brief checkmark
copied-state affordance. No CSP change: Clipboard access is a browser
permission, not a CSP directive.
Search input (OverviewView.vue) filters the providers table to
providers with a matching model (case-insensitive substring) and the
aliases table to aliases whose own name or target matches, auto-
expanding matching providers while a query is active and filtering
each expanded provider's own chip list down to just its matches (so a
72-model provider does not stay a 72-chip wall for a 1-model search).
Manual expand/collapse state is tracked separately from the search's
own auto-expand set, so clearing the query restores exactly what the
user had open by hand - nothing to explicitly "restore", since the
auto-expand set was never written into the manual one.
Integration: TestAdminDashboard's overview assertion extended
(adminProviderModels helper) to check the openai provider's models
array contains dynamic.yml.tmpl's explicit "gpt-mock" model - stays
API-level (asserts the JSON payload), not the rendered HTML, so the
prefixed-display-form request is left at the frontend only per scope.
Regenerated via make admin-ui; admin_assets_gen.go included.
listFor's alias dedupe keyed on the raw owners map, so a colliding
discovered model whose provider is group-denied caused BOTH the real
entry and the alias to be skipped - the id vanished from the listing
entirely even though resolve still serves it via the alias.
Track ids actually appended to out in a listed map instead, and skip
the alias only when listed[alias] is true. Extends
TestModelRegistry_ListFor_Alias_DedupedAgainstDiscoveredCollision with
a provider-denied variant: group providers:["openai"], collision on
the denied "acme" provider -> alias listed exactly once, owned_by
openai.
Review fix, task 6 (1 Important + 3 folded):
1. TestModelAliases now polls GET /v1/models (waitForModelAlias) until
"aliased/mock" appears, bounded 30s, before asserting - removes the
stale-dynamic-config propagation window a plain healthy-200 check
cannot rule out. Confirmed config-propagation, not alias-logic, per
reviewer's config+error-text analysis.
2. registry.go resolve(): check the warned-set (aliasShadowWarned) before
paying bareWinner's O(providerNames) scan on every aliased request -
only the first request for a colliding alias pays that cost now.
3. listFor: skip an alias entry whose id already exists in owners (a real
model already owns it) - previously produced duplicate ids when a
discovered model collided with a configured alias. resolve's own
alias-wins precedence is unchanged; only the listing is deduped. New
test: TestModelRegistry_ListFor_Alias_DedupedAgainstDiscoveredCollision.
4. newAliasTestRegistry doc comment: "grp's given aliases" -> "the given
aliases" (no grp parameter exists).
Spec §5 (v0.2): Config.ModelAliases map[string]string, alias-first
resolution precedence ahead of provider-prefix splitting, alias-OR-target
authorization, lazy target resolution with a 404 naming both ids,
/v1/models listing, admin overview alias table (JSON + dashboard), and
construction-time validation (shape, prefix-shadow, explicit-model
collision, chain, empty target).
routes_unified.go/routes_media.go needed no resolution-call-site changes
- both already pass the client's raw requested id through to
registry.resolve, so aliases inherit the __alias echo and multipart
rewrite machinery for free. cache.go needed no changes either - cacheKey
already hashes requestedModel, so two aliases resolving to the same
target already produce distinct cache entries.
Adds registry_test.go validation table + precedence/authz/lazy-404/
listing coverage, a dedicated cache-key split test, chat/embeddings/
images/audio-transcriptions end-to-end tests, an extended admin overview
test, and an integration test covering the openai-type vs anthropic-type
echo asymmetry under real Traefik+Yaegi.
1. refreshProvider (the plugin's only background goroutine) had no
recover(): a panicking adapter's listModels would crash the whole
Traefik process. Restructured around deferred recover + finishRefresh,
so a panic is turned into a logged error, inFlight always releases, and
the next interval retries. Documented as a scoped exception to
limits.go's no-background-goroutine rule.
2. newGateway wrapped ctx in an outer 5s WithTimeout before warmFill,
but warmFill already derives a fresh 5s budget per provider from
whatever ctx it receives. The outer wrap made that a single budget
shared across every discovery-enabled provider instead of a per-provider
one - providers after the first could get a near-expired context and
fail instantly. Dropped the outer wrapper; construction's worst case is
now N * 5s, accepted.
3. group.allowsModel stripped at any first slash, so a glob like
'deepseek-*' could spuriously match a bare model id that merely
contains a slash (e.g. an upstream's own 'uni/deepseek-v4-flash-0731'
naming) even though the leading segment names no configured provider.
allowsModel is now an exact glob with no internal splitting;
resolveAgainst and listFor generate the provider-prefixed and
bare-suffix candidates themselves, only when a prefix legitimately
names a configured provider. Updated the Task-3 TestIdentify assertion
to match.
4. Annotated the discarded json.Encode error in handleModels.
Gate: go test ./... -count=1, -race, gofmt, vet, fieldalignment,
golangci-lint (incl gosec) all clean.
Adds modelRegistry: per-provider explicit+discovered model sets, bare-id
and provider-prefixed resolution with sorted-name collision precedence,
group authorization pairing (allowsModel AND allowsProvider), and a
group-filtered OpenAI-compatible /v1/models catalog.
Discovery is lazy and throttled per provider: newGateway runs a
synchronous 5s-bounded warm fill for discovery-enabled providers
(non-fatal on error), and Gateway.ServeHTTP triggers a throttled,
inFlight-guarded background refresh per provider per interval
(default 1h, per-provider DiscoveryInterval parsed at construction).
A failed refresh keeps the previous discovered set (stale-while-error).
Wires GET /v1/models into ServeHTTP (auth required, 401 envelope
otherwise) and gives Gateway its adapters/registry fields, both built
in newGateway.
newModelRegistry's signature gained an error return beyond the task
brief's literal listing: an invalid DiscoveryInterval must be a
constructor error per the controller ruling, which the brief's bare
*modelRegistry signature could not express.
lukaszraczylo