mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-19 05:47:48 +00:00
feat(providers): success-rate accounting
Track per-provider and per-(provider,upstream-model) upstream-attempt outcomes using the existing fixed-window counter machinery (limits.go), following mcp_a2a.go's per-target counter pattern: kind-embedded keys (kindProvider="prov", kindProviderModel="provmodel") keep a provider scope collision-safe against a user/group/MCP-target/agent of the same name. limiter.recordProviderAttempt(provider, model, resp, err) increments an "attempts" counter on every call, and a "failures" counter only for a provider-fault outcome — reusing retry.go's own isTransient(resp, err) classification exactly (transport/connection error, HTTP 429, or a 5xx), not forking it. Both counters are written at minute and day granularity only; provider health is a now-and-today question, not a billing one, so there is no month window. limiter.providerUsage batches every scope's read into one storeGetMulti round trip, mirroring targetUsage's own single-batch discipline. Wired at every chokepoint an upstream HTTP attempt resolves: - providers.go's upstreamJSON/upstreamRawBytes path (chat, embeddings, images, audio) via retry.go's retryPolicy.do, which now looks up an optional attemptRecorder from ctx and invokes it once per retry attempt (attempt-accounting: a request retried twice before succeeding reports three attempts, not one). - routes_unified.go's runUnified (chat/embeddings, streaming and non-streaming alike — both funnel through the same upstreamJSON call). - routes_media.go's three media handlers (images.generations, audio.speech, audio.transcriptions). - routes_passthrough.go's proxyUpstream (native provider passthrough) — a single un-retried attempt, provider-level only: the upstream model lives in the response body, read only after the attempt resolves. The attemptRecorder is threaded via context (providers.go's withAttemptRecorder/attemptRecorderFromContext), not a new function parameter on every call site — proxyUpstream and retryPolicy.do are both shared with paths that must NOT record provider accounting (registry.go's discovery listModels calls, and the MCP/A2A target proxy, mcp_a2a.go's handleTargetProxy), so scoping via context keeps their signatures unchanged and those callers correctly opt out by simply never wrapping their context.
This commit is contained in:
@@ -184,6 +184,50 @@ func newAdapterHTTPClient() *http.Client {
|
||||
return &http.Client{Transport: tr}
|
||||
}
|
||||
|
||||
// attemptRecorder is invoked once per upstream HTTP attempt a shared
|
||||
// chokepoint makes — retryPolicy.do's own retry loop (retry.go), for every
|
||||
// adapter method that goes through upstreamJSON/upstreamRawBytes, and
|
||||
// proxyUpstream's single client.Do call (routes_passthrough.go) — with
|
||||
// that attempt's raw resp/err, for Feature A's (v0.22) per-provider
|
||||
// success-rate accounting (limiter.recordProviderAttempt, limits.go). A
|
||||
// recorder must never touch resp.Body (still owned by its caller, read or
|
||||
// forwarded afterward) and must never retain resp or err past the call.
|
||||
type attemptRecorder func(resp *http.Response, err error)
|
||||
|
||||
// attemptRecorderCtxKey is the unexported context.Value key
|
||||
// withAttemptRecorder/attemptRecorderFromContext share. An unexported
|
||||
// struct type, not a string, so no other package's context.WithValue call
|
||||
// can ever collide with it by accident.
|
||||
type attemptRecorderCtxKey struct{}
|
||||
|
||||
// withAttemptRecorder returns a context carrying rec, so a chokepoint
|
||||
// shared across call sites that must NOT all record provider accounting —
|
||||
// retryPolicy.do and proxyUpstream are both also reached by traffic
|
||||
// Feature A is deliberately out of scope for: registry.go's discovery
|
||||
// listModels calls (no recorder ever set on their ctx) and the MCP/A2A
|
||||
// target proxy (handleTargetProxy, mcp_a2a.go, which never wraps its
|
||||
// request's context this way) — can look the recorder up without either
|
||||
// function needing a dedicated parameter every call site would otherwise
|
||||
// have to pass. A nil rec returns ctx unchanged, so a caller may call this
|
||||
// unconditionally.
|
||||
func withAttemptRecorder(ctx context.Context, rec attemptRecorder) context.Context {
|
||||
if rec == nil {
|
||||
return ctx
|
||||
}
|
||||
return context.WithValue(ctx, attemptRecorderCtxKey{}, rec)
|
||||
}
|
||||
|
||||
// attemptRecorderFromContext returns the attemptRecorder ctx carries via
|
||||
// withAttemptRecorder, or nil when none was set — the common case for any
|
||||
// call path Feature A does not account (see withAttemptRecorder's doc
|
||||
// comment). A bare comma-ok type assertion, not errors.As or any reflect-
|
||||
// based check: see providerHTTPError's own doc comment for why a
|
||||
// yaegi-interpreted plugin must avoid errors.As here.
|
||||
func attemptRecorderFromContext(ctx context.Context) attemptRecorder {
|
||||
rec, _ := ctx.Value(attemptRecorderCtxKey{}).(attemptRecorder)
|
||||
return rec
|
||||
}
|
||||
|
||||
// upstreamJSON issues an HTTP request to url: body, when non-nil, is
|
||||
// marshaled as the JSON request body; hdr's values are added to the
|
||||
// request (a caller builds this from its adapter's auth and content-type
|
||||
|
||||
Reference in New Issue
Block a user