mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-23 06:07:47 +00:00
feat(config): advertise concurrency via RateLimit-Policy with autodetect
The gateway now sends the concurrency limit as a RateLimit-Policy field from draft-ietf-httpapi-ratelimit-headers. The value is "concurrency";q=<n>;qu="concurrent-requests". The invented X-Concurrency-Limit header is gone. The new concurrencyAutodetect option is off by default. When on, the gateway learns a limit for each configured provider. A 429, 503, 504 or upstream timeout sets the limit to half the in-flight requests, with a minimum of 1. Four consecutive successes add 1. At 64 the limit counts as unknown again. Responses served via a provider advertise the smaller of maxConcurrency and the learned value. An explicit maxConcurrency always wins. Routing, retry and failover do not change.
This commit is contained in:
@@ -1297,9 +1297,9 @@ func (g *Gateway) handlePassthrough(w http.ResponseWriter, r *http.Request, u *u
|
||||
// (extractPassthroughUsage, below), read only after this attempt
|
||||
// already resolved, so there is no model to attribute it to yet. See
|
||||
// recordProviderAttempt's own doc comment (limits.go).
|
||||
r = r.WithContext(withAttemptRecorder(r.Context(), func(resp *http.Response, attemptErr error) {
|
||||
g.limiter.recordProviderAttempt(providerName, "", resp, attemptErr)
|
||||
}))
|
||||
trackedCtx, endUpstream := g.trackUpstream(r.Context(), w, providerName, "")
|
||||
defer endUpstream()
|
||||
r = r.WithContext(trackedCtx)
|
||||
// feat: instrument upstream latency — gated on metricsEnabled OR
|
||||
// admin.stats.latency (admin-redesign WP-A), same reasoning as
|
||||
// runUnified's identical wiring (routes_unified.go). No model here
|
||||
|
||||
Reference in New Issue
Block a user