feat(config): advertise concurrency via RateLimit-Policy with autodetect

The gateway now sends the concurrency limit as a RateLimit-Policy
field from draft-ietf-httpapi-ratelimit-headers. The value is
"concurrency";q=<n>;qu="concurrent-requests". The invented
X-Concurrency-Limit header is gone.

The new concurrencyAutodetect option is off by default. When on,
the gateway learns a limit for each configured provider. A 429, 503,
504 or upstream timeout sets the limit to half the in-flight
requests, with a minimum of 1. Four consecutive successes add 1.
At 64 the limit counts as unknown again.

Responses served via a provider advertise the smaller of
maxConcurrency and the learned value. An explicit maxConcurrency
always wins. Routing, retry and failover do not change.
This commit is contained in:
2026-10-18 13:44:08 +01:00
parent 3e3517eb62
commit 644b8115c8
7 changed files with 356 additions and 30 deletions
+3 -3
View File
@@ -1297,9 +1297,9 @@ func (g *Gateway) handlePassthrough(w http.ResponseWriter, r *http.Request, u *u
// (extractPassthroughUsage, below), read only after this attempt
// already resolved, so there is no model to attribute it to yet. See
// recordProviderAttempt's own doc comment (limits.go).
r = r.WithContext(withAttemptRecorder(r.Context(), func(resp *http.Response, attemptErr error) {
g.limiter.recordProviderAttempt(providerName, "", resp, attemptErr)
}))
trackedCtx, endUpstream := g.trackUpstream(r.Context(), w, providerName, "")
defer endUpstream()
r = r.WithContext(trackedCtx)
// feat: instrument upstream latency — gated on metricsEnabled OR
// admin.stats.latency (admin-redesign WP-A), same reasoning as
// runUnified's identical wiring (routes_unified.go). No model here