traefik-llmgateway/retry.go

package traefikllmgateway

import (
	"context"
	"errors"
	"fmt"
	"io"
	"math/rand"
	"net/http"
	"strconv"
	"time"
)

// defaultRetryAttempts and defaultRetryBackoff are the values
// newRetryPolicy applies when a Retry config block is Enabled but leaves
// Attempts or Backoff at its zero value — spec §1's stated defaults.
// defaultRetryAttempts counts RETRIES performed after the initial try
// (see retryPolicy's doc comment), so the default of 1 means one retry —
// two tries total — not one try total.
const (
	defaultRetryAttempts = 1
	defaultRetryBackoff  = "250ms"
)

// maxRetryAttempts is the highest RetryConfig.Attempts newRetryPolicy
// accepts — spec §1's "attempts: int (default 1, max 3)". Like
// defaultRetryAttempts, this counts retries after the initial try: the
// maximum makes four tries total, not three.
const maxRetryAttempts = 3

// maxRetryWait caps the wait retryPolicy.do computes for ITSELF — the
// exponential backoff ladder — between attempts. spec §1's "capped at 2s
// per wait". An upstream's own explicit Retry-After request is a
// separate case with its own, much larger cap (finding 12 fix,
// review-routes.md: this comment previously and incorrectly claimed
// maxRetryWait also caps a Retry-After-derived wait) — see
// maxRetryAfterWait, below.
const maxRetryWait = 2 * time.Second

// maxRetryAfterWait caps a wait this gateway will honour when the
// UPSTREAM itself asked for it via a 429's Retry-After header. It is
// deliberately far larger than maxRetryWait, which caps only the wait
// this gateway computes for itself.
//
// Security audit run-1 (retry/failover fan-out record): previously a
// Retry-After larger than maxRetryWait was DISCARDED and replaced by the
// <=2s exponential ladder, so a saturated provider could not tell this
// gateway to back off for longer than two seconds — the gateway answered
// an explicit overload signal by retrying sooner than asked. Honouring it
// up to this ceiling can only ever make the gateway wait LONGER, never
// sooner, so it strictly reduces outbound pressure. p.waitFn is
// ctx-aware, so a client disconnect or request deadline still aborts the
// wait immediately rather than blocking out the full duration.
const maxRetryAfterWait = 30 * time.Second

// retryJitterFn spreads a COMPUTED backoff so that many requests failing
// at the same instant do not retry in lockstep against the same upstream
// — the synchronized-retry shape that turns one upstream blip into a
// self-sustaining thundering herd (security audit run-1). It applies
// EQUAL jitter: half the computed wait, plus a uniform random draw over
// the other half, so a jittered wait is always at least d/2 and never
// collapses to an immediate retry the way full jitter can.
//
// Finding 4 fix (review-routes.md): do, below, applies this ONLY to
// waitBefore's exponential-backoff branch, never to a wait that came
// from an upstream's own Retry-After header. Jittering a Retry-After
// value can shorten it (equal jitter draws anywhere in [d/2, d]), which
// would retry a saturated provider SOONER than it explicitly asked —
// exactly the backwards-under-overload behavior maxRetryAfterWait's own
// doc comment says honouring Retry-After exists to prevent ("can only
// ever make the gateway wait LONGER, never sooner"). waitBefore's second
// return value tells do which branch produced the wait, so this
// function itself needs no such distinction — it always jitters
// unconditionally; the caller decides whether to call it at all.
//
// A package-level var, mirroring retryPolicy.waitFn's own convention, so
// tests can substitute an identity function and keep asserting exact
// durations. It is applied in do, not in waitBefore, so waitBefore stays
// a pure function of (attempt, resp) and its own unit tests keep testing
// the backoff ladder itself rather than the randomness on top of it.
var retryJitterFn = func(d time.Duration) time.Duration {
	if d <= 0 {
		return d
	}
	half := d / 2
	// #nosec G404 -- this randomness spreads retry timing to break up a
	// synchronized herd; it guards no secret and gates no decision, so a
	// predictable draw grants an attacker nothing. crypto/rand would add
	// a syscall per retry wait and an error path, for no security gain.
	return half + time.Duration(rand.Int63n(int64(d-half)+1))
}

// retryDrainLimit caps how much of a retried-away response's body
// drainAndClose reads before closing it. It is generous enough to let
// the underlying connection settle for reuse, but far below
// maxResponseBytes (32MiB): a failed attempt's body is discarded, never
// parsed, so there is no reason to pay for reading a large one in full.
const retryDrainLimit = 64 << 10

// retryPolicy implements spec §1's same-provider retry.
//
// attempts counts RETRIES performed after the initial try, not the
// total try count — ruling (controller review, 2026-08-20): the earlier
// "attempts = total tries" reading made the documented default of 1
// perform zero retries, silently defeating retry:{enabled:true}'s own
// defaults. With this policy, attempts=1 (the default) means one retry —
// two tries total; attempts=3 (the max) means three retries — four
// tries total. totalTries() is the single place that turns attempts
// into the actual try count do's loop uses.
//
// A nil *retryPolicy, or one with enabled=false, makes do try call
// exactly once — the same shape every v0.1 adapter constructor leaves
// its adapter's retry field in, since none of them set it. waitFn is a
// field (not a direct call to a wait implementation) so tests can inject
// a non-blocking spy instead of paying real wall-clock time; the
// production default (defaultWaitFn) is ctx-aware.
type retryPolicy struct {
	waitFn   func(context.Context, time.Duration) bool
	backoff  time.Duration
	attempts int
	enabled  bool
}

// totalTries returns how many times do should call call() in total: 1
// for a nil or disabled policy (no retry — v0.1 behavior), or
// p.attempts+1 for an enabled one, per retryPolicy's own doc comment. A
// malformed attempts value below 1 — reachable only by constructing a
// *retryPolicy directly rather than through newRetryPolicy, which
// already rejects it — is clamped to 1 retry rather than trusted as-is,
// so this can never return less than 1: a totalTries of 0 or less would
// make do's loop skip calling call() entirely and return the disallowed
// (nil, nil) shape.
func (p *retryPolicy) totalTries() int {
	if p == nil || !p.enabled {
		return 1
	}
	retries := p.attempts
	if retries < 1 {
		retries = 1
	}
	return retries + 1
}

// defaultWaitFn is the production (*retryPolicy).waitFn: it blocks for
// d, or returns early the moment ctx is done, whichever comes first. A
// context canceled, or a deadline reached, while do is backing off
// aborts the remaining wait promptly instead of blocking out the full
// duration for a retry the caller no longer wants — this also bounds how
// long newGateway's synchronous warmFill discovery fill can be held up
// by a retrying listModels call. Returns true when the wait completed
// normally (do should proceed with the next attempt), false when ctx
// ended it early (do should stop retrying and return the last result).
func defaultWaitFn(ctx context.Context, d time.Duration) bool {
	select {
	case <-time.After(d):
		return true
	case <-ctx.Done():
		return false
	}
}

// newRetryPolicy validates rc and returns the retryPolicy buildAdapters
// attaches to every adapter it constructs. Disabled config (the v0.1
// default) skips validation and defaulting entirely — an unset
// Attempts/Backoff on a disabled block is never an error, since neither
// value is ever read.
//
// rc.Attempts is the number of retries performed after the initial try,
// not the total try count (see retryPolicy's doc comment): the default,
// applied when rc.Attempts is left at its zero value, is 1 retry (two
// tries total); the maximum, maxRetryAttempts, is 3 retries (four tries
// total). A value outside 1..maxRetryAttempts is a config error.
// rc.Backoff defaults to defaultRetryBackoff when left empty, and must
// parse via time.ParseDuration either way.
func newRetryPolicy(rc RetryConfig) (*retryPolicy, error) {
	if !rc.Enabled {
		return &retryPolicy{enabled: false, waitFn: defaultWaitFn}, nil
	}

	attempts := rc.Attempts
	if attempts == 0 {
		attempts = defaultRetryAttempts
	}
	if attempts < 1 || attempts > maxRetryAttempts {
		return nil, fmt.Errorf("llmgateway: retry.attempts (retries after the first try) must be between 1 and %d, got %d", maxRetryAttempts, attempts)
	}

	backoffStr := rc.Backoff
	if backoffStr == "" {
		backoffStr = defaultRetryBackoff
	}
	backoff, err := time.ParseDuration(backoffStr)
	if err != nil {
		return nil, fmt.Errorf("llmgateway: retry.backoff: %w", err)
	}

	return &retryPolicy{enabled: true, attempts: attempts, backoff: backoff, waitFn: defaultWaitFn}, nil
}

// isTransient reports whether resp/err — the result of one upstream
// attempt, mutually exclusive per http.Client's own contract (a non-nil
// err always means resp is nil) — is a failure retryPolicy.do should
// retry: any error reaching upstreamJSON's client.Do (dial/EOF/reset),
// other than a context cancellation, a deadline, or a failure to build
// the outgoing *http.Request in the first place (errRequestBuildFailed —
// a malformed method or URL fails identically on every attempt, so
// retrying it only wastes attempts) — or a response carrying HTTP 429 or
// a 5xx status. Every other outcome — success, or a non-429 4xx — is not
// transient, per spec §1's "Never on 4xx≠429, translate errors, or
// context cancellation".
func isTransient(resp *http.Response, err error) bool {
	if err != nil {
		if errors.Is(err, errRequestBuildFailed) {
			return false
		}
		return !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded)
	}
	if resp == nil {
		return false
	}
	return resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= http.StatusInternalServerError
}

// waitBefore returns how long do should wait before the attempt after
// attempt (1-based) — the one that just produced resp (nil on a network
// error) — and whether that wait came from the upstream's own
// Retry-After header (fromRetryAfter) rather than this gateway's own
// exponential backoff ladder. do uses fromRetryAfter to decide whether
// retryJitterFn may touch the wait at all (finding 4 fix, review-
// routes.md — see retryJitterFn's own doc comment for why jittering a
// Retry-After value is wrong).
//
// A 429 response carrying a Retry-After header that parses as a
// non-negative whole-second count wins outright, clamped to
// maxRetryAfterWait (NOT maxRetryWait — an upstream's own explicit
// cool-down request is honoured far longer than this gateway's own
// ladder; see maxRetryAfterWait's own doc comment): the upstream said
// exactly how long to back off, and this gateway waits at least that
// long, up to the 30s ceiling. Every other case falls back to
// exponential backoff (p.backoff * 2^(attempt-1)), capped at
// maxRetryWait — that covers a 429 with no Retry-After header, one that
// fails to parse as whole seconds (for example the HTTP-date form), one
// that parses negative, a Retry-After header present on a non-429 5xx
// (ignored: Retry-After is only meaningful on 429 here), and every
// network-error retry.
func (p *retryPolicy) waitBefore(attempt int, resp *http.Response) (wait time.Duration, fromRetryAfter bool) {
	if resp != nil && resp.StatusCode == http.StatusTooManyRequests {
		if ra := resp.Header.Get("Retry-After"); ra != "" {
			if secs, err := strconv.Atoi(ra); err == nil && secs >= 0 {
				// An upstream-requested cool-down is honoured up to
				// maxRetryAfterWait, and CLAMPED to it rather than
				// discarded, when it asks for longer (security audit
				// run-1). Discarding it — the previous behavior — fell
				// back to the <=2s ladder and so retried a provider that
				// had explicitly asked for a longer pause sooner than it
				// asked, which is backwards under overload.
				raWait := time.Duration(secs) * time.Second
				if raWait > maxRetryAfterWait {
					raWait = maxRetryAfterWait
				}
				return raWait, true
			}
		}
	}

	wait = p.backoff * time.Duration(int64(1)<<uint(attempt-1))
	if wait > maxRetryWait {
		wait = maxRetryWait
	}
	return wait, false
}

// drainAndClose discards a retried-away response's remaining body
// (capped at retryDrainLimit, defensively) and closes it, so the
// underlying connection is eligible for reuse on the next attempt
// instead of being forced closed by an un-drained body.
func drainAndClose(resp *http.Response) {
	if resp == nil || resp.Body == nil {
		return
	}
	_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, retryDrainLimit))
	_ = resp.Body.Close()
}

// do executes call, retrying while isTransient(resp, err) reports a
// transient failure, up to totalTries() tries in total (see its doc for
// the retries-vs-total-tries distinction). A nil p, or one with
// enabled=false, tries call exactly once — the disabled/unwired shape.
// Between attempts it drains and closes the failed attempt's response
// body, then waits per waitBefore via p.waitFn — ctx-aware, so a context
// canceled or timed out during that wait aborts the remaining attempts
// immediately rather than blocking out the wait in full. do returns the
// last attempt's (resp, err) as-is on success, on a still-transient
// failure once tries are exhausted, or on a non-transient failure at any
// attempt.
//
// Finding 9 fix (review-routes.md): when the wait itself is aborted
// early (p.waitFn returns false — ctx was canceled or its deadline
// passed while do was backing off, most commonly the CLIENT
// disconnecting during a now-up-to-30s Retry-After wait), do returns
// ctx.Err() instead of the just-drained-and-closed resp/err pair. The
// previous behavior returned that resp as-is: drainAndClose (below) had
// already fully read and closed its body by that point, so a caller
// building an error from it (newProviderHTTPError, providers.go) read a
// closed body and silently produced an error with an EMPTY upstream
// body — the real failure (the wait was aborted, not that the upstream
// answered with nothing) was lost. ctx.Err() is always non-nil here:
// waitFn returning false is p.waitFn's own documented contract for "ctx
// ended the wait early" (defaultWaitFn's doc comment), so this never
// masks a wait that genuinely completed.
//
// Feature A (v0.22) attempt-accounting: every call() invocation — not just
// the final one — is reported to ctx's attemptRecorder, if it carries one
// (attemptRecorderFromContext, providers.go), before the transient check
// above decides whether to retry. A request retried twice before
// succeeding therefore reports THREE attempts, not one — the caller
// (limiter.recordProviderAttempt, limits.go) classifies each with the
// identical isTransient(resp, err) this loop already uses, so "attempt"
// and "failure" here always agree with what actually got retried.
func (p *retryPolicy) do(ctx context.Context, call func() (*http.Response, error)) (*http.Response, error) {
	tries := p.totalTries()
	rec := attemptRecorderFromContext(ctx)

	var resp *http.Response
	var err error
	for attempt := 1; attempt <= tries; attempt++ {
		resp, err = call()
		if rec != nil {
			rec(resp, err)
		}
		if !isTransient(resp, err) || attempt == tries {
			return resp, err
		}

		// Finding 4 fix (review-routes.md): jitter is applied ONLY to the
		// exponential-backoff branch (fromRetryAfter == false), never to a
		// wait the upstream itself requested via Retry-After — see
		// retryJitterFn's own doc comment for why jittering that value
		// would be backwards under overload.
		wait, fromRetryAfter := p.waitBefore(attempt, resp)
		if !fromRetryAfter {
			wait = retryJitterFn(wait)
		}
		drainAndClose(resp)
		if !p.waitFn(ctx, wait) {
			if ctxErr := ctx.Err(); ctxErr != nil {
				return nil, ctxErr
			}
			return resp, err
		}
	}
	return resp, err
}