perf(openai): presize upstream JSON response reads from Content-Length

forwardJSON read upstream response bodies with io.ReadAll, which grows
its buffer many times. readAllLimited returns the same data and errors,
but it uses the declared Content-Length as a capacity hint. A body of
known length then usually needs one allocation.

A wrong Content-Length still reads to EOF or to the limit. The hint is
clamped to 1 MiB, so a peer that declares a large length and then stalls
cannot make the gateway allocate more than that before data arrives. The
client request path keeps io.ReadAll, because the client controls its
own Content-Length.

Tests compare readAllLimited with io.ReadAll for exact, short, long,
unknown and oversized lengths, for a mid-body error and for the clamp.

With both commits, the yaegi harness measures 729565 bytes per embeddings
request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms
instead of 1.31 ms. With 16 embeddings per request, allocation falls from
5.0 MB to 3.0 MB.

Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
This commit is contained in:
2026-09-17 15:27:00 +01:00
parent 5f791d3700
commit 56e112a3ed
3 changed files with 145 additions and 1 deletions
+29
View File
@@ -80,6 +80,35 @@ func newProviderHTTPError(resp *http.Response) *providerHTTPError {
return &providerHTTPError{status: resp.StatusCode, body: body}
}
// readAllHintMax caps how much readAllLimited allocates up front from a
// declared Content-Length. A peer that declares a large length and then
// stalls must not make the gateway hold more than this before any byte
// arrives; a longer body still reads correctly, it just regrows past it.
const readAllHintMax = 1 << 20
// readAllLimited returns the same result as io.ReadAll(io.LimitReader(r,
// limit)); contentLength is only a capacity hint — a wrong value still
// reads to EOF or limit, never trusted for correctness. Used for upstream
// response bodies (forwardJSON), where the sender is the operator-
// configured provider, not an arbitrary client. When it looks usable, it
// lets a body of known length be read into a single allocation instead of
// io.ReadAll's repeated doubling regrowth, but the hint itself is clamped
// to readAllHintMax before allocating — the added bytes.MinRead keeps
// ReadFrom's final probe read (which always tries to read past what it
// expects) from itself forcing one more regrow.
func readAllLimited(r io.Reader, contentLength, limit int64) ([]byte, error) {
if contentLength <= 0 || contentLength > limit {
return io.ReadAll(io.LimitReader(r, limit))
}
hint := contentLength
if hint > readAllHintMax {
hint = readAllHintMax
}
buf := bytes.NewBuffer(make([]byte, 0, hint+bytes.MinRead))
_, err := buf.ReadFrom(io.LimitReader(r, limit))
return buf.Bytes(), err
}
// errUpstream is a sentinel wrapped into errors an adapter returns when it
// cannot reach, or cannot make sense of a response from, its upstream —
// distinct from providerHTTPError, which means the upstream answered, just