mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
perf(openai): presize upstream JSON response reads from Content-Length
forwardJSON read upstream response bodies with io.ReadAll, which grows its buffer many times. readAllLimited returns the same data and errors, but it uses the declared Content-Length as a capacity hint. A body of known length then usually needs one allocation. A wrong Content-Length still reads to EOF or to the limit. The hint is clamped to 1 MiB, so a peer that declares a large length and then stalls cannot make the gateway allocate more than that before data arrives. The client request path keeps io.ReadAll, because the client controls its own Content-Length. Tests compare readAllLimited with io.ReadAll for exact, short, long, unknown and oversized lengths, for a mid-body error and for the clamp. With both commits, the yaegi harness measures 729565 bytes per embeddings request instead of 925587 (-21%), 12.6% fewer allocations, and 1.15 ms instead of 1.31 ms. With 16 embeddings per request, allocation falls from 5.0 MB to 3.0 MB. Claude-Session: https://claude.ai/code/session_01KwtV5ecajCrftcf8TkfPad
This commit is contained in:
+1
-1
@@ -214,7 +214,7 @@ type chatUsagePayload struct {
|
||||
// since a malformed-but-readable body is the upstream's problem to answer
|
||||
// for, not a reason to swallow the response.
|
||||
func (a *openaiAdapter) forwardJSON(w http.ResponseWriter, resp *http.Response) (usage, error) {
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes))
|
||||
body, err := readAllLimited(resp.Body, resp.ContentLength, maxResponseBytes)
|
||||
if err != nil {
|
||||
return usage{}, fmt.Errorf("%w: read response body: %w", errUpstream, err)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user