mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-18 05:47:38 +00:00
fix(limits): cut body-admission bounds to [8,64] on measured heap cost
Round 3 sized these from the isolated json.Unmarshal peak (~51.5MB). Measured end-to-end through the real handler it is ~80-108MB per slot, so a 256 ceiling allows a 21GB peak under adversarial load — above any realistic pod budget, and the reference deployment limits Traefik to 512Mi. The slot is now held only across read+decode, so a smaller bound costs legitimate throughput almost nothing.
This commit is contained in:
+33
-2
@@ -367,10 +367,41 @@ func CreateConfig() *Config {
|
|||||||
// worst-case LIVE-heap arithmetic above still needs to stay defensible
|
// worst-case LIVE-heap arithmetic above still needs to stay defensible
|
||||||
// on a modest node.
|
// on a modest node.
|
||||||
const (
|
const (
|
||||||
minBodyAdmissionCap = 32
|
minBodyAdmissionCap = 8
|
||||||
maxBodyAdmissionCap = 256
|
maxBodyAdmissionCap = 64
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// NOTE on these bounds (round-4 correction, measured): review round 3
|
||||||
|
// shipped [32, 256], reasoning from the isolated json.Unmarshal peak
|
||||||
|
// (~51.5MB for an adversarial 4MiB body). The verifier then measured the
|
||||||
|
// END-TO-END per-slot cost through the real handler — ~80-108MB, since
|
||||||
|
// the raw body []byte stays live alongside the decoded map and GC lags —
|
||||||
|
// and swept the cap against peak HeapInuse under 400 concurrent
|
||||||
|
// adversarial requests:
|
||||||
|
//
|
||||||
|
// cap 16 -> 1.5GB cap 64 -> 6.1GB
|
||||||
|
// cap 32 -> 3.4GB cap 256 -> 21.1GB
|
||||||
|
//
|
||||||
|
// A 256 ceiling therefore does not prevent the OOM this semaphore exists
|
||||||
|
// to prevent: it is above any realistic pod budget. It is still strictly
|
||||||
|
// better than the unbounded base (25GB on the same burst), but "better
|
||||||
|
// than unbounded" was never the goal. The reference deployment runs
|
||||||
|
// Traefik with resources.limits.memory: 512Mi, where even the 8-slot
|
||||||
|
// floor is the binding constraint, not the ceiling.
|
||||||
|
//
|
||||||
|
// So the bounds are cut to [8, 64]: 64 keeps a large-memory host from
|
||||||
|
// being throttled below what it can actually serve, 8 keeps a small pod
|
||||||
|
// from admitting more concurrent decodes than its heap can hold. This
|
||||||
|
// costs legitimate throughput almost nothing because the slot is now
|
||||||
|
// held ONLY across read+decode (round-3 fix) — microseconds to low
|
||||||
|
// milliseconds — not across the upstream call; an operator whose host
|
||||||
|
// genuinely wants more sets Config.MaxInFlightBodyRequests explicitly,
|
||||||
|
// which always wins. Deriving the ceiling from the process's real memory
|
||||||
|
// budget (runtime/debug.SetMemoryLimit(-1), which the chart already sets
|
||||||
|
// via goMemLimitPercentage) is the better long-term shape and is left as
|
||||||
|
// a follow-up: it needs its own Yaegi-interpreted verification, since
|
||||||
|
// runtime/debug's availability under the interpreter is unproven here.
|
||||||
|
|
||||||
// bodyAdmissionCapPerCPU is defaultBodyAdmissionCap's GOMAXPROCS
|
// bodyAdmissionCapPerCPU is defaultBodyAdmissionCap's GOMAXPROCS
|
||||||
// multiplier (revised round 3 — see minBodyAdmissionCap's own doc
|
// multiplier (revised round 3 — see minBodyAdmissionCap's own doc
|
||||||
// comment for the full before/after reasoning): a decode-scoped slot is
|
// comment for the full before/after reasoning): a decode-scoped slot is
|
||||||
|
|||||||
Reference in New Issue
Block a user