mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-21 05:57:53 +00:00
perf(limits): fuse admission reads into one round trip
checkAndCount already batched its request-counter increments into one pipeline. evaluateScope then read every scope's token and cost budgets through up to four further serial round trips per scope. A request with a fully-limited user and group scope paid nine round trips before the upstream call started. This change fuses every budget read into the SAME pipeline as the increments, through a new incrAndGetMulti method on counterStore. The reads are safe to fuse because they touch disjoint keys (req:* versus tokin:*/tokout:*/cost:*), and tokens and cost are only ever written by account() after the response completes, never by checkAndCount itself. Violation precedence stays identical to the pre-fusion implementation: a user's own violation still wins over their group's, and within one scope the order stays requests-per-minute, requests-per-day, tokens-per-day, tokens-per-month, cost-per-day, cost-per-month. New table-driven tests pin this precedence directly, engineering two simultaneous violation conditions per case and asserting only the higher-precedence one is reported. evaluateScope, budgetViolation, and tokenBudgetViolation are removed; buildBudgetProbes plus a loop inline the same logic into checkAndCount, which is what lets their reads share the fused round trip. Measured on the interpreted Yaegi harness (yaegi v0.16.1, the runtime this plugin actually ships under): RTT/req for a fully-limited chat request drops from 10.99 to 3.00, with cmds/req unchanged at 74 (this is a round-trip fix, not a command-count or allocation change). Single-request throughput rises from 112 to 335 req/s; at concurrency 16, from 121 to 380 req/s. counterStore.incrAndGetMulti is a required interface method, not an optional capability behind a type assertion: Yaegi can silently return ok=false when a comma-ok assertion checks a compiled concrete value against a plugin-declared interface (Trap 4, documented in matchesSentinel's own doc comment). A plain interface method call carries no such risk and is the same mechanism incrMulti already uses in production.
This commit is contained in:
@@ -87,6 +87,10 @@ func (alwaysErrStore) incrMulti([]counterIncr) ([]int64, error) {
|
||||
return nil, errStoreDownStub
|
||||
}
|
||||
|
||||
func (alwaysErrStore) incrAndGetMulti([]counterIncr, []string) ([]int64, []int64, error) {
|
||||
return nil, nil, errStoreDownStub
|
||||
}
|
||||
|
||||
// TestHandleChat_HappyPath_NonStreaming_AccountsUsage drives a full
|
||||
// chat-completion request through Gateway.ServeHTTP against a fake
|
||||
// openai-shaped upstream, and asserts the response is forwarded verbatim
|
||||
@@ -1755,7 +1759,7 @@ func TestCacheCaptureWriter_ImplicitStatus_CapturesContentType(t *testing.T) {
|
||||
// builds a user AND a group limitScope unconditionally (v0.21 accounting
|
||||
// fix): whether either entity has Limits configured or not, both scopes
|
||||
// must always be present — a nil Limits field turns off ENFORCEMENT
|
||||
// (evaluateScope's own nil check, limits.go) for that scope, never
|
||||
// (checkAndCount's own nil check, limits.go) for that scope, never
|
||||
// accounting. Table-driven over every nil/non-nil combination of user and
|
||||
// group limits.
|
||||
func TestBuildLimitScopes_AlwaysBuildsBothScopes(t *testing.T) {
|
||||
|
||||
Reference in New Issue
Block a user