mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-21 05:57:53 +00:00
fix(accounting): bound upstream-controlled usage and counter keys
Passthrough accounting read the model id, the usage figures and the response size from the upstream response without limits. - Mint a model counter scope only for ids the provider serves. An upstream that echoed new ids created unbounded counter keys. - Drop a reported token or cost figure above a fixed ceiling. One huge figure locked a user out of every budget for the whole window. - Bill an echoed unpriced model at the price of the model that passed the cost-budget gate. - Cap a body-size prompt estimate at one million tokens. - Bound the total bytes that concurrent passthrough accounting tees hold. A tee that cannot reserve room falls back to the estimate. - Add cache.accountHits. When true, a cache hit charges the cached usage to the user. The default stays false.
This commit is contained in:
@@ -3355,7 +3355,7 @@ func TestHandlePassthrough_ModelLatencyAndUModelSuppressed_ProviderHistogramsOnl
|
||||
|
||||
cfg := CreateConfig()
|
||||
cfg.Providers = map[string]*ProviderConfig{
|
||||
"openai": {Type: "openai", BaseURL: srv.URL, APIKey: "sk-up"},
|
||||
"openai": {Type: "openai", BaseURL: srv.URL, APIKey: "sk-up", Models: []string{"gpt-native-x"}},
|
||||
}
|
||||
cfg.Groups = map[string]*GroupConfig{"default": {}}
|
||||
cfg.Users = &UsersConfig{Inline: []*UserConfig{
|
||||
|
||||
Reference in New Issue
Block a user