mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-20 05:56:56 +00:00
fix(accounting): bound upstream-controlled usage and counter keys
Passthrough accounting read the model id, the usage figures and the response size from the upstream response without limits. - Mint a model counter scope only for ids the provider serves. An upstream that echoed new ids created unbounded counter keys. - Drop a reported token or cost figure above a fixed ceiling. One huge figure locked a user out of every budget for the whole window. - Bill an echoed unpriced model at the price of the model that passed the cost-budget gate. - Cap a body-size prompt estimate at one million tokens. - Bound the total bytes that concurrent passthrough accounting tees hold. A tee that cannot reserve room falls back to the estimate. - Add cache.accountHits. When true, a cache hit charges the cached usage to the user. The default stays false.
This commit is contained in:
@@ -1555,6 +1555,13 @@ func (m *modelRegistry) bareWinner(id string, grp *group, extraIDs ...string) (s
|
||||
return visible[0], true
|
||||
}
|
||||
|
||||
// providerHasModel reports whether provider's explicit or discovered model
|
||||
// set contains id.
|
||||
func (m *modelRegistry) providerHasModel(provider, id string) bool {
|
||||
st, ok := m.states[provider]
|
||||
return ok && st.hasModel(id)
|
||||
}
|
||||
|
||||
// resolve maps a client-requested model id to the adapter that serves it.
|
||||
// It returns the adapter, the upstream model id to send that adapter (the
|
||||
// id form the provider itself knows, with any gateway-side provider prefix
|
||||
|
||||
Reference in New Issue
Block a user