mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-20 05:56:56 +00:00
feat(metrics): expose token-accounting provenance as a metric dimension
Token accounting has three provenances today: reported (real provider usage), estimated (a non-streaming response with no usage, prompt substituted from request body size), and unbilled (a streaming response with no usage, request counted but zero tokens billed). Nothing on any read surface distinguished them before this change. Add a per-process provenanceStore (metrics.go), recorded in runUnified right after the existing estimated/unbilled classification (routes_unified.go), for a candidate whose response actually completed. A failed-over candidate is never recorded, so ordinary failover noise never pollutes the unbilled signal. Expose two new counter families on GET /metrics: llmgateway_usage_provenance_requests_total and llmgateway_usage_provenance_tokens_total, both labeled by provider and provenance. Together they answer the three operator questions this change targets: what fraction of billed tokens came from real reported usage, which providers produce estimated or unbilled accounting, and how often the unbilled case happens at all. Both families are in-process only, so sum() is the correct cross-replica aggregation, matching this file's other per-process families. Billing itself is unchanged: this only records which provenance an already-billed number took, never a new number. Tests cover reported/estimated/unbilled classification, that billing matches the pre-existing contract exactly, and that estimated and unbilled never collapse into one series. Each behavioral assertion was verified against a real mutation of the source before being kept.
This commit is contained in:
@@ -555,6 +555,34 @@ func (g *Gateway) runMeteredCall(sw *statusTrackingWriter, r *http.Request, scop
|
||||
g.logf("%s: usage for model %q logged as estimated (%d prompt tokens derived from request body size, not the provider's reported usage)", logPrefix, cand.canonical, result.prompt)
|
||||
}
|
||||
|
||||
// Usage provenance (feat: expose token-accounting provenance),
|
||||
// recorded ONLY for a candidate whose own response actually
|
||||
// completed (callErr == nil): a candidate that failed over never
|
||||
// bills more than a zero-total no-op (this function's own doc
|
||||
// comment above), and counting THAT as "unbilled" would drown the
|
||||
// one signal this feature exists to surface — a provider silently
|
||||
// serving completions free against every budget — in ordinary
|
||||
// failover noise. "unbilled" is re-derived from streaming &&
|
||||
// result.total() == 0 rather than a separate flag carried out of
|
||||
// the block above: result.prompt/result.estimated are only ever
|
||||
// mutated by that block's non-streaming branch, so this condition
|
||||
// is true here if and only if the streaming branch above ran.
|
||||
// Gated on metricsEnabled exactly like withLatencyRecorder's own
|
||||
// wiring above, so a deployment with metrics off pays nothing
|
||||
// extra here. Observation only: nothing recorded here feeds back
|
||||
// into result or the account call above, both of which already
|
||||
// ran — see g.recordUsageProvenance's own doc comment (metrics.go).
|
||||
if callErr == nil && metricsEnabled(g.cfg) {
|
||||
provenance := provenanceReported
|
||||
switch {
|
||||
case result.estimated:
|
||||
provenance = provenanceEstimated
|
||||
case streaming && result.total() == 0:
|
||||
provenance = provenanceUnbilled
|
||||
}
|
||||
g.recordUsageProvenance(providerName, provenance, result.total())
|
||||
}
|
||||
|
||||
// A miss stores the response after everything above has already
|
||||
// run — accounting must never be skipped or delayed waiting on a
|
||||
// cache write. Only a genuine upstream 200 is stored (spec §2's
|
||||
|
||||
Reference in New Issue
Block a user